diff --git a/docs/INFRASTRUCTURE.md b/docs/INFRASTRUCTURE.md new file mode 100644 index 0000000..76e7e79 --- /dev/null +++ b/docs/INFRASTRUCTURE.md @@ -0,0 +1,230 @@ +# Cariflex EMS - Infrastructure Documentation + +## Architecture Overview + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ CARIFLEX EMS │ +│ Energy Management System for Martinique │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ +│ │ OpenADR │ │ FlexMeasures│ │ Grafana │ │ +│ │ VTN/VEN │───▶│ Server │───▶│ Dashboard │ │ +│ │ (S2+OpenADR)│ │ (EMS Core) │ │ (14 panels)│ │ +│ └──────────────┘ └──────────────┘ └──────────────┘ │ +│ │ │ │ │ +│ ▼ ▼ ▼ │ +│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ +│ │ PostgreSQL │ │ Redis │ │ Traefik │ │ +│ │ (time-series)│ │ (job queue) │ │ (reverse │ │ +│ │ │ │ │ │ proxy) │ │ +│ └──────────────┘ └──────────────┘ └──────────────┘ │ +│ │ +└─────────────────────────────────────────────────────────────────────────────┘ +``` + +## Network Architecture + +### Networks +| Network | Subnet | Purpose | External | +|---------|--------|---------|----------| +| `traefik-public` | 172.29.0.0/16 | Public services (FM, Grafana) | Yes | +| `openadr-internal` | 192.168.240.0/24 | OpenADR VTN/VEN communication | No | + +### DNS/Hosts +| Service | Domain | Container | Port | +|---------|--------|-----------|------| +| FlexMeasures | cariflex.digitribe.fr | flexmeasures-server | 5000 | +| Grafana | grafana.digitribe.fr | smart-city-grafana | 3000 | +| OpenADR VTN | - | openadr-vtn | 8080 | +| OpenADR VEN | - | openadr-ven | - | + +### Firewall (UFW) +- Public ports: 80 (HTTP), 443 (HTTPS), 51820 (WireGuard) +- Internal networks: 172.16.0.0/12, 10.0.0.0/8, 192.168.0.0/16, 192.168.240.0/24 + +## Docker Containers + +### Cariflex EMS Stack +| Container | Image | Network | IP | Port | Status | +|-----------|-------|---------|----|---- |--------| +| flexmeasures-server | lfenergy/flexmeasures:latest | traefik-public | 172.29.0.x | 5000 | ✅ UP | +| flexmeasures-worker | lfenergy/flexmeasures:latest | traefik-public | 172.29.0.x | - | ✅ UP | +| flexmeasures-db | postgres:17 | traefik-public | 172.29.0.x | 5432 | ✅ UP | +| flexmeasures-redis | redis:7-alpine | traefik-public | 172.29.0.x | 6379 | ✅ UP | +| openadr-vtn | flexmeasures-openadr-vtn | openadr-internal | 192.168.240.10 | 8080 | ✅ UP | +| openadr-ven | flexmeasures-openadr-ven | openadr-internal | 192.168.240.11 | - | ✅ UP | +| smart-city-grafana | grafana/grafana | traefik-public | - | 3000 | ✅ UP | + +## IP Address Allocation (phpIPAM) + +### Cariflex Subnet: 192.168.240.0/24 +| IP | Service | Container | MAC | +|----|---------|-----------|-----| +| 192.168.240.1 | Gateway | - | - | +| 192.168.240.10 | OpenADR VTN | openadr-vtn | - | +| 192.168.240.11 | OpenADR VEN | openadr-ven | - | +| 192.168.240.128-254 | Reserved | - | - | + +### Public Subnet (traefik-public): 172.29.0.0/16 +| IP | Service | Domain | +|----|---------|--------| +| 172.29.0.x | FlexMeasures | cariflex.digitribe.fr | +| 172.29.0.x | Grafana | grafana.digitribe.fr | + +## Sensor Mapping + +### FlexMeasures Sensors +| ID | Name | Unit | Type | Source | +|----|------|------|------|--------| +| 41-50 | pv_01..10_power | kW | PV Production | Simulator | +| 51-60 | bat_01..10_power | kWh | Battery SOC | Simulator | +| 61-70 | ev_01..10_power | kWh | EV Charge | Simulator | +| 81 | irradiance | W/m² | Weather | Open-Meteo | +| 82 | temperature | °C | Weather | Open-Meteo | +| 83 | wind_speed | m/s | Weather | Open-Meteo | +| 84 | consumption_price | EUR/MWh | Price | OpenADR VTN | +| 85 | production_price | EUR/MWh | Price | Simulated | +| 86 | load_control_signal | % | DSR | OpenADR VTN | +| 87 | demand_response_signal | dimensionless | DSR | OpenADR VTN | + +### S2 Resource Mapping +| S2 Resource | FM Sensor | Asset | Min Power | Max Power | +|-------------|-----------|-------|-----------|-----------| +| battery_01-05 | 51-55 | Bat_01-05 | -50 kW | +50 kW | +| ev_01-05 | 61-65 | EV_01-05 | -22 kW | +22 kW | + +## OpenADR Protocol + +### VTN (Virtual Top Node) +- **Container**: openadr-vtn +- **Port**: 8080 (internal) +- **Profile**: OpenADR 2.0b +- **Events**: ENERGY_PRICE (24h), LOAD_CONTROL (4h) +- **Schedule**: Every 10 seconds poll + +### VEN (Virtual End Node) +- **Container**: openadr-ven +- **VTN URL**: http://192.168.240.10:8080/OpenADR2/Simple/2.0b +- **Registration ID**: reg-Cariflex-VEN-001 +- **Event Handler**: on_event -> FM API (sensors 84, 86) + +### Event Flow +1. VTN creates events (price, load control) +2. VEN polls VTN every 10s +3. VEN receives oadrDistributeEvent +4. VEN sends event responses (optIn) +5. VEN posts data to FM API +6. Grafana displays real-time data + +## S2 Protocol Integration + +### S2 Service +- **Location**: openadr-ven container (background process) +- **Protocol**: S2 (FRBC - Flexibility Resource Bank Control) +- **Mapping**: S2 resources -> FM sensors (51-65) +- **Functions**: + - process_load_control(): Converts OpenADR load control to S2 FRBC instructions + - process_demand_response(): Converts OpenADR demand response to S2 FRBC instructions + +### S2 Message Types +- FRBCInstruction: Power setpoint commands +- FRBCSystemDescription: Resource capabilities +- FRBCActuatorStatus: Resource status feedback +- FRBCStorageStatus: Battery/EV storage status + +## Backup Strategy + +### Automated Backups +- **Script**: `/home/eric/cariflex/scripts/full_backup.sh` +- **Location**: `/home/eric/backups/cariflex/YYYYMMDD_HHMMSS.tar.gz` +- **Contents**: + - PostgreSQL dump (flexmeasures DB) + - Redis dump (RDB file) + - FM config and templates + - OpenADR scripts and Dockerfiles + - Grafana dashboards (JSON) + - Docker container states + - Network configuration + +### Backup Retention +- Daily backups kept for 30 days +- Weekly backups kept for 12 weeks +- Monthly backups kept indefinitely + +## Deployment Environments + +### Development +- **Host**: Local VM or developer machine +- **Docker Compose**: Single node +- **Data**: Simulated +- **URL**: localhost + +### Test +- **Host**: Test server +- **Docker Compose**: Single node +- **Data**: Anonymized production data +- **URL**: test.cariflex.digitribe.fr + +### Production +- **Host**: Production server (this deployment) +- **Docker Compose**: Single node (scalable to Swarm/K8s) +- **Data**: Real-time production data +- **URL**: cariflex.digitribe.fr + +## Security + +### Credentials +| Service | Username | Password | Location | +|---------|----------|----------|----------| +| FlexMeasures | admin@digitribe.fr | Digitribe972 | FM DB | +| PostgreSQL | flexmeasures | Digitribe972 | PostgreSQL | +| Redis | - | (disabled) | Redis | +| Grafana | admin | admin | Grafana DB | + +### SSL/TLS +- All public services use Let's Encrypt certificates +- Traefik handles SSL termination +- Internal services use HTTP (isolated network) + +## Monitoring + +### Health Checks +- FlexMeasures: https://cariflex.digitribe.fr/health +- Grafana: https://grafana.digitrobe.fr/api/health +- OpenADR VTN: http://192.168.240.10:8080/OpenADR2/Simple/2.0b + +### Logs +- Docker: `docker logs ` +- FM: `/app/flexmeasures/logs/` +- Grafana: `/var/log/grafana/` + +## Troubleshooting + +### Common Issues +1. **Redis ACL error**: `docker exec flexmeasures-redis redis-cli CONFIG SET requirepass ""` +2. **OpenADR connection refused**: Check UFW rules for 192.168.240.0/24 +3. **Grafana no data**: Check datasource UID matches PostgreSQL DS +4. **FM worker not running**: `docker restart flexmeasures-worker` + +### Useful Commands +```bash +# View all containers +docker ps -a | grep -E "flexmeasures|openadr|grafana" + +# Check OpenADR communication +docker logs openadr-ven | grep -i "price\|load\|event" + +# Check FM API +curl -s https://cariflex.digitribe.fr/api/v3_0/sensors + +# Database query +docker exec flexmeasures-db psql -U flexmeasures -d flexmeasures -c "SELECT * FROM sensor WHERE id BETWEEN 80 AND 90;" +``` + +## References +- FlexMeasures: https://flexmeasures.readthedocs.io/ +- OpenLEADR: https://openleadr.org/ +- S2 Protocol: https://s2-standard.github.io/ +- Grafana: https://grafana.com/docs/ diff --git a/iac/ansible/playbook.yml b/iac/ansible/playbook.yml new file mode 100644 index 0000000..420ad8a --- /dev/null +++ b/iac/ansible/playbook.yml @@ -0,0 +1,59 @@ +# Cariflex EMS - Ansible Playbook +# Deploys the full Cariflex stack to a target host + +--- +- name: Deploy Cariflex EMS + hosts: cariflex + become: yes + vars: + cariflex_version: "{{ lookup('env', 'CARIFLEX_VERSION') | default('latest', true) }}" + deploy_env: "{{ lookup('env', 'DEPLOY_ENV') | default('production', true) }}" + + pre_tasks: + - name: Check minimum requirements + assert: + that: + - ansible_memtotal_mb >= 4096 + - ansible_processor_vcpus >= 2 + fail_msg: "Host does not meet minimum requirements (4GB RAM, 2 CPUs)" + + roles: + - role: common + tags: [common] + - role: docker + tags: [docker] + - role: traefik + tags: [traefik] + - role: postgresql + tags: [database] + - role: redis + tags: [cache] + - role: flexmeasures + tags: [app] + - role: openadr + tags: [openadr] + - role: grafana + tags: [monitoring] + - role: backup + tags: [backup] + + post_tasks: + - name: Verify deployment + uri: + url: "https://{{ cariflex_domain }}/api/v3_0/sensors" + status_code: 200 + register: health_check + retries: 10 + delay: 5 + until: health_check.status == 200 + + - name: Display deployment info + debug: + msg: | + ========================================== + Cariflex EMS Deployed Successfully! + Environment: {{ deploy_env }} + URL: https://{{ cariflex_domain }} + Grafana: https://{{ grafana_domain }} + Version: {{ cariflex_version }} + ========================================== diff --git a/iac/environments/README.md b/iac/environments/README.md new file mode 100644 index 0000000..3b0d179 --- /dev/null +++ b/iac/environments/README.md @@ -0,0 +1,85 @@ +# Cariflex EMS - Environment Configuration + +## Development Environment + +```bash +# .env.dev +DEPLOY_ENV=dev +CARIFLEX_VERSION=dev +CARIFLEX_DOMAIN=cariflex.dev.local +GRAFANA_DOMAIN=grafana.dev.local +FM_SECRET_KEY=dev-secret-key-change-me +FM_DB_PASSWORD=devpassword +REDIS_PASSWORD=devpassword +``` + +### Deploy to Dev +```bash +cd /home/eric/cariflex/iac +ansible-playbook playbook.yml -i environments/dev/inventory --extra-vars "deploy_env=dev" +``` + +## Test Environment + +```bash +# .env.test +DEPLOY_ENV=test +CARIFLEX_VERSION=test +CARIFLEX_DOMAIN=cariflex.test.digitribe.fr +GRAFANA_DOMAIN=grafana.test.digitribe.fr +FM_SECRET_KEY={{ vault_fm_secret_key }} +FM_DB_PASSWORD={{ vault_fm_db_password }} +REDIS_PASSWORD={{ vault_redis_password }} +``` + +### Deploy to Test +```bash +cd /home/eric/cariflex/iac +ansible-playbook playbook.yml -i environments/test/inventory --extra-vars "deploy_env=test" --ask-vault-pass +``` + +## Production Environment + +```bash +# .env.prod +DEPLOY_ENV=production +CARIFLEX_VERSION=latest +CARIFLEX_DOMAIN=cariflex.digitribe.fr +GRAFANA_DOMAIN=grafana.digitribe.fr +FM_SECRET_KEY={{ vault_fm_secret_key }} +FM_DB_PASSWORD={{ vault_fm_db_password }} +REDIS_PASSWORD={{ vault_redis_password }} +``` + +### Deploy to Production +```bash +cd /home/eric/cariflex/iac +ansible-playbook playbook.yml -i environments/prod/inventory --extra-vars "deploy_env=production" --ask-vault-pass +``` + +## Docker Compose Override Files + +Each environment has its own override file: +- `docker-compose.dev.yml` - Development (debug mode, local volumes) +- `docker-compose.test.yml` - Testing (test data, isolated network) +- `docker-compose.prod.yml` - Production (optimized, SSL, backups) + +## Kubernetes Deployment + +For K8s deployment, see `k8s/` directory: +- `k8s/namespace.yaml` - Namespace creation +- `k8s/configmap.yaml` - Configuration +- `k8s/secrets.yaml` - Secrets (encrypted) +- `k8s/deployments/` - Application deployments +- `k8s/services/` - Service definitions +- `k8s/ingress/` - Ingress rules + +### Deploy to K8s +```bash +kubectl apply -f k8s/namespace.yaml +kubectl apply -f k8s/configmap.yaml +kubectl apply -f k8s/secrets.yaml +kubectl apply -f k8s/deployments/ +kubectl apply -f k8s/services/ +kubectl apply -f k8s/ingress/ +``` diff --git a/scripts/full_backup.sh b/scripts/full_backup.sh new file mode 100755 index 0000000..d3d296c --- /dev/null +++ b/scripts/full_backup.sh @@ -0,0 +1,56 @@ +#!/bin/bash +# Cariflex Full Backup Script +BACKUP_DIR="/home/eric/backups/cariflex/$(date +%Y%m%d_%H%M%S)" +mkdir -p "$BACKUP_DIR" + +echo "=== Cariflex Full Backup $(date) ===" + +# 1. PostgreSQL dump +echo "Backing up PostgreSQL..." +docker exec flexmeasures-db pg_dump -U flexmeasures flexmeasures | gzip > "$BACKUP_DIR/flexmeasures_db.sql.gz" + +# 2. Redis dump +echo "Backing up Redis..." +docker exec flexmeasures-redis redis-cli BGSAVE +docker cp flexmeasures-redis:/data/dump.rdb "$BACKUP_DIR/redis_dump.rdb" + +# 3. FM config and templates +echo "Backing up FM config..." +cp -r /home/eric/cariflex/templates "$BACKUP_DIR/" +cp -r /home/eric/cariflex/config "$BACKUP_DIR/" 2>/dev/null || true +cp /home/eric/flexmeasures/docker-compose.yml "$BACKUP_DIR/" +cp /home/eric/flexmeasures/docker-compose.openadr.yml "$BACKUP_DIR/" + +# 4. OpenADR scripts +echo "Backing up OpenADR..." +cp -r /home/eric/flexmeasures/*.py "$BACKUP_DIR/" +cp /home/eric/flexmeasures/Dockerfile.openadr "$BACKUP_DIR/" +cp /home/eric/flexmeasures/start_ven.sh "$BACKUP_DIR/" + +# 5. Grafana dashboards +echo "Backing up Grafana dashboards..." +curl -s -u admin:admin "http://localhost:3001/api/dashboards/uid/cariflex-ems-full" | python3 -c " +import json, sys +d = json.load(sys.stdin) +with open('$BACKUP_DIR/grafana_dashboard.json', 'w') as f: + json.dump(d, f, indent=2) +print('Dashboard saved') +" + +# 6. Docker container states +echo "Backing up container states..." +docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}" > "$BACKUP_DIR/containers.txt" +docker inspect openadr-vtn openadr-ven flexmeasures-server flexmeasures-db flexmeasures-redis 2>/dev/null > "$BACKUP_DIR/container_inspects.json" + +# 7. Network config +echo "Backing up network config..." +docker network inspect openadr-internal 2>/dev/null > "$BACKUP_DIR/network_openadr.json" +docker network inspect traefik-public 2>/dev/null > "$BACKUP_DIR/network_traefik.json" + +# 8. Compress +echo "Compressing..." +tar -czf "$BACKUP_DIR.tar.gz" -C "$(dirname $BACKUP_DIR)" "$(basename $BACKUP_DIR)" +rm -rf "$BACKUP_DIR" + +echo "=== Backup complete: $BACKUP_DIR.tar.gz ===" +ls -lh "$BACKUP_DIR.tar.gz"