Infrastructure docs, IaC scripts, skills energy (VOLTTRON, EnergyPlus, OpenStudio, BuildingGym, OSeMOSYS)

2026-06-10 12:09:00 -04:00
parent a0577ed17c
commit 84667c7126
4 changed files with 430 additions and 0 deletions
--- a/docs/INFRASTRUCTURE.md
+++ b/docs/INFRASTRUCTURE.md
@@ -0,0 +1,230 @@
+# Cariflex EMS - Infrastructure Documentation
+
+## Architecture Overview
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                              CARIFLEX EMS                                    │
+│                    Energy Management System for Martinique                   │
+├─────────────────────────────────────────────────────────────────────────────┤
+│                                                                              │
+│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐                   │
+│  │   OpenADR    │    │   FlexMeasures│    │   Grafana    │                   │
+│  │   VTN/VEN    │───▶│   Server      │───▶│   Dashboard  │                   │
+│  │   (S2+OpenADR)│    │   (EMS Core)  │    │   (14 panels)│                   │
+│  └──────────────┘    └──────────────┘    └──────────────┘                   │
+│         │                    │                    │                          │
+│         ▼                    ▼                    ▼                          │
+│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐                   │
+│  │ PostgreSQL   │    │    Redis      │    │   Traefik    │                   │
+│  │ (time-series)│    │  (job queue)  │    │  (reverse    │                   │
+│  │              │    │              │    │   proxy)      │                   │
+│  └──────────────┘    └──────────────┘    └──────────────┘                   │
+│                                                                              │
+└─────────────────────────────────────────────────────────────────────────────┘
+```
+
+## Network Architecture
+
+### Networks
+| Network | Subnet | Purpose | External |
+|---------|--------|---------|----------|
+| `traefik-public` | 172.29.0.0/16 | Public services (FM, Grafana) | Yes |
+| `openadr-internal` | 192.168.240.0/24 | OpenADR VTN/VEN communication | No |
+
+### DNS/Hosts
+| Service | Domain | Container | Port |
+|---------|--------|-----------|------|
+| FlexMeasures | cariflex.digitribe.fr | flexmeasures-server | 5000 |
+| Grafana | grafana.digitribe.fr | smart-city-grafana | 3000 |
+| OpenADR VTN | - | openadr-vtn | 8080 |
+| OpenADR VEN | - | openadr-ven | - |
+
+### Firewall (UFW)
+- Public ports: 80 (HTTP), 443 (HTTPS), 51820 (WireGuard)
+- Internal networks: 172.16.0.0/12, 10.0.0.0/8, 192.168.0.0/16, 192.168.240.0/24
+
+## Docker Containers
+
+### Cariflex EMS Stack
+| Container | Image | Network | IP | Port | Status |
+|-----------|-------|---------|----|---- |--------|
+| flexmeasures-server | lfenergy/flexmeasures:latest | traefik-public | 172.29.0.x | 5000 | ✅ UP |
+| flexmeasures-worker | lfenergy/flexmeasures:latest | traefik-public | 172.29.0.x | - | ✅ UP |
+| flexmeasures-db | postgres:17 | traefik-public | 172.29.0.x | 5432 | ✅ UP |
+| flexmeasures-redis | redis:7-alpine | traefik-public | 172.29.0.x | 6379 | ✅ UP |
+| openadr-vtn | flexmeasures-openadr-vtn | openadr-internal | 192.168.240.10 | 8080 | ✅ UP |
+| openadr-ven | flexmeasures-openadr-ven | openadr-internal | 192.168.240.11 | - | ✅ UP |
+| smart-city-grafana | grafana/grafana | traefik-public | - | 3000 | ✅ UP |
+
+## IP Address Allocation (phpIPAM)
+
+### Cariflex Subnet: 192.168.240.0/24
+| IP | Service | Container | MAC |
+|----|---------|-----------|-----|
+| 192.168.240.1 | Gateway | - | - |
+| 192.168.240.10 | OpenADR VTN | openadr-vtn | - |
+| 192.168.240.11 | OpenADR VEN | openadr-ven | - |
+| 192.168.240.128-254 | Reserved | - | - |
+
+### Public Subnet (traefik-public): 172.29.0.0/16
+| IP | Service | Domain |
+|----|---------|--------|
+| 172.29.0.x | FlexMeasures | cariflex.digitribe.fr |
+| 172.29.0.x | Grafana | grafana.digitribe.fr |
+
+## Sensor Mapping
+
+### FlexMeasures Sensors
+| ID | Name | Unit | Type | Source |
+|----|------|------|------|--------|
+| 41-50 | pv_01..10_power | kW | PV Production | Simulator |
+| 51-60 | bat_01..10_power | kWh | Battery SOC | Simulator |
+| 61-70 | ev_01..10_power | kWh | EV Charge | Simulator |
+| 81 | irradiance | W/m² | Weather | Open-Meteo |
+| 82 | temperature | °C | Weather | Open-Meteo |
+| 83 | wind_speed | m/s | Weather | Open-Meteo |
+| 84 | consumption_price | EUR/MWh | Price | OpenADR VTN |
+| 85 | production_price | EUR/MWh | Price | Simulated |
+| 86 | load_control_signal | % | DSR | OpenADR VTN |
+| 87 | demand_response_signal | dimensionless | DSR | OpenADR VTN |
+
+### S2 Resource Mapping
+| S2 Resource | FM Sensor | Asset | Min Power | Max Power |
+|-------------|-----------|-------|-----------|-----------|
+| battery_01-05 | 51-55 | Bat_01-05 | -50 kW | +50 kW |
+| ev_01-05 | 61-65 | EV_01-05 | -22 kW | +22 kW |
+
+## OpenADR Protocol
+
+### VTN (Virtual Top Node)
+- **Container**: openadr-vtn
+- **Port**: 8080 (internal)
+- **Profile**: OpenADR 2.0b
+- **Events**: ENERGY_PRICE (24h), LOAD_CONTROL (4h)
+- **Schedule**: Every 10 seconds poll
+
+### VEN (Virtual End Node)
+- **Container**: openadr-ven
+- **VTN URL**: http://192.168.240.10:8080/OpenADR2/Simple/2.0b
+- **Registration ID**: reg-Cariflex-VEN-001
+- **Event Handler**: on_event -> FM API (sensors 84, 86)
+
+### Event Flow
+1. VTN creates events (price, load control)
+2. VEN polls VTN every 10s
+3. VEN receives oadrDistributeEvent
+4. VEN sends event responses (optIn)
+5. VEN posts data to FM API
+6. Grafana displays real-time data
+
+## S2 Protocol Integration
+
+### S2 Service
+- **Location**: openadr-ven container (background process)
+- **Protocol**: S2 (FRBC - Flexibility Resource Bank Control)
+- **Mapping**: S2 resources -> FM sensors (51-65)
+- **Functions**:
+  - process_load_control(): Converts OpenADR load control to S2 FRBC instructions
+  - process_demand_response(): Converts OpenADR demand response to S2 FRBC instructions
+
+### S2 Message Types
+- FRBCInstruction: Power setpoint commands
+- FRBCSystemDescription: Resource capabilities
+- FRBCActuatorStatus: Resource status feedback
+- FRBCStorageStatus: Battery/EV storage status
+
+## Backup Strategy
+
+### Automated Backups
+- **Script**: `/home/eric/cariflex/scripts/full_backup.sh`
+- **Location**: `/home/eric/backups/cariflex/YYYYMMDD_HHMMSS.tar.gz`
+- **Contents**:
+  - PostgreSQL dump (flexmeasures DB)
+  - Redis dump (RDB file)
+  - FM config and templates
+  - OpenADR scripts and Dockerfiles
+  - Grafana dashboards (JSON)
+  - Docker container states
+  - Network configuration
+
+### Backup Retention
+- Daily backups kept for 30 days
+- Weekly backups kept for 12 weeks
+- Monthly backups kept indefinitely
+
+## Deployment Environments
+
+### Development
+- **Host**: Local VM or developer machine
+- **Docker Compose**: Single node
+- **Data**: Simulated
+- **URL**: localhost
+
+### Test
+- **Host**: Test server
+- **Docker Compose**: Single node
+- **Data**: Anonymized production data
+- **URL**: test.cariflex.digitribe.fr
+
+### Production
+- **Host**: Production server (this deployment)
+- **Docker Compose**: Single node (scalable to Swarm/K8s)
+- **Data**: Real-time production data
+- **URL**: cariflex.digitribe.fr
+
+## Security
+
+### Credentials
+| Service | Username | Password | Location |
+|---------|----------|----------|----------|
+| FlexMeasures | admin@digitribe.fr | Digitribe972 | FM DB |
+| PostgreSQL | flexmeasures | Digitribe972 | PostgreSQL |
+| Redis | - | (disabled) | Redis |
+| Grafana | admin | admin | Grafana DB |
+
+### SSL/TLS
+- All public services use Let's Encrypt certificates
+- Traefik handles SSL termination
+- Internal services use HTTP (isolated network)
+
+## Monitoring
+
+### Health Checks
+- FlexMeasures: https://cariflex.digitribe.fr/health
+- Grafana: https://grafana.digitrobe.fr/api/health
+- OpenADR VTN: http://192.168.240.10:8080/OpenADR2/Simple/2.0b
+
+### Logs
+- Docker: `docker logs <container>`
+- FM: `/app/flexmeasures/logs/`
+- Grafana: `/var/log/grafana/`
+
+## Troubleshooting
+
+### Common Issues
+1. **Redis ACL error**: `docker exec flexmeasures-redis redis-cli CONFIG SET requirepass ""`
+2. **OpenADR connection refused**: Check UFW rules for 192.168.240.0/24
+3. **Grafana no data**: Check datasource UID matches PostgreSQL DS
+4. **FM worker not running**: `docker restart flexmeasures-worker`
+
+### Useful Commands
+```bash
+# View all containers
+docker ps -a | grep -E "flexmeasures|openadr|grafana"
+
+# Check OpenADR communication
+docker logs openadr-ven | grep -i "price\|load\|event"
+
+# Check FM API
+curl -s https://cariflex.digitribe.fr/api/v3_0/sensors
+
+# Database query
+docker exec flexmeasures-db psql -U flexmeasures -d flexmeasures -c "SELECT * FROM sensor WHERE id BETWEEN 80 AND 90;"
+```
+
+## References
+- FlexMeasures: https://flexmeasures.readthedocs.io/
+- OpenLEADR: https://openleadr.org/
+- S2 Protocol: https://s2-standard.github.io/
+- Grafana: https://grafana.com/docs/
--- a/iac/ansible/playbook.yml
+++ b/iac/ansible/playbook.yml
@@ -0,0 +1,59 @@
+# Cariflex EMS - Ansible Playbook
+# Deploys the full Cariflex stack to a target host
+
+---
+- name: Deploy Cariflex EMS
+  hosts: cariflex
+  become: yes
+  vars:
+    cariflex_version: "{{ lookup('env', 'CARIFLEX_VERSION') | default('latest', true) }}"
+    deploy_env: "{{ lookup('env', 'DEPLOY_ENV') | default('production', true) }}"
+    
+  pre_tasks:
+    - name: Check minimum requirements
+      assert:
+        that:
+          - ansible_memtotal_mb >= 4096
+          - ansible_processor_vcpus >= 2
+        fail_msg: "Host does not meet minimum requirements (4GB RAM, 2 CPUs)"
+
+  roles:
+    - role: common
+      tags: [common]
+    - role: docker
+      tags: [docker]
+    - role: traefik
+      tags: [traefik]
+    - role: postgresql
+      tags: [database]
+    - role: redis
+      tags: [cache]
+    - role: flexmeasures
+      tags: [app]
+    - role: openadr
+      tags: [openadr]
+    - role: grafana
+      tags: [monitoring]
+    - role: backup
+      tags: [backup]
+
+  post_tasks:
+    - name: Verify deployment
+      uri:
+        url: "https://{{ cariflex_domain }}/api/v3_0/sensors"
+        status_code: 200
+      register: health_check
+      retries: 10
+      delay: 5
+      until: health_check.status == 200
+
+    - name: Display deployment info
+      debug:
+        msg: |
+          ==========================================
+          Cariflex EMS Deployed Successfully!
+          Environment: {{ deploy_env }}
+          URL: https://{{ cariflex_domain }}
+          Grafana: https://{{ grafana_domain }}
+          Version: {{ cariflex_version }}
+          ==========================================
--- a/iac/environments/README.md
+++ b/iac/environments/README.md
@@ -0,0 +1,85 @@
+# Cariflex EMS - Environment Configuration
+
+## Development Environment
+
+```bash
+# .env.dev
+DEPLOY_ENV=dev
+CARIFLEX_VERSION=dev
+CARIFLEX_DOMAIN=cariflex.dev.local
+GRAFANA_DOMAIN=grafana.dev.local
+FM_SECRET_KEY=dev-secret-key-change-me
+FM_DB_PASSWORD=devpassword
+REDIS_PASSWORD=devpassword
+```
+
+### Deploy to Dev
+```bash
+cd /home/eric/cariflex/iac
+ansible-playbook playbook.yml -i environments/dev/inventory --extra-vars "deploy_env=dev"
+```
+
+## Test Environment
+
+```bash
+# .env.test
+DEPLOY_ENV=test
+CARIFLEX_VERSION=test
+CARIFLEX_DOMAIN=cariflex.test.digitribe.fr
+GRAFANA_DOMAIN=grafana.test.digitribe.fr
+FM_SECRET_KEY={{ vault_fm_secret_key }}
+FM_DB_PASSWORD={{ vault_fm_db_password }}
+REDIS_PASSWORD={{ vault_redis_password }}
+```
+
+### Deploy to Test
+```bash
+cd /home/eric/cariflex/iac
+ansible-playbook playbook.yml -i environments/test/inventory --extra-vars "deploy_env=test" --ask-vault-pass
+```
+
+## Production Environment
+
+```bash
+# .env.prod
+DEPLOY_ENV=production
+CARIFLEX_VERSION=latest
+CARIFLEX_DOMAIN=cariflex.digitribe.fr
+GRAFANA_DOMAIN=grafana.digitribe.fr
+FM_SECRET_KEY={{ vault_fm_secret_key }}
+FM_DB_PASSWORD={{ vault_fm_db_password }}
+REDIS_PASSWORD={{ vault_redis_password }}
+```
+
+### Deploy to Production
+```bash
+cd /home/eric/cariflex/iac
+ansible-playbook playbook.yml -i environments/prod/inventory --extra-vars "deploy_env=production" --ask-vault-pass
+```
+
+## Docker Compose Override Files
+
+Each environment has its own override file:
+- `docker-compose.dev.yml` - Development (debug mode, local volumes)
+- `docker-compose.test.yml` - Testing (test data, isolated network)
+- `docker-compose.prod.yml` - Production (optimized, SSL, backups)
+
+## Kubernetes Deployment
+
+For K8s deployment, see `k8s/` directory:
+- `k8s/namespace.yaml` - Namespace creation
+- `k8s/configmap.yaml` - Configuration
+- `k8s/secrets.yaml` - Secrets (encrypted)
+- `k8s/deployments/` - Application deployments
+- `k8s/services/` - Service definitions
+- `k8s/ingress/` - Ingress rules
+
+### Deploy to K8s
+```bash
+kubectl apply -f k8s/namespace.yaml
+kubectl apply -f k8s/configmap.yaml
+kubectl apply -f k8s/secrets.yaml
+kubectl apply -f k8s/deployments/
+kubectl apply -f k8s/services/
+kubectl apply -f k8s/ingress/
+```
--- a/scripts/full_backup.sh
+++ b/scripts/full_backup.sh
@@ -0,0 +1,56 @@
+#!/bin/bash
+# Cariflex Full Backup Script
+BACKUP_DIR="/home/eric/backups/cariflex/$(date +%Y%m%d_%H%M%S)"
+mkdir -p "$BACKUP_DIR"
+
+echo "=== Cariflex Full Backup $(date) ==="
+
+# 1. PostgreSQL dump
+echo "Backing up PostgreSQL..."
+docker exec flexmeasures-db pg_dump -U flexmeasures flexmeasures | gzip > "$BACKUP_DIR/flexmeasures_db.sql.gz"
+
+# 2. Redis dump
+echo "Backing up Redis..."
+docker exec flexmeasures-redis redis-cli BGSAVE
+docker cp flexmeasures-redis:/data/dump.rdb "$BACKUP_DIR/redis_dump.rdb"
+
+# 3. FM config and templates
+echo "Backing up FM config..."
+cp -r /home/eric/cariflex/templates "$BACKUP_DIR/"
+cp -r /home/eric/cariflex/config "$BACKUP_DIR/" 2>/dev/null || true
+cp /home/eric/flexmeasures/docker-compose.yml "$BACKUP_DIR/"
+cp /home/eric/flexmeasures/docker-compose.openadr.yml "$BACKUP_DIR/"
+
+# 4. OpenADR scripts
+echo "Backing up OpenADR..."
+cp -r /home/eric/flexmeasures/*.py "$BACKUP_DIR/"
+cp /home/eric/flexmeasures/Dockerfile.openadr "$BACKUP_DIR/"
+cp /home/eric/flexmeasures/start_ven.sh "$BACKUP_DIR/"
+
+# 5. Grafana dashboards
+echo "Backing up Grafana dashboards..."
+curl -s -u admin:admin "http://localhost:3001/api/dashboards/uid/cariflex-ems-full" | python3 -c "
+import json, sys
+d = json.load(sys.stdin)
+with open('$BACKUP_DIR/grafana_dashboard.json', 'w') as f:
+    json.dump(d, f, indent=2)
+print('Dashboard saved')
+"
+
+# 6. Docker container states
+echo "Backing up container states..."
+docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}" > "$BACKUP_DIR/containers.txt"
+docker inspect openadr-vtn openadr-ven flexmeasures-server flexmeasures-db flexmeasures-redis 2>/dev/null > "$BACKUP_DIR/container_inspects.json"
+
+# 7. Network config
+echo "Backing up network config..."
+docker network inspect openadr-internal 2>/dev/null > "$BACKUP_DIR/network_openadr.json"
+docker network inspect traefik-public 2>/dev/null > "$BACKUP_DIR/network_traefik.json"
+
+# 8. Compress
+echo "Compressing..."
+tar -czf "$BACKUP_DIR.tar.gz" -C "$(dirname $BACKUP_DIR)" "$(basename $BACKUP_DIR)"
+rm -rf "$BACKUP_DIR"
+
+echo "=== Backup complete: $BACKUP_DIR.tar.gz ==="
+ls -lh "$BACKUP_DIR.tar.gz"