eric/cariflex
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Infrastructure docs, IaC scripts, skills energy (VOLTTRON, EnergyPlus, OpenStudio, BuildingGym, OSeMOSYS)

Browse Source
This commit is contained in:
Eric F
2026-06-10 12:09:00 -04:00
parent a0577ed17c
commit 84667c7126
4 changed files with 430 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

230
docs/INFRASTRUCTURE.md Normal file
View File

@@ -0,0 +1,230 @@
# Cariflex EMS - Infrastructure Documentation
## Architecture Overview
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ CARIFLEX EMS │
│ Energy Management System for Martinique │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ OpenADR │ │ FlexMeasures│ │ Grafana │ │
│ │ VTN/VEN │───▶│ Server │───▶│ Dashboard │ │
│ │ (S2+OpenADR)│ │ (EMS Core) │ │ (14 panels)│ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ PostgreSQL │ │ Redis │ │ Traefik │ │
│ │ (time-series)│ │ (job queue) │ │ (reverse │ │
│ │ │ │ │ │ proxy) │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
```
## Network Architecture
### Networks
| Network | Subnet | Purpose | External |
|---------|--------|---------|----------|
| `traefik-public` | 172.29.0.0/16 | Public services (FM, Grafana) | Yes |
| `openadr-internal` | 192.168.240.0/24 | OpenADR VTN/VEN communication | No |
### DNS/Hosts
| Service | Domain | Container | Port |
|---------|--------|-----------|------|
| FlexMeasures | cariflex.digitribe.fr | flexmeasures-server | 5000 |
| Grafana | grafana.digitribe.fr | smart-city-grafana | 3000 |
| OpenADR VTN | - | openadr-vtn | 8080 |
| OpenADR VEN | - | openadr-ven | - |
### Firewall (UFW)
- Public ports: 80 (HTTP), 443 (HTTPS), 51820 (WireGuard)
- Internal networks: 172.16.0.0/12, 10.0.0.0/8, 192.168.0.0/16, 192.168.240.0/24
## Docker Containers
### Cariflex EMS Stack
| Container | Image | Network | IP | Port | Status |
|-----------|-------|---------|----|---- |--------|
| flexmeasures-server | lfenergy/flexmeasures:latest | traefik-public | 172.29.0.x | 5000 | ✅ UP |
| flexmeasures-worker | lfenergy/flexmeasures:latest | traefik-public | 172.29.0.x | - | ✅ UP |
| flexmeasures-db | postgres:17 | traefik-public | 172.29.0.x | 5432 | ✅ UP |
| flexmeasures-redis | redis:7-alpine | traefik-public | 172.29.0.x | 6379 | ✅ UP |
| openadr-vtn | flexmeasures-openadr-vtn | openadr-internal | 192.168.240.10 | 8080 | ✅ UP |
| openadr-ven | flexmeasures-openadr-ven | openadr-internal | 192.168.240.11 | - | ✅ UP |
| smart-city-grafana | grafana/grafana | traefik-public | - | 3000 | ✅ UP |
## IP Address Allocation (phpIPAM)
### Cariflex Subnet: 192.168.240.0/24
| IP | Service | Container | MAC |
|----|---------|-----------|-----|
| 192.168.240.1 | Gateway | - | - |
| 192.168.240.10 | OpenADR VTN | openadr-vtn | - |
| 192.168.240.11 | OpenADR VEN | openadr-ven | - |
| 192.168.240.128-254 | Reserved | - | - |
### Public Subnet (traefik-public): 172.29.0.0/16
| IP | Service | Domain |
|----|---------|--------|
| 172.29.0.x | FlexMeasures | cariflex.digitribe.fr |
| 172.29.0.x | Grafana | grafana.digitribe.fr |
## Sensor Mapping
### FlexMeasures Sensors
| ID | Name | Unit | Type | Source |
|----|------|------|------|--------|
| 41-50 | pv_01..10_power | kW | PV Production | Simulator |
| 51-60 | bat_01..10_power | kWh | Battery SOC | Simulator |
| 61-70 | ev_01..10_power | kWh | EV Charge | Simulator |
| 81 | irradiance | W/m² | Weather | Open-Meteo |
| 82 | temperature | °C | Weather | Open-Meteo |
| 83 | wind_speed | m/s | Weather | Open-Meteo |
| 84 | consumption_price | EUR/MWh | Price | OpenADR VTN |
| 85 | production_price | EUR/MWh | Price | Simulated |
| 86 | load_control_signal | % | DSR | OpenADR VTN |
| 87 | demand_response_signal | dimensionless | DSR | OpenADR VTN |
### S2 Resource Mapping
| S2 Resource | FM Sensor | Asset | Min Power | Max Power |
|-------------|-----------|-------|-----------|-----------|
| battery_01-05 | 51-55 | Bat_01-05 | -50 kW | +50 kW |
| ev_01-05 | 61-65 | EV_01-05 | -22 kW | +22 kW |
## OpenADR Protocol
### VTN (Virtual Top Node)
- **Container**: openadr-vtn
- **Port**: 8080 (internal)
- **Profile**: OpenADR 2.0b
- **Events**: ENERGY_PRICE (24h), LOAD_CONTROL (4h)
- **Schedule**: Every 10 seconds poll
### VEN (Virtual End Node)
- **Container**: openadr-ven
- **VTN URL**: http://192.168.240.10:8080/OpenADR2/Simple/2.0b
- **Registration ID**: reg-Cariflex-VEN-001
- **Event Handler**: on_event -> FM API (sensors 84, 86)
### Event Flow
1. VTN creates events (price, load control)
2. VEN polls VTN every 10s
3. VEN receives oadrDistributeEvent
4. VEN sends event responses (optIn)
5. VEN posts data to FM API
6. Grafana displays real-time data
## S2 Protocol Integration
### S2 Service
- **Location**: openadr-ven container (background process)
- **Protocol**: S2 (FRBC - Flexibility Resource Bank Control)
- **Mapping**: S2 resources -> FM sensors (51-65)
- **Functions**:
- process_load_control(): Converts OpenADR load control to S2 FRBC instructions
- process_demand_response(): Converts OpenADR demand response to S2 FRBC instructions
### S2 Message Types
- FRBCInstruction: Power setpoint commands
- FRBCSystemDescription: Resource capabilities
- FRBCActuatorStatus: Resource status feedback
- FRBCStorageStatus: Battery/EV storage status
## Backup Strategy
### Automated Backups
- **Script**: `/home/eric/cariflex/scripts/full_backup.sh`
- **Location**: `/home/eric/backups/cariflex/YYYYMMDD_HHMMSS.tar.gz`
- **Contents**:
- PostgreSQL dump (flexmeasures DB)
- Redis dump (RDB file)
- FM config and templates
- OpenADR scripts and Dockerfiles
- Grafana dashboards (JSON)
- Docker container states
- Network configuration
### Backup Retention
- Daily backups kept for 30 days
- Weekly backups kept for 12 weeks
- Monthly backups kept indefinitely
## Deployment Environments
### Development
- **Host**: Local VM or developer machine
- **Docker Compose**: Single node
- **Data**: Simulated
- **URL**: localhost
### Test
- **Host**: Test server
- **Docker Compose**: Single node
- **Data**: Anonymized production data
- **URL**: test.cariflex.digitribe.fr
### Production
- **Host**: Production server (this deployment)
- **Docker Compose**: Single node (scalable to Swarm/K8s)
- **Data**: Real-time production data
- **URL**: cariflex.digitribe.fr
## Security
### Credentials
| Service | Username | Password | Location |
|---------|----------|----------|----------|
| FlexMeasures | admin@digitribe.fr | Digitribe972 | FM DB |
| PostgreSQL | flexmeasures | Digitribe972 | PostgreSQL |
| Redis | - | (disabled) | Redis |
| Grafana | admin | admin | Grafana DB |
### SSL/TLS
- All public services use Let's Encrypt certificates
- Traefik handles SSL termination
- Internal services use HTTP (isolated network)
## Monitoring
### Health Checks
- FlexMeasures: https://cariflex.digitribe.fr/health
- Grafana: https://grafana.digitrobe.fr/api/health
- OpenADR VTN: http://192.168.240.10:8080/OpenADR2/Simple/2.0b
### Logs
- Docker: `docker logs <container>`
- FM: `/app/flexmeasures/logs/`
- Grafana: `/var/log/grafana/`
## Troubleshooting
### Common Issues
1. **Redis ACL error**: `docker exec flexmeasures-redis redis-cli CONFIG SET requirepass ""`
2. **OpenADR connection refused**: Check UFW rules for 192.168.240.0/24
3. **Grafana no data**: Check datasource UID matches PostgreSQL DS
4. **FM worker not running**: `docker restart flexmeasures-worker`
### Useful Commands
```bash
# View all containers
docker ps -a | grep -E "flexmeasures|openadr|grafana"
# Check OpenADR communication
docker logs openadr-ven | grep -i "price\|load\|event"
# Check FM API
curl -s https://cariflex.digitribe.fr/api/v3_0/sensors
# Database query
docker exec flexmeasures-db psql -U flexmeasures -d flexmeasures -c "SELECT * FROM sensor WHERE id BETWEEN 80 AND 90;"
```
## References
- FlexMeasures: https://flexmeasures.readthedocs.io/
- OpenLEADR: https://openleadr.org/
- S2 Protocol: https://s2-standard.github.io/
- Grafana: https://grafana.com/docs/

59
iac/ansible/playbook.yml Normal file
View File

@@ -0,0 +1,59 @@
# Cariflex EMS - Ansible Playbook
# Deploys the full Cariflex stack to a target host
---
- name: Deploy Cariflex EMS
hosts: cariflex
become: yes
vars:
cariflex_version: "{{ lookup('env', 'CARIFLEX_VERSION') | default('latest', true) }}"
deploy_env: "{{ lookup('env', 'DEPLOY_ENV') | default('production', true) }}"
pre_tasks:
- name: Check minimum requirements
assert:
that:
- ansible_memtotal_mb >= 4096
- ansible_processor_vcpus >= 2
fail_msg: "Host does not meet minimum requirements (4GB RAM, 2 CPUs)"
roles:
- role: common
tags: [common]
- role: docker
tags: [docker]
- role: traefik
tags: [traefik]
- role: postgresql
tags: [database]
- role: redis
tags: [cache]
- role: flexmeasures
tags: [app]
- role: openadr
tags: [openadr]
- role: grafana
tags: [monitoring]
- role: backup
tags: [backup]
post_tasks:
- name: Verify deployment
uri:
url: "https://{{ cariflex_domain }}/api/v3_0/sensors"
status_code: 200
register: health_check
retries: 10
delay: 5
until: health_check.status == 200
- name: Display deployment info
debug:
msg: |
==========================================
Cariflex EMS Deployed Successfully!
Environment: {{ deploy_env }}
URL: https://{{ cariflex_domain }}
Grafana: https://{{ grafana_domain }}
Version: {{ cariflex_version }}
==========================================

85
iac/environments/README.md Normal file
View File

@@ -0,0 +1,85 @@
# Cariflex EMS - Environment Configuration
## Development Environment
```bash
# .env.dev
DEPLOY_ENV=dev
CARIFLEX_VERSION=dev
CARIFLEX_DOMAIN=cariflex.dev.local
GRAFANA_DOMAIN=grafana.dev.local
FM_SECRET_KEY=dev-secret-key-change-me
FM_DB_PASSWORD=devpassword
REDIS_PASSWORD=devpassword
```
### Deploy to Dev
```bash
cd /home/eric/cariflex/iac
ansible-playbook playbook.yml -i environments/dev/inventory --extra-vars "deploy_env=dev"
```
## Test Environment
```bash
# .env.test
DEPLOY_ENV=test
CARIFLEX_VERSION=test
CARIFLEX_DOMAIN=cariflex.test.digitribe.fr
GRAFANA_DOMAIN=grafana.test.digitribe.fr
FM_SECRET_KEY={{ vault_fm_secret_key }}
FM_DB_PASSWORD={{ vault_fm_db_password }}
REDIS_PASSWORD={{ vault_redis_password }}
```
### Deploy to Test
```bash
cd /home/eric/cariflex/iac
ansible-playbook playbook.yml -i environments/test/inventory --extra-vars "deploy_env=test" --ask-vault-pass
```
## Production Environment
```bash
# .env.prod
DEPLOY_ENV=production
CARIFLEX_VERSION=latest
CARIFLEX_DOMAIN=cariflex.digitribe.fr
GRAFANA_DOMAIN=grafana.digitribe.fr
FM_SECRET_KEY={{ vault_fm_secret_key }}
FM_DB_PASSWORD={{ vault_fm_db_password }}
REDIS_PASSWORD={{ vault_redis_password }}
```
### Deploy to Production
```bash
cd /home/eric/cariflex/iac
ansible-playbook playbook.yml -i environments/prod/inventory --extra-vars "deploy_env=production" --ask-vault-pass
```
## Docker Compose Override Files
Each environment has its own override file:
- `docker-compose.dev.yml` - Development (debug mode, local volumes)
- `docker-compose.test.yml` - Testing (test data, isolated network)
- `docker-compose.prod.yml` - Production (optimized, SSL, backups)
## Kubernetes Deployment
For K8s deployment, see `k8s/` directory:
- `k8s/namespace.yaml` - Namespace creation
- `k8s/configmap.yaml` - Configuration
- `k8s/secrets.yaml` - Secrets (encrypted)
- `k8s/deployments/` - Application deployments
- `k8s/services/` - Service definitions
- `k8s/ingress/` - Ingress rules
### Deploy to K8s
```bash
kubectl apply -f k8s/namespace.yaml
kubectl apply -f k8s/configmap.yaml
kubectl apply -f k8s/secrets.yaml
kubectl apply -f k8s/deployments/
kubectl apply -f k8s/services/
kubectl apply -f k8s/ingress/
```

56
scripts/full_backup.sh Executable file
View File

@@ -0,0 +1,56 @@
#!/bin/bash
# Cariflex Full Backup Script
BACKUP_DIR="/home/eric/backups/cariflex/$(date +%Y%m%d_%H%M%S)"
mkdir -p "$BACKUP_DIR"
echo "=== Cariflex Full Backup $(date) ==="
# 1. PostgreSQL dump
echo "Backing up PostgreSQL..."
docker exec flexmeasures-db pg_dump -U flexmeasures flexmeasures | gzip > "$BACKUP_DIR/flexmeasures_db.sql.gz"
# 2. Redis dump
echo "Backing up Redis..."
docker exec flexmeasures-redis redis-cli BGSAVE
docker cp flexmeasures-redis:/data/dump.rdb "$BACKUP_DIR/redis_dump.rdb"
# 3. FM config and templates
echo "Backing up FM config..."
cp -r /home/eric/cariflex/templates "$BACKUP_DIR/"
cp -r /home/eric/cariflex/config "$BACKUP_DIR/" 2>/dev/null || true
cp /home/eric/flexmeasures/docker-compose.yml "$BACKUP_DIR/"
cp /home/eric/flexmeasures/docker-compose.openadr.yml "$BACKUP_DIR/"
# 4. OpenADR scripts
echo "Backing up OpenADR..."
cp -r /home/eric/flexmeasures/*.py "$BACKUP_DIR/"
cp /home/eric/flexmeasures/Dockerfile.openadr "$BACKUP_DIR/"
cp /home/eric/flexmeasures/start_ven.sh "$BACKUP_DIR/"
# 5. Grafana dashboards
echo "Backing up Grafana dashboards..."
curl -s -u admin:admin "http://localhost:3001/api/dashboards/uid/cariflex-ems-full" | python3 -c "
import json, sys
d = json.load(sys.stdin)
with open('$BACKUP_DIR/grafana_dashboard.json', 'w') as f:
json.dump(d, f, indent=2)
print('Dashboard saved')
"
# 6. Docker container states
echo "Backing up container states..."
docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}" > "$BACKUP_DIR/containers.txt"
docker inspect openadr-vtn openadr-ven flexmeasures-server flexmeasures-db flexmeasures-redis 2>/dev/null > "$BACKUP_DIR/container_inspects.json"
# 7. Network config
echo "Backing up network config..."
docker network inspect openadr-internal 2>/dev/null > "$BACKUP_DIR/network_openadr.json"
docker network inspect traefik-public 2>/dev/null > "$BACKUP_DIR/network_traefik.json"
# 8. Compress
echo "Compressing..."
tar -czf "$BACKUP_DIR.tar.gz" -C "$(dirname $BACKUP_DIR)" "$(basename $BACKUP_DIR)"
rm -rf "$BACKUP_DIR"
echo "=== Backup complete: $BACKUP_DIR.tar.gz ==="
ls -lh "$BACKUP_DIR.tar.gz"