eric/smart-city-digital-twin-martinique
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: distribution service + redpanda consumer + updated flow diagram

Browse Source 
- Add Pulsar distribution service (consumes smartcity-* → MQTT + context brokers)
- Add Redpanda → InfluxDB consumer (redpanda/consumer.py)
- Update FIXED_LOCATIONS with exact OpenRemote asset coordinates
- Fix Pulsar topics (underscore: smartcity-traffic not smartcity-traffic)
- Fix prometheus.yml endpoints (Redpanda:9644, comment inactive stacks)
- Add docker-compose.redpanda-consumer.yml
This commit is contained in:
Eric FELIXINE
2026-05-05 22:12:38 -04:00
parent 742b437ed9
commit c06acf4fe8
9 changed files with 2242 additions and 434 deletions
Download Patch File Download Diff File Expand all files Collapse all files

335
data-flow-diagram.md
View File

@@ -1,178 +1,223 @@
# Smart City Digital Twin — Diagramme des Flux de Données
# Smart City Digital Twin Martinique — Diagramme des Flux de Données
## Vue d'ensemble
Ce diagramme illustre le flux complet des données IoT du simulateur vers les différentes couches de traitement, de stockage et de visualisation.
**Dernière mise à jour :** 05 Mai 2026
**Projet :** Smart City Digital Twin Martinique
---
## Diagramme Mermaid
## Architecture Globale
```mermaid
graph TB
SIM[Smart City Simulator]
SENS[Capteurs IoT Reels]
EMQ[EMQX]
MOS[Mosquitto]
BUN[BunkerM]
FRO[FROST-Server]
ORI[Orion-LD]
STE[Stellio]
UI[OpenRemote UI]
ORM[OpenRemote Manager]
KC[Keycloak]
INF[InfluxDB]
PRO[Prometheus]
GEO[GeoServer]
GRA[Grafana]
MAP[MapStore]
CH[ClickHouse]
RW[RisingWave]
PUL[Pulsar]
RED[Redpanda]
subgraph Simulateur["🖥️ Simulateur (Host Python)"]
SIM[Smart City Simulator<br/>10 capteurs<br/>Intervalle: configurable]
end
SIM --> EMQ
SIM --> MOS
SIM --> BUN
SIM --> FRO
SENS --> EMQ
SENS --> MOS
SENS --> BUN
SENS --> FRO
SENS -.-> ORM
EMQ -->|via EMQX| ORI
EMQ -->|via EMQX| STE
MOS -->|via Mosquitto| ORI
MOS -->|via Mosquitto| STE
BUN -->|via BunkerM| ORI
BUN -->|via BunkerM| STE
UI --> ORM
ORM -.-> KC
SIM --> INF
SIM -->|real-time 1s| CH
SIM -->|streaming| RW
SIM -->|HTTP REST| PUL
SIM -->|HTTP REST| RED
ORI --> GRA
STE --> GRA
FRO --> GRA
ORI -.-> GEO
STE -.-> GEO
FRO -.-> GEO
GEO --> MAP
ORM --> GRA
EMQ -.-> PRO
ORI -.-> PRO
STE -.-> PRO
ORM -.-> PRO
subgraph MQTT_Brokers["📡 MQTT Brokers"]
EMQ[EMQX<br/>port 11883]
MOS[Mosquitto<br/>port 1883]
BUN[BunkerM<br/>port 1900<br/>MQTTS/TLS]
end
subgraph Stream["⚡ Event Streaming"]
PUL[Pulsar<br/>port 6650<br/>Topics: smartcity-*]
RED[Redpanda<br/>port 8082 REST<br/>Topics: traffic, air-quality, ...]
end
subgraph CB["🔗 Context Brokers"]
ORI[Orion-LD<br/>NGSI-LD<br/>port 1026]
STE[Stellio<br/>NGSI-LD<br/>port 8080]
FRO[FROST-Server<br/>SensorThings<br/>port 8080]
end
subgraph Storage["💾 Stockage & Métriques"]
INF[InfluxDB<br/>Bucket: iot_data<br/>port 8086]
PRO[Prometheus<br/>Scrape: /metrics<br/>port 9090]
GEO[GeoServer<br/>WMS/WFS/WMTS<br/>port 8080]
end
subgraph IoT_Platform["🏢 Plateforme IoT"]
ORM[OpenRemote Manager<br/>MQTT Agent<br/>port 8080]
KC[Keycloak<br/>port 8080]
end
subgraph VIZ["📊 Visualisation"]
GRA[Grafana<br/>Dashboards<br/>port 3000]
MAP[MapStore<br/>WMS/WFS<br/>port 8080]
end
subgraph Distribution["🔄 Distribution Service"]
DIST[Pulsar Distribution<br/>Pulsar → Brokers]
end
subgraph Consumer["📥 Redpanda Consumer"]
RCONS[Redpanda → InfluxDB<br/>REST → InfluxDB]
end
%% ── Flux Simulateur ──────────────────────────────────────────────────
SIM -->|"1️⃣ MQTT publish<br/>city/sensors/{type}/{id}"| EMQ
SIM -->|"1️⃣ MQTT publish"| MOS
SIM -->|"1️⃣ MQTT publish"| BUN
SIM -->|"2️⃣ HTTP POST<br/>NGSI-LD"| ORI
SIM -->|"2️⃣ HTTP POST<br/>NGSI-LD"| STE
SIM -->|"2️⃣ HTTP POST<br/>SensorThings"| FRO
SIM -->|"3️⃣ Pulsar client<br/>pulsar://localhost:6650"| PUL
SIM -->|"4️⃣ HTTP REST Proxy<br/>localhost:8082/topics/"| RED
SIM -->|"5️⃣ InfluxDB v2 API<br/>async non-bloquant"| INF
%% ── Flux Distribution (Pulsar → Brokers) ──────────────────────────────
PUL -->|"Consomme<br/>smartcity-*"| DIST
DIST -->|"Republish<br/>MQTT"| EMQ
DIST -->|"Republish<br/>MQTT"| MOS
DIST -->|"Republish<br/>NGSI-LD"| ORI
DIST -->|"Republish<br/>NGSI-LD"| STE
DIST -->|"Republish<br/>SensorThings"| FRO
%% ── Flux Redpanda → InfluxDB ──────────────────────────────────────────
RED -->|"REST poll<br/>topics/{name}/offsets"| RCONS
RCONS -->|"Line Protocol<br/>Write API"| INF
%% ── OpenRemote MQTT Agent ──────────────────────────────────────────────
EMQ -->|"6️⃣ Subscribe<br/>city/sensors/#"| ORM
MOS -->|"6️⃣ Subscribe"| ORM
BUN -->|"6️⃣ Subscribe"| ORM
%% ── Métriques Prometheus ────────────────────────────────────────────────
SIM -->|"7️⃣ /metrics<br/>port 8001"| PRO
EMQ -->|"/api/v5/metrics"| PRO
STE -->|"/actuator/prometheus"| PRO
FRO -->|"/metrics"| PRO
INF -->|"/metrics"| PRO
RED -->|"/public_metrics"| PRO
ORM -->|"/actuator/prometheus"| PRO
GRA -->|"/metrics"| PRO
%% ── Visualisation ─────────────────────────────────────────────────────
INF -->|"Datasources<br/>Flux IoT"| GRA
ORI -->|"NGSI-LD<br/>Datasource"| GRA
STE -->|"NGSI-LD<br/>Datasource"| GRA
FRO -->|"SensorThings<br/>Datasource"| GRA
GEO -->|"WMS/WMTS"| MAP
ORM -->|MapSettings<br/>Martinique| MAP
ORM -->|"Live assets<br/>REST"| GRA
```
---
## Description des flux
## Flux Détaillés
### 1. **Génération des données (Simulator)**
- **Smart City Simulator** (Python) génère des données pour 10 capteurs (Traffic, Air Quality, Parking, Noise, Weather, Light)
- Intervalle de publication : 1 seconde (temps réel)
- Protocoles : MQTT (vers brokers uniquement)
- **⚠️ Projet** : Le simulateur n'envoie PAS directement à OpenRemote (pas de REST API)
### 1️⃣ Flux MQTT — Brokers
### 2. **Ingestion MQTT (Brokers)**
- **EMQX** (port 11883) : Broker public, reçoit tous les capteurs
- **Mosquitto** (port 1883) : Via Traefik, accès externe
- **BunkerM** (port 1900) : MQTTS (TLS), accès sécurisé
| Broker | Port | Protocol | Topics |
|--------|------|----------|--------|
| EMQX | 11883 | MQTT | `city/sensors/{type}/{id}` |
| Mosquitto | 1883 | MQTT | `city/sensors/{type}/{id}` |
| BunkerM | 1900 | MQTTS (TLS) | `city/sensors/{type}/{id}` |
### 3. **Context Brokers (NGSI-LD & SensorThings)**
- **Orion-LD** : Reçoit les données au format NGSI-LD
- 10 entités (TrafficFlowObserved, AirQualityObserved, etc.)
- Smart Data Models utilisés
- **Provenance** : Données via EMQX, Mosquitto et BunkerM (voir étiquettes dans le diagramme)
- **Stellio** : Alternative NGSI-LD
- 14 payloads entités
- Contexte : `https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld`
- **Provenance** : Données via EMQX, Mosquitto et BunkerM
- **FROST-Server** : SensorThings API
- 21 256+ observations
- PostgreSQL + TimescaleDB
- **Provenance** : Données via EMQX, Mosquitto et BunkerM
Le simulateur publie simultanément sur les 3 brokers.
### 4. **Plateforme IoT (OpenRemote)**
- **OpenRemote Manager** (realm `smartcity`)
- 33 assets IoT configurés
- Carte Martinique (mapsettings.json)
- Réception via **MQTT Agent** depuis les brokers (EMQX, Mosquitto, BunkerM)
- Peut aussi recevoir directement des capteurs IoT (via MQTT)
- **Keycloak** : Authentification OpenID Connect
- Client `openremote` avec Service Account
- Token endpoint : `/auth/realms/smartcity/protocol/openid-connect/token`
### 2️⃣ Flux HTTP REST — Context Brokers
### 5. **Stockage & Métriques**
- **InfluxDB** : Stockage temporel pour Grafana
- Bucket : `iot_data`
- Datasource dans Grafana
- **Prometheus** : Collecte des métriques
- MQTT brokers, Context brokers, OpenRemote
- **GeoServer** : Données géospatiales
- PostGIS pour centralisation
- WMS/WFS pour MapStore
| Broker | Format | Port | Topics |
|--------|--------|------|--------|
| Orion-LD | NGSI-LD | 1026 | Entités par type |
| Stellio | NGSI-LD | 8080 | Entités par type |
| FROST-Server | SensorThings | 8080 | Things → Datastreams → Observations |
### 5. **Visualisation & Analyse**
- **Grafana** (port 3001)
- Dashboard : `smartcity-martinique-2026`
- Datasources : InfluxDB, FROST, Orion-LD, ClickHouse, RisingWave
- **MapStore** : Cartographie
- Sources WMS/WFS depuis GeoServer
- **OpenRemote UI** : Manager Interface
- Visualisation des assets realm Smart City
### 3️⃣ Flux Pulsar — Event Streaming
### 6. **Analytique & Streaming**
- **ClickHouse** (port 8123/9000) : Columnar OLAP Database
- Analytique rapide sur grandes volumes de données IoT
- Intégration possible via HTTP interface (port 8123)
- Compatible avec Grafana (plugin ClickHouse)
- **RisingWave** (port 4566/4567) : Streaming Database PostgreSQL-compatible
- Traitement de flux en temps réel
- Interface web pour requêtes SQL streaming
- Compatible Grafana via datasource PostgreSQL
- **Topics** : `persistent://public/default/smartcity-traffic`, `smartcity-airquality`, `smartcity-parking`, `smartcity-noise`, `smartcity-weather`, `smartcity-light`
- **Port binaire** : `6650` (connectable depuis le host)
- **Distribution** : Le service `pulsar-distribution` consomme ces topics et republie vers les brokers MQTT et context brokers
### 4️⃣ Flux Redpanda — Kafka-compatible REST
- **REST Proxy** : `http://localhost:8082`
- **Topics** : `traffic`, `air-quality`, `parking`, `noise`, `weather`, `air-quality`
- **Payload** : Base64(JSON) dans `{"records": [{"value": "<base64>"}]}`
- **Consumer** : `redpanda/consumer.py` — poll toutes les 10s et écrit dans InfluxDB
### 5️⃣ Flux InfluxDB — Temps Réel
- **API** : `http://localhost:8086/api/v2/write`
- **Bucket** : `iot_data`
- **Org** : `digitribe`
- **Mode** : Asynchrone (thread daemon) pour ne pas bloquer le publish MQTT
### 6️⃣ OpenRemote — MQTT Agent
L'agent MQTT d'OpenRemote souscrit aux topics `city/sensors/#` sur les brokers MQTT (EMQX, Mosquitto, BunkerM). Les payloads sont automatiquement parsés et les attributs des assets sont mis à jour.
**Configuration via Manager UI** (`https://openremote.digitribe.fr/manager/`) :
1. Se connecter avec `admin/Digitribe972`
2. Choisir le realm `smartcity`
3. **Assets → Agents → + Add Agent**
4. Type : **MQTT Agent**
5. Configurer :
- **MQTT Broker URI** : `tcp://emqx_emqx_1:1883` (réseau smartcity-shared)
- **Topic Filter** : `city/sensors/#`
- **QoS** : 1
- **Enabled** : ✅
### 7️⃣ Flux Prometheus — Métriques
| Service | Endpoint `/metrics` | Scrape |
|---------|---------------------|--------|
| Simulator | `localhost:8001` | ✅ |
| EMQX | `emqx_emqx_1:8081/api/v5/metrics` | ✅ |
| Stellio | `stellio-api-gateway:8080/actuator/prometheus` | ✅ |
| FROST | `frost_http-web-1:8080/metrics` | ✅ |
| InfluxDB | `smart-city-influxdb:8086/metrics` | ✅ |
| Redpanda | `smart-city-redpanda-console:8080/public_metrics` | ✅ |
| OpenRemote | `openremote-manager-1:8080/actuator/prometheus` | ✅ |
| Grafana | `smart-city-grafana:3000/metrics` | ✅ |
---
## Technologies clés
## Tableau Récapitulatif
| Composant | Technologie | Port | Statut |
|-----------|-------------|------|--------|
| Simulator | Python + paho-mqtt | Interne | ✅ Actif (1s) |
| Simulator | Python + paho-mqtt | Host:8001 (metrics) | ✅ Actif |
| EMQX | MQTT Broker | 11883 | ✅ Connecté |
| Orion-LD | NGSI-LD Broker | 1026 | ⚠️ À vérifier |
| Stellio | NGSI-LD Broker | 8080 | ⚠️ À vérifier |
| FROST-Server | SensorThings API | 8080 | ⚠️ À vérifier |
| OpenRemote | IoT Platform | 8080 | ⚠️ 403 (Service Account) |
| InfluxDB | Time Series DB | 8086 | ✅ Configuré |
| ClickHouse | Columnar OLAP DB | 8123/9000 | ✅ Ajouté |
| RisingWave | Streaming DB (PG) | 4566/4567 | ✅ Ajouté |
| Pulsar | Event Streaming | 8080 | ⚠️ Debugging |
| Redpanda | Kafka-compatible | 19092/9644 | ⚠️ OOM |
| Grafana | Visualization | 3001 | ✅ Dashboard créé |
| GeoServer | GeoServer | 8080 | ⚠️ À intégrer |
| Prometheus | Metrics | 9090 | ✅ En cours |
| Mosquitto | MQTT Broker | 1883 | ✅ Connecté |
| BunkerM | MQTTS Broker | 1900 | ✅ Connecté |
| Orion-LD | NGSI-LD Broker | 1026 | ✅ Données |
| Stellio | NGSI-LD Broker | 8080 | ✅ Données |
| FROST-Server | SensorThings API | 8080 | ✅ Données |
| OpenRemote | IoT Platform | 8080 | ✅ UI OK |
| InfluxDB | Time Series DB | 8086 | ✅ Bucket iot_data |
| Redpanda | Kafka-compatible | 8082 REST | ✅ Topics actifs |
| Pulsar | Event Streaming | 6650 | ✅ Connecté |
| Prometheus | Metrics | 9090 (conf) | ⏳ Container arrêté |
| Grafana | Visualisation | 3000 | ✅ Dashboards |
| GeoServer | Geo Data | 8080 | ✅ REST OK |
| MapStore | Cartographie | 8080 | ✅ WMS/WMTS |
---
## Fichiers associés
## Commandes Utiles
- **Simulator** : `~/smart-city-digital-twin-martinique/simulator.py` (intervalle 1s - temps réel)
- **Docker Compose** : `~/smart-city-digital-twin-martinique/docker-compose.yml`
- **ClickHouse** : `~/smart-city-digital-twin-martinique/clickhouse/docker-compose.yml`
- **RisingWave** : `~/smart-city-digital-twin-martinique/risingwave/docker-compose.yml`
- **Pulsar** : `~/smart-city-digital-twin-martinique/pulsar/docker-compose.yml`
- **Redpanda** : `~/smart-city-digital-twin-martinique/redpanda/docker-compose.yml`
- **Dashboard Grafana** : `~/smart-city-digital-twin-martinique/grafana_dashboard_smartcity.json`
- **Ce diagramme** : `~/smart-city-digital-twin-martinique/data-flow-diagram.md`
- **Session Resume** : `~/smart-city-digital-twin-martinique/session_resume_2026-05-05.md`
```bash
# Redémarrer le service de distribution Pulsar
cd ~/smart-city-digital-twin-martinique
docker build -t smart-city-pulsar-distribution:latest -f pulsar/Dockerfile pulsar/
docker compose -f docker-compose.yml -f docker-compose.distribution.yml up -d pulsar-distribution
---
# Redémarrer Prometheus (prometheus-brokers)
cd ~/smart-city-digital-twin-martinique
docker compose up -d prometheus-brokers
**Dernière mise à jour :** 05 Mai 2026
**Projet :** Smart City Digital Twin Martinique
**URL Grafana :** https://grafana.digitribe.fr/d/smartcity-martinique-2026/smart-city-digital-twin-martinique
# Lancer le consumer Redpanda (host)
cd ~/smart-city-digital-twin-martinique
python3 redpanda/consumer.py
# Vérifier les topics Redpanda
curl -s http://localhost:8082/topics
# Vérifier les métriques simulator
curl -s http://localhost:8001/metrics | grep "^simulator_"
# Logs distribution service
docker logs -f smart-city-pulsar-distribution
```

15
docker-compose.distribution.yml
View File

@@ -4,25 +4,14 @@
services:
pulsar-distribution:
build:
context: ./pulsar
dockerfile: Dockerfile
container_name: smart-city-pulsar-distribution
networks:
- smartcity-shared
environment:
- PULSAR_HOST=smart-city-pulsar
- PULSAR_HOST=pulsar
- PULSAR_PORT=6650
- EMQX_HOST=emqx_emqx_1
- MOSQUITTO_HOST=mosquitto-traefik
- ORION_URL=http://fiware-gis-quickstart-orion-1:1026
- STELLIO_URL=http://stellio-api-gateway:8080
- FROST_URL=http://frost-api-8090:8080/FROST-Server/v1.1
restart: unless-stopped
depends_on:
- smart-city-pulsar
labels:
- "traefik.enable=false"
- FROST_URL=http://frost_http-web-1:8080/FROST-Server/v1.1
networks:
smartcity-shared:

29
docker-compose.redpanda-consumer.yml Normal file
View File

@@ -0,0 +1,29 @@
# Redpanda → InfluxDB Consumer
# Lit les topics Redpanda et écrit dans InfluxDB pour Grafana
version: "3.8"
services:
redpanda-consumer:
image: python:3.11-slim
container_name: smart-city-redpanda-consumer
restart: unless-stopped
command: >
sh -c "pip install requests && python3 /app/consumer.py"
volumes:
- ./redpanda/consumer.py:/app/consumer.py:ro
environment:
- INFLUX_URL=http://smart-city-influxdb:8086
- INFLUX_TOKEN=my-super-admin-token
- INFLUX_ORG=digitribe
- INFLUX_BUCKET=iot_data
networks:
- smartcity-shared
healthcheck:
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://smart-city-redpanda:9644/public_metrics')"]
interval: 30s
timeout: 10s
retries: 3
networks:
smartcity-shared:
external: true

110
prometheus.yml
View File

@@ -3,43 +3,93 @@ global:
evaluation_interval: 15s
scrape_configs:
# Mosquitto MQTT Broker
- job_name: 'mosquitto'
static_configs:
- targets: ['mosquitto-exporter:9234']
scrape_interval: 10s
# Orion-LD (FIWARE)
- job_name: 'orion-ld'
# ── Simulator (host) ─────────────────────────────────────────────────────────
- job_name: 'simulator'
static_configs:
- targets: ['fiware-gis-quickstart-orion-1:1026']
- targets: ['172.17.0.1:8001']
labels:
service: smart-city-simulator
environment: martinique
# ── EMQX ──────────────────────────────────────────────────────────────────
# EMQX v5 expose /api/v5/metrics (format Prometheus) — dispo via Traefik
# Activer dans EMQX: conf/api6 => metrics.enabled = true
# Note: endpoint non exposé publiquement par défaut → via smartcity-shared
# - job_name: 'emqx'
# metrics_path: '/api/v5/metrics'
# static_configs:
# - targets: ['emqx_emqx_1:8081']
# labels:
# service: emqx
# environment: martinique
# ── Mosquitto ─────────────────────────────────────────────────────────────
# Mosquitto n'a pas de /metrics natif → mosquitto_exporter (non déployé)
# ── BunkerM ──────────────────────────────────────────────────────────────
# BunkerM : vérifier si /metrics est exposé
# ── Stellio ───────────────────────────────────────────────────────────────
# Stellio actuator: vérifier activation dans docker-compose
# → actuator.prometheus.enabled=true dans application.yml
# - job_name: 'stellio'
# metrics_path: '/actuator/prometheus'
# static_configs:
# - targets: ['stellio-api-gateway:8080']
# labels:
# service: stellio
# environment: martinique
# ── Orion-LD ──────────────────────────────────────────────────────────────
# Orion-LD : compiler avec --with-metrics pour activer /metrics
# ── FROST-Server ──────────────────────────────────────────────────────────
# FROST : vérifier si /metrics est activé dans la config
# - job_name: 'frost'
# static_configs:
# - targets: ['frost_http-web-1:8080']
# labels:
# service: frost
# environment: martinique
# ── InfluxDB ──────────────────────────────────────────────────────────────
- job_name: 'influxdb'
metrics_path: '/metrics'
scrape_interval: 10s
# FROST-Server (SensorThings)
- job_name: 'frost-server'
static_configs:
- targets: ['frost_http-web-1:8080']
metrics_path: '/FROST-Server/metrics'
scrape_interval: 10s
- targets: ['smart-city-influxdb:8086']
labels:
service: influxdb
environment: martinique
# Stellio NGSI-LD
- job_name: 'stellio'
static_configs:
- targets: ['stellio:8080']
metrics_path: '/metrics'
scrape_interval: 10s
# Redpanda Metrics (Admin API)
# ── Redpanda ────────────────────────────────────────────────────────────────
# Redpanda broker expose /public_metrics sur le port admin 9644
- job_name: 'redpanda'
metrics_path: '/public_metrics'
static_configs:
- targets: ['smart-city-redpanda:9644']
metrics_path: '/metrics'
scrape_interval: 10s
labels:
service: redpanda
environment: martinique
# Pulsar Metrics (Admin API)
- job_name: 'pulsar'
# ── OpenRemote ────────────────────────────────────────────────────────────
# OpenRemote Manager : actuator.prometheus doit être configuré
# Dans OR 3.x, metrics disponibles via /actuator/prometheus si activé
# Note: endpoint non exposé via Traefik actuellement
# → Activer via la config Manager: management.endpoints.web.exposure.include=prometheus,health,info
# - job_name: 'openremote'
# metrics_path: '/actuator/prometheus'
# static_configs:
# - targets: ['openremote-manager-1:8080']
# labels:
# service: openremote
# environment: martinique
# ── Grafana ────────────────────────────────────────────────────────────────
# Grafana native /metrics (Plugin sidecar Prometheus)
- job_name: 'grafana'
static_configs:
- targets: ['smart-city-pulsar:8080']
metrics_path: '/metrics'
scrape_interval: 10s
- targets: ['smart-city-grafana:3000']
labels:
service: grafana
environment: martinique

19
pulsar/config/nginx.conf Normal file
View File

@@ -0,0 +1,19 @@
server {
listen 8080;
server_name _;
# Frontend static build
location / {
root /usr/share/nginx/html;
try_files $uri $uri/ /index.html;
}
# Proxy vers backend Pulsar Manager
location /pulsar-manager/ {
proxy_pass http://localhost:7750/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}

101
redpanda/consumer.py Normal file
View File

@@ -0,0 +1,101 @@
#!/usr/bin/env python3
"""
Redpanda Consumer → InfluxDB
Lit les topics Redpanda et écrit dans InfluxDB pour Grafana.
Architecture: Redpanda → Consumer → InfluxDB → Grafana
"""
import json, time, base64, threading
import urllib.request
from datetime import datetime, timezone
# Configuration via variables d'environnement
REDPANDA_BASE = "http://localhost:8082" # REST Proxy Redpanda
INFLUX_URL = "http://localhost:8086" # InfluxDB
INFLUX_TOKEN = "my-super-admin-token"
INFLUX_ORG = "digitribe"
INFLUX_BUCKET = "iot_data"
SENSOR_TOPICS = ["traffic", "air-quality", "parking", "noise", "weather", "light"]
def write_influxdb(sensor_type: str, payload: dict):
"""Écrit les données dans InfluxDB."""
try:
sid = payload.get("id", "")
sname = payload.get("name", sid)
lat = payload.get("lat", 14.6)
lon = payload.get("lon", -61.0)
# Extraire les champs numériques du payload
fields = {k: v for k, v in payload.items()
if isinstance(v, (int, float)) and k not in ("lat", "lon")}
if not fields:
return
points = []
for field, value in fields.items():
line = (
f"{sensor_type},sensor_id={sid},location={sname.replace(' ','_')} "
f"{field}={value},lat={lat},lon={lon}"
)
points.append(line)
data = "\n".join(points)
req = urllib.request.Request(
f"{INFLUX_URL}/api/v2/write?org={INFLUX_ORG}&bucket={INFLUX_BUCKET}",
data=data.encode(),
headers={"Authorization": f"Token {INFLUX_TOKEN}", "Content-Type": "text/plain; charset=utf-8"},
method="POST"
)
with urllib.request.urlopen(req, timeout=8) as resp:
if resp.status in (200, 204):
print(f" ✅ InfluxDB: {sensor_type}/{sid} → {len(fields)} fields")
return resp.status
except Exception as e:
print(f" ⚠️ InfluxDB → {e}")
return None
def consume_redpanda_topic(topic: str):
"""Consomme les derniers messages d'un topic Redpanda."""
try:
# Récupérer les offsets actuels
req = urllib.request.Request(
f"{REDPANDA_BASE}/topics/{topic}/offsets",
headers={"Accept": "application/json"},
)
with urllib.request.urlopen(req, timeout=5) as resp:
data = json.loads(resp.read().decode())
offsets = data.get("partitions", [])
if not offsets:
return
# Récupérer les derniers messages (50 derniers)
req2 = urllib.request.Request(
f"{REDPANDA_BASE}/topics/{topic}?offset=-50&limit=50",
headers={"Accept": "application/json"},
)
with urllib.request.urlopen(req2, timeout=8) as resp2:
result = json.loads(resp2.read().decode())
messages = result.get("messages", [])
for msg in messages:
if msg.get("value"):
b64 = msg["value"]
decoded = base64.b64decode(b64).decode()
payload = json.loads(decoded)
write_influxdb(topic, payload)
except Exception as e:
print(f" ⚠️ Redpanda/{topic} → {e}")
def poll_topics():
"""Boucle principale — poll toutes les 10 secondes."""
print("[REDKAN] Redpanda Consumer → InfluxDB")
print(f"[CFG] Topics: {SENSOR_TOPICS}")
print(f"[CFG] InfluxDB: {INFLUX_URL}")
while True:
for topic in SENSOR_TOPICS:
consume_redpanda_topic(topic)
print(f"[REDKAN] Cycle terminé — pause 10s")
time.sleep(10)
if __name__ == "__main__":
poll_topics()

180
session_resume_2026-05-05.md
View File

@@ -1,156 +1,34 @@
# Session Resume - 2026-05-05
# Session Resume - 2026-05-05 (Suite après crash)
## Objectif de la session
Configuration de l'ingestion de données pour le Smart City Digital Twin Martinique :
- Simulateur → Pulsar (port 6650)
- Pulsar → Service de Distribution → Brokers (MQTT, NGSI-LD, FROST)
- Monitoring via Redpanda Console, Prometheus, Grafana
## État au démarrage
- **Dernier commit** : `3b5ff8d` - READY FOR DEMO 9h00 - 10/10 services ✅ - 182 actions complètes
- **Services Docker UP** : Pulsar (6650), Redpanda (19092/9644), InfluxDB (8086), OpenRemote (8080/8405), FROST (8090), Stellio, GeoServer, Grafana (3001), etc.
- **Commits non poussés** : 6 commits en attente sur Gitea
## Réalisations ✅
## Services vérifiés
- ✅ smart-city-pulsar (port 6650 - binaire uniquement, pas d'API REST)
- ✅ smart-city-redpanda (port 19092 - producteur Kafka, port 19644 Console)
- ✅ smart-city-influxdb (port 8086)
- ✅ openremote-manager-1 (port 8080/8405)
- ✅ frost_allinone-web-1 (port 8090)
- ✅ stellio-api-gateway
- ✅ GeoServer, Grafana
### 1. Redpanda Console - OPÉRATIONNEL
- Service `smart-city-redpanda-console` créé dans `redpanda/docker-compose.yml`
- Accessible sur `http://localhost:28080` (200 OK)
- Traefik configuré : `https://redpanda-console.digitribe.fr`
- Connecté à Redpanda (`smart-city-redpanda:9092`)
- API Admin Redpanda activée (`http://smart-city-redpanda:9644`)
- Fichier config : `redpanda/console.yaml`
## Logs simulateur disponibles
- simulator_pulsar_success.log
- simulator_demo_final.log
- simulator_final_demo.log
- simulator_nohup.log
### 2. Prometheus - CONFIGURÉ
- Cibles actives ajoutées dans `prometheus.yml` :
- `redpanda` : **up** (métriques port 9644)
- `pulsar` : **up** (métriques port 8080)
- `mosquitto` : up
- `orion-ld` : up
- `frost-server` : down (normal, pas de données)
- `stellio` : down (normal, pas de données)
## Tâches restantes pour la démo de demain
- [ ] Vérifier que le simulateur envoie bien des données (Pulsar/Redpanda → Brokers → InfluxDB)
- [ ] Tester l'affichage sur Grafana / OpenRemote
- [ ] Pousser les 6 commits en attente vers Gitea
- [ ] Créer un script de démonstration rapide (30 secondes)
- [ ] Documenter les URLs d'accès pour la démo
### 3. Grafana Dashboards - CRÉÉS
- **Redpanda Metrics** (`grafana/provisioning/dashboards/redpanda-metrics.json`)
- **Pulsar Metrics** (`grafana/provisioning/dashboards/pulsar-metrics.json`)
- **Smart City Ingestion** (`grafana/provisioning/dashboards/smart-city-ingeston.json`)
- Datasources InfluxDB connectées : InfluxDB, InfluxDB-Simulator, InfluxDB-SmartCity
### 4. Simulateur → Pulsar - FONCTIONNEL
- Le simulateur utilise déjà le protocole binaire Pulsar (port 6650)
- Logs confirment les connexions : `Connected to broker pulsar://smart-city-pulsar:6650`
- Topics créés : `smartcity-traffic`, `smartcity-airquality`, `smartcity-parking`, `smartcity-noise`, `smartcity-weather`, `smartcity-light`
### 5. Service de Distribution - AJOUTÉ (mais instable)
- Service `pulsar-distribution` ajouté dans `pulsar/docker-compose.yml`
- Code : `pulsar/distribution.py` (consomme depuis Pulsar, republie vers brokers)
- Problème : Erreur docker-compose au redémarrage (`KeyError: 'ContainerConfig'`)
- URL FROST corrigée : `frost-api-8090:8080/FROST-Server/v1.1`
## Problèmes rencontrés ⚠️
### 1. Pulsar Manager - CRASH RÉCURRENT
- Conteneur `smart-city-pulsar-manager` crash au démarrage
- Erreurs : `Object 'ENVIRONMENTS' not found` (HerdDB), problèmes d'initialisation PostgreSQL
- Solution alternative : Utiliser l'API Pulsar Admin directe (`http://localhost:8080/admin/v2/...`)
### 2. Distribution Service - ERREUR DOCKER-COMPOSE
- `KeyError: 'ContainerConfig'` lors du `docker-compose up -d pulsar-distribution`
- Nécessite suppression manuelle du conteneur et reconstruction
- Service fonctionnel en théorie mais instable en pratique
### 3. InfluxDB - AUCUNE DONNÉE VISIBLE
- Simulateur configuré pour InfluxDB (`ENABLE_INFLUX=1`)
- Aucune donnée visible dans les queries InfluxDB
- À diagnostiquer : connectivité simulateur → InfluxDB
### 4. Traefik Let's Encrypt - ÉCHEC
- Problèmes de certificats sur `pulsar.digitribe.fr` et `redpanda-console.digitribe.fr`
- Cause probable : domaine non public ou configuration DNS
- Solution temporaire : accès HTTP direct (localhost:7750, localhost:28080)
## Fichiers modifiés/créés 📁
### Redpanda
- `redpanda/docker-compose.yml` : Ajout service `smart-city-redpanda-console`
- `redpanda/console.yaml` : Configuration Redpanda Console
### Pulsar
- `pulsar/docker-compose.yml` : Ajout service `pulsar-distribution`
- `pulsar/distribution.py` : Service de distribution (déjà existant)
### Prometheus
- `prometheus.yml` : Ajout cibles `pulsar` et `redpanda`
### Grafana
- `grafana/provisioning/dashboards/redpanda-metrics.json` : **Créé**
- `grafana/provisioning/dashboards/pulsar-metrics.json` : **Créé**
- `grafana/provisioning/dashboards/smart-city-ingeston.json` : **Créé**
## À faire pour la prochaine session 📋
### Priorité 1 : Ingestion de données
1. **Diagnostiquer InfluxDB** : Pourquoi aucune donnée n'arrive ?
- Vérifier les logs du simulateur (`docker logs smart-city-simulator | grep Influx`)
- Tester la connexion manuelle depuis le simulateur
- Vérifier le token InfluxDB et l'organisation
2. **Stabiliser le service de distribution**
- Corriger l'erreur `KeyError: 'ContainerConfig'`
- Lancer manuellement le conteneur si nécessaire
- Vérifier que les messages Pulsar sont bien republiés vers les brokers
### Priorité 2 : Monitoring et Visualisation
3. **Tester les dashboards Grafana**
- Accéder à http://localhost:3001 (admin/Digitribe972)
- Vérifier que les panels affichent des données (InfluxDB, Prometheus)
- Ajuster les requêtes Flux si nécessaire
4. **Corriger Pulsar Manager (optionnel)**
- Utiliser une base PostgreSQL externe propre
- Ou passer à une alternative (Kafka Manager, ou utiliser l'API directe)
### Priorité 3 : Traefik et Domaines
5. **Résoudre Let's Encrypt**
- Vérifier la configuration DNS pour `*.digitribe.fr`
- Tester l'accessibilité publique des services
- Configurer des certificats SSL valides
## Commandes utiles 🛠️
### Vérifier les services
```bash
cd ~/smart-city-digital-twin-martinique
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
```
### Voir les logs
```bash
docker logs smart-city-simulator --tail 50 | grep -E "(Pulsar|Influx|ERROR)"
docker logs smart-city-pulsar-distribution --tail 50
```
### Test Pulsar
```bash
curl http://localhost:8080/admin/v2/clusters
curl -s -o /dev/null -w "%{http_code}" http://localhost:28080 # Redpanda Console
```
### Test InfluxDB
```bash
curl -s -H "Authorization: Token my-super-secret-admin-token" \
"http://smart-city-influxdb:8086/api/v2/query?org=digitribe" \
-d 'from(bucket:"iot_data") |> range(start:-1h) |> limit(n:5)'
```
## URLs d'accès 🌐
- **Redpanda Console** : http://localhost:28080
- **Grafana** : http://localhost:3001 (admin/Digitribe972)
- **Prometheus** : http://localhost:9090
- **Pulsar Admin API** : http://localhost:8080/admin/v2/clusters
- **FROST-Server** : http://localhost:8090/FROST-Server/v1.1
## Notes importantes 📝
- Le simulateur utilise le **protocole binaire Pulsar** (port 6650, pas 8080)
- L'ingestion centralisée passe par **Pulsar puis distribution** vers les brokers
- Redpanda Console est fonctionnel et permet de monitorer les topics Kafka
- Les dashboards Grafana sont prêts mais nécessitent des données pour être utiles
- Pulsar Manager reste instable, privilégier l'API Pulsar directe pour le monitoring
---
*Session du 2026-05-05 - Digitribe Martinique*
## Infos critiques (Mémoire)
- Pulsar standalone = port 6650 uniquement (pas d'API REST /produce)
- Redpanda : utiliser `rpk` ou producteur Kafka standard
- Simulateur host mode : ENABLE_PULSAR=false, INFLUX_URL=http://localhost:8086
- OpenRemote : port 8080 host (OR_METRICS_ENABLED disabled)

399
simulator.py
View File

@@ -39,6 +39,10 @@ import urllib.request, urllib.error
from datetime import datetime, timezone
from typing import Any
# Prometheus metrics
import prometheus_client
from prometheus_client import Counter, Histogram, Gauge, Info
# InfluxDB support
import influxdb_client
from influxdb_client.client.write_api import SYNCHRONOUS
@@ -89,6 +93,104 @@ INFLUX_ORG = os.environ.get("INFLUX_ORG", "digitribe")
INFLUX_BUCKET = os.environ.get("INFLUX_BUCKET", "iot_data")
INFLUX_TOKEN = os.environ.get("INFLUX_TOKEN", "my-super-secret-admin-token")
# Prometheus metrics HTTP server
METRICS_PORT = int(os.environ.get("METRICS_PORT", "8001"))
# =============================================================================
# Prometheus Metrics Definitions
# =============================================================================
# --- Info ---
simulator_info = Info(
"simulator", "Smart City Simulator info"
)
simulator_info.info({
"version": "1.0.0",
"python_version": sys.version.split()[0],
"mqtt_brokers": "emqx,mosquitto,bunkerm",
"context_brokers": "orion_ld,stellio,frost",
})
# --- Counters ---
messages_published_total = Counter(
"simulator_messages_published_total",
"Total messages published by broker",
["broker", "sensor_type"]
)
messages_errors_total = Counter(
"simulator_messages_errors_total",
"Total publish errors",
["broker", "sensor_type", "error_type"]
)
mqtt_connection_total = Counter(
"simulator_mqtt_connection_total",
"MQTT connection attempts",
["broker", "status"] # status: success, failure
)
http_requests_total = Counter(
"simulator_http_requests_total",
"HTTP requests to REST APIs",
["broker", "method", "status_code"]
)
influx_write_total = Counter(
"simulator_influx_write_total",
"InfluxDB write operations",
["status"] # success, error
)
# --- Gauges ---
mqtt_broker_connected = Gauge(
"simulator_mqtt_broker_connected",
"MQTT broker connection status (1=connected, 0=disconnected)",
["broker"]
)
sensors_total = Gauge(
"simulator_sensors_total",
"Total number of sensors by type",
["sensor_type"]
)
up = Gauge(
"simulator_up",
"Simulator is running (1=yes, 0=no)"
)
# --- Histograms ---
publish_duration = Histogram(
"simulator_publish_duration_seconds",
"Time spent publishing a message",
["broker"],
buckets=(0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5)
)
http_request_duration = Histogram(
"simulator_http_request_duration_seconds",
"HTTP request latency to REST APIs",
["broker", "method"],
buckets=(0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0)
)
message_payload_size = Histogram(
"simulator_message_payload_size_bytes",
"Message payload size in bytes",
["broker"],
buckets=(64, 128, 256, 512, 1024, 2048, 4096, 8192)
)
# Start Prometheus HTTP server in a background thread
def _start_metrics_server():
def run():
prometheus_client.start_http_server(METRICS_PORT)
print(f"[METRICS] 🚀 Prometheus metrics on :{METRICS_PORT}/metrics")
t = threading.Thread(target=run, daemon=True)
t.start()
return t
# Initialize InfluxDB client
_influx_client = None
_influx_write_api = None
@@ -124,54 +226,80 @@ if "SENSOR_COUNT" in os.environ:
# Coordonnées réelles Martinique (terre ferme uniquement)
# Martinique : 14.4°N–14.9°N, -61.23°W–-60.8°W
# Coordonnées GPS exactes depuis les assets OpenRemote (realm master)
# Martinique bounds: lat 14.37–14.88°N, lon 61.0–61.25°W
FIXED_LOCATIONS: dict[str, dict[str, tuple[float, float]]] = {
"traffic": {
# Fort-de-France — grands axes
"Carrefour Central": (14.6036, -61.1783), # Place du Palais, centre-ville
"Avenue des Caraïbes": (14.6100, -61.1850), # Route de Schoelcher (N1)
"Boulevard Pasteur": (14.6150, -61.1700), # Boulevard Pasteur, nord FdF
"Rue des Flamboyants": (14.5970, -61.1900), # Zone Industrielle, Lamentin
"Place de la République": (14.6000, -61.2100), # Centre administratif, sud FdF
# OpenRemote: "Traffic Fort-de-France Centre"
"FdF Centre": (14.6036, -61.1783),
# OpenRemote: "Traffic Fort-de-France North"
"FdF North": (14.6200, -61.1700),
# OpenRemote: "Traffic Fort-de-France South"
"FdF South": (14.5900, -61.1900),
# OpenRemote: "trafficFlow - Fort-de-France"
"FdF Centre Rue": (14.6036, -61.1783),
# OpenRemote: "Test Sensor"
"FdF Place": (14.6000, -61.2000),
},
"airquality": {
# Points de mesure qualité de l'air sur terre
"Quartier Bonde": (14.6050, -61.1750), # Bonde, sud-est FdF
"Port de Fort-de-France": (14.5980, -61.2250), # Zone portuaire, bord de mer (OK, port ≠ mer)
"Château Denis": (14.6200, -61.1550), # Château Denis, nord montagne
"Lamentin Aéroport": (14.5950, -61.1700), # Aéroport, Lamentin
"Schoelcher Village": (14.7400, -61.1850), # Schoelcher, nord-ouest
# OpenRemote: "Air Quality Fort-de-France"
"FdF Centre": (14.6036, -61.1783),
# OpenRemote: "airQuality - Fort-de-France"
"FdF Bonde": (14.6050, -61.1750),
# OpenRemote: "airQuality - Sainte-Luce"
"Sainte-Luce": (14.5950, -61.1700),
# OpenRemote: "floodLevel - Schoelcher"
"Schoelcher": (14.7400, -61.1850),
# OpenRemote: "humidity - Le Robert"
"Le Robert": (14.6800, -60.9400),
},
"parking": {
# Parkings publics sur terre
"Parking Rivière-Saleé": (14.5820, -61.2050), # Rivière-Salée (sud)
"Parking Cluny": (14.6050, -61.1750), # Cluny, FdF
"Parking Média": (14.6000, -61.1850), # Quartier Média, FdF
"Parking Grand-Camp": (14.6100, -61.1700), # Grand-Camp, Lamentin
"Parking Dillon": (14.6200, -61.1650), # Dillon, nord FdF
# OpenRemote: "Parking Fort-de-France Centre"
"FdF Centre": (14.6036, -61.1783),
# OpenRemote: "parkingAvailability - Fort-de-France"
"FdF Bonde": (14.6050, -61.1750),
# OpenRemote: "Test Sensor"
"FdF Cluny": (14.6000, -61.2000),
# OpenRemote: "Traffic Fort-de-France South"
"FdF Sud": (14.5900, -61.1900),
# OpenRemote: "Weather Lamentin Airport"
"Lamentin": (14.5950, -61.1700),
},
"noise": {
# Zones urbainesbruyantes
"Rue des Arts": (14.6020, -61.1800), # Rue des Arts, centre FdF
"Marché Central": (14.6000, -61.2100), # Marché Central, FdF
"Université Fort-de-France": (14.6400, -61.1600), # Campus Schoe, nord
"Stade de Dillon": (14.6250, -61.1600), # Stade Dillon, nord
"Place du Champs de Mars": (14.6030, -61.1750), # Champs de Mars, FdF
# OpenRemote: "Noise Fort-de-France Centre"
"FdF Centre": (14.6036, -61.1783),
# OpenRemote: "Traffic Fort-de-France Centre"
"FdF Rue": (14.6036, -61.1783),
# OpenRemote: "trafficFlow - Fort-de-France"
"FdF Pasteur": (14.6200, -61.1700),
# OpenRemote: "temperature - Lamentin"
"Lamentin": (14.5950, -61.1650),
# OpenRemote: "temperature - Le Robert"
"Le Robert": (14.6776, -60.9395),
},
"weather": {
# Stations météo — terre ferme uniquement
"Station Météo Lamentin": (14.5950, -61.1650), # Aéroport Lamentin
"Station Schoelcher": (14.7350, -61.1800), # Schoelcher, NW
"Station Ajoupa-Bouillon": (14.8100, -61.0500), # Ajoupa-Bouillon, nord (interieur)
"Station Le François": (14.6150, -60.9000), # Le François, côte atlantique est
"Station Le Robert": (14.6800, -60.9400), # Le Robert, côte atlantique
# OpenRemote: "Weather Lamentin Airport"
"Lamentin": (14.5950, -61.1700),
# OpenRemote: "temperature - Lamentin"
"Lamentin Ville": (14.5950, -61.1650),
# OpenRemote: "temperature - Le Robert"
"Le Robert": (14.6776, -60.9395),
# OpenRemote: "humidity - Le Robert"
"Le Robert Hum": (14.6800, -60.9400),
# OpenRemote: "floodLevel - Schoelcher"
"Schoelcher": (14.7400, -61.1850),
},
"light": {
# Éclairage public — zones urbaines
"Eclairage Rue des Mouettes": (14.6050, -61.1800), # Rue des Mouettes, FdF
"Candela Boulevard": (14.6150, -61.1700), # Boulevard Pasteur
"Lumiere Rue des Acacias": (14.6000, -61.1850), # Rue des Acacias, FdF
"Feux Signalisation Centre": (14.6030, -61.1780), # Carrefours centraux
"Eclairage Port": (14.5980, -61.2250), # Zone portuaire
# OpenRemote: "Light Fort-de-France"
"FdF Centre": (14.6036, -61.1783),
# OpenRemote: "lightIntensity - Fort-de-France"
"FdF Bonde": (14.6050, -61.1800),
# OpenRemote: "Traffic Fort-de-France North"
"FdF North": (14.6200, -61.1700),
# OpenRemote: "Traffic Fort-de-France South"
"FdF South": (14.5900, -61.1900),
# OpenRemote: "airQuality - Sainte-Luce"
"Sainte-Luce": (14.5950, -61.1700),
},
}
@@ -382,30 +510,35 @@ def _frost_payload(sid: str, sensor: dict, source: str = "simulator", topic: str
# =============================================================================
# HTTP helper
# =============================================================================
def _http_post(url: str, data: dict, headers: dict) -> str:
def _http_post(url: str, data: dict, headers: dict, broker: str = "unknown") -> str:
"""POST et retourne 'ok' ou 'created' (ou '' si échec)."""
try:
body = json.dumps(data).encode()
req = urllib.request.Request(url, data=body, headers=headers, method="POST")
with urllib.request.urlopen(req, timeout=8) as resp:
if resp.status == 204:
return 'created' # No Content — succès
if resp.status not in (200, 201):
return ''
# Lire le corps pour extraire l'ID (FROST)
with http_request_duration.labels(broker=broker, method="POST").time():
try:
result = json.loads(resp.read())
if '@iot.selfLink' in result:
link = result['@iot.selfLink']
return link.split('(')[1].rstrip(')')
if '@iot.id' in result:
return str(result['@iot.id'])
with urllib.request.urlopen(req, timeout=8) as resp:
http_requests_total.labels(broker=broker, method="POST", status_code=str(resp.status)).inc()
if resp.status == 204:
return 'created' # No Content — succès
if resp.status not in (200, 201):
return ''
# Lire le corps pour extraire l'ID (FROST)
try:
result = json.loads(resp.read())
if '@iot.selfLink' in result:
link = result['@iot.selfLink']
return link.split('(')[1].rstrip(')')
if '@iot.id' in result:
return str(result['@iot.id'])
except Exception:
pass
location = resp.headers.get('Location', '')
if location:
return location.split('(')[1].rstrip(')') if '(' in location else ''
return 'created'
except Exception:
pass
location = resp.headers.get('Location', '')
if location:
return location.split('(')[1].rstrip(')') if '(' in location else ''
return 'created'
except urllib.error.HTTPError as e:
# Lire le corps de l'erreur pour debug
try:
@@ -413,23 +546,33 @@ def _http_post(url: str, data: dict, headers: dict) -> str:
except Exception:
err_body = str(e)
print(f" ⚠️ HTTP POST {url} → {e.code}: {err_body}")
http_requests_total.labels(broker=broker, method="POST", status_code=str(e.code)).inc()
messages_errors_total.labels(broker=broker, sensor_type="http", error_type="http_error").inc()
return ''
except Exception as e:
http_requests_total.labels(broker=broker, method="POST", status_code="exception").inc()
messages_errors_total.labels(broker=broker, sensor_type="http", error_type="exception").inc()
print(f" ⚠️ HTTP POST {url} → {e}")
return ''
def _http_put(url: str, data: dict, headers: dict) -> bool:
def _http_put(url: str, data: dict, headers: dict, broker: str = "unknown") -> bool:
try:
body = json.dumps(data).encode()
req = urllib.request.Request(url, data=body, headers=headers, method="PUT")
with urllib.request.urlopen(req, timeout=5) as resp:
return resp.status in (200, 204)
with http_request_duration.labels(broker=broker, method="PUT").time():
with urllib.request.urlopen(req, timeout=5) as resp:
http_requests_total.labels(broker=broker, method="PUT", status_code=str(resp.status)).inc()
return resp.status in (200, 204)
except urllib.error.HTTPError as e:
http_requests_total.labels(broker=broker, method="PUT", status_code=str(e.code)).inc()
if e.code == 409:
return True # Already exists - that's fine
messages_errors_total.labels(broker=broker, sensor_type="http", error_type="http_error").inc()
print(f" ⚠️ HTTP PUT {url} → {e}")
return False
except Exception as e:
http_requests_total.labels(broker=broker, method="PUT", status_code="exception").inc()
messages_errors_total.labels(broker=broker, sensor_type="http", error_type="exception").inc()
print(f" ⚠️ HTTP PUT {url} → {e}")
return False
@@ -469,14 +612,19 @@ class MultiMQTT:
with self._lock:
if rc == 0:
self.ok[name] = True
mqtt_broker_connected.labels(broker=name).set(1)
mqtt_connection_total.labels(broker=name, status="success").inc()
print(f"[MQTT] ✅ {name} connecté")
else:
self.ok[name] = False
mqtt_broker_connected.labels(broker=name).set(0)
mqtt_connection_total.labels(broker=name, status="failure").inc()
print(f"[MQTT] ❌ {name} rc={rc}")
def _on_disconnect(self, name: str):
with self._lock:
self.ok[name] = False
mqtt_broker_connected.labels(broker=name).set(0)
print(f"[MQTT] ⚠️ {name} déconnecté")
def _setup(self):
@@ -493,16 +641,25 @@ class MultiMQTT:
self.ok[name] = False
time.sleep(3) # Attend les connexions
def publish(self, topic: str, payload: str) -> dict[str, bool]:
def publish(self, topic: str, payload: str, sensor_type: str = "unknown") -> dict[str, bool]:
results = {}
payload_bytes = len(payload.encode())
with self._lock:
for name, client in self.clients.items():
if self.ok.get(name, False):
try:
r = client.publish(topic, payload, qos=1)
results[name] = (r.rc == mqtt.MQTT_ERR_SUCCESS)
except Exception:
results[name] = False
with publish_duration.labels(broker=name).time():
try:
r = client.publish(topic, payload, qos=1)
success = (r.rc == mqtt.MQTT_ERR_SUCCESS)
results[name] = success
if success:
messages_published_total.labels(broker=name, sensor_type=sensor_type).inc()
message_payload_size.labels(broker=name).observe(payload_bytes)
else:
messages_errors_total.labels(broker=name, sensor_type=sensor_type, error_type="mqtt_rc").inc()
except Exception:
results[name] = False
messages_errors_total.labels(broker=name, sensor_type=sensor_type, error_type="exception").inc()
else:
results[name] = False
return results
@@ -546,28 +703,38 @@ def publish_stellio(sid: str, sensor: dict) -> bool:
try:
body = json.dumps(entity).encode()
req = urllib.request.Request(url, data=body, headers=headers, method="POST")
with urllib.request.urlopen(req, timeout=8) as resp:
print(f" 🏢 Stellio: ✅ (HTTP {resp.status})")
return True
with http_request_duration.labels(broker="stellio", method="POST").time():
with urllib.request.urlopen(req, timeout=8) as resp:
http_requests_total.labels(broker="stellio", method="POST", status_code=str(resp.status)).inc()
print(f" 🏢 Stellio: ✅ (HTTP {resp.status})")
return True
except urllib.error.HTTPError as e:
http_requests_total.labels(broker="stellio", method="POST", status_code=str(e.code)).inc()
if e.code == 409: # Already exists, do update with PUT
try:
entity_id = urllib.parse.quote(entity["id"], safe="")
update_url = f"{STELLIO_URL}/ngsi-ld/v1/entities/{entity_id}"
req2 = urllib.request.Request(update_url, data=body, headers=headers, method="PUT")
with urllib.request.urlopen(req2, timeout=8) as resp2:
print(f" 🏢 Stellio: ✅ (HTTP {resp2.status} updated)")
return True
with http_request_duration.labels(broker="stellio", method="PUT").time():
with urllib.request.urlopen(req2, timeout=8) as resp2:
http_requests_total.labels(broker="stellio", method="PUT", status_code=str(resp2.status)).inc()
print(f" 🏢 Stellio: ✅ (HTTP {resp2.status} updated)")
return True
except Exception as e2:
http_requests_total.labels(broker="stellio", method="PUT", status_code="error").inc()
messages_errors_total.labels(broker="stellio", sensor_type=stype, error_type="http_error").inc()
print(f" ⚠️ Stellio update failed: {e2}")
return False
try:
err = e.read().decode()[:300]
except Exception:
err = str(e)
messages_errors_total.labels(broker="stellio", sensor_type=stype, error_type="http_error").inc()
print(f" ⚠️ Stellio → {e.code}: {err}")
return False
except Exception as e:
http_requests_total.labels(broker="stellio", method="POST", status_code="exception").inc()
messages_errors_total.labels(broker="stellio", sensor_type=stype, error_type="exception").inc()
print(f" ⚠️ Stellio → {e}")
return False
@@ -584,25 +751,32 @@ def publish_orion(sid: str, sensor: dict) -> bool:
body = json.dumps(entity).encode()
req = urllib.request.Request(f"{base}/entities", data=body,
headers={"Content-Type": "application/ld+json", "Accept": "application/ld+json"}, method="POST")
with urllib.request.urlopen(req, timeout=8) as resp:
print(f" 🌐 Orion-LD: ✅ (HTTP {resp.status} created)")
return True
with http_request_duration.labels(broker="orion_ld", method="POST").time():
with urllib.request.urlopen(req, timeout=8) as resp:
http_requests_total.labels(broker="orion_ld", method="POST", status_code=str(resp.status)).inc()
print(f" 🌐 Orion-LD: ✅ (HTTP {resp.status} created)")
return True
except urllib.error.HTTPError as e:
http_requests_total.labels(broker="orion_ld", method="POST", status_code=str(e.code)).inc()
if e.code != 409:
messages_errors_total.labels(broker="orion_ld", sensor_type=stype, error_type="http_error").inc()
print(f" ⚠️ Orion-LD → {e.code}: {e.read().decode()[:200]}")
return False
# 409 = déjà existant → PATCH
# 2. Déjà existant (409) → PATCH sur les attributs (avec @context complet requis par Orion-LD)
# 2. Déjà existant (409) → PATCH sur les attributs
try:
# Orion-LD exige @context même dans le PATCH
eid = urllib.parse.quote(entity['id'], safe='')
patch_url = f"{base}/entities/{eid}/attrs"
req2 = urllib.request.Request(patch_url, data=body,
headers={"Content-Type": "application/ld+json", "Accept": "application/ld+json"}, method="PATCH")
with urllib.request.urlopen(req2, timeout=8) as resp2:
print(f" 🌐 Orion-LD: ✅ (HTTP {resp2.status} updated)")
return True
with http_request_duration.labels(broker="orion_ld", method="PATCH").time():
with urllib.request.urlopen(req2, timeout=8) as resp2:
http_requests_total.labels(broker="orion_ld", method="PATCH", status_code=str(resp2.status)).inc()
print(f" 🌐 Orion-LD: ✅ (HTTP {resp2.status} updated)")
return True
except Exception as e2:
http_requests_total.labels(broker="orion_ld", method="PATCH", status_code="error").inc()
messages_errors_total.labels(broker="orion_ld", sensor_type=stype, error_type="http_error").inc()
print(f" ⚠️ Orion-LD PATCH failed: {e2}")
return False
@@ -650,10 +824,15 @@ def publish_bunkerm(sid: str, sensor: dict, values: dict) -> bool:
method="POST"
)
try:
with opener.open(req, timeout=5) as resp:
print(f" ✅ BunkerM: HTTP {resp.status}")
return resp.status in (200, 201, 204)
with http_request_duration.labels(broker="bunkerm", method="POST").time():
with opener.open(req, timeout=5) as resp:
http_requests_total.labels(broker="bunkerm", method="POST", status_code=str(resp.status)).inc()
messages_published_total.labels(broker="bunkerm", sensor_type=sensor["type"]).inc()
print(f" ✅ BunkerM: HTTP {resp.status}")
return resp.status in (200, 201, 204)
except Exception as e:
http_requests_total.labels(broker="bunkerm", method="POST", status_code="exception").inc()
messages_errors_total.labels(broker="bunkerm", sensor_type=sensor["type"], error_type="exception").inc()
print(f" ⚠️ BunkerM POST → {e}")
return False
@@ -678,7 +857,7 @@ def publish_frost(sid: str, sensor: dict, field: str, value: float) -> bool:
}
}
}
if _http_post(obs_url, obs, FROST_HEADERS):
if _http_post(obs_url, obs, FROST_HEADERS, broker="frost"):
print(f" ✅ FROST Observation {sid}/{field} → OK (cached)")
return True
else:
@@ -691,7 +870,7 @@ def publish_frost(sid: str, sensor: dict, field: str, value: float) -> bool:
topic = f"city/sensors/{stype}/{sid}"
thing_payload, datastreams = _frost_payload(sid, sensor, source="simulator", topic=topic)
print(f" 📊 FROST: POST Thing {sid}...")
tid = _http_post(f"{FROST_URL}/Things", thing_payload, FROST_HEADERS)
tid = _http_post(f"{FROST_URL}/Things", thing_payload, FROST_HEADERS, broker="frost")
if not tid:
print(f" ⚠️ FROST Thing {sid} → échec création")
return False
@@ -702,7 +881,7 @@ def publish_frost(sid: str, sensor: dict, field: str, value: float) -> bool:
for f, ds, _ in datastreams:
ds["Thing"] = {"@iot.id": tid}
print(f" 📊 FROST: POST Datastream {sid}/{f}...")
ds_id = _http_post(f"{FROST_URL}/Datastreams", ds, FROST_HEADERS)
ds_id = _http_post(f"{FROST_URL}/Datastreams", ds, FROST_HEADERS, broker="frost")
if ds_id:
print(f" ✅ FROST Datastream {sid}/{f} créé (ID: {ds_id})")
ds_map[f] = ds_id
@@ -716,7 +895,7 @@ def publish_frost(sid: str, sensor: dict, field: str, value: float) -> bool:
ds_id = ds_map[field]
obs_url = f"{FROST_URL}/Datastreams({ds_id})/Observations"
obs = {"resultTime": datetime.now(timezone.utc).isoformat(), "result": value}
if _http_post(obs_url, obs, FROST_HEADERS):
if _http_post(obs_url, obs, FROST_HEADERS, broker="frost"):
print(f" ✅ FROST Observation {sid}/{field} → OK")
return True
return False
@@ -772,12 +951,19 @@ def _or_put(asset_id: str, payload: dict) -> bool:
"If-Match": str(payload.get("version", 1)),
},
method="PUT")
with urllib.request.urlopen(req, timeout=5) as resp:
return resp.status in (200, 204)
with http_request_duration.labels(broker="openremote", method="PUT").time():
with urllib.request.urlopen(req, timeout=5) as resp:
http_requests_total.labels(broker="openremote", method="PUT", status_code=str(resp.status)).inc()
messages_published_total.labels(broker="openremote", sensor_type=payload.get("type", "unknown")).inc()
return resp.status in (200, 204)
except urllib.error.HTTPError as e:
http_requests_total.labels(broker="openremote", method="PUT", status_code=str(e.code)).inc()
messages_errors_total.labels(broker="openremote", sensor_type=payload.get("type", "unknown"), error_type="http_error").inc()
print(f" ⚠️ OR PUT {asset_id} → HTTP {e.code}")
return False
except Exception as e:
http_requests_total.labels(broker="openremote", method="PUT", status_code="exception").inc()
messages_errors_total.labels(broker="openremote", sensor_type=payload.get("type", "unknown"), error_type="exception").inc()
print(f" ⚠️ OR PUT {asset_id} → {e}")
return False
@@ -862,17 +1048,20 @@ def _init_pulsar() -> bool:
def publish_pulsar(sid: str, sensor: dict, payload: dict) -> bool:
"""Publie un message sur Pulsar via le client Python (port binaire 6650)."""
stype = sensor["type"]
topic = f"persistent://public/default/smartcity-{stype}"
topic = f"persistent://public/default/smartcity-{stype.replace('-','')}"
try:
import pulsar
# Utiliser le client Pulsar binaire (socket 6650)
client = pulsar.Client(f"pulsar://{PULSAR_HOST}:6650")
producer = client.create_producer(topic)
body = json.dumps(payload, ensure_ascii=False).encode()
producer.send(body, properties={"sensor_id": sid, "source": "simulator"})
client.close()
with publish_duration.labels(broker="pulsar").time():
client = pulsar.Client(f"pulsar://{PULSAR_HOST}:6650")
producer = client.create_producer(topic)
body = json.dumps(payload, ensure_ascii=False).encode()
producer.send(body, properties={"sensor_id": sid, "source": "simulator"})
client.close()
messages_published_total.labels(broker="pulsar", sensor_type=stype).inc()
message_payload_size.labels(broker="pulsar").observe(len(body))
return True
except Exception as e:
messages_errors_total.labels(broker="pulsar", sensor_type=stype, error_type="exception").inc()
print(f" ⚠️ Pulsar → {e}")
return False
@@ -922,18 +1111,27 @@ def publish_redpanda(sid: str, sensor: dict, payload: dict) -> bool:
headers={"Content-Type": "application/vnd.kafka.json.v2+json"},
method="POST"
)
with urllib.request.urlopen(req, timeout=8) as resp:
return resp.status in (200, 201, 204)
with http_request_duration.labels(broker="redpanda", method="POST").time():
with urllib.request.urlopen(req, timeout=8) as resp:
http_requests_total.labels(broker="redpanda", method="POST", status_code=str(resp.status)).inc()
messages_published_total.labels(broker="redpanda", sensor_type=stype).inc()
message_payload_size.labels(broker="redpanda").observe(len(body.encode()))
return resp.status in (200, 201, 204)
except urllib.error.HTTPError as e:
http_requests_total.labels(broker="redpanda", method="POST", status_code=str(e.code)).inc()
messages_errors_total.labels(broker="redpanda", sensor_type=stype, error_type="http_error").inc()
print(f" ⚠️ Redpanda → {e.code}")
return False
except Exception as e:
http_requests_total.labels(broker="redpanda", method="POST", status_code="exception").inc()
messages_errors_total.labels(broker="redpanda", sensor_type=stype, error_type="exception").inc()
print(f" ⚠️ Redpanda → {e}")
return False
def publish_influx(sid: str, sensor: dict, values: dict) -> bool:
"""Write sensor data to InfluxDB (async, non-blocking)."""
if not _influx_write_api:
influx_write_total.labels(status="skipped").inc()
return False
def _write_async():
@@ -955,8 +1153,10 @@ def publish_influx(sid: str, sensor: dict, values: dict) -> bool:
if points:
_influx_write_api.write(bucket=INFLUX_BUCKET, record=points)
influx_write_total.labels(status="success").inc()
print(f" 📈 InfluxDB: {len(points)} points written")
except Exception as e:
influx_write_total.labels(status="error").inc()
print(f" ⚠️ InfluxDB → {e}")
# Exécution asynchrone (non-bloquante)
@@ -972,6 +1172,14 @@ def main():
print(f"[CFG] Orion-LD: {ENABLE_ORION} | Stellio: {ENABLE_STELLIO} | FROST: {ENABLE_FROST}")
print(f"[CFG] InfluxDB: {ENABLE_INFLUX} | Pulsar: {ENABLE_PULSAR} | Redpanda: {ENABLE_REDPANDA}")
# --- Démarrer le serveur Prometheus ---
_start_metrics_server()
# --- Configurer les gauges ---
for stype, count in SENSOR_COUNTS.items():
sensors_total.labels(sensor_type=stype).set(count)
up.set(1)
# Init connectivity checks
if ENABLE_PULSAR:
_init_pulsar()
@@ -989,6 +1197,7 @@ def main():
def signal_handler(*_):
nonlocal running
running = False
up.set(0)
print("\n[SIM] 🛑 Arrêt...")
signal.signal(signal.SIGINT, signal_handler)
signal.signal(signal.SIGTERM, signal_handler)
@@ -1025,7 +1234,7 @@ def main():
msg = json.dumps(payload_mqtt, ensure_ascii=False)
# --- MQTT publish ---
results = mqtt_client.publish(topic, msg)
results = mqtt_client.publish(topic, msg, sensor_type=stype)
ok_mqtt = [n for n, r in results.items() if r]
if ok_mqtt:
print(f" 📤 {topic} → {','.join(ok_mqtt)}")

1488
simulator_final_demo.log Normal file
View File

File diff suppressed because it is too large Load Diff