fix: QuantumLeap + Redis + simulateur MQTT-only + données test CrateDB
This commit is contained in:
Eric FELIXINE
@@ -1,89 +1,83 @@
|
|||||||
# RESUMÉ FINAL - Smart City Digital Twin (06 Mai 2026 - 18h15)
|
# Résumé Final - Smart City Digital Twin (06 Mai 2026 - 19h30)
|
||||||
|
|
||||||
## ✅ RÉALISÉ (en parallèle)
|
## ✅ Ce qui fonctionne
|
||||||
|
1. **MQTT Brokers** : EMQX (11883), Mosquitto (1883), BunkerM (1900) - OK
|
||||||
|
2. **IoT-Agents** : Reçoivent les données MQTT et mettent à jour Orion-LD - OK
|
||||||
|
3. **Orion-LD** : Contient les entités (vérifier via `curl http://localhost:1026/v2/entities`)
|
||||||
|
4. **CrateDB** : Fonctionne parfaitement (INSERT manuel OK)
|
||||||
|
5. **Grafana** : Datasources CrateDB configurées (IDs 23, 24)
|
||||||
|
6. **Redis** : Installé et accessible par QuantumLeap
|
||||||
|
|
||||||
### 1. Pipelines complètes
|
## ❌ Problème bloquant : QuantumLeap → CrateDB
|
||||||
**Pipeline 1 : Orion-LD**
|
**Symptômes :**
|
||||||
- 3 IoT-Agents (EMQX:4041, Mosquitto:4042, BunkerM:4043) ✅
|
- QuantumLeap reçoit les notifications (`/v2/notify` → "Notification successfully processed")
|
||||||
- Orion-LD (port 1026) ✅
|
- Aucune donnée insérée dans CrateDB (`quantumleap.etairqualityobserved`)
|
||||||
- QuantumLeap-Orion (port 8668) ✅
|
- La queue Redis reste vide (`rq:queue:default` n'existe pas)
|
||||||
- CrateDB-Orion (port 5432/4200) ✅ **Contient des données** (`quantumleap.etairqualityobserved`)
|
- `WQ_OFFLOAD_WORK=True` est activé mais les tâches ne sont pas ajoutées à la queue
|
||||||
|
|
||||||
**Pipeline 2 : Stellio**
|
**Investigation :**
|
||||||
- Stellio (port 8080) ✅
|
- `offload_to_work_queue()` retourne `True` ✅
|
||||||
- QuantumLeap-Stellio (port 8669) ✅
|
- `redis_connection()` utilise `REDIS_HOST=smart-city-redis` et `REDIS_PORT=6379` ✅
|
||||||
- CrateDB-Stellio (port 5433/4201) ✅ **Vide (à tester)**
|
- Worker RQ lancé et connecté ✅
|
||||||
|
- Mais `InsertAction.enqueue()` n'ajoute rien à la queue Redis
|
||||||
|
|
||||||
### 2. Grafana ✅
|
**Hypothèses :**
|
||||||
- URL: https://grafana.digitribe.fr
|
1. `InsertAction` n'est pas picklable (échec silencieux de `q.enqueue()`)
|
||||||
- **2 Datasources CrateDB** ajoutées :
|
2. Problème de connexion Redis dans `enqueue()`
|
||||||
- `CrateDB-SmartCity` (ID: 23) → smart-city-cratedb:5432, DB: quantumleap
|
3. La méthode `trans.insert()` échoue silencieusement
|
||||||
- `CrateDB-Stellio` (ID: 24) → smart-city-cratedb-stellio:5432, DB: quantumleap_stellio
|
4. Bug dans le module `wq` de QuantumLeap
|
||||||
|
|
||||||
### 3. Documentation (Gitea) ✅
|
## 🛠️ Solution temporaire (pour Grafana)
|
||||||
- ✅ `architecture-multi-cb.md` → Architecture 2 pipelines (corrigée)
|
Des données de test ont été insérées manuellement dans CrateDB :
|
||||||
- ✅ `data-flow-diagram.md` → Mermaid (3 IoT-Agents)
|
```sql
|
||||||
- ✅ `data-flow-diagram.html` → Version web
|
INSERT INTO quantumleap.etairqualityobserved (entity_id, time_index, no2, temperature, humidity) VALUES
|
||||||
- ✅ `session_resume_2026-05-06.md` → Résumé complet
|
('urn:ngsi-ld:AirQualityObserved:sensor001', 1778112000000, 45.5, 28.0, 85.0),
|
||||||
- ✅ `docker-compose.quantumleap-stellio.yml` → Nouveau fichier
|
...
|
||||||
- ✅ Tout commité et pushé sur Gitea
|
|
||||||
|
|
||||||
### 4. Problèmes résolus
|
|
||||||
- ✅ Variables QuantumLeap (`CRATE_HOST/PORT` au lieu de `QL_*`)
|
|
||||||
- ✅ Images Docker (`fiware/quantum-leap:latest` avec tiret)
|
|
||||||
- ✅ Alias réseau Orion-LD (`orion-ld` + `smart-city-orion-ld`)
|
|
||||||
- ✅ Datasources Grafana (2 CrateDB connectées)
|
|
||||||
|
|
||||||
## ⚠️ EN COURS / PROBLÈMES
|
|
||||||
|
|
||||||
1. **Stellio Subscription** : Échoue en 405 (API NGSI-LD)
|
|
||||||
- Cause probable : Endpoint ou format de requête incorrect
|
|
||||||
- Action : Rechercher la bonne API Stellio
|
|
||||||
|
|
||||||
2. **Orion-LD Subscriptions** : Retourne HTTP 000 (Timeout)
|
|
||||||
- Cause probable : Problème réseau interne (Orion → QuantumLeap)
|
|
||||||
- Action : Vérifier les logs Orion-LD
|
|
||||||
|
|
||||||
3. **Simulateur** : Doit publier sur `json/smartcity-api-key/{id}/attrs`
|
|
||||||
- Action : Modifier `simulator.py` si nécessaire
|
|
||||||
|
|
||||||
## 🎯 PROCHAINES ÉTAPES
|
|
||||||
|
|
||||||
1. **Tester Stellio** : Trouver la bonne API pour les subscriptions NGSI-LD
|
|
||||||
2. **Finaliser le flux** : Valider Stellio → QuantumLeap-Stellio → CrateDB-Stellio
|
|
||||||
3. **Créer les dashboards Grafana** : Utiliser les 2 sources CrateDB
|
|
||||||
4. **Simulateur** : S'assurer qu'il publie correctement sur les 3 brokers
|
|
||||||
|
|
||||||
## 📊 BILAN SERVICES
|
|
||||||
|
|
||||||
| Service | Container | Port | Statut | Datasource Grafana |
|
|
||||||
|---------|-----------|------|--------|-------------------|
|
|
||||||
| **Orion-LD** | smart-city-orion-ld | 1026 | ✅ Up | - |
|
|
||||||
| **QuantumLeap-Orion** | smart-city-quantumleap | 8668 | ✅ Up | - |
|
|
||||||
| **CrateDB-Orion** | smart-city-cratedb | 5432/4200 | ✅ Up | `CrateDB-SmartCity` (23) ✅ |
|
|
||||||
| **Stellio** | stellio-api-gateway | 8080 | ✅ Up | - |
|
|
||||||
| **QuantumLeap-Stellio** | smart-city-quantumleap-stellio | 8669 | ✅ Up | - |
|
|
||||||
| **CrateDB-Stellio** | smart-city-cratedb-stellio | 5433/4201 | ✅ Up | `CrateDB-Stellio` (24) ✅ |
|
|
||||||
| **3 IoT-Agents** | smart-city-iot-agent-* | 4041-4043 | ✅ Up | - |
|
|
||||||
| **Grafana** | smart-city-grafana | 3001 | ✅ Up | 2 sources ✅ |
|
|
||||||
|
|
||||||
## 📡 COMMANDES DE TEST (Flux complet)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# 1. Publier sur EMQX (avec préfixe json/)
|
|
||||||
mosquitto_pub -h localhost -p 11883 \
|
|
||||||
-t "json/smartcity-api-key/airquality_001/attrs" \
|
|
||||||
-m '{"NO2": 80.5, "temperature": 33.0, "humidity": 95.0}'
|
|
||||||
|
|
||||||
# 2. Vérifier Orion-LD
|
|
||||||
sleep 10 && curl -s http://localhost:1026/v2/entities
|
|
||||||
|
|
||||||
# 3. Vérifier CrateDB-Orion
|
|
||||||
docker exec smart-city-cratedb crash -c "SELECT * FROM quantumleap.etairqualityobserved LIMIT 5;"
|
|
||||||
|
|
||||||
# 4. Vérifier Grafana
|
|
||||||
curl -s http://localhost:3001/api/dashboards/home -u admin:Digitribe972
|
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Dashboards Grafana configurés :**
|
||||||
|
- Dashboard Orion-LD (ID: 21)
|
||||||
|
- Dashboard Stellio (ID: 22)
|
||||||
|
- Datasource CrateDB-SmartCity (ID: 23, port 5432)
|
||||||
|
- Datasource CrateDB-Stellio (ID: 24, port 5433)
|
||||||
|
|
||||||
|
## 📋 Actions pour finaliser
|
||||||
|
1. **Stellio Pipeline** :
|
||||||
|
- Corriger `docker-compose.quantumleap-stellio.yml` (CRATE_PORT=4200)
|
||||||
|
- Créer subscription Stellio → QuantumLeap-Stellio
|
||||||
|
- Vérifier `CrateDB-Stellio`
|
||||||
|
|
||||||
|
2. **QuantumLeap Debug** (à faire ultérieurement) :
|
||||||
|
- Vérifier si `InsertAction` est picklable
|
||||||
|
- Ajouter des logs dans `wq/core/task.py` (`enqueue()`)
|
||||||
|
- Tester `trans.insert()` manuellement avec un payload simple
|
||||||
|
- Consulter la documentation QuantumLeap / issues GitHub
|
||||||
|
|
||||||
|
3. **Simulateur** :
|
||||||
|
- `simulator.py` corrigé pour n'utiliser que MQTT (Orion/Stellio désactivés)
|
||||||
|
- MQTT OK, IoT-Agent OK, mais QuantumLeap ne traite pas les notifications
|
||||||
|
|
||||||
|
## 🔧 Commandes utiles
|
||||||
|
```bash
|
||||||
|
# Vérifier CrateDB
|
||||||
|
docker exec smart-city-cratedb crash -c "SELECT * FROM quantumleap.etairqualityobserved LIMIT 10;"
|
||||||
|
|
||||||
|
# Vérifier Redis
|
||||||
|
docker exec smart-city-redis redis-cli keys "*"
|
||||||
|
|
||||||
|
# Voir les logs QuantumLeap
|
||||||
|
docker logs smart-city-quantumleap --tail 100
|
||||||
|
|
||||||
|
# Tester notification manuelle
|
||||||
|
curl -s -X POST http://localhost:8668/v2/notify -H 'Content-Type: application/json' \
|
||||||
|
-d '{"subscriptionId": "test", "data": [{...}]}'
|
||||||
|
```
|
||||||
|
|
||||||
|
## 📊 Fichiers modifiés
|
||||||
|
- `docker-compose.quantumleap.yml` : +Redis, +healthcheck CrateDB, +variables environnement
|
||||||
|
- `simulator.py` : Orion-LD et Stellio désactivés (MQTT uniquement)
|
||||||
|
- `RESUME_FINAL_2026-05-06.md` : Ce fichier
|
||||||
|
|
||||||
---
|
---
|
||||||
*Dernière mise à jour : 06 Mai 2026, 18h15 - 2 Pipelines isolées, Grafana prêt*
|
**Prochaine étape** : Configurer la pipeline Stellio et finaliser les dashboards Grafana avec les données de test.
|
||||||
|
Le problème QuantumLeap nécessite une investigation plus poussée du code source (`wq` module).
|
||||||
|
|||||||
Binary file not shown.
34
docker-compose.mosquitto.yml
Normal file
34
docker-compose.mosquitto.yml
Normal file
@@ -0,0 +1,34 @@
|
|||||||
|
# Mosquitto Broker for Smart City Simulator
|
||||||
|
# Usage: docker compose -f docker-compose.yml -f docker-compose.mosquitto.yml up -d
|
||||||
|
version: '3.8'
|
||||||
|
|
||||||
|
services:
|
||||||
|
mosquitto:
|
||||||
|
image: eclipse-mosquitto:latest
|
||||||
|
container_name: smart-city-mosquitto
|
||||||
|
restart: unless-stopped
|
||||||
|
networks:
|
||||||
|
- smartcity-shared
|
||||||
|
- traefik-public
|
||||||
|
ports:
|
||||||
|
- "1883:1883"
|
||||||
|
- "9001:9001"
|
||||||
|
volumes:
|
||||||
|
- mosquitto-data:/mosquitto/data
|
||||||
|
- mosquitto-logs:/mosquitto/log
|
||||||
|
command: mosquitto -c /mosquitto/config/mosquitto.conf
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD", "mosquitto_sub", "-h", "localhost", "-p", "1883", "-t", "test", "-W", "1"]
|
||||||
|
interval: 30s
|
||||||
|
timeout: 10s
|
||||||
|
retries: 3
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
mosquitto-data:
|
||||||
|
mosquitto-logs:
|
||||||
|
|
||||||
|
networks:
|
||||||
|
smartcity-shared:
|
||||||
|
external: true
|
||||||
|
traefik-public:
|
||||||
|
external: true
|
||||||
@@ -1,15 +1,20 @@
|
|||||||
version: '3.8'
|
version: '3.8'
|
||||||
|
|
||||||
networks:
|
|
||||||
traefik-public:
|
|
||||||
external: true
|
|
||||||
smartcity-shared:
|
|
||||||
external: true
|
|
||||||
|
|
||||||
services:
|
services:
|
||||||
|
redis:
|
||||||
|
image: redis:7-alpine
|
||||||
|
container_name: smart-city-redis
|
||||||
|
networks:
|
||||||
|
- smartcity-shared
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD", "redis-cli", "ping"]
|
||||||
|
interval: 10s
|
||||||
|
timeout: 5s
|
||||||
|
retries: 5
|
||||||
|
|
||||||
cratedb:
|
cratedb:
|
||||||
container_name: smart-city-cratedb
|
|
||||||
image: crate:5.5
|
image: crate:5.5
|
||||||
|
container_name: smart-city-cratedb
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
environment:
|
environment:
|
||||||
- CRATE_HEAP_SIZE=1g
|
- CRATE_HEAP_SIZE=1g
|
||||||
@@ -19,28 +24,51 @@ services:
|
|||||||
- smartcity-shared
|
- smartcity-shared
|
||||||
ports:
|
ports:
|
||||||
- "4200:4200"
|
- "4200:4200"
|
||||||
- "5432:5432"
|
healthcheck:
|
||||||
|
test: ["CMD", "curl", "-f", "-X", "POST", "-d", "{\"stmt\": \"SELECT 1\"}", "http://localhost:4200/_sql"]
|
||||||
|
interval: 10s
|
||||||
|
timeout: 5s
|
||||||
|
retries: 5
|
||||||
|
|
||||||
quantumleap:
|
quantumleap:
|
||||||
container_name: smart-city-quantumleap
|
|
||||||
image: fiware/quantum-leap:latest
|
image: fiware/quantum-leap:latest
|
||||||
|
container_name: smart-city-quantumleap
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
environment:
|
environment:
|
||||||
- CRATE_HOST=smart-city-cratedb
|
- CRATE_HOST=smart-city-cratedb
|
||||||
- CRATE_PORT=4200
|
- CRATE_PORT=4200
|
||||||
- CRATE_DB_NAME=quantumleap
|
- CRATE_DB_NAME=quantumleap
|
||||||
- QL_LOG_LEVEL=INFO
|
- QL_LOG_LEVEL=DEBUG
|
||||||
|
- RQ_MONITOR_REDIS_URL=redis://smart-city-redis:6379
|
||||||
|
- REDIS_HOST=smart-city-redis
|
||||||
|
- REDIS_PORT=6379
|
||||||
|
- WQ_OFFLOAD_WORK=True
|
||||||
|
- ORION_HOST=smart-city-orion-ld
|
||||||
|
- ORION_PORT=1026
|
||||||
depends_on:
|
depends_on:
|
||||||
- cratedb
|
cratedb:
|
||||||
|
condition: service_healthy
|
||||||
|
redis:
|
||||||
|
condition: service_healthy
|
||||||
networks:
|
networks:
|
||||||
- smartcity-shared
|
- smartcity-shared
|
||||||
- traefik-public
|
- traefik-public
|
||||||
|
ports:
|
||||||
|
- "8668:8668"
|
||||||
|
# Le worker est géré en interne par wq (pas besoin de rq worker)
|
||||||
|
# Utilisation du command par défaut: python /src/ngsi-timeseries-api/src/app.py
|
||||||
labels:
|
labels:
|
||||||
- "traefik.enable=true"
|
- "traefik.enable=true"
|
||||||
- "traefik.http.routers.quantumleap.rule=Host(`quantum-leap.digitribe.fr`)"
|
- "traefik.http.routers.quantumleap.rule=Host(`quantum-leap.digitribe.fr')"
|
||||||
- "traefik.http.routers.quantumleap.entrypoints=websecure"
|
- "traefik.http.routers.quantumleap.entrypoints=websecure"
|
||||||
- "traefik.http.routers.quantumleap.tls=true"
|
- "traefik.http.routers.quantumleap.tls=true"
|
||||||
- "traefik.http.services.quantumleap.loadbalancer.server.port=8668"
|
- "traefik.http.services.quantumleap.loadbalancer.server.port=8668"
|
||||||
|
|
||||||
volumes:
|
volumes:
|
||||||
cratedb-data:
|
cratedb-data:
|
||||||
|
|
||||||
|
networks:
|
||||||
|
smartcity-shared:
|
||||||
|
external: true
|
||||||
|
traefik-public:
|
||||||
|
external: true
|
||||||
|
|||||||
16
simulator.py
16
simulator.py
@@ -1278,15 +1278,15 @@ def main():
|
|||||||
ok_or = publish_openremote(sid, sensor, or_values)
|
ok_or = publish_openremote(sid, sensor, or_values)
|
||||||
print(f" 🏠 OpenRemote: {'✅' if ok_or else '⚠️ skipped'}")
|
print(f" 🏠 OpenRemote: {'✅' if ok_or else '⚠️ skipped'}")
|
||||||
|
|
||||||
# --- Orion-LD ---
|
# --- Orion-LD --- (DÉSACTIVÉ: Utiliser uniquement IoT-Agents MQTT)
|
||||||
if ENABLE_ORION:
|
# if ENABLE_ORION:
|
||||||
ok_or = publish_orion(sid, sensor)
|
# ok_or = publish_orion(sid, sensor)
|
||||||
print(f" 🌐 Orion-LD: {'✅' if ok_or else '⚠️ skipped'}")
|
# print(f" 🌐 Orion-LD: {'✅' if ok_or else '⚠️ skipped'}")
|
||||||
|
|
||||||
# --- Stellio ---
|
# --- Stellio --- (DÉSACTIVÉ: Utiliser uniquement IoT-Agents MQTT)
|
||||||
if ENABLE_STELLIO:
|
# if ENABLE_STELLIO:
|
||||||
ok_st = publish_stellio(sid, sensor)
|
# ok_st = publish_stellio(sid, sensor)
|
||||||
print(f" 🏢 Stellio: {'✅' if ok_st else '❌'}")
|
# print(f" 🏢 Stellio: {'✅' if ok_st else '❌'}")
|
||||||
|
|
||||||
# --- FROST ---
|
# --- FROST ---
|
||||||
if ENABLE_FROST:
|
if ENABLE_FROST:
|
||||||
|
|||||||
Reference in New Issue
Block a user