Compare commits

...

48 Commits

Author SHA1 Message Date
Eric FELIXINE
83779cf5d7 fix: telegraf topics, mqtt brokers, docker-compose fixes
- Fix MOSQUITTO_HOST (wrong container name)
- Fix EMQX_PORT (1885 external -> 1883 internal)
- Fix telegraf MQTT topics (city/sensors/#)
- Fix BunkerM dynsec JSON
- Add kepler.yml Traefik config
- Update monitoring script
2026-06-07 20:18:41 -04:00
Eric FELIXINE
7c0cb330d9 chore: update TODO.md timestamp 2026-06-04 10:15 2026-06-04 10:26:34 -04:00
Eric FELIXINE
f45ac0cb6e feat(k8s): add defaults/main.yml, meta/main.yml for all 27 roles + 4 helm templates
- Added defaults/main.yml with production-ready values for all 27 Ansible roles
- Added meta/main.yml with role dependencies (DAG: prereq → namespaces → storage → traefik → cert-manager → services)
- Created 4 missing Helm templates: flink-deployment, kafka-cluster, smartapp-web, smartapp-api
- Fixed YAML syntax error in backup/tasks/main.yml (Velero backupStorageLocation)
- Updated README with domain list, dependencies diagram, and corrected Helm chart names
- All 81 YAML files pass validation
2026-06-04 09:45:16 -04:00
Eric FELIXINE
66ac47b684 docs: add infrastructure snapshot 2026-06-04 2026-06-04 02:26:23 -04:00
Eric FELIXINE
fb62291b3e feat: add helm/ansible deployment files for Kubernetes
Some checks failed
Build & Deploy Smart App Web / lint (push) Failing after 1s
Build & Deploy Smart App Web / build-web (push) Has been skipped
Build & Deploy Smart App Web / docker-build (push) Has been skipped
Build & Deploy Smart App Web / deploy (push) Has been skipped
2026-06-04 02:09:17 -04:00
Eric FELIXINE
8c2251faba TODO: mise a jour 2026-06-04 - cleanup massif, helms ansible generés 2026-06-04 02:05:32 -04:00
Eric FELIXINE
b56749182e chore: update TODO — Honcho API deployed, Gitea Actions configured, Smart App Docker ready
Honcho:
- API UP on honcho.digitribe.fr (port 8089)
- Workspace 'hermes-agent' and session 'smart-city-session' created
- Memory storage working (messages stored via REST API)
- Hermes plugin configured in ~/.hermes/honcho.json
- Dialectic chat requires valid LLM API key

Gitea Actions:
- Runner docker-runner-01 registered and working
- SSH secrets configured (SERVER_HOST, SERVER_USER, SSH_PRIVATE_KEY)
- Workflow: lint + build + deploy

Smart App:
- Dockerfile web: multi-stage node + nginx
- Traefik: smartapp.digitribe.fr
- deploy.sh: web/docker/api/all
- LocalAI config removed (service no longer exists)
2026-06-02 06:57:56 -04:00
Eric FELIXINE
808dbbe4f3 ci: test full pipeline — secrets configured (SERVER_HOST, SERVER_USER, SSH_PRIVATE_KEY) 2026-06-02 01:19:10 -04:00
Eric FELIXINE
9f40e187d8 ci: verify Gitea Actions runner — docker-runner-01 registered and working 2026-06-01 23:57:39 -04:00
Eric FELIXINE
f8e34562d5 feat(smart-app): CI/CD pipeline + deploy scripts
Some checks failed
Build & Deploy Smart App Web / lint (push) Failing after 1s
Build & Deploy Smart App Web / build-web (push) Has been skipped
Build & Deploy Smart App Web / docker-build (push) Has been skipped
Build & Deploy Smart App Web / deploy (push) Has been skipped
- .gitea/workflows/build-and-deploy.yml:
  - Lint + TypeScript check on PR
  - Expo web build on push to master
  - Deploy via SSH to server (copy to /var/www/smartapp)
  - Docker build alternative
  - Artifact upload for builds
- deploy.sh: manual deploy script (web|docker|api|all)
- app.json: Expo config with bundle IDs
- assets/: generated icons (icon, splash, adaptive, favicon)
- Added expo-pwa, html-webpack-plugin, workbox-webpack-plugin deps
2026-06-01 23:03:11 -04:00
Eric FELIXINE
fd583a8b16 feat(smart-app): add Docker web build + Traefik integration 2026-06-01 22:49:31 -04:00
Eric FELIXINE
43ae2ebcac feat(smart-app): complete all remaining components, screens, hooks, services, stores, i18n
- i18n/index.ts: i18next setup with FR/EN/ES/DE translations
- constants.ts: app config, sensor types, alert severity, storage keys, refresh intervals
- store/index.ts: barrel export for all stores
- iotStore.ts: full IoT store (6 sensors, 3 zones, 2 alerts) with actions
- notificationStore.ts: notification store (5 mock notifications) with actions
- uiStore.ts: theme/language store + translation maps for 4 languages
- useSensors.ts: sensor filtering by type/zone, alert sensors selector
- useAlerts.ts: active alerts, critical alerts, acknowledge
- useNotifications.ts: notification CRUD operations
- useLocation.ts: GPS location with expo-location, default Fort-de-France
- SensorCard.tsx: full sensor card with status dot, compact mode
- StatsCard.tsx: stats card with icon, value, trend
- AlertCard.tsx: alert card with severity bar, acknowledge button
- ZoneCard.tsx: zone card with color bar, sensor/alert counts
- LineChart.tsx: bar-based line chart with Y-axis labels
- BarChart.tsx: bar chart with value labels
- GaugeChart.tsx: semi-circular gauge with color thresholds
- MapView.tsx: map placeholder with overlay markers
- MarkerPopup.tsx: popup with title, value, status, detail button
- DashboardScreen.tsx: analytics dashboard with gauges + charts
- SensorDetailScreen.tsx: sensor detail with gauge + history chart
- NotificationPrefsScreen.tsx: notification preference toggles (4)
- LayerDetailScreen.tsx: layer detail placeholder
- iot.service.ts: CRUD operations for sensors, zones, alerts
- gis.service.ts: geocoding, POI search, routing
2026-06-01 22:31:36 -04:00
Eric FELIXINE
a5124b0f0d chore: update TODO.md with smart app MVP and ditto fixes 2026-06-01 18:03:40 -04:00
Eric FELIXINE
e30ae8ed09 feat(smart-app): implement complete mobile app MVP
- App.tsx: full navigation (Auth stack + Main tabs with 5 screens)
- Auth: LoginScreen, RegisterScreen, ForgotPasswordScreen
- HomeScreen: dashboard with IoT metrics, weather widget, alerts, quick actions, sensors
- MapScreen: interactive map with layer toggles (6 layers)
- MarketplaceScreen: categories (6), products (5), search
- ChatScreen: AI chat with quick prompts (4), bot responses
- ProfileScreen: user info, stats, menu (9 items), logout
- AlertsScreen: alert list with severity, acknowledge
- SensorsScreen: sensor list with type filters (6 types), search
- ZonesScreen: zone cards with stats
- SettingsScreen: language picker (FR/EN/ES/DE), privacy, about
- Stores: iotStore (sensors, zones, alerts), notificationStore, uiStore + i18n
- Hooks: useSensors, useAlerts, useNotifications, useLocation
- Components: Card, Button, LoadingSpinner, ErrorBoundary, Header
- Services: iotService, notificationService (with axios API client)
- Utils: formatters (temp, AQI, noise, dates), validators (email, password, IBAN)
- Theme: colors.ts with full design system (Blue Ocean palette)
- Ditto: fixed MongoDB connection, new JWT secrets, official gateway image
2026-06-01 18:00:35 -04:00
Eric FELIXINE
08ca495bde feat: backend FastAPI Smart App City — auth JWT, IoT, GIS, notifications, reporting 2026-06-01 14:47:05 -04:00
Eric FELIXINE
31334b5ce5 chore: session resume final 2026-06-01 — lakehouse traefik routes, network fixes 2026-06-01 14:10:52 -04:00
Eric FELIXINE
ef6e5fbae0 feat: routes Traefik pour lakehouse (trino, kafka-ui, flink, gravitino, minio) 2026-06-01 14:09:55 -04:00
Eric FELIXINE
ae35506db6 chore: backup session 2026-06-01 final — snapshot, resume, TODO, all fixes documented 2026-06-01 12:24:34 -04:00
Eric FELIXINE
8c38a23b4b chore: session resume 2026-06-01 final — JupyterHub spawn fix, all creds documented 2026-06-01 12:07:10 -04:00
Eric FELIXINE
cca9e4aedc fix: JupyterHub spawn - switch to LocalProcessSpawner, fix password hash, eric user
- SimpleLocalProcessSpawner doesn't pass JUPYTERHUB_SERVICE_URL in JH 5.3.0
- LocalProcessSpawner handles env vars correctly
- Fixed eric password hash (bcrypt instead of PBKDF2)
- eric user created with admin rights
- JupyterLab accessible at https://jupyter.digitribe.fr
- Credentials: eric / Digitribe972
2026-06-01 11:45:52 -04:00
Eric FELIXINE
85199fc3f0 chore: session backup 2026-06-01 continue — Kafka/Trino/JupyterHub fixes, TODO update 2026-06-01 10:39:11 -04:00
Eric FELIXINE
cb45b89f1f fix: JupyterHub Dockerfile - add eric user, sudo, fix DB path (4 slashes) 2026-06-01 10:26:47 -04:00
Eric FELIXINE
9e933fea66 chore: session backup 2026-06-01 final — TODO restructuré, état complet 2026-05-30 08:49:31 -04:00
Eric FELIXINE
acdf250470 chore: backup session 2026-06-01b — JupyterHub fix, Hermes Dashboard, OR mbtiles, Trino node.properties
Summary of changes:
- JupyterHub: fix DB path (absolute), Dockerfile cleanup, SimpleLocalProcessSpawner
- JupyterHub: user eric created as admin
- Hermes Dashboard WebUI + TUI chat service (systemd, localhost:9119, auto-boot)
- OR mbtiles: generated Martinique PNG tiles (5690 tiles, 10.9MB) — needs PBF for OR
- OR mbtiles: restored original PBF with corrected metadata (world bounds, Martinique center)
- OR mapsettings: verified center=[-61,14.5], bounds=Martinique, minZoom=0
- Trino: added node.properties (node.environment=production) — needs restart
- TODO.md: updated with current state
- session_resume_consolide.md: created (per-session summary)
2026-05-30 08:14:47 -04:00
Eric FELIXINE
008f1679ce fix: JupyterHub DB path + user eric + OR mbtiles bounds + Hermes Dashboard
- Fix JupyterHub: sqlite db_url absolute path (was double-nested /srv/jupyterhub/srv/jupyterhub)
- Create user eric as admin in JupyterHub (id=2, authorized)
- Fix OpenRemote mbtiles: bounds metadata = world (-180,-85,180,85) for free zoom
- OR map API confirmed working: center=[-61,14.5], minZoom=0, bounds=Martinique
- Add Hermes Dashboard WebUI + TUI chat service (localhost:9119, auto-start at boot)
- Add generate_martinique_mbtiles.py script (future tile generation)

Known issues:
- JupyterHub spawn timeout (singleuser server slow to start, increased to 120s)
- OR mbtiles still contains Netherlands vector tiles (need Martinique tiles)
- Kafka, Trino still in restart loop (separate fix needed)
2026-05-29 07:01:00 -04:00
Eric FELIXINE
a234e808f2 chore: add VRE stack configs (JupyterHub + Zeppelin) + lakehouse components
- Add VRE directory with JupyterHub + Zeppelin docker-compose configs
- Add Gravitino, Flink, Kafka, MinIO, Trino lakehouse stack
- Add Superset, Metabase, StarRocks analytics tools
- Session reprise après crash 2026-06-01

Infrastructure: 86 conteneurs total
Known issues: Kafka (no ZK conn), Trino (node.env null), JupyterHub (DB path)
2026-05-29 02:21:08 -04:00
Eric FELIXINE
486c1d2675 feat: Add OpenRemote stack config (docker-compose + traefik)
- Docker-compose based on GitHub repo openremote/openremote
- Images: timescale/timescaledb-ha:pg15, openremote/keycloak:latest, openremote/manager:1.24.0
- All 3 services healthy and running
- URL: https://openremote.digitribe.fr/manager/
2026-05-28 23:22:25 -04:00
Eric FELIXINE
184f3ca8dd chore: session backup 2026-05-27 — OpenRemote deployed with KC 23.0.7, MindsDB config updated 2026-05-27 13:14:58 -04:00
Eric FELIXINE
feb80694ab chore: update TODO — fix session date references 2026-05-26 22:55:46 -04:00
Eric FELIXINE
a19ee4080f chore: remove erroneous 2026-05-29 snapshot (correct date is 2026-05-26) 2026-05-26 22:54:57 -04:00
Eric FELIXINE
46bb937714 chore: session backup 2026-05-26 — reprise après crash, snapshot + resume + TODO update 2026-05-26 22:54:37 -04:00
Eric FELIXINE
e0d023d372 docs: add architecture inventory 2026-05-27 2026-05-26 21:04:35 -04:00
Eric FELIXINE
19cb678791 chore: session backup 2026-05-27 — ODK deployed, project created, TODO updated 2026-05-26 20:54:50 -04:00
Eric FELIXINE
978280f866 fix: ODK Traefik config — add odk.yml route, ODK now accessible at https://odk.digitribe.fr 2026-05-26 20:26:17 -04:00
Eric FELIXINE
89a821a364 chore: update TODO.md — ODK progress, ChirpStack pw reset, Smart App City arch 2026-05-26 19:06:07 -04:00
Eric FELIXINE
94f74f2dfc feat: add smart-app-city sub-project architecture
- Architecture globale (React Native + NestJS + FastAPI)
- Beckn Protocol (OTN-DPI) integration docs
- AI layer: RAG pipeline + AI Agents (LocalAI + Qdrant)
- i18n: FR/EN/ES/DE support
- Docker Compose for backend services
- Project structure with frontend, backend, ai, beckn directories
2026-05-26 18:47:02 -04:00
Eric FELIXINE
f1e1b98519 chore: session backup 2026-05-28 — TODO update + snapshot 2026-05-26 17:20:04 -04:00
Eric FELIXINE
e4c558c296 Dashboard v7: valeurs temps réel dans panneau stat + dropdown capteur
- Panneau Stat en haut: 7 métriques temps réel (temp, humidité, NO2, O3, CO, batterie, PM2.5)
- Dropdown avec includeAll + allValue '.*'
- Regex  dans toutes les queries Flux
- historique: 7j pour NO2 et PM2.5
- Refresh: 5s
2026-05-26 14:20:00 -04:00
Eric FELIXINE
65e2d42f63 Dashboard v5: variable capteur + tooltip details + coordonnees Sainte-Anne
- Ajout variable  pour filtrage par topic InfluxDB
- Tous les panels filtres par capteur selectionne
- Tooltip markersGeomap en mode details
- Coordonnees Sainte-Anne corrigees (deplacement ~300m sur terre)
- 1001 topics disponibles dans InfluxDB
2026-05-26 14:05:33 -04:00
Eric FELIXINE
a7716102fd Fix GeoMap dashboard v3 - temperature_celsius + Geomap layer config
Fixes:
- Temperature field: temperature_c (wrong) -> temperature_celsius (correct)
- Geomap panel: added explicit location config with lat/lon field mapping
- Added PM2.5 timeseries panel
- Dashboard UID: geomap-test-v1
2026-05-26 13:52:25 -04:00
Eric FELIXINE
7643d88ffb Add GeoMap dashboard + ChirpStack REST API config
- Grafana GeoMap dashboard (PostGIS + InfluxDB) for real-time sensor visualization
- Dashboard accessible at: https://grafana.digitribe.fr/d/geosmart-city-2026/smart-city-geomap-temps-reel
- PostGIS datasource added (user: grafana, network: smartcity-shared)
- InfluxDB-SmartCity-Correct datasource configured
- ChirpStack REST API: added --cors-origins flag
2026-05-26 13:14:01 -04:00
Eric FELIXINE
7df2f6798f feat: deploy Superset and Metabase behind Traefik
- Superset: docker-compose.superset.yml (app + postgres + redis)
  URL: https://superset.digitribe.fr
  Port: 8088 (internal), Traefik routes Host(superset.digitribe.fr)

- Metabase: docker-compose.metabase.yml (app + postgres)
  URL: https://metabase.digitribe.fr
  Port: 3000 (internal), Traefik routes Host(metabase.digitribe.fr)

- Traefik configs: 31-superset.yml, 32-metabase.yml
- Both services use smartcity-shared and traefik-public networks
- Both use letsencrypt TLS certificates

Verified:
- Superset: UP healthy, accessible via https://superset.digitribe.fr
- Metabase: UP healthy, accessible via https://metabase.digitribe.fr
2026-05-25 22:59:25 -04:00
Eric FELIXINE
943836f8fb feat: activate BunkerM MQTT broker + fix Telegraf
- BunkerM: recreated with port 1883 (external) -> 1900 (internal)
- BunkerM: disabled dynsec plugin, using password_file auth (bunker/bunker)
- Simulator: ENABLE_BUNKER=1, BUNKERM_PORT=1900
- Telegraf: reactivated BunkerM consumer on port 1900
- Telegraf: recreated container (3 MQTT consumers connected)
- Grafana: dashboard v4 with corrected Flux queries
- Grafana: datasource fixed (bucket=smartcity, token=my-super-token)

Verified:
- Simulator publishes to EMQX , Mosquitto , BunkerM 
- Telegraf receives from all 3 brokers 
- InfluxDB has data from all brokers 
- Grafana dashboard displays data 
2026-05-25 20:03:55 -04:00
Eric FELIXINE
5bbd5a6e5d fix: Grafana dashboard 'no data' — datasource + Flux queries
- Fix datasource: bucket=smartcity, token=my-super-token, org=digitribe
- Fix dashboard queries: filter by topic tag instead of _measurement
  (all data in measurement 'mqtt_consumer', type in tag 'topic')
- Fix field names: temperature_c→temperature_celsius, luminosity→brightness_lux
- Update dashboard to v3 with 15 panels (airquality, traffic, parking, weather, noise, light)
- Update TODO.md and session_resume

Tested: PM2.5 , Temperature , Vehicle Count  via Grafana API
2026-05-25 16:39:50 -04:00
Eric FELIXINE
6d1d9c8620 fix: telegraf containers names + openremote pg image + session snapshot 2026-05-25
- telegraf.conf: fix Mosquitto/BunkerM container names (hyphens not underscores)
- tegraf.conf: comment out BunkerM consumer (auth fails, simulator not sending)
- openremote/docker-compose.yml: switch PG image to timescaledb-ha:pg15 (fixes timescaledb_toolkit crash)
- Add session_resume + architecture snapshot 2026-05-25
- Update TODO.md with current status
2026-05-25 14:13:39 -04:00
Eric FELIXINE
eb97f2a7dd Fix: Traefik dynamic config, Ditto things startup crash, and OpenRemote sensor coordinates 2026-05-20 18:18:21 -04:00
Eric FELIXINE
45f3ab8a3d docs: session resume 2026-05-20 - stabilization and platforms deployment 2026-05-20 13:19:29 -04:00
Eric FELIXINE
98f0bcb021 Session 2026-05-20: Contexus MQTT devices, OpenRemote agent, 60 capteurs configures 2026-05-20 00:58:48 -04:00
35804 changed files with 3719004 additions and 355 deletions

2
.env.ditto Normal file
View File

@@ -0,0 +1,2 @@
DITTO_JWT_SECRET=NTOT-Vh8WRKWE52eV8zRiLs3a-gd8YUGSrvm5x2InZc
DEVOPS_PASSWORD=OvP9WVB09aFDnYPyK52UIg

View File

@@ -0,0 +1,121 @@
# Gitea Actions — CI/CD Smart App City Web
# Trigger: push sur master ou PR
name: Build & Deploy Smart App Web
on:
push:
branches: [master]
paths:
- 'smart-app-city/frontend/**'
pull_request:
branches: [master]
jobs:
# ─── Lint + Type Check ─────────────────────────────────────────────────
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: smart-app-city/frontend/package-lock.json
- name: Install dependencies
working-directory: smart-app-city/frontend
run: npm ci --legacy-peer-deps
- name: TypeScript check
working-directory: smart-app-city/frontend
run: npx tsc --noEmit
continue-on-error: true # TODO: fix TS strict errors
# ─── Build Web ─────────────────────────────────────────────────────────
build-web:
runs-on: ubuntu-latest
needs: lint
if: github.event_name == 'push'
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: smart-app-city/frontend/package-lock.json
- name: Install dependencies
working-directory: smart-app-city/frontend
run: npm ci --legacy-peer-deps
- name: Build Expo web
working-directory: smart-app-city/frontend
run: npx expo export:web
- name: Upload build artifact
uses: actions/upload-artifact@v4
with:
name: smartapp-web-build
path: smart-app-city/frontend/dist/
retention-days: 7
# ─── Deploy to Server ──────────────────────────────────────────────────
deploy:
runs-on: ubuntu-latest
needs: build-web
if: github.ref == 'refs/heads/master'
steps:
- uses: actions/checkout@v4
- name: Download build artifact
uses: actions/download-artifact@v4
with:
name: smartapp-web-build
path: smart-app-city/frontend/dist/
- name: Deploy to server via SSH
uses: appleboy/scp-action@v0.1.7
with:
host: ${{ secrets.SERVER_HOST }}
username: ${{ secrets.SERVER_USER }}
key: ${{ secrets.SSH_PRIVATE_KEY }}
source: "smart-app-city/frontend/dist/"
target: "/var/www/smartapp/"
strip_components: 3
- name: Restart nginx (or copy to Docker volume)
uses: appleboy/ssh-action@v1.0.7
with:
host: ${{ secrets.SERVER_HOST }}
username: ${{ secrets.SERVER_USER }}
key: ${{ secrets.SSH_PRIVATE_KEY }}
script: |
# Option A: If using Docker volume
docker cp /var/www/smartapp/ smartapp-web:/usr/share/nginx/html/
# Option B: If using Docker build
# cd /home/eric/smart-city-digital-twin-martinique/smart-app-city
# docker compose up -d --build smartapp-web
echo "✅ Smart App Web deployed at $(date)"
# ─── Build Docker Image (alternative) ───────────────────────────────────
docker-build:
runs-on: ubuntu-latest
needs: lint
if: github.ref == 'refs/heads/master'
steps:
- uses: actions/checkout@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build Docker image
working-directory: smart-app-city/frontend
run: |
docker build -t smartapp-web:${{ github.sha }} .
docker tag smartapp-web:${{ github.sha }} smartapp-web:latest
# Note: Pour pousser vers un registry privé, ajouter docker login + push
# - name: Push to registry
# run: docker push registry.digitribe.fr/smartapp-web:latest

24
.gitignore vendored Normal file
View File

@@ -0,0 +1,24 @@
# Dependencies
node_modules/
*/node_modules/
# Build outputs
dist/
build/
*.pyc
__pycache__/
# Environment files
.env
.env.local
*.env
# IDE
.vscode/
.idea/
*.swp
*.swo
# OS
.DS_Store
Thumbs.db

14
32-kafka-ui.yml Normal file
View File

@@ -0,0 +1,14 @@
http:
routers:
kafka-ui:
rule: "Host(`kafka.digitribe.fr`)"
entryPoints:
- websecure
service: kafka-ui-svc
tls:
certResolver: letsencrypt
services:
kafka-ui-svc:
loadBalancer:
servers:
- url: "http://kafka-ui:8080"

14
32-trino.yml Normal file
View File

@@ -0,0 +1,14 @@
http:
routers:
trino:
rule: "Host(`trino.digitribe.fr`)"
entryPoints:
- websecure
service: trino-svc
tls:
certResolver: letsencrypt
services:
trino-svc:
loadBalancer:
servers:
- url: "http://trino:8084"

14
33-flink.yml Normal file
View File

@@ -0,0 +1,14 @@
http:
routers:
flink:
rule: "Host(`flink.digitribe.fr`)"
entryPoints:
- websecure
service: flink-svc
tls:
certResolver: letsencrypt
services:
flink-svc:
loadBalancer:
servers:
- url: "http://flink-jobmanager:8081"

14
34-gravitino.yml Normal file
View File

@@ -0,0 +1,14 @@
http:
routers:
gravitino:
rule: "Host(`gravitino.digitribe.fr`)"
entryPoints:
- websecure
service: gravitino-svc
tls:
certResolver: letsencrypt
services:
gravitino-svc:
loadBalancer:
servers:
- url: "http://gravitino:8090"

14
35-minio.yml Normal file
View File

@@ -0,0 +1,14 @@
http:
routers:
minio:
rule: "Host(`minio.digitribe.fr`)"
entryPoints:
- websecure
service: minio-svc
tls:
certResolver: letsencrypt
services:
minio-svc:
loadBalancer:
servers:
- url: "http://minio:9001"

241
INVENTORY-2026-05-27.md Normal file
View File

@@ -0,0 +1,241 @@
# Smart City Digital Twin — Inventaire Architecture
> **Date** : 2026-05-27 01:00
> **Hôte** : Linux 6.8.0-117-generic (31Gi RAM)
> **Containers actifs** : 59 running / 1 restarting / 61 exited
> **Réseaux Docker** : 30+ networks (smartcity-shared, traefik-public, odk-internal, etc.)
---
## 📊 Vue d'ensemble
| Catégorie | Containers | Statut |
|-----------|------------|--------|
| **ODK Central** | 4 | ✅ Tous UP |
| **IoT / MQTT** | 10 | ✅ Tous UP |
| **Data / Analytics** | 10 | ✅ Tous UP |
| **GIS / Géospatial** | 8 | ✅ Tous UP |
| **Identité** | 3 | ⚠️ Honcho restarting |
| **Smart City Core** | 10 | ✅ Tous UP |
| **Monitoring** | 3 | ✅ Tous UP |
| **AI** | 1 | ✅ UP |
| **Autres** | 7 | ✅ Tous UP |
---
## 🔷 ODK Central (4 containers)
| Container | Image | Ports | Domaine |
|-----------|-------|-------|---------|
| `odk-nginx` | odk-nginx:latest | 80, 443 | odk.digitribe.fr |
| `odk-service` | odk-service:latest | 8383 | — |
| `odk-postgres` | postgres:15-alpine | 5432 | — |
| `odk-pyxform` | ghcr.io/getodk/pyxform-http:v4.4.1 | 80 | — |
**Credentials** : efelixine@digitribe.fr / Digitribe972
**Projet** : Smart-City-Martinique (id=1)
---
## 🔶 IoT / MQTT (10 containers)
| Container | Image | Ports | Rôle |
|-----------|-------|-------|------|
| `bunkerm-bunkerm-1` | bunkeriot/bunkerm:latest | 1883→1900, 2000 | MQTT Broker principal |
| `emqx_emqx_1` | emqx/emqx:latest | 11883, 18081, 18883, 38083 | MQTT Broker v5 |
| `smart-city-digital-twin-martinique-mosquitto-1` | eclipse-mosquitto:2 | 1883 | MQTT v5 (simulateur) |
| `chirpstack-mosquitto-1` | eclipse-mosquitto:2 | 1883 | MQTT ChirpStack |
| `smart-city-digital-twin-martinique-chirpstack-1` | chirpstack/chirpstack:latest | — | LoRaWAN NS |
| `smart-city-digital-twin-martinique-chirpstack-rest-api-1` | chirstack/chirpstack-rest-api:4 | — | ChirpStack REST API |
| `smart-city-digital-twin-martinique-chirpstack-gateway-bridge-1` | chirpstack/chirpstack-gateway-bridge:4 | 1700/udp | Gateway Bridge |
| `smart-city-digital-twin-martinique-chirpstack-gateway-bridge-basicstation-1` | chirpstack/chirpstack-gateway-bridge:4 | — | Basic Station |
| `chirpstack-redis-1` | redis:7-alpine | 6379 | Cache ChirpStack |
| `chirpstack-postgres-1` | postgres:14-alpine | 5432 | DB ChirpStack |
**Credentials MQTT** : bunker / bunker (BunkerM)
**Credentials ChirpStack** : admin / Digitribe972
---
## 📈 Data / Analytics (10 containers)
| Container | Image | Ports | Domaine |
|-----------|-------|-------|---------|
| `smart-city-grafana` | grafana/grafana:10.2.0 | 3001 | grafana.digitribe.fr |
| `grafana_stack-grafana-1` | grafana/grafana:latest | 3000 | — |
| `honcho-grafana-1` | grafana/grafana:11.4.0 | 3088 | — |
| `smart-city-influxdb` | influxdb:2.7-alpine | 8086 | — |
| `smart-city-loki` | grafana/loki:latest | 3100 | — |
| `smart-city-prometheus-brokers` | prom/prometheus:latest | — | — |
| `honcho-prometheus-1` | prom/prometheus:v3.2.1 | 9091 | — |
| `smart-city-redpanda-console` | redpandadata/console:v2.5.0 | 28080 | — |
| `metabase-app` | metabase/metabase:latest | 3000 | metabase.digitribe.fr |
| `metabase-postgres` | postgres:15-alpine | 5432 | — |
**Credentials** : admin / Digitribe972 (Grafana, Metabase, Superset)
**InfluxDB** : token=my-super-token, org=digitribe, bucket=smartcity
---
## 🗺️ GIS / Géospatial (8 containers)
| Container | Image | Ports | Domaine |
|-----------|-------|-------|---------|
| `geoserver_stack-geoserver-1` | oscarfonts/geoserver:2.25.2 | 8080 | geoserver.digitribe.fr |
| `mapstore-app` | geosolutionsit/mapstore2:latest | 8080 | mapstore.digitribe.fr |
| `mapstore-proxy` | nginx | 80 | — |
| `mapstore-postgres` | geosolutions-mapstore/postgis | 5432 | — |
| `postgis-smartcity` | postgis/postgis:15-3.4 | 5433 | — |
| `fiware-gis-quickstart-orion-1` | quay.io/fiware/orion-ld | 2026 | — |
| `fiware-gis-quickstart-orionproxy-1` | fiware-gis-quickstart-orionproxy | 1026 | — |
| `fiware-gis-quickstart-mongo-db-1` | mongo:4.2 | 27017 | — |
| `frost_allinone-web-1` | fraunhoferiosb/frost-server:latest | 8089 | — |
| `frost_http-web-1` | fraunhoferiosb/frost-server-http:latest | 8080 | — |
| `stellio-api-gateway` | stellio/stellio-api-gateway:latest-dev | 8080 | stellio.digitribe.fr |
---
## 🔐 Identité (3 containers)
| Container | Image | Ports | Domaine |
|-----------|-------|-------|---------|
| `openremote-keycloak` | quay.io/keycloak/keycloak:24.0 | 8080, 8443 | openremote.digitribe.fr/auth |
| `honcho-api-1` | honcho:latest | 8000 | — |
| `honcho-deriver-1` | honcho-deriver | — | — |
**Credentials Keycloak** : admin / admin
**Credentials OpenRemote** : admin / Digitribe972
---
## 🏙️ Smart City Core (10 containers)
| Container | Image | Ports | Rôle |
|-----------|-------|-------|------|
| `smart-city-simulator` | smart-city-simulator | 8081 | Simulateur 60 capteurs |
| `smart-city-telegraf` | telegraf:1.28 | 8092, 8125, 8094 | Collecte IoT |
| `contexus-app` | contexusio/contexus:latest | 15000 | Plateforme IoT |
| `contexus-postgres` | postgres:16 | 5432 | DB Contexus |
| `contexus-redis` | redis:7-alpine | 6379 | Cache Contexus |
| `smart-city-ditto-policies` | eclipse/ditto-policies:latest | 8080 | Digital Twin policies |
| `smart-city-ditto-gateway` | eclipse/ditto-gateway:latest | 8080 | Digital Twin gateway |
| `smart-city-ditto-mongodb` | mongo:6 | 27017 | DB Ditto |
| `digital-twin-nodered` | nodered/node-red:3.1 | 1880 | Node-RED |
| `digital-twin-connector` | python:3.11-slim | — | Connector Python |
| `smart-city-digital-twin-martinique-redis-1` | redis:7-alpine | 6379 | Cache simulateur |
**Credentials Contexus** : iotevadmin / Digitribe972
---
## 👁️ Monitoring (3 containers)
| Container | Image | Ports | Rôle |
|-----------|-------|-------|------|
| `traefik` | traefik:v3.1 | 80, 443, 8404 | Reverse Proxy |
| `smart-city-promtail` | grafana/promtail:latest | — | Log shipping |
| `docker-exporter` | docker-exporter:latest | 8005 | Métriques Docker |
---
## 🤖 AI (1 container)
| Container | Image | Ports | Rôle |
|-----------|-------|-------|------|
| `localai-api` | localai/localai:latest | 8080 | LLM local (Llama 3.1 70B) |
**Credentials** : admin / Digitribe972
**API Key** : hermes-localai-secret-key-2024
---
## 📦 Autres (7 containers)
| Container | Image | Ports | Rôle |
|-----------|-------|-------|------|
| `gitea` | gitea/gitea:latest | 22, 3000 | Git (gitea.digitribe.fr) |
| `agentgateway` | cr.agentgateway.dev/agentgateway:latest | 3000, 15000 | Agent Gateway |
| `smart-city-kepler` | smart-city-kepler:latest | 80, 8080 | Kepler.gl |
| `stellio-api-gateway` | stellio/stellio-api-gateway:latest-dev | 8080 | NGSI-LD Context Broker |
| `esperotech` | esperotech/yaade:latest | 9339 | Yaade |
| `phpipam-phpipam-web-1` | phpipam/phpipam-www:latest | 8085 | IPAM |
| `phpipam-phpipam-cron-1` | phpipam/phpipam-cron:latest | — | IPAM cron |
| `docker_zookeeper_1` | zookeeper:3.8.1 | 2181 | ZooKeeper |
---
## 🌐 Réseaux Docker Principaux
| Réseau | Usage |
|--------|-------|
| `smartcity-shared` | Réseau principal partagé entre services |
| `traefik-public` | Exposition web via Traefik |
| `central_odk-internal` | Réseau interne ODK |
| `openremote_default` | OpenRemote stack |
| `contexus-iot-network` | Contexus IoT |
| `fiware-gis-quickstart_fiware` | FIWARE GIS |
| `mapstore2_mapstore-network` | MapStore |
| `superset_default` | Apache Superset |
| `metabase_default` | Metabase |
| `localai_default` | LocalAI |
| `honcho_default` | Honcho |
| `mainflux-network` | Mainflux |
| `stellio-context-broker_default` | Stellio |
---
## 🔑 Credentials Résumé
| Service | Login | Password |
|---------|-------|----------|
| **ODK Central** | efelixine@digitribe.fr | Digitribe972 |
| **Grafana** | admin | Digitribe972 |
| **Metabase** | admin@digitribe.fr | Digitribe972 |
| **Superset** | admin | Digitribe972 |
| **OpenRemote** | admin | Digitribe972 |
| **Keycloak** | admin | admin |
| **Contexus** | iotevadmin | Digitribe972 |
| **ChirpStack** | admin | Digitribe972 |
| **BunkerM MQTT** | bunker | bunker |
| **LocalAI** | admin | Digitribe972 |
| **InfluxDB** | — | token=my-super-token |
| **PostgreSQL Contexus** | contexus | Digitribe972 |
| **Redis Contexus** | — | Digitribe972 |
---
## 📁 Répertoires Projet
```
~/smart-city-digital-twin-martinique/ # Repo principal (Gitea)
~/odk/central/ # ODK Central
~/openremote/ # OpenRemote
~/traefik-config/dynamic/ # Config Traefik
~/smart-app-city/ # Sous-projet app mobile
```
---
## 📊 Pipeline de Données
```
Simulateur (60 capteurs)
┌───────────────────────────────────────┐
│ 3 Brokers MQTT │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ EMQX │ │Mosquitto│ │ BunkerM │ │
│ └────┬────┘ └────┬────┘ └────┬────┘ │
└───────┼───────────┼───────────┼───────┘
└───────────┴───────────┘
Telegraf (3 inputs MQTT)
InfluxDB v2 (bucket: smartcity)
Grafana (Dashboard v7)
```
---
*Inventaire généré le 2026-05-27 01:00 — OWL*

View File

@@ -0,0 +1,31 @@
# Smart City Monitoring Report - 2026-05-22
**Timestamp:** 2026-05-22 00:50:30
## Summary
⚠️ **9 issue(s) detected** - Critical systems are partially unavailable
## Container Status
| Container | Status |
|-----------|--------|
| openremote_manager_1 | 🛑 DOWN |
| openremote_keycloak_1 | 🛑 DOWN |
| stellio-api-gateway | 🛑 DOWN |
| smart-city-prometheus-brokers | 🛑 DOWN |
## Endpoint Status
| Service | URL | Status |
|---------|-----|--------|
| OpenRemote | https://openremote.digitribe.fr | 🌐 DOWN (HTTP 502) |
| Orion-LD | http://fiware-gis-quickstart-orion-1:1026/version | 🌐 DOWN (HTTP 000) |
| Stellio | https://stellio.digitribe.fr | 🌐 DOWN (HTTP 502) |
| FROST | http://frost_http-web-1:8080/FROST-Server/core/v1.0/info | 🌐 DOWN (HTTP 000) |
## Network
- 🔌 Network issue: Traefik → OpenRemote
## Recommendations
1. Restart critical containers: `docker-compose up -d`
2. Check Traefik logs for routing issues
3. Verify network connectivity between services
4. Review container health checks

150
TODO.md
View File

@@ -1,53 +1,121 @@
# Smart City Digital Twin — TODO List
> Dernière mise à jour : 2026-05-19 23:25 (v2)
> Dernière mise à jour : 2026-06-04 02:00 (finalisation)
## ✅ Complété (5/13)
| ID | Tâche |
|----|-------|
| p1-bunkerm | BunkerM: DNS corrigé (underscores → hyphens) |
| p2-geoserver | GeoServer: workspace Digitribe + Data Store PostGIS dédié |
| p2-postgis | PostGIS dédié: conteneur postgis-smartcity UP (PostGIS 3.4) |
| p2-mapstore | MapStore: GeoServer WMS ajouté au CORS, couche sensors créée |
| p5-docs | Documentation + commits Gitea |
## ✅ Complété (session 2026-06-03 / 06-04)
## 🔴 Bloqué (5/13)
| ID | Tâche | Raison |
| ID | Tâche | Détail |
|----|-------|--------|
| p1-or | OpenRemote agents MQTT + map display | PostgreSQL bloqué en recovery (TimescaleDB upgrade) |
| p4-ditto | Ditto.digitribe.fr | MongoDB localhost hardcodé |
| p1-prometheus | Prometheus + Grafana | Réseau interne inaccessible |
| p3-kepler | KeplerGL | Image Docker incomplète, build npm trop long |
| p3-geomesa | GeoMesa | Installation complexe (Maven, binaires) |
| airflow-deploy | Apache Airflow déployé | `airflow.digitribe.fr` — Python 3.11, LocalExecutor |
| openfn-cleanup | OpenFN supprimé | Race condition Cachex/Ecto non résolue |
| ditto-cleanup | Stack Ditto supprimée | API v2 non fonctionnelle (schema-versions) |
| openremote-cleanup | Stack OpenRemote supprimée | Patches bundle appliqués |
| gravitino-cleanup | Gravitino supprimé | Unhealthy |
| fiware-gis-cleanup | FIWARE GIS Quickstart supprimé | |
| contexus-cleanup | Contexus supprimé | Unhealthy |
| kafka-cleanup | Kafka supprimé | Unhealthy + sera redeployé via Helm |
| flink-cleanup | Flink supprimé | Dépendances kafka |
| bi-cleanup | Superset + Metabase supprimés | Seront redeployés via Helm |
| mindsdb-cleanup | MindsDB supprimé | Autoheal unhealthy |
| odk-cleanup | ODK Central supprimé | Sera redeployé via Helm |
| jupyterhub-cleanup | JupyterHub supprimé | Sera redeployé via Helm |
| zeppelin-cleanup | Zeppelin supprimé | Sera redeployé via Helm |
| gis-cleanup | MapStore + GeoServer + FROST supprimés | Seront redeployés via Helm |
| iot-cleanup | Node-RED + phpIPAM + EMQX + Mosquitto + BunkerM + ChirpStack supprimés | Seront redeployés via Helm |
| monitoring-cleanup | Grafana + Loki + Prometheus + InfluxDB + Telegraf supprimés | Seront redeployés via Helm |
| storage-cleanup | MinIO + PostgreSQL + PostGIS + Redis + Zookeeper supprimés | Seront redeployés via Helm |
| misc-cleanup | AgentGateway + Esperotech + Redpanda Console + Docker exporter + Simulator supprimés | |
| backups | Sauvegardes config | Fichiers sauvegardés dans /home/eric/backups/2026-06-04/ |
| helms-ansible | Fichiers Helm/Ansibles générés | 25+ rôles dans helms/ |
| helms-readme | README déploiement K8s | Architecture, installation, troubleshooting |
| helms-vault | Template vault.yml | Variables chiffrées pour le déploiement |
| git-push | Push sur Gitea | 2 commits pushés (TODO + helms) |
## 🔴 En cours
| ID | Tâche | Raison | Prochaine action |
|----|-------|--------|------------------|
| (aucune) | — | — | — |
## ⏳ En attente (déploiement Kubernetes via Ansible)
## ⏳ En attente (3/13)
| ID | Tâche |
|----|-------|
| p3-analyse | Analyse: GeoMesa + KeplerGL (bloqué) |
| p1-ngsi | NGSI-LD: validation pipeline (basse priorité) |
| p0-chirpstack | ChirpStack: login API gRPC-REST |
| k8s-cluster | Créer le cluster Kubernetes (3 nœuds minimum) |
| nfs-server | Configurer le serveur NFS pour le storage |
| traefik-deploy | Déployer Traefik via Helm |
| cert-manager-deploy | Déployer cert-manager pour TLS |
| storage-deploy | Déployer NFS provisioner + StorageClass |
| monitoring-deploy | Déployer Prometheus + Grafana + Loki |
| databases-deploy | Déployer PostgreSQL HA + Redis + MinIO |
| kafka-deploy | Déployer Kafka (Strimzi) |
| flink-deploy | Déployer Apache Flink |
| airflow-deploy | Déployer Apache Airflow |
| iot-deploy | Déployer EMQX + Mosquitto + Node-RED + phpIPAM |
| gitea-deploy | Déployer Gitea |
| jupyterhub-deploy | Déployer JupyterHub |
| bi-deploy | Déployer Superset + Metabase |
| mindsdb-deploy | Déployer MindsDB |
| odk-deploy | Déployer ODK Central |
| gis-deploy | Déployer MapStore + GeoServer + FROST |
| clickhouse-deploy | Déployer ClickHouse (`clickhouse.digitribe.fr`) |
| starrocks-deploy | Déployer StarRocks (`starrocks.digitribe.fr`) |
| trino-deploy | Déployer Trino (`trino.digitribe.fr`) |
| deltalake-deploy | Déployer Delta Lake (`deltalake.digitribe.fr`) |
| streamlit-deploy | Déployer Streamlit (`streamlit.digitribe.fr`) |
| duckdb-deploy | Déployer DuckDB (`duckdb.digitribe.fr`) |
| smartapp-deploy | Déployer Smart App (`smartapp.digitribe.fr`) |
| backup-deploy | Déployer Velero pour les sauvegardes |
## 📦 Skills créées (hors projet)
| Skill | Catégorie | Statut |
|-------|-----------|--------|
| epicollect5 | mobile-data-collection | ✅ |
| odk | mobile-data-collection | ✅ |
| kobotoolbox | mobile-data-collection | ✅ |
| apache-superset | data-visualization | ✅ |
| metabase | data-visualization | ✅ |
| contexus | iot | ✅ |
## 📁 Fichiers Helm / Ansible générés
## 📝 Notes 2026-05-19
- **OpenRemote map display** : Investigation approfondie — points ne s'affichent pas malgré toutes les conditions remplies (location, agentLink, showOnDashboard, bon realm)
- **Décision** : Repartir de zéro avec installation fraîche du Manager
- **PostgreSQL** : Bloqué en recovery — TimescaleDB upgrade (2.26.3) sur cluster vide après `docker-compose down --volumes`. Le volume a été recréé vide et le TimescaleDB essaie de s'installer, ce qui bloque le recovery depuis plus d'une heure.
- **Solution possible** : Supprimer le volume et relancer sans TimescaleDB, ou attendre que le TimescaleDB finisse son installation
- **Prochaines étapes** : Attendre PostgreSQL → Vérifier Manager/Keycloak → Lancer simulateur → Créer dashboard via UI → Vérifier affichage
- **Custom project** : Répertoire `/home/eric/openremote/custom-project/` cloné — prêt pour développement custom
- **GeoJSON proxy** : Correction des coordonnées (lat/lon → lon/lat) — le proxy retourne maintenant les coordonnées dans le bon ordre
Le répertoire `helms/` (dans le repo Gitea) contient les fichiers pour un déploiement modulaire sur Kubernetes via Ansible.
### Structure
```
helms/
├── README.md # Documentation déploiement
├── deploy.yml # Playbook principal
├── undeploy.yml # Playbook de suppression
├── inventory/hosts.yml # Inventory des nœuds K8s
├── group_vars/all.yml # Variables globales
├── group_vars/vault.yml # Variables chiffrées (template)
└── roles/ # 25+ rôles Ansible
```
### Utilisation
```bash
cd helms/
ansible-playbook deploy.yml --ask-vault-pass
ansible-playbook deploy.yml --tags clickhouse --ask-vault-pass
ansible-playbook undeploy.yml
```
## 📝 Infrastructure actuelle (10 containers Docker)
| Service | Image | Statut |
|---------|-------|--------|
| airflow-scheduler | apache/airflow:2.9.3-python3.11 | ✅ healthy |
| airflow-webserver | apache/airflow:2.9.3-python3.11 | ✅ healthy |
| airflow-init | apache/airflow:2.9.3-python3.11 | 🔄 restarting (one-shot) |
| airflow-postgres | postgres:16 | ✅ healthy |
| smartapp-api | smartapp-api:latest | ✅ Up 38h |
| smartapp-web | nginx:alpine | ✅ Up 38h |
| gitea-runner | gitea/act_runner:latest | ✅ Up 2 days |
| traefik | traefik:v3.1 | ✅ Up 2 days |
| smart-city-kepler | smart-city-kepler:latest | ✅ Up 2 weeks |
| gitea | gitea/gitea:latest | ✅ Up 2 jours |
## 📊 Statistiques
- **Containers Docker** : 10 (down from 72)
- **Stacks supprimées** : 6 (OpenFN, Ditto, OpenRemote, Gravitino, FIWARE GIS, Contexus)
- **Services unhealthy** : 0
- **Fichiers Helm/Ansible** : 33 fichiers
- **Rôles Ansible** : 25+
- **Namespaces K8s prévus** : 18
## Credentials
- **GeoServer**: admin / Digitribe972
- **PostGIS dédié**: smartcity / SmartCity972 (port 5433)
- **ChirpStack**: admin / admin1234
- **Gitea**: eric / token configuré
- **Gitea** : eric / (voir config)
- **Airflow** : admin / (changé par Eric)

52
bemserver/Dockerfile Normal file
View File

@@ -0,0 +1,52 @@
# BEMServer - Building Energy Management Server
# Multi-component Dockerfile: core + api + ui + celery
# Based on Python 3.11 slim with TimescaleDB support
FROM python:3.11-slim AS base
# System dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
gcc \
libpq-dev \
libffi-dev \
curl \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /opt/bemserver
# ---- Stage 1: Install bemserver-core ----
FROM base AS core
COPY bemserver/bemserver-core /tmp/bemserver-core
RUN pip install --no-cache-dir /tmp/bemserver-core
# ---- Stage 2: Install bemserver-api ----
FROM core AS api
COPY bemserver/bemserver-api /tmp/bemserver-api
RUN pip install --no-cache-dir /tmp/bemserver-api
# ---- Stage 3: Install bemserver-ui ----
FROM api AS ui
COPY bemserver/bemserver-ui /tmp/bemserver-ui
RUN pip install --no-cache-dir /tmp/bemserver-ui
# ---- Final stage ----
FROM ui AS final
# Create non-root user
RUN groupadd -r bemserver && useradd -r -g bemserver -d /opt/bemserver -s /sbin/nologin bemserver
# Create config directory
RUN mkdir -p /opt/bemserver/config /opt/bemserver/data \
&& chown -R bemserver:bemserver /opt/bemserver
# Copy entrypoint script
COPY bemserver/entrypoint.sh /opt/bemserver/entrypoint.sh
RUN chmod +x /opt/bemserver/entrypoint.sh
# Healthcheck
HEALTHCHECK --interval=30s --timeout=10s --retries=5 --start_period=60s \
CMD curl -f http://localhost:5000/healthz || exit 1
USER bemserver
ENTRYPOINT ["/opt/bemserver/entrypoint.sh"]

44
bemserver/entrypoint.sh Normal file
View File

@@ -0,0 +1,44 @@
#!/bin/bash
# BEMServer entrypoint - runs the specified component
set -e
COMPONENT=${BEMSERVER_COMPONENT:-api}
CONFIG_DIR="/opt/bemserver/config"
case "$COMPONENT" in
api)
echo "Starting BEMServer API on port 5000..."
exec flask --app bemserver_api.app create --config "${CONFIG_DIR}/api-settings.py"
;;
ui)
echo "Starting BEMServer UI on port 5001..."
exec flask --app bemserver_ui.app create --config "${CONFIG_DIR}/ui-settings.cfg"
;;
celery-worker)
echo "Starting BEMServer Celery worker..."
export BEMSERVER_CORE_SETTINGS_FILE="${CONFIG_DIR}/core-settings.py"
exec celery -A bemserver_core.celery_worker worker --loglevel=info
;;
celery-beat)
echo "Starting BEMServer Celery beat..."
export BEMSERVER_CORE_SETTINGS_FILE="${CONFIG_DIR}/core-settings.py"
exec celery -A bemserver_core.celery_worker beat --loglevel=info
;;
init-db)
echo "Initializing BEMServer database..."
export BEMSERVER_CORE_SETTINGS_FILE="${CONFIG_DIR}/core-settings.py"
bemserver_db_upgrade
echo "Database initialized."
;;
create-admin)
echo "Creating admin user..."
export BEMSERVER_CORE_SETTINGS_FILE="${CONFIG_DIR}/core-settings.py"
bemserver_create_user --name "${BEMSERVER_ADMIN_USER:-admin}" --email "${BEMSERVER_ADMIN_EMAIL:-admin@digitribe.fr}" --admin
echo "Admin user created."
;;
*)
echo "Unknown component: $COMPONENT"
echo "Valid components: api, ui, celery-worker, celery-beat, init-db, create-admin"
exit 1
;;
esac

View File

@@ -0,0 +1,6 @@
FROM eclipse/ditto-gateway:latest
USER root
# Copy the modified JAR (with open auth in reference.conf)
COPY ditto-gateway-service-3.8.12-allinone.jar /opt/ditto/ditto-gateway-service-3.8.12-allinone.jar
USER ditto

View File

@@ -0,0 +1,22 @@
# Minimal override - authentication settings only
# This file is loaded after reference.conf by Typesafe config
ditto {
gateway {
authentication {
pre-authentication {
enabled = true
}
devops {
secured = false
devops-authentication-method = "basic"
password = "ditto-devops-secret"
password = ${?DEVOPS_PASSWORD}
status-secured = false
status-authentication-method = "basic"
statusPassword = "ditto-status-secret"
statusPassword = ${?STATUS_PASSWORD}
}
}
}
}

View File

@@ -0,0 +1,7 @@
#!/bin/sh
# Fix permissions for mounted config files
if [ -f /opt/ditto/gateway-extension.conf ]; then
chmod 644 /opt/ditto/gateway-extension.conf
fi
# Start the gateway
exec java -jar /opt/ditto/ditto-gateway-service-3.8.12-allinone.jar

View File

@@ -0,0 +1,2 @@
#!/bin/sh
exec java -Dconfig.file=/opt/ditto/gateway.conf -jar /opt/ditto/ditto-gateway-service-3.8.12-allinone.jar

View File

@@ -0,0 +1,76 @@
#!/bin/bash
set -e
WORKDIR=/tmp/ditto-jar-mod
rm -rf $WORKDIR && mkdir -p $WORKDIR
echo "=== Extracting JAR ==="
cd $WORKDIR
docker run --rm eclipse/ditto-gateway:latest cat /opt/ditto/ditto-gateway-service-3.8.12-allinone.jar > ditto-gateway-service-3.8.12-allinone.jar
jar xf ditto-gateway-service-3.8.12-allinone.jar reference.conf
echo "=== Original reference.conf tail ==="
tail -5 reference.conf
echo "=== Adding auth config to reference.conf ==="
# Remove the last closing brace, add our config, re-add the closing brace
# The reference.conf ends with nested braces - find the very last line
python3 << 'PYEOF'
with open("/tmp/ditto-jar-mod/reference.conf", "r") as f:
lines = f.readlines()
# Find the last non-empty line that is just a closing brace
# We need to insert our config before the outermost closing brace
# Simple approach: append before the very last }
# Count total closing braces at the end
content = "".join(lines)
# The reference.conf has a complex nested structure
# We'll add our ditto config as a new root-level block at the end
# We need to close the last block and add a comma, then our new block
# Actually, simpler: just append if the file ends with }
# Find the position of the very last }
last_brace_pos = content.rfind('}')
if last_brace_pos >= 0:
# Check if there's content after the last } (like kamon.conf include)
rest = content[last_brace_pos+1:].strip()
if rest:
# There's content after the last }, our approach is wrong
print(f"Content after last }}: {rest[:100]}")
# Insert before the last } with a comma
new_content = content[:last_brace_pos].rstrip()
# Remove trailing comma if present
if new_content.endswith(','):
new_content = new_content[:-1]
new_content += ',\n\n# Custom auth overrides\n' + rest[:0] + '\n' + ' authentication {\n pre-authentication {\n enabled = true\n }\n devops {\n secured = false\n }\n }\n}\n'
# Just append
new_content = content.rstrip() + '\n\n# Custom auth overrides\nditto {\n gateway {\n authentication {\n pre-authentication {\n enabled = true\n }\n devops {\n secured = false\n }\n }\n }\n}\n'
else:
# Last char is }, replace it with our config + }
new_content = content[:last_brace_pos].rstrip()
# Remove trailing comma
if new_content.endswith(','):
new_content = new_content[:-1]
new_content += ',\n\n# Custom auth overrides\n gateway {\n authentication {\n pre-authentication {\n enabled = true\n }\n devops {\n secured = false\n }\n }\n }\n}\n'
else:
new_content = content
with open("/tmp/ditto-jar-mod/reference.conf", "w") as f:
f.write(new_content)
print("reference.conf modified successfully")
PYEOF
echo "=== Modified reference.conf tail ==="
tail -20 reference.conf
echo "=== Updating JAR (replacing reference.conf) ==="
jar uf ditto-gateway-service-3.8.12-allinone.jar reference.conf
echo "=== Verifying modified reference.conf in JAR ==="
jar xf ditto-gateway-service-3.8.12-allinone.jar reference.conf
grep -c "pre-authentication" reference.conf || echo "NOT FOUND in JAR"
echo "=== Done. JAR is ready at $WORKDIR ==="

View File

@@ -0,0 +1,7 @@
ditto {
gateway {
http {
enablecors = true
}
}
}

View File

@@ -0,0 +1,363 @@
ditto {
version = "3.8.12"
extensions {
jwt-authorization-subjects-provider = {
extension-class = org.eclipse.ditto.gateway.service.security.authentication.jwt.DittoJwtAuthorizationSubjectsProvider
}
jwt-authentication-result-provider = {
extension-class = org.eclipse.ditto.gateway.service.security.authentication.jwt.DefaultJwtAuthenticationResultProvider
extension-config = {
role = regular
jwt-authorization-subjects-provider = {
extension-class = org.eclipse.ditto.gateway.service.security.authentication.jwt.DittoJwtAuthorizationSubjectsProvider
extension-config = {
role = regular
}
}
}
}
jwt-authentication-result-provider-devops = {
extension-class = org.eclipse.ditto.gateway.service.security.authentication.jwt.DefaultJwtAuthenticationResultProvider
extension-config = {
role = devops
jwt-authorization-subjects-provider = {
extension-class = org.eclipse.ditto.gateway.service.security.authentication.jwt.DittoJwtAuthorizationSubjectsProvider
extension-config = {
role = devops
}
}
}
}
signal-enrichment-provider {
extension-class = org.eclipse.ditto.gateway.service.endpoints.utils.DefaultGatewaySignalEnrichmentProvider
extension-config = {
cache {
enabled = true
maximum-size = 20000
expire-after-create = 2m
}
}
}
http-bind-flow-provider = org.eclipse.ditto.gateway.service.endpoints.routes.LoggingHttpBindFlowProvider
websocket-config-provider = org.eclipse.ditto.gateway.service.endpoints.routes.websocket.NoOpWebSocketConfigProvider
gateway-authentication-directive-factory = org.eclipse.ditto.gateway.service.endpoints.directives.auth.DittoGatewayAuthenticationDirectiveFactory
http-request-actor-props-factory = org.eclipse.ditto.gateway.service.endpoints.actors.DefaultHttpRequestActorPropsFactory
sse-event-sniffer = org.eclipse.ditto.gateway.service.endpoints.routes.sse.NoOpSseEventSniffer
streaming-authorization-enforcer = org.eclipse.ditto.gateway.service.streaming.NoOpAuthorizationEnforcer
incoming-websocket-event-sniffer = org.eclipse.ditto.gateway.service.endpoints.routes.websocket.NoOpIncomingWebSocketEventSniffer
outgoing-websocket-event-sniffer = org.eclipse.ditto.gateway.service.endpoints.routes.websocket.NoOpOutgoingWebSocketEventSniffer
custom-api-routes-provider = org.eclipse.ditto.gateway.service.endpoints.routes.NoopCustomApiRoutesProvider
sse-connection-supervisor = org.eclipse.ditto.gateway.service.endpoints.routes.sse.NoOpSseConnectionSupervisor
websocket-connection-supervisor = "org.eclipse.ditto.gateway.service.endpoints.routes.websocket.NoOpWebSocketSupervisor"
connections-retrieval-actor-props-factory = org.eclipse.ditto.gateway.service.endpoints.actors.DefaultConnectionsRetrievalActorPropsFactory
}
service-name = "gateway"
mapping-strategy.implementation = "org.eclipse.ditto.gateway.service.util.GatewayMappingStrategies"
gateway {
http {
hostname = ""
hostname = ${?HOSTNAME}
hostname = ${?BIND_HOSTNAME}
port = 8080
port = ${?HTTP_PORT}
port = ${?PORT}
coordinated-shutdown-timeout = 65s
coordinated-shutdown-timeout = ${?COORDINATED_SHUTDOWN_REQUEST_TIMEOUT}
schema-versions = [2]
protocol-headers = ["X-Forwarded-Proto", "x_forwarded_proto"]
forcehttps = false
forcehttps = ${?FORCE_HTTPS}
redirect-to-https = false
redirect-to-https = ${?REDIRECT_TO_HTTPS}
redirect-to-https-blocklist-pattern = "/api.*|/ws.*|/status.*|/overall.*"
enablecors = false
enablecors = ${?ENABLE_CORS}
request-timeout = 60s
request-timeout = ${?REQUEST_TIMEOUT}
additional-accepted-media-types = ${?ADDITIONAL_ACCEPTED_MEDIA_TYPES}
query-params-as-headers = [
"accept"
"channel"
"correlation-id"
"requested-acks"
"declared-acks"
"response-required"
"timeout"
"live-channel-timeout-strategy"
"allow-policy-lockout"
"condition"
"live-channel-condition"
"at-historical-revision"
"at-historical-timestamp"
"dry-run"
]
}
streaming {
session-counter-scrape-interval = 30s
parallelism = 64
parallelism = ${?GATEWAY_STREAMING_PARALLELISM}
search-idle-timeout = 60s
search-idle-timeout = ${?GATEWAY_STREAMING_SEARCH_IDLE_TIMEOUT}
subscription-refresh-delay = 5m
subscription-refresh-delay = ${?GATEWAY_STREAMING_SUBSCRIPTION_REFRESH_DELAY}
acknowledgement {
forwarder-fallback-timeout = 65s
}
websocket {
subscriber {
backpressure-queue-size = 100
}
publisher {
backpressure-buffer-size = 200
}
throttling-rejection-factor = 1.25
throttling {
enabled = false
}
streaming-authorization-enforcer = "org.eclipse.ditto.gateway.service.streaming.NoOpAuthorizationEnforcer"
}
sse {
throttling {
enabled = false
}
streaming-authorization-enforcer = "org.eclipse.ditto.gateway.service.streaming.NoOpAuthorizationEnforcer"
}
}
command {
default-timeout = ${ditto.gateway.http.request-timeout}
max-timeout = 1m
smart-channel-buffer = 10s
connections-retrieve-limit = 100
}
message {
default-timeout = 10s
max-timeout = 1m
}
claim-message {
default-timeout = 1m
max-timeout = 10m
}
dns {
address = none
address = ${?DNS_SERVER}
}
authentication {
http {
proxy {
enabled = false
enabled = ${?AUTH_HTTP_PROXY_ENABLED}
hostname = ${?AUTH_HTTP_PROXY_HOST}
port = ${?AUTH_HTTP_PROXY_PORT}
username = ${?AUTH_HTTP_PROXY_USERNAME}
password = ${?AUTH_HTTP_PROXY_PASSWORD}
}
}
oauth {
protocol = "https"
protocol = ${?OAUTH_PROTOCOL}
allowed-clock-skew = 10s
allowed-clock-skew = ${?OAUTH_ALLOWED_CLOCK_SKEW}
openid-connect-issuers = {
google = {
issuer = "accounts.google.com"
}
}
token-integration-subject = "integration:{{policy-entry:label}}:{{jwt:aud}}"
token-integration-subject = ${?OAUTH_TOKEN_INTEGRATION_SUBJECT}
}
# PRE-AUTHENTICATION = open access for /api/2/
pre-authentication {
enabled = true
}
devops {
secured = false
devops-authentication-method = "basic"
password = "ditto-devops-secret"
password = ${?DEVOPS_PASSWORD}
status-secured = false
status-authentication-method = "basic"
statusPassword = "ditto-status-secret"
statusPassword = ${?STATUS_PASSWORD}
}
}
health-check {
enabled = true
enabled = ${?HEALTH_CHECK_ENABLED}
interval = 60s
interval = ${?HEALTH_CHECK_INTERVAL}
service.timeout = 10s
service.timeout = ${?HEALTH_CHECK_SERVICE_TIMEOUT}
cluster-roles = {
enabled = true
enabled = ${?HEALTH_CHECK_ROLES_ENABLED}
expected = [
"policies"
"things"
"search"
"gateway"
"connectivity"
]
}
}
public-health {
cache-timeout = 20s
cache-timeout = ${?GATEWAY_STATUS_HEALTH_EXTERNAL_TIMEOUT}
}
cloud-events {
empty-schema-allowed = true
data-types = [
"application/json"
"application/vnd.eclipse.ditto+json"
]
}
cache {
publickeys {
maxentries = 32
expiry = 60m
maximum-size = ${ditto.gateway.cache.publickeys.maxentries}
expire-after-write = ${ditto.gateway.cache.publickeys.expiry}
}
}
statistics {
ask-timeout = 5s
ask-timeout = ${?STATISTICS_UPDATE_INTERVAL}
update-interval = 15s
update-interval = ${?STATISTICS_UPDATE_INTERVAL}
details-expire-after = 3s
details-expire-after = ${?STATISTICS_DETAILS_EXPIRE_AFTER}
shards = [
{
region = "thing"
role = "things"
root = "/user/thingsRoot"
}
{
region = "policy"
role = "policies"
root = "/user/policiesRoot"
}
{
region = "search-wildcard-updater"
role = "search"
root = "/user/thingsWildcardSearchRoot/searchUpdaterRoot"
}
]
}
}
tracing {
filter = {
includes = ["**"]
excludes = ["GET /ws/2"]
}
}
}
secrets {
devops_password {
name = "devops_password"
name = ${?DEVOPS_PASSWORD_NAME}
}
status_password {
name = "status_password"
name = ${?STATUS_PASSWORD_NAME}
}
}
pekko.http.client {
user-agent-header = eclipse-ditto/${ditto.version}
}
pekko {
actor {
default-dispatcher {
executor = "org.eclipse.ditto.internal.utils.metrics.service.executor.InstrumentedForkJoinExecutorServiceConfigurator"
}
deployment {
/gatewayRoot/proxy {
router = round-robin-pool
resizer {
lower-bound = 5
upper-bound = 100
messages-per-resize = 50
}
}
}
}
cluster {
sharding {
role = ${ditto.service-name}
passivation {
strategy = "off"
}
}
roles = ["gateway"]
}
coordinated-shutdown {
phases {
service-requests-done {
timeout = 70s
}
}
}
http {
server {
server-header = ""
request-timeout = ${ditto.gateway.http.request-timeout}
idle-timeout = 610s
max-connections = 4096
raw-request-uri-header = on
parsing {
max-uri-length = 8k
max-content-length = 1m
uri-parsing-mode = relaxed
}
websocket {
periodic-keep-alive-mode = ping
periodic-keep-alive-max-idle = 30s
}
termination-deadline-exceeded-response {
status = 502
}
}
host-connection-pool {
max-open-requests = 1024
idle-timeout = 60s
}
}
management.health-checks.readiness-checks {
gateway-http-readiness = "org.eclipse.ditto.gateway.service.health.GatewayHttpReadinessCheck"
}
management.health-checks.liveness-checks {
subsystem-health = "org.eclipse.ditto.internal.utils.health.SubsystemHealthCheck"
}
}
authentication-dispatcher {
type = Dispatcher
executor = "org.eclipse.ditto.internal.utils.metrics.service.executor.InstrumentedThreadPoolExecutorServiceConfigurator"
thread-pool-executor {
core-pool-size-min = 4
core-pool-size-factor = 2.0
core-pool-size-max = 8
}
throughput = 100
}
signal-enrichment-cache-dispatcher {
type = Dispatcher
executor = "org.eclipse.ditto.internal.utils.metrics.service.executor.InstrumentedThreadPoolExecutorServiceConfigurator"
}

View File

@@ -0,0 +1,169 @@
#!/usr/bin/env python3
"""
Modify the reference.conf inside the Ditto gateway JAR.
Inserts ditto { gateway { authentication { pre-authentication { enabled = true } } } }
at the root level, before the final closing brace.
"""
import zipfile
import shutil
import os
import subprocess
import sys
JAR_PATH = "/tmp/ditto-jar-mod/ditto-gateway-service-3.8.12-allinone.jar"
AUTH_BLOCK = """
### Custom Ditto auth override - pre-authentication enabled
ditto {
gateway {
authentication {
pre-authentication {
enabled = true
}
devops {
secured = false
devops-authentication-method = "basic"
password = "ditto-devops-secret"
status-secured = false
status-authentication-method = "basic"
statusPassword = "ditto-status-secret"
}
}
}
}
"""
def main():
os.makedirs("/tmp/ditto-jar-mod", exist_ok=True)
print("=== Step 1: Extracting JAR ===")
result = subprocess.run(
["docker", "run", "--rm", "eclipse/ditto-gateway:latest",
"cat", "/opt/ditto/ditto-gateway-service-3.8.12-allinone.jar"],
capture_output=True, check=True
)
with open(JAR_PATH, "wb") as f:
f.write(result.stdout)
print(f"JAR: {len(result.stdout)} bytes")
print("=== Step 2: Modifying reference.conf ===")
with zipfile.ZipFile(JAR_PATH, 'r') as zin:
ref_conf = zin.read("reference.conf").decode("utf-8")
lines = ref_conf.split('\n')
total_lines = len(lines)
print(f"reference.conf: {total_lines} lines")
# Find the LAST closing brace at root level (depth 0)
# Track depth through the entire file
depth = 0
last_root_close_idx = None
for i, line in enumerate(lines):
stripped = line.strip()
if not stripped or stripped.startswith('#'):
continue
# Count braces (simple approach - count { and })
# This isn't perfect for HOCON but works for our case
for ch in stripped:
if ch == '{':
depth += 1
elif ch == '}':
depth -= 1
if depth == 0:
last_root_close_idx = i
if last_root_close_idx is None:
print("ERROR: Could not find root-level closing brace!")
sys.exit(1)
insert_idx = last_root_close_idx
print(f"Last root-level '}}' at line {insert_idx + 1}: '{lines[insert_idx].strip()}'")
# Check if we need a comma before our block
# Look at the non-empty line before insert_idx
prev_idx = insert_idx - 1
while prev_idx >= 0 and lines[prev_idx].strip() == '':
prev_idx -= 1
if prev_idx >= 0:
prev_stripped = lines[prev_idx].strip()
if prev_stripped.endswith('}'):
# Need to add a comma
lines[prev_idx] = lines[prev_idx].rstrip()
if not lines[prev_idx].endswith(','):
lines[prev_idx] += ','
print(f"Added comma to line {prev_idx + 1}")
# Insert our block
auth_lines = AUTH_BLOCK.split('\n')
new_lines = lines[:insert_idx] + auth_lines + lines[insert_idx:]
modified_conf = '\n'.join(new_lines)
print(f"Modified: {total_lines} -> {len(new_lines)} lines")
# Verify brace balance
depth = 0
for i, line in enumerate(new_lines):
stripped = line.strip()
if not stripped or stripped.startswith('#'):
continue
for ch in stripped:
if ch == '{':
depth += 1
elif ch == '}':
depth -= 1
if depth < 0:
print(f"ERROR: Negative depth at line {i+1}")
sys.exit(1)
print(f"Brace depth check: {depth} (should be 0)")
if depth != 0:
print("ERROR: Unbalanced braces!")
sys.exit(1)
# Verify ditto is at root level
for i, line in enumerate(new_lines):
if line.strip() == 'ditto {':
indent = len(line) - len(line.lstrip())
print(f"'ditto {{' at line {i+1}, indent: {indent}")
if indent != 0:
print(f"WARNING: Expected indent 0, got {indent}")
break
print("=== Step 3: Creating unsigned JAR ===")
skip_files = set()
with zipfile.ZipFile(JAR_PATH, 'r') as zin:
for name in zin.namelist():
if name.startswith("META-INF/"):
upper = name.upper()
if upper.endswith(".SF") or upper.endswith(".RSA") or upper.endswith(".DSA") or upper == "MANIFEST.MF":
skip_files.add(name)
with zipfile.ZipFile(JAR_PATH + ".new", 'w', zipfile.ZIP_DEFLATED) as zout:
for item in zin.infolist():
if item.filename in skip_files:
continue
data = zin.read(item.filename)
if item.filename == "reference.conf":
data = modified_conf.encode("utf-8")
info = zipfile.ZipInfo(filename=item.filename, date_time=item.date_time)
info.compress_type = zipfile.ZIP_DEFLATED
info.external_attr = item.external_attr
zout.writestr(info, data)
shutil.move(JAR_PATH + ".new", JAR_PATH)
print("=== Step 4: Verifying ===")
with zipfile.ZipFile(JAR_PATH, 'r') as z:
ref = z.read("reference.conf").decode("utf-8")
assert "pre-authentication" in ref
sig = [n for n in z.namelist() if n.startswith("META-INF/") and n.upper().endswith(('.SF', '.RSA', '.DSA'))]
print(f"OK Signature files: {len(sig)}")
print(f"OK JAR size: {os.path.getsize(JAR_PATH)}")
print("\n=== DONE ===")
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,70 @@
#!/bin/bash
# Minimal JWT/OAuth2 server for Ditto
# Serves a JWKS endpoint and validates tokens signed with DITTO_JWT_SECRET
cat > /tmp/oauth2-server.js << 'EOF'
const http = require('http');
const crypto = require('crypto');
const SECRET = process.env.DITTO_JWT_SECRET || 'my-ditto-jwt-secret-key-12345';
const PORT = 3000;
function base64url(buf) {
return buf.toString('base64').replace(/\+/g, '-').replace(/\//g, '_').replace(/=/g, '');
}
// Generate a token
function generateToken(sub, scope) {
const header = { alg: 'RS256', typ: 'JWT', kid: 'ditto-local' };
const now = Math.floor(Date.now() / 1000);
const payload = {
iss: 'http://localhost:' + PORT,
sub: sub,
aud: 'ditto:cognito',
iat: now,
exp: now + 3600,
scope: scope
};
const h = base64url(Buffer.from(JSON.stringify(header)));
const p = base64url(Buffer.from(JSON.stringify(payload)));
const sig = base64url(
crypto.createHmac('sha256', SECRET).update(h + '.' + p).digest()
);
return h + '.' + p + '.' + sig;
}
const server = http.createServer((req, res) => {
res.setHeader('Content-Type', 'application/json');
if (req.url === '/.well-known/openid-configuration') {
res.end(JSON.stringify({
issuer: 'http://localhost:' + PORT,
jwks_uri: 'http://localhost:' + PORT + '/.well-known/jwks.json',
token_endpoint: 'http://localhost:' + PORT + '/token'
}));
} else if (req.url === '/.well-known/jwks.json') {
// Extract public key from secret (for HS256 we just return the secret as k)
const jwk = {
kty: 'oct',
kid: 'ditto-local',
use: 'sig',
alg: 'HS256',
k: base64url(Buffer.from(SECRET))
};
res.end(JSON.stringify({ keys: [jwk] }));
} else if (req.url === '/token') {
const token = generateToken('ditto', 'READ_WRITE');
res.end(JSON.stringify({ access_token: token, token_type: 'Bearer', expires_in: 3600 }));
} else {
res.statusCode = 404;
res.end('{}');
}
});
server.listen(PORT, '0.0.0.0', () => {
console.log('OAuth2 server listening on port ' + PORT);
console.log('Token: ' + generateToken('ditto', scope='READ_WRITE'));
});
EOF
node /tmp/oauth2-server.js

View File

@@ -0,0 +1,14 @@
ditto {
gateway {
authentication {
pre-authentication {
enabled = true
}
devops {
secured = false
devops-authentication-method = "basic"
password = "ditto-devops-secret"
}
}
}
}

25
ditto-things.yml Normal file
View File

@@ -0,0 +1,25 @@
services:
ditto-things:
image: eclipse/ditto-things:latest
container_name: smart-city-ditto-things
restart: unless-stopped
hostname: ditto-things
environment:
- TZ=Europe/Berlin
- BIND_HOSTNAME=0.0.0.0
- DITTO_JWT_SECRET=my-ditto-jwt-secret-key-12345
- MONGO_HOST=smart-city-ditto-mongodb
- MONGO_PORT=27017
- MONGO_DB=Things
- MONGODB_URI=mongodb://smart-city-ditto-mongodb:27017/Things
- AKKA_REMOTE_ENABLED=false
- JAVA_TOOL_OPTIONS=-Dditto.things.authentication.devops.password=ditto-devops-secret
networks:
traefik-public:
aliases:
- ditto-cluster
- ditto-things
networks:
traefik-public:
external: true

View File

@@ -0,0 +1,35 @@
# BunkerM MQTT Broker - Smart City Digital Twin
version: '3.8'
networks:
smartcity-shared:
external: true
traefik-public:
external: true
volumes:
bunkerm_mosquitto_data:
external: true
services:
bunkerm:
image: bunkeriot/bunkerm:latest
container_name: bunkerm-bunkerm-1
restart: unless-stopped
networks:
- smartcity-shared
- traefik-public
ports:
- "1884:1900"
- "2000:2000"
environment:
- MQTT_PORT=1900
- CONFIG_API_PORT=2000
volumes:
- bunkerm_mosquitto_data:/var/lib/mosquitto
healthcheck:
test: ["CMD-SHELL", "bash -c 'echo > /dev/tcp/localhost/1900' || exit 1"]
interval: 30s
timeout: 10s
retries: 5
start_period: 60s

View File

@@ -2,6 +2,7 @@ version: "3.8"
services:
chirpstack:
container_name: smart-city-chirpstack
image: chirpstack/chirpstack:latest
command: -c /etc/chirpstack
restart: unless-stopped
@@ -22,9 +23,10 @@ services:
- smartcity-shared
chirpstack-rest-api:
container_name: smart-city-chirpstack-rest-api
image: chirpstack/chirpstack-rest-api:4
restart: unless-stopped
command: --server chirpstack:8080 --bind 0.0.0.0:8090 --insecure
command: --server chirpstack:8080 --bind 0.0.0.0:8090 --insecure --cors-origins="*"
depends_on:
- chirpstack
labels:

View File

@@ -4,6 +4,7 @@
services:
pulsar-distribution:
container_name: smart-city-pulsar-distribution
environment:
- PULSAR_HOST=pulsar
- PULSAR_PORT=6650

View File

@@ -1,6 +1,5 @@
# Eclipse Ditto - Smart City Digital Twin
version: '3.8'
# Eclipse Ditto - Smart City Digital Twin - Martinique
# Using official Eclipse Ditto images with Akka cluster
services:
ditto-mongodb:
image: mongo:6
@@ -14,7 +13,7 @@ services:
- ditto-mongo-data:/data/db
ditto-policies:
image: eclipse/ditto-policies:latest
image: eclipse/ditto-policies:3.8.0
container_name: smart-city-ditto-policies
restart: unless-stopped
hostname: ditto-policies
@@ -23,25 +22,20 @@ services:
environment:
- TZ=Europe/Berlin
- BIND_HOSTNAME=0.0.0.0
- DITTO_JWT_SECRET=my-ditto-jwt-secret-key-12345
- MONGO_HOST=smart-city-ditto-mongodb
- MONGO_PORT=27017
- MONGO_DB=Policies
- DITTO_JWT_SECRET=NTOT-Vh8WRKWE52eV8zRiLs3a-gd8YUGSrvm5x2InZc
- MONGODB_URI=mongodb://smart-city-ditto-mongodb:27017/Policies
- AKKA_REMOTE_ENABLED=false
- AKKA_REMOTE_ENABLED=true
- AKKA_REMOTE_CANONICAL_HOSTNAME=ditto-policies
- AKKA_REMOTE_CANONICAL_PORT=2551
- JAVA_TOOL_OPTIONS=-Dditto.mongodb.uri=mongodb://smart-city-ditto-mongodb:27017/Policies -Dditto.mongodb.db-name=Policies
networks:
traefik-public:
aliases:
- ditto-cluster
- ditto-policies
labels:
- "traefik.enable=true"
- "traefik.http.routers.ditto-policies.rule=Host(`ditto-policies.digitribe.fr`)"
- "traefik.http.routers.ditto-policies.entrypoints=web"
- "traefik.http.services.ditto-policies.loadbalancer.server.port=8080"
ditto-things:
image: eclipse/ditto-things:latest
image: eclipse/ditto-things:3.8.0
container_name: smart-city-ditto-things
restart: unless-stopped
hostname: ditto-things
@@ -51,22 +45,17 @@ services:
environment:
- TZ=Europe/Berlin
- BIND_HOSTNAME=0.0.0.0
- DITTO_JWT_SECRET=my-ditto-jwt-secret-key-12345
- MONGO_HOST=smart-city-ditto-mongodb
- MONGO_PORT=27017
- MONGO_DB=Things
- DITTO_JWT_SECRET=NTOT-Vh8WRKWE52eV8zRiLs3a-gd8YUGSrvm5x2InZc
- MONGODB_URI=mongodb://smart-city-ditto-mongodb:27017/Things
- AKKA_REMOTE_ENABLED=false
- AKKA_REMOTE_ENABLED=true
- AKKA_REMOTE_CANONICAL_HOSTNAME=ditto-things
- AKKA_REMOTE_CANONICAL_PORT=2551
- JAVA_TOOL_OPTIONS=-Dditto.mongodb.uri=mongodb://smart-city-ditto-mongodb:27017/Things -Dditto.mongodb.db-name=Things -Dditto.things.authentication.devops.password=OvP9WVB09aFDnYPyK52UIg
networks:
traefik-public:
aliases:
- ditto-cluster
- ditto-things
labels:
- "traefik.enable=true"
- "traefik.http.routers.ditto-things.rule=Host(`ditto-things.digitribe.fr`)"
- "traefik.http.routers.ditto-things.entrypoints=web"
- "traefik.http.services.ditto-things.loadbalancer.server.port=8080"
ditto-gateway:
image: eclipse/ditto-gateway:latest
@@ -79,15 +68,18 @@ services:
environment:
- TZ=Europe/Berlin
- BIND_HOSTNAME=0.0.0.0
- DITTO_JWT_SECRET=my-ditto-jwt-secret-key-12345
- DITTO_JWT_SECRET=NTOT-Vh8WRKWE52eV8zRiLs3a-gd8YUGSrvm5x2InZc
- DITTO_GATEWAY_PROXY_ENABLED=true
- AKKA_REMOTE_ENABLED=false
- AKKA_REMOTE_ENABLED=true
- AKKA_REMOTE_CANONICAL_HOSTNAME=ditto-gateway
- AKKA_REMOTE_CANONICAL_PORT=2551
- DITTO_GW_STREAMING_ENABLED=true
- DITTO_GW_MQTT_BROKER=smart-city-mosquitto:1883
- DITTO_GW_MQTT_BROKER=192.168.192.26:1883
- DITTO_GW_MQTT_TOPIC_FILTER=smartcity/#
- DEVOPS_PASSWORD=ditto-devops-secret
- ENABLE_PRE_AUTHENTICATION=true
- JAVA_TOOL_OPTIONS=-Dditto.gateway.authentication.devops.password=ditto-devops-secret -Dditto.gateway.authentication.devops.secured=true -Dditto.gateway.authentication.devops.devops-authentication-method=basic
- DEVOPS_PASSWORD=OvP9WVB09aFDnYPyK52UIg
- JAVA_TOOL_OPTIONS=-Xms512m -Xmx1024m -Dditto.gateway.http.port=8080 -Dditto.gateway.http.api.enabled=true
- DITTO_APIDOC_ENABLED=true
- DITTO_GATEWAY_HTTP_API_ENABLED=true
networks:
traefik-public:
aliases:
@@ -95,14 +87,28 @@ services:
- ditto-gateway
labels:
- "traefik.enable=true"
- "traefik.http.routers.ditto-gateway.rule=Host(`ditto.digitribe.fr`)"
- "traefik.http.routers.ditto-gateway.entrypoints=websecure"
- "traefik.http.routers.ditto-gateway.tls.certresolver=letsencrypt"
- "traefik.http.services.ditto-gateway.loadbalancer.server.port=8080"
- "traefik.http.routers.ditto.rule=Host(`ditto.digitribe.fr`)"
- "traefik.http.routers.ditto.entrypoints=websecure"
- "traefik.http.routers.ditto.tls.certresolver=letsencrypt"
- "traefik.http.services.ditto.loadbalancer.server.port=8080"
ditto-ui:
image: eclipse/ditto-ui:latest
container_name: smart-city-ditto-ui
restart: unless-stopped
depends_on:
- ditto-gateway
networks:
traefik-public:
aliases:
- ditto-ui
networks:
traefik-public:
external: true
smartcity-shared:
external: true
volumes:
ditto-mongo-data:
name: smart-city-digital-twin-martinique_ditto-mongo-data

29
docker-compose.emqx.yml Normal file
View File

@@ -0,0 +1,29 @@
services:
emqx:
image: emqx/emqx:5.4
container_name: emqx_emqx_1
restart: unless-stopped
networks:
- smartcity-shared
ports:
- "1885:1883"
- "8083:8083"
- "8883:8883"
- "8084:8084"
- "18083:18083"
environment:
- EMQX_NAME=emqx
- EMQX_HOST=emqx_emqx_1
volumes:
- emqx-data:/opt/emqx/data
- emqx-log:/opt/emqx/log
volumes:
emqx-data:
name: smart-city-emqx-data
emqx-log:
name: smart-city-emqx-log
networks:
smartcity-shared:
external: true

View File

@@ -23,7 +23,7 @@ services:
- IOTA_REGISTRY_TYPE=memory
# MQTT Listener - EMQX
- IOTA_MQTT_HOST=emqx_emqx_1
- IOTA_MQTT_PORT=1883
- IOTA_MQTT_PORT=1885
- IOTA_PROVIDER_URL=http://smart-city-iot-agent-emqx:4041
- IOTA_DEFAULT_RESOURCE=/
- IOTA_DEFAULT_APIKEY=smartcity-emqx

View File

@@ -0,0 +1,71 @@
# Metabase - BI Dashboard for Smart City Digital Twin
# Usage: docker compose -f docker-compose.metabase.yml up -d
# Access: https://metabase.digitribe.fr
version: '3.8'
networks:
smartcity-shared:
external: true
traefik-public:
external: true
volumes:
metabase_data:
name: smart-city-metabase-data
services:
metabase-db:
image: postgres:15-alpine
container_name: metabase-postgres
restart: unless-stopped
networks:
- smartcity-shared
environment:
POSTGRES_DB: metabase
POSTGRES_USER: metabase
POSTGRES_PASSWORD: Digitribe972
volumes:
- metabase_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U metabase"]
interval: 10s
timeout: 5s
retries: 5
metabase:
image: metabase/metabase:latest
container_name: metabase-app
restart: unless-stopped
networks:
- smartcity-shared
- traefik-public
depends_on:
metabase-db:
condition: service_healthy
environment:
MB_DB_TYPE: postgres
MB_DB_DBNAME: metabase
MB_DB_PORT: 5432
MB_DB_USER: metabase
MB_DB_PASS: Digitribe972
MB_DB_HOST: metabase-postgres
MB_SITE_NAME: "Smart City Martinique"
MB_SITE_URL: "https://metabase.digitribe.fr"
MB_APPLICATION_DB: "file:/metabase-data/metabase.db"
MB_ENABLE_PASSWORD_LOGIN: "true"
MB_ADMIN_EMAIL: admin@digitribe.fr
MB_ADMIN_PASSWORD: Digitribe972
MB_JETTY_PORT: 3000
labels:
- "traefik.enable=true"
- "traefik.http.routers.metabase.rule=Host(`metabase.digitribe.fr`)"
- "traefik.http.routers.metabase.entrypoints=websecure"
- "traefik.http.routers.metabase.tls.certresolver=letsencrypt"
- "traefik.http.services.metabase.loadbalancer.server.port=3000"
healthcheck:
test: ["CMD-SHELL", "curl -f http://localhost:3000/api/health || exit 1"]
interval: 30s
timeout: 10s
retries: 5
start_period: 120s

View File

@@ -13,7 +13,7 @@ services:
- orion-ld
- smart-city-orion-ld
traefik-public:
command: -dbhost smart-city-mongodb -db orion
command: -dbhost smart-city-iot-mongodb -db orion
healthcheck:
test: ["CMD-SHELL", "curl -f http://localhost:1026/version || exit 1"]
interval: 30s

View File

@@ -1,28 +1,16 @@
# Redpanda → InfluxDB Consumer
# Lit les topics Redpanda et écrit dans InfluxDB pour Grafana
version: "3.8"
# DÉSACTIVÉ — Redpanda broker non démarré
# Usage: docker compose -f docker-compose.redpanda-consumer.yml up -d
services:
redpanda-consumer:
image: python:3.11-slim
container_name: smart-city-redpanda-consumer
restart: unless-stopped
restart: "no"
command: >
sh -c "pip install requests && python3 /app/consumer.py"
volumes:
- ./redpanda/consumer.py:/app/consumer.py:ro
environment:
- INFLUX_URL=http://smart-city-influxdb:8086
- INFLUX_TOKEN=my-super-admin-token
- INFLUX_ORG=digitribe
- INFLUX_BUCKET=iot_data
sh -c "echo 'Redpanda consumer désactivé — Redpanda broker non démarré' && sleep infinity"
networks:
- smartcity-shared
healthcheck:
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://smart-city-redpanda:9644/public_metrics')"]
interval: 30s
timeout: 10s
retries: 3
networks:
smartcity-shared:

View File

@@ -0,0 +1,39 @@
# Apache Superset - Smart City Digital Twin
# Uses official apache/superset Docker Hub image
# Access: https://superset.digitribe.fr
networks:
smartcity-shared:
external: true
traefik-public:
external: true
volumes:
superset_home:
services:
superset:
image: apache/superset:latest
container_name: superset-app
restart: unless-stopped
networks:
- smartcity-shared
- traefik-public
environment:
# Use Superset's built-in SQLite for metadata (simplest setup)
# For production, replace with PostgreSQL
SUPERSET_SECRET_KEY: superset-secret-key-2024-change-me
volumes:
- superset_home:/app/superset_home
labels:
- "traefik.enable=true"
- "traefik.http.routers.superset.rule=Host(`superset.digitribe.fr`)"
- "traefik.http.routers.superset.entrypoints=websecure"
- "traefik.http.routers.superset.tls.certresolver=letsencrypt"
- "traefik.http.services.superset.loadbalancer.server.port=8088"
healthcheck:
test: ["CMD-SHELL", "curl -f http://localhost:8088/health || exit 1"]
interval: 30s
timeout: 15s
retries: 5
start_period: 120s

View File

@@ -11,6 +11,7 @@ version: "3.8"
services:
tts-postgres:
container_name: smart-city-tts-postgres
image: postgres:14
restart: unless-stopped
environment:
@@ -22,6 +23,7 @@ services:
networks:
- smartcity-shared
tts-redis:
container_name: smart-city-tts-redis
image: redis:7
command: redis-server --appendonly yes
restart: unless-stopped
@@ -30,6 +32,7 @@ services:
networks:
- smartcity-shared
tts-stack:
container_name: smart-city-tts-stack
image: thethingsnetwork/lorawan-stack:latest
entrypoint: ttn-lw-stack -c /config/ttn-lw-stack-docker.yml
command: start

View File

@@ -24,28 +24,34 @@ services:
- traefik-public
- openremote_default
environment:
# MQTT Brokers - ALL enabled
# MQTT Brokers
- ENABLE_EMQX=1
- ENABLE_MOSQUITTO=1
- ENABLE_BUNKER=1
- EMQX_HOST=emqx_emqx_1
- EMQX_PORT=1883
- MOSQUITTO_HOST=smart-city-mosquitto-1
- MOSQUITTO_PORT=1883
- BUNKERM_HOST=bunkerm-bunkerm-1
- BUNKERM_PORT=1900
# Context Brokers (DESACTIVE - tout passe par les IoT Agents via MQTT)
- ENABLE_ORION=false
- ENABLE_STELLIO=false
- ENABLE_FROST=false
# Databases
- ENABLE_INFLUX=true
# Databases (DESACTIVE - Telegraf s'occupe de InfluxDB)
- ENABLE_INFLUX=false
- INFLUX_URL=http://smart-city-influxdb:8086
# OpenRemote
- ENABLE_OPENREMOTE=1
- OR_URL=http://openremote_manager_1:8080
- ENABLE_OPENREMOTE=0
- OR_MQTT_USER=admin
- OR_MQTT_PASS=Digitribe972
- OR_URL=http://openremote-manager:8080
- OR_REALM=master
- OR_TOKEN_REALM=master
- OR_ADMIN_USER=admin
- OR_ADMIN_PASS=Digitribe972
- OR_CLIENT_SECRET=0oQjzTfiEELYmj5jFwT4iIuWUDtQDvVa
# Pulsar (Disabled for demo stability)
# Pulsar (Disabled for stability)
- ENABLE_PULSAR=false
# Redpanda (Disabled)
- ENABLE_REDPANDA=false
@@ -66,7 +72,7 @@ services:
- traefik-public
- openremote_default
environment:
- OR_URL=http://openremote_manager_1:8080
- OR_URL=http://openremote-manager:8080
- OR_ADMIN_USER=admin
- OR_ADMIN_PASS=Digitribe972
- OR_REALM=master

File diff suppressed because one or more lines are too long

48
flink/docker-compose.yml Normal file
View File

@@ -0,0 +1,48 @@
# Smart City Digital Twin Martinique — Apache Flink
# Usage: docker compose -f flink/docker-compose.yml up -d
# Image officielle Apache Flink 1.20.1 avec digest vérifié
networks:
smartcity-shared:
external: true
services:
jobmanager:
image: apache/flink:1.20.1-scala_2.12-java17@sha256:ecc5785594eff2d94e29e6b116b3124c0cdb3a9c952ebdf38ef0fef90fb9913d
container_name: flink-jobmanager
command: jobmanager
networks:
- smartcity-shared
ports:
- "8081:8081" # Flink Web UI
environment:
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: jobmanager
jobmanager.memory.process.size: 1024m
taskmanager.memory.process.size: 1024m
taskmanager.numberOfTaskSlots: 4
parallelism.default: 2
rest.port: 8081
restart: unless-stopped
labels:
- "traefik.enable=false"
taskmanager:
image: apache/flink:1.20.1-scala_2.12-java17@sha256:ecc5785594eff2d94e29e6b116b3124c0cdb3a9c952ebdf38ef0fef90fb9913d
container_name: flink-taskmanager
command: taskmanager
networks:
- smartcity-shared
depends_on:
- jobmanager
environment:
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: jobmanager
taskmanager.memory.process.size: 1024m
taskmanager.numberOfTaskSlots: 4
parallelism.default: 2
restart: unless-stopped
labels:
- "traefik.enable=false"

View File

@@ -1,7 +1,5 @@
{
"annotations": {
"list": []
},
"annotations": {"list": []},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
@@ -9,135 +7,166 @@
"links": [],
"panels": [
{
"title": "Air Quality (PM2.5)",
"title": "Air Quality PM2.5 (µg/m³)",
"type": "timeseries",
"datasource": {
"type": "influxdb",
"uid": "influxdb-smartcity"
},
"datasource": {"type": "influxdb", "uid": "influxdb-smartcity"},
"targets": [
{
"query": "from(bucket:\"smartcity\") |> range(start: v.timeRangeStart, stop:v.timeRangeStop) |> filter(fn: (r) => r[\"_measurement\"] == \"airquality\") |> filter(fn: (r) => r[\"_field\"] == \"pm25_ugm3\") |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false) |> yield(name: \"mean\")"
"query": "from(bucket:\"smartcity\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"topic\"] =~ /smartcity\\/airquality/)\n |> filter(fn: (r) => r[\"_field\"] == \"pm25_ugm3\")\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> yield(name: \"mean\")"
}
],
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 0
}
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 0}
},
{
"title": "Traffic Flow (Vehicles)",
"title": "Air Quality — NO2 (µg/m³)",
"type": "timeseries",
"datasource": {
"type": "influxdb",
"uid": "influxdb-smartcity"
},
"datasource": {"type": "influxdb", "uid": "influxdb-smartcity"},
"targets": [
{
"query": "from(bucket:\"smartcity\") |> range(start: v.timeRangeStart, stop:v.timeRangeStop) |> filter(fn: (r) => r[\"_measurement\"] == \"traffic\") |> filter(fn: (r) => r[\"_field\"] == \"vehicle_count\") |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false) |> yield(name: \"mean\")"
"query": "from(bucket:\"smartcity\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"topic\"] =~ /smartcity\\/airquality/)\n |> filter(fn: (r) => r[\"_field\"] == \"no2_ugm3\")\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> yield(name: \"mean\")"
}
],
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 0
}
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 0}
},
{
"title": "Parking Occupancy (%)",
"title": "Temperature (°C)",
"type": "timeseries",
"datasource": {
"type": "influxdb",
"uid": "influxdb-smartcity"
},
"datasource": {"type": "influxdb", "uid": "influxdb-smartcity"},
"targets": [
{
"query": "from(bucket:\"smartcity\") |> range(start: v.timeRangeStart, stop:v.timeRangeStop) |> filter(fn: (r) => r[\"_measurement\"] == \"parking\") |> filter(fn: (r) => r[\"_field\"] == \"occupancy_percent\") |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false) |> yield(name: \"mean\")"
"query": "from(bucket:\"smartcity\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"topic\"] =~ /smartcity\\/weather/)\n |> filter(fn: (r) => r[\"_field\"] == \"temperature_celsius\")\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> yield(name: \"mean\")"
}
],
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 8
}
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 8}
},
{
"title": "Noise Levels (dB)",
"title": "Humidity (%)",
"type": "timeseries",
"datasource": {
"type": "influxdb",
"uid": "influxdb-smartcity"
},
"datasource": {"type": "influxdb", "uid": "influxdb-smartcity"},
"targets": [
{
"query": "from(bucket:\"smartcity\") |> range(start: v.timeRangeStart, stop:v.timeRangeStop) |> filter(fn: (r) => r[\"_measurement\"] == \"noise\") |> filter(fn: (r) => r[\"_field\"] == \"noise_level_db\") |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false) |> yield(name: \"mean\")"
"query": "from(bucket:\"smartcity\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"topic\"] =~ /smartcity\\/weather/)\n |> filter(fn: (r) => r[\"_field\"] == \"humidity_percent\")\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> yield(name: \"mean\")"
}
],
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 8
}
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 8}
},
{
"title": "Weather (Temperature \u00b0C)",
"title": "Wind Speed (km/h)",
"type": "timeseries",
"datasource": {
"type": "influxdb",
"uid": "influxdb-smartcity"
},
"datasource": {"type": "influxdb", "uid": "influxdb-smartcity"},
"targets": [
{
"query": "from(bucket:\"smartcity\") |> range(start: v.timeRangeStart, stop:v.timeRangeStop) |> filter(fn: (r) => r[\"_measurement\"] == \"weather\") |> filter(fn: (r) => r[\"_field\"] == \"temperature_c\") |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false) |> yield(name: \"mean\")"
"query": "from(bucket:\"smartcity\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"topic\"] =~ /smartcity\\/weather/)\n |> filter(fn: (r) => r[\"_field\"] == \"wind_speed_kmh\")\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> yield(name: \"mean\")"
}
],
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 16
}
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 16}
},
{
"title": "Light Levels",
"title": "Rain (mm)",
"type": "timeseries",
"datasource": {
"type": "influxdb",
"uid": "influxdb-smartcity"
},
"datasource": {"type": "influxdb", "uid": "influxdb-smartcity"},
"targets": [
{
"query": "from(bucket:\"smartcity\") |> range(start: v.timeRangeStart, stop:v.timeRangeStop) |> filter(fn: (r) => r[\"_measurement\"] == \"light\") |> filter(fn: (r) => r[\"_field\"] == \"luminosity\") |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false) |> yield(name: \"mean\")"
"query": "from(bucket:\"smartcity\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"topic\"] =~ /smartcity\\/weather/)\n |> filter(fn: (r) => r[\"_field\"] == \"rain_mm\")\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> yield(name: \"mean\")"
}
],
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 16
}
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 16}
},
{
"title": "Traffic — Vehicle Count",
"type": "timeseries",
"datasource": {"type": "influxdb", "uid": "influxdb-smartcity"},
"targets": [
{
"query": "from(bucket:\"smartcity\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"topic\"] =~ /smartcity\\/traffic/)\n |> filter(fn: (r) => r[\"_field\"] == \"vehicle_count\")\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> yield(name: \"mean\")"
}
],
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 24}
},
{
"title": "Traffic — Avg Speed (km/h)",
"type": "timeseries",
"datasource": {"type": "influxdb", "uid": "influxdb-smartcity"},
"targets": [
{
"query": "from(bucket:\"smartcity\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"topic\"] =~ /smartcity\\/traffic/)\n |> filter(fn: (r) => r[\"_field\"] == \"average_speed_kmh\")\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> yield(name: \"mean\")"
}
],
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 24}
},
{
"title": "Parking — Available Spots",
"type": "timeseries",
"datasource": {"type": "influxdb", "uid": "influxdb-smartcity"},
"targets": [
{
"query": "from(bucket:\"smartcity\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"topic\"] =~ /smartcity\\/parking/)\n |> filter(fn: (r) => r[\"_field\"] == \"available_spots\")\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> yield(name: \"mean\")"
}
],
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 32}
},
{
"title": "Parking — Occupancy (%)",
"type": "timeseries",
"datasource": {"type": "influxdb", "uid": "influxdb-smartcity"},
"targets": [
{
"query": "from(bucket:\"smartcity\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"topic\"] =~ /smartcity\\/parking/)\n |> filter(fn: (r) => r[\"_field\"] == \"occupancy_percent\")\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> yield(name: \"mean\")"
}
],
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 32}
},
{
"title": "Noise Level (dB)",
"type": "timeseries",
"datasource": {"type": "influxdb", "uid": "influxdb-smartcity"},
"targets": [
{
"query": "from(bucket:\"smartcity\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"topic\"] =~ /smartcity\\/noise/)\n |> filter(fn: (r) => r[\"_field\"] == \"noise_level_db\")\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> yield(name: \"mean\")"
}
],
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 40}
},
{
"title": "Light — Brightness (lux)",
"type": "timeseries",
"datasource": {"type": "influxdb", "uid": "influxdb-smartcity"},
"targets": [
{
"query": "from(bucket:\"smartcity\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"topic\"] =~ /smartcity\\/light/)\n |> filter(fn: (r) => r[\"_field\"] == \"brightness_lux\")\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> yield(name: \"mean\")"
}
],
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 40}
},
{
"title": "UV Index",
"type": "timeseries",
"datasource": {"type": "influxdb", "uid": "influxdb-smartcity"},
"targets": [
{
"query": "from(bucket:\"smartcity\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"topic\"] =~ /smartcity\\/weather/)\n |> filter(fn: (r) => r[\"_field\"] == \"uv_index\")\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> yield(name: \"mean\")"
}
],
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 48}
},
{
"title": "Pressure (hPa)",
"type": "timeseries",
"datasource": {"type": "influxdb", "uid": "influxdb-smartcity"},
"targets": [
{
"query": "from(bucket:\"smartcity\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"topic\"] =~ /smartcity\\/weather/)\n |> filter(fn: (r) => r[\"_field\"] == \"pressure_hpa\")\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> yield(name: \"mean\")"
}
],
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 48}
}
],
"schemaVersion": 36,
"style": "dark",
"tags": [
"smartcity",
"martinique",
"iot"
],
"templating": {
"list": []
},
"time": {
"from": "now-1h",
"to": "now"
},
"title": "Smart City Digital Twin - Martinique",
"tags": ["smartcity", "martinique", "iot"],
"templating": {"list": []},
"time": {"from": "now-1h", "to": "now"},
"title": "Smart City Digital Twin — Martinique",
"uid": "smartcity-martinique-v2",
"version": 1
}
"version": 2
}

View File

@@ -2,13 +2,18 @@
apiVersion: 1
datasources:
- name: InfluxDB
- name: influxdb-smartcity
type: influxdb
access: proxy
url: http://docker-influxdb-1:8086
database: iot_data
user: admin
password: digitribe972
url: http://smart-city-influxdb:8086
database: smartcity
jsonData:
version: Flux
organization: digitribe
defaultBucket: smartcity
tlsSkipVerify: true
secureJsonData:
token: my-super-token
isDefault: true
readOnly: false

View File

@@ -1,16 +0,0 @@
{
"annotations": {"list": []},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"links": [],
"panels": [],
"schemaVersion": 36,
"style": "dark",
"tags": ["smart-city"],
"templating": {"list": []},
"time": {"from": "now-24h", "to": "now"},
"title": "Smart City Dashboards",
"timezone": "Americas/Martinique",
"uid": "smart-city-dashboards"
}

View File

@@ -0,0 +1,172 @@
{
"annotations": {"list": []},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"id": null,
"links": [],
"panels": [
{
"title": "Air Quality — PM2.5 (µg/m³)",
"type": "timeseries",
"datasource": {"type": "influxdb", "name": "Influxdb-v2"},
"targets": [
{
"query": "from(bucket:\"smartcity\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"topic\"] =~ /smartcity\\/airquality/)\n |> filter(fn: (r) => r[\"_field\"] == \"pm25_ugm3\")\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> yield(name: \"mean\")"
}
],
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 0}
},
{
"title": "Air Quality — NO2 (µg/m³)",
"type": "timeseries",
"datasource": {"type": "influxdb", "name": "Influxdb-v2"},
"targets": [
{
"query": "from(bucket:\"smartcity\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"topic\"] =~ /smartcity\\/airquality/)\n |> filter(fn: (r) => r[\"_field\"] == \"no2_ugm3\")\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> yield(name: \"mean\")"
}
],
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 0}
},
{
"title": "Temperature (°C)",
"type": "timeseries",
"datasource": {"type": "influxdb", "name": "Influxdb-v2"},
"targets": [
{
"query": "from(bucket:\"smartcity\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"topic\"] =~ /smartcity\\/weather/)\n |> filter(fn: (r) => r[\"_field\"] == \"temperature_celsius\")\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> yield(name: \"mean\")"
}
],
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 8}
},
{
"title": "Humidity (%)",
"type": "timeseries",
"datasource": {"type": "influxdb", "name": "Influxdb-v2"},
"targets": [
{
"query": "from(bucket:\"smartcity\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"topic\"] =~ /smartcity\\/weather/)\n |> filter(fn: (r) => r[\"_field\"] == \"humidity_percent\")\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> yield(name: \"mean\")"
}
],
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 8}
},
{
"title": "Wind Speed (km/h)",
"type": "timeseries",
"datasource": {"type": "influxdb", "name": "Influxdb-v2"},
"targets": [
{
"query": "from(bucket:\"smartcity\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"topic\"] =~ /smartcity\\/weather/)\n |> filter(fn: (r) => r[\"_field\"] == \"wind_speed_kmh\")\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> yield(name: \"mean\")"
}
],
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 16}
},
{
"title": "Rain (mm)",
"type": "timeseries",
"datasource": {"type": "influxdb", "name": "Influxdb-v2"},
"targets": [
{
"query": "from(bucket:\"smartcity\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"topic\"] =~ /smartcity\\/weather/)\n |> filter(fn: (r) => r[\"_field\"] == \"rain_mm\")\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> yield(name: \"mean\")"
}
],
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 16}
},
{
"title": "Traffic — Vehicle Count",
"type": "timeseries",
"datasource": {"type": "influxdb", "name": "Influxdb-v2"},
"targets": [
{
"query": "from(bucket:\"smartcity\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"topic\"] =~ /smartcity\\/traffic/)\n |> filter(fn: (r) => r[\"_field\"] == \"vehicle_count\")\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> yield(name: \"mean\")"
}
],
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 24}
},
{
"title": "Traffic — Avg Speed (km/h)",
"type": "timeseries",
"datasource": {"type": "influxdb", "name": "Influxdb-v2"},
"targets": [
{
"query": "from(bucket:\"smartcity\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"topic\"] =~ /smartcity\\/traffic/)\n |> filter(fn: (r) => r[\"_field\"] == \"average_speed_kmh\")\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> yield(name: \"mean\")"
}
],
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 24}
},
{
"title": "Parking — Available Spots",
"type": "timeseries",
"datasource": {"type": "influxdb", "name": "Influxdb-v2"},
"targets": [
{
"query": "from(bucket:\"smartcity\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"topic\"] =~ /smartcity\\/parking/)\n |> filter(fn: (r) => r[\"_field\"] == \"available_spots\")\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> yield(name: \"mean\")"
}
],
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 32}
},
{
"title": "Parking — Occupancy (%)",
"type": "timeseries",
"datasource": {"type": "influxdb", "name": "Influxdb-v2"},
"targets": [
{
"query": "from(bucket:\"smartcity\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"topic\"] =~ /smartcity\\/parking/)\n |> filter(fn: (r) => r[\"_field\"] == \"occupancy_percent\")\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> yield(name: \"mean\")"
}
],
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 32}
},
{
"title": "Noise Level (dB)",
"type": "timeseries",
"datasource": {"type": "influxdb", "name": "Influxdb-v2"},
"targets": [
{
"query": "from(bucket:\"smartcity\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"topic\"] =~ /smartcity\\/noise/)\n |> filter(fn: (r) => r[\"_field\"] == \"noise_level_db\")\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> yield(name: \"mean\")"
}
],
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 40}
},
{
"title": "Light — Brightness (lux)",
"type": "timeseries",
"datasource": {"type": "influxdb", "name": "Influxdb-v2"},
"targets": [
{
"query": "from(bucket:\"smartcity\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"topic\"] =~ /smartcity\\/light/)\n |> filter(fn: (r) => r[\"_field\"] == \"brightness_lux\")\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> yield(name: \"mean\")"
}
],
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 40}
},
{
"title": "UV Index",
"type": "timeseries",
"datasource": {"type": "influxdb", "name": "Influxdb-v2"},
"targets": [
{
"query": "from(bucket:\"smartcity\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"topic\"] =~ /smartcity\\/weather/)\n |> filter(fn: (r) => r[\"_field\"] == \"uv_index\")\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> yield(name: \"mean\")"
}
],
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 48}
},
{
"title": "Pressure (hPa)",
"type": "timeseries",
"datasource": {"type": "influxdb", "name": "Influxdb-v2"},
"targets": [
{
"query": "from(bucket:\"smartcity\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"topic\"] =~ /smartcity\\/weather/)\n |> filter(fn: (r) => r[\"_field\"] == \"pressure_hpa\")\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> yield(name: \"mean\")"
}
],
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 48}
}
],
"schemaVersion": 36,
"style": "dark",
"tags": ["smartcity", "martinique", "iot"],
"templating": {"list": []},
"time": {"from": "now-1h", "to": "now"},
"title": "Smart City Digital Twin — Martinique",
"uid": "smartcity-martinique-v2",
"version": 2
}

View File

@@ -1,9 +1,8 @@
# Grafana datasources - Smart City Digital Twin Martinique
# Each datasource is editable and uses the container DNS name in smartcity-shared network
apiVersion: 1
datasources:
# ── InfluxDB v2 (time-series IoT data) ──────────────────────────────────────
# InfluxDB v2 (time-series IoT data)
- name: InfluxDB-v2
type: influxdb
access: proxy
@@ -13,12 +12,12 @@ datasources:
jsonData:
version: Flux
organization: digitribe
defaultBucket: iot_data
defaultBucket: smartcity
tlsSkipVerify: true
secureJsonData:
token: my-super-secret-admin-token
token: my-super-token
# ── FIWARE Orion-LD (NGSI-LD context broker) ────────────────────────────────
# Requires grafana-simple-json-datasource plugin
# FIWARE Orion-LD
- name: FIWARE Orion
type: grafana-simple-json-datasource
access: proxy
@@ -28,8 +27,7 @@ datasources:
queryURLTemplate: "/ngsi-ld/v1/entities?type={{type}}"
method: GET
# ── GeoServer WMS (spatial data) ────────────────────────────────────────────
# GeoServer is an external service reachable via its container name
# GeoServer WMS
- name: GeoServer WMS
type: grafana-simple-json-datasource
access: proxy
@@ -39,8 +37,7 @@ datasources:
queryURLTemplate: "/geoserver/wfs?service=WFS&version=2.0&request=GetFeature&typeName={{type}}"
method: GET
# ── FROST-Server (SensorThings API) ──────────────────────────────────────────
# Requires grafana-simple-json-datasource plugin
# FROST-Server
- name: FROST-Server
type: grafana-simple-json-datasource
access: proxy

246
helms/README.md Normal file
View File

@@ -0,0 +1,246 @@
# Smart City Martinique - Déploiement Kubernetes
## Architecture
```
┌─────────────────────────────────────────────────────────┐
│ TRAEFIK (Ingress) │
│ ports 80/443 │
└─────────────────────────────────────────────────────────┘
┌─────────────────────────────────┼─────────────────────────────────┐
│ │ │
┌────▼────┐ ┌──────────┐ ┌──────────▼──────────┐ ┌─────────────────┐
│ Airflow │ │ Kafka │ │ Data & Storage │ │ Monitoring │
│ │ │ Cluster │ │ │ │ │
│ web │ │ 3 brokers│ │ PostgreSQL HA │ │ Prometheus │
│ sched │ │ connect │ │ Redis Cluster │ │ Grafana │
│ worker │ │ ui │ │ MinIO │ │ Loki │
└─────────┘ └──────────┘ │ ClickHouse │ │ Promtail │
│ StarRocks │ └─────────────────┘
┌──────────┐ ┌──────────┐ │ Trino │
│ Flink │ │ IoT │ │ Delta Lake │ ┌─────────────────┐
│ │ │ │ │ DuckDB │ │ BI & Analytics │
│ jobmgr │ │ EMQX │ └─────────────────────┘ │ │
│ taskmgr │ │ Mosquitto│ │ Superset │
└──────────┘ │ Node-RED │ ┌─────────────────────┐ │ Metabase │
│ phpIPAM │ │ Git & Notebooks │ │ MindsDB │
┌──────────┐ │ ChirpStk │ │ │ └─────────────────┘
│ GIS │ └──────────┘ │ Gitea │
│ │ │ JupyterHub │ ┌─────────────────┐
│ MapStore │ ┌──────────┐ │ Zeppelin │ │ Web Apps │
│ GeoServer│ │ ODK │ └─────────────────────┘ │ │
│ FROST │ │ │ │ Smart App │
│ Stellio │ │ nginx │ ┌─────────────────────┐ │ Streamlit │
│ FIWARE │ │ service │ │ Data Collection │ │ Kepler │
└──────────┘ │ postgres │ │ │ └─────────────────┘
└──────────┘ │ Telegraf │
│ InfluxDB │
│ Simulator │
└─────────────────────┘
```
## Prérequis
### Cluster Kubernetes
- 3 nœuds minimum (1 master + 2 workers)
- Kubernetes 1.28+
- containerd
- Cilium (CNI)
### Serveur NFS
- 1 serveur NFS pour le stockage persistant
- Minimum 500Go d'espace disque
### Outils
- kubectl
- helm
- ansible 2.15+
- ansible-galaxy collection install kubernetes.core
## Installation
### 1. Cloner le repository
```bash
git clone https://gitea.digitribe.fr/eric/smart-city-digital-twin-martinique.git
cd smart-city-digital-twin-martinique/helms
```
### 2. Configurer l'inventory
Éditer `inventory/hosts.yml` avec les IPs de vos nœuds :
```yaml
k8s_masters:
hosts:
k8s-master-1:
ansible_host: "192.168.1.100"
k8s_workers:
hosts:
k8s-worker-1:
ansible_host: "192.168.1.101"
k8s-worker-2:
ansible_host: "192.168.1.102"
nfs_server:
hosts:
nfs-1:
ansible_host: "192.168.1.200"
```
### 3. Configurer les variables
Éditer `group_vars/all.yml` selon vos besoins (ressources, domaines, etc.)
### 4. Chiffrer les secrets
```bash
ansible-vault encrypt group_vars/vault.yml
```
### 5. Déployer
```bash
# Déployer toute la stack
ansible-playbook deploy.yml --ask-vault-pass
# Déployer un service spécifique
ansible-playbook deploy.yml --tags clickhouse --ask-vault-pass
ansible-playbook deploy.yml --tags trino --ask-vault-pass
ansible-playbook deploy.yml --tags streamlit --ask-vault-pass
ansible-playbook deploy.yml --tags kafka --ask-vault-pass
ansible-playbook deploy.yml --tags monitoring --ask-vault-pass
```
### 6. Vérifier
```bash
kubectl get pods --all-namespaces
kubectl get ingress --all-namespaces
```
## Services déployés
| Service | Domaine | Namespace | Helm Chart |
|---------|---------|-----------|------------|
| Traefik | traefik.digitribe.fr | traefik | traefik/traefik |
| Airflow | airflow.digitribe.fr | airflow | apache/airflow |
| Kafka | kafka-bootstrap.digitribe.fr | kafka | strimzi/kafka-operator |
| Flink | flink.digitribe.fr | flink | apache/flink-kubernetes-operator |
| ClickHouse | clickhouse.digitribe.fr | clickhouse | bitnami/clickhouse |
| StarRocks | starrocks.digitribe.fr | starrocks | community/starrocks |
| Trino | trino.digitribe.fr | trino | trinodb/trino |
| Delta Lake | deltalake.digitribe.fr | deltalake | custom |
| Streamlit | streamlit.digitribe.fr | streamlit | custom |
| DuckDB | duckdb.digitribe.fr | duckdb | custom |
| EMQX | emqx.digitribe.fr | iot | emqx/emqx-operator |
| Mosquitto | mqtt.digitribe.fr | iot | custom |
| Node-RED | nodered.digitribe.fr | iot | custom |
| phpIPAM | phpipam.digitribe.fr | phpipam | custom |
| Gitea | gitea.digitribe.fr | gitea | gitea-charts/gitea |
| JupyterHub | jupyter.digitribe.fr | jupyterhub | jupyterhub/jupyterhub |
| Superset | superset.digitribe.fr | superset | apache/superset |
| Metabase | metabase.digitribe.fr | metabase | bitnami/metabase |
| MindsDB | mindsdb.digitribe.fr | mindsdb | bitnami/mindsdb |
| ODK Central | odk.digitribe.fr | odk | custom |
| MapStore | mapstore.digitribe.fr | gis | custom |
| GeoServer | geoserver.digitribe.fr | gis | custom |
| Smart App | smartapp.digitribe.fr | smartapp | custom |
| Smart App API | api-smartapp.digitribe.fr | smartapp | custom |
| Grafana | grafana.digitribe.fr | monitoring | grafana/grafana |
| MinIO | minio.digitribe.fr | databases | bitnami/minio |
| PostgreSQL | — (interne) | databases | bitnami/postgresql-ha |
| Redis | — (interne) | databases | bitnami/redis-cluster |
## Dépendances entre rôles
```
prerequisites → namespaces → storage → traefik → cert-manager
┌─────────────────────┼─────────────────────┐
↓ ↓ ↓
databases monitoring kafka
(postgres, (prometheus, ↓
redis, minio) grafana, loki) flink
↓ ↓ ↓
└─────────────────────┼─────────────────────┘
┌─────────────────────┼─────────────────────┐
↓ ↓ ↓
airflow bi iot
gitea jupyterhub superset metabase emqx mosquitto
odk mindsdb trino nodered phpipam
gis clickhouse streamlit
smartapp deltalake duckdb
backup (Velero)
```
## Commandes utiles
```bash
# Lister tous les pods
kubectl get pods --all-namespaces
# Voir les logs d'un pod
kubectl logs -f <pod-name> -n <namespace>
# Voir les événements
kubectl get events --all-namespaces --sort-by='.lastTimestamp'
# Voir les ingress
kubectl get ingress --all-namespaces
# Voir les PVC
kubectl get pvc --all-namespaces
# Redéployer un service
ansible-playbook deploy.yml --tags <service> --ask-vault-pass
# Supprimer un service
kubectl delete namespace <namespace>
# Supprimer toute la stack
ansible-playbook undeploy.yml
```
## Troubleshooting
### Pod en CrashLoopBackOff
```bash
kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> --previous
```
### PVC en Pending
```bash
kubectl get storageclass
kubectl get pv
kubectl describe pvc <pvc-name> -n <namespace>
```
### Ingress non accessible
```bash
kubectl get ingress -n <namespace>
kubectl describe ingress <ingress-name> -n <namespace>
kubectl logs -f deployment/traefik -n traefik
```
## Maintenance
### Backup
Les sauvegardes sont configurées via Velero :
```bash
kubectl get schedules -n velero
kubectl get backups -n velero
```
### Mise à jour d'un service
```bash
ansible-playbook deploy.yml --tags <service> --ask-vault-pass
```
### Scaling
```bash
kubectl scale deployment <deployment> --replicas=<n> -n <namespace>
```

77
helms/deploy.yml Normal file
View File

@@ -0,0 +1,77 @@
---
# Playbook principal pour le déploiement Kubernetes
# Fichier: deploy.yml
- name: Déploiement Smart City Martinique sur Kubernetes
hosts: localhost
connection: local
gather_facts: false
vars_files:
- group_vars/all.yml
- group_vars/vault.yml
pre_tasks:
- name: Vérifier que kubectl est installé
command: kubectl version --client
changed_when: false
- name: Vérifier la connexion au cluster
command: kubectl cluster-info
changed_when: false
roles:
- role: prerequisites
tags: [prerequisites]
- role: namespaces
tags: [namespaces]
- role: storage
tags: [storage]
- role: traefik
tags: [traefik, ingress]
- role: cert-manager
tags: [cert-manager, tls]
- role: monitoring
tags: [monitoring]
- role: databases
tags: [databases]
- role: kafka
tags: [kafka]
- role: flink
tags: [flink]
- role: airflow
tags: [airflow]
- role: iot
tags: [iot, mqtt]
- role: gitea
tags: [gitea]
- role: jupyterhub
tags: [jupyterhub]
- role: bi
tags: [bi, superset, metabase]
- role: mindsdb
tags: [mindsdb]
- role: odk
tags: [odk]
- role: gis
tags: [gis, mapstore, geoserver, frost]
- role: clickhouse
tags: [clickhouse]
- role: starrocks
tags: [starrocks]
- role: trino
tags: [trino]
- role: deltalake
tags: [deltalake]
- role: streamlit
tags: [streamlit]
- role: duckdb
tags: [duckdb]
- role: nodered
tags: [nodered]
- role: phpipam
tags: [phpipam]
- role: smartapp
tags: [smartapp]
- role: backup
tags: [backup]

535
helms/group_vars/all.yml Normal file
View File

@@ -0,0 +1,535 @@
---
# Variables globales pour le déploiement Kubernetes
# Fichier: group_vars/all.yml
# ============================================================
# Configuration du cluster Kubernetes
# ============================================================
cluster_name: smart-city-martinique
k8s_version: "1.28.0"
container_runtime: containerd
network_plugin: cilium
# ============================================================
# Configuration réseau
# ============================================================
domain: digitribe.fr
traefik_namespace: traefik
ingress_class: traefik
# TLS
tls_enabled: true
tls_certresolver: letsencrypt
acme_email: admin@digitribe.fr
# ============================================================
# Storage
# ============================================================
storage_class: nfs-client
nfs_server: "192.168.1.200"
nfs_path: /data/k8s
# Persistent Volume sizes
storage_sizes:
postgres: 50Gi
minio: 500Gi
kafka: 100Gi
influxdb: 50Gi
loki: 100Gi
grafana: 10Gi
jupyterhub: 20Gi
gitea: 20Gi
metabase: 10Gi
superset: 10Gi
mindsdb: 20Gi
odk: 10Gi
mapstore: 10Gi
geoserver: 20Gi
airflow: 20Gi
flink: 20Gi
emqx: 10Gi
mosquitto: 5Gi
redis: 10Gi
elasticsearch: 50Gi
# ============================================================
# Helm Charts versions
# ============================================================
helm_charts:
traefik:
chart: traefik/traefik
version: "28.0.0"
ingress_nginx:
chart: ingress-nginx/ingress-nginx
version: "4.8.0"
cert_manager:
chart: jetstack/cert-manager
version: "1.13.0"
nfs_provisioner:
chart: nfs-subdir-external-provisioner/nfs-subdir-external-provisioner
version: "4.0.18"
postgresql:
chart: bitnami/postgresql
version: "13.2.0"
postgresql_ha:
chart: bitnami/postgresql-ha
version: "12.2.0"
redis:
chart: bitnami/redis
version: "18.0.0"
minio:
chart: bitnami/minio
version: "12.10.0"
kafka:
chart: strimzi/kafka-operator
version: "0.38.0"
flink:
chart: apache/flink-kubernetes-operator
version: "1.7.0"
airflow:
chart: apache/airflow
version: "1.11.0"
grafana:
chart: grafana/grafana
version: "7.0.0"
loki:
chart: grafana/loki-stack
version: "2.9.0"
prometheus:
chart: prometheus/kube-prometheus-stack
version: "51.0.0"
emqx:
chart: emqx/emqx-operator
version: "2.2.0"
mosquitto:
chart: k8s-at-home/mosquitto
version: "4.8.0"
gitea:
chart: gitea/gitea
version: "9.0.0"
jupyterhub:
chart: jupyterhub/jupyterhub
version: "3.0.0"
superset:
chart: apache/superset
version: "0.11.0"
metabase:
chart: bitnami/metabase
version: "0.13.0"
mindsdb:
chart: bitnami/mindsdb
version: "0.1.0"
odk:
chart: odk/odk-central
version: "1.0.0"
mapstore:
chart: geosolutionsit/mapstore
version: "1.0.0"
geoserver:
chart: kartoza/geoserver
version: "2.2.0"
frost:
chart: fraunhoferiosb/frost-server
version: "1.0.0"
nodered:
chart: k8s-at-home/node-red
version: "4.8.0"
phpipam:
chart: phpipam/phpipam
version: "1.0.0"
clickhouse:
chart: bitnami/clickhouse
version: "4.0.0"
starrocks:
chart: starrocks/starrocks-community
version: "1.0.0"
trino:
chart: trinodb/trino
version: "0.10.0"
deltalake:
chart: delta-io/delta-lake
version: "1.0.0"
streamlit:
chart: streamlit/streamlit
version: "1.0.0"
duckdb:
chart: duckdb/duckdb
version: "1.0.0"
elasticsearch:
chart: elastic/elasticsearch
version: "8.11.0"
kibana:
chart: elastic/kibana
version: "8.11.0"
# ============================================================
# Services configuration
# ============================================================
services:
airflow:
enabled: true
namespace: airflow
replicas: 2
resources:
requests:
cpu: "500m"
memory: "1Gi"
limits:
cpu: "2000m"
memory: "4Gi"
kafka:
enabled: true
namespace: kafka
replicas: 3
resources:
requests:
cpu: "500m"
memory: "1Gi"
limits:
cpu: "2000m"
memory: "4Gi"
flink:
enabled: true
namespace: flink
replicas: 2
resources:
requests:
cpu: "500m"
memory: "1Gi"
limits:
cpu: "2000m"
memory: "4Gi"
emqx:
enabled: true
namespace: iot
replicas: 3
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "2Gi"
mosquitto:
enabled: true
namespace: iot
replicas: 2
resources:
requests:
cpu: "100m"
memory: "256Mi"
limits:
cpu: "500m"
memory: "1Gi"
postgresql:
enabled: true
namespace: default
replicas: 1
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "2Gi"
redis:
enabled: true
namespace: default
replicas: 3
resources:
requests:
cpu: "100m"
memory: "256Mi"
limits:
cpu: "500m"
memory: "1Gi"
minio:
enabled: true
namespace: default
replicas: 4
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "2Gi"
grafana:
enabled: true
namespace: monitoring
replicas: 1
resources:
requests:
cpu: "100m"
memory: "256Mi"
limits:
cpu: "500m"
memory: "1Gi"
loki:
enabled: true
namespace: monitoring
replicas: 1
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "2Gi"
prometheus:
enabled: true
namespace: monitoring
replicas: 1
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "2Gi"
gitea:
enabled: true
namespace: gitea
replicas: 1
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "2Gi"
jupyterhub:
enabled: true
namespace: jupyterhub
replicas: 1
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "2Gi"
superset:
enabled: true
namespace: superset
replicas: 1
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "2Gi"
metabase:
enabled: true
namespace: metabase
replicas: 1
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "2Gi"
mindsdb:
enabled: true
namespace: mindsdb
replicas: 1
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "2Gi"
odk:
enabled: true
namespace: odk
replicas: 1
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "2Gi"
mapstore:
enabled: true
namespace: mapstore
replicas: 1
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "2Gi"
geoserver:
enabled: true
namespace: geoserver
replicas: 1
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "2Gi"
frost:
enabled: true
namespace: iot
replicas: 1
resources:
requests:
cpu: "100m"
memory: "256Mi"
limits:
cpu: "500m"
memory: "1Gi"
nodered:
enabled: true
namespace: iot
replicas: 1
resources:
requests:
cpu: "100m"
memory: "256Mi"
limits:
cpu: "500m"
memory: "1Gi"
phpipam:
enabled: true
namespace: phpipam
replicas: 1
resources:
requests:
cpu: "100m"
memory: "256Mi"
limits:
cpu: "500m"
memory: "1Gi"
smartapp:
enabled: true
namespace: smartapp
replicas: 2
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "2Gi"
clickhouse:
enabled: true
namespace: clickhouse
replicas: 1
resources:
requests:
cpu: "500m"
memory: "1Gi"
limits:
cpu: "2000m"
memory: "4Gi"
starrocks:
enabled: true
namespace: starrocks
replicas: 1
resources:
requests:
cpu: "500m"
memory: "1Gi"
limits:
cpu: "2000m"
memory: "4Gi"
trino:
enabled: true
namespace: trino
replicas: 1
resources:
requests:
cpu: "500m"
memory: "1Gi"
limits:
cpu: "2000m"
memory: "4Gi"
deltalake:
enabled: true
namespace: deltalake
replicas: 1
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "2Gi"
streamlit:
enabled: true
namespace: streamlit
replicas: 1
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "2Gi"
duckdb:
enabled: true
namespace: duckdb
replicas: 1
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "2Gi"
# ============================================================
# Monitoring
# ============================================================
monitoring:
enabled: true
namespace: monitoring
grafana_admin_password: "{{ vault_grafana_password }}"
prometheus_retention: 30d
loki_retention: 30d
# ============================================================
# Backup
# ============================================================
backup:
enabled: true
schedule: "0 2 * * *"
retention: 30
storage_class: nfs-client
storage_size: 100Gi

View File

@@ -0,0 +1,60 @@
---
# Vault Ansible - Variables chiffrées
# Fichier: group_vars/vault.yml
# Chiffrer avec: ansible-vault encrypt group_vars/vault.yml
# PostgreSQL
vault_postgres_password: "Digitribe972"
vault_postgres_repmgr_password: "Digitribe972"
# Redis
vault_redis_password: "Digitribe972"
# MinIO
vault_minio_root_user: "minioadmin"
vault_minio_root_password: "Digitribe972"
# Grafana
vault_grafana_admin_password: "Digitribe972"
# Airflow
vault_airflow_fernet_key: "Digitribe972SecretKeyForAirflow2024"
vault_airflow_admin_password: "Digitribe972"
# Gitea
vault_gitea_admin_password: "Digitribe972"
# Superset
vault_superset_admin_password: "Digitribe972"
vault_superset_db_password: "Digitribe972"
# Metabase
vault_metabase_db_password: "Digitribe972"
# MindsDB
vault_mindsdb_password: "Digitribe972"
# ClickHouse
vault_clickhouse_password: "Digitribe972"
# Trino
vault_trino_db_password: "Digitribe972"
# MQTT
vault_mosquitto_password: "Digitribe972"
vault_emqx_admin_password: "Digitribe972"
# phpIPAM
vault_phpipam_admin_password: "Digitribe972"
# ODK
vault_odk_admin_password: "Digitribe972"
# GeoServer
vault_geoserver_admin_password: "Digitribe972"
# MapStore
vault_mapstore_admin_password: "Digitribe972"
# StarRocks
vault_starrocks_root_password: "Digitribe972"

79
helms/inventory/hosts.yml Normal file
View File

@@ -0,0 +1,79 @@
---
# Inventory pour le déploiement Kubernetes via Ansible
# Fichier: inventory/hosts.yml
all:
children:
k8s_masters:
hosts:
k8s-master-1:
ansible_host: "{{ k8s_master_ip | default('192.168.1.100') }}"
ansible_user: "{{ k8s_user | default('root') }}"
k8s_workers:
hosts:
k8s-worker-1:
ansible_host: "{{ k8s_worker1_ip | default('192.168.1.101') }}"
ansible_user: "{{ k8s_user | default('root') }}"
k8s-worker-2:
ansible_host: "{{ k8s_worker2_ip | default('192.168.1.102') }}"
ansible_user: "{{ k8s_user | default('root') }}"
nfs_server:
hosts:
nfs-1:
ansible_host: "{{ nfs_server_ip | default('192.168.1.200') }}"
ansible_user: "{{ nfs_user | default('root') }}"
vars:
# Configuration globale
cluster_name: smart-city-martinique
k8s_version: "1.28"
container_runtime: containerd
network_plugin: cilium
domain: digitribe.fr
# Namespaces Kubernetes
namespaces:
- airflow
- kafka
- flink
- monitoring
- iot
- gitea
- jupyterhub
- odk
- smartapp
- superset
- metabase
- mindsdb
- mapstore
- geoserver
- frost
- nodered
- phpipam
- traefik
- ingress-nginx
- clickhouse
- starrocks
- trino
- deltalake
- streamlit
- duckdb
# Storage
storage_class: nfs-client
nfs_path: /data/k8s
# Helm repositories
helm_repos:
- name: bitnami
url: https://charts.bitnami.com/bitnami
- name: apache
url: https://charts.apache.org
- name: grafana
url: https://grafana.github.io/helm-charts
- name: prometheus
url: https://prometheus-community.github.io/helm-charts
- name: strimzi
url: https://strimzi.io/charts/
- name: flink-operator
url: https://downloads.apache.org/flink/flink-kubernetes-operator-1.7.0/

View File

@@ -0,0 +1,19 @@
---
# Role: airflow
# Valeurs par défaut pour Apache Airflow
# Réplicas des workers Airflow
services:
airflow:
replicas: 2
resources:
requests:
cpu: "500m"
memory: "1Gi"
limits:
cpu: "2000m"
memory: "4Gi"
# Stockage des logs Airflow
storage_sizes:
airflow: "20Gi"

View File

@@ -0,0 +1,13 @@
---
galaxy_info:
author: Eric FELIXINE
description: Deploy Apache Airflow for workflow orchestration on Kubernetes
license: MIT
min_ansible_version: "2.15"
platforms:
- name: Kubernetes
versions:
- "1.28"
dependencies:
- role: databases
- role: kafka

View File

@@ -0,0 +1,34 @@
---
# Role: airflow
# Déploie Apache Airflow
- name: Installer Airflow
kubernetes.core.helm:
name: airflow
chart_ref: "{{ helm_charts.airflow.chart }}"
release_namespace: airflow
create_namespace: true
values:
executor: CeleryExecutor
fernetKey: "{{ vault_airflow_fernet_key }}"
webserver:
defaultUser:
username: admin
password: "{{ vault_airflow_admin_password }}"
dags:
persistence:
enabled: true
size: 10Gi
logs:
persistence:
enabled: true
size: "{{ storage_sizes.airflow }}"
scheduler:
resources: "{{ services.airflow.resources }}"
webserver:
resources: "{{ services.airflow.resources }}"
workers:
replicas: "{{ services.airflow.replicas }}"
resources: "{{ services.airflow.resources }}"
triggerer:
resources: "{{ services.airflow.resources }}"

View File

@@ -0,0 +1,8 @@
---
# Role: backup
# Valeurs par défaut pour les sauvegardes Velero
# Planification des sauvegardes (cron format)
backup:
schedule: "0 2 * * *"
retention: "168" # 7 jours en heures

View File

@@ -0,0 +1,11 @@
---
galaxy_info:
author: Eric FELIXINE
description: Deploy Velero backup and disaster recovery solution on Kubernetes
license: MIT
min_ansible_version: "2.15"
platforms:
- name: Kubernetes
versions:
- "1.28"
dependencies: []

View File

@@ -0,0 +1,34 @@
---
# Role: backup
# Configure les sauvegardes Velero
- name: Installer Velero
kubernetes.core.helm:
name: velero
chart_ref: vmware-tanzu/velero
release_namespace: velero
create_namespace: true
values:
configuration:
backupStorageLocation:
- name: default
provider: aws
bucket: smart-city-backup
config:
region: eu-west-3
s3ForcePathStyle: true
schedules:
daily:
schedule: "{{ backup.schedule }}"
template:
includedNamespaces:
- "{{ item }}"
snapshotVolumes: true
ttl: "{{ backup.retention }}h0m0s"
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "2Gi"

View File

@@ -0,0 +1,23 @@
---
# Role: bi
# Valeurs par défaut pour Superset et Metabase
services:
superset:
replicas: 1
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "2Gi"
metabase:
replicas: 1
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "2Gi"

View File

@@ -0,0 +1,12 @@
---
galaxy_info:
author: Eric FELIXINE
description: Deploy Business Intelligence tools on Kubernetes
license: MIT
min_ansible_version: "2.15"
platforms:
- name: Kubernetes
versions:
- "1.28"
dependencies:
- role: databases

View File

@@ -0,0 +1,44 @@
---
# Role: bi
# Déploie Superset et Metabase
- name: Installer Superset
kubernetes.core.helm:
name: superset
chart_ref: "{{ helm_charts.superset.chart }}"
release_namespace: superset
create_namespace: true
values:
supersetNode:
connections:
redis_password: "{{ vault_redis_password }}"
db_user: superset
db_pass: "{{ vault_superset_db_password }}"
resources: "{{ services.superset.resources }}"
supersetWorker:
replicas: 2
resources: "{{ services.superset.resources }}"
bootstrapScript: |
#!/bin/bash
pip install psycopg2-binary redis
init:
adminUser:
username: admin
password: "{{ vault_superset_admin_password }}"
email: admin@digitribe.fr
- name: Installer Metabase
kubernetes.core.helm:
name: metabase
chart_ref: "{{ helm_charts.metabase.chart }}"
release_namespace: metabase
create_namespace: true
values:
database:
type: postgres
host: postgresql-ha-pgpool.default.svc.cluster.local
port: 5432
dbname: metabase
username: metabase
password: "{{ vault_metabase_db_password }}"
resources: "{{ services.metabase.resources }}"

View File

@@ -0,0 +1,6 @@
---
# Role: cert-manager
# Valeurs par défaut pour cert-manager
# Email pour les certificats Let's Encrypt
acme_email: "admin@digitribe.fr"

View File

@@ -0,0 +1,12 @@
---
galaxy_info:
author: Eric FELIXINE
description: Deploy cert-manager for automated TLS certificate management on Kubernetes
license: MIT
min_ansible_version: "2.15"
platforms:
- name: Kubernetes
versions:
- "1.28"
dependencies:
- role: traefik

View File

@@ -0,0 +1,39 @@
---
# Role: cert-manager
# Déploie cert-manager pour la gestion des certificats TLS
- name: Installer cert-manager
kubernetes.core.helm:
name: cert-manager
chart_ref: "{{ helm_charts.cert_manager.chart }}"
chart_version: "{{ helm_charts.cert_manager.version }}"
release_namespace: cert-manager
create_namespace: true
values:
installCRDs: true
resources:
requests:
cpu: "100m"
memory: "128Mi"
limits:
cpu: "500m"
memory: "256Mi"
- name: Créer le ClusterIssuer Let's Encrypt
kubernetes.core.k8s:
state: present
definition:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: "{{ acme_email }}"
privateKeySecretRef:
name: letsencrypt-key
solvers:
- http01:
ingress:
class: traefik

View File

@@ -0,0 +1,17 @@
---
# Role: clickhouse
# Valeurs par défaut pour ClickHouse
services:
clickhouse:
replicas: 2
resources:
requests:
cpu: "500m"
memory: "1Gi"
limits:
cpu: "2000m"
memory: "4Gi"
storage_sizes:
clickhouse: "50Gi"

View File

@@ -0,0 +1,12 @@
---
galaxy_info:
author: Eric FELIXINE
description: Deploy ClickHouse columnar database for analytics on Kubernetes
license: MIT
min_ansible_version: "2.15"
platforms:
- name: Kubernetes
versions:
- "1.28"
dependencies:
- role: databases

View File

@@ -0,0 +1,34 @@
---
# Role: clickhouse
# Déploie ClickHouse
- name: Installer ClickHouse
kubernetes.core.helm:
name: clickhouse
chart_ref: "{{ helm_charts.clickhouse.chart }}"
chart_version: "{{ helm_charts.clickhouse.version }}"
release_namespace: clickhouse
create_namespace: true
values:
shards: 1
replicaCount: "{{ services.clickhouse.replicas }}"
persistence:
size: "{{ storage_sizes.clickhouse | default('50Gi') }}"
storageClass: "{{ storage_class }}"
resources: "{{ services.clickhouse.resources }}"
auth:
username: default
password: "{{ vault_clickhouse_password }}"
service:
type: ClusterIP
ingress:
enabled: true
hosts:
- host: clickhouse.digitribe.fr
paths:
- path: /
pathType: Prefix
tls:
- secretName: clickhouse-tls
hosts:
- clickhouse.digitribe.fr

View File

@@ -0,0 +1,27 @@
---
# Role: databases
# Valeurs par défaut pour PostgreSQL, Redis et MinIO
services:
postgresql:
replicas: 2
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "2Gi"
# Stockages
storage_sizes:
postgresql: "50Gi"
redis: "10Gi"
minio: "100Gi"
# Mots de passe Vault (valeurs DUMMY — overridés par group_vars/vault.yml)
vault_postgres_password: "DUMMY_POSTGRES_PASSWORD"
vault_postgres_repmgr_password: "DUMMY_REPMGR_PASSWORD"
vault_redis_password: "DUMMY_REDIS_PASSWORD"
vault_minio_root_user: "DUMMY_MINIO_USER"
vault_minio_root_password: "DUMMY_MINIO_PASSWORD"

View File

@@ -0,0 +1,13 @@
---
galaxy_info:
author: Eric FELIXINE
description: Deploy and manage core database services (PostgreSQL, MySQL, Redis) on Kubernetes
license: MIT
min_ansible_version: "2.15"
platforms:
- name: Kubernetes
versions:
- "1.28"
dependencies:
- role: storage
- role: cert-manager

View File

@@ -0,0 +1,54 @@
---
# Role: databases
# Déploie PostgreSQL, Redis et MinIO
- name: Installer PostgreSQL HA
kubernetes.core.helm:
name: postgresql
chart_ref: "{{ helm_charts.postgresql_ha.chart }}"
release_namespace: default
values:
postgresql:
password: "{{ vault_postgres_password }}"
repmgrPassword: "{{ vault_postgres_repmgr_password }}"
persistence:
size: "{{ storage_sizes.postgresql }}"
storageClass: "{{ storage_class }}"
resources:
requests:
cpu: "{{ services.postgresql.resources.requests.cpu }}"
memory: "{{ services.postgresql.resources.requests.memory }}"
- name: Installer Redis Cluster
kubernetes.core.helm:
name: redis
chart_ref: "{{ helm_charts.redis.chart }}"
release_namespace: default
values:
cluster:
nodes: 3
password: "{{ vault_redis_password }}"
persistence:
size: "{{ storage_sizes.redis }}"
storageClass: "{{ storage_class }}"
resources:
requests:
cpu: "100m"
memory: "256Mi"
- name: Installer MinIO
kubernetes.core.helm:
name: minio
chart_ref: "{{ helm_charts.minio.chart }}"
release_namespace: default
values:
auth:
rootUser: "{{ vault_minio_root_user }}"
rootPassword: "{{ vault_minio_root_password }}"
persistence:
size: "{{ storage_sizes.minio }}"
storageClass: "{{ storage_class }}"
resources:
requests:
cpu: "250m"
memory: "512Mi"

View File

@@ -0,0 +1,17 @@
---
# Role: deltalake
# Valeurs par défaut pour Delta Lake
services:
deltalake:
replicas: 1
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "2Gi"
storage_sizes:
deltalake: "100Gi"

View File

@@ -0,0 +1,12 @@
---
galaxy_info:
author: Eric FELIXINE
description: Deploy Delta Lake storage layer for data lakehouse architecture on Kubernetes
license: MIT
min_ansible_version: "2.15"
platforms:
- name: Kubernetes
versions:
- "1.28"
dependencies:
- role: databases

View File

@@ -0,0 +1,30 @@
---
# Role: deltalake
# Déploie Delta Lake
- name: Installer Delta Lake
kubernetes.core.helm:
name: deltalake
chart_ref: "{{ helm_charts.deltalake.chart }}"
chart_version: "{{ helm_charts.deltalake.version }}"
release_namespace: deltalake
create_namespace: true
values:
replicaCount: "{{ services.deltalake.replicas }}"
resources: "{{ services.deltalake.resources }}"
storage:
size: "{{ storage_sizes.deltalake | default('100Gi') }}"
storageClass: "{{ storage_class }}"
service:
type: ClusterIP
ingress:
enabled: true
hosts:
- host: deltalake.digitribe.fr
paths:
- path: /
pathType: Prefix
tls:
- secretName: deltalake-tls
hosts:
- deltalake.digitribe.fr

View File

@@ -0,0 +1,17 @@
---
# Role: duckdb
# Valeurs par défaut pour DuckDB
services:
duckdb:
replicas: 1
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "2Gi"
storage_sizes:
duckdb: "50Gi"

View File

@@ -0,0 +1,12 @@
---
galaxy_info:
author: Eric FELIXINE
description: Deploy DuckDB embedded analytical database on Kubernetes
license: MIT
min_ansible_version: "2.15"
platforms:
- name: Kubernetes
versions:
- "1.28"
dependencies:
- role: databases

View File

@@ -0,0 +1,30 @@
---
# Role: duckdb
# Déploie DuckDB
- name: Installer DuckDB
kubernetes.core.helm:
name: duckdb
chart_ref: "{{ helm_charts.duckdb.chart }}"
chart_version: "{{ helm_charts.duckdb.version }}"
release_namespace: duckdb
create_namespace: true
values:
replicaCount: "{{ services.duckdb.replicas }}"
resources: "{{ services.duckdb.resources }}"
storage:
size: "{{ storage_sizes.duckdb | default('50Gi') }}"
storageClass: "{{ storage_class }}"
service:
type: ClusterIP
ingress:
enabled: true
hosts:
- host: duckdb.digitribe.fr
paths:
- path: /
pathType: Prefix
tls:
- secretName: duckdb-tls
hosts:
- duckdb.digitribe.fr

View File

@@ -0,0 +1,14 @@
---
# Role: flink
# Valeurs par défaut pour Apache Flink
services:
flink:
replicas: 2
resources:
requests:
cpu: "1000m"
memory: "2Gi"
limits:
cpu: "2000m"
memory: "4Gi"

View File

@@ -0,0 +1,12 @@
---
galaxy_info:
author: Eric FELIXINE
description: Deploy Apache Flink for stream processing on Kubernetes
license: MIT
min_ansible_version: "2.15"
platforms:
- name: Kubernetes
versions:
- "1.28"
dependencies:
- role: kafka

View File

@@ -0,0 +1,18 @@
---
# Role: flink
# Déploie Apache Flink via l'opérateur
- name: Installer l'opérateur Flink
kubernetes.core.helm:
name: flink-kubernetes-operator
chart_ref: "{{ helm_charts.flink.chart }}"
release_namespace: flink
create_namespace: true
- name: Créer le déploiement Flink
kubernetes.core.k8s:
state: present
template: flink-deployment.yml.j2
vars:
flink_namespace: flink
flink_replicas: "{{ services.flink.replicas }}"

View File

@@ -0,0 +1,140 @@
---
# Role: flink
# Template: flink-deployment.yml.j2
# Déploiement d'un cluster Apache Flink via FlinkKubernetesOperator
# Variables:
# {{ flink_namespace }} - Namespace Kubernetes (défaut: flink)
# {{ flink_replicas }} - Nombre de TaskManagers (défaut: 2)
---
apiVersion: v1
kind: Namespace
metadata:
name: {{ flink_namespace | default('flink') }}
labels:
app: flink
version: "1.18"
---
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
name: flink-cluster
namespace: {{ flink_namespace | default('flink') }}
labels:
app: flink
version: "1.18"
spec:
image: flink:1.18-scala_2.12
flinkVersion: v1_18
imagePullPolicy: IfNotPresent
# --- JobManager ---
jobmanager:
resource:
memory: "2048m"
cpu: 1
replicas: 1
# --- TaskManager ---
taskmanager:
resource:
memory: "4096m"
cpu: 2
replicas: {{ flink_replicas | default(2) }}
# --- Configuration Flink ---
flinkConfiguration:
taskmanager.numberOfTaskSlots: "2"
state.backend: rocksdb
state.checkpoints.dir: s3://flink-checkpoints
state.savepoints.dir: s3://flink-savepoints
high-availability: zookeeper
high-availability.zookeeper.quorum: zk-cs.{{ flink_namespace | default('flink') }}.svc.cluster.local:2181
web.upload.dir: /tmp/flink-web-upload
---
apiVersion: v1
kind: Service
metadata:
name: flink-jobmanager
namespace: {{ flink_namespace | default('flink') }}
labels:
app: flink
component: jobmanager
version: "1.18"
spec:
type: ClusterIP
selector:
app: flink
component: jobmanager
ports:
- name: rpc
port: 6123
targetPort: 6123
protocol: TCP
- name: blob
port: 6124
targetPort: 6124
protocol: TCP
- name: webui
port: 8081
targetPort: 8081
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
name: flink-taskmanager
namespace: {{ flink_namespace | default('flink') }}
labels:
app: flink
component: taskmanager
version: "1.18"
spec:
type: ClusterIP
selector:
app: flink
component: taskmanager
ports:
- name: rpc
port: 6122
targetPort: 6122
protocol: TCP
- name: data
port: 6125
targetPort: 6125
protocol: TCP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: flink-webui
namespace: {{ flink_namespace | default('flink') }}
labels:
app: flink
component: webui
version: "1.18"
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/ssl-redirect: "true"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
ingressClassName: nginx
tls:
- hosts:
- flink.digitribe.fr
secretName: flink-tls
rules:
- host: flink.digitribe.fr
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: flink-jobmanager
port:
number: 8081

View File

@@ -0,0 +1,36 @@
---
# Role: gis
# Valeurs par défaut pour MapStore, GeoServer et FROST
services:
mapstore:
replicas: 1
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "2Gi"
geoserver:
replicas: 1
resources:
requests:
cpu: "500m"
memory: "1Gi"
limits:
cpu: "2000m"
memory: "4Gi"
frost:
replicas: 1
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "2Gi"
storage_sizes:
mapstore: "10Gi"
geoserver: "20Gi"

View File

@@ -0,0 +1,12 @@
---
galaxy_info:
author: Eric FELIXINE
description: Deploy Geographic Information System (GIS) services on Kubernetes
license: MIT
min_ansible_version: "2.15"
platforms:
- name: Kubernetes
versions:
- "1.28"
dependencies:
- role: databases

View File

@@ -0,0 +1,37 @@
---
# Role: gis
# Déploie MapStore, GeoServer et FROST
- name: Installer MapStore
kubernetes.core.helm:
name: mapstore
chart_ref: "{{ helm_charts.mapstore.chart }}"
release_namespace: mapstore
create_namespace: true
values:
persistence:
size: "{{ storage_sizes.mapstore }}"
resources: "{{ services.mapstore.resources }}"
- name: Installer GeoServer
kubernetes.core.helm:
name: geoserver
chart_ref: "{{ helm_charts.geoserver.chart }}"
release_namespace: geoserver
create_namespace: true
values:
persistence:
geodataDir:
storageClass: "{{ storage_class }}"
size: "{{ storage_sizes.geoserver }}"
resources: "{{ services.geoserver.resources }}"
- name: Installer FROST
kubernetes.core.helm:
name: frost
chart_ref: "{{ helm_charts.frost.chart }}"
release_namespace: iot
values:
persistence:
size: 10Gi
resources: "{{ services.frost.resources }}"

View File

@@ -0,0 +1,17 @@
---
# Role: gitea
# Valeurs par défaut pour Gitea
services:
gitea:
replicas: 1
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "2Gi"
storage_sizes:
gitea: "20Gi"

View File

@@ -0,0 +1,12 @@
---
galaxy_info:
author: Eric FELIXINE
description: Deploy Gitea - self-hosted Git service on Kubernetes
license: MIT
min_ansible_version: "2.15"
platforms:
- name: Kubernetes
versions:
- "1.28"
dependencies:
- role: databases

View File

@@ -0,0 +1,28 @@
---
# Role: gitea
# Déploie Gitea
- name: Installer Gitea
kubernetes.core.helm:
name: gitea
chart_ref: "{{ helm_charts.gitea.chart }}"
release_namespace: gitea
create_namespace: true
values:
gitea:
admin:
username: eric
password: "{{ vault_gitea_admin_password }}"
email: admin@digitribe.fr
config:
server:
DOMAIN: gitea.digitribe.fr
ROOT_URL: https://gitea.digitribe.fr
SSH_DOMAIN: gitea.digitribe.fr
SSH_PORT: 22
persistence:
enabled: true
size: "{{ storage_sizes.gitea }}"
postgresql:
enabled: true
resources: "{{ services.gitea.resources }}"

View File

@@ -0,0 +1,27 @@
---
# Role: iot
# Valeurs par défaut pour EMQX et Mosquitto
services:
emqx:
replicas: 2
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "2Gi"
mosquitto:
replicas: 1
resources:
requests:
cpu: "100m"
memory: "256Mi"
limits:
cpu: "500m"
memory: "512Mi"
storage_sizes:
emqx: "10Gi"
mosquitto: "5Gi"

View File

@@ -0,0 +1,13 @@
---
galaxy_info:
author: Eric FELIXINE
description: Deploy IoT platform services on Kubernetes
license: MIT
min_ansible_version: "2.15"
platforms:
- name: Kubernetes
versions:
- "1.28"
dependencies:
- role: databases
- role: kafka

View File

@@ -0,0 +1,35 @@
---
# Role: iot
# Déploie les brokers MQTT
- name: Installer EMQX
kubernetes.core.helm:
name: emqx
chart_ref: "{{ helm_charts.emqx.chart }}"
release_namespace: iot
create_namespace: true
values:
replicaCount: "{{ services.emqx.replicas }}"
persistence:
enabled: true
size: "{{ storage_sizes.emqx }}"
resources: "{{ services.emqx.resources }}"
- name: Installer Mosquitto
kubernetes.core.helm:
name: mosquitto
chart_ref: "{{ helm_charts.mosquitto.chart }}"
release_namespace: iot
values:
replicaCount: "{{ services.mosquitto.replicas }}"
persistence:
enabled: true
size: "{{ storage_sizes.mosquitto }}"
resources: "{{ services.mosquitto.resources }}"
config: |
listener 1883
allow_anonymous false
password_file /etc/mosquitto/passwd
auth:
password: "{{ vault_mosquitto_password }}"

View File

@@ -0,0 +1,17 @@
---
# Role: jupyterhub
# Valeurs par défaut pour JupyterHub
services:
jupyterhub:
replicas: 1
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "2Gi"
storage_sizes:
jupyterhub: "20Gi"

View File

@@ -0,0 +1,12 @@
---
galaxy_info:
author: Eric FELIXINE
description: Deploy JupyterHub for multi-user notebook environments on Kubernetes
license: MIT
min_ansible_version: "2.15"
platforms:
- name: Kubernetes
versions:
- "1.28"
dependencies:
- role: databases

View File

@@ -0,0 +1,31 @@
---
# Role: jupyterhub
# Déploie JupyterHub
- name: Installer JupyterHub
kubernetes.core.helm:
name: hub
chart_ref: "{{ helm_charts.jupyterhub.chart }}"
release_namespace: jupyterhub
create_namespace: true
values:
hub:
config:
Authenticator:
admin_users:
- eric
JupyterHub:
admin_access: true
db:
pvc:
storage: "{{ storage_sizes.jupyterhub }}"
singleuser:
storage:
capacity: "{{ storage_sizes.jupyterhub }}"
dynamic:
pvcNameTemplate: "jupyterhub-{userid}"
volumeNameTemplate: "jupyterhub-{userid}"
storageClass: "{{ storage_class }}"
proxy:
service:
type: ClusterIP

View File

@@ -0,0 +1,17 @@
---
# Role: kafka
# Valeurs par défaut pour Kafka (Strimzi)
services:
kafka:
replicas: 3
resources:
requests:
cpu: "1000m"
memory: "2Gi"
limits:
cpu: "2000m"
memory: "4Gi"
storage_sizes:
kafka: "100Gi"

View File

@@ -0,0 +1,13 @@
---
galaxy_info:
author: Eric FELIXINE
description: Deploy and manage Apache Kafka cluster on Kubernetes
license: MIT
min_ansible_version: "2.15"
platforms:
- name: Kubernetes
versions:
- "1.28"
dependencies:
- role: storage
- role: cert-manager

View File

@@ -0,0 +1,19 @@
---
# Role: kafka
# Déploie Kafka via l'opérateur Strimzi
- name: Installer l'opérateur Strimzi
kubernetes.core.helm:
name: strimzi-kafka-operator
chart_ref: "{{ helm_charts.kafka.chart }}"
release_namespace: kafka
create_namespace: true
- name: Créer le cluster Kafka
kubernetes.core.k8s:
state: present
template: kafka-cluster.yml.j2
vars:
kafka_namespace: kafka
kafka_replicas: "{{ services.kafka.replicas }}"
kafka_storage_size: "{{ storage_sizes.kafka }}"

View File

@@ -0,0 +1,295 @@
---
# Role: kafka
# Template: kafka-cluster.yml.j2
# Cluster Kafka via Strimzi KafkaOperator
# Variables:
# {{ kafka_namespace }} - Namespace Kubernetes (défaut: kafka)
# {{ kafka_replicas }} - Nombre de brokers Kafka (défaut: 3)
# {{ kafka_storage_size }} - Taille du stockage par broker (défaut: 100Gi)
---
apiVersion: v1
kind: Namespace
metadata:
name: {{ kafka_namespace | default('kafka') }}
labels:
app: kafka
version: "3.6"
---
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: kafka-cluster
namespace: {{ kafka_namespace | default('kafka') }}
labels:
app: kafka
version: "3.6"
spec:
kafka:
version: 3.6.0
replicas: {{ kafka_replicas | default(3) }}
listeners:
- name: plain
port: 9092
type: internal
tls: false
- name: tls
port: 9093
type: internal
tls: true
- name: external
port: 9094
type: ingress
tls: true
configuration:
bootstrap:
host: kafka-bootstrap.digitribe.fr
brokers:
- broker: 0
host: kafka-broker-0.digitribe.fr
- broker: 1
host: kafka-broker-1.digitribe.fr
- broker: 2
host: kafka-broker-2.digitribe.fr
config:
offsets.topic.replication.factor: 3
transaction.state.log.replication.factor: 3
transaction.state.log.min.isr: 2
default.replication.factor: 3
min.insync.replicas: 2
inter.broker.protocol.version: "3.6"
log.message.format.version: "3.6"
storage:
type: jbod
volumes:
- id: 0
type: persistent-claim
size: {{ kafka_storage_size | default('100Gi') }}
class: standard
deleteClaim: false
resources:
requests:
cpu: "1"
memory: "2Gi"
limits:
cpu: "2"
memory: "4Gi"
livenessProbe:
initialDelaySeconds: 30
timeoutSeconds: 5
readinessProbe:
initialDelaySeconds: 10
timeoutSeconds: 5
metricsConfig:
type: jmxPrometheusExporter
valueFrom:
configMapKeyRef:
name: kafka-metrics
key: kafka-metrics-config.yml
template:
pod:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: strimzi.io/name
operator: In
values:
- kafka-cluster-kafka
topologyKey: kubernetes.io/hostname
zookeeper:
replicas: 3
storage:
type: persistent-claim
size: 20Gi
class: standard
deleteClaim: false
resources:
requests:
cpu: "500m"
memory: "1Gi"
limits:
cpu: "1"
memory: "2Gi"
livenessProbe:
initialDelaySeconds: 30
timeoutSeconds: 5
readinessProbe:
initialDelaySeconds: 10
timeoutSeconds: 5
entityOperator:
topicOperator:
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "500m"
memory: "1Gi"
userOperator:
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "500m"
memory: "1Gi"
kafkaExporter:
topicRegex: ".*"
groupRegex: ".*"
resources:
requests:
cpu: "200m"
memory: "256Mi"
limits:
cpu: "500m"
memory: "512Mi"
---
apiVersion: v1
kind: ConfigMap
metadata:
name: kafka-metrics
namespace: {{ kafka_namespace | default('kafka') }}
labels:
app: kafka
version: "3.6"
data:
kafka-metrics-config.yml: |
# See https://github.com/prometheus/jmx_exporter for more info about JMX Prometheus Exporter metrics
lowercaseOutputName: true
rules:
# Special cases and very specific rules
- pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>Value
name: kafka_server_$1_$2
type: GAUGE
labels:
clientId: "$3"
topic: "$4"
partition: "$5"
- pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>Value
name: kafka_server_$1_$2
type: GAUGE
labels:
clientId: "$3"
broker: "$4:$5"
# Generic per-second counters with 0-2 key/value pairs
- pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+), (.+)=(.+)><>Count
name: kafka_$1_$2_$3_total
type: COUNTER
labels:
"$4": "$5"
"$6": "$7"
- pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+)><>Count
name: kafka_$1_$2_$3_total
type: COUNTER
labels:
"$4": "$5"
- pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*><>Count
name: kafka_$1_$2_$3_total
type: COUNTER
# Generic gauges with 0-2 key/value pairs
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Value
name: kafka_$1_$2_$3
type: GAUGE
labels:
"$4": "$5"
"$6": "$7"
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Value
name: kafka_$1_$2_$3
type: GAUGE
labels:
"$4": "$5"
- pattern: kafka.(\w+)<type=(.+), name=(.+)><>Value
name: kafka_$1_$2_$3
type: GAUGE
# Emulate Prometheus 'Summary' metrics for the exported 'Histogram's
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Count
name: kafka_$1_$2_$3_count
type: COUNTER
labels:
"$4": "$5"
"$6": "$7"
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*), (.+)=(.+)><>(\d+)thPercentile
name: kafka_$1_$2_$3
type: SUMMARY
labels:
"$4": "$5"
"$6": "$7"
quantile: 0.95
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Count
name: kafka_$1_$2_$3_count
type: COUNTER
labels:
"$4": "$5"
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*)><>(\d+)thPercentile
name: kafka_$1_$2_$3
type: SUMMARY
labels:
"$4": "$5"
quantile: 0.95
- pattern: kafka.(\w+)<type=(.+), name=(.+)><>Count
name: kafka_$1_$2_$3_count
type: COUNTER
---
apiVersion: v1
kind: Service
metadata:
name: kafka-bootstrap
namespace: {{ kafka_namespace | default('kafka') }}
labels:
app: kafka
component: bootstrap
version: "3.6"
spec:
type: ClusterIP
selector:
strimzi.io/cluster: kafka-cluster
strimzi.io/name: kafka-cluster-kafka
ports:
- name: tcp-internal
port: 9092
targetPort: 9092
protocol: TCP
- name: tcp-tls
port: 9093
targetPort: 9093
protocol: TCP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: kafka-external
namespace: {{ kafka_namespace | default('kafka') }}
labels:
app: kafka
component: external
version: "3.6"
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/backend-protocol: "TCP"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
ingressClassName: nginx
tls:
- hosts:
- kafka-bootstrap.digitribe.fr
secretName: kafka-bootstrap-tls
rules:
- host: kafka-bootstrap.digitribe.fr
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: kafka-cluster-kafka-external-bootstrap
port:
number: 9094

View File

@@ -0,0 +1,17 @@
---
# Role: mindsdb
# Valeurs par défaut pour MindsDB
services:
mindsdb:
replicas: 1
resources:
requests:
cpu: "500m"
memory: "1Gi"
limits:
cpu: "2000m"
memory: "4Gi"
storage_sizes:
mindsdb: "20Gi"

View File

@@ -0,0 +1,12 @@
---
galaxy_info:
author: Eric FELIXINE
description: Deploy MindsDB - open-source AI/ML database on Kubernetes
license: MIT
min_ansible_version: "2.15"
platforms:
- name: Kubernetes
versions:
- "1.28"
dependencies:
- role: databases

View File

@@ -0,0 +1,18 @@
---
# Role: mindsdb
# Déploie MindsDB
- name: Installer MindsDB
kubernetes.core.helm:
name: mindsdb
chart_ref: "{{ helm_charts.mindsdb.chart }}"
release_namespace: mindsdb
create_namespace: true
values:
mindsdb:
auth:
username: admin
password: "{{ vault_mindsdb_password }}"
storage:
size: "{{ storage_sizes.mindsdb }}"
resources: "{{ services.mindsdb.resources }}"

View File

@@ -0,0 +1,12 @@
---
# Role: monitoring
# Valeurs par défaut pour Prometheus, Grafana, Loki et Promtail
monitoring:
prometheus_retention: "30d"
grafana_admin_password: "DUMMY_GRAFANA_ADMIN_PASSWORD"
storage_sizes:
prometheus: "50Gi"
grafana: "10Gi"
loki: "50Gi"

View File

@@ -0,0 +1,13 @@
---
galaxy_info:
author: Eric FELIXINE
description: Deploy monitoring stack (Prometheus, Grafana, Alertmanager) on Kubernetes
license: MIT
min_ansible_version: "2.15"
platforms:
- name: Kubernetes
versions:
- "1.28"
dependencies:
- role: storage
- role: cert-manager

View File

@@ -0,0 +1,41 @@
---
# Role: monitoring
# Déploie Prometheus, Grafana, Loki et Promtail
- name: Installer kube-prometheus-stack
kubernetes.core.helm:
name: prometheus
chart_ref: "{{ helm_charts.prometheus.chart }}"
release_namespace: monitoring
create_namespace: true
values:
prometheus:
prometheusSpec:
retention: "{{ monitoring.prometheus_retention }}"
storageSpec:
volumeClaimTemplate:
spec:
storageClassName: "{{ storage_class }}"
resources:
requests:
storage: "{{ storage_sizes.prometheus }}"
grafana:
adminPassword: "{{ monitoring.grafana_admin_password }}"
persistence:
enabled: true
size: "{{ storage_sizes.grafana }}"
alertmanager:
enabled: false
- name: Installer Loki Stack
kubernetes.core.helm:
name: loki
chart_ref: "{{ helm_charts.loki.chart }}"
release_namespace: monitoring
values:
loki:
persistence:
enabled: true
size: "{{ storage_sizes.loki }}"
promtail:
enabled: true

View File

@@ -0,0 +1,5 @@
---
# Role: namespaces
# Crée les namespaces Kubernetes
# Les namespaces sont définis dans group_vars (variable: namespaces)
# Aucune variable custom supplémentaire requise pour ce rôle.

Some files were not shown because too many files have changed in this diff Show More