Add extracted tools: CitrineOS, OpenOCPP, ShapeShifter

- CitrineOS core extracted (CSMS OCPP 2.0.1) - OpenOCPP extracted (firmware OCPP 1.6J/2.0.1) - ShapeShifter library installed (pip install -e) - ShapeShifter specification extracted - EVerest extracted TODO updated with progress
2026-06-08 00:38:27 -04:00
parent 468cfeaa50
commit d398a6ced2
7326 changed files with 1177561 additions and 7 deletions
--- a/tools/citrineos-core-main/apps/Server/README.md
+++ b/tools/citrineos-core-main/apps/Server/README.md
@@ -0,0 +1,295 @@
+<!--
+SPDX-FileCopyrightText: 2025 Contributors to the CitrineOS Project
+
+SPDX-License-Identifier: Apache-2.0
+-->
+
+![CitrineOS Logo](../../logo_white.png#gh-dark-mode-only)
+![CitrineOS Logo](../../logo_black.png#gh-light-mode-only)
+
+# CitrineOS Server (`@citrineos/server`)
+
+This is the OCPP server application for CitrineOS — the runnable entrypoint that wires together
+[`@citrineos/base`](../../packages/base) and [`@citrineos/core`](../../packages/core) into a deployable
+service. It hosts the WebSocket endpoints that charging stations connect to, the OCPP message router, the
+HTTP/REST Data and Message APIs, and the Sequelize database migrations.
+
+It is one workspace member of the `citrineos-core` pnpm monorepo. For repository-wide setup (cloning,
+`pnpm install`, building, the full-stack Docker Compose files, and the operator UI), see the
+[root README](../../README.md).
+
+## Table of Contents
+
+- [Running the Server](#running-the-server)
+  - [With Docker (backend only)](#with-docker-backend-only)
+  - [Without Docker](#without-docker)
+- [Attaching a Debugger](#attaching-a-debugger)
+- [Server Ports](#server-ports)
+- [Database Sync vs. Migration](#database-sync-vs-migration)
+- [Runtime Configuration](#runtime-configuration)
+- [Bootstrap Configuration Environment Variables](#bootstrap-configuration-environment-variables)
+- [Generating OCPP Interfaces](#generating-ocpp-interfaces)
+- [Validating Custom OCPP DataTransfer Messages](#validating-custom-ocpp-datatransfer-messages)
+- [Allow Unknown Charging Stations & Auto-Commissioning](#allow-unknown-charging-stations--auto-commissioning)
+- [Hasura Metadata](#hasura-metadata)
+- [Testing with EVerest](#testing-with-everest)
+
+## Running the Server
+
+Make sure the workspace has been installed and built first (from the repository root: `pnpm install && pnpm run build`).
+
+### With Docker (backend only)
+
+`apps/Server/docker-compose.yml` brings up the server plus its supporting services — RabbitMQ, PostgreSQL, MinIO,
+and Hasura — but **not** the operator UI. The server image is built from local source and the source tree is mounted
+as volumes for live reload. Run it from this directory:
+
+```shell
+cd apps/Server
+docker compose up -d
+```
+
+To run the full stack including the operator UI, use one of the root-level Compose files instead — see the
+[root README](../../README.md#information-on-docker-setup).
+
+### Without Docker
+
+To start the server directly with pnpm, run from the repository root:
+
+```shell
+pnpm run start
+```
+
+Or from this directory:
+
+```shell
+cd apps/Server
+pnpm run start
+```
+
+This launches the server via `nodemon` (see `nodemon.json`), which builds the workspace, runs database migrations,
+and then starts the process with the Node.js inspector listening on port 9229.
+
+CitrineOS requires configuration to allow your OCPP 1.6 and OCPP 2.0.1 compliant charging stations to connect.
+To change the configuration used outside of Docker, adjust the configuration file at
+`apps/Server/src/config/envs/local.ts`. Make sure any changes to the local configuration do not make it into your PR.
+
+## Attaching a Debugger
+
+Whether you run the application with Docker or locally with pnpm, you can attach a debugger to port 9229 and set
+breakpoints in the TypeScript code directly from your IDE.
+
+To make the process **wait for the debugger to attach** before executing, modify the `nodemon.json` exec command from:
+
+```shell
+pnpm run build --prefix ../../ && pnpm run migrate && node --inspect=0.0.0.0:9229 ./dist/index.js
+```
+
+to:
+
+```shell
+pnpm run build --prefix ../../ && pnpm run migrate && node --inspect-brk=0.0.0.0:9229 ./dist/index.js
+```
+
+## Server Ports
+
+When running, the server container exposes the following ports (see `docker-compose.yml`):
+
+- `8080`: webserver HTTP — [Swagger](http://localhost:8080/docs)
+- `8081`: websocket server TCP connection without auth
+- `8082`: websocket server TCP connection with basic HTTP auth
+- `8083`: additional websocket server
+- `8443` / `8444`: TLS websocket servers
+- `9229`: Node.js debugger
+
+## Database Sync vs. Migration
+
+By default, CitrineOS uses migrations to manage database schema changes. This is the recommended approach for
+production environments. The `pnpm run migrate` script (run automatically on start via `nodemon.json`) applies
+Sequelize migrations.
+
+For development purposes, you can also use `sync` to automatically synchronize your database schema with the models.
+Two sync scripts are available at the repository root:
+
+- `pnpm run sync-db`: synchronizes the database schema with the models without altering existing tables. Useful for
+  development when you want to quickly update the schema without losing data.
+- `pnpm run force-sync-db`: drops all tables and recreates them based on the models. Useful when you want to start
+  with a fresh database.
+
+**Disclaimer:** Using `sync` in a production environment is not recommended as it can lead to data loss. Always use
+migrations for production deployments.
+
+## Runtime Configuration
+
+Values from configuration files (`local.ts`, `docker.ts`, `swarm.docker.ts`) may be overridden at runtime via
+environment variables. Environment variables prefixed with `citrineos_` and hierarchically separated by an
+underscore will override the corresponding value. For example, the amqp URL:
+
+```json
+util: {
+    (...)
+    messageBroker: {
+        amqp: {
+            url: 'amqp://guest:guest@localhost:5672'
+            (...)
+        }
+        (...)
+    }
+    (...)
+}
+```
+
+may be overridden by setting the environment variable `CITRINEOS_util_messageBroker_amqp_url` (case-insensitive).
+
+## Bootstrap Configuration Environment Variables
+
+All environment variables use the `CITRINEOS_` prefix.
+Additional prefixes can be added by passing the `--env-prefix` argument to nodemon (see `start:instance1` in
+`package.json`).
+Here's the complete list of environment variables used in bootstrapping the application (this is not the full system
+configuration):
+
+### Basic Bootstrap Configuration
+
+- `BOOTSTRAP_CITRINEOS_CONFIG_FILENAME` - Name of the main config file (default: `config.json`)
+- `BOOTSTRAP_CITRINEOS_CONFIG_DIR` - Directory containing the config file (optional)
+- `BOOTSTRAP_CITRINEOS_FILE_ACCESS_TYPE` - Type of file access: `local`, `s3`, or `gcp`
+
+### Database Configuration
+
+Database connection details (moved from system config to bootstrap config for better security and 12-factor compliance):
+
+- `BOOTSTRAP_CITRINEOS_DATABASE_HOST` - Database host (default: `localhost`)
+- `BOOTSTRAP_CITRINEOS_DATABASE_PORT` - Database port (default: `5432`)
+- `BOOTSTRAP_CITRINEOS_DATABASE_NAME` - Database name (default: `citrine`)
+- `BOOTSTRAP_CITRINEOS_DATABASE_DIALECT` - Database dialect (default: `postgres`)
+- `BOOTSTRAP_CITRINEOS_DATABASE_USERNAME` - Database username (optional)
+- `BOOTSTRAP_CITRINEOS_DATABASE_PASSWORD` - Database password (optional)
+- `BOOTSTRAP_CITRINEOS_DATABASE_SYNC` - Enable database sync (via sequelize) (true/false, default: `false`)
+- `BOOTSTRAP_CITRINEOS_DATABASE_ALTER` - Enable database alter (via sequelize) (true/false, default: `false`)
+- `BOOTSTRAP_CITRINEOS_DATABASE_MAX_RETRIES` - Maximum connection retries (default: `3`)
+- `BOOTSTRAP_CITRINEOS_DATABASE_RETRY_DELAY` - Retry delay in milliseconds (default: `1000`)
+
+### Local File Access
+
+When `BOOTSTRAP_CITRINEOS_FILE_ACCESS_TYPE=local`:
+
+- `BOOTSTRAP_CITRINEOS_FILE_ACCESS_LOCAL_DEFAULT_FILE_PATH` - Default file path (default: `/data`)
+
+### S3 File Access
+
+When `BOOTSTRAP_CITRINEOS_FILE_ACCESS_TYPE=s3`:
+
+- `BOOTSTRAP_CITRINEOS_FILE_ACCESS_S3_REGION` - AWS region (optional)
+- `BOOTSTRAP_CITRINEOS_FILE_ACCESS_S3_ENDPOINT` - S3 endpoint URL (for MinIO or custom S3)
+- `BOOTSTRAP_CITRINEOS_FILE_ACCESS_S3_DEFAULT_BUCKET_NAME` - S3 bucket name (default: `citrineos-s3-bucket`)
+- `BOOTSTRAP_CITRINEOS_FILE_ACCESS_S3_FORCE_PATH_STYLE` - Force path style (true/false, default: `true`)
+- `BOOTSTRAP_CITRINEOS_FILE_ACCESS_S3_ACCESS_KEY_ID` - S3 access key ID
+- `BOOTSTRAP_CITRINEOS_FILE_ACCESS_S3_SECRET_ACCESS_KEY` - S3 secret access key
+
+### GCP File Access
+
+When `BOOTSTRAP_CITRINEOS_FILE_ACCESS_TYPE=gcp`:
+
+- `BOOTSTRAP_CITRINEOS_FILE_ACCESS_GCP_PROJECTID` - Project ID
+- `BOOTSTRAP_CITRINEOS_FILE_ACCESS_GCP_CREDENTIALS` - GCP Credentials object (Optional, if not set will use Application Default Credentials such as the `GOOGLE_APPLICATION_CREDENTIALS` environment variable or gcloud CLI credentials)
+
+## Generating OCPP Interfaces
+
+All CitrineOS interfaces for OCPP 1.6, 2.0.1, and 2.1-defined schemas were procedurally generated using a processing
+script. Schemas are sourced from official OCPP JSON files.
+As of release 1.8.0, the schema files used by CitrineOS are not the raw output of this function; we have added
+field-level validation that the official schemas lack.
+
+## Validating Custom OCPP DataTransfer Messages
+
+It is possible to add custom JSON schemas to validate the data fields of DataTransfer messages, which are supported by
+all OCPP versions.
+In the `apps/Server/src/index.ts` code, there is a function `ajvInstance()` that creates the AJV instance. Here, you
+could register DataTransfer schemas:
+
+```
+import { MyDataTransferRequestSchema } from './path'
+...
+ajvInstance.compile(MyDataTransferRequestSchema);
+```
+
+Note: The schema's `$id` field must follow this format:
+
+```
+${protocol}-${dataTransferRequest.vendorId}${dataTransferRequest.messageId ? `-${dataTransferRequest.messageId}` : ''}
+```
+
+'Protocol' is the OCPP websocket subprotocol, i.e. "ocpp1.6", "ocpp2.0.1", or so on.
+
+CitrineOS's validation logic assumes that the data field is a string field with JSON structure, and uses `JSON.parse`
+before validation. Other approaches to custom DataTransfer message types are not supported.
+
+## Allow Unknown Charging Stations & Auto-Commissioning
+
+The System Configuration defines websocket servers with certain properties, one of which is 'Allow Unknown Charging
+Stations', a boolean that permits charging stations which are not commissioned to connect to CitrineOS.
+This triggers an auto-commissioning flow which creates the station on its first connection, and creates evses and
+connectors for that station in response to StatusNotifications.
+This is not recommended for production; it is exclusively for testing and is enabled by the default configuration only
+on the websocket server at port 8081 — which also has no security.
+Since not all information on the charger is necessarily available in the OCPP messages, commissioning may be wrong and
+will be incomplete. In 1.6 in particular, multi-evse stations will not commission properly because 1.6 does not have a
+concept of 'evses'. This will lead to improper behavior if a 1.6 station with multiple evses is auto-commissioned:
+CitrineOS will assume each new transaction is on the same evse and will automatically mark older transactions on that
+evse as inactive, leading to an inconsistent state with the charging station.
+
+## Hasura Metadata
+
+In order for Hasura to track the existing Citrine tables and relationships, this repository comes with Hasura metadata
+already exported into the `apps/Server/hasura-metadata` folder.
+Running the Docker container will automatically import this metadata and track all tables and relationships.
+
+Unfortunately, Hasura doesn't currently support importing metadata from a JSON (which is the format if you export your
+metadata from the Hasura UI or API).
+Refer to this issue for more information: https://github.com/hasura/graphql-engine/issues/8423#issuecomment-1115996153.
+
+Therefore, you must use the Hasura CLI to re-export your metadata, should something change with it. As explained in the
+Hasura docs https://hasura.io/docs/2.0/migrations-metadata-seeds/auto-apply-migrations/#auto-apply-metadata,
+Hasura provides an image called `hasura/graphql-engine:<version>.cli-migrations-v3` that will process and import the
+metadata first before starting the server and runs the Hasura CLI internally. This is the image CitrineOS normally uses
+in order to automatically load accurate metadata. However, if you want to capture the current state of your database,
+you should use a normal version tag (such as `v2.40.3` instead of `v2.40.3.cli-migrations-v3`). Then proceed to the
+Hasura console at `localhost:8090`, go to the data tab, use the sidebar to navigate to the database schema at
+default>public, and track all of the tables, relationships, and functions you need. Then proceed with the below
+instructions.
+
+You can follow these steps to re-export your metadata via the Hasura CLI in the `graphql-engine` container:
+
+- (if the hasura cli isn't installed):
+
+```
+curl -L https://github.com/hasura/graphql-engine/raw/stable/cli/get.sh | bash
+```
+
+- (If not yet initialized) Initialize the Hasura project in the `graphql-engine` container (you can do this via the Docker Desktop `exec` view):
+
+```
+hasura-cli init
+OR
+hasura init
+
+enter any name you wish for the project (i.e. citrine)
+```
+
+- Export the metadata by executing this command in the `graphql-engine` container:
+
+```
+hasura-cli metadata export
+OR
+hasura metadata export
+```
+
+- Find the exported files in the `graphql-engine` container's files in the metadata filepath `<name of project i.e. citrine>/metadata` and pull that metadata backup onto your local machine
+- Copy the contents of the copied `metadata` folder into the `apps/Server/hasura-metadata` folder in this repository
+
+## Testing with EVerest
+
+In case you don't have a charger that supports OCPP to experiment with, you can run the EVerest charger simulator
+locally and point it at CitrineOS. Helper scripts (`pnpm run start-everest` and `pnpm run start-everest-16`) and full
+instructions live in [`everest/README.md`](./everest/README.md).