Add extracted tools: CitrineOS, OpenOCPP, ShapeShifter

- CitrineOS core extracted (CSMS OCPP 2.0.1)
- OpenOCPP extracted (firmware OCPP 1.6J/2.0.1)
- ShapeShifter library installed (pip install -e)
- ShapeShifter specification extracted
- EVerest extracted

TODO updated with progress

tools/shapeshifter-library-python-main/docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = source
+BUILDDIR      = build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

tools/shapeshifter-library-python-main/docs/make.bat

@@ -0,0 +1,35 @@
+@ECHO OFF
+
+pushd %~dp0
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+	set SPHINXBUILD=sphinx-build
+)
+set SOURCEDIR=source
+set BUILDDIR=build
+
+%SPHINXBUILD% >NUL 2>NUL
+if errorlevel 9009 (
+	echo.
+	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+	echo.installed, then set the SPHINXBUILD environment variable to point
+	echo.to the full path of the 'sphinx-build' executable. Alternatively you
+	echo.may add the Sphinx directory to PATH.
+	echo.
+	echo.If you don't have Sphinx installed, grab it from
+	echo.https://www.sphinx-doc.org/
+	exit /b 1
+)
+
+if "%1" == "" goto help
+
+%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+goto end
+
+:help
+%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+
+:end
+popd

tools/shapeshifter-library-python-main/docs/requirements.txt

@@ -0,0 +1 @@
+shapeshifter-uftp

tools/shapeshifter-library-python-main/docs/source/api/clients.rst

@@ -0,0 +1,7 @@
+Client API Reference
+====================
+
+You can use any of the four prepared clients to communicate to services of the other parties.
+
+.. automodule:: shapeshifter_uftp.client
+   :exclude-members: ShapeshifterClient

tools/shapeshifter-library-python-main/docs/source/api/index.rst

@@ -0,0 +1,13 @@
+API Reference
+=============
+
+This is the low-level API reference.
+
+.. toctree::
+   :maxdepth: 2
+   :caption: API Reference:
+
+   clients
+   services
+   types
+   transport

tools/shapeshifter-library-python-main/docs/source/api/services.rst

@@ -0,0 +1,8 @@
+Service API Reference
+=====================
+
+The servers listen for request from other clients.
+
+.. automodule:: shapeshifter_uftp.service.agr_service
+.. automodule:: shapeshifter_uftp.service.dso_service
+.. automodule:: shapeshifter_uftp.service.cro_service

tools/shapeshifter-library-python-main/docs/source/api/transport.rst

@@ -0,0 +1,7 @@
+Transport API Reference
+=======================
+
+Low-level transport details for sealing and unsealing messages
+
+.. automodule:: shapeshifter_uftp.transport
+

tools/shapeshifter-library-python-main/docs/source/api/types.rst

@@ -0,0 +1,10 @@
+Types API Reference
+===================
+
+These objects correspond directly to the UFTP objects that are sent across the wire. You will encounter these in the messages and response when using Shapeshifter UFTP.
+
+.. automodule:: shapeshifter_uftp.uftp.common
+.. automodule:: shapeshifter_uftp.uftp.agr_cro
+.. automodule:: shapeshifter_uftp.uftp.agr_dso
+.. automodule:: shapeshifter_uftp.uftp.cro_dso
+.. automodule:: shapeshifter_uftp.uftp.metering

tools/shapeshifter-library-python-main/docs/source/cli.rst

@@ -0,0 +1,44 @@
+CLI (Command Line Interface)
+============================
+
+These utilities are provided to developers to speed up the development process.
+
+
+Create a new keypair
+--------------------
+
+You can generate a signing keypair using the shapeshifter-keypair command line tool:
+
+.. code-block:: bash
+
+    $ shapeshifter-keypair
+    Private key (base64): Wyg81Lki5Ib4YkEFtqfkR6edTFQywjXoZybtBQLbNJbOz+ZRPsx4RptptEDEd9Pn4UE/RWuYP/gmlbYX8Kgr8g==
+    Public key (base64):  zs/mUT7MeEababRAxHfT5+FBP0VrmD/4JpW2F/CoK/I=
+
+
+Perform a DNS-lookup for a party's endpoints and keys
+-----------------------------------------------------
+
+The UFTP specification deals a way of publishing service discovery information over DNS. This CLI tool will quicly look up the well-known DNS endpoints and tell you the information that is available:
+
+.. code-block:: bash
+
+    $ shapeshifter-lookup -d enexis.dev -r dso
+    --------------------------------------------------------------------------------
+    Shapeshifer version: 3.0.0
+    Endpoint URL:        https://shapeshifter-dso.enexis.dev/shapeshifter/api/v3/message
+    Signing key:         zs/mUT7MeEababRAxHfT5+FBP0VrmD/4JpW2F/CoK/I=
+    Decryption Key:      dW+UWxFrGVE1eu1OkPSMl+qXT5/rwKzVSRoU0XSJ0RY=
+    --------------------------------------------------------------------------------
+
+
+If no information is found, it will tell you the DNS names it was looking at, to make it easier to publish your own DNS records:
+
+.. code-block:: bash
+
+    $ shapeshifter-lookup --domain example.com --role dso
+    --------------------------------------------------------------------------------
+    Could not retrieve version at _usef.example.com: DNS name not found
+    Could not retrieve endpoint at _http._dso._usef.example.com: DNS name not found.
+    Could not retrieve public keys at _dso._usef.example.com: DNS name not found.
+    --------------------------------------------------------------------------------

tools/shapeshifter-library-python-main/docs/source/concepts.rst

@@ -0,0 +1,136 @@
+Concepts
+========
+
+Communication Structure
+-----------------------
+
+This library provides all the parts you need to build a Shapeshifter-compliant participant.
+
+Each request is a subclass of :code:`PayloadMessage`. The python library performs some checks on the validity of the message, and responds with an appropriate HTTP status code. If the message was valid, it is then handed to one of your functions, so that you can send the response to this message.
+
+After receiving the message, the receiving party usually wants to send an actual response of some kind. For instance, a :code:`FlexRequest` message from the Distribution System Operator DSO might be replied to using a :code:`FlexOffer` message from the Aggregator (AGR). In shapeshifter-uftp this is called the **processing** step, and happens separately from the request context. A typical post-processing step might look like this:
+
+.. code-block:: python3
+
+    from shapeshifter_uftp import (
+        ShapeshifterAgrService,
+        AcceptedRejected,
+        FlexRequest,
+        FlexOffer,
+        PayloadMessageResponse
+    )
+
+    class MyAggregatorService(ShapeshifterAggregatorService):
+
+        ...
+
+        def process_flex_request(self, message: FlexRequest):
+            # Do some work to determine what flexibility we can offer to the DSO
+            available_flex = my_backend.get_available_flexibility(...)
+
+            # Send the FlexOffer message to the DSO
+            with self.dso_client(message.sender_domain) as client:
+                response = client.send_flex_offer(FlexOffer(...))
+
+        ...
+
+This pattern repeats for all the messages that are exchanged between the participants:
+
+
+**Aggregator (AGR) Service:**
+
+- Messages from the DSO:
+    - :code:`process_d_prognosis_response`
+    - :code:`process_flex_request`
+    - :code:`process_flex_offer_response`
+    - :code:`process_flex_offer_revocation_response`
+    - :code:`process_flex_order`
+    - :code:`process_flex_reservation_update`
+    - :code:`process_flex_settlement`
+    - :code:`process_metering_response`
+- Messages from the CRO:
+    - :code:`process_agr_portfolio_query_response`
+    - :code:`process_agr_portfolio_update_response`
+
+**Common Reference Operator (CRO) Service:**
+
+- Messages from the Aggregator
+    - :code:`process_agr_portfolio_query`
+    - :code:`process_agr_portfolio_update`
+- Messages from the DSO
+    - :code:`process_dso_portfolio_query`
+    - :code:`process_dso_portfolio_update`
+
+
+**Distribution System Operator (DSO) Service:**
+
+- Messages from the Aggregator:
+    - :code:`process_d_prognosis`
+    - :code:`process_flex_request_response`
+    - :code:`process_flex_offer`
+    - :code:`process_flex_order_response`
+    - :code:`process_flex_offer_revocation`
+    - :code:`process_flex_reservation_update_response`
+    - :code:`process_flex_settlement_response`
+    - :code:`process_metering`
+- Messages from the CRO:
+    - :code:`process_dso_portfolio_query_response`
+    - :code:`process_dso_portfolio_update_response`
+
+
+Identification of participants
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Shapeshifter Messages always come inside and envelope called SignedMessage. This envelope contains the following items:
+- :code:`sender_domain`: the canonical domain name of the sender. This is not a full URL, but merely an identificiation.
+- :code:`sender_role`: the role of the sender of the message, either :code:`AGR` for Aggregator, :code:`CRO` for Common Reference Operator, or :code:`DSO` for Distribution System Operator.
+- :code:`body`: a base64-encoded signed message that can be decoded using the sender's public key.
+
+The recipient of a message will look at the sender_domain and sender_role and determine if they know this party. If they do, they can use some key lookup function to retrieve the public key with which the message can be opened. See: Looking up keys.
+
+Looking up participant keys
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When encountering a combination of sender_domain and sender_role, you can look up their public key in two ways:
+
+- Using DNS: a well-known system of DNS names is specified in the UFTP protocol definition. Shapsehifter-UFTP implements this en encourages you to use it.
+- USing a custom key-lookup method that takes the sender_role and sender_domain as arguments, and should return the public key in base64 format. This way, you can implement your own lookup function, which might look up the information in your own database, or perform an external API call to the GOPACS Shapeshifter Address Book, for example.
+
+.. code-block:: python3
+
+    from shapeshifter_uftp.exceptions import AuthenticationTimeoutException
+
+    def key_lookup(sender_domain, sender_role):
+        cursor = database.cursor()
+        cursor.execute("SELECT public_key FROM shapeshifter_participants WHERE sender_role = %s AND sender_domain = %s", (sender_role, sender_domain))
+        if cursor.rowcount == 0:
+            raise AuthenticationTimeoutException()
+        public_key = cursor.fetchone()[0]
+        return public_key
+
+
+
+Message Schema and Default Values
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The structure of UFTP messages looks like this:
+
+- SignedMessage
+    - SenderDomain
+    - SenderRole
+    - Body: a base64-encoded blob that contains the PayloadMessage and the signature.
+
+These :code:`SignedMessage` s are never exposed to you, the developer, and are taken care of within shapeshifter-uftp.
+
+What you deal with is the contents of the body of that message, which is always a subclass of :code:`PayloadMessage`.
+
+Each :code:`PayloadMessage` contains the following default properties:
+
+- :code:`Version`: the protocol version that this message complies to
+- :code:`SenderDomain`: the domain of the sending participant
+- :code:`RecipientDomain`: the demain of the recipient
+- :code:`TimeStamp`: the timestamp at which the message was created
+- :code:`MessageID`: a unique identifier for this message
+- :code:`ConversationID`: an identifier of the conversation this message belongs to
+
+All of these are required properties, but all of these can be calculated by the framework during message transmission. You as a developer don't need to supply these arguments for each message you create. If you want to override any of these, you can.

tools/shapeshifter-library-python-main/docs/source/conf.py

@@ -0,0 +1,51 @@
+# Configuration file for the Sphinx documentation builder.
+#
+# For the full list of built-in configuration values, see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Project information -----------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
+
+import os
+import sys
+
+cwd = os.path.dirname(__file__)
+project_root = os.path.join(os.path.dirname(os.path.dirname(cwd)))
+sys.path.insert(0, project_root)
+
+import shapeshifter_uftp
+
+project = "Shapeshifter UFTP"
+copyright = "2023, Enexis Groep"
+author = "Stan Janssen"
+release = "1.0.0"
+
+# -- General configuration ---------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
+
+extensions = [
+    "sphinx.ext.autodoc",
+    "sphinx.ext.autosummary",
+    "sphinx.ext.napoleon",
+    "sphinx.ext.viewcode",
+]
+
+autosummary_generate = True
+autodoc_default_options = {
+    # The ones below should be optional but work nicely together with
+    # example_package/autodoctest/doc/source/_templates/autosummary/class.rst
+    # and other defaults in sphinx-autodoc.
+    "inherited-members": True,
+    "no-special-members": True,
+    "member-order": "bysource",
+}
+
+templates_path = ["_templates"]
+exclude_patterns = []
+
+
+# -- Options for HTML output -------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
+
+html_theme = "sphinx_rtd_theme"
+html_static_path = ["_static"]

tools/shapeshifter-library-python-main/docs/source/examples/index.rst

@@ -0,0 +1,148 @@
+Examples
+========
+
+
+Self-contained Aggregator service and client
+--------------------------------------------
+
+.. code-block:: python3
+
+    from datetime import datetime, timedelta, timezone
+
+    from shapeshifter_uftp import ShapeshifterAgrService
+    from shapeshifter_uftp.uftp import (FlexOffer, FlexOfferOption,
+                                        FlexOfferOptionISP, FlexRequest,
+                                        FlexRequestResponse, FlexOrder, FlexOrderResponse,
+                                        AcceptedRejected)
+    from xsdata.models.datatype import XmlDate
+
+
+    class DemoAggregator(ShapeshifterAgrService):
+        def process_agr_portfolio_query_response(self, message):
+            print(f"Received a message: {message}")
+
+        def process_agr_portfolio_update_response(self, message):
+            print(f"Received a message: {message}")
+
+        def process_d_prognosis_response(self, message):
+            print(f"Received a message: {message}")
+
+        def process_flex_request(self, message: FlexRequest):
+            print(f"Received a message: {message}")
+
+            # Example of how to send a new message after
+            # processing an incoming message.
+            dso_client = self.dso_client(message.sender_domain)
+
+            # Send the FlexRequestResponse
+            dso_client.send_flex_request_response(
+                FlexRequestResponse(
+                    flex_request_message_id=message.message_id,
+                    conversation_id=message.conversation_id,
+                    result=AcceptedRejected.ACCEPTED
+                )
+            )
+
+            # Send the FlexOffer
+            dso_client.send_flex_offer(
+                FlexOffer(
+                    flex_request_message_id=message.message_id,
+                    conversation_id=message.conversation_id,
+                    isp_duration="PT15M",
+                    period=XmlDate(2023, 1, 1),
+                    congestion_point="ean.123456789012",
+                    expiration_date_time=datetime.now(timezone.utc).isoformat(),
+                    offer_options=[
+                        FlexOfferOption(
+                            isps=[FlexOfferOptionISP(power=1, start=1, duration=1)],
+                            option_reference="MyOption",
+                            price=2.30,
+                            min_activation_factor=0.5,
+                        )
+                    ],
+                )
+            )
+
+        def process_flex_offer_response(self, message: FlexOffer):
+            print(f"Received a message: {message}")
+
+        def process_flex_offer_revocation_response(self, message):
+            print(f"Received a message: {message}")
+
+        def process_flex_order(self, message: FlexOrder):
+            print(f"Received a message: {message}")
+
+            dso_client = self.dso_client(message.sender_domain)
+            dso_client.send_flex_order_response(
+                FlexOrderResponse(
+                    flex_order_message_id=message.message_id,
+                    conversation_id=message.conversation_id,
+                    result=AcceptedRejected.ACCEPTED
+                )
+            )
+
+        def process_flex_reservation_update(self, message):
+            print(f"Received a message: {message}")
+
+        def process_flex_settlement(self, message):
+            print(f"Received a message: {message}")
+
+        def process_metering_response(self, message):
+            print(f"Received a message: {message}")
+
+
+    def key_lookup(sender_domain, sender_role):
+        known_senders = {
+            ("dso.demo", "DSO"): "NsTbq/iABU6tbsjriBg/Z5dSfQstulD0GpMI2fLDWec=",
+            ("cro.demo", "CRO"): "ySUYU87usErRFKGJafwvVDLGhnBVJCCNYfQvmwv8ObM=",
+        }
+        return known_senders.get((sender_domain, sender_role))
+
+
+    def endpoint_lookup(sender_domain, sender_role):
+        known_senders = {
+            ("dso.demo", "DSO"): "http://localhost:8081/shapeshifter/api/v3/message",
+            ("cro.demo", "CRO"): "http://localhost:8082/shapeshifter/api/v3/message",
+        }
+        return known_senders.get((sender_domain, sender_role))
+
+    if __name__ == "__main__":
+        aggregator = DemoAggregator(
+            sender_domain="aggregator.demo",
+            signing_key="mz5XYCNKxpx48K+9oipUhsjBZed3L7rTVKLsWmG1HOqRLIeuGpIa1KAt6AlbVGqJvewd8v1J0uVUTqpGt7F8tw==",
+            key_lookup_function=key_lookup,
+            endpoint_lookup_function=endpoint_lookup,
+            port=8080,
+        )
+
+        # Start the Aggregator Service
+        aggregator.run_in_thread()
+
+        # Create a client object to talk to a DSO
+        dso_client = aggregator.dso_client("dso.demo")
+
+        # Create a Flex Offer Message
+        flex_offer_message = FlexOffer(
+            isp_duration="PT15M",
+            period=XmlDate(2023, 1, 1),
+            congestion_point="ean.123456789012",
+            expiration_date_time=datetime.now(timezone.utc).isoformat(),
+            offer_options=[
+                FlexOfferOption(
+                    isps=[FlexOfferOptionISP(power=1, start=1, duration=1)],
+                    option_reference="MyOption",
+                    price=2.30,
+                    min_activation_factor=0.5,
+                )
+            ],
+        )
+
+        # As a demo, press enter to send another FlexOffer message to the DSO.
+        while True:
+            try:
+                input("Press return to send a FlexOffer message to the DSO")
+                response = dso_client.send_flex_offer(flex_offer_message)
+                print(f"Response was: {response}")
+            except:
+                aggregator.stop()
+                break

tools/shapeshifter-library-python-main/docs/source/index.rst

@@ -0,0 +1,27 @@
+.. shapeshifter-uftp documentation master file, created by
+   sphinx-quickstart on Wed May 31 15:17:01 2023.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+Welcome to Shapeshifter UFTP's documentation!
+=============================================
+
+This library allows you to quickly build applications that talk to Distribution System Operators, Aggregators and Common Reference Operators as defined in the UFTP Specification.
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Contents:
+
+   readme
+   concepts
+   examples/index
+   cli
+   api/index
+
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`

tools/shapeshifter-library-python-main/docs/source/readme.rst

@@ -0,0 +1 @@
+.. include:: ../../README.rst