Add extracted tools: CitrineOS, OpenOCPP, ShapeShifter

- CitrineOS core extracted (CSMS OCPP 2.0.1)
- OpenOCPP extracted (firmware OCPP 1.6J/2.0.1)
- ShapeShifter library installed (pip install -e)
- ShapeShifter specification extracted
- EVerest extracted

TODO updated with progress
This commit is contained in:
Eric F
2026-06-08 00:38:27 -04:00
parent 468cfeaa50
commit d398a6ced2
7326 changed files with 1177561 additions and 7 deletions

View File

@@ -0,0 +1,9 @@
from .agr_service import ShapeshifterAgrService
from .cro_service import ShapeshifterCroService
from .dso_service import ShapeshifterDsoService
__all__ = [
"ShapeshifterAgrService",
"ShapeshifterCroService",
"ShapeshifterDsoService",
]

View File

@@ -0,0 +1,192 @@
from abc import ABC, abstractmethod
from ..client import ShapeshifterAgrCroClient, ShapeshifterAgrDsoClient
from ..uftp import (
AgrPortfolioQueryResponse,
AgrPortfolioUpdateResponse,
DPrognosisResponse,
FlexOfferResponse,
FlexOfferRevocationResponse,
FlexOrder,
FlexRequest,
FlexReservationUpdate,
FlexSettlement,
MeteringResponse,
TestMessage,
TestMessageResponse,
UsefRole,
)
from .base_service import ShapeshifterService
class ShapeshifterAgrService(
ShapeshifterService, ABC
): # pylint: disable=too-many-public-methods
"""
Service that represents the Aggregator in the UFTP communication.
This service can receive requests from the DSO.
"""
sender_role = UsefRole.AGR
acceptable_messages = [
AgrPortfolioQueryResponse,
AgrPortfolioUpdateResponse,
DPrognosisResponse,
FlexOfferResponse,
FlexOfferRevocationResponse,
FlexOrder,
FlexRequest,
FlexReservationUpdate,
FlexSettlement,
MeteringResponse,
TestMessage,
TestMessageResponse,
]
@abstractmethod
def process_d_prognosis_response(self, message: DPrognosisResponse):
"""
FlexOffer messages are used by AGRs to make DSOs an offer for provision
of flexibility. A FlexOffer message contains a list of ISPs and, for
each ISP, the change in consumption or production offered and the price
for the total amount of flexibility offered. FlexOffer messages can be
sent once a FlexRequest message has been received but can also be sent
unsolicited. Note that multiple FlexOffer messages may be sent based on
a single FlexRequest, e.g. to increase the chance that the DSO will
order at least part of its available flexibility. The AGR must make
sure that it can actually provide the flexibility offered across all of
its FlexOffers.
"""
@abstractmethod
def process_flex_request(self, message: FlexRequest):
"""
This method should probably end by sending some Flex Offers to the DSO::
with self.dso_client(message.sender_domain) as client:
response = client.send_flex_offer(FlexOffer(...)
# Do something with the response here.
"""
@abstractmethod
def process_flex_offer_response(self, message: FlexOfferResponse):
"""
This method should probably end by sending some Flex Offers to the DSO::
with self.dso_client(message.sender_domain) as client:
response = client.send_flex_offer(FlexOffer(...)
# Do something with the response here.
"""
@abstractmethod
def process_flex_offer_revocation_response(
self, message: FlexOfferRevocationResponse
):
"""
Upon receiving and processing a FlexOfferRevocation message, the
receiving implementation must reply with a FlexOfferRevocationResponse,
indicating whether the revocation was handled successfully.
It is advised that this method ends by sending a FlexSettlementResponse to the DSO::
with self.dso_client(message.sender_domain):
response = client.send_flex_settlement_response(FlexSettlementResponse(...))
# do something with the response here.
"""
@abstractmethod
def process_flex_order(self, message: FlexOrder):
"""
FlexOrder messages are used by DSOs to purchase flexibility from an AGR
based on a previous FlexOffer. A FlexOrder message contains a list of
ISPs, with, for each ISP, the change in consumption or production to be
realized by the AGR, and the accepted price to be paid by the DSO for
this amount of flexibility. This ISP list should be copied from the
FlexOffer message without modification: AGR implementations will
(and must) reject FlexOrder messages where the ISP list is not exactly
the same as offered.
"""
@abstractmethod
def process_flex_reservation_update(self, message: FlexReservationUpdate):
"""
For bilateral contracts, FlexReservationUpdate messages are used by DSOs
to signal to an AGR which part of the contracted volume is still
reserved and which part is not needed and may be used for other
purposes. For each ISP, a power value is given which indicates how much
power is still reserved. Zero power means that no power is reserved for
that ISP and the sign of the power indicates the direction.
"""
@abstractmethod
def process_flex_settlement(self, message: FlexSettlement):
"""
The FlexSettlement message is sent by DSOs on a regular basis
(typically monthly) to AGRs, in order to initiate settlement. It
includes a list of all FlexOrders placed by the originating party
during the settlement period.
It is advised that this method ends by sending a FlexSettlementResponse to the DSO::
with self.get_dso_client(message.sender_domain):
response = client.send_flex_settlement_response(FlexSettlementResponse(...))
# do something with the response here.
"""
@abstractmethod
def process_metering_response(self, message: MeteringResponse):
"""
Upon receiving and processing a Metering message, the
receiving implementation must reply with a MeteringResponse,
indicating whether the update was handled successfully.
"""
@abstractmethod
def process_agr_portfolio_query_response(self, message: AgrPortfolioQueryResponse):
"""
The AgrPortfolioQueryResponse is sent by the CRO after you sent a
AgrPortfolioQuery. It contains the list of your connections. It is
recommended that you do not perform any long-running operations inside
this function, but return a PayloadMessageResponse quickly.
Longer-running operations (like a database sync) should be done inside
the process_agr_portfolio_query_response method.
If the list of connections does not match what you expected it
to be, you can send an AgrPortfolioUpdate message at the end
of this method::
with self.get_dso_client(message.sender_domain) as client:
response = client.send_portfolio_update(AgrPortfolioUpdate(...))
# Do something with the response here
"""
@abstractmethod
def process_agr_portfolio_update_response(
self, message: AgrPortfolioUpdateResponse
):
"""
The AgrPortfolioUptadeResponse is sent by the CRO after you sent a
AgrPortfolioUpdate.
"""
# ------------------------------------------------------------ #
# Convenience methods for getting a client to the designated #
# participant. #
# ------------------------------------------------------------ #
def cro_client(self, recipient_domain, version="3.1.0") -> ShapeshifterAgrCroClient:
"""
Retrieve a client object for sending messages to the CRO.
"""
return self._get_client(recipient_domain, "CRO", version)
def dso_client(self, recipient_domain, version="3.1.0") -> ShapeshifterAgrDsoClient:
"""
Retrieve a client object for sending messages to the DSO.
"""
return self._get_client(recipient_domain, "DSO", version)

View File

@@ -0,0 +1,310 @@
import re
from concurrent.futures import ThreadPoolExecutor
from threading import Thread
from time import sleep
import uvicorn
from fastapi import FastAPI, Response
from fastapi.exceptions import HTTPException
from fastapi_xml import XmlAppResponse, XmlRoute
from .. import transport
from ..client import client_map
from ..exceptions import (
FunctionalException,
InvalidMessageException,
InvalidSenderException,
TransportException,
)
from ..logging import logger
from ..uftp import (
AcceptedRejected,
PayloadMessage,
SignedMessage,
TestMessage,
TestMessageResponse,
UsefRole,
request_response_map,
)
class ShapeshifterService():
"""
Basis for all Shapeshifter Services. Defines the web service, the
message ingestion, the post-processing queue mechanics, and
threading and context options.
"""
sender_domain = None
sender_role = None
acceptable_messages = []
num_inbound_threads = 10
num_outbound_threads = 10
def __init__(
self,
sender_domain,
signing_key,
key_lookup_function=None,
endpoint_lookup_function=None,
oauth_lookup_function=None,
host: str = "0.0.0.0",
port: int = 8080,
path: str = "/shapeshifter/api/v3/message",
version: str = "3.1.0"
):
"""
:param sender_domain: our sender domain (FQDN) that the recipient uses to look us up.
:param signing_key: the private singing key that we use to sign outgoing messages.
:param key_lookup_function: A callable that takes a (sender_domain, sender_role)
pair and returns a verify_key (str or bytes).
Omit parameter to use DNS for key lookup.
:param key_lookup_function: A callable that takes a (sender_domain, sender_role)
pair and returns a full endpoint URL (str).
Omit parameter to use DNS for endpoint lookup.
:param oauth_lookup_function: A callable that takes a (sender_domain, sender_role)
pair and returns in instance of shapeshifter_uftp.OAuthClient
if OAuth authentication is required.
:param host: the host to bind the server to (usually 127.0.0.1 or 0.0.0.0)
:param port: the port to bind the server to (default: 8080)
:param path: the URL path that the server listens on (default: /shapeshifter/api/v3/message)
"""
if version not in ("3.0.0", "3.1.0"):
raise ValueError(f"'version' must be one of '3.0.0' or '3.1.0', not {version}")
self.version = version
# Set the sender domain, which is used
# to identify us to the other party.
self.sender_domain = sender_domain
# The signing key is used to sign outgoing messages. The
# corresponding public key should be published via DNS or
# given to the recipient out-of-band.
self.signing_key = signing_key
# The key lookup method is used to look up keys for the other
# party. If omitted, use DNS lookups using well-known DNS names.
self.key_lookup_function = key_lookup_function or transport.get_key
# The endpoint lookup method is used to look up the endpoint
# that we send all messages to. If omitted, use DNS lookups
# using well-known DNS names.
self.endpoint_lookup_function = endpoint_lookup_function or transport.get_endpoint
# The OAuth lookup function is used to get the OAuth instance
# used to authenticate outgoing requests.
self.oauth_lookup_function = oauth_lookup_function
# The FastAPI web app takes care of routing messages to the
# (one) endpoint, and by virtue of FastAPI-XML convert the
# python-friendly objects into XML and vice versa.
self.app = FastAPI(default_response_class=XmlAppResponse)
self.app.router.route_class = XmlRoute
self.app.router.add_api_route(
path,
endpoint=self._receive_message,
response_model=None,
methods=["POST"],
status_code=200,
)
# The web server hosts the FastAPI application and takes care
# of the HTTP transport.
config = uvicorn.Config(app=self.app, host=host, port=port)
self.server = uvicorn.Server(config)
self.server_thread = None
# Create an inbound executor worker
self.inbound_executor = ThreadPoolExecutor(max_workers=self.num_inbound_threads)
self.outbound_executor = ThreadPoolExecutor(max_workers=self.num_outbound_threads)
def run(self):
"""
Start the web server that hosts the FastAPI application. Other
participants can now send messages to us.
"""
# Start the service and start accepting incoming requests.
self.server.run()
def run_in_thread(self):
"""
Run the service in a background thread.
"""
self.server_thread = Thread(target=self.run)
self.server_thread.start()
while not self.server.started:
sleep(0.1)
def stop(self):
"""
Stop the service if it was running in a separate thread.
"""
self.server.should_exit = True
if self.server_thread and self.server_thread.is_alive():
self.server_thread.join()
self.server_thread = None
# ------------------------------------------------------------ #
# Message handling and processing methods, internal to the #
# Shapeshifter UFTP implementation. #
# ------------------------------------------------------------ #
def _receive_message(self, message: SignedMessage) -> Response:
"""
The default entrypoint for the route. This will unpack the
message and validate the signature. It will thes pass the
PayloadMessago to the pre processing function for a
response.
"""
logger.info(f"Got a request: {message}")
# Get the public key that is used to decrypt the message
signing_key = self.key_lookup_function(
message.sender_domain, message.sender_role.value
)
logger.debug(f"The signing key is {signing_key}")
# Unseal the message, returning an error if required
try:
unsealed_message = transport.unseal_message(message.body, signing_key)
# Verify that the sender_domain inside the message is the
# same as the sender_domain of the SignedMessage
# envelope
if unsealed_message.sender_domain != message.sender_domain:
logger.warning(
"Received a message with mismatching sender_domain in the "
f"SignedMessage envelope ({message.sender_domain}) and the "
f"inner PayloadMessage ({unsealed_message.sender_domain})"
)
raise InvalidSenderException()
if type(unsealed_message) not in self.acceptable_messages:
logger.warning(
f"Received a misdirected message of type {unsealed_message.__class__.__name__} "
f"from {unsealed_message.sender_domain}.")
raise InvalidMessageException(unsealed_message)
except TransportException as err:
logger.warning(f"The original transport error is {err.__class__.__name__}: {err}")
raise HTTPException(err.http_status_code) from err
except FunctionalException as err:
self.outbound_executor.submit(self._reject_message, message, unsealed_message, err.rejection_reason)
else:
# If the initial checks passed, process the message in the
# user-defined pipeline.
self.inbound_executor.submit(self._process_message, unsealed_message, message.sender_role)
return Response(status_code=200)
def _process_message(self, message: PayloadMessage, sender_role: UsefRole):
"""
Find the relevant post-processing method to handle the message
outside of the request context, and run it.
"""
process_method_name = f"process_{snake_case(message.__class__.__name__)}"
process_method = getattr(self, process_method_name)
try:
if isinstance(message, TestMessage):
process_method(message, sender_role)
else:
process_method(message)
except Exception as err: # pylint: disable=broad-exception-caught
logger.error(
f"An error occurred during the post-processing of a {message.__class__.__name__} message."
f"{err.__class__.__name__}: {err}"
)
def _get_client(self, recipient_domain: str, recipient_role: UsefRole, version: str = "3.1.0"):
"""
Method to get a relevant client to communicate to the
indicated participant.
"""
client_cls = client_map[(self.sender_role, recipient_role)]
recipient_endpoint = self.endpoint_lookup_function(recipient_domain, recipient_role)
recipient_signing_key = self.key_lookup_function(recipient_domain, recipient_role)
oauth_client = self.oauth_lookup_function(recipient_domain, recipient_role) if self.oauth_lookup_function else None
return client_cls(
sender_domain = self.sender_domain,
signing_key = self.signing_key,
recipient_domain = recipient_domain,
recipient_endpoint = recipient_endpoint,
recipient_signing_key = recipient_signing_key,
oauth_client = oauth_client,
version=version,
)
def _reject_message(self, message, unsealed_message, reason):
"""
Send a rejection to the sending party.
"""
if type(unsealed_message) not in request_response_map:
return
client = self._get_client(message.sender_domain, message.sender_role, unsealed_message.version)
response_type = request_response_map[type(unsealed_message)]
response_id_field = snake_case(type(unsealed_message).__name__) + "_message_id"
message_contents = {
"recipient_domain": message.sender_domain,
"conversation_id": unsealed_message.conversation_id,
"result": AcceptedRejected.REJECTED,
"rejection_reason": reason,
response_id_field: unsealed_message.message_id
}
response_message = response_type(**message_contents)
client._send_message(response_message)
def __enter__(self):
"""
Context-manager method that allows an instance of this class to be
used in a temporary context.
Starts the uvicorn server in a separate thread and waits for it
to be started. Useful in testing scenarios.
Usage:
with MyShapeshifterServiceSubClass(...) as server:
# your test code here, server exists cleanly when leaving
# the 'with' block.
"""
self.run_in_thread()
return self
def __exit__(self, *args, **kwargs):
"""
Tell uvicorn we should exit and wait for the thread to finish.
"""
self.stop()
# ------------------------------------------------------------ #
# Common messages to all parties #
# ------------------------------------------------------------ #
def process_test_message(self, message: TestMessage, sender_role: UsefRole):
logger.info(
"Received a TestMessage, will respond with TestMessageResponse. "
f"Implement the process_test_message method on your {self.__class__.__qualname__} "
f"to implement custom behavior. The message was: {message}"
)
response = TestMessageResponse(conversation_id=message.conversation_id)
client = self._get_client(message.sender_domain, sender_role, version=message.version or self.version)
client.send_test_message_response(response)
def process_test_message_response(self, message: TestMessage):
logger.info(f"Received a TestMessageResponse: {message}")
def snake_case(text):
"""
Convert text from CamelCase to snake_case.
"""
return re.sub(r"(.)([A-Z][a-z])", r"\1_\2", text).lower()

View File

@@ -0,0 +1,86 @@
from abc import ABC, abstractmethod
from ..client import ShapeshifterCroAgrClient, ShapeshifterCroDsoClient
from ..uftp import (
AgrPortfolioQuery,
AgrPortfolioUpdate,
DsoPortfolioQuery,
DsoPortfolioUpdate,
TestMessage,
TestMessageResponse,
UsefRole,
)
from .base_service import ShapeshifterService
class ShapeshifterCroService(ShapeshifterService, ABC):
"""
Service that represent the Common Reference Operator in the UFTP communication.
It can receive requests from the Aggregator and from the DSO.
"""
sender_role = UsefRole.CRO
acceptable_messages = [
DsoPortfolioQuery,
DsoPortfolioUpdate,
AgrPortfolioQuery,
AgrPortfolioUpdate,
TestMessage,
TestMessageResponse,
]
# ------------------------------------------------------------ #
# Methods related to Agr Portfolio Query messages #
# ------------------------------------------------------------ #
@abstractmethod
def process_agr_portfolio_query(self, message: AgrPortfolioQuery):
"""
The AGRPortfolioQuery is used by the AGR to retrieve
additional information on the connections.
"""
@abstractmethod
def process_agr_portfolio_update(self, message: AgrPortfolioUpdate):
"""
The AGRPortfolioUpdate is used by the AGR to indicate on which
Connections it represents prosumers.
"""
@abstractmethod
def process_dso_portfolio_query(self, message: DsoPortfolioQuery):
"""
DSOPortfolioQuery is used by DSOs to discover which AGRs represent
connections on its registered congestion point(s).
You should end this method by sending a
DsoPortfolioQueryResponse back to the DSO::
with self.get_dso_client(message.sender_domain) as client:
client.send_portfolio_query_response
"""
@abstractmethod
def process_dso_portfolio_update(self, message: DsoPortfolioUpdate):
"""
The DSOPortfolioUpdate is used by the DSO to indicate on which
congestion points it wants to engage in flexibility trading.
"""
# ------------------------------------------------------------ #
# Convenience methods for getting a client to the designated #
# participant. #
# ------------------------------------------------------------ #
def agr_client(self, recipient_domain, version="3.1.0") -> ShapeshifterCroAgrClient:
"""
Retrieve a client object for sending messages to the AGR.
"""
return self._get_client(recipient_domain, "AGR", version)
def dso_client(self, recipient_domain, version="3.1.0") -> ShapeshifterCroDsoClient:
"""
Retrieve a client object for sending messages to the DSO.
"""
return self._get_client(recipient_domain, "DSO", version)

View File

@@ -0,0 +1,173 @@
from abc import ABC, abstractmethod
from ..client import ShapeshifterDsoAgrClient, ShapeshifterDsoCroClient
from ..uftp import (
DPrognosis,
DsoPortfolioQueryResponse,
DsoPortfolioUpdateResponse,
FlexOffer,
FlexOfferRevocation,
FlexOrderResponse,
FlexRequestResponse,
FlexReservationUpdateResponse,
FlexSettlementResponse,
Metering,
TestMessage,
TestMessageResponse,
UsefRole,
)
from .base_service import ShapeshifterService
# pylint: disable=too-many-public-methods
class ShapeshifterDsoService(ShapeshifterService, ABC):
"""
Service that represents the Distribution System Operator in the UFTP communication.
It can receive requests from the Aggregator.
You should subclass this class and implement your own message handling methods.
"""
sender_role = UsefRole.DSO
acceptable_messages = [
DPrognosis,
DsoPortfolioQueryResponse,
DsoPortfolioUpdateResponse,
FlexOffer,
FlexOfferRevocation,
FlexOrderResponse,
FlexRequestResponse,
FlexReservationUpdateResponse,
FlexSettlementResponse,
Metering,
TestMessage,
TestMessageResponse,
]
@abstractmethod
def process_d_prognosis(self, message: DPrognosis):
"""
D-Prognosis messages are used to communicate D-prognoses between AGRs
and DSOs. D-Prognosis messages always contain data for all ISPs for the
period they apply to, even if a prognosis is sent after the start of
the period, when one or more ISPs are already in the operate or
settlement phase. Receiving implementations should ignore the
information supplied for those ISPs.
"""
@abstractmethod
def process_flex_request_response(self, message: FlexRequestResponse):
"""
FlexOffer messages are used by AGRs to make DSOs an offer for provision
of flexibility. A FlexOffer message contains a list of ISPs and, for
each ISP, the change in consumption or production offered and the price
for the total amount of flexibility offered. FlexOffer messages can be
sent once a FlexRequest message has been received but can also be sent
unsolicited. Note that multiple FlexOffer messages may be sent based on
a single FlexRequest, e.g. to increase the chance that the DSO will
order at least part of its available flexibility. The AGR must make
sure that it can actually provide the flexibility offered across all of
its FlexOffers.
"""
@abstractmethod
def process_flex_offer(self, message: FlexOffer):
"""
FlexOffer messages are used by AGRs to make DSOs an offer for provision
of flexibility. A FlexOffer message contains a list of ISPs and, for
each ISP, the change in consumption or production offered and the price
for the total amount of flexibility offered. FlexOffer messages can be
sent once a FlexRequest message has been received but can also be sent
unsolicited. Note that multiple FlexOffer messages may be sent based on
a single FlexRequest, e.g. to increase the chance that the DSO will
order at least part of its available flexibility. The AGR must make
sure that it can actually provide the flexibility offered across all of
its FlexOffers.
"""
@abstractmethod
def process_flex_order_response(self, message: FlexOrderResponse):
"""
Upon receiving and processing a FlexOrder message, the receiving
implementation must reply with a FlexOrderResponse, indicating whether
the update was handled successfully.
"""
@abstractmethod
def process_flex_offer_revocation(self, message: FlexOfferRevocation):
"""
The FlexOfferRevocation message is used by the AGR to revoke a FlexOffer
previously sent to a DSO. It voids the FlexOffer, even if its validity
time has not yet expired. Revocation is not allowed for FlexOffers that
already have associated accepted FlexOrders.
"""
@abstractmethod
def process_flex_reservation_update_response(
self, message: FlexReservationUpdateResponse
):
"""
The FlexOfferRevocation message is used by the AGR to revoke a FlexOffer
previously sent to a DSO. It voids the FlexOffer, even if its validity
time has not yet expired. Revocation is not allowed for FlexOffers that
already have associated accepted FlexOrders.
"""
@abstractmethod
def process_flex_settlement_response(self, message: FlexSettlementResponse):
"""
Upon receiving and processing a FlexSettlement message, the AGR must
reply with a FlexSettlementResponse, indicating whether the initial
message was handled successfully. When a FlexSettlement message is
rejected, the DSO should consider all FlexOrderSettlement elements of
that message related to potential dispute.
"""
@abstractmethod
def process_dso_portfolio_query_response(self, message: DsoPortfolioQueryResponse):
"""
Upon receiving and processing a DSOPortfolioQuery message, the receiving
implementation must reply with a DSOPortfolioQueryResponse, indicating
whether the query executed successfully, and if it did, including the
query results. Most queries will return zero or more congestion points
"""
@abstractmethod
def process_dso_portfolio_update_response(
self, message: DsoPortfolioUpdateResponse
):
"""
Upon receiving and processing a DSOPortfolioUpdate message, the
receiving implementation must reply with a DSOPortfolioUpdateResponse,
indicating whether the update was handled successfully.
"""
@abstractmethod
def process_metering(self, message: Metering):
"""
The Metering message is an optional message. The DSO will specify
whether metering messages are required for a given program. If metering
messages are used then the AGR must send metering messages, with one
message sent per connection point per day. The metering messages must
all be sent before the settlement can be performed. It is recommend to
send the metering messages daily, once the metering data has been
collected for the day.
"""
# ------------------------------------------------------------ #
# Convenience methods for getting a client to the designated #
# participant. #
# ------------------------------------------------------------ #
def agr_client(self, recipient_domain, version="3.1.0") -> ShapeshifterDsoAgrClient:
"""
Retrieve a client object for sending messages to the AGR.
"""
return self._get_client(recipient_domain, "AGR", version)
def cro_client(self, recipient_domain, version="3.1.0") -> ShapeshifterDsoCroClient:
"""
Retrieve a client object for sending messages to the CRO.
"""
return self._get_client(recipient_domain, "CRO", version)