Add extracted tools: CitrineOS, OpenOCPP, ShapeShifter

- CitrineOS core extracted (CSMS OCPP 2.0.1)
- OpenOCPP extracted (firmware OCPP 1.6J/2.0.1)
- ShapeShifter library installed (pip install -e)
- ShapeShifter specification extracted
- EVerest extracted

TODO updated with progress
This commit is contained in:
Eric F
2026-06-08 00:38:27 -04:00
parent 468cfeaa50
commit d398a6ced2
7326 changed files with 1177561 additions and 7 deletions

View File

@@ -0,0 +1,121 @@
<!--
SPDX-FileCopyrightText: 2020-2023 Contributors to the Shapeshifter project
SPDX-License-Identifier: Apache-2.0
-->
# Contract phase
## Overall description
The contract phase is when DSOs and AGRs begin interaction.
Typically, this involves a pre-qualification process, where the AGRs capability to deliver flexibility, portfolio and IT systems are assessed.
AGR pre-qualification is out-of-scope for UFTP and therefore not further described in this document.
See [^B6] for more information.
In addition, the contract phase includes the exchange of information related to congestion points and associated connections through the common reference.
Note that any contract negotiations for bilateral contracts also takes place in the contract phase (see below).
[^B6]: USEF, "Recommended practices and key considerations for a regulatory framework and market design on explicit Demand Response," 2017. [Online]. Available: [https://www.usef.energy/app/uploads/2017/09/Recommended-practices-for-DR-market-design-2.pdf](https://www.usef.energy/app/uploads/2017/09/Recommended-practices-for-DR-market-design-2.pdf).
<figure markdown>
![General information flow in the Contract phase](../diagrams/contract-phase.puml){ .no-lightbox }
<figcaption>General information flow in the Contract phase</figcaption>
</figure>
As part of its internal grid planning process, a DSO determines where congestion may possibly take place.
Congestion points are declared whenever necessary, most likely several times a year, depending on the volume of trend analyses performed by a DSO and the condition of the grid.
By publishing a particular congestion point, the DSO opens a flexibility market for it.
AGRs publish the customers that they serve in the common reference.
As a result, a DSO can determine whether there is liquidity on its market and AGRs know which of their customers can participate in a DSOs market.
USEF introduces two alternative options for the DSO to procure flexibility from AGRs:
- **Long-term flexibility options:** activation of flexibility options in prearranged bilateral contracts.
Contracts of this nature oblige the AGR to offer a fixed amount of flexibility to the market via daily FlexOffers.
This product guarantees a certain regular availability of flexibility but the price (model) has to be arranged in advance.
As a result, the DSO can select the contract(s) that provide the required flexibility at the lowest costs although these, typically, will not reflect all flexibility available in the market or the actual marginal costs for any flexibility supplied.
This method is comparable to the way in which national TSOs contract balancing services and a tender or auction procedure can be implemented to manage it, alongside long-term flexibility options subject to prearranged specific conditions; e.g. which include a maximum or fixed price for the activation of the flexibility.
- **Short-term flexibility options:** procuring flexibility that AGRs have offered for a specific day.
In this situation, the AGR has no contractual obligation to offer the flexibility to the market but decides to do so on a day-to-day basis.
This product inherently deals with short-term flexibility which is only valid for a specific day.
The price offered by AGRs will better reflect the marginal costs although the availability of the flexibility in this product is not guaranteed.
This method is comparable with the way market parties are free to offer capacity to the TSO on the secondary and tertiary markets (regulating and reserve power) on a daily basis.
The first option seems attractive for the DSO as it guarantees a minimum supply of flexibility, therefore reducing availability risks in the validate phase, but this comes at a higher price.
The DSO is responsible for finding the right balance between costs and certainty, while ensuring that grid reliability is always maintained.
It is important to note that both products are part of the same flexibility market and therefore it provides natural growth options for the evolution of flexibility trading.
While, initially, the availability of flexibility supply and demand might be a primary concern for both buyers and sellers, leading them to focus on long-term contracts, this will likely change as the market matures and availability becomes less of an issue.
At that point, both buyers and sellers would be willing to have greater reliance on short-term flexibility which, in turn, would offer increased market liquidity.
## Bilateral contract: FlexOption
For long-term flexibility, USEF introduces the **FlexOption**. This type of contract enables the DSO to take an option on future flexibility. In relation to this, USEF specifies that:
- a FlexOption is a bilateral contract between DSO and AGR to deliver a specified amount of flexibility, at a specific location (i.e. congestion point), for a specified time schedule and duration.
- FlexOptions force the AGR to send flexibility offers when the flexibility requests match the contract details.
- FlexOptions are tradable via secondary markets.
!!! success "Recommended practice:"
In order to make FlexOptions attractive to AGRs, a DSO could add remuneration components; e.g. availability remuneration and/or a minimum number of activations.
A FlexOption contract typically has the following ingredients:
| Element | Description | Example(s) |
|-------------------------------|----------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------|
| ISPs and durations | The Imbalance Settlement Period (ISP) in which the flexibility should be delivered and the duration of the contract. | <ul><li>Every ISP between 17:00 and 19:00 on Fridays</li><li>from December through March for the coming two years</li></ul> |
| Lead time | Time before the (recurring) flexibility option expires. | <ul><li>One week before dispatch</li><li>The day before dispatch at 16:00</li><li>Does not expire</li></ul> |
| Congestion Point | DSO-specific: Congestion Point at which the flexibility should be realized. | |
| Amount | Amount of flexibility being offered, specified as a delta to a baseline value or drop to mechanism. | <ul><li>10kW (average power per ISP)</li></ul> |
| Capacity remuneration | Price the AGR receives for offering the flexibility option | <ul><li>€20/MW/day</li><li>€40/MW/ISP</li></ul> |
| Volume remuneration | Maximum or predefined price for the flexibility, to be paid when the flexibility is ordered by the DSO. | <ul><li>€20/MWh</li><li>Less than €20/MWh</li><li>€0</li></ul> |
| Maximum number of activations | The Maximum number of activations at which the flexibility in the LT contract can be called upon. | <ul><li>Once per contract</li><li>Once per month</li><li>Unlimited</li></ul> |
| Recovery Time | Minimum time between activiations | |
| Penalties | Amount the AGR is penalized for not offering the option of flexibility and/or the flexibility itself. | n/a |
UFTP does not include messages to communicate the contents of a FlexOption; this is assumed to be out-of-scope.
## Common Reference
A DSO declares its congestion points in the contract phase.
An AGR must be able to retrieve information about these points since this allows it to fulfil its obligation to provide the compulsory D-prognoses to DSOs for congestion points at which it represents customers.
It also aids the AGRs portfolio optimization activities.
Since a prosumer can switch AGRs, and the congestion points in the grid change over time, USEF introduces a common reference.
The common reference contains a list of connection identifiers (for example EANs) for each congestion point, as registered by the participating DSOs.
Each AGR also registers the connections on which it represents prosumers.
The common reference is shared between all involved parties, while respecting privacy and security principles [^B5] which ensures, amongst other things, that only necessary information is shared and therefore included in the common reference.
The common reference is operated by the Common Reference Operator (CRO) role.
Conceptually, it contains the entities and relationships shown in the figure below:
[^B5]: USEF Foundation, "USEF: The Privacy and Security Guideline," USEF Foundation, Arnhem, 2015. Available: [https://www.usef.energy/app/uploads/2016/12/USEF_PrivacySecurityGuideline_3nov15.pdf](https://www.usef.energy/app/uploads/2016/12/USEF_PrivacySecurityGuideline_3nov15.pdf)
<figure markdown>
![Logical structure of the Common Reference](../assets/images/image8.emf.odg.svg.png){ width=1000px }
<figcaption>Logical structure of the Common Reference</figcaption>
</figure>
The common reference is, typically, setup by the DSO and includes all of its congestion points.
However, it could be extended to form a common reference at national or European grid level.
UFTP allows for the registering and tracking of changes in the common reference; e.g. a DSO might announce a new congestion point for the following week which is registered in the common reference.
The history of all changes to the common reference is retained and can be retrieved for a certain date using the query function functions (see [AGRPortfolioQuery](../message-descriptions/message-catalog/agr-portfolio-query.md) and [DSOPortfolioUpdate](../message-descriptions/message-catalog/dso-portfolio-update.md)); this will also help resolve any potential disputes.
### Accessibility of the data in the common reference
Access to the common reference data is limited according to the legal or contractual requirements of the various parties consulting it[^3].
This implies that a DSO can only see the data for the connections it is responsible for.
Consequently, it can see which AGRs are serving its connections, in order to ensure that it receives D-prognoses from them.
AGRs can only see information about the congestion points for which they serve connections.
In addition, they can retrieve the number of connections per congestion point.
Implementing the common reference this way ensures that the DSO does not need to share details of its grid topology.
The common reference can be operated in two modes: open mode and closed mode.
If operating in open mode, the CRO will accept updates from any USEF-compliant participants.
In closed mode, participants will need to be pre-configured in order for updates to be accepted.
[^3]:
Another option is to treat the common reference as open data, therefore implying that everyone can access it.
Although the open data model can ease the implementation of the common reference, the acceptance of such a model by prosumers is highly dependent on the culture in the geographical area in which the model is used.

View File

@@ -0,0 +1,15 @@
<!--
SPDX-FileCopyrightText: 2020-2023 Contributors to the Shapeshifter project
SPDX-License-Identifier: Apache-2.0
-->
# General description
This chapter describes the global information flows for flexibility trading between AGR and DSO for grid contraint purposes.
In this respect, UFTP acts as a subset of USEFs market-based coordination mechanism (MCM), where AGR-DSO trading is an embedded part of the broader mechanism which applies to multiple flexibility services for all relevant market roles.
In order to understand how UFTP is embedded in USEF, we have used some of USEFs general concepts to describe the UFTP processes.
For further information on these, please see USEF: The Framework explained [^B1].
Sections describe the interactions in the different Phases of MCM: [Contract](contract-phase.md), [Plan](plan-phase.md), [Validate](validate-phase.md), [Operate](operate-phase.md) and [Settle](settle-phase.md).
[^B1]: USEF, "Flexibility Value Chain (update 2018)," 8 10 2018. [Online]. Available: [https://www.usef.energy/app/uploads/2018/11/USEF-White-paper-Flexibility-Value-Chain-2018-version-1.0_Oct18.pdf](https://www.usef.energy/app/uploads/2018/11/USEF-White-paper-Flexibility-Value-Chain-2018-version-1.0_Oct18.pdf).

View File

@@ -0,0 +1,39 @@
<!--
SPDX-FileCopyrightText: 2020-2023 Contributors to the Shapeshifter project
SPDX-License-Identifier: Apache-2.0
-->
# Operate phase
While (similar to the current liberalized energy market) the total energy system will remain in balance, without any congestion issues, as long as there are no deviations from the plans, it is unlikely that all plans will be executed exactly.
Deviations can arise from a range of sources, ranging from changing weather expectations to an extension of a football match.
Deviations can lead to: imbalances in supply and demand of energy at total system level (affecting the BRP); changes within the AGR portfolio (affecting the AGR); and local congestion in the distribution system (affecting the DSO).
USEFs MCM is designed such that, during the operate phase, additional flexibility can be applied in order to compensate for these deviations.
The processes for invoking this flexibility are depicted in the following figure.
<!-- TODO: check why this image has been shortened in the .pdf file -->
<figure markdown>
![General information flows in the Operate phase.](../diagrams/operate-phase.puml){ .no-lightbox }
<figcaption>General information flows in the Operate phase.</figcaption>
</figure>
During the operate phase, the AGRs main goal is to adhere to its plan and respect the D-prognoses.[^12]
In order to achieve this, FlexOfferRevocation the AGR must first plan to ensure assets operate in accordance with its forecast portfolio performance and that any flexibility sold is actually delivered.
[^12]: UFTP allows for alternative baselines, see [General description](index.md). In case of alternative baselines, D-prognosis is optional.
Next, the AGR measures the net demand or supply of its portfolio, in order to detect deviations from its plan and D-prognoses.
Where deviations occur, the AGR re-optimizes its portfolio.
It is possible that deviations could be solved using the flexibility contained within the portfolio itself.
If this is not the case, the AGR must change its plan (and probably limit its liabilities due to non-performance, to minimize fines) and control the assets to ensure that the new plans are realized and send an updated D-prognosis.
If a FlexOrder has already been accepted, but the AGR is no longer able to comply, the AGR should inform the DSO as soon as possible through direct means outside of the UFTP protocol.
During the operate phase, the AGR may wish to revoke outstanding (i.e. not ordered) FlexOffers if it is no longer able or willing to deliver the flexibility.
Although the DSO will reduce congestion risks in the validate phase, the DSO can still request that AGRs dispatch flexible power options to resolve potential grid problems in the operate phase.
Again, due to time constraints, this will be done based on flexibility offers from the validate phase, which are still valid.
When this flexibility is used, the corresponding BRP portfolio will no longer be in balance.
Where the AGR is responsible for the balance, it will most likely charge the additional costs to the DSO.
This is not considered further.

View File

@@ -0,0 +1,31 @@
<!--
SPDX-FileCopyrightText: 2020-2023 Contributors to the Shapeshifter project
SPDX-License-Identifier: Apache-2.0
-->
# Plan phase
USEFs plan phase aims to find an economically optimized program to supply the energy demand of both AGR and BRP portfolios for a certain period.
In this phase, the AGR optimizes its portfolio and typically arbitrages between different flexibility markets to maximize the value of the available flexibility.
The result of this process is a balanced portfolio.
The exchange between AGR and BRPs and corresponding trades on the energy markets are out-of-scope for UFTP.
The result, however, will lead to a D-prognosis in the validate phase.
The processes that take place during the plan phase are schematically depicted in the following figure.
<figure markdown>
![General information flow in the Plan phase](../diagrams/plan-phase.puml){ .no-lightbox }
<figcaption>General information flow in the Plan phase</figcaption>
</figure>
Since the list of connections belonging to a congestion point, and the list of customers that are served by the AGR, may switch from day to day, USEF specifies that this information is requested on a day-to-day basis.
The AGRs portfolio optimization is out-of-scope and depicted only for reference.
See [^3] for a detailed description of this process.
For UFTP, it is important to realize that the AGR optimizes its portfolio based on its clients needs, optionally taking into account its long-term contractual obligations.
For bilateral contracts, the DSO might provide a FlexReservationUpdate message e.g. signaling which part of the contracted volume is still reserved and which part is not needed and may be used for other purposes.
This will typically re-trigger the AGRs portfolio optimization process. More information about the flex reservation mechanism is added in [Rationale for information exchange in flexibility request](../appendix/rationale-for-information-exchange-in-flexibility-request.md).
[^3]: USEF, "USEF - The FrameWork Specifications 2015," 2015. [Online].

View File

@@ -0,0 +1,137 @@
<!--
SPDX-FileCopyrightText: 2020-2023 Contributors to the Shapeshifter project
SPDX-License-Identifier: Apache-2.0
-->
# Settle phase
## Overall description
In the settle phase, the AGR is remunerated for the delivered flexibility.
There should be a check first, to ensure that the procured flexibility was actually delivered.
A penalty may be applied when there is a mismatch between the baseline and the actual profile.
Except for metering data, USEF assumes that both AGR and DSO have all the information to make these calculations.
The DSO sends its calculation to the AGR for verification.
<figure markdown>
![General information flow flex settlement between DSO and AGR](../diagrams/settle-phase.puml){ .no-lightbox }
<figcaption>General information flow flex settlement between DSO and AGR</figcaption>
</figure>
In order to perform the settlement, it is necessary for the DSO to compare the metering for each ISP against the baseline, as contained in the demand-prognosis message.
In some regions the metering for the ISP periods for the fiscal meter may already be available to the DSO, through its own systems.
However, where the DSO does not have access to the meter data, or where plant metering is used for settlement assessment, the metering data must be communicated to the DSO by the AGR.
For this purpose the AGR will use the Metering messages to send the daily metering per ISP from each connection point.
**Recommended Practice:** It is recommended to the include the Power profile in the settlement messages.
Each month, the DSO will calculate the flexibility settlement volumes and prices for each AGR according to the steps below.
1. **Calculate procured flexibility**</br>
For each congestion point and ISP, both the volumes and prices of the acquired flexibility are collected.
2. **Gather Metering Data**</br>
This is done either based on the DSOs own metering records or using the metering message
3. **Validate if baselines are met**</br>
This step is only performed where flexibility has been procured from the specific AGR, for a congestion point and ISP.
The deviation between the final forecast (taking into account all procured flexibility) and the realized profile is calculated.
4. **Calculate flex settlement volumes and prices**</br>
The settlement volumes and prices are calculated (see example below).
5. **Send settlement message to AGR**</br>
This shows, per month, the accumulated settlement volume, accumulated deviation from the agreed baselines, and accumulated settlement price (accumulated over the ISPs and the congestion points).
The AGR will use its own information to do the same calculations
1. **Calculate sold flexibility** (equivalent to 1)
2. **Validate if baselines are met** (equivalent to 2)
3. **Plausibility check** (equivalent to 3)</br>
The results are compared to the settlement message received from the DSO.
If the difference in the previous plausibility check is within limits, the AGR sends an acknowledgement to the DSO.
If the difference exceeds predefined limits, the DSOs calculation is disputed.
This is signaled via the SettlementResponseMessage.
The dispute process is not part of USEF.
## Settlement details
The DSO is responsible for settling the flexibility that it has acquired from the AGR.
Within this settlement, the DSO needs to check whether the acquired flexibility has been delivered according to the agreements.
Penalties can arise where agreements have not been met and this is considered an integral part of the settlement process.
This section describes how the supply of flexibility from the AGR to the DSO is settled.
Several methods are possible, depending on how the flexibility is offered to the DSO and, more precisely, whether the DSO uses the mechanism of long-term and short- term flex options, and whether these flex options are rewarded, even if they are not exercised.
Passive contribution to constraint management may also be rewarded.
Calculations are (in general) performed at ISP level, with settlement calculated over a one-month period.
### Settlement components
The settlement consists of five components:
1. **Settlement of acquired flexibility**</br>
The DSO will acquire flexibility through the market mechanism if congestion is expected, either ahead of time (validate phase) or real-time (operate phase).
On average, the market price is determined by the equilibrium of demanded and offered flexibility (merit order).
Following the recommendation given in Section [Flexibility trading between the AGR and DSO](validate-phase.md#flexibility-trading-between-the-agr-and-dso), pay-as-bid pricing is used for individual settlements.
2. **Settlement of deviations from baselines**</br>
AGRs that have sold flexibility to the DSO need to limit their capacity to the value stated in the baseline, corrected for the sold flexibility.
A penalty is raised for each ISP where the agreed capacity has (on average) been exceeded.
The penalty is single sided, meaning that the AGR is allowed to deviate from its baseline as long as the deviation contributes to avoiding the grid constraint.
3. **[optional] Settlement of long-term flexibility options (bilateral contracts)**</br>
The DSO may acquire long-term flexibility options to ensure that congestion can be avoided at all times.
This may be necessary to justify a decision to delay or defer grid investments, or where grid constraints are expected in the (near) future.
The DSO will determine (e.g. through audits) that the AGR is always capable of providing the contracted flexibility (at contracted times).
4. **[optional] Settlement of short-term flexibility options (FlexOffers which are not procured)**</br>
When grid constraints are expected during the validate phase by the DSO, all AGRs are asked to place flexibility offers.
AGRs whose flexibility bids are accepted by the DSO are expected to meet their (adjusted) baselines.
However, it is just as important for the DSO that the other AGRs are also bound to their baselines.
To enforce and reward this, the DSO may also accept an option on the flexibility offered by the AGR, even if the bid is not accepted, to ensure the AGR will not jeopardize grid constraints by invoking its flexibility for other purposes.
The DSO will pay the AGR for the option, as this will put a constraint on the total capacity of the AGRs portfolio.
5. **[Optional] Settlement of passive contribution to constraint management**</br>
Possible reward for AGRs that passively support the management of grid constraints either by staying below the agreed value expressed in their baselines or by reducing more power than agreed in the exercised flexibility bid.
This is specifically relevant if constraints have been resolved by passive contributions, where other AGRs have exceeded the agreed capacities.
In this situation, the passive contribution can be rewarded using the raised penalties (zero sum calculations).
The settlement method implicitly assumes that each AGR has the motivation and capacity to produce high quality baselines.
### Settlement calculations
(numbers refer to the settlement components above)
| Nr. | Amount | P = | Q = | Remark(s) |
|-------|------------|---------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| **1** | P x Q | P = flexiblity fee | Q = power called ("Delivered Flex Quantity") | The flexibility fee is determined by the equilibrium of demanded and offered flexibility (flexibility market). |
| **2** | P x Q | P = excess penalty (asymmetrical, only applied when deviation has contributed to congestion) | Q = allocation initial baseline Σ FlexOrders ("Power Deficiency Quantity"), where allocation is the average realized power during the ISP | This penalty only applies to AGRs that have sold flexibility (for this ISP), and corresponds with not meeting the agreements of the flexibility bid. |
| **3** | negotiated | n.a. | n.a. | The DSO will pay compensation for the long-term flexibility contract in proportion to the agreed power, the duration of the time slot, the nomination lead-time and the duration of the contract. Fixed fee may be derived from:<ul><li>The costs for allowing thermal limitations to be violated;</li><li>The avoided costs for grid reinforcements;</li><li>The value of flexibility for alternative uses;</li><li>The costs of alternative flexibility options.</li></ul> |
| **4** | P x Q | P = flexibility option fee | Q = power option | In this settlement component, an AGR can be rewarded for participating in the flexibility market through short-term products. Rationale for relating the flexibility option settlement to the capacity offered is that an option to cap the capacity of an AGR with twice the capacity of another AGR, has twice the value to the DSO. |
| **5** | P x Q | P = contribution fee (asymmetrical, only applied when deviation has relieved congestion). Derived from excess penalties raised. | Q = allocation - baselines power flexibility called during Operate phase where allocation is the average realized power during the ISP. | |
### Settlement example
For a specific ISP we assume:
- The AGR has specified a total load of 10 MW in its initial baseline
- During one or more flexibility requests, a total of 2 MW of flexibility has been acquired by the DSO, yielding an adjusted baseline of 8 MW
- The table below shows how different realizations are settled. The realization is specified by the allocation, representing the average power during one specific ISP for one congestion point. In this example the following (illustrative) prices are applied:
- Flex price equals 7 € / MW
- Baseline deviation penalty equals 11 € / MW (single sided)
| Allocation (MW) | Flex realized (MW) | Delivered flex quantity (MW) | Flex paid (€) | baseline deviation (MW) | Power deficiency quantity (MW) | Penalty raised (€) | Settlement (€) |
|-----------------|--------------------|------------------------------|---------------|-------------------------|--------------------------------|--------------------|----------------|
| 7 | 3 | 2 | 14 | -1 | 0 | 0 | 14 |
| 8 | 2 | 2 | 14 | 0 | 0 | 0 | 14 |
| 9 | 1 | 1 | 7 | 1 | 1 | -11 | -4 |
| 10 | 0 | 0 | 0 | 2 | 2 | -22 | -22 |
| 11 | -1 | 0 | 0 | 3 | 3 | -33 | -33 |
Elaboration of the calculations performed in the table:
1. _Allocation_ shows possible results of the realization of the AGR portfolio
2. The _flex realized_ equals the baseline associated with the flex offer minus the allocation
3. The delivered _flex quantity_, represents the flexibility that is both acquired (therefore maximum of 2) and delivered.
Passive contributions (as in the row with allocation=7) are not rewarded.
4. The _flex paid_ equals the _delivered flex quantity_ times the flex price
5. The _baseline deviation_ equals the allocation minus the adjusted baseline
6. The _power deficiency quantity_ equals the _baseline deviation_, where negative deviations are set to 0 since these are not penalized.
7. The _penalty raised_ equals the _power deficiency quantity_ times the penalty
8. The settlement price equals the sum of the _flex paid_ and _penalties raised_.
9. The monthly totals of the elements: _delivered flex quantity, power deficiency quantity and Settlement_ are sent to the AGR.

View File

@@ -0,0 +1,101 @@
<!--
SPDX-FileCopyrightText: 2020-2023 Contributors to the Shapeshifter project
SPDX-License-Identifier: Apache-2.0
-->
# USEF general concepts
## USEF market-based coordination mechanism (MCM)
USEFs MCM facilitates the delivery of value propositions (i.e. marketable services) to various market parties without imposing
limitations on the diversity and customization of the propositions.
The USEF MCM is designed for all energy commodities and enables the market to optimize in time, capacity and power.
The MCM provides access, under equal conditions, for all stakeholders to a single integrated market.
This unique approach aims to deliver a future-proof market design. The USEF MCM operations scheme distinguishes five phases:
![](../assets/images/image5.png)
In UFTP, the general MCM phases are followed. However, processes in each phase are limited to the interactions between the
AGR and DSO for flexibility trading. This includes interaction with the Common Reference:
- **Contract:** AGR and DSO negotiate FlexOptions.
This is flexibility that is reserved for DSO purposes and can be invoked by the DSO when needed.
Typically, a contract includes availability remunerations and activation remunerations.
- **Plan:** information exchange between DSO and AGRs related to congestion points.
This information exchange through the Common Reference involves communication with the Common Reference Operator (CRO).
The AGR carries out an initial portfolio optimization.
- **Validate:** the DSO uses D-prognoses to validate whether the demand and supply of energy can be distributed safely without any limitations.
If congestion occurs, the DSO can procure flexibility from AGRs to resolve grid capacity issues.
- **Operate:** in the operate phase, the actual assets and appliances are dispatched and the AGR adheres to its D-prognoses.
When required, DSOs can invoke additional flexibility from AGRs to resolve unexpected congestion.
- **Settle:** in the settle phase, the flexibility that the AGR has sold to DSOs is settled.
For this purpose, the actual consumed and produced volumes are allocated to the responsible parties first.
Any unresolved or disputed volumes are reconciled shortly afterwards.
## USEF Operating regimes
USEF recognizes four different operating regimes.
The green regime is the classical grid without any limitations (copper plate) and the red regime is where power outages occur.
USEF introduces two additional regimes. In the green and yellow regimes, the MCM assures optimal use of the flexibility available for BRPs (green and yellow) and DSOs (yellow).
The orange regime is introduced as a fallback for situations where there is insufficient flexibility available for the DSO to avoid an outage—the DSO can limit connections to temporarily overrule the market.
<figure markdown>
![USEF Operating regimes](../assets/images/image6.png)
<figcaption>USEF Operating regimes</figcaption>
</figure>
UFTP flexibility trading takes place solely in the yellow regime. Full descriptions of all other regimes can be found in [^B3].
[^B3]: USEF, "USEF - The FrameWork Specifications 2015," 2015. [Online].
## Day-head and intraday flexiblity trading
The plan and validate phases aim to make optimal use of flexibility and maximize the freedom of dispatch and transactions of all stakeholders before the actual delivery of energy takes place.
The timescale of these phases already ranges from years and months ahead, towards one day ahead and, ultimately, hours ahead before the actual delivery of energy (i.e. the operate phase) starts.
This broad time window supports trading on various energy markets (i.e. forward market, day-ahead spot market and intraday spot-market) and the monitoring of changes in the required grid capacity.
USEF specifies that iterations between the plan and validate phases take place at least twice: initially, during the day ahead and, secondly, during intraday.
Hence, two points in time are determined when the markets close: 1) end of day-ahead 2) end of intraday.
This specification eases the iterative process of creating, aligning and converging plans and D-prognoses and fits well with many current national processes applied in the day ahead and intraday trading market.
Please find more details in [Iterations between the plan and validate phases](validate-phase.md#iterations-between-the-plan-and-validate-phases).
UFTP supports both day-ahead and intraday trading. Intraday trading is optional for UFTP.
## Congestions points
A congestion point is a set of connections which (directly) relate to a part of the grid where grid capacity might be exceeded because it may be insufficient to distribute the requested amount of energy; e.g. the secondary side of an LV transformer.
Please note that a congestion point is a part of the grid where congestion may possibly occur, rather than actually occurs.
It is the DSOs responsibility to determine congestion points.
!!! success "Recommended practice:"
How to determine which grid points are congestion points is the DSOs responsibility and therefore outside of the scope of USEF.
However, USEF recommends declaring congestion points at the lowest possible level in the grid as this allows for detailed insight about local congestion while simultaneously, through aggregation, safeguarding the reliability of the grid safety analysis.
Congestion points are registered in the common reference, see [Common Reference](contract-phase.md#common-reference).
## Flexiblity market time granularity
All flexibility exchange in USEF is based on a time granularity that is in line with the imbalance settlement period (ISP)[^2], which is the time unit for which the imbalance of the balance responsible parties is calculated.
For most EU countries this is a 15-minute interval, leading to 96 ISPs per day.
Forecasts, flexibility offers, and orders and settlement are all based on ISPs.
Flexibility is expressed in Power [Watts] which is to be interpreted as the average Power during an ISP.
[^2]: Commission Regulation (EU) 2017/2195 of 23 November 2017 establishing a guideline on electricity balancing defines an imbalance settlement period (ISP) as the time unit for which the imbalance of the balance responsible parties is calculated (Article 2(10)).
## Alternative baselines
All flexibility trading in USEF assumes that flexibility offers take the form of deviation from baseline, i.e. the default situation that would occur if no flexibility were activated.
Typically, the baseline is derived via an AGR nomination referred to as the D-prognosis.
However, UFTP allows for alternative baseline, where a D-prognosis is not required, and it is assumed that the AGR and DSO agree upon the baseline.
An alternative baseline could be based on a measurement (MBMA-method), a mathematical formula or a reference group, etc.
The choice of the alternative baseline and its establishment is out-of-scope for UFTP.
The UFTP trading messages either refer to D-prognosis or to an external baseline reference.
Throughout this document, the D-prognosis is used as the default baseline method.
If UFTP is used with alternative baseline, then all references to the D-prognosis can be ignored.

View File

@@ -0,0 +1,281 @@
<!--
SPDX-FileCopyrightText: 2020-2023 Contributors to the Shapeshifter project
SPDX-License-Identifier: Apache-2.0
-->
# Validate phase
## Overall description
<figure markdown>
![Validate phase](../diagrams/validate-phase.puml){ .no-lightbox }
<figcaption>General information flow in Validate phase</figcaption>
</figure>
Assuming that D-prognoses are used as baseline[^4], each AGR creates a D-prognosis per declared congestion point and sends it to the DSO.
Sending D-prognoses is not required when an alternative baseline method has been agreed.
When all D-prognoses for a congestion point are received, the DSO combines these with the forecasts of non-AGR connections to execute a final grid safety analysis.
This analysis determines whether it is possible to distribute the planned energy, or the limits of the distribution grid are reached.
In the latter situation, USEF moves to the yellow regime and the DSO procures flexibility in the market to resolve these congestion issues.
The process of flexibility trading is as follows:
[^4]:
UFTP allows for alternative baselines, see [General description](index.md).
In case of alternative baselines, D-prognosis is optional.
1. The DSO requests all AGRs at the congestion point to provide flexibility.
In this request, the DSO indicates the magnitude (amount of excess power) and timing (ISP) of the expected congestion, and how much capacity is available in the remaining ISPs. This is handled via a _FlexRequest_.
2. AGRs who are able and willing to change their load/generation profile in line with the request create _FlexOffers_.
3. If the offers are a good fit; i.e. help to mitigate congestion, the DSO can procure flexibility by placing a _FlexOrder_.
4. AGRs send an updated D-prognosis which incorporates the flexibility ordered.
If the flexibility offered is not sufficient to resolve the expected congestion, or no flexibility is offered, USEF moves to the orange regime.
Please note that in this situation, USEF specifies that the DSO procures all offered flexibility from the market (i.e. from the yellow regime) to solve parts of the congestion.
This is because any flexibility offered will still help to reduce the impact of the orange regime (e.g. the number of connections impacted by the graceful degradation).
USEFs orange regime is outside the scope of UFTP, and, therefore, not further discussed in this document.
<figure markdown>
![Validate phase with multiple FlexOffers](../diagrams/validate-phase-multiple-flexoffers.puml){ .no-lightbox }
<figcaption>AGR may send multiple FlexOffers on the same request, and open FlexOffers, which are not yet ordered may be revoked by the AGR.</figcaption>
</figure>
AGRs may respond with one or more FlexOffers.
Where an AGR responds with multiple FlexOffers, the DSO can freely choose the most appropriate offer(s).
Provided the flexibility has not been ordered via a FlexOrder, a FlexOffer may be revoked by the AGR (FlexOfferRevocation).
Alternatively, if there is an existing agreement between the DSO and AGR, the DSO can procure the flexibility by directly placing a _FlexOrder_, without sending a _FlexRequest_ first:
<figure markdown>
![Direct FlexOrder without FlexRequest](../diagrams/../diagrams/validate-phase-direct-flexorder.puml){ .no-lightbox }
<figcaption>If there is an existing agreement, DSO can procure flexibility by sending a FlexOrder directly, without FlexRequest</figcaption>
</figure>
Flexibility trading may go on until gate closure time:
<figure markdown>
![Flexibility trading may go on until gate closure time](../diagrams/../diagrams/validate-phase-gate-closure-time.puml){ .no-lightbox }
<figcaption>Flexibility trading may go on until gate closure time. This includes iterations between plan phase and validate phase. The last accepted D-prognosis before gate closure time serves as a basis for the next phase. Any open FlexOffers, which are not expired, may be ordered after gate closure.</figcaption>
</figure>
## Exchange of D-prognoses between AGR and DSO
When using D-prognoses as the baseline methodology, an AGR sends its D-prognosis per congestion point to the DSO.
The DSO assesses the validity of each received D-prognosis and informs the AGR whether or not it is accepted.
As long as the gate closure time has not passed, the AGR can update its D-prognoses and resend it to the DSO.
During these iterations, an accepted D-prognosis[^5] is assumed actual, provided no associated updates are received and accepted by the DSO.
For the exchange of D-Prognoses, USEF specifies the following:
- An AGR is obliged to create a D-prognosis for its connections as soon as its connection point is declared by a DSO.
The AGR is informed of this congestion point via the mandatory (daily) CRO queries.
- The D-prognosis only includes the forecasted load of those prosumers served by the AGR and that have a connection related to a congestion point.
- The D-Prognosis contains one full calendar day for the day-ahead process.
In the intraday process, all remaining ISPs for that calendar day must be included
- The average power is given per ISP as a minimum time granularity
- The D-Prognosis does not include additional information; e.g. details about the available flexibility or prices
!!! success "USEF provides the following recommended practices for the exchange of D-Prognoses:"
- The D-Prognosis is sent day-ahead to the DSO; at the latest, two hours before the gate closure of the national balancing regime (day-ahead gate closure)[^6]
- The DSO can perform a monthly evaluation of the accuracy and reliability of D-prognoses in order to motivate AGRs to improve the D-prognoses they supply
[^5]: The accepted D-prognosis is the D-prognosis that used as a basis for the next period (Intraday) or phase (operate).
[^6]: For the difference between day-ahead and intraday, please refer to [Iterations between the plan and validate phases](validate-phase.md#iterations-between-the-plan-and-validate-phases)
## Processing D-prognoses
When all D-prognoses relating to a particular congestion point are received, the DSO creates a total prognosis for each congestion point.
When accumulating the D-prognoses, the DSO must estimate and add the distribution requirements required for those connections which are not served by an AGR, and, therefore, not included in the D-prognoses.
Using the accumulated D-prognosis, the DSO can determine whether it is possible to distribute the planned energy, or whether the limits of the distribution grid will be reached.
The timing of the FlexRequest is determined by the DSO, who must make a trade-off between waiting for all AGRs to send their D-prognoses[^7] before identifying grid constraints or issuing a FlexRequest based on its own grid forecasting process.
It is the DSOs responsibility to combine the two sources of information.
Where alternative baseline methods are used, the DSO will issue a FlexRequest based solely on its own information.
[^7]:
USEF 2015 specified that the DSO should wait for all D-prognoses to be submitted.
This would allow for a more precise determination of possible congestion and also ensure a fair process of offering and procuring flexibility.
However, this constraint hinders the process in situations where there are missing or late D-prognoses.
The grid safety analysis can lead to either of the following conclusions:
- That no congestion is expected. All power flows are within calculated safety margins.
- That congestion is expected as the energy flows exceed calculated safety margins. The DSO then starts the process of acquiring flexibility from the market.
Where no AGRs are active at the congestion point, there is no market to supply the DSO with flexibility to reduce its energy flows to levels below the safety margins.
In this situation, the system moves to the orange regime.
The resulting DSO processes are outside the scope of this document.
See [^B4] for a global description.
## Flexibility trading between the AGR and DSO
If the outcome of the grid safety analysis is that congestion will arise, the system is moved to the yellow regime and the DSO will need to take action.
In order to solve the expected congestion, the following steps are taken:
1. The DSO requests all AGRs at the congestion point to provide flexibility.
In its request, the DSO indicates the magnitude (amount of excess power) and timing (ISP) of the expected congestion and (optionally) how much capacity is available in the remaining ISPs
2. The AGRs receive the flexibility request for adjusting[^8] distribution requirements
3. AGRs generate offers for the flexibility. Note that these offers could either be the result of a voluntary offering process or compulsory, where there are prearranged bilateral contracts between DSO and AGR (FlexOptions)
4. The DSO receives the flexibility offers
5. The DSO procures flexibility to resolve the congestion issues by placing flexibility orders.
6. The DSO determines whether the expected congestion is solved using the ordered flexibility.
If this is not the case, the system moves to the orange regime (see [^B4] for more details)
7. The AGRs receive the flexibility orders which result in the actual procurement of flexibility by the DSO.
8. AGRs provide an (updated) D-prognosis, incorporating the ordered flexibility.
This step only applies when D-prognoses are used as baseline methodology.
[^B4]: USEF Foundation, "USEF: The Framework Explained," USEF Foundation, Arnhem, 2015.
[^8]: Either by decreasing energy use, increasing local generation, or time-shifting load.
When trading via a market platform AGRs typically offer their flexibility without an underlying request (unsolicited flex offers) and the DSO selects appropriate offers.
Steps 1 and 2 are not applicable in this situation.
- Information about expected congestion is provided to all registered AGRs;
- Expected congestion information includes the direction of the overload (production/consumption);
- Expected congestion information includes the amount of reduction needed;
- Expected congestion information includes available grid capacity for other ISPs.
See [Rationale for information exchange in flexibility request](../appendix/rationale-for-information-exchange-in-flexibility-request.md) for more details about this information exchange including rationale.
In the [FlexRequest messages](../message-descriptions/message-catalog/flex-request.md), there is no single power value, rather a power space for each ISP, bound by two power values (“MinPower” and “MaxPower”).
In addition, there is a distinction between ISPs with a requested disposition meaning that there is a request for a deviation on the power consumption/production and ISPs with an available disposition meaning that there is available space to deviate on the power consumption/production.
<figure markdown>
![Example forecast for a Congestion point, leading to an example FlexRequest, both displayed in graphs](../assets/images/image13.svg.png){ width=1000px }
<figcaption>Example forecast for a Congestion point, leading to an example FlexRequest, both displayed in graphs</figcaption>
</figure>
In the example used in the figure above, congestion is expected at ISP 2 at a certain congestion point.
The FlexRequest is transmitted to all AGRs that have prosumers on the congestion point, requesting a decrease of consumption (or an increase of production) on ISP 2.
If AGRs choose to shift consumption from ISP 2 to another ISP, the available spaces in the ISPs from the FlexRequest indicate that there is more room for consumption in ISP 5 than in ISP 1, 3 and 4.
A flexibility offer with a consumption shift to ISP 5 is therefore more likely to be ordered.
Note that a request is always relative to the agreed baseline.
The essence of broadcasting the same request to all AGRs is that the requested deviation is the same for each AGR, relative to its own baseline.
Decreasing power consumption is thereby equivalent to increasing power production and vice versa.
The distribution of the FlexRequest message is limited to AGRs which are registered via the CRO (see [Plan phase](plan-phase.md)).
The information on direction, amount and available capacity is provided in specific data structures that are part of the FlexRequest message (see [Power](../message-descriptions/message-catalog/power.md)).
For the procurement of flexibility by the DSOs, USEF specifies the following rules:
- A flexibility offer is valid until it expires or is revoked.
It is the AGRs responsibility to determine the acceptance deadline.
Revocation is not possible after the offer has been ordered.
- A flexibility order is definite and binding once it has been placed.
- The DSO chooses which flexibility offers it accepts to solve the congestion.
In this regard, it is not obligatory for the DSO to start with the offer that has the lowest price.
The DSO has the freedom to assess the balance between the price and quality[^9] of the flexibility offered in both long-term contracts and short-term offers.
- Bidding takes place at congestion point level, making every congestion point a local flexibility market.
- The DSO sends a flexibility request when the result of the grid safety analysis indicates possible congestion.
- A flexibility order is linked to a flexibility offer and a baseline for settlement purposes.
[^9]: For example, it is desirable for the DSO to procure flexibility that moves energy use from a moment in time of high load to a moment in time of very low load, rather than procuring flexibility that moves energy use from a moment in of high load to a moment in time at which the maximum of the distribution capacity is almost reached.
UFTP has two optional structures in the trading messages:
- [optional] A FlexOffer can contain multiple mutually exclusive options.
The DSO is able to pick one of the options in its FlexOrder.
Whether or not an AGR is allowed to add multiple options is signaled in the CRO.
- [optional] AGRs may indicate the capability for partial activation in their offers.
In all messages, a minimum activation percentage is provided i.e. all values lower than 100% mean that partial activation is possible.
The DSO could use partial activation to better match the requested flexibility with available offers.
Partial activation applies to all ISPs in the offer and also to the offered price.
For example, if the DSO procures 80% of the offer, all ISPs in the FlexOrder should be 80% of the corresponding ISPs in the FlexOffer, and the settlement price is set to 80% of the original price in the offer.
If the DSO has not implemented the option, it may simply ignore the structure.
### Updates after flex has been ordered
USEF specifies that when ISPs are altered as a result of a flexibility deal, the resulting load for these ISPs is used for settlement.
This means that deviation on ISPs with requested disposition that have been traded, might lead to penalties for the associated AGR.
Updates on other ISPs are always possible for the AGR but will require a new D-prognosis.
Other AGRs can still change their D-prognoses for all ISPs without incurring a penalty.
More details about the penalty calculation can be found in the [settlement](settle-phase.md).
The load profile resulting from a flexibility deal will be stored independently from any subsequent updates of the D-prognosis, because it allows the DSO to verify the actual delivery of the flexibility for settlement purposes.
!!! success "Recommended Practice for flexibility trading:"
For the procurement of flexibility by the DSOs, USEF recommends the following:
- Publishing is in an anonymized way
- Pay-as-bid is used for pay-out.
- Timing of the flexibility orders is between 24 and 2 hours before delivery (day-ahead trading).
## Iterations between the plan and validate phases
As a result of flexibility exchange with the DSO, the AGRs portfolio is not balanced and so it may wish to re-optimize it.
By doing so, the MCM returns to the [plan phase](plan-phase.md), the only difference being that the AGR now takes the flexibility it has already sold into account.
In addition, during the plan phase, the AGR has the freedom to continuously re-optimize its portfolio whenever deemed necessary (e.g. when new and improved forecasts are available).
The rationale behind this high degree of freedom for the AGRs is to ensure that the flexibility they offer is applied in the most optimal and cost-effective way (i.e. flexibility is used there where it creates most value).
However, these iterative processes also lead to the issue that D-prognoses change continuously, and continuous alignment is required between AGR/DSO.
As a result, there is a risk that the iterative process between the plan and validate phases does not converge.
In conclusion, there is a balance to be struck between making sure that the iterative processes converge, and offering a high degree of freedom to ensure that the available flexibility is used in the most optimal and cost-effective way.
### Gate closure times
For reasons mentioned above, USEF specifies that there needs to be at least two moments in time when the iterations are forced to end. This moment is called the gate closure time.
In alignment with the current practices in liberalized energy markets, the two gate closure times are:
1. Day-ahead closure time: the moment at which the day-ahead market closes.
During the day-ahead period, the plan and validate phases can be iterated as often as needed, as long as the system converges before the gate closure time.
After the day-ahead closure time, the processes in the plan phase can only be restarted during the intraday period.
2. Intraday closure time[^10]: the moment at which the intraday market closes.
During the intraday period, the plan and validate phases can be iterated as often as needed, as long as the system converges before the gate closure time.
After the intraday closure time, the processes in the plan phase cannot be restarted at all, as the MCM moves to the operate phase.
[^10]: Note that intraday flexibility trading is optional in UFTP so intraday gate closure applies only when intraday trading is enabled by the DSO
<figure markdown>
![Example of USEF day-ahead gate closure time.](../assets/images/image14.emf.odg.svg.png){ width=1000px }
<figcaption>Example of USEF day-ahead gate closure time.</figcaption>
</figure>
An example of the day-ahead gate closure is given in the figure above.
Please note that while there is a single day-ahead gate closure time per day, there will be multiple intra-day gate closures.
This is illustrated in the figure below.
The ISP corresponding to the current time is in operate phase and ISPs more than one hour ahead are still in plan or validate phase.
<figure markdown>
![Example of USEF intra-day gate closure times.](../assets/images/image15.emf.odg.svg.png){ width=1000px }
<figcaption>Example of USEF intra-day gate closure times.</figcaption>
</figure>
!!! success "Recommended Practice:"
The national regulator determines the specific timing details of the gate closures. USEF recommends:
- A day-ahead gate closure time of 2 hours before the gate closure of the national balancing market
- An intraday gate closure time of one hour before the start of the ISP
## Balance responsibility and redispatch options
The procurement of flexibility by the DSO may impact the AGRs balance position (nomination)[^11].
Therefore, USEFs validate phase is iterative with the plan phase, i.e. the AGR can make adjustments to its nomination via energy exchange with its associated BRP and/or energy trades on the energy markets.
When the gate closure time is reached, all issues need to be resolved or there will be in an imbalance in the AGRs portfolio (which is the AGRs responsibility).
[^11]: his is not the case when the AGR can manage the deviation itself, e.g. one prosumer uses more energy, another prosumer uses less energy, and these two level each other out.
As an alternative, the procurement of flexibility is organized as an energy trade between AGR and DSO.
In this situation, the AGR portfolio stays balanced and there is no need to iterate to the plan phase.
Nevertheless, the AGR is free to re-optimize its portfolio at any time, for any reason, which could induce iterations.
The DSO faces imbalance as a result of the trade and is responsible for this.
The DSOs imbalance can be restored via a counter trade outside the congested area (redispatch).
USEF supports both ways of balance responsibility.
The DSO determines the method of trading for a given congestion point and signals this via the CRO. See [DSOPortfolioUpdate](../message-descriptions/message-catalog/dso-portfolio-update.md).
!!! success "Recommended practices for balance responsibility:"
- USEF recommends that the AGR retains balance responsibility for day-ahead flexibility trading.
Any imbalance risk as a result of DSO trades can be mitigated via day-ahead energy trades.
- USEF recommends shifting the balance responsibility to the DSO for intraday flexibility trading.
This is equivalent to an energy trade from AGR to DSO.
In order to neutralize its position, the DSO should then trigger a counter-trade (redispatch).