eric/cariflex
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add FlexMeasures plugins, USEF protocol, and Cariflex simulator

Browse Source 
- flexmeasures-entsoe: ENTSO-E data plugin
- flexmeasures-weather: Weather data plugin
- USEF Flex Trading Protocol PDF (2.4MB)
- Cariflex simulator (publishes to Redis)
- Dashboard Grafana updated with correct InfluxDB queries
- All tools extracted in /tools/
This commit is contained in:
Eric F
2026-06-08 07:38:57 -04:00
parent 3fb90a8033
commit d4974e3241
72 changed files with 5185 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
tools/flexmeasures-entsoe/.flake8 Executable file
View File

@@ -0,0 +1,7 @@
[flake8]
exclude = .git,__pycache__,documentation
max-line-length = 160
max-complexity = 13
select = B,C,E,F,W,B9
ignore = E501, W503, E203

42
tools/flexmeasures-entsoe/.github/workflows/ci.yml vendored Normal file
View File

@@ -0,0 +1,42 @@
name: CI
on:
push:
branches:
- main
pull_request:
permissions:
contents: read
jobs:
test:
name: "Python ${{ matrix.python-version }} / FlexMeasures ${{ matrix.flexmeasures.version }}"
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
python-version: ["3.10", "3.11", "3.12"]
flexmeasures:
- version: "0.31.*"
requirement: "flexmeasures==0.31.*"
- version: "0.32.*"
requirement: "flexmeasures==0.32.*"
- version: "latest"
requirement: "flexmeasures"
steps:
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Install FlexMeasures compatibility target
run: |
python -m pip install --upgrade pip setuptools wheel
python -m pip install "${{ matrix.flexmeasures.requirement }}"
- name: Run tests
run: make test

75
tools/flexmeasures-entsoe/.gitignore vendored Normal file
View File

@@ -0,0 +1,75 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
pip-wheel-metadata/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
# Flask stuff:
instance/
.webassets-cache
# Sphinx documentation
docs/_build/
# IPython
profile_default/
ipython_config.py
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
# mypy
.mypy_cache/
.dmypy.json
dmypy.json
# custom project files
flexmeasures.log

19
tools/flexmeasures-entsoe/.pre-commit-config.yaml Executable file
View File

@@ -0,0 +1,19 @@
repos:
- repo: https://github.com/pycqa/flake8
rev: 6.0.0 # New version tags can be found here: https://github.com/pycqa/flake8/tags
hooks:
- id: flake8
name: flake8 (code linting)
- repo: https://github.com/psf/black
rev: 22.10.0 # New version tags can be found here: https://github.com/psf/black/tags
hooks:
- id: black
name: black (code formatting)
- repo: local
hooks:
- id: mypy
name: mypy (static typing)
pass_filenames: false
language: script
entry: run_mypy.sh
verbose: true

201
tools/flexmeasures-entsoe/LICENSE Normal file
View File

@@ -0,0 +1,201 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

20
tools/flexmeasures-entsoe/Makefile Normal file
View File

@@ -0,0 +1,20 @@
# Note: use tabs
# actions which are virtual, i.e. not a script
.PHONY: install install-for-dev test
install:
pip install -e .
# ---- Development ---
test:
make install-for-dev
pytest
install-for-dev:
pip install -r requirements/app.in -r requirements/dev.in -r requirements/test.in
make install
pre-commit install

133
tools/flexmeasures-entsoe/README.md Normal file
View File

@@ -0,0 +1,133 @@
# ENTSO-E forecasts & data
Importing data which can be relevant for energy flexibility services via ENTSO-E's API into FlexMeasures.
We start with data about the upcoming day.
- Generation forecasts for the upcoming day
- Based on these, CO2 content for the upcoming day
- Day-ahead prices
## Usage
Importing tomorrow's prices:
flexmeasures entsoe import-day-ahead-prices
Importing tomorrow's generation (incl. CO2 estimated content):
flexmeasures entsoe import-day-ahead-generation
Use ``--help`` to learn more usage details.
### October 1st 2025 go-live for ENTSO-E moving to 15-minute day-ahead prices
ENTSO-E is moving from 1-hour day-ahead prices 15-minute day-ahead prices on October 1st 2025.
To prepare for this transition, you have two choices:
1. resample your existing price sensor in FlexMeasures from 1 hour to 15 minutes, or
2. get a new sensor for the 15-minute data.
If you do this *after* the go-live moment, the `flexmeasures-entsoe` package just keeps resampling the 15-minute ENTSO-E data to hourly data.
#### 1. Resampling
**The upside** of resampling your existing price data is that the sensor ID of your price sensor in FlexMeasures will remain the same.
Depending on your system setup, `Forecaster`/`Reporter`/`Scheduler` configurations (such as an asset's `flex-context`) may depend on it, and your users may expect the 15-minute data to live under the same sensor.
**The downside** is that it quadruples your data for that sensor, due to the fact that FlexMeasures only supports a fixed resolution for any given sensor. Although there should be no noticeable hit in performance, it obviously leads to redundant data in the price history before October 1st 2025.
**To resample** your historical data, use:
```bash
flexmeasures edit resample-data --sensor <ID of your day-ahead price sensor> --event-resolution 15
```
The `flexmeasures-entsoe` package already automatically resamples the ENTSO-E data to the resolution of your sensor.
If you use a `Reporter` to derive retail prices or to compute energy costs, there is no need to update its configuration; just resample these sensors too, using the previous command (replacing the sensor ID as needed).
Alternatively, if you want to keep these sensors in their original resolution, and find that your reporters fail with an `AssertionError` about mismatched resolutions, you may need to add the `--resolution PT1H` option when using the `flexmeasures add report` command.
#### 2. Getting a new sensor
**The upside** is that this doesn't quadruple your historic data (see *the downside* of resampling, above).
**The downside** is that you may need to revise `Forecaster`/`Reporter`/`Scheduler` configurations (such as an asset's `flex-context`) and notify users (see *the upside* of resampling, above).
**To get a new sensor**, rename your existing *Day-ahead prices* sensor in the FlexMeasures UI.
The `flexmeasures-entsoe` package will then automatically create a new 15-minute price sensor the next time `flexmeasures entsoe import-day-ahead-prices` is run, assigning it a new sensor ID.
If you have any price or costs sensors using a `Reporter` to derive values from the day-ahead wholesale prices, update the sensor ID in the configuration of each `Reporter`.
Finally, either resample each derived sensor using:
```bash
flexmeasures edit resample-data --sensor <ID of your derivative sensor> --event-resolution 15
```
or, if you want to keep these sensors in their original resolution, and find that your reporters fail with an `AssertionError` about mismatched resolutions, you may need to add the `--resolution PT1H` option when using the `flexmeasures add report` command.
## Installation
First of all, this is a FlexMeasures plugin. Consult the FlexMeasures documentation for setup.
1. Add the plugin to [the `FLEXMEASURES_PLUGINS` setting](https://flexmeasures.readthedocs.io/stable/configuration.html#flexmeasures-plugins). Either use `/path/to/flexmeasures-entsoe/flexmeasures_entsoe` or `flexmeasures_entsoe` if you installed this as a package locally (see below).
2. Add `ENTSOE_AUTH_TOKEN` to your FlexMeasures config (e.g. ~/.flexmeasures.cfg).
You can generate this token after you made an account at ENTSO-E, read more [here](https://transparencyplatform.zendesk.com/hc/en-us/articles/12845911031188-How-to-get-security-token).
Optionally, override other settings (defaults shown here):
ENTSOE_COUNTRY_CODE = "NL"
ENTSOE_COUNTRY_TIMEZONE = "Europe/Amsterdam"
ENTSOE_DERIVED_DATA_SOURCE = "FlexMeasures ENTSO-E"
The `ENTSOE_DERIVED_DATA_SOURCE` option is used to name the source of data that this plugin derives from ENTSO-E data, like a CO₂ signal.
Original ENTSO-E data is reported as being sourced by `"ENTSO-E"`.
3. To install this plugin locally as a package, try `pip install .`.
## Testing
ENTSO-E provides a test server (iop) for development. It's good practice not to overwhelm their production server.
Set ``ENTSOE_USE_TEST_SERVER=True`` to enable this.
In that case, this plugin will look for the auth token in the config setting ``ENTSOE_AUTH_TOKEN_TEST_SERVER``.
Note, however, that ENTSO-E usually does not seem to make the latest data available there. Asking for the next day can often get an empty response.
## Supported FlexMeasures versions
This plugin targets two distinct FlexMeasures capability tiers:
| FlexMeasures version | Behavior |
|---|---|
| `< 0.32` | Uses the legacy `get_data_source` factory; no account is linked to the ENTSO-E source. |
| `>= 0.32` | Uses the account-linked source API (`get_or_create_source` with an `Account`). |
This package supports Python 3.10 through 3.12, following the Python support policy of the currently supported FlexMeasures releases.
The oldest supported FlexMeasures release line is `0.31.*`.
CI is run against `0.31.*` (minimum supported legacy release), `0.32.*` (first account-linked release), and the latest released FlexMeasures version across all supported Python versions.
When a new FlexMeasures release introduces breaking changes the matrix should be updated accordingly.
## Development
To keep our code quality high, we use pre-commit:
pip install pre-commit black flake8 mypy
pre-commit install
or:
make install-for-dev
Try it:
pre-commit run --all-files --show-diff-on-failure

54
tools/flexmeasures-entsoe/flexmeasures_entsoe/__init__.py Normal file
View File

@@ -0,0 +1,54 @@
import os
import sys
from flask import Blueprint
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, HERE)
DEFAULT_COUNTRY_CODE = "NL"
DEFAULT_COUNTRY_TIMEZONE = "Europe/Amsterdam" # This is what we receive, even if ENTSO-E documents Europe/Brussels
DEFAULT_DATA_SOURCE_NAME = "ENTSO-E"
DEFAULT_DERIVED_DATA_SOURCE = "FlexMeasures ENTSO-E"
__version__ = "0.9"
__settings__ = {
"ENTSOE_AUTH_TOKEN": dict(
description="You can generate this token after you made an account at ENTSO-E.",
level="error",
),
"ENTSOE_COUNTRY_CODE": dict(
level="warning",
message_if_missing=f"'{DEFAULT_COUNTRY_CODE}' will be used as a default.",
),
"ENTSOE_COUNTRY_TIMEZONE": dict(
description="IANA timezone name used to localize ENTSO-E sensors.",
level="info",
message_if_missing=f"'{DEFAULT_COUNTRY_TIMEZONE}' will be used as a default.",
),
"ENTSOE_USE_TEST_SERVER": dict(
description="Boolean to indicate whether to use the ENTSO-E's iop test server instead of their production server",
level="debug",
),
"ENTSOE_AUTH_TOKEN_TEST_SERVER": dict(
description="You can generate this token after you made an account at ENTSO-E.",
level="debug",
),
"ENTSOE_DERIVED_DATA_SOURCE": dict(
description="String used to name the source of data that this plugin derives from ENTSO-E data, like a CO₂ signal.",
level="info",
message_if_missing=f"'{DEFAULT_DERIVED_DATA_SOURCE}' will be used as a default.",
),
"ENTSOE_DATA_SOURCE_NAME": dict(
description="String used to name the ENTSO-E data source and the account associated with it.",
level="info",
message_if_missing=f"'{DEFAULT_DATA_SOURCE_NAME}' will be used as a default.",
),
}
entsoe_data_bp = Blueprint("entsoe", __name__, cli_group="entsoe")
entsoe_data_bp.cli.help = "ENTSO-E Data commands"
from .generation import day_ahead as day_ahead_generation # noqa: E402,F401
from .prices import day_ahead as day_ahead_prices # noqa: E402,F401

10
tools/flexmeasures-entsoe/flexmeasures_entsoe/generation/__init__.py Normal file
View File

@@ -0,0 +1,10 @@
from datetime import timedelta
# sensor_name, unit, event_resolution, data sourced directly by ENTSO-E or not (i.e. derived)
generation_sensors = (
("Scheduled generation", "MW", timedelta(minutes=15), True),
("Solar", "MW", timedelta(hours=1), True),
("Wind Onshore", "MW", timedelta(hours=1), True),
("Wind Offshore", "MW", timedelta(hours=1), True),
("CO₂ intensity", "kg/MWh", timedelta(minutes=15), False),
)

214
tools/flexmeasures-entsoe/flexmeasures_entsoe/generation/day_ahead.py Normal file
View File

@@ -0,0 +1,214 @@
from typing import Optional
from datetime import datetime
import click
from flask.cli import with_appcontext
from flask import current_app
# from entsoe.entsoe import URL
import pandas as pd
from flexmeasures.data.transactional import task_with_status_report
from .. import (
entsoe_data_bp,
) # noqa: E402
from . import generation_sensors
from ..utils import (
create_entsoe_client,
ensure_country_code_and_timezone,
ensure_data_source,
ensure_data_source_for_derived_data,
abort_if_data_empty,
parse_from_and_to_dates,
save_entsoe_series,
ensure_sensors,
resample_if_needed,
start_import_log,
)
"""
Get the CO₂ content from tomorrow's generation forecasts.
We get the overall forecast and the solar&wind forecast, so we know the share of green energy.
For now, we'll compute the CO₂ mix from some assumptions.
"""
# TODO: Decide which sources to use ― https://github.com/SeitaBV/flexmeasures-entsoe/issues/2
# Source for these ratios: https://ourworldindata.org/energy/country/netherlands#what-sources-does-the-country-get-its-electricity-from (2020 data)
grey_energy_mix = dict(gas=0.598, oil=0.045, coal=0.0718)
# Source for kg CO₂ per MWh: https://energy.utexas.edu/news/nuclear-and-wind-power-estimated-have-lowest-levelized-co2-emissions
kg_CO2_per_MWh = dict(
coal=870, # lignite
gas=464, # natural
solar=44.5, # mix of utility/residential, difference isn't large
oil=652, # ca. 75% of coal, see https://www.volker-quaschning.de/datserv/CO2-spez/index_e.php
wind_onshore=14,
wind_offshore=17, # factor of ~ 1.1, see https://www.mdpi.com/2071-1050/10/6/2022
)
@entsoe_data_bp.cli.command("import-day-ahead-generation")
@click.option(
"--from-date",
required=False,
type=click.DateTime(["%Y-%m-%d"]),
help="Query data from this date onwards. If not specified, defaults to today",
)
@click.option(
"--to-date",
required=False,
type=click.DateTime(["%Y-%m-%d"]),
help="Query data until this date (inclusive). If not specified, defaults to tomorrow.",
)
@click.option(
"--dryrun/--no-dryrun",
default=False,
help="In dry run mode, do not save the data to the db.",
)
@click.option(
"--country",
"country_code",
required=False,
help="ENTSO-E country code (such as BE, DE, FR or NL).",
)
@click.option(
"--timezone",
"country_timezone",
required=False,
help="Timezone for the country (such as 'Europe/Amsterdam').",
)
@click.option(
"--for",
"default_import_timerange",
required=False,
default="today-and-tomorrow",
type=click.Choice(["today", "tomorrow", "today-and-tomorrow"]),
help="Easy-to-use time range setting, only used if --from-date and --to-date are not used. If set to 'today' or 'tomorrow' or 'today-and-tomorrow', only import data for thes days. The default is today-and-tomorrow.",
)
@with_appcontext
@task_with_status_report("entsoe-import-day-ahead-generation")
def import_day_ahead_generation(
dryrun: bool = False,
from_date: Optional[datetime] = None,
to_date: Optional[datetime] = None,
country_code: Optional[str] = None,
country_timezone: Optional[str] = None,
default_import_timerange: str = "today-and-tomorrow",
):
"""
Import forecasted generation for any date range, defaulting to today and tomorrow.
This will save overall generation, solar, offshore and onshore wind, and the estimated CO₂ content per hour.
Possibly best to run this script somewhere around or maybe two or three hours after 13:00,
when tomorrow's prices are announced.
"""
# Set up FlexMeasures data structure
country_code, country_timezone = ensure_country_code_and_timezone(
country_code, country_timezone
)
entsoe_data_source = ensure_data_source()
derived_data_source = ensure_data_source_for_derived_data()
sensors = ensure_sensors(generation_sensors, country_code, country_timezone)
# Parse CLI options (or set defaults)
from_time, until_time = parse_from_and_to_dates(
from_date, to_date, country_timezone, default_to=default_import_timerange
)
# Start import
client = create_entsoe_client()
log, now = start_import_log(
"day-ahead generation", from_time, until_time, country_code, country_timezone
)
log.info("Getting scheduled generation ...")
# We assume that the green (solar & wind) generation is not included in this (it is not scheduled)
scheduled_generation: pd.Series = client.query_generation_forecast(
country_code, start=from_time, end=until_time
)
abort_if_data_empty(scheduled_generation)
log.debug("Overall aggregated generation: \n%s" % scheduled_generation)
scheduled_generation = resample_if_needed(
scheduled_generation,
sensors["Scheduled generation"],
)
log.info("Getting green generation ...")
green_generation_df: pd.DataFrame = client.query_wind_and_solar_forecast(
country_code, start=from_time, end=until_time, psr_type=None
)
abort_if_data_empty(green_generation_df)
log.debug("Green generation: \n%s" % green_generation_df)
log.info("Aggregating green energy columns ...")
all_green_generation = green_generation_df.sum(axis="columns")
log.debug("Aggregated green generation: \n%s" % all_green_generation)
log.info("Computing combined generation forecast ...")
all_generation = scheduled_generation + all_green_generation
log.debug("Combined generation: \n%s" % all_generation)
log.info("Computing CO₂ content from the MWh values ...")
co2_in_kg = calculate_CO2_content_in_kg(scheduled_generation, green_generation_df)
log.debug("Overall CO₂ content (kg): \n%s" % co2_in_kg)
forecasted_kg_CO2_per_MWh = co2_in_kg / all_generation
log.debug("Overall CO₂ content (kg/MWh): \n%s" % forecasted_kg_CO2_per_MWh)
def get_series_for_sensor(sensor):
if sensor.name == "Scheduled generation":
return scheduled_generation
elif sensor.name == "Solar":
return green_generation_df["Solar"]
elif sensor.name == "Wind Onshore":
return green_generation_df["Wind Onshore"]
elif sensor.name == "Wind Offshore":
return green_generation_df["Wind Offshore"]
elif sensor.name == "CO₂ intensity":
return forecasted_kg_CO2_per_MWh
else:
log.error(f"Cannot connect data to sensor {sensor.name}.")
raise click.Abort
if not dryrun:
for sensor in sensors.values():
series = get_series_for_sensor(sensor)
log.info(f"Saving {len(series)} beliefs for Sensor {sensor.name} ...")
entsoe_source = (
entsoe_data_source if sensor.data_by_entsoe else derived_data_source
)
save_entsoe_series(series, sensor, entsoe_source, country_timezone, now)
def calculate_CO2_content_in_kg(
grey_generation: pd.Series, green_generation: pd.DataFrame
) -> pd.Series:
grey_CO2_intensity_factor = ( # TODO: a factor per hour of the day
(grey_energy_mix["coal"] * kg_CO2_per_MWh["coal"])
+ (grey_energy_mix["gas"] * kg_CO2_per_MWh["gas"])
+ (grey_energy_mix["oil"] * kg_CO2_per_MWh["oil"])
)
current_app.logger.debug(f"Grey intensity factor: {grey_CO2_intensity_factor}")
grey_CO2_content = grey_generation * grey_CO2_intensity_factor
current_app.logger.debug("Grey CO₂ content (tonnes): \n%s" % grey_CO2_content)
green_generation["solar CO₂"] = (
green_generation["Solar"] * kg_CO2_per_MWh["solar"] / 1000.0
)
green_generation["wind_onshore CO₂"] = (
green_generation["Wind Onshore"] * kg_CO2_per_MWh["wind_onshore"]
)
green_generation["wind_offshore CO₂"] = (
green_generation["Wind Offshore"] * kg_CO2_per_MWh["wind_offshore"]
)
current_app.logger.debug(
"Green generation and CO₂ content: \n%s" % green_generation
)
return (
grey_CO2_content
+ green_generation["solar CO₂"]
+ green_generation["wind_onshore CO₂"]
+ green_generation["wind_offshore CO₂"]
)

57
tools/flexmeasures-entsoe/flexmeasures_entsoe/generation/utils.py Normal file
View File

@@ -0,0 +1,57 @@
import pandas as pd
def determine_net_emission_factors(shares: pd.DataFrame) -> pd.Series:
"""Given production shares, determine the net emission factors.
Or given production by type, determine the net emissions.
Use column headers that match production types listed below.
Use any index.
For example:
print(shares)
fossil_gas other fossil_hard_coal waste nuclear
hour
0 0.443685 0.206033 0.237596 0.050915 0.059455
1 0.443910 0.205065 0.235022 0.052614 0.060987
print(determine_net_emission_factors(shares))
hour
0 644.753221
1 641.410093
Name: Average emissions from Dutch electricity production (kg CO₂ eq/MWh), dtype: float64
"""
emission_factors = dict(
biomass=50.4,
fossil_brown_coal_or_lignite=None, # unknown
fossil_coal_derived_gas=None, # unknown
fossil_gas=464,
fossil_hard_coal=1030,
fossil_oil=1010,
fossil_oil_shale=None, # unknown
fossil_peat=None, # unknown
geothermal=0.00664,
hydro_pumped_storage=611,
hydro_run_of_river_and_poundage=0.0253,
hydro_water_reservoir=8.13,
marine=None, # unknown
nuclear=10.1,
other=927, # for EU28
other_renewable=None, # unknown
solar=0.00591,
waste=None, # unknown
wind_offshore=0.133,
wind_onshore=0.133,
) # supplementary material from "Real-time carbon accounting method for the European electricity markets, Tranberg et al. (2019)"
# todo: substitute placeholder for unknown emission factor of waste
emission_factors["waste"] = emission_factors["biomass"]
for production_type in shares.columns:
shares[production_type] = (
shares[production_type] * emission_factors[production_type]
)
return shares.sum(axis=1).rename(
"Average emissions from Dutch electricity production (kg CO₂ eq/MWh)"
)

4
tools/flexmeasures-entsoe/flexmeasures_entsoe/prices/__init__.py Normal file
View File

@@ -0,0 +1,4 @@
from datetime import timedelta
# sensor_name, unit, even_resolution, data sourced directly by ENTSO-E or not (i.e. derived)
pricing_sensors = (("Day-ahead prices", "EUR/MWh", timedelta(minutes=15), True),)

155
tools/flexmeasures-entsoe/flexmeasures_entsoe/prices/day_ahead.py Normal file
View File

@@ -0,0 +1,155 @@
from typing import Optional
from datetime import datetime
import click
from flask.cli import with_appcontext
import pandas as pd
from flexmeasures import Source, Sensor
from flexmeasures.data.transactional import task_with_status_report
from flexmeasures.data.schemas import SensorIdField
from flexmeasures.data.schemas.sources import DataSourceIdField
from . import pricing_sensors
from .. import (
entsoe_data_bp,
) # noqa: E402
from ..utils import (
create_entsoe_client,
ensure_country_code_and_timezone,
ensure_data_source,
parse_from_and_to_dates,
ensure_sensors,
save_entsoe_series,
abort_if_data_empty,
abort_if_data_incomplete,
resample_if_needed,
start_import_log,
)
@entsoe_data_bp.cli.command("import-day-ahead-prices")
@click.option(
"--from-date",
required=False,
type=click.DateTime(["%Y-%m-%d"]),
help="Query data from this date onwards. If not specified, defaults to today",
)
@click.option(
"--to-date",
required=False,
type=click.DateTime(["%Y-%m-%d"]),
help="Query data until this date (inclusive). If not specified, defaults to tomorrow.",
)
@click.option(
"--dryrun/--no-dryrun",
default=False,
help="In dry run mode, do not save the data to the db.",
)
@click.option(
"--country",
"country_code",
required=False,
help="ENTSO-E country code (such as BE, DE, FR or NL).",
)
@click.option(
"--timezone",
"country_timezone",
required=False,
help="Timezone for the country (such as 'Europe/Amsterdam').",
)
@click.option(
"--sensor",
"sensor",
type=SensorIdField(),
required=False,
help="Sensor to store the data into. If not provided, the sensor `Day-ahead prices` is used.",
)
@click.option(
"--source",
"source",
type=DataSourceIdField(),
required=False,
help="Source of the price data. If not provided, the source `ENTSO-E` is used.",
)
@click.option(
"--for",
"default_import_timerange",
required=False,
default="today-and-tomorrow",
type=click.Choice(["today", "tomorrow", "today-and-tomorrow"]),
help="Easy-to-use time range setting, which defines the defaults for start and end to be used when --from-date and/or --to-date are not used. Can be set to 'today' or 'tomorrow' or 'today-and-tomorrow' (which is the default value).",
)
@click.option(
"--fail-on-incomplete-data",
"fail_on_incomplete_data",
is_flag=True,
default=False,
help="If set, the import will abort if the data received is incomplete.",
)
@with_appcontext
@task_with_status_report("entsoe-import-day-ahead-prices")
def import_day_ahead_prices(
dryrun: bool = False,
from_date: Optional[datetime] = None,
to_date: Optional[datetime] = None,
country_code: Optional[str] = None,
country_timezone: Optional[str] = None,
sensor: Optional[Sensor] = None,
source: Optional[Source] = None,
default_import_timerange: str = "today-and-tomorrow",
fail_on_incomplete_data: bool = False,
):
"""
Import forecasted prices for any date range, defaulting to today and tomorrow.
Possibly best to run this script somewhere around or maybe two or three hours after 13:00,
when tomorrow's prices are announced.
"""
# Set up FlexMeasures data structure
country_code, country_timezone = ensure_country_code_and_timezone(
country_code, country_timezone
)
if source is None:
entsoe_data_source = ensure_data_source()
else:
entsoe_data_source = source
if sensor is None:
# For now, we only have one pricing sensor ...
sensors = ensure_sensors(pricing_sensors, country_code, country_timezone)
pricing_sensor = sensors["Day-ahead prices"]
assert pricing_sensor.name == "Day-ahead prices"
else:
pricing_sensor = sensor
# Parse CLI options (or set defaults)
from_time, until_time = parse_from_and_to_dates(
from_date, to_date, country_timezone, default_to=default_import_timerange
)
# Start import
client = create_entsoe_client()
log, now = start_import_log(
"day-ahead price", from_time, until_time, country_code, country_timezone
)
log.info("Getting prices ...")
prices: pd.Series = client.query_day_ahead_prices(
country_code, start=from_time, end=until_time
)
abort_if_data_empty(prices)
if fail_on_incomplete_data:
abort_if_data_incomplete(
prices, from_time, until_time, pricing_sensor.event_resolution
)
prices = resample_if_needed(prices, pricing_sensor)
log.debug("Prices: \n%s" % prices)
if not dryrun:
log.info(f"Saving {len(prices)} beliefs for Sensor {pricing_sensor.name} ...")
save_entsoe_series(
prices, pricing_sensor, entsoe_data_source, country_timezone, now
)

0
tools/flexmeasures-entsoe/flexmeasures_entsoe/tests/__init__.py Normal file
View File

308
tools/flexmeasures-entsoe/flexmeasures_entsoe/tests/test_utils.py Normal file
View File

@@ -0,0 +1,308 @@
from datetime import datetime, timedelta
from types import SimpleNamespace
import click
import pandas as pd
import pytz
import pytest
from flexmeasures_entsoe import DEFAULT_DATA_SOURCE_NAME, DEFAULT_DERIVED_DATA_SOURCE
from flexmeasures_entsoe.utils import (
FM_SUPPORTS_ACCOUNT_LINKED_SOURCES,
_ensure_entsoe_source,
abort_if_data_incomplete,
ensure_data_source,
ensure_data_source_for_derived_data,
parse_from_and_to_dates,
)
def test_abort_if_data_incomplete():
"""
Tests that the function raises click.Abort if data is incomplete.
1. Data is complete: No exception raised.
2. Data is incomplete: click.Abort is raised.
"""
start = pd.Timestamp("2025-01-01 00:00")
end = pd.Timestamp("2025-01-02 00:00")
resolution = pd.Timedelta(hours=1)
# Case 1: Data is complete (24 items for 24 hours)
complete_data = pd.DataFrame({"val": range(24)})
try:
abort_if_data_incomplete(complete_data, start, end, resolution)
except click.Abort:
pytest.fail("Function raised Abort unexpectedly on complete data")
# Case 2: Data is incomplete (20 items for 24 hours)
incomplete_data = pd.DataFrame({"val": range(20)})
with pytest.raises(click.Abort):
abort_if_data_incomplete(incomplete_data, start, end, resolution)
def test_parse_from_and_to_dates():
"""
Tests CLI date parsing logic:
1. Explicit dates are timezone-localized correctly.
2. 'None' defaults to tomorrow (start of day) -> day after tomorrow.
"""
tz_str = "UTC"
tz = pytz.timezone(tz_str)
now = datetime.now(tz)
today = datetime(now.year, now.month, now.day, tzinfo=tz)
# Case 1: Explicit inputs
input_start = datetime(2025, 5, 1)
input_end = datetime(2025, 5, 2)
s, e = parse_from_and_to_dates(
from_date=input_start, until_date=input_end, country_timezone=tz_str
)
assert s.tzinfo.zone == tz.zone
assert (e - s) == timedelta(days=2)
assert e == datetime(2025, 5, 3, tzinfo=tz)
# Case 2: default_to="tomorrow"
s_tom, e_tom = parse_from_and_to_dates(
from_date=None, until_date=None, country_timezone=tz_str, default_to="tomorrow"
)
assert e_tom - s_tom == timedelta(days=1)
assert s_tom == today + timedelta(days=1)
assert e_tom == today + timedelta(days=2)
# Case 3: default_to="today-and-tomorrow"
s_tod, e_tod = parse_from_and_to_dates(
from_date=None, until_date=None, country_timezone=tz_str
)
assert e_tod - s_tod == timedelta(days=2)
assert s_tod == today
assert e_tod == today + timedelta(days=2)
# Case 4: only providing until_date (today midnight == start of tomorrow), while start comes from "today-and-tomorrow"
today_midnight = datetime(now.year, now.month, now.day) + timedelta(days=1)
s_none, e_none = parse_from_and_to_dates(
from_date=None, until_date=today_midnight, country_timezone=tz_str
)
assert e_none - s_none == timedelta(days=2)
assert s_none == today
assert e_none == today + timedelta(days=2)
# The version-branch tests below still use monkeypatching to isolate source
# creation side effects and to simulate upgrade reuse of legacy ENTSO-E
# sources without requiring multiple FlexMeasures installs in one test run.
@pytest.mark.skipif(
not FM_SUPPORTS_ACCOUNT_LINKED_SOURCES,
reason="Account-linked ENTSO-E sources are only supported on FlexMeasures >= 0.32.",
)
def test_ensure_data_source_passes_entsoe_account_when_supported(monkeypatch):
"""Test that ensure_data_source() creates a market-type source and passes the ENTSO-E account."""
from flask import Flask
app = Flask(__name__)
captured_kwargs = {}
def fake_get_or_create_source(source, source_type, account, flush):
captured_kwargs.update(
dict(
source=source,
source_type=source_type,
account=account,
flush=flush,
)
)
return SimpleNamespace(type=source_type, account=account, name=source)
fake_account = SimpleNamespace(name=DEFAULT_DATA_SOURCE_NAME)
monkeypatch.setattr(
"flexmeasures_entsoe.utils.get_or_create_source",
fake_get_or_create_source,
)
monkeypatch.setattr(
"flexmeasures_entsoe.utils._find_existing_source",
lambda source_name, source_type: None,
)
monkeypatch.setattr(
"flexmeasures_entsoe.utils.get_or_create_entsoe_account",
lambda: fake_account,
)
with app.app_context():
data_source = ensure_data_source()
assert data_source.type == "market"
assert captured_kwargs["source"] == DEFAULT_DATA_SOURCE_NAME
assert captured_kwargs["account"].name == DEFAULT_DATA_SOURCE_NAME
@pytest.mark.skipif(
not FM_SUPPORTS_ACCOUNT_LINKED_SOURCES,
reason="Account-linked ENTSO-E sources are only supported on FlexMeasures >= 0.32.",
)
def test_ensure_data_source_for_derived_data_passes_entsoe_account_when_supported(
monkeypatch,
):
"""Test that ensure_data_source_for_derived_data() passes the ENTSO-E account."""
from flask import Flask
app = Flask(__name__)
captured_kwargs = {}
def fake_get_or_create_source(source, source_type, account, flush):
captured_kwargs.update(
dict(
source=source,
source_type=source_type,
account=account,
flush=flush,
)
)
return SimpleNamespace(type=source_type, account=account, name=source)
fake_account = SimpleNamespace(name=DEFAULT_DATA_SOURCE_NAME)
monkeypatch.setattr(
"flexmeasures_entsoe.utils.get_or_create_source",
fake_get_or_create_source,
)
monkeypatch.setattr(
"flexmeasures_entsoe.utils._find_existing_source",
lambda source_name, source_type: None,
)
monkeypatch.setattr(
"flexmeasures_entsoe.utils.get_or_create_entsoe_account",
lambda: fake_account,
)
with app.app_context():
data_source = ensure_data_source_for_derived_data()
assert data_source.type == "forecasting script"
assert captured_kwargs["source"] == DEFAULT_DERIVED_DATA_SOURCE
assert captured_kwargs["account"].name == DEFAULT_DATA_SOURCE_NAME
@pytest.mark.skipif(
FM_SUPPORTS_ACCOUNT_LINKED_SOURCES,
reason="Legacy get_data_source fallback is only used on FlexMeasures < 0.32.",
)
def test_ensure_data_source_omits_account_when_not_supported(monkeypatch):
"""Test that ensure_data_source() falls back to the legacy source factory without an account."""
from flask import Flask
app = Flask(__name__)
captured_kwargs = {}
def fake_get_data_source(data_source_name, data_source_type):
captured_kwargs.update(
data_source_name=data_source_name,
data_source_type=data_source_type,
)
return SimpleNamespace(name=data_source_name, type=data_source_type)
monkeypatch.setattr(
"flexmeasures_entsoe.utils._find_existing_source",
lambda source_name, source_type: None,
)
monkeypatch.setattr(
"flexmeasures_entsoe.utils.get_data_source",
fake_get_data_source,
)
with app.app_context():
data_source = ensure_data_source()
assert data_source.type == "market"
assert captured_kwargs == {
"data_source_name": DEFAULT_DATA_SOURCE_NAME,
"data_source_type": "market",
}
@pytest.mark.skipif(
FM_SUPPORTS_ACCOUNT_LINKED_SOURCES,
reason="Legacy get_data_source fallback is only used on FlexMeasures < 0.32.",
)
def test_ensure_data_source_for_derived_data_omits_account_when_not_supported(
monkeypatch,
):
"""Test that ensure_data_source_for_derived_data() falls back to the legacy source factory."""
from flask import Flask
app = Flask(__name__)
captured_kwargs = {}
def fake_get_data_source(data_source_name, data_source_type):
captured_kwargs.update(
data_source_name=data_source_name,
data_source_type=data_source_type,
)
return SimpleNamespace(name=data_source_name, type=data_source_type)
monkeypatch.setattr(
"flexmeasures_entsoe.utils._find_existing_source",
lambda source_name, source_type: None,
)
monkeypatch.setattr(
"flexmeasures_entsoe.utils.get_data_source",
fake_get_data_source,
)
with app.app_context():
data_source = ensure_data_source_for_derived_data()
assert data_source.type == "forecasting script"
assert captured_kwargs == {
"data_source_name": DEFAULT_DERIVED_DATA_SOURCE,
"data_source_type": "forecasting script",
}
@pytest.mark.skipif(
not FM_SUPPORTS_ACCOUNT_LINKED_SOURCES,
reason="Legacy source upgrade reuse matters in the account-linked source path only.",
)
def test_ensure_entsoe_source_reuses_legacy_source_and_sets_account(monkeypatch):
legacy_source = SimpleNamespace(
name=DEFAULT_DATA_SOURCE_NAME,
type="forecasting script",
account=None,
)
fake_account = SimpleNamespace(name=DEFAULT_DATA_SOURCE_NAME)
def fake_find_existing_source(source_name, source_type):
if source_type == "market":
return None
if source_type == "forecasting script":
return legacy_source
return None
monkeypatch.setattr(
"flexmeasures_entsoe.utils._find_existing_source",
fake_find_existing_source,
)
monkeypatch.setattr(
"flexmeasures_entsoe.utils.get_or_create_entsoe_account",
lambda: fake_account,
)
monkeypatch.setattr(
"flexmeasures_entsoe.utils.get_or_create_source",
lambda **kwargs: pytest.fail(
"Should reuse a legacy ENTSO-E source before creating a new one."
),
)
data_source = _ensure_entsoe_source(
source_name=DEFAULT_DATA_SOURCE_NAME,
source_type="market",
legacy_source_type="forecasting script",
)
assert data_source is legacy_source
assert data_source.type == "market"
assert data_source.account is fake_account

369
tools/flexmeasures-entsoe/flexmeasures_entsoe/utils.py Normal file
View File

@@ -0,0 +1,369 @@
from typing import Dict, Optional, Tuple, Union
from datetime import datetime, timedelta
from logging import Logger
from entsoe import EntsoePandasClient
from flask import current_app
from packaging import version
from pandas.tseries.frequencies import to_offset
import pandas as pd
import click
import pytz
import entsoe
from flexmeasures.data.utils import get_data_source, save_to_db
from flexmeasures import Asset, AssetType, Sensor, Source, __version__ as flexmeasures_version
from flexmeasures.data import db
from flexmeasures.utils.time_utils import server_now
from timely_beliefs import BeliefsDataFrame
from flexmeasures.cli.utils import MsgStyle
from . import (
DEFAULT_DATA_SOURCE_NAME,
DEFAULT_DERIVED_DATA_SOURCE,
DEFAULT_COUNTRY_CODE,
DEFAULT_COUNTRY_TIMEZONE,
) # noqa: E402
FM_SUPPORTS_ACCOUNT_LINKED_SOURCES = version.parse(
flexmeasures_version
) >= version.parse("0.32")
if FM_SUPPORTS_ACCOUNT_LINKED_SOURCES:
from flexmeasures import Account
from flexmeasures.data.services.data_sources import get_or_create_source
def _find_existing_source(source_name: str, source_type: str) -> Optional[Source]:
return (
Source.query.filter(
Source.name == source_name,
Source.type == source_type,
)
.order_by(Source.id)
.first()
)
def get_or_create_entsoe_account():
"""Make sure we have an account for the ENTSO-E provider service."""
account_name = current_app.config.get(
"ENTSOE_DATA_SOURCE_NAME", DEFAULT_DATA_SOURCE_NAME
)
entsoe_account = Account.query.filter(
Account.name == account_name,
).one_or_none()
if entsoe_account is None:
entsoe_account = Account(name=account_name)
db.session.add(entsoe_account)
db.session.flush()
return entsoe_account
def _ensure_entsoe_source(
source_name: str,
source_type: str,
legacy_source_type: Optional[str] = None,
) -> Source:
"""Reuse legacy sources when possible while branching explicitly on FM version."""
entsoe_account = None
if FM_SUPPORTS_ACCOUNT_LINKED_SOURCES:
entsoe_account = get_or_create_entsoe_account()
existing_source = _find_existing_source(source_name, source_type)
if existing_source is None and legacy_source_type is not None:
existing_source = _find_existing_source(source_name, legacy_source_type)
if existing_source is not None:
existing_source.type = source_type
if existing_source is not None:
if entsoe_account is not None and getattr(existing_source, "account", None) is None:
existing_source.account = entsoe_account
return existing_source
if not FM_SUPPORTS_ACCOUNT_LINKED_SOURCES:
return get_data_source(
data_source_name=source_name,
data_source_type=source_type,
)
source_kwargs = dict(
source=source_name,
source_type=source_type,
flush=False,
)
if entsoe_account is not None:
source_kwargs["account"] = entsoe_account
return get_or_create_source(**source_kwargs)
def ensure_data_source() -> Source:
"""Make sure we have a raw ENTSO-E data source of type "market"."""
return _ensure_entsoe_source(
source_name=current_app.config.get(
"ENTSOE_DATA_SOURCE_NAME", DEFAULT_DATA_SOURCE_NAME
),
source_type="market",
legacy_source_type="forecasting script",
)
def ensure_data_source_for_derived_data() -> Source:
"""Make sure we have a data source for data derived from ENTSO-E data."""
return _ensure_entsoe_source(
source_name=current_app.config.get(
"ENTSOE_DERIVED_DATA_SOURCE", DEFAULT_DERIVED_DATA_SOURCE
),
source_type="forecasting script",
)
def ensure_transmission_zone_asset(country_code: str) -> Asset:
"""
Ensure a GenericAsset exists to model the transmission zone for which this plugin gathers data.
"""
transmission_zone_type = AssetType.query.filter(
AssetType.name == "transmission zone"
).one_or_none()
if not transmission_zone_type:
current_app.logger.info("Adding transmission zone type ...")
transmission_zone_type = AssetType(
name="transmission zone",
description="A grid regulated & balanced as a whole, usually a national grid.",
)
db.session.add(transmission_zone_type)
ga_name = f"{country_code} transmission zone"
transmission_zone = Asset.query.filter(Asset.name == ga_name).one_or_none()
if not transmission_zone:
current_app.logger.info(f"Adding {ga_name} ...")
transmission_zone = Asset(
name=ga_name,
generic_asset_type=transmission_zone_type,
account_id=None, # public
)
db.session.add(transmission_zone)
db.session.commit()
return transmission_zone
def ensure_sensors(
sensor_specifications: Tuple,
country_code: str,
timezone: str,
) -> Dict[str, Sensor]:
"""
Ensure a GenericAsset exists to model the transmission zone for which this plugin gathers
generation data, then add specified sensors for relevant data we collect.
If new sensors got created, the session has been flushed.
"""
sensors = {}
sensors_created: bool = False
transmission_zone = ensure_transmission_zone_asset(country_code)
for sensor_name, unit, event_resolution, data_by_entsoe in sensor_specifications:
sensor = Sensor.query.filter(
Sensor.name == sensor_name,
Sensor.unit == unit,
Sensor.generic_asset == transmission_zone,
).one_or_none()
if not sensor:
current_app.logger.info(f"Adding sensor {sensor_name} ...")
sensor = Sensor(
name=sensor_name,
unit=unit,
generic_asset=transmission_zone,
timezone=timezone,
event_resolution=event_resolution,
)
db.session.add(sensor)
sensors_created = True
elif sensor.event_resolution != event_resolution:
current_app.logger.warning(
f"The {sensor_name} sensor exists, but has a resolution of {sensor.event_resolution} instead of {event_resolution}. Please refer the 'October 1st 2025 go-live' instructions in `README.md`."
)
sensor.data_by_entsoe = data_by_entsoe
sensors[sensor_name] = sensor
if sensors_created:
db.session.flush()
return sensors
def get_auth_token_from_config_and_set_server_url() -> str:
"""
Read ENTSOE auth token from config, raise if not given.
If test server is supposed to be used, we'll try to read the token
usable for that, and also change the URL.
"""
use_test_server = current_app.config.get("ENTSOE_USE_TEST_SERVER", False)
if use_test_server:
auth_token = current_app.config.get("ENTSOE_AUTH_TOKEN_TEST_SERVER")
entsoe.entsoe.URL = "https://iop-transparency.entsoe.eu/api"
else:
auth_token = current_app.config.get("ENTSOE_AUTH_TOKEN")
entsoe.entsoe.URL = "https://web-api.tp.entsoe.eu/api"
if not auth_token:
click.echo("Setting ENTSOE_AUTH_TOKEN seems empty!")
raise click.Abort
return auth_token
def ensure_country_code_and_timezone(
country_code: Optional[str] = None,
country_timezone: Optional[str] = None,
) -> Tuple[str, str]:
if country_code is None:
country_code = current_app.config.get(
"ENTSOE_COUNTRY_CODE", DEFAULT_COUNTRY_CODE
)
if country_timezone is None:
country_timezone = current_app.config.get(
"ENTSOE_COUNTRY_TIMEZONE", DEFAULT_COUNTRY_TIMEZONE
)
return country_code, country_timezone
def create_entsoe_client() -> EntsoePandasClient:
auth_token = get_auth_token_from_config_and_set_server_url()
client = EntsoePandasClient(api_key=auth_token)
return client
def abort_if_data_empty(data: Union[pd.DataFrame, pd.Series]):
if data.empty:
click.echo(
"Result is empty. Probably ENTSO-E does not provide these forecasts yet ..."
)
raise click.Abort
def abort_if_data_incomplete(
data: Union[pd.DataFrame, pd.Series],
from_time: pd.Timestamp,
until_time: pd.Timestamp,
resolution: pd.Timedelta,
):
expected_periods = int((until_time - from_time) / resolution)
if len(data) < expected_periods:
click.secho(
f"Result is incomplete. Expected {expected_periods} periods but got {len(data)}. Probably ENTSO-E does not provide these forecasts yet ...",
**MsgStyle.ERROR,
)
raise click.Abort
def parse_from_and_to_dates(
from_date: Optional[datetime],
until_date: Optional[datetime],
country_timezone: str,
default_to: str = "today-and-tomorrow", # Can be "tomorrow" or "today"
) -> Tuple[pd.Timestamp, pd.Timestamp]:
"""
Parse CLI options for start and end date (or set default to today and tomorrow) for inout to entsoe-py
Note: we expect only dates as input here, and until_date is inclusive, so we extend it with 24h - so if from_date is equal to until_date, we return 00:00 and 24:00 of that day.
Note: entsoe-py expects time params as pd.Timestamp
"""
tz = pytz.timezone(country_timezone)
now = datetime.now(tz)
today_start = now.replace(hour=0, minute=0, second=0, microsecond=0)
if default_to == "today":
default_start = today_start
default_end = today_start + timedelta(days=1)
elif default_to == "tomorrow":
default_start = today_start + timedelta(days=1)
default_end = default_start + timedelta(days=1)
elif default_to == "today-and-tomorrow":
default_start = today_start
default_end = default_start + timedelta(days=2)
else:
raise ValueError(
f"Invalid default_to value: {default_to}. Expected 'today', 'tomorrow' or 'today-and-tomorrow'."
)
if from_date is None:
start_date = pd.Timestamp(default_start)
else:
start_date = pd.Timestamp(from_date, tzinfo=pytz.timezone(country_timezone))
if until_date is None:
end_date = pd.Timestamp(default_end)
else:
end_date = pd.Timestamp(until_date, tzinfo=pytz.timezone(country_timezone))
# The until_date provided is considered inclusive, so we add 24 hours to include the entire day
end_date += pd.Timedelta(hours=24)
return start_date, end_date
def resample_if_needed(s: pd.Series, sensor: Sensor) -> pd.Series:
inferred_frequency = pd.infer_freq(s.index)
if inferred_frequency is None:
raise ValueError(
"Data has no discernible frequency from which to derive an event resolution."
)
inferred_resolution = pd.to_timedelta(to_offset(inferred_frequency))
target_resolution = sensor.event_resolution
if inferred_resolution == target_resolution:
return s
elif inferred_resolution > target_resolution:
current_app.logger.debug(f"Upsampling data for {sensor.name} ...")
index = pd.date_range(
s.index[0],
s.index[-1] + inferred_resolution,
freq=target_resolution,
inclusive="left",
)
s = s.reindex(index).pad()
elif inferred_resolution < target_resolution:
current_app.logger.debug(f"Downsampling data for {sensor.name} ...")
s = s.resample(target_resolution).mean()
current_app.logger.debug(f"Resampled data for {sensor.name}: \n%s" % s)
return s
def save_entsoe_series(
series: pd.Series,
sensor: Sensor,
entsoe_source: Source,
country_timezone: str,
now: Optional[datetime] = None,
):
"""
Save a series gotten from ENTSO-E to a FlexMeasures database.
"""
if not now:
now = server_now().astimezone(pytz.timezone(country_timezone))
belief_times = (
(series.index.floor("D") - pd.Timedelta("6h"))
.to_frame(name="clipped_belief_times")
.clip(upper=now)
.set_index("clipped_belief_times")
.index
) # published no later than D-1 18:00 Brussels time
bdf = BeliefsDataFrame(
series,
source=entsoe_source,
sensor=sensor,
belief_time=belief_times,
)
# TODO: evaluate some traits of the data via FlexMeasures, see https://github.com/SeitaBV/flexmeasures-entsoe/issues/3
status = save_to_db(bdf)
if status == "success_but_nothing_new":
current_app.logger.info("Done. These beliefs had already been saved before.")
elif status == "success_with_unchanged_beliefs_skipped":
current_app.logger.info("Done. Some beliefs had already been saved before.")
def start_import_log(
import_type: str,
from_time: pd.Timestamp,
until_time: pd.Timestamp,
country_code: str,
country_timezone: str,
) -> Tuple[Logger, datetime]:
log = current_app.logger
log.info(
f"Importing {import_type} data for {country_code} (timezone {country_timezone}), starting at {from_time}, up until {until_time}, from ENTSO-E at {entsoe.entsoe.URL} ..."
)
now = server_now().astimezone(pytz.timezone(country_timezone))
return log, now

51
tools/flexmeasures-entsoe/pyproject.toml Normal file
View File

@@ -0,0 +1,51 @@
[build-system]
requires = ["setuptools>=62", "setuptools_scm[toml]>=6.2", "wheel>=0.29.0"]
build-backend = "setuptools.build_meta"
[project]
name = "flexmeasures-entsoe"
description = "Integrating FlexMeasures with ENTSO-E"
readme = "README.md"
requires-python = ">=3.10"
license = "Apache-2.0"
license-files = [
"LICENSE",
]
authors = [
{name = "Seita BV", email = "nicolas@seita.nl"}
]
keywords = ["smart grid", "renewables", "balancing", "forecasting", "scheduling"]
classifiers = [
"Environment :: Console",
"Environment :: Web Environment",
"Programming Language :: Python",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Framework :: Flask",
"Development Status :: 5 - Production/Stable",
"Operating System :: POSIX :: Linux",
"Operating System :: MacOS :: MacOS X",
"Natural Language :: English"
]
dynamic = ["version", "dependencies"]
[project.urls]
Homepage = "https://github.com/SeitaBV/flexmeasures-entsoe"
Documentation = "https://github.com/SeitaBV/flexmeasures-entsoe"
"Source code" = "https://github.com/SeitaBV/flexmeasures-entsoe"
[project.scripts]
flexmeasures = "flexmeasures.utils.app_utils:flexmeasures_cli"
[tool.setuptools]
include-package-data = true
[tool.setuptools.packages.find]
where = ["."]
include = ["flexmeasures*"]
[tool.setuptools_scm]
local_scheme = "no-local-version"
version_scheme = "guess-next-dev"

23
tools/flexmeasures-entsoe/requirements/Readme.md Normal file
View File

@@ -0,0 +1,23 @@
# Requirements
All FlexMeasures requirements are specified in this directory.
We separate by use case:
- app: All requirements for running the FlexMeasures platform
- test: Additional requirements used for running automated tests
- dev: Additional requirements used for developers (this includes testing)
Also note the following distinction:
## .in files
Here, we describe the requirements. We give the name of a requirement or even a range (e.g. `>=1.0.`).
## .txt files
These files are not to be edited by hand. They are created by `pip-compile` (or `make freeze-deps`).
They are usually not needed, only for development environments. When distributing FlexMeasures with pinned dependency versions and this plugin, only the extra app dependencies (see .in file) need extra care beyond the .txt files.
Each requirement is pinned to a specific version in these files. The great benefit is reproducibility across environments (local dev as well as staging or production).

3
tools/flexmeasures-entsoe/requirements/app.in Normal file
View File

@@ -0,0 +1,3 @@
# only listing extra dependencies that flexmeasures does not have
entsoe-py
timely-beliefs>=3.2.3

11
tools/flexmeasures-entsoe/requirements/dev.in Normal file
View File

@@ -0,0 +1,11 @@
# include flexmeasures as a dev dependency so a fresh environment has it
flexmeasures>=0.28.2
pre-commit
black
flake8
flake8-blind-except
mypy
pytest-runner
types-pytz
setuptools_scm
watchdog

4
tools/flexmeasures-entsoe/requirements/test.in Normal file
View File

@@ -0,0 +1,4 @@
pytest
pytest-flask
pytest-sugar
pytest-cov

7
tools/flexmeasures-entsoe/run_mypy.sh Executable file
View File

@@ -0,0 +1,7 @@
#!/bin/bash
set -e
pip install mypy
# We are checking python files which have type hints
files=$(find . -name \*.py -not \( -path "./venv/*" -prune \) -not \( -path "./.eggs/*" -prune \) )
mypy --follow-imports skip --ignore-missing-imports $files

10
tools/flexmeasures-entsoe/setup.cfg Normal file
View File

@@ -0,0 +1,10 @@
[aliases]
test = pytest
flake8 = flake8
[flake8]
exclude = .git,__pycache__,documentation
max-line-length = 160
max-complexity = 13
select = B,C,E,F,W,B9
ignore = E501, W503, E203

26
tools/flexmeasures-entsoe/setup.py Normal file
View File

@@ -0,0 +1,26 @@
from setuptools import setup
def load_requirements(use_case):
"""
Loading range requirements.
Packaging should be used for installing the package into existing stacks.
We therefore read the .in file for the use case.
.txt files include the exact pins, and are useful for deployments or dev
environments with exactly comparable environments.
"""
reqs = []
with open("requirements/%s.in" % use_case, "r") as f:
reqs = [
req
for req in f.read().splitlines()
if not req.strip() == ""
and not req.strip().startswith("#")
and not req.strip().startswith("-c")
and not req.strip().startswith("--find-links")
]
return reqs
setup(install_requires=load_requirements("app"))