cariflex/tools/EVerest-main/tests

EVerest tests

This folder contains the basic test functions for EVerest's e2e testing. You can either extend the '*_tests.py' files in the "EVerest/tests/core_tests" folder or write your own!

Prerequisites

In order to run any of the e2e tests, you need to have EVerest compiled and installed on your system. Please also make sure to install the python requirements.

cd EVerest/
cmake -S . -B build -DBUILD_TESTING=ON
cmake --build build --target install --parallel -j$(nproc)
. build/venv/bin/activate
cmake --build build --target everestpy_pip_install_dist # install everestpy
cmake --build build --target everest-testing_pip_install_dist # install everest-testing
cmake --build build --target iso15118_pip_install_dist # install iso15118 for ev side simulation
python3 -m pip install "aiofile>=3.7.4"
python3 -m pip install "netifaces>=0.11.0"

Execute locally

All test suites are run via the unified run-tests.sh script:

cd ~/checkout/everest-workspace/EVerest/tests

./run-tests.sh all          # all tests (core, framework, async, OCPP)
./run-tests.sh core         # core tests only
./run-tests.sh framework    # framework tests only
./run-tests.sh asyncapi        # async API tests only
./run-tests.sh integration  # core + framework + async
./run-tests.sh ocpp         # all OCPP tests (1.6, 2.0.1, 2.1)
./run-tests.sh --serial all # run serially (no parallelism)
./run-tests.sh -j4 all      # limit to 4 parallel workers
./run-tests.sh --help       # show all options

The script automatically sets up network isolation for parallel ISO 15118 tests (requires sudo or CAP_NET_ADMIN) and falls back to sequential execution otherwise.

After execution result.xml and report.html are written to the tests/ directory.

Running individual tests directly

For a single test file or test case, invoke pytest directly from the tests/ directory:

cd ~/checkout/everest-workspace/EVerest/tests

# Run a single test file
python3 -m pytest core_tests/smoke_tests.py

# Run a single test by name
python3 -m pytest core_tests/smoke_tests.py::test_iso15118_ac_session

Network isolation is not set up automatically in this mode — ISO 15118 tests will run sequentially via their xdist_group marker.

Add own test sets

To create own test sets, you need to write a new python file in the "EVerest/tests/core_tests/" folder. A basic file template would look like this:

import logging
import pytest

from everest.testing.core_utils.fixtures import *
from validations.base_functions import wait_for_and_validate_event
from everest.testing.core_utils.test_control_module import TestControlModule
from everest.testing.core_utils.everest_core import EverestCore
from validations.user_functions import *

@ pytest.mark.asyncio
async def test_001_my_first_test(everest_core: EverestCore, 
                                 test_control_module: TestControlModule):
    logging.info(">>>>>>>>> test_001_my_first_test <<<<<<<<<")
    everest_core.start()

    assert await wait_for_and_validate_event(test_control_module, 
                                             exp_event='my_event', 
                                             exp_data={"my_data":"test"},
                                             validation_function=my_validation_function,
                                             timeout=30) 

*Note: All test functions' names NEED to start with "test" (see pytest documentation: https://docs.pytest.org/)

In the above example you then would also have to write a (user-)validation function by the name of "my_validation_function()" in the "EVerest/tests/core_tests/validations/user_functions.py" file.

For this, you can utilize helper functions from the "EVerest/tests/core_tests/validations/base_functions.py" file (e.g. "get_key_if_exists()" to traverse and retrieve event data from an event-object).

Attention: When you change something in the test- or validation files, please do not forget to run "make install" on EVerest before starting the tests, as otherwise the changes are not reflected in the test run!

Controlling more functionality with PyTestControlModule

If you would like to receive other everest-internal data, send commands to other EVerest modules or would like to change the received events for everest-testing, you can modify the PyTestControlModule in the "EVerest/modules/PyTestControlModule" folder.

Attention: Please be aware, though, that the use of this module is preliminary and changes in the everest-framework may render this module obsolete in the future!

Troubleshooting

1. Problem: Multiple concatenated tests that use the PyTestControlModule (in standalone configuration) result in an exception and subsequent crash of everest-testing.

Cause: There is currently no way to unload a module in EVerest. But because pytest runs out of the same context for all tests in a testset, a once loaded PyTestControlModule stays active for the rest of the pytest/everest-testing run. When a new test tries to load a second instance of the PyTestControlModule it collides with the already running instance and causes everest-testing to crash.

Solution: Unfortunately, this requires a change in the everest-framework. This change is currently under development, but not yet merged into main. For the moment, there is a workaround available: Use a new test-set (another python file containing a new set of tests) for every test that requires control via the PyTestControlModule. As soon as a fix for the underlying issue is merged, this will be updated!

2. Problem: On an unmodified run of "basic_charging_tests.py" the test test_001_charge_defined_ammount fails and reports an amount of 0.0kWh charged.

Cause: Currently there is a bug in the EVSE manager implementation in EVerest, causing a race-condition between resetting the "amount charged" parameter and the event reporting for a "transaction_finished" event. Sometimes it reports correctly, other times it reports a value of 0.0kWh charged.

Solution: This problem is known and a fix is under way. For the moment, just re-run the test. Usually, this should then produce a passing result.

Code coverage

To activate code coverage, you need to configure the project with BUILD_TESTING enabled. Furthermore, gcovr needs to be installed.

Generating coverage reports

Before generating a coverage report you will typically need to run some executables or tests, which contain or link to code, that generates coverage statistics.

After that, just run:

cmake --build build --target everest-core_create_coverage
# or ninja -C build everest-core_create_coverage

The generated index.html will be available at build/everest-core_create_coverage/index.html.

Note: the code coverage for a specific compilation unit (cpp file) will accumulate if you run multiple executables that use this compilation unit. The code coverage is recorded in gcda files. If you want to get separate coverage reports for different executables, these gcda files need to be deleted for each new run.

Adding code/targets to code coverage

By default, all cpp modules are added automatically to a list of targets that will be compiled with coverage flags. If additional libraries or tests need to be added to code coverage, you should use the following functions in your cmake code:

ev_register_library_target(LIBRARY_TARGET_NAME)
ev_register_test_target(TEST_TARGET_NAME)

Conventions

For the ev_register_*_target functions, call these function close to their scope of detail - meaning near add_library for libraries like:

add_library(evse_security_conversions STATIC)
add_library(everest::evse_security_conversions ALIAS evse_security_conversions)
ev_register_library_target(evse_security_conversions)

and near add_test for tests like:

add_test(${TLS_GTEST_NAME} ${TLS_GTEST_NAME})
ev_register_test_target(${TLS_GTEST_NAME})

and not scattered around.

---

By default, all directories ending with tests will be excluded from the code coverage report (because we're not interested in the code coverage of the unit test code itself). So when adding unit tests for code coverage, place them into a tests folder.

Troubleshooting

no_working_dir_found: while developing code (e.g. an example.cpp), it might happen, that the created object file example.cpp.o will get out of sync with its corresponding example.cpp.gcno (gcov note file) and/or example.cpp.gcda (gcov data file). When creating a report, gcovr will complain, because it can't find the corresponding example.cpp.o file for a stale example.cpp.gcno file. It might also not be able to find the example.cpp.gcno file for a stale example.cpp.gcda file or an older gcda file might got out of sync due to a newer gcno file and so forth. Do not simply ignore these warnings/errors, as the report might not be accurate anymore.

The current sledgehammer approach is to manually delete the files in question or just do (be careful with rm commands and wildcards!):

find ./build -name "*.gcno" -delete
find ./build -name "*.gcda" -delete
ninja -C build clean
ninja -C build
# run code
# create coverage

We're currently trying to write some tooling around this to make this more usable.