Developer guide

Developer Install

This package can be installed through the following commands:

## Update to real URL !
git clone https://github.com/CNES/slurp
cd slurp
make install
source venv/bin/activate # to go in installed dev environment

Dependencies : git, make

Packages is available in virtualenv in developer mode !

Please use make help to use all preconfigured possibilities.

Coding guide

Here are some rules to apply when developing a new functionality:

Comments: Include a comments ratio high enough and use explicit variables names. A comment by code block of several lines is necessary to explain a new functionality.
Test: Each new functionality shall have a corresponding test in its module’s test file. This test shall, if possible, check the function’s outputs and the corresponding degraded cases.
Documentation: All functions shall be documented (object, parameters, return values).
Use type hints: Use the type hints provided by the typing python module.
Use doctype: Follow sphinx default doctype for automatic API
Quality code: Correct project quality code errors with pre-commit automatic workflow (see below)
Factorization: Factorize the code as much as possible. The command line tools shall only include the main workflow and rely on the slurp python modules.
Be careful with user interface upgrade: If major modifications of the user interface or of the tool’s behaviour are done, update the user documentation (and the notebooks if necessary).
Logging and no print: The usage of the print() function is forbidden: use the logging python standard module instead.
Limit classes: If possible, limit the use of classes as much as possible and opt for a functional approach. The classes are reserved for data modelling if it is impossible to do so using xarray and for the good level of modularity.
Limit new dependencies: Do not add new dependencies unless it is absolutely necessary, and only if it has a permissive license.

Tests

This section explains how to run the project tests with pytest and how to automatically generate documentation from them.

Running the tests

Tests are managed with pytest and use markers to select subsets of tests.

Available markers:

ci: tests for Continuous Integration (CI). Example: .. code-block:: bash

pytest -m ci –config config_tests_ci.json –main_config main_config_tests_ci.json –cov=slurp
validation: validate all algorithms (compute and compare against reference results). Example: .. code-block:: bash

pytest -m validation
features: validate specific features. Example: .. code-block:: bash

pytest -m “features or validation”

Tests are located in the tests/ folder.

Test Documentation

Test prepare module with differents features and different arguments values

tests.test_features.test_features_prepare.test_absolute_analyse_glcm(main_config, features_test_img, output_dir)[source]: Tests the prepare module with glcm analysis enabled glcm: Use a global land cover map to calculate the better number of vegetation cluster to use for mask computation

tests.test_features.test_features_prepare.test_absolute_analyse_glcm_ci(main_config, features_test_img, valid_stack, output_dir)[source]: Run the test_absolute_analyse_glcm with a specified valid_stack (for GithubCI).

tests.test_features.test_features_prepare.test_prepare_update_config(main_config, features_test_img, output_dir)[source]: test that the effective_used_config.json file created during slurp_prepare is correctly updated.

tests.test_features.test_features_prepare.write_command_compute_prepare(nb_workers, main_config, features_test_img, output_dir, valid_stack=None)[source]: Builds a command string to run the prepare module with specified worker count and valid stack.

Test shadow mask with differents features and different arguments values

tests.test_features.test_features_shadowmask.test_absolute_threshold(main_config, features_test_img, output_dir, ref_dir)[source]: Tests the shadow mask computation with absolute thresholding enabled.

tests.test_features.test_features_shadowmask.test_absolute_threshold_ci(main_config, features_test_img, output_dir, ref_dir, valid_stack)[source]: Run the test_absolute_threshold test with a specified valid stack (for GithubCI).

tests.test_features.test_features_shadowmask.test_percentile(percentile, main_config, features_test_img, output_dir, ref_dir)[source]: Tests the shadow mask computation with different percentile values. The percentile value is used to cut histogram and estimate shadow threshold

tests.test_features.test_features_shadowmask.test_percentile_ci(percentile, main_config, features_test_img, output_dir, ref_dir, valid_stack)[source]: Run the test_percentile with a specified valid_stack (for GithubCI).

tests.test_features.test_features_shadowmask.test_percentile_nir_rgb(th_rgb, th_nir, main_config, features_test_img, output_dir, ref_dir)[source]: Tests the shadow mask computation with different threshold values for the nir and rgb bands.

tests.test_features.test_features_shadowmask.test_percentile_nir_rgb_ci(th_rgb, th_nir, main_config, features_test_img, output_dir, ref_dir, valid_stack)[source]: Run test_percentile_nir_rgb_ci with a specified valid_stack (for GithubCI).

tests.test_features.test_features_shadowmask.write_command_compute_shadowmask(nb_workers, main_config, features_test_img, output_dir, ref_dir, valid_stack=None)[source]: Builds a command string to compute a shadow mask using the shadowmask module.

Test urban mask with differents features and different arguments values

tests.test_features.test_features_urbanmask.test_nb_samples(nb_samples_other, nb_samples_urban, main_config, features_test_img, output_dir, ref_dir)[source]: Tests the urban mask computation with different sample counts for other and urban classes. nb_samples_other: Number of samples in other for learning. nb_samples_urban: Number of samples in buildings for learning

tests.test_features.test_features_urbanmask.test_nb_samples_ci(nb_samples_other, nb_samples_urban, main_config, features_test_img, output_dir, ref_dir, valid_stack)[source]: Run test_nb_samples with a specified valid_stack (for GithubCI).

tests.test_features.test_features_urbanmask.test_vegmask_max_value(vegmask_min_value, main_config, features_test_img, output_dir, ref_dir)[source]: Tests the urban mask computation with different vegetation mask minimum values. vegmask_min_value: Vegetation min value for vegetated areas : all pixels with lower value will be predicted

tests.test_features.test_features_urbanmask.test_vegmask_max_value_ci(vegmask_min_value, main_config, features_test_img, output_dir, ref_dir, valid_stack)[source]: Run the test test_vegmask_max_value with a specified valid_stack (for GithubCI).

tests.test_features.test_features_urbanmask.write_command_compute_urbanmask(nb_workers, main_config, features_test_img, output_dir, ref_dir, valid_stack=None)[source]: Builds a command string to compute an urban mask using the urbanmask module.

Test vegetation mask with differents features and different arguments values

tests.test_features.test_features_vegetation.test_debug(main_config, features_test_img, output_dir, ref_dir)[source]: Tests the vegetation mask computation with debug mode enabled.

tests.test_features.test_features_vegetation.test_max_low_veg(main_config, features_test_img, output_dir, ref_dir)[source]: Tests the vegetation mask computation with a specified maximum value for low vegetation clusters. nb_clusters_low_veg: Nb of clusters considered as low vegetation(1-NB_CLUSTERS).

tests.test_features.test_features_vegetation.test_nb_clusters(nb_clusters_veg, nb_clusters_low_veg, main_config, features_test_img, output_dir, ref_dir, valid_stack)[source]: Tests the vegetation mask computation with different numbers of clusters for vegetation and low vegetation. nb_cluster_veg: Nb of clusters considered as vegetation (1-NB_CLUSTERS). nb_clusters_low_veg: Nb of clusters considered as low vegetation(1-NB_CLUSTERS).

tests.test_features.test_features_vegetation.test_percentile(min_ndvi_veg, max_ndvi_noveg, main_config, features_test_img, output_dir, ref_dir)[source]: Tests the vegetation mask computation with different NDVI thresholds for vegetation and non-vegetation. min_ndvi_veg: Minimal mean NDVI value to consider a cluster as vegetation (overload nb clusters choice). max_ndvi_noveg: Maximal mean NDVI value to consider a cluster as non-vegetation (overload nb clusters choice).

tests.test_features.test_features_vegetation.test_texture_mode(main_config, features_test_img, output_dir, ref_dir)[source]: “Tests the vegetation mask computation with texture mode disabled. texture_mode: Labelize vegetation with (yes) or without (no) distinction low/high, ” f”or get all NB_CLUSTERS vegetation clusters without distinction low/high.

tests.test_features.test_features_vegetation.test_vegetation_mask_ci(main_config, features_test_img, output_dir, ref_dir, valid_stack)[source]: Run the vegetation mask with a specified valid_stack (for GithubCI).

tests.test_features.test_features_vegetation.test_vegmask_max_value(main_config, features_test_img, output_dir, ref_dir)[source]: Tests the vegetation mask computation with non-vegetation clusters enabled. non_veg_clusters: Labelize each ‘non vegetation cluster’ as 0, 1, 2 (..) instead of single label (0)

tests.test_features.test_features_vegetation.write_command_compute_vegetationmask(nb_workers, main_config, features_test_img, output_dir, ref_dir, valid_stack=None)[source]: Builds a command string to compute a vegetation mask using the vegetationmask module.

Test water mask with differents features and different arguments values

tests.test_features.test_features_watermask.test_hand_filter(main_config, features_test_img, output_dir, ref_dir)[source]: Tests the water mask computation with HAND filtering enabled: Postprocess with Hand (set to 0 when hand > thresh), incompatible with hand_strict

tests.test_features.test_features_watermask.test_hand_filter_ci(main_config, features_test_img, output_dir, ref_dir, valid_stack)[source]: Run test_hand_filter with a specified valid_stack (for GithubCI).

tests.test_features.test_features_watermask.test_hand_strict(main_config, features_test_img, output_dir, ref_dir)[source]: Tests the water mask computation with HAND strict filtering enabled. hand_strict: Use not(pekelxx) for other (no water) samples.

tests.test_features.test_features_watermask.test_hand_strict_ci(main_config, features_test_img, output_dir, ref_dir, valid_stack)[source]: Run test_hand_strict with a specified valid_stack (for GithubCI).

tests.test_features.test_features_watermask.test_nb_samples(main_config, features_test_img, output_dir, ref_dir, nb_samples_water, nb_samples_other)[source]: Tests the water mask computation with different sample counts for water and other classes. nb_samples_water: Number of samples in water for learning. nb_samples_other: Number of samples in ‘other’ class for learning.

tests.test_features.test_features_watermask.test_nb_samples_auto(main_config, features_test_img, output_dir, ref_dir)[source]: Tests the water mask computation with automatic sample count selection.

tests.test_features.test_features_watermask.test_pekel_filter(main_config, features_test_img, output_dir, ref_dir)[source]: Tests the water mask computation with the Pekel filter disabled: Deactivate postprocess with pekel which only keeps surfaces already known by pekel.

tests.test_features.test_features_watermask.test_pekel_filter_ci(main_config, features_test_img, output_dir, ref_dir, valid_stack)[source]: Run test_pekel_filter with a specified valid_stack (for GithubCI).

tests.test_features.test_features_watermask.test_samples_method(main_config, features_test_img, output_dir, ref_dir, samples_method)[source]: Tests the water mask computation with different sample selection methods. samples_method: Select method for choosing learning samples

tests.test_features.test_features_watermask.test_samples_method_ci(main_config, features_test_img, output_dir, ref_dir, valid_stack, samples_method)[source]: Run test_samples_method with a specified valid_stack (for GithubCI).

tests.test_features.test_features_watermask.test_simple_ndwi_threshold(main_config, features_test_img, output_dir, ref_dir)[source]: Tests the water mask computation with a simple NDWI threshold enabled. simple_ndwi_threshold: Compute water mask as a simple NDWI threshold, useful in arid places where no water is known by Peckel

tests.test_features.test_features_watermask.test_simple_ndwi_threshold_ci(main_config, features_test_img, output_dir, ref_dir, valid_stack)[source]: Run test_simple_nwdi_threshold with specified valid_stack (for GithubCI).

tests.test_features.test_features_watermask.write_command_compute_watermask(nb_workers, main_config, features_test_img, output_dir, ref_dir, valid_stack=None)[source]: Builds a command string to compute a water mask using the watermask module.

Tests for stack mask generation.

tests.test_valid.test_valid_sensor_geom.input_files(input_images)[source]

tests.test_valid.test_valid_sensor_geom.input_images(sensor_geom_dir)[source]

tests.test_valid.test_valid_sensor_geom.predict_pekels(output_dir)[source]

tests.test_valid.test_valid_sensor_geom.predict_wsfs(output_dir)[source]

tests.test_valid.test_valid_sensor_geom.prepare_sensor_geom(main_config, wsf, pekel, output_dir, file, dtm, nb_workers)[source]: Prepares sensor geometry data and validates output files.

tests.test_valid.test_valid_sensor_geom.test_prepare_sensor_geom(main_config, wsf, pekel, output_dir, ref_dir, input_files)[source]: Tests the sensor geometry preparation and validates output masks.

tests.test_valid.test_valid_sensor_geom.test_validation_sensor_geom_pekel(predict_pekels, ref_dir)[source]: Tests the validation of Pekel masks generated from sensor geometry.

tests.test_valid.test_valid_sensor_geom.test_validation_sensor_geom_wsf(predict_wsfs, ref_dir)[source]: Tests the validation of WSF masks generated from sensor geometry.

Tests for shadowmask generation.

tests.test_valid.test_valid_shadowmask.compute_shadowmask(main_config, output_dir, ref_dir, file, nb_workers, valid_stack=None)[source]: Computes the shadow mask for a given image and validates output.

tests.test_valid.test_valid_shadowmask.input_files(data_dir)[source]

tests.test_valid.test_valid_shadowmask.prepare_shadowmask(main_config, output_dir, file, nb_workers)[source]: Prepares the valid stack for shadow mask computation.

tests.test_valid.test_valid_shadowmask.test_prepare_computation_and_validation_shadowmask(main_config, output_dir, ref_dir, input_files)[source]: Tests the full workflow of preparation, computation, and validation of shadow mask for each input file.

Tests for stack mask generation.

tests.test_valid.test_valid_stackmask.compute_stackmask(file, main_config, output_dir, data_dir, ref_dir, nb_workers)[source]: Computes the stack mask for a given image and validates output.

tests.test_valid.test_valid_stackmask.input_files(data_dir)[source]

tests.test_valid.test_valid_stackmask.test_computation_and_validation_stackask(input_files, main_config, output_dir, data_dir, ref_dir)[source]: Tests both computation and validation of stack mask for each input file.

tests.test_valid.test_valid_stackmask.test_computation_stackmask_ci(features_test_img, main_config, output_dir, valid_stack)[source]: Tests the computation of stack mask in a CI environment using test input masks.

Tests for urbanmask generation.

tests.test_valid.test_valid_urbanmask.compute_urbanmask(file, main_config, nb_workers, ref_dir, valid_stack=None, ndvi=None, ndwi=None, wsf=None)[source]: Computes the urban mask for a given image and validates output.

tests.test_valid.test_valid_urbanmask.input_files(data_dir)[source]

tests.test_valid.test_valid_urbanmask.prepare_urbanmask(file, main_config, output_dir, wsf, nb_workers)[source]: Prepares the valid stack, NDVI, and NDWI files for urban mask computation.

tests.test_valid.test_valid_urbanmask.test_prepare_computation_and_validation_urbanmask(input_files, ref_dir, main_config, output_dir, wsf)[source]: Tests the full workflow of preparation, computation, and validation of urban mask for each input file.

Tests for vegetationmask generation.

tests.test_valid.test_valid_vegetationmask.compute_vegetationmask(file, main_config, output_dir, ref_dir, nb_workers, valid_stack=None, ndvi=None, ndwi=None, texture=None)[source]: Computes the vegetation mask for a given image and validates output.

tests.test_valid.test_valid_vegetationmask.input_files(data_dir)[source]

tests.test_valid.test_valid_vegetationmask.prepare_vegetationmask(file, main_config, output_dir, nb_workers)[source]: Prepares the valid stack, NDVI, NDWI, and texture files for vegetation mask computation.

tests.test_valid.test_valid_vegetationmask.test_prepare_computation_and_validation_vegetationmask(input_files, main_config, output_dir, ref_dir)[source]: Tests the full workflow of preparation, computation, and validation of vegetation mask for each input file.

Tests for watermask generation.

tests.test_valid.test_valid_watermask.compute_watermask(file, main_config, output_dir, ref_dir, nb_workers, valid_stack=None, ndvi=None, ndwi=None, pekel=None, hand=None)[source]: Computes the water mask for a given image and validates output.

tests.test_valid.test_valid_watermask.input_files(data_dir)[source]

tests.test_valid.test_valid_watermask.prepare_watermask(file, main_config, output_dir, pekel, hand, nb_workers)[source]: Prepares the valid stack, NDVI, and NDWI files for water mask computation.

tests.test_valid.test_valid_watermask.test_prepare_computation_and_validation_watermask(input_files, main_config, output_dir, ref_dir, pekel, hand)[source]: Tests the full workflow of preparation, computation, and validation of water mask for each input file.

Documentation

slurp documentation can be generated with following command:

make docs

It cleans documentation from docs/build/ directory and builds the sphinx documentation from docs/source/ into docs/build/:

sphinx-build -M clean docs/source/ docs/build
sphinx-build -M html docs/source/ docs/build

Pre-commit validation

A pre-commit validation is installed with code quality tools (see below). It is installed automatically by make install-dev command.

Here is the way to install it manually:

$ pre-commit install -t pre-commit
$ pre-commit install -t pre-push

This installs the pre-commit hook in .git/hooks/pre-commit and .git/hooks/pre-push from .pre-commit-config.yaml file configuration.

It is possible to test pre-commit before commiting:

$ pre-commit run --all-files                # Run all hooks on all files
$ pre-commit run --files slurp/__init__.py   # Run all hooks on one file
$ pre-commit run pylint                     # Run only pylint hook
$ pre-commit run --hook-stage push --all-files # Run with push hook

Code quality

The project uses Isort, Black, Flake8 and Pylint quality code checking.

Use the following command in virtualenv to check the code with these tools:

$ make lint

Use the following command to format the code with isort and black:

$ make format

Isort

Isort is a Python utility / library to sort imports alphabetically, and automatically separated into sections and by type.

isort configuration is done in .pyproject.toml Isort manual usage examples:

$ cd slurp_HOME
$ isort --check slurp tests  # Check code with isort, does nothing
$ isort --diff slurp tests   # Show isort diff modifications
$ isort slurp tests          # Apply modifications

Isort messages can be avoided when really needed with “# isort:skip” on the incriminated line.

Black

Black is a quick and deterministic code formatter to help focus on the content.

black configuration is done in .pyproject.toml

If necessary, Black doesn’t reformat blocks that start with “# fmt: off” and end with # fmt: on, or lines that ends with “# fmt: skip”. “# fmt: on/off” have to be on the same level of indentation.

Black manual usage examples:

$ cd slurp_HOME
$ black --check slurp tests  # Check code with black with no modifications
$ black --diff slurp tests   # Show black diff modifications
$ black slurp tests          # Apply modifications

Flake8

Flake8 is a command-line utility for enforcing style consistency across Python projects. By default it includes lint checks provided by the PyFlakes project, PEP-0008 inspired style checks provided by the PyCodeStyle project, and McCabe complexity checking provided by the McCabe project. It will also run third-party extensions if they are found and installed.

flake8 configuration is done in setup.cfg

Flake8 messages can be avoided (in particular cases !) adding “# noqa” in the file or line for all messages. It is better to choose filter message with “# noqa: E731” (with E371 example being the error number). Look at examples in source code.

Flake8 manual usage examples:

$ cd slurp_HOME
$ flake8 slurp tests           # Run all flake8 tests

Pylint

Pylint is a global linting tool which helps to have many information on source code.

pylint configuration is done in dedicated .pylintrc file.

Pylint messages can be avoided (in particular cases !) adding “# pylint: disable=error-message-name” in the file or line. Look at examples in source code.

Pylint manual usage examples:

$ cd slurp_HOME
$ pylint tests slurp       # Run all pylint tests
$ pylint --list-msgs          # Get pylint detailed errors informations

Bug report

Any proven or suspected malfunction should be traced in a bug report, the latter being an issue in the slurp github repository.

Don’t hesitate to do so: It is best to open a bug report and quickly resolve it than to let a problem remain in the project. Notifying the potential bugs is the first way for contributing to a software.

In the problem description, be as accurate as possible. Include:

The procedure used to initialize the environment
The incriminated command line or python function
The content of the input and output configuration files (content.json)

Contributing workflow

Any code modification requires a Merge Request. It is forbidden to push patches directly into master (this branch is protected).

It is recommended to open your Merge Request as soon as possible in order to inform the developers of your ongoing work. Please add WIP: before your Merge Request title if your work is in progress: This prevents an accidental merge and informs the other developers of the unfinished state of your work.

The Merge Request shall have a short description of the proposed changes. If it is relative to an issue, you can signal it by adding Closes xx where xx is the reference number of the issue.

Likewise, if you work on a branch (which is recommended), prefix the branch’s name by xx- in order to link it to the xx issue.

slurp’s Classical workflow is :

Check Licence and sign Contribution license agreement (Individual or Corporate)
Create an issue (or begin from an existing one)
Create a Merge Request from the issue: a MR is created accordingly with WIP:, Closes xx and associated xx-name-issue branch
Modify slurp’s code from a local working directory or from the forge (less possibilities)
Git add, commit and push from local working clone directory or from the forge directly
Follow Conventional commits specifications for commit messages
Beware that pre-commit hooks can be installed for code analysis (see below pre-commit validation).
Launch the tests with pytest on your modifications (or don’t forget to add ones).
When finished, change your Merge Request name (erase WIP: in title ) and ask to review the code.

Contribution license agreement

slurp requires that contributors sign out a Contributor LicenseAgreement. The purpose of this CLA is to ensure that the project has the necessary ownership or grants of rights over all contributions to allow them to distribute under the chosen license (Apache License Version 2.0)

To accept your contribution, we need you to complete, sign and email to cars@cnes.fr an Individual Contributor LicensingAgreement (ICLA) form and a Corporate Contributor Licensing Agreement (CCLA) form if you are contributing on behalf of your company or another entity which retains copyright for your contribution.

The copyright owner (or owner’s agent) must be mentioned in headers of all modified source files and also added to the AUTHORS.md file.