Skip to content

GitLab

Commit 35dc5ae3 authored by Lukas Riedel's avatar Lukas Riedel
Browse files
Options

Merge branch 'master' into update-ci-config-for-new-runners 

Remove 'unit_tests_parallel' target and add parallel unit tests to
'unit_tests' target.
parents cabfeba9 372da0ed
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 4236 additions and 1980 deletions
.gitignore
View file @ 35dc5ae3
......@@ -5,6 +5,7 @@ build-cmake/
python/dorie/wrapper/pf_from_file.py
python/dorie/wrapper/test_dorie.py
python/dorie/cli/cmds.py
test/maps/cell_ids.h5
# Ignore temporary and auto-generated files #
*~
......
.gitlab-ci.yml
View file @ 35dc5ae3
......@@ -12,6 +12,7 @@ variables:
-DDUNE_PYTHON_ALLOW_GET_PIP=True
MAKE_FLAGS:
-j $CPUS_MULTICORE
RUN_IN_DUNE_ENV: $CI_PROJECT_DIR/build-cmake/run-in-dune-env
image: $DUNE_ENV_IMAGE
......
CHANGELOG.md
View file @ 35dc5ae3
......@@ -37,12 +37,7 @@
to call `FlowParameters::bind` with a grid entity to cache the appropriate
data. All classes and functions querying parameter data have been updated.
The old parameterization classes located in
[`param_base.hh`](dune/dorie/solver/param_base.hh),
[`param_factory.hh`](dune/dorie/solver/param_factory.hh), and
[`param_van_genuchten.hh`](dune/dorie/solver/param_van_genuchten.hh)
are still used for reading in data for the new storage scheme.
The respective objects are freed once a simulation commences.
The old parameterization classes have been removed.
* `RichardsSimulation` now uses one vector of coefficients instead of two, which
now is a `shared_ptr` instead a `unique_ptr`.
* Every `VTKAdapter` is now able to be used as `GridFunctionAdapter` of the
......@@ -72,6 +67,23 @@
* The `dorie` CLI has been moved into the DORiE Python package. It can now only
be invoked after activating the DUNE virtual environment. The usage
instructions in docs and `README.md` have been updated accordingly.
* Soil architecture and parameter input has been completely revised.
DORiE now expects a YAML file containing parameterization data and separate
domain mapping data which can be supplied through the input GMSH file, or as
H5 dataset. See the documentation for more information. Several keys in the
main routine config file have been changed.
See [#86](https://ts-gitlab.iup.uni-heidelberg.de/dorie/dorie/issues/86).
* The Parameter Field Generator (PFG) module has been revised and does not
write parameterization data anymore.
It can be used to create mappings for heterogeneous media through the
binary converter, or continuous scaling fields (feature pending).
Several keys in the PFG config file have been changed.
See [#86](https://ts-gitlab.iup.uni-heidelberg.de/dorie/dorie/issues/86).
### Deprecated
......
README.md
View file @ 35dc5ae3
......@@ -242,20 +242,24 @@ by calling
dorie create
DORiE encapsulates two main routines, the Parameter Field Generator (PFG) and
the main program routine. Each takes a single `.ini` configuration file as
argument.
Tweak the parameters of `parfield.ini` to your liking and then call
DORiE implements a lightweight wrapper around the `dune-randomfield`
generator. You can use it to easily create a heterogeneous soil architecture.
This step is optional. Tweak the parameters of `parfield.ini` to your liking
and then call
dorie pfg parfield.ini
to create a parameter field file. Do the same with the `config.ini`, while
ensuring that you use the parameter field file you just created. Then call
The DORiE main routine is executed with the `run` command.
Tweak the parameters of `config.ini` to your liking. You will need to
reference several additional input files for soil parameters, boundary
conditions, GMSH grid files (optional), and grid mappings (optional).
Refer to the documentation for further information.
Once prepared, call
dorie run config.ini
to execute the main program.
to execute the solver.
## Troubleshooting
CMake heavily caches the results of its configuration process. In case you encounter errors or strange behavior, especially after an update, you should delete the DORiE build folder (called `build-cmake` by default) and re-install DORiE using `dunecontrol`.
......
cmake/modules/DorieTesting.cmake
View file @ 35dc5ae3
......@@ -10,6 +10,13 @@ add_custom_target(system_tests
COMMAND ctest --output-on-failure --exclude-regex ^ut.+$
)
# create the mapping datafile as preparation for all tests
add_custom_target(prepare_testing
COMMAND ${DUNE_PYTHON_VIRTUALENV_EXECUTABLE} ${PROJECT_SOURCE_DIR}/test/maps/create_param_maps.py ${PROJECT_SOURCE_DIR}/test/maps/cell_ids.h5
)
add_dependencies(system_tests prepare_testing)
add_dependencies(unit_tests prepare_testing)
#
# .. cmake_function:: add_coverage_links
#
......@@ -134,7 +141,7 @@ function(dorie_add_metaini_test)
# configure meta ini file or just copy.
get_filename_component(metaini-name ${SYSTEM_TEST_METAINI} NAME_WE)
get_filename_component(metaini-extension ${SYSTEM_TEST_METAINI} EXT)
if(metaini-extension EQUAL ".mini.in")
if(metaini-extension STREQUAL ".mini.in")
configure_file(${SYSTEM_TEST_METAINI} ${metaini-name}.mini)
set(SYSTEM_TEST_METAINI "${metaini-name}.mini")
else()
......
doc/default_files/field-parameters.xml
View file @ 35dc5ae3
......@@ -32,215 +32,108 @@ adding an empty line, make text **bold** or ``monospaced``.
<dorie>
<category name="general" hidden="true">
<parameter name="generator">
<definition>
Specifies which parameter field generator module should be used.
</definition>
<values> image, csv, fft, hdf5 </values>
<suggestion> csv </suggestion>
<comment> One of: image, csv, fft, hdf5 </comment>
</parameter>
<parameter name="inputFile">
<definition>
Path to file holding the parameter distribution data (i.e., image, CSV file,
or HDF5 file). Has no effect when using FFT generator.
</definition>
<values> path </values>
</parameter>
<parameter name="outputFile">
<definition>
Output file name of the parameter field array.
Output file path of the parameter field array.
</definition>
<values> path </values>
<suggestion> field.h5 </suggestion>
</parameter>
<parameter name="parameterization">
<parameter name="dataset">
<definition>
Parameterization to be used when mapping field value to parameter.
Essentially sets which ``parameters.&lt;parameterization&gt;.&lt;x&gt;``
values should be used.
The dataset name inside the ``outputFile``.
</definition>
<values> vanGenuchten </values>
<suggestion> vanGenuchten </suggestion>
<values> path </values>
</parameter>
<parameter name="overwrite">
<definition>
Whether an existing parameter field array should be overwritten.
<parameter name="converter">
<definition> Identifier of the random field converter. The converter is
applied after the random field is created and modifies it.
</definition>
<values> bool </values>
<suggestion> false </suggestion>
<values> none, binary </values>
<suggestion> binary </suggestion>
<comment> none, binary </comment>
</parameter>
</category>
<category name="parameters.vanGenuchten.1" hidden="true">
<parameter name="theta_r">
<parameter name="tempDir">
<definition>
Residual water content of medium 1, used in the Mualem-van Genuchten
parameterization. Default values for sand, according to Soil Physics, p. 92.
Temporary output directory of the dune-randomfield module.
Note that this only accepts a folder (file names are fixed).
</definition>
<values> float </values>
<suggestion> 0.03 </suggestion>
</parameter>
<parameter name="theta_s">
<definition> Saturated water content. </definition>
<values> float </values>
<suggestion> 0.32 </suggestion>
</parameter>
<parameter name="alpha">
<definition> log10(&alpha;) in the van Genuchten parameterization. Units: [1/m] </definition>
<values> float </values>
<suggestion> -2.3 </suggestion>
</parameter>
<parameter name="tau">
<definition> &tau; in the Mualem parameterization of the hydraulic permeability. </definition>
<values> float </values>
<suggestion> -1.1 </suggestion>
</parameter>
<parameter name="n">
<definition> n in the van Genuchten parameterization. </definition>
<values> float </values>
<suggestion> 4.17 </suggestion>
</parameter>
<parameter name="k0">
<definition> Saturated hydraulic conductivity. Units: [m/s] </definition>
<values> float </values>
<suggestion> 2.2E-5 </suggestion>
<values> path </values>
<suggestion> /tmp/fft_generator/ </suggestion>
</parameter>
</category>
<category name="parameters.vanGenuchten.2" hidden="true">
<parameter name="theta_r">
<definition>
Residual water content of medium 2, used in the Mualem-van Genuchten
parameterization. Default values for silt, according to Soil Physics, p. 92.
</definition>
<values> float </values>
<suggestion> 0.01 </suggestion>
</parameter>
<parameter name="theta_s">
<definition> Saturated water content. </definition>
<values> float </values>
<suggestion> 0.41 </suggestion>
</parameter>
<parameter name="alpha">
<definition> log10(&alpha;) in the van Genuchten parameterization. Units: [1/m] </definition>
<values> float </values>
<suggestion> -0.7 </suggestion>
</parameter>
<parameter name="tau">
<definition> &tau; in the Mualem parameterization of the hydraulic permeability. </definition>
<values> float </values>
<suggestion> .0 </suggestion>
</parameter>
<parameter name="n">
<definition> n in the van Genuchten parameterization. </definition>
<values> float </values>
<suggestion> 1.3 </suggestion>
<category name="grid">
<parameter name="dimensions">
<definition> Physical dimensions of the field. </definition>
<suggestion> 2 </suggestion>
<values> 2, 3 </values>
</parameter>
<parameter name="k0">
<definition> Saturated hydraulic conductivity. Units: [m/s] </definition>
<values> float </values>
<suggestion> 1E-5 </suggestion>
<parameter name="extensions">
<definition> Physical extensions of the created random field.
This information is not used by DORiE but required for meaningful
correlation lengths (see ``grid.corrLength`` below).
</definition>
<values> int &times; int (&times; int) </values>
<suggestion> 1 1 </suggestion>
</parameter>
</category>
<category name="generator">
<parameter name="millerSimilarity" hidden="true">
<definition>
Whether to use Miller similarity or not. If set to true, only
the first specified parameter set is used. Also, make sure to set a proper
variance.
<parameter name="cells">
<definition> Resolution (in grid cells) of the created random field.
</definition>
<values> bool </values>
<suggestion> false </suggestion>
<values> int &times; int (&times; int) </values>
<suggestion> 100 100 </suggestion>
</parameter>
</category>
<parameter name="dimensions">
<definition>
Spatial dimension of the created field. 3-dimensional fields are only supported
by the fft and hdf5 generators.
<category name="stochastic">
<parameter name="seed">
<definition> Seed of all random number generators involved.
The same seed will lead to the same random field, if the application
is used with the same settings
(compiler, number of parallel processes, ...)
</definition>
<values> 2, 3 </values>
<suggestion> 2 </suggestion>
<values> int </values>
<suggestion> 1 </suggestion>
</parameter>
<parameter name="variance">
<definition>
Variance of the resulting field, if ``millerSimilarity`` is used.
Variance of the resulting field.
</definition>
<values> float </values>
<suggestion> 0.2 </suggestion>
</parameter>
<parameter name="extensions">
<definition>
Physical extensions of the created field.
<parameter name="corrLength">
<definition> Physical correlation lengths for axiparallel anisotropy.
Specify one value for each dimension of the field.
</definition>
<values> float &times; float (&times; float) </values>
<suggestion> 1 1 </suggestion>
</parameter>
</category>
<category name="generator.fft">
<parameter name="outputPath">
<definition>
Path to the output folder of the generated field. Note that this only accepts
a folder, not a file name (file names are fixed).
</definition>
<values> path </values>
<suggestion> /tmp/fft_generator/ </suggestion>
</parameter>
<parameter name="N">
<definition> Suggested size (in cells) of the created random field. </definition>
<values> int &times; int (&times; int) </values>
<suggestion> 1000 1000 </suggestion>
</parameter>
<parameter name="correlationLengths">
<definition> Correlation lengths in each dimension of the random field, in
meters. </definition>
<values> float &times; float (&times; float) </values>
<suggestion> .1 .05 </suggestion>
</parameter>
<parameter name="seed">
<definition> Seed of all random number generators involved. Ensures reproducibility
(the same seed will always lead to the same random field). </definition>
<values> int </values>
<suggestion> 123456789 </suggestion>
</parameter>
<parameter name="covariance">
<definition> Defines which variogram function should be used to create the
random field. </definition>
<values> exponential, gaussian, spherical </values>
<values> exponential, gaussian, spherical, whiteNoise </values>
<suggestion> gaussian </suggestion>
<comment> exponential, gaussian, spherical </comment>
<comment> exponential, gaussian, spherical, whiteNoise </comment>
</parameter>
</category>
<category name="generator.image" hidden="true">
</category>
<category name="generator.csv" hidden="true">
</category>
<category name="generator.h5" hidden="true">
<category name="converter.binary">
<parameter name="indices">
<definition> Binary values A and B to transform the field into.
The field will be set to A where its value is less equal 0, and B where
its value is greater 0.
</definition>
<values> int </values>
<suggestion> 0 1 </suggestion>
</parameter>
</category>
</dorie>
doc/default_files/parameters.xml
View file @ 35dc5ae3
......@@ -135,6 +135,29 @@ adding an empty line, make text **bold** or ``monospaced``.
<values> int </values>
<suggestion> 0 </suggestion>
</parameter>
<parameter name="mappingFile">
<definition> The H5 file containing the mapping from cell to medium index.
Specify the dataset inside the file with the next key. Leave empty or
set to ``None`` for a global homogeneous medium. In this case,
``grid.globalIndex`` has to be specified.
</definition>
<values> path </values>
</parameter>
<parameter name="mappingFileDataset">
<definition> The H5 dataset inside ``grid.mappingFile`` containing the
mapping from cell to medium index.
</definition>
<values> path </values>
</parameter>
<parameter name="globalIndex">
<definition> The medium index to use for all grid cells, if
``mappingFile`` is unset or ``None``.
</definition>
<values> int </values>
</parameter>
</category>
<category name="boundary">
......@@ -248,33 +271,10 @@ adding an empty line, make text **bold** or ``monospaced``.
</category>
<category name="parameters">
<parameter name="arrayFile">
<definition> Path to an HDF5 array file containing all parameters used in the chosen
parameterization. Can be created using the ``dorie pfg`` command. </definition>
<values> path </values>
<comment> Create with 'dorie pfg' </comment>
</parameter>
<parameter name="scale">
<definition> Scaling factor of the parameter field. A value &gt; 1 zooms into
the field; a value &lt; 1 zooms out. This may of course change the statistical
properties of the field (e.g. correlation lengths).
<parameter name="file">
<definition> YAML file containing the parameter definitions.
</definition>
<values> float &times; float (&times; float) </values>
<suggestion> 1 1 </suggestion>
</parameter>
<parameter name="offset">
<definition> The parameter field is shifted by this value (in physical units) </definition>
<values> float &times; float (&times; float) </values>
<suggestion> 0 0 </suggestion>
</parameter>
<parameter name="interpolation">
<definition> DEPRECATED: Sets the interpolation behavior when querying parameter field values
at a certain grid cell. </definition>
<values> nearest </values>
<suggestion> nearest </suggestion>
<values> path </values>
</parameter>
</category>
......
doc/index.rst
View file @ 35dc5ae3
......@@ -50,6 +50,7 @@ Manual
man-config-file
man-bcfile
man-cookbook
man-parameter-file
public-api
.. _c-reference:
......
doc/man-parameter-file.rst
View file @ 35dc5ae3
***********************************
Parameter Field File Specifications
***********************************
Soil Parameters and Architecture
================================
The parameter field file is used to supply soil parameter information to the
simulation. It is supplied as
`H5 File <https://support.hdfgroup.org/HDF5/doc/H5.intro.html>`_.
Specifying the domain properties requires input of the soil architecture
(in relation to the grid) and of the soil parameters. The latter incorporate
the *subscale physics*, the representation of the soil hydraulic properties
below the REV scale. DORiE requires several input files for retrieving this
information, depending on the type of grid used for the computation.
The Parameter Field Generator module always generates files compliant to this
specification.
A YAML_ parameter file is always required. It specifies a set of soil
parameterizations, along with a medium index. This index is used to reference
the medium specified in the architecture files. There are two types of these:
If you insert an unstructured grid with GMSH, you have to incorporate the
medium information. If you create a rectangular grid, you need to add a mapping
dataset, specifying the medium index for each (initial) grid cell.
The structure of this file is as follows:
YAML Parameter File
-------------------
This file is used to specify the parameterization for each medium inside the
simulated domain. It follows a simple hierarchical syntax. The file path and
name must be specified via the ``parameters.file`` key of the
:doc:`config file <man-config-file>`.
* H5Group: `parameters`
The top-level mapping must contain the key ``volumes``. This key contains a
sequence of arbitrary parameterization names. These, in turn, contain the
parameterization ``type``, the medium ``index``, and the actual ``parameters``.
Attributes:
.. code-block:: yaml
* array: `extensions` : n-dimensional spatial extensions of the field
* bool: `millerSimilarity` : whether Miller scaling shall be used
* string: `parameterization` : Only "vanGenuchten" supported
* double: `variance` : Variance of the Gaussian random field for Miller scaling
volumes:
<name-0>:
type: <type>
index: <index>
parameters:
<param-name-0>: <value>
# ...
# ...
* H5Group: `vanGenuchten`
You find a documentation of the parameterizations implemented in DORiE
along with the parameters they require below.
The parameterization ``type`` must match the static ``type`` member of one of
them. Likewise, the parameters must match the names of parameters
these objects use. The medium index must be an **integer**.
Datasets for each parameter in a n-dimensional field. The extensions of
all datasets must match. Additionally, there is the `raw_field` dataset,
specifying the natural logarithm of the Miller scaling factor.
Medium Mapping Dataset
----------------------
This dataset is required when creating a rectangular grid, whether adaptive or
not. It has to be realized as dataset inside an HDF5_ file. The dataset must
have the same dimensions as the target grid, and must specify the medium index
for every cell of this grid. As all medium indices are integers, the dataset
must contain **integers, not floats**! An HDF5 file with such a dataset can
easily be created with the Python module ``h5py``.
* H5Dataset: `alpha`
* H5Dataset: `k0`
* H5Dataset: `n`
* H5Dataset: `tau`
* H5Dataset: `theta_r`
* H5Dataset: `theta_s`
* H5Dataset: `raw_field`
\ No newline at end of file
The dataset layout follows the C memory layout, with the primary dimension
running fastest. A 3D HDF5 dataset or a 3D ``numpy`` array have the layout
``array[z][y][x]``, and 2D arrays have the layout ``array[y][x]``. The first
dimension is always the vertical axis.
The following Python code creates a dataset ``layered`` in a file
``mapping.h5``, where the lower half of the domain maps to medium 0, and the
upper half to medium 1. The 3D domain contains 10 cells in each direction.
.. code-block:: python
import numpy as np
import h5py
# Create dataset. Mind the data type!
size = 10
layered = np.zeros((size, size, size), dtype=np.int_)
layered[5:, ...] = 1
# Write dataset to file
with h5py.File("mapping.h5", 'a') as file:
f.create_dataset("layered", data=layered)
The dataset and the file containing it are specified in the
:doc:`config file <man-config-file>` via the keys ``grid.mappingFile`` and
``grid.mappingFileDataset``.
Homogeneous Media
^^^^^^^^^^^^^^^^^
In the special case of a globally homogeneous medium, you can omit the mapping
dataset. This is indicated by leaving ``grid.mappingFile`` undefined or setting
its value to ``None``. You can then define the medium index for the entire
domain with the key ``grid.globalIndex``, which is unused otherwise.
Mapping Soil Layers with GMSH
-----------------------------
GMSH_ supports mapping lines, surfaces, and volumes to *physical* entities.
These entities may combine multiple of the aforementioned *geometrical*
entities. Physical entities are assigned a non-negative index upon creation.
These indices can be set by the user in any of the GMSH frontends.
Indices may not occur multiple times, even if they are assigned to different
types of physical entities.
The following ``.geo`` GMSH input file creates a rectangular domain that is
split in half. The lower part is mapped to index 1, the upper part to index 2.
This code can directly tranferred to a Python script using the pygmsh_ module.
It writes a ``.geo`` file while checking for a correct syntax within your
script. It is readily installed into the
:doc:`virtual environment <python-dorie-wrapper>`.
.. code-block:: default
// define geometric entities
Point(1) = {0, 0, 0, 0.1};
Point(2) = {2, 0, 0, 0.1};
Point(3) = {2, 2, 0, 0.1};
Point(4) = {0, 2, 0, 0.1};
Point(5) = {0, 1, 0, 0.1};
Point(6) = {2, 1, 0, 0.1};
Line(1) = {1, 2};
Line(2) = {2, 6};
Line(3) = {6, 3};
Line(4) = {3, 4};
Line(5) = {4, 5};