Commit 1483aa27 authored by Lukas Riedel's avatar Lukas Riedel Committed by Santiago Ospina De Los Ríos

Add Parameter attributes for version changes

* New attributes store version changes as dictionaries with version,
  description pairs.
* Adjust XML parser to parse values of new attributes.
* Update __str__ and __eq__ methods of Parameter.
parent c1dd6d8c
......@@ -6,6 +6,8 @@ We're rolling out version `X.Y.0`! :tada:
### 1 — In the Code
- [ ] `master`: Update version numbers in `VERSION`, `CHANGELOG.md`,
`dune.module` to `X.Y.0`.
- [ ] `master`: Update all `version="unreleased"` tags in the default config
file XML sources to `version="X.Y.0"`.
### 2 — On GitLab
- [ ] [Create branch][new branch] `X.Y-stable` from `master`
......
......@@ -11,6 +11,8 @@ The MRs to be considered in this update are listed in #
- [ ] Cherry-pick the listed commits into `X.Y-patch`
- [ ] `CHANGELOG.md`: Move appropriate entries into new section
- [ ] Update version numbers in `VERSION`, `CHANGELOG.md`, `dune.module`
- [ ] Update `version="unreleased"` tags in the default config file XML
sources <!-- only if applicable -->
#### Help on Cherry-Picking
Cherry-picking merge commits requires specifying the "mainline" parent, which
......
......@@ -30,6 +30,8 @@
* Cookbook tutorial on infiltration into homogeneous sand !157
* GitLab Description Templates for Releases !166
* ParaView tutorial for analyzing DORiE output files !173
* Parameter XML files support tags for version changes !171
* Unit test for Parameter XML file parser !171
### Changed
* Data structures for storing and accessing parameter information !55
......
# Configuration File Parameters
DORiE reads several interdependent parameters from INI configuration files
during execution. These parameters are listed in XML files inside this
directory. The `dorie.parscraper` module reads these XML files, tracks where
the respective parameters are queried in the source code, writes default
INI files for users, and creates a documentation of all parameters in RST.
There are four XML files:
* `field-parameters.xml`: The parameter tree used by the Parameter Field
Generator module.
* `common-parameters.xml`: Parameters used by both models.
* `richards-parameters.xml`: Parameters used by the Richards model only.
* `transport-parameters.xml`: Parameters used by the Transport model only.
Model-exclusive parameter categories are prefixed with the lowercase model
name during processing.
## XML File Hierarchy
The XML files must have the following hierarchy:
```xml
<dorie> -> <category> -> <parameter>
```
Category attributes:
* `name` (required): The name as it appears in the config file.
Parameter attributes:
* `name` (required): The name as it appears in the config file.
* `hidden`: If a parameter is not queried by the DORiE source code itself
(set to `false` by default).
Parameter tags with text only:
* `definition`: Meaning of the parameter as running text, displayed in docs.
* `suggestion`: Default value in created config files
* `values`: Allowed values, displayed in docs
* `comment`: Extra comment, will only show up in INI config files
Parameter tags with `version` attributes and text:
* `versionadded`: At which version the parameter was added.
* `versionchanged`: At which version the parameter key or behavior changed.
* `deprecated`: At which version the parameter became deprecated.
**Note:** When adding one of the version tags, set the `version` attribute to
`unreleased`. The proper version will be set during release.
All tags are optional. For any tag, the respective text field may be empty.
Text showing up in the docs may include RST markup. Paragraph breaks
may be indicated by two line breaks. This is useful for writing lists.
Parameter tags with `version` attributes strip all line breaks.
### Example XML Parameter Definition
```xml
<dorie>
<category name="output">
<parameter name="file" hidden="false">
<definition>
The path to write the output file to.
Note:
* The folder must already exist.
* An existing file will be overwritten.
</definition>
<values> path </values>
<suggestion> out.bin </suggestion>
<comment> Choose an existing path </comment>
<versionadded version="1.0"> </versionadded>
<versionchanged version="1.1"> Target folder must exist
</versionchanged>
</parameter>
</category>
</dorie>
```
......@@ -13,22 +13,7 @@ A full list is found at https://www.w3.org/TR/REC-html40/sgml/entities.html
<!ENTITY times "&#215;">
]>
<!--
XML file hierarchy:
<dorie> -> <category> -> <parameter> -> (parameter attributes)
Possible parameter attributes:
definition: meaning of the parameter, will only show up in html output
suggestion: standard value in created parameter files
values: possible values, will only show up in html output
comment: extra comment, will only show up in parameter files
All attributes are optional.
The parser supports rudimentary markdown / styling. You can add a paragraph by
adding an empty line, make text **bold** or ``monospaced``.
-->
<dorie>
<category name="simulation">
<parameter name="mode">
......
......@@ -13,22 +13,6 @@ A full list is found at https://www.w3.org/TR/REC-html40/sgml/entities.html
<!ENTITY times "&#215;">
]>
<!--
XML file hierarchy:
<dorie> -> <category> -> <parameter> -> (parameter attributes)
Possible parameter attributes:
definition: meaning of the parameter, will only show up in html output
suggestion: standard value in created parameter files
values: possible values, will only show up in html output
comment: extra comment, will only show up in parameter files
All attributes are optional.
The parser supports rudimentary markdown / styling. You can add a paragraph by
adding an empty line, make text **bold** or ``monospaced``.
-->
<dorie>
<category name="general" hidden="true">
......
......@@ -13,22 +13,6 @@ A full list is found at https://www.w3.org/TR/REC-html40/sgml/entities.html
<!ENTITY times "&#215;">
]>
<!--
XML file hierarchy:
<dorie> -> <category> -> <parameter> -> (parameter attributes)
Possible parameter attributes:
definition: meaning of the parameter, will only show up in html output
suggestion: standard value in created parameter files
values: possible values, will only show up in html output
comment: extra comment, will only show up in parameter files
All attributes are optional.
The parser supports rudimentary markdown / styling. You can add a paragraph by
adding an empty line, make text **bold** or ``monospaced``.
-->
<dorie>
<category name="output">
......
......@@ -13,22 +13,6 @@ A full list is found at https://www.w3.org/TR/REC-html40/sgml/entities.html
<!ENTITY times "&#215;">
]>
<!--
XML file hierarchy:
<dorie> -> <category> -> <parameter> -> (parameter attributes)
Possible parameter attributes:
definition: meaning of the parameter, will only show up in html output
suggestion: standard value in created parameter files
values: possible values, will only show up in html output
comment: extra comment, will only show up in parameter files
All attributes are optional.
The parser supports rudimentary markdown / styling. You can add a paragraph by
adding an empty line, make text **bold** or ``monospaced``.
-->
<dorie>
<category name="output">
......
......@@ -18,6 +18,9 @@ class Parameter:
self.values = None #: Possible values
self.comment = None #: Extra comment showing up in the generated ini files
self.hidden = False #: Do not issue an error if parameter is not found
self.versionadded = {} #: Versions and descriptions on feature added
self.versionchanged = {} #: Versions and descriptions on feature changed
self.deprecated = {} #: Versions and descriptions on feature deprecated
#: List of all the positions in the code where the parameter was found,
#: encoded in a tuple: (filename, lineno, C++ type)
......@@ -38,17 +41,30 @@ class Parameter:
string += "Allowed values: {0}\n".format(self.values)
string += "Comment: {0}\n".format(self.comment)
string += "Hidden: {}\n".format("yes" if self.hidden else "no")
string += self.__write_version_dict(self.versionadded, "Version added")
string += self.__write_version_dict(self.versionchanged, "Version changed")
string += self.__write_version_dict(self.deprecated, "Deprecated")
string += "Found at:\n"
for f, l, t in self._sources:
string += "\t{0}:{1} as {2}\n".format(f,l,t)
return string
def __write_version_dict(self, data, heading):
if not data:
return ""
out = heading + ":\n"
for ver, descr in data.items():
out += " v{}".format(ver)
if descr:
out += ": " + descr
out += "\n"
return out
def __eq__(self, other):
"""
Two Parameters are considered equal if both their category and key properties
match. This is used to quickly check if a Parameter is already defined.
"""
if isinstance(other,Parameter):
return self.category == other.category and self.key == other.key
"""Compare two parameters by all their attributes"""
if isinstance(other, Parameter):
return self.__dict__ == other.__dict__
else:
return False
......@@ -67,9 +67,16 @@ def parse(filename,model=None):
p.hidden = child.attrib["hidden"].lower() in ["true","1","yes"]
for attribute in grandchild:
if hasattr(p,attribute.tag) and not attribute.tag.startswith("_"):
stripped_text = "\n".join([x.strip() for x in attribute.text.split("\n")])
setattr(p,attribute.tag,stripped_text)
if hasattr(p, attribute.tag) and not attribute.tag.startswith("_"):
p_attr = getattr(p, attribute.tag)
stripped_text = attribute.text.lstrip().rstrip()
text_lines = [x.strip() for x in stripped_text.splitlines()]
if isinstance(p_attr, dict):
description = " ".join(text_lines)
p_attr[attribute.attrib["version"]] = description
else:
description = "\n".join(text_lines)
setattr(p, attribute.tag, description)
else:
warn("Unrecognized attribute {0} for {1}.{2}".format(attribute.tag,category,key),XMLWarning)
......
<dorie>
<category name="cat1">
<parameter name="param1">
<definition> Definition
</definition>
<suggestion> 0 </suggestion>
<values> 0, 1 </values>
<versionadded version="1.0"> </versionadded>
<versionchanged version="1.1"> change </versionchanged>
<versionchanged version="1.2"> change2 </versionchanged>
<deprecated version="2.0">
deprecated
deprecated
</deprecated>
<comment> comment </comment>
</parameter>
<parameter name="param2" hidden="true">
<definition>
Definition
Definition
</definition>
<suggestion> 2 </suggestion>
<values> 2 </values>
</parameter>
</category>
</dorie>
import pytest
import numpy as np
from collections import OrderedDict
from dorie.parscraper.readers import xml
from dorie.parscraper.parameter import Parameter
XML_FILE = "param_defaults.xml"
CATEGORY = "cat1"
PARAM_1 = "param1"
PARAM_2 = "param2"
# Fixtures ---------------------------------------------------------------
@pytest.fixture
def xml_param ():
"""Parse the XML file and return the parameter instances"""
return xml.parse(XML_FILE)
@pytest.fixture
def param_1():
"""The resulting parameter instance of 'param1' to test against"""
param = Parameter(None, CATEGORY, PARAM_1)
param.definition = "Definition"
param.suggestion = "0"
param.values = "0, 1"
param.comment = "comment"
param.hidden = False
param.versionadded = {"1.0": ""}
param.versionchanged = {"1.1": "change", "1.2": "change2"}
# NOTE: Newlines are removed
param.deprecated = {"2.0": "deprecated deprecated"}
return param
@pytest.fixture
def param_2():
"""The resulting parameter instance of 'param2' to test against"""
param = Parameter(None, CATEGORY, PARAM_2)
# NOTE: Internal newlines persist
param.definition = "Definition\nDefinition"
param.suggestion = "2"
param.values = "2"
param.comment = None
param.hidden = True
param.versionadded = {}
param.versionchanged = {}
param.deprecated = {}
return param
# Tests ------------------------------------------------------------------
def test_parse(xml_param, param_1, param_2):
"""Check if the parser returns the correct instances"""
assert isinstance(xml_param, OrderedDict)
assert CATEGORY in xml_param
print(xml_param[CATEGORY][0])
print(param_1)
assert param_1 == xml_param[CATEGORY][0]
assert param_2 == xml_param[CATEGORY][1]
......@@ -6,7 +6,7 @@ import os
import warnings
import argparse
from collections import OrderedDict
from yaml import load
from yaml import unsafe_load
from dorie.parscraper import writers
from dorie.parscraper.parameter import Parameter
......@@ -35,7 +35,7 @@ def write(data, args):
def read_data(args):
with open(args.input, 'r') as file:
return load(file)
return unsafe_load(file)
def parse_args():
parser = argparse.ArgumentParser(
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment