Merge branch...

Merge branch '166-support-xml-tags-specifying-version-changes-of-config-file-parameters' into 'master'

Resolve "Support XML tags specifying version changes of config file parameters"

Closes #166

See merge request !171

.gitlab/issue_templates/version-rollout.md

@@ -6,6 +6,8 @@ We're rolling out version `X.Y.0`! :tada:
 ### 1 — In the Code
 - [ ] `master`: Update version numbers in `VERSION`, `CHANGELOG.md`,
    `dune.module` to `X.Y.0`.
+- [ ] `master`: Update all `version="unreleased"` tags in the default config
+   file XML sources to `version="X.Y.0"`.
 
 ### 2 — On GitLab
 - [ ] [Create branch][new branch] `X.Y-stable` from `master`

.gitlab/merge_request_templates/patch-release.md

@@ -11,6 +11,8 @@ The MRs to be considered in this update are listed in #
 - [ ] Cherry-pick the listed commits into `X.Y-patch`
 - [ ] `CHANGELOG.md`: Move appropriate entries into new section
 - [ ] Update version numbers in `VERSION`, `CHANGELOG.md`, `dune.module`
+- [ ] Update `version="unreleased"` tags in the default config file XML
+    sources <!-- only if applicable -->
 
 #### Help on Cherry-Picking
 Cherry-picking merge commits requires specifying the "mainline" parent, which

CHANGELOG.md

@@ -30,6 +30,8 @@
 * Cookbook tutorial on infiltration into homogeneous sand !157
 * GitLab Description Templates for Releases !166
 * ParaView tutorial for analyzing DORiE output files !173
+* Parameter XML files support tags for version changes !171
+* Unit test for Parameter XML file parser !171
 
 ### Changed
 * Data structures for storing and accessing parameter information !55

doc/default_files/README.md

+# Configuration File Parameters
+
+DORiE reads several interdependent parameters from INI configuration files
+during execution. These parameters are listed in XML files inside this
+directory. The `dorie.parscraper` module reads these XML files, tracks where
+the respective parameters are queried in the source code, writes default
+INI files for users, and creates a documentation of all parameters in RST.
+
+There are four XML files:
+* `field-parameters.xml`: The parameter tree used by the Parameter Field
+  Generator module.
+* `common-parameters.xml`: Parameters used by both models.
+* `richards-parameters.xml`: Parameters used by the Richards model only.
+* `transport-parameters.xml`: Parameters used by the Transport model only.
+
+Model-exclusive parameter categories are prefixed with the lowercase model
+name during processing.
+
+## XML File Hierarchy
+
+The XML files must have the following hierarchy:
+```xml
+<dorie> -> <category> -> <parameter>
+```
+
+Category attributes:
+* `name` (required): The name as it appears in the config file.
+
+Parameter attributes:
+* `name` (required): The name as it appears in the config file.
+* `hidden`: If a parameter is not queried by the DORiE source code itself
+    (set to `false` by default).
+
+Parameter tags with text only:
+* `definition`: Meaning of the parameter as running text, displayed in docs.
+* `suggestion`: Default value in created config files
+* `values`: Allowed values, displayed in docs
+* `comment`: Extra comment, will only show up in INI config files
+
+Parameter tags with `version` attributes and text:
+* `versionadded`: At which version the parameter was added.
+* `versionchanged`: At which version the parameter key or behavior changed.
+* `deprecated`: At which version the parameter became deprecated.
+
+**Note:** When adding one of the version tags, set the `version` attribute to
+`unreleased`. The proper version will be set during release.
+
+All tags are optional. For any tag, the respective text field may be empty.
+
+Text showing up in the docs may include RST markup. Paragraph breaks
+may be indicated by two line breaks. This is useful for writing lists.
+Parameter tags with `version` attributes strip all line breaks.
+
+### Example XML Parameter Definition
+
+```xml
+<dorie>
+    <category name="output">
+        <parameter name="file" hidden="false">
+            <definition>
+                The path to write the output file to.
+
+                Note:
+
+                * The folder must already exist.
+
+                * An existing file will be overwritten.
+            </definition>
+            <values> path </values>
+            <suggestion> out.bin </suggestion>
+            <comment> Choose an existing path </comment>
+            <versionadded version="1.0"> </versionadded>
+            <versionchanged version="1.1"> Target folder must exist
+            </versionchanged>
+        </parameter>
+    </category>
+</dorie>
+```

doc/default_files/common-parameters.xml

@@ -13,22 +13,7 @@ A full list is found at https://www.w3.org/TR/REC-html40/sgml/entities.html
     <!ENTITY times "&#215;">
 ]>
 
-<!--
-XML file hierarchy:
-<dorie> -> <category> -> <parameter> -> (parameter attributes)
-
-Possible parameter attributes:
-
-  definition: meaning of the parameter, will only show up in html output
-  suggestion: standard value in created parameter files
-  values: possible values, will only show up in html output
-  comment: extra comment, will only show up in parameter files
 
-All attributes are optional.
-
-The parser supports rudimentary markdown / styling. You can add a paragraph by
-adding an empty line, make text **bold** or ``monospaced``.
--->
 <dorie>
   <category name="simulation">
     <parameter name="mode">

doc/default_files/field-parameters.xml

@@ -13,22 +13,6 @@ A full list is found at https://www.w3.org/TR/REC-html40/sgml/entities.html
     <!ENTITY times "&#215;">
 ]>
 
-<!--
-XML file hierarchy:
-<dorie> -> <category> -> <parameter> -> (parameter attributes)
-
-Possible parameter attributes:
-
-  definition: meaning of the parameter, will only show up in html output
-  suggestion: standard value in created parameter files
-  values: possible values, will only show up in html output
-  comment: extra comment, will only show up in parameter files
-
-All attributes are optional.
-
-The parser supports rudimentary markdown / styling. You can add a paragraph by
-adding an empty line, make text **bold** or ``monospaced``.
--->
 
 <dorie>
   <category name="general" hidden="true">

doc/default_files/richards-parameters.xml

@@ -13,22 +13,6 @@ A full list is found at https://www.w3.org/TR/REC-html40/sgml/entities.html
     <!ENTITY times "&#215;">
 ]>
 
-<!--
-XML file hierarchy:
-<dorie> -> <category> -> <parameter> -> (parameter attributes)
-
-Possible parameter attributes:
-
-  definition: meaning of the parameter, will only show up in html output
-  suggestion: standard value in created parameter files
-  values: possible values, will only show up in html output
-  comment: extra comment, will only show up in parameter files
-
-All attributes are optional.
-
-The parser supports rudimentary markdown / styling. You can add a paragraph by
-adding an empty line, make text **bold** or ``monospaced``.
--->
 
 <dorie>
   <category name="output">

doc/default_files/transport-parameters.xml

@@ -13,22 +13,6 @@ A full list is found at https://www.w3.org/TR/REC-html40/sgml/entities.html
     <!ENTITY times "&#215;">
 ]>
 
-<!--
-XML file hierarchy:
-<dorie> -> <category> -> <parameter> -> (parameter attributes)
-
-Possible parameter attributes:
-
-  definition: meaning of the parameter, will only show up in html output
-  suggestion: standard value in created parameter files
-  values: possible values, will only show up in html output
-  comment: extra comment, will only show up in parameter files
-
-All attributes are optional.
-
-The parser supports rudimentary markdown / styling. You can add a paragraph by
-adding an empty line, make text **bold** or ``monospaced``.
--->
 
 <dorie>
   <category name="output">

python/dorie/dorie/parscraper/parameter.py

@@ -18,6 +18,9 @@ class Parameter:
         self.values = None #: Possible values
         self.comment = None #: Extra comment showing up in the generated ini files
         self.hidden = False #: Do not issue an error if parameter is not found
+        self.versionadded = {} #: Versions and descriptions on feature added
+        self.versionchanged = {} #: Versions and descriptions on feature changed
+        self.deprecated = {} #: Versions and descriptions on feature deprecated
 
         #: List of all the positions in the code where the parameter was found,
         #: encoded in a tuple: (filename, lineno, C++ type)
@@ -38,17 +41,30 @@ class Parameter:
         string += "Allowed values: {0}\n".format(self.values)
         string += "Comment: {0}\n".format(self.comment)
         string += "Hidden: {}\n".format("yes" if self.hidden else "no")
+        string += self.__write_version_dict(self.versionadded, "Version added")
+        string += self.__write_version_dict(self.versionchanged, "Version changed")
+        string += self.__write_version_dict(self.deprecated, "Deprecated")
         string += "Found at:\n"
         for f, l, t in self._sources:
             string += "\t{0}:{1} as {2}\n".format(f,l,t)
         return string
 
+    def __write_version_dict(self, data, heading):
+        if not data:
+            return ""
+
+        out = heading + ":\n"
+        for ver, descr in data.items():
+            out += "  v{}".format(ver)
+            if descr:
+                out += ": " + descr
+            out += "\n"
+
+        return out
+
     def __eq__(self, other):
-        """
-        Two Parameters are considered equal if both their category and key properties
-        match. This is used to quickly check if a Parameter is already defined.
-        """
-        if isinstance(other,Parameter):
-            return self.category == other.category and self.key == other.key
+        """Compare two parameters by all their attributes"""
+        if isinstance(other, Parameter):
+            return self.__dict__ == other.__dict__
         else:
             return False

python/dorie/dorie/parscraper/readers/xml.py

@@ -67,9 +67,16 @@ def parse(filename,model=None):
                 p.hidden = child.attrib["hidden"].lower() in ["true","1","yes"]
 
             for attribute in grandchild:
-                if hasattr(p,attribute.tag) and not attribute.tag.startswith("_"):
-                    stripped_text = "\n".join([x.strip() for x in attribute.text.split("\n")])
-                    setattr(p,attribute.tag,stripped_text)
+                if hasattr(p, attribute.tag) and not attribute.tag.startswith("_"):
+                    p_attr = getattr(p, attribute.tag)
+                    stripped_text = attribute.text.lstrip().rstrip()
+                    text_lines = [x.strip() for x in stripped_text.splitlines()]
+                    if isinstance(p_attr, dict):
+                        description = " ".join(text_lines)
+                        p_attr[attribute.attrib["version"]] = description
+                    else:
+                        description = "\n".join(text_lines)
+                        setattr(p, attribute.tag, description)
                 else:
                     warn("Unrecognized attribute {0} for {1}.{2}".format(attribute.tag,category,key),XMLWarning)
 

python/dorie/dorie/test/param_defaults.xml

+<dorie>
+    <category name="cat1">
+        <parameter name="param1">
+            <definition> Definition
+            </definition>
+            <suggestion> 0 </suggestion>
+            <values> 0, 1 </values>
+            <versionadded version="1.0"> </versionadded>
+            <versionchanged version="1.1"> change </versionchanged>
+            <versionchanged version="1.2"> change2 </versionchanged>
+            <deprecated version="2.0">
+                deprecated
+                deprecated
+            </deprecated>
+            <comment> comment </comment>
+        </parameter>
+
+        <parameter name="param2" hidden="true">
+            <definition>
+                Definition
+                Definition
+            </definition>
+            <suggestion> 2 </suggestion>
+            <values> 2 </values>
+        </parameter>
+    </category>
+</dorie>

python/dorie/dorie/test/test_xml_param_parser.py

+import pytest
+import numpy as np
+
+from collections import OrderedDict
+
+from dorie.parscraper.readers import xml
+from dorie.parscraper.parameter import Parameter
+
+XML_FILE = "param_defaults.xml"
+CATEGORY = "cat1"
+PARAM_1 = "param1"
+PARAM_2 = "param2"
+
+# Fixtures ---------------------------------------------------------------
+@pytest.fixture
+def xml_param ():
+    """Parse the XML file and return the parameter instances"""
+    return xml.parse(XML_FILE)
+
+@pytest.fixture
+def param_1():
+    """The resulting parameter instance of 'param1' to test against"""
+    param = Parameter(None, CATEGORY, PARAM_1)
+    param.definition = "Definition"
+    param.suggestion = "0"
+    param.values = "0, 1"
+    param.comment = "comment"
+    param.hidden = False
+    param.versionadded = {"1.0": ""}
+    param.versionchanged = {"1.1": "change", "1.2": "change2"}
+    # NOTE: Newlines are removed
+    param.deprecated = {"2.0": "deprecated deprecated"}
+
+    return param
+
+@pytest.fixture
+def param_2():
+    """The resulting parameter instance of 'param2' to test against"""
+    param = Parameter(None, CATEGORY, PARAM_2)
+    # NOTE: Internal newlines persist
+    param.definition = "Definition\nDefinition"
+    param.suggestion = "2"
+    param.values = "2"
+    param.comment = None
+    param.hidden = True
+    param.versionadded = {}
+    param.versionchanged = {}
+    param.deprecated = {}
+
+    return param
+
+# Tests ------------------------------------------------------------------
+
+def test_parse(xml_param, param_1, param_2):
+    """Check if the parser returns the correct instances"""
+    assert isinstance(xml_param, OrderedDict)
+    assert CATEGORY in xml_param
+
+    print(xml_param[CATEGORY][0])
+    print(param_1)
+    assert param_1 == xml_param[CATEGORY][0]
+    assert param_2 == xml_param[CATEGORY][1]

python/dorie/wrapper/write_config.py

@@ -6,7 +6,7 @@ import os
 import warnings
 import argparse
 from collections import OrderedDict
-from yaml import load
+from yaml import unsafe_load
 
 from dorie.parscraper import writers
 from dorie.parscraper.parameter import Parameter
@@ -35,7 +35,7 @@ def write(data, args):
 
 def read_data(args):
     with open(args.input, 'r') as file:
-        return load(file)
+        return unsafe_load(file)
 
 def parse_args():
     parser = argparse.ArgumentParser(