Merge pull request #8211 from bernt-matthias/topic/filter-static-regex

new regular expression filter, and efficient sort filter

lib/galaxy/tool_util/xsd/galaxy.xsd

@@ -3566,7 +3566,7 @@ is the ``none`` preset.</xs:documentation>
   <xs:complexType name="Filter">
     <xs:annotation>
       <xs:documentation xml:lang="en"><![CDATA[Optionally contained within an
-``<options>`` tag set - filter out values obtained from a locally stored file (e.g.
+``<options>`` tag set - modify (e.g. remove, add, sort, ...) the list of values obtained from a locally stored file (e.g.
 a tool data table) or a dataset in the current history.
 
 ### Examples
@@ -3669,19 +3669,39 @@ demonstrates splitting up strings into multiple values.
     <xs:attribute name="type" type="FilterType" use="required">
       <xs:annotation>
         <xs:documentation xml:lang="en"><![CDATA[
+Currently the following filters are defined:
+
+* ``static_value`` filter options for which the entry in a given ``column`` of the referenced file based on equality to the ``value`` attribute of the filter. 
+* ``regexp`` similar to the ``static_value`` filter, but checks if the regular expression given by ``value`` matches the entry.
+* ``param_value`` filter options for which the entry in a given ``column`` of the referenced file based on properties of another input parameter specified by ``ref``. This property is by default the value of the parameter, but also the values of another attribute (``ref_attribute``) of the parameter can be used, e.g. the extension of a data input.
+* ``data_meta`` If a ``column`` is given options are filtered for which the entry in this column ``column`` is equal to metadata of another input parameter specified by ``ref``. 
+If no ``column`` is given the metadata value of the referenced input is added to the options list (in this case the corresponding ``options`` tag must not have the ``from_data_table`` or ``from_dataset`` attributes). 
+In both cases the desired metadata is selected by ``key``.
+
+The above filters can be inverted by setting ``keep`` to true.
+
+* ``add_value``: add an option with a given ``name`` and ``value`` to the options. By default the new option is appended, with ``index`` the insertion position can be specified.
+* ``remove_value``: remove a value from the options. Either specified explicitly with ``value``, the value of another input specifified with ``ref``, or the metatdata ``key`` of another input ``meta_ref``.
+* ``unique_value``: remove options that have duplicate entries in the given ``column``.
+* ``sort_by``: sort options by the entries of a given ``column``.
+* ``multiple_splitter``: split the entries of the specified ``column``(s) in the referenced file using a ``separator``. Thereby the number of columns is increased. 
+
+Note that filters that are applyed after one of the latter two filters must not refer to 
+
 These values are defined in the module
 [/lib/galaxy/tools/parameters/dynamic_options.py](https://github.com/galaxyproject/galaxy/blob/master/lib/galaxy/tools/parameters/dynamic_options.py)
-in the ``filter_types`` dictionary. Currently defined values are: ``data_meta``,
-``param_value``, ``static_value``, ``unique_value``, ``multiple_splitter``,
-``attribute_value_splitter``, ``add_value``, ``remove_value``, and
-``sort_by``]]></xs:documentation>
+in the ``filter_types`` dictionary. 
+
+Deprecated filter types:
+
+* ``attribute_value_splitter`` 
+        ]]></xs:documentation>
       </xs:annotation>
     </xs:attribute>
     <xs:attribute name="column" type="xs:string">
       <xs:annotation>
-        <xs:documentation xml:lang="en">Column targeted by this filter - this
-attribute is unused and invalid if ``type`` is ``add_value`` or ``remove_value``.
-This can be a column index or a column name.</xs:documentation>
+        <xs:documentation xml:lang="en">Column targeted by this filter given as column index or a column name. Invalid if ``type`` is ``add_value`` or ``remove_value``.
+</xs:documentation>
       </xs:annotation>
     </xs:attribute>
     <xs:attribute name="name" type="xs:string">
@@ -3723,7 +3743,7 @@ only used if ``multiple`` is set to ``true``.]]></xs:documentation>
       <xs:annotation>
         <xs:documentation xml:lang="en">If ``true``, keep columns matching the
 value, if ``false`` discard columns matching the value. Used when ``type`` is
-either ``static_value`` or ``param_value``.</xs:documentation>
+either ``static_value``, ``regexp`` or ``param_value``.</xs:documentation>
       </xs:annotation>
     </xs:attribute>
     <xs:attribute name="value" type="xs:string">
@@ -3731,7 +3751,7 @@ either ``static_value`` or ``param_value``.</xs:documentation>
         <xs:documentation xml:lang="en">Target value of the operations - has
 slightly different meanings depending on ``type``. For instance when ``type`` is
 ``add_value`` it is the value to add to the list and when ``type`` is
-``static_value`` it is the value compared against.</xs:documentation>
+``static_value`` or ``regexp`` it is the value compared against.</xs:documentation>
       </xs:annotation>
     </xs:attribute>
     <xs:attribute name="ref_attribute" type="xs:string">
@@ -5509,6 +5529,7 @@ and ``bibtex`` are the only supported options.</xs:documentation>
       <xs:enumeration value="data_meta"/>
       <xs:enumeration value="param_value"/>
       <xs:enumeration value="static_value"/>
+      <xs:enumeration value="regexp"/>
       <xs:enumeration value="unique_value"/>
       <xs:enumeration value="multiple_splitter"/>
       <xs:enumeration value="add_value"/>

lib/galaxy/tools/parameters/dynamic_options.py

@@ -4,6 +4,7 @@ on the values of other parameters or other aspects of the current state)
 """
 import logging
 import os
+import re
 
 from six import StringIO
 
@@ -74,7 +75,44 @@ class StaticValueFilter(Filter):
         except Exception:
             pass
         for fields in options:
-            if (self.keep and fields[self.column] == filter_value) or (not self.keep and fields[self.column] != filter_value):
+            if self.keep == (filter_value == fields[self.column]):
+                rval.append(fields)
+        return rval
+
+
+class RegexpFilter(Filter):
+    """
+    Filters a list of options on a column by a regular expression.
+
+    Type: regexp
+
+    Required Attributes:
+        value: regular expression to compare to
+        column: column in options to compare with
+    Optional Attributes:
+        keep: Keep columns matching the regexp (True)
+              Discard columns matching the regexp (False)
+    """
+
+    def __init__(self, d_option, elem):
+        Filter.__init__(self, d_option, elem)
+        self.value = elem.get("value", None)
+        assert self.value is not None, "Required 'value' attribute missing from filter"
+        column = elem.get("column", None)
+        assert column is not None, "Required 'column' attribute missing from filter, when loading from file"
+        self.column = d_option.column_spec_to_index(column)
+        self.keep = string_as_bool(elem.get("keep", 'True'))
+
+    def filter_options(self, options, trans, other_values):
+        rval = []
+        filter_value = self.value
+        try:
+            filter_value = User.expand_user_properties(trans.user, filter_value)
+        except Exception:
+            pass
+        filter_pattern = re.compile(filter_value)
+        for fields in options:
+            if self.keep == (not filter_pattern.match(fields[self.column]) is None):
                 rval.append(fields)
         return rval
 
@@ -141,6 +179,9 @@ class DataMetaFilter(Filter):
         if not isinstance(ref, HistoryDatasetAssociation) and not is_data_or_data_list:
             return []  # not a valid dataset
 
+        # get the metadata value. for lists (of data sets) and collections
+        # the meta data value of all elements is determined if its the same
+        # for all, if different are found the filter returns an empty list
         if is_data_list:
             meta_value = None
             for single_ref in ref:
@@ -155,6 +196,8 @@ class DataMetaFilter(Filter):
         else:
             meta_value = ref.metadata.get(self.key, None)
 
+        # if no meta data value could be determined just return a copy
+        # of the original options
         if meta_value is None:
             return [(disp_name, optval, selected) for disp_name, optval, selected in options]
 
@@ -226,7 +269,7 @@ class ParamValueFilter(Filter):
         ref = str(ref)
         rval = []
         for fields in options:
-            if (self.keep and fields[self.column] == ref) or (not self.keep and fields[self.column] != ref):
+            if self.keep == (fields[self.column] == ref):
                 rval.append(fields)
         return rval
 
@@ -292,6 +335,8 @@ class AttributeValueSplitterFilter(Filter):
     """
     Filters a list of attribute-value pairs to be unique attribute names.
 
+    DEPRECATED: just replace with 2 rounds of MultipleSplitterFilter
+
     Type: attribute_value_splitter
 
     Required Attributes:
@@ -442,20 +487,13 @@ class SortByColumnFilter(Filter):
         self.column = d_option.column_spec_to_index(column)
 
     def filter_options(self, options, trans, other_values):
-        rval = []
-        for fields in options:
-            for j in range(0, len(rval)):
-                if fields[self.column] < rval[j][self.column]:
-                    rval.insert(j, fields)
-                    break
-            else:
-                rval.append(fields)
-        return rval
+        return sorted(options, key=lambda x: x[self.column])
 
 
 filter_types = dict(data_meta=DataMetaFilter,
                     param_value=ParamValueFilter,
                     static_value=StaticValueFilter,
+                    regexp=RegexpFilter,
                     unique_value=UniqueValueFilter,
                     multiple_splitter=MultipleSplitterFilter,
                     attribute_value_splitter=AttributeValueSplitterFilter,

test/functional/tools/dbkey_filter_multi_input.xml

@@ -1,5 +1,5 @@
 <tool id="dbkey_filter_multi_input" name="dbkey_filter_multi_input" version="0.1.0">
-    <description>Filter select on dbkey of multiple inputs</description>
+    <description>Filter select on dbkey of multiple inputs and use of named column</description>
     <command><![CDATA[
         #for $input in $inputs#
         cat $input >> $output;
@@ -10,7 +10,7 @@
         <param format="txt" name="inputs" type="data" label="Inputs" multiple="true" help="" />
         <param name="index" type="select" label="Using reference genome">
           <options from_data_table="test_fasta_indexes">
-            <filter type="data_meta" ref="inputs" key="dbkey" column="1" />
+            <filter type="data_meta" ref="inputs" key="dbkey" column="dbkey" />
             <validator type="no_options" message="No reference genome is available for the build associated with the selected input dataset" />
           </options>
         </param>
@@ -31,4 +31,4 @@
 
     <help>
     </help>
-</tool>
+</tool>

test/functional/tools/filter_static_regexp.xml

@@ -0,0 +1,94 @@
+<tool id="filter_static_regexp" name="filter_static_regexp" version="0.1.0">
+    <description>Filter by static value and regexp</description>
+    <command>
+        echo $index_static > '$output'
+        echo $index_static_keep >> '$output'
+        echo $index_regexp >> '$output'
+        echo $index_regexp_keep >> '$output'
+    </command>
+    <inputs>
+        <!-- tests for static_value filter: remove hp18 from the options by 
+             a) removing it explicitly (due to keep="false") 
+             b) keeping only the other option, i.e. hg19 -->
+        <param name="index" type="select" label="Using reference genome">
+          <options from_data_table="test_fasta_indexes">
+            <filter type="static_value" column="dbkey" value="hg18" />
+            <validator type="no_options" message="No reference genome is available for the build associated with the selected input dataset" />
+          </options>
+        </param>
+        <param name="index2" type="select" label="Using reference genome">
+          <options from_data_table="test_fasta_indexes">
+            <filter type="static_value" column="dbkey" value="hg19" keep="true" />
+            <validator type="no_options" message="No reference genome is available for the build associated with the selected input dataset" />
+          </options>
+        </param>
+        <!-- tests for regexp filter: remove hp18 from the options 
+             essentially the same as for the static value filter, just using a simple regexp -->
+        <param name="index_regexp" type="select" label="Using reference genome">
+          <options from_data_table="test_fasta_indexes">
+            <filter type="regexp" column="dbkey" value="hg.8" />
+            <validator type="no_options" message="No reference genome is available for the build associated with the selected input dataset" />
+          </options>
+        </param>
+        <param name="index_regexp_keep" type="select" label="Using reference genome">
+          <options from_data_table="test_fasta_indexes">
+            <filter type="regexp" column="dbkey" value="hg.9" keep="true" />
+            <validator type="no_options" message="No reference genome is available for the build associated with the selected input dataset" />
+          </options>
+        </param>
+    </inputs>
+
+    <outputs>
+        <data format="txt" name="output" />
+    </outputs>
+
+    <tests>
+        <!-- can choose a dbkey if it matches input -->
+        <test>
+            <param name="index_static" value="hg19" />
+            <param name="index_static_keep" value="hg19" />
+            <param name="index_regexp" value="hg19" />
+            <param name="index_regexp_keep" value="hg19" />
+            <output name="output">
+                <assert_contents>
+                    <has_text text="hg19" />    
+                    <not_has_text text="hg18" />
+                </assert_contents>
+            </output>
+        </test>
+        <!-- cannot pick index otherwise -->
+        <!-- Does this make sense - if no dbkey is defined there is no option
+             available? -->
+        <test expect_failure="true">
+            <param name="index_static" value="hg18" />
+            <param name="index_static_keep" value="hg19" />
+            <param name="index_regexp" value="hg19" />
+            <param name="index_regexp_keep" value="hg19" />
+            <output name="output"/>
+        </test>
+        <test expect_failure="true">
+            <param name="index_static" value="hg19" />
+            <param name="index_static_keep" value="hg18" />
+            <param name="index_regexp" value="hg19" />
+            <param name="index_regexp_keep" value="hg19" />
+            <output name="output"/>
+        </test>
+        <test expect_failure="true">
+            <param name="index_static" value="hg19" />
+            <param name="index_static_keep" value="hg19" />
+            <param name="index_regexp" value="hg18" />
+            <param name="index_regexp_keep" value="hg19" />
+            <output name="output"/>
+        </test>
+        <test expect_failure="true">
+            <param name="index_static" value="hg19" />
+            <param name="index_static_keep" value="hg19" />
+            <param name="index_regexp" value="hg19" />
+            <param name="index_regexp_keep" value="hg18" />
+            <output name="output"/>
+        </test>
+    </tests>
+
+    <help>
+    </help>
+</tool>

test/functional/tools/samples_tool_conf.xml

@@ -34,6 +34,7 @@
   <tool file="empty_datasets.xml" />
   <tool file="inputs_as_json.xml" />
   <tool file="inputs_as_json_with_paths.xml" />
+  <tool file="filter_static_regexp.xml" />
   <tool file="dbkey_filter_input.xml" />
   <tool file="dbkey_filter_multi_input.xml" />
   <tool file="dbkey_output_action.xml" />