This would affect for example the mapping over of fastq-dump (and all other
tools in the sra-toolkit).
If one had attempted this previously the discovery phase would fail with:
```
galaxy.tools.parameters.output_collect ERROR 2018-01-30 08:56:46,369 Problem gathering output collection.
Traceback (most recent call last):
File "/bioinfo/guests/mvandenb/galaxy/lib/galaxy/tools/parameters/output_collect.py", line 169, in collect_dynamic_collections
collection
File "/bioinfo/guests/mvandenb/galaxy/lib/galaxy/managers/collections.py", line 216, in collection_builder_for
return builder.BoundCollectionBuilder(dataset_collection, collection_type_description)
File "/bioinfo/guests/mvandenb/galaxy/lib/galaxy/dataset_collections/builder.py", line 84, in __init__
raise Exception("Cannot reset elements of an already populated dataset collection.")
Exception: Cannot reset elements of an already populated dataset collection.
```
Instead we force the collection state to be new when we are discovering output collection datasets,
which seems reasonable to me. This includes an API testcase that would have failed previously.
This is using the simplest of workflows. I had to
remove the chromInfo part from the workflow, because the
path referenced would be changed dynamically and so the
job search wouldn't recognize this as being equivalent.
Also includes a simple test that runs cat1 three times,
the second time with `use_cached_job=False` (the default),
and the third time with `use_cached_job=True` and verifies
that the second output is different from the third, and that the
first and third output are identical.
This does not work at the moment, since we need to verify a jobs' input name,
dataset and metadata, for which we don't have an explicit trail. The only trace
is via the update_time (i.e if a input dataset hasn't been updated since the
job was created we know these were the actual parameters in use during the
execution of this job).
This specifically addresses the problem where some jobs of a mapped-over
collection have failed. Instead of filtering the failed collection and
restarting the workflow at this position (involving a lot of copy-paste ...)
the user can now limit the rerun to the problematic jobs and the workflow
should resume from there.
Should fix https://github.com/galaxyproject/galaxy/issues/2235.
This is one possible implementation, it would also be feasible to not
manipulate the original collection, but to copy the HDCA and then to replace
collection elements and replace all references for jobs that depend on the HDCA,
as we do for HDAs. This implementation seems simpler, but let me know if you
see problems with this approach.
This test runs a workflow whose first step fails, followed by a tool that uses
the first step's output as an input, which is behind a nested conditional. This
recapitulates the bug described in https://github.com/galaxyproject/galaxy/issues/5222.
With new test case to verify this fix is needed. The previous test case that was verifying collection mapping and rescheduling of subworkflows turned out to really be testing the connections with respect to the outer workflow since only the last step of the subworkflow required rescheduling iterations. This newer test case verifies the connections within the subworkflow also - in addition to re-testing the connections and mapping in the outer workflow.
These are sort of an error in the workflow definition but we don't have a means to competently display this information to the user via the UI so I think we need to just allow this to happen.
I've attached a test case that exhibits this behavior and passes under 17.09 but fails under the current dev branch without these fixes. This was broken in https://github.com/galaxyproject/galaxy/commit/d5f98f49045bca351e982f17cac9386308089eeb when we started to actually persist pointers to workflow invocation outputs (as part of https://github.com/galaxyproject/galaxy/pull/5013).
There are no places where this count is not set atomically in one thread so this should not have any performance downsides versus the older approach of calculating it on the fly (which was very expensive).
These aggregate all the jobs contributing to an implicitly created HistoryDatasetCollectionAssociation.
- Model enhancements that allow this.
- API endpoint for jobs summary of history content (somewhat uniform across HDAs and HDCAs just for consistency).
- Test cases.
- Rework history contents API a bit to eliminate more costly state summary checks by default.
Create mapped over collections ahead of time instead of at the end only if they are valid.
- This way if there is a problem that problem can be reported in the state of the mapped collection.
- This way datasets can just be hidden instead of displayed and then hidden (fixes#1790).
- I think this should improve performance and prevent locks because we no longer have conflicting threads trying to write dataset state.
- We populate the collection as we go so we can recover and continue scheduling the mapped step (needed for #3883)
If one maps a tool over an empty collection, one should get an empty output collection for each of the tool's outputs.
This should have always been a valid operation I think, the code just wasn't quite structured right when the collection work was merged - there were too many places I was assuming the existence of at least one actual output or input that was mapped over to continue going. This reworks all of that and seems to handle empty collections appropriately.
Fixes#4025 (not being informing the user of the operation, but actually by allowing the operation - which it should be allowed I think).
Workflow Invocations
--------------------
The workflow invocation outputs half of this is relatively straight forward. It is modelled somewhat on job outputs, output datasets and output dataset collections are now tracked for each workflow invocation and exposed via the workflow invocation API. This required adding new tables (linked to WorkflowInvocations and WorkflowOutputs) that track these output associations.
Previously one could imagine backtracking this information for simple tool steps via the WorkflowInvocationStep -> Job table, but for steps that have many jobs (i.e. mapping over a collection) or for non-tool steps such information was more difficult to recover (and simply couldn't be recovered from the API at all or even internally without significant knowledge of the underlying workflow).
Workflow Invocation Steps
-------------------------
Tracking the outputs of WorkflowInvocationSteps was not previously done at all, one would have to follow the Job table as well. A signficant downside to this is that one cannot map over empty collections in a workflow - since no such job would exist. Tracking job outputs for WorkflowInvocationSteps is not a simple matter of just attaching outputs to an existing table because we had no concept of a workflow step tracked - since there could be many WorklfowInvocationSteps corresponding to the same combination of WorkflowInvocation and WorkflowStep. That should feel wrong and that is because it is - when collections were added the possiblity of having many jobs for the same combination of WorkflowInvocation and WorkflowStep was added. I should have split WorkflowInvocationSteps into WorkflowInvocationSteps and WorkflowInvocationStepJobAssociations at that time but didn't. This commit now does it - effectively normalizing the ``workflow_invocation_step`` table by introducing the new ``workflow_invocation_step_job_association`` table.
Splitting up the WorkflowInvocationStep table this way allows recovering the mapped over output (e.g. the implicitly created collection from all the jobs) as well the outputs from the individual jobs (by walking WorkflowInvocationStep -> WorkflowInvocationStepJobAssociation -> Job -> JobToOutput*Association).
This split up involves failrly substantial changes to the workflow module interface. Any place a list of WorkflowInvocationSteps was assumed, I reworked it to just expect a single WorkflowInvocationStep. I vastly simplified recover_mapping to just use the persisted outputs (this was needed in order to also implment empty collection mapping in workflows). This also fixes a bug (or implements a missing feature) where Subworkflow moudles had no recover_mapping methods - so for instance if a tool that produces dynamic collections appeared anywhere in a workflow after a subworkflow step - that workflow would not complete scheduling properly.
Now that we have a way to reference the set of jobs corresponding to a workflow step within an invocation, we can start to track partial scheduling of such steps. This is outlined in https://github.com/galaxyproject/galaxy/issues/3883 and refactoring toward this goal is included here - including adding a state to WorkflowInvocationStep so Galaxy can determine if it has started scheduling this step and an index when scheduling jobs so it can tell how far into a scheduling things have gone as well as augmenting the tool executor to take a maximum number of jobs to execute and allow recovery of existing jobs for collection building purposes.
*Applications*
These changes will enable:
- A simple, consistent API for finding workflow outputs that can be consumed by Planemo for testing workflows.
- Mapping over empty collections in workflows.
- Re-scheduling workflow invocations that include subworkflow steps.
- Partial scheduling within steps requiring a large number of jobs when scheduling workflow invocations.
Checked through tools-iuc and I couldn't find any tools that use the actual input data parameters in filter statements so I think it is a fairly good approximation to assume that all mapped jobs will either filter or not at least with current execution model.
John Chilton