This is using the simplest of workflows. I had to
remove the chromInfo part from the workflow, because the
path referenced would be changed dynamically and so the
job search wouldn't recognize this as being equivalent.
This specifically addresses the problem where some jobs of a mapped-over
collection have failed. Instead of filtering the failed collection and
restarting the workflow at this position (involving a lot of copy-paste ...)
the user can now limit the rerun to the problematic jobs and the workflow
should resume from there.
Should fix https://github.com/galaxyproject/galaxy/issues/2235.
This is one possible implementation, it would also be feasible to not
manipulate the original collection, but to copy the HDCA and then to replace
collection elements and replace all references for jobs that depend on the HDCA,
as we do for HDAs. This implementation seems simpler, but let me know if you
see problems with this approach.
This test runs a workflow whose first step fails, followed by a tool that uses
the first step's output as an input, which is behind a nested conditional. This
recapitulates the bug described in https://github.com/galaxyproject/galaxy/issues/5222.
With new test case to verify this fix is needed. The previous test case that was verifying collection mapping and rescheduling of subworkflows turned out to really be testing the connections with respect to the outer workflow since only the last step of the subworkflow required rescheduling iterations. This newer test case verifies the connections within the subworkflow also - in addition to re-testing the connections and mapping in the outer workflow.
These are sort of an error in the workflow definition but we don't have a means to competently display this information to the user via the UI so I think we need to just allow this to happen.
I've attached a test case that exhibits this behavior and passes under 17.09 but fails under the current dev branch without these fixes. This was broken in https://github.com/galaxyproject/galaxy/commit/d5f98f49045bca351e982f17cac9386308089eeb when we started to actually persist pointers to workflow invocation outputs (as part of https://github.com/galaxyproject/galaxy/pull/5013).
Create mapped over collections ahead of time instead of at the end only if they are valid.
- This way if there is a problem that problem can be reported in the state of the mapped collection.
- This way datasets can just be hidden instead of displayed and then hidden (fixes#1790).
- I think this should improve performance and prevent locks because we no longer have conflicting threads trying to write dataset state.
- We populate the collection as we go so we can recover and continue scheduling the mapped step (needed for #3883)
If one maps a tool over an empty collection, one should get an empty output collection for each of the tool's outputs.
This should have always been a valid operation I think, the code just wasn't quite structured right when the collection work was merged - there were too many places I was assuming the existence of at least one actual output or input that was mapped over to continue going. This reworks all of that and seems to handle empty collections appropriately.
Fixes#4025 (not being informing the user of the operation, but actually by allowing the operation - which it should be allowed I think).
Workflow Invocations
--------------------
The workflow invocation outputs half of this is relatively straight forward. It is modelled somewhat on job outputs, output datasets and output dataset collections are now tracked for each workflow invocation and exposed via the workflow invocation API. This required adding new tables (linked to WorkflowInvocations and WorkflowOutputs) that track these output associations.
Previously one could imagine backtracking this information for simple tool steps via the WorkflowInvocationStep -> Job table, but for steps that have many jobs (i.e. mapping over a collection) or for non-tool steps such information was more difficult to recover (and simply couldn't be recovered from the API at all or even internally without significant knowledge of the underlying workflow).
Workflow Invocation Steps
-------------------------
Tracking the outputs of WorkflowInvocationSteps was not previously done at all, one would have to follow the Job table as well. A signficant downside to this is that one cannot map over empty collections in a workflow - since no such job would exist. Tracking job outputs for WorkflowInvocationSteps is not a simple matter of just attaching outputs to an existing table because we had no concept of a workflow step tracked - since there could be many WorklfowInvocationSteps corresponding to the same combination of WorkflowInvocation and WorkflowStep. That should feel wrong and that is because it is - when collections were added the possiblity of having many jobs for the same combination of WorkflowInvocation and WorkflowStep was added. I should have split WorkflowInvocationSteps into WorkflowInvocationSteps and WorkflowInvocationStepJobAssociations at that time but didn't. This commit now does it - effectively normalizing the ``workflow_invocation_step`` table by introducing the new ``workflow_invocation_step_job_association`` table.
Splitting up the WorkflowInvocationStep table this way allows recovering the mapped over output (e.g. the implicitly created collection from all the jobs) as well the outputs from the individual jobs (by walking WorkflowInvocationStep -> WorkflowInvocationStepJobAssociation -> Job -> JobToOutput*Association).
This split up involves failrly substantial changes to the workflow module interface. Any place a list of WorkflowInvocationSteps was assumed, I reworked it to just expect a single WorkflowInvocationStep. I vastly simplified recover_mapping to just use the persisted outputs (this was needed in order to also implment empty collection mapping in workflows). This also fixes a bug (or implements a missing feature) where Subworkflow moudles had no recover_mapping methods - so for instance if a tool that produces dynamic collections appeared anywhere in a workflow after a subworkflow step - that workflow would not complete scheduling properly.
Now that we have a way to reference the set of jobs corresponding to a workflow step within an invocation, we can start to track partial scheduling of such steps. This is outlined in https://github.com/galaxyproject/galaxy/issues/3883 and refactoring toward this goal is included here - including adding a state to WorkflowInvocationStep so Galaxy can determine if it has started scheduling this step and an index when scheduling jobs so it can tell how far into a scheduling things have gone as well as augmenting the tool executor to take a maximum number of jobs to execute and allow recovery of existing jobs for collection building purposes.
*Applications*
These changes will enable:
- A simple, consistent API for finding workflow outputs that can be consumed by Planemo for testing workflows.
- Mapping over empty collections in workflows.
- Re-scheduling workflow invocations that include subworkflow steps.
- Partial scheduling within steps requiring a large number of jobs when scheduling workflow invocations.
At the insistence of @nsoranzo, be more explicit about waiting for workflow test in API tests. While I was in there I also setup new abstractions for waiting on state and updated various workflow cancelling tests to use test_history context for better error reporting.
xref #1675
This is of limited utility since we don't really expose the name - and intentionally so. Related open bugs/enhancements that still need to be addressed are:
- Applying rename to the collection (in addition to the elements) - #1680.
- Download of collection elements with element identifier instead of the name: #2023 / #2140.
John Chilton