Implement maximum_workflow_jobs_per_scheduling_iteration to test this and mitigate memory issues associated with scheduling large workflow steps (maybe).
Fixes#3883.
Create mapped over collections ahead of time instead of at the end only if they are valid.
- This way if there is a problem that problem can be reported in the state of the mapped collection.
- This way datasets can just be hidden instead of displayed and then hidden (fixes#1790).
- I think this should improve performance and prevent locks because we no longer have conflicting threads trying to write dataset state.
- We populate the collection as we go so we can recover and continue scheduling the mapped step (needed for #3883)
If one maps a tool over an empty collection, one should get an empty output collection for each of the tool's outputs.
This should have always been a valid operation I think, the code just wasn't quite structured right when the collection work was merged - there were too many places I was assuming the existence of at least one actual output or input that was mapped over to continue going. This reworks all of that and seems to handle empty collections appropriately.
Fixes#4025 (not being informing the user of the operation, but actually by allowing the operation - which it should be allowed I think).
Workflow Invocations
--------------------
The workflow invocation outputs half of this is relatively straight forward. It is modelled somewhat on job outputs, output datasets and output dataset collections are now tracked for each workflow invocation and exposed via the workflow invocation API. This required adding new tables (linked to WorkflowInvocations and WorkflowOutputs) that track these output associations.
Previously one could imagine backtracking this information for simple tool steps via the WorkflowInvocationStep -> Job table, but for steps that have many jobs (i.e. mapping over a collection) or for non-tool steps such information was more difficult to recover (and simply couldn't be recovered from the API at all or even internally without significant knowledge of the underlying workflow).
Workflow Invocation Steps
-------------------------
Tracking the outputs of WorkflowInvocationSteps was not previously done at all, one would have to follow the Job table as well. A signficant downside to this is that one cannot map over empty collections in a workflow - since no such job would exist. Tracking job outputs for WorkflowInvocationSteps is not a simple matter of just attaching outputs to an existing table because we had no concept of a workflow step tracked - since there could be many WorklfowInvocationSteps corresponding to the same combination of WorkflowInvocation and WorkflowStep. That should feel wrong and that is because it is - when collections were added the possiblity of having many jobs for the same combination of WorkflowInvocation and WorkflowStep was added. I should have split WorkflowInvocationSteps into WorkflowInvocationSteps and WorkflowInvocationStepJobAssociations at that time but didn't. This commit now does it - effectively normalizing the ``workflow_invocation_step`` table by introducing the new ``workflow_invocation_step_job_association`` table.
Splitting up the WorkflowInvocationStep table this way allows recovering the mapped over output (e.g. the implicitly created collection from all the jobs) as well the outputs from the individual jobs (by walking WorkflowInvocationStep -> WorkflowInvocationStepJobAssociation -> Job -> JobToOutput*Association).
This split up involves failrly substantial changes to the workflow module interface. Any place a list of WorkflowInvocationSteps was assumed, I reworked it to just expect a single WorkflowInvocationStep. I vastly simplified recover_mapping to just use the persisted outputs (this was needed in order to also implment empty collection mapping in workflows). This also fixes a bug (or implements a missing feature) where Subworkflow moudles had no recover_mapping methods - so for instance if a tool that produces dynamic collections appeared anywhere in a workflow after a subworkflow step - that workflow would not complete scheduling properly.
Now that we have a way to reference the set of jobs corresponding to a workflow step within an invocation, we can start to track partial scheduling of such steps. This is outlined in https://github.com/galaxyproject/galaxy/issues/3883 and refactoring toward this goal is included here - including adding a state to WorkflowInvocationStep so Galaxy can determine if it has started scheduling this step and an index when scheduling jobs so it can tell how far into a scheduling things have gone as well as augmenting the tool executor to take a maximum number of jobs to execute and allow recovery of existing jobs for collection building purposes.
*Applications*
These changes will enable:
- A simple, consistent API for finding workflow outputs that can be consumed by Planemo for testing workflows.
- Mapping over empty collections in workflows.
- Re-scheduling workflow invocations that include subworkflow steps.
- Partial scheduling within steps requiring a large number of jobs when scheduling workflow invocations.
Also, merge its 3 parameters `gzip_only`, `bz2_only`, `zip_only` into
`compressed_formats` (a list of allowed formats).
As a consequence of the changes in `get_fileobj()`, update:
- `files_diff()`
- `get_file_peek()`, which now determines that a file is binary when a
`UnicodeDecodeError` exception is raised and doesn't need
`is_multi_byte` any more
- `iter_headers()` and `get_headers()`, which now return Unicode and don't
need `is_multi_byte` parameter any more
As a consequence of the changes in `get_file_peek()`, update:
- `set_peek()`, which now doesn't need `is_multi_byte` any more
As a consequence of the changes in `get_headers()`, update:
- `guess_ext` and `is_column_based()`, which now determine that a file is
binary when a `UnicodeDecodeError` exception is raised and don't need
`is_multi_byte` any more
As a consequence of the changes to `guess_ext`, update:
- `handle_uploaded_dataset_file() doesn't need `is_multi_byte` any more
Also, remove duplicated calls to `get_file_peek()` in
lib/galaxy/datatypes/molecules.py and lib/galaxy/datatypes/msa.py
The `is_multi_byte` was not removed from the signature of `get_file_peek()`
and `set_peek()` in order to preserve compatibility for ToolShed datatypes,
thanks @jmchilton for the review.
We now set a non-zero exit code to actually set a job to failed.
I belive this is a reasonable action, and it avoids green, empty datasets
with log messages like:
```
galaxy.jobs.runners ERROR 2017-11-14 13:32:06,469 (19020/860246.torque6.curie.fr) Job output not returned from cluster: [Errno 2] No such file or directory: '/data/users/mvandenb/gx
124/tmp_nfs/jwd/019/19020/galaxy_19020.o'
galaxy.jobs.runners DEBUG 2017-11-14 13:32:06,491 (19020/860246.torque6.curie.fr) Unable to cleanup /data/users/mvandenb/gx124/tmp_nfs/jwd/019/19020/galaxy_19020.o: [Errno 2] No suc
h file or directory: '/data/users/mvandenb/gx124/tmp_nfs/jwd/019/19020/galaxy_19020.o'
galaxy.jobs.runners DEBUG 2017-11-14 13:32:06,503 (19020/860246.torque6.curie.fr) Unable to cleanup /data/users/mvandenb/gx124/tmp_nfs/jwd/019/19020/galaxy_19020.e: [Errno 2] No suc
h file or directory: '/data/users/mvandenb/gx124/tmp_nfs/jwd/019/19020/galaxy_19020.e'
galaxy.jobs.runners DEBUG 2017-11-14 13:32:06,513 (19020/860246.torque6.curie.fr) Unable to cleanup /data/users/mvandenb/gx124/tmp_nfs/jwd/019/19020/galaxy_19020.ec: [Errno 2] No su
ch file or directory: '/data/users/mvandenb/gx124/tmp_nfs/jwd/019/19020/galaxy_19020.ec'
```
This would seem to be a fairly serious memory leak in the abstract but I don't have data that it fixes anything. None the less if it gets into dev and the tests seem to pass I'll open a PR to backport it to at least 17.05 and maybe back even more.
John Chilton