Previously I was attempting to mimic Galaxy's behavior of copying these files to a fixed output location after execution on the remote server before setting metadata. This was an un-needed copy since it appears to easier to just send the metadata generation code the expected path on the working directory.
This directory is required to exist but usually unneeded because Galaxy creates this directory on startup. However, when setting metadata remotely with the LWR that remote Galaxy instance may have never been started and this directory may not exist.
Add separate examples for MQ driven stuff - this should clear up things and make it clear which new parameters are required when using message queue and which are no longer required. Add more documentation for remote dependency resolution. Document rewrite_parameters.
Among other smaller fixes, this update causes the LWR client/runner to pass more tool dependency context through to the remote LWR server when `dependency_resolution` is set to `remote`. This additional context allows tool shed packages to be resolved remotely (i.e. allows proper use of ToolShedPackageDependencyResolver on the remote LWR server).
It remains the responsibility of the deployer to ensure identical versions of tool shed packages are installed on the remote LWR server and the local Galaxy instance.
Previously LWR jut serialized tool requirements, by transitioning to this class LWR should be able to resolve tool shed installed packages as well - requiring the additional repository and tool dependency context.
With expanded test case that checks such a connection. This takes care of of subsequent connections by iteratively updating hid_to_output_pair correctly (... I think, test works with at least two).
I changed this in some other places in the code before committing but this part stayed broken. Collections will match on order, type, and number of elements (pushing the problem to UI) - element identifiers can be different but will be preserved when possible.
Tool test is mildly useful to verify this works for single execution - but more useful as basis for future changesets testing mixed collection and subcollection mapping tool executions.
Collections can be mapped over 'data' parameters and sufficiently nested collections can be mapped over 'data_collection' parameters (for instance a list of 5 pairs can be supplied to a tool taking in a pair and 5 jobs will be executed). I (perhaps poorly) term these concepts collection mapping and subcollection mapping.
Prior to this changeset - the API for doing collection 'mapping' and 'subcollection mapping' was somewhat more divergent and the tool execution code explicitly forbid doing both kinds of mappings in the same tool execution even if the effective collections could be matched (e.g. it could not map a 'data' parameter over a list of 5 datasets and a pair parameter a list of 5 pairs in the same execution).
This changeset should remedy this - as long as the effective collection mappings can match up such jobs should be possible. The workflow editor (and I think runner) already thought this was possible, so this changeset reduces the tool-workflow impedance mismatch - an existing problem exacerbated by recent dataset collections introduction.
This all needs much more testing - test workflows execute this way, functional test of a tool execution that combines collection mapping and subcollection mapping, etc....
Daniel Blankenberg