Add RemoteTransferAction - to allow LWR server to stage files up and down by downloading/uploading them. Still need to augment Galaxy to support such an endpoint for job-related files.
At high-level there are two core enhancements here - ability to kill message queue driven LWR jobs (though due to regressions in latest galaxy release there is not longer a way to initiate this from the GUI I don't think) and additional state transition (LWR+MQ jobs will now transition from queue to running properly, previously they just went from queue to complete).
A lot of other small changes to LWR client aimed at getting file size of lwr_client/client.py and lwr_client/manager.py under control - see LWR commit log for more details.
This updates the LWR client through LWR changeset 2d9f333.
To use LWR via message queue, ensure each such destination defines "default_file_action" as "remote_copy" and the LWR base staging directory pointed to by the "jobs_directory" parameter. Finally add a "url" argument to LWR job runner plugin in job_conf.xml specifing the amqp URL to target (by default this will target the default ("_default_") LWR job manager - this can be overridden by specifing a "manager" attribute the plugin element in job_conf.xml).
Some of the implementation details here are bit hacky now to localize changes to LWR, once Dannon has merged his work - a new job runner base class or option should be implemented to account for this truely asynchronous job running implementation. (Comments inline about this.)
This updates the LWR client through LWR changeset 59ad1ea03448.
The LWR server and client can be driven via message queues, though this changeset doesn't update the LWR runner itself to allow this - only the client itself. The runner however has been updated to support some more minor changes that enabled the LWR client to be message queue driven.
In particular, a new action type was added "remote_copy" that does a file-system copy of files being staged in or out like "copy" - but it does so on the LWR side instead of the Galaxy client. In this scenario these actions are serialized and transferred with the "launch" command to the LWR - potentially eliminating many HTTP calls.
Paired with this, more processing can happen on the client side - namely calculating remote paths and properties - and if configured in this fashion the LWR "setup" step can be eliminated - allowing the LWR client to completely specify a job with a single one-directional message (very useful for message queues). While modifying the LWR to be able to behave in this fashion was done to enable message queues - a traditional HTTP driven LWR can use these features (remote copy, client-side path calculation, etc...) and job_conf.xml.sample_advanced has been updated to reflect this.
For more information on changes to the LWR client this encompasses see the following LWR changesets:
https://bitbucket.org/jmchilton/lwr/commits/bacaa5c840bbf41952fcafe6e0deadd306c868fahttps://bitbucket.org/jmchilton/lwr/commits/c8254d4525fa19f6761c6a75506d7b2bbdbced9ehttps://bitbucket.org/jmchilton/lwr/commits/1f068927a677bbfb932c475e5a4dad9885c61071https://bitbucket.org/jmchilton/lwr/commits/bde8cbe6f5736a82d5fa20b322cb34d9d4e8fbeehttps://bitbucket.org/jmchilton/lwr/commits/993910fa0c5a0ac2be72b2d70eeaaa0767e18032
Carl Eberhard