Relative 'tool_path' directories are resolved relative to GALAXY_ROOT (or really working directory). This is probably the expected behavior - but I think it is advantageous to be able to find tools relative to the tool_conf file also. This can now be done using string.Template resolution of ${tool_conf_dir}.
LWR runner and client remains in Galaxy as is and can continue to target old LWR server. But these should be considered deprecated and will likely be dropped at some point. The LWR documentaiton in job_conf.xml.sample_advanced has been replaced with Pulsar documentation to reflect this.
Pulsar runners will need to target a Pulsar server (now with a RESTful web interface - MQ option also available). Information on upgrading an LWR instance to pulsar can be found in the [Pulsar docs](http://pulsar.readthedocs.org/en/latest/#upgrading-from-the-lwr).
Three new job runners have been added to replace the LWR runner. The `PulsarLegacyJobRunner` more or less has all of the old defaults that the LWR runner had and this one should be easiest to get working for people who are porting over old LWR servers (and will likely be most compatible with remote Windows hosts).
The `PulsarRESTJobRunner` targets a remote Pulsar server over HTTP(S) but has newer standard configuration options such as rewriting parameters at tool stage by default, remote dependency evaluation by default, etc....
The `PulsarMQJobRunner` targets a remote Pulsar server via AMQP - and has the newer defaults of the RESTful runner as well as targetting the Galaxy job files API by default. The `url` runner parameter has been renamed `amqp_url` for clarity.
These newer runners improve parameter handling in other ways such as warning deployers when ignored parameters are supplied.
$output.extra_files_path will now mimic $output.files_path behavior - writing these files to the working directory to have them copied over later. This is more correct when using outputs_to_working_directory, running jobs as the real user, running jobs through LWR/Pulsar, or when using the NestedObjectStore. This is less efficient the 90% of the time when none of these conditions hold.
There is still an open issue that for inputs - $input.extra_files_path returns a non-exist path for DiskObjectStores but throws an error for NestedObjectStores. This behavior could be synchronized by this patch (https://gist.github.com/jmchilton/d05bcf02874762cdb829) - though I do not know if this is the intended behavior of NestedObjectStores.
- Runner states - Allow runner plugins to provide job finish/failure
conditions back to Galaxy so that actions can be taken on specific actions.
Currently only the WALLTIME_REACHED state is defined. This can be set on
the (Asynchronous)JobState's `runner_state` attribute. Only the slurm
runner currently does this.
- Runner state handlers - Pluggable interface for defining actions to take
when runner state actions occur. Any (non _) python file in
galaxy.jobs.runners.state_handlers will be loaded, but handler function
names should match the step in the job lifecycle where they should be used.
Only the 'failure' method is currently implemented, but adding more would
be trivial. Clever parameterization a la the dynamic runner would be a nice
improvement here.
- Destination resubmission - Destinations in the job config can specify a new
destination that jobs should be resubmitted to under certain conditions
(currently the only condition implemented is walltime_reached on the
original destination...)
- Resubmit on walltime reached state handler plugin - The actual resubmission
implementation.
- RESUBMITTED Job state - Allows resubmitted jobs to bypass the normal ready
to run checks and begin execution immediately.
- RESUBMITTED DatasetInstance state - This was the best method Carl and I
could come up with for persisting the resubmitted state so that it would be
visible in the UI. It's not perfect but I didn't want to alter the schema
and the only place it could go (job table) is not eagerloaded on history
status updates. The resubmission code will not actully set this state yet
(it is commented out) until the UI can cope with it. Bonus: once this is
done we can pretty easily add a "job concurrency limit reached" to give
users a visual cue on jobs waiting for that reason.
This has been tested pretty extensively with job recovery, concurrency limits
and multiprocess setups, which is to say that it will surely fail miserably in
production.
John Chilton