- Runner states - Allow runner plugins to provide job finish/failure
conditions back to Galaxy so that actions can be taken on specific actions.
Currently only the WALLTIME_REACHED state is defined. This can be set on
the (Asynchronous)JobState's `runner_state` attribute. Only the slurm
runner currently does this.
- Runner state handlers - Pluggable interface for defining actions to take
when runner state actions occur. Any (non _) python file in
galaxy.jobs.runners.state_handlers will be loaded, but handler function
names should match the step in the job lifecycle where they should be used.
Only the 'failure' method is currently implemented, but adding more would
be trivial. Clever parameterization a la the dynamic runner would be a nice
improvement here.
- Destination resubmission - Destinations in the job config can specify a new
destination that jobs should be resubmitted to under certain conditions
(currently the only condition implemented is walltime_reached on the
original destination...)
- Resubmit on walltime reached state handler plugin - The actual resubmission
implementation.
- RESUBMITTED Job state - Allows resubmitted jobs to bypass the normal ready
to run checks and begin execution immediately.
- RESUBMITTED DatasetInstance state - This was the best method Carl and I
could come up with for persisting the resubmitted state so that it would be
visible in the UI. It's not perfect but I didn't want to alter the schema
and the only place it could go (job table) is not eagerloaded on history
status updates. The resubmission code will not actully set this state yet
(it is commented out) until the UI can cope with it. Bonus: once this is
done we can pretty easily add a "job concurrency limit reached" to give
users a visual cue on jobs waiting for that reason.
This has been tested pretty extensively with job recovery, concurrency limits
and multiprocess setups, which is to say that it will surely fail miserably in
production.
Approach outlined by Carl, introduces separation between Backbone model id and underlying Galaxy id. Backbone id now includes type in the id. Lots of switching around which id is used where - probably introduces some bugs if I failed to update which id is to be used or switched to wrong one.
Aysam Guerler