All binned scores moved to scofield:/cache, chrominfo.txt is copied
nightly from herbie via a cron job. This can go away when galaxyOps is
no longer in use.
spinning on the queue the monitor method now takes all of the items from the
queue at once and moves them to a waiting list, processes the waiting list,
and then sleeps.
The main job manager uses an interruptible sleep so that when new jobs arrive
they are dispatched immediately.
This should help with processor utilization quite a bit (spinning on the
queues with no sleep was pretty heavy).
Biomart doesn't work on the trunk yet (Before or after this commit).
axt to lav tool won't be functional until multiple datasets per history item is available on trunk - although I could
make it so that only the lav file is generated and no fasta files, in which case it wouldn't be an issue.
- Creating the command line has been broken up into several methods
associated with 'Tool'
- Creation of the command line is now done by DefaultToolAction and stored in
the database (thus if the tool config changes between queueing and running
of a job things will not break, and if the JobQueue was running on a
different machine from the web frontend it would not need knowledge of the
tool config)
- Broke the old JobQueue up quite a bit. New classes are:
jobs.JobQueue: jobs to be run still get added here. It has a single monitor
thread that iterates the jobs waiting for one to become runnable, then it
tries to dispatch the job
jobs.DefaultJobDispatcher: responsible for deciding how to run a job once
it is ready to run. Just a trivial implementation for now, as before is PBS
is enabled and the tool is not a 'data_source' tool, try to run it using
PBS, otherwise run locally
jobs.runners.local.LocalJobRunner: the old pool of worker threads that runs
jobs processes locally
jobs.runners.local.PBSJobRunner: PBS job runner. Only uses a single thread
to watch all queued/running jobs. This is TOTALLY UNTESTED. I haven't even
imported it. I just tried to break up Nate's code to make a state machine
that could be managed with one thread. Need to know more about the PBS
setup to test.
Next steps:
- Clean up locally running jobs when the queue is shut down (kill the jobs
and mark the state change appropriately)
- For all jobs it should be possible to requeue and jobs in the 'waiting'
state on startup
- For the PBS queue, it should be possible to recover the state of any jobs
that were queued or running on startup. However we might need more data
in the database for this
needed on the interface. It should be possible to undelete items as well.
As far as purging, if a dataset is in the deleted state and not modified for
a certain period of time we can purge the data file and set the state to
"purged" or something similar. At this point datasets can no longer be
undeleted (but we still have all the job history in the database for as
long as we want).
Daniel Blankenberg