In particular break out mixins defining context-based utilities for interacting with app, an user, and a history. In downstream work on workflow scheduling this proves nessecary and sufficient to create an context for 'execute'-ing tools (create jobs) outside of web threads in response to workflow requests.
Unit tests for some of this.
Add utility to RuleHelper to simplify bursting reasoning slightly, setup a stock rule for bursting between two static job destinations - mostly for demonstration purprose but it might be useful in some settings.
... if only in a limitted sort of way. Provide ability to hash jobs by history, workflow, user, etc... and choose among various destinations semi-randomly based on that to distribute work across destinations based on these factors.
Add a stock rule as in the form of a new dynamic destination type ("choose_one") that demonstrate using this to quickly proxy to some fixed static destinations. Add examples to job_conf.xml.sample_advanced.
Generate a workflow invocation UUID that is stored with each job in the workflow and allow dynamic job destinations to consume these. Should allow for grouping jobs from the same workflow together during resource allocation (a sort of first attempt at dealing with data locality in workflows).
In other words, send extra job destination parameters to dynamic rule functions as arguments (in addition to those dynamically populated by Galaxy itself). This enables greater parameterization of rule functions and should lead to cleaner separation of logic and data (i.e. sites can program rules that restrict access to users, but which users can be populated at a higher level in `job_conf.xml`).
For example the following dynamic job rule:
def cluster1(app, memory="4096", cores="1", hours="48"):
native_spec = "--time=%s:00:00 --nodes=1 --ntasks=%s --mem=%s" % ( hours, cores, memory )
return JobDestination( "cluster1", params=dict( native_specification=native_spec ) )
Could then be called with various parameters in job_conf.xml as follows:
<destination id="short_job" type="dyanmic">
<param id="function">cluster1</param>
<param id="hours">1</param>
</destination>
<destination id="big_job" type="dynamic">
<param id="function">cluster1</param>
<param id="cores">8</param>
<param id="memory">32768</param>
</destination>
If multiple destinations map to the same underlying resource (a very typical case), this could allow rule developer to reason about cluster as a whole - though perhaps clunkily - by supplying all destination ids mapping to that cluster.
A bunch of unit tests for galaxy.jobs.mapper and galaxy.jobs.rule_helper testing various aspects of static and dynamic job mapping and the utility class (RuleHelper) that can be supplied to dynamic job rules.
... this will allow linking to actual steps (referential integrity) if/when we store workflow requests in the database and failing faster.
Also more unit tests.
... so internals of tool XML description are only utilized in the galaxy.tools module (and submodules). Add some unit tests for this new method for finding externally referenced files.
Write tests against this module to make me feel more confident white-listing is only allowing select statements as well as tests to make me feel more confident that actual SELECT statements cannot be turned into update statements or access external resources.
Fixes multirun inside of conditional, repeats if tool form state updated (e.g. because conditional param updated or repeat block added).
More tests - ugly tests - but tests.
Allows testing both how URLs are going to map to controllers and actions as well as testing what paths will be produced by url_for.
Requires some refactoring of Galaxy framework classes and buildapp to allow stubbing out actual instantiation of controllers, etc....
This is based on work by Hector del Risco at the University of Florida.
Various tool-style parameters can be defined in job_resource_params_conf.xml (see sample) and these define the pool of all possible resource parameters that can be injected into a tool. These can be groupped together and assigned to tools in job_conf.xml (see documented changes to job_conf.xml.sample_advanced).
A conditional is then injected into each tool that gives the end-user the ability to select these parameters. The values of these parameters (as a dict) can then be obtained by dynamic job destination that define a 'resource_params' argument.
Lots here still needs to be tested/verified - can workflows exported from Galaxies without resource parameters (or different ones, be reused in Galaxies with resource parameters) (I think yes?)? can workflows exported with resource parameters be used in other Galaxies (I think yes?), how best to present UI while running workflows? (maybe nothing needs to be changed), should exported job parameter data be swept clean (say when exporting a workflow for instance).
A great extension to this would be to support conditional when defining parameters.
Finally, the Galaxy team will make a reasonable effort to support this syntax for job_conf.xml, job_resource_params_conf.xml, and internal interface for job parameters for sometime once this changeset has reached -stable but until these interfaces may change.
Unit tests verifing the fixed, correct behavior included. Bug introduced in e8c84dd715782e7c1d709d8068e6033b835f7f39 as an unintended consequence of duplicating the first dataset in the dictionary of input datasets generated by the tool action code.
Fixes https://trello.com/c/LCEKxImR.
Previously LWR jut serialized tool requirements, by transitioning to this class LWR should be able to resolve tool shed installed packages as well - requiring the additional repository and tool dependency context.
When running normal tools with normal data inputs across dataset collections in parallel - a collection will be created for each output with a "structure" matching that of the input (for instance pairs will create pairs, lists will create lists with the same identifiers, etc...). A previous changeset added the ability to run the tool in parallel - this changeset extends that functionality to create analogous collections from these parallel runs.
For example, if one filters a pair of FASTQ files a pair is created out of the result. Likewise, if one filters a list of FASTQ files - a list collection with same cardinality is built from the results.
There is a lot left TODO here - for one a lot of this logic should be moved into the dataset_collection module. The matching needs to be exact right now - not a problem for pairs (every 'element' has name 'left' or 'right') but for lists with element names - these have to match exactly - but a 'list' like samp1_l, samp2_l, samp3_l should be able to match against samp1_r, samp2_r, samp3_r and create a new list samp1, samp2, and samp3. Even if there is no matching prefixes a new 'unlabeled' list should be able to be created.
Allow replacing data parameter inputs with collections - this will cause the tool to produce multiple jobs for the submission - one for each combination of input parameters after being matched up (linked). In addition to various unit tests, functional tests demonstrate the API usage in `test/functional/api/test_tools.py`.
John Chilton