This is based on work by Hector del Risco at the University of Florida.
Various tool-style parameters can be defined in job_resource_params_conf.xml (see sample) and these define the pool of all possible resource parameters that can be injected into a tool. These can be groupped together and assigned to tools in job_conf.xml (see documented changes to job_conf.xml.sample_advanced).
A conditional is then injected into each tool that gives the end-user the ability to select these parameters. The values of these parameters (as a dict) can then be obtained by dynamic job destination that define a 'resource_params' argument.
Lots here still needs to be tested/verified - can workflows exported from Galaxies without resource parameters (or different ones, be reused in Galaxies with resource parameters) (I think yes?)? can workflows exported with resource parameters be used in other Galaxies (I think yes?), how best to present UI while running workflows? (maybe nothing needs to be changed), should exported job parameter data be swept clean (say when exporting a workflow for instance).
A great extension to this would be to support conditional when defining parameters.
Finally, the Galaxy team will make a reasonable effort to support this syntax for job_conf.xml, job_resource_params_conf.xml, and internal interface for job parameters for sometime once this changeset has reached -stable but until these interfaces may change.
and add a destination_total_concurrent_jobs limit that prevents Galaxy from
queueing *any* jobs at a destination or tagged group of destinations once the
limit is reached.
In particular ampq_publish_retry can now be set to True to retry disconnected connections on publication. Hopefully, this will fix producer-related disconnect problems seen in production.
Add new module - galaxy.util.specs - with utilities for things I predict would be common spec types.
This simplifies some things in terms of defaults and empty values that means other simplifications outside the LWR job runner constructor as well (e.g. for ssl_ parameters).
... or they will not work correctly with the LWR. This is making explicit a convention that was already followed by all devteam tools. If need arises for multiple paths in the same data table or there is an important tool out there that does not label it's paths as "path" let me know and we can try to implement some XML markup in the tool_data_table.xml file to declare these fields as paths.
Use new assumption to cleanup what LWR considers unstructured paths (unstructured paths is no longer even a good name for this concept unfortunately).
Create ParamsWithSpecs class in galaxy.util. If we are going to build out specs for all LWR client parameters - would be nice if they could be reused on the LWR server side where relevant (e.g. AMQP connection parameters).
Slightly modified implementation that extends collections.defaultdict so that operations like iterating over parameters, fetching keys, etc... are available so LWR runner param handling code can look like destination param handling code.
Bug report from takadonet of collection request with 280 items only creating 60 - want to know if request got chopped off by proxy or paste, there was some logic error in my code, or if it hit some sort of db/transaction limit. This logging should help dianose that (hopefully). If I find the bug, I will remove this extra logging.
Bug report from takadonet of collection request with 280 items only creating 60 - want to know if request got chopped off by proxy or paste, there was some logic error in my code, or if it hit some sort of db/transaction limit. This logging should help dianose that (hopefully). If I find the bug, I will remove this extra logging.
Daniel Blankenberg