Long term this could allow Galaxy to support - multiple tooling formats (Galaxy-like YAML, CWL http://bit.ly/cwltooldesc, etc...). But I think it is also important from a purely design perspective - this is a core logic class integrating different components - they should not also be doing XML parsing.
To verify the interface for parsing tools is expressive enough to allow multiple useful implementations, I built a test YAML tool description that implements many of the same features as Galaxy but smooths out rough edges (uses exit codes for job failure by default for instance). Loading these tools is disabled by default and it is not documented how to enable them because they are not intended to be part of Galaxy's public API.
Rather than relying solely on exceptions back to nose/test framework - add option (--structured_data_report_file) to run_tests.sh that causes a bunch of detailed data to be dumped to the specified file in a very structured way. Includes full to_dict of the job from the API which in turn includes job metrics, command-line, job's standard error and outputs (instead of the test frameworks), as well as the tool inputs, and exceptions broken out for tool execution versus output checking.
Its all indexed in the file by the test id (without the actual test toolbox depending on knowing the test id) - so one could pair this information with the XUnit output to produce much more detailed breakdowns of the tests.
That change forced $output.extra_files_path to be the same thing as $output.files_path. See rational in changeset.
Some datatypes however consume this path when generated metadata - post d781366 these datatypes would be accessing the wrong directory. This fixes them.
The suggested best practice I would put forward is $input.extra_files_path and $output.files_path. $output.extra_files_path redirects to $output.files_path for newer Galaxy versions (August 2014+) so that Galaxy configurations requiring this directory be in the jobs working directory work. Datatypes consuming dataset.extra_files_path for these outputs will be broken when used with $output.files_path for the August release of Galaxy - but if this changeset is back-ported to the October 2014 release then these types and tools will work going forward (without modification).
It can be verified that the older releases are broken with the following existing test (fixed with this changeset).
./run_functional_tests.sh -framework -id composite_output
At least for the local job runner. This variant of property loading does correctly interpolate %(here) so there is no longer a need to catch those exceptions and object store related properties can use %(here).
Functional tests don't work without the tweak to imports in cloudlaunch - not sure if that has always been a problem and I am just running the tests in a different Galaxy instance or if the earlier import of util caused the problem.
Regardless I guess we shoud update to a version of bioblend that doesn't require simplejson.
Update scripts/check_eggs.py and scripts/fetch_eggs.py with logic to find the correct config file. These files should be refactored so they can share some of this code the way I refactored the database management scripts I guess.
Now appends test tools to sample tools instead of to default tools Galaxy would other wise load (config/tool_conf.xml or tool_conf.xml if these exist instead of sample) - so this is different behavior but I am unsure if it is worse. --with_framework_test_tools was added for API tests and these should probably target the distribution and not random tools someone might have configured so perhaps this new behavior is superior.
Database and tool migrations won't work if you use galaxy.ini.sample. Perhaps it should automatically try appending .sample? Or we should really be encouraging people to use the config file properly.
Fix thanks to Chong Chen at IBM - "When invoking the LSF drmma api to submit a job, the API will print the message like 'Job <78> is submitted to default queue <normal>'. So if the Galaxy want to use this API, need modify the drmaa_external_runner.py script to handle this message. LSF already support a env variable to disable this message print out, "BSUB_QUIET", set this into env can disable this message output."
A second issue was related to subsequent querying of the job state as a user other than the submitter - according to Chong Chen this has been fixed in latest LSF DRMAA changes on github (https://github.com/PlatformLSF/lsf-drmaa/commit/43e62546172b90d9f95e90273d77f4c35a0bb404).
This directory is required to exist but usually unneeded because Galaxy creates this directory on startup. However, when setting metadata remotely with the LWR that remote Galaxy instance may have never been started and this directory may not exist.
-framework will still cause just a small set of non-default, non-user facing tools to be tested and exerise various aspects of the tool and test frameworks, but now an argument can be passed in -with_framework_test_tools to allow these tools to be used as part of the normal test framework (i.e. when not using -framework). This serves two purposes - to ensure the feature coverage of existing tests remains high as the tests are migrated out of the distribution and to allow a set of tests to be available for functional (e.g. API tests) that users don't need to see.
Dave Bouvier