The -api parameter takes an optional argument so it can be thought of as "-api [testpath]". So in the example -with_framework_test_tools needs to come first.
Allows run_tests.sh to be executed from directories outside the galaxy home directory without returning the error "python: can't open file './scripts/functional_tests.py': [Errno 2] No such file or directory"
Runs tests in a Dockerfile with pre-migrated databases and all testing dependencies (postgres, mysql, npm, casper, phantomjs, etc...). Hopefully this makes it easier to run more interesting tests - down the road I would like to add more options for things like condor, slurm, run as real user, etc....
Rather than relying solely on exceptions back to nose/test framework - add option (--structured_data_report_file) to run_tests.sh that causes a bunch of detailed data to be dumped to the specified file in a very structured way. Includes full to_dict of the job from the API which in turn includes job metrics, command-line, job's standard error and outputs (instead of the test frameworks), as well as the tool inputs, and exceptions broken out for tool execution versus output checking.
Its all indexed in the file by the test id (without the actual test toolbox depending on knowing the test id) - so one could pair this information with the XUnit output to produce much more detailed breakdowns of the tests.
-framework will still cause just a small set of non-default, non-user facing tools to be tested and exerise various aspects of the tool and test frameworks, but now an argument can be passed in -with_framework_test_tools to allow these tools to be used as part of the normal test framework (i.e. when not using -framework). This serves two purposes - to ensure the feature coverage of existing tests remains high as the tests are migrated out of the distribution and to allow a set of tests to be available for functional (e.g. API tests) that users don't need to see.
Should be backward compatible but features better argument handling - things can appear in other orders (-id x -data_managers is the same as -data_managers -id x for instance), better error messages if arguments are missing, and more standard short argument with single dash or long argument with double dash variants of each argument are available.
John Chilton