I was unable to reproduce #1531 in testing, but if the problem is something to do with stale state the following sledge hammer should fix it.
Runt the new test case with:
./run_tests.sh -api test/api/test_workflows.py:WorkflowsApiTestCase.test_delete_intermediate_datasets_pja_1
Prior to PR #1302 workflows wouldn't validate call validate at all on inputs. 1302 was perhaps an over correction that causes validate to be called on datasets that aren't yet READY but require metadata. This rolls that back somewhat, other parameters will still validate - but validation asserting the existence of metadata won't get called in workflows. This is still not ideal but after 1302 and this - we are still in a better, more secure place than beforehand.
One might argue that since we need to introduce a new validation point (right before a job is scheduled and set to QUEUED) - that we would be better off rolling back #1302 entirely. I've considered that, but I still think it is better to fail fast on the other problems, also I would rather call the validation code too many times than not enough and it would be very late in the release cycle for such a change.
Just switch the default for tool caching and only cache during initial loads of tool panels - as during a toolbox refresh. This should mean that all code from tool shed will ignore the cache and should fix#1515.
Additionally, reloading the tool was broken since if would call the load tool code and then just trust the cache. This should fix tool reloading as well.
I'm making more things configurable in case someday the BSD user wants to use Galaxy - the default shell for jobs can be configured at the galaxy or job destination level now.
Including 'TypeError: _get_config_option() got multiple values for keyword argument 'prefix' and problem when planemo's cwltool is installed in Galaxy's virtualenv.
Previously the job script underwent these checks and I had assumed since that got written later it would never be the case that tool_script.sh wouldn't be synchronized by the job script was - turns out file systems are more creative and less logical than one might expected.
Also increases the count for the number of syncs that will execute because 15 on rare occasion proved not to be enough as well as making both the count and sleep time configurable.
Long term this is just not a solution, but it is the strategy we use for mapped over output HDAs also.
Rebased and fixed based on comments from @nsoranzo indicating the previous attempt did absolutely nothing to fix the problem.
xref https://github.com/galaxyproject/tools-iuc/pull/412/files
Conflicts:
lib/galaxy/tools/actions/__init__.py
Workaround for issue #1283, which causes the loaded tool to have a
StockLineage instead of a ToolShedLineage, thus creating multiple entries
for the tool in the panel.
Conflicts:
lib/galaxy/tools/toolbox/base.py
John Chilton