From https://docs.python-requests.org/en/master/user/advanced/#timeouts:
> Most requests to external servers should have a timeout attached, in
> case the server is not responding in a timely manner. By default,
> requests do not time out unless a timeout value is set explicitly.
> Without a timeout, your code may hang for minutes or more.
That should make it significantly easier to debug issues when the
container monitor dies unexpectedly (which I sort of expect in the
future given that we parse output from the docker CLI).
Tested with Pulsar but not sure there is anything Pulsar specific about the implementation - simply set container_monitor_result on the job destination to 'callback' (instead of implicit default of 'file'). This causes the monitor.py script to hit a new job_ports API (that mirrors job_files API) and uses the same security mechanism (per-job keys that are only active while the job is active).
PR contains fixes for embedded Pulsar configuration handling, embedded Pulsar under uwsgi, and the introduction of new feature that allows dynamic expansion of ``$UWSGI_PORT`` inside of galaxy_infrastructure_url.
which makes some tools fail by printing to stderr:
```
galaxy.tool_util.output_checker DEBUG 2020-01-07 02:59:07,720 job failed, detected state generic_error, standard error is - [/galaxy_venv/local/lib/python2.7/site-packages/cwltool/__init__.py:17: CWLToolDeprecationWarning:
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020.
Please upgrade your Python as the Python 2.7 version of cwltool won't be
maintained after that date.
""", category=CWLToolDeprecationWarning)
]
```
which makes some tools fail by printing to stderr:
```
galaxy.tool_util.output_checker DEBUG 2020-01-07 02:59:07,720 job failed, detected state generic_error, standard error is - [/galaxy_venv/local/lib/python2.7/site-packages/cwltool/__init__.py:17: CWLToolDeprecationWarning:
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020.
Please upgrade your Python as the Python 2.7 version of cwltool won't be
maintained after that date.
""", category=CWLToolDeprecationWarning)
]
```
- enables docker destination for interactivetool tests
- fixes a wrong argument in entrypoint management
- fixes the test setup that targeted a previous iteration (which was
still called realtimetools)
- prevents deleting the working directory before python has a chance to
start the container monitor (that's the sleep command)
- fixes some odd interactive tool test assertions
- uses unicode arguments in key_type_token_mapping.py.
uwsgi arguments are always passed as bytestrings.
It breaks the Mac setup and seems just wrong, docker might not be on localhost. This needs to be made more specific in some way and should handle those socker commands not working on OS X.
PR #6925 introduced a GUI for connecting non-data (e.g. integer, boolean, color, etc..) workflow input parameter to tool input parameters (the backend for this was originally added in #1306). That ideally was just the beginning of work toward using such values in structured ways in workflows.
This PR extends tool output handling to allow producing of non-data parameters. These can serve as a source for non-data values in workflows the same work workflow input parameters can. To make such values more easy to produce, this PR also introduces Galaxy expression tools - mirroring functionality regularly used in CWL. These small JavaScript-based tools that consume inputs just like a regular Galaxy tool but that produce dictionary of non-data values.
I think these expressions will be maximally useful when paired with format 2 workflows once we allow users to load arbitrary tools (I make the case more in full here https://github.com/galaxyproject/galaxy/pull/7545#issuecomment-473894424), but I outline some potential uses there as well.
Because there is always a checklist in my PR descriptions:
- Tool definition language and plumbing and datatype for expressing expressions as jobs.
- Allow connecting expression tools to parameters in workflows, will delay evaluation of workflow so calculated value
- Example test expression tools for testing and demonstration.
There is are somethings that happen in set_metadata.py that assume the old style galaxy.json. This sets the stage for proper external metadata collection of newer style galaxy.json files.
Variant of external metadata generation that doesn't use command-line arguments or absolute paths and simply takes operates on the current, job working directory and knows what to do.
- Uses relative paths allowing the directory to be staged elsewhere via Pulsar, Condor, or Docker for instance.
- Avoids DB objects for hard-coded paths making everything more robust.
- Avoids huge command-lines which for huge collection reduction jobs might hit system limits for command-line length.
- Uses JSON dicts to communicate parameters - meaning we can avoid awkward argument parsing as this evolves (see old max_metadata_value_size handling in set_metadata.py for instance).
- Uses output names instead of database IDs to key files - so file naming is consistent and human readable.
Michael R. Crusoe