I like this better for three reasons:
- Since usually it is scripts producing this JSON - we have the most control at that point for determining the failure and we don't have to deal with an artificial dependency between the tool's stdio and the output.
- At some point we could potentially allow some datasets to be ok now even though the job fails.
- It is a cleaner interface at the Python level between job finish and output collection IMO (no need for isinstance checking).
This is a bugfix for the future bug of missing documentation for 17.09 and before. 18.01 will require a significant rewrite of these pages for the uWSGI and configuration changes to Galaxy and the release_17.09 branch of these docs is now being hosted https://docs.galaxyproject.org/en/release_17.09/, so backporting this configuration documentation will allow us to provide permenant links to this 17.09 varaint of all this documentation (with information about paste and galaxy.ini for instance).
This content currently includes:
- the production configuration page
- the cluster configuration page
- the job configuration page
- the scaling information page
- the ftp configuration page
- the top-level nginx page (for external auth content still linking out to the hub)
- the top-level apache page (for external auth content still linking out to the hub)
- the job metrics configuration page
Includes small tweaks to sphinx documentation generation.
- Add a quick build mode that skips source generation / Python docs (export GALAXY_DOCS_SKIP_SOURCE=1 to build this way). This speeds up review of core Sphinx docs.
- Use AutoStructify for better Markdown integration.
Updates to the documentation above and beyond what is on the corresponding Hub pages to reflect more of the state of art as of 17.09 including:
- Update the job configuration page with information about XML macros.
- Mention and link to newer style resubmission condition expressions (not just an enumerated string anymore).
- Updated list of dynamic job runner configuration parameters.
- Mention the ``raw`` attribute for ``env`` directives in the job configuration page.
- Mention the ``total_walltime`` limit added in #3217.
- Drop mentions of older URL-based job conf configuration stuff.
- Mention newer Local and Paramiko based shell runners in CLI section.
- Mention newer slurm job styles in CLI section.
- Improved Pulsar links.
- Fix dataset security link in Apache display sites config.
- Update job metrics page to link to a new Github issue instead of old Trello issues.
Without this updates of data managers fails with
```
AttributeError: 'NoneType' object has no attribute 'get'
File "galaxy/web/framework/middleware/sentry.py", line 40, in __call__
iterable = self.application(environ, start_response)
File "/bioinfo/guests/mvandenb/galaxy/.venv/local/lib/python2.7/site-packages/paste/recursive.py", line 85, in __call__
return self.application(environ, start_response)
File "/bioinfo/guests/mvandenb/galaxy/.venv/local/lib/python2.7/site-packages/paste/httpexceptions.py", line 640, in __call__
return self.application(environ, start_response)
File "galaxy/web/framework/base.py", line 136, in __call__
return self.handle_request(environ, start_response)
File "galaxy/web/framework/base.py", line 215, in handle_request
body = method(trans, **kwargs)
File "galaxy/web/framework/decorators.py", line 98, in decorator
return func(self, trans, *args, **kwargs)
File "galaxy/webapps/galaxy/controllers/admin_toolshed.py", line 1849, in update_to_changeset_revision
persist=True)
File "tool_shed/galaxy_install/metadata/installed_repository_metadata_manager.py", line 30, in __init__
metadata_dict=metadata_dict, user=None)
File "tool_shed/metadata/metadata_generator.py", line 63, in __init__
self.metadata_dict = {'shed_config_filename': self.shed_config_dict.get('config_filename', None)}
```
This should prevent errors such as:
```
galaxy.util ERROR 2017-11-17 09:20:12,842 Error parsing file /data/users/mvandenb/gx/config/shed_tool_data_table_conf.xml
Traceback (most recent call last):
File "lib/galaxy/util/__init__.py", line 217, in parse_xml
root = tree.parse(fname, parser=ElementTree.XMLParser(target=DoctypeSafeCallbackTarget()))
File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 657, in parse
self._root = parser.close()
File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1654, in close
self._raiseerror(v)
File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1506, in _raiseerror
raise err
ParseError: no element found: line 1, column 0
```
reported in https://github.com/galaxyproject/galaxy/issues/5031.
TypeError: Non-hexadecimal digit found
File "galaxy/web/framework/middleware/sentry.py", line 40, in __call__
iterable = self.application(environ, start_response)
File "/bioinfo/guests/mvandenb/galaxy/.venv/local/lib/python2.7/site-packages/paste/recursive.py", line 85, in __call__
return self.application(environ, start_response)
File "/bioinfo/guests/mvandenb/galaxy/.venv/local/lib/python2.7/site-packages/paste/httpexceptions.py", line 640, in __call__
return self.application(environ, start_response)
File "galaxy/web/framework/base.py", line 136, in __call__
return self.handle_request(environ, start_response)
File "galaxy/web/framework/base.py", line 215, in handle_request
body = method(trans, **kwargs)
File "galaxy/web/framework/decorators.py", line 98, in decorator
return func(self, trans, *args, **kwargs)
File "galaxy/webapps/galaxy/controllers/admin_toolshed.py", line 660, in manage_repositories
return self.repository_installation_grid(trans, **kwd)
File "galaxy/web/framework/helpers/grids.py", line 85, in __call__
query = self.build_initial_query(trans, **kwargs)
File "tool_shed/galaxy_install/grids/admin_toolshed_grids.py", line 334, in build_initial_query
clause_list.append(self.model_class.table.c.id == trans.security.decode_id(tool_shed_repository_id))
File "galaxy/web/security/__init__.py", line 107, in decode_id
return int(id_cipher.decrypt(obj_id.decode('hex')).lstrip("!"))
File "encodings/hex_codec.py", line 42, in hex_decode
output = binascii.a2b_hex(input)
This is due to `tool_shed_repository_ids` being a json dump of a list
in certain consitions (noticed this after installing a new repository).
This would seem to be a fairly serious memory leak in the abstract but I don't have data that it fixes anything. None the less if it gets into dev and the tests seem to pass I'll open a PR to backport it to at least 17.05 and maybe back even more.
I broke this in https://github.com/galaxyproject/galaxy/pull/4388.
I went through the other changes in that PR, and I believe this should be the
only error.
Figured out through the great detective work of @abernard.
This fixes the first error in https://github.com/galaxyproject/galaxy/issues/5031:
```
Exception in thread ToolConfWatcher.thread:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "lib/galaxy/tools/toolbox/watcher.py", line 138, in check
self.reload_callback()
File "lib/galaxy/webapps/galaxy/config_watchers.py", line 24, in <lambda>
self.tool_config_watcher = get_tool_conf_watcher(reload_callback=lambda: reload_toolbox(self.app), tool_cache=self.app.tool_cache)
File "lib/galaxy/queue_worker.py", line 92, in reload_toolbox
_get_new_toolbox(app)
File "lib/galaxy/queue_worker.py", line 111, in _get_new_toolbox
new_toolbox = tools.ToolBox(tool_configs, app.config.tool_path, app)
File "lib/galaxy/tools/__init__.py", line 226, in __init__
app=app,
File "lib/galaxy/tools/toolbox/base.py", line 1061, in __init__
super(BaseGalaxyToolBox, self).__init__(config_filenames, tool_root_dir, app)
File "lib/galaxy/tools/toolbox/base.py", line 87, in __init__
self._save_integrated_tool_panel()
File "lib/galaxy/tools/toolbox/integrated_panel.py", line 46, in _save_integrated_tool_panel
self._write_integrated_tool_panel_config_file()
File "lib/galaxy/tools/toolbox/integrated_panel.py", line 106, in _write_integrated_tool_panel_config_file
os.chmod(self._integrated_tool_panel_config, 0o644)
OSError: [Errno 2] No such file or directory: '/data/users/mvandenb/gx/config/integrated_tool_panel.xml'
```
Dannon Baker