Upgrade syntax using `pyupgrade --py36-plus` .
Manually drop several `six` imports.
Also:
- Remove broken pr_cache in scripts/bootstrap_history.py
- Fix broken prefix removal in lib/galaxy/tool_util/deps/mulled/mulled_build.py
After updating the MTS to python 3 we saw this:
```
AttributeError: 'NoneType' object has no attribute 'data'
File "galaxy/web/framework/middleware/sentry.py", line 43, in __call__
iterable = self.application(environ, start_response)
File "/srv/toolshed/main/venv/lib/python3.8/site-packages/paste/translogger.py", line 69, in __call__
return self.application(environ, replacement_start_response)
File "/srv/toolshed/main/venv/lib/python3.8/site-packages/paste/recursive.py", line 85, in __call__
return self.application(environ, start_response)
File "routes/middleware.py", line 141, in __call__
response = self.app(environ, start_response)
File "/srv/toolshed/main/venv/lib/python3.8/site-packages/paste/httpexceptions.py", line 640, in __call__
return self.application(environ, start_response)
File "galaxy/web/framework/base.py", line 145, in __call__
return self.handle_request(environ, start_response)
File "galaxy/web/framework/base.py", line 224, in handle_request
body = method(trans, **kwargs)
File "tool_shed/webapp/controllers/repository.py", line 880, in display_tool
repository, tool, valid, message = tv.load_tool_from_changeset_revision(repository_id,
File "tool_shed/tools/tool_validator.py", line 216, in load_tool_from_changeset_revision
self.handle_sample_files_and_load_tool_from_tmp_config(repo,
File "tool_shed/tools/tool_validator.py", line 164, in handle_sample_files_and_load_tool_from_tmp_config
sample_files, deleted_sample_files = self.get_list_of_copied_sample_files(repo, changeset_revision, dir=work_dir)
File "tool_shed/tools/tool_validator.py", line 136, in get_list_of_copied_sample_files
fh.write(fctx.data())
```
in https://sentry.galaxyproject.org/sentry/toolshed/issues/903796/.
This should fix that.
`ctx_file` was a bytestring, so in `get_file_context_from_ctx` we'd
never find the sample file we're looking for, because
`if filename == ctx_file_name:` would fail since `filename` is a
bytestring and ctx_file_name a unicode string.
by creating the `hg_repo` property on the `Repository` object. This is
only used in the toolshed, meaning we can move get_repo_for_repository
to lib/tool_shed/util/hg_util.py.
`get_repo_for_repository` takes up considerable time when hitting the
`/api/repositories/updates` route in the tool shed. This is hit at
least whice per call
(https://gist.github.com/mvdbeek/6a58b5f4fe9489dd9a15109da74a0c86#file-gistfile1-txt-L7),
so making this a cached property should result in a nice speedup.
This is (so far a subset of) the tool utilities from galaxy-lib. galaxy-lib mocked out a fake galaxy.tools to allow these to all exist below the galaxy.tools - but this was a terrible hack - in order for the new packages to be true subsets of Galaxy they need to be refactored into a new package that doesn't depend on models, SA, galaxy.jobs, etc...
[x] galaxy.tools.deps -> galaxy.tool_util.deps
[x] galaxy.tools.loader -> galaxy.tool_util.loader
[x] galaxy.tools.cwl -> galaxy.tool_util.cwl
[x] galaxy.tools.loader -> galaxy.tool_util.loader
[x] galaxy.tools.fetcher -> galaxy.tool_util.fetcher
[x] galaxy.tools.locations -> galaxy.tool_util.locations
[x] galaxy.tools.parser -> galaxy.tool_util.parser
Fixes the following exception:
```
AttributeError: 'NoneType' object has no attribute 'id'
File "galaxy/web/framework/middleware/sentry.py", line 43, in __call__
iterable = self.application(environ, start_response)
File "/srv/toolshed/main/venv/local/lib/python2.7/site-packages/paste/translogger.py", line 69, in __call__
return self.application(environ, replacement_start_response)
File "/srv/toolshed/main/venv/local/lib/python2.7/site-packages/paste/recursive.py", line 85, in __call__
return self.application(environ, start_response)
File "routes/middleware.py", line 141, in __call__
response = self.app(environ, start_response)
File "/srv/toolshed/main/venv/local/lib/python2.7/site-packages/paste/httpexceptions.py", line 640, in __call__
return self.application(environ, start_response)
File "galaxy/web/framework/base.py", line 143, in __call__
return self.handle_request(environ, start_response)
File "galaxy/web/framework/base.py", line 222, in handle_request
body = method(trans, **kwargs)
File "galaxy/webapps/tool_shed/controllers/repository.py", line 888, in display_tool
tool_state = tool_util.new_state(trans, tool, invalid=False)
File "tool_shed/util/tool_util.py", line 211, in new_state
log.debug('Failed to build tool state for tool "%s" using standard method, will try to fall back on custom method: %s', tool.id, e)
```
(from https://sentry.galaxyproject.org/sentry/toolshed/issues/279820/).
Might also fix some other issues when redering workflow svg for
workflows with invalid tools.
- Add flake8-import-order to flake8 Pipfile and remove py27-lint-imports
and py27-lint-imports-include-list tox envs
- Fix most E201 and E202 errors reported by flake8-import-order v0.15,
but pin flake8-import-order to v0.14.3 until
https://github.com/PyCQA/flake8-import-order/issues/123
is fixed
This let us drop 2 jobs on Travis per each job.
This means we don't rely on the Tool Shed metadata anymore and instead
deduce the tool lineage via the tool_id, which contains the version.
Version comparison then happens via LooseVersions.
This works for toolshed and non-toolshed tools, restoring proper
grouping in the tool panel and version switches, even for tool shed
tools whose lineage is broken.
John Chilton