- Create a new project galaxy-files that does not depend on anything but galaxy-util.
- Add a dependency in galaxy-data on galaxy-files, use typing in galaxy.data.sniff to make this clear.
- Cleanup the galaxy-files unit tests to not depend on galaxy.datatypes.sniff code - this wasn't ideal to start with.
- Add a small integration-ish unit test test_sniff_file_sources.py to test that piece of sniff.py that depends on galaxy-files.
Fix https://github.com/galaxyproject/galaxy/issues/12412 .
We have a dependency on `Tempita` (through `sqlalchemy-migrate`), which is
abandonware and uses the `use_2to3=True` option of `setup()` in its
`setup.py` .
The new Setuptools v58.0.0 removed support for 2to3 during builds:
https://github.com/pypa/setuptools/blob/main/CHANGES.rst#v5800
As a result Tempita is installed at even older version and importing it
raises a SyntaxError.
For Galaxy as a full application this has been solved by creating a wheel on
https://wheels.galaxyproject.org . But the `galaxy-data` package (which also
depends on `sqlalchemy-migrate`) cannot automatically install the
tempita-0.5.2 wheel from our alternative package index when installed via
`pip install galaxy-data` as there is no syntax to specify that.
I've therefore resurrected Tempita from the Bitbucket Mercurial repo (thanks
to Software Heritage backups!
https://bitbucket-archive.softwareheritage.org/projects/ia/ianb/tempita.html
), converted to a GitHub repo and updated its code to Python 3.6 with 2to3
and pyupgrade.
I'm still seeing errors in some places, and just changing everything to
SimpleJSON seems a bit risky at this point.
We'll have to live with
```
File "/cvmfs/main.galaxyproject.org/galaxy/lib/galaxy/datatypes/registry.py", line 590, in change_datatype
data.init_meta(copy_from=data)
File "/cvmfs/main.galaxyproject.org/galaxy/lib/galaxy/model/__init__.py", line 2783, in init_meta
return self.datatype.init_meta(self, copy_from=copy_from)
File "/cvmfs/main.galaxyproject.org/galaxy/lib/galaxy/datatypes/data.py", line 171, in init_meta
dataset.metadata = copy_from.metadata
File "/cvmfs/main.galaxyproject.org/galaxy/lib/galaxy/model/__init__.py", line 2705, in set_metadata
self._metadata = self.metadata.make_dict_copy(bunch)
File "/cvmfs/main.galaxyproject.org/galaxy/lib/galaxy/model/metadata.py", line 161, in make_dict_copy
rval[key] = self.spec[key].param.make_copy(value, target_context=self, source_context=to_copy)
File "/cvmfs/main.galaxyproject.org/galaxy/lib/galaxy/model/metadata.py", line 281, in make_copy
return copy.deepcopy(value)
File "/cvmfs/main.galaxyproject.org/deps/_conda/envs/_galaxy_/lib/python3.6/copy.py", line 161, in deepcopy
y = copier(memo)
File "/cvmfs/main.galaxyproject.org/galaxy/lib/galaxy/model/custom_types.py", line 232, in __deepcopy__
return MutationList(MutationObj.coerce(self._key, copy.deepcopy(self[:])))
AttributeError: 'MutationList' object has no attribute '_key'
```
for now.
I was going back and forth between sqlalchemy-json and
sqlalchemy-mutable. sqlalchemy-mutable is more lenient with JSON
primitive values being set on a mutable field, so I preferred that.
However we're seeing
```
Traceback (most recent call last):
File "/Users/mvandenb/src/galaxy/lib/galaxy/tools/__init__.py", line 1649, in handle_single_execution
rval = self.execute(
File "/Users/mvandenb/src/galaxy/lib/galaxy/tools/__init__.py", line 1747, in execute
return self.tool_action.execute(self, trans, incoming=incoming, set_output_hid=set_output_hid, history=history, **kwargs)
File "/Users/mvandenb/src/galaxy/lib/galaxy/tools/actions/__init__.py", line 529, in execute
handle_output(name, output)
File "/Users/mvandenb/src/galaxy/lib/galaxy/tools/actions/__init__.py", line 442, in handle_output
data.init_meta(copy_from=metadata_source)
File "/Users/mvandenb/src/galaxy/lib/galaxy/model/__init__.py", line 2897, in init_meta
return self.datatype.init_meta(self, copy_from=copy_from)
File "/Users/mvandenb/src/galaxy/lib/galaxy/datatypes/data.py", line 181, in init_meta
dataset.metadata = copy_from.metadata
File "/Users/mvandenb/src/galaxy/lib/galaxy/model/__init__.py", line 2808, in set_metadata
self._metadata = self.metadata.make_dict_copy(bunch)
File "/Users/mvandenb/src/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 272, in __set__
self.impl.set(
File "/Users/mvandenb/src/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 865, in set
value = self.fire_replace_event(
File "/Users/mvandenb/src/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 873, in fire_replace_event
value = fn(
File "/Users/mvandenb/src/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/ext/mutable.py", line 483, in set_
value = cls.coerce(key, value)
File "/Users/mvandenb/src/sqlalchemy-mutable/sqlalchemy_mutable/mutable.py", line 280, in coerce
converted_obj = cls._convert(obj)
File "/Users/mvandenb/src/sqlalchemy-mutable/sqlalchemy_mutable/mutable.py", line 317, in _convert
return tracked_type(obj, root)
File "/Users/mvandenb/src/sqlalchemy-mutable/sqlalchemy_mutable/mutable_dict.py", line 80, in __init__
super().__init__(self._convert_mapping(source))
File "/Users/mvandenb/src/sqlalchemy-mutable/sqlalchemy_mutable/mutable.py", line 342, in _convert_mapping
return {
File "/Users/mvandenb/src/sqlalchemy-mutable/sqlalchemy_mutable/mutable.py", line 343, in <dictcomp>
key: self._convert_item(item) for key, item in mapping.items()
File "/Users/mvandenb/src/sqlalchemy-mutable/sqlalchemy_mutable/mutable.py", line 334, in _convert_item
return self._convert(item, self.root)
File "/Users/mvandenb/src/sqlalchemy-mutable/sqlalchemy_mutable/mutable.py", line 319, in _convert
obj.root = root
File "/Users/mvandenb/src/sqlalchemy-mutable/sqlalchemy_mutable/mutable.py", line 431, in __setattr__
return super().__setattr__(name, obj)
File "/Users/mvandenb/src/sqlalchemy-mutable/sqlalchemy_mutable/mutable.py", line 390, in root
for child in self._tracked_children:
File "/Users/mvandenb/src/sqlalchemy-mutable/sqlalchemy_mutable/mutable.py", line 398, in _tracked_children
self.__getattribute__(name) for name in self._tracked_attr_names
AttributeError: 'MutableList' object has no attribute '_tracked_attr_names'
```
with jobs that are started on before 20.09 and that finish on 21.01.
sqlalchemy-json doesn't have that problem.
sqlalchemy-mutable subclasses types, and not all libraries handle this
well (we would need to add CoercedBool isinstance check to the json
encoder), and pyyaml doesn't work with subclassed built-in types at all.
John Chilton