I'm still seeing errors in some places, and just changing everything to
SimpleJSON seems a bit risky at this point.
We'll have to live with
```
File "/cvmfs/main.galaxyproject.org/galaxy/lib/galaxy/datatypes/registry.py", line 590, in change_datatype
data.init_meta(copy_from=data)
File "/cvmfs/main.galaxyproject.org/galaxy/lib/galaxy/model/__init__.py", line 2783, in init_meta
return self.datatype.init_meta(self, copy_from=copy_from)
File "/cvmfs/main.galaxyproject.org/galaxy/lib/galaxy/datatypes/data.py", line 171, in init_meta
dataset.metadata = copy_from.metadata
File "/cvmfs/main.galaxyproject.org/galaxy/lib/galaxy/model/__init__.py", line 2705, in set_metadata
self._metadata = self.metadata.make_dict_copy(bunch)
File "/cvmfs/main.galaxyproject.org/galaxy/lib/galaxy/model/metadata.py", line 161, in make_dict_copy
rval[key] = self.spec[key].param.make_copy(value, target_context=self, source_context=to_copy)
File "/cvmfs/main.galaxyproject.org/galaxy/lib/galaxy/model/metadata.py", line 281, in make_copy
return copy.deepcopy(value)
File "/cvmfs/main.galaxyproject.org/deps/_conda/envs/_galaxy_/lib/python3.6/copy.py", line 161, in deepcopy
y = copier(memo)
File "/cvmfs/main.galaxyproject.org/galaxy/lib/galaxy/model/custom_types.py", line 232, in __deepcopy__
return MutationList(MutationObj.coerce(self._key, copy.deepcopy(self[:])))
AttributeError: 'MutationList' object has no attribute '_key'
```
for now.
I was going back and forth between sqlalchemy-json and
sqlalchemy-mutable. sqlalchemy-mutable is more lenient with JSON
primitive values being set on a mutable field, so I preferred that.
However we're seeing
```
Traceback (most recent call last):
File "/Users/mvandenb/src/galaxy/lib/galaxy/tools/__init__.py", line 1649, in handle_single_execution
rval = self.execute(
File "/Users/mvandenb/src/galaxy/lib/galaxy/tools/__init__.py", line 1747, in execute
return self.tool_action.execute(self, trans, incoming=incoming, set_output_hid=set_output_hid, history=history, **kwargs)
File "/Users/mvandenb/src/galaxy/lib/galaxy/tools/actions/__init__.py", line 529, in execute
handle_output(name, output)
File "/Users/mvandenb/src/galaxy/lib/galaxy/tools/actions/__init__.py", line 442, in handle_output
data.init_meta(copy_from=metadata_source)
File "/Users/mvandenb/src/galaxy/lib/galaxy/model/__init__.py", line 2897, in init_meta
return self.datatype.init_meta(self, copy_from=copy_from)
File "/Users/mvandenb/src/galaxy/lib/galaxy/datatypes/data.py", line 181, in init_meta
dataset.metadata = copy_from.metadata
File "/Users/mvandenb/src/galaxy/lib/galaxy/model/__init__.py", line 2808, in set_metadata
self._metadata = self.metadata.make_dict_copy(bunch)
File "/Users/mvandenb/src/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 272, in __set__
self.impl.set(
File "/Users/mvandenb/src/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 865, in set
value = self.fire_replace_event(
File "/Users/mvandenb/src/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 873, in fire_replace_event
value = fn(
File "/Users/mvandenb/src/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/ext/mutable.py", line 483, in set_
value = cls.coerce(key, value)
File "/Users/mvandenb/src/sqlalchemy-mutable/sqlalchemy_mutable/mutable.py", line 280, in coerce
converted_obj = cls._convert(obj)
File "/Users/mvandenb/src/sqlalchemy-mutable/sqlalchemy_mutable/mutable.py", line 317, in _convert
return tracked_type(obj, root)
File "/Users/mvandenb/src/sqlalchemy-mutable/sqlalchemy_mutable/mutable_dict.py", line 80, in __init__
super().__init__(self._convert_mapping(source))
File "/Users/mvandenb/src/sqlalchemy-mutable/sqlalchemy_mutable/mutable.py", line 342, in _convert_mapping
return {
File "/Users/mvandenb/src/sqlalchemy-mutable/sqlalchemy_mutable/mutable.py", line 343, in <dictcomp>
key: self._convert_item(item) for key, item in mapping.items()
File "/Users/mvandenb/src/sqlalchemy-mutable/sqlalchemy_mutable/mutable.py", line 334, in _convert_item
return self._convert(item, self.root)
File "/Users/mvandenb/src/sqlalchemy-mutable/sqlalchemy_mutable/mutable.py", line 319, in _convert
obj.root = root
File "/Users/mvandenb/src/sqlalchemy-mutable/sqlalchemy_mutable/mutable.py", line 431, in __setattr__
return super().__setattr__(name, obj)
File "/Users/mvandenb/src/sqlalchemy-mutable/sqlalchemy_mutable/mutable.py", line 390, in root
for child in self._tracked_children:
File "/Users/mvandenb/src/sqlalchemy-mutable/sqlalchemy_mutable/mutable.py", line 398, in _tracked_children
self.__getattribute__(name) for name in self._tracked_attr_names
AttributeError: 'MutableList' object has no attribute '_tracked_attr_names'
```
with jobs that are started on before 20.09 and that finish on 21.01.
sqlalchemy-json doesn't have that problem.
sqlalchemy-mutable subclasses types, and not all libraries handle this
well (we would need to add CoercedBool isinstance check to the json
encoder), and pyyaml doesn't work with subclassed built-in types at all.
Upgrade syntax using `pyupgrade --py36-plus` .
Manually drop several `six` imports.
Also:
- Remove broken pr_cache in scripts/bootstrap_history.py
- Fix broken prefix removal in lib/galaxy/tool_util/deps/mulled/mulled_build.py
clean_multiline_string was used in multiple places across modules in different packages so I simply refactored it into galaxy.util and added a test case (used in multiple places and has tests a lower bar then some other code in galaxy.util).
The tempfilecache abstraction was only used in one place, so I've refactored it right into that file.
Together these should simplify the package tests for galaxy-data (and hopefully other packages also).
Nicola Soranzo