Pydantic <2.7.4 is broken on Python>=3.12.4 , see
https://github.com/pydantic/pydantic/issues/9609 .
Fixed upstream in https://github.com/pydantic/pydantic/pull/9639 .
Fix the following traceback in startup test on Python 3.12:
```
Traceback (most recent call last):
File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/bin/galaxyctl", line 10, in <module>
sys.exit(galaxyctl())
^^^^^^^^^^^
File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.12/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.12/site-packages/click/core.py", line 1682, in invoke
cmd_name, cmd, args = self.resolve_command(ctx, args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.12/site-packages/click/core.py", line 1729, in resolve_command
cmd = self.get_command(ctx, cmd_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.12/site-packages/gravity/cli.py", line 59, in get_command
return name_to_command(name)
^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.12/site-packages/gravity/cli.py", line 45, in name_to_command
mod = __import__(mod_name, None, None, ["cli"])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.12/site-packages/gravity/commands/cmd_stop.py", line 4, in <module>
from gravity import process_manager
File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.12/site-packages/gravity/process_manager/__init__.py", line 14, in <module>
from gravity.config_manager import ConfigManager
File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.12/site-packages/gravity/config_manager.py", line 18, in <module>
from gravity.state import ConfigFile, service_for_service_type
File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.12/site-packages/gravity/state.py", line 42, in <module>
class ConfigFile(BaseModel):
File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.12/site-packages/pydantic/v1/main.py", line 286, in __new__
cls.__try_update_forward_refs__()
File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.12/site-packages/pydantic/v1/main.py", line 807, in __try_update_forward_refs__
update_model_forward_refs(cls, cls.__fields__.values(), cls.__config__.json_encoders, localns, (NameError,))
File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.12/site-packages/pydantic/v1/typing.py", line 554, in update_model_forward_refs
update_field_forward_refs(f, globalns=globalns, localns=localns)
File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.12/site-packages/pydantic/v1/typing.py", line 520, in update_field_forward_refs
field.type_ = evaluate_forwardref(field.type_, globalns, localns or None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.12/site-packages/pydantic/v1/typing.py", line 66, in evaluate_forwardref
return cast(Any, type_)._evaluate(globalns, localns, set())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: ForwardRef._evaluate() missing 1 required keyword-only argument: 'recursive_guard'
```
Example: https://github.com/galaxyproject/galaxy/actions/runs/9581802216/job/26419972297
When enabling the `warn_unused_ignores = True` option of mypy, this
was making the `test_galaxy_packages` tests fail with:
```
mypy .
Traceback (most recent call last):
File "/tmp/gxpkgtestenvu3fgDr/bin/mypy", line 8, in <module>
sys.exit(console_entry())
File "/tmp/gxpkgtestenvu3fgDr/lib/python3.10/site-packages/mypy/__main__.py", line 15, in console_entry
main()
File "mypy/main.py", line 100, in main
File "mypy/main.py", line 182, in run_build
File "mypy/build.py", line 192, in build
File "mypy/build.py", line 266, in _build
File "mypy/build.py", line 2942, in dispatch
File "mypy/build.py", line 3333, in process_graph
File "mypy/build.py", line 3414, in process_fresh_modules
File "mypy/build.py", line 2110, in fix_cross_refs
File "mypy/fixup.py", line 52, in fixup_module
File "mypy/fixup.py", line 127, in visit_symbol_table
File "mypy/lookup.py", line 49, in lookup_fully_qualified
AssertionError: Cannot find component 'util' for 'galaxy.util'
make: *** [Makefile:96: mypy] Error 1
```
Moreover, installing packages in editable mode was hiding some issues
with how files from other packages were accessed.
This commit creates two new packages - below a description of these packages and why they are being created.
galaxy-schema: This package contains the pydantic models that power the API. The purpose of packaging them is to ensure they can be reused by clients with minimal external dependencies. Reusing the schema in client code would very quickly provide rapid documentation and validation and static checking for Python clients using the Galaxy API.
galaxy-tool-shed: This package contains the tool shed server code. This has been a long term project to allow the tool shed to be spun out but maintain real galaxy dependencies and maintain testing. More discussions around this project can be found as part of https://github.com/galaxyproject/galaxy/pull/8830.
This reverts commit 353c08cbb9 .
pysam 0.20.0 doesn't build for Python 3.11 on macOS, so we need to
temporarily add the unnecessary Cython runtime dependency that
pysam 0.21 brings in.
- Update Python dependencies that didn't have a cp311 wheel: h5py and
psycopg2-binary.
- For packages for which there is no recent version that works on all
Python versions supported by Galaxy (numpy and scipy), replace old
requirement with multiple Python-version-specific requirements.
Fix `pysam.index()` calls which now use samtools 1.16 where
`samtools index input output` doesn't work any more if `output` is the path
of an already existing file, but `samtools index -o output input` works
fine.
Also:
- Pre-create the (empty) index file for mock test datasets as a unit test
reproducer of the above issue.
- Fix `test_cram` unit test that wasn't checking the return value of
`CRAM.set_index_file()`, which was in fact returning False because
`test-data/2.cram` was not sorted (updated now).
Matthias Bernt