Commit Graph
125 Commits
Author SHA1 Message Date
davelopez e699452006 Add missing download scheme configuration to Rucio object store template for tests 2025-05-08 12:49:01 +02:00
Yakubov, Sergey 5282940a32 fix liniting 2025-05-07 16:10:47 +02:00
Yakubov, Sergey 50a1d5b6a2 add examples, refactor templates 2025-05-07 16:10:47 +02:00
mvdbeek a726a956e5 Run in extended metadata mode again 2025-03-21 10:51:01 +01:00
mvdbeek 0bb228d134 Disable celery and extended metadata
Either of these options are not compatible with rucio. Maybe more than
one connection isn't permitted ?
2025-03-21 09:33:02 +01:00
Yakubov, Sergey ae29750eec refactor rucio broker config 2025-03-20 15:29:16 -04:00
Yakubov, Sergey bd9f584163 create rucio.cfg from inside objectstore 2025-03-20 12:44:17 -04:00
Nicola Soranzo caa4995999 Update rucio image URL 2025-02-25 19:20:47 +00:00
Nicola Soranzo edd8bccf4f Drop support for Python 3.8 2025-02-25 19:20:46 +00:00
mvdbeek 7f6a243281 Merge branch 'release_24.1' into release_24.2 2025-02-11 17:07:08 +01:00
davelopez adef4f8f88 Add integration tests for user object store quota handling 2025-02-11 14:49:14 +01:00
John Chilton 7f7710d392 Implement recovering normalized workflow request from invocation. 2024-10-15 09:12:05 -04:00
davelopez 946bded098 Use UUID instead of ID in user object store test 2024-10-15 11:33:13 +02:00
John Chilton 01c42c888b Expand pre-checking of file source/object store configuration.
- UI for detailed display of errors.
- UI option to test configuration from management menu.
- API + UI for checking configuration before upgrading to new version of template.
- API + UI for checking configuration before updating current template's settings.
- Add an option during update/upgrade to allow forcing the update even if configuration doesn't validate - I don't allow creation of invalid things, but if there are problems with an existing thing - admins and power users should have recourse. It is their data.
2024-10-14 14:27:05 +02:00
mvdbeek 7572e5c41f Merge branch 'release_24.1' into dev 2024-09-20 08:54:14 +02:00
mvdbeek 2bbae08220 Merge branch 'release_24.0' into release_24.1 2024-09-20 08:27:18 +02:00
mvdbeek e1d6c93a21 Fix discovered outputs with directory metadata and distributed object
stores

The problem is that `object_store.is_private` would check in which
object store the dataset is stored, and that fails if we haven't written
to the object store yet (which is why extended metadata is a
workaround).

Fixes https://github.com/galaxyproject/galaxy/issues/17208
2024-09-19 19:51:51 +02:00
mvdbeek 03a7fda6a3 Merge branch 'release_24.1' into dev 2024-08-09 15:57:48 +02:00
mvdbeek 302328f05f Merge branch 'release_24.0' into release_24.1 2024-08-09 15:57:24 +02:00
davelopez d9051005bc Remove unnecessary assert for file_size
The file_size should be preserved in the database for traceability and other uses.
2024-08-09 14:03:53 +02:00
Anthony Bretaudeau 41741041e0 Fix worklfow typo here and there 2024-07-10 15:18:31 +02:00
mvdbeek e395ea7d6d Stabilize test_purge_while_job_running test
In the extended_metadata + remote tool_evaluation_strategy the
commandline is templated as part of the job, so we need to give the test
more time to start running before we delete outputs (as then we can't
template the command line anymore).
2024-07-04 17:35:57 +02:00
mvdbeek b8e2bfc9c7 Merge branch 'release_24.0' into release_24.1 2024-06-11 13:42:48 +02:00
mvdbeek f0e09cc8bd Fix up tests 2024-06-11 08:18:14 +02:00
mvdbeek a35033d821 Extend test to make sure disk path remains purged 2024-06-10 11:58:46 +02:00
mvdbeek 435a8cd37f Fix up test cases
Add a sleep so we can delay output purging until command line is
templated. If we don't sleep we generate a command line where the
output path is just `''`. That needs another fix!
2024-06-10 10:34:15 +02:00
John Chilton 821b31c3b2 More structured indexing for user data objects.
I think I clung to the idea of having two ways to do this too long. I did a bunch of testing with the old way (exposing integer references based on numeric database primary keys). Having a config option that if changed would break everything exisiting is also a symptom of maybe me clinging too hard for too long.

The result of dropping this option is a much cleaner API schema, simpler API objects, and better typing throughout. We can also be more certain of how things entering the Vault are stored - I think being more strucutured about this is good.

Ultimately, the future facing stuff (e.g. OAuth 2.0) is going to require UUIDs so that I can store things like refresh tokens in the store before the object has been fully created.
2024-06-07 19:20:11 -04:00
John Chilton f95da89b3d Implement file source templates 2024-05-20 12:18:47 -04:00
John Chilton d3991a696b Object Store creation templates. 2024-05-20 12:16:02 -04:00
John Chilton 9d2a45f58d Merge pull request #17540 from bwalkowi/add-onedata-objectstore
Add onedata objectstore
2024-05-19 16:41:00 -04:00
John Chilton 3c991d3b20 Fix irods test cases for code refactoring. 2024-05-14 08:45:59 -04:00
Bartosz Walkowicz 2d995bb178 Format new onedata related code 2024-05-06 11:50:00 +02:00
Bartosz Walkowicz 1f6b6a0827 Merge branch 'dev' of https://github.com/galaxyproject/galaxy into add-onedata-objectstore 2024-05-06 06:34:16 +02:00
John Chilton e03d162429 Update Azure infrastructure. 2024-05-03 13:57:58 -04:00
Łukasz Opioła cdca217f34 Docs improvements, fix camelCasing of the tls validation option in Onedata file source plugin 2024-03-06 09:33:23 +01:00
Bartosz Walkowicz bf8db0cb5d Add //space/@path option to onedata objectstore 2024-03-06 09:33:20 +01:00
Bartosz Walkowicz 86d88559ff Use onedatafilerestclient instead on onedatarestfs 2024-03-06 09:25:11 +01:00
Bartosz Walkowicz 64b70b5ffe Remove debug prints 2024-03-06 09:25:11 +01:00
Bartosz Walkowicz 48f13dd546 Add Onedata objectstore implementation integration tests 2024-03-06 09:25:08 +01:00
Marius van den Beek abe930a471 Merge pull request #17156 from SergeyYakubov/rucio-plugin
Adding object store plugin for Rucio
2024-03-05 16:11:03 +01:00
John Chilton 4bd70b47ba Merge pull request #17460 from jmchilton/filter_storage
Allow filtering history datasets by object store ID and quota source.
2024-02-20 10:45:11 -05:00
Nicola Soranzo 6ffdf8489e Rename classes derived from `IntegrationInstance` which are not test cases 2024-02-16 10:53:04 +00:00
Nicola Soranzo dcd739a063 Type annotation and cleanup of `GalaxyTestDriver and integration_module_instance()` 2024-02-16 10:53:04 +00:00
John Chilton fcdcd8dc05 WIP: history filter by objectstore/quotasource 2024-02-15 14:13:49 -05:00
Yakubov, Sergey 442632f863 refactor tests skips, better wait to rucio to start 2024-02-14 13:24:38 -05:00
Yakubov, Sergey f23ab02059 rename password variable 2024-02-14 09:03:21 -05:00
Yakubov, Sergey 0b9d28eaa1 make rucio conditional dependency 2024-02-14 09:03:21 -05:00
Yakubov, Sergey 0beef65cc9 Adding object store plugin for Rucio
https://rucio.cern.ch
2024-02-14 09:03:21 -05:00
John Chilton c106a87a90 Implement device source ID for object stores.
Allow attaching a device ID on object stores that indicates they produce equivalent paths. Implement an API endpoint that allows swapping datsets between equivalent object stores if they have not been copied by new users.
2024-02-07 11:39:51 -05:00
Nicola Soranzo e975ca3754 Update to black 2024 stable style 2024-01-30 18:09:51 +00:00