mvdbeek
b8e2bfc9c7
Merge branch 'release_24.0' into release_24.1
2024-06-11 13:42:48 +02:00
Marius van den Beek
96c9be32ce
Merge pull request #18342 from mvdbeek/fix_copying_purged_files
...
[24.0] Do not copy purged outputs to object store
2024-06-11 13:35:10 +02:00
mvdbeek
f0e09cc8bd
Fix up tests
2024-06-11 08:18:14 +02:00
John Davis
13de716c2f
Merge pull request #18365 from jdavcs/241_merge_240_2
...
[24.1] Merge 24.0 into 24.1
2024-06-11 00:11:44 -04:00
John Davis
8351709800
Fix mypy after merge
2024-06-11 00:01:27 -04:00
John Davis
b2bbdcbb7f
Merge branch 'release_24.0' into release_24.1
2024-06-10 23:59:39 -04:00
John Davis
cea7eb0075
Merge pull request #18364 from jdavcs/241_bug_reverse_bool
...
[24.1] Fix check for anonymous
2024-06-10 23:35:49 -04:00
Martin Cech
d7dad3a779
Merge pull request #18358 from davelopez/24.0_fix_condition_anonymous_jobs
...
[24.0] Fix anonymous user job retrieval logic
2024-06-11 02:28:44 +02:00
John Davis
d0414c3d95
Fix check for anonymous
2024-06-10 18:45:38 -04:00
davelopez and Nicola Soranzo
9df23a64d3
Require session to list jobs instead of returning empty job list for anonymous
...
Co-authored-by: Nicola Soranzo <nicola.soranzo@earlham.ac.uk >
2024-06-10 16:33:47 +02:00
mvdbeek
3c7730c199
Delete purged outputs from object store when importing dataset for purged output
2024-06-10 16:21:22 +02:00
mvdbeek
50b7f8f8da
Do not overwrite purged or deleted states when importing datasets from job
2024-06-10 16:21:22 +02:00
mvdbeek
5ab3120270
Purge dataset if finishing job for purged output
2024-06-10 16:21:21 +02:00
davelopez and mvdbeek
a6b45e561c
Filter jobs by session instead of using the session current history for anonymous
...
Co-authored-by: mvdbeek <m.vandenbeek@gmail.com >
2024-06-10 13:34:27 +02:00
Marius van den Beek
2585d1e604
Merge pull request #18291 from jmchilton/cleanup_user_data_indexing
...
[24.1] More structured indexing for user data objects.
2024-06-10 13:27:26 +02:00
davelopez
af923d6b8f
Fix anonymous user job retrieval logic
...
For anonymous users, if a history_id is provided, do not override with the current session history and instead rely on the history accessibility
2024-06-10 12:33:11 +02:00
mvdbeek
c489eb20af
Set file_size and total_size to 0 for discovered but purged datasets
2024-06-10 12:10:53 +02:00
mvdbeek
a35033d821
Extend test to make sure disk path remains purged
2024-06-10 11:58:46 +02:00
mvdbeek
fc42b61d30
Fix metadata setting on purged outputs
2024-06-10 11:55:36 +02:00
mvdbeek
eb2e318b3d
Elevate external metadata setting to warning
...
This is a tricky one. There could be systematic issues here that an
admin would want to fix, but it could also just be the case that a user
forced a wrong datatype. Ideally we'd probably tag this as job execution
issue ? It's not very different from a job error IMO.
2024-06-10 11:53:41 +02:00
mvdbeek
03eca7cc52
Fix handling of collecting discovered but purged outputs
2024-06-10 10:34:15 +02:00
mvdbeek
435a8cd37f
Fix up test cases
...
Add a sleep so we can delay output purging until command line is
templated. If we don't sleep we generate a command line where the
output path is just `''`. That needs another fix!
2024-06-10 10:34:15 +02:00
mvdbeek
65e9605f4a
Don't fail _finish_or_resubmit if tool_stdout / tool_stderr not written
...
These are only written if the command actually ran, so we'd fail here
for instance if the outputs are deleted before the job had a chance to
run.
2024-06-08 20:09:18 +02:00
mvdbeek
8bc3f2151a
Add tool that produces all types of outputs
2024-06-08 16:27:49 +02:00
mvdbeek
a2fdcb3c72
Merge branch 'release_24.0' into release_24.1
2024-06-08 15:54:19 +02:00
mvdbeek
c6930fff72
Merge branch 'release_23.2' into release_24.0
2024-06-08 15:53:41 +02:00
Marius van den Beek
18d9bbba45
Merge pull request #18349 from mvdbeek/fix_dce_to_cwl
...
[23.2] Allow DCE as outer input to to_cwl
2024-06-08 15:43:01 +02:00
John Davis
9d15f2f1ce
Merge pull request #18353 from jdavcs/241_merge3
...
[24.1] Merge 24.0 into 24.1
2024-06-07 20:44:33 -04:00
John Chilton
821b31c3b2
More structured indexing for user data objects.
...
I think I clung to the idea of having two ways to do this too long. I did a bunch of testing with the old way (exposing integer references based on numeric database primary keys). Having a config option that if changed would break everything exisiting is also a symptom of maybe me clinging too hard for too long.
The result of dropping this option is a much cleaner API schema, simpler API objects, and better typing throughout. We can also be more certain of how things entering the Vault are stored - I think being more strucutured about this is good.
Ultimately, the future facing stuff (e.g. OAuth 2.0) is going to require UUIDs so that I can store things like refresh tokens in the store before the object has been fully created.
2024-06-07 19:20:11 -04:00
John Davis
189184e2ae
Fix mypy error (surfaced after merge)
2024-06-07 17:03:50 -04:00
John Davis
bbb10cbf61
Merge branch 'release_24.0' into release_24.1
2024-06-07 14:48:42 -04:00
John Davis
d4a4b91ac4
Merge pull request #18352 from jmchilton/241_file_sources_anon
...
Fix #18316 (anonymous file sources)
2024-06-07 14:43:44 -04:00
John Chilton
7acf35f9a5
Fix #18316... I think.
2024-06-07 14:07:52 -04:00
Marius van den Beek
f38461bd82
Merge pull request #18333 from davelopez/24.0_fix_500_for_anonymous_job_listing
...
[24.0] Fix authentication error for anonymous users querying jobs
2024-06-07 17:45:54 +02:00
mvdbeek
dff6307e91
Allow DCE as outer input to to_cwl
2024-06-07 17:29:07 +02:00
Nicola Soranzo
0fb77c875e
Merge branch 'release_24.0' into release_24.1
2024-06-07 16:28:23 +01:00
Nicola Soranzo
62eadc87c5
Merge branch 'release_23.2' into release_24.0
2024-06-07 16:26:04 +01:00
Nicola Soranzo
3817559039
Merge branch 'release_23.1' into release_23.2
2024-06-07 16:24:43 +01:00
mvdbeek
5339fabbf3
Merge branch 'release_24.0' into release_24.1
2024-06-07 16:52:46 +02:00
mvdbeek
8e6e25d7fb
Don't push purged dataset contents to object store
...
This in particular needs a lot of new tests.
We will also need to actively purge datasets in the model store import
code, since users might have purged datasets while the job ran.
Again, more tests needed.
2024-06-07 15:20:45 +02:00
mvdbeek
a1d41ab0d6
Raise appropriate exception if require path parameter is missing
2024-06-07 15:20:45 +02:00
mvdbeek
c3a5d61d61
Don't include purged outputs in from_work_dir outputs
2024-06-07 15:20:44 +02:00
mvdbeek
87124a396a
Don't copy from_work_dir outputs to purged path in shell fragment
2024-06-07 15:20:44 +02:00
John Davis
bf60526254
Merge pull request #18343 from davelopez/24.1_fix_visualizations_dataset_filtering
...
[24.1] Fix visualizations compatible dataset filtering
2024-06-07 08:34:26 -04:00
David López
d0f01af9e1
Merge pull request #18344 from davelopez/24.1_fix_typing_index_users
...
[24.1] Fix typing issue in reused variable
2024-06-07 13:15:06 +02:00
davelopez
28e0a5b77e
Fix linting unnecessary list literal
2024-06-07 11:51:26 +02:00
davelopez
67fd14046b
Fix typing issue in reused variable
2024-06-07 11:06:06 +02:00
davelopez
3aea494035
Fix possible undefined value
2024-06-07 10:56:42 +02:00
davelopez
8496d00369
Fix total items count in DataDialog
2024-06-07 10:12:26 +02:00
davelopez
205601fb0d
Fix Editor visualization to accept only text-based datatypes
2024-06-07 09:49:51 +02:00
mvdbeek