` element, which best describes the content of the current page.
- - When increasing a heading level, do not skip Levels.
+- When increasing a heading level, do not skip Levels.
- - Do not use a headings level to determine it's size. Use one of the heading utility classes (e.g. `h-lg`) instead.
+- Do not use a headings level to determine it's size. Use one of the heading utility classes (e.g. `h-lg`) instead.
- - Do not use a heading tag to make a non-heading text large. Use a heading utility class on a `` instead.
+- Do not use a heading tag to make a non-heading text large. Use a heading utility class on a `` instead.
## Sizing Headings
There are several utility classes which size headings:
- - `h-xl` - Extra large headings
- - `h-lg` - Large headings (most main headings should have this size)
- - `h-md` - Medium headings
- - `h-sm` - Small headings (many sub-headings have this size)
- - `h-text` - Text sized headings
+- `h-xl` - Extra large headings
+- `h-lg` - Large headings (most main headings should have this size)
+- `h-md` - Medium headings
+- `h-sm` - Small headings (many sub-headings have this size)
+- `h-text` - Text sized headings
Galaxy uses `h-lg` for most top-level (``) headings.
@@ -45,8 +45,8 @@ Set the headings level, by setting aa `h1 ... h6` prop:
Following properties allow for further styling the component:
- - `size=["xl", "lg", "md", "sm", "text"]` - sets the headings size class
- - `bold` - makes a heading bold
- - `inline` - displays the heading inline
- - `separator` - draws a separating line, to better distinguish sections
- - `icon="..."` - adds a font-awesome icon decoration to the left of the heading. Make sure to also load the icon with `library.add(...)`.
+- `size=["xl", "lg", "md", "sm", "text"]` - sets the headings size class
+- `bold` - makes a heading bold
+- `inline` - displays the heading inline
+- `separator` - draws a separating line, to better distinguish sections
+- `icon="..."` - adds a font-awesome icon decoration to the left of the heading. Make sure to also load the icon with `library.add(...)`.
diff --git a/client/docs/never-use-jquery.md b/client/docs/never-use-jquery.md
index c1e07181928..df7a5370da5 100644
--- a/client/docs/never-use-jquery.md
+++ b/client/docs/never-use-jquery.md
@@ -1,6 +1,6 @@
Did you know, jQuery is old enough to drive? It's old enough to get a driver's license. jQuery is a
tool that was built to deal with inconsistencies in browsers that NO LONGER EXIST. In a couple
-years, jQuery will be voting, drinking, and capable of being tried as an adult.
+years, jQuery will be voting, drinking, and capable of being tried as an adult.
If you think you need jQuery, you are mistaken. Please seek help from somebody in the wg-ui-ux
workgroup. There is nothing jQuery can provide you that isn't already part of vanilla javascript or
@@ -14,5 +14,6 @@ One of our most important goals in redesigning Galaxy is the complete eliminatio
from our source, along with all its invasive plugins.
### References
-* [You Don't Need jQuery](https://github.com/nefe/You-Dont-Need-jQuery)
-* [document.querySelector](https://developer.mozilla.org/en-US/docs/Web/API/Document/querySelector)
\ No newline at end of file
+
+- [You Don't Need jQuery](https://github.com/nefe/You-Dont-Need-jQuery)
+- [document.querySelector](https://developer.mozilla.org/en-US/docs/Web/API/Document/querySelector)
diff --git a/client/docs/providers-and-renderers.md b/client/docs/providers-and-renderers.md
index 1eca93b3e13..2637fcbf9f9 100644
--- a/client/docs/providers-and-renderers.md
+++ b/client/docs/providers-and-renderers.md
@@ -22,34 +22,27 @@ component we previously made, but you are free to putput whatever you want in th
doodad and saveDoddad properties as desired, as well as any other local data with the only
restriction that Vue needs a single root element in which to render.
-
## The Renderer
```html static
- A very Long Button Text
+A very Long Button Text
```
Notice the strange positioning of the `>` brackets.
@@ -277,7 +273,7 @@ Do not add space between elements connected by conditionals.
>
> condition met
>
->
+>
>
> condition not met
>
@@ -293,7 +289,7 @@ Add space between non-connected elements.
>
> First span.
>
->
+>
>
> Second span.
>
@@ -325,7 +321,7 @@ Add space between logical blocks of elements.
>
> condition 1 not met
>
->
+>
>
> condition 2 met
>
diff --git a/client/docs/unit-testing/debugging-unit-tests.md b/client/docs/unit-testing/debugging-unit-tests.md
index d7916cf6fa5..74097ef365b 100644
--- a/client/docs/unit-testing/debugging-unit-tests.md
+++ b/client/docs/unit-testing/debugging-unit-tests.md
@@ -9,7 +9,7 @@ code updates its test extensions again.
#### To debug a single Jest test:
-1. Open a jest test (a file that ends with *.test.js)
+1. Open a jest test (a file that ends with \*.test.js)
1. Make sure the test file is selected, especially if you have multiple files open. This process
will fail confusingly and without obvious error if you have not launched the debugger with the
@@ -34,8 +34,6 @@ code updates its test extensions again.
variables near your breakpoint and see their values in the "Variables" section of the Run and
Debug pane.
-
-
```json
// sample launch.json
{
@@ -45,16 +43,16 @@ code updates its test extensions again.
"type": "node",
"name": "debug selected jest test",
"request": "launch",
-
+
// launches version of jest from inside the node_modules
// this means you need to have run yarn first
"program": "${workspaceFolder}/client/node_modules/jest/bin/jest",
"args": [
- // Alias -i.
+ // Alias -i.
// Normally jest opens up a bunch of workers to run all your tests faster
// but we don't want that right now.
"--runInBand",
-
+
// finds jest config
"--config",
"${workspaceFolder}/client/tests/jest/jest.config.js",
@@ -68,9 +66,8 @@ code updates its test extensions again.
"console": "integratedTerminal",
// allows you to place breakpoints right in vscode's gutter
- "disableOptimisticBPs": true,
- },
-
+ "disableOptimisticBPs": true
+ }
]
}
```
diff --git a/client/docs/unit-testing/readme.md b/client/docs/unit-testing/readme.md
index 393c1a0c117..72c3ef2e582 100644
--- a/client/docs/unit-testing/readme.md
+++ b/client/docs/unit-testing/readme.md
@@ -1,20 +1,18 @@
-
[Galaxy uses Jest](https://jestjs.io/) for its client-side unit testing
framework.
For testing Vue components, we use the [Vue testing
utils](https://vue-test-utils.vuejs.org/) to mount individual components in a
-test bed and check them for rendered features. Please use jest-based mocking
+test bed and check them for rendered features. Please use jest-based mocking
for isolating test functionality.
-
### Specific test scenarios & examples
-* [Mocking an imported
-dependency](https://github.com/galaxyproject/galaxy/blob/dev/client/src/components/Tags/tagService.test.js)
-* [Testing async
-operations](https://github.com/galaxyproject/galaxy/blob/dev/client/src/components/Tags/tagService.test.js)
-* [Testing a Vue component for expected rendering
-output](https://github.com/galaxyproject/galaxy/blob/dev/client/src/components/Tags/StatelessTags.test.js)
-* [Firing an event against a shallow mounted vue
-component](https://github.com/galaxyproject/galaxy/blob/dev/client/src/components/Tags/StatelessTags.test.js)
+- [Mocking an imported
+ dependency](https://github.com/galaxyproject/galaxy/blob/dev/client/src/components/Tags/tagService.test.js)
+- [Testing async
+ operations](https://github.com/galaxyproject/galaxy/blob/dev/client/src/components/Tags/tagService.test.js)
+- [Testing a Vue component for expected rendering
+ output](https://github.com/galaxyproject/galaxy/blob/dev/client/src/components/Tags/StatelessTags.test.js)
+- [Firing an event against a shallow mounted vue
+ component](https://github.com/galaxyproject/galaxy/blob/dev/client/src/components/Tags/StatelessTags.test.js)
diff --git a/client/docs/unit-testing/strategies.md b/client/docs/unit-testing/strategies.md
index 64c343cc850..d06353193c3 100644
--- a/client/docs/unit-testing/strategies.md
+++ b/client/docs/unit-testing/strategies.md
@@ -1,4 +1,4 @@
-Part of making good code is making that code easy to test.
+Part of making good code is making that code easy to test.
### Implement logic in pure functions when possible
@@ -10,14 +10,13 @@ There is almost definitely no such thing as a well-written 1000 line function.
Most of whatever happened in that thing was probably deterministic and can be
broken up into easily testable chunks.
-
### Wrap native browser resources in a function so they can be easily mocked
If your javascript needs to talk to the window object, or navigator, etc. wrap
that in a function call so that it can be easily mocked during testing.
-
#### Your Module
+
```js static
// myModule.js
@@ -35,6 +34,7 @@ export function theThingYouReallyCareAbout() {
```
#### Your test file
+
```js static
// myModule.test.js
import { theThingYouReallyCareAbout, redirectTo } from "./myModule";
diff --git a/client/docs/unit-testing/writing-tests.md b/client/docs/unit-testing/writing-tests.md
index 8951b5797de..9a49db6c510 100644
--- a/client/docs/unit-testing/writing-tests.md
+++ b/client/docs/unit-testing/writing-tests.md
@@ -2,14 +2,13 @@
Please remember that these tests are not _for_ you. They're for the people who
come after you. It will be a lot easier to modify, repair and upgrade your code
-if they can figure out what you were originally hoping to accomplish. Try to
+if they can figure out what you were originally hoping to accomplish. Try to
use as detailed 'expect' statements as possible -- overuse of 'toBeTruthy()'
for example, can hide the intent of your test.
-Add a couple of comments. Use variable names that mean something. Nobody's
+Add a couple of comments. Use variable names that mean something. Nobody's
code is as self-documenting as they believe it to be.
-
### Only test the public API that you define
Internal implementations come and go with library upgrades and new tech. But
@@ -20,17 +19,16 @@ Separate your concerns and identify the developer-facing methods and functions
you expect them to use. Test THOSE. Everything else should probably be
considered an implementation detail.
-The other side of the same coin is to test *only* the unit in question. If your
+The other side of the same coin is to test _only_ the unit in question. If your
component has a model that uses a service that touches Vuex, which then uses
-Axios to fetch some data -- don't test all that at once. Break things apart and
-mock functionality to isolate testing to units. End to end testing is a
+Axios to fetch some data -- don't test all that at once. Break things apart and
+mock functionality to isolate testing to units. End to end testing is a
separate thing that shouldn't be attempted using spec tests in Jest.
Assume nobody cares _how_ your code works, we just need to know that the public
API you designed _does_ work. If performance problems or new tech necessitate a
re-write, these tests become a guide for the next implementation.
-
### Writing a test file
Jest will try to test any file ending in "\*.test.js". Please place your test
@@ -65,9 +63,7 @@ describe("some module you wrote", () => {
});
```
-
### Check out the Jest helper functions
We have created some [common helpers for common testing
-scenarios](https://github.com/galaxyproject/galaxy/blob/dev/client/tests/jest/helpers.js).
-
+scenarios](https://github.com/galaxyproject/galaxy/blob/dev/client/tests/jest/helpers.js).
diff --git a/client/package.json b/client/package.json
index dbcf569c61f..6e3f76bfc7e 100644
--- a/client/package.json
+++ b/client/package.json
@@ -1,5 +1,5 @@
{
- "name": "galaxy-client",
+ "name": "@galaxyproject/galaxy-client",
"version": "0.1.0",
"description": "Galaxy client application build system",
"keywords": [
@@ -9,7 +9,10 @@
"type": "git",
"url": "https://github.com/galaxyproject/galaxy.git"
},
- "license": "AFL-3.0",
+ "license": "MIT",
+ "files": [
+ "dist"
+ ],
"browserslist": [
"defaults",
"not ie <= 11",
@@ -110,20 +113,18 @@
"xml-beautifier": "^0.5.0"
},
"scripts": {
- "watch": "gulp && yarn run save-build-hash && yarn run webpack-watch",
- "serve": "NODE_OPTIONS=--max-old-space-size=4096 && NODE_ENV=development gulp && webpack serve",
- "build": "NODE_ENV=development gulp && webpack && yarn run save-build-hash",
- "build-production": "NODE_ENV=production gulp && yarn run webpack-production && yarn run save-build-hash",
- "build-production-maps": "NODE_ENV=production gulp && yarn run webpack-production-maps && yarn run save-build-hash",
+ "develop": "NODE_OPTIONS=--max-old-space-size=4096 NODE_ENV=development gulp && webpack-dev-server",
+ "build": "NODE_ENV=development gulp && webpack && yarn run stage-build",
+ "build-production": "NODE_ENV=production gulp && yarn run webpack-production && yarn run stage-build",
+ "build-production-maps": "NODE_ENV=production gulp && yarn run webpack-production-maps && yarn run stage-build",
"build-stats": "NODE_ENV=production webpack --profile --json=webpack-stats.json",
"view-stats": "webpack-bundle-analyzer webpack-stats.json ../static/dist/",
- "webpack-watch": "webpack --watch",
"webpack-production": "NODE_ENV=production webpack",
"webpack-production-maps": "GXY_BUILD_SOURCEMAPS=1 NODE_ENV=production webpack",
"gulp": "gulp",
- "save-build-hash": "(git rev-parse HEAD 2>/dev/null || echo '') >../static/client_build_hash.txt",
- "format": "prettier --write 'src/style/scss/**/*.scss' 'src/**/{*.ts,*.js,*.vue}' 'tests/jest/standalone/{*.ts,*.js}' '!src/libs/**'",
- "format-check": "prettier --check 'src/style/scss/**/*.scss' 'src/**/{*.js,*.vue}' 'tests/jest/standalone/{*.ts,*.js}' '!src/libs/**'",
+ "stage-build": "cpy 'dist/*' '../static/dist' && (git rev-parse HEAD 2>/dev/null || echo '') >../static/client_build_hash.txt",
+ "format": "prettier --write .",
+ "format-check": "prettier --check .",
"prettier": "yarn run format",
"test": "yarn run qunit && yarn run jest",
"jest": "jest --config tests/jest/jest.config.js",
@@ -151,6 +152,7 @@
"axios-mock-adapter": "^1.20.0",
"babel-jest": "^29.3.1",
"buffer": "^6.0.3",
+ "cpy-cli": "^4.2.0",
"css-loader": "^6.7.1",
"css-minimizer-webpack-plugin": "^4.0.0",
"del": "^6.0.0",
diff --git a/client/prettier.config.js b/client/prettier.config.js
index df792433737..d9cc69f4d8c 100644
--- a/client/prettier.config.js
+++ b/client/prettier.config.js
@@ -1,5 +1,5 @@
module.exports = {
tabWidth: 4,
printWidth: 120,
- bracketSameLine: true
+ bracketSameLine: true,
};
diff --git a/client/src/components/Alert.md b/client/src/components/Alert.md
index 021efd67e56..61038ebc71a 100644
--- a/client/src/components/Alert.md
+++ b/client/src/components/Alert.md
@@ -8,7 +8,7 @@
```js
- A {{variant}} message
+ A {{ variant }} message
```
diff --git a/client/src/components/Dataset/DatasetName.vue b/client/src/components/Dataset/DatasetName.vue
index 1625cd497ac..9751155ea3d 100644
--- a/client/src/components/Dataset/DatasetName.vue
+++ b/client/src/components/Dataset/DatasetName.vue
@@ -22,11 +22,11 @@
diff --git a/client/src/components/DatasetInformation/DatasetAttributes.vue b/client/src/components/DatasetInformation/DatasetAttributes.vue
index dd942966dd0..73874c5f601 100644
--- a/client/src/components/DatasetInformation/DatasetAttributes.vue
+++ b/client/src/components/DatasetInformation/DatasetAttributes.vue
@@ -20,12 +20,16 @@
@click="submit('attribute', 'attributes')">
+
-
+
-
+
{{ source.source_uri }}
-
diff --git a/client/src/components/Form/Elements/FormUpload.test.js b/client/src/components/Form/Elements/FormUpload.test.js
new file mode 100644
index 00000000000..b140b31d0e5
--- /dev/null
+++ b/client/src/components/Form/Elements/FormUpload.test.js
@@ -0,0 +1,30 @@
+import { mount } from "@vue/test-utils";
+import { getLocalVue } from "tests/jest/helpers";
+import FormUpload from "./FormUpload";
+
+const localVue = getLocalVue();
+
+describe("FormUpload", () => {
+ const mountFormUpload = (props) =>
+ mount(FormUpload, {
+ propsData: props,
+ localVue,
+ });
+
+ it("should display selected file in disabled textarea", async () => {
+ const v = "H1, H2, H3\nv1, v2, v3";
+ const wrapper = mountFormUpload({ value: v });
+ const el = wrapper.find("textarea");
+ expect(el.element.value).toEqual(v);
+ expect(el.element.disabled).toBe(true);
+ });
+
+ it("should not display text box if file has not been selected", async () => {
+ const wrapper = mountFormUpload({ value: null });
+ const el = wrapper.find("textarea");
+ expect(el.isVisible()).toBe(false);
+ expect(el.text()).toBe("");
+ const noInput = wrapper.find("label");
+ expect(noInput.text()).toBe("No file chosen");
+ });
+});
diff --git a/client/src/components/Form/Elements/FormUpload.vue b/client/src/components/Form/Elements/FormUpload.vue
new file mode 100644
index 00000000000..bd1d46d820c
--- /dev/null
+++ b/client/src/components/Form/Elements/FormUpload.vue
@@ -0,0 +1,45 @@
+
+
+
+
+
import HistoryList from "./history-list";
export default {
- props: {
- actionId: {
- type: String,
- default: null,
- },
- },
mounted() {
- new HistoryList.View({
- action_id: this.actionId,
- }).$el.appendTo(this.$refs.target);
+ new HistoryList.View().$el.appendTo(this.$refs.target);
},
};
diff --git a/client/src/components/Grid/history-list.js b/client/src/components/Grid/history-list.js
index e8a768413af..42b8c21ff2c 100644
--- a/client/src/components/Grid/history-list.js
+++ b/client/src/components/Grid/history-list.js
@@ -62,15 +62,10 @@ var View = Backbone.View.extend({
initialize: function (options) {
const Galaxy = getGalaxyInstance();
LoadingIndicator.markViewAsLoading(this);
-
- if (options.action_id == "list_published") {
- this.active_tab = "shared";
- } else if (options.action_id == "list") {
- this.active_tab = "user";
- }
+ this.active_tab = "user";
this.model = new Backbone.Model();
Utils.get({
- url: `${getAppRoot()}history/${options.action_id}?${$.param(Galaxy.params)}`,
+ url: `${getAppRoot()}history/list?${$.param(Galaxy.params)}`,
success: (response) => {
this.model.set(response);
this.render();
diff --git a/client/src/components/History/Content/Dataset/DatasetActions.vue b/client/src/components/History/Content/Dataset/DatasetActions.vue
index 46fce47e359..118010babe4 100644
--- a/client/src/components/History/Content/Dataset/DatasetActions.vue
+++ b/client/src/components/History/Content/Dataset/DatasetActions.vue
@@ -44,7 +44,7 @@
{
+ let wrapper;
+
+ beforeEach(() => {
+ wrapper = mount(GenericElement, {
+ propsData: {
+ dsc: {
+ elements: [
+ {
+ element_index: 1,
+ element_identifier: "element-1",
+ element_type: "hda",
+ object: {
+ id: "item-1",
+ },
+ },
+ {
+ element_index: 2,
+ element_identifier: "element-2",
+ element_type: "hdca",
+ object: {
+ id: "item-2",
+ collection_type: "list",
+ element_count: 2,
+ elements_datatypes: ["txt"],
+ elements: [
+ {
+ element_index: 3,
+ element_identifier: "element-3",
+ element_type: "hda",
+ object: {
+ id: "item_3",
+ },
+ },
+ {
+ element_index: 4,
+ element_identifier: "element-4",
+ element_type: "hda",
+ object: {
+ id: "item_4",
+ },
+ },
+ ],
+ },
+ },
+ ],
+ },
+ },
+ localVue,
+ });
+ });
+
+ it("check basics", async () => {
+ const contentItems = wrapper.findAll(".content-item");
+ expect(contentItems.length).toBe(2);
+ expect(contentItems.at(0).attributes("data-hid")).toBe("1");
+ expect(contentItems.at(1).attributes("data-hid")).toBe("2");
+ await contentItems.at(1).find(".cursor-pointer").trigger("click");
+ const contentExpanded = wrapper.findAll(".content-item");
+ expect(contentExpanded.length).toBe(4);
+ expect(contentExpanded.at(2).attributes("data-hid")).toBe("3");
+ expect(contentExpanded.at(3).attributes("data-hid")).toBe("4");
+ });
+});
diff --git a/client/src/components/History/Content/GenericElement.vue b/client/src/components/History/Content/GenericElement.vue
index 7b8f4068501..d12d9f1de86 100644
--- a/client/src/components/History/Content/GenericElement.vue
+++ b/client/src/components/History/Content/GenericElement.vue
@@ -1,36 +1,40 @@
+
+
-
@@ -23,7 +24,8 @@
data-description="switch to another history"
size="sm"
variant="link"
- title="Switch to history">
+ :disabled="currentUser.isAnonymous"
+ :title="userTitle('Switch to history')">
Add Selected
+
+ Add Selected
+
@@ -69,13 +74,8 @@ export default {
};
},
computed: {
- formattedItems() {
- return this.histories.map((item) => {
- if (item.id === this.currentHistoryId) {
- item._rowVariant = "info";
- }
- return item;
- });
+ isEmptySelection() {
+ return this.selectedHistories.length === 0;
},
},
watch: {
diff --git a/client/src/components/History/README.md b/client/src/components/History/README.md
index 3bf90443ce9..9388c5b3f6e 100644
--- a/client/src/components/History/README.md
+++ b/client/src/components/History/README.md
@@ -1,13 +1,12 @@
### History Panel Component Tree
+
This is not intended to be a complete listing, but a general idea of how the components are intended
to interact with each other.
```html static
-
-
@@ -18,7 +17,7 @@ to interact with each other.
-
+
-
+
(
-```
\ No newline at end of file
+```
diff --git a/client/src/components/HistoryExport/ExportLink.vue b/client/src/components/HistoryExport/ExportLink.vue
index fece9bbe0ce..b4bbdb95b82 100644
--- a/client/src/components/HistoryExport/ExportLink.vue
+++ b/client/src/components/HistoryExport/ExportLink.vue
@@ -3,13 +3,9 @@
{{ link }}
- view job details )
diff --git a/client/src/components/Indices/SharingIndicators.vue b/client/src/components/Indices/SharingIndicators.vue
index 38d754eb33c..98031f5f07f 100644
--- a/client/src/components/Indices/SharingIndicators.vue
+++ b/client/src/components/Indices/SharingIndicators.vue
@@ -1,19 +1,17 @@
-
-
@@ -16,7 +18,9 @@
-
diff --git a/client/src/components/LoadingSpan.vue b/client/src/components/LoadingSpan.vue
index 85e79a25c7e..c4c3b9d26b5 100644
--- a/client/src/components/LoadingSpan.vue
+++ b/client/src/components/LoadingSpan.vue
@@ -1,7 +1,7 @@
-
-
+
+
+
+
diff --git a/client/src/components/User/UserPreferences.vue b/client/src/components/User/UserPreferences.vue
index 8a1e02dec11..09afa1e0532 100644
--- a/client/src/components/User/UserPreferences.vue
+++ b/client/src/components/User/UserPreferences.vue
@@ -82,6 +82,11 @@
description="Click here to make all data private."
@click="makeDataPrivate" />
+
+ >
+
+
-
+
{{ error }}
-
+
- ![ui thumbnails]()
-
+ ![ui thumbnails]()
+
- {{ plugin.html }}
- {{ plugin.description }}
+ {{ plugin.html }}
+ {{ plugin.description }}
@@ -155,3 +151,38 @@ export default {
},
};
+
+
diff --git a/client/src/components/Workflow/Editor/ConnectionMenu.vue b/client/src/components/Workflow/Editor/ConnectionMenu.vue
index 31dac4eb940..e3be1d641de 100644
--- a/client/src/components/Workflow/Editor/ConnectionMenu.vue
+++ b/client/src/components/Workflow/Editor/ConnectionMenu.vue
@@ -24,7 +24,12 @@
+
+
+ (label?.value || contentId?.value || name.value)!);
+const stepTitle = computed(() => {
+ if (label?.value) {
+ return label.value;
+ }
+ if (isSubworkflow.value) {
+ return name.value;
+ } else {
+ return contentId?.value || name.value;
+ }
+});
const nodeIcon = computed(() => WorkflowIcons[type.value]);
const formDisplayId = computed(() => stepId.value.toString());
const isSubworkflow = computed(() => type.value === "subworkflow");
diff --git a/client/src/components/Workflow/Editor/Forms/FormTool.test.js b/client/src/components/Workflow/Editor/Forms/FormTool.test.js
index 2db5af07c0e..01bdcfb973c 100644
--- a/client/src/components/Workflow/Editor/Forms/FormTool.test.js
+++ b/client/src/components/Workflow/Editor/Forms/FormTool.test.js
@@ -11,9 +11,7 @@ import { createTestingPinia } from "@pinia/testing";
const localVue = getLocalVue();
describe("FormTool", () => {
- let wrapper;
-
- beforeEach(() => {
+ function mountTarget() {
const store = new Vuex.Store({
modules: {
user: mockModule(userStore),
@@ -21,7 +19,7 @@ describe("FormTool", () => {
},
});
- wrapper = mount(FormTool, {
+ return mount(FormTool, {
propsData: {
id: "input",
datatypes: [],
@@ -32,7 +30,7 @@ describe("FormTool", () => {
name: "tool_name",
version: "1.0",
description: "description",
- inputs: [],
+ inputs: [{ name: "input", label: "input", type: "text", value: "value" }],
help: "help_text",
versions: ["1.0", "2.0", "3.0"],
citations: false,
@@ -46,15 +44,16 @@ describe("FormTool", () => {
stubs: {
CurrentUser: MockCurrentUser({ id: "fakeuser" }),
ConfigProvider: MockConfigProvider({ id: "fakeconfig" }),
- FormElement: { template: "form-element" },
ToolFooter: { template: "tool-footer" },
},
pinia: createTestingPinia(),
provide: { store },
});
- });
+ }
it("changes between different versions", async () => {
+ const wrapper = mountTarget();
+
const dropdowns = wrapper.findAll(".tool-versions .dropdown-item");
let version = dropdowns.at(1);
expect(version.text()).toBe("Switch to 2.0");
diff --git a/client/src/components/Workflow/Editor/Forms/FormTool.vue b/client/src/components/Workflow/Editor/Forms/FormTool.vue
index 09e76c5de96..d56446087be 100644
--- a/client/src/components/Workflow/Editor/Forms/FormTool.vue
+++ b/client/src/components/Workflow/Editor/Forms/FormTool.vue
@@ -27,6 +27,7 @@
:area="true"
help="Add an annotation or notes to this step. Annotations are available when a workflow is viewed."
@input="onAnnotation" />
+ Tool Parameters
diff --git a/client/src/components/Workflow/Editor/TerminalConnector.vue b/client/src/components/Workflow/Editor/TerminalConnector.vue
index 20dfc4a2540..5830982e5d0 100644
--- a/client/src/components/Workflow/Editor/TerminalConnector.vue
+++ b/client/src/components/Workflow/Editor/TerminalConnector.vue
@@ -4,14 +4,16 @@
:id="connectionId"
:position="position"
:output-is-mapped-over="outputIsMappedOver"
- :input-is-mapped-over="inputIsMappedOver">
+ :input-is-mapped-over="inputIsMappedOver"
+ :connection-is-valid="connectionIsValid"
+ :nullable="outputIsOptional">
@@ -35,7 +39,8 @@ const dragStyle = computed(() => {
v-if="draggingTerminal && draggingConnection"
:position="draggingConnection"
:input-is-mapped-over="false"
- :output-is-mapped-over="draggingTerminal.mapOver.isCollection">
+ :output-is-mapped-over="draggingTerminal.mapOver.isCollection"
+ :nullable="dragIsOptional">
= ref(null);
const elementBounding = useElementBounding(canvas, { windowResize: false, windowScroll: false });
const scroll = useScroll(canvas);
-const { transform, panBy, setZoom, moveTo } = useZoom(1, 0.2, 5, canvas, scroll);
+const { transform, panBy, setZoom, moveTo } = useD3Zoom(1, minZoom, maxZoom, canvas, scroll);
const isDragging = ref(false);
provide("isDragging", isDragging);
diff --git a/client/src/components/Workflow/Editor/ZoomControl.vue b/client/src/components/Workflow/Editor/ZoomControl.vue
index 61df33f52f6..081ec21fa7a 100644
--- a/client/src/components/Workflow/Editor/ZoomControl.vue
+++ b/client/src/components/Workflow/Editor/ZoomControl.vue
@@ -1,46 +1,29 @@
+
+
+
+
+ {{ infoString }}
+
+
+ Problem occurred at this step:
+
+
+
getRootFromIndexLink() + path;
export default {
components: {
+ InvocationMessage,
ProgressBar,
LoadingSpan,
},
diff --git a/client/src/components/WorkflowInvocationState/invocationMessageModel.ts b/client/src/components/WorkflowInvocationState/invocationMessageModel.ts
new file mode 100644
index 00000000000..7c86d43ad17
--- /dev/null
+++ b/client/src/components/WorkflowInvocationState/invocationMessageModel.ts
@@ -0,0 +1,242 @@
+/* tslint:disable */
+/* eslint-disable */
+/**
+/* This file was automatically generated from pydantic models by running pydantic2ts.
+/* Do not modify it by hand - just update the pydantic models and then re-run the script
+*/
+
+export type InvocationMessageResponseModel =
+ | GenericInvocationCancellationReviewFailedEncodedDatabaseIdField
+ | GenericInvocationCancellationHistoryDeletedEncodedDatabaseIdField
+ | GenericInvocationCancellationUserRequestEncodedDatabaseIdField
+ | GenericInvocationFailureDatasetFailedEncodedDatabaseIdField
+ | GenericInvocationFailureCollectionFailedEncodedDatabaseIdField
+ | GenericInvocationFailureJobFailedEncodedDatabaseIdField
+ | GenericInvocationFailureOutputNotFoundEncodedDatabaseIdField
+ | GenericInvocationFailureExpressionEvaluationFailedEncodedDatabaseIdField
+ | GenericInvocationFailureWhenNotBooleanEncodedDatabaseIdField
+ | GenericInvocationUnexpectedFailureEncodedDatabaseIdField
+ | GenericInvocationEvaluationWarningWorkflowOutputNotFoundEncodedDatabaseIdField;
+
+export interface GenericInvocationCancellationHistoryDeletedEncodedDatabaseIdField {
+ reason: "history_deleted";
+ /**
+ * History ID of history that was deleted.
+ */
+ history_id: string;
+}
+export interface GenericInvocationCancellationHistoryDeletedInt {
+ reason: "history_deleted";
+ /**
+ * History ID of history that was deleted.
+ */
+ history_id: number;
+}
+export interface GenericInvocationCancellationReviewFailedEncodedDatabaseIdField {
+ reason: "cancelled_on_review";
+ /**
+ * Workflow step id of paused step that did not pass review.
+ */
+ workflow_step_id: number;
+}
+export interface GenericInvocationCancellationReviewFailedInt {
+ reason: "cancelled_on_review";
+ /**
+ * Workflow step id of paused step that did not pass review.
+ */
+ workflow_step_id: number;
+}
+export interface GenericInvocationCancellationUserRequestEncodedDatabaseIdField {
+ reason: "user_request";
+}
+export interface GenericInvocationCancellationUserRequestInt {
+ reason: "user_request";
+}
+export interface GenericInvocationEvaluationWarningWorkflowOutputNotFoundEncodedDatabaseIdField {
+ reason: "workflow_output_not_found";
+ workflow_step_id: number;
+ /**
+ * Output that was designated as workflow output but that has not been found
+ */
+ output_name: string;
+}
+export interface GenericInvocationEvaluationWarningWorkflowOutputNotFoundInt {
+ reason: "workflow_output_not_found";
+ workflow_step_id: number;
+ /**
+ * Output that was designated as workflow output but that has not been found
+ */
+ output_name: string;
+}
+export interface GenericInvocationFailureCollectionFailedEncodedDatabaseIdField {
+ reason: "collection_failed";
+ /**
+ * Workflow step id of step that failed.
+ */
+ workflow_step_id: number;
+ /**
+ * HistoryDatasetCollectionAssociation ID that relates to failure.
+ */
+ hdca_id?: string;
+ /**
+ * Workflow step id of step that caused failure.
+ */
+ dependent_workflow_step_id: number;
+}
+export interface GenericInvocationFailureCollectionFailedInt {
+ reason: "collection_failed";
+ /**
+ * Workflow step id of step that failed.
+ */
+ workflow_step_id: number;
+ /**
+ * HistoryDatasetCollectionAssociation ID that relates to failure.
+ */
+ hdca_id?: number;
+ /**
+ * Workflow step id of step that caused failure.
+ */
+ dependent_workflow_step_id: number;
+}
+export interface GenericInvocationFailureDatasetFailedEncodedDatabaseIdField {
+ reason: "dataset_failed";
+ /**
+ * Workflow step id of step that failed.
+ */
+ workflow_step_id: number;
+ /**
+ * HistoryDatasetAssociation ID that relates to failure.
+ */
+ hda_id: string;
+ /**
+ * Workflow step id of step that caused failure.
+ */
+ dependent_workflow_step_id?: number;
+}
+export interface GenericInvocationFailureDatasetFailedInt {
+ reason: "dataset_failed";
+ /**
+ * Workflow step id of step that failed.
+ */
+ workflow_step_id: number;
+ /**
+ * HistoryDatasetAssociation ID that relates to failure.
+ */
+ hda_id: number;
+ /**
+ * Workflow step id of step that caused failure.
+ */
+ dependent_workflow_step_id?: number;
+}
+export interface GenericInvocationFailureExpressionEvaluationFailedEncodedDatabaseIdField {
+ reason: "expression_evaluation_failed";
+ /**
+ * Workflow step id of step that failed.
+ */
+ workflow_step_id: number;
+ /**
+ * May contain details to help troubleshoot this problem.
+ */
+ details?: string;
+}
+export interface GenericInvocationFailureExpressionEvaluationFailedInt {
+ reason: "expression_evaluation_failed";
+ /**
+ * Workflow step id of step that failed.
+ */
+ workflow_step_id: number;
+ /**
+ * May contain details to help troubleshoot this problem.
+ */
+ details?: string;
+}
+export interface GenericInvocationFailureJobFailedEncodedDatabaseIdField {
+ reason: "job_failed";
+ /**
+ * Workflow step id of step that failed.
+ */
+ workflow_step_id: number;
+ /**
+ * Job ID that relates to failure.
+ */
+ job_id?: string;
+ /**
+ * Workflow step id of step that caused failure.
+ */
+ dependent_workflow_step_id: number;
+}
+export interface GenericInvocationFailureJobFailedInt {
+ reason: "job_failed";
+ /**
+ * Workflow step id of step that failed.
+ */
+ workflow_step_id: number;
+ /**
+ * Job ID that relates to failure.
+ */
+ job_id?: number;
+ /**
+ * Workflow step id of step that caused failure.
+ */
+ dependent_workflow_step_id: number;
+}
+export interface GenericInvocationFailureOutputNotFoundEncodedDatabaseIdField {
+ reason: "output_not_found";
+ /**
+ * Workflow step id of step that failed.
+ */
+ workflow_step_id: number;
+ output_name: string;
+ /**
+ * Workflow step id of step that caused failure.
+ */
+ dependent_workflow_step_id: number;
+}
+export interface GenericInvocationFailureOutputNotFoundInt {
+ reason: "output_not_found";
+ /**
+ * Workflow step id of step that failed.
+ */
+ workflow_step_id: number;
+ output_name: string;
+ /**
+ * Workflow step id of step that caused failure.
+ */
+ dependent_workflow_step_id: number;
+}
+export interface GenericInvocationFailureWhenNotBooleanEncodedDatabaseIdField {
+ reason: "when_not_boolean";
+ /**
+ * Workflow step id of step that failed.
+ */
+ workflow_step_id: number;
+ /**
+ * Contains details to help troubleshoot this problem.
+ */
+ details: string;
+}
+export interface GenericInvocationFailureWhenNotBooleanInt {
+ reason: "when_not_boolean";
+ /**
+ * Workflow step id of step that failed.
+ */
+ workflow_step_id: number;
+ /**
+ * Contains details to help troubleshoot this problem.
+ */
+ details: string;
+}
+export interface GenericInvocationUnexpectedFailureEncodedDatabaseIdField {
+ reason: "unexpected_failure";
+ /**
+ * May contains details to help troubleshoot this problem.
+ */
+ details?: string;
+}
+export interface GenericInvocationUnexpectedFailureInt {
+ reason: "unexpected_failure";
+ /**
+ * May contains details to help troubleshoot this problem.
+ */
+ details?: string;
+}
diff --git a/client/src/composables/datatypes.ts b/client/src/composables/datatypes.ts
index 88a38f045f4..cb29633e4ec 100644
--- a/client/src/composables/datatypes.ts
+++ b/client/src/composables/datatypes.ts
@@ -29,7 +29,7 @@ export function useDetailedDatatypes() {
async function getDatatypes() {
try {
- const datatypesPromise = datatypesFetcher({ extension_only: true });
+ const datatypesPromise = datatypesFetcher({ extension_only: false });
const datatypeEDAMFormatsPromise = edamFormatsFetcher({});
const datatypeEDAMDataPromise = edamDataFetcher({});
diff --git a/client/src/composables/fileDrop.js b/client/src/composables/fileDrop.ts
similarity index 52%
rename from client/src/composables/fileDrop.js
rename to client/src/composables/fileDrop.ts
index 76e8ed1609f..0d71b92425b 100644
--- a/client/src/composables/fileDrop.js
+++ b/client/src/composables/fileDrop.ts
@@ -1,5 +1,8 @@
-import { ref, unref } from "vue";
-import { useEventListener } from "@vueuse/core";
+import { ref, unref, type Ref } from "vue";
+import { useEventListener, type MaybeComputedRef } from "@vueuse/core";
+import { wait } from "@/utils/wait";
+
+export type FileDropHandler = (event: DragEvent) => void;
/**
* Custom File-Drop composable
@@ -7,18 +10,26 @@ import { useEventListener } from "@vueuse/core";
* @param onDrop callback function called when drop occurs
* @param solo when true, only reacts if no modal is open
*/
-export function useFileDrop(dropZone, onDrop, solo) {
+export function useFileDrop(
+ dropZone: MaybeComputedRef,
+ onDrop: Ref | FileDropHandler,
+ solo: MaybeComputedRef
+) {
const isFileOverDocument = ref(false);
const isFileOverDropZone = ref(false);
- const dragBlocked = ref(false);
+ // blocks drag events in this composable, to avoid drag events in unwanted situations
+ let dragBlocked = false;
+
+ // keeps track if the drag has exited, to avoid premature drag canceling
+ let hasExited = true;
// Don't react to page-internal drag events
useEventListener(
document.body,
"dragstart",
() => {
- dragBlocked.value = true;
+ dragBlocked = true;
},
true
);
@@ -27,9 +38,10 @@ export function useFileDrop(dropZone, onDrop, solo) {
document.body,
"dragover",
(event) => {
- if (!dragBlocked.value) {
+ if (!dragBlocked) {
// prevent the browser from opening the file
event.preventDefault();
+ hasExited = false;
}
},
true
@@ -39,40 +51,54 @@ export function useFileDrop(dropZone, onDrop, solo) {
document.body,
"drop",
(event) => {
- if (!dragBlocked.value) {
+ if (!dragBlocked) {
// prevent the browser from opening the file
event.preventDefault();
if (isFileOverDropZone.value && isFileOverDocument.value) {
- unref(onDrop)(event);
+ const dropHandler = unref(onDrop);
+ dropHandler(event as DragEvent);
}
}
isFileOverDocument.value = false;
- dragBlocked.value = false;
+ dragBlocked = false;
+ hasExited = true;
},
true
);
- useEventListener(
- document.body,
- "dragend",
- () => {
- // reset on drag end
- isFileOverDocument.value = false;
- isFileOverDropZone.value = false;
- dragBlocked.value = false;
- },
- true
- );
+ /** Reset all variables */
+ const reset = () => {
+ isFileOverDocument.value = false;
+ isFileOverDropZone.value = false;
+ dragBlocked = false;
+ hasExited = true;
+ };
+
+ useEventListener(document.body, "dragend", reset, true);
+
+ useEventListener(document.body, "dragleave", async () => {
+ hasExited = true;
+
+ // This event may have been triggered by components
+ // which have not been properly childed to the body yet.
+ // Wait a bit, and check if hasExited is still true.
+ await wait(100);
+
+ if (hasExited) {
+ reset();
+ }
+ });
useEventListener(
document.body,
"dragenter",
(event) => {
// init values if drag is possible
- if (!dragBlocked.value && !(unref(solo) && isAnyModalOpen())) {
+ if (!dragBlocked && !(unref(solo) && isAnyModalOpen())) {
isFileOverDocument.value = true;
isFileOverDropZone.value = false;
+ hasExited = false;
event.preventDefault();
}
@@ -90,6 +116,7 @@ export function useFileDrop(dropZone, onDrop, solo) {
"dragenter",
() => {
isFileOverDropZone.value = true;
+ hasExited = false;
},
true
);
@@ -99,6 +126,7 @@ export function useFileDrop(dropZone, onDrop, solo) {
"dragleave",
() => {
isFileOverDropZone.value = false;
+ hasExited = false;
},
true
);
diff --git a/client/src/composables/useWorkflowInstance.ts b/client/src/composables/useWorkflowInstance.ts
new file mode 100644
index 00000000000..4e1a376397b
--- /dev/null
+++ b/client/src/composables/useWorkflowInstance.ts
@@ -0,0 +1,23 @@
+import { useWorkflowStore } from "@/stores/workflowStore";
+import { ref } from "vue";
+
+export function useWorkflowInstance(workflowId: string) {
+ const workflowStore = useWorkflowStore();
+ const workflow = ref(workflowStore.getWorkflowByInstanceId(workflowId));
+ const loading = ref(false);
+
+ async function getWorkflowInstance() {
+ if (!workflow.value) {
+ loading.value = true;
+ try {
+ await workflowStore.fetchWorkflowForInstanceId(workflowId);
+ } catch (e) {
+ loading.value = false;
+ console.error("unable to fetch workflow \n", e);
+ }
+ }
+ }
+ getWorkflowInstance();
+
+ return { workflow, loading };
+}
diff --git a/client/src/entry/analysis/router.js b/client/src/entry/analysis/router.js
index 56e2bdac7e9..558e962322f 100644
--- a/client/src/entry/analysis/router.js
+++ b/client/src/entry/analysis/router.js
@@ -235,6 +235,12 @@ export function getRouter(Galaxy) {
component: HistoryPublishedList,
props: true,
},
+ {
+ path: "histories/list",
+ component: GridHistory,
+ props: true,
+ redirect: redirectAnon(),
+ },
{
path: "histories/:historyId/export",
get component() {
@@ -242,11 +248,6 @@ export function getRouter(Galaxy) {
},
props: true,
},
- {
- path: "histories/:actionId",
- component: GridHistory,
- props: true,
- },
{
path: "interactivetool_entry_points/list",
component: InteractiveTools,
diff --git a/client/src/mvc/dataset/data.js b/client/src/mvc/dataset/data.js
index 2705f3a6960..78cadb2c434 100644
--- a/client/src/mvc/dataset/data.js
+++ b/client/src/mvc/dataset/data.js
@@ -3,7 +3,7 @@ import Backbone from "backbone";
import { getAppRoot } from "onload/loadConfig";
//temporary
-import { appendVueComponent } from "utils/mountVueComponent";
+import { replaceChildrenWithComponent } from "utils/mountVueComponent";
import TabularChunkedView from "components/Visualizations/Tabular/TabularChunkedView.vue";
/**
@@ -72,5 +72,5 @@ export var DatasetCollection = Backbone.Collection.extend({
export const createTabularDatasetChunkedView = (options) => {
// We'll always have a parent_elt in options, so create a div and insert it into that.
- return appendVueComponent(options.parent_elt, TabularChunkedView, { options });
+ return replaceChildrenWithComponent(options.parent_elt, TabularChunkedView, { options });
};
diff --git a/client/src/mvc/visualization/chart/chart-client.js b/client/src/mvc/visualization/chart/chart-client.js
index 3b94a33dda4..4a030f5c10b 100644
--- a/client/src/mvc/visualization/chart/chart-client.js
+++ b/client/src/mvc/visualization/chart/chart-client.js
@@ -31,12 +31,7 @@ export default Backbone.View.extend({
this.$buttons = this.$(".charts-buttons");
this.chart = new Chart({}, options);
this.chart.plugin = options.visualization_plugin;
- this.chart.plugin.specs = this.chart.plugin.specs || {};
- if (Object.keys(this.chart.plugin.specs).length === 0) {
- this.chart.requiresConfirmation = false;
- } else {
- this.chart.requiresConfirmation = asBoolean(this.chart.plugin.specs.confirm);
- }
+ this.chart.requiresConfirmation = asBoolean(this.chart.plugin.specs?.confirm);
this.chart_load = options.chart_load;
this.message = new Ui.Message();
this.deferred = new Deferred();
diff --git a/client/src/mvc/visualization/chart/views/groups.js b/client/src/mvc/visualization/chart/views/groups.js
index 8fdaafdaa9c..150e13ea2b5 100644
--- a/client/src/mvc/visualization/chart/views/groups.js
+++ b/client/src/mvc/visualization/chart/views/groups.js
@@ -8,7 +8,7 @@ import Utils from "utils/utils";
import Repeat from "./repeat";
import { visitInputs } from "components/Form/utilities";
import FormDisplay from "components/Form/FormDisplay";
-import { appendVueComponent } from "utils/mountVueComponent";
+import { replaceChildrenWithComponent } from "utils/mountVueComponent";
var GroupView = Backbone.View.extend({
initialize: function (app, options) {
@@ -75,7 +75,7 @@ var GroupView = Backbone.View.extend({
params[name] = input.value;
});
self.redraw(params);
- const instance = appendVueComponent(self.$el, FormDisplay, {
+ const instance = replaceChildrenWithComponent(self.el, FormDisplay, {
inputs: inputs,
});
instance.$on("onChange", (data) => {
diff --git a/client/src/mvc/visualization/chart/views/menu.js b/client/src/mvc/visualization/chart/views/menu.js
index e1f259704cb..283c329767f 100644
--- a/client/src/mvc/visualization/chart/views/menu.js
+++ b/client/src/mvc/visualization/chart/views/menu.js
@@ -134,14 +134,14 @@ export default Backbone.View.extend({
},
render: function () {
- var visible = this.model.get("visible");
+ const visible = this.model.get("visible");
this.app.$el[visible ? "removeClass" : "addClass"]("charts-fullscreen");
- this.execute_button.model.set("visible", visible && !!this.app.chart.plugin.specs.confirm);
+ this.execute_button.model.set("visible", visible && !!this.app.chart.plugin.specs?.confirm);
this.save_button.model.set("visible", visible);
this.export_button.model.set("visible", visible);
this.right_button.model.set("visible", visible);
this.left_button.model.set("visible", !visible);
- var exports = this.app.chart.plugin.specs.exports || [];
+ const exports = this.app.chart.plugin.specs?.exports ?? [];
this.export_button.collection.each((model) => {
model.set("visible", exports.indexOf(model.get("key")) !== -1);
});
diff --git a/client/src/mvc/visualization/chart/views/settings.js b/client/src/mvc/visualization/chart/views/settings.js
index db0475bebdc..80f6e03e042 100644
--- a/client/src/mvc/visualization/chart/views/settings.js
+++ b/client/src/mvc/visualization/chart/views/settings.js
@@ -3,7 +3,7 @@ import Backbone from "backbone";
import Utils from "utils/utils";
import { visitInputs } from "components/Form/utilities";
import FormDisplay from "components/Form/FormDisplay";
-import { appendVueComponent } from "utils/mountVueComponent";
+import { replaceChildrenWithComponent } from "utils/mountVueComponent";
export default Backbone.View.extend({
initialize: function (app) {
@@ -27,7 +27,7 @@ export default Backbone.View.extend({
self.chart.settings.set(name, input.value);
}
});
- const instance = appendVueComponent(this.$el, FormDisplay, {
+ const instance = replaceChildrenWithComponent(this.el, FormDisplay, {
inputs: inputs,
});
instance.$on("onChange", (data) => {
diff --git a/client/src/schema/schema.ts b/client/src/schema/schema.ts
index 29886efe1fb..4b0b6e8dca8 100644
--- a/client/src/schema/schema.ts
+++ b/client/src/schema/schema.ts
@@ -1169,6 +1169,18 @@ export interface paths {
/** Return the user's API key with extra information. */
get: operations["get_api_key_detailed_api_users__user_id__api_key_detailed_get"];
};
+ "/api/users/{user_id}/beacon": {
+ /**
+ * Returns information about beacon share settings
+ * @description **Warning**: This endpoint is experimental and might change or disappear in future versions.
+ */
+ get: operations["get_beacon_api_users__user_id__beacon_get"];
+ /**
+ * Changes beacon setting
+ * @description **Warning**: This endpoint is experimental and might change or disappear in future versions.
+ */
+ post: operations["set_beacon_api_users__user_id__beacon_post"];
+ };
"/api/version": {
/**
* Return Galaxy version information: major/minor version, optional extra info
@@ -4087,6 +4099,12 @@ export interface components {
* @default 0
*/
running?: number;
+ /**
+ * Skipped jobs
+ * @description Number of jobs that were skipped due to conditional workflow step execution.
+ * @default 0
+ */
+ skipped?: number;
/**
* Upload jobs
* @description Number of jobs in the `upload` state.
@@ -7188,6 +7206,17 @@ export interface components {
*/
url: string;
};
+ /**
+ * UserBeaconSetting
+ * @description Base model definition with common configuration used by all derived models.
+ */
+ UserBeaconSetting: {
+ /**
+ * Enabled
+ * @description True if beacon sharing is enabled
+ */
+ enabled: boolean;
+ };
/**
* UserEmail
* @description Base model definition with common configuration used by all derived models.
@@ -7493,7 +7522,8 @@ export interface components {
| "deleted"
| "deleted_new"
| "stop"
- | "stopped";
+ | "stopped"
+ | "skipped";
/**
* populated_states
* @description An enumeration.
@@ -13916,6 +13946,71 @@ export interface operations {
};
};
};
+ get_beacon_api_users__user_id__beacon_get: {
+ /**
+ * Returns information about beacon share settings
+ * @description **Warning**: This endpoint is experimental and might change or disappear in future versions.
+ */
+ parameters: {
+ /** @description The user ID that will be used to effectively make this API call. Only admins and designated users can make API calls on behalf of other users. */
+ header?: {
+ "run-as"?: string;
+ };
+ /** @description The ID of the user to get. */
+ path: {
+ user_id: string;
+ };
+ };
+ responses: {
+ /** @description Successful Response */
+ 200: {
+ content: {
+ "application/json": components["schemas"]["UserBeaconSetting"];
+ };
+ };
+ /** @description Validation Error */
+ 422: {
+ content: {
+ "application/json": components["schemas"]["HTTPValidationError"];
+ };
+ };
+ };
+ };
+ set_beacon_api_users__user_id__beacon_post: {
+ /**
+ * Changes beacon setting
+ * @description **Warning**: This endpoint is experimental and might change or disappear in future versions.
+ */
+ parameters: {
+ /** @description The user ID that will be used to effectively make this API call. Only admins and designated users can make API calls on behalf of other users. */
+ header?: {
+ "run-as"?: string;
+ };
+ /** @description The ID of the user to get. */
+ path: {
+ user_id: string;
+ };
+ };
+ requestBody: {
+ content: {
+ "application/json": components["schemas"]["UserBeaconSetting"];
+ };
+ };
+ responses: {
+ /** @description Successful Response */
+ 200: {
+ content: {
+ "application/json": components["schemas"]["UserBeaconSetting"];
+ };
+ };
+ /** @description Validation Error */
+ 422: {
+ content: {
+ "application/json": components["schemas"]["HTTPValidationError"];
+ };
+ };
+ };
+ };
version_api_version_get: {
/**
* Return Galaxy version information: major/minor version, optional extra info
diff --git a/client/src/stores/workflowConnectionStore.ts b/client/src/stores/workflowConnectionStore.ts
index d2730db36f1..04a46a09f2c 100644
--- a/client/src/stores/workflowConnectionStore.ts
+++ b/client/src/stores/workflowConnectionStore.ts
@@ -1,8 +1,15 @@
import { defineStore } from "pinia";
import { useWorkflowStepStore } from "@/stores/workflowStepStore";
+import { state } from "@/store/tagStore";
+import Vue from "vue";
+
+interface InvalidConnections {
+ [index: string]: string | undefined;
+}
export interface State {
connections: Connection[];
+ invalidConnections: InvalidConnections;
}
export class Connection {
@@ -27,6 +34,7 @@ export interface BaseTerminal {
export interface InputTerminal extends BaseTerminal {
connectorType: "input";
+ input_subworkflow_step_id?: number;
}
export interface OutputTerminal extends BaseTerminal {
@@ -58,6 +66,7 @@ function pushOrSet(object: { [key: string | number]: Array }, key: string
export const useConnectionStore = defineStore("workflowConnectionStore", {
state: (): State => ({
connections: [] as Connection[],
+ invalidConnections: {} as InvalidConnections,
}),
getters: {
getOutputTerminalsForInputTerminal(state: State) {
@@ -111,12 +120,19 @@ export const useConnectionStore = defineStore("workflowConnectionStore", {
const stepStore = useWorkflowStepStore();
stepStore.addConnection(connection);
},
+ markInvalidConnection(this: State, connectionId: string, reason: string) {
+ Vue.set(this.invalidConnections, connectionId, reason);
+ },
+ dropFromInvalidConnections(this: State, connectionId: string) {
+ Vue.delete(this.invalidConnections, connectionId);
+ },
removeConnection(this: State, terminal: InputTerminal | OutputTerminal | Connection["id"]) {
const stepStore = useWorkflowStepStore();
this.connections = this.connections.filter((connection) => {
if (typeof terminal === "string") {
if (connection.id == terminal) {
stepStore.removeConnection(connection);
+ Vue.delete(this.invalidConnections, connection.id);
return false;
} else {
return true;
@@ -124,6 +140,7 @@ export const useConnectionStore = defineStore("workflowConnectionStore", {
} else if (terminal.connectorType === "input") {
if (connection.input.stepId == terminal.stepId && connection.input.name == terminal.name) {
stepStore.removeConnection(connection);
+ Vue.delete(this.invalidConnections, connection.id);
return false;
} else {
return true;
@@ -131,6 +148,7 @@ export const useConnectionStore = defineStore("workflowConnectionStore", {
} else {
if (connection.output.stepId == terminal.stepId && connection.output.name == terminal.name) {
stepStore.removeConnection(connection);
+ Vue.delete(this.invalidConnections, connection.id);
return false;
} else {
return true;
diff --git a/client/src/stores/workflowStepStore.ts b/client/src/stores/workflowStepStore.ts
index 262fe91271c..9bf5e39cb94 100644
--- a/client/src/stores/workflowStepStore.ts
+++ b/client/src/stores/workflowStepStore.ts
@@ -9,6 +9,7 @@ interface State {
steps: { [index: string]: Step };
stepIndex: number;
stepMapOver: { [index: number]: CollectionTypeDescriptor };
+ stepInputMapOver: StepInputMapOver;
}
interface StepPosition {
@@ -63,6 +64,7 @@ interface BaseStepInput {
extensions: string[];
optional: boolean;
input_type: string;
+ input_subworkflow_step_id?: number;
}
export interface DataStepInput extends BaseStepInput {
@@ -79,14 +81,14 @@ export interface ParameterStepInput extends Omit {
type: typeof ParameterTypes;
}
-type InputTerminalSource = DataStepInput | DataCollectionStepInput | ParameterStepInput;
-type OutputTerminalSource = DataOutput | CollectionOutput | ParameterOutput;
+export type InputTerminalSource = DataStepInput | DataCollectionStepInput | ParameterStepInput;
+export type OutputTerminalSource = DataOutput | CollectionOutput | ParameterOutput;
export type TerminalSource = InputTerminalSource | OutputTerminalSource;
interface WorkflowOutput {
output_name: string;
- label?: string;
- uuid?: string;
+ label?: string | null;
+ uuid?: string | null;
}
export interface NewStep {
@@ -97,7 +99,7 @@ export interface NewStep {
errors?: string[] | null;
input_connections: StepInputConnection;
inputs: Array;
- label?: string;
+ label?: string | null;
name: string;
outputs: Array;
position?: StepPosition;
@@ -106,6 +108,7 @@ export interface NewStep {
tooltip?: string;
type: "tool" | "data_input" | "data_collection_input" | "subworkflow" | "parameter_input" | "pause";
uuid?: string;
+ when?: string | null;
workflow_outputs?: WorkflowOutput[];
}
@@ -118,12 +121,13 @@ export interface Steps {
}
export interface StepInputConnection {
- [index: string]: ConnectionOutputLink | ConnectionOutputLink[];
+ [index: string]: ConnectionOutputLink | ConnectionOutputLink[] | undefined;
}
export interface ConnectionOutputLink {
output_name: string;
id: number;
+ input_subworkflow_step_id?: number;
}
interface WorkflowOutputs {
@@ -133,10 +137,15 @@ interface WorkflowOutputs {
};
}
+interface StepInputMapOver {
+ [index: number]: { [index: string]: CollectionTypeDescriptor };
+}
+
export const useWorkflowStepStore = defineStore("workflowStepStore", {
state: (): State => ({
steps: {} as Steps,
stepMapOver: {} as { [index: number]: CollectionTypeDescriptor },
+ stepInputMapOver: {} as StepInputMapOver,
stepIndex: -1,
}),
getters: {
@@ -145,6 +154,32 @@ export const useWorkflowStepStore = defineStore("workflowStepStore", {
return state.steps[stepId.toString()];
};
},
+ getStepExtraInputs(state: State) {
+ const extraInputs: { [index: number]: InputTerminalSource[] } = {};
+ Object.values(state.steps).forEach((step) => {
+ if (step?.when !== undefined) {
+ Object.keys(step.input_connections).forEach((inputName) => {
+ if (!step.inputs.find((input) => input.name === inputName) && step.when?.includes(inputName)) {
+ const terminalSource = {
+ name: inputName,
+ optional: false,
+ input_type: "parameter" as const,
+ type: "boolean" as const,
+ multiple: false,
+ label: inputName,
+ extensions: [],
+ };
+ if (extraInputs[step.id]) {
+ extraInputs[step.id].push(terminalSource);
+ } else {
+ extraInputs[step.id] = [terminalSource];
+ }
+ }
+ });
+ }
+ });
+ return (stepId: number) => extraInputs[stepId] || [];
+ },
getStepIndex(state: State) {
return Math.max(...Object.values(state.steps).map((step) => step.id), state.stepIndex);
},
@@ -185,7 +220,17 @@ export const useWorkflowStepStore = defineStore("workflowStepStore", {
this.steps[step.id.toString()] = step;
},
changeStepMapOver(stepId: number, mapOver: CollectionTypeDescriptor) {
- this.stepMapOver[stepId] = mapOver;
+ Vue.set(this.stepMapOver, stepId, mapOver);
+ },
+ resetStepInputMapOver(stepId: number) {
+ Vue.set(this.stepInputMapOver, stepId, {});
+ },
+ changeStepInputMapOver(stepId: number, inputName: string, mapOver: CollectionTypeDescriptor) {
+ if (this.stepInputMapOver[stepId]) {
+ Vue.set(this.stepInputMapOver[stepId], inputName, mapOver);
+ } else {
+ Vue.set(this.stepInputMapOver, stepId, { [inputName]: mapOver });
+ }
},
addConnection(connection: Connection) {
const inputStep = this.getStep(connection.input.stepId);
@@ -193,11 +238,19 @@ export const useWorkflowStepStore = defineStore("workflowStepStore", {
inputStep,
`Failed to add connection, because step with id ${connection.input.stepId} is undefined`
);
+ const input = inputStep.inputs.find((input) => input.name === connection.input.name);
+ const connectionLink: ConnectionOutputLink = {
+ output_name: connection.output.name,
+ id: connection.output.stepId,
+ };
+ if (input && "input_subworkflow_step_id" in input && input.input_subworkflow_step_id !== undefined) {
+ connectionLink["input_subworkflow_step_id"] = input.input_subworkflow_step_id;
+ }
const updatedStep = {
...inputStep,
input_connections: {
...inputStep.input_connections,
- [connection.input.name]: { output_name: connection.output.name, id: connection.output.stepId },
+ [connection.input.name]: connectionLink,
},
};
this.updateStep(updatedStep);
@@ -224,7 +277,10 @@ export const useWorkflowStepStore = defineStore("workflowStepStore", {
export function stepToConnections(step: Step): Connection[] {
const connections: Connection[] = [];
if (step.input_connections) {
- Object.entries(step?.input_connections).forEach(([input_name, outputArray]) => {
+ Object.entries(step?.input_connections).forEach(([inputName, outputArray]) => {
+ if (outputArray === undefined) {
+ return;
+ }
if (!Array.isArray(outputArray)) {
outputArray = [outputArray];
}
@@ -232,7 +288,7 @@ export function stepToConnections(step: Step): Connection[] {
const connection = new Connection(
{
stepId: step.id,
- name: input_name,
+ name: inputName,
connectorType: "input",
},
{
@@ -241,6 +297,10 @@ export function stepToConnections(step: Step): Connection[] {
connectorType: "output",
}
);
+ const connectionInput = step.inputs.find((input) => input.name == inputName);
+ if (connectionInput && "input_subworkflow_step_id" in connectionInput) {
+ connection.input.input_subworkflow_step_id = connectionInput.input_subworkflow_step_id;
+ }
connections.push(connection);
});
});
diff --git a/client/src/stores/workflowStore.js b/client/src/stores/workflowStore.ts
similarity index 65%
rename from client/src/stores/workflowStore.js
rename to client/src/stores/workflowStore.ts
index 5ba6400e1fe..5cf2b8b8d9d 100644
--- a/client/src/stores/workflowStore.js
+++ b/client/src/stores/workflowStore.ts
@@ -1,20 +1,25 @@
import { defineStore } from "pinia";
import axios from "axios";
+import type { Steps } from "@/stores/workflowStepStore";
+import { getAppRoot } from "@/onload/loadConfig";
-import { getAppRoot } from "onload/loadConfig";
+interface Workflow {
+ [index: string]: any;
+ steps: Steps;
+}
export const useWorkflowStore = defineStore("workflowStore", {
state: () => ({
- workflowsByInstanceId: {},
+ workflowsByInstanceId: {} as { [index: string]: Workflow },
}),
getters: {
getWorkflowByInstanceId: (state) => {
- return (workflowId) => {
- state.workflowsByInstanceId[workflowId];
+ return (workflowId: string) => {
+ return state.workflowsByInstanceId[workflowId];
};
},
getWorkflowNameByInstanceId: (state) => {
- return (workflowId) => {
+ return (workflowId: string) => {
const details = state.workflowsByInstanceId[workflowId];
if (details && details.name) {
return details.name;
@@ -24,19 +29,19 @@ export const useWorkflowStore = defineStore("workflowStore", {
};
},
getStoredWorkflowIdByInstanceId: (state) => {
- return (workflowId) => {
+ return (workflowId: string) => {
const storedWorkflow = state.workflowsByInstanceId[workflowId];
return storedWorkflow?.id;
};
},
},
actions: {
- async fetchWorkflowForInstanceId(workflowId) {
+ async fetchWorkflowForInstanceId(workflowId: string) {
console.debug("Fetching workflow details for", workflowId);
const params = { instance: "true" };
const { data } = await axios.get(`${getAppRoot()}api/workflows/${workflowId}`, { params });
this.$patch((state) => {
- state.workflowsByInstanceId[workflowId] = data;
+ state.workflowsByInstanceId[workflowId] = data as Workflow;
});
},
},
diff --git a/client/src/style/scss/ui.scss b/client/src/style/scss/ui.scss
index 9726e862dfa..8f2e316092d 100644
--- a/client/src/style/scss/ui.scss
+++ b/client/src/style/scss/ui.scss
@@ -102,37 +102,6 @@ $ui-margin-horizontal-large: $margin-v * 2;
cursor: pointer;
}
-// thumbnails
-.ui-thumbnails {
- .ui-thumbnails-item {
- cursor: pointer;
- .ui-thumbnails-image {
- padding: 1rem;
- width: 5rem;
- height: 4.3rem;
- }
- .ui-thumbnails-icon {
- padding: 1rem;
- width: 5rem;
- height: 4.3rem;
- font-size: 2em;
- text-align: center;
- color: $text-color;
- }
- .ui-thumbnails-text {
- word-wrap: break-word;
- }
- }
- .ui-thumbnails-item:hover {
- .ui-thumbnails-title {
- text-decoration: underline;
- }
- .ui-thumbnails-text {
- text-decoration: underline;
- }
- }
-}
-
.ui-form-composite {
height: 100%;
flex-direction: column;
diff --git a/client/src/style/scss/workflow.scss b/client/src/style/scss/workflow.scss
index eb94d9c3dfa..a3ca99cf01d 100644
--- a/client/src/style/scss/workflow.scss
+++ b/client/src/style/scss/workflow.scss
@@ -117,6 +117,9 @@
.ribbon-inner-invalid {
stroke: $brand-warning;
}
+ &.dashed {
+ stroke-dasharray: 5, 3;
+ }
&:hover .ribbon-outer {
stroke: $brand-success;
}
diff --git a/client/src/utils/mountVueComponent.js b/client/src/utils/mountVueComponent.js
index 02efe8e2e5b..c7b1f240f07 100644
--- a/client/src/utils/mountVueComponent.js
+++ b/client/src/utils/mountVueComponent.js
@@ -32,10 +32,9 @@ export const mountVueComponent = (ComponentDefinition) => {
return (propsData, el) => new component({ store, propsData, el });
};
-export const appendVueComponent = ($el, ComponentDefinition, propsData = {}) => {
- // TODO: this doesn't append, it wipes out the element and replaces contents?
+export const replaceChildrenWithComponent = (el, ComponentDefinition, propsData = {}) => {
const container = document.createElement("div");
- $el.replaceChildren(container);
+ el.replaceChildren(container);
const component = Vue.extend(ComponentDefinition);
const mountFn = (propsData, el) => new component({ propsData, el });
return mountFn(propsData, container);
diff --git a/client/src/utils/navigation/navigation.yml b/client/src/utils/navigation/navigation.yml
index 5b8a7a8120c..7114c497544 100644
--- a/client/src/utils/navigation/navigation.yml
+++ b/client/src/utils/navigation/navigation.yml
@@ -183,6 +183,7 @@ history_panel:
edit_button: '${_} .edit-btn'
delete_button: '${_} .delete-btn'
rerun_button: '${_} .rerun-btn'
+ visualize_button: '${_} .visualize-btn'
collection_job_details_button: '${_} .collection-job-details-btn'
# Action buttons...
@@ -523,6 +524,11 @@ workflows:
annotation: '#workflow_annotation'
submit: '#submit'
+visualization:
+ selectors:
+ _: '#center div.plugin-list'
+ plugin_item: '[data-plugin-name*="${id}"]'
+
trs_search:
selectors:
search: "#trs-search-query"
@@ -643,6 +649,7 @@ workflow_editor:
//div[@data-label='Remove Tags']//input
tool_version_button: ".tool-versions"
connector_for: "#connection-${sink_id}-${source_id}"
+ connector_invalid_for: "#connection-${sink_id}-${source_id} .ribbon-inner-invalid"
connector_destroy_callout: '.delete-terminal'
save_button: '.editor-button-save'
state_modal_body: '.state-upgrade-modal'
diff --git a/client/src/utils/wait.ts b/client/src/utils/wait.ts
new file mode 100644
index 00000000000..9c9930b5488
--- /dev/null
+++ b/client/src/utils/wait.ts
@@ -0,0 +1,5 @@
+export function wait(milliseconds: number) {
+ return new Promise((resolve) => {
+ setTimeout(() => resolve(), milliseconds);
+ });
+}
diff --git a/client/tests/jest/__mocks__/config.js b/client/tests/jest/__mocks__/config.js
index 0aaa35c1744..61763815697 100644
--- a/client/tests/jest/__mocks__/config.js
+++ b/client/tests/jest/__mocks__/config.js
@@ -5,6 +5,6 @@ export default {
caching: {
adapter: "memory",
revs_limit: 1,
- pageSize: 50
- }
-}
+ pageSize: 50,
+ },
+};
diff --git a/client/tests/jest/__mocks__/trackster.js b/client/tests/jest/__mocks__/trackster.js
index 34f0f96bfab..ce75d703b2b 100644
--- a/client/tests/jest/__mocks__/trackster.js
+++ b/client/tests/jest/__mocks__/trackster.js
@@ -8,4 +8,4 @@ export class TracksterUI extends Backbone.Model {
initialize(baseURL) {
this.baseURL = baseURL;
}
-}
\ No newline at end of file
+}
diff --git a/client/tests/jest/jest-raw-loader.js b/client/tests/jest/jest-raw-loader.js
index 3ef7c744106..087224e0497 100644
--- a/client/tests/jest/jest-raw-loader.js
+++ b/client/tests/jest/jest-raw-loader.js
@@ -1,3 +1,5 @@
module.exports = {
- process: content => { return { code: "module.exports = " + JSON.stringify(content)}}
+ process: (content) => {
+ return { code: "module.exports = " + JSON.stringify(content) };
+ },
};
diff --git a/client/tests/jest/jest.setup.js b/client/tests/jest/jest.setup.js
index 474a5845a15..3dda52917b4 100644
--- a/client/tests/jest/jest.setup.js
+++ b/client/tests/jest/jest.setup.js
@@ -1,5 +1,5 @@
import "@testing-library/jest-dom";
-import Vue from 'vue';
+import Vue from "vue";
// Set Vue to suppress production / devtools / etc. warnings
Vue.config.productionTip = false;
diff --git a/client/tests/jest/yaml-jest.js b/client/tests/jest/yaml-jest.js
index 8e84063fe5b..2034043ff0e 100644
--- a/client/tests/jest/yaml-jest.js
+++ b/client/tests/jest/yaml-jest.js
@@ -1,13 +1,13 @@
-const transformer = require("yaml-jest").default
+const transformer = require("yaml-jest").default;
const newTransformer = {
- ...transformer,
- process: function (...params) {
- return {
- code: transformer?.process(...params),
- map: null,
- }
- },
-}
+ ...transformer,
+ process: function (...params) {
+ return {
+ code: transformer?.process(...params),
+ map: null,
+ };
+ },
+};
-module.exports = newTransformer
+module.exports = newTransformer;
diff --git a/client/tests/qunit/test-data/paired-collection-creator.data.js b/client/tests/qunit/test-data/paired-collection-creator.data.js
index a6c3c954568..1e2638c8a75 100644
--- a/client/tests/qunit/test-data/paired-collection-creator.data.js
+++ b/client/tests/qunit/test-data/paired-collection-creator.data.js
@@ -19,23 +19,23 @@ var datasets1 = [
];
var datasets2 = [
- { name: "DP134_1_FS_PSII_FSB_42C_A10.1.fastq", state: STATES.OK},
- { name: "DP134_1_FS_PSII_FSB_42C_A10.2.fastq", state: STATES.OK}
-]
+ { name: "DP134_1_FS_PSII_FSB_42C_A10.1.fastq", state: STATES.OK },
+ { name: "DP134_1_FS_PSII_FSB_42C_A10.2.fastq", state: STATES.OK },
+];
var datasets3 = [
- {name: "UII_moo_1.1.fastq", state: STATES.OK},
- {name: "UII_moo_1.2.fastq", state: STATES.OK}
-]
+ { name: "UII_moo_1.1.fastq", state: STATES.OK },
+ { name: "UII_moo_1.2.fastq", state: STATES.OK },
+];
-var datasets4= [
+var datasets4 = [
{ name: "SET1-01_R1.fastq", state: STATES.OK },
{ name: "SET1-01_R2.fastq", state: STATES.OK },
{ name: "SET1-02_R1.fastq", state: STATES.OK },
{ name: "SET1-02_R2.fastq", state: STATES.OK },
{ name: "SET1-03_R1.fastq", state: STATES.OK },
{ name: "SET1-03_R2.fastq", state: STATES.OK },
-]
+];
var datasets1CreateRequestJSON = {
type: "dataset_collection",
@@ -171,5 +171,5 @@ export default {
_1requestJSON: datasets1CreateRequestJSON,
_2: datasets2,
_3: datasets3,
- _4: datasets4
+ _4: datasets4,
};
diff --git a/client/types/globals.d.ts b/client/types/globals.d.ts
index abf750be2e1..97ec1d6ccf9 100644
--- a/client/types/globals.d.ts
+++ b/client/types/globals.d.ts
@@ -1,4 +1,4 @@
// Webpack-injected globals.
-declare const __webpack_public_path__:string;
+declare const __webpack_public_path__: string;
declare const __buildTimestamp__: string;
-declare const __license__: string;
\ No newline at end of file
+declare const __license__: string;
diff --git a/client/types/shim-vue.d.ts b/client/types/shim-vue.d.ts
index 0660bd67a54..d6e6b864e50 100644
--- a/client/types/shim-vue.d.ts
+++ b/client/types/shim-vue.d.ts
@@ -1,4 +1,4 @@
declare module "*.vue" {
- import Vue from "vue";
- export default Vue;
+ import Vue from "vue";
+ export default Vue;
}
diff --git a/client/types/types.d.ts b/client/types/types.d.ts
index efe9231ef21..423222e511c 100644
--- a/client/types/types.d.ts
+++ b/client/types/types.d.ts
@@ -1,4 +1,4 @@
-declare module '*.yml' {
- const data: any
- export default data
+declare module "*.yml" {
+ const data: any;
+ export default data;
}
diff --git a/client/webpack.config.js b/client/webpack.config.js
index 9a892104993..f9493e6ec7c 100644
--- a/client/webpack.config.js
+++ b/client/webpack.config.js
@@ -40,8 +40,9 @@ module.exports = (env = {}, argv = {}) => {
toolshed: ["polyfills", "bundleToolshed", "entry/generic"],
},
output: {
- path: path.join(__dirname, "../", "/static/dist"),
+ path: path.join(__dirname, "dist"),
filename: "[name].bundled.js",
+ clean: true,
},
resolve: {
plugins: [new TsconfigPathsPlugin({ extensions: [".ts", ".js", ".json", ".vue", ".scss"] })],
@@ -249,16 +250,20 @@ module.exports = (env = {}, argv = {}) => {
allowedHosts: process.env.GITPOD_WORKSPACE_ID ? "all" : "auto",
devMiddleware: {
publicPath: "/static/dist",
+ writeToDisk: true,
},
hot: true,
- port: 8081,
+ port: process.env.WEBPACK_PORT || 8081,
host: "0.0.0.0",
// proxy *everything* to the galaxy server.
// someday, when we have a fully API-driven independent client, this
// can be a more limited set -- e.g. `/api`, `/auth`
proxy: {
"**": {
- target: process.env.GALAXY_URL || "http://localhost:8080",
+ // We explicitly use ipv4 loopback instead of localhost to
+ // avoid ipv6/ipv4 resolution order issues; this should
+ // align with Galaxy's default.
+ target: process.env.GALAXY_URL || "http://127.0.0.1:8080",
secure: process.env.CHANGE_ORIGIN ? !process.env.CHANGE_ORIGIN : true,
changeOrigin: !!process.env.CHANGE_ORIGIN,
logLevel: "debug",
diff --git a/client/yarn.lock b/client/yarn.lock
index ecccf5a872b..d7f18e64d3a 100644
--- a/client/yarn.lock
+++ b/client/yarn.lock
@@ -2295,11 +2295,21 @@
resolved "https://registry.npmjs.org/@types/mime/-/mime-3.0.1.tgz"
integrity sha512-Y4XFY5VJAuw0FgAqPNd6NNoV44jbq9Bz2L7Rh/J6jLTiHBSBJa9fxqQIvkIld4GsoDOcCbvzOUAbLPsSKKg+uA==
+"@types/minimist@^1.2.2":
+ version "1.2.2"
+ resolved "https://registry.yarnpkg.com/@types/minimist/-/minimist-1.2.2.tgz#ee771e2ba4b3dc5b372935d549fd9617bf345b8c"
+ integrity sha512-jhuKLIRrhvCPLqwPcx6INqmKeiA5EWrsCOPhrlFSrbrmU4ZMPjj5Ul/oLCMDO98XRUIwVm78xICz4EPCektzeQ==
+
"@types/node@*", "@types/node@>=10.0.0":
version "18.6.5"
resolved "https://registry.npmjs.org/@types/node/-/node-18.6.5.tgz"
integrity sha512-Xjt5ZGUa5WusGZJ4WJPbOT8QOqp6nDynVFRKcUt32bOgvXEoc6o085WNkYTMO7ifAj2isEfQQ2cseE+wT6jsRw==
+"@types/normalize-package-data@^2.4.0":
+ version "2.4.1"
+ resolved "https://registry.yarnpkg.com/@types/normalize-package-data/-/normalize-package-data-2.4.1.tgz#d3357479a0fdfdd5907fe67e17e0a85c906e1301"
+ integrity sha512-Gj7cI7z+98M282Tqmp2K5EIsoouUEzbBJhQQzDE3jSIRk6r9gsz0oUokqIUR4u1R3dMHo0pDHM7sNOHyhulypw==
+
"@types/parse-json@^4.0.0":
version "4.0.0"
resolved "https://registry.npmjs.org/@types/parse-json/-/parse-json-4.0.0.tgz"
@@ -2788,6 +2798,14 @@ aggregate-error@^3.0.0:
clean-stack "^2.0.0"
indent-string "^4.0.0"
+aggregate-error@^4.0.0:
+ version "4.0.1"
+ resolved "https://registry.yarnpkg.com/aggregate-error/-/aggregate-error-4.0.1.tgz#25091fe1573b9e0be892aeda15c7c66a545f758e"
+ integrity sha512-0poP0T7el6Vq3rstR8Mn4V/IQrpBLO6POkUSrN7RhyY+GF/InCFShQzsQ39T25gkHhLgSLByyAz+Kjb+c2L98w==
+ dependencies:
+ clean-stack "^4.0.0"
+ indent-string "^5.0.0"
+
ajv-formats@^2.1.1:
version "2.1.1"
resolved "https://registry.npmjs.org/ajv-formats/-/ajv-formats-2.1.1.tgz"
@@ -3035,6 +3053,16 @@ array-unique@^0.3.2:
resolved "https://registry.npmjs.org/array-unique/-/array-unique-0.3.2.tgz"
integrity sha512-SleRWjh9JUud2wH1hPs9rZBZ33H6T9HOiL0uwGnGx9FpE6wKGyfWugmbkEOIs6qWrZhg0LWeLziLrEwQJhs5mQ==
+arrify@^1.0.1:
+ version "1.0.1"
+ resolved "https://registry.yarnpkg.com/arrify/-/arrify-1.0.1.tgz#898508da2226f380df904728456849c1501a4b0d"
+ integrity sha512-3CYzex9M9FGQjCGMGyi6/31c8GJbgb0qGyrx5HWxPd0aCwh4cB2YjMb2Xf9UuoogrMrlO9cTqnB5rI5GHZTcUA==
+
+arrify@^3.0.0:
+ version "3.0.0"
+ resolved "https://registry.yarnpkg.com/arrify/-/arrify-3.0.0.tgz#ccdefb8eaf2a1d2ab0da1ca2ce53118759fd46bc"
+ integrity sha512-tLkvA81vQG/XqE2mjDkGQHoOINtMHtysSnemrmoGe6PydDPMRbVugqyk4A6V/WDWEfm3l+0d8anA9r8cv/5Jaw==
+
assert@^2.0.0:
version "2.0.0"
resolved "https://registry.npmjs.org/assert/-/assert-2.0.0.tgz"
@@ -3506,6 +3534,16 @@ callsites@^3.0.0:
resolved "https://registry.npmjs.org/callsites/-/callsites-3.1.0.tgz"
integrity sha512-P8BjAsXvZS+VIDUI11hHCQEv74YT67YUi5JJFNWIqL235sBmjX4+qx9Muvls5ivyNENctx46xQLQ3aTuE7ssaQ==
+camelcase-keys@^7.0.0:
+ version "7.0.2"
+ resolved "https://registry.yarnpkg.com/camelcase-keys/-/camelcase-keys-7.0.2.tgz#d048d8c69448745bb0de6fc4c1c52a30dfbe7252"
+ integrity sha512-Rjs1H+A9R+Ig+4E/9oyB66UC5Mj9Xq3N//vcLf2WzgdTi/3gUu3Z9KoqmlrEG4VuuLK8wJHofxzdQXz/knhiYg==
+ dependencies:
+ camelcase "^6.3.0"
+ map-obj "^4.1.0"
+ quick-lru "^5.1.1"
+ type-fest "^1.2.1"
+
camelcase@^3.0.0:
version "3.0.0"
resolved "https://registry.npmjs.org/camelcase/-/camelcase-3.0.0.tgz"
@@ -3516,7 +3554,7 @@ camelcase@^5.3.1:
resolved "https://registry.npmjs.org/camelcase/-/camelcase-5.3.1.tgz"
integrity sha512-L28STB170nwWS63UjtlEOE3dldQApaJXZkOI1uMFfzf3rRuPegHaHesyee+YxQ+W6SvRDQV6UrdOdRiR153wJg==
-camelcase@^6.2.0:
+camelcase@^6.2.0, camelcase@^6.3.0:
version "6.3.0"
resolved "https://registry.npmjs.org/camelcase/-/camelcase-6.3.0.tgz"
integrity sha512-Gmy6FhYlCY7uOElZUSbxo2UCDH8owEk996gkbrpsgGtrJLM3J7jGxl9Ic7Qwwj4ivOE5AWZWRMecDdF7hqGjFA==
@@ -3662,6 +3700,13 @@ clean-stack@^2.0.0:
resolved "https://registry.npmjs.org/clean-stack/-/clean-stack-2.2.0.tgz"
integrity sha512-4diC9HaTE+KRAMWhDhrGOECgWZxoevMc5TlkObMqNSsVU62PYzXZ/SMTjzyGAFF1YusgxGcSWTEXBhp0CPwQ1A==
+clean-stack@^4.0.0:
+ version "4.2.0"
+ resolved "https://registry.yarnpkg.com/clean-stack/-/clean-stack-4.2.0.tgz#c464e4cde4ac789f4e0735c5d75beb49d7b30b31"
+ integrity sha512-LYv6XPxoyODi36Dp976riBtSY27VmFo+MKqEU9QCCWyTrdEPDog+RWA7xQWHi6Vbp61j5c4cdzzX1NidnwtUWg==
+ dependencies:
+ escape-string-regexp "5.0.0"
+
cliui@^3.2.0:
version "3.2.0"
resolved "https://registry.npmjs.org/cliui/-/cliui-3.2.0.tgz"
@@ -4012,6 +4057,38 @@ cosmiconfig@^7.0.0:
path-type "^4.0.0"
yaml "^1.10.0"
+cp-file@^9.1.0:
+ version "9.1.0"
+ resolved "https://registry.yarnpkg.com/cp-file/-/cp-file-9.1.0.tgz#e98e30db72d57d47b5b1d444deb70d05e5684921"
+ integrity sha512-3scnzFj/94eb7y4wyXRWwvzLFaQp87yyfTnChIjlfYrVqp5lVO3E2hIJMeQIltUT0K2ZAB3An1qXcBmwGyvuwA==
+ dependencies:
+ graceful-fs "^4.1.2"
+ make-dir "^3.0.0"
+ nested-error-stacks "^2.0.0"
+ p-event "^4.1.0"
+
+cpy-cli@^4.2.0:
+ version "4.2.0"
+ resolved "https://registry.yarnpkg.com/cpy-cli/-/cpy-cli-4.2.0.tgz#d60bf9ac776486af20aac2fa4fb95f8f10b35ce8"
+ integrity sha512-b04b+cbdr29CdpREPKw/itrfjO43Ty0Aj7wRM6M6LoE4GJxZJCk9Xp+Eu1IqztkKh3LxIBt1tDplENsa6KYprg==
+ dependencies:
+ cpy "^9.0.0"
+ meow "^10.1.2"
+
+cpy@^9.0.0:
+ version "9.0.1"
+ resolved "https://registry.yarnpkg.com/cpy/-/cpy-9.0.1.tgz#7f3ad0ad5bafe0bc70645c4bb567969927cadb9f"
+ integrity sha512-D9U0DR5FjTCN3oMTcFGktanHnAG5l020yvOCR1zKILmAyPP7I/9pl6NFgRbDcmSENtbK1sQLBz1p9HIOlroiNg==
+ dependencies:
+ arrify "^3.0.0"
+ cp-file "^9.1.0"
+ globby "^13.1.1"
+ junk "^4.0.0"
+ micromatch "^4.0.4"
+ nested-error-stacks "^2.1.0"
+ p-filter "^3.0.0"
+ p-map "^5.3.0"
+
cross-spawn@^7.0.2, cross-spawn@^7.0.3:
version "7.0.3"
resolved "https://registry.npmjs.org/cross-spawn/-/cross-spawn-7.0.3.tgz"
@@ -4489,11 +4566,24 @@ debug@4, debug@^4.1.0, debug@^4.1.1, debug@^4.3.2, debug@^4.3.4, debug@~4.3.1, d
dependencies:
ms "2.1.2"
-decamelize@^1.1.1:
+decamelize-keys@^1.1.0:
+ version "1.1.1"
+ resolved "https://registry.yarnpkg.com/decamelize-keys/-/decamelize-keys-1.1.1.tgz#04a2d523b2f18d80d0158a43b895d56dff8d19d8"
+ integrity sha512-WiPxgEirIV0/eIOMcnFBA3/IJZAZqKnwAwWyvvdi4lsr1WCN22nhdf/3db3DoZcUjTV2SqfzIwNyp6y2xs3nmg==
+ dependencies:
+ decamelize "^1.1.0"
+ map-obj "^1.0.0"
+
+decamelize@^1.1.0, decamelize@^1.1.1:
version "1.2.0"
resolved "https://registry.npmjs.org/decamelize/-/decamelize-1.2.0.tgz"
integrity sha512-z2S+W9X73hAUUki+N+9Za2lBlun89zigOyGrsax+KUQ6wKW4ZoWpEYBkGhQjwAjjDCkWxhY0VKEhk8wzY7F5cA==
+decamelize@^5.0.0:
+ version "5.0.1"
+ resolved "https://registry.yarnpkg.com/decamelize/-/decamelize-5.0.1.tgz#db11a92e58c741ef339fb0a2868d8a06a9a7b1e9"
+ integrity sha512-VfxadyCECXgQlkoEAjeghAr5gY3Hf+IKjKb+X8tGVDtveCjN+USwprd2q3QXBR9T1+x2DG0XZF5/w+7HAtSaXA==
+
decimal.js@^10.4.2:
version "10.4.3"
resolved "https://registry.yarnpkg.com/decimal.js/-/decimal.js-10.4.3.tgz#1044092884d245d1b7f65725fa4ad4c6f781cc23"
@@ -5052,6 +5142,11 @@ escape-html@~1.0.3:
resolved "https://registry.npmjs.org/escape-html/-/escape-html-1.0.3.tgz"
integrity sha512-NiSupZ4OeuGwr68lGIeym/ksIZMJodUGOSCZ/FSnTxcrekbvqrgdUxlJOMpijaKZVjAJrWrGs/6Jy8OMuyj9ow==
+escape-string-regexp@5.0.0:
+ version "5.0.0"
+ resolved "https://registry.yarnpkg.com/escape-string-regexp/-/escape-string-regexp-5.0.0.tgz#4683126b500b61762f2dbebace1806e8be31b1c8"
+ integrity sha512-/veY75JbMK4j1yjvuUxuVsiS/hr/4iHs9FTT6cgTexxdE0Ly/glccBAkloH/DofkjRbZU3bnoj38mOmhkZ0lHw==
+
escape-string-regexp@^1.0.5:
version "1.0.5"
resolved "https://registry.npmjs.org/escape-string-regexp/-/escape-string-regexp-1.0.5.tgz"
@@ -5409,7 +5504,7 @@ fast-deep-equal@^3.1.1, fast-deep-equal@^3.1.3:
resolved "https://registry.npmjs.org/fast-deep-equal/-/fast-deep-equal-3.1.3.tgz"
integrity sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q==
-fast-glob@^3.2.12:
+fast-glob@^3.2.11, fast-glob@^3.2.12:
version "3.2.12"
resolved "https://registry.yarnpkg.com/fast-glob/-/fast-glob-3.2.12.tgz#7f39ec99c2e6ab030337142da9e0c18f37afae80"
integrity sha512-DVj4CQIYYow0BlaelwK1pHl5n5cRSJfM60UA0zK891sVInoPri2Ekj7+e1CT3/3qxXenpI+nBBmQAcJPJgaj4w==
@@ -5926,6 +6021,17 @@ globby@^11.0.1, globby@^11.1.0:
merge2 "^1.4.1"
slash "^3.0.0"
+globby@^13.1.1:
+ version "13.1.3"
+ resolved "https://registry.yarnpkg.com/globby/-/globby-13.1.3.tgz#f62baf5720bcb2c1330c8d4ef222ee12318563ff"
+ integrity sha512-8krCNHXvlCgHDpegPzleMq07yMYTO2sXKASmZmquEYWEmCx6J5UTRbp5RwMJkTJGtcQ44YpiUYUiN0b9mzy8Bw==
+ dependencies:
+ dir-glob "^3.0.1"
+ fast-glob "^3.2.11"
+ ignore "^5.2.0"
+ merge2 "^1.4.1"
+ slash "^4.0.0"
+
globrex@^0.1.2:
version "0.1.2"
resolved "https://registry.npmjs.org/globrex/-/globrex-0.1.2.tgz"
@@ -6010,6 +6116,11 @@ handsontable@^4.0.0:
numbro "^2.0.6"
pikaday "1.5.1"
+hard-rejection@^2.1.0:
+ version "2.1.0"
+ resolved "https://registry.yarnpkg.com/hard-rejection/-/hard-rejection-2.1.0.tgz#1c6eda5c1685c63942766d79bb40ae773cecd883"
+ integrity sha512-VIZB+ibDhx7ObhAe7OVtoEbuP4h/MuOTHJ+J8h/eBXotJYl0fBgR72xDFCKgIh22OJZIOVNxBMWuhAr10r8HdA==
+
has-bigints@^1.0.1, has-bigints@^1.0.2:
version "1.0.2"
resolved "https://registry.npmjs.org/has-bigints/-/has-bigints-1.0.2.tgz"
@@ -6104,6 +6215,13 @@ hosted-git-info@^2.1.4:
resolved "https://registry.npmjs.org/hosted-git-info/-/hosted-git-info-2.8.9.tgz"
integrity sha512-mxIDAb9Lsm6DoOJ7xH+5+X4y1LU/4Hi50L9C5sIswK3JzULS4bwk1FvjdBgvYR4bzT4tuUQiC15FE2f5HbLvYw==
+hosted-git-info@^4.0.1:
+ version "4.1.0"
+ resolved "https://registry.yarnpkg.com/hosted-git-info/-/hosted-git-info-4.1.0.tgz#827b82867e9ff1c8d0c4d9d53880397d2c86d224"
+ integrity sha512-kyCuEOWjJqZuDbRHzL8V93NzQhwIB71oFWSyzVo+KPZI+pnQPPxucdkrOZvkLRnrf5URsQM+IJ09Dw29cRALIA==
+ dependencies:
+ lru-cache "^6.0.0"
+
hpack.js@^2.1.6:
version "2.1.6"
resolved "https://registry.npmjs.org/hpack.js/-/hpack.js-2.1.6.tgz"
@@ -6300,6 +6418,11 @@ indent-string@^4.0.0:
resolved "https://registry.npmjs.org/indent-string/-/indent-string-4.0.0.tgz"
integrity sha512-EdDDZu4A2OyIK7Lr/2zG+w5jmbuk1DVBnEwREQvBzspBJkCEbRa8GxU1lghYcaGJCnRWibjDXlq779X1/y5xwg==
+indent-string@^5.0.0:
+ version "5.0.0"
+ resolved "https://registry.yarnpkg.com/indent-string/-/indent-string-5.0.0.tgz#4fd2980fccaf8622d14c64d694f4cf33c81951a5"
+ integrity sha512-m6FAo/spmsW2Ab2fU35JTYwtOKa2yAwXSwgjSv1TJzh4Mh7mC3lzAOVLBprb72XsTrgkEIsl7YrFNAiDiRhIGg==
+
inflight@^1.0.4:
version "1.0.6"
resolved "https://registry.npmjs.org/inflight/-/inflight-1.0.6.tgz"
@@ -6434,6 +6557,13 @@ is-callable@^1.1.3, is-callable@^1.1.4, is-callable@^1.2.4:
resolved "https://registry.npmjs.org/is-callable/-/is-callable-1.2.4.tgz"
integrity sha512-nsuwtxZfMX67Oryl9LCQ+upnC0Z0BgpwntpS89m1H/TLF0zNfzfLMV/9Wa/6MZsj0acpEjAO0KF1xT6ZdLl95w==
+is-core-module@^2.5.0:
+ version "2.11.0"
+ resolved "https://registry.yarnpkg.com/is-core-module/-/is-core-module-2.11.0.tgz#ad4cb3e3863e814523c96f3f58d26cc570ff0144"
+ integrity sha512-RRjxlvLDkD1YJwDbroBHMb+cukurkDWNyHx7D3oNB5x9rb5ogcksMC5wHCadcXoo67gVr/+3GFySh3134zi6rw==
+ dependencies:
+ has "^1.0.3"
+
is-core-module@^2.9.0:
version "2.10.0"
resolved "https://registry.npmjs.org/is-core-module/-/is-core-module-2.10.0.tgz"
@@ -6597,6 +6727,11 @@ is-path-inside@^3.0.2, is-path-inside@^3.0.3:
resolved "https://registry.npmjs.org/is-path-inside/-/is-path-inside-3.0.3.tgz"
integrity sha512-Fd4gABb+ycGAmKou8eMftCupSir5lRxqf4aD/vd0cD2qc4HL07OjCeuHMr8Ro4CoMaeCKDB0/ECBOVWjTwUvPQ==
+is-plain-obj@^1.1.0:
+ version "1.1.0"
+ resolved "https://registry.yarnpkg.com/is-plain-obj/-/is-plain-obj-1.1.0.tgz#71a50c8429dfca773c92a390a4a03b39fcd51d3e"
+ integrity sha512-yvkRyxmFKEOQ4pNXCmJG5AEQNlXJS5LaONXo5/cLdTZdWvsZ1ioJEonLGAosKlMWE8lwUy/bJzMjcw8az73+Fg==
+
is-plain-obj@^3.0.0:
version "3.0.0"
resolved "https://registry.npmjs.org/is-plain-obj/-/is-plain-obj-3.0.0.tgz"
@@ -7425,6 +7560,11 @@ jspdf@^2.5.1:
dompurify "^2.2.0"
html2canvas "^1.0.0-rc.5"
+junk@^4.0.0:
+ version "4.0.0"
+ resolved "https://registry.yarnpkg.com/junk/-/junk-4.0.0.tgz#9b1104ddf5281cd24ffa3c8a7443d19ce192b37f"
+ integrity sha512-ojtSU++zLJ3jQG9bAYjg94w+/DOJtRyD7nPaerMFrBhmdVmiV5/exYH5t4uHga4G/95nT6hr1OJoKIFbYbrW5w==
+
just-debounce@^1.0.0:
version "1.1.0"
resolved "https://registry.npmjs.org/just-debounce/-/just-debounce-1.1.0.tgz"
@@ -7510,7 +7650,7 @@ kind-of@^5.0.0, kind-of@^5.0.2:
resolved "https://registry.npmjs.org/kind-of/-/kind-of-5.1.0.tgz"
integrity sha512-NGEErnH6F2vUuXDh+OlbcKW7/wOcfdRHaZ7VWtqCztfHri/++YKmP51OdWeGPuqCOba6kk2OTe5d02VmTB80Pw==
-kind-of@^6.0.0, kind-of@^6.0.2:
+kind-of@^6.0.0, kind-of@^6.0.2, kind-of@^6.0.3:
version "6.0.3"
resolved "https://registry.npmjs.org/kind-of/-/kind-of-6.0.3.tgz"
integrity sha512-dcS1ul+9tmeD95T+x28/ehLgd9mENa3LsvDTtzm3vyBEO7RPptvAD+t44WVXaUjTBRcrpFeFlC8WCruUR456hw==
@@ -7832,6 +7972,16 @@ map-cache@^0.2.0, map-cache@^0.2.2:
resolved "https://registry.npmjs.org/map-cache/-/map-cache-0.2.2.tgz"
integrity sha512-8y/eV9QQZCiyn1SprXSrCmqJN0yNRATe+PO8ztwqrvrbdRLA3eYJF0yaR0YayLWkMbsQSKWS9N2gPcGEc4UsZg==
+map-obj@^1.0.0:
+ version "1.0.1"
+ resolved "https://registry.yarnpkg.com/map-obj/-/map-obj-1.0.1.tgz#d933ceb9205d82bdcf4886f6742bdc2b4dea146d"
+ integrity sha512-7N/q3lyZ+LVCp7PzuxrJr4KMbBE2hW7BT7YNia330OFxIf4d3r5zVpicP2650l7CPN6RM9zOJRl3NGpqSiw3Eg==
+
+map-obj@^4.1.0:
+ version "4.3.0"
+ resolved "https://registry.yarnpkg.com/map-obj/-/map-obj-4.3.0.tgz#9304f906e93faae70880da102a9f1df0ea8bb05a"
+ integrity sha512-hdN1wVrZbb29eBGiGjJbeP8JbKjq1urkHJ/LIP/NY48MZ1QVXUsQBV1G1zvYFHn1XE06cwjBsOI2K3Ulnj1YXQ==
+
map-visit@^1.0.0:
version "1.0.0"
resolved "https://registry.npmjs.org/map-visit/-/map-visit-1.0.0.tgz"
@@ -7892,6 +8042,24 @@ memfs@^3.4.3:
dependencies:
fs-monkey "^1.0.3"
+meow@^10.1.2:
+ version "10.1.5"
+ resolved "https://registry.yarnpkg.com/meow/-/meow-10.1.5.tgz#be52a1d87b5f5698602b0f32875ee5940904aa7f"
+ integrity sha512-/d+PQ4GKmGvM9Bee/DPa8z3mXs/pkvJE2KEThngVNOqtmljC6K7NMPxtc2JeZYTmpWb9k/TmxjeL18ez3h7vCw==
+ dependencies:
+ "@types/minimist" "^1.2.2"
+ camelcase-keys "^7.0.0"
+ decamelize "^5.0.0"
+ decamelize-keys "^1.1.0"
+ hard-rejection "^2.1.0"
+ minimist-options "4.1.0"
+ normalize-package-data "^3.0.2"
+ read-pkg-up "^8.0.0"
+ redent "^4.0.0"
+ trim-newlines "^4.0.2"
+ type-fest "^1.2.2"
+ yargs-parser "^20.2.9"
+
merge-descriptors@1.0.1:
version "1.0.1"
resolved "https://registry.npmjs.org/merge-descriptors/-/merge-descriptors-1.0.1.tgz"
@@ -7973,7 +8141,7 @@ mimic-fn@^2.1.0:
resolved "https://registry.npmjs.org/mimic-fn/-/mimic-fn-2.1.0.tgz"
integrity sha512-OqbOk5oEQeAZ8WXWydlu9HJjz9WVdEIvamMCcXmuqUYjTknH/sqsWvhQ3vgwKFRR1HpjvNBKQ37nbJgYzGqGcg==
-min-indent@^1.0.0:
+min-indent@^1.0.0, min-indent@^1.0.1:
version "1.0.1"
resolved "https://registry.npmjs.org/min-indent/-/min-indent-1.0.1.tgz"
integrity sha512-I9jwMn07Sy/IwOj3zVkVik2JTvgpaykDZEigL6Rx6N9LbMywwUSMtxET+7lVoDLLd3O3IXwJwvuuns8UB/HeAg==
@@ -8004,6 +8172,15 @@ minimatch@^5.0.1:
dependencies:
brace-expansion "^2.0.1"
+minimist-options@4.1.0:
+ version "4.1.0"
+ resolved "https://registry.yarnpkg.com/minimist-options/-/minimist-options-4.1.0.tgz#c0655713c53a8a2ebd77ffa247d342c40f010619"
+ integrity sha512-Q4r8ghd80yhO/0j1O3B2BjweX3fiHg9cdOwjJd2J76Q135c+NDxGCqdYKQ1SKBuFfgWbAUzBfvYjPUEeNgqN1A==
+ dependencies:
+ arrify "^1.0.1"
+ is-plain-obj "^1.1.0"
+ kind-of "^6.0.3"
+
minimist@^1.2.0, minimist@^1.2.6:
version "1.2.6"
resolved "https://registry.npmjs.org/minimist/-/minimist-1.2.6.tgz"
@@ -8109,6 +8286,11 @@ neo-async@^2.6.2:
resolved "https://registry.npmjs.org/neo-async/-/neo-async-2.6.2.tgz"
integrity sha512-Yd3UES5mWCSqR+qNT93S3UoYUkqAZ9lLg8a7g9rimsWmYGK8cVToA4/sF3RrshdyV3sAGMXVUmpMYOw+dLpOuw==
+nested-error-stacks@^2.0.0, nested-error-stacks@^2.1.0:
+ version "2.1.1"
+ resolved "https://registry.yarnpkg.com/nested-error-stacks/-/nested-error-stacks-2.1.1.tgz#26c8a3cee6cc05fbcf1e333cd2fc3e003326c0b5"
+ integrity sha512-9iN1ka/9zmX1ZvLV9ewJYEk9h7RyRRtqdK0woXcqohu8EWIerfPUjYJPg0ULy0UqP7cslmdGc8xKDJcojlKiaw==
+
next-tick@^1.1.0:
version "1.1.0"
resolved "https://registry.npmjs.org/next-tick/-/next-tick-1.1.0.tgz"
@@ -8177,6 +8359,16 @@ normalize-package-data@^2.3.2:
semver "2 || 3 || 4 || 5"
validate-npm-package-license "^3.0.1"
+normalize-package-data@^3.0.2:
+ version "3.0.3"
+ resolved "https://registry.yarnpkg.com/normalize-package-data/-/normalize-package-data-3.0.3.tgz#dbcc3e2da59509a0983422884cd172eefdfa525e"
+ integrity sha512-p2W1sgqij3zMMyRC067Dg16bfzVH+w7hyegmpIvZ4JNjqtGOVAIvLmjBx3yP7YTe9vKJgkoNOPjwQGogDoMXFA==
+ dependencies:
+ hosted-git-info "^4.0.1"
+ is-core-module "^2.5.0"
+ semver "^7.3.4"
+ validate-npm-package-license "^3.0.1"
+
normalize-path@^2.1.1:
version "2.1.1"
resolved "https://registry.npmjs.org/normalize-path/-/normalize-path-2.1.1.tgz"
@@ -8436,6 +8628,25 @@ os-locale@^1.4.0:
dependencies:
lcid "^1.0.0"
+p-event@^4.1.0:
+ version "4.2.0"
+ resolved "https://registry.yarnpkg.com/p-event/-/p-event-4.2.0.tgz#af4b049c8acd91ae81083ebd1e6f5cae2044c1b5"
+ integrity sha512-KXatOjCRXXkSePPb1Nbi0p0m+gQAwdlbhi4wQKJPI1HsMQS9g+Sqp2o+QHziPr7eYJyOZet836KoHEVM1mwOrQ==
+ dependencies:
+ p-timeout "^3.1.0"
+
+p-filter@^3.0.0:
+ version "3.0.0"
+ resolved "https://registry.yarnpkg.com/p-filter/-/p-filter-3.0.0.tgz#ce50e03b24b23930e11679ab8694bd09a2d7ed35"
+ integrity sha512-QtoWLjXAW++uTX67HZQz1dbTpqBfiidsB6VtQUC9iR85S120+s0T5sO6s+B5MLzFcZkrEd/DGMmCjR+f2Qpxwg==
+ dependencies:
+ p-map "^5.1.0"
+
+p-finally@^1.0.0:
+ version "1.0.0"
+ resolved "https://registry.yarnpkg.com/p-finally/-/p-finally-1.0.0.tgz#3fbcfb15b899a44123b34b6dcc18b724336a2cae"
+ integrity sha512-LICb2p9CB7FS+0eR1oqWnHhp0FljGLZCWBE9aix0Uye9W8LTQPwMTYVGWQWIw9RdQiDg4+epXQODwIYJtSJaow==
+
p-limit@^2.2.0:
version "2.3.0"
resolved "https://registry.npmjs.org/p-limit/-/p-limit-2.3.0.tgz"
@@ -8471,6 +8682,13 @@ p-map@^4.0.0:
dependencies:
aggregate-error "^3.0.0"
+p-map@^5.1.0, p-map@^5.3.0:
+ version "5.5.0"
+ resolved "https://registry.yarnpkg.com/p-map/-/p-map-5.5.0.tgz#054ca8ca778dfa4cf3f8db6638ccb5b937266715"
+ integrity sha512-VFqfGDHlx87K66yZrNdI4YGtD70IRyd+zSvgks6mzHPRNkoKy+9EKP4SFC77/vTTQYmRmti7dvqC+m5jBrBAcg==
+ dependencies:
+ aggregate-error "^4.0.0"
+
p-retry@^4.5.0:
version "4.6.2"
resolved "https://registry.npmjs.org/p-retry/-/p-retry-4.6.2.tgz"
@@ -8479,6 +8697,13 @@ p-retry@^4.5.0:
"@types/retry" "0.12.0"
retry "^0.13.1"
+p-timeout@^3.1.0:
+ version "3.2.0"
+ resolved "https://registry.yarnpkg.com/p-timeout/-/p-timeout-3.2.0.tgz#c7e17abc971d2a7962ef83626b35d635acf23dfe"
+ integrity sha512-rhIwUycgwwKcP9yTOOFK/AKsAopjjCakVqLHePO3CC6Mir1Z99xT+R63jZxAT5lFZLa2inS5h+ZS2GvR99/FBg==
+ dependencies:
+ p-finally "^1.0.0"
+
p-try@^2.0.0:
version "2.2.0"
resolved "https://registry.npmjs.org/p-try/-/p-try-2.2.0.tgz"
@@ -9137,6 +9362,11 @@ queue-microtask@^1.2.2:
resolved "https://registry.npmjs.org/queue-microtask/-/queue-microtask-1.2.3.tgz"
integrity sha512-NuaNSa6flKT5JaSYQzJok04JzTL1CA6aGhv5rfLW3PgqA+M2ChpZQnAC8h8i4ZFkBS8X5RqkDBHA7r4hej3K9A==
+quick-lru@^5.1.1:
+ version "5.1.1"
+ resolved "https://registry.yarnpkg.com/quick-lru/-/quick-lru-5.1.1.tgz#366493e6b3e42a3a6885e2e99d18f80fb7a8c932"
+ integrity sha512-WuyALRjWPDGtt/wzJiadO5AXY+8hZ80hVpe6MyivgraREW751X3SbhRvG3eLKOYN+8VEvqLcf3wdnt44Z4S4SA==
+
qunit@^2.17.2:
version "2.19.1"
resolved "https://registry.npmjs.org/qunit/-/qunit-2.19.1.tgz"
@@ -9201,6 +9431,15 @@ read-pkg-up@^1.0.1:
find-up "^1.0.0"
read-pkg "^1.0.0"
+read-pkg-up@^8.0.0:
+ version "8.0.0"
+ resolved "https://registry.yarnpkg.com/read-pkg-up/-/read-pkg-up-8.0.0.tgz#72f595b65e66110f43b052dd9af4de6b10534670"
+ integrity sha512-snVCqPczksT0HS2EC+SxUndvSzn6LRCwpfSvLrIfR5BKDQQZMaI6jPRC9dYvYFDRAuFEAnkwww8kBBNE/3VvzQ==
+ dependencies:
+ find-up "^5.0.0"
+ read-pkg "^6.0.0"
+ type-fest "^1.0.1"
+
read-pkg@^1.0.0:
version "1.1.0"
resolved "https://registry.npmjs.org/read-pkg/-/read-pkg-1.1.0.tgz"
@@ -9210,6 +9449,16 @@ read-pkg@^1.0.0:
normalize-package-data "^2.3.2"
path-type "^1.0.0"
+read-pkg@^6.0.0:
+ version "6.0.0"
+ resolved "https://registry.yarnpkg.com/read-pkg/-/read-pkg-6.0.0.tgz#a67a7d6a1c2b0c3cd6aa2ea521f40c458a4a504c"
+ integrity sha512-X1Fu3dPuk/8ZLsMhEj5f4wFAF0DWoK7qhGJvgaijocXxBmSToKfbFtqbxMO7bVjNA1dmE5huAzjXj/ey86iw9Q==
+ dependencies:
+ "@types/normalize-package-data" "^2.4.0"
+ normalize-package-data "^3.0.2"
+ parse-json "^5.2.0"
+ type-fest "^1.0.1"
+
readable-stream@^2.0.0, readable-stream@^2.0.1, readable-stream@^2.0.5, readable-stream@^2.1.5, readable-stream@^2.2.2, readable-stream@^2.3.3, readable-stream@^2.3.5, readable-stream@^2.3.6, readable-stream@~2.3.6:
version "2.3.7"
resolved "https://registry.npmjs.org/readable-stream/-/readable-stream-2.3.7.tgz"
@@ -9261,6 +9510,14 @@ redent@^3.0.0:
indent-string "^4.0.0"
strip-indent "^3.0.0"
+redent@^4.0.0:
+ version "4.0.0"
+ resolved "https://registry.yarnpkg.com/redent/-/redent-4.0.0.tgz#0c0ba7caabb24257ab3bb7a4fd95dd1d5c5681f9"
+ integrity sha512-tYkDkVVtYkSVhuQ4zBgfvciymHaeuel+zFKXShfDnFP5SyVEP7qo70Rf1jTOTCx3vGNAbnEi/xFkcfQVMIBWag==
+ dependencies:
+ indent-string "^5.0.0"
+ strip-indent "^4.0.0"
+
regenerate-unicode-properties@^10.0.1:
version "10.0.1"
resolved "https://registry.npmjs.org/regenerate-unicode-properties/-/regenerate-unicode-properties-10.0.1.tgz"
@@ -9856,6 +10113,11 @@ slash@^3.0.0:
resolved "https://registry.npmjs.org/slash/-/slash-3.0.0.tgz"
integrity sha512-g9Q1haeby36OSStwb4ntCGGGaKsaVSjQ68fBxoQcutl5fS1vuY18H3wSt3jFyFtrkx+Kz0V1G85A4MyAdDMi2Q==
+slash@^4.0.0:
+ version "4.0.0"
+ resolved "https://registry.yarnpkg.com/slash/-/slash-4.0.0.tgz#2422372176c4c6c5addb5e2ada885af984b396a7"
+ integrity sha512-3dOsAHXXUkQTpOYcoAxLIorMTp4gIQr5IW3iVb7A7lFIp0VHhnynm9izx6TssdrIcVIESAlVjtnO2K8bg+Coew==
+
slugify@^1.6.5:
version "1.6.5"
resolved "https://registry.npmjs.org/slugify/-/slugify-1.6.5.tgz"
@@ -10258,6 +10520,13 @@ strip-indent@^3.0.0:
dependencies:
min-indent "^1.0.0"
+strip-indent@^4.0.0:
+ version "4.0.0"
+ resolved "https://registry.yarnpkg.com/strip-indent/-/strip-indent-4.0.0.tgz#b41379433dd06f5eae805e21d631e07ee670d853"
+ integrity sha512-mnVSV2l+Zv6BLpSD/8V87CW/y9EmmbYzGCIavsnsI6/nwn26DwffM/yztm30Z/I2DY9wdS3vXVCMnHDgZaVNoA==
+ dependencies:
+ min-indent "^1.0.1"
+
strip-json-comments@^2.0.0:
version "2.0.1"
resolved "https://registry.yarnpkg.com/strip-json-comments/-/strip-json-comments-2.0.1.tgz#3c531942e908c2697c0ec344858c286c7ca0a60a"
@@ -10537,6 +10806,11 @@ tr46@~0.0.3:
resolved "https://registry.npmjs.org/tr46/-/tr46-0.0.3.tgz"
integrity sha512-N3WMsuqV66lT30CrXNbEjx4GEwlow3v6rr4mCcv6prnfwhS01rkgyFdjPNBYd9br7LpXV1+Emh01fHnq2Gdgrw==
+trim-newlines@^4.0.2:
+ version "4.0.2"
+ resolved "https://registry.yarnpkg.com/trim-newlines/-/trim-newlines-4.0.2.tgz#d6aaaf6a0df1b4b536d183879a6b939489808c7c"
+ integrity sha512-GJtWyq9InR/2HRiLZgpIKv+ufIKrVrvjQWEj7PxAXNc5dwbNJkqhAUoAGgzRmULAnoOM5EIpveYd3J2VeSAIew==
+
ts-jest@^29.0.3:
version "29.0.3"
resolved "https://registry.yarnpkg.com/ts-jest/-/ts-jest-29.0.3.tgz#63ea93c5401ab73595440733cefdba31fcf9cb77"
@@ -10648,6 +10922,11 @@ type-fest@^0.21.3:
resolved "https://registry.npmjs.org/type-fest/-/type-fest-0.21.3.tgz"
integrity sha512-t0rzBq87m3fVcduHDUFhKmyyX+9eo6WQjZvf51Ea/M0Q7+T374Jp1aUiyUl0GKxp8M/OETVHSDvmkyPgvX+X2w==
+type-fest@^1.0.1, type-fest@^1.2.1, type-fest@^1.2.2:
+ version "1.4.0"
+ resolved "https://registry.yarnpkg.com/type-fest/-/type-fest-1.4.0.tgz#e9fb813fe3bf1744ec359d55d1affefa76f14be1"
+ integrity sha512-yGSza74xk0UG8k+pLh5oeoYirvIiWo5t0/o3zHHAO2tRDiZcxWP7fywNlXhqb6/r6sWvwi+RsyQMWhVLe4BVuA==
+
type-is@~1.6.18:
version "1.6.18"
resolved "https://registry.npmjs.org/type-is/-/type-is-1.6.18.tgz"
@@ -10682,9 +10961,9 @@ typescript@^4.9.3:
integrity sha512-CIfGzTelbKNEnLpLdGFgdyKhG23CKdKgQPOBc+OUNrkJ2vr+KSzsSV5kq5iWhEQbok+quxgGzrAtGWCyU7tHnA==
ua-parser-js@^0.7.30:
- version "0.7.31"
- resolved "https://registry.npmjs.org/ua-parser-js/-/ua-parser-js-0.7.31.tgz"
- integrity sha512-qLK/Xe9E2uzmYI3qLeOmI0tEOt+TBBQyUIAh4aAgU05FVYzeZrKUdkAZfBNVGRaHVgV0TDkdEngJSw/SyQchkQ==
+ version "0.7.33"
+ resolved "https://registry.yarnpkg.com/ua-parser-js/-/ua-parser-js-0.7.33.tgz#1d04acb4ccef9293df6f70f2c3d22f3030d8b532"
+ integrity sha512-s8ax/CeZdK9R/56Sui0WM6y9OFREJarMRHqLB2EwkovemBxNQ+Bqu8GAsUnVcXKgphb++ghr/B2BZx4mahujPw==
uc.micro@^1.0.1, uc.micro@^1.0.5:
version "1.0.6"
@@ -11499,7 +11778,7 @@ yaml@^2.0.0:
resolved "https://registry.npmjs.org/yaml/-/yaml-2.1.1.tgz"
integrity sha512-o96x3OPo8GjWeSLF+wOAbrPfhFOGY0W00GNaxCDv+9hkcDJEnev1yh8S7pgHF0ik6zc8sQLuL8hjHjJULZp8bw==
-yargs-parser@^20.2.2:
+yargs-parser@^20.2.2, yargs-parser@^20.2.9:
version "20.2.9"
resolved "https://registry.npmjs.org/yargs-parser/-/yargs-parser-20.2.9.tgz"
integrity sha512-y11nGElTIV+CT3Zv9t7VKl+Q3hTQoT9a1Qzezhhl6Rp21gJ/IVTW7Z3y9EWXhuUBC2Shnf+DX0antecpAwSP8w==
diff --git a/config/plugins/tours/core.windows.yaml b/config/plugins/tours/core.windows.yaml
index 226150dadf6..d2318ce3c5d 100644
--- a/config/plugins/tours/core.windows.yaml
+++ b/config/plugins/tours/core.windows.yaml
@@ -6,15 +6,14 @@ tags:
- "UI"
- "visualization"
+requirements:
+ - new_history
+
steps:
- content: "This short tour will walk you through Galaxy's Window Manager feature"
- - element: "#current-history-panel .create-hist-btn"
- intro: "Before using the Window Manager, we will create a new history for this tour."
- postclick: true
-
- element: "#tool-panel-upload-button"
- intro: "Now, we will upload some tabular data into our new history."
+ intro: "First, we will upload some tabular data into our new history."
postclick: true
- element: "#btn-new"
@@ -78,7 +77,7 @@ steps:
- element: "#right"
intro: "This is your history. It contains our two tables being uploaded."
-
+
- component: history_panel.item(hid=1,state=ok).display_button
intro: "Clicking the eye-icon usually displays a dataset in the center panel."
postclick: true
diff --git a/config/plugins/visualizations/annotate_image/src/script.js b/config/plugins/visualizations/annotate_image/src/script.js
index e8167257e4f..d97fe758420 100644
--- a/config/plugins/visualizations/annotate_image/src/script.js
+++ b/config/plugins/visualizations/annotate_image/src/script.js
@@ -5,6 +5,12 @@ import _ from "underscore";
// Use lighter weight 'core' version of paper since we don't need paperscript
import paper from "../node_modules/paper/dist/paper-core.js";
+/* This will be part of the charts/viz standard lib in 23.1 */
+const slashCleanup = /(\/)+/g;
+function prefixedDownloadUrl(root, path) {
+ return `${root}/${path}`.replace(slashCleanup, "/");
+}
+
const CommandManager = (function() {
function CommandManager() {}
@@ -437,12 +443,12 @@ window.bundleEntries.load = function (opt) {
});
};
- const safe_download_url = `${options.root}${dataset.download_url}`;
+ const downloadUrl = prefixedDownloadUrl(opt.root, dataset.download_url);
$.ajax({
- url: safe_download_url,
+ url: downloadUrl,
success: function(content) {
const $chartViewer = $("#" + opt.target);
- $chartViewer.html("![](" + safe_download_url + ")
");
+ $chartViewer.html("![](" + downloadUrl + ")
");
$chartViewer.css("overflow", "auto");
$chartViewer.css("position", "relative");
const $image = $chartViewer.find("img");
diff --git a/config/plugins/visualizations/cytoscape/src/script.js b/config/plugins/visualizations/cytoscape/src/script.js
index fbcb9510c7d..a2d0fdd2d01 100644
--- a/config/plugins/visualizations/cytoscape/src/script.js
+++ b/config/plugins/visualizations/cytoscape/src/script.js
@@ -1,5 +1,11 @@
import Cytoscape from 'cytoscape';
+/* This will be part of the charts/viz standard lib in 23.1 */
+const slashCleanup = /(\/)+/g;
+function prefixedDownloadUrl(root, path) {
+ return `${root}/${path}`.replace(slashCleanup, "/");
+}
+
// Public method. Return graph data as JSON
var parse_sif = function( text ) {
// Private variables and methods
@@ -166,9 +172,8 @@ window.bundleEntries.load = function (options) {
cytoscape = null,
sif_file_ext = "sif",
highlighted_color = settings.get( 'color_picker_highlighted' );
- const safe_download_url = `${options.root}${dataset.download_url}`;
$.ajax({
- url : safe_download_url,
+ url : prefixedDownloadUrl(options.root, dataset.download_url),
success : function( content ) {
// Select data for the graph
if( dataset.file_ext === sif_file_ext ) {
diff --git a/config/plugins/visualizations/drawrna/src/script.js b/config/plugins/visualizations/drawrna/src/script.js
index 2b0ac8faf20..ce081c07266 100644
--- a/config/plugins/visualizations/drawrna/src/script.js
+++ b/config/plugins/visualizations/drawrna/src/script.js
@@ -1,11 +1,17 @@
import DrawRNA from "./drawrnajs/drawrna";
window.bundleEntries = window.bundleEntries || {};
+
+/* This will be part of the charts/viz standard lib in 23.1 */
+const slashCleanup = /(\/)+/g;
+function prefixedDownloadUrl(root, path) {
+ return `${root}/${path}`.replace(slashCleanup, "/");
+}
+
window.bundleEntries.load = function (options) {
var chart = options.chart;
var dataset = options.dataset;
- const safe_download_url = `${options.root}${dataset.download_url}`;
$.ajax({
- url: safe_download_url,
+ url: prefixedDownloadUrl(options.root, dataset.download_url),
success: function(response) {
var input = response.split('\n');
var app = new DrawRNA({
diff --git a/config/plugins/visualizations/msa/src/script.js b/config/plugins/visualizations/msa/src/script.js
index a89020f424b..72dabe78577 100755
--- a/config/plugins/visualizations/msa/src/script.js
+++ b/config/plugins/visualizations/msa/src/script.js
@@ -1,5 +1,11 @@
import "./msa.min.js";
+/* This will be part of the charts/viz standard lib in 23.1 */
+const slashCleanup = /(\/)+/g;
+function prefixedDownloadUrl(root, path) {
+ return `${root}/${path}`.replace(slashCleanup, "/");
+}
+
Object.assign(window.bundleEntries || {}, {
load: function (options) {
const chart = options.chart;
@@ -14,8 +20,7 @@ Object.assign(window.bundleEntries || {}, {
menu: "small",
bootstrapMenu: "true" == settings.get("menu"),
});
- const safe_download_url = `${options.root}${dataset.download_url}`;
- msaViz.u.file.importURL(safe_download_url, () => {
+ msaViz.u.file.importURL(prefixedDownloadUrl(options.root, dataset.download_url), () => {
msaViz.render();
chart.state("ok", "Chart drawn.");
options.process.resolve();
diff --git a/config/plugins/visualizations/ngl/src/ngl.js b/config/plugins/visualizations/ngl/src/ngl.js
index b9d92f73bec..5d9ee9cfd6f 100644
--- a/config/plugins/visualizations/ngl/src/ngl.js
+++ b/config/plugins/visualizations/ngl/src/ngl.js
@@ -1,31 +1,37 @@
import * as ngl from "./viewer";
/** Get boolean as string */
-function asBoolean (value) {
+function asBoolean(value) {
return String(value).toLowerCase() == "true";
}
+/* This will be part of the charts/viz standard lib in 23.1 */
+const slashCleanup = /(\/)+/g;
+function prefixedDownloadUrl(root, path) {
+ return `${root}/${path}`.replace(slashCleanup, "/");
+}
+
window.bundleEntries = window.bundleEntries || {};
window.bundleEntries.load = function (options) {
- var dataset = options.dataset,
- settings = options.chart.settings,
- stage = new ngl.Stage(options.target, { backgroundColor: settings.get("backcolor") }),
- representation_parameters = {},
- stage_parameters = {};
- representation_parameters = {
+ const dataset = options.dataset;
+ const settings = options.chart.settings;
+ const stage = new ngl.Stage(options.target, { backgroundColor: settings.get("backcolor") });
+ const representationParameters = {
radius: settings.get("radius"),
assembly: settings.get("assembly"),
color: settings.get("colorscheme"),
opacity: settings.get("opacity"),
};
- stage_parameters = { ext: dataset.extension, defaultRepresentation: true };
- const safe_download_url = `${options.root}${dataset.download_url}`;
+ const stageParameters = { ext: dataset.extension, defaultRepresentation: true };
+
try {
- stage.loadFile(safe_download_url, stage_parameters).then(function (component) {
- component.addRepresentation(settings.get("mode"), representation_parameters);
- options.chart.state("ok", "Chart drawn.");
- options.process.resolve();
- });
+ stage
+ .loadFile(prefixedDownloadUrl(options.root, dataset.download_url), stageParameters)
+ .then(function (component) {
+ component.addRepresentation(settings.get("mode"), representationParameters);
+ options.chart.state("ok", "Chart drawn.");
+ options.process.resolve();
+ });
} catch (e) {
options.chart.state("failed", "Could not load PDB file.");
options.process.resolve();
diff --git a/config/plugins/visualizations/openlayers/src/script.js b/config/plugins/visualizations/openlayers/src/script.js
index b4ca4af179f..90922403fa7 100644
--- a/config/plugins/visualizations/openlayers/src/script.js
+++ b/config/plugins/visualizations/openlayers/src/script.js
@@ -13,6 +13,12 @@ import { saveAs } from "file-saver";
import shp from "shpjs";
import axios from "axios";
+/* This will be part of the charts/viz standard lib in 23.1 */
+const slashCleanup = /(\/)+/g;
+function prefixedDownloadUrl(root, path) {
+ return `${root}/${path}`.replace(slashCleanup, "/");
+}
+
var MapViewer = (function(mv) {
mv.gMap = null;
@@ -240,11 +246,11 @@ window.bundleEntries = window.bundleEntries || {};
window.bundleEntries.load = function (options) {
const chart = options.chart;
const dataset = options.dataset;
- const safe_download_url = `${options.root}${dataset.download_url}`;
+ const downloadUrl = prefixedDownloadUrl(options.root, dataset.download_url);
$.ajax({
- url: safe_download_url,
+ url: downloadUrl,
success: () => {
- MapViewer.loadFile(safe_download_url, dataset.extension, options, chart);
+ MapViewer.loadFile(downloadUrl, dataset.extension, options, chart);
},
error: () => {
chart.state("failed", "Failed to access dataset.");
diff --git a/config/plugins/visualizations/phylocanvas/src/script.js b/config/plugins/visualizations/phylocanvas/src/script.js
index d437bfe6942..f2b3c6a60e3 100644
--- a/config/plugins/visualizations/phylocanvas/src/script.js
+++ b/config/plugins/visualizations/phylocanvas/src/script.js
@@ -1,12 +1,18 @@
import * as Phylocanvas from "phylocanvas";
+
+/* This will be part of the charts/viz standard lib in 23.1 */
+const slashCleanup = /(\/)+/g;
+function prefixedDownloadUrl(root, path) {
+ return `${root}/${path}`.replace(slashCleanup, "/");
+}
+
_.extend(window.bundleEntries || {}, {
load: function(options) {
var chart = options.chart;
var dataset = options.dataset;
var settings = options.chart.settings;
- const safe_download_url = `${options.root}${dataset.download_url}`;
$.ajax( {
- url : safe_download_url,
+ url : prefixedDownloadUrl(options.root, dataset.download_url),
success : function( content ) {
try {
var tree = Phylocanvas.default.createTree( options.target ),
diff --git a/config/plugins/visualizations/pv/src/pv.js b/config/plugins/visualizations/pv/src/pv.js
index 14a9bdf7db3..6dc41a292a5 100644
--- a/config/plugins/visualizations/pv/src/pv.js
+++ b/config/plugins/visualizations/pv/src/pv.js
@@ -1,5 +1,11 @@
import * as pv from "bio-pv";
+/* This will be part of the charts/viz standard lib in 23.1 */
+const slashCleanup = /(\/)+/g;
+function prefixedDownloadUrl(root, path) {
+ return `${root}/${path}`.replace(slashCleanup, "/");
+}
+
window.bundleEntries = window.bundleEntries || {};
window.bundleEntries.load = function (options) {
var settings = options.chart.settings;
@@ -10,9 +16,8 @@ window.bundleEntries.load = function (options) {
antialias: true,
outline: true,
});
- const safe_download_url = `${options.root}${dataset.download_url}`;
var xhr = new XMLHttpRequest();
- xhr.open("GET", safe_download_url);
+ xhr.open("GET", prefixedDownloadUrl(options.root, options.dataset.download_url));
xhr.onload = function () {
if (xhr.status === 200) {
var structure = pv.io.pdb(xhr.response);
diff --git a/doc/source/admin/galaxy_options.rst b/doc/source/admin/galaxy_options.rst
index 7253a266130..f36e3162429 100644
--- a/doc/source/admin/galaxy_options.rst
+++ b/doc/source/admin/galaxy_options.rst
@@ -5077,3 +5077,13 @@
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+``enable_beacon_integration``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+:Description:
+ Enables user preferences and api endpoint for the beacon integration.
+:Default: ``false``
+:Type: bool
+
+
diff --git a/doc/source/dev/run_tests_help.txt b/doc/source/dev/run_tests_help.txt
index 4e002913339..5e646326027 100644
--- a/doc/source/dev/run_tests_help.txt
+++ b/doc/source/dev/run_tests_help.txt
@@ -75,8 +75,8 @@ Run a specific selenium test (under Linux or Mac OS X after installing geckodriv
Run a selenium test against a running server while watching client (fastest iterating on client tests):
./run.sh & # run Galaxy on 8080
- make client-watch & # watch for client changes
- export GALAXY_TEST_EXTERNAL=http://localhost:8080/ # Target tests at server.
+ make client-dev-server & # watch for client changes
+ export GALAXY_TEST_EXTERNAL=http://localhost:8081/ # Target tests at server.
. .venv/bin/activate # source the virtualenv so can skip run_tests.sh.
pytest lib/galaxy_test/selenium/test_workflow_editor.py::TestWorkflowEditor::test_data_input
diff --git a/lib/galaxy/config/sample/datatypes_conf.xml.sample b/lib/galaxy/config/sample/datatypes_conf.xml.sample
index 2efd24f4af4..49aab75ddbb 100644
--- a/lib/galaxy/config/sample/datatypes_conf.xml.sample
+++ b/lib/galaxy/config/sample/datatypes_conf.xml.sample
@@ -418,7 +418,6 @@
@@ -445,6 +444,7 @@
+
+
+
+
+
+
- pysam
- coreutils
+ coreutils
+ htslib
'$output1'
]]>
-
+
@@ -29,27 +40,55 @@
-
+
+
+
+
+
-
+
+
+
-
+
+
+
-
+
+
+
@@ -58,6 +97,10 @@
+
diff --git a/lib/galaxy/datatypes/converters/interval_to_tabix_converter.py b/lib/galaxy/datatypes/converters/interval_to_tabix_converter.py
deleted file mode 100644
index 29e66b518a0..00000000000
--- a/lib/galaxy/datatypes/converters/interval_to_tabix_converter.py
+++ /dev/null
@@ -1,59 +0,0 @@
-#!/usr/bin/env python
-
-"""
-Uses pysam to index a bgzipped interval file with tabix
-Supported presets: bed, gff, vcf
-
-usage: %prog in_file out_file
-"""
-import optparse
-import os
-import sys
-
-import pysam
-
-
-def main():
- # Read options, args.
- parser = optparse.OptionParser()
- parser.add_option("-c", "--chr-col", type="int", dest="chrom_col")
- parser.add_option("-s", "--start-col", type="int", dest="start_col")
- parser.add_option("-e", "--end-col", type="int", dest="end_col")
- parser.add_option("-P", "--preset", dest="preset")
- (options, args) = parser.parse_args()
- _, bgzip_fname, out_fname = args
- to_tabix(
- bgzip_fname=bgzip_fname,
- out_fname=out_fname,
- preset=options.preset,
- chrom_col=options.chrom_col,
- start_col=options.start_col,
- end_col=options.end_col,
- )
-
-
-def to_tabix(bgzip_fname, out_fname, preset=None, chrom_col=None, start_col=None, end_col=None):
- # Create index.
- if preset:
- # Preset type.
- bgzip_fname = pysam.tabix_index(
- filename=bgzip_fname, preset=preset, keep_original=True, index=out_fname, force=True
- )
- else:
- # For interval files; column indices are 0-based.
- bgzip_fname = pysam.tabix_index(
- filename=bgzip_fname,
- seq_col=(chrom_col - 1),
- start_col=(start_col - 1),
- end_col=(end_col - 1),
- keep_original=True,
- index=out_fname,
- force=True,
- )
- if os.path.getsize(out_fname) == 0:
- sys.exit("The converted tabix index file is empty, meaning the input data is invalid.")
- return bgzip_fname
-
-
-if __name__ == "__main__":
- main()
diff --git a/lib/galaxy/datatypes/converters/interval_to_tabix_converter.xml b/lib/galaxy/datatypes/converters/interval_to_tabix_converter.xml
index 54dbd1ad28e..2f94c32737c 100644
--- a/lib/galaxy/datatypes/converters/interval_to_tabix_converter.xml
+++ b/lib/galaxy/datatypes/converters/interval_to_tabix_converter.xml
@@ -1,18 +1,31 @@
-
+
- pysam
+ htslib
&2 echo "The converted tabix index file is empty, meaning the input data is invalid.";
+ exit 1;
+ fi
+ &&
+ tabix -l input.bgz
]]>
@@ -23,12 +36,22 @@
-
+
-
+
+
+
+
+
@@ -37,19 +60,66 @@
+
+
+
-
+
+
+
+
+
-
+
+
+
+
+
-
+
+
+
+
+
+
+
+
+
+
+
diff --git a/lib/galaxy/datatypes/converters/tabular_to_dbnsfp.py b/lib/galaxy/datatypes/converters/tabular_to_dbnsfp.py
deleted file mode 100644
index a8c7c06ee5a..00000000000
--- a/lib/galaxy/datatypes/converters/tabular_to_dbnsfp.py
+++ /dev/null
@@ -1,35 +0,0 @@
-#!/usr/bin/env python
-"""
-Uses pysam to bgzip a file
-
-usage: %prog in_file out_file
-"""
-
-import optparse
-import os.path
-
-import pysam
-
-
-def main():
- # Read options, args.
- usage = "Usage: %prog [options] tabular_input_file bgzip_output_file"
- parser = optparse.OptionParser(usage=usage)
- parser.add_option("-c", "--chr-col", type="int", default=0, dest="chrom_col")
- parser.add_option("-s", "--start-col", type="int", default=1, dest="start_col")
- parser.add_option("-e", "--end-col", type="int", default=1, dest="end_col")
- (options, args) = parser.parse_args()
- if len(args) != 2:
- parser.print_usage()
- exit(1)
- input_fname, output_fname = args
- output_dir = os.path.dirname(output_fname)
- if not os.path.exists(output_dir):
- os.makedirs(output_dir)
- pysam.tabix_compress(input_fname, output_fname, force=True)
- # Column indices are 0-based.
- pysam.tabix_index(output_fname, seq_col=options.chrom_col, start_col=options.start_col, end_col=options.end_col)
-
-
-if __name__ == "__main__":
- main()
diff --git a/lib/galaxy/datatypes/converters/tabular_to_dbnsfp.xml b/lib/galaxy/datatypes/converters/tabular_to_dbnsfp.xml
index a0b7c255736..7727d107dc7 100644
--- a/lib/galaxy/datatypes/converters/tabular_to_dbnsfp.xml
+++ b/lib/galaxy/datatypes/converters/tabular_to_dbnsfp.xml
@@ -1,9 +1,13 @@
-
+
- pysam
+ htslib
- python '$__tool_directory__/tabular_to_dbnsfp.py' '$input' '$dbnsfp.extra_files_path/dbNSFP.gz'
+ '$dbnsfp.extra_files_path/dbNSFP.gz' &&
+ tabix -s 1 -b 2 -e 2 '$dbnsfp.extra_files_path/dbNSFP.gz'
+ ]]>
diff --git a/lib/galaxy/datatypes/converters/vcf_bgzip_to_tabix_converter.xml b/lib/galaxy/datatypes/converters/vcf_bgzip_to_tabix_converter.xml
index 0d459aa011e..b6bdcd0c53a 100644
--- a/lib/galaxy/datatypes/converters/vcf_bgzip_to_tabix_converter.xml
+++ b/lib/galaxy/datatypes/converters/vcf_bgzip_to_tabix_converter.xml
@@ -1,9 +1,14 @@
-
+
- pysam
+ htslib
- python '$__tool_directory__/interval_to_tabix_converter.py' -P 'vcf' '' '$input1' '$output1'
+
@@ -13,7 +18,14 @@
-
+
+
+
diff --git a/lib/galaxy/datatypes/converters/vcf_to_vcf_bgzip.py b/lib/galaxy/datatypes/converters/vcf_to_vcf_bgzip.py
deleted file mode 100644
index 91ca2fe88bc..00000000000
--- a/lib/galaxy/datatypes/converters/vcf_to_vcf_bgzip.py
+++ /dev/null
@@ -1,25 +0,0 @@
-#!/usr/bin/env python
-
-"""
-Uses pysam to bgzip a vcf file as-is.
-Headers, which are important, are kept.
-Original ordering, which may be specifically needed by tools or external display applications, is also maintained.
-
-usage: %prog in_file out_file
-"""
-import optparse
-
-import pysam
-
-
-def main():
- # Read options, args.
- parser = optparse.OptionParser()
- (options, args) = parser.parse_args()
- input_fname, output_fname = args
-
- pysam.tabix_compress(input_fname, output_fname, force=True)
-
-
-if __name__ == "__main__":
- main()
diff --git a/lib/galaxy/datatypes/converters/vcf_to_vcf_bgzip_converter.xml b/lib/galaxy/datatypes/converters/vcf_to_vcf_bgzip_converter.xml
index b680b973a48..41617707e2b 100644
--- a/lib/galaxy/datatypes/converters/vcf_to_vcf_bgzip_converter.xml
+++ b/lib/galaxy/datatypes/converters/vcf_to_vcf_bgzip_converter.xml
@@ -1,10 +1,11 @@
-
+
- pysam
- coreutils
+ htslib
- python '$__tool_directory__/vcf_to_vcf_bgzip.py' '$input1' '$output1'
+ '$output1'
+ ]]>
@@ -14,7 +15,11 @@
-
+
diff --git a/lib/galaxy/datatypes/data.py b/lib/galaxy/datatypes/data.py
index d8ff7f93e53..636e5c398bc 100644
--- a/lib/galaxy/datatypes/data.py
+++ b/lib/galaxy/datatypes/data.py
@@ -20,12 +20,21 @@ from typing import (
)
from markupsafe import escape
-from typing_extensions import (
- Literal,
- Protocol,
-)
+from typing_extensions import Literal
from galaxy import util
+from galaxy.datatypes._protocols import (
+ DatasetHasHidProtocol,
+ DatasetProtocol,
+ HasClearAssociatedFiles,
+ HasCreatingJob,
+ HasExt,
+ HasExtraFilesAndMetadata,
+ HasFileName,
+ HasInfo,
+ HasMetadata,
+ HasName,
+)
from galaxy.datatypes.metadata import (
MetadataElement, # import directly to maintain ease of use in Datatype class definitions
)
@@ -58,11 +67,6 @@ from . import (
if TYPE_CHECKING:
from galaxy.datatypes.display_applications.application import DisplayApplication
from galaxy.datatypes.registry import Registry
- from galaxy.model import (
- DatasetInstance,
- HistoryDatasetAssociation,
- HistoryDatasetCollectionAssociation,
- )
XSS_VULNERABLE_MIME_TYPES = [
"image/svg+xml", # Unfiltered by Galaxy and may contain JS that would be executed by some browsers.
@@ -83,11 +87,6 @@ DEFAULT_MAX_PEEK_SIZE = 1000000 # 1 MB
Headers = Dict[str, Any]
-class GeneratePrimaryFileDataset(Protocol):
- extra_files_path: str
- metadata: Any
-
-
class DatatypeConverterNotFoundException(Exception):
pass
@@ -113,7 +112,7 @@ class DatatypeValidation:
return f"DatatypeValidation[state={self.state},message={self.message}]"
-def validate(dataset_instance: "DatasetInstance") -> DatatypeValidation:
+def validate(dataset_instance: DatasetProtocol) -> DatatypeValidation:
try:
datatype_validation = dataset_instance.datatype.validate(dataset_instance)
except Exception as e:
@@ -237,7 +236,7 @@ class Data(metaclass=DataMeta):
def groom_dataset_content(self, file_name: str) -> None:
"""This function is called on an output dataset file if dataset_content_needs_grooming returns True."""
- def init_meta(self, dataset: "DatasetInstance", copy_from: Optional["DatasetInstance"] = None) -> None:
+ def init_meta(self, dataset: HasMetadata, copy_from: Optional[HasMetadata] = None) -> None:
# Metadata should be left mostly uninitialized. Dataset will
# handle returning default values when metadata is not set.
# copy_from allows metadata to be passed in that will be
@@ -247,12 +246,10 @@ class Data(metaclass=DataMeta):
if copy_from:
dataset.metadata = copy_from.metadata
- def set_meta(self, dataset: "DatasetInstance", *, overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, *, overwrite: bool = True, **kwd) -> None:
"""Unimplemented method, allows guessing of metadata from contents of file"""
- def missing_meta(
- self, dataset: "DatasetInstance", check: Optional[List] = None, skip: Optional[List] = None
- ) -> bool:
+ def missing_meta(self, dataset: HasMetadata, check: Optional[List] = None, skip: Optional[List] = None) -> bool:
"""
Checks for empty metadata values.
Returns False if no non-optional metadata is missing and the missing metadata key otherwise.
@@ -294,7 +291,7 @@ class Data(metaclass=DataMeta):
max_optional_metadata_filesize = property(get_max_optional_metadata_filesize, set_max_optional_metadata_filesize)
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
"""
Set the peek and blurb text
"""
@@ -305,7 +302,7 @@ class Data(metaclass=DataMeta):
dataset.peek = "file does not exist"
dataset.blurb = "file purged from disk"
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
"""Create HTML table, used for displaying peek"""
out = ['']
try:
@@ -346,7 +343,7 @@ class Data(metaclass=DataMeta):
return error, msg, messagetype
def _archive_composite_dataset(
- self, trans, data: "DatasetInstance", headers: Headers, do_action: str = "zip"
+ self, trans, data: DatasetHasHidProtocol, headers: Headers, do_action: str = "zip"
) -> Tuple[Union[ZipstreamWrapper, str], Headers]:
# save a composite object into a compressed archive for downloading
outfname = data.name[0:150]
@@ -392,7 +389,7 @@ class Data(metaclass=DataMeta):
yield fpath, rpath
def _serve_raw(
- self, dataset: "HistoryDatasetAssociation", to_ext: Optional[str], headers: Headers, **kwd
+ self, dataset: DatasetHasHidProtocol, to_ext: Optional[str], headers: Headers, **kwd
) -> Tuple[IO, Headers]:
headers["Content-Length"] = str(os.stat(dataset.file_name).st_size)
headers[
@@ -408,7 +405,7 @@ class Data(metaclass=DataMeta):
headers["Content-Disposition"] = f'attachment; filename="{filename}"'
return open(dataset.file_name, mode="rb"), headers
- def to_archive(self, dataset: "DatasetInstance", name: str = "") -> Iterable:
+ def to_archive(self, dataset: DatasetProtocol, name: str = "") -> Iterable:
"""
Collect archive paths and file handles that need to be exported when archiving `dataset`.
@@ -433,7 +430,7 @@ class Data(metaclass=DataMeta):
def display_data(
self,
trans,
- dataset: "HistoryDatasetAssociation",
+ dataset: DatasetHasHidProtocol,
preview: bool = False,
filename: Optional[str] = None,
to_ext: Optional[str] = None,
@@ -547,7 +544,7 @@ class Data(metaclass=DataMeta):
headers,
)
- def display_as_markdown(self, dataset_instance: "DatasetInstance") -> str:
+ def display_as_markdown(self, dataset_instance: DatasetProtocol) -> str:
"""Prepare for embedding dataset into a basic Markdown document.
This is a somewhat experimental interface and should not be implemented
@@ -575,7 +572,7 @@ class Data(metaclass=DataMeta):
result += indicate_data_truncated()
return result
- def _yield_user_file_content(self, trans, from_dataset: "DatasetInstance", filename: str, headers: Headers) -> IO:
+ def _yield_user_file_content(self, trans, from_dataset: HasCreatingJob, filename: str, headers: Headers) -> IO:
"""This method is responsible for sanitizing the HTML if needed."""
if trans.app.config.sanitize_all_html and headers.get("content-type", None) == "text/html":
# Sanitize anytime we respond with plain text/html content.
@@ -596,9 +593,9 @@ class Data(metaclass=DataMeta):
def _download_filename(
self,
- dataset: "HistoryDatasetAssociation",
+ dataset: DatasetHasHidProtocol,
to_ext: Optional[str] = None,
- hdca: Optional["HistoryDatasetCollectionAssociation"] = None,
+ hdca: Optional[DatasetHasHidProtocol] = None,
element_identifier: Optional[str] = None,
filename_pattern: Optional[str] = None,
) -> str:
@@ -630,14 +627,14 @@ class Data(metaclass=DataMeta):
return string.Template(filename_pattern).substitute(**template_values)
- def display_name(self, dataset: "DatasetInstance") -> str:
+ def display_name(self, dataset: HasName) -> str:
"""Returns formatted html of dataset name"""
try:
return escape(unicodify(dataset.name, "utf-8"))
except Exception:
return "name unavailable"
- def display_info(self, dataset: "DatasetInstance") -> str:
+ def display_info(self, dataset: HasInfo) -> str:
"""Returns formatted html of dataset info"""
try:
# Change new line chars to html
@@ -697,7 +694,7 @@ class Data(metaclass=DataMeta):
def get_display_application(self, key: str, default: Optional["DisplayApplication"] = None) -> "DisplayApplication":
return self.display_applications.get(key, default)
- def get_display_applications_by_dataset(self, dataset: "DatasetInstance", trans) -> Dict[str, "DisplayApplication"]:
+ def get_display_applications_by_dataset(self, dataset: DatasetProtocol, trans) -> Dict[str, "DisplayApplication"]:
rval = {}
for key, value in self.display_applications.items():
value = value.filter_by_dataset(dataset, trans)
@@ -716,7 +713,7 @@ class Data(metaclass=DataMeta):
except Exception:
return "unknown"
- def as_display_type(self, dataset: "DatasetInstance", type: str, **kwd) -> Union[FileObjType, str]:
+ def as_display_type(self, dataset: DatasetProtocol, type: str, **kwd) -> Union[FileObjType, str]:
"""Returns modified file contents for a particular display type"""
try:
if type in self.get_display_types():
@@ -731,7 +728,7 @@ class Data(metaclass=DataMeta):
return f"This display type ({type}) is not implemented for this datatype ({dataset.ext})."
def get_display_links(
- self, dataset: "DatasetInstance", type: str, app, base_url: str, target_frame: str = "_blank", **kwd
+ self, dataset: DatasetProtocol, type: str, app, base_url: str, target_frame: str = "_blank", **kwd
):
"""
Returns a list of tuples of (name, link) for a particular display type. No check on
@@ -753,15 +750,13 @@ class Data(metaclass=DataMeta):
)
return target_frame, []
- def get_converter_types(
- self, original_dataset: "DatasetInstance", datatypes_registry: "Registry"
- ) -> Dict[str, Dict]:
+ def get_converter_types(self, original_dataset: HasExt, datatypes_registry: "Registry") -> Dict[str, Dict]:
"""Returns available converters by type for this dataset"""
return datatypes_registry.get_converters_by_datatype(original_dataset.ext)
def find_conversion_destination(
- self, dataset, accepted_formats: List[str], datatypes_registry, **kwd
- ) -> Tuple[bool, Optional[str], Optional["DatasetInstance"]]:
+ self, dataset: DatasetProtocol, accepted_formats: List[str], datatypes_registry, **kwd
+ ) -> Tuple[bool, Optional[str], Any]:
"""Returns ( direct_match, converted_ext, existing converted dataset )"""
return datatypes_registry.find_conversion_destination_for_dataset_by_extensions(
dataset, accepted_formats, **kwd
@@ -770,7 +765,7 @@ class Data(metaclass=DataMeta):
def convert_dataset(
self,
trans,
- original_dataset: "HistoryDatasetAssociation",
+ original_dataset: DatasetHasHidProtocol,
target_type: str,
return_output: bool = False,
visible: bool = True,
@@ -806,11 +801,11 @@ class Data(metaclass=DataMeta):
# We need to clear associated files before we set metadata
# so that as soon as metadata starts to be set, e.g. implicitly converted datasets are deleted and no longer available 'while' metadata is being set, not just after
# We'll also clear after setting metadata, for backwards compatibility
- def after_setting_metadata(self, dataset: "DatasetInstance") -> None:
+ def after_setting_metadata(self, dataset: HasClearAssociatedFiles) -> None:
"""This function is called on the dataset after metadata is set."""
dataset.clear_associated_files(metadata_safe=True)
- def before_setting_metadata(self, dataset: "DatasetInstance") -> None:
+ def before_setting_metadata(self, dataset: HasClearAssociatedFiles) -> None:
"""This function is called on the dataset before metadata is set."""
dataset.clear_associated_files(metadata_safe=True)
@@ -851,7 +846,7 @@ class Data(metaclass=DataMeta):
files[key] = value
return files
- def get_writable_files_for_dataset(self, dataset: "DatasetInstance") -> Dict:
+ def get_writable_files_for_dataset(self, dataset: Optional[HasMetadata]) -> Dict:
files = {}
if self.composite_type != "auto_primary_file":
files[self.primary_file_name] = self.__new_composite_file(self.primary_file_name)
@@ -859,7 +854,7 @@ class Data(metaclass=DataMeta):
files[key] = value
return files
- def get_composite_files(self, dataset=None):
+ def get_composite_files(self, dataset: Optional[HasMetadata] = None):
def substitute_composite_key(key, composite_file):
if composite_file.substitute_name_with_metadata:
if dataset:
@@ -874,7 +869,7 @@ class Data(metaclass=DataMeta):
files[substitute_composite_key(key, value)] = value
return files
- def generate_primary_file(self, dataset: GeneratePrimaryFileDataset) -> str:
+ def generate_primary_file(self, dataset: HasExtraFilesAndMetadata) -> str:
raise Exception("generate_primary_file is not implemented for this datatype.")
@property
@@ -911,7 +906,7 @@ class Data(metaclass=DataMeta):
"""
return data_format in self.dataproviders
- def dataprovider(self, dataset: "DatasetInstance", data_format: str, **settings):
+ def dataprovider(self, dataset: DatasetProtocol, data_format: str, **settings):
"""
Base dataprovider factory for all datatypes that returns the proper provider
for the given `data_format` or raises a `NoProviderAvailable`.
@@ -920,22 +915,22 @@ class Data(metaclass=DataMeta):
return self.dataproviders[data_format](self, dataset, **settings)
raise p_dataproviders.exceptions.NoProviderAvailable(self, data_format)
- def validate(self, dataset: "DatasetInstance", **kwd) -> DatatypeValidation:
+ def validate(self, dataset: DatasetProtocol, **kwd) -> DatatypeValidation:
return DatatypeValidation.unvalidated()
@p_dataproviders.decorators.dataprovider_factory("base")
- def base_dataprovider(self, dataset: "DatasetInstance", **settings) -> p_dataproviders.base.DataProvider:
+ def base_dataprovider(self, dataset: DatasetProtocol, **settings) -> p_dataproviders.base.DataProvider:
dataset_source = p_dataproviders.dataset.DatasetDataProvider(dataset)
return p_dataproviders.base.DataProvider(dataset_source, **settings)
@p_dataproviders.decorators.dataprovider_factory("chunk", p_dataproviders.chunk.ChunkDataProvider.settings)
- def chunk_dataprovider(self, dataset: "DatasetInstance", **settings) -> p_dataproviders.chunk.ChunkDataProvider:
+ def chunk_dataprovider(self, dataset: DatasetProtocol, **settings) -> p_dataproviders.chunk.ChunkDataProvider:
dataset_source = p_dataproviders.dataset.DatasetDataProvider(dataset)
return p_dataproviders.chunk.ChunkDataProvider(dataset_source, **settings)
@p_dataproviders.decorators.dataprovider_factory("chunk64", p_dataproviders.chunk.Base64ChunkDataProvider.settings)
def chunk64_dataprovider(
- self, dataset: "DatasetInstance", **settings
+ self, dataset: DatasetProtocol, **settings
) -> p_dataproviders.chunk.Base64ChunkDataProvider:
dataset_source = p_dataproviders.dataset.DatasetDataProvider(dataset)
return p_dataproviders.chunk.Base64ChunkDataProvider(dataset_source, **settings)
@@ -946,7 +941,7 @@ class Data(metaclass=DataMeta):
mime = DEFAULT_MIME_TYPE
headers["content-type"] = mime
- def handle_dataset_as_image(self, hda: "DatasetInstance") -> str:
+ def handle_dataset_as_image(self, hda: DatasetProtocol) -> str:
raise Exception("Unimplemented Method")
def __getstate__(self) -> Dict[str, Any]:
@@ -978,13 +973,13 @@ class Text(Data):
"""Returns the mime type of the datatype"""
return "text/plain"
- def set_meta(self, dataset: "DatasetInstance", *, overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, *, overwrite: bool = True, **kwd) -> None:
"""
Set the number of lines of data in dataset.
"""
dataset.metadata.data_lines = self.count_data_lines(dataset)
- def estimate_file_lines(self, dataset: "DatasetInstance") -> Optional[int]:
+ def estimate_file_lines(self, dataset: DatasetProtocol) -> Optional[int]:
"""
Perform a rough estimate by extrapolating number of lines from a small read.
"""
@@ -998,7 +993,7 @@ class Text(Data):
log.error(f"Unable to estimate lines in file {dataset.file_name}")
return None
- def count_data_lines(self, dataset: "DatasetInstance") -> Optional[int]:
+ def count_data_lines(self, dataset: HasFileName) -> Optional[int]:
"""
Count the number of lines of data in dataset,
skipping all blank lines and comments.
@@ -1019,7 +1014,7 @@ class Text(Data):
return None
return data_lines
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
"""
Set the peek. This method is used by various subclasses of Text.
"""
@@ -1132,9 +1127,7 @@ class Text(Data):
# ------------- Dataproviders
@p_dataproviders.decorators.dataprovider_factory("line", p_dataproviders.line.FilteredLineDataProvider.settings)
- def line_dataprovider(
- self, dataset: "DatasetInstance", **settings
- ) -> p_dataproviders.line.FilteredLineDataProvider:
+ def line_dataprovider(self, dataset: DatasetProtocol, **settings) -> p_dataproviders.line.FilteredLineDataProvider:
"""
Returns an iterator over the dataset's lines (that have been stripped)
optionally excluding blank lines and lines that start with a comment character.
@@ -1144,7 +1137,7 @@ class Text(Data):
@p_dataproviders.decorators.dataprovider_factory("regex-line", p_dataproviders.line.RegexLineDataProvider.settings)
def regex_line_dataprovider(
- self, dataset: "DatasetInstance", **settings
+ self, dataset: DatasetProtocol, **settings
) -> p_dataproviders.line.RegexLineDataProvider:
"""
Returns an iterator over the dataset's lines
diff --git a/lib/galaxy/datatypes/flow.py b/lib/galaxy/datatypes/flow.py
index e520be5a1ea..6f43d8316cf 100644
--- a/lib/galaxy/datatypes/flow.py
+++ b/lib/galaxy/datatypes/flow.py
@@ -3,8 +3,8 @@ Flow analysis datatypes.
"""
import logging
-from typing import TYPE_CHECKING
+from galaxy.datatypes._protocols import DatasetProtocol
from galaxy.datatypes.binary import Binary
from galaxy.datatypes.sniff import (
build_sniff_from_prefix,
@@ -12,9 +12,6 @@ from galaxy.datatypes.sniff import (
)
from . import data
-if TYPE_CHECKING:
- from galaxy.model import DatasetInstance
-
log = logging.getLogger(__name__)
@@ -24,7 +21,7 @@ class FCS(Binary):
file_ext = "fcs"
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
dataset.peek = "Binary FCS file"
dataset.blurb = data.nice_size(dataset.get_size())
@@ -32,7 +29,7 @@ class FCS(Binary):
dataset.peek = "file does not exist"
dataset.blurb = "file purged from disk"
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
try:
return dataset.peek
except Exception:
diff --git a/lib/galaxy/datatypes/genetics.py b/lib/galaxy/datatypes/genetics.py
index 1ebb8398cb6..bfb2ad2ac14 100644
--- a/lib/galaxy/datatypes/genetics.py
+++ b/lib/galaxy/datatypes/genetics.py
@@ -20,7 +20,6 @@ from typing import (
IO,
List,
Optional,
- TYPE_CHECKING,
Union,
)
from urllib.parse import quote_plus
@@ -28,9 +27,13 @@ from urllib.parse import quote_plus
from markupsafe import escape
from galaxy.datatypes import metadata
+from galaxy.datatypes._protocols import (
+ DatasetProtocol,
+ HasExtraFilesAndMetadata,
+ HasMetadata,
+)
from galaxy.datatypes.data import (
DatatypeValidation,
- GeneratePrimaryFileDataset,
Text,
)
from galaxy.datatypes.metadata import MetadataElement
@@ -46,9 +49,6 @@ from galaxy.util import (
)
from galaxy.util.compression_utils import FileObjType
-if TYPE_CHECKING:
- from galaxy.model import DatasetInstance
-
gal_Log = logging.getLogger(__name__)
verbose = False
@@ -75,7 +75,7 @@ class GenomeGraphs(Tabular):
super().__init__(**kwd)
self.add_display_app("ucsc", "Genome Graph", "as_ucsc_display_file", "ucsc_links")
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
super().set_meta(dataset, overwrite=overwrite, **kwd)
dataset.metadata.markerCol = 1
header = open(dataset.file_name).readlines()[0].strip().split("\t")
@@ -84,13 +84,13 @@ class GenomeGraphs(Tabular):
t[0] = "string"
dataset.metadata.column_types = t
- def as_ucsc_display_file(self, dataset: "DatasetInstance", **kwd) -> Union[FileObjType, str]:
+ def as_ucsc_display_file(self, dataset: DatasetProtocol, **kwd) -> Union[FileObjType, str]:
"""
Returns file
"""
return open(dataset.file_name, "rb")
- def ucsc_links(self, dataset: "DatasetInstance", type: str, app, base_url: str) -> List:
+ def ucsc_links(self, dataset: DatasetProtocol, type: str, app, base_url: str) -> List:
"""
from the ever-helpful angie hinrichs angie@soe.ucsc.edu
a genome graphs call looks like this
@@ -142,7 +142,7 @@ class GenomeGraphs(Tabular):
ret_val.append((site_name, link))
return ret_val
- def make_html_table(self, dataset: "DatasetInstance", **kwargs) -> str:
+ def make_html_table(self, dataset: DatasetProtocol, **kwargs) -> str:
"""
Create HTML table, used for displaying peek
"""
@@ -173,7 +173,7 @@ class GenomeGraphs(Tabular):
except Exception as exc:
return f"Can't create peek {exc}"
- def validate(self, dataset: "DatasetInstance", **kwd) -> DatatypeValidation:
+ def validate(self, dataset: DatasetProtocol, **kwd) -> DatatypeValidation:
"""
Validate a gg file - all numeric after header row
"""
@@ -237,7 +237,7 @@ class rgTabList(Tabular):
super().__init__(**kwd)
self.column_names = []
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
"""Returns formated html of peek"""
return self.make_html_table(dataset, column_names=self.column_names)
@@ -302,7 +302,7 @@ class Rgenetics(Html):
composite_type = "auto_primary_file"
file_ext = "rgenetics"
- def generate_primary_file(self, dataset: GeneratePrimaryFileDataset) -> str:
+ def generate_primary_file(self, dataset: HasExtraFilesAndMetadata) -> str:
rval = ["Rgenetics Galaxy Composite Dataset "]
rval.append("This composite dataset is composed of the following files:")
for composite_name, composite_file in self.get_composite_files(dataset=dataset).items():
@@ -319,7 +319,7 @@ class Rgenetics(Html):
rval.append("
")
return "\n".join(rval)
- def regenerate_primary_file(self, dataset: "DatasetInstance") -> None:
+ def regenerate_primary_file(self, dataset: DatasetProtocol) -> None:
"""
cannot do this until we are setting metadata
"""
@@ -341,7 +341,7 @@ class Rgenetics(Html):
"""Returns the mime type of the datatype"""
return "text/html"
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""
for lped/pbed eg
@@ -381,7 +381,7 @@ class SNPMatrix(Rgenetics):
file_ext = "snpmatrix"
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
dataset.peek = "Binary RGenetics file"
dataset.blurb = nice_size(dataset.get_size())
@@ -577,7 +577,7 @@ class IdeasPre(Html):
self.add_composite_file("IDEAS_input_config.txt", description="IDEAS input config", is_binary=False)
self.add_composite_file("tmp.tar.gz", description="Compressed archive of compressed bed files", is_binary=True)
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
super().set_meta(dataset, overwrite=overwrite, **kwd)
for fname in os.listdir(dataset.extra_files_path):
if fname.startswith("chromosomes"):
@@ -590,7 +590,7 @@ class IdeasPre(Html):
dataset.metadata.tmp_archive = os.path.join(dataset.extra_files_path, fname)
self.regenerate_primary_file(dataset)
- def generate_primary_file(self, dataset: GeneratePrimaryFileDataset) -> str:
+ def generate_primary_file(self, dataset: HasExtraFilesAndMetadata) -> str:
rval = [""]
rval.append("Files prepared for IDEAS
")
rval.append("")
@@ -600,7 +600,7 @@ class IdeasPre(Html):
rval.append("
\n")
return "\n".join(rval)
- def regenerate_primary_file(self, dataset: "DatasetInstance") -> None:
+ def regenerate_primary_file(self, dataset: DatasetProtocol) -> None:
# Cannot do this until we are setting metadata.
rval = [""]
rval.append("Files prepared for IDEAS
")
@@ -654,7 +654,7 @@ class RexpBase(Html):
is_binary=False,
)
- def generate_primary_file(self, dataset: GeneratePrimaryFileDataset) -> str:
+ def generate_primary_file(self, dataset: HasExtraFilesAndMetadata) -> str:
"""
This is called only at upload to write the html file
cannot rename the datasets here - they come with the default unfortunately
@@ -764,7 +764,7 @@ class RexpBase(Html):
p = []
return "\n".join(p)
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
"""
expects a .pheno file in the extra_files_dir - ugh
note that R is weird and does not include the row.name in
@@ -813,7 +813,7 @@ class RexpBase(Html):
except Exception:
return "## rexpression get_file_peek: no file found"
- def regenerate_primary_file(self, dataset: "DatasetInstance") -> None:
+ def regenerate_primary_file(self, dataset: DatasetProtocol) -> None:
"""
cannot do this until we are setting metadata
"""
@@ -830,11 +830,11 @@ class RexpBase(Html):
f.write("\n".join(rval))
f.write("\n")
- def init_meta(self, dataset: "DatasetInstance", copy_from: Optional["DatasetInstance"] = None) -> None:
+ def init_meta(self, dataset: HasMetadata, copy_from: Optional[HasMetadata] = None) -> None:
if copy_from:
dataset.metadata = copy_from.metadata
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""
NOTE we apply the tabular machinary to the phenodata extracted
from a BioC eSet or affybatch.
@@ -910,7 +910,7 @@ class RexpBase(Html):
except Exception as exc:
return f"Can't create html table {unicodify(exc)}"
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
"""
Returns formatted html of peek
"""
diff --git a/lib/galaxy/datatypes/gis.py b/lib/galaxy/datatypes/gis.py
index ead789aa2d9..b3b37dce200 100644
--- a/lib/galaxy/datatypes/gis.py
+++ b/lib/galaxy/datatypes/gis.py
@@ -1,13 +1,11 @@
"""
GIS classes
"""
-from typing import TYPE_CHECKING
-
+from galaxy.datatypes._protocols import (
+ DatasetProtocol,
+ HasExtraFilesAndMetadata,
+)
from galaxy.datatypes.binary import Binary
-from galaxy.datatypes.data import GeneratePrimaryFileDataset
-
-if TYPE_CHECKING:
- from galaxy.model import DatasetInstance
class Shapefile(Binary):
@@ -66,7 +64,7 @@ class Shapefile(Binary):
"shapefile.shp.xml", description="Geospatial metadata in XML format (xml)", is_binary=False, optional=True
)
- def generate_primary_file(self, dataset: GeneratePrimaryFileDataset) -> str:
+ def generate_primary_file(self, dataset: HasExtraFilesAndMetadata) -> str:
rval = ["Shapefile Galaxy Composite Dataset "]
rval.append("This composite dataset is composed of the following files:")
for composite_name, composite_file in self.get_composite_files(dataset=dataset).items():
@@ -83,7 +81,7 @@ class Shapefile(Binary):
rval.append("
\n")
return "\n".join(rval)
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
"""Set the peek and blurb text."""
if not dataset.dataset.purged:
dataset.peek = "Shapefile data"
@@ -92,7 +90,7 @@ class Shapefile(Binary):
dataset.peek = "file does not exist"
dataset.blurb = "file purged from disk"
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
"""Create HTML content, used for displaying peek."""
try:
return dataset.peek
diff --git a/lib/galaxy/datatypes/goldenpath.py b/lib/galaxy/datatypes/goldenpath.py
index d5ef0405449..e3204ac3067 100644
--- a/lib/galaxy/datatypes/goldenpath.py
+++ b/lib/galaxy/datatypes/goldenpath.py
@@ -3,10 +3,10 @@ import logging
import os
from typing import (
Set,
- TYPE_CHECKING,
Union,
)
+from galaxy.datatypes._protocols import DatasetProtocol
from galaxy.datatypes.sniff import (
build_sniff_from_prefix,
FilePrefix,
@@ -14,9 +14,6 @@ from galaxy.datatypes.sniff import (
)
from .tabular import Tabular
-if TYPE_CHECKING:
- from galaxy.model import DatasetInstance
-
log = logging.getLogger(__name__)
@@ -27,7 +24,7 @@ class GoldenPath(Tabular):
edam_format = "format_3693"
file_ext = "agp"
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
# AGPFile reads and validates entire file.
AGPFile(dataset.file_name)
super().set_meta(dataset, overwrite=overwrite, **kwd)
diff --git a/lib/galaxy/datatypes/graph.py b/lib/galaxy/datatypes/graph.py
index babdb0c8489..3ecea4e1ede 100644
--- a/lib/galaxy/datatypes/graph.py
+++ b/lib/galaxy/datatypes/graph.py
@@ -2,11 +2,9 @@
Graph content classes.
"""
import logging
-from typing import (
- List,
- TYPE_CHECKING,
-)
+from typing import List
+from galaxy.datatypes._protocols import DatasetProtocol
from galaxy.datatypes.dataproviders.column import ColumnarDataProvider
from galaxy.datatypes.dataproviders.dataset import DatasetDataProvider
from galaxy.datatypes.dataproviders.hierarchy import XMLDataProvider
@@ -18,9 +16,6 @@ from . import (
xml,
)
-if TYPE_CHECKING:
- from galaxy.model import DatasetInstance
-
log = logging.getLogger(__name__)
@@ -33,7 +28,7 @@ class Xgmml(xml.GenericXml):
file_ext = "xgmml"
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
"""
Set the peek and blurb text
"""
@@ -63,7 +58,7 @@ class Xgmml(xml.GenericXml):
data.Text.merge(split_files, output_file)
@dataproviders.decorators.dataprovider_factory("node-edge", XMLDataProvider.settings)
- def node_edge_dataprovider(self, dataset: "DatasetInstance", **settings) -> "XGMMLGraphDataProvider":
+ def node_edge_dataprovider(self, dataset: DatasetProtocol, **settings) -> "XGMMLGraphDataProvider":
dataset_source = DatasetDataProvider(dataset)
return XGMMLGraphDataProvider(dataset_source, **settings)
@@ -81,7 +76,7 @@ class Sif(tabular.Tabular):
file_ext = "sif"
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
"""
Set the peek and blurb text
"""
@@ -103,7 +98,7 @@ class Sif(tabular.Tabular):
data.Text.merge(split_files, output_file)
@dataproviders.decorators.dataprovider_factory("node-edge", ColumnarDataProvider.settings)
- def node_edge_dataprovider(self, dataset: "DatasetInstance", **settings) -> "SIFGraphDataProvider":
+ def node_edge_dataprovider(self, dataset: DatasetProtocol, **settings) -> "SIFGraphDataProvider":
dataset_source = DatasetDataProvider(dataset)
return SIFGraphDataProvider(dataset_source, **settings)
diff --git a/lib/galaxy/datatypes/hdf5.py b/lib/galaxy/datatypes/hdf5.py
index 6a2b42c58be..2bc15251420 100644
--- a/lib/galaxy/datatypes/hdf5.py
+++ b/lib/galaxy/datatypes/hdf5.py
@@ -2,20 +2,15 @@
This datatype was created for use with the iSEE interactive tool.
"""
-from typing import (
- Optional,
- TYPE_CHECKING,
-)
+from typing import Optional
-from galaxy.datatypes.data import (
- Data,
- GeneratePrimaryFileDataset,
+from galaxy.datatypes._protocols import (
+ HasExtraFilesAndMetadata,
+ HasMetadata,
)
+from galaxy.datatypes.data import Data
from galaxy.datatypes.metadata import MetadataElement
-if TYPE_CHECKING:
- from galaxy.model import DatasetInstance
-
class HDF5SummarizedExperiment(Data):
"""Composite datatype to represent HDF5SummarizedExperiment objects.
@@ -56,11 +51,11 @@ class HDF5SummarizedExperiment(Data):
description="Summarized experiment data array",
)
- def init_meta(self, dataset: "DatasetInstance", copy_from: Optional["DatasetInstance"] = None) -> None:
+ def init_meta(self, dataset: HasMetadata, copy_from: Optional[HasMetadata] = None) -> None:
"""Override parent init metadata."""
Data.init_meta(self, dataset, copy_from=copy_from)
- def generate_primary_file(self, dataset: GeneratePrimaryFileDataset) -> str:
+ def generate_primary_file(self, dataset: HasExtraFilesAndMetadata) -> str:
"""Generate primary file to represent dataset."""
return f"""
diff --git a/lib/galaxy/datatypes/images.py b/lib/galaxy/datatypes/images.py
index 8dd6f79b7c6..6576b88b871 100644
--- a/lib/galaxy/datatypes/images.py
+++ b/lib/galaxy/datatypes/images.py
@@ -4,17 +4,17 @@ Image classes
import base64
import json
import logging
-from typing import (
- Optional,
- TYPE_CHECKING,
-)
+from typing import Optional
import mrcfile
import numpy as np
import tifffile
+from galaxy.datatypes._protocols import (
+ DatasetProtocol,
+ HasExtraFilesAndMetadata,
+)
from galaxy.datatypes.binary import Binary
-from galaxy.datatypes.data import GeneratePrimaryFileDataset
from galaxy.datatypes.metadata import (
FileParameter,
MetadataElement,
@@ -29,9 +29,6 @@ from galaxy.util.image_util import check_image_type
from . import data
from .xml import GenericXml
-if TYPE_CHECKING:
- from galaxy.model import DatasetInstance
-
log = logging.getLogger(__name__)
# TODO: Uploading image files of various types is supported in Galaxy, but on
@@ -56,7 +53,7 @@ class Image(data.Data):
super().__init__(**kwd)
self.image_formats = [self.file_ext.upper()]
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
dataset.peek = f"Image in {dataset.extension} format"
dataset.blurb = nice_size(dataset.get_size())
@@ -68,7 +65,7 @@ class Image(data.Data):
"""Determine if the file is in this format"""
return check_image_type(filename, self.image_formats)
- def handle_dataset_as_image(self, hda: "DatasetInstance") -> str:
+ def handle_dataset_as_image(self, hda: DatasetProtocol) -> str:
dataset = hda.dataset
name = hda.name or ""
with open(dataset.file_name, "rb") as f:
@@ -108,7 +105,7 @@ class OMETiff(Tiff):
)
def set_meta(
- self, dataset: "DatasetInstance", overwrite: bool = True, metadata_tmp_files_dir: Optional[str] = None, **kwd
+ self, dataset: DatasetProtocol, overwrite: bool = True, metadata_tmp_files_dir: Optional[str] = None, **kwd
) -> None:
spec_key = "offsets"
offsets_file = dataset.metadata.offsets
@@ -383,7 +380,7 @@ class Analyze75(Binary):
self.add_composite_file("t2m", description="The Analyze75 t2m file.", optional=True, is_binary=True)
- def generate_primary_file(self, dataset: GeneratePrimaryFileDataset) -> str:
+ def generate_primary_file(self, dataset: HasExtraFilesAndMetadata) -> str:
rval = ["Analyze75 Composite Dataset. "]
rval.append("This composite dataset is composed of the following files:")
for composite_name, composite_file in self.get_composite_files(dataset=dataset).items():
@@ -493,7 +490,7 @@ class Star(data.Text):
file_ext = "star"
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
"""Set the peek and blurb text"""
if not dataset.dataset.purged:
dataset.peek = data.get_file_peek(dataset.file_name)
diff --git a/lib/galaxy/datatypes/interval.py b/lib/galaxy/datatypes/interval.py
index 2c4e710d9e1..a21cb2368c4 100644
--- a/lib/galaxy/datatypes/interval.py
+++ b/lib/galaxy/datatypes/interval.py
@@ -9,7 +9,6 @@ from typing import (
List,
Optional,
Tuple,
- TYPE_CHECKING,
Union,
)
from urllib.parse import quote_plus
@@ -22,6 +21,11 @@ from bx.intervals.io import (
from galaxy import util
from galaxy.datatypes import metadata
+from galaxy.datatypes._protocols import (
+ DatasetProtocol,
+ HasId,
+ HasMetadata,
+)
from galaxy.datatypes.data import DatatypeValidation
from galaxy.datatypes.dataproviders.dataset import (
DatasetDataProvider,
@@ -48,9 +52,6 @@ from . import (
dataproviders,
)
-if TYPE_CHECKING:
- from galaxy.model import DatasetInstance
-
log = logging.getLogger(__name__)
# Contains the meta columns and the words that map to it; list aliases on the
@@ -123,11 +124,11 @@ class Interval(Tabular):
Tabular.__init__(self, **kwd)
self.add_display_app("ucsc", "display at UCSC", "as_ucsc_display_file", "ucsc_links")
- def init_meta(self, dataset: "DatasetInstance", copy_from: Optional["DatasetInstance"] = None) -> None:
+ def init_meta(self, dataset: HasMetadata, copy_from: Optional[HasMetadata] = None) -> None:
Tabular.init_meta(self, dataset, copy_from=copy_from)
def set_meta(
- self, dataset: "DatasetInstance", *, overwrite: bool = True, first_line_is_header: bool = False, **kwd
+ self, dataset: DatasetProtocol, *, overwrite: bool = True, first_line_is_header: bool = False, **kwd
) -> None:
"""Tries to guess from the line the location number of the column for the chromosome, region start-end and strand"""
Tabular.set_meta(self, dataset, overwrite=overwrite, skip=0)
@@ -188,7 +189,7 @@ class Interval(Tabular):
else:
empty_line_count += 1
- def displayable(self, dataset: "DatasetInstance"):
+ def displayable(self, dataset: DatasetProtocol) -> bool:
try:
return (
not dataset.dataset.purged
@@ -205,7 +206,7 @@ class Interval(Tabular):
def get_estimated_display_viewport(
self,
- dataset: "DatasetInstance",
+ dataset: DatasetProtocol,
chrom_col: Optional[int] = None,
start_col: Optional[int] = None,
end_col: Optional[int] = None,
@@ -267,7 +268,7 @@ class Interval(Tabular):
log.exception("Exception caught attempting to generate viewport for dataset '%d'", dataset.id)
return (None, None, None)
- def as_ucsc_display_file(self, dataset: "DatasetInstance", **kwd) -> Union[FileObjType, str]:
+ def as_ucsc_display_file(self, dataset: DatasetProtocol, **kwd) -> Union[FileObjType, str]:
"""Returns file contents with only the bed data"""
with tempfile.NamedTemporaryFile(delete=False, mode="w") as fh:
c, s, e, t, n = (
@@ -301,7 +302,7 @@ class Interval(Tabular):
fh.write("%s\n" % "\t".join(tmp))
return compression_utils.get_fileobj(fh.name, mode="rb")
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
"""Returns formated html of peek"""
return self.make_html_table(
dataset,
@@ -314,7 +315,7 @@ class Interval(Tabular):
},
)
- def ucsc_links(self, dataset: "DatasetInstance", type: str, app, base_url: str) -> List:
+ def ucsc_links(self, dataset: DatasetProtocol, type: str, app, base_url: str) -> List:
"""
Generate links to UCSC genome browser sites based on the dbkey
and content of dataset.
@@ -348,7 +349,7 @@ class Interval(Tabular):
ret_val.append((site_name, link))
return ret_val
- def validate(self, dataset: "DatasetInstance", **kwd) -> DatatypeValidation:
+ def validate(self, dataset: DatasetProtocol, **kwd) -> DatatypeValidation:
"""Validate an interval file using the bx GenomicIntervalReader"""
c, s, e, t = (
dataset.metadata.chromCol,
@@ -403,20 +404,20 @@ class Interval(Tabular):
# ------------- Dataproviders
@dataproviders.decorators.dataprovider_factory("genomic-region", GenomicRegionDataProvider.settings)
- def genomic_region_dataprovider(self, dataset: "DatasetInstance", **settings) -> GenomicRegionDataProvider:
+ def genomic_region_dataprovider(self, dataset: DatasetProtocol, **settings) -> GenomicRegionDataProvider:
return GenomicRegionDataProvider(dataset, **settings)
@dataproviders.decorators.dataprovider_factory("genomic-region-dict", GenomicRegionDataProvider.settings)
- def genomic_region_dict_dataprovider(self, dataset: "DatasetInstance", **settings) -> GenomicRegionDataProvider:
+ def genomic_region_dict_dataprovider(self, dataset: DatasetProtocol, **settings) -> GenomicRegionDataProvider:
settings["named_columns"] = True
return self.genomic_region_dataprovider(dataset, **settings)
@dataproviders.decorators.dataprovider_factory("interval", IntervalDataProvider.settings)
- def interval_dataprovider(self, dataset: "DatasetInstance", **settings) -> IntervalDataProvider:
+ def interval_dataprovider(self, dataset: DatasetProtocol, **settings) -> IntervalDataProvider:
return IntervalDataProvider(dataset, **settings)
@dataproviders.decorators.dataprovider_factory("interval-dict", IntervalDataProvider.settings)
- def interval_dict_dataprovider(self, dataset: "DatasetInstance", **settings) -> IntervalDataProvider:
+ def interval_dict_dataprovider(self, dataset: DatasetProtocol, **settings) -> IntervalDataProvider:
settings["named_columns"] = True
return self.interval_dataprovider(dataset, **settings)
@@ -429,7 +430,7 @@ class BedGraph(Interval):
track_type = "LineTrack"
data_sources = {"data": "bigwig", "index": "bigwig"}
- def as_ucsc_display_file(self, dataset: "DatasetInstance", **kwd) -> Union[FileObjType, str]:
+ def as_ucsc_display_file(self, dataset: DatasetProtocol, **kwd) -> Union[FileObjType, str]:
"""
Returns file contents as is with no modifications.
TODO: this is a functional stub and will need to be enhanced moving forward to provide additional support for bedgraph.
@@ -438,7 +439,7 @@ class BedGraph(Interval):
def get_estimated_display_viewport(
self,
- dataset: "DatasetInstance",
+ dataset: DatasetProtocol,
chrom_col: Optional[int] = 0,
start_col: Optional[int] = 1,
end_col: Optional[int] = 2,
@@ -497,7 +498,7 @@ class Bed(Interval):
)
# do we need to repeat these? they are the same as should be inherited from interval type
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""Sets the metadata information for datasets previously determined to be in bed format."""
if dataset.has_data():
i = 0
@@ -518,7 +519,7 @@ class Bed(Interval):
break
Tabular.set_meta(self, dataset, overwrite=overwrite, skip=i)
- def as_ucsc_display_file(self, dataset: "DatasetInstance", **kwd) -> Union[FileObjType, str]:
+ def as_ucsc_display_file(self, dataset: DatasetProtocol, **kwd) -> Union[FileObjType, str]:
"""Returns file contents with only the bed data. If bed 6+, treat as interval."""
for line in open(dataset.file_name):
line = line.strip()
@@ -730,7 +731,7 @@ class BedStrict(Bed):
Tabular.__init__(self, **kwd)
self.clear_display_apps() # only new style display applications for this datatype
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
Tabular.set_meta(self, dataset, overwrite=overwrite, **kwd) # need column count first
if dataset.metadata.columns >= 4:
dataset.metadata.nameCol = 4
@@ -757,7 +758,7 @@ class Bed12(BedStrict):
class _RemoteCallMixin:
def _get_remote_call_url(
- self, redirect_url: str, site_name: str, dataset: "DatasetInstance", type: str, app, base_url: str
+ self, redirect_url: str, site_name: str, dataset: HasId, type: str, app, base_url: str
) -> str:
"""Retrieve the URL to call out to an external site and retrieve data.
This routes our external URL through a local galaxy instance which makes
@@ -814,7 +815,7 @@ class Gff(Tabular, _RemoteCallMixin):
self.add_display_app("ucsc", "display at UCSC", "as_ucsc_display_file", "ucsc_links")
self.add_display_app("gbrowse", "display in Gbrowse", "as_gbrowse_display_file", "gbrowse_links")
- def set_attribute_metadata(self, dataset: "DatasetInstance") -> None:
+ def set_attribute_metadata(self, dataset: DatasetProtocol) -> None:
"""
Sets metadata elements for dataset's attributes.
"""
@@ -854,7 +855,7 @@ class Gff(Tabular, _RemoteCallMixin):
dataset.metadata.attribute_types = attribute_types
dataset.metadata.attributes = len(attribute_types)
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
self.set_attribute_metadata(dataset)
i = 0
@@ -872,12 +873,12 @@ class Gff(Tabular, _RemoteCallMixin):
pass
Tabular.set_meta(self, dataset, overwrite=overwrite, skip=i)
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
"""Returns formated html of peek"""
return self.make_html_table(dataset, column_names=self.column_names)
def get_estimated_display_viewport(
- self, dataset: "DatasetInstance"
+ self, dataset: DatasetProtocol
) -> Tuple[Optional[str], Optional[str], Optional[str]]:
"""
Return a chrom, start, stop tuple for viewing a file. There are slight differences between gff 2 and gff 3
@@ -953,7 +954,7 @@ class Gff(Tabular, _RemoteCallMixin):
log.exception("Unexpected error")
return (None, None, None) # could not determine viewport
- def ucsc_links(self, dataset: "DatasetInstance", type: str, app, base_url: str) -> List:
+ def ucsc_links(self, dataset: DatasetProtocol, type: str, app, base_url: str) -> List:
ret_val = []
seqid, start, stop = self.get_estimated_display_viewport(dataset)
if seqid is not None:
@@ -966,7 +967,7 @@ class Gff(Tabular, _RemoteCallMixin):
ret_val.append((site_name, link))
return ret_val
- def gbrowse_links(self, dataset: "DatasetInstance", type: str, app, base_url: str) -> List:
+ def gbrowse_links(self, dataset: DatasetProtocol, type: str, app, base_url: str) -> List:
ret_val = []
seqid, start, stop = self.get_estimated_display_viewport(dataset)
if seqid is not None:
@@ -1032,20 +1033,20 @@ class Gff(Tabular, _RemoteCallMixin):
# ------------- Dataproviders
# redefine bc super is Tabular
@dataproviders.decorators.dataprovider_factory("genomic-region", GenomicRegionDataProvider.settings)
- def genomic_region_dataprovider(self, dataset: "DatasetInstance", **settings) -> GenomicRegionDataProvider:
+ def genomic_region_dataprovider(self, dataset: DatasetProtocol, **settings) -> GenomicRegionDataProvider:
return GenomicRegionDataProvider(dataset, 0, 3, 4, **settings)
@dataproviders.decorators.dataprovider_factory("genomic-region-dict", GenomicRegionDataProvider.settings)
- def genomic_region_dict_dataprovider(self, dataset: "DatasetInstance", **settings) -> GenomicRegionDataProvider:
+ def genomic_region_dict_dataprovider(self, dataset: DatasetProtocol, **settings) -> GenomicRegionDataProvider:
settings["named_columns"] = True
return self.genomic_region_dataprovider(dataset, **settings)
@dataproviders.decorators.dataprovider_factory("interval", IntervalDataProvider.settings)
- def interval_dataprovider(self, dataset: "DatasetInstance", **settings):
+ def interval_dataprovider(self, dataset: DatasetProtocol, **settings):
return IntervalDataProvider(dataset, 0, 3, 4, 6, 2, **settings)
@dataproviders.decorators.dataprovider_factory("interval-dict", IntervalDataProvider.settings)
- def interval_dict_dataprovider(self, dataset: "DatasetInstance", **settings):
+ def interval_dict_dataprovider(self, dataset: DatasetProtocol, **settings):
settings["named_columns"] = True
return self.interval_dataprovider(dataset, **settings)
@@ -1073,7 +1074,7 @@ class Gff3(Gff):
"""Initialize datatype, by adding GBrowse display app"""
Gff.__init__(self, **kwd)
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
self.set_attribute_metadata(dataset)
i = 0
with compression_utils.get_fileobj(dataset.file_name) as in_fh:
@@ -1289,7 +1290,7 @@ class Wiggle(Tabular, _RemoteCallMixin):
self.add_display_app("gbrowse", "display in Gbrowse", "as_gbrowse_display_file", "gbrowse_links")
def get_estimated_display_viewport(
- self, dataset: "DatasetInstance"
+ self, dataset: DatasetProtocol
) -> Tuple[Optional[str], Optional[str], Optional[str]]:
"""Return a chrom, start, stop tuple for viewing a file."""
viewport_feature_count = 100 # viewport should check at least 100 features; excludes comment lines
@@ -1355,7 +1356,7 @@ class Wiggle(Tabular, _RemoteCallMixin):
log.exception("Unexpected error")
return (None, None, None) # could not determine viewport
- def gbrowse_links(self, dataset: "DatasetInstance", type: str, app, base_url: str) -> List:
+ def gbrowse_links(self, dataset: DatasetProtocol, type: str, app, base_url: str) -> List:
ret_val = []
chrom, start, stop = self.get_estimated_display_viewport(dataset)
if chrom is not None:
@@ -1368,7 +1369,7 @@ class Wiggle(Tabular, _RemoteCallMixin):
ret_val.append((site_name, link))
return ret_val
- def ucsc_links(self, dataset: "DatasetInstance", type: str, app, base_url: str) -> List:
+ def ucsc_links(self, dataset: DatasetProtocol, type: str, app, base_url: str) -> List:
ret_val = []
chrom, start, stop = self.get_estimated_display_viewport(dataset)
if chrom is not None:
@@ -1381,11 +1382,11 @@ class Wiggle(Tabular, _RemoteCallMixin):
ret_val.append((site_name, link))
return ret_val
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
"""Returns formated html of peek"""
return self.make_html_table(dataset, skipchars=["track", "#"])
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
max_data_lines = None
i = 0
for i, line in enumerate(open(dataset.file_name)): # noqa: B007
@@ -1443,12 +1444,12 @@ class Wiggle(Tabular, _RemoteCallMixin):
# ------------- Dataproviders
@dataproviders.decorators.dataprovider_factory("wiggle", WiggleDataProvider.settings)
- def wiggle_dataprovider(self, dataset: "DatasetInstance", **settings) -> WiggleDataProvider:
+ def wiggle_dataprovider(self, dataset: DatasetProtocol, **settings) -> WiggleDataProvider:
dataset_source = DatasetDataProvider(dataset)
return WiggleDataProvider(dataset_source, **settings)
@dataproviders.decorators.dataprovider_factory("wiggle-dict", WiggleDataProvider.settings)
- def wiggle_dict_dataprovider(self, dataset: "DatasetInstance", **settings) -> WiggleDataProvider:
+ def wiggle_dict_dataprovider(self, dataset: DatasetProtocol, **settings) -> WiggleDataProvider:
dataset_source = DatasetDataProvider(dataset)
settings["named_columns"] = True
return WiggleDataProvider(dataset_source, **settings)
@@ -1466,16 +1467,16 @@ class CustomTrack(Tabular):
Tabular.__init__(self, **kwd)
self.add_display_app("ucsc", "display at UCSC", "as_ucsc_display_file", "ucsc_links")
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
Tabular.set_meta(self, dataset, overwrite=overwrite, skip=1)
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
"""Returns formated html of peek"""
return self.make_html_table(dataset, skipchars=["track", "#"])
def get_estimated_display_viewport(
self,
- dataset: "DatasetInstance",
+ dataset: DatasetProtocol,
chrom_col: Optional[int] = None,
start_col: Optional[int] = None,
end_col: Optional[int] = None,
@@ -1537,7 +1538,7 @@ class CustomTrack(Tabular):
log.exception("Unexpected error")
return (None, None, None) # could not determine viewport
- def ucsc_links(self, dataset: "DatasetInstance", type: str, app, base_url: str) -> List:
+ def ucsc_links(self, dataset: DatasetProtocol, type: str, app, base_url: str) -> List:
ret_val = []
chrom, start, stop = self.get_estimated_display_viewport(dataset)
if chrom is not None:
@@ -1782,7 +1783,7 @@ class IntervalTabix(Interval):
# Ideally the tabix_index would be regenerated when the metadataElements are updated
def set_meta(
self,
- dataset: "DatasetInstance",
+ dataset: DatasetProtocol,
overwrite: bool = True,
first_line_is_header: bool = False,
metadata_tmp_files_dir: Optional[str] = None,
diff --git a/lib/galaxy/datatypes/isa.py b/lib/galaxy/datatypes/isa.py
index 6b5f81687d2..38edcd22dfd 100644
--- a/lib/galaxy/datatypes/isa.py
+++ b/lib/galaxy/datatypes/isa.py
@@ -26,21 +26,19 @@ from isatools import (
from markupsafe import escape
from galaxy import util
-from galaxy.datatypes.data import (
- Data,
- GeneratePrimaryFileDataset,
+from galaxy.datatypes._protocols import (
+ DatasetHasHidProtocol,
+ DatasetProtocol,
+ HasExtraFilesAndMetadata,
+ HasExtraFilesPath,
)
+from galaxy.datatypes.data import Data
from galaxy.util.compression_utils import CompressedFile
from galaxy.util.sanitize_html import sanitize_html
if TYPE_CHECKING:
from isatools.model import Investigation
- from galaxy.model import (
- DatasetInstance,
- HistoryDatasetAssociation,
- )
-
# CONSTANTS {{{1
################################################################
@@ -92,7 +90,7 @@ class _Isa(Data):
# Get ISA folder path {{{2
################################################################
- def _get_isa_folder_path(self, dataset: "DatasetInstance") -> str:
+ def _get_isa_folder_path(self, dataset: HasExtraFilesPath) -> str:
isa_folder = dataset.extra_files_path
if not isa_folder:
raise Exception("Unvalid dataset object, or no extra files path found for this dataset.")
@@ -101,7 +99,7 @@ class _Isa(Data):
# Get main file {{{2
################################################################
- def _get_main_file(self, dataset: "DatasetInstance") -> Optional[str]:
+ def _get_main_file(self, dataset: HasExtraFilesPath) -> Optional[str]:
"""Get the main file of the ISA archive. Either the investigation file i_*.txt for ISA-Tab, or the JSON file for ISA-JSON."""
main_file = None
@@ -127,7 +125,7 @@ class _Isa(Data):
# Get investigation {{{2
################################################################
- def _get_investigation(self, dataset: "DatasetInstance") -> Optional["Investigation"]:
+ def _get_investigation(self, dataset: HasExtraFilesPath) -> Optional["Investigation"]:
"""Create a contained instance specific to the exact ISA type (Tab or Json).
We will use it to parse and access information from the archive."""
@@ -164,7 +162,7 @@ class _Isa(Data):
# Set peek {{{2
################################################################
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
"""Set the peek and blurb text. Get first lines of the main file and set it as the peek."""
main_file = self._get_main_file(dataset)
@@ -190,7 +188,7 @@ class _Isa(Data):
# Display peek {{{2
################################################################
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
"""Create the HTML table used for displaying peek, from the peek text found by set_peek() method."""
out = ['']
@@ -211,7 +209,7 @@ class _Isa(Data):
# Generate primary file {{{2
################################################################
- def generate_primary_file(self, dataset: GeneratePrimaryFileDataset) -> str:
+ def generate_primary_file(self, dataset: HasExtraFilesAndMetadata) -> str:
"""Generate the primary file. It is an HTML file containing description of the composite dataset
as well as a list of the composite files that it contains."""
@@ -270,7 +268,7 @@ class _Isa(Data):
def display_data(
self,
trans,
- dataset: "HistoryDatasetAssociation",
+ dataset: DatasetHasHidProtocol,
preview: bool = False,
filename: Optional[str] = None,
to_ext: Optional[str] = None,
diff --git a/lib/galaxy/datatypes/media.py b/lib/galaxy/datatypes/media.py
index 987df7fc928..1b7baf4ddd7 100644
--- a/lib/galaxy/datatypes/media.py
+++ b/lib/galaxy/datatypes/media.py
@@ -6,9 +6,9 @@ from functools import lru_cache
from typing import (
List,
Tuple,
- TYPE_CHECKING,
)
+from galaxy.datatypes._protocols import DatasetProtocol
from galaxy.datatypes.binary import Binary
from galaxy.datatypes.metadata import (
ListParameter,
@@ -16,9 +16,6 @@ from galaxy.datatypes.metadata import (
)
from galaxy.util import which
-if TYPE_CHECKING:
- from galaxy.model import DatasetInstance
-
@lru_cache(maxsize=128)
def _ffprobe(path):
@@ -75,7 +72,7 @@ class Audio(Binary):
no_value=0,
)
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
if which("ffprobe"):
metadata, streams = ffprobe(dataset.file_name)
@@ -162,7 +159,7 @@ class Video(Binary):
w = h = fps = 0
return w, h, fps
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
if which("ffprobe"):
metadata, streams = ffprobe(dataset.file_name)
(w, h, fps) = self._get_resolution(streams)
@@ -283,7 +280,7 @@ class Wav(Audio):
with wave.open(filename, "rb"):
return True
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""Set the metadata for this dataset from the file contents."""
try:
with wave.open(dataset.dataset.file_name, "rb") as fd:
diff --git a/lib/galaxy/datatypes/microarrays.py b/lib/galaxy/datatypes/microarrays.py
index aa8ba733a1a..1aeea27a6c5 100644
--- a/lib/galaxy/datatypes/microarrays.py
+++ b/lib/galaxy/datatypes/microarrays.py
@@ -1,7 +1,7 @@
import logging
-from typing import TYPE_CHECKING
from galaxy.datatypes import data
+from galaxy.datatypes._protocols import DatasetProtocol
from galaxy.datatypes.binary import Cel # noqa: F401
from galaxy.datatypes.data import get_file_peek
from galaxy.datatypes.metadata import MetadataElement
@@ -11,9 +11,6 @@ from galaxy.datatypes.sniff import (
get_headers,
)
-if TYPE_CHECKING:
- from galaxy.model import DatasetInstance
-
log = logging.getLogger(__name__)
@@ -80,7 +77,7 @@ class GenericMicroarrayFile(data.Text):
name="block_type", default=0, desc="Type of block", readonly=True, visible=True, optional=True, no_value=0
)
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
if dataset.metadata.block_count == 1:
dataset.blurb = f"{dataset.metadata.file_type} {dataset.metadata.version_number}: Format {dataset.metadata.file_format}, 1 block, {dataset.metadata.number_of_optional_header_records} headers and {dataset.metadata.number_of_data_columns} columns"
@@ -120,7 +117,7 @@ class Gal(GenericMicroarrayFile):
headers = get_headers(file_prefix, sep="\t", count=3)
return "ATF" in headers[0][0] and "GenePix ArrayList" in headers[2][0]
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""
Set metadata for Gal file.
"""
@@ -162,7 +159,7 @@ class Gpr(GenericMicroarrayFile):
headers = get_headers(file_prefix, sep="\t", count=3)
return "ATF" in headers[0][0] and "GenePix Results" in headers[2][0]
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""
Set metadata for Gpr file.
"""
diff --git a/lib/galaxy/datatypes/molecules.py b/lib/galaxy/datatypes/molecules.py
index fa2d103353e..69fbc1a4515 100644
--- a/lib/galaxy/datatypes/molecules.py
+++ b/lib/galaxy/datatypes/molecules.py
@@ -5,10 +5,10 @@ from typing import (
Callable,
Dict,
List,
- TYPE_CHECKING,
)
from galaxy.datatypes import metadata
+from galaxy.datatypes._protocols import DatasetProtocol
from galaxy.datatypes.binary import Binary
from galaxy.datatypes.data import (
get_file_peek,
@@ -29,9 +29,6 @@ from galaxy.util import (
unicodify,
)
-if TYPE_CHECKING:
- from galaxy.model import DatasetInstance
-
# optional import to enhance metadata
try:
from ase import io as ase_io
@@ -72,7 +69,7 @@ class GenericMolFile(Text):
no_value=0,
)
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
if dataset.metadata.number_of_molecules == 1:
dataset.blurb = "1 molecule"
@@ -204,7 +201,7 @@ class GenericMolFile(Text):
class MOL(GenericMolFile):
file_ext = "mol"
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""
Set the number molecules, in the case of MOL its always one.
"""
@@ -257,7 +254,7 @@ class SDF(GenericMolFile):
break
return False
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""
Set the number of molecules in dataset.
"""
@@ -340,7 +337,7 @@ class MOL2(GenericMolFile):
break
return False
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""
Set the number of lines of data in dataset.
"""
@@ -426,7 +423,7 @@ class FPS(GenericMolFile):
else:
return False
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""
Set the number of lines of data in dataset.
"""
@@ -531,7 +528,7 @@ class OBFS(Binary):
self.add_composite_file("molecule.mol2", optional=True, is_binary=False, description="Molecule File")
self.add_composite_file("molecule.cml", optional=True, is_binary=False, description="Molecule File")
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
"""Set the peek and blurb text."""
if not dataset.dataset.purged:
dataset.peek = "OpenBabel Fastsearch Index"
@@ -540,7 +537,7 @@ class OBFS(Binary):
dataset.peek = "file does not exist"
dataset.blurb = "file purged from disk"
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
"""Create HTML content, used for displaying peek."""
try:
return dataset.peek
@@ -567,7 +564,7 @@ class OBFS(Binary):
class DRF(GenericMolFile):
file_ext = "drf"
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""
Set the number of lines of data in dataset.
"""
@@ -581,7 +578,7 @@ class PHAR(GenericMolFile):
file_ext = "phar"
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
dataset.peek = get_file_peek(dataset.file_name)
dataset.blurb = "pharmacophore"
@@ -634,7 +631,7 @@ class PDB(GenericMolFile):
else:
return False
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""
Find Chain_IDs for metadata.
"""
@@ -650,7 +647,7 @@ class PDB(GenericMolFile):
log.error("Error finding chain_ids: %s", unicodify(e))
raise
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
atom_numbers = count_special_lines("^ATOM", dataset.file_name)
hetatm_numbers = count_special_lines("^HETATM", dataset.file_name)
@@ -703,7 +700,7 @@ class PDBQT(GenericMolFile):
else:
return False
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
root_numbers = count_special_lines("^ROOT", dataset.file_name)
branch_numbers = count_special_lines("^BRANCH", dataset.file_name)
@@ -788,7 +785,7 @@ class PQR(GenericMolFile):
else:
return False
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""
Find Optional Chain_IDs for metadata.
"""
@@ -807,7 +804,7 @@ class PQR(GenericMolFile):
log.error("Error finding chain_ids: %s", unicodify(e))
raise
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
atom_numbers = count_special_lines("^ATOM", dataset.file_name)
hetatm_numbers = count_special_lines("^HETATM", dataset.file_name)
@@ -885,7 +882,7 @@ class Cell(GenericMolFile):
return True
return False
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""
Find Atom IDs for metadata.
"""
@@ -925,7 +922,7 @@ class Cell(GenericMolFile):
dataset.metadata.is_periodic = bool(pbc.any())
dataset.metadata.lattice_parameters = list(lattice_parameters)
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
dataset.peek = get_file_peek(dataset.file_name)
dataset.info = self.get_dataset_info(dataset.metadata)
@@ -1046,7 +1043,7 @@ class CIF(GenericMolFile):
return False
return False
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""
Find Atom IDs for metadata.
"""
@@ -1092,7 +1089,7 @@ class CIF(GenericMolFile):
dataset.metadata.is_periodic = is_periodic
dataset.metadata.lattice_parameters = list(lattice_parameters)
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
dataset.peek = get_file_peek(dataset.file_name)
dataset.info = self.get_dataset_info(dataset.metadata)
@@ -1243,7 +1240,7 @@ class XYZ(GenericMolFile):
raise
return blocks
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""
Find Atom IDs for metadata.
"""
@@ -1290,7 +1287,7 @@ class XYZ(GenericMolFile):
dataset.metadata.is_periodic = is_periodic
dataset.metadata.lattice_parameters = list(lattice_parameters)
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
dataset.peek = get_file_peek(dataset.file_name)
dataset.info = self.get_dataset_info(dataset.metadata)
@@ -1445,7 +1442,7 @@ class ExtendedXYZ(XYZ):
blocks.append({"number_of_atoms": n_atoms, "comment": comment, "atom_data": atoms})
return blocks
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
super().set_peek(dataset)
dataset.blurb = "Extended " + dataset.blurb
@@ -1453,7 +1450,7 @@ class ExtendedXYZ(XYZ):
class grd(Text):
file_ext = "grd"
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
dataset.peek = get_file_peek(dataset.file_name)
dataset.blurb = "grids for docking"
@@ -1465,7 +1462,7 @@ class grd(Text):
class grdtgz(Binary):
file_ext = "grd.tgz"
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
dataset.peek = "binary data"
dataset.blurb = "compressed grids for docking"
@@ -1497,13 +1494,13 @@ class InChI(Tabular):
no_value=0,
)
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""
Set the number of lines of data in dataset.
"""
dataset.metadata.number_of_molecules = self.count_data_lines(dataset)
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
if dataset.metadata.number_of_molecules == 1:
dataset.blurb = "1 molecule"
@@ -1562,13 +1559,13 @@ class SMILES(Tabular):
no_value=0,
)
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""
Set the number of lines of data in dataset.
"""
dataset.metadata.number_of_molecules = self.count_data_lines(dataset)
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
if dataset.metadata.number_of_molecules == 1:
dataset.blurb = "1 molecule"
@@ -1598,13 +1595,13 @@ class CML(GenericXml):
no_value=0,
)
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""
Set the number of lines of data in dataset.
"""
dataset.metadata.number_of_molecules = count_special_lines(r"^\s* None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
if dataset.metadata.number_of_molecules == 1:
dataset.blurb = "1 molecule"
@@ -1762,7 +1759,7 @@ class GRO(GenericMolFile):
return False
return True
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
dataset.peek = get_file_peek(dataset.file_name)
atom_number = int(dataset.peek.split("\n")[1])
diff --git a/lib/galaxy/datatypes/mothur.py b/lib/galaxy/datatypes/mothur.py
index a82771dcd8f..c798e6fd14e 100644
--- a/lib/galaxy/datatypes/mothur.py
+++ b/lib/galaxy/datatypes/mothur.py
@@ -6,9 +6,12 @@ import re
from typing import (
List,
Optional,
- TYPE_CHECKING,
)
+from galaxy.datatypes._protocols import (
+ DatasetProtocol,
+ HasMetadata,
+)
from galaxy.datatypes.data import Text
from galaxy.datatypes.metadata import MetadataElement
from galaxy.datatypes.sniff import (
@@ -20,9 +23,6 @@ from galaxy.datatypes.sniff import (
from galaxy.datatypes.tabular import Tabular
from galaxy.util import unicodify
-if TYPE_CHECKING:
- from galaxy.model import DatasetInstance
-
log = logging.getLogger(__name__)
@@ -36,7 +36,7 @@ class Otu(Text):
def __init__(self, **kwd):
super().__init__(**kwd)
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""
Set metadata for Otu files.
@@ -126,7 +126,7 @@ class Sabund(Otu):
"""
super().__init__(**kwd)
- def init_meta(self, dataset: "DatasetInstance", copy_from: Optional["DatasetInstance"] = None) -> None:
+ def init_meta(self, dataset: HasMetadata, copy_from: Optional[HasMetadata] = None) -> None:
super().init_meta(dataset, copy_from=copy_from)
def sniff_prefix(self, file_prefix: FilePrefix) -> bool:
@@ -170,10 +170,10 @@ class GroupAbund(Otu):
def __init__(self, **kwd):
super().__init__(**kwd)
- def init_meta(self, dataset: "DatasetInstance", copy_from: Optional["DatasetInstance"] = None) -> None:
+ def init_meta(self, dataset: HasMetadata, copy_from: Optional[HasMetadata] = None) -> None:
super().init_meta(dataset, copy_from=copy_from)
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, skip: Optional[int] = 1, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, skip: Optional[int] = 1, **kwd) -> None:
super().set_meta(dataset, overwrite=overwrite, **kwd)
# See if file starts with header line
@@ -297,7 +297,7 @@ class AlignCheck(Tabular):
self.column_types = ["str", "int", "int", "int", "int", "int", "int", "int"]
self.comment_lines = 1
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
super().set_meta(dataset, overwrite=overwrite, **kwd)
dataset.metadata.column_names = self.column_names
@@ -351,10 +351,10 @@ class DistanceMatrix(Text):
no_value="?",
)
- def init_meta(self, dataset: "DatasetInstance", copy_from: Optional["DatasetInstance"] = None) -> None:
+ def init_meta(self, dataset: HasMetadata, copy_from: Optional[HasMetadata] = None) -> None:
super().init_meta(dataset, copy_from=copy_from)
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, skip: Optional[int] = 0, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, skip: Optional[int] = 0, **kwd) -> None:
super().set_meta(dataset, overwrite=overwrite, skip=skip, **kwd)
headers = iter_headers(dataset.file_name, sep="\t")
@@ -376,7 +376,7 @@ class LowerTriangleDistanceMatrix(DistanceMatrix):
"""Initialize secondary structure map datatype"""
super().__init__(**kwd)
- def init_meta(self, dataset: "DatasetInstance", copy_from: Optional["DatasetInstance"] = None) -> None:
+ def init_meta(self, dataset: HasMetadata, copy_from: Optional[HasMetadata] = None) -> None:
super().init_meta(dataset, copy_from=copy_from)
def sniff_prefix(self, file_prefix: FilePrefix) -> bool:
@@ -441,7 +441,7 @@ class SquareDistanceMatrix(DistanceMatrix):
def __init__(self, **kwd):
super().__init__(**kwd)
- def init_meta(self, dataset: "DatasetInstance", copy_from: Optional["DatasetInstance"] = None) -> None:
+ def init_meta(self, dataset: HasMetadata, copy_from: Optional[HasMetadata] = None) -> None:
super().init_meta(dataset, copy_from=copy_from)
def sniff_prefix(self, file_prefix: FilePrefix) -> bool:
@@ -507,7 +507,7 @@ class PairwiseDistanceMatrix(DistanceMatrix, Tabular):
self.column_names = ["Sequence", "Sequence", "Distance"]
self.column_types = ["str", "str", "float"]
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, skip: Optional[int] = None, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, skip: Optional[int] = None, **kwd) -> None:
super().set_meta(dataset, overwrite=overwrite, skip=skip, **kwd)
def sniff_prefix(self, file_prefix: FilePrefix) -> bool:
@@ -597,7 +597,7 @@ class Group(Tabular):
def set_meta(
self,
- dataset: "DatasetInstance",
+ dataset: DatasetProtocol,
overwrite: bool = True,
skip: Optional[int] = None,
max_data_lines: Optional[int] = None,
@@ -845,7 +845,7 @@ class CountTable(Tabular):
def set_meta(
self,
- dataset: "DatasetInstance",
+ dataset: DatasetProtocol,
overwrite: bool = True,
skip: Optional[int] = 1,
max_data_lines: Optional[int] = None,
@@ -1057,7 +1057,7 @@ class SffFlow(Tabular):
def set_meta(
self,
- dataset: "DatasetInstance",
+ dataset: DatasetProtocol,
overwrite: bool = True,
skip: Optional[int] = 1,
max_data_lines: Optional[int] = None,
@@ -1072,7 +1072,7 @@ class SffFlow(Tabular):
except Exception as e:
log.warning(f"SffFlow set_meta {e}")
- def make_html_table(self, dataset: "DatasetInstance", skipchars: Optional[List] = None, **kwargs) -> str:
+ def make_html_table(self, dataset: DatasetProtocol, skipchars: Optional[List] = None, **kwargs) -> str:
"""Create HTML table, used for displaying peek"""
skipchars = skipchars or []
try:
diff --git a/lib/galaxy/datatypes/msa.py b/lib/galaxy/datatypes/msa.py
index 6dc5f3e5c03..901709c0f16 100644
--- a/lib/galaxy/datatypes/msa.py
+++ b/lib/galaxy/datatypes/msa.py
@@ -6,9 +6,9 @@ from typing import (
Callable,
Dict,
List,
- TYPE_CHECKING,
)
+from galaxy.datatypes._protocols import DatasetProtocol
from galaxy.datatypes.binary import Binary
from galaxy.datatypes.data import (
get_file_peek,
@@ -25,9 +25,6 @@ from galaxy.util import (
unicodify,
)
-if TYPE_CHECKING:
- from galaxy.model import DatasetInstance
-
log = logging.getLogger(__name__)
STOCKHOLM_SEARCH_PATTERN = re.compile(r"#\s+STOCKHOLM\s+1\.0")
@@ -57,7 +54,7 @@ class InfernalCM(Text):
no_value=0,
)
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
dataset.peek = get_file_peek(dataset.file_name)
if dataset.metadata.number_of_models == 1:
@@ -81,7 +78,7 @@ class InfernalCM(Text):
"""
return file_prefix.startswith("INFERNAL")
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""
Set the number of models and the version of CM file in dataset.
"""
@@ -97,7 +94,7 @@ class Hmmer(Text):
edam_data = "data_1364"
edam_format = "format_1370"
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
dataset.peek = get_file_peek(dataset.file_name)
dataset.blurb = "HMMER Database"
@@ -105,7 +102,7 @@ class Hmmer(Text):
dataset.peek = "file does not exist"
dataset.blurb = "file purged from disc"
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
try:
return dataset.peek
except Exception:
@@ -140,7 +137,7 @@ class HmmerPress(Binary):
file_ext = "hmmpress"
composite_type = "basic"
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
"""Set the peek and blurb text."""
if not dataset.dataset.purged:
dataset.peek = "HMMER Binary database"
@@ -149,7 +146,7 @@ class HmmerPress(Binary):
dataset.peek = "file does not exist"
dataset.blurb = "file purged from disk"
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
"""Create HTML content, used for displaying peek."""
try:
return dataset.peek
@@ -184,7 +181,7 @@ class Stockholm_1_0(Text):
no_value=0,
)
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
if dataset.metadata.number_of_models == 1:
dataset.blurb = "1 alignment"
@@ -198,7 +195,7 @@ class Stockholm_1_0(Text):
def sniff_prefix(self, file_prefix: FilePrefix) -> bool:
return file_prefix.search(STOCKHOLM_SEARCH_PATTERN)
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""
Set the number of models in dataset.
@@ -275,7 +272,7 @@ class MauveXmfa(Text):
no_value=0,
)
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
if dataset.metadata.number_of_models == 1:
dataset.blurb = "1 alignment"
@@ -289,7 +286,7 @@ class MauveXmfa(Text):
def sniff_prefix(self, file_prefix: FilePrefix) -> bool:
return file_prefix.startswith("#FormatVersion Mauve1")
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
dataset.metadata.number_of_models = generic_util.count_special_lines(
"^#Sequence([[:digit:]]+)Entry", dataset.file_name
)
diff --git a/lib/galaxy/datatypes/neo4j.py b/lib/galaxy/datatypes/neo4j.py
index 4c53dccb56a..27fede42da8 100644
--- a/lib/galaxy/datatypes/neo4j.py
+++ b/lib/galaxy/datatypes/neo4j.py
@@ -2,18 +2,15 @@
Neo4j Composite Dataset
"""
import logging
-from typing import TYPE_CHECKING
-from galaxy.datatypes.data import (
- Data,
- GeneratePrimaryFileDataset,
+from galaxy.datatypes._protocols import (
+ DatasetProtocol,
+ HasExtraFilesAndMetadata,
)
+from galaxy.datatypes.data import Data
from galaxy.datatypes.images import Html
from galaxy.datatypes.metadata import MetadataElement
-if TYPE_CHECKING:
- from galaxy.model import DatasetInstance
-
gal_Log = logging.getLogger(__name__)
verbose = True
@@ -25,7 +22,7 @@ class Neo4j(Html):
stored in extra files path
"""
- def generate_primary_file(self, dataset: GeneratePrimaryFileDataset) -> str:
+ def generate_primary_file(self, dataset: HasExtraFilesAndMetadata) -> str:
"""
This is called only at upload to write the html file
cannot rename the datasets here - they come with the default unfortunately
@@ -48,7 +45,7 @@ class Neo4j(Html):
"""Returns the mime type of the datatype"""
return "text/html"
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
"""Set the peek and blurb text"""
if not dataset.dataset.purged:
dataset.peek = "Neo4j database (multiple files)"
@@ -57,7 +54,7 @@ class Neo4j(Html):
dataset.peek = "file does not exist"
dataset.blurb = "file purged from disk"
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
"""Create HTML content, used for displaying peek."""
try:
return dataset.peek
diff --git a/lib/galaxy/datatypes/ngsindex.py b/lib/galaxy/datatypes/ngsindex.py
index 198fb994b0f..2da0df85d96 100644
--- a/lib/galaxy/datatypes/ngsindex.py
+++ b/lib/galaxy/datatypes/ngsindex.py
@@ -3,15 +3,14 @@ NGS indexes
"""
import logging
import os
-from typing import TYPE_CHECKING
-from galaxy.datatypes.data import GeneratePrimaryFileDataset
+from galaxy.datatypes._protocols import (
+ DatasetProtocol,
+ HasExtraFilesAndMetadata,
+)
from .metadata import MetadataElement
from .text import Html
-if TYPE_CHECKING:
- from galaxy.model import DatasetInstance
-
log = logging.getLogger(__name__)
@@ -38,14 +37,14 @@ class BowtieIndex(Html):
composite_type = "auto_primary_file"
- def generate_primary_file(self, dataset: GeneratePrimaryFileDataset) -> str:
+ def generate_primary_file(self, dataset: HasExtraFilesAndMetadata) -> str:
"""
This is called only at upload to write the html file
cannot rename the datasets here - they come with the default unfortunately
"""
return "AutoGenerated Primary File for Composite Dataset"
- def regenerate_primary_file(self, dataset: "DatasetInstance") -> None:
+ def regenerate_primary_file(self, dataset: DatasetProtocol) -> None:
"""
cannot do this until we are setting metadata
"""
@@ -62,7 +61,7 @@ class BowtieIndex(Html):
f.write("\n".join(rval))
f.write("\n")
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
dataset.peek = f"Bowtie index file ({dataset.metadata.sequence_space})"
dataset.blurb = f"{dataset.metadata.sequence_space} space"
@@ -70,7 +69,7 @@ class BowtieIndex(Html):
dataset.peek = "file does not exist"
dataset.blurb = "file purged from disk"
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
try:
return dataset.peek
except Exception:
diff --git a/lib/galaxy/datatypes/phylip.py b/lib/galaxy/datatypes/phylip.py
index 551043080cf..5c468fe77eb 100644
--- a/lib/galaxy/datatypes/phylip.py
+++ b/lib/galaxy/datatypes/phylip.py
@@ -10,6 +10,7 @@ Phylip datatype sniffer
from typing import TYPE_CHECKING
from galaxy import util
+from galaxy.datatypes._protocols import DatasetProtocol
from galaxy.datatypes.data import (
get_file_peek,
Text,
@@ -24,8 +25,6 @@ from .metadata import MetadataElement
if TYPE_CHECKING:
from io import StringIO
- from galaxy.model import DatasetInstance
-
@build_sniff_from_prefix
class Phylip(Text):
@@ -39,7 +38,7 @@ class Phylip(Text):
name="sequences", default=0, desc="Number of sequences", readonly=True, visible=False, optional=True, no_value=0
)
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""
Set the number of sequences and the number of data lines in dataset.
"""
@@ -49,7 +48,7 @@ class Phylip(Text):
except Exception:
raise Exception("Header does not correspond to PHYLIP header.")
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
dataset.peek = get_file_peek(dataset.file_name)
if dataset.metadata.sequences:
diff --git a/lib/galaxy/datatypes/plant_tribes.py b/lib/galaxy/datatypes/plant_tribes.py
index a0528fe0e3d..929a648deb5 100644
--- a/lib/galaxy/datatypes/plant_tribes.py
+++ b/lib/galaxy/datatypes/plant_tribes.py
@@ -1,7 +1,7 @@
import logging
import re
-from typing import TYPE_CHECKING
+from galaxy.datatypes._protocols import DatasetProtocol
from galaxy.datatypes.data import (
get_file_peek,
Text,
@@ -15,9 +15,6 @@ from galaxy.datatypes.sniff import (
from galaxy.datatypes.tabular import Tabular
from galaxy.util import nice_size
-if TYPE_CHECKING:
- from galaxy.model import DatasetInstance
-
log = logging.getLogger(__name__)
@@ -25,13 +22,13 @@ log = logging.getLogger(__name__)
class Smat(Text):
file_ext = "smat"
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
try:
return dataset.peek
except Exception:
return f"ESTScan scores matrices ({nice_size(dataset.get_size())})"
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
dataset.peek = get_file_peek(dataset.file_name)
dataset.blurb = "ESTScan scores matrices"
@@ -97,13 +94,13 @@ class PlantTribesKsComponents(Tabular):
no_value=0,
)
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
try:
return dataset.peek
except Exception:
return f"Significant components in the Ks distribution ({nice_size(dataset.get_size())})"
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""
Set the number of significant components in the Ks distribution.
The dataset will always be on the order of less than 10 lines.
@@ -125,7 +122,7 @@ class PlantTribesKsComponents(Tabular):
if len(significant_components) > 0:
dataset.metadata.number_comp = max(significant_components)
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
dataset.peek = get_file_peek(dataset.file_name)
if dataset.metadata.number_comp == 1:
diff --git a/lib/galaxy/datatypes/proteomics.py b/lib/galaxy/datatypes/proteomics.py
index 7bda20361aa..dae3f7c5f81 100644
--- a/lib/galaxy/datatypes/proteomics.py
+++ b/lib/galaxy/datatypes/proteomics.py
@@ -7,15 +7,15 @@ from typing import (
IO,
List,
Optional,
- TYPE_CHECKING,
)
from galaxy.datatypes import data
-from galaxy.datatypes.binary import Binary
-from galaxy.datatypes.data import (
- GeneratePrimaryFileDataset,
- Text,
+from galaxy.datatypes._protocols import (
+ DatasetProtocol,
+ HasExtraFilesAndMetadata,
)
+from galaxy.datatypes.binary import Binary
+from galaxy.datatypes.data import Text
from galaxy.datatypes.sequence import Sequence
from galaxy.datatypes.sniff import (
build_sniff_from_prefix,
@@ -28,9 +28,6 @@ from galaxy.datatypes.tabular import (
from galaxy.datatypes.xml import GenericXml
from galaxy.util import nice_size
-if TYPE_CHECKING:
- from galaxy.model import DatasetInstance
-
log = logging.getLogger(__name__)
@@ -58,7 +55,7 @@ class Wiff(Binary):
is_binary=True,
)
- def generate_primary_file(self, dataset: GeneratePrimaryFileDataset) -> str:
+ def generate_primary_file(self, dataset: HasExtraFilesAndMetadata) -> str:
rval = ["Wiff Composite Dataset "]
rval.append("This composite dataset is composed of the following files:")
for composite_name, composite_file in self.get_composite_files(dataset=dataset).items():
@@ -100,7 +97,7 @@ class Wiff2(Binary):
is_binary=True,
)
- def generate_primary_file(self, dataset: GeneratePrimaryFileDataset) -> str:
+ def generate_primary_file(self, dataset: HasExtraFilesAndMetadata) -> str:
rval = ["Wiff2 Composite Dataset "]
rval.append("This composite dataset is composed of the following files:")
for composite_name, composite_file in self.get_composite_files(dataset=dataset).items():
@@ -148,7 +145,7 @@ class MzTab(Text):
def __init__(self, **kwd):
super().__init__(**kwd)
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
"""Set the peek and blurb text"""
if not dataset.dataset.purged:
dataset.peek = data.get_file_peek(dataset.file_name)
@@ -199,7 +196,7 @@ class MzTab2(MzTab):
def __init__(self, **kwd):
super().__init__(**kwd)
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
"""Set the peek and blurb text"""
if not dataset.dataset.purged:
dataset.peek = data.get_file_peek(dataset.file_name)
@@ -244,7 +241,7 @@ class Kroenik(Tabular):
"Modifications",
]
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
"""Returns formated html of peek"""
return self.make_html_table(dataset, column_names=self.column_names)
@@ -283,7 +280,7 @@ class PepList(Tabular):
super().__init__(**kwd)
self.column_names = ["m/z", "rt(min)", "snr", "charge", "intensity"]
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
"""Returns formated html of peek"""
return self.make_html_table(dataset, column_names=self.column_names)
@@ -320,7 +317,7 @@ class PSMS(Tabular):
super().__init__(**kwd)
self.column_names = ["PSMId", "score", "q-value", "posterior_error_prob", "peptide", "proteinIds"]
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
"""Returns formated html of peek"""
return self.make_html_table(dataset, column_names=self.column_names)
@@ -380,7 +377,7 @@ class PepXmlReport(Tabular):
"Interprophet Probability",
]
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
"""Returns formated html of peek"""
return self.make_html_table(dataset, column_names=self.column_names)
@@ -421,7 +418,7 @@ class ProtXmlReport(Tabular):
"Is Evidence?",
]
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
"""Returns formated html of peek"""
return self.make_html_table(dataset, column_names=self.column_names)
@@ -436,7 +433,7 @@ class Dta(TabularData):
file_ext = "dta"
comment_lines = 0
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
column_types = []
data_row: List = []
data_lines = 0
@@ -509,7 +506,7 @@ class Dta2d(TabularData):
return False
return True
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
data_lines = 0
delim = None
if dataset.has_data():
@@ -645,7 +642,7 @@ class Edta(TabularData):
line[idx] += str(idx // 4)
return line
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
data_lines = 0
delim = None
tpe = None
@@ -719,7 +716,7 @@ class ProteomicsXml(GenericXml):
pattern = r"<(\w*:)?%s" % self.root
return re.search(pattern, line) is not None
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
"""Set the peek and blurb text"""
if not dataset.dataset.purged:
dataset.peek = data.get_file_peek(dataset.file_name)
@@ -888,7 +885,7 @@ class Mgf(Text):
edam_format = "format_3651"
file_ext = "mgf"
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
"""Set the peek and blurb text"""
if not dataset.dataset.purged:
dataset.peek = data.get_file_peek(dataset.file_name)
@@ -918,7 +915,7 @@ class MascotDat(Text):
edam_format = "format_3713"
file_ext = "mascotdat"
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
"""Set the peek and blurb text"""
if not dataset.dataset.purged:
dataset.peek = data.get_file_peek(dataset.file_name)
@@ -962,7 +959,7 @@ class ThermoRAW(Binary):
except Exception:
return False
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
dataset.peek = "Thermo Finnigan RAW file"
dataset.blurb = nice_size(dataset.get_size())
@@ -970,7 +967,7 @@ class ThermoRAW(Binary):
dataset.peek = "file does not exist"
dataset.blurb = "file purged from disk"
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
try:
return dataset.peek
except Exception:
@@ -1004,7 +1001,7 @@ class SPLibNoIndex(Text):
file_ext = "splib_noindex"
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
"""Set the peek and blurb text"""
if not dataset.dataset.purged:
dataset.peek = data.get_file_peek(dataset.file_name)
@@ -1029,7 +1026,7 @@ class SPLib(Msp):
self.add_composite_file("library.spidx", description="Spectrum index", is_binary=False)
self.add_composite_file("library.pepidx", description="Peptide index", is_binary=False)
- def generate_primary_file(self, dataset: GeneratePrimaryFileDataset) -> str:
+ def generate_primary_file(self, dataset: HasExtraFilesAndMetadata) -> str:
rval = ["Spectral Library Composite Dataset "]
rval.append("This composite dataset is composed of the following files:")
for composite_name, composite_file in self.get_composite_files(dataset=dataset).items():
@@ -1046,7 +1043,7 @@ class SPLib(Msp):
rval.append("
")
return "\n".join(rval)
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
"""Set the peek and blurb text"""
if not dataset.dataset.purged:
dataset.peek = data.get_file_peek(dataset.file_name)
@@ -1118,7 +1115,7 @@ class ImzML(Binary):
self.add_composite_file("ibd", description="The mass spectral data component.", is_binary=True)
- def generate_primary_file(self, dataset: GeneratePrimaryFileDataset) -> str:
+ def generate_primary_file(self, dataset: HasExtraFilesAndMetadata) -> str:
rval = ["imzML Composite Dataset "]
rval.append("This composite dataset is composed of the following files:")
for composite_name, composite_file in self.get_composite_files(dataset=dataset).items():
diff --git a/lib/galaxy/datatypes/qiime2.py b/lib/galaxy/datatypes/qiime2.py
index 4a2ed5cc9dc..987ca9576a6 100644
--- a/lib/galaxy/datatypes/qiime2.py
+++ b/lib/galaxy/datatypes/qiime2.py
@@ -7,11 +7,14 @@ from typing import (
Dict,
List,
Optional,
- TYPE_CHECKING,
)
import yaml
+from galaxy.datatypes._protocols import (
+ DatasetProtocol,
+ HasMetadata,
+)
from galaxy.datatypes.binary import CompressedZipArchive
from galaxy.datatypes.metadata import MetadataElement
from galaxy.datatypes.sniff import (
@@ -20,9 +23,6 @@ from galaxy.datatypes.sniff import (
)
from galaxy.datatypes.tabular import Tabular
-if TYPE_CHECKING:
- from galaxy.model import DatasetInstance
-
class _QIIME2ResultBase(CompressedZipArchive):
"""Base class for QIIME2Artifact and QIIME2Visualization"""
@@ -33,7 +33,7 @@ class _QIIME2ResultBase(CompressedZipArchive):
MetadataElement(name="format", optional=True, no_value="", readonly=True)
MetadataElement(name="version", readonly=True)
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
metadata = _get_metadata_from_archive(dataset.file_name)
for key, value in metadata.items():
if value:
@@ -41,7 +41,7 @@ class _QIIME2ResultBase(CompressedZipArchive):
dataset.metadata.semantic_type_simple = _strip_properties(dataset.metadata.semantic_type)
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if dataset.metadata.semantic_type == "Visualization":
dataset.blurb = "QIIME 2 Visualization"
else:
@@ -49,7 +49,7 @@ class _QIIME2ResultBase(CompressedZipArchive):
dataset.peek = "\n".join(map(": ".join, self._peek(dataset)))
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
def make_row(item):
return "%s %s " % tuple(html.escape(x) for x in item)
@@ -59,7 +59,7 @@ class _QIIME2ResultBase(CompressedZipArchive):
return "".join(table)
- def _peek(self, dataset: "DatasetInstance", simple: bool = False) -> List:
+ def _peek(self, dataset: HasMetadata, simple: bool = False) -> List:
peek = [("Type", dataset.metadata.semantic_type), ("UUID", dataset.metadata.uuid)]
if not simple:
if dataset.metadata.semantic_type != "Visualization":
@@ -115,7 +115,7 @@ class QIIME2Metadata(Tabular):
def get_column_names(self, first_line: str) -> Optional[List[str]]:
return first_line.strip().split("\t")
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""
Let Galaxy's Tabular format handle most of this. We will just jump
in at the last minute to (potentially) override some column types.
diff --git a/lib/galaxy/datatypes/qualityscore.py b/lib/galaxy/datatypes/qualityscore.py
index 49ddc9c3eff..8349fe9a070 100644
--- a/lib/galaxy/datatypes/qualityscore.py
+++ b/lib/galaxy/datatypes/qualityscore.py
@@ -2,17 +2,14 @@
Qualityscore class
"""
import logging
-from typing import TYPE_CHECKING
+from galaxy.datatypes._protocols import DatasetProtocol
from galaxy.datatypes.sniff import (
build_sniff_from_prefix,
FilePrefix,
)
from . import data
-if TYPE_CHECKING:
- from galaxy.model import DatasetInstance
-
log = logging.getLogger(__name__)
@@ -69,7 +66,7 @@ class QualityScoreSOLiD(QualityScore):
return False
return goodblock > 0
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
if self.max_optional_metadata_filesize >= 0 and dataset.get_size() > self.max_optional_metadata_filesize:
dataset.metadata.data_lines = None
return
diff --git a/lib/galaxy/datatypes/registry.py b/lib/galaxy/datatypes/registry.py
index 746761d9a74..b6bc708cfc2 100644
--- a/lib/galaxy/datatypes/registry.py
+++ b/lib/galaxy/datatypes/registry.py
@@ -9,16 +9,18 @@ import pkgutil
from pathlib import Path
from string import Template
from typing import (
+ cast,
Dict,
List,
Optional,
Tuple,
- TYPE_CHECKING,
+ Union,
)
import yaml
import galaxy.util
+from galaxy.datatypes._protocols import DatasetProtocol
from galaxy.tool_util.edam_util import load_edam_tree
from galaxy.util import RW_R__R__
from galaxy.util.bunch import Bunch
@@ -37,9 +39,6 @@ from . import (
)
from .display_applications.application import DisplayApplication
-if TYPE_CHECKING:
- from galaxy.model import DatasetInstance
-
class ConfigurationError(Exception):
pass
@@ -832,8 +831,8 @@ class Registry:
return None
def find_conversion_destination_for_dataset_by_extensions(
- self, dataset_or_ext, accepted_formats: List[str], converter_safe: bool = True
- ) -> Tuple[bool, Optional[str], Optional["DatasetInstance"]]:
+ self, dataset_or_ext: Union[str, DatasetProtocol], accepted_formats: List[str], converter_safe: bool = True
+ ) -> Tuple[bool, Optional[str], Optional[DatasetProtocol]]:
"""
returns (direct_match, converted_ext, converted_dataset)
- direct match is True iff no the data set already has an accepted format
@@ -841,7 +840,7 @@ class Registry:
"""
if hasattr(dataset_or_ext, "ext"):
ext = dataset_or_ext.ext
- dataset = dataset_or_ext
+ dataset = cast(DatasetProtocol, dataset_or_ext)
else:
ext = dataset_or_ext
dataset = None
@@ -854,7 +853,7 @@ class Registry:
convert_ext_datatype = self.get_datatype_by_extension(convert_ext)
if convert_ext_datatype is None:
self.log.warning(
- f"Datatype class not found for extension '{convert_ext}', which is used as target for conversion from datatype '{dataset.ext}'"
+ f"Datatype class not found for extension '{convert_ext}', which is used as target for conversion from datatype '{ext}'"
)
elif convert_ext_datatype.matches_any(accepted_formats):
converted_dataset = dataset and dataset.get_converted_files_by_type(convert_ext)
diff --git a/lib/galaxy/datatypes/sequence.py b/lib/galaxy/datatypes/sequence.py
index 040d68ede75..7eef083dfcb 100644
--- a/lib/galaxy/datatypes/sequence.py
+++ b/lib/galaxy/datatypes/sequence.py
@@ -17,7 +17,6 @@ from typing import (
Iterable,
List,
Optional,
- TYPE_CHECKING,
)
import bx.align.maf
@@ -25,6 +24,11 @@ from markupsafe import escape
from galaxy import util
from galaxy.datatypes import metadata
+from galaxy.datatypes._protocols import (
+ DatasetHasHidProtocol,
+ DatasetProtocol,
+ HasMetadata,
+)
from galaxy.datatypes.binary import Binary
from galaxy.datatypes.data import DatatypeValidation
from galaxy.datatypes.metadata import (
@@ -45,12 +49,6 @@ from galaxy.util.checkers import is_gzip
from galaxy.util.image_util import check_image_type
from . import data
-if TYPE_CHECKING:
- from galaxy.model import (
- DatasetInstance,
- HistoryDatasetAssociation,
- )
-
log = logging.getLogger(__name__)
@@ -71,7 +69,7 @@ class SequenceSplitLocations(data.Text):
file_ext = "fqtoc"
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
try:
parsed_data = json.load(open(dataset.file_name))
@@ -108,7 +106,7 @@ class Sequence(data.Text):
name="sequences", default=0, desc="Number of sequences", readonly=True, visible=False, optional=True, no_value=0
)
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""
Set the number of sequences and the number of data lines in dataset.
"""
@@ -128,7 +126,7 @@ class Sequence(data.Text):
dataset.metadata.data_lines = data_lines
dataset.metadata.sequences = sequences
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
dataset.peek = data.get_file_peek(dataset.file_name)
if dataset.metadata.sequences:
@@ -564,7 +562,7 @@ class csFasta(Sequence):
return False
return False
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
if self.max_optional_metadata_filesize >= 0 and dataset.get_size() > self.max_optional_metadata_filesize:
dataset.metadata.data_lines = None
dataset.metadata.sequences = None
@@ -641,7 +639,7 @@ class Fastg(Sequence):
break # we found a non-empty line, but it's not a header
return False
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
with open(dataset.file_name) as fh:
for i, line in enumerate(fh):
if not line:
@@ -666,7 +664,7 @@ class Fastg(Sequence):
return
return Sequence.set_meta(self, dataset, overwrite=overwrite, **kwd)
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
dataset.peek = data.get_file_peek(dataset.file_name)
if dataset.metadata.sequences:
@@ -690,7 +688,7 @@ class BaseFastq(Sequence):
file_ext = "fastq"
bases_regexp = re.compile(r"^[NGTAC 0123\.]*$", re.IGNORECASE)
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""
Set the number of sequences and the number of data lines
in dataset.
@@ -771,7 +769,7 @@ class BaseFastq(Sequence):
def display_data(
self,
trans,
- dataset: "HistoryDatasetAssociation",
+ dataset: DatasetHasHidProtocol,
preview: bool = False,
filename: Optional[str] = None,
to_ext: Optional[str] = None,
@@ -874,7 +872,7 @@ class BaseFastq(Sequence):
return True
return False
- def validate(self, dataset: "DatasetInstance", **kwd) -> DatatypeValidation:
+ def validate(self, dataset: DatasetProtocol, **kwd) -> DatatypeValidation:
headers = iter_headers(dataset.file_name, sep="\n", count=-1)
# check to see if the base qualities match
if not self.quality_check(headers):
@@ -1011,11 +1009,11 @@ class Maf(Alignment):
optional=True,
)
- def init_meta(self, dataset: "DatasetInstance", copy_from: Optional["DatasetInstance"] = None) -> None:
+ def init_meta(self, dataset: HasMetadata, copy_from: Optional[HasMetadata] = None) -> None:
Alignment.init_meta(self, dataset, copy_from=copy_from)
def set_meta(
- self, dataset: "DatasetInstance", overwrite: bool = True, metadata_tmp_files_dir: Optional[str] = None, **kwd
+ self, dataset: DatasetProtocol, overwrite: bool = True, metadata_tmp_files_dir: Optional[str] = None, **kwd
) -> None:
"""
Parses and sets species, chromosomes, index from MAF file.
@@ -1049,7 +1047,7 @@ class Maf(Alignment):
indexes.write(open(index_file.file_name, "wb"))
dataset.metadata.maf_index = index_file
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
# The file must exist on disk for the get_file_peek() method
dataset.peek = data.get_file_peek(dataset.file_name)
@@ -1063,11 +1061,11 @@ class Maf(Alignment):
dataset.peek = "file does not exist"
dataset.blurb = "file purged from disk"
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
"""Returns formated html of peek"""
return self.make_html_table(dataset)
- def make_html_table(self, dataset: "DatasetInstance", skipchars: Optional[List] = None) -> str:
+ def make_html_table(self, dataset: DatasetProtocol, skipchars: Optional[List] = None) -> str:
"""Create HTML table, used for displaying peek"""
skipchars = skipchars or []
try:
@@ -1142,7 +1140,7 @@ class MafCustomTrack(data.Text):
name="vp_end", default="100", desc="Viewport End", readonly=True, optional=True, visible=False, no_value=""
)
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""
Parses and sets viewport metadata from MAF file.
"""
@@ -1275,7 +1273,7 @@ class RNADotPlotMatrix(data.Data):
edam_format = "format_3466"
file_ext = "rna_eps"
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
dataset.peek = "RNA Dot Plot format (Postscript derivative)"
dataset.blurb = nice_size(dataset.get_size())
@@ -1313,7 +1311,7 @@ class DotBracket(Sequence):
sequence_regexp = re.compile(r"^[ACGTURYKMSWBDHVN]+$", re.I)
structure_regexp = re.compile(r"^[\(\)\.\[\]{}]+$")
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""
Set the number of sequences and the number of data lines
in dataset.
diff --git a/lib/galaxy/datatypes/spaln.py b/lib/galaxy/datatypes/spaln.py
index 6ce094e341d..1bc7f0fd950 100644
--- a/lib/galaxy/datatypes/spaln.py
+++ b/lib/galaxy/datatypes/spaln.py
@@ -9,22 +9,17 @@ from typing import (
Dict,
List,
Optional,
- TYPE_CHECKING,
)
-from galaxy.datatypes.data import (
- Data,
- GeneratePrimaryFileDataset,
+from galaxy.datatypes._protocols import (
+ DatasetHasHidProtocol,
+ DatasetProtocol,
+ HasExtraFilesAndMetadata,
)
+from galaxy.datatypes.data import Data
from galaxy.datatypes.metadata import MetadataElement
from galaxy.util import smart_str
-if TYPE_CHECKING:
- from galaxy.model import (
- DatasetInstance,
- HistoryDatasetAssociation,
- )
-
log = logging.getLogger(__name__)
verbose = True
@@ -68,7 +63,7 @@ class _SpalnDb(Data):
substitute_name_with_metadata="spalndb_name",
)
- def generate_primary_file(self, dataset: GeneratePrimaryFileDataset) -> str:
+ def generate_primary_file(self, dataset: HasExtraFilesAndMetadata) -> str:
rval = ["Spaln Database "]
rval.append("This composite dataset is composed of the following files:")
for composite_name, composite_file in self.get_composite_files(dataset=dataset).items():
@@ -84,7 +79,7 @@ class _SpalnDb(Data):
rval.append("
")
return "\n".join(rval)
- def regenerate_primary_file(self, dataset: "DatasetInstance") -> None:
+ def regenerate_primary_file(self, dataset: DatasetProtocol) -> None:
"""
cannot do this until we are setting metadata
"""
@@ -103,7 +98,7 @@ class _SpalnDb(Data):
f.write("\n".join(rval))
f.write("\n")
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
"""Set the peek and blurb text."""
if not dataset.dataset.purged:
dataset.peek = "spaln database (multiple files)"
@@ -112,7 +107,7 @@ class _SpalnDb(Data):
dataset.peek = "file does not exist"
dataset.blurb = "file purged from disk"
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
"""Create HTML content, used for displaying peek."""
try:
return dataset.peek
@@ -122,7 +117,7 @@ class _SpalnDb(Data):
def display_data(
self,
trans,
- dataset: "HistoryDatasetAssociation",
+ dataset: DatasetHasHidProtocol,
preview: bool = False,
filename: Optional[str] = None,
to_ext: Optional[str] = None,
@@ -183,7 +178,7 @@ class _SpalnDb(Data):
return None
raise NotImplementedError("Can't split spaln database")
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
super().set_meta(dataset, overwrite=overwrite, **kwd)
efp = dataset.extra_files_path
for filename in os.listdir(efp):
diff --git a/lib/galaxy/datatypes/speech.py b/lib/galaxy/datatypes/speech.py
index 559cffee5d4..5f22a4da077 100644
--- a/lib/galaxy/datatypes/speech.py
+++ b/lib/galaxy/datatypes/speech.py
@@ -1,5 +1,4 @@
-from typing import TYPE_CHECKING
-
+from galaxy.datatypes._protocols import DatasetProtocol
from galaxy.datatypes.data import Text
from galaxy.datatypes.metadata import (
ListParameter,
@@ -7,9 +6,6 @@ from galaxy.datatypes.metadata import (
)
from galaxy.datatypes.sniff import get_headers
-if TYPE_CHECKING:
- from galaxy.model import DatasetInstance
-
class TextGrid(Text):
"""Praat Textgrid file for speech annotations
@@ -102,7 +98,7 @@ class BPF(Text):
"SAO",
]
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""Set the metadata for this dataset from the file contents"""
types = set()
with open(dataset.dataset.file_name) as fd:
diff --git a/lib/galaxy/datatypes/tabular.py b/lib/galaxy/datatypes/tabular.py
index 442cff28e59..3cfbacc43f1 100644
--- a/lib/galaxy/datatypes/tabular.py
+++ b/lib/galaxy/datatypes/tabular.py
@@ -17,7 +17,6 @@ from typing import (
Dict,
List,
Optional,
- TYPE_CHECKING,
Union,
)
@@ -30,6 +29,12 @@ from galaxy.datatypes import (
data,
metadata,
)
+from galaxy.datatypes._protocols import (
+ DatasetHasHidProtocol,
+ DatasetProtocol,
+ HasFileName,
+ HasMetadata,
+)
from galaxy.datatypes.binary import _BamOrSam
from galaxy.datatypes.data import (
DatatypeValidation,
@@ -71,12 +76,6 @@ from galaxy.util.markdown import (
)
from . import dataproviders
-if TYPE_CHECKING:
- from galaxy.model import (
- DatasetInstance,
- HistoryDatasetAssociation,
- )
-
log = logging.getLogger(__name__)
MAX_DATA_LINES = 100000
@@ -122,16 +121,16 @@ class TabularData(Text):
)
@abc.abstractmethod
- def set_meta(self, dataset: "DatasetInstance", *, overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, *, overwrite: bool = True, **kwd) -> None:
raise NotImplementedError
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
kwd.setdefault("line_wrap", False)
super().set_peek(dataset, **kwd)
if dataset.metadata.comment_lines:
dataset.blurb = f"{dataset.blurb}, {util.commaify(str(dataset.metadata.comment_lines))} comments"
- def displayable(self, dataset: "DatasetInstance"):
+ def displayable(self, dataset: DatasetProtocol) -> bool:
try:
return (
not dataset.dataset.purged
@@ -143,7 +142,7 @@ class TabularData(Text):
except Exception:
return False
- def get_chunk(self, trans, dataset: "DatasetInstance", offset: int = 0, ck_size: Optional[int] = None) -> str:
+ def get_chunk(self, trans, dataset: HasFileName, offset: int = 0, ck_size: Optional[int] = None) -> str:
ck_data, last_read = self._read_chunk(trans, dataset, offset, ck_size)
return dumps(
{
@@ -153,7 +152,7 @@ class TabularData(Text):
}
)
- def _read_chunk(self, trans, dataset: "DatasetInstance", offset: int, ck_size: Optional[int] = None):
+ def _read_chunk(self, trans, dataset: HasFileName, offset: int, ck_size: Optional[int] = None):
with compression_utils.get_fileobj(dataset.file_name) as f:
f.seek(offset)
ck_data = f.read(ck_size or trans.app.config.display_chunk_size)
@@ -168,7 +167,7 @@ class TabularData(Text):
def display_data(
self,
trans,
- dataset: "HistoryDatasetAssociation",
+ dataset: DatasetHasHidProtocol,
preview: bool = False,
filename: Optional[str] = None,
to_ext: Optional[str] = None,
@@ -225,7 +224,7 @@ class TabularData(Text):
headers,
)
- def display_as_markdown(self, dataset_instance: "DatasetInstance") -> str:
+ def display_as_markdown(self, dataset_instance: DatasetProtocol) -> str:
with open(dataset_instance.file_name) as f:
contents = f.read(data.DEFAULT_MAX_PEEK_SIZE)
markdown = self.make_html_table(dataset_instance, peek=contents)
@@ -233,7 +232,7 @@ class TabularData(Text):
markdown += indicate_data_truncated()
return pre_formatted_contents(markdown)
- def make_html_table(self, dataset: "DatasetInstance", **kwargs) -> str:
+ def make_html_table(self, dataset: DatasetProtocol, **kwargs) -> str:
"""Create HTML table, used for displaying peek"""
try:
out = ['']
@@ -246,7 +245,7 @@ class TabularData(Text):
def make_html_peek_header(
self,
- dataset: "DatasetInstance",
+ dataset: DatasetProtocol,
skipchars: Optional[List] = None,
column_names: Optional[List] = None,
column_number_format: str = "%s",
@@ -300,7 +299,7 @@ class TabularData(Text):
raise Exception(f"Can't create peek header: {util.unicodify(exc)}")
return "".join(out)
- def make_html_peek_rows(self, dataset: "DatasetInstance", skipchars: Optional[List] = None, **kwargs) -> str:
+ def make_html_peek_rows(self, dataset: DatasetProtocol, skipchars: Optional[List] = None, **kwargs) -> str:
if skipchars is None:
skipchars = []
out = []
@@ -337,7 +336,7 @@ class TabularData(Text):
raise Exception(f"Can't create peek rows: {util.unicodify(exc)}")
return "".join(out)
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
"""Returns formatted html of peek"""
return self.make_html_table(dataset)
@@ -373,27 +372,27 @@ class TabularData(Text):
# ------------- Dataproviders
@dataproviders.decorators.dataprovider_factory("column", ColumnarDataProvider.settings)
- def column_dataprovider(self, dataset: "DatasetInstance", **settings) -> ColumnarDataProvider:
+ def column_dataprovider(self, dataset: DatasetProtocol, **settings) -> ColumnarDataProvider:
"""Uses column settings that are passed in"""
dataset_source = DatasetDataProvider(dataset)
delimiter = dataset.metadata.delimiter
return ColumnarDataProvider(dataset_source, deliminator=delimiter, **settings)
@dataproviders.decorators.dataprovider_factory("dataset-column", ColumnarDataProvider.settings)
- def dataset_column_dataprovider(self, dataset: "DatasetInstance", **settings) -> DatasetColumnarDataProvider:
+ def dataset_column_dataprovider(self, dataset: DatasetProtocol, **settings) -> DatasetColumnarDataProvider:
"""Attempts to get column settings from dataset.metadata"""
delimiter = dataset.metadata.delimiter
return DatasetColumnarDataProvider(dataset, deliminator=delimiter, **settings)
@dataproviders.decorators.dataprovider_factory("dict", DictDataProvider.settings)
- def dict_dataprovider(self, dataset: "DatasetInstance", **settings) -> DictDataProvider:
+ def dict_dataprovider(self, dataset: DatasetProtocol, **settings) -> DictDataProvider:
"""Uses column settings that are passed in"""
dataset_source = DatasetDataProvider(dataset)
delimiter = dataset.metadata.delimiter
return DictDataProvider(dataset_source, deliminator=delimiter, **settings)
@dataproviders.decorators.dataprovider_factory("dataset-dict", DictDataProvider.settings)
- def dataset_dict_dataprovider(self, dataset: "DatasetInstance", **settings) -> DatasetDictDataProvider:
+ def dataset_dict_dataprovider(self, dataset: DatasetProtocol, **settings) -> DatasetDictDataProvider:
"""Attempts to get column settings from dataset.metadata"""
delimiter = dataset.metadata.delimiter
return DatasetDictDataProvider(dataset, deliminator=delimiter, **settings)
@@ -410,7 +409,7 @@ class Tabular(TabularData):
def set_meta(
self,
- dataset: "DatasetInstance",
+ dataset: DatasetProtocol,
*,
overwrite: bool = True,
skip: Optional[int] = None,
@@ -579,10 +578,10 @@ class Tabular(TabularData):
if column_names is not None:
dataset.metadata.column_names = column_names
- def as_gbrowse_display_file(self, dataset: "DatasetInstance", **kwd) -> Union[FileObjType, str]:
+ def as_gbrowse_display_file(self, dataset: HasFileName, **kwd) -> Union[FileObjType, str]:
return open(dataset.file_name, "rb")
- def as_ucsc_display_file(self, dataset: "DatasetInstance", **kwd) -> Union[FileObjType, str]:
+ def as_ucsc_display_file(self, dataset: DatasetProtocol, **kwd) -> Union[FileObjType, str]:
return open(dataset.file_name, "rb")
@@ -592,7 +591,7 @@ class SraManifest(Tabular):
file_ext = "sra_manifest.tabular"
data_line_offset = 1
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
super().set_meta(dataset, overwrite=overwrite, **kwd)
dataset.metadata.comment_lines = 1
@@ -633,7 +632,7 @@ class Taxonomy(Tabular):
"Subspecies",
]
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
"""Returns formated html of peek"""
return self.make_html_table(dataset, column_names=self.column_names)
@@ -724,7 +723,7 @@ class Sam(Tabular, _BamOrSam):
"OPT",
]
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
"""Returns formated html of peek"""
return self.make_html_table(dataset, column_names=self.column_names)
@@ -782,7 +781,7 @@ class Sam(Tabular, _BamOrSam):
def set_meta(
self,
- dataset: "DatasetInstance",
+ dataset: DatasetProtocol,
overwrite: bool = True,
skip: Optional[int] = None,
max_data_lines: Optional[int] = 5,
@@ -864,55 +863,55 @@ class Sam(Tabular, _BamOrSam):
# sam does not use '#' to indicate comments/headers - we need to strip out those headers from the std. providers
# TODO:?? seems like there should be an easier way to do this - metadata.comment_char?
@dataproviders.decorators.dataprovider_factory("line", FilteredLineDataProvider.settings)
- def line_dataprovider(self, dataset: "DatasetInstance", **settings) -> FilteredLineDataProvider:
+ def line_dataprovider(self, dataset: DatasetProtocol, **settings) -> FilteredLineDataProvider:
settings["comment_char"] = "@"
return super().line_dataprovider(dataset, **settings)
@dataproviders.decorators.dataprovider_factory("regex-line", RegexLineDataProvider.settings)
- def regex_line_dataprovider(self, dataset: "DatasetInstance", **settings) -> RegexLineDataProvider:
+ def regex_line_dataprovider(self, dataset: DatasetProtocol, **settings) -> RegexLineDataProvider:
settings["comment_char"] = "@"
return super().regex_line_dataprovider(dataset, **settings)
@dataproviders.decorators.dataprovider_factory("column", ColumnarDataProvider.settings)
- def column_dataprovider(self, dataset: "DatasetInstance", **settings) -> ColumnarDataProvider:
+ def column_dataprovider(self, dataset: DatasetProtocol, **settings) -> ColumnarDataProvider:
settings["comment_char"] = "@"
return super().column_dataprovider(dataset, **settings)
@dataproviders.decorators.dataprovider_factory("dataset-column", ColumnarDataProvider.settings)
- def dataset_column_dataprovider(self, dataset: "DatasetInstance", **settings) -> DatasetColumnarDataProvider:
+ def dataset_column_dataprovider(self, dataset: DatasetProtocol, **settings) -> DatasetColumnarDataProvider:
settings["comment_char"] = "@"
return super().dataset_column_dataprovider(dataset, **settings)
@dataproviders.decorators.dataprovider_factory("dict", DictDataProvider.settings)
- def dict_dataprovider(self, dataset: "DatasetInstance", **settings) -> DictDataProvider:
+ def dict_dataprovider(self, dataset: DatasetProtocol, **settings) -> DictDataProvider:
settings["comment_char"] = "@"
return super().dict_dataprovider(dataset, **settings)
@dataproviders.decorators.dataprovider_factory("dataset-dict", DictDataProvider.settings)
- def dataset_dict_dataprovider(self, dataset: "DatasetInstance", **settings) -> DatasetDictDataProvider:
+ def dataset_dict_dataprovider(self, dataset: DatasetProtocol, **settings) -> DatasetDictDataProvider:
settings["comment_char"] = "@"
return super().dataset_dict_dataprovider(dataset, **settings)
@dataproviders.decorators.dataprovider_factory("header", RegexLineDataProvider.settings)
- def header_dataprovider(self, dataset: "DatasetInstance", **settings) -> RegexLineDataProvider:
+ def header_dataprovider(self, dataset: DatasetProtocol, **settings) -> RegexLineDataProvider:
dataset_source = DatasetDataProvider(dataset)
headers_source = RegexLineDataProvider(dataset_source, regex_list=["^@"])
return RegexLineDataProvider(headers_source, **settings)
@dataproviders.decorators.dataprovider_factory("id-seq-qual", dict_dataprovider.settings)
- def id_seq_qual_dataprovider(self, dataset: "DatasetInstance", **settings) -> DictDataProvider:
+ def id_seq_qual_dataprovider(self, dataset: DatasetProtocol, **settings) -> DictDataProvider:
# provided as an example of a specified column dict (w/o metadata)
settings["indeces"] = [0, 9, 10]
settings["column_names"] = ["id", "seq", "qual"]
return self.dict_dataprovider(dataset, **settings)
@dataproviders.decorators.dataprovider_factory("genomic-region", GenomicRegionDataProvider.settings)
- def genomic_region_dataprovider(self, dataset: "DatasetInstance", **settings) -> GenomicRegionDataProvider:
+ def genomic_region_dataprovider(self, dataset: DatasetProtocol, **settings) -> GenomicRegionDataProvider:
settings["comment_char"] = "@"
return GenomicRegionDataProvider(dataset, 2, 3, 3, **settings)
@dataproviders.decorators.dataprovider_factory("genomic-region-dict", GenomicRegionDataProvider.settings)
- def genomic_region_dict_dataprovider(self, dataset: "DatasetInstance", **settings) -> GenomicRegionDataProvider:
+ def genomic_region_dict_dataprovider(self, dataset: DatasetProtocol, **settings) -> GenomicRegionDataProvider:
settings["comment_char"] = "@"
return GenomicRegionDataProvider(dataset, 2, 3, 3, True, **settings)
@@ -937,10 +936,10 @@ class Pileup(Tabular):
MetadataElement(name="endCol", default=2, desc="End column", param=metadata.ColumnParameter)
MetadataElement(name="baseCol", default=3, desc="Reference base column", param=metadata.ColumnParameter)
- def init_meta(self, dataset: "DatasetInstance", copy_from: Optional["DatasetInstance"] = None) -> None:
+ def init_meta(self, dataset: HasMetadata, copy_from: Optional[HasMetadata] = None) -> None:
super().init_meta(dataset, copy_from=copy_from)
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
"""Returns formated html of peek"""
return self.make_html_table(
dataset, column_parameter_alias={"chromCol": "Chrom", "startCol": "Start", "baseCol": "Base"}
@@ -993,11 +992,11 @@ class Pileup(Tabular):
# Dataproviders
@dataproviders.decorators.dataprovider_factory("genomic-region", GenomicRegionDataProvider.settings)
- def genomic_region_dataprovider(self, dataset: "DatasetInstance", **settings) -> GenomicRegionDataProvider:
+ def genomic_region_dataprovider(self, dataset: DatasetProtocol, **settings) -> GenomicRegionDataProvider:
return GenomicRegionDataProvider(dataset, **settings)
@dataproviders.decorators.dataprovider_factory("genomic-region-dict", GenomicRegionDataProvider.settings)
- def genomic_region_dict_dataprovider(self, dataset: "DatasetInstance", **settings) -> GenomicRegionDataProvider:
+ def genomic_region_dict_dataprovider(self, dataset: DatasetProtocol, **settings) -> GenomicRegionDataProvider:
settings["named_columns"] = True
return self.genomic_region_dataprovider(dataset, **settings)
@@ -1042,11 +1041,11 @@ class BaseVcf(Tabular):
headers = get_headers(fname_or_file_prefix, "\n", count=1)
return headers[0][0].startswith("##fileformat=VCF")
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
"""Returns formated html of peek"""
return self.make_html_table(dataset, column_names=self.column_names)
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
super().set_meta(dataset, overwrite=overwrite, **kwd)
line = None
with compression_utils.get_fileobj(dataset.file_name) as fh:
@@ -1072,7 +1071,7 @@ class BaseVcf(Tabular):
if exit_code != 0:
raise Exception(f"Error merging VCF files: {stderr!r}")
- def validate(self, dataset: "DatasetInstance", **kwd) -> DatatypeValidation:
+ def validate(self, dataset: DatasetProtocol, **kwd) -> DatatypeValidation:
def validate_row(row):
if len(row) < 8:
raise Exception("Not enough columns in row %s" % row.join("\t"))
@@ -1082,11 +1081,11 @@ class BaseVcf(Tabular):
# Dataproviders
@dataproviders.decorators.dataprovider_factory("genomic-region", GenomicRegionDataProvider.settings)
- def genomic_region_dataprovider(self, dataset: "DatasetInstance", **settings) -> GenomicRegionDataProvider:
+ def genomic_region_dataprovider(self, dataset: DatasetProtocol, **settings) -> GenomicRegionDataProvider:
return GenomicRegionDataProvider(dataset, 0, 1, 1, **settings)
@dataproviders.decorators.dataprovider_factory("genomic-region-dict", GenomicRegionDataProvider.settings)
- def genomic_region_dict_dataprovider(self, dataset: "DatasetInstance", **settings) -> GenomicRegionDataProvider:
+ def genomic_region_dict_dataprovider(self, dataset: DatasetProtocol, **settings) -> GenomicRegionDataProvider:
settings["named_columns"] = True
return self.genomic_region_dataprovider(dataset, **settings)
@@ -1127,7 +1126,7 @@ class VcfGz(BaseVcf, binary.Binary):
return binascii.hexlify(last28) == b"1f8b08040000000000ff0600424302001b0003000000000000000000"
def set_meta(
- self, dataset: "DatasetInstance", overwrite: bool = True, metadata_tmp_files_dir: Optional[str] = None, **kwd
+ self, dataset: DatasetProtocol, overwrite: bool = True, metadata_tmp_files_dir: Optional[str] = None, **kwd
) -> None:
super().set_meta(dataset, overwrite=overwrite, **kwd)
# Creates the index for the VCF file.
@@ -1230,7 +1229,7 @@ class Eland(Tabular):
]
def make_html_table(
- self, dataset: "DatasetInstance", skipchars: Optional[List] = None, peek: Optional[List] = None, **kwargs
+ self, dataset: DatasetProtocol, skipchars: Optional[List] = None, peek: Optional[List] = None, **kwargs
) -> str:
"""Create HTML table, used for displaying peek"""
skipchars = skipchars or []
@@ -1291,7 +1290,7 @@ class Eland(Tabular):
def set_meta(
self,
- dataset: "DatasetInstance",
+ dataset: DatasetProtocol,
overwrite: bool = True,
skip: Optional[int] = None,
max_data_lines: Optional[int] = 5,
@@ -1453,7 +1452,7 @@ class BaseCSV(TabularData):
return False
return True
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
column_types = []
header_row = []
data_row = []
@@ -1535,7 +1534,7 @@ class ConnectivityTable(Tabular):
self.column_names = ["base_index", "base", "neighbor_left", "neighbor_right", "partner", "natural_numbering"]
self.column_types = ["int", "str", "int", "int", "int", "int"]
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
data_lines = 0
with open(dataset.file_name) as fh:
@@ -1602,7 +1601,7 @@ class ConnectivityTable(Tabular):
i += 1
return False
- def get_chunk(self, trans, dataset: "DatasetInstance", offset: int = 0, ck_size: Optional[int] = None) -> str:
+ def get_chunk(self, trans, dataset: HasFileName, offset: int = 0, ck_size: Optional[int] = None) -> str:
ck_data, last_read = self._read_chunk(trans, dataset, offset, ck_size)
try:
# The ConnectivityTable format has several derivatives of which one is delimited by (multiple) spaces.
@@ -1670,7 +1669,7 @@ class MatrixMarket(TabularData):
def set_meta(
self,
- dataset: "DatasetInstance",
+ dataset: DatasetProtocol,
overwrite: bool = True,
skip: Optional[int] = None,
max_data_lines: Optional[int] = 5,
@@ -1798,7 +1797,7 @@ class CMAP(TabularData):
def set_meta(
self,
- dataset: "DatasetInstance",
+ dataset: DatasetProtocol,
overwrite: bool = True,
skip: Optional[int] = None,
max_data_lines: Optional[int] = 7,
diff --git a/lib/galaxy/datatypes/text.py b/lib/galaxy/datatypes/text.py
index 28fe2fa8964..fd64370ab67 100644
--- a/lib/galaxy/datatypes/text.py
+++ b/lib/galaxy/datatypes/text.py
@@ -12,13 +12,17 @@ from typing import (
IO,
Optional,
Tuple,
- TYPE_CHECKING,
)
import yaml
+from galaxy.datatypes._protocols import (
+ DatasetHasHidProtocol,
+ DatasetProtocol,
+ HasCreatingJob,
+ HasExtraFilesAndMetadata,
+)
from galaxy.datatypes.data import (
- GeneratePrimaryFileDataset,
get_file_peek,
Headers,
Text,
@@ -39,12 +43,6 @@ from galaxy.util import (
unicodify,
)
-if TYPE_CHECKING:
- from galaxy.model import (
- DatasetInstance,
- HistoryDatasetAssociation,
- )
-
log = logging.getLogger(__name__)
@@ -55,7 +53,7 @@ class Html(Text):
edam_format = "format_2331"
file_ext = "html"
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
dataset.peek = "HTML file"
dataset.blurb = nice_size(dataset.get_size())
@@ -91,7 +89,7 @@ class Json(Text):
edam_format = "format_3464"
file_ext = "json"
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
dataset.peek = get_file_peek(dataset.file_name)
dataset.blurb = "JavaScript Object Notation (JSON)"
@@ -128,7 +126,7 @@ class Json(Text):
return start.startswith("[") or start.startswith("{")
return False
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
try:
return dataset.peek
except Exception:
@@ -143,7 +141,7 @@ class ExpressionJson(Json):
name="json_type", default=None, desc="JavaScript or JSON type of expression", readonly=True, visible=True
)
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
""" """
if dataset.has_data():
json_type = "null"
@@ -170,7 +168,7 @@ class ExpressionJson(Json):
class Ipynb(Json):
file_ext = "ipynb"
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
dataset.peek = get_file_peek(dataset.file_name)
dataset.blurb = "Jupyter Notebook"
@@ -197,7 +195,7 @@ class Ipynb(Json):
def display_data(
self,
trans,
- dataset: "HistoryDatasetAssociation",
+ dataset: DatasetHasHidProtocol,
preview: bool = False,
filename: Optional[str] = None,
to_ext: Optional[str] = None,
@@ -218,7 +216,7 @@ class Ipynb(Json):
def _display_data_trusted(
self,
trans,
- dataset: "HistoryDatasetAssociation",
+ dataset: DatasetHasHidProtocol,
preview: bool = False,
filename: Optional[str] = None,
to_ext: Optional[str] = None,
@@ -253,7 +251,7 @@ class Ipynb(Json):
)
return open(ofilename, mode="rb"), headers
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""
Set the number of models in dataset.
"""
@@ -389,7 +387,7 @@ class Biom1(Json):
no_value=[],
)
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
super().set_peek(dataset)
if not dataset.dataset.purged:
dataset.blurb = "Biological Observation Matrix v1"
@@ -426,7 +424,7 @@ class Biom1(Json):
pass
return is_biom
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""
Store metadata information from the BIOM file.
"""
@@ -488,7 +486,7 @@ class ImgtJson(Json):
MetadataElement(name="taxon_names", default=[], desc="taxonID: names", readonly=True, visible=True, no_value=[])
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
super().set_peek(dataset)
if not dataset.dataset.purged:
dataset.blurb = "IMGT Library"
@@ -527,7 +525,7 @@ class ImgtJson(Json):
pass
return is_imgt
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""
Store metadata information from the imgt file.
"""
@@ -554,7 +552,7 @@ class GeoJson(Json):
file_ext = "geojson"
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
super().set_peek(dataset)
if not dataset.dataset.purged:
dataset.blurb = "GeoJSON"
@@ -615,7 +613,7 @@ class Obo(Text):
edam_format = "format_2549"
file_ext = "obo"
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
dataset.peek = get_file_peek(dataset.file_name)
dataset.blurb = "Open Biomedical Ontology (OBO)"
@@ -657,7 +655,7 @@ class Arff(Text):
)
MetadataElement(name="columns", default=0, desc="Number of columns", readonly=True, visible=True, no_value=0)
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
dataset.peek = get_file_peek(dataset.file_name)
dataset.blurb = "Attribute-Relation File Format (ARFF)"
@@ -693,7 +691,7 @@ class Arff(Text):
return True
return False
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
"""
Trying to count the comment lines and the number of columns included.
A typical ARFF data block looks like this:
@@ -778,7 +776,7 @@ class SnpEffDb(Text):
pass
return snpeff_version
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
super().set_meta(dataset, overwrite=overwrite, **kwd)
data_dir = dataset.extra_files_path
# search data_dir/genome_version for files
@@ -864,14 +862,14 @@ class SnpSiftDbNSFP(Text):
"%s.gz.tbi", description="Tabix Index File", substitute_name_with_metadata="reference_name", is_binary=True
)
- def generate_primary_file(self, dataset: GeneratePrimaryFileDataset) -> str:
+ def generate_primary_file(self, dataset: HasExtraFilesAndMetadata) -> str:
"""
This is called only at upload to write the html file
cannot rename the datasets here - they come with the default unfortunately
"""
return "SnpSiftDbNSFP Composite Dataset "
- def regenerate_primary_file(self, dataset: "DatasetInstance") -> None:
+ def regenerate_primary_file(self, dataset: DatasetProtocol) -> None:
"""
cannot do this until we are setting metadata
"""
@@ -883,7 +881,7 @@ class SnpSiftDbNSFP(Text):
f.write("\n")
f.write(annotations)
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
try:
efp = dataset.extra_files_path
if os.path.exists(efp):
@@ -909,7 +907,7 @@ class SnpSiftDbNSFP(Text):
unicodify(e),
)
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
dataset.peek = f"{dataset.metadata.reference_name} : {','.join(dataset.metadata.annotation)}"
dataset.blurb = f"{dataset.metadata.reference_name}"
@@ -1094,7 +1092,7 @@ class Yaml(Text):
"""Returns the mime type of the datatype"""
return "application/yaml"
- def _yield_user_file_content(self, trans, from_dataset: "DatasetInstance", filename: str, headers: Headers) -> IO:
+ def _yield_user_file_content(self, trans, from_dataset: HasCreatingJob, filename: str, headers: Headers) -> IO:
# Override non-standard application/yaml mediatype with
# text/plain, so preview is shown in preview iframe,
# instead of downloading the file.
@@ -1152,7 +1150,7 @@ class BCSLts(Json):
is_bcsl_ts = self._looks_like_bcsl_ts(file_prefix)
return is_bcsl_ts
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
lines = "States: {}\nTransitions: {}\nUnique agents: {}\nInitial state: {}"
ts = json.load(open(dataset.file_name))
@@ -1186,7 +1184,7 @@ class StormSample(Text):
keywords = ["Storm-pars", "Result (initial states)"]
return all(keyword in file_prefix.contents_header for keyword in keywords)
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
dataset.peek = "Storm-pars sample results."
dataset.blurb = nice_size(dataset.get_size())
@@ -1211,7 +1209,7 @@ class StormCheck(Text):
keywords = ["Storm ", "Result (for initial states)"]
return all(keyword in file_prefix.contents_header for keyword in keywords)
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
with open(dataset.file_name) as result:
answer = ""
@@ -1239,7 +1237,7 @@ class CTLresult(Text):
keywords = ["Result:", "Number of satisfying states:"]
return all(keyword in file_prefix.contents_header for keyword in keywords)
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
with open(dataset.file_name) as result:
answer = ""
diff --git a/lib/galaxy/datatypes/tracks.py b/lib/galaxy/datatypes/tracks.py
index bf5e29e6b8c..2af7decc628 100644
--- a/lib/galaxy/datatypes/tracks.py
+++ b/lib/galaxy/datatypes/tracks.py
@@ -2,15 +2,14 @@
Datatype classes for tracks/track views within galaxy.
"""
import logging
-from typing import TYPE_CHECKING
-from galaxy.datatypes.data import GeneratePrimaryFileDataset
+from galaxy.datatypes._protocols import (
+ DatasetProtocol,
+ HasExtraFilesAndMetadata,
+)
from galaxy.datatypes.text import Html
from . import binary
-if TYPE_CHECKING:
- from galaxy.model import DatasetInstance
-
log = logging.getLogger(__name__)
@@ -30,7 +29,7 @@ class UCSCTrackHub(Html):
file_ext = "trackhub"
composite_type = "auto_primary_file"
- def generate_primary_file(self, dataset: GeneratePrimaryFileDataset) -> str:
+ def generate_primary_file(self, dataset: HasExtraFilesAndMetadata) -> str:
"""
This is called only at upload to write the html file
cannot rename the datasets here - they come with the default unfortunately
@@ -48,14 +47,14 @@ class UCSCTrackHub(Html):
rval.append("")
return "\n".join(rval)
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
dataset.peek = "Track Hub structure: Visualization in UCSC Track Hub"
else:
dataset.peek = "file does not exist"
dataset.blurb = "file purged from disk"
- def display_peek(self, dataset: "DatasetInstance") -> str:
+ def display_peek(self, dataset: DatasetProtocol) -> str:
try:
return dataset.peek
except Exception:
diff --git a/lib/galaxy/datatypes/triples.py b/lib/galaxy/datatypes/triples.py
index debd78bb366..d908eae1dd9 100644
--- a/lib/galaxy/datatypes/triples.py
+++ b/lib/galaxy/datatypes/triples.py
@@ -3,8 +3,8 @@ Triple format classes
"""
import logging
import re
-from typing import TYPE_CHECKING
+from galaxy.datatypes._protocols import DatasetProtocol
from galaxy.datatypes.sniff import (
build_sniff_from_prefix,
FilePrefix,
@@ -16,9 +16,6 @@ from . import (
xml,
)
-if TYPE_CHECKING:
- from galaxy.model import DatasetInstance
-
log = logging.getLogger(__name__)
TURTLE_PREFIX_PATTERN = re.compile(r"@prefix\s+[^:]*:\s+<[^>]*>\s\.")
@@ -40,7 +37,7 @@ class Triples(data.Data):
"""
return False
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
"""Set the peek and blurb text"""
if not dataset.dataset.purged:
dataset.peek = data.get_file_peek(dataset.file_name)
@@ -65,7 +62,7 @@ class NTriples(data.Text, Triples):
return True
return False
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
"""Set the peek and blurb text"""
if not dataset.dataset.purged:
dataset.peek = data.get_file_peek(dataset.file_name)
@@ -89,7 +86,7 @@ class N3(data.Text, Triples):
"""
return False
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
"""Set the peek and blurb text"""
if not dataset.dataset.purged:
dataset.peek = data.get_file_peek(dataset.file_name)
@@ -117,7 +114,7 @@ class Turtle(data.Text, Triples):
return True
return False
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
"""Set the peek and blurb text"""
if not dataset.dataset.purged:
dataset.peek = data.get_file_peek(dataset.file_name)
@@ -146,7 +143,7 @@ class Rdf(xml.GenericXml, Triples):
return True
return False
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
"""Set the peek and blurb text"""
if not dataset.dataset.purged:
dataset.peek = data.get_file_peek(dataset.file_name)
@@ -172,7 +169,7 @@ class Jsonld(text.Json, Triples):
return True
return False
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
"""Set the peek and blurb text"""
if not dataset.dataset.purged:
dataset.peek = data.get_file_peek(dataset.file_name)
@@ -196,7 +193,7 @@ class HDT(binary.Binary, Triples):
return True
return False
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
"""Set the peek and blurb text"""
if not dataset.dataset.purged:
dataset.peek = data.get_file_peek(dataset.file_name)
diff --git a/lib/galaxy/datatypes/xml.py b/lib/galaxy/datatypes/xml.py
index 00c1fe94ca0..475cc5cd1ff 100644
--- a/lib/galaxy/datatypes/xml.py
+++ b/lib/galaxy/datatypes/xml.py
@@ -3,12 +3,10 @@ XML format classes
"""
import logging
import re
-from typing import (
- List,
- TYPE_CHECKING,
-)
+from typing import List
from galaxy import util
+from galaxy.datatypes._protocols import DatasetProtocol
from galaxy.datatypes.dataproviders.dataset import DatasetDataProvider
from galaxy.datatypes.dataproviders.hierarchy import XMLDataProvider
from galaxy.datatypes.metadata import MetadataElement
@@ -22,9 +20,6 @@ from . import (
dataproviders,
)
-if TYPE_CHECKING:
- from galaxy.model import DatasetInstance
-
log = logging.getLogger(__name__)
OWL_MARKER = re.compile(r"\ None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
"""Set the peek and blurb text"""
if not dataset.dataset.purged:
dataset.peek = data.get_file_peek(dataset.file_name)
@@ -81,7 +76,7 @@ class GenericXml(data.Text):
data.Text.merge(split_files, output_file)
@dataproviders.decorators.dataprovider_factory("xml", XMLDataProvider.settings)
- def xml_dataprovider(self, dataset: "DatasetInstance", **settings) -> XMLDataProvider:
+ def xml_dataprovider(self, dataset: DatasetProtocol, **settings) -> XMLDataProvider:
dataset_source = DatasetDataProvider(dataset)
return XMLDataProvider(dataset_source, **settings)
@@ -92,7 +87,7 @@ class MEMEXml(GenericXml):
file_ext = "memexml"
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
"""Set the peek and blurb text"""
if not dataset.dataset.purged:
dataset.peek = data.get_file_peek(dataset.file_name)
@@ -108,7 +103,7 @@ class CisML(GenericXml):
file_ext = "cisml"
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
"""Set the peek and blurb text"""
if not dataset.dataset.purged:
dataset.peek = data.get_file_peek(dataset.file_name)
@@ -152,7 +147,7 @@ class Dzi(GenericXml):
def __init__(self, **kwd):
super().__init__(**kwd)
- def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None:
+ def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None:
tree = util.parse_xml(dataset.file_name)
root = tree.getroot()
dataset.metadata.format = root.get("Format")
@@ -169,7 +164,7 @@ class Dzi(GenericXml):
dataset.metadata.width = elem.get("Width")
dataset.metadata.height = elem.get("Height")
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
dataset.peek = data.get_file_peek(dataset.file_name)
dataset.blurb = "Deep Zoom Image"
@@ -202,7 +197,7 @@ class Phyloxml(GenericXml):
edam_format = "format_3159"
file_ext = "phyloxml"
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
"""Set the peek and blurb text"""
if not dataset.dataset.purged:
dataset.peek = data.get_file_peek(dataset.file_name)
@@ -237,7 +232,7 @@ class Owl(GenericXml):
edam_format = "format_3262"
file_ext = "owl"
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
dataset.peek = data.get_file_peek(dataset.file_name)
dataset.blurb = "Web Ontology Language OWL"
@@ -262,7 +257,7 @@ class Sbml(GenericXml):
edam_data = "data_2024"
edam_format = "format_2585"
- def set_peek(self, dataset: "DatasetInstance", **kwd) -> None:
+ def set_peek(self, dataset: DatasetProtocol, **kwd) -> None:
if not dataset.dataset.purged:
dataset.peek = data.get_file_peek(dataset.file_name)
dataset.blurb = "System Biology Markup Language SBML"
diff --git a/lib/galaxy/dependencies/dev-requirements.txt b/lib/galaxy/dependencies/dev-requirements.txt
index 2d83fdd2c9f..afb44df79cc 100644
--- a/lib/galaxy/dependencies/dev-requirements.txt
+++ b/lib/galaxy/dependencies/dev-requirements.txt
@@ -53,7 +53,7 @@ lockfile==0.12.2 ; python_version >= "3.7" and python_version < "3.11"
lxml==4.9.2 ; python_version >= "3.7" and python_version < "3.11"
markdown-it-reporter==0.0.2 ; python_version >= "3.7" and python_version < "3.11"
markdown==3.4.1 ; python_version >= "3.7" and python_version < "3.11"
-markupsafe==2.1.1 ; python_version >= "3.7" and python_version < "3.11"
+markupsafe==2.1.2 ; python_version >= "3.7" and python_version < "3.11"
matplotlib==3.5.3 ; python_version >= "3.7" and python_version < "3.11"
mirakuru==2.4.2 ; python_version >= "3.7" and python_version < "3.11"
mistune==2.0.4 ; python_version >= "3.7" and python_version < "3.11"
@@ -89,10 +89,10 @@ pytest-metadata==2.0.4 ; python_version >= "3.7" and python_version < "3.11"
pytest-mock==3.10.0 ; python_version >= "3.7" and python_version < "3.11"
pytest-postgresql==4.1.1 ; python_version >= "3.7" and python_version < "3.11"
pytest-shard==0.1.2 ; python_version >= "3.7" and python_version < "3.11"
-pytest==7.2.0 ; python_version >= "3.7" and python_version < "3.11"
+pytest==7.2.1 ; python_version >= "3.7" and python_version < "3.11"
python-dateutil==2.8.2 ; python_version >= "3.7" and python_version < "3.11"
-python-irodsclient==1.1.5 ; python_version >= "3.7" and python_version < "3.11"
-pytz==2022.7 ; python_version >= "3.7" and python_version < "3.11"
+python-irodsclient==1.1.6 ; python_version >= "3.7" and python_version < "3.11"
+pytz==2022.7.1 ; python_version >= "3.7" and python_version < "3.11"
rdflib==6.2.0 ; python_version >= "3.7" and python_version < "3.11"
recommonmark==0.7.1 ; python_version >= "3.7" and python_version < "3.11"
requests==2.28.2 ; python_version >= "3.7" and python_version < "3.11"
@@ -104,7 +104,7 @@ schema-salad==8.3.20230109181936 ; python_version >= "3.7" and python_version <
scipy==1.7.3 ; python_version >= "3.7" and python_version < "3.11"
selenium==4.7.2 ; python_version >= "3.7" and python_version < "3.11"
seletools==1.3.0 ; python_version >= "3.7" and python_version < "3.11"
-setuptools==65.7.0 ; python_version >= "3.7" and python_version < "3.11"
+setuptools==66.1.1 ; python_version >= "3.7" and python_version < "3.11"
six==1.16.0 ; python_version >= "3.7" and python_version < "3.11"
sniffio==1.3.0 ; python_version >= "3.7" and python_version < "3.11"
snowballstemmer==2.2.0 ; python_version >= "3.7" and python_version < "3.11"
@@ -120,7 +120,7 @@ sphinxcontrib-qthelp==1.0.3 ; python_version >= "3.7" and python_version < "3.11
sphinxcontrib-serializinghtml==1.1.5 ; python_version >= "3.7" and python_version < "3.11"
statsd==4.0.1 ; python_version >= "3.7" and python_version < "3.11"
testfixtures==7.0.4 ; python_version >= "3.7" and python_version < "3.11"
-tinydb==4.7.0 ; python_version >= "3.7" and python_version < "3.11"
+tinydb==4.7.1 ; python_version >= "3.7" and python_version < "3.11"
toml==0.10.2 ; python_version >= "3.7" and python_version < "3.11"
tomli==2.0.1 ; python_version >= "3.7" and python_version < "3.11"
trio-websocket==0.9.2 ; python_version >= "3.7" and python_version < "3.11"
@@ -134,7 +134,7 @@ urllib3==1.26.14 ; python_version >= "3.7" and python_version < "3.11"
urllib3[socks]==1.26.14 ; python_version >= "3.7" and python_version < "3.11"
vine==5.0.0 ; python_version >= "3.7" and python_version < "3.11"
watchdog==2.2.1 ; python_version >= "3.7" and python_version < "3.11"
-wcwidth==0.2.5 ; python_version >= "3.7" and python_version < "3.11"
+wcwidth==0.2.6 ; python_version >= "3.7" and python_version < "3.11"
wrapt==1.14.1 ; python_version >= "3.7" and python_version < "3.11"
wsproto==1.2.0 ; python_version >= "3.7" and python_version < "3.11"
yarl==1.8.2 ; python_version >= "3.7" and python_version < "3.11"
diff --git a/lib/galaxy/dependencies/pinned-lint-requirements.txt b/lib/galaxy/dependencies/pinned-lint-requirements.txt
index 4cf3961e3dd..e5ca6fafb5d 100644
--- a/lib/galaxy/dependencies/pinned-lint-requirements.txt
+++ b/lib/galaxy/dependencies/pinned-lint-requirements.txt
@@ -1,10 +1,10 @@
attrs==22.2.0
flake8==5.0.4
-flake8-bugbear==22.12.6
+flake8-bugbear==23.1.20
importlib-metadata==4.2.0
mccabe==0.7.0
pycodestyle==2.9.1
pyflakes==2.5.0
-ruff==0.0.223
+ruff==0.0.228
typing_extensions==4.4.0
zipp==3.11.0
diff --git a/lib/galaxy/dependencies/pinned-requirements.txt b/lib/galaxy/dependencies/pinned-requirements.txt
index d369ca9eb30..3cf5b20fd73 100644
--- a/lib/galaxy/dependencies/pinned-requirements.txt
+++ b/lib/galaxy/dependencies/pinned-requirements.txt
@@ -5,7 +5,8 @@ adal==1.2.7 ; python_version >= "3.7" and python_version < "3.11"
aiofiles==22.1.0 ; python_version >= "3.7" and python_version < "3.11"
aiohttp==3.8.3 ; python_version >= "3.7" and python_version < "3.11"
aiosignal==1.3.1 ; python_version >= "3.7" and python_version < "3.11"
-alembic==1.9.1 ; python_version >= "3.7" and python_version < "3.11"
+alembic-utils==0.7.8 ; python_version >= "3.7" and python_version < "3.11"
+alembic==1.9.2 ; python_version >= "3.7" and python_version < "3.11"
amqp==5.1.1 ; python_version >= "3.7" and python_version < "3.11"
anyio==3.6.2 ; python_version >= "3.7" and python_version < "3.11"
apispec==6.0.2 ; python_version >= "3.7" and python_version < "3.11"
@@ -45,7 +46,6 @@ cloudauthz==0.6.0 ; python_version >= "3.7" and python_version < "3.11"
cloudbridge==3.1.0 ; python_version >= "3.7" and python_version < "3.11"
colorama==0.4.6 ; python_version >= "3.7" and python_version < "3.11" and platform_system == "Windows"
coloredlogs==15.0.1 ; python_version >= "3.7" and python_version < "3.11"
-commonmark==0.9.1 ; python_version >= "3.7" and python_version < "3.11"
cryptography==39.0.0 ; python_version >= "3.7" and python_version < "3.11"
cwl-upgrader==1.2.4 ; python_version >= "3.7" and python_version < "3.11"
cwl-utils==0.21 ; python_version >= "3.7" and python_version < "3.11"
@@ -54,7 +54,7 @@ decorator==5.1.1 ; python_version >= "3.7" and python_version < "3.11"
defusedxml==0.7.1 ; python_version >= "3.7" and python_version < "3.11"
deprecation==2.1.0 ; python_version >= "3.7" and python_version < "3.11"
dictobj==0.4 ; python_version >= "3.7" and python_version < "3.11"
-dnspython==2.2.1 ; python_version >= "3.7" and python_version < "3.11"
+dnspython==2.3.0 ; python_version >= "3.7" and python_version < "3.11"
docopt==0.6.2 ; python_version >= "3.7" and python_version < "3.11"
docutils==0.16 ; python_version >= "3.7" and python_version < "3.11"
ecdsa==0.18.0 ; python_version >= "3.7" and python_version < "3.11"
@@ -62,6 +62,7 @@ edam-ontology==1.25.2 ; python_version >= "3.7" and python_version < "3.11"
email-validator==1.3.0 ; python_version >= "3.7" and python_version < "3.11"
fastapi-utils==0.2.1 ; python_version >= "3.7" and python_version < "3.11"
fastapi==0.89.1 ; python_version >= "3.7" and python_version < "3.11"
+flupy==1.2.0 ; python_version >= "3.7" and python_version < "3.11"
frozenlist==1.3.3 ; python_version >= "3.7" and python_version < "3.11"
fs==2.4.16 ; python_version >= "3.7" and python_version < "3.11"
funcsigs==1.0.2 ; python_version >= "3.7" and python_version < "3.11"
@@ -81,15 +82,17 @@ importlib-resources==5.10.2 ; python_version >= "3.7" and python_version < "3.11
isa-rwval==0.10.10 ; python_version >= "3.7" and python_version < "3.11"
isodate==0.6.1 ; python_version >= "3.7" and python_version < "3.11"
jinja2==3.1.2 ; python_version >= "3.7" and python_version < "3.11"
-jsonref==1.0.1 ; python_version >= "3.7" and python_version < "3.11"
+jsonref==1.1.0 ; python_version >= "3.7" and python_version < "3.11"
jsonschema==4.17.3 ; python_version >= "3.7" and python_version < "3.11"
kombu==5.2.4 ; python_version >= "3.7" and python_version < "3.11"
lagom==2.2.0 ; python_version >= "3.7" and python_version < "3.11"
lockfile==0.12.2 ; python_version >= "3.7" and python_version < "3.11"
lxml==4.9.2 ; python_version >= "3.7" and python_version < "3.11"
mako==1.2.4 ; python_version >= "3.7" and python_version < "3.11"
+markdown-it-py==2.1.0 ; python_version >= "3.7" and python_version < "3.11"
markdown==3.4.1 ; python_version >= "3.7" and python_version < "3.11"
-markupsafe==2.1.1 ; python_version >= "3.7" and python_version < "3.11"
+markupsafe==2.1.2 ; python_version >= "3.7" and python_version < "3.11"
+mdurl==0.1.2 ; python_version >= "3.7" and python_version < "3.11"
mercurial==6.3.2 ; python_version >= "3.7" and python_version < "3.11"
mistune==2.0.4 ; python_version >= "3.7" and python_version < "3.11"
mrcfile==1.4.3 ; python_version >= "3.7" and python_version < "3.11"
@@ -102,7 +105,8 @@ numpy==1.21.6 ; python_version >= "3.7" and python_version < "3.11"
oauthlib==3.2.2 ; python_version >= "3.7" and python_version < "3.11"
oyaml==1.0 ; python_version >= "3.7" and python_version < "3.11"
packaging==21.3 ; python_version >= "3.7" and python_version < "3.11"
-paramiko==2.12.0 ; python_version >= "3.7" and python_version < "3.11"
+paramiko==3.0.0 ; python_version >= "3.7" and python_version < "3.11"
+parse==1.19.0 ; python_version >= "3.7" and python_version < "3.11"
parsley==1.3 ; python_version >= "3.7" and python_version < "3.11"
paste==3.5.2 ; python_version >= "3.7" and python_version < "3.11"
pastedeploy==3.0.1 ; python_version >= "3.7" and python_version < "3.11"
@@ -137,7 +141,7 @@ python-jose==3.3.0 ; python_version >= "3.7" and python_version < "3.11"
python-magic==0.4.27 ; python_version >= "3.7" and python_version < "3.11"
python-multipart==0.0.5 ; python_version >= "3.7" and python_version < "3.11"
python3-openid==3.2.0 ; python_version >= "3.7" and python_version < "3.11"
-pytz==2022.7 ; python_version >= "3.7" and python_version < "3.11"
+pytz==2022.7.1 ; python_version >= "3.7" and python_version < "3.11"
pyyaml==6.0 ; python_version >= "3.7" and python_version < "3.11"
pyzmq==25.0.0 ; python_version >= "3.7" and python_version < "3.11"
rdflib==6.2.0 ; python_version >= "3.7" and python_version < "3.11"
@@ -147,7 +151,7 @@ requests-oauthlib==1.3.1 ; python_version >= "3.7" and python_version < "3.11"
requests-toolbelt==0.10.1 ; python_version >= "3.7" and python_version < "3.11"
requests-unixsocket==0.3.0 ; python_version >= "3.7" and python_version < "3.11"
requests==2.28.2 ; python_version >= "3.7" and python_version < "3.11"
-rich==13.0.1 ; python_version >= "3.7" and python_version < "3.11"
+rich==13.2.0 ; python_version >= "3.7" and python_version < "3.11"
rocrate==0.7.0 ; python_version >= "3.7" and python_version < "3.11"
routes==2.5.1 ; python_version >= "3.7" and python_version < "3.11"
rsa==4.9 ; python_version >= "3.7" and python_version < "3.11"
@@ -155,7 +159,7 @@ ruamel-yaml-clib==0.2.7 ; platform_python_implementation == "CPython" and python
ruamel-yaml==0.17.21 ; python_version < "3.11" and python_version >= "3.7"
schema-salad==8.3.20230109181936 ; python_version >= "3.7" and python_version < "3.11"
setuptools-scm==5.0.2 ; python_version >= "3.7" and python_version < "3.11"
-setuptools==65.7.0 ; python_version >= "3.7" and python_version < "3.11"
+setuptools==66.1.1 ; python_version >= "3.7" and python_version < "3.11"
shellescape==3.8.1 ; python_version >= "3.7" and python_version < "3.11"
six==1.16.0 ; python_version >= "3.7" and python_version < "3.11"
sniffio==1.3.0 ; python_version >= "3.7" and python_version < "3.11"
@@ -172,7 +176,7 @@ svgwrite==1.4.3 ; python_version >= "3.7" and python_version < "3.11"
tempita==0.5.2 ; python_version >= "3.7" and python_version < "3.11"
tenacity==8.1.0 ; python_version >= "3.7" and python_version < "3.11"
tifffile==2021.11.2 ; python_version >= "3.7" and python_version < "3.11"
-tinydb==4.7.0 ; python_version >= "3.7" and python_version < "3.11"
+tinydb==4.7.1 ; python_version >= "3.7" and python_version < "3.11"
tornado==6.2 ; python_version >= "3.7" and python_version < "3.11"
tqdm==4.64.1 ; python_version >= "3.7" and python_version < "3.11"
tuspy==1.0.0 ; python_version >= "3.7" and python_version < "3.11"
@@ -184,7 +188,7 @@ urllib3==1.26.14 ; python_version >= "3.7" and python_version < "3.11"
uvicorn==0.20.0 ; python_version >= "3.7" and python_version < "3.11"
uvloop==0.17.0 ; python_version >= "3.7" and python_version < "3.11"
vine==5.0.0 ; python_version >= "3.7" and python_version < "3.11"
-wcwidth==0.2.5 ; python_version >= "3.7" and python_version < "3.11"
+wcwidth==0.2.6 ; python_version >= "3.7" and python_version < "3.11"
webencodings==0.5.1 ; python_version >= "3.7" and python_version < "3.11"
webob==1.8.7 ; python_version >= "3.7" and python_version < "3.11"
whoosh==2.7.4 ; python_version >= "3.7" and python_version < "3.11"
diff --git a/lib/galaxy/dependencies/pinned-typecheck-requirements.txt b/lib/galaxy/dependencies/pinned-typecheck-requirements.txt
index f1e1c5305b3..5b68529c79c 100644
--- a/lib/galaxy/dependencies/pinned-typecheck-requirements.txt
+++ b/lib/galaxy/dependencies/pinned-typecheck-requirements.txt
@@ -6,16 +6,16 @@ pycparser==2.21
tomli==2.0.1
typed-ast==1.5.4
types-bleach==5.0.3.1
-types-boto==2.49.18.4
+types-boto==2.49.18.5
types-contextvars==2.4.7
types-dataclasses==0.6.6
-types-docutils==0.19.1.1
-types-Markdown==3.4.2.1
-types-paramiko==2.12.0.2
+types-docutils==0.19.1.2
+types-Markdown==3.4.2.2
+types-paramiko==2.12.0.3
types-pkg-resources==0.1.3
-types-python-dateutil==2.8.19.5
-types-PyYAML==6.0.12.2
-types-requests==2.28.11.7
+types-python-dateutil==2.8.19.6
+types-PyYAML==6.0.12.3
+types-requests==2.28.11.8
types-six==1.16.21.4
types-urllib3==1.26.25.4
typing_extensions==4.4.0
diff --git a/lib/galaxy/job_execution/actions/post.py b/lib/galaxy/job_execution/actions/post.py
index c377681640a..47b466110cb 100644
--- a/lib/galaxy/job_execution/actions/post.py
+++ b/lib/galaxy/job_execution/actions/post.py
@@ -109,6 +109,9 @@ class ChangeDatatypeAction(DefaultJobAction):
@classmethod
def execute(cls, app, sa_session, action, job, replacement_dict, final_job_state=None):
+ if job.state == job.states.SKIPPED:
+ # Don't change datatype, must remain expression.json
+ return
for dataset_assoc in job.output_datasets:
if action.output_name == "" or dataset_assoc.name == action.output_name:
app.datatypes_registry.change_datatype(dataset_assoc.dataset, action.action_arguments["newtype"])
@@ -330,7 +333,7 @@ class ColumnSetAction(DefaultJobAction):
@classmethod
def get_short_str(cls, pja):
- return f"Set the following metadata values:
{'
'.join('{} : {}'.format(escape(k), escape(v)) for k, v in pja.action_arguments.items())}"
+ return f"Set the following metadata values:
{'
'.join(f'{escape(k)} : {escape(v)}' for k, v in pja.action_arguments.items())}"
class SetMetadataAction(DefaultJobAction):
diff --git a/lib/galaxy/managers/collections.py b/lib/galaxy/managers/collections.py
index 70370edd452..1cd673b50c0 100644
--- a/lib/galaxy/managers/collections.py
+++ b/lib/galaxy/managers/collections.py
@@ -829,7 +829,9 @@ class DatasetCollectionManager:
DCE = model.DatasetCollectionElement
qry = Query(DCE).filter(DCE.dataset_collection_id == parent_id)
qry = qry.order_by(DCE.element_index)
- qry = qry.options(joinedload("child_collection"), joinedload("hda"))
+ qry = qry.options(
+ joinedload(model.DatasetCollectionElement.child_collection), joinedload(model.DatasetCollectionElement.hda)
+ )
if limit is not None:
qry = qry.limit(int(limit))
if offset is not None:
diff --git a/lib/galaxy/managers/configuration.py b/lib/galaxy/managers/configuration.py
index 2485e4b75e6..5a2998c583c 100644
--- a/lib/galaxy/managers/configuration.py
+++ b/lib/galaxy/managers/configuration.py
@@ -155,6 +155,7 @@ class ConfigSerializer(base.ModelSerializer):
"matomo_site_id": _use_config,
"enable_unique_workflow_defaults": _use_config,
"enable_beta_markdown_export": _use_config,
+ "enable_beacon_integration": _use_config,
"use_legacy_history": _use_config,
"simplified_workflow_run_ui": _use_config,
"simplified_workflow_run_ui_target_history": _use_config,
diff --git a/lib/galaxy/managers/history_contents.py b/lib/galaxy/managers/history_contents.py
index 055a8b94687..077b8fff1fd 100644
--- a/lib/galaxy/managers/history_contents.py
+++ b/lib/galaxy/managers/history_contents.py
@@ -438,7 +438,7 @@ class HistoryContentsManager(base.SortableManager):
.query(component_class)
.filter(component_class.id.in_(id_list))
.options(undefer(component_class._metadata))
- .options(joinedload("dataset.actions")) # TODO: use class attr after moving Dataset to declarative mapping.
+ .options(joinedload(component_class.dataset).joinedload(model.Dataset.actions))
.options(joinedload(component_class.tags))
.options(joinedload(component_class.annotations)) # type: ignore[attr-defined]
)
diff --git a/lib/galaxy/managers/session.py b/lib/galaxy/managers/session.py
index 5d9fc238af8..4f6ca157f0e 100644
--- a/lib/galaxy/managers/session.py
+++ b/lib/galaxy/managers/session.py
@@ -30,7 +30,7 @@ class GalaxySessionManager:
self.model.GalaxySession.table.c.is_valid == true(),
)
)
- .options(joinedload("user"))
+ .options(joinedload(self.model.GalaxySession.user))
.first()
)
return galaxy_session
diff --git a/lib/galaxy/managers/sharable.py b/lib/galaxy/managers/sharable.py
index aebef5bb039..20eb3442f72 100644
--- a/lib/galaxy/managers/sharable.py
+++ b/lib/galaxy/managers/sharable.py
@@ -222,7 +222,7 @@ class SharableModelManager(
Return a query for this model already filtered to models shared
with a particular user.
"""
- query = self.session().query(self.model_class).join("users_shared_with")
+ query = self.session().query(self.model_class).join(self.model_class.users_shared_with)
if eagerloads is False:
query = query.enable_eagerloads(False)
# TODO: as filter in FilterParser also
diff --git a/lib/galaxy/managers/workflows.py b/lib/galaxy/managers/workflows.py
index f2d88ea5d3f..9e2e011bf77 100644
--- a/lib/galaxy/managers/workflows.py
+++ b/lib/galaxy/managers/workflows.py
@@ -10,6 +10,7 @@ from typing import (
NamedTuple,
Optional,
Tuple,
+ Union,
)
from gxformat2 import (
@@ -57,6 +58,7 @@ from galaxy.model.index_filter_util import (
text_column_filter,
)
from galaxy.model.item_attrs import UsesAnnotations
+from galaxy.schema.invocation import InvocationCancellationUserRequest
from galaxy.schema.schema import WorkflowIndexQueryPayload
from galaxy.structured_app import MinimalManagerApp
from galaxy.tools.parameters import (
@@ -161,10 +163,10 @@ class WorkflowsManager(sharable.SharableModelManager, deletable.DeletableManager
query = query.outerjoin(model.StoredWorkflow.users_shared_with)
query = query.outerjoin(model.StoredWorkflow.tags)
- latest_workflow_load = joinedload("latest_workflow")
+ latest_workflow_load = joinedload(model.StoredWorkflow.latest_workflow)
if not payload.skip_step_counts:
latest_workflow_load = latest_workflow_load.undefer("step_count")
- latest_workflow_load = latest_workflow_load.lazyload("steps")
+ latest_workflow_load = latest_workflow_load.lazyload(model.Workflow.steps)
query = query.options(joinedload(model.StoredWorkflow.annotations))
query = query.options(latest_workflow_load)
@@ -266,9 +268,11 @@ class WorkflowsManager(sharable.SharableModelManager, deletable.DeletableManager
)
)
stored_workflow = workflow_query.options(
- joinedload("annotations"),
- joinedload("tags"),
- subqueryload("latest_workflow").joinedload("steps").joinedload("*"),
+ joinedload(trans.app.model.StoredWorkflow.annotations),
+ joinedload(trans.app.model.StoredWorkflow.tags),
+ subqueryload(trans.app.model.StoredWorkflow.latest_workflow)
+ .joinedload(trans.app.model.Workflow.steps)
+ .joinedload("*"),
).first()
if stored_workflow is None:
if not by_stored_id:
@@ -359,15 +363,15 @@ class WorkflowsManager(sharable.SharableModelManager, deletable.DeletableManager
return True
- def get_invocation(self, trans, decoded_invocation_id, eager=False):
- q = trans.sa_session.query(self.app.model.WorkflowInvocation)
+ def get_invocation(self, trans, decoded_invocation_id, eager=False) -> model.WorkflowInvocation:
+ q = trans.sa_session.query(model.WorkflowInvocation)
if eager:
q = q.options(
- subqueryload(self.app.model.WorkflowInvocation.steps)
- .joinedload("implicit_collection_jobs")
- .joinedload("jobs")
- .joinedload("job")
- .joinedload("input_datasets")
+ subqueryload(model.WorkflowInvocation.steps)
+ .joinedload(model.WorkflowInvocationStep.implicit_collection_jobs)
+ .joinedload(model.ImplicitCollectionJobs.jobs)
+ .joinedload(model.ImplicitCollectionJobsJobAssociation.job)
+ .joinedload(model.Job.input_datasets)
)
workflow_invocation = q.get(decoded_invocation_id)
if not workflow_invocation:
@@ -401,6 +405,7 @@ class WorkflowsManager(sharable.SharableModelManager, deletable.DeletableManager
cancelled = workflow_invocation.cancel()
if cancelled:
+ workflow_invocation.add_message(InvocationCancellationUserRequest(reason="user_request"))
trans.sa_session.add(workflow_invocation)
trans.sa_session.flush()
else:
@@ -934,6 +939,7 @@ class WorkflowContentsManager(UsesAnnotations):
else:
inputs = step.module.get_runtime_inputs(connections=step.output_connections)
step_model = {"inputs": [input.to_dict(trans) for input in inputs.values()]}
+ step_model["when"] = step.when_expression
step_model["replacement_parameters"] = step.module.get_replacement_parameters(step)
step_model["step_type"] = step.type
step_model["step_label"] = step.label
@@ -1012,13 +1018,16 @@ class WorkflowContentsManager(UsesAnnotations):
nested_input_dict = {}
index = repeat_values[i]["__index__"]
nested_input_dict["title"] = "%i. %s" % (i + 1, input.title)
- nested_input_dict["inputs"] = do_inputs(
- input.inputs,
- repeat_values[i],
- f"{prefix + input.name}_{str(index)}|",
- step,
- other_values,
- )
+ try:
+ nested_input_dict["inputs"] = do_inputs(
+ input.inputs,
+ repeat_values[i],
+ f"{prefix + input.name}_{str(index)}|",
+ step,
+ other_values,
+ )
+ except KeyError:
+ continue
nested_input_dicts.append(nested_input_dict)
input_dict["inputs"] = nested_input_dicts
elif input.type == "conditional":
@@ -1028,14 +1037,20 @@ class WorkflowContentsManager(UsesAnnotations):
row_for_param(
input_dict, input.test_param, group_values[input.test_param.name], other_values, prefix, step
)
- input_dict["inputs"] = do_inputs(
- input.cases[current_case].inputs, group_values, new_prefix, step, other_values
- )
+ try:
+ input_dict["inputs"] = do_inputs(
+ input.cases[current_case].inputs, group_values, new_prefix, step, other_values
+ )
+ except KeyError:
+ continue
elif input.type == "section":
new_prefix = f"{prefix + input.name}|"
group_values = values[input.name]
input_dict["title"] = input.title
- input_dict["inputs"] = do_inputs(input.inputs, group_values, new_prefix, step, other_values)
+ try:
+ input_dict["inputs"] = do_inputs(input.inputs, group_values, new_prefix, step, other_values)
+ except KeyError:
+ continue
else:
row_for_param(input_dict, input, values[input.name], other_values, prefix, step)
input_dicts.append(input_dict)
@@ -1128,6 +1143,7 @@ class WorkflowContentsManager(UsesAnnotations):
"annotation": annotation_str,
"post_job_actions": module.get_post_job_actions({}),
"uuid": str(step.uuid) if step.uuid else None,
+ "when": step.when_expression,
"workflow_outputs": [],
}
if tooltip:
@@ -1360,6 +1376,7 @@ class WorkflowContentsManager(UsesAnnotations):
"uuid": str(step.uuid),
"label": step.label or None,
"annotation": annotation_str,
+ "when": step.when_expression,
}
if step.type == "tool":
step_dict["tool_id"] = content_id if allow_upgrade else step.tool_id
@@ -1547,6 +1564,7 @@ class WorkflowContentsManager(UsesAnnotations):
"annotation": self.get_item_annotation_str(sa_session, stored.user, step),
"tool_inputs": step.tool_inputs,
"input_steps": {},
+ "when": step.when_expression,
}
if step_type == "subworkflow":
@@ -1668,7 +1686,11 @@ class WorkflowContentsManager(UsesAnnotations):
self.add_item_annotation(sa_session, trans.get_user(), step, annotation)
# Stick this in the step temporarily
- step.temp_input_connections = step_dict.get("input_connections", {})
+ DictConnection = Dict[str, Union[int, str]]
+ temp_input_connections: Dict[str, Union[List[DictConnection], DictConnection]] = step_dict.get(
+ "input_connections", {}
+ )
+ step.temp_input_connections = temp_input_connections
# Create the model class for the step
steps.append(step)
@@ -1709,8 +1731,11 @@ class WorkflowContentsManager(UsesAnnotations):
step_input.default_value = default
step_input.default_value_set = True
+ if "when" in step_dict:
+ step.when_expression = step_dict["when"]
if dry_run and step in trans.sa_session:
trans.sa_session.expunge(step)
+
return module, step
def __load_subworkflow_from_step_dict(
diff --git a/lib/galaxy/model/__init__.py b/lib/galaxy/model/__init__.py
index 2d74b52e335..8375e61e572 100644
--- a/lib/galaxy/model/__init__.py
+++ b/lib/galaxy/model/__init__.py
@@ -152,6 +152,9 @@ from galaxy.util.hash_util import (
from galaxy.util.json import safe_loads
from galaxy.util.sanitize_html import sanitize_html
+if TYPE_CHECKING:
+ from galaxy.schema.invocation import InvocationMessageUnion
+
log = logging.getLogger(__name__)
_datatypes_registry = None
@@ -1072,6 +1075,7 @@ class Job(Base, JobLike, UsesCreateAndUpdateTime, Dictifiable, Serializable):
DELETED_NEW = "deleted_new" # now DELETING, remove after 21.0
STOPPING = "stop"
STOPPED = "stopped"
+ SKIPPED = "skipped"
terminal_states = [states.OK, states.ERROR, states.DELETED]
#: job states where the job hasn't finished and the model may still change
@@ -2958,10 +2962,10 @@ class History(Base, HasTags, Dictifiable, UsesAnnotations, HasName, Serializable
.filter(not_(HistoryDatasetAssociation.deleted))
.order_by(HistoryDatasetAssociation.table.c.hid.asc())
.options(
- joinedload("dataset"),
- joinedload("dataset.actions"),
- joinedload("dataset.actions.role"),
- joinedload("tags"),
+ joinedload(HistoryDatasetAssociation.dataset)
+ .joinedload(Dataset.actions)
+ .joinedload(DatasetPermissions.role),
+ joinedload(HistoryDatasetAssociation.tags),
)
)
@@ -2989,7 +2993,10 @@ class History(Base, HasTags, Dictifiable, UsesAnnotations, HasName, Serializable
.filter(not_(HistoryDatasetCollectionAssociation.deleted))
.filter(HistoryDatasetCollectionAssociation.visible)
.order_by(HistoryDatasetCollectionAssociation.table.c.hid.asc())
- .options(joinedload("collection"), joinedload("tags"))
+ .options(
+ joinedload(HistoryDatasetCollectionAssociation.collection),
+ joinedload(HistoryDatasetCollectionAssociation.tags),
+ )
)
self._active_visible_dataset_collections = query.all()
return self._active_visible_dataset_collections
@@ -4260,7 +4267,7 @@ class DatasetInstance(UsesCreateAndUpdateTime, _HasTable):
self, accepted_formats: List[str], **kwd
) -> Tuple[bool, Optional[str], Optional["DatasetInstance"]]:
"""Returns ( target_ext, existing converted dataset )"""
- return self.datatype.find_conversion_destination(self, accepted_formats, _get_datatypes_registry(), **kwd)
+ return self.datatype.find_conversion_destination(self, accepted_formats, _get_datatypes_registry(), **kwd) # type: ignore[arg-type]
def add_validation_error(self, validation_error):
self.validation_errors.append(validation_error)
@@ -7040,6 +7047,7 @@ class WorkflowStep(Base, RepresentById):
position = Column(MutableJSONType)
config = Column(JSONType)
order_index: int = Column(Integer)
+ when_expression = Column(JSONType)
uuid = Column(UUIDType)
label = Column(Unicode(255))
temp_input_connections: Optional[InputConnDictType]
@@ -7084,10 +7092,12 @@ class WorkflowStep(Base, RepresentById):
def __init__(self):
self.uuid = uuid4()
self._input_connections_by_name = None
+ self._inputs_by_name = None
@reconstructor
def init_on_load(self):
self._input_connections_by_name = None
+ self._inputs_by_name = None
@property
def tool_uuid(self):
@@ -7113,6 +7123,23 @@ class WorkflowStep(Base, RepresentById):
tool_state = self.tool_inputs
return tool_state.get("optional") or False
+ def setup_inputs_by_name(self):
+ # Ensure input_connections has already been set.
+
+ # Make connection information available on each step by input name.
+ inputs_by_name = {}
+ for step_input in self.inputs:
+ input_name = step_input.name
+ assert input_name not in inputs_by_name
+ inputs_by_name[input_name] = step_input
+ self._inputs_by_name = inputs_by_name
+
+ @property
+ def inputs_by_name(self):
+ if self._inputs_by_name is None:
+ self.setup_inputs_by_name()
+ return self._inputs_by_name
+
def get_input(self, input_name):
for step_input in self.inputs:
if step_input.name == input_name:
@@ -7136,8 +7163,20 @@ class WorkflowStep(Base, RepresentById):
conn.output_name = output_name
add_object_to_object_session(conn, output_step)
conn.output_step = output_step
- if input_subworkflow_step_index is not None:
- input_subworkflow_step = self.subworkflow.step_by_index(input_subworkflow_step_index)
+ if self.subworkflow:
+ if input_subworkflow_step_index is not None:
+ input_subworkflow_step = self.subworkflow.step_by_index(input_subworkflow_step_index)
+ else:
+ input_subworkflow_steps = [step for step in self.subworkflow.input_steps if step.label == input_name]
+ if not input_subworkflow_steps:
+ inferred_order_index = input_name.split(":", 1)[0]
+ if inferred_order_index.isdigit():
+ input_subworkflow_steps = [self.subworkflow.step_by_index(int(inferred_order_index))]
+ if len(input_subworkflow_steps) != 1:
+ raise galaxy.exceptions.MessageException(
+ f"Invalid subworkflow connection at step index {self.order_index + 1}"
+ )
+ input_subworkflow_step = input_subworkflow_steps[0]
conn.input_subworkflow_step = input_subworkflow_step
return conn
@@ -7482,9 +7521,26 @@ class WorkflowInvocation(Base, UsesCreateAndUpdateTime, Dictifiable, Serializabl
)
output_datasets = relationship("WorkflowInvocationOutputDatasetAssociation", back_populates="workflow_invocation")
output_values = relationship("WorkflowInvocationOutputValue", back_populates="workflow_invocation")
+ messages = relationship("WorkflowInvocationMessage", back_populates="workflow_invocation")
- dict_collection_visible_keys = ["id", "update_time", "create_time", "workflow_id", "history_id", "uuid", "state"]
- dict_element_visible_keys = ["id", "update_time", "create_time", "workflow_id", "history_id", "uuid", "state"]
+ dict_collection_visible_keys = [
+ "id",
+ "update_time",
+ "create_time",
+ "workflow_id",
+ "history_id",
+ "uuid",
+ "state",
+ ]
+ dict_element_visible_keys = [
+ "id",
+ "update_time",
+ "create_time",
+ "workflow_id",
+ "history_id",
+ "uuid",
+ "state",
+ ]
class states(str, Enum):
NEW = "new" # Brand new workflow invocation... maybe this should be same as READY
@@ -7560,7 +7616,7 @@ class WorkflowInvocation(Base, UsesCreateAndUpdateTime, Dictifiable, Serializabl
step_invocations[step_id] = invocation_step
return step_invocations
- def step_invocation_for_step_id(self, step_id):
+ def step_invocation_for_step_id(self, step_id: int) -> Optional["WorkflowInvocationStep"]:
target_invocation_step = None
for invocation_step in self.steps:
if step_id == invocation_step.workflow_step_id:
@@ -7857,6 +7913,19 @@ class WorkflowInvocation(Base, UsesCreateAndUpdateTime, Dictifiable, Serializabl
attach_step(request_to_content)
self.input_step_parameters.append(request_to_content)
+ def add_message(self, message: "InvocationMessageUnion"):
+ self.messages.append(
+ WorkflowInvocationMessage(
+ workflow_invocation_id=self.id,
+ **message.dict(
+ exclude_unset=True,
+ exclude={
+ "history_id"
+ }, # history_id comes in through workflow_invocation and isn't persisted in database
+ ),
+ )
+ )
+
@property
def resource_parameters(self):
resource_type = WorkflowRequestInputParameter.types.RESOURCE_PARAMETERS
@@ -7918,6 +7987,36 @@ class WorkflowInvocationToSubworkflowInvocationAssociation(Base, Dictifiable, Re
dict_element_visible_keys = ["id", "workflow_step_id", "workflow_invocation_id", "subworkflow_invocation_id"]
+class WorkflowInvocationMessage(Base, Dictifiable, Serializable):
+ __tablename__ = "workflow_invocation_message"
+ id = Column(Integer, primary_key=True)
+ workflow_invocation_id = Column(Integer, ForeignKey("workflow_invocation.id"), index=True, nullable=False)
+ reason = Column(String(32))
+ details = Column(TrimmedString(255), nullable=True)
+ output_name = Column(String(255), nullable=True)
+ workflow_step_id = Column(Integer, ForeignKey("workflow_step.id"), nullable=True)
+ dependent_workflow_step_id = Column(Integer, ForeignKey("workflow_step.id"), nullable=True)
+ job_id = Column(Integer, ForeignKey("job.id"), nullable=True)
+ hda_id = Column(Integer, ForeignKey("history_dataset_association.id"), nullable=True)
+ hdca_id = Column(Integer, ForeignKey("history_dataset_collection_association.id"), nullable=True)
+
+ workflow_invocation = relationship("WorkflowInvocation", back_populates="messages", lazy=True)
+ workflow_step = relationship("WorkflowStep", foreign_keys=workflow_step_id, lazy=True)
+ dependent_workflow_step = relationship("WorkflowStep", foreign_keys=dependent_workflow_step_id, lazy=True)
+
+ @property
+ def workflow_step_index(self):
+ return self.workflow_step and self.workflow_step.order_index
+
+ @property
+ def dependent_workflow_step_index(self):
+ return self.dependent_workflow_step and self.dependent_workflow_step.order_index
+
+ @property
+ def history_id(self):
+ return self.workflow_invocation.history_id
+
+
class WorkflowInvocationStep(Base, Dictifiable, Serializable):
__tablename__ = "workflow_invocation_step"
diff --git a/lib/galaxy/model/dataset_collections/matching.py b/lib/galaxy/model/dataset_collections/matching.py
index 4b1d7efbff6..948e317ec69 100644
--- a/lib/galaxy/model/dataset_collections/matching.py
+++ b/lib/galaxy/model/dataset_collections/matching.py
@@ -48,6 +48,7 @@ class MatchingCollections:
self.collections = {}
self.subcollection_types = {}
self.action_tuples = {}
+ self.when_values = None
def __attempt_add_to_linked_match(self, input_name, hdca, collection_type_description, subcollection_type):
structure = get_structure(hdca, collection_type_description, leaf_subcollection_type=subcollection_type)
@@ -62,6 +63,7 @@ class MatchingCollections:
self.subcollection_types[input_name] = subcollection_type
def slice_collections(self):
+ self.linked_structure.when_values = self.when_values
return self.linked_structure.walk_collections(self.collections)
def subcollection_mapping_type(self, input_name):
@@ -77,6 +79,7 @@ class MatchingCollections:
if linked_structure is None:
linked_structure = leaf
effective_structure = effective_structure.multiply(linked_structure)
+ effective_structure.when_values = self.when_values
return None if effective_structure.is_leaf else effective_structure
def map_over_action_tuples(self, input_name):
diff --git a/lib/galaxy/model/dataset_collections/structure.py b/lib/galaxy/model/dataset_collections/structure.py
index 79c85b76edd..159f5ad5562 100644
--- a/lib/galaxy/model/dataset_collections/structure.py
+++ b/lib/galaxy/model/dataset_collections/structure.py
@@ -63,9 +63,10 @@ class UninitializedTree(BaseTree):
class Tree(BaseTree):
children_known = True
- def __init__(self, children, collection_type_description):
+ def __init__(self, children, collection_type_description, when_values=None):
super().__init__(collection_type_description)
self.children = children
+ self.when_values = when_values
@staticmethod
def for_dataset_collection(dataset_collection, collection_type_description):
@@ -94,10 +95,11 @@ class Tree(BaseTree):
return collection[index] # noqa: B023
if substructure.is_leaf:
- yield dict_map(get_element, collection_dict)
+ yield dict_map(get_element, collection_dict), self.when_values[index] if self.when_values else None
else:
sub_collections = dict_map(lambda collection: get_element(collection).child_collection, collection_dict)
- yield from substructure._walk_collections(sub_collections)
+ for element, _when_value in substructure._walk_collections(sub_collections):
+ yield element, self.when_values[index] if self.when_values else None
@property
def is_leaf(self):
@@ -139,7 +141,7 @@ class Tree(BaseTree):
return Tree(cloned_children, self.collection_type_description)
def __str__(self):
- return f"Tree[collection_type={self.collection_type_description},children={','.join(map(lambda identifier_and_element: '{}={}'.format(identifier_and_element[0], identifier_and_element[1]), self.children))}]"
+ return f"Tree[collection_type={self.collection_type_description},children={','.join(map(lambda identifier_and_element: f'{identifier_and_element[0]}={identifier_and_element[1]}', self.children))}]"
def tool_output_to_structure(get_sliced_input_collection_structure, tool_output, collections_manager):
diff --git a/lib/galaxy/model/migrations/alembic/versions_gxy/3100452fa030_add_workflow_invocation_message_table.py b/lib/galaxy/model/migrations/alembic/versions_gxy/3100452fa030_add_workflow_invocation_message_table.py
new file mode 100644
index 00000000000..263a47110cc
--- /dev/null
+++ b/lib/galaxy/model/migrations/alembic/versions_gxy/3100452fa030_add_workflow_invocation_message_table.py
@@ -0,0 +1,46 @@
+"""Add reason column on invocation table
+
+Revision ID: 3100452fa030
+Revises: 518c8438a91b
+Create Date: 2023-01-13 16:13:09.578391
+
+"""
+from alembic import op
+from sqlalchemy import (
+ Column,
+ ForeignKey,
+ Integer,
+ String,
+)
+
+from galaxy.model.custom_types import TrimmedString
+
+# revision identifiers, used by Alembic.
+revision = "3100452fa030"
+down_revision = "518c8438a91b"
+branch_labels = None
+depends_on = None
+
+
+# database object names used in this revision
+table_name = "workflow_invocation_message"
+
+
+def upgrade():
+ op.create_table(
+ table_name,
+ Column("id", Integer, primary_key=True),
+ Column("reason", String(32)),
+ Column("details", TrimmedString(255)),
+ Column("output_name", String(255)),
+ Column("workflow_invocation_id", Integer, ForeignKey("workflow_invocation.id"), index=True, nullable=False),
+ Column("workflow_step_id", Integer, ForeignKey("workflow_step.id"), nullable=True),
+ Column("dependent_workflow_step_id", Integer, ForeignKey("workflow_step.id"), nullable=True),
+ Column("job_id", Integer, ForeignKey("job.id"), nullable=True),
+ Column("hda_id", Integer, ForeignKey("history_dataset_association.id"), nullable=True),
+ Column("hdca_id", Integer, ForeignKey("history_dataset_collection_association.id"), nullable=True),
+ )
+
+
+def downgrade():
+ op.drop_table(table_name)
diff --git a/lib/galaxy/model/migrations/alembic/versions_gxy/518c8438a91b_add_when_expression_column.py b/lib/galaxy/model/migrations/alembic/versions_gxy/518c8438a91b_add_when_expression_column.py
new file mode 100644
index 00000000000..879ec4e35ac
--- /dev/null
+++ b/lib/galaxy/model/migrations/alembic/versions_gxy/518c8438a91b_add_when_expression_column.py
@@ -0,0 +1,34 @@
+"""Add when_expression column
+
+Revision ID: 518c8438a91b
+Revises: 59e024ceaca1
+Create Date: 2022-10-24 16:43:39.565871
+
+"""
+import sqlalchemy as sa
+from alembic import op
+
+from galaxy.model.custom_types import JSONType
+from galaxy.model.migrations.util import (
+ column_exists,
+ drop_column,
+)
+
+# revision identifiers, used by Alembic.
+revision = "518c8438a91b"
+down_revision = "59e024ceaca1"
+branch_labels = None
+depends_on = None
+
+# database object names used in this revision
+table_name = "workflow_step"
+column_name = "when_expression"
+
+
+def upgrade():
+ if not column_exists(table_name, column_name):
+ op.add_column(table_name, sa.Column(column_name, JSONType))
+
+
+def downgrade():
+ drop_column(table_name, column_name)
diff --git a/lib/galaxy/model/migrations/alembic/versions_gxy/c39f1de47a04_add_skipped_state_to_collection_job_.py b/lib/galaxy/model/migrations/alembic/versions_gxy/c39f1de47a04_add_skipped_state_to_collection_job_.py
new file mode 100644
index 00000000000..8474c471106
--- /dev/null
+++ b/lib/galaxy/model/migrations/alembic/versions_gxy/c39f1de47a04_add_skipped_state_to_collection_job_.py
@@ -0,0 +1,99 @@
+"""Add skipped state to collection_job_state_summary_view
+
+Revision ID: c39f1de47a04
+Revises: 3100452fa030
+Create Date: 2023-01-16 11:53:59.783836
+
+"""
+from typing import Generator
+
+from alembic import (
+ context,
+ op,
+)
+from alembic_utils.pg_view import PGView
+from sqlalchemy import text as sql_text
+from sqlalchemy.engine.url import make_url
+from sqlalchemy.sql.elements import TextClause
+
+from galaxy.model.view import HistoryDatasetCollectionJobStateSummary
+
+# revision identifiers, used by Alembic.
+revision = "c39f1de47a04"
+down_revision = "3100452fa030"
+branch_labels = None
+depends_on = None
+
+PREVIOUS_AGGREGATE_QUERY = """
+SELECT
+ hdca_id,
+ SUM(CASE WHEN state = 'new' THEN 1 ELSE 0 END) AS new,
+ SUM(CASE WHEN state = 'resubmitted' THEN 1 ELSE 0 END) AS resubmitted,
+ SUM(CASE WHEN state = 'waiting' THEN 1 ELSE 0 END) AS waiting,
+ SUM(CASE WHEN state = 'queued' THEN 1 ELSE 0 END) AS queued,
+ SUM(CASE WHEN state = 'running' THEN 1 ELSE 0 END) AS running,
+ SUM(CASE WHEN state = 'ok' THEN 1 ELSE 0 END) AS ok,
+ SUM(CASE WHEN state = 'error' THEN 1 ELSE 0 END) AS error,
+ SUM(CASE WHEN state = 'failed' THEN 1 ELSE 0 END) AS failed,
+ SUM(CASE WHEN state = 'paused' THEN 1 ELSE 0 END) AS paused,
+ SUM(CASE WHEN state = 'deleted' THEN 1 ELSE 0 END) AS deleted,
+ SUM(CASE WHEN state = 'deleted_new' THEN 1 ELSE 0 END) AS deleted_new,
+ SUM(CASE WHEN state = 'upload' THEN 1 ELSE 0 END) AS upload,
+ SUM(CASE WHEN job_id IS NOT NULL THEN 1 ELSE 0 END) AS all_jobs
+FROM (
+ SELECT hdca.id AS hdca_id, job.id AS job_id, job.state as state
+ FROM history_dataset_collection_association hdca
+ LEFT JOIN implicit_collection_jobs icj
+ ON icj.id = hdca.implicit_collection_jobs_id
+ LEFT JOIN implicit_collection_jobs_job_association icjja
+ ON icj.id = icjja.implicit_collection_jobs_id
+ LEFT JOIN job
+ ON icjja.job_id = job.id
+
+ UNION
+
+ SELECT hdca.id AS hdca_id, job.id AS job_id, job.state AS state
+ FROM history_dataset_collection_association hdca
+ LEFT JOIN job
+ ON hdca.job_id = job.id
+) jobstates
+GROUP BY jobstates.hdca_id
+"""
+
+
+class SqlitePGView(PGView):
+ def to_sql_statement_create_or_replace(self) -> Generator[TextClause, None, None]:
+ """Generates a SQL "create or replace view" statement"""
+
+ yield sql_text(f"""DROP VIEW IF EXISTS {self.literal_schema}."{self.signature}";""")
+ yield sql_text(f"""CREATE VIEW {self.literal_schema}."{self.signature}" AS {self.definition};""")
+
+
+def get_view_instance(definition: str):
+
+ try:
+ url = make_url(context.config.get_main_option("sqlalchemy.url"))
+ dialect = url.get_dialect().name
+ except Exception:
+ # If someone's doing offline migrations they're probably using PostgreSQL
+ dialect = "postgresql"
+ if dialect == "postgresql":
+ ViewClass = PGView
+ schema = "public"
+ elif dialect == "sqlite":
+ ViewClass = SqlitePGView
+ schema = "main"
+ else:
+ raise Exception(f"Don't know how to generate view for dialect '{dialect}'")
+ return ViewClass(schema=schema, signature="collection_job_state_summary_view", definition=definition)
+
+
+def upgrade():
+ view = get_view_instance(HistoryDatasetCollectionJobStateSummary.aggregate_state_query)
+ # op.replace_entity comes from alembic_utils plugin
+ op.replace_entity(view) # type: ignore[attr-defined]
+
+
+def downgrade():
+ view = get_view_instance(PREVIOUS_AGGREGATE_QUERY)
+ op.replace_entity(view) # type: ignore[attr-defined]
diff --git a/lib/galaxy/model/security.py b/lib/galaxy/model/security.py
index 29159b70d0e..881e3276bf4 100644
--- a/lib/galaxy/model/security.py
+++ b/lib/galaxy/model/security.py
@@ -1507,12 +1507,15 @@ class GalaxyRBACAgent(RBACAgent):
return True, ""
action = self.permitted_actions.DATASET_ACCESS
+ raise
lddas = (
self.sa_session.query(self.model.LibraryDatasetDatasetAssociation)
.join("library_dataset")
.filter(self.model.LibraryDataset.folder == folder)
.join("dataset")
- .options(joinedload("dataset").joinedload("actions"))
+ .options(
+ joinedload(self.model.LibraryDatasetDatasetAssociation.dataset).joinedload(self.model.Dataset.actions)
+ )
.all()
)
diff --git a/lib/galaxy/model/view/__init__.py b/lib/galaxy/model/view/__init__.py
index 0bb5c2771a1..04fd7d64870 100644
--- a/lib/galaxy/model/view/__init__.py
+++ b/lib/galaxy/model/view/__init__.py
@@ -26,6 +26,7 @@ SELECT
SUM(CASE WHEN state = 'error' THEN 1 ELSE 0 END) AS error,
SUM(CASE WHEN state = 'failed' THEN 1 ELSE 0 END) AS failed,
SUM(CASE WHEN state = 'paused' THEN 1 ELSE 0 END) AS paused,
+ SUM(CASE WHEN state = 'skipped' THEN 1 ELSE 0 END) AS skipped,
SUM(CASE WHEN state = 'deleted' THEN 1 ELSE 0 END) AS deleted,
SUM(CASE WHEN state = 'deleted_new' THEN 1 ELSE 0 END) AS deleted_new,
SUM(CASE WHEN state = 'upload' THEN 1 ELSE 0 END) AS upload,
@@ -61,6 +62,7 @@ GROUP BY jobstates.hdca_id
column("error", Integer),
column("failed", Integer),
column("paused", Integer),
+ column("skipped", Integer),
column("deleted", Integer),
column("deleted_new", Integer),
column("upload", Integer),
diff --git a/lib/galaxy/schema/invocation.py b/lib/galaxy/schema/invocation.py
new file mode 100644
index 00000000000..6803d2183e9
--- /dev/null
+++ b/lib/galaxy/schema/invocation.py
@@ -0,0 +1,214 @@
+from enum import Enum
+from typing import (
+ Any,
+ Generic,
+ Optional,
+ TypeVar,
+ Union,
+)
+
+from pydantic import (
+ BaseModel,
+ Field,
+)
+from pydantic.generics import GenericModel
+from pydantic.utils import GetterDict
+from typing_extensions import (
+ Annotated,
+ Literal,
+)
+
+from galaxy.schema.fields import EncodedDatabaseIdField
+
+
+class WarningReason(str, Enum):
+ workflow_output_not_found = "workflow_output_not_found"
+
+
+class FailureReason(str, Enum):
+ dataset_failed = "dataset_failed"
+ collection_failed = "collection_failed"
+ job_failed = "job_failed"
+ output_not_found = "output_not_found"
+ expression_evaluation_failed = "expression_evaluation_failed"
+ when_not_boolean = "when_not_boolean"
+ unexpected_failure = "unexpected_failure"
+
+
+class CancelReason(str, Enum):
+ """Possible reasons for a cancelled workflow."""
+
+ history_deleted = "history_deleted"
+ user_request = "user_request"
+ cancelled_on_review = "cancelled_on_review"
+
+
+DatabaseIdT = TypeVar("DatabaseIdT")
+
+
+class StepOrderIndexGetter(GetterDict):
+ def get(self, key: Any, default: Any = None) -> Any:
+
+ # Fetch the order_index when serializing for the API,
+ # which makes much more sense when pointing to steps.
+ if key == "workflow_step_id":
+ return self._obj.workflow_step.order_index
+ elif key == "dependent_workflow_step_id":
+ return self._obj.dependent_workflow_step.order_index
+
+ return super().get(key, default)
+
+
+class InvocationMessageBase(GenericModel):
+ reason: Union[CancelReason, FailureReason, WarningReason]
+
+ class Config:
+ orm_mode = True
+ getter_dict = StepOrderIndexGetter
+
+
+class GenericInvocationCancellationReviewFailed(InvocationMessageBase, Generic[DatabaseIdT]):
+ reason: Literal[CancelReason.cancelled_on_review]
+ workflow_step_id: int = Field(..., description="Workflow step id of paused step that did not pass review.")
+
+
+class GenericInvocationCancellationHistoryDeleted(InvocationMessageBase, Generic[DatabaseIdT]):
+ reason: Literal[CancelReason.history_deleted]
+ history_id: DatabaseIdT = Field(..., title="History ID", description="History ID of history that was deleted.")
+
+
+class GenericInvocationCancellationUserRequest(InvocationMessageBase, Generic[DatabaseIdT]):
+ reason: Literal[CancelReason.user_request]
+
+
+class InvocationFailureMessageBase(InvocationMessageBase, Generic[DatabaseIdT]):
+ workflow_step_id: int = Field(..., description="Workflow step id of step that failed.")
+
+
+class GenericInvocationFailureDatasetFailed(InvocationFailureMessageBase[DatabaseIdT], Generic[DatabaseIdT]):
+ reason: Literal[FailureReason.dataset_failed]
+ hda_id: DatabaseIdT = Field(
+ ..., title="HistoryDatasetAssociation ID", description="HistoryDatasetAssociation ID that relates to failure."
+ )
+ dependent_workflow_step_id: Optional[int] = Field(None, description="Workflow step id of step that caused failure.")
+
+
+class GenericInvocationFailureCollectionFailed(InvocationFailureMessageBase[DatabaseIdT], Generic[DatabaseIdT]):
+ reason: Literal[FailureReason.collection_failed]
+ hdca_id: DatabaseIdT = Field(
+ None,
+ title="HistoryDatasetCollectionAssociation ID",
+ description="HistoryDatasetCollectionAssociation ID that relates to failure.",
+ )
+ dependent_workflow_step_id: int = Field(..., description="Workflow step id of step that caused failure.")
+
+
+class GenericInvocationFailureJobFailed(InvocationFailureMessageBase[DatabaseIdT], Generic[DatabaseIdT]):
+ reason: Literal[FailureReason.job_failed]
+ job_id: DatabaseIdT = Field(None, title="Job ID", description="Job ID that relates to failure.")
+ dependent_workflow_step_id: int = Field(..., description="Workflow step id of step that caused failure.")
+
+
+class GenericInvocationFailureOutputNotFound(InvocationFailureMessageBase[DatabaseIdT], Generic[DatabaseIdT]):
+ reason: Literal[FailureReason.output_not_found]
+ output_name: str = Field(..., title="Tool or module output name that was referenced but not produced")
+ dependent_workflow_step_id: int = Field(..., description="Workflow step id of step that caused failure.")
+
+
+class GenericInvocationFailureExpressionEvaluationFailed(
+ InvocationFailureMessageBase[DatabaseIdT], Generic[DatabaseIdT]
+):
+ reason: Literal[FailureReason.expression_evaluation_failed]
+ details: Optional[str] = Field(None, description="May contain details to help troubleshoot this problem.")
+
+
+class GenericInvocationFailureWhenNotBoolean(InvocationFailureMessageBase[DatabaseIdT], Generic[DatabaseIdT]):
+ reason: Literal[FailureReason.when_not_boolean]
+ details: str = Field(..., description="Contains details to help troubleshoot this problem.")
+
+
+class GenericInvocationUnexpectedFailure(InvocationMessageBase, Generic[DatabaseIdT]):
+ reason: Literal[FailureReason.unexpected_failure]
+ details: Optional[str] = Field(None, description="May contains details to help troubleshoot this problem.")
+
+
+class GenericInvocationWarning(InvocationMessageBase, Generic[DatabaseIdT]):
+ reason: WarningReason = Field(..., title="Failure Reason", description="Reason for warning")
+ workflow_step_id: Optional[int] = Field(None, title="Workflow step id of step that caused a warning.")
+
+
+class GenericInvocationEvaluationWarningWorkflowOutputNotFound(
+ GenericInvocationWarning[DatabaseIdT], Generic[DatabaseIdT]
+):
+ reason: Literal[WarningReason.workflow_output_not_found]
+ workflow_step_id: int = Field(..., title="Workflow step id of step that caused a warning.")
+ output_name: str = Field(
+ ..., description="Output that was designated as workflow output but that has not been found"
+ )
+
+
+InvocationCancellationReviewFailed = GenericInvocationCancellationReviewFailed[int]
+InvocationCancellationHistoryDeleted = GenericInvocationCancellationHistoryDeleted[int]
+InvocationCancellationUserRequest = GenericInvocationCancellationUserRequest[int]
+InvocationFailureDatasetFailed = GenericInvocationFailureDatasetFailed[int]
+InvocationFailureCollectionFailed = GenericInvocationFailureCollectionFailed[int]
+InvocationFailureJobFailed = GenericInvocationFailureJobFailed[int]
+InvocationFailureOutputNotFound = GenericInvocationFailureOutputNotFound[int]
+InvocationFailureExpressionEvaluationFailed = GenericInvocationFailureExpressionEvaluationFailed[int]
+InvocationFailureWhenNotBoolean = GenericInvocationFailureWhenNotBoolean[int]
+InvocationUnexpectedFailure = GenericInvocationUnexpectedFailure[int]
+InvocationWarningWorkflowOutputNotFound = GenericInvocationEvaluationWarningWorkflowOutputNotFound[int]
+
+InvocationMessageUnion = Union[
+ InvocationCancellationReviewFailed,
+ InvocationCancellationHistoryDeleted,
+ InvocationCancellationUserRequest,
+ InvocationFailureDatasetFailed,
+ InvocationFailureCollectionFailed,
+ InvocationFailureJobFailed,
+ InvocationFailureOutputNotFound,
+ InvocationFailureExpressionEvaluationFailed,
+ InvocationFailureWhenNotBoolean,
+ InvocationUnexpectedFailure,
+ InvocationWarningWorkflowOutputNotFound,
+]
+
+InvocationCancellationReviewFailedResponseModel = GenericInvocationCancellationReviewFailed[EncodedDatabaseIdField]
+InvocationCancellationHistoryDeletedResponseModel = GenericInvocationCancellationHistoryDeleted[EncodedDatabaseIdField]
+InvocationCancellationUserRequestResponseModel = GenericInvocationCancellationUserRequest[EncodedDatabaseIdField]
+InvocationFailureDatasetFailedResponseModel = GenericInvocationFailureDatasetFailed[EncodedDatabaseIdField]
+InvocationFailureCollectionFailedResponseModel = GenericInvocationFailureCollectionFailed[EncodedDatabaseIdField]
+InvocationFailureJobFailedResponseModel = GenericInvocationFailureJobFailed[EncodedDatabaseIdField]
+InvocationFailureOutputNotFoundResponseModel = GenericInvocationFailureOutputNotFound[EncodedDatabaseIdField]
+InvocationFailureExpressionEvaluationFailedResponseModel = GenericInvocationFailureExpressionEvaluationFailed[
+ EncodedDatabaseIdField
+]
+InvocationFailureWhenNotBooleanResponseModel = GenericInvocationFailureWhenNotBoolean[EncodedDatabaseIdField]
+InvocationUnexpectedFailureResponseModel = GenericInvocationUnexpectedFailure[EncodedDatabaseIdField]
+InvocationWarningWorkflowOutputNotFoundResponseModel = GenericInvocationEvaluationWarningWorkflowOutputNotFound[
+ EncodedDatabaseIdField
+]
+
+InvocationMessageResponseUnion = Annotated[
+ Union[
+ InvocationCancellationReviewFailedResponseModel,
+ InvocationCancellationHistoryDeletedResponseModel,
+ InvocationCancellationUserRequestResponseModel,
+ InvocationFailureDatasetFailedResponseModel,
+ InvocationFailureCollectionFailedResponseModel,
+ InvocationFailureJobFailedResponseModel,
+ InvocationFailureOutputNotFoundResponseModel,
+ InvocationFailureExpressionEvaluationFailedResponseModel,
+ InvocationFailureWhenNotBooleanResponseModel,
+ InvocationUnexpectedFailureResponseModel,
+ InvocationWarningWorkflowOutputNotFoundResponseModel,
+ ],
+ Field(discriminator="reason"),
+]
+
+
+class InvocationMessageResponseModel(BaseModel):
+ __root__: InvocationMessageResponseUnion
+
+ class Config:
+ orm_mode = True
diff --git a/lib/galaxy/schema/schema.py b/lib/galaxy/schema/schema.py
index 2c81fab87f9..658a20c3eaf 100644
--- a/lib/galaxy/schema/schema.py
+++ b/lib/galaxy/schema/schema.py
@@ -733,6 +733,11 @@ class HDCJobStateSummary(Model):
title="Paused jobs",
description="Number of jobs in the `paused` state.",
)
+ skipped: int = Field(
+ 0,
+ title="Skipped jobs",
+ description="Number of jobs that were skipped due to conditional workflow step execution.",
+ )
deleted_new: int = Field(
0,
title="Deleted new jobs",
@@ -3043,6 +3048,14 @@ class UserEmail(Model):
)
+class UserBeaconSetting(Model):
+ enabled: bool = Field(
+ ...,
+ title="Enabled",
+ description="True if beacon sharing is enabled",
+ )
+
+
class SharingStatus(Model):
id: DecodedDatabaseIdField = Field(
...,
diff --git a/lib/galaxy/tool_util/cwl/parser.py b/lib/galaxy/tool_util/cwl/parser.py
index 97d79c1fc98..5f0bd18563c 100644
--- a/lib/galaxy/tool_util/cwl/parser.py
+++ b/lib/galaxy/tool_util/cwl/parser.py
@@ -4,12 +4,10 @@ adapt cwltool to Galaxy features and abstract the library away from the rest
of the framework.
"""
-import base64
import copy
import json
import logging
import os
-import pickle
from abc import (
ABCMeta,
abstractmethod,
@@ -31,7 +29,6 @@ from galaxy.exceptions import MessageException
from galaxy.util import (
listify,
safe_makedirs,
- unicodify,
)
from galaxy.util.bunch import Bunch
from .cwltool_deps import (
@@ -86,8 +83,6 @@ SUPPORTED_TOOL_REQUIREMENTS = [
SUPPORTED_WORKFLOW_REQUIREMENTS = SUPPORTED_TOOL_REQUIREMENTS + []
-PERSISTED_REPRESENTATION = "cwl_tool_object"
-
ToolStateType = Dict[str, Union[None, str, bool, Dict[str, str]]]
@@ -173,29 +168,29 @@ class ToolProxy(metaclass=ABCMeta):
def to_persistent_representation(self):
"""Return a JSON representation of this tool. Not for serialization
over the wire, but serialization in a database."""
- # TODO: Replace this with some more readable serialization,
- # I really don't like using pickle here.
- if PERSISTED_REPRESENTATION == "cwl_tool_object":
- persisted_obj = remove_pickle_problems(self._tool)
- else:
- persisted_obj = self._raw_process_reference
+ persisted_obj = self._tool.tool
+ persisted_obj["requirements"] = self.requirements
+ if not persisted_obj.get("cwlVersion"):
+ # This happens for any inline process, but getting it from metadata is correct for inline processes at least
+ persisted_obj["cwlVersion"] = self._tool.metadata["cwlVersion"]
return {
"class": self._class,
- "pickle": unicodify(base64.b64encode(pickle.dumps(persisted_obj, pickle.HIGHEST_PROTOCOL))),
+ # Should maybe be yaml instead
+ "raw_process_reference": persisted_obj,
"uuid": self.uuid,
}
@staticmethod
- def from_persistent_representation(as_object) -> "ToolProxy":
+ def from_persistent_representation(as_object, strict_cwl_validation=True, tool_directory=None) -> "ToolProxy":
"""Recover an object serialized with to_persistent_representation."""
if "class" not in as_object:
raise Exception("Failed to deserialize tool proxy from JSON object - no class found.")
- if "pickle" not in as_object:
- raise Exception("Failed to deserialize tool proxy from JSON object - no pickle representation found.")
- if "uuid" not in as_object:
- raise Exception("Failed to deserialize tool proxy from JSON object - no uuid found.")
- to_unpickle = base64.b64decode(as_object["pickle"])
- loaded_object = pickle.loads(to_unpickle)
+ loaded_object = tool_proxy(
+ tool_object=as_object["raw_process_reference"],
+ strict_cwl_validation=strict_cwl_validation,
+ tool_directory=tool_directory,
+ uuid=as_object.get("uuid"),
+ )
return loaded_object
@property
@@ -686,7 +681,6 @@ class WorkflowProxy:
"input_connections": {}, # Should the Galaxy API really require this? - Seems to.
"workflow_outputs": self.get_outputs_for_label(label),
}
-
if input_type == "File" and "default" not in input:
input_as_dict["type"] = "data_input"
elif isinstance(input_type, dict) and input_type.get("type") == "array":
@@ -744,14 +738,8 @@ def tool_proxy_from_persistent_representation(
) -> ToolProxy:
"""Load a ToolProxy from a previously persisted representation."""
ensure_cwltool_available()
- if PERSISTED_REPRESENTATION == "cwl_tool_object":
- kwds = {"cwl_tool_object": ToolProxy.from_persistent_representation(persisted_tool)}
- else:
- raw_process_reference = persisted_tool # ???
- kwds = {"raw_process_reference": ToolProxy.from_persistent_representation(raw_process_reference)}
- uuid = persisted_tool["uuid"]
- return _to_cwl_tool_object(
- uuid=uuid, strict_cwl_validation=strict_cwl_validation, tool_directory=tool_directory, **kwds
+ return ToolProxy.from_persistent_representation(
+ persisted_tool, strict_cwl_validation=strict_cwl_validation, tool_directory=tool_directory
)
@@ -766,15 +754,10 @@ def load_job_proxy(job_directory, strict_cwl_validation=True) -> JobProxy:
job_objects = json.load(open(job_objects_path))
job_inputs = job_objects["job_inputs"]
output_dict = job_objects["output_dict"]
- # Any reason to retain older tool_path variant of this? Probably not?
- if "tool_path" in job_objects:
- tool_path = job_objects["tool_path"]
- cwl_tool = tool_proxy(tool_path, strict_cwl_validation=strict_cwl_validation)
- else:
- persisted_tool = job_objects["tool_representation"]
- cwl_tool = tool_proxy_from_persistent_representation(
- persisted_tool=persisted_tool, strict_cwl_validation=strict_cwl_validation
- )
+ persisted_tool = job_objects["tool_representation"]
+ cwl_tool = tool_proxy_from_persistent_representation(
+ persisted_tool=persisted_tool, strict_cwl_validation=strict_cwl_validation
+ )
return cwl_tool.job_proxy(job_inputs, output_dict, job_directory=job_directory)
@@ -1057,7 +1040,8 @@ class ToolStepProxy(BaseStepProxy):
tool_state[input_name] = None
outputs = self.galaxy_workflow_outputs_list()
- return {
+ when_expression = self._step.tool.get("when")
+ rval = {
"id": self._index,
"tool_uuid": self.tool_proxy.uuid, # TODO: make sure this is respected...
"label": self.label,
@@ -1068,6 +1052,9 @@ class ToolStepProxy(BaseStepProxy):
"inputs": self.inputs_to_dicts(),
"workflow_outputs": outputs,
}
+ if when_expression:
+ rval["when"] = when_expression
+ return rval
class SubworkflowStepProxy(BaseStepProxy):
@@ -1104,17 +1091,6 @@ class SubworkflowStepProxy(BaseStepProxy):
return self._subworkflow_proxy
-def remove_pickle_problems(obj):
- """doc_loader does not pickle correctly"""
- if hasattr(obj, "doc_loader"):
- obj.doc_loader = None
- if hasattr(obj, "embedded_tool"):
- obj.embedded_tool = remove_pickle_problems(obj.embedded_tool)
- if hasattr(obj, "steps"):
- obj.steps = [remove_pickle_problems(s) for s in obj.steps]
- return obj
-
-
def _outer_field_to_input_instance(field):
field_type = field_to_field_type(field) # Must be a list if in here?
if not isinstance(field_type, list):
diff --git a/lib/galaxy/tool_util/deps/conda_util.py b/lib/galaxy/tool_util/deps/conda_util.py
index b5bdfd72473..bc91af0c433 100644
--- a/lib/galaxy/tool_util/deps/conda_util.py
+++ b/lib/galaxy/tool_util/deps/conda_util.py
@@ -24,6 +24,7 @@ import packaging.version
from galaxy.util import (
commands,
+ download_to_file,
listify,
shlex_join,
smart_str,
@@ -483,16 +484,13 @@ def hash_conda_packages(conda_packages: Iterable[CondaTarget]) -> str:
def install_conda(conda_context: CondaContext, force_conda_build: bool = False) -> int:
with tempfile.NamedTemporaryFile(suffix=".sh", prefix="conda_install", delete=False) as temp:
script_path = temp.name
- download_cmd = commands.download_command(conda_link(), to=script_path)
install_cmd = ["bash", script_path, "-b", "-p", conda_context.conda_prefix]
package_targets = list(CONDA_PACKAGE_SPECS)
if force_conda_build or conda_context.use_local:
package_targets.extend(CONDA_BUILD_SPECS)
log.info("Installing conda, this may take several minutes.")
try:
- exit_code = conda_context.shell_exec(download_cmd)
- if exit_code:
- return exit_code
+ download_to_file(conda_link(), script_path)
exit_code = conda_context.shell_exec(install_cmd)
except Exception:
log.exception("Failed to install conda")
diff --git a/lib/galaxy/tool_util/deps/mulled/mulled_build.py b/lib/galaxy/tool_util/deps/mulled/mulled_build.py
index 7f99a829b3f..82ca88e156e 100644
--- a/lib/galaxy/tool_util/deps/mulled/mulled_build.py
+++ b/lib/galaxy/tool_util/deps/mulled/mulled_build.py
@@ -29,6 +29,7 @@ from galaxy.tool_util.deps.conda_util import (
from galaxy.tool_util.deps.docker_util import command_list as docker_command_list
from galaxy.util import (
commands,
+ download_to_file,
safe_makedirs,
shlex_join,
unicodify,
@@ -428,10 +429,12 @@ def ensure_installed(involucro_context, auto_init):
def install_involucro(involucro_context):
install_path = os.path.abspath(involucro_context.involucro_bin)
involucro_context.involucro_bin = install_path
- download_cmd = commands.download_command(involucro_link(), to=install_path)
- exit_code = involucro_context.shell_exec(download_cmd)
- if exit_code:
- return exit_code
+
+ try:
+ download_to_file(involucro_link(), install_path)
+ except Exception:
+ log.exception(f"Failed to download involucro from url '{involucro_link()}'")
+ return 1
try:
os.chmod(install_path, os.stat(install_path).st_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH)
return 0
diff --git a/lib/galaxy/tool_util/xsd/galaxy.xsd b/lib/galaxy/tool_util/xsd/galaxy.xsd
index 59fcba6ea45..0c7ccdea0d5 100644
--- a/lib/galaxy/tool_util/xsd/galaxy.xsd
+++ b/lib/galaxy/tool_util/xsd/galaxy.xsd
@@ -3485,7 +3485,11 @@ or ``select``.
-
diff --git a/lib/galaxy/tools/__init__.py b/lib/galaxy/tools/__init__.py
index fba2844f7d0..a569f884a82 100644
--- a/lib/galaxy/tools/__init__.py
+++ b/lib/galaxy/tools/__init__.py
@@ -1899,6 +1899,7 @@ class Tool(Dictifiable):
collection_info=None,
job_callback=None,
flush_job=True,
+ skip=False,
):
"""
Return a pair with whether execution is successful as well as either
@@ -1916,6 +1917,7 @@ class Tool(Dictifiable):
collection_info=collection_info,
job_callback=job_callback,
flush_job=flush_job,
+ skip=skip,
)
job = rval[0]
out_data = rval[1]
diff --git a/lib/galaxy/tools/actions/__init__.py b/lib/galaxy/tools/actions/__init__.py
index c4c77f5648b..45eba238fd3 100644
--- a/lib/galaxy/tools/actions/__init__.py
+++ b/lib/galaxy/tools/actions/__init__.py
@@ -23,6 +23,7 @@ from galaxy.model import (
)
from galaxy.model.dataset_collections.builder import CollectionBuilder
from galaxy.model.none_like import NoneDataset
+from galaxy.objectstore import ObjectStorePopulator
from galaxy.tools.parameters import update_dataset_ids
from galaxy.tools.parameters.basic import (
DataCollectionToolParameter,
@@ -367,6 +368,7 @@ class DefaultToolAction(ToolAction):
collection_info=None,
job_callback=None,
flush_job=True,
+ skip=False,
):
"""
Executes a tool, creating job and tool outputs, associating them, and
@@ -506,7 +508,7 @@ class DefaultToolAction(ToolAction):
)
data.copy_tags_to(preserved_tags.values())
- # This may not be neccesary with the new parent/child associations
+ # This may not be necessary with the new parent/child associations
data.designation = name
# Copy metadata from one of the inputs if requested.
@@ -638,6 +640,17 @@ class DefaultToolAction(ToolAction):
job_setup_timer = ExecutionTimer()
# Create the job object
job, galaxy_session = self._new_job_for_session(trans, tool, history)
+ if skip:
+ job.state = job.states.SKIPPED
+ for output_collection in output_collections.out_collections.values():
+ output_collection.mark_as_populated()
+ object_store_populator = ObjectStorePopulator(trans.app, trans.user)
+ for data in out_data.values():
+ object_store_populator.set_object_store_id(data)
+ data.extension = "expression.json"
+ data.state = "ok"
+ with open(data.dataset.file_name, "w") as out:
+ out.write(json.dumps(None))
self._record_inputs(trans, tool, job, incoming, inp_data, inp_dataset_collections)
self._record_outputs(job, out_data, output_collections)
# execute immediate post job actions and associate post job actions that are to be executed after the job is complete
@@ -694,7 +707,7 @@ class DefaultToolAction(ToolAction):
trans.sa_session.flush()
log.info(f"Flushed transaction for job {job.log_str()} {job_flush_timer}")
- return job, out_data, history
+ return job, out_data, history
def _remap_job_on_rerun(self, trans, galaxy_session, rerun_remap_job_id, current_job, out_data):
"""
diff --git a/lib/galaxy/tools/actions/upload_common.py b/lib/galaxy/tools/actions/upload_common.py
index 94f752c8faf..412e83d0b54 100644
--- a/lib/galaxy/tools/actions/upload_common.py
+++ b/lib/galaxy/tools/actions/upload_common.py
@@ -422,10 +422,11 @@ def active_folders(trans, folder):
# Stolen from galaxy.web.controllers.library_common (importing from which causes a circular issues).
# Much faster way of retrieving all active sub-folders within a given folder than the
# performance of the mapper. This query also eagerloads the permissions on each folder.
+ raise
return (
trans.sa_session.query(LibraryFolder)
.filter_by(parent=folder, deleted=False)
- .options(joinedload("actions"))
+ .options(joinedload(LibraryFolder.actions))
.order_by(LibraryFolder.table.c.name)
.all()
)
diff --git a/lib/galaxy/tools/execute.py b/lib/galaxy/tools/execute.py
index 507b68abaf5..a325966fb60 100644
--- a/lib/galaxy/tools/execute.py
+++ b/lib/galaxy/tools/execute.py
@@ -90,7 +90,7 @@ def execute(
)
execution_cache = ToolExecutionCache(trans)
- def execute_single_job(execution_slice, completed_job):
+ def execute_single_job(execution_slice, completed_job, skip=False):
job_timer = tool.app.execution_timer_factory.get_timer(
"internals.galaxy.tools.execute.job_single", SINGLE_EXECUTION_SUCCESS_MESSAGE
)
@@ -121,6 +121,7 @@ def execute(
collection_info,
job_callback=job_callback,
flush_job=False,
+ skip=skip,
)
if job:
log.debug(job_timer.to_str(tool_id=tool.id, job_id=job.id))
@@ -161,7 +162,8 @@ def execute(
has_remaining_jobs = True
break
else:
- execute_single_job(execution_slice, completed_jobs[i])
+ skip = execution_slice.param_combination.pop("__when_value__", None) is False
+ execute_single_job(execution_slice, completed_jobs[i], skip=skip)
history = execution_slice.history or history
jobs_executed += 1
@@ -526,7 +528,7 @@ class ToolExecutionTracker(ExecutionTracker):
self.outputs_by_output_name[job_output.name].append(job_output.dataset_collection)
def new_collection_execution_slices(self):
- for job_index, (param_combination, dataset_collection_elements) in enumerate(
+ for job_index, (param_combination, (dataset_collection_elements, _when_value)) in enumerate(
zip(self.param_combinations, self.walk_implicit_collections())
):
completed_job = self.completed_jobs and self.completed_jobs[job_index]
@@ -550,7 +552,7 @@ class WorkflowStepExecutionTracker(ExecutionTracker):
self.invocation_step.job = job
def new_collection_execution_slices(self):
- for job_index, (param_combination, dataset_collection_elements) in enumerate(
+ for job_index, (param_combination, (dataset_collection_elements, _when_value)) in enumerate(
zip(self.param_combinations, self.walk_implicit_collections())
):
completed_job = self.completed_jobs and self.completed_jobs[job_index]
diff --git a/lib/galaxy/tools/parameters/wrapped_json.py b/lib/galaxy/tools/parameters/wrapped_json.py
index 1cac8c2e2e5..912ea3eb2c7 100644
--- a/lib/galaxy/tools/parameters/wrapped_json.py
+++ b/lib/galaxy/tools/parameters/wrapped_json.py
@@ -1,3 +1,4 @@
+import json
import logging
from typing import (
Any,
@@ -168,6 +169,15 @@ def _json_wrap_input(input, value_wrapper, profile, handle_files="skip"):
def _hda_to_object(hda):
+ if hda.extension == "expression.json":
+ # We may have a null data value
+ with open(str(hda)) as inp:
+ try:
+ rval = json.loads(inp.read(5))
+ if rval is None:
+ return rval
+ except Exception:
+ pass
hda_dict = hda.to_dict()
metadata_dict = {}
diff --git a/lib/galaxy/util/sanitize_html.py b/lib/galaxy/util/sanitize_html.py
index cdfe5229168..d915a80a7a1 100644
--- a/lib/galaxy/util/sanitize_html.py
+++ b/lib/galaxy/util/sanitize_html.py
@@ -252,5 +252,5 @@ _acceptable_attributes = [
def sanitize_html(htmlSource, allow_data_urls=False):
kwd = dict(tags=_acceptable_elements, attributes=_acceptable_attributes, strip=True)
if allow_data_urls:
- kwd["protocols"] = bleach.ALLOWED_PROTOCOLS + ["data"]
+ kwd["protocols"] = list(bleach.ALLOWED_PROTOCOLS) + ["data"]
return bleach.clean(unicodify(htmlSource), **kwd)
diff --git a/lib/galaxy/version.py b/lib/galaxy/version.py
index db94f44b5b7..3ffcc1d351b 100644
--- a/lib/galaxy/version.py
+++ b/lib/galaxy/version.py
@@ -1,3 +1,3 @@
-VERSION_MAJOR = "23.0"
+VERSION_MAJOR = "23.1"
VERSION_MINOR = "dev0"
VERSION = VERSION_MAJOR + (f".{VERSION_MINOR}" if VERSION_MINOR else "")
diff --git a/lib/galaxy/webapps/galaxy/api/users.py b/lib/galaxy/webapps/galaxy/api/users.py
index b7ea6e00079..8eb658c81bf 100644
--- a/lib/galaxy/webapps/galaxy/api/users.py
+++ b/lib/galaxy/webapps/galaxy/api/users.py
@@ -7,6 +7,7 @@ import logging
import re
from fastapi import (
+ Body,
Path,
Response,
status,
@@ -35,6 +36,7 @@ from galaxy.model import (
)
from galaxy.schema import APIKeyModel
from galaxy.schema.fields import DecodedDatabaseIdField
+from galaxy.schema.schema import UserBeaconSetting
from galaxy.security.validate_user_input import (
validate_email,
validate_password,
@@ -138,6 +140,44 @@ class FastAPIHistories:
self.service.delete_api_key(trans, user_id)
return Response(status_code=status.HTTP_204_NO_CONTENT)
+ @router.get(
+ "/api/users/{user_id}/beacon",
+ summary="Returns information about beacon share settings",
+ )
+ def get_beacon(
+ self,
+ trans: ProvidesUserContext = DependsOnTrans,
+ user_id: DecodedDatabaseIdField = UserIdPathParam,
+ ) -> UserBeaconSetting:
+ """
+ **Warning**: This endpoint is experimental and might change or disappear in future versions.
+ """
+ user = self.service._get_user(trans, user_id)
+
+ enabled = user.preferences["beacon_enabled"] if "beacon_enabled" in user.preferences else False
+
+ return UserBeaconSetting(enabled=enabled)
+
+ @router.post(
+ "/api/users/{user_id}/beacon",
+ summary="Changes beacon setting",
+ )
+ def set_beacon(
+ self,
+ trans: ProvidesUserContext = DependsOnTrans,
+ user_id: DecodedDatabaseIdField = UserIdPathParam,
+ payload: UserBeaconSetting = Body(...),
+ ) -> UserBeaconSetting:
+ """
+ **Warning**: This endpoint is experimental and might change or disappear in future versions.
+ """
+ user = self.service._get_user(trans, user_id)
+
+ user.preferences["beacon_enabled"] = payload.enabled
+ trans.sa_session.flush()
+
+ return payload
+
class UserAPIController(BaseGalaxyAPIController, UsesTagsMixin, BaseUIController, UsesFormDefinitionsMixin):
user_manager: users.UserManager = depends(users.UserManager)
diff --git a/lib/galaxy/webapps/galaxy/api/workflows.py b/lib/galaxy/webapps/galaxy/api/workflows.py
index d2e4d0e891d..c4f3b0c97c0 100644
--- a/lib/galaxy/webapps/galaxy/api/workflows.py
+++ b/lib/galaxy/webapps/galaxy/api/workflows.py
@@ -48,6 +48,7 @@ from galaxy.managers.workflows import (
from galaxy.model.item_attrs import UsesAnnotations
from galaxy.model.store import BcoExportOptions
from galaxy.schema.fields import DecodedDatabaseIdField
+from galaxy.schema.invocation import InvocationMessageResponseModel
from galaxy.schema.schema import (
AsyncFile,
AsyncTaskResultSummary,
@@ -236,6 +237,7 @@ class WorkflowsAPIController(
"""
ways_to_create = {
+ "archive_file",
"archive_source",
"from_history_id",
"from_path",
@@ -254,8 +256,8 @@ class WorkflowsAPIController(
message = f"Only one parameter among - {', '.join(ways_to_create)} - must be specified"
raise exceptions.RequestParameterInvalidException(message)
- if "archive_source" in payload:
- archive_source = payload["archive_source"]
+ if "archive_source" in payload or "archive_file" in payload:
+ archive_source = payload.get("archive_source")
archive_file = payload.get("archive_file")
archive_data = None
if archive_source:
@@ -764,12 +766,19 @@ class WorkflowsAPIController(
invocations.append(workflow_invocation)
trans.sa_session.flush()
- invocations = [self.encode_all_ids(trans, invocation.to_dict(), recursive=True) for invocation in invocations]
+ encoded_invocations = []
+ for invocation in invocations:
+ as_dict = workflow_invocation.to_dict()
+ as_dict = self.encode_all_ids(trans, as_dict, recursive=True)
+ as_dict["messages"] = [
+ InvocationMessageResponseModel.parse_obj(message).__root__.dict() for message in invocation.messages
+ ]
+ encoded_invocations.append(as_dict)
if is_batch:
- return invocations
+ return encoded_invocations
else:
- return invocations[0]
+ return encoded_invocations[0]
@expose_api
def index_invocations(self, trans: GalaxyWebTransaction, **kwd):
diff --git a/lib/galaxy/webapps/galaxy/controllers/dataset.py b/lib/galaxy/webapps/galaxy/controllers/dataset.py
index 0473f7d2a1f..fe1578907b9 100644
--- a/lib/galaxy/webapps/galaxy/controllers/dataset.py
+++ b/lib/galaxy/webapps/galaxy/controllers/dataset.py
@@ -291,6 +291,10 @@ class DatasetInterface(BaseUIController, UsesAnnotations, UsesItemRatings, UsesE
if data.missing_meta():
message = 'Required metadata values are missing. Some of these values may not be editable by the user. Selecting "Auto-detect" will attempt to fix these values.'
status = "warning"
+ metadata_disable = data.state not in [
+ trans.model.Dataset.states.OK,
+ trans.model.Dataset.states.FAILED_METADATA,
+ ]
# datatype conversion
conversion_options = [
(f"{convert_id} (using '{convert_name}')", convert_id)
@@ -369,6 +373,7 @@ class DatasetInterface(BaseUIController, UsesAnnotations, UsesItemRatings, UsesE
"message": message,
"status": status,
"dataset_id": dataset_id,
+ "metadata_disable": metadata_disable,
"attribute_inputs": attribute_inputs,
"conversion_inputs": conversion_inputs,
"conversion_disable": conversion_disable,
diff --git a/lib/galaxy/webapps/galaxy/controllers/history.py b/lib/galaxy/webapps/galaxy/controllers/history.py
index 73d9717e7e0..6411a3a158b 100644
--- a/lib/galaxy/webapps/galaxy/controllers/history.py
+++ b/lib/galaxy/webapps/galaxy/controllers/history.py
@@ -208,7 +208,7 @@ class SharedHistoryListGrid(grids.Grid):
]
def build_initial_query(self, trans, **kwargs):
- return trans.sa_session.query(self.model_class).join("users_shared_with")
+ return trans.sa_session.query(self.model_class).join(self.model_class.users_shared_with)
def apply_query_filter(self, trans, query, **kwargs):
return query.filter(model.HistoryUserShareAssociation.user == trans.user)
diff --git a/lib/galaxy/webapps/galaxy/controllers/page.py b/lib/galaxy/webapps/galaxy/controllers/page.py
index 180d782dea9..b753e1a0786 100644
--- a/lib/galaxy/webapps/galaxy/controllers/page.py
+++ b/lib/galaxy/webapps/galaxy/controllers/page.py
@@ -161,7 +161,11 @@ class PageAllPublishedGrid(grids.Grid):
trans.sa_session.query(self.model_class)
.join("user")
.filter(model.User.deleted == false())
- .options(joinedload("user").load_only("username"), joinedload("annotations"), undefer("average_rating"))
+ .options(
+ joinedload(self.model_class.user).load_only("username"),
+ joinedload(self.model_class.annotations),
+ undefer("average_rating"),
+ )
)
def apply_query_filter(self, trans, query, **kwargs):
diff --git a/lib/galaxy/webapps/galaxy/controllers/visualization.py b/lib/galaxy/webapps/galaxy/controllers/visualization.py
index e56b249d8a6..248b777ef90 100644
--- a/lib/galaxy/webapps/galaxy/controllers/visualization.py
+++ b/lib/galaxy/webapps/galaxy/controllers/visualization.py
@@ -287,8 +287,12 @@ class VisualizationAllPublishedGrid(grids.Grid):
# See optimization description comments and TODO for tags in matching public histories query.
return (
trans.sa_session.query(self.model_class)
- .join("user")
- .options(joinedload("user").load_only("username"), joinedload("annotations"), undefer("average_rating"))
+ .join(self.model_class.user)
+ .options(
+ joinedload(self.model_class.user).load_only("username"),
+ joinedload(self.model_class.annotations),
+ undefer("average_rating"),
+ )
)
def apply_query_filter(self, trans, query, **kwargs):
diff --git a/lib/galaxy/webapps/galaxy/controllers/workflow.py b/lib/galaxy/webapps/galaxy/controllers/workflow.py
index 3e51a7d8308..88fae496993 100644
--- a/lib/galaxy/webapps/galaxy/controllers/workflow.py
+++ b/lib/galaxy/webapps/galaxy/controllers/workflow.py
@@ -163,12 +163,12 @@ class StoredWorkflowAllPublishedGrid(grids.Grid):
# of its steps to be eagerly loaded.
return (
trans.sa_session.query(self.model_class)
- .join("user")
+ .join(self.model_class.user)
.options(
- lazyload("latest_workflow"),
- joinedload("user").load_only("username"),
- joinedload("annotations"),
- undefer("average_rating"),
+ lazyload(self.model_class.latest_workflow),
+ joinedload(self.model_class.user).load_only(model.User.username),
+ joinedload(self.model_class.annotations),
+ undefer(self.model_class.average_rating),
)
)
@@ -487,7 +487,7 @@ class WorkflowController(BaseUIController, SharableMixin, UsesStoredWorkflowMixi
trans.sa_session.query(model.StoredWorkflow)
.filter_by(user=trans.user, deleted=False, hidden=False)
.order_by(desc(model.StoredWorkflow.table.c.update_time))
- .options(joinedload("latest_workflow").joinedload("steps"))
+ .options(joinedload(model.StoredWorkflow.latest_workflow).joinedload(model.Workflow.steps))
.all()
)
if version is None:
@@ -706,14 +706,11 @@ class WorkflowController(BaseUIController, SharableMixin, UsesStoredWorkflowMixi
)
# Index page with message
workflow_id = trans.security.encode_id(stored_workflow.id)
+ edit_url = url_for(f"/workflows/edit?id={workflow_id}")
+ run_url = url_for(f"/workflows/run?id={workflow_id}")
return trans.show_message(
- 'Workflow "%s" created from current history. '
- 'You can edit or run the workflow.'
- % (
- escape(workflow_name),
- url_for(controller="workflow", action="editor", id=workflow_id),
- url_for(controller="workflows", action="run", id=workflow_id),
- )
+ f'Workflow "{escape(workflow_name)}" created from current history. '
+ f'You can edit or run the workflow.'
)
def get_item(self, trans, id):
diff --git a/lib/galaxy/webapps/galaxy/services/datasets.py b/lib/galaxy/webapps/galaxy/services/datasets.py
index b83f9c5ace2..20182007816 100644
--- a/lib/galaxy/webapps/galaxy/services/datasets.py
+++ b/lib/galaxy/webapps/galaxy/services/datasets.py
@@ -654,7 +654,7 @@ class DatasetsService(ServiceBase, UsesVisualizationMixin):
if dataset.src == DatasetSourceType.hda:
self.hda_manager.error_if_uploading(dataset_instance)
if payload.purge:
- manager.purge(dataset_instance, flush=False)
+ manager.purge(dataset_instance, flush=True)
else:
manager.delete(dataset_instance, flush=False)
success_count += 1
diff --git a/lib/galaxy/webapps/galaxy/services/history_contents.py b/lib/galaxy/webapps/galaxy/services/history_contents.py
index 0c72c4b497c..c388bae518c 100644
--- a/lib/galaxy/webapps/galaxy/services/history_contents.py
+++ b/lib/galaxy/webapps/galaxy/services/history_contents.py
@@ -1413,7 +1413,7 @@ class HistoryItemOperator:
return
if isinstance(item, HistoryDatasetCollectionAssociation):
return self.dataset_collection_manager.delete(trans, "history", item.id, recursive=True, purge=True)
- self.hda_manager.purge(item, flush=self.flush)
+ self.hda_manager.purge(item, flush=True)
def _change_datatype(
self, item: HistoryItemModel, params: ChangeDatatypeOperationParams, trans: ProvidesHistoryContext
diff --git a/lib/galaxy/webapps/galaxy/services/invocations.py b/lib/galaxy/webapps/galaxy/services/invocations.py
index 82e1bb536d2..44d0655a3fe 100644
--- a/lib/galaxy/webapps/galaxy/services/invocations.py
+++ b/lib/galaxy/webapps/galaxy/services/invocations.py
@@ -24,11 +24,13 @@ from galaxy.exceptions import (
)
from galaxy.managers.histories import HistoryManager
from galaxy.managers.workflows import WorkflowsManager
+from galaxy.model import WorkflowInvocation
from galaxy.model.store import (
BcoExportOptions,
get_export_store_factory,
)
from galaxy.schema.fields import DecodedDatabaseIdField
+from galaxy.schema.invocation import InvocationMessageResponseModel
from galaxy.schema.schema import (
AsyncFile,
AsyncTaskResultSummary,
@@ -190,7 +192,7 @@ class InvocationsService(ServiceBase):
def serialize_workflow_invocation(
self,
- invocation,
+ invocation: WorkflowInvocation,
params: InvocationSerializationParams,
default_view: InvocationSerializationView = InvocationSerializationView.element,
):
@@ -198,7 +200,11 @@ class InvocationsService(ServiceBase):
step_details = params.step_details
legacy_job_state = params.legacy_job_state
as_dict = invocation.to_dict(view, step_details=step_details, legacy_job_state=legacy_job_state)
- return self.security.encode_all_ids(as_dict, recursive=True)
+ as_dict = self.security.encode_all_ids(as_dict, recursive=True)
+ as_dict["messages"] = [
+ InvocationMessageResponseModel.parse_obj(message).__root__.dict() for message in invocation.messages
+ ]
+ return as_dict
def serialize_workflow_invocations(
self,
diff --git a/lib/galaxy/webapps/reports/controllers/system.py b/lib/galaxy/webapps/reports/controllers/system.py
index 0c8bfbe63ee..e26446c75d2 100644
--- a/lib/galaxy/webapps/reports/controllers/system.py
+++ b/lib/galaxy/webapps/reports/controllers/system.py
@@ -115,7 +115,7 @@ class System(BaseUIController):
model.History.update_time < cutoff_time,
)
)
- .options(joinedload("datasets"))
+ .options(joinedload(model.History.datasets))
)
for history in histories:
diff --git a/lib/galaxy/workflow/modules.py b/lib/galaxy/workflow/modules.py
index 1a7cd41037b..ce0445dd17b 100644
--- a/lib/galaxy/workflow/modules.py
+++ b/lib/galaxy/workflow/modules.py
@@ -13,10 +13,12 @@ from typing import (
List,
Optional,
Type,
+ TYPE_CHECKING,
Union,
)
import packaging.version
+from cwl_utils.expression import do_eval
from typing_extensions import TypedDict
from galaxy import (
@@ -36,6 +38,15 @@ from galaxy.model import (
WorkflowStepConnection,
)
from galaxy.model.dataset_collections import matching
+from galaxy.schema.invocation import (
+ CancelReason,
+ FailureReason,
+ InvocationCancellationReviewFailed,
+ InvocationFailureDatasetFailed,
+ InvocationFailureExpressionEvaluationFailed,
+ InvocationFailureWhenNotBoolean,
+)
+from galaxy.tool_util.cwl.util import set_basename_and_derived_properties
from galaxy.tool_util.parser.output_objects import ToolExpressionOutput
from galaxy.tools import (
DatabaseOperationTool,
@@ -86,6 +97,9 @@ from galaxy.util.rules_dsl import RuleSet
from galaxy.util.template import fill_template
from galaxy.util.tool_shed.common_util import get_tool_shed_url_from_tool_shed_registry
+if TYPE_CHECKING:
+ from galaxy.schema.invocation import InvocationMessageUnion
+
log = logging.getLogger(__name__)
# Key into Tool state to describe invocation-specific runtime properties.
@@ -104,6 +118,126 @@ class NoReplacement:
NO_REPLACEMENT = NoReplacement()
+class ConditionalStepWhen(BooleanToolParameter):
+ pass
+
+
+def evaluate_value_from_expressions(progress, step, execution_state, extra_step_state):
+ when_expression = step.when_expression
+ value_from_expressions = {}
+
+ if execution_state:
+ for key in execution_state.inputs.keys():
+ step_input = step.inputs_by_name.get(key)
+ if step_input and step_input.value_from is not None:
+ value_from_expressions[key] = step_input.value_from
+
+ if not value_from_expressions and when_expression is None:
+ return {}
+
+ hda_references = []
+
+ def to_cwl(value):
+ element_identifier = None
+ if isinstance(value, model.DatasetCollectionElement) and value.hda:
+ element_identifier = value.element_identifier
+ value = value.hda
+ if isinstance(value, model.HistoryDatasetAssociation):
+ # I think the following two checks are needed but they may
+ # not be needed.
+ if not value.dataset.in_ready_state():
+ why = "dataset [%s] is needed for valueFrom expression and is non-ready" % value.id
+ raise DelayedWorkflowEvaluation(why=why)
+ if not value.is_ok:
+ raise FailWorkflowEvaluation(
+ why=InvocationFailureDatasetFailed(
+ reason=FailureReason.dataset_failed, hda_id=value.id, workflow_step_id=step.id
+ )
+ )
+ if value.ext == "expression.json":
+ with open(value.file_name) as f:
+ # OUR safe_loads won't work, will not load numbers, etc...
+ return json.load(f)
+ else:
+ hda_references.append(value)
+ properties = {
+ "class": "File",
+ "location": "step_input://%d" % len(hda_references),
+ }
+ set_basename_and_derived_properties(
+ properties, value.dataset.created_from_basename or element_identifier or value.name
+ )
+ return properties
+ elif hasattr(value, "collection"):
+ collection = value.collection
+ if collection.collection_type == "list":
+ return [to_cwl(dce) for dce in collection.dataset_elements]
+ else:
+ # Could be record or nested lists
+ rval = {}
+ for element in collection.elements:
+ rval[element.element_identifier] = to_cwl(element.element_object)
+ return rval
+ else:
+ return value
+
+ def from_cwl(value):
+ # TODO: turn actual files into HDAs here ... somehow I suppose. Things with
+ # file:// locations for instance.
+ if isinstance(value, dict) and "class" in value and "location" in value:
+ if value["class"] == "File":
+ # This is going to re-file -> HDA this each iteration I think, not a good
+ # implementation.
+ return progress.raw_to_galaxy(value)
+ assert value["location"].startswith("step_input://"), "Invalid location %s" % value
+ return hda_references[int(value["location"][len("step_input://") :]) - 1]
+ elif isinstance(value, dict):
+ raise NotImplementedError()
+ else:
+ return value
+
+ step_state = {}
+ for key, value in extra_step_state.items():
+ step_state[key] = to_cwl(value)
+ if execution_state:
+ for key, value in execution_state.inputs.items():
+ step_state[key] = to_cwl(value)
+
+ if when_expression is not None:
+ try:
+ as_cwl_value = do_eval(
+ when_expression,
+ step_state,
+ [{"class": "InlineJavascriptRequirement"}],
+ None,
+ None,
+ {},
+ )
+ except Exception:
+ # Exception contains script and traceback, which could be helpful for debugging workflows,
+ # but both could conceivably contain secrets.
+ # CWL has a secret hint that should cause values to be sanitized,
+ # but Galaxy does not, so we can't really display anything here at this point.
+ # In any case I believe the CWL secret hint can be bypassed if the value is passed on
+ # to another step input that doesn't have the secret set.
+ # Complicated stuff, ignore for now.
+ raise FailWorkflowEvaluation(
+ InvocationFailureExpressionEvaluationFailed(
+ reason=FailureReason.expression_evaluation_failed, workflow_step_id=step.id
+ )
+ )
+ when_value = from_cwl(as_cwl_value)
+ if not isinstance(when_value, bool):
+ raise FailWorkflowEvaluation(
+ InvocationFailureWhenNotBoolean(
+ reason=FailureReason.when_not_boolean,
+ workflow_step_id=step.id,
+ details=f"Type is: {when_value.__class__.__name__}",
+ )
+ )
+ return when_value
+
+
class WorkflowModule:
label: str
@@ -361,19 +495,15 @@ class WorkflowModule:
return []
def compute_collection_info(self, progress, step, all_inputs):
- """Use get_all_inputs (if implemented) to determine collection mapping for execution.
-
- Hopefully this can be reused for Tool and Subworkflow modules.
"""
-
+ Use get_all_inputs (if implemented) to determine collection mapping for execution.
+ """
collections_to_match = self._find_collections_to_match(progress, step, all_inputs)
# Have implicit collections...
- if collections_to_match.has_collections():
- collection_info = self.trans.app.dataset_collection_manager.match_collections(collections_to_match)
- else:
- collection_info = None
-
- return collection_info
+ collection_info = self.trans.app.dataset_collection_manager.match_collections(collections_to_match)
+ if collection_info and progress.subworkflow_collection_info:
+ collection_info.when_values = progress.subworkflow_collection_info.when_values
+ return collection_info or progress.subworkflow_collection_info
def _find_collections_to_match(self, progress, step, all_inputs):
collections_to_match = matching.CollectionsToMatch()
@@ -440,6 +570,20 @@ class WorkflowModule:
collections_to_match.add(name, data)
continue
+ known_input_names = {input_dict["name"] for input_dict in all_inputs}
+
+ if step.when_expression:
+ for step_input in step.inputs:
+ step_input_name = step_input.name
+ input_in_execution_state = step_input_name not in known_input_names
+ if input_in_execution_state:
+ maybe_collection = progress.replacement_for_connection(
+ step.input_connections_by_name[step_input_name][0]
+ )
+ if hasattr(maybe_collection, "collection"):
+ # Is that always right ?
+ collections_to_match.add(step_input_name, maybe_collection)
+
return collections_to_match
@@ -596,10 +740,47 @@ class SubWorkflowModule(WorkflowModule):
inputs, etc...
"""
step = invocation_step.workflow_step
- collection_info = self.compute_collection_info(progress, step, self.get_all_inputs())
- structure = collection_info.structure if collection_info else None
+ all_inputs = self.get_all_inputs()
+ collection_info = self.compute_collection_info(progress, step, all_inputs)
+
+ if collection_info:
+ iteration_elements_iter = collection_info.slice_collections()
+ else:
+ if progress.when_values:
+ # If we have more than one item in when_values it must have come from an expression.json
+ # collection, so we'd have a collection_info instance ... I think.
+ assert len(progress.when_values) == 1, "Got more than 1 when value, this shouldn't be possible"
+ iteration_elements_iter = [(None, progress.when_values[0] if progress.when_values else None)]
+
+ when_values = []
+ if step.when_expression:
+ for (iteration_elements, when_value) in iteration_elements_iter:
+ if when_value is False:
+ when_values.append(when_value)
+ continue
+ extra_step_state = {}
+ for step_input in step.inputs:
+ step_input_name = step_input.name
+ if iteration_elements and step_input_name in iteration_elements: # noqa: B023
+ value = iteration_elements[step_input_name] # noqa: B023
+ else:
+ value = progress.replacement_for_connection(step_input.connections[0], is_data=True)
+ extra_step_state[step_input_name] = value
+
+ when_values.append(
+ evaluate_value_from_expressions(
+ progress, step, execution_state={}, extra_step_state=extra_step_state
+ )
+ )
+ if collection_info:
+ collection_info.when_values = when_values
+
subworkflow_invoker = progress.subworkflow_invoker(
- trans, step, use_cached_job=use_cached_job, subworkflow_structure=structure
+ trans,
+ step,
+ use_cached_job=use_cached_job,
+ subworkflow_collection_info=collection_info,
+ when_values=when_values,
)
subworkflow_invoker.invoke()
subworkflow = subworkflow_invoker.workflow
@@ -1457,7 +1638,11 @@ class PauseModule(WorkflowModule):
progress.set_step_outputs(invocation_step, {"output": replacement})
return
elif action is False:
- raise CancelWorkflowEvaluation()
+ raise CancelWorkflowEvaluation(
+ why=InvocationCancellationReviewFailed(
+ reason=CancelReason.cancelled_on_review, workflow_step_id=step.id
+ )
+ )
delayed_why = "workflow paused at this step waiting for review"
raise DelayedWorkflowEvaluation(why=delayed_why)
@@ -1919,15 +2104,18 @@ class ToolModule(WorkflowModule):
step = invocation_step.workflow_step
tool = trans.app.toolbox.get_tool(step.tool_id, tool_version=step.tool_version, tool_uuid=step.tool_uuid)
if not tool.is_workflow_compatible:
- message = f"Specified tool [{tool.id}] in workflow is not workflow-compatible."
- raise Exception(message)
+ # TODO: why do we even create an invocation, seems like something we could check on submit?
+ message = f"Specified tool [{tool.id}] in step {step.order_index + 1} is not workflow-compatible."
+ raise exceptions.MessageException(message)
tool_state = step.state
+ tool_inputs = tool.inputs.copy()
# Not strictly needed - but keep Tool state clean by stripping runtime
# metadata parameters from it.
if RUNTIME_STEP_META_STATE_KEY in tool_state.inputs:
del tool_state.inputs[RUNTIME_STEP_META_STATE_KEY]
all_inputs = self.get_all_inputs()
+
all_inputs_by_name = {}
for input_dict in all_inputs:
all_inputs_by_name[input_dict["name"]] = input_dict
@@ -1937,10 +2125,12 @@ class ToolModule(WorkflowModule):
if collection_info:
iteration_elements_iter = collection_info.slice_collections()
else:
- iteration_elements_iter = [None]
+ if progress.when_values:
+ assert len(progress.when_values) == 1, "Got more than 1 when value, this shouldn't be possible"
+ iteration_elements_iter = [(None, progress.when_values[0] if progress.when_values else None)]
resource_parameters = invocation.resource_parameters
- for iteration_elements in iteration_elements_iter:
+ for (iteration_elements, when_value) in iteration_elements_iter:
execution_state = tool_state.copy()
# TODO: Move next step into copy()
execution_state.inputs = make_dict_copy(execution_state.inputs)
@@ -1971,22 +2161,58 @@ class ToolModule(WorkflowModule):
replacement = json.load(f)
found_replacement_keys.add(prefixed_name) # noqa: B023
+ # bool cast should be fine, can only have true/false on ConditionalStepWhen
+ # also terrible of course and it's not needed for API requests
+ if isinstance(input, ConditionalStepWhen) and bool(replacement) is False:
+ raise SkipWorkflowStepEvaluation
+
return replacement
try:
# Replace DummyDatasets with historydatasetassociations
visit_input_values(
- tool.inputs,
+ tool_inputs,
execution_state.inputs,
callback,
no_replacement_value=NO_REPLACEMENT,
replace_optional_connections=True,
)
except KeyError as k:
- message_template = "Error due to input mapping of '%s' in '%s'. A common cause of this is conditional outputs that cannot be determined until runtime, please review your workflow."
- message = message_template % (tool.name, unicodify(k))
+ message = f"Error due to input mapping of '{unicodify(k)}' in tool '{tool.id}'. A common cause of this is conditional outputs that cannot be determined until runtime, please review workflow step {step.order_index + 1}."
raise exceptions.MessageException(message)
+ if step.when_expression and when_value is not False:
+ extra_step_state = {}
+ for step_input in step.inputs:
+ step_input_name = step_input.name
+ input_in_execution_state = step_input_name not in execution_state.inputs
+ if input_in_execution_state:
+ if step_input_name in all_inputs_by_name:
+ if iteration_elements and step_input_name in iteration_elements: # noqa: B023
+ value = iteration_elements[step_input_name] # noqa: B023
+ else:
+ value = progress.replacement_for_input(step, all_inputs_by_name[step_input_name])
+ # TODO: only do this for values... is everything with a default
+ # this way a field parameter? I guess not?
+ extra_step_state[step_input_name] = value
+ # Might be needed someday...
+ # elif step_input.default_value_set:
+ # extra_step_state[step_input_name] = step_input.default_value
+ else:
+ if iteration_elements and step_input_name in iteration_elements: # noqa: B023
+ value = iteration_elements[step_input_name] # noqa: B023
+ else:
+ value = progress.replacement_for_connection(step_input.connections[0], is_data=True)
+ extra_step_state[step_input_name] = value
+
+ if when_value is not False:
+ when_value = evaluate_value_from_expressions(
+ progress, step, execution_state=execution_state, extra_step_state=extra_step_state
+ )
+ if when_value is not None:
+ # Track this more formally ?
+ execution_state.inputs["__when_value__"] = when_value
+
unmatched_input_connections = expected_replacement_keys - found_replacement_keys
if unmatched_input_connections:
log.warning(f"Failed to use input connections for inputs [{unmatched_input_connections}]")
@@ -2052,8 +2278,9 @@ class ToolModule(WorkflowModule):
self._handle_mapped_over_post_job_actions(step, step_inputs, step_outputs, progress.replacement_dict)
if execution_tracker.execution_errors:
- message = "Failed to create one or more job(s) for workflow step."
- raise Exception(message)
+ # TODO: formalize into InvocationFailure ?
+ message = f"Failed to create {len(execution_tracker.execution_errors)} job(s) for workflow step {step.order_index + 1}: {str(execution_tracker.execution_errors[0])}"
+ raise exceptions.MessageException(message)
return complete
@@ -2196,6 +2423,16 @@ class DelayedWorkflowEvaluation(Exception):
class CancelWorkflowEvaluation(Exception):
+ def __init__(self, why: "InvocationMessageUnion"):
+ self.why = why
+
+
+class FailWorkflowEvaluation(Exception):
+ def __init__(self, why: "InvocationMessageUnion"):
+ self.why = why
+
+
+class SkipWorkflowStepEvaluation(Exception):
pass
diff --git a/lib/galaxy/workflow/run.py b/lib/galaxy/workflow/run.py
index 002b93ee211..ef447aaabb0 100644
--- a/lib/galaxy/workflow/run.py
+++ b/lib/galaxy/workflow/run.py
@@ -13,10 +13,23 @@ from typing import (
from typing_extensions import Protocol
from galaxy import model
+from galaxy.exceptions import MessageException
from galaxy.model import (
WorkflowInvocation,
WorkflowInvocationStep,
)
+from galaxy.schema.invocation import (
+ CancelReason,
+ FailureReason,
+ InvocationCancellationHistoryDeleted,
+ InvocationFailureCollectionFailed,
+ InvocationFailureDatasetFailed,
+ InvocationFailureJobFailed,
+ InvocationFailureOutputNotFound,
+ InvocationUnexpectedFailure,
+ InvocationWarningWorkflowOutputNotFound,
+ WarningReason,
+)
from galaxy.util import ExecutionTimer
from galaxy.workflow import modules
from galaxy.workflow.run_request import (
@@ -73,21 +86,31 @@ def __invoke(
workflow_invocation=workflow_invocation,
)
workflow_invocation = invoker.workflow_invocation
+ outputs = {}
try:
outputs = invoker.invoke()
- except modules.CancelWorkflowEvaluation:
+ except modules.CancelWorkflowEvaluation as e:
if workflow_invocation.cancel():
- trans.sa_session.add(workflow_invocation)
- outputs = {}
+ workflow_invocation.add_message(e.why)
+ except modules.FailWorkflowEvaluation as e:
+ workflow_invocation.fail()
+ workflow_invocation.add_message(e.why)
+ except MessageException as e:
+ # Convention for safe message we can show to users
+ workflow_invocation.fail()
+ failure = InvocationUnexpectedFailure(reason=FailureReason.unexpected_failure, details=str(e))
+ workflow_invocation.add_message(failure)
except Exception:
+ # Could potentially be large and/or contain raw ids or other secrets, don't add details
log.exception("Failed to execute scheduled workflow.")
# Running workflow invocation in background, just mark
# persistent workflow invocation as failed.
+ failure = InvocationUnexpectedFailure(reason=FailureReason.unexpected_failure)
workflow_invocation.fail()
- trans.sa_session.add(workflow_invocation)
- outputs = {}
+ workflow_invocation.add_message(failure)
# Be sure to update state of workflow_invocation.
+ trans.sa_session.add(workflow_invocation)
trans.sa_session.flush()
return outputs, workflow_invocation
@@ -176,8 +199,11 @@ class WorkflowInvoker:
return self.progress.outputs
if workflow_invocation.history.deleted:
- log.info("Cancelled workflow evaluation due to deleted history")
- raise modules.CancelWorkflowEvaluation()
+ raise modules.CancelWorkflowEvaluation(
+ why=InvocationCancellationHistoryDeleted(
+ reason=CancelReason.history_deleted, history_id=workflow_invocation.history_id
+ )
+ )
remaining_steps = self.progress.remaining_steps()
delayed_steps = False
@@ -244,9 +270,9 @@ class WorkflowInvoker:
for input_connection in step.input_connections:
if input_connection.non_data_connection:
output_id = input_connection.output_step.id
- self.__check_implicitly_dependent_step(output_id)
+ self.__check_implicitly_dependent_step(output_id, step.id)
- def __check_implicitly_dependent_step(self, output_id):
+ def __check_implicitly_dependent_step(self, output_id: int, step_id: int):
step_invocation = self.workflow_invocation.step_invocation_for_step_id(output_id)
# No steps created yet - have to delay evaluation.
@@ -268,7 +294,14 @@ class WorkflowInvoker:
raise modules.DelayedWorkflowEvaluation(why=delayed_why)
if job.state != job.states.OK:
- raise modules.CancelWorkflowEvaluation()
+ raise modules.FailWorkflowEvaluation(
+ why=InvocationFailureJobFailed(
+ reason=FailureReason.job_failed,
+ job_id=job.id,
+ workflow_step_id=step_id,
+ dependent_workflow_step_id=output_id,
+ )
+ )
def _invoke_step(self, invocation_step: WorkflowInvocationStep) -> Optional[bool]:
incomplete_or_none = invocation_step.workflow_step.module.execute(
@@ -305,7 +338,8 @@ class WorkflowProgress:
copy_inputs_to_history: bool = False,
use_cached_job: bool = False,
replacement_dict: Optional[Dict[str, str]] = None,
- subworkflow_structure=None,
+ subworkflow_collection_info=None,
+ when_values=None,
) -> None:
self.outputs: Dict[int, Any] = {}
self.module_injector = module_injector
@@ -317,7 +351,9 @@ class WorkflowProgress:
self.copy_inputs_to_history = copy_inputs_to_history
self.use_cached_job = use_cached_job
self.replacement_dict = replacement_dict or {}
- self.subworkflow_structure = subworkflow_structure
+ self.subworkflow_collection_info = subworkflow_collection_info
+ self.subworkflow_structure = subworkflow_collection_info.structure if subworkflow_collection_info else None
+ self.when_values = when_values
@property
def maximum_jobs_to_schedule_or_none(self) -> Optional[int]:
@@ -346,9 +382,10 @@ class WorkflowProgress:
step_args = self.param_map.get(step_id, {})
self.module_injector.compute_runtime_state(step, step_args=step_args)
if step_id not in step_states:
- raise Exception(
- f"Workflow invocation [{self.workflow_invocation.id}] has no step state for step {step.log_str()}. States ids are {list(step_states.keys())}."
- )
+ # Can this ever happen?
+ public_message = f"Workflow invocation has no step state for step {step.order_index + 1}"
+ log.error(f"{public_message}. State is known for these step ids: {list(step_states.keys())}.")
+ raise MessageException(public_message)
runtime_state = step_states[step_id].value
assert step.module
step.state = step.module.decode_runtime_state(runtime_state)
@@ -390,29 +427,45 @@ class WorkflowProgress:
def replacement_for_connection(self, connection: "WorkflowStepConnection", is_data: bool = True) -> Any:
output_step_id = connection.output_step.id
+ output_name = connection.output_name
if output_step_id not in self.outputs:
- message = f"No outputs found for step id {output_step_id}, outputs are {self.outputs}"
- raise Exception(message)
+ raise modules.FailWorkflowEvaluation(
+ why=InvocationFailureOutputNotFound(
+ reason=FailureReason.output_not_found,
+ workflow_step_id=connection.input_step_id,
+ output_name=output_name,
+ dependent_workflow_step_id=output_step_id,
+ )
+ )
step_outputs = self.outputs[output_step_id]
if step_outputs is STEP_OUTPUT_DELAYED:
delayed_why = f"dependent step [{output_step_id}] delayed, so this step must be delayed"
raise modules.DelayedWorkflowEvaluation(why=delayed_why)
- output_name = connection.output_name
try:
replacement = step_outputs[output_name]
except KeyError:
- # Must resolve.
- template = "Workflow evaluation problem - failed to find output_name %s in step_outputs %s"
- message = template % (output_name, step_outputs)
- raise Exception(message)
+ raise modules.FailWorkflowEvaluation(
+ why=InvocationFailureOutputNotFound(
+ reason=FailureReason.output_not_found,
+ workflow_step_id=connection.input_step_id,
+ output_name=output_name,
+ dependent_workflow_step_id=output_step_id,
+ )
+ )
if isinstance(replacement, model.HistoryDatasetCollectionAssociation):
if not replacement.collection.populated:
if not replacement.waiting_for_elements:
# If we are not waiting for elements, there was some
# problem creating the collection. Collection will never
# be populated.
- # TODO: consider distinguish between cancelled and failed?
- raise modules.CancelWorkflowEvaluation()
+ raise modules.FailWorkflowEvaluation(
+ why=InvocationFailureCollectionFailed(
+ reason=FailureReason.collection_failed,
+ hdca_id=replacement.id,
+ workflow_step_id=connection.input_step_id,
+ dependent_workflow_step_id=output_step_id,
+ )
+ )
delayed_why = f"dependent collection [{replacement.id}] not yet populated with datasets"
raise modules.DelayedWorkflowEvaluation(why=delayed_why)
@@ -426,7 +479,14 @@ class WorkflowProgress:
if replacement.is_pending:
raise modules.DelayedWorkflowEvaluation()
if not replacement.is_ok:
- raise modules.CancelWorkflowEvaluation()
+ raise modules.FailWorkflowEvaluation(
+ why=InvocationFailureDatasetFailed(
+ reason=FailureReason.dataset_failed,
+ hda_id=replacement.id,
+ workflow_step_id=connection.input_step_id,
+ dependent_workflow_step_id=output_step_id,
+ )
+ )
else:
if not replacement.collection.populated:
raise modules.DelayedWorkflowEvaluation()
@@ -435,7 +495,14 @@ class WorkflowProgress:
if dataset_instance.is_pending:
pending = True
elif not dataset_instance.is_ok:
- raise modules.CancelWorkflowEvaluation()
+ raise modules.FailWorkflowEvaluation(
+ why=InvocationFailureDatasetFailed(
+ reason=FailureReason.dataset_failed,
+ hda_id=replacement.id,
+ workflow_step_id=connection.input_step_id,
+ dependent_workflow_step_id=output_step_id,
+ )
+ )
if pending:
raise modules.DelayedWorkflowEvaluation()
@@ -466,7 +533,15 @@ class WorkflowProgress:
if default_value or step.input_optional:
outputs["output"] = default_value
else:
- raise ValueError(f"{step.log_str()} not found in inputs_step_id {self.inputs_by_step_id}")
+ log.error(f"{step.log_str()} not found in inputs_step_id {self.inputs_by_step_id}")
+ raise modules.FailWorkflowEvaluation(
+ why=InvocationFailureOutputNotFound(
+ reason=FailureReason.output_not_found,
+ workflow_step_id=invocation_step.workflow_step_id,
+ output_name="output",
+ dependent_workflow_step_id=invocation_step.workflow_step_id,
+ )
+ )
elif step_id in self.inputs_by_step_id:
outputs["output"] = self.inputs_by_step_id[step_id]
@@ -494,13 +569,14 @@ class WorkflowProgress:
for workflow_output in step.workflow_outputs:
output_name = workflow_output.output_name
if output_name not in outputs:
+ invocation_step.workflow_invocation.add_message(
+ InvocationWarningWorkflowOutputNotFound(
+ reason=WarningReason.workflow_output_not_found,
+ workflow_step_id=step.id,
+ output_name=output_name,
+ )
+ )
message = f"Failed to find expected workflow output [{output_name}] in step outputs [{outputs}]"
- # raise KeyError(message)
- # Pre-18.01 we would have never even detected this output wasn't configured
- # and even in 18.01 we don't have a way to tell the user something bad is
- # happening so I guess we just log a debug message and continue sadly for now.
- # Once https://github.com/galaxyproject/galaxy/issues/5142 is complete we could
- # at least tell the user what happened, give them a warning.
log.debug(message)
continue
output = outputs[output_name]
@@ -523,7 +599,7 @@ class WorkflowProgress:
workflow_invocation = self.workflow_invocation
subworkflow_invocation = workflow_invocation.get_subworkflow_invocation_for_step(step)
if subworkflow_invocation is None:
- raise Exception(f"Failed to find persisted workflow invocation for step [{step.id}]")
+ raise MessageException(f"Failed to find persisted subworkflow invocation for step [{step.order_index + 1}]")
return subworkflow_invocation
def subworkflow_invoker(
@@ -531,12 +607,17 @@ class WorkflowProgress:
trans: "WorkRequestContext",
step: "WorkflowStep",
use_cached_job: bool = False,
- subworkflow_structure=None,
+ subworkflow_collection_info=None,
+ when_values=None,
) -> WorkflowInvoker:
subworkflow_invocation = self._subworkflow_invocation(step)
workflow_run_config = workflow_request_to_run_config(subworkflow_invocation, use_cached_job)
subworkflow_progress = self.subworkflow_progress(
- subworkflow_invocation, step, workflow_run_config.param_map, subworkflow_structure=subworkflow_structure
+ subworkflow_invocation,
+ step,
+ workflow_run_config.param_map,
+ subworkflow_collection_info=subworkflow_collection_info,
+ when_values=when_values,
)
subworkflow_invocation = subworkflow_progress.workflow_invocation
return WorkflowInvoker(
@@ -551,7 +632,8 @@ class WorkflowProgress:
subworkflow_invocation: WorkflowInvocation,
step: "WorkflowStep",
param_map: Dict,
- subworkflow_structure=None,
+ subworkflow_collection_info=None,
+ when_values=None,
) -> "WorkflowProgress":
subworkflow = subworkflow_invocation.workflow
subworkflow_inputs = {}
@@ -570,7 +652,14 @@ class WorkflowProgress:
break
if not connection_found and not input_subworkflow_step.input_optional:
- raise Exception("Could not find connections for all subworkflow inputs.")
+ raise modules.FailWorkflowEvaluation(
+ InvocationFailureOutputNotFound(
+ reason=FailureReason.output_not_found,
+ workflow_step_id=step.id,
+ output_name=input_connection.output_name,
+ dependent_workflow_step_id=input_connection.output_step.id,
+ )
+ )
return WorkflowProgress(
subworkflow_invocation,
@@ -579,7 +668,8 @@ class WorkflowProgress:
param_map=param_map,
use_cached_job=self.use_cached_job,
replacement_dict=self.replacement_dict,
- subworkflow_structure=subworkflow_structure,
+ subworkflow_collection_info=subworkflow_collection_info,
+ when_values=when_values,
)
def _recover_mapping(self, step_invocation: WorkflowInvocationStep) -> None:
diff --git a/lib/galaxy_test/api/test_authenticate.py b/lib/galaxy_test/api/test_authenticate.py
index a63ae40bed6..41f437d4913 100644
--- a/lib/galaxy_test/api/test_authenticate.py
+++ b/lib/galaxy_test/api/test_authenticate.py
@@ -4,6 +4,7 @@ from requests import get
from galaxy_test.base.api_util import baseauth_headers
from galaxy_test.base.decorators import requires_new_user
+from galaxy_test.base.populators import skip_without_tool
from ._framework import ApiTestCase
TEST_USER_EMAIL = "auth_user_test@bx.psu.edu"
@@ -26,6 +27,7 @@ class TestAuthenticateApi(ApiTestCase):
random_api_response = get(random_api_url, params=dict(key=auth_dict["api_key"]))
self._assert_status_code_is(random_api_response, 200)
+ @skip_without_tool("test_data_source")
def test_tool_runner_session_cookie_handling(self):
response = get(self.url)
tool_runner_session_cookie = response.cookies["galaxytoolrunnersession"]
diff --git a/lib/galaxy_test/api/test_dataset_collections.py b/lib/galaxy_test/api/test_dataset_collections.py
index 0798c809da8..af218d8a096 100644
--- a/lib/galaxy_test/api/test_dataset_collections.py
+++ b/lib/galaxy_test/api/test_dataset_collections.py
@@ -4,6 +4,7 @@ from typing import List
from galaxy.util.unittest_utils import skip_if_github_down
from galaxy_test.base.api_asserts import assert_object_id_error
+from galaxy_test.base.decorators import requires_new_user
from galaxy_test.base.populators import (
DatasetCollectionPopulator,
DatasetPopulator,
@@ -181,6 +182,7 @@ class TestDatasetCollectionsApi(ApiTestCase):
namelist = archive.namelist()
assert len(namelist) == 3, f"Expected 3 elements in [{namelist}]"
+ @requires_new_user
def test_hda_security(self):
with self.dataset_populator.test_history(require_new=False) as history_id:
element_identifiers = self.dataset_collection_populator.pair_identifiers(history_id)
@@ -374,6 +376,7 @@ class TestDatasetCollectionsApi(ApiTestCase):
def _download_dataset_collection(self, history_id: str, hdca_id: str):
return self._get(f"histories/{history_id}/contents/dataset_collections/{hdca_id}/download")
+ @requires_new_user
def test_collection_contents_security(self, history_id):
# request contents on an hdca that doesn't belong to user
hdca, contents_url = self._create_collection_contents_pair(history_id)
diff --git a/lib/galaxy_test/api/test_datasets.py b/lib/galaxy_test/api/test_datasets.py
index 76d2a41e32f..003ed7e767a 100644
--- a/lib/galaxy_test/api/test_datasets.py
+++ b/lib/galaxy_test/api/test_datasets.py
@@ -470,6 +470,8 @@ class TestDatasetsApi(ApiTestCase):
hda = self.dataset_populator.new_dataset(history_id)
dataset_map[index] = hda["id"]
+ self.dataset_populator.wait_for_history(history_id)
+
expected_deleted_source_ids = [
{"id": dataset_map[1], "src": "hda"},
{"id": dataset_map[2], "src": "hda"},
diff --git a/lib/galaxy_test/api/test_drs.py b/lib/galaxy_test/api/test_drs.py
index 81d77dee25b..f942c7761e2 100644
--- a/lib/galaxy_test/api/test_drs.py
+++ b/lib/galaxy_test/api/test_drs.py
@@ -4,7 +4,10 @@ from typing import (
List,
)
from unittest import SkipTest
-from urllib.parse import urlparse
+from urllib.parse import (
+ urljoin,
+ urlparse,
+)
import requests
@@ -31,18 +34,18 @@ class TestDrsApi(ApiTestCase):
self.dataset_populator = DatasetPopulator(self.galaxy_interactor)
def test_service_info(self):
- api_url = f"{self.url}ga4gh/drs/v1/service-info"
+ api_url = self._url_join("ga4gh/drs/v1/service-info")
info_response = requests.get(api_url)
info_response.raise_for_status()
- def test_404_on_private_datsets(self):
+ def test_404_on_private_datasets(self):
history_id = self.dataset_populator.new_history()
hda = self.dataset_populator.new_dataset(history_id, content=CONTENT, wait=True)
dataset_id = hda["id"]
drs_id = hda["drs_id"]
self.dataset_populator.make_private(history_id=history_id, dataset_id=dataset_id)
for method in HTTP_METHODS:
- api_url = f"{self.url}ga4gh/drs/v1/objects/{drs_id}"
+ api_url = self._url_join(f"ga4gh/drs/v1/objects/{drs_id}")
show_response = method(api_url)
assert show_response.status_code == 403
@@ -52,7 +55,7 @@ class TestDrsApi(ApiTestCase):
drs_id = hda["drs_id"]
for method in HTTP_METHODS:
for _ in range(10):
- api_url = f"{self.url}ga4gh/drs/v1/objects/{drs_id}"
+ api_url = self._url_join(f"ga4gh/drs/v1/objects/{drs_id}")
show_response = method(api_url)
show_response.raise_for_status()
if show_response.status_code != 202:
@@ -102,10 +105,13 @@ class TestDrsApi(ApiTestCase):
drs_id = hda["drs_id"]
for method in HTTP_METHODS:
- api_url = f"{self.url}ga4gh/drs/v1/objects/{drs_id}/access/fakeid"
+ api_url = self._url_join(f"ga4gh/drs/v1/objects/{drs_id}/access/fakeid")
error_response = method(api_url)
assert type(error_response.status_code) == int
assert error_response.status_code == 404
error_as_dict = error_response.json()
assert "status_code" in error_as_dict
assert "msg" in error_as_dict
+
+ def _url_join(self, suffix):
+ return urljoin(self.url, suffix)
diff --git a/lib/galaxy_test/api/test_histories.py b/lib/galaxy_test/api/test_histories.py
index b4ab38be3ff..3e0ffae41dc 100644
--- a/lib/galaxy_test/api/test_histories.py
+++ b/lib/galaxy_test/api/test_histories.py
@@ -11,6 +11,10 @@ from galaxy.model.unittest_utils.store_fixtures import (
)
from galaxy_test.api.sharable import SharingApiTests
from galaxy_test.base.api_asserts import assert_has_keys
+from galaxy_test.base.decorators import (
+ requires_admin,
+ requires_new_user,
+)
from galaxy_test.base.populators import (
DatasetCollectionPopulator,
DatasetPopulator,
@@ -249,6 +253,7 @@ class TestHistoriesApi(ApiTestCase, BaseHistories):
create_response = self._post("histories", data=post_data, anon=True)
self._assert_status_code_is(create_response, 403)
+ @requires_admin
def test_create_without_session_fails(self):
post_data = dict(name="SessionNeeded")
# Using admin=True will boostrap an Admin user without session
@@ -581,6 +586,7 @@ class TestSharingHistory(ApiTestCase, BaseHistories, SharingApiTests):
super().setUp()
self.dataset_populator = DatasetPopulator(self.galaxy_interactor)
+ @requires_new_user
def test_sharing_with_private_datasets(self):
history_id = self.dataset_populator.new_history()
hda = self.dataset_populator.new_dataset(history_id)
@@ -607,6 +613,8 @@ class TestSharingHistory(ApiTestCase, BaseHistories, SharingApiTests):
assert sharing_response["users_shared_with"]
assert sharing_response["users_shared_with"][0]["id"] == target_user_id
+ @requires_admin
+ @requires_new_user
def test_sharing_without_manage_permissions(self):
history_id = self.dataset_populator.new_history()
hda = self.dataset_populator.new_dataset(history_id)
@@ -649,6 +657,7 @@ class TestSharingHistory(ApiTestCase, BaseHistories, SharingApiTests):
assert sharing_response["users_shared_with"]
assert sharing_response["users_shared_with"][0]["id"] == target_user_id
+ @requires_new_user
def test_sharing_empty_not_allowed(self):
history_id = self.dataset_populator.new_history()
@@ -661,6 +670,7 @@ class TestSharingHistory(ApiTestCase, BaseHistories, SharingApiTests):
assert sharing_response["errors"]
assert "empty" in sharing_response["errors"][0]
+ @requires_new_user
def test_sharing_with_duplicated_users(self):
history_id = self.create("HistoryToShareWithDuplicatedUser")
@@ -674,6 +684,7 @@ class TestSharingHistory(ApiTestCase, BaseHistories, SharingApiTests):
assert len(sharing_response["users_shared_with"]) == 1
assert sharing_response["users_shared_with"][0]["id"] == target_user_id
+ @requires_new_user
def test_sharing_private_history_makes_datasets_public(self):
history_id = self.dataset_populator.new_history()
hda = self.dataset_populator.new_dataset(history_id)
diff --git a/lib/galaxy_test/api/test_history_contents.py b/lib/galaxy_test/api/test_history_contents.py
index a51528d96fe..de9267a9a10 100644
--- a/lib/galaxy_test/api/test_history_contents.py
+++ b/lib/galaxy_test/api/test_history_contents.py
@@ -7,6 +7,12 @@ from typing import (
)
from galaxy_test.api._framework import ApiTestCase
+from galaxy_test.base.decorators import (
+ requires_admin,
+ requires_celery,
+ requires_new_library,
+ requires_new_user,
+)
from galaxy_test.base.populators import (
DatasetCollectionPopulator,
DatasetPopulator,
@@ -35,6 +41,7 @@ class TestHistoryContentsApi(ApiTestCase):
hda_summary = self.__check_for_hda(contents_response, hda1)
assert "display_types" not in hda_summary # Quick summary, not full details
+ @requires_admin
def test_make_private_and_public(self, history_id):
hda1 = self._wait_for_new_hda(history_id)
update_url = f"histories/{history_id}/contents/{hda1['id']}/permissions"
@@ -64,9 +71,11 @@ class TestHistoryContentsApi(ApiTestCase):
self._assert_status_code_is(update_response, 200)
self._assert_other_user_can_access(history_id, hda1["id"])
+ @requires_new_user
def test_set_permissions_add_admin_history_contents(self, history_id):
self._verify_dataset_permissions(history_id, "history_contents")
+ @requires_new_user
def test_set_permissions_add_admin_datasets(self, history_id):
self._verify_dataset_permissions(history_id, "dataset")
@@ -204,6 +213,7 @@ class TestHistoryContentsApi(ApiTestCase):
inheritance_chain = inheritance_chain_response.json()
assert len(inheritance_chain) == 1
+ @requires_new_library
def test_library_copy(self, history_id):
ld = self.library_populator.new_library_dataset("lda_test_library")
create_data = dict(
@@ -591,6 +601,7 @@ class TestHistoryContentsApi(ApiTestCase):
return element0["object"], element1["object"]
+ @requires_new_library
def test_hdca_from_library_datasets(self, history_id):
ld = self.library_populator.new_library_dataset("el1")
ldda_id = ld["ldda_id"]
@@ -612,6 +623,7 @@ class TestHistoryContentsApi(ApiTestCase):
assert hda["copied_from_ldda_id"] == ldda_id
assert hda["history_id"] == history_id
+ @requires_new_library
def test_hdca_from_inaccessible_library_datasets(self, history_id):
library, library_dataset = self.library_populator.new_library_dataset_in_private_library(
"HDCACreateInaccesibleLibrary"
@@ -943,6 +955,7 @@ class TestHistoryContentsApiBulkOperation(ApiTestCase):
if item["history_content_type"] == "dataset":
self.dataset_populator.wait_for_purge(history_id=history_id, content_id=item["id"])
+ @requires_new_user
def test_only_owner_can_apply_bulk_operations(self):
with self.dataset_populator.test_history() as history_id:
self._create_test_history_contents(history_id)
@@ -993,6 +1006,7 @@ class TestHistoryContentsApiBulkOperation(ApiTestCase):
for expected_tag in expected_tags:
assert expected_tag in item["tags"]
+ @requires_celery
def test_bulk_dbkey_change(self):
with self.dataset_populator.test_history() as history_id:
_, _, history_contents = self._create_test_history_contents(history_id)
@@ -1015,6 +1029,7 @@ class TestHistoryContentsApiBulkOperation(ApiTestCase):
if item["history_content_type"] == "dataset":
assert item["dbkey"] == expected_dbkey
+ @requires_celery
def test_bulk_dbkey_change_dataset_collection(self):
with self.dataset_populator.test_history() as history_id:
_, collection_ids, history_contents = self._create_test_history_contents(history_id)
diff --git a/lib/galaxy_test/api/test_users.py b/lib/galaxy_test/api/test_users.py
index 5701521921d..186bd039a77 100644
--- a/lib/galaxy_test/api/test_users.py
+++ b/lib/galaxy_test/api/test_users.py
@@ -8,6 +8,10 @@ from requests import (
from galaxy_test.api._framework import ApiTestCase
from galaxy_test.base.api_asserts import assert_object_id_error
+from galaxy_test.base.decorators import (
+ requires_admin,
+ requires_new_user,
+)
from galaxy_test.base.populators import skip_without_tool
TEST_USER_EMAIL = "user_for_users_index_test@bx.psu.edu"
@@ -17,6 +21,8 @@ TEST_USER_EMAIL_UNDELETE = "user_for_undelete_test@bx.psu.edu"
class TestUsersApi(ApiTestCase):
+ @requires_admin
+ @requires_new_user
def test_index(self):
self._setup_user(TEST_USER_EMAIL)
all_users_response = self._get("users", admin=True)
@@ -28,6 +34,7 @@ class TestUsersApi(ApiTestCase):
# new user.
assert len(all_users) > 1
+ @requires_new_user
def test_index_only_self_for_nonadmins(self):
self._setup_user(TEST_USER_EMAIL)
with self._different_user():
@@ -35,6 +42,7 @@ class TestUsersApi(ApiTestCase):
# Non admin users can only see themselves
assert len(all_users_response.json()) == 1
+ @requires_new_user
def test_show(self):
user = self._setup_user(TEST_USER_EMAIL)
with self._different_user(email=TEST_USER_EMAIL):
@@ -42,6 +50,7 @@ class TestUsersApi(ApiTestCase):
self._assert_status_code_is(show_response, 200)
self.__assert_matches_user(user, show_response.json())
+ @requires_new_user
def test_update(self):
new_name = "linnaeus"
user = self._setup_user(TEST_USER_EMAIL)
@@ -68,6 +77,8 @@ class TestUsersApi(ApiTestCase):
update_response = put(update_url, data=json.dumps(dict(username=new_name)))
assert_object_id_error(update_response)
+ @requires_admin
+ @requires_new_user
def test_admin_update(self):
new_name = "flexo"
user = self._setup_user(TEST_USER_EMAIL)
@@ -77,12 +88,16 @@ class TestUsersApi(ApiTestCase):
update_json = update_response.json()
assert update_json["username"] == new_name
+ @requires_admin
+ @requires_new_user
def test_delete_user(self):
user = self._setup_user(TEST_USER_EMAIL_DELETE)
self._delete(f"users/{user['id']}", admin=True)
updated_user = self._get(f"users/deleted/{user['id']}", admin=True).json()
assert updated_user["deleted"] is True, updated_user
+ @requires_admin
+ @requires_new_user
def test_purge_user(self):
"""Delete user and then purge them."""
user = self._setup_user(TEST_USER_EMAIL_PURGE)
@@ -96,6 +111,8 @@ class TestUsersApi(ApiTestCase):
assert purged_user["deleted"] is True, purged_user
assert purged_user["purged"] is True, purged_user
+ @requires_admin
+ @requires_new_user
def test_undelete_user(self):
"""Delete user and then undelete them."""
user = self._setup_user(TEST_USER_EMAIL_UNDELETE)
@@ -107,6 +124,7 @@ class TestUsersApi(ApiTestCase):
undeleted_user = self._get(f"users/{user['id']}", admin=True).json()
assert undeleted_user["deleted"] is False, undeleted_user
+ @requires_new_user
def test_information(self):
user = self._setup_user(TEST_USER_EMAIL)
url = self.__url("information/inputs", user)
@@ -126,6 +144,7 @@ class TestUsersApi(ApiTestCase):
assert len(response["addresses"]) == 1
assert response["addresses"][0]["desc"] == "_desc"
+ @requires_new_user
def test_manage_api_key(self):
with self._different_user("manage-api-key-test@user.com"):
user_id = self._get_current_user_id()
@@ -150,6 +169,7 @@ class TestUsersApi(ApiTestCase):
assert new_api_key
assert new_api_key != user_api_key
+ @requires_new_user
def test_only_admin_can_manage_other_users_api_key(self):
with self._different_user():
other_user_id = self._get_current_user_id()
@@ -171,6 +191,8 @@ class TestUsersApi(ApiTestCase):
response = self._delete(f"users/{other_user_id}/api_key", admin=True)
self._assert_status_code_is_ok(response)
+ @requires_admin
+ @requires_new_user
@skip_without_tool("cat1")
def test_favorites(self):
user = self._setup_user(TEST_USER_EMAIL)
@@ -233,3 +255,21 @@ class TestUsersApi(ApiTestCase):
users = users_response.json()
assert len(users) == 1
return users[0]["id"]
+
+ def test_manage_beacon_settings(self):
+ user_id = self._get_current_user_id()
+
+ # Assert that beacon sharing is initially disabled
+ response = self._get(f"users/{user_id}/beacon")
+ user_beacon_settings = response.json()
+ assert not user_beacon_settings["enabled"]
+
+ # Check if post request is successful
+ response = self._post(f"users/{user_id}/beacon", data={"enabled": True}, json=True)
+ user_beacon_settings = response.json()
+ assert user_beacon_settings["enabled"]
+
+ # Check if the setting has been persisted
+ response = self._get(f"users/{user_id}/beacon")
+ user_beacon_settings = response.json()
+ assert user_beacon_settings["enabled"]
diff --git a/lib/galaxy_test/api/test_workflows.py b/lib/galaxy_test/api/test_workflows.py
index 921f4ad3dcc..92bbd17a2aa 100644
--- a/lib/galaxy_test/api/test_workflows.py
+++ b/lib/galaxy_test/api/test_workflows.py
@@ -36,6 +36,7 @@ from galaxy_test.base.populators import (
WorkflowPopulator,
)
from galaxy_test.base.workflow_fixtures import (
+ NESTED_WORKFLOW_WITH_CONDITIONAL_SUBWORKFLOW_AND_DISCONNECTED_MAP_OVER_SOURCE,
WORKFLOW_INPUTS_AS_OUTPUTS,
WORKFLOW_NESTED_REPLACEMENT_PARAMETER,
WORKFLOW_NESTED_RUNTIME_PARAMETER,
@@ -1930,6 +1931,434 @@ test_data:
history_id=history_id,
)
+ def test_run_workflow_simple_conditional_step(self):
+ with self.dataset_populator.test_history() as history_id:
+ summary = self._run_workflow(
+ """class: GalaxyWorkflow
+inputs:
+ should_run:
+ type: boolean
+ some_file:
+ type: data
+steps:
+ cat1:
+ tool_id: cat1
+ in:
+ input1: some_file
+ should_run: should_run
+ when: $(inputs.should_run)
+""",
+ test_data="""
+some_file:
+ value: 1.bed
+ type: File
+should_run:
+ value: false
+ type: raw
+""",
+ history_id=history_id,
+ )
+ invocation_details = self.workflow_populator.get_invocation(summary.invocation_id, step_details=True)
+ for step in invocation_details["steps"]:
+ if step["workflow_step_label"] == "cat1":
+ assert sum(1 for j in step["jobs"] if j["state"] == "skipped") == 1
+
+ def test_run_workflow_invalid_when_expression(self):
+ with self.dataset_populator.test_history() as history_id:
+ summary = self._run_workflow(
+ """class: GalaxyWorkflow
+inputs:
+ should_run:
+ type: boolean
+ some_file:
+ type: data
+steps:
+ cat1:
+ tool_id: cat1
+ in:
+ input1: some_file
+ should_run: should_run
+ when: $(:syntaxError:)
+""",
+ test_data="""
+some_file:
+ value: 1.bed
+ type: File
+should_run:
+ value: false
+ type: raw
+""",
+ history_id=history_id,
+ wait=True,
+ assert_ok=False,
+ )
+ invocation_details = self.workflow_populator.get_invocation(summary.invocation_id, step_details=True)
+ assert invocation_details["state"] == "failed"
+ assert len(invocation_details["messages"]) == 1
+ message = invocation_details["messages"][0]
+ assert message["reason"] == "expression_evaluation_failed"
+
+ def test_run_workflow_fails_when_expression_not_boolean(self):
+ with self.dataset_populator.test_history() as history_id:
+ summary = self._run_workflow(
+ """class: GalaxyWorkflow
+inputs:
+ should_run:
+ type: boolean
+ some_file:
+ type: data
+steps:
+ cat1:
+ tool_id: cat1
+ in:
+ input1: some_file
+ should_run: should_run
+ when: $("false")
+""",
+ test_data="""
+some_file:
+ value: 1.bed
+ type: File
+should_run:
+ value: false
+ type: raw
+""",
+ history_id=history_id,
+ wait=True,
+ assert_ok=False,
+ )
+ invocation_details = self.workflow_populator.get_invocation(summary.invocation_id, step_details=True)
+ assert invocation_details["state"] == "failed"
+ assert len(invocation_details["messages"]) == 1
+ message = invocation_details["messages"][0]
+ assert message["reason"] == "when_not_boolean"
+ assert message["details"] == "Type is: str"
+ assert message["workflow_step_id"] == 2
+
+ def test_run_workflow_subworkflow_conditional_step(self):
+ with self.dataset_populator.test_history() as history_id:
+ summary = self._run_workflow(
+ """class: GalaxyWorkflow
+inputs:
+ should_run:
+ type: boolean
+ some_file:
+ type: data
+steps:
+ subworkflow:
+ run:
+ class: GalaxyWorkflow
+ inputs:
+ some_file:
+ type: data
+ should_run:
+ type: boolean
+ steps:
+ a_tool_step:
+ tool_id: cat1
+ in:
+ input1: some_file
+ in:
+ some_file: some_file
+ should_run: should_run
+ outputs:
+ inner_out: a_tool_step/out_file1
+ when: $(inputs.should_run)
+outputs:
+ outer_output:
+ outputSource: subworkflow/inner_out
+""",
+ test_data="""
+some_file:
+ value: 1.bed
+ type: File
+should_run:
+ value: false
+ type: raw
+""",
+ history_id=history_id,
+ wait=True,
+ assert_ok=True,
+ )
+ invocation_details = self.workflow_populator.get_invocation(summary.invocation_id, step_details=True)
+ subworkflow_invocation_id = invocation_details["steps"][-1]["subworkflow_invocation_id"]
+ self.workflow_populator.wait_for_invocation_and_jobs(
+ history_id=history_id, workflow_id="whatever", invocation_id=subworkflow_invocation_id
+ )
+ invocation_details = self.workflow_populator.get_invocation(subworkflow_invocation_id, step_details=True)
+ for step in invocation_details["steps"]:
+ if step["workflow_step_label"] == "a_tool_step":
+ assert sum(1 for j in step["jobs"] if j["state"] == "skipped") == 1
+
+ def test_run_workflow_conditional_step_map_over_expression_tool(self):
+ with self.dataset_populator.test_history() as history_id:
+ summary = self._run_workflow(
+ """
+class: GalaxyWorkflow
+inputs:
+ boolean_input_files: collection
+steps:
+- label: param_out
+ tool_id: param_value_from_file
+ in:
+ input1: boolean_input_files
+ state:
+ param_type: boolean
+- label: consume_expression_parameter
+ tool_id: cat1
+ in:
+ input1: boolean_input_files
+ should_run: param_out/boolean_param
+ out:
+ out_file1:
+ change_datatype: txt
+ when: $(inputs.should_run)
+test_data:
+ boolean_input_files:
+ collection_type: list
+ elements:
+ - identifier: true
+ content: true
+ - identifier: false
+ content: false
+""",
+ history_id=history_id,
+ )
+ invocation_details = self.workflow_populator.get_invocation(summary.invocation_id, step_details=True)
+ for step in invocation_details["steps"]:
+ if step["workflow_step_label"] == "consume_expression_parameter":
+ skipped_jobs = [j for j in step["jobs"] if j["state"] == "skipped"]
+ assert len(skipped_jobs) == 1
+ # also assert that change_datatype was ignored for null output
+ job_details = self.dataset_populator.get_job_details(skipped_jobs[0]["id"], full=True).json()
+ skipped_hda_id = job_details["outputs"]["out_file1"]["id"]
+ dataset_details = self.dataset_populator.get_history_dataset_details(
+ history_id=history_id, content_id=skipped_hda_id
+ )
+ assert dataset_details["file_ext"] == "expression.json", dataset_details
+
+ def test_run_workflow_conditional_subworkflow_step_map_over_expression_tool(self):
+ with self.dataset_populator.test_history() as history_id:
+ summary = self._run_workflow(
+ """
+class: GalaxyWorkflow
+inputs:
+ boolean_input_files: collection
+steps:
+ create_list_of_boolean:
+ tool_id: param_value_from_file
+ in:
+ input1: boolean_input_files
+ state:
+ param_type: boolean
+ subworkflow:
+ run:
+ class: GalaxyWorkflow
+ inputs:
+ boolean_input_file: data
+ should_run: boolean
+ steps:
+ consume_expression_parameter:
+ tool_id: cat1
+ in:
+ input1: boolean_input_file
+ out:
+ out_file1:
+ change_datatype: txt
+ outputs:
+ inner_output:
+ outputSource: consume_expression_parameter/out_file1
+ in:
+ boolean_input_file: boolean_input_files
+ should_run: create_list_of_boolean/boolean_param
+ when: $(inputs.should_run)
+outputs:
+ outer_output:
+ outputSource: subworkflow/inner_output
+test_data:
+ boolean_input_files:
+ collection_type: list
+ elements:
+ - identifier: true
+ content: true
+ - identifier: false
+ content: false
+""",
+ history_id=history_id,
+ )
+ invocation_details = self.workflow_populator.get_invocation(summary.invocation_id, step_details=True)
+ assert "outer_output" in invocation_details["output_collections"]
+ outer_output = invocation_details["output_collections"]["outer_output"]
+ outer_hdca = self.dataset_populator.get_history_collection_details(
+ history_id, content_id=outer_output["id"]
+ )
+ assert outer_hdca["job_state_summary"]["all_jobs"] == 2
+ assert outer_hdca["job_state_summary"]["ok"] == 1
+ assert outer_hdca["job_state_summary"]["skipped"] == 1
+
+ def test_run_workflow_conditional_subworkflow_step_map_over_expression_tool_with_extra_nesting(self):
+ with self.dataset_populator.test_history() as history_id:
+ summary = self._run_workflow(
+ NESTED_WORKFLOW_WITH_CONDITIONAL_SUBWORKFLOW_AND_DISCONNECTED_MAP_OVER_SOURCE,
+ test_data="""boolean_input_files:
+ collection_type: list
+ elements:
+ - identifier: true
+ content: true
+ - identifier: false
+ content: false
+""",
+ history_id=history_id,
+ )
+ invocation_details = self.workflow_populator.get_invocation(summary.invocation_id, step_details=True)
+ outer_create_nested_id = invocation_details["output_collections"]["outer_create_nested"]["id"]
+ outer_create_nested = self.dataset_populator.get_history_collection_details(
+ history_id, content_id=outer_create_nested_id
+ )
+ assert outer_create_nested["job_state_summary"]["all_jobs"] == 2
+ assert outer_create_nested["job_state_summary"]["ok"] == 1
+ assert outer_create_nested["job_state_summary"]["skipped"] == 1
+
+ for cat1_output in ["outer_output_1", "outer_output_2"]:
+ outer_output = invocation_details["output_collections"][cat1_output]
+ outer_hdca = self.dataset_populator.get_history_collection_details(
+ history_id, content_id=outer_output["id"]
+ )
+ # You might expect 12 total jobs, 6 ok and 6 skipped,
+ # but because we're not actually running one branch of collection_creates_dynamic_nested
+ # there's no input to consume_expression_parameter.
+ # It's unclear if that's a problem or not ... probably not a major one,
+ # since we keep producing "empty" outer collections, which seems somewhat correct.
+ assert outer_hdca["job_state_summary"]["all_jobs"] == 6
+ assert outer_hdca["job_state_summary"]["ok"] == 6
+ assert outer_hdca["collection_type"] == "list:list:list"
+ elements = outer_hdca["elements"]
+ assert elements[0]["element_identifier"] == "True"
+ assert elements[0]["object"]["element_count"] == 3
+ assert elements[1]["element_identifier"] == "False"
+ assert elements[1]["object"]["element_count"] == 0
+
+ def test_run_workflow_conditional_subworkflow_step_map_over_expression_tool_with_extra_nesting_skip_all(self):
+ with self.dataset_populator.test_history() as history_id:
+ summary = self._run_workflow(
+ NESTED_WORKFLOW_WITH_CONDITIONAL_SUBWORKFLOW_AND_DISCONNECTED_MAP_OVER_SOURCE,
+ test_data="""boolean_input_files:
+ collection_type: list
+ elements:
+ - identifier: false
+ content: false
+ - identifier: also_false
+ content: false
+""",
+ history_id=history_id,
+ )
+ invocation_details = self.workflow_populator.get_invocation(summary.invocation_id, step_details=True)
+ outer_create_nested_id = invocation_details["output_collections"]["outer_create_nested"]["id"]
+ outer_create_nested = self.dataset_populator.get_history_collection_details(
+ history_id, content_id=outer_create_nested_id
+ )
+ assert outer_create_nested["job_state_summary"]["all_jobs"] == 2
+ assert outer_create_nested["job_state_summary"]["skipped"] == 2
+
+ for cat1_output in ["outer_output_1", "outer_output_2"]:
+ outer_output = invocation_details["output_collections"][cat1_output]
+ outer_hdca = self.dataset_populator.get_history_collection_details(
+ history_id, content_id=outer_output["id"]
+ )
+ assert outer_hdca["job_state_summary"]["all_jobs"] == 0
+ assert outer_hdca["collection_type"] == "list:list:list"
+
+ def test_run_workflow_conditional_step_map_over_expression_tool_pick_value(self):
+ with self.dataset_populator.test_history() as history_id:
+ summary = self._run_workflow(
+ """
+class: GalaxyWorkflow
+inputs:
+ boolean_input_files_1: collection
+ boolean_input_files_2: collection
+outputs:
+ my_output:
+ outputSource: pick_value/data_param
+steps:
+- label: param_out_1
+ tool_id: param_value_from_file
+ in:
+ input1: boolean_input_files_1
+ state:
+ param_type: boolean
+- label: param_out_2
+ tool_id: param_value_from_file
+ in:
+ input1: boolean_input_files_2
+ state:
+ param_type: boolean
+- label: consume_expression_parameter_1
+ tool_id: cat1
+ in:
+ input1: boolean_input_files_1
+ should_run: param_out_1/boolean_param
+ when: $(inputs.should_run)
+- label: consume_expression_parameter_2
+ tool_id: cat1
+ in:
+ input1: boolean_input_files_2
+ should_run: param_out_2/boolean_param
+ when: $(inputs.should_run)
+- label: pick_value
+ tool_id: pick_value
+ tool_state:
+ style_cond:
+ __current_case__: 2
+ pick_style: first_or_error
+ type_cond:
+ __current_case__: 4
+ param_type: data
+ pick_from:
+ - __index__: 0
+ value:
+ __class__: RuntimeValue
+ - __index__: 1
+ value:
+ __class__: RuntimeValue
+ in:
+ style_cond|type_cond|pick_from_0|value:
+ source: consume_expression_parameter_1/out_file1
+ style_cond|type_cond|pick_from_1|value:
+ source: consume_expression_parameter_2/out_file1
+test_data:
+ boolean_input_files_1:
+ collection_type: list
+ elements:
+ - identifier: true
+ content: true
+ - identifier: false
+ content: false
+ boolean_input_files_2:
+ collection_type: list
+ elements:
+ - identifier: false
+ content: false
+ - identifier: true
+ content: true
+""",
+ history_id=history_id,
+ )
+ invocation_details = self.workflow_populator.get_invocation(summary.invocation_id, step_details=True)
+ output_collection_id = invocation_details["output_collections"]["my_output"]["id"]
+ hdca_details = self.dataset_populator.get_history_collection_details(
+ history_id=history_id, content_id=output_collection_id
+ )
+ elements = hdca_details["elements"]
+ assert len(elements) == 2
+ for element in elements:
+ content = self.dataset_populator.get_history_dataset_content(
+ history_id, content_id=element["object"]["id"]
+ )
+ assert content == "True"
+ for step in invocation_details["steps"]:
+ if step["workflow_step_label"].startswith("consume_expression_parameter_"):
+ assert sum(1 for j in step["jobs"] if j["state"] == "skipped") == 1
+
def test_run_subworkflow_simple(self) -> None:
with self.dataset_populator.test_history() as history_id:
summary = self._run_workflow(
@@ -3096,6 +3525,12 @@ input1:
self._delete(f"histories/{history_id}")
invocation_cancelled = self._wait_for_invocation_state(uploaded_workflow_id, invocation_id, "cancelled")
+ workflow_details = self._invocation_details(uploaded_workflow_id, invocation_id)
+ assert len(workflow_details["messages"]) == 1
+ message = workflow_details["messages"][0]
+ assert message["history_id"] == history_id
+ assert message["reason"] == "history_deleted"
+
assert invocation_cancelled, "Workflow state is not cancelled..."
@skip_without_tool("cat")
@@ -3117,6 +3552,11 @@ input1:
invocation_cancelled = self._wait_for_invocation_state(uploaded_workflow_id, invocation_id, "cancelled")
assert invocation_cancelled, "Workflow state is not cancelled..."
+ workflow_details = self._invocation_details(uploaded_workflow_id, invocation_id)
+ assert len(workflow_details["messages"]) == 1
+ message = workflow_details["messages"][0]
+ assert message["history_id"] == history_id
+ assert message["reason"] == "history_deleted"
@skip_without_tool("cat")
def test_workflow_pause(self):
@@ -3164,6 +3604,11 @@ input1:
# Ensure the workflow eventually becomes cancelled.
invocation_cancelled = self._wait_for_invocation_state(uploaded_workflow_id, invocation_id, "cancelled")
+ workflow_details = self._invocation_details(uploaded_workflow_id, invocation_id)
+ assert len(workflow_details["messages"]) == 1
+ message = workflow_details["messages"][0]
+ assert "workflow_step_id" in message
+ assert message["reason"] == "cancelled_on_review"
assert invocation_cancelled, "Workflow state is not cancelled..."
@skip_without_tool("head")
@@ -3220,6 +3665,71 @@ input1:
invocation = self._invocation_details(uploaded_workflow_id, invocation_id)
assert invocation["state"] == "cancelled"
+ message = invocation["messages"][0]
+ assert message["reason"] == "user_request"
+
+ def test_workflow_failed_output_not_found(self, history_id):
+ summary = self._run_workflow(
+ """
+class: GalaxyWorkflow
+inputs: []
+steps:
+ create_2:
+ tool_id: create_2
+ state:
+ sleep_time: 0
+ outputs:
+ out_file1:
+ rename: "my new name"
+ out_file2:
+ rename: "my other new name"
+ first_cat1:
+ tool_id: cat
+ in:
+ input1: create_2/does_not_exist
+ """,
+ history_id=history_id,
+ assert_ok=False,
+ wait=True,
+ )
+ invocation = self.workflow_populator.get_invocation(summary.invocation_id)
+ assert invocation["state"] == "failed"
+ assert len(invocation["messages"]) == 1
+ message = invocation["messages"][0]
+ assert message["reason"] == "output_not_found"
+ assert message["workflow_step_id"] == 1
+ assert message["dependent_workflow_step_id"] == 0
+
+ def test_workflow_warning_workflow_output_not_found(self, history_id):
+ summary = self._run_workflow(
+ """
+class: GalaxyWorkflow
+inputs: []
+steps:
+ create_2:
+ tool_id: create_2
+ state:
+ sleep_time: 0
+ outputs:
+ out_file1:
+ rename: "my new name"
+ out_file2:
+ rename: "my other new name"
+outputs:
+ main_out:
+ outputSource: create_2/does_not_exist
+ """,
+ history_id=history_id,
+ assert_ok=False,
+ wait=True,
+ )
+ invocation = self.workflow_populator.get_invocation(summary.invocation_id)
+ assert invocation["state"] == "scheduled"
+ assert len(invocation["messages"]) == 1
+ message = invocation["messages"][0]
+ assert message["reason"] == "workflow_output_not_found"
+ assert "workflow_step_id" in message
+ assert message["output_name"] == "does_not_exist"
@skip_without_tool("identifier_multiple")
def test_invocation_map_over(self, history_id):
diff --git a/lib/galaxy_test/base/decorators.py b/lib/galaxy_test/base/decorators.py
index 8e8be0b6c05..91613faa439 100644
--- a/lib/galaxy_test/base/decorators.py
+++ b/lib/galaxy_test/base/decorators.py
@@ -18,6 +18,7 @@ from typing_extensions import Literal
KnownRequirementT = Union[
Literal["admin"],
+ Literal["celery"],
Literal["new_history"],
Literal["new_library"],
Literal["new_published_objects"],
@@ -92,9 +93,14 @@ def requires_admin(method):
return _wrap_method_with_galaxy_requirement(method, "admin")
+def requires_celery(method):
+ return _wrap_method_with_galaxy_requirement(method, "celery")
+
+
__all__ = (
"has_requirement",
"requires_admin",
+ "requires_celery",
"requires_new_history",
"requires_new_library",
"requires_new_published_objects",
diff --git a/lib/galaxy_test/base/workflow_fixtures.py b/lib/galaxy_test/base/workflow_fixtures.py
index f01a3c7a13e..0f164c087b0 100644
--- a/lib/galaxy_test/base/workflow_fixtures.py
+++ b/lib/galaxy_test/base/workflow_fixtures.py
@@ -994,3 +994,60 @@ bed_input:
file_type: bed
type: File
"""
+
+
+NESTED_WORKFLOW_WITH_CONDITIONAL_SUBWORKFLOW_AND_DISCONNECTED_MAP_OVER_SOURCE = """
+class: GalaxyWorkflow
+inputs:
+ boolean_input_files: collection
+steps:
+ create_list_of_boolean:
+ tool_id: param_value_from_file
+ in:
+ input1: boolean_input_files
+ state:
+ param_type: boolean
+ subworkflow:
+ run:
+ class: GalaxyWorkflow
+ inputs:
+ boolean_input_file: data
+ should_run: boolean
+ steps:
+ create_more_inputs:
+ tool_id: collection_creates_dynamic_nested
+ consume_expression_parameter:
+ tool_id: cat1
+ state:
+ input1:
+ $link: create_more_inputs/list_output
+ queries:
+ - input2:
+ $link: boolean_input_file
+ out:
+ out_file1:
+ change_datatype: txt
+ consume_expression_parameter_2:
+ tool_id: cat1
+ state:
+ input1:
+ $link: consume_expression_parameter/out_file1
+ outputs:
+ inner_create_nested:
+ outputSource: create_more_inputs/list_output
+ inner_output_1:
+ outputSource: consume_expression_parameter/out_file1
+ inner_output_2:
+ outputSource: consume_expression_parameter_2/out_file1
+ in:
+ boolean_input_file: boolean_input_files
+ should_run: create_list_of_boolean/boolean_param
+ when: $(inputs.should_run)
+outputs:
+ outer_create_nested:
+ outputSource: subworkflow/inner_create_nested
+ outer_output_1:
+ outputSource: subworkflow/inner_output_1
+ outer_output_2:
+ outputSource: subworkflow/inner_output_2
+"""
diff --git a/lib/galaxy_test/selenium/test_anon_history.py b/lib/galaxy_test/selenium/test_anon_history.py
index d8f7ab1329a..ab1ea2289c4 100644
--- a/lib/galaxy_test/selenium/test_anon_history.py
+++ b/lib/galaxy_test/selenium/test_anon_history.py
@@ -1,3 +1,4 @@
+from galaxy_test.base.decorators import requires_new_user
from .framework import (
selenium_test,
SeleniumTestCase,
@@ -25,12 +26,14 @@ class TestAnonymousHistories(SeleniumTestCase):
self.components.history_panel.empty_message.assert_absent_or_hidden()
@selenium_test
+ @requires_new_user
def test_anon_history_after_registration(self):
self._upload_file_anonymous_then_register_user()
self.home()
self.history_panel_wait_for_hid_state(1, "ok")
@selenium_test
+ @requires_new_user
def test_clean_anon_history_after_logout(self):
self._upload_file_anonymous_then_register_user()
self.logout_if_needed()
diff --git a/lib/galaxy_test/selenium/test_collection_builders.py b/lib/galaxy_test/selenium/test_collection_builders.py
index 21f3c492d37..25b6d0afa0f 100644
--- a/lib/galaxy_test/selenium/test_collection_builders.py
+++ b/lib/galaxy_test/selenium/test_collection_builders.py
@@ -1,4 +1,5 @@
from .framework import (
+ managed_history,
selenium_test,
SeleniumTestCase,
)
@@ -9,6 +10,7 @@ class TestCollectionBuilders(SeleniumTestCase):
ensure_registered = True
@selenium_test
+ @managed_history
def test_build_list_simple_hidden(self):
self.perform_upload(self.get_filename("1.fasta"))
self._wait_for_and_select([1])
@@ -19,6 +21,7 @@ class TestCollectionBuilders(SeleniumTestCase):
self._wait_for_hid_visible(2)
@selenium_test
+ @managed_history
def test_build_list_and_show_items(self):
self.perform_upload(self.get_filename("1.fasta"))
self._wait_for_and_select([1])
@@ -32,6 +35,7 @@ class TestCollectionBuilders(SeleniumTestCase):
self._wait_for_hid_visible(3)
@selenium_test
+ @managed_history
def test_build_pair_simple_hidden(self):
self.perform_upload(self.get_filename("1.tabular"))
self.perform_upload(self.get_filename("2.tabular"))
@@ -43,6 +47,7 @@ class TestCollectionBuilders(SeleniumTestCase):
self._wait_for_hid_visible(3)
@selenium_test
+ @managed_history
def test_build_paired_list_simple(self):
self.perform_upload(self.get_filename("1.tabular"))
self.perform_upload(self.get_filename("2.tabular"))
@@ -61,6 +66,7 @@ class TestCollectionBuilders(SeleniumTestCase):
self._wait_for_hid_visible(2)
@selenium_test
+ @managed_history
def test_build_paired_list_show_original(self):
self.perform_upload(self.get_filename("1.tabular"))
self.perform_upload(self.get_filename("2.tabular"))
@@ -85,6 +91,7 @@ class TestCollectionBuilders(SeleniumTestCase):
self._wait_for_hid_visible(4)
@selenium_test
+ @managed_history
def test_build_simple_list_via_rules_hidden(self):
self.perform_upload(self.get_filename("1.fasta"))
self._wait_for_and_select([1])
diff --git a/lib/galaxy_test/selenium/test_collection_edit.py b/lib/galaxy_test/selenium/test_collection_edit.py
index 279724d97b9..73148a36816 100644
--- a/lib/galaxy_test/selenium/test_collection_edit.py
+++ b/lib/galaxy_test/selenium/test_collection_edit.py
@@ -1,4 +1,5 @@
from .framework import (
+ managed_history,
selenium_test,
SeleniumTestCase,
)
@@ -9,6 +10,7 @@ class TestCollectionEdit(SeleniumTestCase):
ensure_registered = True
@selenium_test
+ @managed_history
def test_change_dbkey_simple_list(self):
self.create_simple_list_collection()
self.open_collection_edit_view()
@@ -23,6 +25,7 @@ class TestCollectionEdit(SeleniumTestCase):
self.check_current_data_value(dataNew)
@selenium_test
+ @managed_history
def test_change_datatype_simple_list(self):
self.create_simple_list_collection_txt()
self.open_collection_edit_view()
diff --git a/lib/galaxy_test/selenium/test_dataset_metadata_download.py b/lib/galaxy_test/selenium/test_dataset_metadata_download.py
index 3df1e638dce..0a36a93078f 100644
--- a/lib/galaxy_test/selenium/test_dataset_metadata_download.py
+++ b/lib/galaxy_test/selenium/test_dataset_metadata_download.py
@@ -1,4 +1,5 @@
from .framework import (
+ managed_history,
selenium_test,
SeleniumTestCase,
UsesHistoryItemAssertions,
@@ -11,6 +12,7 @@ class TestHistoryDatasetState(SeleniumTestCase, UsesHistoryItemAssertions):
ensure_registered = True
@selenium_test
+ @managed_history
def test_dataset_metadata_download(self):
self._prepare_dataset()
item = self.history_panel_item_component(hid=FIRST_HID)
diff --git a/lib/galaxy_test/selenium/test_history_dataset_state.py b/lib/galaxy_test/selenium/test_history_dataset_state.py
index e4b059ecff5..eae0342fc4f 100644
--- a/lib/galaxy_test/selenium/test_history_dataset_state.py
+++ b/lib/galaxy_test/selenium/test_history_dataset_state.py
@@ -4,6 +4,7 @@ from galaxy.model.unittest_utils.store_fixtures import (
TEST_SOURCE_URI,
)
from .framework import (
+ managed_history,
selenium_test,
SeleniumTestCase,
UsesHistoryItemAssertions,
@@ -26,6 +27,7 @@ class TestHistoryDatasetState(SeleniumTestCase, UsesHistoryItemAssertions):
ensure_registered = True
@selenium_test
+ @managed_history
def test_dataset_state(self):
self._prepare_dataset()
self.history_panel_item_body_component(FIRST_HID, wait=True)
@@ -40,6 +42,7 @@ class TestHistoryDatasetState(SeleniumTestCase, UsesHistoryItemAssertions):
self.screenshot("dataset_details_ok")
@selenium_test
+ @managed_history
def test_dataset_change_dbkey(self):
item = self._prepare_dataset()
self.assert_item_dbkey_displayed_as(FIRST_HID, "?")
@@ -57,6 +60,7 @@ class TestHistoryDatasetState(SeleniumTestCase, UsesHistoryItemAssertions):
self.assert_item_dbkey_displayed_as(FIRST_HID, "apiMel3")
@selenium_test
+ @managed_history
def test_dataset_state_discarded(self):
self.history_panel_create_new()
history_id = self.current_history_id()
@@ -77,6 +81,7 @@ class TestHistoryDatasetState(SeleniumTestCase, UsesHistoryItemAssertions):
self.screenshot("dataset_details_discarded")
@selenium_test
+ @managed_history
def test_dataset_state_deferred(self):
self.history_panel_create_new()
history_id = self.current_history_id()
diff --git a/lib/galaxy_test/selenium/test_history_multi_view.py b/lib/galaxy_test/selenium/test_history_multi_view.py
index 785c2c84f28..03db2c2c418 100644
--- a/lib/galaxy_test/selenium/test_history_multi_view.py
+++ b/lib/galaxy_test/selenium/test_history_multi_view.py
@@ -1,4 +1,5 @@
from .framework import (
+ managed_history,
selenium_test,
SeleniumTestCase,
)
@@ -22,6 +23,7 @@ class TestHistoryMultiView(SeleniumTestCase):
self.screenshot("multi_history_collection")
@selenium_test
+ @managed_history
def test_list_list_display(self):
history_id = self.current_history_id()
method = self.dataset_collection_populator.create_list_of_list_in_history(history_id, wait=True).json
diff --git a/lib/galaxy_test/selenium/test_visualizations.py b/lib/galaxy_test/selenium/test_visualizations.py
new file mode 100644
index 00000000000..ca662748a64
--- /dev/null
+++ b/lib/galaxy_test/selenium/test_visualizations.py
@@ -0,0 +1,20 @@
+from .framework import (
+ selenium_test,
+ SeleniumTestCase,
+)
+
+
+class TestVisualizations(SeleniumTestCase):
+ @selenium_test
+ def test_charts_framework(self):
+ self.register()
+ self.perform_upload(self.get_filename("1.sam"))
+ self.history_panel_wait_for_hid_ok(1)
+ dataset_component = self.history_panel_click_item_title(1, wait=True)
+ dataset_component.visualize_button.wait_for_and_click()
+
+ self.components.visualization.plugin_item(id="nvd3_bar").wait_for_and_click()
+
+ with self.main_panel():
+ self.wait_for_selector("g.nvd3")
+ self.screenshot("test_charts_framework -- nvd3_bar")
diff --git a/lib/galaxy_test/selenium/test_workflow_editor.py b/lib/galaxy_test/selenium/test_workflow_editor.py
index f58142eef8b..eb40f60ff96 100644
--- a/lib/galaxy_test/selenium/test_workflow_editor.py
+++ b/lib/galaxy_test/selenium/test_workflow_editor.py
@@ -557,7 +557,18 @@ steps:
editor.select_datatype_text_search.wait_for_and_send_keys("bam")
editor.select_datatype(datatype="bam").wait_for_and_click()
editor.node.output_data_row(output_name="out_file1", extension="bam").wait_for_visible()
- self.assert_not_connected("create_2#out_file1", "checksum#input")
+ self.assert_connection_invalid("create_2#out_file1", "checksum#input")
+ save_button = self.components.workflow_editor.save_button
+ # Assert save button is disabled
+ assert save_button.has_class("disabled")
+ # Make connection valid again
+ editor.change_datatype.wait_for_and_click()
+ editor.select_datatype_text_search.wait_for_and_send_keys("tabular")
+ editor.select_datatype(datatype="tabular").wait_for_and_click()
+ # Assert connection is valid
+ self.assert_connected("create_2#out_file1", "checksum#input")
+ # Assert save button is enabled
+ assert not save_button.has_class("disabled")
@selenium_test
def test_change_datatype_post_job_action_lost_regression(self):
@@ -686,7 +697,10 @@ steps:
workflow_populator.upload_yaml_workflow(WORKFLOW_OPTIONAL_TRUE_INPUT_COLLECTION, name=child_workflow_name)
parent_workflow_id = workflow_populator.upload_yaml_workflow(
"""class: GalaxyWorkflow
-inputs: []
+inputs:
+ input_collection:
+ type: collection
+ collection_type: "list"
steps:
- tool_id: multiple_versions
tool_version: 0.1
@@ -705,10 +719,16 @@ steps:
self.sleep_for(self.wait_types.UX_RENDER)
self.assert_workflow_has_changes_and_save()
workflow = self.workflow_populator.download_workflow(parent_workflow_id)
- subworkflow_step = workflow["steps"]["1"]
+ subworkflow_step = workflow["steps"]["2"]
assert subworkflow_step["name"] == child_workflow_name
assert subworkflow_step["type"] == "subworkflow"
assert subworkflow_step["subworkflow"]["a_galaxy_workflow"] == "true"
+ self.workflow_editor_connect("input_collection#output", f"{child_workflow_name}#input1")
+ self.assert_connected("input_collection#output", f"{child_workflow_name}#input1")
+ self.assert_workflow_has_changes_and_save()
+ workflow = self.workflow_populator.download_workflow(parent_workflow_id)
+ subworkflow_step = workflow["steps"]["2"]
+ assert subworkflow_step["input_connections"]["input1"]["input_subworkflow_step_id"] == 0
@selenium_test
def test_editor_insert_steps(self):
@@ -862,6 +882,10 @@ steps:
source_id, sink_id = self.workflow_editor_source_sink_terminal_ids(source, sink)
self.components.workflow_editor.connector_for(source_id=source_id, sink_id=sink_id).wait_for_visible()
+ def assert_connection_invalid(self, source, sink):
+ source_id, sink_id = self.workflow_editor_source_sink_terminal_ids(source, sink)
+ self.components.workflow_editor.connector_invalid_for(source_id=source_id, sink_id=sink_id).wait_for_present()
+
def assert_not_connected(self, source, sink):
source_id, sink_id = self.workflow_editor_source_sink_terminal_ids(source, sink)
self.components.workflow_editor.connector_for(source_id=source_id, sink_id=sink_id).wait_for_absent()
diff --git a/lib/tool_shed/webapp/api/repositories.py b/lib/tool_shed/webapp/api/repositories.py
index 5dfc86d2ecf..86f6a9f859e 100644
--- a/lib/tool_shed/webapp/api/repositories.py
+++ b/lib/tool_shed/webapp/api/repositories.py
@@ -50,6 +50,7 @@ from tool_shed.util import (
repository_util,
tool_util,
)
+from tool_shed.webapp import model
from tool_shed.webapp.search.repo_search import RepoSearch
log = logging.getLogger(__name__)
@@ -118,7 +119,7 @@ class RepositoriesController(BaseAPIController):
if owner is None:
owner = kwd.get("owner", None)
tsr_id = kwd.get("tsr_id", None)
- eagerload_columns = ["downloadable_revisions"]
+ eagerload_columns = [model.Repository.downloadable_revisions]
if None not in [name, owner]:
# Get the repository information.
repository = repository_util.get_repository_by_name_and_owner(
@@ -217,7 +218,7 @@ class RepositoriesController(BaseAPIController):
if name and owner and changeset_revision:
# Get the repository information.
repository = repository_util.get_repository_by_name_and_owner(
- self.app, name, owner, eagerload_columns=["downloadable_revisions"]
+ self.app, name, owner, eagerload_columns=[model.Repository.downloadable_revisions]
)
if repository is None:
log.debug(f"Cannot locate repository {name} owned by {owner}")
@@ -284,7 +285,7 @@ class RepositoriesController(BaseAPIController):
tsr_id = kwd.get("tsr_id", None)
if tsr_id is not None:
repository = repository_util.get_repository_in_tool_shed(
- self.app, tsr_id, eagerload_columns=["downloadable_revisions"]
+ self.app, tsr_id, eagerload_columns=[model.Repository.downloadable_revisions]
)
else:
error_message = "Error in the Tool Shed repositories API in get_ordered_installable_revisions: "
@@ -737,7 +738,7 @@ class RepositoriesController(BaseAPIController):
changeset_revision = kwd.get("changeset_revision", None)
hexlify_this = util.asbool(kwd.get("hexlify", True))
repository = repository_util.get_repository_by_name_and_owner(
- trans.app, name, owner, eagerload_columns=["downloadable_revisions"]
+ trans.app, name, owner, eagerload_columns=[model.Repository.downloadable_revisions]
)
if repository and repository.downloadable_revisions:
repository_metadata = metadata_util.get_repository_metadata_by_changeset_revision(
@@ -837,7 +838,7 @@ class RepositoriesController(BaseAPIController):
downloadable_only = util.asbool(kwd.get("downloadable_only", "True"))
all_metadata = {}
repository = repository_util.get_repository_in_tool_shed(
- self.app, id, eagerload_columns=["downloadable_revisions"]
+ self.app, id, eagerload_columns=[model.Repository.downloadable_revisions]
)
for changeset, changehash in metadata_util.get_metadata_revisions(
self.app, repository, sort_revisions=True, downloadable=downloadable_only
diff --git a/package.json b/package.json
new file mode 100644
index 00000000000..05aabcb42eb
--- /dev/null
+++ b/package.json
@@ -0,0 +1,24 @@
+{
+ "name": "@galaxyproject/galaxy",
+ "version": "1.0.0",
+ "description": ".. figure:: https://galaxyproject.org/images/galaxy-logos/galaxy_project_logo.jpg :alt: Galaxy Logo",
+ "main": "index.js",
+ "devDependencies": {},
+ "scripts": {
+ "stage": "cpy './node_modules/@galaxyproject/galaxy-client/dist/*' './static/dist'"
+ },
+ "repository": {
+ "type": "git",
+ "url": "git+https://github.com/galaxyproject/galaxy.git"
+ },
+ "author": "Galaxy Contributors",
+ "license": "MIT",
+ "bugs": {
+ "url": "https://github.com/galaxyproject/galaxy/issues"
+ },
+ "homepage": "https://github.com/galaxyproject/galaxy#readme",
+ "dependencies": {
+ "@galaxyproject/galaxy-client": "^0.1.0",
+ "cpy-cli": "^4.2.0"
+ }
+}
diff --git a/pyproject.toml b/pyproject.toml
index b1e7c1f6e1a..d8b3484a6e4 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -30,6 +30,7 @@ url = "https://wheels.galaxyproject.org/simple"
a2wsgi = "*"
aiofiles = "*"
alembic = "*"
+alembic_utils = "*"
apispec = "*"
Babel = "*"
bdbag = ">=1.6.3"
diff --git a/scripts/common_startup.sh b/scripts/common_startup.sh
index a2716829aa3..245421136da 100755
--- a/scripts/common_startup.sh
+++ b/scripts/common_startup.sh
@@ -17,6 +17,8 @@ FETCH_WHEELS=1
CREATE_VENV=1
COPY_SAMPLE_FILES=1
SKIP_CLIENT_BUILD=${GALAXY_SKIP_CLIENT_BUILD:-0}
+SKIP_NODE=${GALAXY_SKIP_NODE:-0}
+INSTALL_PREBUILT_CLIENT=${GALAXY_INSTALL_PREBUILT_CLIENT:-0}
NODE_VERSION=${GALAXY_NODE_VERSION:-"$(cat client/.node_version)"}
YARN_INSTALL_OPTS=${YARN_INSTALL_OPTS:-"--network-timeout 300000 --check-files"}
@@ -28,6 +30,7 @@ for arg in "$@"; do
[ "$arg" = "--stop-daemon" ] && FETCH_WHEELS=0
[ "$arg" = "--skip-samples" ] && COPY_SAMPLE_FILES=0
[ "$arg" = "--skip-client-build" ] && SKIP_CLIENT_BUILD=1
+ [ "$arg" = "--skip-node" ] && SKIP_NODE=1
done
SAMPLES="
@@ -197,19 +200,21 @@ if [ $FETCH_WHEELS -eq 1 ]; then
fi
# Install node if not installed
-if [ -n "$VIRTUAL_ENV" ]; then
- if ! in_venv "$(command -v node)" || [ "$(node --version)" != "v${NODE_VERSION}" ]; then
- echo "Installing node into $VIRTUAL_ENV with nodeenv."
- if [ -d "${VIRTUAL_ENV}/lib/node_modules" ]; then
- echo "Removing old ${VIRTUAL_ENV}/lib/node_modules directory."
- rm -rf "${VIRTUAL_ENV}/lib/node_modules"
+if [ $SKIP_NODE -eq 0 ]; then
+ if [ -n "$VIRTUAL_ENV" ]; then
+ if ! in_venv "$(command -v node)" || [ "$(node --version)" != "v${NODE_VERSION}" ]; then
+ echo "Installing node into $VIRTUAL_ENV with nodeenv."
+ if [ -d "${VIRTUAL_ENV}/lib/node_modules" ]; then
+ echo "Removing old ${VIRTUAL_ENV}/lib/node_modules directory."
+ rm -rf "${VIRTUAL_ENV}/lib/node_modules"
+ fi
+ nodeenv -n "$NODE_VERSION" -p
+ fi
+ elif [ -n "$CONDA_DEFAULT_ENV" ] && [ -n "$CONDA_EXE" ]; then
+ if ! in_conda_env "$(command -v node)"; then
+ echo "Installing node into '$CONDA_DEFAULT_ENV' Conda environment with conda."
+ $CONDA_EXE install --yes --override-channels --channel conda-forge --name "$CONDA_DEFAULT_ENV" nodejs="$NODE_VERSION"
fi
- nodeenv -n "$NODE_VERSION" -p
- fi
-elif [ -n "$CONDA_DEFAULT_ENV" ] && [ -n "$CONDA_EXE" ]; then
- if ! in_conda_env "$(command -v node)"; then
- echo "Installing node into '$CONDA_DEFAULT_ENV' Conda environment with conda."
- $CONDA_EXE install --yes --override-channels --channel conda-forge --name "$CONDA_DEFAULT_ENV" nodejs="$NODE_VERSION"
fi
fi
@@ -257,16 +262,32 @@ if [ $SKIP_CLIENT_BUILD -eq 0 ]; then
set_galaxy_config_file_var
# Set plugin path
GALAXY_PLUGIN_PATH=$(python scripts/config_parse.py --setting=plugin_path --config-file="$GALAXY_CONFIG_FILE")
- # Build client
- cd client
- if yarn install $YARN_INSTALL_OPTS; then
- if ! (export GALAXY_PLUGIN_PATH="$GALAXY_PLUGIN_PATH"; yarn run build-production-maps;) then
- echo "ERROR: Galaxy client build failed. See ./client/README.md for more information, including how to get help."
+
+ if [ $INSTALL_PREBUILT_CLIENT -eq 0 ]; then
+ # If we have not opted to use a prebuilt client, then build client.
+ cd client
+ if yarn install $YARN_INSTALL_OPTS; then
+ if ! (export GALAXY_PLUGIN_PATH="$GALAXY_PLUGIN_PATH"; yarn run build-production-maps;) then
+ echo "ERROR: Galaxy client build failed. See ./client/README.md for more information, including how to get help."
+ exit 1
+ fi
+ else
+ echo "ERROR: Galaxy client dependency installation failed. See ./client/README.md for more information, including how to get help."
exit 1
fi
+ cd -
else
- echo "ERROR: Galaxy client dependency installation failed. See ./client/README.md for more information, including how to get help."
- exit 1
+ # Install prebuilt client
+ if yarn install $YARN_INSTALL_OPTS; then
+ if ! (yarn run stage) then
+ echo "ERROR: Galaxy prebuilt client install failed. See ./client/README.md for more information, including how to get help."
+ exit 1
+ fi
+ else
+ echo "ERROR: Galaxy prebuilt client dependency installation failed. See ./client/README.md for more information, including how to get help."
+ exit 1
+ fi
+
fi
- cd -
+
fi
diff --git a/scripts/cwl_conformance_to_test_cases.py b/scripts/cwl_conformance_to_test_cases.py
index 44e6bb79b1e..ad63d88fbd4 100644
--- a/scripts/cwl_conformance_to_test_cases.py
+++ b/scripts/cwl_conformance_to_test_cases.py
@@ -44,8 +44,6 @@ RED_TESTS = {
"directory_secondaryfiles",
"docker_entrypoint",
"dockeroutputdir",
- "dynamic_initial_workdir",
- "dynamic_resreq_filesizes",
"dynamic_resreq_wf_optional_file_default",
"dynamic_resreq_wf_optional_file_wf_default",
"env_home_tmpdir",
@@ -102,7 +100,6 @@ RED_TESTS = {
"directory_secondaryfiles",
"docker_entrypoint",
"dockeroutputdir",
- "dynamic_resreq_filesizes",
"dynamic_resreq_wf_optional_file_default",
"dynamic_resreq_wf_optional_file_wf_default",
"embedded_subworkflow",
@@ -174,13 +171,13 @@ RED_TESTS = {
"step_input_default_value_overriden_2nd_step_noexp",
"step_input_default_value_overriden_noexp",
# not required
- "225",
- "226",
- "233",
- "235",
- "236",
- "306",
- "307",
+ "initial_work_dir_for_null_and_arrays",
+ "initial_work_dir_for_array_dirs",
+ "illegal_symlink",
+ "modify_file_content",
+ "modify_directory_content",
+ "cond-with-defaults-1",
+ "cond-with-defaults-2",
"all_non_null_all_null",
"all_non_null_all_null_nojs",
"all_non_null_multi_non_null",
@@ -209,7 +206,6 @@ RED_TESTS = {
"directory_secondaryfiles",
"docker_entrypoint",
"dockeroutputdir",
- "dynamic_resreq_filesizes",
"dynamic_resreq_wf_optional_file_default",
"dynamic_resreq_wf_optional_file_wf_default",
"embedded_subworkflow",
diff --git a/test/deployment/usegalaxystar.bash b/test/deployment/usegalaxystar.bash
new file mode 100644
index 00000000000..7d7711c8657
--- /dev/null
+++ b/test/deployment/usegalaxystar.bash
@@ -0,0 +1,110 @@
+#!/bin/bash
+
+# Once-a-release deployment tests for usegalaxy*
+# This script is designed for use with .github/workflows/deployment.yaml,
+# a manually triggered GitHub Actions workflow.
+# The variables this script consumes include:
+# - GALAXY_TEST_DEPLOYMENT_TARGET
+# target Galaxy deployment to test against - this should be
+# usegalaxymain or usegalaxytest currently.
+# - GALAXY_TEST_DEPLOYMENT_DEBUG
+# should be the string "true" to enable additional debugging
+# - GALAXY_TEST_USEGALAXYMAIN_USER_KEY
+# GALAXY_TEST_USEGALAXYTEST_USER_KEY
+# API key to select for given target.
+
+set -e
+
+echo "Deployment target is $GALAXY_TEST_DEPLOYMENT_TARGET"
+
+# Setup dependencies and activate resulting Python environment.
+./scripts/common_startup.sh --skip-client-build --skip-node
+. .venv/bin/activate
+
+if [ $GALAXY_TEST_DEPLOYMENT_DEBUG = "true" ];
+then
+ PYTEST_DEBUG_ARGS="-s -v"
+else
+ PYTEST_DEBUG_ARGS=""
+fi
+
+if [ "$GALAXY_TEST_DEPLOYMENT_TARGET" = "usegalaxymain" ];
+then
+ GALAXY_TEST_EXTERNAL="https://usegalaxy.org"
+ GALAXY_TEST_USER_API_KEY="$GALAXY_TEST_USEGALAXYMAIN_USER_KEY"
+ GALAXY_TEST_USER_EMAIL="$GALAXY_TEST_USEGALAXYMAIN_USER_EMAIL"
+ GALAXY_TEST_SELENIUM_USER_PASSWORD="$GALAXY_TEST_USEGALAXYMAIN_USER_PASSWORD"
+elif [ "$GALAXY_TEST_DEPLOYMENT_TARGET" = "usegalaxytest" ];
+then
+ GALAXY_TEST_EXTERNAL="https://test.galaxyproject.org"
+ GALAXY_TEST_USER_API_KEY="$GALAXY_TEST_USEGALAXYTEST_USER_KEY"
+ GALAXY_TEST_USER_EMAIL="$GALAXY_TEST_USEGALAXYTEST_USER_EMAIL"
+ GALAXY_TEST_SELENIUM_USER_PASSWORD="$GALAXY_TEST_USEGALAXYTEST_USER_PASSWORD"
+elif [ "$GALAXY_TEST_DEPLOYMENT_TARGET" = "usegalaxyeu" ];
+then
+ GALAXY_TEST_EXTERNAL="https://usegalaxy.eu"
+ GALAXY_TEST_USER_API_KEY="$GALAXY_TEST_USEGALAXYEU_USER_KEY"
+ GALAXY_TEST_USER_EMAIL="$GALAXY_TEST_USEGALAXYEU_USER_EMAIL"
+ GALAXY_TEST_SELENIUM_USER_PASSWORD="$GALAXY_TEST_USEGALAXYEU_USER_PASSWORD"
+else
+ echo "ERROR: Unknown deployment test target ${GALAXY_TEST_DEPLOYMENT_TARGET}"
+ exit 1
+fi
+export GALAXY_TEST_EXTERNAL
+export GALAXY_TEST_USER_API_KEY
+export GALAXY_TEST_USER_EMAIL
+GALAXY_TEST_SELENIUM_USER_EMAIL="$GALAXY_TEST_USER_EMAIL"
+export GALAXY_TEST_SELENIUM_USER_EMAIL
+export GALAXY_TEST_SELENIUM_USER_PASSWORD
+
+api_tests=(
+ lib/galaxy_test/api/test_datatypes.py
+ lib/galaxy_test/api/test_workflow_extraction.py
+ lib/galaxy_test/api/test_users.py
+ lib/galaxy_test/api/test_roles.py
+ lib/galaxy_test/api/test_authenticate.py
+ lib/galaxy_test/api/test_configuration.py
+ lib/galaxy_test/api/test_dataset_collections.py
+ lib/galaxy_test/api/test_display_applications.py
+ lib/galaxy_test/api/test_drs.py
+ lib/galaxy_test/api/test_folders.py
+ lib/galaxy_test/api/test_folder_contents.py
+ lib/galaxy_test/api/test_framework.py
+ lib/galaxy_test/api/test_histories.py
+ lib/galaxy_test/api/test_history_contents.py
+)
+selenium_tests=(
+ lib/galaxy_test/selenium/test_anon_history.py
+ lib/galaxy_test/selenium/test_collection_builders.py
+ lib/galaxy_test/selenium/test_dataset_metadata_download.py
+ lib/galaxy_test/selenium/test_collection_edit.py
+ lib/galaxy_test/selenium/test_history_copy_elements.py
+ lib/galaxy_test/selenium/test_history_dataset_state.py
+ lib/galaxy_test/selenium/test_history_multi_view.py
+)
+if [ "$GALAXY_TEST_DEPLOYMENT_TEST_TYPE" = "all" ];
+then
+ tests=("${api_tests[@]}" "${selenium_tests[@]}")
+elif [ "$GALAXY_TEST_DEPLOYMENT_TEST_TYPE" = "api" ];
+then
+ tests=("${api_tests[@]}")
+elif [ "$GALAXY_TEST_DEPLOYMENT_TEST_TYPE" = "selenium" ];
+then
+ tests=("${selenium_tests[@]}")
+else
+ echo "Unknown GALAXY_TEST_DEPLOYMENT_TEST_TYPE encountered ${GALAXY_TEST_DEPLOYMENT_TEST_TYPE}"
+ exit 1
+fi
+
+echo "Running tests: ${tests[@]}"
+# pytest markers to select against. In order to not cause an explosion of
+# semi-permanent artifacts that are difficult to manage tests which require
+# new libraries or users are disabled. Tests which require admin keys are
+# also disabled for security/simplicity reasons.
+PYTEST_MARKER_EXPRESSION="not requires_admin and not requires_new_library and not requires_new_user and not requires_celery"
+PYTEST_HTML_REPORT="deployment_tests.html"
+pytest \
+ $PYTEST_DEBUG_ARGS \
+ --html "$PYTEST_HTML_REPORT" \
+ -m "${PYTEST_MARKER_EXPRESSION}" \
+ "${tests[@]}"
diff --git a/test/integration/test_containerized_jobs.py b/test/integration/test_containerized_jobs.py
index a10bc084986..68178fbf053 100644
--- a/test/integration/test_containerized_jobs.py
+++ b/test/integration/test_containerized_jobs.py
@@ -24,20 +24,20 @@ class MulledJobTestCases:
def test_explicit(self, history_id: str) -> None:
self.dataset_populator.run_tool("mulled_example_explicit", {}, history_id)
- self.dataset_populator.wait_for_history(history_id, assert_ok=True)
- output = self.dataset_populator.get_history_dataset_content(history_id, timeout=EXTENDED_TIMEOUT)
+ self.dataset_populator.wait_for_history(history_id, assert_ok=True, timeout=EXTENDED_TIMEOUT)
+ output = self.dataset_populator.get_history_dataset_content(history_id)
assert "0.7.15-r1140" in output
def test_mulled_simple(self, history_id: str) -> None:
self.dataset_populator.run_tool("mulled_example_simple", {}, history_id)
- self.dataset_populator.wait_for_history(history_id, assert_ok=True)
- output = self.dataset_populator.get_history_dataset_content(history_id, timeout=EXTENDED_TIMEOUT)
+ self.dataset_populator.wait_for_history(history_id, assert_ok=True, timeout=EXTENDED_TIMEOUT)
+ output = self.dataset_populator.get_history_dataset_content(history_id)
assert "0.7.15-r1140" in output
def test_mulled_explicit_invalid_case(self, history_id: str) -> None:
self.dataset_populator.run_tool("mulled_example_invalid_case", {}, history_id)
- self.dataset_populator.wait_for_history(history_id, assert_ok=True)
- output = self.dataset_populator.get_history_dataset_content(history_id, timeout=EXTENDED_TIMEOUT)
+ self.dataset_populator.wait_for_history(history_id, assert_ok=True, timeout=EXTENDED_TIMEOUT)
+ output = self.dataset_populator.get_history_dataset_content(history_id)
assert "0.7.15-r1140" in output
diff --git a/test/integration/test_purge_datasets.py b/test/integration/test_purge_datasets.py
new file mode 100644
index 00000000000..726f698fd3d
--- /dev/null
+++ b/test/integration/test_purge_datasets.py
@@ -0,0 +1,84 @@
+import os
+from typing import (
+ Callable,
+ Optional,
+)
+
+from galaxy_test.base.populators import DatasetPopulator
+from galaxy_test.driver import integration_util
+
+
+class TestPurgeDatasetsIntegration(integration_util.IntegrationTestCase):
+
+ dataset_populator: DatasetPopulator
+
+ def setUp(self):
+ super().setUp()
+ self.dataset_populator = DatasetPopulator(self.galaxy_interactor)
+ self.history_id = self.dataset_populator.new_history()
+
+ @classmethod
+ def handle_galaxy_config_kwds(cls, config):
+ super().handle_galaxy_config_kwds(config)
+ config["allow_user_dataset_purge"] = True
+
+ def test_purge_dataset_batch_removes_underlying_dataset_from_disk(self):
+ self._expect_dataset_purged_on(self._purge_hda_using_batch)
+
+ def test_purge_history_content_bulk_removes_underlying_dataset_from_disk(self):
+ self._expect_dataset_purged_on(self._purge_hda_using_bulk)
+
+ def _expect_dataset_purged_on(self, purge_operation: Callable):
+ hda = self.dataset_populator.new_dataset(self.history_id, wait=True)
+ hda_id = hda["id"]
+
+ # Ensure dataset file exists on disk
+ dataset_file = self._get_underlying_dataset_on_disk(hda_id)
+ assert self._file_exists_on_disk(dataset_file)
+
+ # Purge dataset
+ purge_operation(hda_id)
+
+ # Ensure dataset is purged
+ self.dataset_populator.wait_for_purge(self.history_id, hda_id)
+
+ # Ensure dataset file is removed from disk after purge
+ assert not self._file_exists_on_disk(dataset_file)
+
+ def _purge_hda_using_batch(self, hda_id):
+ payload = {
+ "purge": True,
+ "datasets": [
+ {"id": hda_id, "src": "hda"},
+ ],
+ }
+ purge_response = self._delete("datasets", data=payload, json=True)
+ self._assert_status_code_is_ok(purge_response)
+ purge_result = purge_response.json()
+ assert purge_result["success_count"] == 1
+
+ def _purge_hda_using_bulk(self, hda_id):
+ payload = {
+ "operation": "purge",
+ "items": [
+ {
+ "id": hda_id,
+ "history_content_type": "dataset",
+ },
+ ],
+ }
+ purge_response = self._put(
+ f"histories/{self.history_id}/contents/bulk",
+ data=payload,
+ json=True,
+ )
+ self._assert_status_code_is_ok(purge_response)
+ purge_result = purge_response.json()
+ assert purge_result["success_count"] == 1
+
+ def _get_underlying_dataset_on_disk(self, hda_id: str) -> Optional[str]:
+ detailed_response = self._get(f"datasets/{hda_id}", admin=True).json()
+ return detailed_response.get("file_name")
+
+ def _file_exists_on_disk(self, filename: Optional[str]) -> bool:
+ return os.path.isfile(filename) if filename else False
diff --git a/test/unit/data/datatypes/converters/__init__.py b/test/unit/data/datatypes/converters/__init__.py
deleted file mode 100644
index e69de29bb2d..00000000000
diff --git a/test/unit/data/datatypes/converters/test_interval_to_tabix.py b/test/unit/data/datatypes/converters/test_interval_to_tabix.py
deleted file mode 100644
index b1ce9570d67..00000000000
--- a/test/unit/data/datatypes/converters/test_interval_to_tabix.py
+++ /dev/null
@@ -1,15 +0,0 @@
-import pysam
-
-from galaxy.datatypes.converters.interval_to_tabix_converter import to_tabix
-from ..util import (
- get_input_files,
- get_tmp_path,
-)
-
-
-def test_to_tabix():
- with get_input_files("1.vcf") as input_files:
- with get_tmp_path(suffix=".tbi") as index:
- bgzip_fname = to_tabix(input_files[0], index, preset="vcf")
- f = pysam.TabixFile(bgzip_fname, index=index)
- f.close()
diff --git a/test/unit/util/test_bool_expressions.py b/test/unit/util/test_bool_expressions.py
index 58e9766a679..1c8596ca528 100644
--- a/test/unit/util/test_bool_expressions.py
+++ b/test/unit/util/test_bool_expressions.py
@@ -1,4 +1,5 @@
import pytest
+from pyparsing.exceptions import ParseException
from galaxy.util.bool_expressions import (
BooleanExpressionEvaluator,
@@ -61,7 +62,7 @@ def test_expression_evaluates_as_expected(expr: str, expected: bool, contained_e
@pytest.mark.parametrize("expr", INVALID_EXPRESSIONS_TESTS)
def test_invalid_expression_raises_exception(expr: str, contained_evaluator: BooleanExpressionEvaluator):
- with pytest.raises(Exception):
+ with pytest.raises(ParseException):
contained_evaluator.evaluate_expression(expr)
diff --git a/test/unit/util/test_sqlite_utils.py b/test/unit/util/test_sqlite_utils.py
index 8e627614316..758733579c5 100644
--- a/test/unit/util/test_sqlite_utils.py
+++ b/test/unit/util/test_sqlite_utils.py
@@ -21,17 +21,19 @@ def test_sqlite_exploits():
connection = sqlite.connect(":memory:")
connection.execute("create TABLE FOO (foo1 text)")
__assert_has_n_rows(connection, "select * from FOO", 0)
- __assert_query_errors(connection, "select * from FOOX")
+ __assert_query_errors(connection, "select * from FOOX", "no such table")
# Make sure sqlite query cannot execute multiple statements
- __assert_query_errors(connection, "select * from FOO; select * from FOO")
+ __assert_query_errors(
+ connection, "select * from FOO; select * from FOO", "You can only execute one statement at a time."
+ )
# Make sure sqlite cannot select on PRAGMA results
- __assert_query_errors(connection, "select * from (PRAGMA database_list)")
+ __assert_query_errors(connection, "select * from (PRAGMA database_list)", "no such table: PRAGMA")
__assert_has_n_rows(connection, "select * from FOO where foo1 in (SELECT foo1 from FOO)", 0)
# Ensure nested queries cannot modify database.
- __assert_query_errors(connection, "select * from FOO where foo1 in (INSERT INTO FOO VALUES ('bar')")
+ __assert_query_errors(connection, "select * from FOO where foo1 in (INSERT INTO FOO VALUES ('bar')", "syntax error")
# Should access to the schema be disallowed?
# __assert_has_n_rows(connection, "select * from SQLITE_MASTER", 0)
@@ -44,8 +46,8 @@ def __assert_has_n_rows(connection, query, n):
assert count == n
-def __assert_query_errors(connection, query):
- with pytest.raises(Exception):
+def __assert_query_errors(connection, query, match):
+ with pytest.raises(Exception, match=match):
connection.cursor().execute(query)
diff --git a/tools/interactive/interactivetool_ml_jupyter_notebook.xml b/tools/interactive/interactivetool_ml_jupyter_notebook.xml
index f11cec41589..5d19185d4ef 100644
--- a/tools/interactive/interactivetool_ml_jupyter_notebook.xml
+++ b/tools/interactive/interactivetool_ml_jupyter_notebook.xml
@@ -1,6 +1,6 @@
- 0.1
+ 0.2
docker.io/anupkumar/docker-ml-jupyterlab:galaxy-integration-@VERSION@
@@ -19,71 +19,221 @@
true
+
+
+
+
+
+
+
-
-
-
-
+
+
+
-
-
+
+
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
-
+
+
+
@@ -97,10 +247,9 @@
JupyterLab is a next-generation web-based user interface for Project Jupyter. JupyterLab enables you to work with documents and activities such as Jupyter notebooks, text editors,
terminals, and custom components in a flexible, integrated, and extensible manner.
- Galaxy offers you to use Jupyter Lab directly in Galaxy accessing and interacting with Galaxy datasets as you like. A very common use-case is to
+ Galaxy offers you to use JupyterLab directly in Galaxy accessing and interacting with Galaxy datasets as you like. A very common use-case is to
do the heavy lifting such as performing computation on GPUs and data reduction steps in Galaxy and the plotting and more `interactive` part on smaller datasets in Jupyter Lab.
- You can start with a new Jupyter Lab notebook from scratch and wait until the job starts running. Running job will provide you a link which can be opened in the same or another
- browser tab. This link opens Jupyter Lab notebook which can be used to prototype machine learning solutions.
+ You can start with a new JupyterLab notebook from scratch and wait until the job starts running. Running job will provide you a link which can be opened in the same or another browser tab. This link opens JupyterLab notebook which can be used to prototype machine learning solutions.
]]>
- A {{variant}} message
+ A {{ variant }} message
```
diff --git a/client/src/components/Dataset/DatasetName.vue b/client/src/components/Dataset/DatasetName.vue
index 1625cd497ac..9751155ea3d 100644
--- a/client/src/components/Dataset/DatasetName.vue
+++ b/client/src/components/Dataset/DatasetName.vue
@@ -22,11 +22,11 @@
diff --git a/client/src/components/DatasetInformation/DatasetAttributes.vue b/client/src/components/DatasetInformation/DatasetAttributes.vue
index dd942966dd0..73874c5f601 100644
--- a/client/src/components/DatasetInformation/DatasetAttributes.vue
+++ b/client/src/components/DatasetInformation/DatasetAttributes.vue
@@ -20,12 +20,16 @@
@click="submit('attribute', 'attributes')">
+
+
diff --git a/client/src/components/Form/FormElement.vue b/client/src/components/Form/FormElement.vue
index 9e74614f742..ecc6fa340a7 100644
--- a/client/src/components/Form/FormElement.vue
+++ b/client/src/components/Form/FormElement.vue
@@ -10,6 +10,7 @@ import FormNumber from "./Elements/FormNumber.vue";
import FormText from "./Elements/FormText.vue";
import FormOptionalText from "./Elements/FormOptionalText.vue";
import FormRulesEdit from "./Elements/FormRulesEdit.vue";
+import FormUpload from "./Elements/FormUpload.vue";
import { FontAwesomeIcon } from "@fortawesome/vue-fontawesome";
import { ref, computed, useAttrs } from "vue";
import { library } from "@fortawesome/fontawesome-svg-core";
@@ -265,6 +266,7 @@ const isOptional = computed(() => !isRequired.value && attrs.value["optional"] !
:multiple="attrs.multiple" />
+
+
+
-
-
-
diff --git a/client/src/components/History/Content/GenericItem.vue b/client/src/components/History/Content/GenericItem.vue
index 4b93fe579b0..adda51b27de 100644
--- a/client/src/components/History/Content/GenericItem.vue
+++ b/client/src/components/History/Content/GenericItem.vue
@@ -15,9 +15,7 @@
@undelete="onUndelete(item)"
@unhide="onUnhide(item)" />
+
+
+
-
diff --git a/client/src/components/History/Content/model/states.js b/client/src/components/History/Content/model/states.js
index 150d990b228..f37800d8282 100644
--- a/client/src/components/History/Content/model/states.js
+++ b/client/src/components/History/Content/model/states.js
@@ -33,6 +33,7 @@ export const STATES = {
/** metadata discovery/setting failed or errored (but otherwise ok) */
failed_metadata: {
status: "danger",
+ text: "Metadata generation failed. Please retry.",
icon: "exclamation-triangle",
},
/** was created without a tool */
diff --git a/client/src/components/History/CurrentHistory/HistoryNavigation.test.js b/client/src/components/History/CurrentHistory/HistoryNavigation.test.js
index e0486579328..3fffa8e50ec 100644
--- a/client/src/components/History/CurrentHistory/HistoryNavigation.test.js
+++ b/client/src/components/History/CurrentHistory/HistoryNavigation.test.js
@@ -56,6 +56,11 @@ describe("History Navigation", () => {
provide: { store },
});
+ const createButton = wrapper.find("*[data-description='create new history']");
+ expect(createButton.attributes().disabled).toBeFalsy();
+ const switchButton = wrapper.find("*[data-description='switch to another history']");
+ expect(switchButton.attributes().disabled).toBeFalsy();
+
const dropDown = wrapper.find("*[data-description='history options']");
const optionElements = dropDown.findAll("b-dropdown-item-stub");
const optionTexts = optionElements.wrappers.map((el) => el.text());
@@ -76,8 +81,12 @@ describe("History Navigation", () => {
provide: { store },
});
- const dropDown = wrapper.find("*[data-description='history options']");
+ const createButton = wrapper.find("*[data-description='create new history']");
+ expect(createButton.attributes().disabled).toBeTruthy();
+ const switchButton = wrapper.find("*[data-description='switch to another history']");
+ expect(switchButton.attributes().disabled).toBeTruthy();
+ const dropDown = wrapper.find("*[data-description='history options']");
const enabledOptionElements = dropDown.findAll("b-dropdown-item-stub:not([disabled])");
const enabledOptionTexts = enabledOptionElements.wrappers.map((el) => el.text());
expect(enabledOptionTexts).toStrictEqual(anonymousOptions);
diff --git a/client/src/components/History/CurrentHistory/HistoryNavigation.vue b/client/src/components/History/CurrentHistory/HistoryNavigation.vue
index 9d73f41cf6a..adc4d45a35e 100644
--- a/client/src/components/History/CurrentHistory/HistoryNavigation.vue
+++ b/client/src/components/History/CurrentHistory/HistoryNavigation.vue
@@ -12,7 +12,8 @@
data-description="create new history"
size="sm"
variant="link"
- title="Create new history"
+ :disabled="currentUser.isAnonymous"
+ :title="userTitle('Create new history')"
@click="$emit('createNewHistory')">
-
+
@@ -16,7 +18,9 @@
-
diff --git a/client/src/components/LoadingSpan.vue b/client/src/components/LoadingSpan.vue
index 85e79a25c7e..c4c3b9d26b5 100644
--- a/client/src/components/LoadingSpan.vue
+++ b/client/src/components/LoadingSpan.vue
@@ -1,7 +1,7 @@
-
-
+
+
+
+
diff --git a/client/src/components/User/UserPreferences.vue b/client/src/components/User/UserPreferences.vue
index 8a1e02dec11..09afa1e0532 100644
--- a/client/src/components/User/UserPreferences.vue
+++ b/client/src/components/User/UserPreferences.vue
@@ -82,6 +82,11 @@
description="Click here to make all data private."
@click="makeDataPrivate" />
+
+ >
+
+
-
+
-
{{ error }}
|
- |
- {{ plugin.html }}
- {{ plugin.description }}
+ {{ plugin.html }}
+ {{ plugin.description }}
|
| %s | %s | " % tuple(html.escape(x) for x in item) @@ -59,7 +59,7 @@ class _QIIME2ResultBase(CompressedZipArchive): return "".join(table) - def _peek(self, dataset: "DatasetInstance", simple: bool = False) -> List: + def _peek(self, dataset: HasMetadata, simple: bool = False) -> List: peek = [("Type", dataset.metadata.semantic_type), ("UUID", dataset.metadata.uuid)] if not simple: if dataset.metadata.semantic_type != "Visualization": @@ -115,7 +115,7 @@ class QIIME2Metadata(Tabular): def get_column_names(self, first_line: str) -> Optional[List[str]]: return first_line.strip().split("\t") - def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None: + def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None: """ Let Galaxy's Tabular format handle most of this. We will just jump in at the last minute to (potentially) override some column types. diff --git a/lib/galaxy/datatypes/qualityscore.py b/lib/galaxy/datatypes/qualityscore.py index 49ddc9c3eff..8349fe9a070 100644 --- a/lib/galaxy/datatypes/qualityscore.py +++ b/lib/galaxy/datatypes/qualityscore.py @@ -2,17 +2,14 @@ Qualityscore class """ import logging -from typing import TYPE_CHECKING +from galaxy.datatypes._protocols import DatasetProtocol from galaxy.datatypes.sniff import ( build_sniff_from_prefix, FilePrefix, ) from . import data -if TYPE_CHECKING: - from galaxy.model import DatasetInstance - log = logging.getLogger(__name__) @@ -69,7 +66,7 @@ class QualityScoreSOLiD(QualityScore): return False return goodblock > 0 - def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None: + def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None: if self.max_optional_metadata_filesize >= 0 and dataset.get_size() > self.max_optional_metadata_filesize: dataset.metadata.data_lines = None return diff --git a/lib/galaxy/datatypes/registry.py b/lib/galaxy/datatypes/registry.py index 746761d9a74..b6bc708cfc2 100644 --- a/lib/galaxy/datatypes/registry.py +++ b/lib/galaxy/datatypes/registry.py @@ -9,16 +9,18 @@ import pkgutil from pathlib import Path from string import Template from typing import ( + cast, Dict, List, Optional, Tuple, - TYPE_CHECKING, + Union, ) import yaml import galaxy.util +from galaxy.datatypes._protocols import DatasetProtocol from galaxy.tool_util.edam_util import load_edam_tree from galaxy.util import RW_R__R__ from galaxy.util.bunch import Bunch @@ -37,9 +39,6 @@ from . import ( ) from .display_applications.application import DisplayApplication -if TYPE_CHECKING: - from galaxy.model import DatasetInstance - class ConfigurationError(Exception): pass @@ -832,8 +831,8 @@ class Registry: return None def find_conversion_destination_for_dataset_by_extensions( - self, dataset_or_ext, accepted_formats: List[str], converter_safe: bool = True - ) -> Tuple[bool, Optional[str], Optional["DatasetInstance"]]: + self, dataset_or_ext: Union[str, DatasetProtocol], accepted_formats: List[str], converter_safe: bool = True + ) -> Tuple[bool, Optional[str], Optional[DatasetProtocol]]: """ returns (direct_match, converted_ext, converted_dataset) - direct match is True iff no the data set already has an accepted format @@ -841,7 +840,7 @@ class Registry: """ if hasattr(dataset_or_ext, "ext"): ext = dataset_or_ext.ext - dataset = dataset_or_ext + dataset = cast(DatasetProtocol, dataset_or_ext) else: ext = dataset_or_ext dataset = None @@ -854,7 +853,7 @@ class Registry: convert_ext_datatype = self.get_datatype_by_extension(convert_ext) if convert_ext_datatype is None: self.log.warning( - f"Datatype class not found for extension '{convert_ext}', which is used as target for conversion from datatype '{dataset.ext}'" + f"Datatype class not found for extension '{convert_ext}', which is used as target for conversion from datatype '{ext}'" ) elif convert_ext_datatype.matches_any(accepted_formats): converted_dataset = dataset and dataset.get_converted_files_by_type(convert_ext) diff --git a/lib/galaxy/datatypes/sequence.py b/lib/galaxy/datatypes/sequence.py index 040d68ede75..7eef083dfcb 100644 --- a/lib/galaxy/datatypes/sequence.py +++ b/lib/galaxy/datatypes/sequence.py @@ -17,7 +17,6 @@ from typing import ( Iterable, List, Optional, - TYPE_CHECKING, ) import bx.align.maf @@ -25,6 +24,11 @@ from markupsafe import escape from galaxy import util from galaxy.datatypes import metadata +from galaxy.datatypes._protocols import ( + DatasetHasHidProtocol, + DatasetProtocol, + HasMetadata, +) from galaxy.datatypes.binary import Binary from galaxy.datatypes.data import DatatypeValidation from galaxy.datatypes.metadata import ( @@ -45,12 +49,6 @@ from galaxy.util.checkers import is_gzip from galaxy.util.image_util import check_image_type from . import data -if TYPE_CHECKING: - from galaxy.model import ( - DatasetInstance, - HistoryDatasetAssociation, - ) - log = logging.getLogger(__name__) @@ -71,7 +69,7 @@ class SequenceSplitLocations(data.Text): file_ext = "fqtoc" - def set_peek(self, dataset: "DatasetInstance", **kwd) -> None: + def set_peek(self, dataset: DatasetProtocol, **kwd) -> None: if not dataset.dataset.purged: try: parsed_data = json.load(open(dataset.file_name)) @@ -108,7 +106,7 @@ class Sequence(data.Text): name="sequences", default=0, desc="Number of sequences", readonly=True, visible=False, optional=True, no_value=0 ) - def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None: + def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None: """ Set the number of sequences and the number of data lines in dataset. """ @@ -128,7 +126,7 @@ class Sequence(data.Text): dataset.metadata.data_lines = data_lines dataset.metadata.sequences = sequences - def set_peek(self, dataset: "DatasetInstance", **kwd) -> None: + def set_peek(self, dataset: DatasetProtocol, **kwd) -> None: if not dataset.dataset.purged: dataset.peek = data.get_file_peek(dataset.file_name) if dataset.metadata.sequences: @@ -564,7 +562,7 @@ class csFasta(Sequence): return False return False - def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None: + def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None: if self.max_optional_metadata_filesize >= 0 and dataset.get_size() > self.max_optional_metadata_filesize: dataset.metadata.data_lines = None dataset.metadata.sequences = None @@ -641,7 +639,7 @@ class Fastg(Sequence): break # we found a non-empty line, but it's not a header return False - def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None: + def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None: with open(dataset.file_name) as fh: for i, line in enumerate(fh): if not line: @@ -666,7 +664,7 @@ class Fastg(Sequence): return return Sequence.set_meta(self, dataset, overwrite=overwrite, **kwd) - def set_peek(self, dataset: "DatasetInstance", **kwd) -> None: + def set_peek(self, dataset: DatasetProtocol, **kwd) -> None: if not dataset.dataset.purged: dataset.peek = data.get_file_peek(dataset.file_name) if dataset.metadata.sequences: @@ -690,7 +688,7 @@ class BaseFastq(Sequence): file_ext = "fastq" bases_regexp = re.compile(r"^[NGTAC 0123\.]*$", re.IGNORECASE) - def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None: + def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None: """ Set the number of sequences and the number of data lines in dataset. @@ -771,7 +769,7 @@ class BaseFastq(Sequence): def display_data( self, trans, - dataset: "HistoryDatasetAssociation", + dataset: DatasetHasHidProtocol, preview: bool = False, filename: Optional[str] = None, to_ext: Optional[str] = None, @@ -874,7 +872,7 @@ class BaseFastq(Sequence): return True return False - def validate(self, dataset: "DatasetInstance", **kwd) -> DatatypeValidation: + def validate(self, dataset: DatasetProtocol, **kwd) -> DatatypeValidation: headers = iter_headers(dataset.file_name, sep="\n", count=-1) # check to see if the base qualities match if not self.quality_check(headers): @@ -1011,11 +1009,11 @@ class Maf(Alignment): optional=True, ) - def init_meta(self, dataset: "DatasetInstance", copy_from: Optional["DatasetInstance"] = None) -> None: + def init_meta(self, dataset: HasMetadata, copy_from: Optional[HasMetadata] = None) -> None: Alignment.init_meta(self, dataset, copy_from=copy_from) def set_meta( - self, dataset: "DatasetInstance", overwrite: bool = True, metadata_tmp_files_dir: Optional[str] = None, **kwd + self, dataset: DatasetProtocol, overwrite: bool = True, metadata_tmp_files_dir: Optional[str] = None, **kwd ) -> None: """ Parses and sets species, chromosomes, index from MAF file. @@ -1049,7 +1047,7 @@ class Maf(Alignment): indexes.write(open(index_file.file_name, "wb")) dataset.metadata.maf_index = index_file - def set_peek(self, dataset: "DatasetInstance", **kwd) -> None: + def set_peek(self, dataset: DatasetProtocol, **kwd) -> None: if not dataset.dataset.purged: # The file must exist on disk for the get_file_peek() method dataset.peek = data.get_file_peek(dataset.file_name) @@ -1063,11 +1061,11 @@ class Maf(Alignment): dataset.peek = "file does not exist" dataset.blurb = "file purged from disk" - def display_peek(self, dataset: "DatasetInstance") -> str: + def display_peek(self, dataset: DatasetProtocol) -> str: """Returns formated html of peek""" return self.make_html_table(dataset) - def make_html_table(self, dataset: "DatasetInstance", skipchars: Optional[List] = None) -> str: + def make_html_table(self, dataset: DatasetProtocol, skipchars: Optional[List] = None) -> str: """Create HTML table, used for displaying peek""" skipchars = skipchars or [] try: @@ -1142,7 +1140,7 @@ class MafCustomTrack(data.Text): name="vp_end", default="100", desc="Viewport End", readonly=True, optional=True, visible=False, no_value="" ) - def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None: + def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None: """ Parses and sets viewport metadata from MAF file. """ @@ -1275,7 +1273,7 @@ class RNADotPlotMatrix(data.Data): edam_format = "format_3466" file_ext = "rna_eps" - def set_peek(self, dataset: "DatasetInstance", **kwd) -> None: + def set_peek(self, dataset: DatasetProtocol, **kwd) -> None: if not dataset.dataset.purged: dataset.peek = "RNA Dot Plot format (Postscript derivative)" dataset.blurb = nice_size(dataset.get_size()) @@ -1313,7 +1311,7 @@ class DotBracket(Sequence): sequence_regexp = re.compile(r"^[ACGTURYKMSWBDHVN]+$", re.I) structure_regexp = re.compile(r"^[\(\)\.\[\]{}]+$") - def set_meta(self, dataset: "DatasetInstance", overwrite: bool = True, **kwd) -> None: + def set_meta(self, dataset: DatasetProtocol, overwrite: bool = True, **kwd) -> None: """ Set the number of sequences and the number of data lines in dataset. diff --git a/lib/galaxy/datatypes/spaln.py b/lib/galaxy/datatypes/spaln.py index 6ce094e341d..1bc7f0fd950 100644 --- a/lib/galaxy/datatypes/spaln.py +++ b/lib/galaxy/datatypes/spaln.py @@ -9,22 +9,17 @@ from typing import ( Dict, List, Optional, - TYPE_CHECKING, ) -from galaxy.datatypes.data import ( - Data, - GeneratePrimaryFileDataset, +from galaxy.datatypes._protocols import ( + DatasetHasHidProtocol, + DatasetProtocol, + HasExtraFilesAndMetadata, ) +from galaxy.datatypes.data import Data from galaxy.datatypes.metadata import MetadataElement from galaxy.util import smart_str -if TYPE_CHECKING: - from galaxy.model import ( - DatasetInstance, - HistoryDatasetAssociation, - ) - log = logging.getLogger(__name__) verbose = True @@ -68,7 +63,7 @@ class _SpalnDb(Data): substitute_name_with_metadata="spalndb_name", ) - def generate_primary_file(self, dataset: GeneratePrimaryFileDataset) -> str: + def generate_primary_file(self, dataset: HasExtraFilesAndMetadata) -> str: rval = ["
|---|