760 Commits
Author SHA1 Message Date
John Chilton 1c4a4ed8cb Merge remote-tracking branch 'galaxyproject/release_18.05' into release_18.09 2018-10-04 11:43:05 -04:00
John Chilton 4a345b99c3 Fix security problems when extracting untrusted tar files.
Fixes GX-2018-0006.
2018-10-04 11:32:52 -04:00
mvdbeek becb922dc1 Strip all control characters from rules
Similar to 366701cfb2, which is valid on its own, but here we just apply it to all rules.
2018-09-27 16:36:31 +02:00
Martin Cech 3072889905 Merge branch 'release_18.05' into release_18.09 2018-09-20 11:48:45 -04:00
John Chilton 56f5bf743a Make newer unit tests more robust with a looped wait...
... instead of raw sleep.
2018-09-19 20:14:08 -04:00
mvdbeek 5635a05a2a Look up collection_type if collection_type_source is set
We do this in the wf-editor and in the backend, so this works for
subworkflows. This should make quite some tools more usable in a
workflow scenario. Should fix
- https://github.com/galaxyproject/galaxy/issues/6514
- https://github.com/galaxyproject/galaxy/issues/6012
- https://github.com/galaxyproject/galaxy/issues/6569
- https://github.com/galaxyproject/galaxy/issues/1889
and perhaps more issues related to this.
2018-09-19 09:48:25 -04:00
mvdbeek 94082a878d Unit test that fails if we don't return the latest version upon version mismatch 2018-09-19 15:39:00 +02:00
mvdbeek dc49c3ea0e Provide visual feedback if tool is broken 2018-09-08 17:59:45 +02:00
mvdbeek 947389887b Add unit test that exercises reloading for broken tools 2018-09-08 11:29:24 +02:00
John Chilton 7806a84d1c Implement runtime parameters in nested workflows. 2018-09-06 14:43:22 -04:00
John Chilton 8148bd91de Merge pull request #6638 from mvdbeek/move_tests_to_pytest_setup
Move tests to pytest setup
2018-09-05 06:50:34 -04:00
mvdbeek 366701cfb2 Strip unicode control characters from tags
Leaving them in can trip up postgresql:
```
Traceback (most recent call last):
  File "lib/galaxy/web/framework/decorators.py", line 283, in decorator
    rval = func(self, trans, *args, **kwargs)
  File "lib/galaxy/webapps/galaxy/api/history_contents.py", line 396, in create
    return self.__create_dataset_collection(trans, history, payload, **kwd)
  File "lib/galaxy/webapps/galaxy/api/history_contents.py", line 525, in __create_dataset_collection
    **create_params
  File "lib/galaxy/managers/collections.py", line 108, in create
    copy_elements=copy_elements,
  File "lib/galaxy/managers/collections.py", line 181, in create_dataset_collection
    copy_elements=copy_elements)
  File "lib/galaxy/managers/collections.py", line 217, in _element_identifiers_to_elements
    copy_elements=copy_elements)
  File "lib/galaxy/managers/collections.py", line 389, in __load_elements
    copy_elements=copy_elements)
  File "lib/galaxy/managers/collections.py", line 434, in __load_element
    self.tag_manager.apply_item_tags(user=trans.user, item=element, tags_str=tag_str)
  File "lib/galaxy/managers/tags.py", line 182, in apply_item_tags
    self.apply_item_tag(user, item, name, value)
  File "lib/galaxy/managers/tags.py", line 154, in apply_item_tag
    tag = self._get_or_create_tag(lc_name)
  File "lib/galaxy/managers/tags.py", line 233, in _get_or_create_tag
    tag = self.get_tag_by_name(scrubbed_tag_str)
  File "lib/galaxy/managers/tags.py", line 205, in get_tag_by_name
    return self.sa_session.query(galaxy.model.Tag).filter_by(name=tag_name.lower()).first()
  File "/bioinfo/guests/mvandenb/galaxy/.venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2789, in first
    ret = list(self[0:1])
  File "/bioinfo/guests/mvandenb/galaxy/.venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2581, in __getitem__
    return list(res)
  File "/bioinfo/guests/mvandenb/galaxy/.venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2889, in __iter__
    return self._execute_and_instances(context)
  File "/bioinfo/guests/mvandenb/galaxy/.venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2912, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/bioinfo/guests/mvandenb/galaxy/.venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/bioinfo/guests/mvandenb/galaxy/.venv/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/bioinfo/guests/mvandenb/galaxy/.venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/bioinfo/guests/mvandenb/galaxy/.venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/bioinfo/guests/mvandenb/galaxy/.venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1416, in _handle_dbapi_exception
    util.reraise(*exc_info)
  File "/bioinfo/guests/mvandenb/galaxy/.venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/bioinfo/guests/mvandenb/galaxy/.venv/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
ValueError: A string literal cannot contain NUL (0x00) characters.
```
2018-08-26 22:29:23 +02:00
mvdbeek 50a524496a Add unit test for copy_element option 2018-08-26 22:13:50 +02:00
mvdbeek 9409322a73 Revert "Adapt unit tests to copying HDAs when creating collections"
This reverts commit 408a003f171be73e4b347579ee6f662a18fc1990.
2018-08-26 22:13:50 +02:00
mvdbeek d4a80e61e1 Adapt unit tests to copying HDAs when creating collections 2018-08-26 22:13:50 +02:00
Marius van den Beek a3fc0b9f32 Merge pull request #6312 from jmchilton/tuple_operations
Lists-as-tuples collection operations
2018-08-26 09:19:49 +02:00
mvdbeek 66c64ccfe4 Adapt dynamic_tool_destination unit tests to pytest as driver
The log capturing seems to be a bit different
2018-08-25 14:18:13 +02:00
John Chilton 6067fee3d4 Remove unused dataset deserializer stuff.
An unfinished thread left in the code base. It was a fine idea but I don't think anyone is left to pick it up and run with it.
2018-08-10 19:39:26 -04:00
John Chilton 84597a1235 Lists-as-tuples collection operations. 2018-08-08 10:28:56 -04:00
John Chilton 55436c3800 Merge pull request #5457 from mvdbeek/select_tag_parameter
Allow selecting subset of collection by group tag
2018-08-07 13:09:18 -04:00
Dannon Baker d1423b6e18 Merge branch 'release_18.05' into dev 2018-08-06 20:20:02 -04:00
Dannon 561e0099bc Merge pull request #6550 from jmchilton/objectstore_setup
Configure object store for datasets in job handler instead of web thread.
2018-08-06 11:00:43 -04:00
mvdbeek 11f9222122 Fix redirection unit test 2018-08-02 13:21:19 +02:00
mvdbeek cb7ebd7349 py3: Fix history import/export tool
- Open history tarfile in binary mode
 - Fix base64 encoding in history import tool
 - test history export more extensively
2018-08-02 13:21:19 +02:00
John Chilton 1b5dff11a7 Setup objectstore for datasets in job handler instead of web thread.
This is good for a few reasons

* It speeds up the bottleneck that is the job creation code in the web thread (the job handler is much easier to parallelize). Skipping the flush for all those datasets and skipping the disk write per dataset should really speed things up nicely when submitting large numbers of jobs and jobs with a large number of inputs.
* This prevents a disk slow down from grinding the web response to a halt. We don't want the handler to freeze up either - but if something is going to freeze it is a better user experience and easier to recover from the job handler freezing while their are disk problems.
* This allows per-job-handler object store configurations - simply configure different job handlers with different object store configurations. This is cool and important and I can imagine some awesome applications but I will wait for them to materialize.
2018-07-30 12:03:44 -04:00
John Chilton 5f19b8752d Fix empty handling rule. 2018-07-26 09:52:07 -04:00
Dannon 4955add0a3 Merge pull request #6524 from jmchilton/cached_identity
Use cached database identity to avoid many DB requests when creating jobs.
2018-07-25 15:28:47 -04:00
Dannon Baker 43a0482f1c Minor (comment-only) wording tweaks. 2018-07-25 15:27:36 -04:00
John Chilton 45b2b7f434 Merge remote-tracking branch 'jmchilton/dev' into select_tag_parameter 2018-07-24 13:18:13 -04:00
John Chilton fe3e284bf9 Fix HDCA annotations.
Backend for these simply did not work.
2018-07-24 13:09:06 -04:00
John Chilton 63c2ffd3ba Implement simple history copy unit tests. 2018-07-24 13:09:06 -04:00
John Chilton dd6b6436c8 Use cached database identity to avoid certain DB requests when creating jobs.
When creating thousands of jobs for large database transactions - previously Galaxy would query the database thousands of times for the same history id, user id, and session id.
2018-07-23 11:30:29 -04:00
mvdbeek 59bd1ba20a Add unittest that catches https://github.com/galaxyproject/galaxy/issues/6495 2018-07-13 11:24:43 +02:00
John Chilton 0b98e3ed13 Merge pull request #6443 from ebolyen/multiple_filter_datasetmatcher
Allows DatasetMatcher to use multiple filters
2018-07-05 13:31:36 -04:00
mvdbeek 7e29d5d83b py3: fix paramiko shell runner by casting to unicode 2018-07-05 02:49:51 +01:00
mvdbeek 25b93c78b6 Fix CORS unit test 2018-07-05 02:49:51 +01:00
mvdbeek c03a86caea py3: fix test_job_param unit test 2018-07-05 02:49:51 +01:00
mvdbeek c52d147612 No need to explicitly compare to bytestring 2018-07-05 02:49:51 +01:00
mvdbeek af6b572eac py3: Fix history/import export 2018-07-05 02:49:51 +01:00
mvdbeek 57f5b42974 py3: fix workflow extract summary 2018-07-05 02:49:51 +01:00
John Chilton 0d59c9cd54 Refactor galaxy.jobs.error_level -> galaxy.tools.parser.error_level.
Needed after 46170e9bcdcc2eb104ebab72a7afc9b7238bc314 to keep galaxy-lib from importing things outside of galaxy-lib but clearly a good refactoring on its own since it eliminates mid-file imports that were there to prevent circular dependencies from loading Galaxy.
2018-07-04 11:57:51 +01:00
Nicola Soranzo 64c591a606 More proper fix for some test_tool_deps.py unit tests
Follow up to https://github.com/galaxyproject/galaxy/pull/6436/files#r199368686
2018-07-03 00:24:04 +01:00
Dannon 75511506a8 Merge pull request #6439 from mvdbeek/py3_db_interaction
Python3: Encode return values of JSON encoder
2018-07-02 10:10:10 -07:00
Evan Bolyen d145d9ac92 Allows DatasetMatcher to use multiple filters
In particular this enables multiple add_value filters to be used with
<options>'s option_filter_attribute.
2018-07-01 15:52:32 -07:00
mvdbeek 44300668d4 Fix registry import in test_galaxy_mapping.py 2018-07-01 14:44:52 -07:00
Bjoern Gruening 41f25b928a fix a few tool_deps test errors 2018-07-01 21:25:25 +02:00
Nicola Soranzo 29cf68f0b0 Do not encode a rendered Mako template
Fix 1 error and 2 failures in unit tests under Python 3, e.g.:

```
FAIL: test_render (unit.visualizations.plugins.test_VisualizationPlugin.VisualizationsPlugin_TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/galaxyproject/galaxy/test/unit/visualizations/plugins/test_VisualizationPlugin.py", line 180, in test_render
    self.assertIsInstance(response, string_types)
AssertionError: b'\nTrue\n' is not an instance of (<class 'str'>,)
```
2018-07-01 00:19:00 -07:00
Nic Herndon 03a4172680 Fixed bugs with unit test on Python 3 2018-06-30 23:06:42 +00:00
Nic Herndon d95c32f4b8 Fixed comparison with binary string 2018-06-30 16:28:43 +00:00
Nic Herndon a221a37af3 Fixed comparison with binary string 2018-06-30 16:28:43 +00:00