Commit Graph
30 Commits
Author SHA1 Message Date
John Chilton 37892dcabc Refactor sniff.stream_url_to_file to galaxy.files.uris.
The value of this can be seen for instance in unpack_tar_gz_archive - which should have nothing to do with datatypes and now doesn't need to load the whole datatypes module.

Use this in workflows API to allow downloading workflow files from configured URIs.
2022-05-21 17:22:41 -04:00
Dannon Baker 03351d2a9f Apply black to tools as well 2022-02-03 09:12:13 -05:00
Dannon Baker 623a5e165a Apply black formatting.
Apply isort.
2022-02-03 07:42:20 -05:00
mvdbeek 4dabdf4282 Add socket timeout for urlopen 2021-06-20 14:04:23 +02:00
Nicola Soranzo 9d74bba7fb Drop support for retired Python 3.5
Upgrade syntax using `pyupgrade --py36-plus` .

Manually drop several `six` imports.

Also:
- Remove broken pr_cache in scripts/bootstrap_history.py
- Fix broken prefix removal in lib/galaxy/tool_util/deps/mulled/mulled_build.py
2020-10-07 11:52:13 +01:00
mvdbeek 8d99791afb Fix data_source tools on python 3 2019-01-18 12:13:13 +01:00
Nicola Soranzo 0e0acb5200 Remove is_multi_byte from upload.py
Also, when sniffing binary files, sniff images together with the other
formats and respect sniff order.

Also remove `is_multi_byte` from:
- `stream_to_open_named_file()`
- `stream_to_file()`

Remove the now unused `get_image_ext()` and `Binary.is_sniffable_binary()`
and all the calls to `Binary.register_sniffable_binary_format()`.
2017-11-30 18:06:37 +00:00
Nicola Soranzo 59f44cf2be Update get_fileobj() to use utf-8 encoding in text mode
Also, merge its 3 parameters `gzip_only`, `bz2_only`, `zip_only` into
`compressed_formats` (a list of allowed formats).

As a consequence of the changes in `get_fileobj()`, update:
- `files_diff()`
- `get_file_peek()`, which now determines that a file is binary when a
  `UnicodeDecodeError` exception is raised and doesn't need
  `is_multi_byte` any more
- `iter_headers()` and `get_headers()`, which now return Unicode and don't
  need `is_multi_byte` parameter any more

As a consequence of the changes in `get_file_peek()`, update:
- `set_peek()`, which now doesn't need `is_multi_byte` any more

As a consequence of the changes in `get_headers()`, update:
- `guess_ext` and `is_column_based()`, which now determine that a file is
  binary when a `UnicodeDecodeError` exception is raised and don't need
  `is_multi_byte` any more

As a consequence of the changes to `guess_ext`, update:
- `handle_uploaded_dataset_file() doesn't need `is_multi_byte` any more

Also, remove duplicated calls to `get_file_peek()` in
lib/galaxy/datatypes/molecules.py and lib/galaxy/datatypes/msa.py

The `is_multi_byte` was not removed from the signature of `get_file_peek()`
and `set_peek()` in order to preserve compatibility for ToolShed datatypes,
thanks @jmchilton for the review.
2017-11-28 17:46:32 +00:00
Nicola Soranzo 9b4de72ca6 Fix all E722 errors and ignore E741
Introduced in flake8 3.5.0
Fix import order.
2017-10-24 11:29:09 +01:00
Nate Coraor 3a70d0c648 [GX-2017-0003]: Fix for the reported issue, only allow http, https, and
ftp schemes in the data_source tool.
2017-10-23 10:25:22 -04:00
Nicola Soranzo a353b2e88f Fix E127 errors introduced by previous commit
using:
```
autopep8 -i -r --exclude $(sed -e 's|^|./|' -e 's|/$||' .ci/flake8_blacklist.txt | paste -sd,) --select E127 .
```
2017-08-17 11:35:39 +01:00
Nicola Soranzo 21b44bf348 Fix all E201 and E202 style errors
using the following command:
```
autopep8 -i -r --exclude $(sed -e 's|^|./|' -e 's|/$||' .ci/flake8_blacklist.txt | paste -sd,) --select E201,E202 .
```
2017-08-17 11:35:39 +01:00
Nicola Soranzo 1d17a29c79 Fixes for pycodestyle 2.2.0 2016-11-15 16:56:50 +00:00
Nicola Soranzo 11b4f3a60c Fix import order and Python3 compatibility for tools/
xref #1715
2016-09-29 19:26:22 +01:00
Nicola Soranzo cc336f61c0 Make some files compatible with Python3
Also enlarge the set of files tested with flake8 under Python3
2016-06-03 19:24:46 +01:00
Nicola Soranzo 8a43b2b492 Python 3: use "as" instead of comma in except clause 2016-05-17 23:53:16 +01:00
John Chilton 499357e508 Remove unneeded galaxy.model import in upload.py...
... and document the ability to remove from other data sources.
2016-01-21 14:31:42 +00:00
Nicola Soranzo 69cda48690 Remove remaining references to galaxy.eggs . flake8 some files in tools/ . 2015-10-26 19:29:13 +00:00
Nicola Soranzo 34d9388456 Use os.pardir everywhere instead of '..' . 2015-07-14 16:20:29 +01:00
Dannon Baker 9d85ca9e23 Switch to_json_string/from_json_string in galaxy/tools. 2014-09-09 10:06:35 -04:00
Daniel Blankenberg d84ab3226e Handle non-ascii unicode in data source tools. Add util.is_binary() method that returns true when provided string contains a null byte. 2012-08-28 19:23:39 -04:00
Greg Von Kuster 17c5b27cdc Re-engineer the datatypes registry so that it is initialized once when the Galaxy server is started, but data types can continue to be loaded throughout the Galaxy server's session (hopefully this doesn't break anything).
Add support for a single "import_module" to be passed to the new load_datatypes() method in the datatypes registry.  This provides the ability to load a single class module from an installed tool shed repository along with a datatypes_conf.xml file included in the installed repository and pass them to the new load_datatypes() method.  In the future, multiple imported modules may be allowed.  The datatypes_conf.xml file included in the repository must conform to a slightly different definition than the same named file that comes with the distribution.  This new definition will be documented in the Galaxy tool shed wiki.

We now have the ability to load new data types into the Galaxy server from an installed tool shed repository without restarting the Galaxy server.
2011-11-23 16:16:15 -05:00
Daniel Blankenberg 694300ce97 Some reworking of data_source tools and the standard data_source.py script. Remove hard-coded special-case handling of UCSC Table Browser and GBrowse datasource tools; functionality remains, but is now a part of the individual tools' XML configuration files. Auto-detect is now available by providing data_type=auto parameters. 2011-04-21 13:12:10 -04:00
Greg Von Kuster 08e0d5e532 Fixes for Bam data type's set_meta() and sam_to_bam tool, indexes will now be properly created for bam datasets. Fixes also for uploading Bam files, they will no longer be uncompressed on upload. 2009-12-04 19:51:46 -05:00
Daniel Blankenberg b2e2b801ea Enhance data_source.py to take advantage of the content length when known. If content length provided by external application is
greater than config.output_size_limit, no data will be retrieved and an error message will be provided to the user.

Tools using this script have been updated to provide the max file size on the command line.

Resolves ticket #93.
2009-08-03 12:02:54 -04:00
Greg Von Kuster a9cd544be9 Add a defaulttimeout setting of 10 minutes to the urlopen() call for requests to remote data sources. 2009-05-28 11:10:44 -04:00
Greg Von Kuster 2647f72e4a Add ability for data_source tools to append parameters passed in the initial response to the value of URL prior to Galaxy's post to the URL. This is a cleaner method for Biomart and also gets GBrowse to wrok. 2008-12-13 23:35:29 -05:00
Greg Von Kuster b74f176613 Add a new URL_method attribute to data_source tool types whose value is either "get" or "post" ( some require a get request while others require a post request ). This fixes the Biomart problem ( along with a new, well documented hack that can be eliminated when Biomart encodes the value of URL in the initial response - they'll tell us when they've fixed this ). Also added some requested info to the "send to EpiGRAPH" tool. 2008-10-27 16:03:43 -04:00
Daniel Blankenberg b9e242e4eb Add a new metadata type of Metadata Files.
These are now used to store the list of chromosomes for species as well as the index for MAF files.

MAF tools have been enhanced to make use of index files when available.

TODO: When datasets are purged from disk, these files should also be purged.
2008-10-22 13:49:22 -04:00
Greg Von Kuster b3b6c54247 Use only 1 underlying executable ( data_source.py ) for data source tools. A new tag set is added to the data source tool configs to handle tranlsation of request param names sent by remote apps ( something like <param_trans galaxy_name="dbkey" remote_name="GENOME" missing="?" /> ). 2008-10-07 15:21:46 -04:00