There seems to be no UnvalidatedValue class anymore. So I removed these
checks which seems to make the tool functional again.
Note that there is one more instance of such a check in the Galaxy
sources (in tools/parameters/basic.py). I guess this can also be
removed?
I like this better for three reasons:
- Since usually it is scripts producing this JSON - we have the most control at that point for determining the failure and we don't have to deal with an artificial dependency between the tool's stdio and the output.
- At some point we could potentially allow some datasets to be ok now even though the job fails.
- It is a cleaner interface at the Python level between job finish and output collection IMO (no need for isinstance checking).
I think setting each of these variables once and simplifing the context they are used in (in the case of purge_upload) makes it more clear what each variable is and how it is set. I also think one, more detailed comment for each variable helps.
Note: This will break run-as-user uploads started prior to the upgrade to 18.XX and executed after the upgrade. It is a small switch to restore the old behavior but I'm not sure it is worth the complexity it adds to the file.
```
run_as_real_user = dataset.get('run_as_real_user', False) or dataset_get.('in_place', True)
```
Remove the need to call `Binary.register_unsniffable_binary_ext()` for
each unsniffable binary datatype.
Fix https://github.com/galaxyproject/galaxy/issues/3441 , where the upload
of files of a datatype defined in datatypes_conf.xml as subclass of an
unsniffable binary datatype ended up with "The uploaded binary file
contains inappropriate content" because it was not possible to register the
subclassed datatype as unsniffable.
Also remove unused `stop_err()` function in upload.py .
Also, when sniffing binary files, sniff images together with the other
formats and respect sniff order.
Also remove `is_multi_byte` from:
- `stream_to_open_named_file()`
- `stream_to_file()`
Remove the now unused `get_image_ext()` and `Binary.is_sniffable_binary()`
and all the calls to `Binary.register_sniffable_binary_format()`.
Also, merge its 3 parameters `gzip_only`, `bz2_only`, `zip_only` into
`compressed_formats` (a list of allowed formats).
As a consequence of the changes in `get_fileobj()`, update:
- `files_diff()`
- `get_file_peek()`, which now determines that a file is binary when a
`UnicodeDecodeError` exception is raised and doesn't need
`is_multi_byte` any more
- `iter_headers()` and `get_headers()`, which now return Unicode and don't
need `is_multi_byte` parameter any more
As a consequence of the changes in `get_file_peek()`, update:
- `set_peek()`, which now doesn't need `is_multi_byte` any more
As a consequence of the changes in `get_headers()`, update:
- `guess_ext` and `is_column_based()`, which now determine that a file is
binary when a `UnicodeDecodeError` exception is raised and don't need
`is_multi_byte` any more
As a consequence of the changes to `guess_ext`, update:
- `handle_uploaded_dataset_file() doesn't need `is_multi_byte` any more
Also, remove duplicated calls to `get_file_peek()` in
lib/galaxy/datatypes/molecules.py and lib/galaxy/datatypes/msa.py
The `is_multi_byte` was not removed from the signature of `get_file_peek()`
and `set_peek()` in order to preserve compatibility for ToolShed datatypes,
thanks @jmchilton for the review.
strings can not be written to byte mode file handlers.
writing resulted in: TypeError: a bytes-like object is required, not 'str'
Alternative would be to make string byte.
- Add flake8-import-order to flake8 Pipfile and remove py27-lint-imports
and py27-lint-imports-include-list tox envs
- Fix most E201 and E202 errors reported by flake8-import-order v0.15,
but pin flake8-import-order to v0.14.3 until
https://github.com/PyCQA/flake8-import-order/issues/123
is fixed
This let us drop 2 jobs on Travis per each job.
shutil.move tries to move files by renaming them. If that fails with
OSError (due to permission or cross-filesystem rename) it falls back to
copying files followed by removing them
(https://github.com/python/cpython/blob/2.7/Lib/shutil.py#L279). By
using shutil.move and catching permission problems we avoid an
unnecessary copy if source and destination are on the same filesystem.
Also avoids shutil.move if the upload tool is run as real-user which
should fix https://github.com/galaxyproject/galaxy/issues/4300.
When importing files from the FTP folder the admin can choose to prevent
purging of imported files. In this case the source file should not be
modified in-place by galaxy.
This also modifies relevant bare `except:` statements and modifies
the meaning of `in_place` in upload.py from no external chown script to
do not edit files in place if we keep the source file.
This fixes https://github.com/galaxyproject/galaxy/issues/4527.
- Add an API test for datatype-defined composite uploads - including exercising newline conversion and the space_to_tab parameter.
- Add a test decorator skip_without_datatype to mirror skip_without_tool for this test, improve both decorators.
- The ftype parameter in the composite test tools does nothing - drop it and drop it from the XSD spec.
- Slightly improve the documentation for these composite_data elements in the XSD.
Nicola Soranzo