John Chilton
57ff68edd0
Merge pull request #4539 from mvdbeek/purge_source
...
Prevent in-place editing of uploaded files if files are imported from the FTP folder
2017-09-07 10:11:04 -04:00
mvdbeek
cccf79ac5f
Make bz2file dependency conditional on python being below version 3.3
2017-09-05 18:30:25 +02:00
mvdbeek
46c85ffa98
Allow working with pbzip2 compressed files
...
This adds supprt for uploading and uncompressing pbzip2 compressed
files by exchanging the bz2 module with bz2file (recommended as workaround in
https://docs.python.org/2/library/bz2.html ). The problem was reported by
@alpapan in #4538 and this commit should fix #4538 .
Depends on https://github.com/galaxyproject/starforge/pull/147 for the
bz2file wheel.
2017-09-05 16:54:34 +02:00
mvdbeek
06db29413e
Move instead of copying converted datasets when possible
...
shutil.move tries to move files by renaming them. If that fails with
OSError (due to permission or cross-filesystem rename) it falls back to
copying files followed by removing them
(https://github.com/python/cpython/blob/2.7/Lib/shutil.py#L279 ). By
using shutil.move and catching permission problems we avoid an
unnecessary copy if source and destination are on the same filesystem.
Also avoids shutil.move if the upload tool is run as real-user which
should fix https://github.com/galaxyproject/galaxy/issues/4300 .
2017-09-03 12:03:36 +02:00
mvdbeek
62ef8b52ae
Fix sniffing for non-binary files
2017-09-03 10:49:23 +02:00
mvdbeek
14497719ba
Prevent in-place editing if purge_source is false
...
When importing files from the FTP folder the admin can choose to prevent
purging of imported files. In this case the source file should not be
modified in-place by galaxy.
This also modifies relevant bare `except:` statements and modifies
the meaning of `in_place` in upload.py from no external chown script to
do not edit files in place if we keep the source file.
This fixes https://github.com/galaxyproject/galaxy/issues/4527 .
2017-09-03 09:48:39 +02:00
John Chilton
0fa287f76a
Add autopep8 script to help rebasing branches after #4440 .
...
Fixes a few more PEP8 issues as part of the autopep8'ing as well.
2017-08-17 13:47:03 -04:00
Nicola Soranzo
a353b2e88f
Fix E127 errors introduced by previous commit
...
using:
```
autopep8 -i -r --exclude $(sed -e 's|^|./|' -e 's|/$||' .ci/flake8_blacklist.txt | paste -sd,) --select E127 .
```
2017-08-17 11:35:39 +01:00
Nicola Soranzo
21b44bf348
Fix all E201 and E202 style errors
...
using the following command:
```
autopep8 -i -r --exclude $(sed -e 's|^|./|' -e 's|/$||' .ci/flake8_blacklist.txt | paste -sd,) --select E201,E202 .
```
2017-08-17 11:35:39 +01:00
John Chilton
fec1eab82d
Add ability to arbitrarily not decompress files on upload via API.
2017-08-12 06:36:50 -04:00
John Chilton
59aa11864a
Add configuration option to disable upload content checking.
2017-08-12 06:36:50 -04:00
guerler
7a6961e30d
Undo dataset renaming
2017-05-01 11:57:07 -04:00
Nate Coraor
f645259dd0
Only keep the last path element when naming files uploaded via the URL
...
paste method
2017-02-27 10:59:21 -05:00
Nicola Soranzo
1d17a29c79
Fixes for pycodestyle 2.2.0
2016-11-15 16:56:50 +00:00
Dannon Baker
cb6e25278c
Refactoring, because it was bugging me.
2016-10-13 10:25:18 -04:00
Dannon Baker
bb44288968
Failthrough to using imghdr when PIL does not detect an image correctly.
...
Some misc cleanup, avoiding use of system functions as variable names,
etc.
2016-10-13 09:55:01 -04:00
Nicola Soranzo
cc336f61c0
Make some files compatible with Python3
...
Also enlarge the set of files tested with flake8 under Python3
2016-06-03 19:24:46 +01:00
Nicola Soranzo
8a43b2b492
Python 3: use "as" instead of comma in except clause
2016-05-17 23:53:16 +01:00
John Chilton
997b53f3ae
Implement option to prevent FTP uploads from being deleted on import.
...
- Simply set `ftp_upload_purge` to `False` in galaxy.ini to enable this.
- xref http://dev.list.galaxyproject.org/Data-removal-after-ftp-import-td4667111.html
- slight xref http://dev.list.galaxyproject.org/Uploading-files-to-galaxy-from-a-folder-td4664614.html
2016-04-14 11:34:21 -04:00
Nicola Soranzo
501a10c3f1
Do not import dumps and loads from galaxy.util.json .
2016-03-02 19:04:24 +00:00
John Chilton
448aee9f00
Fix for multi-byte char problem in PR #1556...
...
... caught by @nsoranzo of course.
2016-02-08 13:48:43 +00:00
John Chilton
499357e508
Remove unneeded galaxy.model import in upload.py...
...
... and document the ability to remove from other data sources.
2016-01-21 14:31:42 +00:00
John Chilton
50e531aa59
Refactor galaxy.datatypes.checkers into galaxy.util.checkers.
...
- Primary reason is because it is used all over the place in contexts that have nothing to do with Galaxy datatype classes or datatype registery or sniffing, most importantly by the code in the tool shed for determining if something is a tool or not.
- This code has no dependencies on the datatypes module or any hard dependencies on things not in the Python standard library.
2015-12-17 15:41:06 +00:00
John Chilton
83f57c56c1
Refactor is_multi_byte out of galaxy.util.
...
It introduces a fairly esoteric dependency in galaxy.util for just a couple uses in Galaxy. Separating it out like this will allow reuse of the entire galaxy.util with only one external dependency (docutils, which planemo for instance already depends on).
2015-12-16 21:36:16 +00:00
Mark Einon
d6d8e5ef86
upload.py: Remove unused function parameter
2015-11-13 11:33:56 +00:00
Nicola Soranzo
7830bc62e0
Another fix for 3c40f32fa2 .
2015-06-17 19:21:13 +01:00
Nicola Soranzo
3c40f32fa2
PEP-8 and pylint fixes.
2015-06-16 16:53:49 +01:00
Bjoern Gruening
5b96f80b48
Add CompressedArchive as datatype and do not uncomress it during upload.
2015-01-12 22:36:20 +01:00
Daniel Blankenberg
b022e92f5b
Add gemini.sqlite datatype.
2015-01-09 14:48:56 -05:00
Dannon Baker
9d85ca9e23
Switch to_json_string/from_json_string in galaxy/tools.
2014-09-09 10:06:35 -04:00
John Chilton
6d4d6dd91a
Bugfix: Fix shutil.move for converted files in upload.py.
2014-06-10 09:36:28 -05:00
Dannon Baker
c30b7838cc
Fix for when check_binary fails against an actual binary datatype. The logic in upload needs to be revisited per the comments.
2014-06-02 13:16:15 -04:00
Trevor Wennblom
1848cf2e3d
whitespace
2014-02-20 18:14:09 -06:00
Gert Hulselmans
fb01c3cc96
Avoid corruption of binary files embedded in gzip, bz2 and zip archives in the upload tool.
...
Add an option in the upload tool to disable the conversion of universal
line endings to Posix line endings.
This is useful for avoiding corruption of uploaded files when a binary
file is contained inside a gzip, bz2 and zip archive.
This fixes bug report: https://trello.com/card/issue-with-uploaded-2bit-gz-files/506338ce32ae458f6d15e4b3/702
2013-05-27 20:55:52 +02:00
Dannon Baker
8161b70d89
Merged in nsoranzo/galaxy-central (pull request #135 )
...
Add generic ASN.1 text and binary datatypes
2013-04-04 08:41:54 -04:00
Nicola Soranzo
2f650b8866
Fix file extension determination
...
Extension was determined uncorrectly for filenames with more than 1 dot
2013-03-06 12:37:08 +01:00
Kyle Ellrott
4aad0ad15e
Enabling dataset UUID setting through library upload, upload1 tool, and galaxy.json file
2013-01-29 15:58:58 -08:00
John Chilton
886a5c61b9
Merge.
2012-09-03 22:14:36 -05:00
Daniel Blankenberg
48548341fc
Handle non-ascii unicode in upload tool. Significant pre-existing refactoring still needed.
2012-08-28 19:23:39 -04:00
John Chilton
d281e835ef
Remove hard-coding of unsniffable binary types array and manually
...
checking each sniffable type with a seperate function in
upload.py. Information on both types is now stored dynamically as
static variables in the Binary class.
2012-08-15 23:35:28 -05:00
Nate Coraor
c362e6202e
A bit more actual user cleaning.
2012-01-18 14:34:00 -05:00
Nate Coraor
61062a3ae5
Merge galaxy-central.
2012-01-18 10:34:39 -05:00
Nate Coraor
85ba013e6d
Actual User: Fix for newline conversion on upload, tighten file permissions for files in upload. Please make sure you clean your temp directory regularly.
2011-12-08 13:22:39 -05:00
Greg Von Kuster
17c5b27cdc
Re-engineer the datatypes registry so that it is initialized once when the Galaxy server is started, but data types can continue to be loaded throughout the Galaxy server's session (hopefully this doesn't break anything).
...
Add support for a single "import_module" to be passed to the new load_datatypes() method in the datatypes registry. This provides the ability to load a single class module from an installed tool shed repository along with a datatypes_conf.xml file included in the installed repository and pass them to the new load_datatypes() method. In the future, multiple imported modules may be allowed. The datatypes_conf.xml file included in the repository must conform to a slightly different definition than the same named file that comes with the distribution. This new definition will be documented in the Galaxy tool shed wiki.
We now have the ability to load new data types into the Galaxy server from an installed tool shed repository without restarting the Galaxy server.
2011-11-23 16:16:15 -05:00
ichorny
236276b132
add actual user changes to galaxy-central
2011-10-21 14:01:48 -07:00
Nate Coraor
7d8b701df6
Fix uploading sorted BAMs to libraries when using the link method rather than copying.
2011-08-26 15:59:29 -04:00
Greg Von Kuster
d8279f0e5c
Add an altered version of Jelle Scholtalbers' enhancement code to support uploding various image data types. I've moved some of the code components around from Jelle's version, and added some fixes. The cod ein the new image_util.py file enables detections of images types without the need to create a new Image() class.
...
I've also added baseline support of the HDF5 data type in this change set, but it is currently required to be in the unsniffable_data_types list.
2011-07-27 16:14:15 -04:00
Nate Coraor
3959897d3d
Uploading a compressed library dataset using either the server directory or path paste methods was previously overwriting the input file, which should never happen. This changeset also makes it possible to upload compressed files that remain compressed (by choosing to "link files only"), however, Galaxy does not yet open the compressed file to properly set metadata on its contents.
2011-06-08 15:35:36 -04:00
Greg Von Kuster
61c3ed305f
Move duplicate data type checker methods from sniff and upload into a new ~/datatypes/checkers.py.
2011-06-03 15:36:59 -04:00
Greg Von Kuster
7640af8e06
Don't alter the contents of a file while uploading to a data library if using the filesystem_paths option. This partially resolves the issue where a supposedly sorted BAM file was being resorted upon upload to a data library when using this option. A better imlementation of determining whether a BAM file has been sorted (so that it does not get resorted) remains to be done.
2011-03-15 09:38:29 -04:00
John Chilton