John Chilton
50e531aa59
Refactor galaxy.datatypes.checkers into galaxy.util.checkers.
...
- Primary reason is because it is used all over the place in contexts that have nothing to do with Galaxy datatype classes or datatype registery or sniffing, most importantly by the code in the tool shed for determining if something is a tool or not.
- This code has no dependencies on the datatypes module or any hard dependencies on things not in the Python standard library.
2015-12-17 15:41:06 +00:00
John Chilton
83f57c56c1
Refactor is_multi_byte out of galaxy.util.
...
It introduces a fairly esoteric dependency in galaxy.util for just a couple uses in Galaxy. Separating it out like this will allow reuse of the entire galaxy.util with only one external dependency (docutils, which planemo for instance already depends on).
2015-12-16 21:36:16 +00:00
Mark Einon
d6d8e5ef86
upload.py: Remove unused function parameter
2015-11-13 11:33:56 +00:00
Nicola Soranzo
7830bc62e0
Another fix for 3c40f32fa2 .
2015-06-17 19:21:13 +01:00
Nicola Soranzo
3c40f32fa2
PEP-8 and pylint fixes.
2015-06-16 16:53:49 +01:00
Bjoern Gruening
5b96f80b48
Add CompressedArchive as datatype and do not uncomress it during upload.
2015-01-12 22:36:20 +01:00
Daniel Blankenberg
b022e92f5b
Add gemini.sqlite datatype.
2015-01-09 14:48:56 -05:00
Dannon Baker
9d85ca9e23
Switch to_json_string/from_json_string in galaxy/tools.
2014-09-09 10:06:35 -04:00
John Chilton
6d4d6dd91a
Bugfix: Fix shutil.move for converted files in upload.py.
2014-06-10 09:36:28 -05:00
Dannon Baker
c30b7838cc
Fix for when check_binary fails against an actual binary datatype. The logic in upload needs to be revisited per the comments.
2014-06-02 13:16:15 -04:00
Trevor Wennblom
1848cf2e3d
whitespace
2014-02-20 18:14:09 -06:00
Gert Hulselmans
fb01c3cc96
Avoid corruption of binary files embedded in gzip, bz2 and zip archives in the upload tool.
...
Add an option in the upload tool to disable the conversion of universal
line endings to Posix line endings.
This is useful for avoiding corruption of uploaded files when a binary
file is contained inside a gzip, bz2 and zip archive.
This fixes bug report: https://trello.com/card/issue-with-uploaded-2bit-gz-files/506338ce32ae458f6d15e4b3/702
2013-05-27 20:55:52 +02:00
Dannon Baker
8161b70d89
Merged in nsoranzo/galaxy-central (pull request #135 )
...
Add generic ASN.1 text and binary datatypes
2013-04-04 08:41:54 -04:00
Nicola Soranzo
2f650b8866
Fix file extension determination
...
Extension was determined uncorrectly for filenames with more than 1 dot
2013-03-06 12:37:08 +01:00
Kyle Ellrott
4aad0ad15e
Enabling dataset UUID setting through library upload, upload1 tool, and galaxy.json file
2013-01-29 15:58:58 -08:00
John Chilton
886a5c61b9
Merge.
2012-09-03 22:14:36 -05:00
Daniel Blankenberg
48548341fc
Handle non-ascii unicode in upload tool. Significant pre-existing refactoring still needed.
2012-08-28 19:23:39 -04:00
John Chilton
d281e835ef
Remove hard-coding of unsniffable binary types array and manually
...
checking each sniffable type with a seperate function in
upload.py. Information on both types is now stored dynamically as
static variables in the Binary class.
2012-08-15 23:35:28 -05:00
Nate Coraor
c362e6202e
A bit more actual user cleaning.
2012-01-18 14:34:00 -05:00
Nate Coraor
61062a3ae5
Merge galaxy-central.
2012-01-18 10:34:39 -05:00
Nate Coraor
85ba013e6d
Actual User: Fix for newline conversion on upload, tighten file permissions for files in upload. Please make sure you clean your temp directory regularly.
2011-12-08 13:22:39 -05:00
Greg Von Kuster
17c5b27cdc
Re-engineer the datatypes registry so that it is initialized once when the Galaxy server is started, but data types can continue to be loaded throughout the Galaxy server's session (hopefully this doesn't break anything).
...
Add support for a single "import_module" to be passed to the new load_datatypes() method in the datatypes registry. This provides the ability to load a single class module from an installed tool shed repository along with a datatypes_conf.xml file included in the installed repository and pass them to the new load_datatypes() method. In the future, multiple imported modules may be allowed. The datatypes_conf.xml file included in the repository must conform to a slightly different definition than the same named file that comes with the distribution. This new definition will be documented in the Galaxy tool shed wiki.
We now have the ability to load new data types into the Galaxy server from an installed tool shed repository without restarting the Galaxy server.
2011-11-23 16:16:15 -05:00
ichorny
236276b132
add actual user changes to galaxy-central
2011-10-21 14:01:48 -07:00
Nate Coraor
7d8b701df6
Fix uploading sorted BAMs to libraries when using the link method rather than copying.
2011-08-26 15:59:29 -04:00
Greg Von Kuster
d8279f0e5c
Add an altered version of Jelle Scholtalbers' enhancement code to support uploding various image data types. I've moved some of the code components around from Jelle's version, and added some fixes. The cod ein the new image_util.py file enables detections of images types without the need to create a new Image() class.
...
I've also added baseline support of the HDF5 data type in this change set, but it is currently required to be in the unsniffable_data_types list.
2011-07-27 16:14:15 -04:00
Nate Coraor
3959897d3d
Uploading a compressed library dataset using either the server directory or path paste methods was previously overwriting the input file, which should never happen. This changeset also makes it possible to upload compressed files that remain compressed (by choosing to "link files only"), however, Galaxy does not yet open the compressed file to properly set metadata on its contents.
2011-06-08 15:35:36 -04:00
Greg Von Kuster
61c3ed305f
Move duplicate data type checker methods from sniff and upload into a new ~/datatypes/checkers.py.
2011-06-03 15:36:59 -04:00
Greg Von Kuster
7640af8e06
Don't alter the contents of a file while uploading to a data library if using the filesystem_paths option. This partially resolves the issue where a supposedly sorted BAM file was being resorted upon upload to a data library when using this option. A better imlementation of determining whether a BAM file has been sorted (so that it does not get resorted) remains to be done.
2011-03-15 09:38:29 -04:00
Greg Von Kuster
df6f94b1d0
Per request from Peter Cock, turn the "Copy data into Galaxy?" check box into a select list that clarifies the behavior. The default is to copy files into Galaxy.
2011-03-11 14:56:11 -05:00
Nate Coraor
82c4a11c88
Handle a minor bug in the upload tool.
2011-01-03 12:05:13 -05:00
Nate Coraor
b140866d0b
Heh heh. Whoops.
2010-11-30 14:37:19 -05:00
Nate Coraor
a07ef2cdd3
Add Job stdout/stderr to library item info page to aid in debugging.
2010-11-30 14:34:17 -05:00
Nate Coraor
8367cdc4a6
"Fix" zip uploads for Python < 2.6. The scare quotes are because in Python < 2.6 you can't read a zip member without loading the entire member into memory...
2010-10-25 16:10:09 -04:00
Nate Coraor
2fb759bd99
Read zipfiles in chunks when uncompressing in the upload tool. And I continue to wish zipfile was a lot more like tarfile...
2010-10-20 17:00:06 -04:00
Nate Coraor
4ca447aa70
Add support for bz2 compressed uploads.
2010-09-10 14:07:16 -04:00
Greg Von Kuster
a7e30c4441
Apply patch from Brad Chapman providing support for detecting, uploading and displaying UCSC bigWig and bigBed. Add new functional tests for uploading and detecting bigbed and bigwig formats, and correct and clean up the test_get_data.py functional test script.
2010-08-19 13:58:10 -04:00
Nate Coraor
8a66864aee
Stop ignoring the sniff order set in the datatypes config file.
2010-08-18 14:56:10 -04:00
Kanwei Li
810c2291de
Detect pdf files on upload [Brad Chapman]. Resolves #357
2010-07-02 18:10:45 -04:00
Nate Coraor
7c71894533
Remove obsolete binseq.zip and txtseq.zip formats, and allow for uploading single files in a zip archive. Adapted from a patch from Pablo Cingolani.
2010-06-23 16:48:13 -04:00
Ross Lazarus
6c39995d19
Patch to upload.py for composite file uploads where the user supplies urls for components
...
At present, the path for each component is passed as a url and there's no upload type for
composite file components, so this
temporary fix searches for :// in the path to figure out if this is an url upload.
Not ideal and in need of a more robust repair.
2010-06-11 21:21:33 -04:00
Daniel Blankenberg
183c2d3d13
Bug fix for handling 'spaces to tab' for non-binary composite datatype files.
...
Add upload tests for testing 'space to tab' when uploading composite and non-composite datatypes.
2010-06-07 15:34:20 -04:00
Daniel Blankenberg
64c68b5a66
Several Fixes for Uploading Composite datatypes. Revert changeset 3769:69e70e588a8e.
2010-05-12 15:57:05 -04:00
Ross Lazarus
84b05c8113
Fixes to rgQC top table ordering and add offset so can use rgManQQ to make plots
...
Fixes to rgManQQ to handle unsorted data - now sorts before manhattan plot
2010-04-05 14:14:09 -04:00
Nate Coraor
933944c12d
Fix bug in upload where source import files could be removed if an error occurs during detection and galaxy owns the source files.
2010-03-23 12:45:08 -04:00
Nate Coraor
482b69b749
Modify the newline conversion methods in sniff so converting in place is optional. This is necessary to fix a bug that occurs if using the 'server_dir' or 'path_paste' library upload methods: previously, they would modify the file to be imported in-place if permissions allowed (probably not what the admin wanted) or fail if permissions did not allow. New functionality is to return the converted tempfile if 'server_dir' or 'path_paste' methods are used. Also, no newline conversion will be done if the administrator uses the symlink checkbox.
2010-03-11 11:56:20 -05:00
Greg Von Kuster
75753f1783
Add error handling to upload's check_zip.
2009-12-09 16:20:12 -05:00
Greg Von Kuster
cf83cecb0b
Better approach to altering initial content of output dataset if necessary. The upload tool will now call the data type's groom_output_dataset() method ( a better name than before_setting_metadata since it is not related to metadata ). This will now also run on the cluster.
2009-12-08 11:46:13 -05:00
Greg Von Kuster
bcd6751cdf
Fix for uploading a Bam file that has not yet been sorted, the call to samtools sort has been moved from the sam_to_bam tool to the Bam()set_meta() method to ensure all Bam datasets are sorted prior to indexing. Added new functional tests to cover uploaing unsorted Bam files. Also cleaned up code in the upload tool for uploaing various binary data formats.
2009-12-07 16:04:33 -05:00
Greg Von Kuster
08e0d5e532
Fixes for Bam data type's set_meta() and sam_to_bam tool, indexes will now be properly created for bam datasets. Fixes also for uploading Bam files, they will no longer be uncompressed on upload.
2009-12-04 19:51:46 -05:00
Greg Von Kuster
e38e78f5e4
Fixes, cleanup and new functional tests for data types and upload:
...
- moved all supported binary data types to the new binary.py
- changed GeneTrack data type to subclass from Text rather than Binary
- added Sff data type to datatypes_conf.xml.sample
- merged test_sniffinad_and_metadata_settings.py test scritp into test_get_data.py
- added several additional functional test for data types to test_get_data.py
- fixed some bugs in upload.py when uploading binary data types
2009-11-13 14:13:03 -05:00
John Chilton