checking each sniffable type with a seperate function in
upload.py. Information on both types is now stored dynamically as
static variables in the Binary class.
Add support for a single "import_module" to be passed to the new load_datatypes() method in the datatypes registry. This provides the ability to load a single class module from an installed tool shed repository along with a datatypes_conf.xml file included in the installed repository and pass them to the new load_datatypes() method. In the future, multiple imported modules may be allowed. The datatypes_conf.xml file included in the repository must conform to a slightly different definition than the same named file that comes with the distribution. This new definition will be documented in the Galaxy tool shed wiki.
We now have the ability to load new data types into the Galaxy server from an installed tool shed repository without restarting the Galaxy server.
At present, the path for each component is passed as a url and there's no upload type for
composite file components, so this
temporary fix searches for :// in the path to figure out if this is an url upload.
Not ideal and in need of a more robust repair.
- moved all supported binary data types to the new binary.py
- changed GeneTrack data type to subclass from Text rather than Binary
- added Sff data type to datatypes_conf.xml.sample
- merged test_sniffinad_and_metadata_settings.py test scritp into test_get_data.py
- added several additional functional test for data types to test_get_data.py
- fixed some bugs in upload.py when uploading binary data types
("Upload files from filesystem paths") to the admin-side library upload pages.
This form contains a textarea that allows Galaxy admins to paste any number of
filesystem paths (files or directories) from which Galaxy will import library
datasets, saving the directory structure (if desired). Since such ability
allows admins access to any file on the Galaxy server which is readable by
Galaxy's system user, this option is disabled by default, and system
administrators should take care in assigning Galaxy administrators when this
feature is enabled. Controls on what files are accessible to this tool based
on ownership or other properties can be added at a later date if there is
sufficient interest for such features.
This commit also includes a checkbox on the "Upload directory of files" page
(as well as the new "Upload files from filesystem paths" page above) that will
prevent Galaxy from copying data to its files directory (by default,
'database/files/'). This is useful for large library datasets that live in
their own managed locations on the filesystem, this will prevent the existence
of duplicate copies of datasets (but means administrators must take care to
manage data - moving or removing the data from its Galaxy-external location
will render these datasets invalid within Galaxy).
One unique feature to be aware of: when using the "Copy data into Galaxy?"
checkbox on the "Upload directory of files" page, any symbolic links
encountered in the chosen import directory will be made absolute and
dereferenced ONCE. This allows administrators to link large datasets to the
import directory, rather than having to make full copies, while being able to
delete such links after importing. Only the first symlink (the one in the
import directory itself) is dereferenced; all others remain. See the following
for an example:
library_import_dir = /galaxy/import
% ls -lR /galaxy/import
/galaxy/import:
total 6
drwxr-xr-x 2 nate nate 512 Oct 1 11:31 link/
/galaxy/import/link:
total 10
lrwxrwxrwx 1 nate nate 71 Oct 1 10:38 1.bed -> ../../../home/nate/galaxy/test-data/1.bed
lrwxrwxrwx 1 nate nate 60 Oct 1 10:38 2.bed -> /home/nate/galaxy/test-data/2.bed
lrwxrwxrwx 1 nate nate 11 Oct 1 10:38 3.bed -> ../../3.bed
lrwxrwxrwx 1 nate nate 35 Oct 1 11:30 4.bed -> ../../galaxy_symlink/test-data/4.bed
lrwxrwxrwx 1 nate nate 41 Oct 1 11:31 5.bed -> /galaxy/galaxy_symlink/test-data/5.bed
% ls -l /galaxy/3.bed
lrwxrwxrwx 1 nate nate 60 Oct 1 10:39 /galaxy/3.bed -> /home/nate/galaxy/test-data/3.bed
% ls -l /galaxy/galaxy_symlink
lrwxrwxrwx 1 nate nate 44 Oct 1 11:30 /galaxy/galaxy_symlink -> /home/nate/galaxy/
In this example,
1.bed is a relative symbolic link to the real 1.bed.
2.bed is an absolute symlink to the real 2.bed.
3.bed is a relative symlink to ../../3.bed, aka /galaxy/3.bed, which itself is
a symlink to the real 3.bed.
4.bed is a relative symlink which follows another symlink
(/galaxy/galaxy_symlink) to the real 4.bed.
5.bed is an absolute symlink in the same fashion as 4.bed
If the 'link' server directory is chosen on the "Upload directory of files"
page, and "Copy data into Galaxy?" is checked "No", the following files will be
referenced by Galaxy:
/home/nate/galaxy/test-data/1.bed
/home/nate/galaxy/test-data/2.bed
/galaxy/3.bed
/galaxy/galaxy_symlink/test-data/4.bed
/galaxy/galaxy_symlink/test-data/5.bed
The Galaxy administrator may now safely delete /galaxy/import/link, but should
take care not to remove the referenced symbolic links (/galaxy/3.bed,
/galaxy/galaxy_symlink).
Not all symbolic links are dereferenced because it is assumed that if an
administrator links to a path in the import directory which itself is (or
contains) links, that is the preferred path for accessing the data.
Bjoern Gruening