Make sure to backup your database before updating.
Also fix a long standing bug in the async controller dealing with updating output datasets upon job completion.
Credit for database migration directions go to Greg.
*** This is a significant change, be sure to follow the migration steps below exactly and in order: ***
---------------
0. Stop Galaxy server
---------------
1. Backup database and galaxy_root/database directory
---------------
2. The current validation_error table is not being used, it is left over from Ian's work that was not made functional. However, if we want to keep it around, we should drop the old version of the table so it will get re-created correctly when the Galaxy server is restarted:
any database - sql command(s):
------------------------------
DROP TABLE validation_error;
---------------
3. Stop server, perform svn update to get latest code, start server to create new history_dataset_association and implicitly_converted_dataset_association tables
---------------
4. Add new columns to dataset table:
postgres sql command(s):
------------------------
ALTER TABLE dataset ADD COLUMN purgable boolean DEFAULT 't';
ALTER TABLE dataset ADD COLUMN external_filename text;
ALTER TABLE dataset ADD COLUMN _extra_files_path text;
mysql sql command(s):
---------------------
ALTER TABLE dataset ADD COLUMN purgable boolean DEFAULT TRUE;
ALTER TABLE dataset ADD COLUMN external_filename text;
ALTER TABLE dataset ADD COLUMN _extra_files_path text;
sqlite sql command(s):
----------------------
ALTER TABLE dataset ADD COLUMN purgable boolean DEFAULT 0;
ALTER TABLE dataset ADD COLUMN external_filename text;
ALTER TABLE dataset ADD COLUMN _extra_files_path text;
---------------
5. Populate new columns:
postgres sql command(s):
------------------------
UPDATE
dataset
SET
external_filename = dataset_filename.filename,
_extra_files_path = dataset_filename.extra_files_path
FROM
dataset_filename
WHERE
dataset.filename_id = dataset_filename.id;
mysql or sqlite sql command(s):
-------------------------------
UPDATE
dataset,
dataset_filename
SET
dataset.external_filename = dataset_filename.filename,
dataset._extra_files_path = dataset_filename.extra_files_path
WHERE
dataset.filename_id = dataset_filename.id;
---------------
6. Populate dataset table with info from dataset_child_association table:
postgres sql command(s):
------------------------
UPDATE
dataset
SET
parent_id = dataset_child_association.parent_dataset_id
FROM
dataset_child_association
WHERE
dataset.id = dataset_child_association.child_dataset_id;
mysql or sqlite sql command(s):
-------------------------------
UPDATE
dataset,
dataset_child_association
SET
dataset.parent_id = dataset_child_association.parent_dataset_id
WHERE
dataset.id = dataset_child_association.child_dataset_id;
---------------
7. Drop dataset_child_association table:
any database - sql command(s):
------------------------------
DROP TABLE dataset_child_association;
---------------
8. Copy parts of the current dataset table to the new history_dataset_association table:
postgres sql command(s):
------------------------
INSERT INTO history_dataset_association
SELECT
id,
history_id,
id AS dataset_id,
create_time,
update_time,
hid,
name,
info,
blurb,
peek,
extension,
metadata,
parent_id,
designation,
deleted,
visible
FROM dataset
ORDER BY id;
mysql or sqlite sql command(s):
-------------------------------
INSERT INTO
history_dataset_association
(
history_id,
dataset_id,
create_time,
update_time,
hid,
name,
info,
blurb,
peek,
extension,
metadata,
parent_id,
designation,
deleted,
visible
)
SELECT
history_id,
id,
create_time,
update_time,
hid,
name,
info,
blurb,
peek,
extension,
metadata,
parent_id,
designation,
deleted,
visible
FROM
dataset
ORDER BY
id;
---------------
9. Update the nextval value of the primary key sequence in the history_dataset_association table
NOTE: THIS IS NOT NECESSARY FOR MYSQL OR SQLITE!
postgres sql command(s):
------------------------
SELECT
setval('history_dataset_association_id_seq', max(id))
FROM
history_dataset_association;
---------------
10. Update dataset table to mark dataset_associated_file dataset as deleted:
postgres sql command(s):
------------------------
UPDATE dataset
SET
deleted = 't'
WHERE
id IN
(SELECT dataset_id FROM dataset_associated_file ORDER BY dataset_id DESC);
mysql or sqlite sql command(s):
-------------------------------
UPDATE dataset
SET
deleted = TRUE
WHERE
id IN
(SELECT dataset_id FROM dataset_associated_file ORDER BY dataset_id DESC);
---------------
11. Drop the dataset_associated_file table:
any database - sql command(s):
------------------------------
DROP TABLE dataset_associated_file;
---------------
12. Alter foreign keys on job_to_input_dataset table:
postgres or sqlite sql command(s):
------------------------
ALTER TABLE
job_to_input_dataset
DROP CONSTRAINT
job_to_input_dataset_dataset_id_fkey;
ALTER TABLE
job_to_input_dataset
ADD CONSTRAINT job_to_input_dataset_dataset_id_fkey
FOREIGN KEY
(dataset_id)
REFERENCES
history_dataset_association(id);
mysql sql command(s):
-------------------------------
ALTER TABLE
job_to_input_dataset
DROP FOREIGN KEY
ix_job_to_input_dataset_dataset_id;
ALTER TABLE
job_to_input_dataset
ADD FOREIGN KEY
ix_job_to_input_dataset_dataset_id (dataset_id)
REFERENCES
history_dataset_association(id);
---------------
13. Alter foreign keys on job_to_output_dataset table:
postgres or sqlite sql command(s):
------------------------
ALTER TABLE
job_to_output_dataset
DROP CONSTRAINT
job_to_output_dataset_dataset_id_fkey;
ALTER TABLE
job_to_output_dataset
ADD CONSTRAINT
job_to_output_dataset_dataset_id_fkey
FOREIGN KEY (dataset_id)
REFERENCES history_dataset_association(id);
mysql sql command(s):
-------------------------------
ALTER TABLE
job_to_output_dataset
DROP FOREIGN KEY
ix_job_to_output_dataset_dataset_id;
ALTER TABLE
job_to_output_dataset
ADD FOREIGN KEY
ix_job_to_output_dataset_dataset_id
FOREIGN KEY (dataset_id)
REFERENCES history_dataset_association(id);
---------------
14. Eliminate columns from the dataset table previously copied to history_dataset_association:
any database - sql command(s):
------------------------------
ALTER TABLE dataset DROP COLUMN hid;
ALTER TABLE dataset DROP COLUMN history_id;
ALTER TABLE dataset DROP COLUMN name;
ALTER TABLE dataset DROP COLUMN info;
ALTER TABLE dataset DROP COLUMN blurb;
ALTER TABLE dataset DROP COLUMN peek;
ALTER TABLE dataset DROP COLUMN extension;
ALTER TABLE dataset DROP COLUMN dbkey;
ALTER TABLE dataset DROP COLUMN metadata;
ALTER TABLE dataset DROP COLUMN parent_id;
ALTER TABLE dataset DROP COLUMN designation;
ALTER TABLE dataset DROP COLUMN visible;
ALTER TABLE dataset DROP COLUMN filename_id;
of downloading its own SGE and linking, scramble now just links to a
user-supplied SGE in $SGE_ROOT.
Added a seperate script to make it build right on sparc/x86 32/64
Solaris.
If a dataset is not the proper format required by a tool, but there is a converter available, the dataset will appear as a valid option in DataToolParameters.
The converted dataset will be created (and subsequently reused) when the job is executed. Conversion utilizes the job runner, and will run on the cluster.
Setting metadata will invalidate the converted dataset, and a new one will be automatically generated as needed.
Whether running locally or via PBS, $PATH should now be handled in your
user's environment.
This also fixes problems like PYTHONPATH on the new blast nodes being
wrong (because they are x86_64 and the frontends are i686).
Note that you can actually start Galaxy without having an eggs dir now.
- Added an option to remove dataset from disk when purging, or leave them for renaming and later removal
- Moved History.purge() and Dataset.purge() methods to the cleanup_datasets.py script
- Eliminated duplicate functionality from admin controller, moving appropriate components to Galaxy reports.
- Re-implemented the long-term process we will use for deleting / purging histories and datasets
(we will initially write external shell scripts to rename dataset files rather then deleting them)
distribution point and copy them to Galaxy's eggs/ directory.
fetch_eggs_tarball.py and check_python_platform.py are mostly for offline
systems. fetch_eggs_tarball.py fetches eggs for your platform (or a
specified platform) and then tars them up, for copying to another system.
check_python_platform.py returns the string for your platform/interpreter
that will be used to download from the egg distribution site.
For those that may have installed revision 2256, use the following SQL command to evolve your schema:
alter table dataset drop column removed;
We have changed direction again on the process we will use to cleanup datasets and histories.
- When we are comfortable with the purge process, the Dataset.purge() function will be modified to delete the file from disk rather than simply renaming it, and the Dataset.remove_from_disk() function will be eliminated.
- The entire process below must be executed at least 1 time ( prior to making these changes ) to ensure that purged files are properly removed from disk. This process is:
delete_userless_histories() -> purge_histories() -> purge_datasets() -> remove_datasets()
Shell scripts have been added for each of these stages with the exception of remove_datasets(), which will eventually be eliminated.
SQL commands to alter schema:
alter table dataset add column removed boolean default 'f';
alter history add column purged boolean default 'f';
- The lifecycle of a history after creation is now deleted -> purged
- Deleting a history will delete all associated datasets (only the database is update, nothing removed from disk).
- Purging a history will purge all associated datasets (datasets are renamed, nothing removed from disk).
- The lifecycle of a dataset after creation is now deleted -> purged -> removed
- Deleting a dataset will only update the database.
- Purging a dataset will rename the file on disk and update the database.
- Removing a dataset will remove the renamed, purged file from disk and update the database.
- Also reverted the behavior of history_delete() to pre-rev 2250 so that no datasets are deleted at teh time the history was deleted by the user.
- The admin controller and the cleanup_datasets.py script include all of the above functionality.
Greg Von Kuster