+ «
+
+
+
diff --git a/doc/source/admin/index.rst b/doc/source/admin/index.rst
new file mode 100644
index 00000000000..a3abf3d072e
--- /dev/null
+++ b/doc/source/admin/index.rst
@@ -0,0 +1,11 @@
+Special Topics in Galaxy Administration & Deployment Documentation
+==================================================================
+
+This documentation intends to function as a version-specific supplement to the
+`wiki `__, not the primary admin
+documentation. These resources should be used together.
+
+.. toctree::
+ :maxdepth: 3
+
+ interactive_environments.rst
diff --git a/doc/source/admin/interactive_environments.png b/doc/source/admin/interactive_environments.png
new file mode 100644
index 00000000000..49a31646fd8
Binary files /dev/null and b/doc/source/admin/interactive_environments.png differ
diff --git a/doc/source/admin/interactive_environments.rst b/doc/source/admin/interactive_environments.rst
new file mode 100644
index 00000000000..8d7821babcb
--- /dev/null
+++ b/doc/source/admin/interactive_environments.rst
@@ -0,0 +1,685 @@
+Galaxy Interactive Environments (GIEs)
+======================================
+
+GIEs were a new feature back in Galaxy 15.05, leading with the release of the
+IPython IE. They were presented at GCC2015, and the RStudio IE was released as
+part of 15.07.
+
+A GIE is a docker container, launched by Galaxy, proxied by Galaxy, with some
+extra sugar inside the container to allow users to interact easily with their
+Galaxy histories.
+
+How GIEs Work
+-------------
+
+A GIE is primarily composed of a Docker container, and the Galaxy visualization
+component. GIE's are essentially visualization plugins which abuse the fact
+that they're rendered with mako templates. The mako templating language used in
+Viz plugins allows running python code, which in turn allows for us to launch
+docker containers. Once this container is launched, we notify a proxy built
+into Galaxy which helps coordinate a 1:1 mapping of users and their docker containers.
+
+Here's a simple diagram recapping the above:
+
+.. image:: interactive_environments.png
+
+Deploying GIEs
+--------------
+
+Deploying GIEs is not a trivial operation. They have complex interactions with
+numerous services, you'll need to be a fairly competent SysAdmin to debug all
+of the possible problems that can occur during deployment. After the initial
+hurdle, most find that GIEs require little to no maintenance.
+
+Setting up the Proxy
+^^^^^^^^^^^^^^^^^^^^
+
+Currently the Galaxy proxy is a NodeJS+Sqlite3 proxy.
+
+- Node has recently upgraded, and our proxy is pinned to an old version of
+ sqlite3. As such you'll currently need to have an older version of Node
+ available (0.10.X - 0.11.X vintage).
+- We're working on solutions in this space to provide a better deployment
+ mechanism here and fewer dependencies.
+- Please note that if you have NodeJS installed under Ubuntu, it often
+ installs to ``/usr/bin/nodejs``, whereas ``npm`` expects it to be
+ ``/usr/bin/node``. You will need to create that symlink yourself.
+
+Once Node and npm are ready to go, you'll need to install the dependencies
+
+.. code-block:: console
+
+ $ cd $GALAXY_ROOT/lib/galaxy/web/proxy/js
+ $ npm install
+
+Running ``node lib/main.js --help`` should produce some useful help text
+
+.. code-block::
+
+ Usage: main [options]
+
+ Options:
+
+ -h, --help output usage information
+ -V, --version output the version number
+ --ip Public-facing IP of the proxy
+ --port Public-facing port of the proxy
+ --cookie Cookie proving authentication
+ --sessions Routes file to monitor
+ --verbose
+
+There are two ways to handle actually running the proxy. The first is to have
+Galaxy automatically launch the proxy as needed, the second is to manage it
+with something like Supervisord. The command for launching the proxy manually
+(or via supervisord) is
+
+.. code-block:: console
+
+ $ node $GALAXY_ROOT/lib/galaxy/web/proxy/js/lib/main.js --ip 0.0.0.0 \
+ --port 8800 --sessions $GALAXY_ROOT/database/session_map.sqlite \
+ --cookie galaxysession --verbose
+
+Configuring the Proxy
+^^^^^^^^^^^^^^^^^^^^^
+
+Configuration is all managed in ``galaxy.ini``
+
+.. code-block:: ini
+
+ dynamic_proxy_manage=True
+ dynamic_proxy_session_map=database/session_map.sqlite
+ dynamic_proxy_bind_port=8800
+ dynamic_proxy_bind_ip=0.0.0.0
+ dynamic_proxy_debug=True
+
+As you can see most of these variables map directly to the command line
+arguments to the NodeJS script. There are a few extra parameters which merit
+individual discussion
+
+.. code-block:: ini
+
+ dynamic_proxy_external_proxy=True
+ dynamic_proxy_prefix=gie_proxy
+
+The first option says that you have Galaxy and the Galaxy NodeJS proxy wrapped
+in an upstream proxy like Apache or NGINX. If you're using an upstream proxy, then
+you'll need to set this to true. This will cause Galaxy to connect users to the
+same port as Galaxy is being served on (so 80/443), rather than directing them
+to ``:8800``.
+
+The second option is closely entertwined with the first option. When Galaxy is
+accessed, it sets a cookie. This cookie generally cannot be sent with requests
+to different domains and different ports, so Galaxy and the dynamic proxy must
+be accessible on the same port and protocol. A further restriction is the
+cookie path; if you're running Galaxy under a URL like
+``https://f.q.d.n/galaxy/``, the cookie is only accessible to URLs that look
+like ``https://f.q.d.n/galaxy/*``. Galaxy and the Galaxy NodeJS proxy will take
+care of most of this for you, but you should be aware of how it functions. If
+your ``galaxysession`` cookie is not available in your request to the proxy
+URL, it will through up an error and you won't be able to connect. Back on
+topic, specifically, the second option sets the URL path that's used to
+differentiate requests that should go through the proxy to those that should go
+to Galaxy. You will need to add special upstream proxy configuration to handle
+this, and you'll need to use the same ``dynamic_proxy_prefix`` in your
+``galaxy.ini`` that you use in your URL routes.
+
+Apache
+
+.. code-block:: apache
+
+ # IPython specific. Other IEs may require their own routes.
+ ProxyPass /galaxy/gie_proxy/ipython/api/kernels ws://localhost:8800/galaxy/gie_proxy/ipython/api/kernels
+
+ # Global GIE configuration
+ ProxyPass /galaxy/gie_proxy http://localhost:8800/galaxy/gie_proxy
+ ProxyPassReverse /galaxy/gie_proxy http://localhost:8800/galaxy/gie_proxy
+
+ # Normal Galaxy configuration
+ ProxyPass /galaxy http://localhost:8000/galaxy
+ ProxyPassReverse /galaxy http://localhost:8000/galaxy
+
+Please note you will need to be using apache2.4 with ``mod_proxy_wstunnel``.
+
+Nginx
+
+.. code-block::
+
+ # TODO, please PR / ping erasche on IRC if you have samples
+
+
+Docker on Another Host
+^^^^^^^^^^^^^^^^^^^^^^
+
+There are many reasons to run Interactive Environments on a separate host and
+not on your webserver, serving Galaxy. This feature has been available since
+15.07 and is used in production at the University of Freiburg.
+
+First you need to configure a second host to be Docker enabled. In the
+following we call this host ``gx-docker`` You need to start the Docker daemon
+and bind it to a TCP port, not to a socket as is the default. For example
+you can start the daemon with
+
+.. code-block:: console
+
+ $ docker -H 0.0.0.0:4243 -d
+
+On your client, the Galaxy webserver, you can now install a Docker client. This
+can also be done on older Systems like Scientific-Linux, CentOS 6, which does
+not have Docker support by default. The client just talks to the Docker daemon
+on host ``gx-docker``, and does not run anything itself, locally. You can test
+your configuration for example by starting busybox from your client on the
+Docker host with
+
+.. code-block:: console
+
+ $ docker -H tcp://gx-docker:4243 run -it busybox sh
+
+So far so good! Now we need to configure Galaxy to use our new Docker host
+to start the Interactive Environments. For that we need to edit the IPython GIE
+configuration, ``ipython.ini`` to use our custom docker host
+
+.. code-block:: ini
+
+ [main]
+
+ [docker]
+ command = docker -H tcp://gx-docker:4243 {docker_args}
+ image = bgruening/docker-ipython-notebook:dev
+ docker_hostname = gx-docker
+
+Please adapt your ``command`` and the ``image`` as needed.
+
+As next step we need to configure a share mount point between the Docker host
+and Galaxy. Unfortunately, this can not be a NFS mount. Docker does not like
+NFS yet. You could for example use a sshfs mount with the following script
+
+.. code-block:: bash
+
+ if mount | grep ^gx-docker:/var/tmp/gx-docker; then
+ echo "/var/tmp/gx-docker already mounted."
+ else
+ sshfs gx-docker:/var/tmp/gx-docker /var/tmp/gx-docker
+ echo 'Mounting ...'
+ fi
+
+This will let Galaxy and the Docker host share temporary files.
+
+Interactive Environments in Detail (and How to Build Your Own)
+--------------------------------------------------------------
+
+Unfortunately building a GIE isn't completely straightforward, and it's
+certainly not as simple as picking out an existing container and plugging it
+in. Here we'll go through build a "Hello, World" GIE which just displays a file
+from a user's history.
+
+Directory Layout
+^^^^^^^^^^^^^^^^
+
+The GIE directory layout looks identical to that of normal visualization
+plugins, for those familiar with developing those
+
+.. code-block:: console
+
+ $ tree $GALAXY_ROOT/config/plugins/interactive_environments/ipython/
+ config/plugins/interactive_environments/ipython/
+ ├── config
+ │ ├── ipython.ini
+ │ ├── ipython.ini.sample
+ │ └── ipython.xml
+ ├── static
+ │ └── js
+ │ └── ipython.js
+ └── templates
+ ├── ipython.mako
+ └── notebook.ipynb
+
+We'll use the variable ``{gie}`` to stand in for the name of your GIE. It
+should match ``[a-z]+``, like ``ipython`` or ``rstudio``. Here you can see the
+``config/`` directory with a ``{gie}.ini.sample`` providing docker and image
+configuration, and then ``{gie}.xml`` which outlines that it is a GIE.
+
+The static directory can hold resources such as javascript and css files. If
+you are actively developing a GIE, you'll need to restart Galaxy after adding
+any resources to that file, before they can be accessed in the browser.
+
+Lastly, and most importantly, there's the templates folder. This normally just
+contains ``{gie}.mako``, however the IPython file needs an extra template file.
+
+First Steps, Configuration
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+We will name our GIE "helloworld", but you are free to name your's differently.
+We'll first need to create the directory structure and set up our
+configuration
+
+.. code-block:: console
+
+ $ mkdir -p $GALAXY_ROOT/config/plugins/interactive_environments/helloworld/{config,static,templates}
+ $ cd $GALAXY_ROOT/config/plugins/interactive_environments/helloworld/
+
+Next, you'll need to create the GIE plugin XML file ``config/helloworld.xml``
+
+.. code-block:: xml
+
+
+
+
+
+
+
+ HistoryDatasetAssociation
+
+
+ tabular.Tabular
+ data.Text
+ dataset_id
+
+
+
+ dataset_id
+
+
+ helloworld.mako
+
+
+Once this is done, we can set up our INI file, ``config/helloworld.ini.sample`` which controlls docker interaction
+
+.. code-block:: ini
+
+ [main]
+ # Unused
+
+ [docker]
+ # Command to execute docker. For example `sudo docker` or `docker-lxc`.
+ #command = docker {docker_args}
+
+ # The docker image name that should be started.
+ image = hello-ie
+
+ # Additional arguments that are passed to the `docker run` command.
+ #command_inject = --sig-proxy=true -e DEBUG=false
+
+ # URL to access the Galaxy API with from the spawn Docker container, if empty
+ # this falls back to galaxy.ini's galaxy_infrastructure_url and finally to the
+ # Docker host of the spawned container if that is also not set.
+ #galaxy_url =
+
+ # The Docker hostname. It can be useful to run the Docker daemon on a different
+ # host than Galaxy.
+ #docker_hostname = localhost
+
+We've named our image ``hello-ie``, we'll get to creating that in a minute.
+
+Mako Templates
+^^^^^^^^^^^^^^
+
+Mako templates are very easy to use, and they allow significantly more
+flexibility than many other templating languages. It's because of this
+flexibility (and ability to write plain python code in them) that GIEs were
+possible to develop easily.
+
+In our ``templates/helloworld.mako``, we'll add the following
+
+.. code-block:: html+mako
+
+ <%namespace name="ie" file="ie.mako" />
+
+This line says to inherit from the ``ie.mako`` file that's available in
+``$GALAXY_ROOT/config/plugins/interactive_environments/common/templates/ie.mako``.
+Next we'll add the following
+
+.. code-block:: html+mako
+
+ <%
+ # Sets ID and sets up a lot of other variables
+ ie_request.load_deploy_config()
+
+ # Define a volume that will be mounted into the container.
+ # This is a useful way to provide access to large files in the container,
+ # if the user knows ahead of time that they will need it.
+ user_file = ie_request.volume(
+ hda.file_name, '/import/file.dat', how='ro')
+
+ # Launch the IE. This builds and runs the docker command in the background.
+ ie_request.launch(
+ volumes=[user_file],
+ env_override={
+ 'custom': '42'
+ }
+ )
+
+ # Only once the container is launched can we template our URLs. The ie_request
+ # doesn't have all of the information needed until the container is running.
+ url = ie_request.url_template('${PROXY_URL}/helloworld/')
+ %>
+
+That mako snippet loaded the configuration from the INI files, launched the
+docker container, and then built a URL to the correct endpoint, through the
+Galaxy NodeJS proxy. Additionally we've set an environment variable named ``CUSTOM`` with the value ``42`` to be passed to the container, and we've attached the dataset the user selected (available in ``hda``) to the container as a read-only volume.
+
+We'll continue appending to our ``helloworld.mako`` the HTML code that's actually displayed to the user, when this template is rendered
+
+.. code-block:: html+mako
+
+
+
+
+ ${ ie.load_default_js() }
+
+
+
+
+
+
+
+
+
+We've glossed over some of the features of this file, but most IEs do a significant amount of "magic" in the top half of the mako template. For instance, the IPython notebook:
+
+- If the user is trying to run the IPython GIE Visualization on an existing notebook in their history, then that gets loaded into the docker container via the temp directory and set as the default notebook
+- Otherwise a default notebook is built for the user.
+
+The RStudio notebook:
+
+- generates a random password and configures the image to use this password
+- Copies in an RData file if the user has loaded one
+- sets some custom environment variables.
+
+
+Connecting the User to the Container via Javascript
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+With the mako template above finished, if you were to load this in your
+browser, not a lot would happen because we haven't built the hello-ie image,
+and we haven't used Javascript to connect the user with the container. In the
+tail end of the template, we set a variable ``notebook_access_url``. These are
+partially a legacy of how things used to be done and you're welcome to clean up
+your code according to your desires. Galaxy's NodeJS proxy handles the
+authentication of users, so you don't have to worry about it, and can just
+assume that only the correct user will have access to a given notebook.
+
+In the ``static/`` directory, we generally create a ``js/`` directory below that, and create a ``{gie}.js`` (so, ``static/js/helloworld.js``) file in there. That file will have a function, ``load_notebook`` which will check if the GIE is available, and when it is, display it to the user.
+
+We start by writing the load notebook function, which is pretty generic
+
+.. code-block:: javascript
+
+ // Load an interactive environment (IE) from a remote URL
+ // @param {String} notebook_access_url: the URL embeded in the page and loaded
+ function load_notebook(notebook_access_url){
+ // When the page has completely loaded...
+ $( document ).ready(function() {
+ // Test if we can access the GIE, and if so, execute the function
+ // to load the GIE for the user.
+ test_ie_availability(notebook_access_url, function(){
+ _handle_notebook_loading(notebook_access_url);
+ });
+ });
+ }
+
+This function will display a spinner to the user to indicate process, and then make multiple requests to ``notebook_access_url``. That MUST return a 200 OK for the ``_handle_notebook_loading`` function to ever be called. 302s do not count!
+
+With that, we've almost completed the Javascript portion, just need to implement the function to display the GIE to the user in an iframe
+
+.. code-block:: javascript
+
+ function _handle_notebook_loading(notebook_access_url){
+ append_notebook(notebook_access_url);
+ }
+
+
+This function is very short. Historically, the GIE process involved a complex dance of:
+
+- generating a random password in the mako template
+- setting it as a javascript variable
+- passing it to the docker container
+- once the container was available, have the javascript automatically log a
+ user in (something browsers try to prevent since that's otherwise an XSS
+ vulnerability.)
+- hope everything worked
+
+Since the NodeJS proxy takes care of authentication/authorization, we can
+reduce the helloworld ``_handle_notebook_loading`` function to a simple
+``append_notebook`` call. You may wish to look at the IPython and RStudio GIEs
+for examples of the complex things that can be done at every step.
+
+The GIE Container
+^^^^^^^^^^^^^^^^^
+
+We'll build a simple container that just displays the dataset a user has
+selected to them. Remember when we attached a volume to the container? We'll
+make use of that now.
+
+GIE Containers (often) consist of:
+
+- Dockerfile
+- NGINX Proxy configuration
+- A custom startup script/entrypoint
+- A script to monitor traffic and kill unused containers
+
+We have to monitor the container's traffic and kill off unused containers,
+bceause no one is watching them. The user launches the container in Galaxy, and
+Galaxy immediately forgets the container exists. Thus, we say that if a
+container has no connections to TCP connections to itself, then it should
+commit suicide by killing the root process.
+
+Here's an example Dockerfile for our helloworld container
+
+.. code-block:: dockerfile
+
+ FROM ubuntu:14.04
+ # These environment variables are passed from Galaxy to the container
+ # and help you enable connectivity to Galaxy from within the container.
+ # This means your user can import/export data from/to Galaxy.
+ ENV DEBIAN_FRONTEND=noninteractive \
+ API_KEY=none \
+ DEBUG=false \
+ PROXY_PREFIX=none \
+ GALAXY_URL=none \
+ GALAXY_WEB_PORT=10000 \
+ HISTORY_ID=none \
+ REMOTE_HOST=none
+
+ RUN apt-get -qq update && \
+ apt-get install --no-install-recommends -y \
+ wget procps nginx python python-pip net-tools nginx
+
+ # Our very important scripts. Make sure you've run `chmod +x startup.sh
+ # monitor_traffic.sh` outside of the container!
+ ADD ./startup.sh /startup.sh
+ ADD ./monitor_traffic.sh /monitor_traffic.sh
+
+ # /import will be the universal mount-point for IPython
+ # The Galaxy instance can copy in data that needs to be present to the
+ # container
+ RUN mkdir /import
+
+ # Nginx configuration
+ COPY ./proxy.conf /proxy.conf
+
+ VOLUME ["/import"]
+ WORKDIR /import/
+
+ # EXTREMELY IMPORTANT! You must expose a SINGLE port on your container.
+ EXPOSE 80
+ CMD /startup.sh
+
+If you have questions on this, please feel free to contact us on IRC
+(`irc.freenode.net#galaxyproject `__).
+
+The proxy configuration is interesting, here we'll point NGINX to reverse proxy
+a service running on ``:8000`` inside the container. That port will be hosting
+a python process which serves up the directory contents of ``/import``, i.e.
+the file the user selected which was mounted as a volume into
+``/import/file.dat``
+
+.. code-block:: nginx
+
+ server {
+ listen 80;
+ server_name localhost;
+ access_log /var/log/nginx/localhost.access.log;
+
+ # Note the trailing slash used everywhere!
+ location PROXY_PREFIX/helloworld/ {
+ proxy_buffering off;
+ proxy_pass http://127.0.0.1:8000/;
+ proxy_redirect http://127.0.0.1:8000/ PROXY_PREFIX/helloworld/;
+ }
+ }
+
+
+And here we'll run that service in our ``startup.sh`` file
+
+.. code-block:: bash
+
+ #!/bin/bash
+ # First, replace the PROXY_PREFIX value in /proxy.conf with the value from
+ # the environment variable.
+ sed -i "s|PROXY_PREFIX|${PROXY_PREFIX}|" /proxy.conf;
+ # Then copy into the default location for ubuntu+nginx
+ cp /proxy.conf /etc/nginx/sites-enabled/default;
+
+ # Here you would normally start whatever service you want to start. In our
+ # example we start a simple directory listing service on port 8000
+ cd /import/ && python -mSimpleHTTPServer &
+
+ # Launch traffic monitor which will automatically kill the container if
+ # traffic stops
+ /monitor_traffic.sh &
+ # And finally launch nginx in foreground mode. This will make debugging
+ # easier as logs will be available from `docker logs ...`
+ nginx -g 'daemon off;'
+
+Lastly, our ``monitor_traffic.sh`` file is often re-used between containers, the only adjustment being the port that is looked at
+
+.. code-block:: bash
+
+ #!/bin/bash
+ while true; do
+ sleep 60
+ if [ `netstat -t | grep -v CLOSE_WAIT | grep ':80' | wc -l` -lt 3 ]
+ then
+ pkill nginx
+ fi
+ done
+
+With those files, ``monitor_traffic.sh``, ``Dockerfile``, ``startup.sh``, and ``proxy.conf``, you should be able to build your ``hello-ie`` container
+
+.. code-block:: bash
+
+ $ cd hello-ie
+ $ docker build -t hello-ie .
+
+Now, if everything went smoothly, you should be able to restart Galaxy and try out your new GIE on a tabular or text file!
+
+Debugging
+^^^^^^^^^
+
+When you launch your new GIE in Galaxy, your Galaxy logs should show something like the following:
+
+.. code-block:: console
+
+ Starting docker container for IE helloworld with command [docker run --sig-proxy=true -e DEBUG=false -e "GALAXY_URL=http://localhost/galaxy/" -e "CORS_ORIGIN=http://localhost" -e "GALAXY_WEB_PORT=8000" -e "HISTORY_ID=f2db41e1fa331b3e" -e "CUSTOM=42" -e "GALAXY_PASTER_PORT=8000" -e "PROXY_PREFIX=/galaxy/gie_proxy" -e "API_KEY=1712364174a0ff79b34e9a78fee3ca1c" -e "REMOTE_HOST=127.0.0.1" -e "USER_EMAIL=hxr@local.host" -d -P -v "/home/hxr/work/galaxy/database/tmp/tmp5HaqZy:/import/" -v "/home/hxr/work/galaxy/database/files/000/dataset_68.dat:/import/file.dat:ro" hello-ie]
+
+Here's the docker command written out in a more readable manner:
+
+.. code-block:: console
+
+ $ docker run --sig-proxy=true \
+ -d -P \
+ -e "API_KEY=1712364174a0ff79b34e9a78fee3ca1c" \
+ -e "CORS_ORIGIN=http://localhost" \
+ -e "CUSTOM=42" \
+ -e "DEBUG=false" \
+ -e "GALAXY_PASTER_PORT=8000" \
+ -e "GALAXY_URL=http://localhost/galaxy/" \
+ -e "GALAXY_WEB_PORT=8000" \
+ -e "HISTORY_ID=f2db41e1fa331b3e" \
+ -e "PROXY_PREFIX=/galaxy/gie_proxy" \
+ -e "REMOTE_HOST=127.0.0.1" \
+ -e "USER_EMAIL=hxr@local.host" \
+ -v "/home/hxr/work/galaxy/database/tmp/tmp5HaqZy:/import/" \
+ -v "/home/hxr/work/galaxy/database/files/000/dataset_68.dat:/import/file.dat:ro" \
+ hello-ie
+
+As you can see, a LOT is going on! We'll break it down further:
+
+- ``-d`` runs the container in daemon mode, it launches and the client
+ submitting the container finished
+- ``-P`` randomly assigns an unused port to the container for each ``EXPOSE``d
+ port from our ``Dockerfile``. This is why you must expose a port, and only
+ one port.
+- A large number of environment variables are set:
+
+ - The user's API key is provided, allowing you to access datasets and
+ submit jobs on their behalf. If you have an environment like
+ IPython/RStudio, it is **highly recommended** that you provide some magic
+ by which the user can use their API key without embedding it in the
+ ntoebook. If you do embed it somehow in a document that gets saved to
+ their history, anyone can impersonate that user if they get a hold of it.
+ In the IPython GIE we have a variable that just runs
+ ``os.environ.get('API_KEY')`` to avoid embedding it in their notebook.
+ - A CORS Origin is provided for very strict servers, but it may be easier
+ to simply void CORS requirements within the nginx proxy in your
+ container.
+ - Custom variables specified in your ``launch()`` command are available
+ - A ``DEBUG`` environment variable should be used to help admins debug
+ existing containers. You should use it to increase logging, not cleanup
+ temporary files, and so on.
+ - ``GALAXY_PASTER_PORT`` (deprecated) and ``GALAXY_WEB_PORT`` are the raw
+ port that Galaxy is listening on. You can use this to help decide how to
+ takl to Galaxy.
+ - ``GALAXY_URL`` is the URL that Galaxy should be accessible at. For
+ various reasons this may not be true. We recommend looking at our
+ implementation of `galaxy.py
+ `__
+ which is a small utility script to provide API access to Galaxy to get
+ and fetch data, based on those environment variables.
+ - The ``HISTORY_ID`` of the current history the user is on is provided. In
+ the IPython/RStudio containers, we provide a dead simple method for users
+ to download datasets from their current history which will be visible to
+ them on the right hand side of their screen.
+ - A ``PROXY_PREFIX`` is provided which should be used in the nginx conf.
+ - ``REMOTE_HOST`` is another component used to test for a possible Galaxy
+ access path
+ - The user's email is made available, very convenient for webservices like
+ Entrez which require the user's email address. You can pre-fill it out
+ for them, making their life easier.
+ - Two volumes are mounted, one a temporary directory from Galaxy (rw), and one
+ the dataset the user selected (ro).
+
+- and finally the image is specified.
+
+Most of this information is usually required to build friendly, easy-to-use
+GIEs. One of the strong points of GIEs is their magic interaction with Galaxy.
+Here we've mounted a volume read-only, but in real life you may wish to provide
+connectivity like IPython and RStudio provide, allowing the user to load
+datasets on demand for interactive analysis, and then to store analysis
+artefacts (and a log of what was done inside the container, Ã la IPython's
+"notebooks") back to their current history.
+
+If everything went well, at this point you should see a directory listing show up:
+
+.. image:: interactive_environments_success.png
+
+If you find yourself encountering difficulties, the "Hello, World" IE is
+available in a `GitHub repo `__, and there are people on the IRC channel who can help debug.
diff --git a/doc/source/admin/interactive_environments.svg b/doc/source/admin/interactive_environments.svg
new file mode 100644
index 00000000000..564913e6bc5
--- /dev/null
+++ b/doc/source/admin/interactive_environments.svg
@@ -0,0 +1,2536 @@
+
+
+
+
diff --git a/doc/source/admin/interactive_environments_success.png b/doc/source/admin/interactive_environments_success.png
new file mode 100644
index 00000000000..2f881909379
Binary files /dev/null and b/doc/source/admin/interactive_environments_success.png differ
diff --git a/doc/source/dev/faq.rst b/doc/source/dev/faq.rst
index bc4be9a5a7d..d8883d7e82c 100644
--- a/doc/source/dev/faq.rst
+++ b/doc/source/dev/faq.rst
@@ -7,6 +7,6 @@ to provide relatively concrete answers for specific tool development scenarios.
... interact with the Galaxy codebase interactively?
----------------------------------------------------
-This can be done with either IPython/Jupyter or a plain Python console, depending on your preferences::
+This can be done with either IPython/Jupyter or a plain python console, depending on your preferences::
python -i scripts/db_shell.py
diff --git a/doc/source/index.rst b/doc/source/index.rst
index 04b95d047e4..16de4d33ac0 100644
--- a/doc/source/index.rst
+++ b/doc/source/index.rst
@@ -38,6 +38,8 @@ Contents
Developer Documentation
+ Special topics in Administration
+
Indices and tables
==================
diff --git a/doc/source/lib/galaxy.jobs.runners.lwr_client.rst b/doc/source/lib/galaxy.jobs.runners.lwr_client.rst
deleted file mode 100644
index 1c0be4e8fac..00000000000
--- a/doc/source/lib/galaxy.jobs.runners.lwr_client.rst
+++ /dev/null
@@ -1,132 +0,0 @@
-galaxy.jobs.runners.lwr_client package
-======================================
-
-.. automodule:: galaxy.jobs.runners.lwr_client
- :members:
- :undoc-members:
- :show-inheritance:
-
-Subpackages
------------
-
-.. toctree::
-
- galaxy.jobs.runners.lwr_client.staging
- galaxy.jobs.runners.lwr_client.transport
-
-Submodules
-----------
-
-galaxy.jobs.runners.lwr_client.action_mapper module
----------------------------------------------------
-
-.. automodule:: galaxy.jobs.runners.lwr_client.action_mapper
- :members:
- :undoc-members:
- :show-inheritance:
-
-galaxy.jobs.runners.lwr_client.amqp_exchange module
----------------------------------------------------
-
-.. automodule:: galaxy.jobs.runners.lwr_client.amqp_exchange
- :members:
- :undoc-members:
- :show-inheritance:
-
-galaxy.jobs.runners.lwr_client.amqp_exchange_factory module
------------------------------------------------------------
-
-.. automodule:: galaxy.jobs.runners.lwr_client.amqp_exchange_factory
- :members:
- :undoc-members:
- :show-inheritance:
-
-galaxy.jobs.runners.lwr_client.client module
---------------------------------------------
-
-.. automodule:: galaxy.jobs.runners.lwr_client.client
- :members:
- :undoc-members:
- :show-inheritance:
-
-galaxy.jobs.runners.lwr_client.config_util module
--------------------------------------------------
-
-.. automodule:: galaxy.jobs.runners.lwr_client.config_util
- :members:
- :undoc-members:
- :show-inheritance:
-
-galaxy.jobs.runners.lwr_client.decorators module
-------------------------------------------------
-
-.. automodule:: galaxy.jobs.runners.lwr_client.decorators
- :members:
- :undoc-members:
- :show-inheritance:
-
-galaxy.jobs.runners.lwr_client.destination module
--------------------------------------------------
-
-.. automodule:: galaxy.jobs.runners.lwr_client.destination
- :members:
- :undoc-members:
- :show-inheritance:
-
-galaxy.jobs.runners.lwr_client.interface module
------------------------------------------------
-
-.. automodule:: galaxy.jobs.runners.lwr_client.interface
- :members:
- :undoc-members:
- :show-inheritance:
-
-galaxy.jobs.runners.lwr_client.job_directory module
----------------------------------------------------
-
-.. automodule:: galaxy.jobs.runners.lwr_client.job_directory
- :members:
- :undoc-members:
- :show-inheritance:
-
-galaxy.jobs.runners.lwr_client.manager module
----------------------------------------------
-
-.. automodule:: galaxy.jobs.runners.lwr_client.manager
- :members:
- :undoc-members:
- :show-inheritance:
-
-galaxy.jobs.runners.lwr_client.object_client module
----------------------------------------------------
-
-.. automodule:: galaxy.jobs.runners.lwr_client.object_client
- :members:
- :undoc-members:
- :show-inheritance:
-
-galaxy.jobs.runners.lwr_client.path_mapper module
--------------------------------------------------
-
-.. automodule:: galaxy.jobs.runners.lwr_client.path_mapper
- :members:
- :undoc-members:
- :show-inheritance:
-
-galaxy.jobs.runners.lwr_client.setup_handler module
----------------------------------------------------
-
-.. automodule:: galaxy.jobs.runners.lwr_client.setup_handler
- :members:
- :undoc-members:
- :show-inheritance:
-
-galaxy.jobs.runners.lwr_client.util module
-------------------------------------------
-
-.. automodule:: galaxy.jobs.runners.lwr_client.util
- :members:
- :undoc-members:
- :show-inheritance:
-
-
diff --git a/doc/source/lib/galaxy.jobs.runners.lwr_client.staging.rst b/doc/source/lib/galaxy.jobs.runners.lwr_client.staging.rst
deleted file mode 100644
index 7ae6a269d3a..00000000000
--- a/doc/source/lib/galaxy.jobs.runners.lwr_client.staging.rst
+++ /dev/null
@@ -1,28 +0,0 @@
-galaxy.jobs.runners.lwr_client.staging package
-==============================================
-
-.. automodule:: galaxy.jobs.runners.lwr_client.staging
- :members:
- :undoc-members:
- :show-inheritance:
-
-Submodules
-----------
-
-galaxy.jobs.runners.lwr_client.staging.down module
---------------------------------------------------
-
-.. automodule:: galaxy.jobs.runners.lwr_client.staging.down
- :members:
- :undoc-members:
- :show-inheritance:
-
-galaxy.jobs.runners.lwr_client.staging.up module
-------------------------------------------------
-
-.. automodule:: galaxy.jobs.runners.lwr_client.staging.up
- :members:
- :undoc-members:
- :show-inheritance:
-
-
diff --git a/doc/source/lib/galaxy.jobs.runners.lwr_client.transport.rst b/doc/source/lib/galaxy.jobs.runners.lwr_client.transport.rst
deleted file mode 100644
index f884599ba2d..00000000000
--- a/doc/source/lib/galaxy.jobs.runners.lwr_client.transport.rst
+++ /dev/null
@@ -1,28 +0,0 @@
-galaxy.jobs.runners.lwr_client.transport package
-================================================
-
-.. automodule:: galaxy.jobs.runners.lwr_client.transport
- :members:
- :undoc-members:
- :show-inheritance:
-
-Submodules
-----------
-
-galaxy.jobs.runners.lwr_client.transport.curl module
-----------------------------------------------------
-
-.. automodule:: galaxy.jobs.runners.lwr_client.transport.curl
- :members:
- :undoc-members:
- :show-inheritance:
-
-galaxy.jobs.runners.lwr_client.transport.standard module
---------------------------------------------------------
-
-.. automodule:: galaxy.jobs.runners.lwr_client.transport.standard
- :members:
- :undoc-members:
- :show-inheritance:
-
-
diff --git a/doc/source/lib/galaxy.jobs.runners.rst b/doc/source/lib/galaxy.jobs.runners.rst
index 30c2f598717..cdb20be1e44 100644
--- a/doc/source/lib/galaxy.jobs.runners.rst
+++ b/doc/source/lib/galaxy.jobs.runners.rst
@@ -11,7 +11,6 @@ Subpackages
.. toctree::
- galaxy.jobs.runners.lwr_client
galaxy.jobs.runners.state_handlers
galaxy.jobs.runners.util
@@ -50,13 +49,6 @@ galaxy.jobs.runners.local module
:undoc-members:
:show-inheritance:
-galaxy.jobs.runners.lwr module
-------------------------------
-
-.. automodule:: galaxy.jobs.runners.lwr
- :members:
- :undoc-members:
- :show-inheritance:
galaxy.jobs.runners.pbs module
------------------------------
diff --git a/lib/galaxy/config.py b/lib/galaxy/config.py
index 89df234fd90..2955d7fa0ad 100644
--- a/lib/galaxy/config.py
+++ b/lib/galaxy/config.py
@@ -171,6 +171,7 @@ class Configuration( object ):
self.outputs_to_working_directory = string_as_bool( kwargs.get( 'outputs_to_working_directory', False ) )
self.output_size_limit = int( kwargs.get( 'output_size_limit', 0 ) )
self.retry_job_output_collection = int( kwargs.get( 'retry_job_output_collection', 0 ) )
+ self.check_job_script_integrity = string_as_bool( kwargs.get( "check_job_script_integrity", True ) )
self.job_walltime = kwargs.get( 'job_walltime', None )
self.job_walltime_delta = None
if self.job_walltime is not None:
@@ -423,6 +424,9 @@ class Configuration( object ):
self.new_lib_browse = string_as_bool( kwargs.get( 'new_lib_browse', False ) )
# Error logging with sentry
self.sentry_dsn = kwargs.get( 'sentry_dsn', None )
+ # Statistics and profiling with statsd
+ self.statsd_host = kwargs.get( 'statsd_host', '')
+ self.statsd_port = int( kwargs.get( 'statsd_port', 8125 ) )
# Logging with fluentd
self.fluent_log = string_as_bool( kwargs.get( 'fluent_log', False ) )
self.fluent_host = kwargs.get( 'fluent_host', 'localhost' )
@@ -442,6 +446,7 @@ class Configuration( object ):
self.dynamic_proxy_bind_port = int( kwargs.get( "dynamic_proxy_bind_port", "8800" ) )
self.dynamic_proxy_bind_ip = kwargs.get( "dynamic_proxy_bind_ip", "0.0.0.0" )
self.dynamic_proxy_external_proxy = string_as_bool( kwargs.get( "dynamic_proxy_external_proxy", "False" ) )
+ self.dynamic_proxy_prefix = kwargs.get( "dynamic_proxy_prefix", "gie_proxy" )
# Default chunk size for chunkable datatypes -- 64k
self.display_chunk_size = int( kwargs.get( 'display_chunk_size', 65536) )
diff --git a/lib/galaxy/dataset_collections/matching.py b/lib/galaxy/dataset_collections/matching.py
index 88763df069c..3398aec4a65 100644
--- a/lib/galaxy/dataset_collections/matching.py
+++ b/lib/galaxy/dataset_collections/matching.py
@@ -1,6 +1,6 @@
from galaxy.util import bunch
from galaxy import exceptions
-from .structure import get_structure
+from .structure import get_structure, leaf
CANNOT_MATCH_ERROR_MESSAGE = "Cannot match collection types."
@@ -13,10 +13,11 @@ class CollectionsToMatch( object ):
def __init__( self ):
self.collections = {}
- def add( self, input_name, hdca, subcollection_type=None ):
+ def add( self, input_name, hdca, subcollection_type=None, linked=True ):
self.collections[ input_name ] = bunch.Bunch(
hdca=hdca,
subcollection_type=subcollection_type,
+ linked=linked,
)
def has_collections( self ):
@@ -37,21 +38,32 @@ class MatchingCollections( object ):
"""
def __init__( self ):
- self.structure = None
+ self.linked_structure = None
+ self.unlinked_structures = []
self.collections = {}
- def __attempt_add_to_match( self, input_name, hdca, collection_type_description, subcollection_type ):
+ def __attempt_add_to_linked_match( self, input_name, hdca, collection_type_description, subcollection_type ):
structure = get_structure( hdca, collection_type_description, leaf_subcollection_type=subcollection_type )
- if not self.structure:
- self.structure = structure
+ if not self.linked_structure:
+ self.linked_structure = structure
self.collections[ input_name ] = hdca
else:
- if not self.structure.can_match( structure ):
+ if not self.linked_structure.can_match( structure ):
raise exceptions.MessageException( CANNOT_MATCH_ERROR_MESSAGE )
self.collections[ input_name ] = hdca
def slice_collections( self ):
- return self.structure.walk_collections( self.collections )
+ return self.linked_structure.walk_collections( self.collections )
+
+ @property
+ def structure( self ):
+ """Yield cross product of all unlinked datasets to linked dataset."""
+ effective_structure = leaf
+ for unlinked_structure in self.unlinked_structures:
+ effective_structure = effective_structure.multiply( unlinked_structure )
+ linked_structure = self.linked_structure or leaf
+ effective_structure = effective_structure.multiply( linked_structure )
+ return None if effective_structure.is_leaf else effective_structure
@staticmethod
def for_collections( collections_to_match, collection_type_descriptions ):
@@ -61,8 +73,13 @@ class MatchingCollections( object ):
matching_collections = MatchingCollections()
for input_key, to_match in collections_to_match.iteritems():
hdca = to_match.hdca
- subcollection_type = to_match.subcollection_type
collection_type_description = collection_type_descriptions.for_collection_type( hdca.collection.collection_type )
- matching_collections.__attempt_add_to_match( input_key, hdca, collection_type_description, subcollection_type )
+ subcollection_type = to_match.subcollection_type
+
+ if to_match.linked:
+ matching_collections.__attempt_add_to_linked_match( input_key, hdca, collection_type_description, subcollection_type )
+ else:
+ structure = get_structure( hdca, collection_type_description, leaf_subcollection_type=subcollection_type )
+ matching_collections.unlinked_structures.append( structure )
return matching_collections
diff --git a/lib/galaxy/dataset_collections/structure.py b/lib/galaxy/dataset_collections/structure.py
index b08d62748bc..8f1eab5a020 100644
--- a/lib/galaxy/dataset_collections/structure.py
+++ b/lib/galaxy/dataset_collections/structure.py
@@ -15,23 +15,34 @@ class Leaf( object ):
def is_leaf( self ):
return True
+ def clone( self ):
+ return self
+
+ def multiply( self, other_structure ):
+ return other_structure.clone()
+
+
leaf = Leaf()
class Tree( object ):
- def __init__( self, dataset_collection, collection_type_description ):
+ def __init__( self, children, collection_type_description ):
+ self.children = children
self.collection_type_description = collection_type_description
+
+ @staticmethod
+ def for_dataset_collection( dataset_collection, collection_type_description ):
children = []
for element in dataset_collection.elements:
if collection_type_description.has_subcollections():
child_collection = element.child_collection
subcollection_type_description = collection_type_description.subcollection_type_description() # Type description of children
- children.append( ( element.element_identifier, Tree( child_collection, collection_type_description=subcollection_type_description ) ) )
+ tree = Tree.for_dataset_collection( child_collection, collection_type_description=subcollection_type_description )
+ children.append( ( element.element_identifier, tree ) )
else:
children.append( ( element.element_identifier, leaf ) )
-
- self.children = children
+ return Tree( children, collection_type_description )
def walk_collections( self, hdca_dict ):
return self._walk_collections( dict_map( lambda hdca: hdca.collection, hdca_dict ) )
@@ -96,6 +107,21 @@ class Tree( object ):
element_identifiers=element_identifiers,
)
+ def multiply( self, other_structure ):
+ if other_structure.is_leaf:
+ return self.clone()
+
+ new_collection_type = self.collection_type_description.multiply( other_structure.collection_type_description )
+ new_children = []
+ for (identifier, structure) in self.children:
+ new_children.append( (identifier, structure.multiply( other_structure ) ) )
+
+ return Tree( new_children, new_collection_type )
+
+ def clone( self ):
+ cloned_children = map( lambda (identifier, element): (identifier, element.clone()), self.children )
+ return Tree( cloned_children, self.collection_type_description )
+
def dict_map( func, input_dict ):
return dict( [ ( k, func(v) ) for k, v in input_dict.iteritems() ] )
@@ -105,4 +131,4 @@ def get_structure( dataset_collection_instance, collection_type_description, lea
if leaf_subcollection_type:
collection_type_description = collection_type_description.effective_collection_type_description( leaf_subcollection_type )
- return Tree( dataset_collection_instance.collection, collection_type_description )
+ return Tree.for_dataset_collection( dataset_collection_instance.collection, collection_type_description )
diff --git a/lib/galaxy/dataset_collections/type_description.py b/lib/galaxy/dataset_collections/type_description.py
index 8d0aa8cbbb3..0bf85b3e367 100644
--- a/lib/galaxy/dataset_collections/type_description.py
+++ b/lib/galaxy/dataset_collections/type_description.py
@@ -102,6 +102,10 @@ class CollectionTypeDescription( object ):
def rank_type_plugin( self ):
return self.collection_type_description_factory.type_registry.get( self.rank_collection_type() )
+ def multiply( self, other_collection_type ):
+ collection_type = map_over_collection_type( self, other_collection_type )
+ return self.collection_type_description_factory.for_collection_type( collection_type )
+
def __str__( self ):
return "CollectionTypeDescription[%s]" % self.collection_type
diff --git a/lib/galaxy/datatypes/binary.py b/lib/galaxy/datatypes/binary.py
index ee20c164075..e5a7f54cca6 100644
--- a/lib/galaxy/datatypes/binary.py
+++ b/lib/galaxy/datatypes/binary.py
@@ -33,11 +33,11 @@ class Binary( data.Data ):
@staticmethod
def register_sniffable_binary_format(data_type, ext, type_class):
- Binary.sniffable_binary_formats.append({"type": data_type, "ext": ext, "class": type_class})
+ Binary.sniffable_binary_formats.append({"type": data_type, "ext": ext.lower(), "class": type_class})
@staticmethod
def register_unsniffable_binary_ext(ext):
- Binary.unsniffable_binary_formats.append(ext)
+ Binary.unsniffable_binary_formats.append(ext.lower())
@staticmethod
def is_sniffable_binary( filename ):
diff --git a/lib/galaxy/datatypes/converters/bgzip.py b/lib/galaxy/datatypes/converters/bgzip.py
index 2eb5c063292..5d2547019f8 100644
--- a/lib/galaxy/datatypes/converters/bgzip.py
+++ b/lib/galaxy/datatypes/converters/bgzip.py
@@ -5,10 +5,9 @@ Uses pysam to bgzip a file
usage: %prog in_file out_file
"""
-
-import tempfile
import optparse
import subprocess
+import tempfile
from pysam import ctabix
diff --git a/lib/galaxy/datatypes/converters/gff_to_interval_index_converter.py b/lib/galaxy/datatypes/converters/gff_to_interval_index_converter.py
index 1b82489c235..eaca2e6240d 100644
--- a/lib/galaxy/datatypes/converters/gff_to_interval_index_converter.py
+++ b/lib/galaxy/datatypes/converters/gff_to_interval_index_converter.py
@@ -6,15 +6,15 @@ Convert from GFF file to interval index file.
usage:
python gff_to_interval_index_converter.py [input] [output]
"""
-
from __future__ import division
-import sys
import fileinput
+import sys
-from galaxy.datatypes.util.gff_util import GFFReaderWrapper, GenomicInterval, convert_gff_coords_to_bed
from bx.interval_index_file import Indexes
+from galaxy.datatypes.util.gff_util import convert_gff_coords_to_bed, GenomicInterval, GFFReaderWrapper
+
def main():
# Arguments
diff --git a/lib/galaxy/datatypes/converters/interval_to_bed_converter.py b/lib/galaxy/datatypes/converters/interval_to_bed_converter.py
index 241a03f57d5..6942449d3b2 100644
--- a/lib/galaxy/datatypes/converters/interval_to_bed_converter.py
+++ b/lib/galaxy/datatypes/converters/interval_to_bed_converter.py
@@ -1,7 +1,7 @@
#!/usr/bin/env python
# Dan Blankenberg
-
import sys
+
import bx.intervals.io
assert sys.version_info[:2] >= ( 2, 4 )
diff --git a/lib/galaxy/datatypes/converters/interval_to_bedstrict_converter.py b/lib/galaxy/datatypes/converters/interval_to_bedstrict_converter.py
index 33175022609..87ef467807f 100644
--- a/lib/galaxy/datatypes/converters/interval_to_bedstrict_converter.py
+++ b/lib/galaxy/datatypes/converters/interval_to_bedstrict_converter.py
@@ -1,7 +1,7 @@
#!/usr/bin/env python
# Dan Blankenberg
-
import sys
+
import bx.intervals.io
assert sys.version_info[:2] >= ( 2, 4 )
diff --git a/lib/galaxy/datatypes/converters/interval_to_coverage.py b/lib/galaxy/datatypes/converters/interval_to_coverage.py
index fce5faa3174..1d367ccb5fb 100644
--- a/lib/galaxy/datatypes/converters/interval_to_coverage.py
+++ b/lib/galaxy/datatypes/converters/interval_to_coverage.py
@@ -6,12 +6,13 @@ usage: %prog bed_file out_file
-1, --cols1=N,N,N,N: Columns for chrom, start, end, strand in interval file
-2, --cols2=N,N,N,N: Columns for chrom, start, end, strand in coverage file
"""
-from bx.intervals import io
-from bx.cookbook import doc_optparse
import commands
-from os import environ
import tempfile
from bisect import bisect
+from os import environ
+
+from bx.cookbook import doc_optparse
+from bx.intervals import io
INTERVAL_METADATA = ('chromCol',
'startCol',
diff --git a/lib/galaxy/datatypes/converters/interval_to_fli.py b/lib/galaxy/datatypes/converters/interval_to_fli.py
index 2501022516c..94d94601583 100644
--- a/lib/galaxy/datatypes/converters/interval_to_fli.py
+++ b/lib/galaxy/datatypes/converters/interval_to_fli.py
@@ -14,8 +14,10 @@ where location is formatted as:
and symbols are sorted in lexigraphical order.
'''
import optparse
+
from bx.tabular.io import Comment
-from galaxy.datatypes.util.gff_util import GFFReaderWrapper, read_unordered_gtf, convert_gff_coords_to_bed
+
+from galaxy.datatypes.util.gff_util import convert_gff_coords_to_bed, GFFReaderWrapper, read_unordered_gtf
def main():
diff --git a/lib/galaxy/datatypes/converters/interval_to_tabix_converter.py b/lib/galaxy/datatypes/converters/interval_to_tabix_converter.py
index a36e6ae7f9d..dc2fa1f938e 100644
--- a/lib/galaxy/datatypes/converters/interval_to_tabix_converter.py
+++ b/lib/galaxy/datatypes/converters/interval_to_tabix_converter.py
@@ -6,10 +6,9 @@ Supported presets: bed, gff, vcf
usage: %prog in_file out_file
"""
-
-import sys
-import os
import optparse
+import os
+import sys
from pysam import ctabix
diff --git a/lib/galaxy/datatypes/converters/maf_to_fasta_converter.py b/lib/galaxy/datatypes/converters/maf_to_fasta_converter.py
index 29f13ade850..068cf8584bc 100644
--- a/lib/galaxy/datatypes/converters/maf_to_fasta_converter.py
+++ b/lib/galaxy/datatypes/converters/maf_to_fasta_converter.py
@@ -1,8 +1,9 @@
#!/usr/bin/env python
# Dan Blankenberg
-
import sys
+
import bx.align.maf
+
from galaxy.tools.util import maf_utilities
assert sys.version_info[:2] >= ( 2, 4 )
diff --git a/lib/galaxy/datatypes/converters/maf_to_interval_converter.py b/lib/galaxy/datatypes/converters/maf_to_interval_converter.py
index 762bb4ce051..812fdeac25f 100644
--- a/lib/galaxy/datatypes/converters/maf_to_interval_converter.py
+++ b/lib/galaxy/datatypes/converters/maf_to_interval_converter.py
@@ -1,8 +1,9 @@
#!/usr/bin/env python
# Dan Blankenberg
-
import sys
+
import bx.align.maf
+
from galaxy.tools.util import maf_utilities
assert sys.version_info[:2] >= ( 2, 4 )
diff --git a/lib/galaxy/datatypes/converters/vcf_to_interval_index_converter.py b/lib/galaxy/datatypes/converters/vcf_to_interval_index_converter.py
index c4d4bc51aef..076cd6e2f0b 100644
--- a/lib/galaxy/datatypes/converters/vcf_to_interval_index_converter.py
+++ b/lib/galaxy/datatypes/converters/vcf_to_interval_index_converter.py
@@ -3,13 +3,14 @@
"""
Convert from VCF file to interval index file.
"""
-
from __future__ import division
import optparse
-import galaxy_utils.sequence.vcf
+
from bx.interval_index_file import Indexes
+import galaxy_utils.sequence.vcf
+
def main():
# Read options, args.
diff --git a/lib/galaxy/datatypes/converters/wiggle_to_simple_converter.py b/lib/galaxy/datatypes/converters/wiggle_to_simple_converter.py
index 8cd2aeeaebf..a409745aff8 100644
--- a/lib/galaxy/datatypes/converters/wiggle_to_simple_converter.py
+++ b/lib/galaxy/datatypes/converters/wiggle_to_simple_converter.py
@@ -7,7 +7,9 @@ Read a wiggle track and print out a series of lines containing
and fixedStep wiggle lines.
"""
import sys
+
import bx.wiggle
+
from galaxy.tools.exception_handling import UCSCOutWrapper, UCSCLimitException
diff --git a/lib/galaxy/datatypes/genetics.py b/lib/galaxy/datatypes/genetics.py
index 61ae72f70fb..161849f2591 100644
--- a/lib/galaxy/datatypes/genetics.py
+++ b/lib/galaxy/datatypes/genetics.py
@@ -163,12 +163,22 @@ class GenomeGraphs( Tabular ):
def sniff( self, filename ):
"""
Determines whether the file is in gg format
+
+ >>> from galaxy.datatypes.sniff import get_test_fname
+ >>> fname = get_test_fname( 'test_space.txt' )
+ >>> GenomeGraphs().sniff( fname )
+ False
+ >>> fname = get_test_fname( '1.gg' )
+ >>> GenomeGraphs().sniff( fname )
+ True
"""
f = open(filename, 'r')
f.readline() # header
- rows = [f.readline().split()[1:] for x in range(3)] # small sample
- # headers = get_headers( filename, '\t' )
+ rows = [f.readline().split()[1:] for x in range(3)] # small sample, trimming first column
for row in rows:
+ if len(row) < 1:
+ # Must actually have at least one value
+ return False
try:
[float(x) for x in row] # first col has been removed
except:
diff --git a/lib/galaxy/datatypes/interval.py b/lib/galaxy/datatypes/interval.py
index f3501efde7d..07fc6c75550 100644
--- a/lib/galaxy/datatypes/interval.py
+++ b/lib/galaxy/datatypes/interval.py
@@ -1,22 +1,25 @@
"""
Interval datatypes
"""
-
+import logging
+import math
import os
import sys
-import logging
import tempfile
-import data
-from galaxy import util
-from galaxy.web import url_for
import urllib
+
+import numpy
from bx.intervals.io import GenomicIntervalReader, ParseError
+
+from galaxy import util
from galaxy.datatypes import metadata
from galaxy.datatypes.metadata import MetadataElement
+from galaxy.datatypes.sniff import get_headers
from galaxy.datatypes.tabular import Tabular
from galaxy.datatypes.util.gff_util import parse_gff_attributes
-from galaxy.datatypes.sniff import get_headers
-import math
+from galaxy.web import url_for
+
+import data
import dataproviders
log = logging.getLogger(__name__)
@@ -1245,9 +1248,6 @@ class Wiggle( Tabular, _RemoteCallMixin ):
"""
Assumes we have a numpy file.
"""
- # Maybe if we import here people will still be able to use Galaxy when numpy kills it
- import numpy
-
range = end - start
# Determine appropriate resolution to plot ~1000 points
resolution = ( 10 ** math.ceil( math.log10( range / 1000 ) ) )
diff --git a/lib/galaxy/datatypes/registry.py b/lib/galaxy/datatypes/registry.py
index 509eb97c7a2..843d1ec06e5 100644
--- a/lib/galaxy/datatypes/registry.py
+++ b/lib/galaxy/datatypes/registry.py
@@ -634,32 +634,9 @@ class Registry( object ):
def load_external_metadata_tool( self, toolbox ):
"""Adds a tool which is used to set external metadata"""
# We need to be able to add a job to the queue to set metadata. The queue will currently only accept jobs with an associated
- # tool. We'll create a special tool to be used for Auto-Detecting metadata; this is less than ideal, but effective
- # Properly building a tool without relying on parsing an XML file is near impossible...so we'll create a temporary file
- tool_xml_text = """
-
-
-
- samtools
-
-
- python "${set_metadata}" ${__SET_EXTERNAL_METADATA_COMMAND_LINE__}
-
-
-
-
-
-
-
-
- from galaxy_ext.metadata.set_metadata import set_metadata; set_metadata()
-
-
- """
- tmp_name = tempfile.NamedTemporaryFile()
- tmp_name.write( tool_xml_text )
- tmp_name.flush()
- set_meta_tool = toolbox.load_hidden_tool( tmp_name.name )
+ # tool. We'll load a special tool to be used for Auto-Detecting metadata; this is less than ideal, but effective
+ # Properly building a tool without relying on parsing an XML file is near difficult...so we bundle with Galaxy.
+ set_meta_tool = toolbox.load_hidden_lib_tool( "galaxy/datatypes/set_metadata_tool.xml" )
self.set_external_metadata_tool = set_meta_tool
self.log.debug( "Loaded external metadata tool: %s", self.set_external_metadata_tool.id )
diff --git a/lib/galaxy/datatypes/set_metadata_tool.xml b/lib/galaxy/datatypes/set_metadata_tool.xml
new file mode 100644
index 00000000000..7d3d1146db2
--- /dev/null
+++ b/lib/galaxy/datatypes/set_metadata_tool.xml
@@ -0,0 +1,18 @@
+
+
+
+ samtools
+
+
+ python "${set_metadata}" ${__SET_EXTERNAL_METADATA_COMMAND_LINE__}
+
+
+
+
+
+
+
+
+ from galaxy_ext.metadata.set_metadata import set_metadata; set_metadata()
+
+
diff --git a/lib/galaxy/datatypes/tabular.py b/lib/galaxy/datatypes/tabular.py
index 447b566530a..2c4bf3be072 100644
--- a/lib/galaxy/datatypes/tabular.py
+++ b/lib/galaxy/datatypes/tabular.py
@@ -1,21 +1,24 @@
"""
Tabular datatype
-
"""
+import csv
import gzip
import logging
import os
-import csv
+import re
+import tempfile
+import subprocess
+
from cgi import escape
+
from galaxy import util
-from galaxy.datatypes import data
-from galaxy.datatypes import metadata
+from galaxy.datatypes import data, metadata
from galaxy.datatypes.checkers import is_gzip
from galaxy.datatypes.metadata import MetadataElement
from galaxy.datatypes.sniff import get_headers
from galaxy.util.json import dumps
+
import dataproviders
-import re
log = logging.getLogger(__name__)
@@ -691,6 +694,19 @@ class Vcf( Tabular ):
# Found header line, get sample names.
dataset.metadata.sample_names = line.split()[ 9: ]
+ @staticmethod
+ def merge(split_files, output_file):
+ stderr_f = tempfile.NamedTemporaryFile(prefix="bam_merge_stderr")
+ stderr_name = stderr_f.name
+ command = ["bcftools", "concat"] + split_files + ["-o", output_file]
+ log.info("Merging vcf files with command [%s]" % " ".join(command))
+ exit_code = subprocess.call( args=command, stderr=open( stderr_name, 'wb' ) )
+ with open(stderr_name, "rb") as f:
+ stderr = f.read().strip()
+ # Did merge succeed?
+ if exit_code != 0:
+ raise Exception("Error merging VCF files: %s" % stderr)
+
# Dataproviders
@dataproviders.decorators.dataprovider_factory( 'genomic-region',
dataproviders.dataset.GenomicRegionDataProvider.settings )
diff --git a/lib/galaxy/datatypes/test/1.gg b/lib/galaxy/datatypes/test/1.gg
new file mode 100644
index 00000000000..24b16fad91b
--- /dev/null
+++ b/lib/galaxy/datatypes/test/1.gg
@@ -0,0 +1,4 @@
+chrX 100000 1.23
+chrX 100000 1.23
+chrX 100000 1.23
+chrX 100000 1.23
diff --git a/lib/galaxy/datatypes/util/gff_util.py b/lib/galaxy/datatypes/util/gff_util.py
index 430b7393731..68e35adf481 100644
--- a/lib/galaxy/datatypes/util/gff_util.py
+++ b/lib/galaxy/datatypes/util/gff_util.py
@@ -1,10 +1,11 @@
"""
Provides utilities for working with GFF files.
"""
-
import copy
+
from bx.intervals.io import GenomicInterval, MissingFieldError, NiceReaderWrapper, ParseError, GenomicIntervalReader
from bx.tabular.io import Header, Comment
+
from galaxy.util.odict import odict
diff --git a/lib/galaxy/jobs/__init__.py b/lib/galaxy/jobs/__init__.py
index 49ba58f083f..ebb7205afa8 100644
--- a/lib/galaxy/jobs/__init__.py
+++ b/lib/galaxy/jobs/__init__.py
@@ -289,9 +289,8 @@ class JobConfiguration( object ):
"""
log.debug('Loading job configuration from %s' % self.app.config.config_file)
- # Always load local and lwr
- self.runner_plugins = [dict(id='local', load='local', workers=self.app.config.local_job_queue_workers),
- dict(id='lwr', load='lwr', workers=self.app.config.cluster_job_queue_workers)]
+ # Always load local
+ self.runner_plugins = [dict(id='local', load='local', workers=self.app.config.local_job_queue_workers)]
# Load tasks if configured
if self.app.config.use_tasked_jobs:
self.runner_plugins.append(dict(id='tasks', load='tasks', workers=self.app.config.local_task_queue_workers))
@@ -1406,7 +1405,7 @@ class JobWrapper( object ):
job = has_metrics.get_job()
per_plugin_properties = self.app.job_metrics.collect_properties( job.destination_id, self.job_id, self.working_directory )
if per_plugin_properties:
- log.info( "Collecting job metrics for %s" % has_metrics )
+ log.info( "Collecting metrics for %s %s" % ( type(has_metrics).__name__, getattr( has_metrics, 'id', None ) ) )
for plugin, properties in per_plugin_properties.iteritems():
for metric_name, metric_value in properties.iteritems():
if metric_value is not None:
diff --git a/lib/galaxy/jobs/command_factory.py b/lib/galaxy/jobs/command_factory.py
index 92044dfa8aa..927143d1660 100644
--- a/lib/galaxy/jobs/command_factory.py
+++ b/lib/galaxy/jobs/command_factory.py
@@ -73,9 +73,9 @@ def build_command(
def __externalize_commands(job_wrapper, commands_builder, remote_command_params, script_name="tool_script.sh"):
local_container_script = join( job_wrapper.working_directory, script_name )
- fh = file( local_container_script, "w" )
- fh.write( "#!%s\n%s" % (DEFAULT_SHELL, commands_builder.build()))
- fh.close()
+ with open( local_container_script, "w" ) as f:
+ script_contents = "#!%s\n%s" % (DEFAULT_SHELL, commands_builder.build())
+ f.write( script_contents )
chmod( local_container_script, 0755 )
commands = local_container_script
diff --git a/lib/galaxy/jobs/runners/__init__.py b/lib/galaxy/jobs/runners/__init__.py
index 5c5f5d6e9f9..42e7bedccf4 100644
--- a/lib/galaxy/jobs/runners/__init__.py
+++ b/lib/galaxy/jobs/runners/__init__.py
@@ -188,8 +188,6 @@ class BaseJobRunner( object ):
raise NotImplementedError()
def build_command_line( self, job_wrapper, include_metadata=False, include_work_dir_outputs=True ):
- # TODO: Eliminate extra kwds no longer used (since LWR skips
- # abstraction and calls build_command directly).
container = self._find_container( job_wrapper )
return build_command(
self,
@@ -248,8 +246,8 @@ class BaseJobRunner( object ):
def _handle_metadata_externally( self, job_wrapper, resolve_requirements=False ):
"""
- Set metadata externally. Used by the local and lwr job runners where this
- shouldn't be attached to command-line to execute.
+ Set metadata externally. Used by the Pulsar job runner where this
+ shouldn't be attached to command line to execute.
"""
# run the metadata setting script here
# this is terminate-able when output dataset/job is deleted
@@ -305,6 +303,41 @@ class BaseJobRunner( object ):
options.update(**kwds)
return job_script(**options)
+ def write_executable_script( self, path, contents, mode=0o755 ):
+ with open( path, 'w' ) as f:
+ f.write( contents )
+ os.chmod( path, mode )
+ self._handle_script_integrity( path )
+
+ def _handle_script_integrity( self, path ):
+ if not getattr( self.app.config, "check_job_script_integrity", True ):
+ return
+
+ script_integrity_verified = False
+ for i in range(5):
+ try:
+ proc = subprocess.Popen( [path], shell=True, env={"ABC_TEST_JOB_SCRIPT_INTEGRITY_XYZ": "1"} )
+ proc.wait()
+ if proc.returncode == 42:
+ script_integrity_verified = True
+ break
+
+ # Else we will sync and wait to see if the script becomes
+ # executable.
+ try:
+ # sync file system to avoid "Text file busy" problems.
+ # These have occurred both in Docker containers and on EC2 clusters
+ # under high load.
+ subprocess.check_call(["/bin/sync"])
+ except Exception:
+ pass
+ time.sleep( .1 )
+ except Exception:
+ pass
+
+ if not script_integrity_verified:
+ raise Exception("Failed to write job script, could not verify job script integrity.")
+
def _complete_terminal_job( self, ajs, **kwargs ):
if ajs.job_wrapper.get_state() != model.Job.states.DELETED:
self.work_queue.put( ( self.finish_job, ajs ) )
diff --git a/lib/galaxy/jobs/runners/cli.py b/lib/galaxy/jobs/runners/cli.py
index 795202e8c30..7d8b5477957 100644
--- a/lib/galaxy/jobs/runners/cli.py
+++ b/lib/galaxy/jobs/runners/cli.py
@@ -72,9 +72,7 @@ class ShellJobRunner( AsynchronousJobRunner ):
)
try:
- fh = file(ajs.job_file, "w")
- fh.write(script)
- fh.close()
+ self.write_executable_script( ajs.job_file, script )
except:
log.exception("(%s) failure writing job script" % galaxy_id_tag )
job_wrapper.fail("failure preparing job script", exception=True)
diff --git a/lib/galaxy/jobs/runners/condor.py b/lib/galaxy/jobs/runners/condor.py
index 6ae20f94396..4b14bbb5e9f 100644
--- a/lib/galaxy/jobs/runners/condor.py
+++ b/lib/galaxy/jobs/runners/condor.py
@@ -90,10 +90,7 @@ class CondorJobRunner( AsynchronousJobRunner ):
slots_statement=galaxy_slots_statement,
)
try:
- fh = file( executable, "w" )
- fh.write( script )
- fh.close()
- os.chmod( executable, 0o750 )
+ self.write_executable_script( executable.job_file, script )
except:
job_wrapper.fail( "failure preparing job script", exception=True )
log.exception( "(%s) failure preparing job script" % galaxy_id_tag )
diff --git a/lib/galaxy/jobs/runners/drmaa.py b/lib/galaxy/jobs/runners/drmaa.py
index ff02cfb6257..98b020ea5cd 100644
--- a/lib/galaxy/jobs/runners/drmaa.py
+++ b/lib/galaxy/jobs/runners/drmaa.py
@@ -148,10 +148,7 @@ class DRMAAJobRunner( AsynchronousJobRunner ):
# fill in the DRM's job run template
script = self.get_job_file(job_wrapper, exit_code_path=ajs.exit_code_file)
try:
- fh = file( ajs.job_file, "w" )
- fh.write( script )
- fh.close()
- os.chmod( ajs.job_file, 0o755 )
+ self.write_executable_script( ajs.job_file, script )
except:
job_wrapper.fail( "failure preparing job script", exception=True )
log.exception( "(%s) failure writing job script" % galaxy_id_tag )
diff --git a/lib/galaxy/jobs/runners/local.py b/lib/galaxy/jobs/runners/local.py
index b91c6e01a4f..ecd5038264b 100644
--- a/lib/galaxy/jobs/runners/local.py
+++ b/lib/galaxy/jobs/runners/local.py
@@ -68,8 +68,7 @@ class LocalJobRunner( BaseJobRunner ):
'working_directory': job_wrapper.working_directory,
}
job_file_contents = self.get_job_file( job_wrapper, **job_script_props )
- open( job_file, 'w' ).write( job_file_contents )
- os.chmod( job_file, 0o755 )
+ self.write_executable_script( job_file, job_file_contents )
return job_file, exit_code_path
def queue_job( self, job_wrapper ):
diff --git a/lib/galaxy/jobs/runners/lwr.py b/lib/galaxy/jobs/runners/lwr.py
deleted file mode 100644
index d7df067e08f..00000000000
--- a/lib/galaxy/jobs/runners/lwr.py
+++ /dev/null
@@ -1,648 +0,0 @@
-import logging
-
-from galaxy import model
-from galaxy.jobs.runners import AsynchronousJobState, AsynchronousJobRunner
-from galaxy.jobs import ComputeEnvironment
-from galaxy.jobs import JobDestination
-from galaxy.jobs.command_factory import build_command
-from galaxy.tools.deps import dependencies
-from galaxy.util import string_as_bool_or_none
-from galaxy.util.bunch import Bunch
-from galaxy.util import specs
-
-import errno
-from time import sleep
-import os
-
-from .lwr_client import build_client_manager
-from .lwr_client import url_to_destination_params
-from .lwr_client import finish_job as lwr_finish_job
-from .lwr_client import submit_job as lwr_submit_job
-from .lwr_client import ClientJobDescription
-from .lwr_client import LwrOutputs
-from .lwr_client import ClientOutputs
-from .lwr_client import PathMapper
-
-log = logging.getLogger( __name__ )
-
-__all__ = [ 'LwrJobRunner' ]
-
-NO_REMOTE_GALAXY_FOR_METADATA_MESSAGE = "LWR misconfiguration - LWR client configured to set metadata remotely, but remote LWR isn't properly configured with a galaxy_home directory."
-NO_REMOTE_DATATYPES_CONFIG = "LWR client is configured to use remote datatypes configuration when setting metadata externally, but LWR is not configured with this information. Defaulting to datatypes_conf.xml."
-GENERIC_REMOTE_ERROR = "Failed to communicate with remote job server."
-
-# Is there a good way to infer some default for this? Can only use
-# url_for from web threads. https://gist.github.com/jmchilton/9098762
-DEFAULT_GALAXY_URL = "http://localhost:8080"
-
-LWR_PARAM_SPECS = dict(
- transport=dict(
- map=specs.to_str_or_none,
- valid=specs.is_in("urllib", "curl", None),
- default=None
- ),
- cache=dict(
- map=specs.to_bool_or_none,
- default=None,
- ),
- url=dict(
- map=specs.to_str_or_none,
- default=None,
- ),
- galaxy_url=dict(
- map=specs.to_str_or_none,
- default=DEFAULT_GALAXY_URL,
- ),
- manager=dict(
- map=specs.to_str_or_none,
- default=None,
- ),
- amqp_consumer_timeout=dict(
- map=lambda val: None if val == "None" else float(val),
- default=None,
- ),
- amqp_connect_ssl_ca_certs=dict(
- map=specs.to_str_or_none,
- default=None,
- ),
- amqp_connect_ssl_keyfile=dict(
- map=specs.to_str_or_none,
- default=None,
- ),
- amqp_connect_ssl_certfile=dict(
- map=specs.to_str_or_none,
- default=None,
- ),
- amqp_connect_ssl_cert_reqs=dict(
- map=specs.to_str_or_none,
- default=None,
- ),
- # http://kombu.readthedocs.org/en/latest/reference/kombu.html#kombu.Producer.publish
- amqp_publish_retry=dict(
- map=specs.to_bool,
- default=False,
- ),
- amqp_publish_priority=dict(
- map=int,
- valid=lambda x: 0 <= x and x <= 9,
- default=0,
- ),
- # http://kombu.readthedocs.org/en/latest/reference/kombu.html#kombu.Exchange.delivery_mode
- amqp_publish_delivery_mode=dict(
- map=str,
- valid=specs.is_in("transient", "persistent"),
- default="persistent",
- ),
- amqp_publish_retry_max_retries=dict(
- map=int,
- default=None,
- ),
- amqp_publish_retry_interval_start=dict(
- map=int,
- default=None,
- ),
- amqp_publish_retry_interval_step=dict(
- map=int,
- default=None,
- ),
- amqp_publish_retry_interval_max=dict(
- map=int,
- default=None,
- ),
-)
-
-
-class LwrJobRunner( AsynchronousJobRunner ):
- """
- LWR Job Runner
- """
- runner_name = "LWRRunner"
-
- def __init__( self, app, nworkers, **kwds ):
- """Start the job runner """
- super( LwrJobRunner, self ).__init__( app, nworkers, runner_param_specs=LWR_PARAM_SPECS, **kwds )
- self._init_worker_threads()
- galaxy_url = self.runner_params.galaxy_url
- if galaxy_url:
- galaxy_url = galaxy_url.rstrip("/")
- self.galaxy_url = galaxy_url
- self.__init_client_manager()
- if self.runner_params.url:
- # This is a message queue driven runner, don't monitor
- # just setup required callback.
- self.client_manager.ensure_has_status_update_callback(self.__async_update)
- else:
- self._init_monitor_thread()
-
- def __init_client_manager( self ):
- client_manager_kwargs = {}
- for kwd in 'url', 'manager', 'cache', 'transport':
- client_manager_kwargs[ kwd ] = self.runner_params[ kwd ]
- for kwd in self.runner_params.keys():
- if kwd.startswith( 'amqp_' ):
- client_manager_kwargs[ kwd ] = self.runner_params[ kwd ]
- self.client_manager = build_client_manager(**client_manager_kwargs)
-
- def url_to_destination( self, url ):
- """Convert a legacy URL to a job destination"""
- return JobDestination( runner="lwr", params=url_to_destination_params( url ) )
-
- def check_watched_item(self, job_state):
- try:
- client = self.get_client_from_state(job_state)
- status = client.get_status()
- except Exception:
- # An orphaned job was put into the queue at app startup, so remote server went down
- # either way we are done I guess.
- self.mark_as_finished(job_state)
- return None
- job_state = self.__update_job_state_for_lwr_status(job_state, status)
- return job_state
-
- def __update_job_state_for_lwr_status(self, job_state, lwr_status):
- if lwr_status == "complete":
- self.mark_as_finished(job_state)
- return None
- if lwr_status == "failed":
- self.fail_job(job_state)
- return None
- if lwr_status == "running" and not job_state.running:
- job_state.running = True
- job_state.job_wrapper.change_state( model.Job.states.RUNNING )
- return job_state
-
- def __async_update( self, full_status ):
- while not hasattr( self.app, 'job_manager' ):
- # The status update thread can start consuming before app is done initializing
- log.debug( 'Received a status update message before app is initialized, waiting 5 seconds' )
- sleep( 5 )
- job_id = None
- try:
- job_id = full_status[ "job_id" ]
- job, job_wrapper = self.app.job_manager.job_handler.job_queue.job_pair_for_id( job_id )
- job_state = self.__job_state( job, job_wrapper )
- self.__update_job_state_for_lwr_status(job_state, full_status["status"])
- except Exception:
- log.exception( "Failed to update LWR job status for job_id %s" % job_id )
- raise
- # Nothing else to do? - Attempt to fail the job?
-
- def queue_job(self, job_wrapper):
- job_destination = job_wrapper.job_destination
-
- command_line, client, remote_job_config, compute_environment = self.__prepare_job( job_wrapper, job_destination )
-
- if not command_line:
- return
-
- try:
- dependencies_description = LwrJobRunner.__dependencies_description( client, job_wrapper )
- rewrite_paths = not LwrJobRunner.__rewrite_parameters( client )
- unstructured_path_rewrites = {}
- if compute_environment:
- unstructured_path_rewrites = compute_environment.unstructured_path_rewrites
-
- client_job_description = ClientJobDescription(
- command_line=command_line,
- input_files=self.get_input_files(job_wrapper),
- client_outputs=self.__client_outputs(client, job_wrapper),
- working_directory=job_wrapper.working_directory,
- tool=job_wrapper.tool,
- config_files=job_wrapper.extra_filenames,
- dependencies_description=dependencies_description,
- env=client.env,
- rewrite_paths=rewrite_paths,
- arbitrary_files=unstructured_path_rewrites,
- )
- job_id = lwr_submit_job(client, client_job_description, remote_job_config)
- log.info("lwr job submitted with job_id %s" % job_id)
- job_wrapper.set_job_destination( job_destination, job_id )
- job_wrapper.change_state( model.Job.states.QUEUED )
- except Exception:
- job_wrapper.fail( "failure running job", exception=True )
- log.exception("failure running job %d" % job_wrapper.job_id)
- return
-
- lwr_job_state = AsynchronousJobState()
- lwr_job_state.job_wrapper = job_wrapper
- lwr_job_state.job_id = job_id
- lwr_job_state.old_state = True
- lwr_job_state.running = False
- lwr_job_state.job_destination = job_destination
- self.monitor_job(lwr_job_state)
-
- def __prepare_job(self, job_wrapper, job_destination):
- """ Build command-line and LWR client for this job. """
- command_line = None
- client = None
- remote_job_config = None
- compute_environment = None
- try:
- client = self.get_client_from_wrapper(job_wrapper)
- tool = job_wrapper.tool
- remote_job_config = client.setup(tool.id, tool.version)
- rewrite_parameters = LwrJobRunner.__rewrite_parameters( client )
- prepare_kwds = {}
- if rewrite_parameters:
- compute_environment = LwrComputeEnvironment( client, job_wrapper, remote_job_config )
- prepare_kwds[ 'compute_environment' ] = compute_environment
- job_wrapper.prepare( **prepare_kwds )
- self.__prepare_input_files_locally(job_wrapper)
- remote_metadata = LwrJobRunner.__remote_metadata( client )
- dependency_resolution = LwrJobRunner.__dependency_resolution( client )
- metadata_kwds = self.__build_metadata_configuration(client, job_wrapper, remote_metadata, remote_job_config)
- remote_command_params = dict(
- working_directory=remote_job_config['working_directory'],
- metadata_kwds=metadata_kwds,
- dependency_resolution=dependency_resolution,
- )
- remote_working_directory = remote_job_config['working_directory']
- # TODO: Following defs work for LWR, always worked for LWR but should be
- # calculated at some other level.
- remote_job_directory = os.path.abspath(os.path.join(remote_working_directory, os.path.pardir))
- remote_tool_directory = os.path.abspath(os.path.join(remote_job_directory, "tool_files"))
- container = self._find_container(
- job_wrapper,
- compute_working_directory=remote_working_directory,
- compute_tool_directory=remote_tool_directory,
- compute_job_directory=remote_job_directory,
- )
- command_line = build_command(
- self,
- job_wrapper=job_wrapper,
- container=container,
- include_metadata=remote_metadata,
- include_work_dir_outputs=False,
- remote_command_params=remote_command_params,
- )
- except Exception:
- job_wrapper.fail( "failure preparing job", exception=True )
- log.exception("failure running job %d" % job_wrapper.job_id)
-
- # If we were able to get a command line, run the job
- if not command_line:
- job_wrapper.finish( '', '' )
-
- return command_line, client, remote_job_config, compute_environment
-
- def __prepare_input_files_locally(self, job_wrapper):
- """Run task splitting commands locally."""
- prepare_input_files_cmds = getattr(job_wrapper, 'prepare_input_files_cmds', None)
- if prepare_input_files_cmds is not None:
- for cmd in prepare_input_files_cmds: # run the commands to stage the input files
- if 0 != os.system(cmd):
- raise Exception('Error running file staging command: %s' % cmd)
- job_wrapper.prepare_input_files_cmds = None # prevent them from being used in-line
-
- def get_output_files(self, job_wrapper):
- output_paths = job_wrapper.get_output_fnames()
- return [ str( o ) for o in output_paths ] # Force job_path from DatasetPath objects.
-
- def get_input_files(self, job_wrapper):
- input_paths = job_wrapper.get_input_paths()
- return [ str( i ) for i in input_paths ] # Force job_path from DatasetPath objects.
-
- def get_client_from_wrapper(self, job_wrapper):
- job_id = job_wrapper.job_id
- if hasattr(job_wrapper, 'task_id'):
- job_id = "%s_%s" % (job_id, job_wrapper.task_id)
- params = job_wrapper.job_destination.params.copy()
- for key, value in params.iteritems():
- if value:
- params[key] = model.User.expand_user_properties( job_wrapper.get_job().user, value )
- env = getattr( job_wrapper.job_destination, "env", [] )
- return self.get_client( params, job_id, env )
-
- def get_client_from_state(self, job_state):
- job_destination_params = job_state.job_destination.params
- job_id = job_state.job_id
- return self.get_client( job_destination_params, job_id )
-
- def get_client( self, job_destination_params, job_id, env=[] ):
- # Cannot use url_for outside of web thread.
- # files_endpoint = url_for( controller="job_files", job_id=encoded_job_id )
-
- encoded_job_id = self.app.security.encode_id(job_id)
- job_key = self.app.security.encode_id( job_id, kind="jobs_files" )
- files_endpoint = "%s/api/jobs/%s/files?job_key=%s" % (
- self.galaxy_url,
- encoded_job_id,
- job_key
- )
- get_client_kwds = dict(
- job_id=str( job_id ),
- files_endpoint=files_endpoint,
- env=env
- )
- return self.client_manager.get_client( job_destination_params, **get_client_kwds )
-
- def finish_job( self, job_state ):
- stderr = stdout = ''
- job_wrapper = job_state.job_wrapper
- try:
- client = self.get_client_from_state(job_state)
- run_results = client.full_status()
- remote_working_directory = run_results.get("working_directory", None)
- stdout = run_results.get('stdout', '')
- stderr = run_results.get('stderr', '')
- exit_code = run_results.get('returncode', None)
- lwr_outputs = LwrOutputs.from_status_response(run_results)
- # Use LWR client code to transfer/copy files back
- # and cleanup job if needed.
- completed_normally = \
- job_wrapper.get_state() not in [ model.Job.states.ERROR, model.Job.states.DELETED ]
- cleanup_job = self.app.config.cleanup_job
- client_outputs = self.__client_outputs(client, job_wrapper)
- finish_args = dict( client=client,
- job_completed_normally=completed_normally,
- cleanup_job=cleanup_job,
- client_outputs=client_outputs,
- lwr_outputs=lwr_outputs )
- failed = lwr_finish_job( **finish_args )
-
- if failed:
- job_wrapper.fail("Failed to find or download one or more job outputs from remote server.", exception=True)
- except Exception:
- message = GENERIC_REMOTE_ERROR
- job_wrapper.fail( message, exception=True )
- log.exception("failure finishing job %d" % job_wrapper.job_id)
- return
- if not LwrJobRunner.__remote_metadata( client ):
- self._handle_metadata_externally( job_wrapper, resolve_requirements=True )
- # Finish the job
- try:
- job_wrapper.finish(
- stdout,
- stderr,
- exit_code,
- remote_working_directory=remote_working_directory
- )
- except Exception:
- log.exception("Job wrapper finish method failed")
- job_wrapper.fail("Unable to finish job", exception=True)
-
- def fail_job( self, job_state ):
- """
- Seperated out so we can use the worker threads for it.
- """
- self.stop_job( self.sa_session.query( self.app.model.Job ).get( job_state.job_wrapper.job_id ) )
- job_state.job_wrapper.fail( getattr( job_state, "fail_message", GENERIC_REMOTE_ERROR ) )
-
- def check_pid( self, pid ):
- try:
- os.kill( pid, 0 )
- return True
- except OSError as e:
- if e.errno == errno.ESRCH:
- log.debug( "check_pid(): PID %d is dead" % pid )
- else:
- log.warning( "check_pid(): Got errno %s when attempting to check PID %d: %s" % ( errno.errorcode[e.errno], pid, e.strerror ) )
- return False
-
- def stop_job( self, job ):
- # if our local job has JobExternalOutputMetadata associated, then our primary job has to have already finished
- job_ext_output_metadata = job.get_external_output_metadata()
- if job_ext_output_metadata:
- pid = job_ext_output_metadata[0].job_runner_external_pid # every JobExternalOutputMetadata has a pid set, we just need to take from one of them
- if pid in [ None, '' ]:
- log.warning( "stop_job(): %s: no PID in database for job, unable to stop" % job.id )
- return
- pid = int( pid )
- if not self.check_pid( pid ):
- log.warning( "stop_job(): %s: PID %d was already dead or can't be signaled" % ( job.id, pid ) )
- return
- for sig in [ 15, 9 ]:
- try:
- os.killpg( pid, sig )
- except OSError as e:
- log.warning( "stop_job(): %s: Got errno %s when attempting to signal %d to PID %d: %s" % ( job.id, errno.errorcode[e.errno], sig, pid, e.strerror ) )
- return # give up
- sleep( 2 )
- if not self.check_pid( pid ):
- log.debug( "stop_job(): %s: PID %d successfully killed with signal %d" % ( job.id, pid, sig ) )
- return
- else:
- log.warning( "stop_job(): %s: PID %d refuses to die after signaling TERM/KILL" % ( job.id, pid ) )
- else:
- # Remote kill
- lwr_url = job.job_runner_name
- job_id = job.job_runner_external_id
- log.debug("Attempt remote lwr kill of job with url %s and id %s" % (lwr_url, job_id))
- client = self.get_client(job.destination_params, job_id)
- client.kill()
-
- def recover( self, job, job_wrapper ):
- """Recovers jobs stuck in the queued/running state when Galaxy started"""
- job_state = self.__job_state( job, job_wrapper )
- job_wrapper.command_line = job.get_command_line()
- state = job.get_state()
- if state in [model.Job.states.RUNNING, model.Job.states.QUEUED]:
- log.debug( "(LWR/%s) is still in running state, adding to the LWR queue" % ( job.get_id()) )
- job_state.old_state = True
- job_state.running = state == model.Job.states.RUNNING
- self.monitor_queue.put( job_state )
-
- def shutdown( self ):
- super( LwrJobRunner, self ).shutdown()
- self.client_manager.shutdown()
-
- def __job_state( self, job, job_wrapper ):
- job_state = AsynchronousJobState()
- # TODO: Determine why this is set when using normal message queue updates
- # but not CLI submitted MQ updates...
- raw_job_id = job.get_job_runner_external_id() or job_wrapper.job_id
- job_state.job_id = str( raw_job_id )
- job_state.runner_url = job_wrapper.get_job_runner_url()
- job_state.job_destination = job_wrapper.job_destination
- job_state.job_wrapper = job_wrapper
- return job_state
-
- def __client_outputs( self, client, job_wrapper ):
- work_dir_outputs = self.get_work_dir_outputs( job_wrapper )
- output_files = self.get_output_files( job_wrapper )
- client_outputs = ClientOutputs(
- working_directory=job_wrapper.working_directory,
- work_dir_outputs=work_dir_outputs,
- output_files=output_files,
- version_file=job_wrapper.get_version_string_path(),
- )
- return client_outputs
-
- @staticmethod
- def __dependencies_description( lwr_client, job_wrapper ):
- dependency_resolution = LwrJobRunner.__dependency_resolution( lwr_client )
- remote_dependency_resolution = dependency_resolution == "remote"
- if not remote_dependency_resolution:
- return None
- requirements = job_wrapper.tool.requirements or []
- installed_tool_dependencies = job_wrapper.tool.installed_tool_dependencies or []
- return dependencies.DependenciesDescription(
- requirements=requirements,
- installed_tool_dependencies=installed_tool_dependencies,
- )
-
- @staticmethod
- def __dependency_resolution( lwr_client ):
- dependency_resolution = lwr_client.destination_params.get( "dependency_resolution", "local" )
- if dependency_resolution not in ["none", "local", "remote"]:
- raise Exception("Unknown dependency_resolution value encountered %s" % dependency_resolution)
- return dependency_resolution
-
- @staticmethod
- def __remote_metadata( lwr_client ):
- remote_metadata = string_as_bool_or_none( lwr_client.destination_params.get( "remote_metadata", False ) )
- return remote_metadata
-
- @staticmethod
- def __use_remote_datatypes_conf( lwr_client ):
- """ When setting remote metadata, use integrated datatypes from this
- Galaxy instance or use the datatypes config configured via the remote
- LWR.
-
- Both options are broken in different ways for same reason - datatypes
- may not match. One can push the local datatypes config to the remote
- server - but there is no guarentee these datatypes will be defined
- there. Alternatively, one can use the remote datatype config - but
- there is no guarentee that it will contain all the datatypes available
- to this Galaxy.
- """
- use_remote_datatypes = string_as_bool_or_none( lwr_client.destination_params.get( "use_remote_datatypes", False ) )
- return use_remote_datatypes
-
- @staticmethod
- def __rewrite_parameters( lwr_client ):
- return string_as_bool_or_none( lwr_client.destination_params.get( "rewrite_parameters", False ) ) or False
-
- def __build_metadata_configuration(self, client, job_wrapper, remote_metadata, remote_job_config):
- metadata_kwds = {}
- if remote_metadata:
- remote_system_properties = remote_job_config.get("system_properties", {})
- remote_galaxy_home = remote_system_properties.get("galaxy_home", None)
- if not remote_galaxy_home:
- raise Exception(NO_REMOTE_GALAXY_FOR_METADATA_MESSAGE)
- metadata_kwds['exec_dir'] = remote_galaxy_home
- outputs_directory = remote_job_config['outputs_directory']
- configs_directory = remote_job_config['configs_directory']
- working_directory = remote_job_config['working_directory']
- # For metadata calculation, we need to build a list of of output
- # file objects with real path indicating location on Galaxy server
- # and false path indicating location on compute server. Since the
- # LWR disables from_work_dir copying as part of the job command
- # line we need to take the list of output locations on the LWR
- # server (produced by self.get_output_files(job_wrapper)) and for
- # each work_dir output substitute the effective path on the LWR
- # server relative to the remote working directory as the
- # false_path to send the metadata command generation module.
- work_dir_outputs = self.get_work_dir_outputs(job_wrapper, job_working_directory=working_directory)
- outputs = [Bunch(false_path=os.path.join(outputs_directory, os.path.basename(path)), real_path=path) for path in self.get_output_files(job_wrapper)]
- for output in outputs:
- for lwr_workdir_path, real_path in work_dir_outputs:
- if real_path == output.real_path:
- output.false_path = lwr_workdir_path
- metadata_kwds['output_fnames'] = outputs
- metadata_kwds['compute_tmp_dir'] = working_directory
- metadata_kwds['config_root'] = remote_galaxy_home
- default_config_file = os.path.join(remote_galaxy_home, 'galaxy.ini')
- metadata_kwds['config_file'] = remote_system_properties.get('galaxy_config_file', default_config_file)
- metadata_kwds['dataset_files_path'] = remote_system_properties.get('galaxy_dataset_files_path', None)
- if LwrJobRunner.__use_remote_datatypes_conf( client ):
- remote_datatypes_config = remote_system_properties.get('galaxy_datatypes_config_file', None)
- if not remote_datatypes_config:
- log.warn(NO_REMOTE_DATATYPES_CONFIG)
- remote_datatypes_config = os.path.join(remote_galaxy_home, 'datatypes_conf.xml')
- metadata_kwds['datatypes_config'] = remote_datatypes_config
- else:
- integrates_datatypes_config = self.app.datatypes_registry.integrated_datatypes_configs
- # Ensure this file gets pushed out to the remote config dir.
- job_wrapper.extra_filenames.append(integrates_datatypes_config)
-
- metadata_kwds['datatypes_config'] = os.path.join(configs_directory, os.path.basename(integrates_datatypes_config))
- return metadata_kwds
-
-
-class LwrComputeEnvironment( ComputeEnvironment ):
-
- def __init__( self, lwr_client, job_wrapper, remote_job_config ):
- self.lwr_client = lwr_client
- self.job_wrapper = job_wrapper
- self.local_path_config = job_wrapper.default_compute_environment()
- self.unstructured_path_rewrites = {}
- # job_wrapper.prepare is going to expunge the job backing the following
- # computations, so precalculate these paths.
- self._wrapper_input_paths = self.local_path_config.input_paths()
- self._wrapper_output_paths = self.local_path_config.output_paths()
- self.path_mapper = PathMapper(lwr_client, remote_job_config, self.local_path_config.working_directory())
- self._config_directory = remote_job_config[ "configs_directory" ]
- self._working_directory = remote_job_config[ "working_directory" ]
- self._sep = remote_job_config[ "system_properties" ][ "separator" ]
- self._tool_dir = remote_job_config[ "tools_directory" ]
- version_path = self.local_path_config.version_path()
- new_version_path = self.path_mapper.remote_version_path_rewrite(version_path)
- if new_version_path:
- version_path = new_version_path
- self._version_path = version_path
-
- def output_paths( self ):
- local_output_paths = self._wrapper_output_paths
-
- results = []
- for local_output_path in local_output_paths:
- wrapper_path = str( local_output_path )
- remote_path = self.path_mapper.remote_output_path_rewrite( wrapper_path )
- results.append( self._dataset_path( local_output_path, remote_path ) )
- return results
-
- def input_paths( self ):
- local_input_paths = self._wrapper_input_paths
-
- results = []
- for local_input_path in local_input_paths:
- wrapper_path = str( local_input_path )
- # This will over-copy in some cases. For instance in the case of task
- # splitting, this input will be copied even though only the work dir
- # input will actually be used.
- remote_path = self.path_mapper.remote_input_path_rewrite( wrapper_path )
- results.append( self._dataset_path( local_input_path, remote_path ) )
- return results
-
- def _dataset_path( self, local_dataset_path, remote_path ):
- remote_extra_files_path = None
- if remote_path:
- remote_extra_files_path = "%s_files" % remote_path[ 0:-len( ".dat" ) ]
- return local_dataset_path.with_path_for_job( remote_path, remote_extra_files_path )
-
- def working_directory( self ):
- return self._working_directory
-
- def config_directory( self ):
- return self._config_directory
-
- def new_file_path( self ):
- return self.working_directory() # Problems with doing this?
-
- def sep( self ):
- return self._sep
-
- def version_path( self ):
- return self._version_path
-
- def rewriter( self, parameter_value ):
- unstructured_path_rewrites = self.unstructured_path_rewrites
- if parameter_value in unstructured_path_rewrites:
- # Path previously mapped, use previous mapping.
- return unstructured_path_rewrites[ parameter_value ]
- if parameter_value in unstructured_path_rewrites.itervalues():
- # Path is a rewritten remote path (this might never occur,
- # consider dropping check...)
- return parameter_value
-
- rewrite, new_unstructured_path_rewrites = self.path_mapper.check_for_arbitrary_rewrite( parameter_value )
- if rewrite:
- unstructured_path_rewrites.update(new_unstructured_path_rewrites)
- return rewrite
- else:
- # Did need to rewrite, use original path or value.
- return parameter_value
-
- def unstructured_path_rewriter( self ):
- return self.rewriter
diff --git a/lib/galaxy/jobs/runners/lwr_client/__init__.py b/lib/galaxy/jobs/runners/lwr_client/__init__.py
deleted file mode 100644
index 32cbdc082d3..00000000000
--- a/lib/galaxy/jobs/runners/lwr_client/__init__.py
+++ /dev/null
@@ -1,62 +0,0 @@
-"""
-lwr_client
-==========
-
-This module contains logic for interfacing with an external LWR server.
-
-------------------
-Configuring Galaxy
-------------------
-
-Galaxy job runners are configured in Galaxy's ``job_conf.xml`` file. See ``job_conf.xml.sample_advanced``
-in your Galaxy code base or on
-`Bitbucket `_
-for information on how to configure Galaxy to interact with the LWR.
-
-Galaxy also supports an older, less rich configuration of job runners directly
-in its main ``galaxy.ini`` file. The following section describes how to
-configure Galaxy to communicate with the LWR in this legacy mode.
-
-Legacy
-------
-
-A Galaxy tool can be configured to be executed remotely via LWR by
-adding a line to the ``galaxy.ini`` file under the ``galaxy:tool_runners``
-section with the format::
-
- = lwr://http://:
-
-As an example, if a host named remotehost is running the LWR server
-application on port ``8913``, then the tool with id ``test_tool`` can
-be configured to run remotely on remotehost by adding the following
-line to ``galaxy.ini``::
-
- test_tool = lwr://http://remotehost:8913
-
-Remember this must be added after the ``[galaxy:tool_runners]`` header
-in the ``universe.ini`` file.
-
-
-"""
-
-from .staging.down import finish_job
-from .staging.up import submit_job
-from .staging import ClientJobDescription
-from .staging import LwrOutputs
-from .staging import ClientOutputs
-from .client import OutputNotFoundException
-from .manager import build_client_manager
-from .destination import url_to_destination_params
-from .path_mapper import PathMapper
-
-__all__ = [
- build_client_manager,
- OutputNotFoundException,
- url_to_destination_params,
- finish_job,
- submit_job,
- ClientJobDescription,
- LwrOutputs,
- ClientOutputs,
- PathMapper,
-]
diff --git a/lib/galaxy/jobs/runners/lwr_client/action_mapper.py b/lib/galaxy/jobs/runners/lwr_client/action_mapper.py
deleted file mode 100644
index 13de35c865b..00000000000
--- a/lib/galaxy/jobs/runners/lwr_client/action_mapper.py
+++ /dev/null
@@ -1,567 +0,0 @@
-from json import load
-from os import makedirs
-from os.path import exists
-from os.path import abspath
-from os.path import dirname
-from os.path import join
-from os.path import basename
-from os.path import sep
-import fnmatch
-from re import compile
-from re import escape
-import galaxy.util
-from galaxy.util.bunch import Bunch
-from .config_util import read_file
-from .util import directory_files
-from .util import unique_path_prefix
-from .transport import get_file
-from .transport import post_file
-
-
-DEFAULT_MAPPED_ACTION = 'transfer' # Not really clear to me what this should be, exception?
-DEFAULT_PATH_MAPPER_TYPE = 'prefix'
-
-STAGING_ACTION_REMOTE = "remote"
-STAGING_ACTION_LOCAL = "local"
-STAGING_ACTION_NONE = None
-STAGING_ACTION_DEFAULT = "default"
-
-# Poor man's enum.
-path_type = Bunch(
- # Galaxy input datasets and extra files.
- INPUT="input",
- # Galaxy config and param files.
- CONFIG="config",
- # Files from tool's tool_dir (for now just wrapper if available).
- TOOL="tool",
- # Input work dir files - e.g. metadata files, task-split input files, etc..
- WORKDIR="workdir",
- # Galaxy output datasets in their final home.
- OUTPUT="output",
- # Galaxy from_work_dir output paths and other files (e.g. galaxy.json)
- OUTPUT_WORKDIR="output_workdir",
- # Other fixed tool parameter paths (likely coming from tool data, but not
- # nessecarily). Not sure this is the best name...
- UNSTRUCTURED="unstructured",
-)
-
-
-ACTION_DEFAULT_PATH_TYPES = [
- path_type.INPUT,
- path_type.CONFIG,
- path_type.TOOL,
- path_type.WORKDIR,
- path_type.OUTPUT,
- path_type.OUTPUT_WORKDIR,
-]
-ALL_PATH_TYPES = ACTION_DEFAULT_PATH_TYPES + [path_type.UNSTRUCTURED]
-
-
-class FileActionMapper(object):
- """
- Objects of this class define how paths are mapped to actions.
-
- >>> json_string = r'''{"paths": [ \
- {"path": "/opt/galaxy", "action": "none"}, \
- {"path": "/galaxy/data", "action": "transfer"}, \
- {"path": "/cool/bamfiles/**/*.bam", "action": "copy", "match_type": "glob"}, \
- {"path": ".*/dataset_\\\\d+.dat", "action": "copy", "match_type": "regex"} \
- ]}'''
- >>> from tempfile import NamedTemporaryFile
- >>> from os import unlink
- >>> def mapper_for(default_action, config_contents):
- ... f = NamedTemporaryFile(delete=False)
- ... f.write(config_contents.encode('UTF-8'))
- ... f.close()
- ... mock_client = Bunch(default_file_action=default_action, action_config_path=f.name, files_endpoint=None)
- ... mapper = FileActionMapper(mock_client)
- ... mapper = FileActionMapper(config=mapper.to_dict()) # Serialize and deserialize it to make sure still works
- ... unlink(f.name)
- ... return mapper
- >>> mapper = mapper_for(default_action='none', config_contents=json_string)
- >>> # Test first config line above, implicit path prefix mapper
- >>> action = mapper.action('/opt/galaxy/tools/filters/catWrapper.py', 'input')
- >>> action.action_type == u'none'
- True
- >>> action.staging_needed
- False
- >>> # Test another (2nd) mapper, this one with a different action
- >>> action = mapper.action('/galaxy/data/files/000/dataset_1.dat', 'input')
- >>> action.action_type == u'transfer'
- True
- >>> action.staging_needed
- True
- >>> # Always at least copy work_dir outputs.
- >>> action = mapper.action('/opt/galaxy/database/working_directory/45.sh', 'workdir')
- >>> action.action_type == u'copy'
- True
- >>> action.staging_needed
- True
- >>> # Test glob mapper (matching test)
- >>> mapper.action('/cool/bamfiles/projectABC/study1/patient3.bam', 'input').action_type == u'copy'
- True
- >>> # Test glob mapper (non-matching test)
- >>> mapper.action('/cool/bamfiles/projectABC/study1/patient3.bam.bai', 'input').action_type == u'none'
- True
- >>> # Regex mapper test.
- >>> mapper.action('/old/galaxy/data/dataset_10245.dat', 'input').action_type == u'copy'
- True
- >>> # Doesn't map unstructured paths by default
- >>> mapper.action('/old/galaxy/data/dataset_10245.dat', 'unstructured').action_type == u'none'
- True
- >>> input_only_mapper = mapper_for(default_action="none", config_contents=r'''{"paths": [ \
- {"path": "/", "action": "transfer", "path_types": "input"} \
- ] }''')
- >>> input_only_mapper.action('/dataset_1.dat', 'input').action_type == u'transfer'
- True
- >>> input_only_mapper.action('/dataset_1.dat', 'output').action_type == u'none'
- True
- >>> unstructured_mapper = mapper_for(default_action="none", config_contents=r'''{"paths": [ \
- {"path": "/", "action": "transfer", "path_types": "*any*"} \
- ] }''')
- >>> unstructured_mapper.action('/old/galaxy/data/dataset_10245.dat', 'unstructured').action_type == u'transfer'
- True
- """
-
- def __init__(self, client=None, config=None):
- if config is None and client is None:
- message = "FileActionMapper must be constructed from either a client or a config dictionary."
- raise Exception(message)
- if config is None:
- config = self.__client_to_config(client)
- self.default_action = config.get("default_action", "transfer")
- self.mappers = mappers_from_dicts(config.get("paths", []))
- self.files_endpoint = config.get("files_endpoint", None)
-
- def action(self, path, type, mapper=None):
- mapper = self.__find_mapper(path, type, mapper)
- action_class = self.__action_class(path, type, mapper)
- file_lister = DEFAULT_FILE_LISTER
- action_kwds = {}
- if mapper:
- file_lister = mapper.file_lister
- action_kwds = mapper.action_kwds
- action = action_class(path, file_lister=file_lister, **action_kwds)
- self.__process_action(action, type)
- return action
-
- def unstructured_mappers(self):
- """ Return mappers that will map 'unstructured' files (i.e. go beyond
- mapping inputs, outputs, and config files).
- """
- return filter(lambda m: path_type.UNSTRUCTURED in m.path_types, self.mappers)
-
- def to_dict(self):
- return dict(
- default_action=self.default_action,
- files_endpoint=self.files_endpoint,
- paths=map(lambda m: m.to_dict(), self.mappers)
- )
-
- def __client_to_config(self, client):
- action_config_path = client.action_config_path
- if action_config_path:
- config = read_file(action_config_path)
- else:
- config = dict()
- config["default_action"] = client.default_file_action
- config["files_endpoint"] = client.files_endpoint
- return config
-
- def __load_action_config(self, path):
- config = load(open(path, 'rb'))
- self.mappers = mappers_from_dicts(config.get('paths', []))
-
- def __find_mapper(self, path, type, mapper=None):
- if not mapper:
- normalized_path = abspath(path)
- for query_mapper in self.mappers:
- if query_mapper.matches(normalized_path, type):
- mapper = query_mapper
- break
- return mapper
-
- def __action_class(self, path, type, mapper):
- action_type = self.default_action if type in ACTION_DEFAULT_PATH_TYPES else "none"
- if mapper:
- action_type = mapper.action_type
- if type in ["workdir", "output_workdir"] and action_type == "none":
- # We are changing the working_directory relative to what
- # Galaxy would use, these need to be copied over.
- action_type = "copy"
- action_class = actions.get(action_type, None)
- if action_class is None:
- message_template = "Unknown action_type encountered %s while trying to map path %s"
- message_args = (action_type, path)
- raise Exception(message_template % message_args)
- return action_class
-
- def __process_action(self, action, file_type):
- """ Extension point to populate extra action information after an
- action has been created.
- """
- if action.action_type == "remote_transfer":
- url_base = self.files_endpoint
- if not url_base:
- raise Exception("Attempted to use remote_transfer action with defining a files_endpoint")
- if "?" not in url_base:
- url_base = "%s?" % url_base
- # TODO: URL encode path.
- url = "%s&path=%s&file_type=%s" % (url_base, action.path, file_type)
- action.url = url
-
-REQUIRED_ACTION_KWD = object()
-
-
-class BaseAction(object):
- action_spec = {}
-
- def __init__(self, path, file_lister=None):
- self.path = path
- self.file_lister = file_lister or DEFAULT_FILE_LISTER
-
- def unstructured_map(self, path_helper):
- unstructured_map = self.file_lister.unstructured_map(self.path)
- if self.staging_needed:
- # To ensure uniqueness, prepend unique prefix to each name
- prefix = unique_path_prefix(self.path)
- for path, name in unstructured_map.iteritems():
- unstructured_map[path] = join(prefix, name)
- else:
- path_rewrites = {}
- for path in unstructured_map:
- rewrite = self.path_rewrite(path_helper, path)
- if rewrite:
- path_rewrites[path] = rewrite
- unstructured_map = path_rewrites
- return unstructured_map
-
- @property
- def staging_needed(self):
- return self.staging != STAGING_ACTION_NONE
-
- @property
- def staging_action_local(self):
- return self.staging == STAGING_ACTION_LOCAL
-
-
-class NoneAction(BaseAction):
- """ This action indicates the corresponding path does not require any
- additional action. This should indicate paths that are available both on
- the LWR client (i.e. Galaxy server) and remote LWR server with the same
- paths. """
- action_type = "none"
- staging = STAGING_ACTION_NONE
-
- def to_dict(self):
- return dict(path=self.path, action_type=self.action_type)
-
- @classmethod
- def from_dict(cls, action_dict):
- return NoneAction(path=action_dict["path"])
-
- def path_rewrite(self, path_helper, path=None):
- return None
-
-
-class RewriteAction(BaseAction):
- """ This actin indicates the LWR server should simply rewrite the path
- to the specified file.
- """
- action_spec = dict(
- source_directory=REQUIRED_ACTION_KWD,
- destination_directory=REQUIRED_ACTION_KWD
- )
- action_type = "rewrite"
- staging = STAGING_ACTION_NONE
-
- def __init__(self, path, file_lister=None, source_directory=None, destination_directory=None):
- self.path = path
- self.file_lister = file_lister or DEFAULT_FILE_LISTER
- self.source_directory = source_directory
- self.destination_directory = destination_directory
-
- def to_dict(self):
- return dict(
- path=self.path,
- action_type=self.action_type,
- source_directory=self.source_directory,
- destination_directory=self.destination_directory,
- )
-
- @classmethod
- def from_dict(cls, action_dict):
- return RewriteAction(
- path=action_dict["path"],
- source_directory=action_dict["source_directory"],
- destination_directory=action_dict["destination_directory"],
- )
-
- def path_rewrite(self, path_helper, path=None):
- if not path:
- path = self.path
- new_path = path_helper.from_posix_with_new_base(self.path, self.source_directory, self.destination_directory)
- return None if new_path == self.path else new_path
-
-
-class TransferAction(BaseAction):
- """ This actions indicates that the LWR client should initiate an HTTP
- transfer of the corresponding path to the remote LWR server before
- launching the job. """
- action_type = "transfer"
- staging = STAGING_ACTION_LOCAL
-
-
-class CopyAction(BaseAction):
- """ This action indicates that the LWR client should execute a file system
- copy of the corresponding path to the LWR staging directory prior to
- launching the corresponding job. """
- action_type = "copy"
- staging = STAGING_ACTION_LOCAL
-
-
-class RemoteCopyAction(BaseAction):
- """ This action indicates the LWR server should copy the file before
- execution via direct file system copy. This is like a CopyAction, but
- it indicates the action should occur on the LWR server instead of on
- the client.
- """
- action_type = "remote_copy"
- staging = STAGING_ACTION_REMOTE
-
- def to_dict(self):
- return dict(path=self.path, action_type=self.action_type)
-
- @classmethod
- def from_dict(cls, action_dict):
- return RemoteCopyAction(path=action_dict["path"])
-
- def write_to_path(self, path):
- galaxy.util.copy_to_path(open(self.path, "rb"), path)
-
- def write_from_path(self, lwr_path):
- destination = self.path
- parent_directory = dirname(destination)
- if not exists(parent_directory):
- makedirs(parent_directory)
- with open(lwr_path, "rb") as f:
- galaxy.util.copy_to_path(f, destination)
-
-
-class RemoteTransferAction(BaseAction):
- """ This action indicates the LWR server should copy the file before
- execution via direct file system copy. This is like a CopyAction, but
- it indicates the action should occur on the LWR server instead of on
- the client.
- """
- action_type = "remote_transfer"
- staging = STAGING_ACTION_REMOTE
-
- def __init__(self, path, file_lister=None, url=None):
- super(RemoteTransferAction, self).__init__(path, file_lister=file_lister)
- self.url = url
-
- def to_dict(self):
- return dict(path=self.path, action_type=self.action_type, url=self.url)
-
- @classmethod
- def from_dict(cls, action_dict):
- return RemoteTransferAction(path=action_dict["path"], url=action_dict["url"])
-
- def write_to_path(self, path):
- get_file(self.url, path)
-
- def write_from_path(self, lwr_path):
- post_file(self.url, lwr_path)
-
-
-class MessageAction(object):
- """ Sort of pseudo action describing "files" store in memory and
- transferred via message (HTTP, Python-call, MQ, etc...)
- """
- action_type = "message"
- staging = STAGING_ACTION_DEFAULT
-
- def __init__(self, contents, client=None):
- self.contents = contents
- self.client = client
-
- @property
- def staging_needed(self):
- return True
-
- @property
- def staging_action_local(self):
- # Ekkk, cannot be called if created through from_dict.
- # Shouldn't be a problem the way it is used - but is an
- # object design problem.
- return self.client.prefer_local_staging
-
- def to_dict(self):
- return dict(contents=self.contents, action_type=MessageAction.action_type)
-
- @classmethod
- def from_dict(cls, action_dict):
- return MessageAction(contents=action_dict["contents"])
-
- def write_to_path(self, path):
- open(path, "w").write(self.contents)
-
-
-DICTIFIABLE_ACTION_CLASSES = [RemoteCopyAction, RemoteTransferAction, MessageAction]
-
-
-def from_dict(action_dict):
- action_type = action_dict.get("action_type", None)
- target_class = None
- for action_class in DICTIFIABLE_ACTION_CLASSES:
- if action_type == action_class.action_type:
- target_class = action_class
- if not target_class:
- message = "Failed to recover action from dictionary - invalid action type specified %s." % action_type
- raise Exception(message)
- return target_class.from_dict(action_dict)
-
-
-class BasePathMapper(object):
-
- def __init__(self, config):
- action_type = config.get('action', DEFAULT_MAPPED_ACTION)
- action_class = actions.get(action_type, None)
- action_kwds = action_class.action_spec.copy()
- for key, value in action_kwds.items():
- if key in config:
- action_kwds[key] = config[key]
- elif value is REQUIRED_ACTION_KWD:
- message_template = "action_type %s requires key word argument %s"
- message = message_template % (action_type, key)
- raise Exception(message)
- self.action_type = action_type
- self.action_kwds = action_kwds
- path_types_str = config.get('path_types', "*defaults*")
- path_types_str = path_types_str.replace("*defaults*", ",".join(ACTION_DEFAULT_PATH_TYPES))
- path_types_str = path_types_str.replace("*any*", ",".join(ALL_PATH_TYPES))
- self.path_types = path_types_str.split(",")
- self.file_lister = FileLister(config)
-
- def matches(self, path, path_type):
- path_type_matches = path_type in self.path_types
- return path_type_matches and self._path_matches(path)
-
- def _extend_base_dict(self, **kwds):
- base_dict = dict(
- action=self.action_type,
- path_types=",".join(self.path_types),
- match_type=self.match_type
- )
- base_dict.update(self.file_lister.to_dict())
- base_dict.update(self.action_kwds)
- base_dict.update(**kwds)
- return base_dict
-
-
-class PrefixPathMapper(BasePathMapper):
- match_type = 'prefix'
-
- def __init__(self, config):
- super(PrefixPathMapper, self).__init__(config)
- self.prefix_path = abspath(config['path'])
-
- def _path_matches(self, path):
- return path.startswith(self.prefix_path)
-
- def to_pattern(self):
- pattern_str = "(%s%s[^\s,\"\']+)" % (escape(self.prefix_path), escape(sep))
- return compile(pattern_str)
-
- def to_dict(self):
- return self._extend_base_dict(path=self.prefix_path)
-
-
-class GlobPathMapper(BasePathMapper):
- match_type = 'glob'
-
- def __init__(self, config):
- super(GlobPathMapper, self).__init__(config)
- self.glob_path = config['path']
-
- def _path_matches(self, path):
- return fnmatch.fnmatch(path, self.glob_path)
-
- def to_pattern(self):
- return compile(fnmatch.translate(self.glob_path))
-
- def to_dict(self):
- return self._extend_base_dict(path=self.glob_path)
-
-
-class RegexPathMapper(BasePathMapper):
- match_type = 'regex'
-
- def __init__(self, config):
- super(RegexPathMapper, self).__init__(config)
- self.pattern_raw = config['path']
- self.pattern = compile(self.pattern_raw)
-
- def _path_matches(self, path):
- return self.pattern.match(path) is not None
-
- def to_pattern(self):
- return self.pattern
-
- def to_dict(self):
- return self._extend_base_dict(path=self.pattern_raw)
-
-MAPPER_CLASSES = [PrefixPathMapper, GlobPathMapper, RegexPathMapper]
-MAPPER_CLASS_DICT = dict(map(lambda c: (c.match_type, c), MAPPER_CLASSES))
-
-
-def mappers_from_dicts(mapper_def_list):
- return map(lambda m: __mappper_from_dict(m), mapper_def_list)
-
-
-def __mappper_from_dict(mapper_dict):
- map_type = mapper_dict.get('match_type', DEFAULT_PATH_MAPPER_TYPE)
- return MAPPER_CLASS_DICT[map_type](mapper_dict)
-
-
-class FileLister(object):
-
- def __init__(self, config):
- self.depth = int(config.get("depth", "0"))
-
- def to_dict(self):
- return dict(
- depth=self.depth
- )
-
- def unstructured_map(self, path):
- depth = self.depth
- if self.depth == 0:
- return {path: basename(path)}
- else:
- while depth > 0:
- path = dirname(path)
- depth -= 1
- return dict([(join(path, f), f) for f in directory_files(path)])
-
-DEFAULT_FILE_LISTER = FileLister(dict(depth=0))
-
-ACTION_CLASSES = [
- NoneAction,
- RewriteAction,
- TransferAction,
- CopyAction,
- RemoteCopyAction,
- RemoteTransferAction,
-]
-actions = dict([(clazz.action_type, clazz) for clazz in ACTION_CLASSES])
-
-
-__all__ = [
- FileActionMapper,
- path_type,
- from_dict,
- MessageAction,
- RemoteTransferAction, # For testing
-]
diff --git a/lib/galaxy/jobs/runners/lwr_client/amqp_exchange.py b/lib/galaxy/jobs/runners/lwr_client/amqp_exchange.py
deleted file mode 100644
index 96ba0db559d..00000000000
--- a/lib/galaxy/jobs/runners/lwr_client/amqp_exchange.py
+++ /dev/null
@@ -1,139 +0,0 @@
-try:
- import kombu
- from kombu import pools
-except ImportError:
- kombu = None
-
-import socket
-import logging
-import threading
-from time import sleep
-log = logging.getLogger(__name__)
-
-
-KOMBU_UNAVAILABLE = "Attempting to bind to AMQP message queue, but kombu dependency unavailable"
-
-DEFAULT_EXCHANGE_NAME = "lwr"
-DEFAULT_EXCHANGE_TYPE = "direct"
-# Set timeout to periodically give up looking and check if polling should end.
-DEFAULT_TIMEOUT = 0.2
-DEFAULT_HEARTBEAT = 580
-
-DEFAULT_RECONNECT_CONSUMER_WAIT = 1
-DEFAULT_HEARTBEAT_WAIT = 1
-
-
-class LwrExchange(object):
- """ Utility for publishing and consuming structured LWR queues using kombu.
- This is shared between the server and client - an exchange should be setup
- for each manager (or in the case of the client, each manager one wished to
- communicate with.)
-
- Each LWR manager is defined solely by name in the scheme, so only one LWR
- should target each AMQP endpoint or care should be taken that unique
- manager names are used across LWR servers targetting same AMQP endpoint -
- and in particular only one such LWR should define an default manager with
- name _default_.
- """
-
- def __init__(
- self,
- url,
- manager_name,
- connect_ssl=None,
- timeout=DEFAULT_TIMEOUT,
- publish_kwds={},
- ):
- """
- """
- if not kombu:
- raise Exception(KOMBU_UNAVAILABLE)
- self.__url = url
- self.__manager_name = manager_name
- self.__connect_ssl = connect_ssl
- self.__exchange = kombu.Exchange(DEFAULT_EXCHANGE_NAME, DEFAULT_EXCHANGE_TYPE)
- self.__timeout = timeout
- # Be sure to log message publishing failures.
- if publish_kwds.get("retry", False):
- if "retry_policy" not in publish_kwds:
- publish_kwds["retry_policy"] = {}
- if "errback" not in publish_kwds["retry_policy"]:
- publish_kwds["retry_policy"]["errback"] = self.__publish_errback
- self.__publish_kwds = publish_kwds
-
- @property
- def url(self):
- return self.__url
-
- def consume(self, queue_name, callback, check=True, connection_kwargs={}):
- queue = self.__queue(queue_name)
- log.debug("Consuming queue '%s'", queue)
- while check:
- heartbeat_thread = None
- try:
- with self.connection(self.__url, heartbeat=DEFAULT_HEARTBEAT, **connection_kwargs) as connection:
- with kombu.Consumer(connection, queues=[queue], callbacks=[callback], accept=['json']):
- heartbeat_thread = self.__start_heartbeat(queue_name, connection)
- while check and connection.connected:
- try:
- connection.drain_events(timeout=self.__timeout)
- except socket.timeout:
- pass
- except (IOError, socket.error) as exc:
- # In testing, errno is None
- log.warning('Got %s, will retry: %s', exc.__class__.__name__, exc)
- if heartbeat_thread:
- heartbeat_thread.join()
- sleep(DEFAULT_RECONNECT_CONSUMER_WAIT)
-
- def heartbeat(self, connection):
- log.debug('AMQP heartbeat thread alive')
- while connection.connected:
- connection.heartbeat_check()
- sleep(DEFAULT_HEARTBEAT_WAIT)
- log.debug('AMQP heartbeat thread exiting')
-
- def publish(self, name, payload):
- with self.connection(self.__url) as connection:
- with pools.producers[connection].acquire() as producer:
- key = self.__queue_name(name)
- producer.publish(
- payload,
- serializer='json',
- exchange=self.__exchange,
- declare=[self.__exchange],
- routing_key=key,
- **self.__publish_kwds
- )
-
- def __publish_errback(self, exc, interval):
- log.error("Connection error while publishing: %r", exc, exc_info=1)
- log.info("Retrying in %s seconds", interval)
-
- def connection(self, connection_string, **kwargs):
- if "ssl" not in kwargs:
- kwargs["ssl"] = self.__connect_ssl
- return kombu.Connection(connection_string, **kwargs)
-
- def __queue(self, name):
- queue_name = self.__queue_name(name)
- queue = kombu.Queue(queue_name, self.__exchange, routing_key=queue_name)
- return queue
-
- def __queue_name(self, name):
- key_prefix = self.__key_prefix()
- queue_name = '%s_%s' % (key_prefix, name)
- return queue_name
-
- def __key_prefix(self):
- if self.__manager_name == "_default_":
- key_prefix = "lwr_"
- else:
- key_prefix = "lwr_%s_" % self.__manager_name
- return key_prefix
-
- def __start_heartbeat(self, queue_name, connection):
- thread_name = "consume-heartbeat-%s" % (self.__queue_name(queue_name))
- thread = threading.Thread(name=thread_name, target=self.heartbeat, args=(connection,))
- thread.start()
- return thread
diff --git a/lib/galaxy/jobs/runners/lwr_client/amqp_exchange_factory.py b/lib/galaxy/jobs/runners/lwr_client/amqp_exchange_factory.py
deleted file mode 100644
index c35a9648800..00000000000
--- a/lib/galaxy/jobs/runners/lwr_client/amqp_exchange_factory.py
+++ /dev/null
@@ -1,41 +0,0 @@
-from .amqp_exchange import LwrExchange
-from .util import filter_destination_params
-
-
-def get_exchange(url, manager_name, params):
- connect_ssl = parse_amqp_connect_ssl_params(params)
- exchange_kwds = dict(
- manager_name=manager_name,
- connect_ssl=connect_ssl,
- publish_kwds=parse_amqp_publish_kwds(params)
- )
- timeout = params.get('amqp_consumer_timeout', False)
- if timeout is not False:
- exchange_kwds['timeout'] = timeout
- exchange = LwrExchange(url, **exchange_kwds)
- return exchange
-
-
-def parse_amqp_connect_ssl_params(params):
- ssl_params = filter_destination_params(params, "amqp_connect_ssl_")
- if not ssl_params:
- return
-
- ssl = __import__('ssl')
- if 'cert_reqs' in ssl_params:
- value = ssl_params['cert_reqs']
- ssl_params['cert_reqs'] = getattr(ssl, value.upper())
- return ssl_params
-
-
-def parse_amqp_publish_kwds(params):
- all_publish_params = filter_destination_params(params, "amqp_publish_")
- retry_policy_params = {}
- for key in all_publish_params.keys():
- if key.startswith("retry_"):
- value = all_publish_params[key]
- retry_policy_params[key[len("retry_"):]] = value
- del all_publish_params[key]
- if retry_policy_params:
- all_publish_params["retry_policy"] = retry_policy_params
- return all_publish_params
diff --git a/lib/galaxy/jobs/runners/lwr_client/client.py b/lib/galaxy/jobs/runners/lwr_client/client.py
deleted file mode 100644
index 7656b05202c..00000000000
--- a/lib/galaxy/jobs/runners/lwr_client/client.py
+++ /dev/null
@@ -1,428 +0,0 @@
-import os
-from json import dumps
-
-from .destination import submit_params
-from .setup_handler import build as build_setup_handler
-from .job_directory import RemoteJobDirectory
-from .decorators import parseJson
-from .decorators import retry
-from .util import copy
-from .util import ensure_directory
-from .util import to_base64_json
-
-
-import logging
-log = logging.getLogger(__name__)
-
-CACHE_WAIT_SECONDS = 3
-
-
-class OutputNotFoundException(Exception):
-
- def __init__(self, path):
- self.path = path
-
- def __str__(self):
- return "No remote output found for path %s" % self.path
-
-
-class BaseJobClient(object):
-
- def __init__(self, destination_params, job_id):
- self.destination_params = destination_params
- self.job_id = job_id
- if "jobs_directory" in (destination_params or {}):
- staging_directory = destination_params["jobs_directory"]
- sep = destination_params.get("remote_sep", os.sep)
- job_directory = RemoteJobDirectory(
- remote_staging_directory=staging_directory,
- remote_id=job_id,
- remote_sep=sep,
- )
- else:
- job_directory = None
- self.env = destination_params.get("env", [])
- self.files_endpoint = destination_params.get("files_endpoint", None)
- self.job_directory = job_directory
-
- self.default_file_action = self.destination_params.get("default_file_action", "transfer")
- self.action_config_path = self.destination_params.get("file_action_config", None)
-
- self.setup_handler = build_setup_handler(self, destination_params)
-
- def setup(self, tool_id=None, tool_version=None):
- """
- Setup remote LWR server to run this job.
- """
- setup_args = {"job_id": self.job_id}
- if tool_id:
- setup_args["tool_id"] = tool_id
- if tool_version:
- setup_args["tool_version"] = tool_version
- return self.setup_handler.setup(**setup_args)
-
- @property
- def prefer_local_staging(self):
- # If doing a job directory is defined, calculate paths here and stage
- # remotely.
- return self.job_directory is None
-
-
-class JobClient(BaseJobClient):
- """
- Objects of this client class perform low-level communication with a remote LWR server.
-
- **Parameters**
-
- destination_params : dict or str
- connection parameters, either url with dict containing url (and optionally `private_token`).
- job_id : str
- Galaxy job/task id.
- """
-
- def __init__(self, destination_params, job_id, job_manager_interface):
- super(JobClient, self).__init__(destination_params, job_id)
- self.job_manager_interface = job_manager_interface
-
- def launch(self, command_line, dependencies_description=None, env=[], remote_staging=[], job_config=None):
- """
- Queue up the execution of the supplied `command_line` on the remote
- server. Called launch for historical reasons, should be renamed to
- enqueue or something like that.
-
- **Parameters**
-
- command_line : str
- Command to execute.
- """
- launch_params = dict(command_line=command_line, job_id=self.job_id)
- submit_params_dict = submit_params(self.destination_params)
- if submit_params_dict:
- launch_params['params'] = dumps(submit_params_dict)
- if dependencies_description:
- launch_params['dependencies_description'] = dumps(dependencies_description.to_dict())
- if env:
- launch_params['env'] = dumps(env)
- if remote_staging:
- launch_params['remote_staging'] = dumps(remote_staging)
- if job_config and self.setup_handler.local:
- # Setup not yet called, job properties were inferred from
- # destination arguments. Hence, must have LWR setup job
- # before queueing.
- setup_params = _setup_params_from_job_config(job_config)
- launch_params["setup_params"] = dumps(setup_params)
- return self._raw_execute("launch", launch_params)
-
- def full_status(self):
- """ Return a dictionary summarizing final state of job.
- """
- return self.raw_check_complete()
-
- def kill(self):
- """
- Cancel remote job, either removing from the queue or killing it.
- """
- return self._raw_execute("kill", {"job_id": self.job_id})
-
- @retry()
- @parseJson()
- def raw_check_complete(self):
- """
- Get check_complete response from the remote server.
- """
- check_complete_response = self._raw_execute("check_complete", {"job_id": self.job_id})
- return check_complete_response
-
- def get_status(self):
- check_complete_response = self.raw_check_complete()
- # Older LWR instances won't set status so use 'complete', at some
- # point drop backward compatibility.
- status = check_complete_response.get("status", None)
- if status in ["status", None]:
- # LEGACY: Bug in certains older LWR instances returned literal
- # "status".
- complete = check_complete_response["complete"] == "true"
- old_status = "complete" if complete else "running"
- status = old_status
- return status
-
- def clean(self):
- """
- Cleanup the remote job.
- """
- self._raw_execute("clean", {"job_id": self.job_id})
-
- @parseJson()
- def remote_setup(self, **setup_args):
- """
- Setup remote LWR server to run this job.
- """
- return self._raw_execute("setup", setup_args)
-
- def put_file(self, path, input_type, name=None, contents=None, action_type='transfer'):
- if not name:
- name = os.path.basename(path)
- args = {"job_id": self.job_id, "name": name, "input_type": input_type}
- input_path = path
- if contents:
- input_path = None
- if action_type == 'transfer':
- return self._upload_file(args, contents, input_path)
- elif action_type == 'copy':
- lwr_path = self._raw_execute('input_path', args)
- copy(path, lwr_path)
- return {'path': lwr_path}
-
- def fetch_output(self, path, name, working_directory, action_type, output_type):
- """
- Fetch (transfer, copy, etc...) an output from the remote LWR server.
-
- **Parameters**
-
- path : str
- Local path of the dataset.
- name : str
- Remote name of file (i.e. path relative to remote staging output
- or working directory).
- working_directory : str
- Local working_directory for the job.
- action_type : str
- Where to find file on LWR (output_workdir or output). legacy is also
- an option in this case LWR is asked for location - this will only be
- used if targetting an older LWR server that didn't return statuses
- allowing this to be inferred.
- """
- if output_type == 'legacy':
- self._fetch_output_legacy(path, working_directory, action_type=action_type)
- elif output_type == 'output_workdir':
- self._fetch_work_dir_output(name, working_directory, path, action_type=action_type)
- elif output_type == 'output':
- self._fetch_output(path=path, name=name, action_type=action_type)
- else:
- raise Exception("Unknown output_type %s" % output_type)
-
- def _raw_execute(self, command, args={}, data=None, input_path=None, output_path=None):
- return self.job_manager_interface.execute(command, args, data, input_path, output_path)
-
- # Deprecated
- def _fetch_output_legacy(self, path, working_directory, action_type='transfer'):
- # Needs to determine if output is task/working directory or standard.
- name = os.path.basename(path)
-
- output_type = self._get_output_type(name)
- if output_type == "none":
- # Just make sure the file was created.
- if not os.path.exists(path):
- raise OutputNotFoundException(path)
- return
- elif output_type in ["task"]:
- path = os.path.join(working_directory, name)
-
- self.__populate_output_path(name, path, output_type, action_type)
-
- def _fetch_output(self, path, name=None, check_exists_remotely=False, action_type='transfer'):
- if not name:
- # Extra files will send in the path.
- name = os.path.basename(path)
-
- output_type = "direct" # Task/from_work_dir outputs now handled with fetch_work_dir_output
- self.__populate_output_path(name, path, output_type, action_type)
-
- def _fetch_work_dir_output(self, name, working_directory, output_path, action_type='transfer'):
- ensure_directory(output_path)
- if action_type == 'transfer':
- self.__raw_download_output(name, self.job_id, "work_dir", output_path)
- else: # Even if action is none - LWR has a different work_dir so this needs to be copied.
- lwr_path = self._output_path(name, self.job_id, 'work_dir')['path']
- copy(lwr_path, output_path)
-
- def __populate_output_path(self, name, output_path, output_type, action_type):
- ensure_directory(output_path)
- if action_type == 'transfer':
- self.__raw_download_output(name, self.job_id, output_type, output_path)
- elif action_type == 'copy':
- lwr_path = self._output_path(name, self.job_id, output_type)['path']
- copy(lwr_path, output_path)
-
- @parseJson()
- def _upload_file(self, args, contents, input_path):
- return self._raw_execute(self._upload_file_action(args), args, contents, input_path)
-
- def _upload_file_action(self, args):
- # Hack for backward compatibility, instead of using new upload_file
- # path. Use old paths.
- input_type = args['input_type']
- action = {
- # For backward compatibility just target upload_input_extra for all
- # inputs, it allows nested inputs. Want to do away with distinction
- # inputs and extra inputs.
- 'input': 'upload_extra_input',
- 'config': 'upload_config_file',
- 'workdir': 'upload_working_directory_file',
- 'tool': 'upload_tool_file',
- 'unstructured': 'upload_unstructured_file',
- }[input_type]
- del args['input_type']
- return action
-
- @parseJson()
- def _get_output_type(self, name):
- return self._raw_execute("get_output_type", {"name": name,
- "job_id": self.job_id})
-
- @parseJson()
- def _output_path(self, name, job_id, output_type):
- return self._raw_execute("output_path",
- {"name": name,
- "job_id": self.job_id,
- "output_type": output_type})
-
- @retry()
- def __raw_download_output(self, name, job_id, output_type, output_path):
- output_params = {
- "name": name,
- "job_id": self.job_id,
- "output_type": output_type
- }
- self._raw_execute("download_output", output_params, output_path=output_path)
-
-
-class BaseMessageJobClient(BaseJobClient):
-
- def __init__(self, destination_params, job_id, client_manager):
- super(BaseMessageJobClient, self).__init__(destination_params, job_id)
- if not self.job_directory:
- error_message = "Message-queue based LWR client requires destination define a remote job_directory to stage files into."
- raise Exception(error_message)
- self.client_manager = client_manager
-
- def clean(self):
- del self.client_manager.status_cache[self.job_id]
-
- def full_status(self):
- full_status = self.client_manager.status_cache.get(self.job_id, None)
- if full_status is None:
- raise Exception("full_status() called before a final status was properly cached with cilent manager.")
- return full_status
-
- def _build_setup_message(self, command_line, dependencies_description, env, remote_staging, job_config):
- """
- """
- launch_params = dict(command_line=command_line, job_id=self.job_id)
- submit_params_dict = submit_params(self.destination_params)
- if submit_params_dict:
- launch_params['submit_params'] = submit_params_dict
- if dependencies_description:
- launch_params['dependencies_description'] = dependencies_description.to_dict()
- if env:
- launch_params['env'] = env
- if remote_staging:
- launch_params['remote_staging'] = remote_staging
- if job_config and self.setup_handler.local:
- # Setup not yet called, job properties were inferred from
- # destination arguments. Hence, must have LWR setup job
- # before queueing.
- setup_params = _setup_params_from_job_config(job_config)
- launch_params["setup_params"] = setup_params
- return launch_params
-
-
-class MessageJobClient(BaseMessageJobClient):
-
- def launch(self, command_line, dependencies_description=None, env=[], remote_staging=[], job_config=None):
- """
- """
- launch_params = self._build_setup_message(
- command_line,
- dependencies_description=dependencies_description,
- env=env,
- remote_staging=remote_staging,
- job_config=job_config
- )
- response = self.client_manager.exchange.publish("setup", launch_params)
- log.info("Job published to setup message queue.")
- return response
-
- def kill(self):
- self.client_manager.exchange.publish("kill", dict(job_id=self.job_id))
-
-
-class MessageCLIJobClient(BaseMessageJobClient):
-
- def __init__(self, destination_params, job_id, client_manager, shell):
- super(MessageCLIJobClient, self).__init__(destination_params, job_id, client_manager)
- self.remote_lwr_path = destination_params["remote_lwr_path"]
- self.shell = shell
-
- def launch(self, command_line, dependencies_description=None, env=[], remote_staging=[], job_config=None):
- """
- """
- launch_params = self._build_setup_message(
- command_line,
- dependencies_description=dependencies_description,
- env=env,
- remote_staging=remote_staging,
- job_config=job_config
- )
- base64_message = to_base64_json(launch_params)
- submit_command = os.path.join(self.remote_lwr_path, "scripts", "submit.bash")
- # TODO: Allow configuration of manager, app, and ini path...
- self.shell.execute("nohup %s --base64 %s &" % (submit_command, base64_message))
-
- def kill(self):
- # TODO
- pass
-
-
-class InputCachingJobClient(JobClient):
- """
- Beta client that cache's staged files to prevent duplication.
- """
-
- def __init__(self, destination_params, job_id, job_manager_interface, client_cacher):
- super(InputCachingJobClient, self).__init__(destination_params, job_id, job_manager_interface)
- self.client_cacher = client_cacher
-
- @parseJson()
- def _upload_file(self, args, contents, input_path):
- action = self._upload_file_action(args)
- if contents:
- input_path = None
- return self._raw_execute(action, args, contents, input_path)
- else:
- event_holder = self.client_cacher.acquire_event(input_path)
- cache_required = self.cache_required(input_path)
- if cache_required:
- self.client_cacher.queue_transfer(self, input_path)
- while not event_holder.failed:
- available = self.file_available(input_path)
- if available['ready']:
- token = available['token']
- args["cache_token"] = token
- return self._raw_execute(action, args)
- event_holder.event.wait(30)
- if event_holder.failed:
- raise Exception("Failed to transfer file %s" % input_path)
-
- @parseJson()
- def cache_required(self, path):
- return self._raw_execute("cache_required", {"path": path})
-
- @parseJson()
- def cache_insert(self, path):
- return self._raw_execute("cache_insert", {"path": path}, None, path)
-
- @parseJson()
- def file_available(self, path):
- return self._raw_execute("file_available", {"path": path})
-
-
-def _setup_params_from_job_config(job_config):
- job_id = job_config.get("job_id", None)
- tool_id = job_config.get("tool_id", None)
- tool_version = job_config.get("tool_version", None)
- return dict(
- job_id=job_id,
- tool_id=tool_id,
- tool_version=tool_version
- )
diff --git a/lib/galaxy/jobs/runners/lwr_client/config_util.py b/lib/galaxy/jobs/runners/lwr_client/config_util.py
deleted file mode 100644
index 60d5f0d6d62..00000000000
--- a/lib/galaxy/jobs/runners/lwr_client/config_util.py
+++ /dev/null
@@ -1,72 +0,0 @@
-""" Generic interface for reading YAML/INI/JSON config files into nested dictionaries.
-"""
-
-import yaml
-
-try:
- from ConfigParser import ConfigParser
-except ImportError:
- from configparser import ConfigParser
-import json
-
-
-CONFIG_TYPE_JSON = "json"
-CONFIG_TYPE_YAML = "yaml"
-CONFIG_TYPE_INI = "ini"
-
-DEFAULT_CONFIG_TYPE = CONFIG_TYPE_YAML
-
-JSON_EXTS = [".json"]
-YAML_EXTS = [".yaml", ".yml"]
-INI_EXTS = [".ini"]
-
-EXT_MAP = {
- CONFIG_TYPE_JSON: JSON_EXTS,
- CONFIG_TYPE_YAML: YAML_EXTS,
- CONFIG_TYPE_INI: INI_EXTS,
-}
-
-
-def read_file(path, type=None, default_type=DEFAULT_CONFIG_TYPE):
- if path is None:
- raise ValueError("Undefined path supplied.")
-
- config_type = __find_type(path, type, default_type)
- return EXT_READERS[config_type](path)
-
-
-def __find_type(path, explicit_type, default_type):
- if explicit_type:
- return explicit_type
-
- for config_type, config_exts in EXT_MAP.items():
- for ext in config_exts:
- if path.endswith(ext):
- return config_type
-
- return default_type
-
-
-def __read_yaml(path):
- if yaml is None:
- raise ImportError("Attempting to read YAML configuration file - but PyYAML dependency unavailable.")
-
- with open(path, "rb") as f:
- return yaml.load(f)
-
-
-def __read_ini(path):
- config = ConfigParser()
- config.read(path)
- return config._sections
-
-
-def __read_json(path):
- with open(path, "rb") as f:
- return json.load(f)
-
-EXT_READERS = {
- CONFIG_TYPE_JSON: __read_json,
- CONFIG_TYPE_YAML: __read_yaml,
- CONFIG_TYPE_INI: __read_ini,
-}
diff --git a/lib/galaxy/jobs/runners/lwr_client/decorators.py b/lib/galaxy/jobs/runners/lwr_client/decorators.py
deleted file mode 100644
index 94035f5ba1b..00000000000
--- a/lib/galaxy/jobs/runners/lwr_client/decorators.py
+++ /dev/null
@@ -1,35 +0,0 @@
-import time
-import json
-
-MAX_RETRY_COUNT = 5
-RETRY_SLEEP_TIME = 0.1
-
-
-class parseJson(object):
-
- def __call__(self, func):
- def replacement(*args, **kwargs):
- response = func(*args, **kwargs)
- return json.loads(response)
- return replacement
-
-
-class retry(object):
-
- def __call__(self, func):
-
- def replacement(*args, **kwargs):
- max_count = MAX_RETRY_COUNT
- count = 0
- while True:
- count += 1
- try:
- return func(*args, **kwargs)
- except:
- if count >= max_count:
- raise
- else:
- time.sleep(RETRY_SLEEP_TIME)
- continue
-
- return replacement
diff --git a/lib/galaxy/jobs/runners/lwr_client/destination.py b/lib/galaxy/jobs/runners/lwr_client/destination.py
deleted file mode 100644
index 53569f6f260..00000000000
--- a/lib/galaxy/jobs/runners/lwr_client/destination.py
+++ /dev/null
@@ -1,58 +0,0 @@
-
-from re import match
-from .util import filter_destination_params
-
-SUBMIT_PREFIX = "submit_"
-
-
-def url_to_destination_params(url):
- """Convert a legacy runner URL to a job destination
-
- >>> params_simple = url_to_destination_params("http://localhost:8913/")
- >>> params_simple["url"]
- 'http://localhost:8913/'
- >>> params_simple["private_token"] is None
- True
- >>> advanced_url = "https://1234x@example.com:8914/managers/longqueue"
- >>> params_advanced = url_to_destination_params(advanced_url)
- >>> params_advanced["url"]
- 'https://example.com:8914/managers/longqueue/'
- >>> params_advanced["private_token"]
- '1234x'
- >>> runner_url = "lwr://http://localhost:8913/"
- >>> runner_params = url_to_destination_params(runner_url)
- >>> runner_params['url']
- 'http://localhost:8913/'
- """
-
- if url.startswith("lwr://"):
- url = url[len("lwr://"):]
-
- if not url.endswith("/"):
- url += "/"
-
- # Check for private token embedded in the URL. A URL of the form
- # https://moo@cow:8913 will try to contact https://cow:8913
- # with a private key of moo
- private_token_format = "https?://(.*)@.*/?"
- private_token_match = match(private_token_format, url)
- private_token = None
- if private_token_match:
- private_token = private_token_match.group(1)
- url = url.replace("%s@" % private_token, '', 1)
-
- destination_args = {"url": url,
- "private_token": private_token}
-
- return destination_args
-
-
-def submit_params(destination_params):
- """
-
- >>> destination_params = {"private_token": "12345", "submit_native_specification": "-q batch"}
- >>> result = submit_params(destination_params)
- >>> result
- {'native_specification': '-q batch'}
- """
- return filter_destination_params(destination_params, SUBMIT_PREFIX)
diff --git a/lib/galaxy/jobs/runners/lwr_client/interface.py b/lib/galaxy/jobs/runners/lwr_client/interface.py
deleted file mode 100644
index 95538d224da..00000000000
--- a/lib/galaxy/jobs/runners/lwr_client/interface.py
+++ /dev/null
@@ -1,102 +0,0 @@
-from abc import ABCMeta
-from abc import abstractmethod
-try:
- from StringIO import StringIO as BytesIO
-except ImportError:
- from io import BytesIO
-try:
- from six import text_type
-except ImportError:
- from galaxy.util import unicodify as text_type
-try:
- from urllib import urlencode
-except ImportError:
- from urllib.parse import urlencode
-
-
-class LwrInteface(object):
- """
- Abstract base class describes how synchronous client communicates with
- (potentially remote) LWR procedures. Obvious implementation is HTTP based
- but LWR objects wrapped in routes can also be directly communicated with
- if in memory.
- """
- __metaclass__ = ABCMeta
-
- @abstractmethod
- def execute(self, command, args={}, data=None, input_path=None, output_path=None):
- """
- Execute the correspond command against configured LWR job manager. Arguments are
- method parameters and data or input_path describe essentially POST bodies. If command
- results in a file, resulting path should be specified as output_path.
- """
-
-
-class HttpLwrInterface(LwrInteface):
-
- def __init__(self, destination_params, transport):
- self.transport = transport
- remote_host = destination_params.get("url")
- assert remote_host is not None, "Failed to determine url for LWR client."
- if not remote_host.endswith("/"):
- remote_host = "%s/" % remote_host
- if not remote_host.startswith("http"):
- remote_host = "http://%s" % remote_host
- self.remote_host = remote_host
- self.private_key = destination_params.get("private_token", None)
-
- def execute(self, command, args={}, data=None, input_path=None, output_path=None):
- url = self.__build_url(command, args)
- response = self.transport.execute(url, data=data, input_path=input_path, output_path=output_path)
- return response
-
- def __build_url(self, command, args):
- if self.private_key:
- args["private_key"] = self.private_key
- arg_bytes = dict([(k, text_type(args[k]).encode('utf-8')) for k in args])
- data = urlencode(arg_bytes)
- url = self.remote_host + command + "?" + data
- return url
-
-
-class LocalLwrInterface(LwrInteface):
-
- def __init__(self, destination_params, job_manager=None, file_cache=None, object_store=None):
- self.job_manager = job_manager
- self.file_cache = file_cache
- self.object_store = object_store
-
- def __app_args(self):
- # Arguments that would be specified from LwrApp if running
- # in web server.
- return {
- 'manager': self.job_manager,
- 'file_cache': self.file_cache,
- 'object_store': self.object_store,
- 'ip': None
- }
-
- def execute(self, command, args={}, data=None, input_path=None, output_path=None):
- # If data set, should be unicode (on Python 2) or str (on Python 3).
- from lwr.web import routes
- from lwr.web.framework import build_func_args
- controller = getattr(routes, command)
- action = controller.func
- body_args = dict(body=self.__build_body(data, input_path))
- args = build_func_args(action, args.copy(), self.__app_args(), body_args)
- result = action(**args)
- if controller.response_type != 'file':
- return controller.body(result)
- else:
- # TODO: Add to Galaxy.
- from galaxy.util import copy_to_path
- with open(result, 'rb') as result_file:
- copy_to_path(result_file, output_path)
-
- def __build_body(self, data, input_path):
- if data is not None:
- return BytesIO(data.encode('utf-8'))
- elif input_path is not None:
- return open(input_path, 'rb')
- else:
- return None
diff --git a/lib/galaxy/jobs/runners/lwr_client/job_directory.py b/lib/galaxy/jobs/runners/lwr_client/job_directory.py
deleted file mode 100644
index b07e1c539ee..00000000000
--- a/lib/galaxy/jobs/runners/lwr_client/job_directory.py
+++ /dev/null
@@ -1,137 +0,0 @@
-"""
-"""
-import os.path
-from collections import deque
-import posixpath
-
-from .util import PathHelper
-from galaxy.util import in_directory
-
-from logging import getLogger
-log = getLogger(__name__)
-
-
-TYPES_TO_METHOD = dict(
- input="inputs_directory",
- input_extra="inputs_directory",
- unstructured="unstructured_files_directory",
- config="configs_directory",
- tool="tool_files_directory",
- work_dir="working_directory",
- workdir="working_directory",
- output="outputs_directory",
- output_workdir="working_directory",
-)
-
-
-class RemoteJobDirectory(object):
- """ Representation of a (potentially) remote LWR-style staging directory.
- """
-
- def __init__(self, remote_staging_directory, remote_id, remote_sep):
- self.path_helper = PathHelper(remote_sep)
- self.job_directory = self.path_helper.remote_join(
- remote_staging_directory,
- remote_id
- )
-
- def working_directory(self):
- return self._sub_dir('working')
-
- def inputs_directory(self):
- return self._sub_dir('inputs')
-
- def outputs_directory(self):
- return self._sub_dir('outputs')
-
- def configs_directory(self):
- return self._sub_dir('configs')
-
- def tool_files_directory(self):
- return self._sub_dir('tool_files')
-
- def unstructured_files_directory(self):
- return self._sub_dir('unstructured')
-
- @property
- def path(self):
- return self.job_directory
-
- @property
- def separator(self):
- return self.path_helper.separator
-
- def calculate_path(self, remote_relative_path, input_type):
- """ Only for used by LWR client, should override for managers to
- enforce security and make the directory if needed.
- """
- directory, allow_nested_files = self._directory_for_file_type(input_type)
- return self.path_helper.remote_join(directory, remote_relative_path)
-
- def _directory_for_file_type(self, file_type):
- allow_nested_files = False
- # work_dir and input_extra are types used by legacy clients...
- # Obviously this client won't be legacy because this is in the
- # client module, but this code is reused on server which may
- # serve legacy clients.
- allow_nested_files = file_type in ['input', 'input_extra', 'unstructured', 'output', 'output_workdir']
- directory_function = getattr(self, TYPES_TO_METHOD.get(file_type, None), None)
- if not directory_function:
- raise Exception("Unknown file_type specified %s" % file_type)
- return directory_function(), allow_nested_files
-
- def _sub_dir(self, name):
- return self.path_helper.remote_join(self.job_directory, name)
-
-
-def get_mapped_file(directory, remote_path, allow_nested_files=False, local_path_module=os.path, mkdir=True):
- """
-
- >>> import ntpath
- >>> get_mapped_file(r'C:\\lwr\\staging\\101', 'dataset_1_files/moo/cow', allow_nested_files=True, local_path_module=ntpath, mkdir=False)
- 'C:\\\\lwr\\\\staging\\\\101\\\\dataset_1_files\\\\moo\\\\cow'
- >>> get_mapped_file(r'C:\\lwr\\staging\\101', 'dataset_1_files/moo/cow', allow_nested_files=False, local_path_module=ntpath)
- 'C:\\\\lwr\\\\staging\\\\101\\\\cow'
- >>> get_mapped_file(r'C:\\lwr\\staging\\101', '../cow', allow_nested_files=True, local_path_module=ntpath, mkdir=False)
- Traceback (most recent call last):
- Exception: Attempt to read or write file outside an authorized directory.
- """
- if not allow_nested_files:
- name = local_path_module.basename(remote_path)
- path = local_path_module.join(directory, name)
- else:
- local_rel_path = __posix_to_local_path(remote_path, local_path_module=local_path_module)
- local_path = local_path_module.join(directory, local_rel_path)
- verify_is_in_directory(local_path, directory, local_path_module=local_path_module)
- local_directory = local_path_module.dirname(local_path)
- if mkdir and not local_path_module.exists(local_directory):
- os.makedirs(local_directory)
- path = local_path
- return path
-
-
-def __posix_to_local_path(path, local_path_module=os.path):
- """
- Converts a posix path (coming from Galaxy), to a local path (be it posix or Windows).
-
- >>> import ntpath
- >>> __posix_to_local_path('dataset_1_files/moo/cow', local_path_module=ntpath)
- 'dataset_1_files\\\\moo\\\\cow'
- >>> import posixpath
- >>> __posix_to_local_path('dataset_1_files/moo/cow', local_path_module=posixpath)
- 'dataset_1_files/moo/cow'
- """
- partial_path = deque()
- while True:
- if not path or path == '/':
- break
- (path, base) = posixpath.split(path)
- partial_path.appendleft(base)
- return local_path_module.join(*partial_path)
-
-
-def verify_is_in_directory(path, directory, local_path_module=os.path):
- if not in_directory(path, directory, local_path_module):
- msg = "Attempt to read or write file outside an authorized directory."
- log.warn("%s Attempted path: %s, valid directory: %s" % (msg, path, directory))
- raise Exception(msg)
diff --git a/lib/galaxy/jobs/runners/lwr_client/manager.py b/lib/galaxy/jobs/runners/lwr_client/manager.py
deleted file mode 100644
index fb6d6c63a84..00000000000
--- a/lib/galaxy/jobs/runners/lwr_client/manager.py
+++ /dev/null
@@ -1,223 +0,0 @@
-import threading
-try:
- from Queue import Queue
-except ImportError:
- from queue import Queue
-from os import getenv
-
-from .client import JobClient
-from .client import InputCachingJobClient
-from .client import MessageJobClient
-from .client import MessageCLIJobClient
-from .interface import HttpLwrInterface
-from .interface import LocalLwrInterface
-from .object_client import ObjectStoreClient
-from .transport import get_transport
-from .util import TransferEventManager
-from .destination import url_to_destination_params
-from .amqp_exchange_factory import get_exchange
-
-
-from logging import getLogger
-log = getLogger(__name__)
-
-DEFAULT_TRANSFER_THREADS = 2
-
-
-def build_client_manager(**kwargs):
- if 'job_manager' in kwargs:
- return ClientManager(**kwargs) # TODO: Consider more separation here.
- elif kwargs.get('url', None):
- return MessageQueueClientManager(**kwargs)
- else:
- return ClientManager(**kwargs)
-
-
-class ClientManager(object):
- """
- Factory to create LWR clients, used to manage potential shared
- state between multiple client connections.
- """
- def __init__(self, **kwds):
- if 'job_manager' in kwds:
- self.job_manager_interface_class = LocalLwrInterface
- self.job_manager_interface_args = dict(job_manager=kwds['job_manager'], file_cache=kwds['file_cache'])
- else:
- self.job_manager_interface_class = HttpLwrInterface
- transport_type = kwds.get('transport', None)
- transport = get_transport(transport_type)
- self.job_manager_interface_args = dict(transport=transport)
- cache = kwds.get('cache', None)
- if cache is None:
- cache = _environ_default_int('LWR_CACHE_TRANSFERS')
- if cache:
- log.info("Setting LWR client class to caching variant.")
- self.client_cacher = ClientCacher(**kwds)
- self.client_class = InputCachingJobClient
- self.extra_client_kwds = {"client_cacher": self.client_cacher}
- else:
- log.info("Setting LWR client class to standard, non-caching variant.")
- self.client_class = JobClient
- self.extra_client_kwds = {}
-
- def get_client(self, destination_params, job_id, **kwargs):
- destination_params = _parse_destination_params(destination_params)
- destination_params.update(**kwargs)
- job_manager_interface_class = self.job_manager_interface_class
- job_manager_interface_args = dict(destination_params=destination_params, **self.job_manager_interface_args)
- job_manager_interface = job_manager_interface_class(**job_manager_interface_args)
- return self.client_class(destination_params, job_id, job_manager_interface, **self.extra_client_kwds)
-
- def shutdown(self):
- pass
-
-
-try:
- from galaxy.jobs.runners.util.cli import factory as cli_factory
-except ImportError:
- from lwr.managers.util.cli import factory as cli_factory
-
-
-class MessageQueueClientManager(object):
-
- def __init__(self, **kwds):
- self.url = kwds.get('url')
- self.manager_name = kwds.get("manager", None) or "_default_"
- self.exchange = get_exchange(self.url, self.manager_name, kwds)
- self.status_cache = {}
- self.callback_lock = threading.Lock()
- self.callback_thread = None
- self.active = True
-
- def ensure_has_status_update_callback(self, callback):
- with self.callback_lock:
- if self.callback_thread is not None:
- return
-
- def callback_wrapper(body, message):
- try:
- if "job_id" in body:
- job_id = body["job_id"]
- self.status_cache[job_id] = body
- log.debug("Handling asynchronous status update from remote LWR.")
- callback(body)
- except Exception:
- log.exception("Failure processing job status update message.")
- except BaseException as e:
- log.exception("Failure processing job status update message - BaseException type %s" % type(e))
- finally:
- message.ack()
-
- def run():
- self.exchange.consume("status_update", callback_wrapper, check=self)
- log.debug("Leaving LWR client status update thread, no additional LWR updates will be processed.")
-
- thread = threading.Thread(
- name="lwr_client_%s_status_update_callback" % self.manager_name,
- target=run
- )
- thread.daemon = False # Lets not interrupt processing of this.
- thread.start()
- self.callback_thread = thread
-
- def shutdown(self):
- self.active = False
-
- def __nonzero__(self):
- return self.active
-
- def get_client(self, destination_params, job_id, **kwargs):
- if job_id is None:
- raise Exception("Cannot generate LWR client for empty job_id.")
- destination_params = _parse_destination_params(destination_params)
- destination_params.update(**kwargs)
- if 'shell_plugin' in destination_params:
- shell = cli_factory.get_shell(destination_params)
- return MessageCLIJobClient(destination_params, job_id, self, shell)
- else:
- return MessageJobClient(destination_params, job_id, self)
-
-
-class ObjectStoreClientManager(object):
-
- def __init__(self, **kwds):
- if 'object_store' in kwds:
- self.interface_class = LocalLwrInterface
- self.interface_args = dict(object_store=kwds['object_store'])
- else:
- self.interface_class = HttpLwrInterface
- transport_type = kwds.get('transport', None)
- transport = get_transport(transport_type)
- self.interface_args = dict(transport=transport)
- self.extra_client_kwds = {}
-
- def get_client(self, client_params):
- interface_class = self.interface_class
- interface_args = dict(destination_params=client_params, **self.interface_args)
- interface = interface_class(**interface_args)
- return ObjectStoreClient(interface)
-
-
-class ClientCacher(object):
-
- def __init__(self, **kwds):
- self.event_manager = TransferEventManager()
- default_transfer_threads = _environ_default_int('LWR_CACHE_THREADS', DEFAULT_TRANSFER_THREADS)
- num_transfer_threads = int(kwds.get('transfer_threads', default_transfer_threads))
- self.__init_transfer_threads(num_transfer_threads)
-
- def queue_transfer(self, client, path):
- self.transfer_queue.put((client, path))
-
- def acquire_event(self, input_path):
- return self.event_manager.acquire_event(input_path)
-
- def _transfer_worker(self):
- while True:
- transfer_info = self.transfer_queue.get()
- try:
- self.__perform_transfer(transfer_info)
- except BaseException as e:
- log.warn("Transfer failed.")
- log.exception(e)
- pass
- self.transfer_queue.task_done()
-
- def __perform_transfer(self, transfer_info):
- (client, path) = transfer_info
- event_holder = self.event_manager.acquire_event(path, force_clear=True)
- failed = True
- try:
- client.cache_insert(path)
- failed = False
- finally:
- event_holder.failed = failed
- event_holder.release()
-
- def __init_transfer_threads(self, num_transfer_threads):
- self.num_transfer_threads = num_transfer_threads
- self.transfer_queue = Queue()
- for i in range(num_transfer_threads):
- t = threading.Thread(target=self._transfer_worker)
- t.daemon = True
- t.start()
-
-
-def _parse_destination_params(destination_params):
- try:
- unicode_type = unicode
- except NameError:
- unicode_type = str
- if isinstance(destination_params, str) or isinstance(destination_params, unicode_type):
- destination_params = url_to_destination_params(destination_params)
- return destination_params
-
-
-def _environ_default_int(variable, default="0"):
- val = getenv(variable, default)
- int_val = int(default)
- if str(val).isdigit():
- int_val = int(val)
- return int_val
-
-__all__ = [ClientManager, ObjectStoreClientManager, HttpLwrInterface]
diff --git a/lib/galaxy/jobs/runners/lwr_client/object_client.py b/lib/galaxy/jobs/runners/lwr_client/object_client.py
deleted file mode 100644
index 5ec9acdb4f5..00000000000
--- a/lib/galaxy/jobs/runners/lwr_client/object_client.py
+++ /dev/null
@@ -1,53 +0,0 @@
-from .decorators import parseJson
-
-
-class ObjectStoreClient(object):
-
- def __init__(self, lwr_interface):
- self.lwr_interface = lwr_interface
-
- @parseJson()
- def exists(self, **kwds):
- return self._raw_execute("object_store_exists", args=self.__data(**kwds))
-
- @parseJson()
- def file_ready(self, **kwds):
- return self._raw_execute("object_store_file_ready", args=self.__data(**kwds))
-
- @parseJson()
- def create(self, **kwds):
- return self._raw_execute("object_store_create", args=self.__data(**kwds))
-
- @parseJson()
- def empty(self, **kwds):
- return self._raw_execute("object_store_empty", args=self.__data(**kwds))
-
- @parseJson()
- def size(self, **kwds):
- return self._raw_execute("object_store_size", args=self.__data(**kwds))
-
- @parseJson()
- def delete(self, **kwds):
- return self._raw_execute("object_store_delete", args=self.__data(**kwds))
-
- @parseJson()
- def get_data(self, **kwds):
- return self._raw_execute("object_store_get_data", args=self.__data(**kwds))
-
- @parseJson()
- def get_filename(self, **kwds):
- return self._raw_execute("object_store_get_filename", args=self.__data(**kwds))
-
- @parseJson()
- def update_from_file(self, **kwds):
- return self._raw_execute("object_store_update_from_file", args=self.__data(**kwds))
-
- @parseJson()
- def get_store_usage_percent(self):
- return self._raw_execute("object_store_get_store_usage_percent", args={})
-
- def __data(self, **kwds):
- return kwds
-
- def _raw_execute(self, command, args={}):
- return self.lwr_interface.execute(command, args, data=None, input_path=None, output_path=None)
diff --git a/lib/galaxy/jobs/runners/lwr_client/path_mapper.py b/lib/galaxy/jobs/runners/lwr_client/path_mapper.py
deleted file mode 100644
index 758c0901638..00000000000
--- a/lib/galaxy/jobs/runners/lwr_client/path_mapper.py
+++ /dev/null
@@ -1,98 +0,0 @@
-import os.path
-from .action_mapper import FileActionMapper
-from .action_mapper import path_type
-from .util import PathHelper
-
-from galaxy.util import in_directory
-
-
-class PathMapper(object):
- """ Ties together a FileActionMapper and remote job configuration returned
- by the LWR setup method to pre-determine the location of files for staging
- on the remote LWR server.
-
- This is not useful when rewrite_paths (as has traditionally been done with
- the LWR) because when doing that the LWR determines the paths as files are
- uploaded. When rewrite_paths is disabled however, the destination of files
- needs to be determined prior to transfer so an object of this class can be
- used.
- """
-
- def __init__(
- self,
- client,
- remote_job_config,
- local_working_directory,
- action_mapper=None,
- ):
- self.local_working_directory = local_working_directory
- if not action_mapper:
- action_mapper = FileActionMapper(client)
- self.action_mapper = action_mapper
- self.input_directory = remote_job_config["inputs_directory"]
- self.output_directory = remote_job_config["outputs_directory"]
- self.working_directory = remote_job_config["working_directory"]
- self.unstructured_files_directory = remote_job_config["unstructured_files_directory"]
- self.config_directory = remote_job_config["configs_directory"]
- separator = remote_job_config["system_properties"]["separator"]
- self.path_helper = PathHelper(separator)
-
- def remote_output_path_rewrite(self, local_path):
- output_type = path_type.OUTPUT
- if in_directory(local_path, self.local_working_directory):
- output_type = path_type.OUTPUT_WORKDIR
- remote_path = self.__remote_path_rewrite(local_path, output_type)
- return remote_path
-
- def remote_input_path_rewrite(self, local_path):
- remote_path = self.__remote_path_rewrite(local_path, path_type.INPUT)
- return remote_path
-
- def remote_version_path_rewrite(self, local_path):
- remote_path = self.__remote_path_rewrite(local_path, path_type.OUTPUT, name="COMMAND_VERSION")
- return remote_path
-
- def check_for_arbitrary_rewrite(self, local_path):
- path = str(local_path) # Use false_path if needed.
- action = self.action_mapper.action(path, path_type.UNSTRUCTURED)
- if not action.staging_needed:
- return action.path_rewrite(self.path_helper), []
- unique_names = action.unstructured_map()
- name = unique_names[path]
- remote_path = self.path_helper.remote_join(self.unstructured_files_directory, name)
- return remote_path, unique_names
-
- def __remote_path_rewrite(self, dataset_path, dataset_path_type, name=None):
- """ Return remote path of this file (if staging is required) else None.
- """
- path = str(dataset_path) # Use false_path if needed.
- action = self.action_mapper.action(path, dataset_path_type)
- if action.staging_needed:
- if name is None:
- name = os.path.basename(path)
- remote_directory = self.__remote_directory(dataset_path_type)
- remote_path_rewrite = self.path_helper.remote_join(remote_directory, name)
- else:
- # Actions which don't require staging MUST define a path_rewrite
- # method.
- remote_path_rewrite = action.path_rewrite(self.path_helper)
-
- return remote_path_rewrite
-
- def __action(self, dataset_path, dataset_path_type):
- path = str(dataset_path) # Use false_path if needed.
- action = self.action_mapper.action(path, dataset_path_type)
- return action
-
- def __remote_directory(self, dataset_path_type):
- if dataset_path_type in [path_type.OUTPUT]:
- return self.output_directory
- elif dataset_path_type in [path_type.WORKDIR, path_type.OUTPUT_WORKDIR]:
- return self.working_directory
- elif dataset_path_type in [path_type.INPUT]:
- return self.input_directory
- else:
- message = "PathMapper cannot handle path type %s" % dataset_path_type
- raise Exception(message)
-
-__all__ = [PathMapper]
diff --git a/lib/galaxy/jobs/runners/lwr_client/setup_handler.py b/lib/galaxy/jobs/runners/lwr_client/setup_handler.py
deleted file mode 100644
index 19906edf667..00000000000
--- a/lib/galaxy/jobs/runners/lwr_client/setup_handler.py
+++ /dev/null
@@ -1,103 +0,0 @@
-import os
-from .util import filter_destination_params
-
-REMOTE_SYSTEM_PROPERTY_PREFIX = "remote_property_"
-
-
-def build(client, destination_args):
- """ Build a SetupHandler object for client from destination parameters.
- """
- # Have defined a remote job directory, lets do the setup locally.
- if client.job_directory:
- handler = LocalSetupHandler(client, destination_args)
- else:
- handler = RemoteSetupHandler(client)
- return handler
-
-
-class LocalSetupHandler(object):
- """ Parse destination params to infer job setup parameters (input/output
- directories, etc...). Default is to get this configuration data from the
- remote LWR server.
-
- Downside of this approach is that it requires more and more dependent
- configuraiton of Galaxy. Upside is that it is asynchronous and thus makes
- message queue driven configurations possible.
-
- Remote system properties (such as galaxy_home) can be specified in
- destination args by prefixing property with remote_property_ (e.g.
- remote_property_galaxy_home).
- """
-
- def __init__(self, client, destination_args):
- self.client = client
- system_properties = self.__build_system_properties(destination_args)
- system_properties["separator"] = client.job_directory.separator
- self.system_properties = system_properties
- self.jobs_directory = destination_args["jobs_directory"]
-
- def setup(self, job_id, tool_id=None, tool_version=None):
- return build_job_config(
- job_id=job_id,
- job_directory=self.client.job_directory,
- system_properties=self.system_properties,
- tool_id=tool_id,
- tool_version=tool_version,
- )
-
- @property
- def local(self):
- """
- """
- return True
-
- def __build_system_properties(self, destination_params):
- return filter_destination_params(destination_params, REMOTE_SYSTEM_PROPERTY_PREFIX)
-
-
-class RemoteSetupHandler(object):
- """ Default behavior. Fetch setup information from remote LWR server.
- """
- def __init__(self, client):
- self.client = client
-
- def setup(self, **setup_args):
- return self.client.remote_setup(**setup_args)
-
- @property
- def local(self):
- """
- """
- return False
-
-
-def build_job_config(job_id, job_directory, system_properties={}, tool_id=None, tool_version=None):
- """
- """
- inputs_directory = job_directory.inputs_directory()
- working_directory = job_directory.working_directory()
- outputs_directory = job_directory.outputs_directory()
- configs_directory = job_directory.configs_directory()
- tools_directory = job_directory.tool_files_directory()
- unstructured_files_directory = job_directory.unstructured_files_directory()
- sep = system_properties.get("sep", os.sep)
- job_config = {
- "working_directory": working_directory,
- "outputs_directory": outputs_directory,
- "configs_directory": configs_directory,
- "tools_directory": tools_directory,
- "inputs_directory": inputs_directory,
- "unstructured_files_directory": unstructured_files_directory,
- # Poorly named legacy attribute. Drop at some point.
- "path_separator": sep,
- "job_id": job_id,
- "system_properties": system_properties,
- }
- if tool_id:
- job_config["tool_id"] = tool_id
- if tool_version:
- job_config["tool_version"] = tool_version
- return job_config
-
-
-__all__ = [build_job_config, build]
diff --git a/lib/galaxy/jobs/runners/lwr_client/staging/__init__.py b/lib/galaxy/jobs/runners/lwr_client/staging/__init__.py
deleted file mode 100644
index c2a51da1870..00000000000
--- a/lib/galaxy/jobs/runners/lwr_client/staging/__init__.py
+++ /dev/null
@@ -1,174 +0,0 @@
-from os.path import basename
-from os.path import join
-from os.path import dirname
-from os import sep
-
-from ..util import PathHelper
-
-COMMAND_VERSION_FILENAME = "COMMAND_VERSION"
-
-
-class ClientJobDescription(object):
- """ A description of how client views job - command_line, inputs, etc..
-
- **Parameters**
-
- command_line : str
- The local command line to execute, this will be rewritten for
- the remote server.
- config_files : list
- List of Galaxy 'configfile's produced for this job. These will
- be rewritten and sent to remote server.
- input_files : list
- List of input files used by job. These will be transferred and
- references rewritten.
- client_outputs : ClientOutputs
- Description of outputs produced by job (at least output files along
- with optional version string and working directory outputs.
- tool_dir : str
- Directory containing tool to execute (if a wrapper is used, it will
- be transferred to remote server).
- working_directory : str
- Local path created by Galaxy for running this job.
- dependencies_description : list
- galaxy.tools.deps.dependencies.DependencyDescription object describing
- tool dependency context for remote depenency resolution.
- env: list
- List of dict object describing environment variables to populate.
- version_file : str
- Path to version file expected on the client server
- arbitrary_files : dict()
- Additional non-input, non-tool, non-config, non-working directory files
- to transfer before staging job. This is most likely data indices but
- can be anything. For now these are copied into staging working
- directory but this will be reworked to find a better, more robust
- location.
- rewrite_paths : boolean
- Indicates whether paths should be rewritten in job inputs (command_line
- and config files) while staging files).
- """
-
- def __init__(
- self,
- tool,
- command_line,
- config_files,
- input_files,
- client_outputs,
- working_directory,
- dependencies_description=None,
- env=[],
- arbitrary_files=None,
- rewrite_paths=True,
- ):
- self.tool = tool
- self.command_line = command_line
- self.config_files = config_files
- self.input_files = input_files
- self.client_outputs = client_outputs
- self.working_directory = working_directory
- self.dependencies_description = dependencies_description
- self.env = env
- self.rewrite_paths = rewrite_paths
- self.arbitrary_files = arbitrary_files or {}
-
- @property
- def output_files(self):
- return self.client_outputs.output_files
-
- @property
- def version_file(self):
- return self.client_outputs.version_file
-
- @property
- def tool_dependencies(self):
- if not self.remote_dependency_resolution:
- return None
- return dict(
- requirements=(self.tool.requirements or []),
- installed_tool_dependencies=(self.tool.installed_tool_dependencies or [])
- )
-
-
-class ClientOutputs(object):
- """ Abstraction describing the output datasets EXPECTED by the Galaxy job
- runner client.
- """
-
- def __init__(self, working_directory, output_files, work_dir_outputs=None, version_file=None):
- self.working_directory = working_directory
- self.work_dir_outputs = work_dir_outputs
- self.output_files = output_files
- self.version_file = version_file
-
- def to_dict(self):
- return dict(
- working_directory=self.working_directory,
- work_dir_outputs=self.work_dir_outputs,
- output_files=self.output_files,
- version_file=self.version_file
- )
-
- @staticmethod
- def from_dict(config_dict):
- return ClientOutputs(
- working_directory=config_dict.get('working_directory'),
- work_dir_outputs=config_dict.get('work_dir_outputs'),
- output_files=config_dict.get('output_files'),
- version_file=config_dict.get('version_file'),
- )
-
-
-class LwrOutputs(object):
- """ Abstraction describing the output files PRODUCED by the remote LWR
- server. """
-
- def __init__(self, working_directory_contents, output_directory_contents, remote_separator=sep):
- self.working_directory_contents = working_directory_contents
- self.output_directory_contents = output_directory_contents
- self.path_helper = PathHelper(remote_separator)
-
- @staticmethod
- def from_status_response(complete_response):
- # Default to None instead of [] to distinguish between empty contents and it not set
- # by the LWR - older LWR instances will not set these in complete response.
- working_directory_contents = complete_response.get("working_directory_contents", None)
- output_directory_contents = complete_response.get("outputs_directory_contents", None)
- # Older (pre-2014) LWR servers will not include separator in response,
- # so this should only be used when reasoning about outputs in
- # subdirectories (which was not previously supported prior to that).
- remote_separator = complete_response.get("system_properties", {}).get("separator", sep)
- return LwrOutputs(
- working_directory_contents,
- output_directory_contents,
- remote_separator
- )
-
- def has_output_file(self, output_file):
- if self.output_directory_contents is None:
- # Legacy LWR doesn't report this, return None indicating unsure if
- # output was generated.
- return None
- else:
- return basename(output_file) in self.output_directory_contents
-
- def has_output_directory_listing(self):
- return self.output_directory_contents is not None
-
- def output_extras(self, output_file):
- """
- Returns dict mapping local path to remote name.
- """
- if not self.has_output_directory_listing():
- # Fetching $output.extra_files_path is not supported with legacy
- # LWR (pre-2014) severs.
- return {}
-
- output_directory = dirname(output_file)
-
- def local_path(name):
- return join(output_directory, self.path_helper.local_name(name))
-
- files_directory = "%s_files%s" % (basename(output_file)[0:-len(".dat")], self.path_helper.separator)
- names = filter(lambda o: o.startswith(files_directory), self.output_directory_contents)
- return dict(map(lambda name: (local_path(name), name), names))
diff --git a/lib/galaxy/jobs/runners/lwr_client/staging/down.py b/lib/galaxy/jobs/runners/lwr_client/staging/down.py
deleted file mode 100644
index 08e9db410a2..00000000000
--- a/lib/galaxy/jobs/runners/lwr_client/staging/down.py
+++ /dev/null
@@ -1,162 +0,0 @@
-from os.path import join
-from os.path import relpath
-from re import compile
-from contextlib import contextmanager
-
-from ..staging import COMMAND_VERSION_FILENAME
-from ..action_mapper import FileActionMapper
-
-
-from logging import getLogger
-log = getLogger(__name__)
-
-# All output files marked with from_work_dir attributes will copied or downloaded
-# this pattern picks up attiditional files to copy back - such as those
-# associated with multiple outputs and metadata configuration. Set to .* to just
-# copy everything
-COPY_FROM_WORKING_DIRECTORY_PATTERN = compile(r"primary_.*|galaxy.json|metadata_.*|dataset_\d+\.dat|__instrument_.*|dataset_\d+_files.+")
-
-
-def finish_job(client, cleanup_job, job_completed_normally, client_outputs, lwr_outputs):
- """ Responsible for downloading results from remote server and cleaning up
- LWR staging directory (if needed.)
- """
- collection_failure_exceptions = []
- if job_completed_normally:
- output_collector = ClientOutputCollector(client)
- action_mapper = FileActionMapper(client)
- results_stager = ResultsCollector(output_collector, action_mapper, client_outputs, lwr_outputs)
- collection_failure_exceptions = results_stager.collect()
- __clean(collection_failure_exceptions, cleanup_job, client)
- return collection_failure_exceptions
-
-
-class ClientOutputCollector(object):
-
- def __init__(self, client):
- self.client = client
-
- def collect_output(self, results_collector, output_type, action, name):
- # This output should have been handled by the LWR.
- if not action.staging_action_local:
- return False
-
- working_directory = results_collector.client_outputs.working_directory
- self.client.fetch_output(
- path=action.path,
- name=name,
- working_directory=working_directory,
- output_type=output_type,
- action_type=action.action_type
- )
- return True
-
-
-class ResultsCollector(object):
-
- def __init__(self, output_collector, action_mapper, client_outputs, lwr_outputs):
- self.output_collector = output_collector
- self.action_mapper = action_mapper
- self.client_outputs = client_outputs
- self.lwr_outputs = lwr_outputs
- self.downloaded_working_directory_files = []
- self.exception_tracker = DownloadExceptionTracker()
- self.output_files = client_outputs.output_files
- self.working_directory_contents = lwr_outputs.working_directory_contents or []
-
- def collect(self):
- self.__collect_working_directory_outputs()
- self.__collect_outputs()
- self.__collect_version_file()
- self.__collect_other_working_directory_files()
- return self.exception_tracker.collection_failure_exceptions
-
- def __collect_working_directory_outputs(self):
- working_directory = self.client_outputs.working_directory
- # Fetch explicit working directory outputs.
- for source_file, output_file in self.client_outputs.work_dir_outputs:
- name = relpath(source_file, working_directory)
- lwr_name = self.lwr_outputs.path_helper.remote_name(name)
- if self._attempt_collect_output('output_workdir', path=output_file, name=lwr_name):
- self.downloaded_working_directory_files.append(lwr_name)
- # Remove from full output_files list so don't try to download directly.
- try:
- self.output_files.remove(output_file)
- except ValueError:
- raise Exception("Failed to remove %s from %s" % (output_file, self.output_files))
-
- def __collect_outputs(self):
- # Legacy LWR not returning list of files, iterate over the list of
- # expected outputs for tool.
- for output_file in self.output_files:
- # Fetch output directly...
- output_generated = self.lwr_outputs.has_output_file(output_file)
- if output_generated is None:
- self._attempt_collect_output('legacy', output_file)
- elif output_generated:
- self._attempt_collect_output('output', output_file)
-
- for galaxy_path, lwr_name in self.lwr_outputs.output_extras(output_file).iteritems():
- self._attempt_collect_output('output', path=galaxy_path, name=lwr_name)
- # else not output generated, do not attempt download.
-
- def __collect_version_file(self):
- version_file = self.client_outputs.version_file
- # output_directory_contents may be none for legacy LWR servers.
- lwr_output_directory_contents = (self.lwr_outputs.output_directory_contents or [])
- if version_file and COMMAND_VERSION_FILENAME in lwr_output_directory_contents:
- self._attempt_collect_output('output', version_file, name=COMMAND_VERSION_FILENAME)
-
- def __collect_other_working_directory_files(self):
- working_directory = self.client_outputs.working_directory
- # Fetch remaining working directory outputs of interest.
- for name in self.working_directory_contents:
- if name in self.downloaded_working_directory_files:
- continue
- if COPY_FROM_WORKING_DIRECTORY_PATTERN.match(name):
- output_file = join(working_directory, self.lwr_outputs.path_helper.local_name(name))
- if self._attempt_collect_output(output_type='output_workdir', path=output_file, name=name):
- self.downloaded_working_directory_files.append(name)
-
- def _attempt_collect_output(self, output_type, path, name=None):
- # path is final path on galaxy server (client)
- # name is the 'name' of the file on the LWR server (possible a relative)
- # path.
- collected = False
- with self.exception_tracker():
- # output_action_type cannot be 'legacy' but output_type may be
- # eventually drop support for legacy mode (where type wasn't known)
- # ahead of time.
- output_action_type = 'output_workdir' if output_type == 'output_workdir' else 'output'
- action = self.action_mapper.action(path, output_action_type)
- if self._collect_output(output_type, action, name):
- collected = True
-
- return collected
-
- def _collect_output(self, output_type, action, name):
- return self.output_collector.collect_output(self, output_type, action, name)
-
-
-class DownloadExceptionTracker(object):
-
- def __init__(self):
- self.collection_failure_exceptions = []
-
- @contextmanager
- def __call__(self):
- try:
- yield
- except Exception as e:
- self.collection_failure_exceptions.append(e)
-
-
-def __clean(collection_failure_exceptions, cleanup_job, client):
- failed = (len(collection_failure_exceptions) > 0)
- if (not failed and cleanup_job != "never") or cleanup_job == "always":
- try:
- client.clean()
- except Exception:
- log.warn("Failed to cleanup remote LWR job")
-
-__all__ = [finish_job]
diff --git a/lib/galaxy/jobs/runners/lwr_client/staging/up.py b/lib/galaxy/jobs/runners/lwr_client/staging/up.py
deleted file mode 100644
index c8996fd8f4e..00000000000
--- a/lib/galaxy/jobs/runners/lwr_client/staging/up.py
+++ /dev/null
@@ -1,420 +0,0 @@
-from os.path import abspath, basename, join, exists
-from os.path import dirname
-from os.path import relpath
-from os import listdir, sep
-from re import findall
-from io import open
-
-from ..staging import COMMAND_VERSION_FILENAME
-from ..action_mapper import FileActionMapper
-from ..action_mapper import path_type
-from ..action_mapper import MessageAction
-from ..util import PathHelper
-from ..util import directory_files
-
-from logging import getLogger
-log = getLogger(__name__)
-
-
-def submit_job(client, client_job_description, job_config=None):
- """
- """
- file_stager = FileStager(client, client_job_description, job_config)
- rebuilt_command_line = file_stager.get_command_line()
- job_id = file_stager.job_id
- launch_kwds = dict(
- command_line=rebuilt_command_line,
- dependencies_description=client_job_description.dependencies_description,
- env=client_job_description.env,
- )
- if file_stager.job_config:
- launch_kwds["job_config"] = file_stager.job_config
- remote_staging = {}
- remote_staging_actions = file_stager.transfer_tracker.remote_staging_actions
- if remote_staging_actions:
- remote_staging["setup"] = remote_staging_actions
- # Somehow make the following optional.
- remote_staging["action_mapper"] = file_stager.action_mapper.to_dict()
- remote_staging["client_outputs"] = client_job_description.client_outputs.to_dict()
-
- if remote_staging:
- launch_kwds["remote_staging"] = remote_staging
-
- client.launch(**launch_kwds)
- return job_id
-
-
-class FileStager(object):
- """
- Objects of the FileStager class interact with an LWR client object to
- stage the files required to run jobs on a remote LWR server.
-
- **Parameters**
-
- client : JobClient
- LWR client object.
- client_job_description : client_job_description
- Description of client view of job to stage and execute remotely.
- """
-
- def __init__(self, client, client_job_description, job_config):
- """
- """
- self.client = client
- self.command_line = client_job_description.command_line
- self.config_files = client_job_description.config_files
- self.input_files = client_job_description.input_files
- self.output_files = client_job_description.output_files
- self.tool_id = client_job_description.tool.id
- self.tool_version = client_job_description.tool.version
- self.tool_dir = abspath(client_job_description.tool.tool_dir)
- self.working_directory = client_job_description.working_directory
- self.version_file = client_job_description.version_file
- self.arbitrary_files = client_job_description.arbitrary_files
- self.rewrite_paths = client_job_description.rewrite_paths
-
- # Setup job inputs, these will need to be rewritten before
- # shipping off to remote LWR server.
- self.job_inputs = JobInputs(self.command_line, self.config_files)
-
- self.action_mapper = FileActionMapper(client)
-
- self.__handle_setup(job_config)
-
- self.transfer_tracker = TransferTracker(client, self.path_helper, self.action_mapper, self.job_inputs, rewrite_paths=self.rewrite_paths)
-
- self.__initialize_referenced_tool_files()
- if self.rewrite_paths:
- self.__initialize_referenced_arbitrary_files()
-
- self.__upload_tool_files()
- self.__upload_input_files()
- self.__upload_working_directory_files()
- self.__upload_arbitrary_files()
-
- if self.rewrite_paths:
- self.__initialize_output_file_renames()
- self.__initialize_task_output_file_renames()
- self.__initialize_config_file_renames()
- self.__initialize_version_file_rename()
-
- self.__handle_rewrites()
-
- self.__upload_rewritten_config_files()
-
- def __handle_setup(self, job_config):
- if not job_config:
- job_config = self.client.setup(self.tool_id, self.tool_version)
-
- self.new_working_directory = job_config['working_directory']
- self.new_outputs_directory = job_config['outputs_directory']
- # Default configs_directory to match remote working_directory to mimic
- # behavior of older LWR servers.
- self.new_configs_directory = job_config.get('configs_directory', self.new_working_directory)
- self.remote_separator = self.__parse_remote_separator(job_config)
- self.path_helper = PathHelper(self.remote_separator)
- # If remote LWR server assigned job id, use that otherwise
- # just use local job_id assigned.
- galaxy_job_id = self.client.job_id
- self.job_id = job_config.get('job_id', galaxy_job_id)
- if self.job_id != galaxy_job_id:
- # Remote LWR server assigned an id different than the
- # Galaxy job id, update client to reflect this.
- self.client.job_id = self.job_id
- self.job_config = job_config
-
- def __parse_remote_separator(self, job_config):
- separator = job_config.get("system_properties", {}).get("separator", None)
- if not separator: # Legacy LWR
- separator = job_config["path_separator"] # Poorly named
- return separator
-
- def __initialize_referenced_tool_files(self):
- self.referenced_tool_files = self.job_inputs.find_referenced_subfiles(self.tool_dir)
-
- def __initialize_referenced_arbitrary_files(self):
- referenced_arbitrary_path_mappers = dict()
- for mapper in self.action_mapper.unstructured_mappers():
- mapper_pattern = mapper.to_pattern()
- # TODO: Make more sophisticated, allow parent directories,
- # grabbing sibbling files based on patterns, etc...
- paths = self.job_inputs.find_pattern_references(mapper_pattern)
- for path in paths:
- if path not in referenced_arbitrary_path_mappers:
- referenced_arbitrary_path_mappers[path] = mapper
- for path, mapper in referenced_arbitrary_path_mappers.iteritems():
- action = self.action_mapper.action(path, path_type.UNSTRUCTURED, mapper)
- unstructured_map = action.unstructured_map(self.path_helper)
- self.arbitrary_files.update(unstructured_map)
-
- def __upload_tool_files(self):
- for referenced_tool_file in self.referenced_tool_files:
- self.transfer_tracker.handle_transfer(referenced_tool_file, path_type.TOOL)
-
- def __upload_arbitrary_files(self):
- for path, name in self.arbitrary_files.iteritems():
- self.transfer_tracker.handle_transfer(path, path_type.UNSTRUCTURED, name=name)
-
- def __upload_input_files(self):
- for input_file in self.input_files:
- self.__upload_input_file(input_file)
- self.__upload_input_extra_files(input_file)
-
- def __upload_input_file(self, input_file):
- if self.__stage_input(input_file):
- if exists(input_file):
- self.transfer_tracker.handle_transfer(input_file, path_type.INPUT)
- else:
- message = "LWR: __upload_input_file called on empty or missing dataset." + \
- " So such file: [%s]" % input_file
- log.debug(message)
-
- def __upload_input_extra_files(self, input_file):
- files_path = "%s_files" % input_file[0:-len(".dat")]
- if exists(files_path) and self.__stage_input(files_path):
- for extra_file_name in directory_files(files_path):
- extra_file_path = join(files_path, extra_file_name)
- remote_name = self.path_helper.remote_name(relpath(extra_file_path, dirname(files_path)))
- self.transfer_tracker.handle_transfer(extra_file_path, path_type.INPUT, name=remote_name)
-
- def __upload_working_directory_files(self):
- # Task manager stages files into working directory, these need to be
- # uploaded if present.
- working_directory_files = listdir(self.working_directory) if exists(self.working_directory) else []
- for working_directory_file in working_directory_files:
- path = join(self.working_directory, working_directory_file)
- self.transfer_tracker.handle_transfer(path, path_type.WORKDIR)
-
- def __initialize_version_file_rename(self):
- version_file = self.version_file
- if version_file:
- remote_path = self.path_helper.remote_join(self.new_outputs_directory, COMMAND_VERSION_FILENAME)
- self.transfer_tracker.register_rewrite(version_file, remote_path, path_type.OUTPUT)
-
- def __initialize_output_file_renames(self):
- for output_file in self.output_files:
- remote_path = self.path_helper.remote_join(self.new_outputs_directory, basename(output_file))
- self.transfer_tracker.register_rewrite(output_file, remote_path, path_type.OUTPUT)
-
- def __initialize_task_output_file_renames(self):
- for output_file in self.output_files:
- name = basename(output_file)
- task_file = join(self.working_directory, name)
- remote_path = self.path_helper.remote_join(self.new_working_directory, name)
- self.transfer_tracker.register_rewrite(task_file, remote_path, path_type.OUTPUT_WORKDIR)
-
- def __initialize_config_file_renames(self):
- for config_file in self.config_files:
- remote_path = self.path_helper.remote_join(self.new_configs_directory, basename(config_file))
- self.transfer_tracker.register_rewrite(config_file, remote_path, path_type.CONFIG)
-
- def __handle_rewrites(self):
- """
- For each file that has been transferred and renamed, updated
- command_line and configfiles to reflect that rewrite.
- """
- self.transfer_tracker.rewrite_input_paths()
-
- def __upload_rewritten_config_files(self):
- for config_file, new_config_contents in self.job_inputs.config_files.items():
- self.transfer_tracker.handle_transfer(config_file, type=path_type.CONFIG, contents=new_config_contents)
-
- def get_command_line(self):
- """
- Returns the rewritten version of the command line to execute suitable
- for remote host.
- """
- return self.job_inputs.command_line
-
- def __stage_input(self, file_path):
- # If we have disabled path rewriting, just assume everything needs to be transferred,
- # else check to ensure the file is referenced before transferring it.
- return (not self.rewrite_paths) or self.job_inputs.path_referenced(file_path)
-
-
-class JobInputs(object):
- """
- Abstractions over dynamic inputs created for a given job (namely the command to
- execute and created configfiles).
-
- **Parameters**
-
- command_line : str
- Local command to execute for this job. (To be rewritten.)
- config_files : str
- Config files created for this job. (To be rewritten.)
-
-
- >>> import tempfile
- >>> tf = tempfile.NamedTemporaryFile()
- >>> def setup_inputs(tf):
- ... open(tf.name, "w").write(u"world /path/to/input the rest")
- ... inputs = JobInputs(u"hello /path/to/input", [tf.name])
- ... return inputs
- >>> inputs = setup_inputs(tf)
- >>> inputs.rewrite_paths(u"/path/to/input", u'C:\\input')
- >>> inputs.command_line == u'hello C:\\\\input'
- True
- >>> inputs.config_files[tf.name] == u'world C:\\\\input the rest'
- True
- >>> tf.close()
- >>> tf = tempfile.NamedTemporaryFile()
- >>> inputs = setup_inputs(tf)
- >>> inputs.find_referenced_subfiles('/path/to') == [u'/path/to/input']
- True
- >>> inputs.path_referenced('/path/to')
- True
- >>> inputs.path_referenced(u'/path/to')
- True
- >>> inputs.path_referenced('/path/to/input')
- True
- >>> inputs.path_referenced('/path/to/notinput')
- False
- >>> tf.close()
- """
-
- def __init__(self, command_line, config_files):
- self.command_line = command_line
- self.config_files = {}
- for config_file in config_files or []:
- config_contents = _read(config_file)
- self.config_files[config_file] = config_contents
-
- def find_pattern_references(self, pattern):
- referenced_files = set()
- for input_contents in self.__items():
- referenced_files.update(findall(pattern, input_contents))
- return list(referenced_files)
-
- def find_referenced_subfiles(self, directory):
- """
- Return list of files below specified `directory` in job inputs. Could
- use more sophisticated logic (match quotes to handle spaces, handle
- subdirectories, etc...).
-
- **Parameters**
-
- directory : str
- Full path to directory to search.
-
- """
- pattern = r"(%s%s\S+)" % (directory, sep)
- return self.find_pattern_references(pattern)
-
- def path_referenced(self, path):
- pattern = r"%s" % path
- found = False
- for input_contents in self.__items():
- if findall(pattern, input_contents):
- found = True
- break
- return found
-
- def rewrite_paths(self, local_path, remote_path):
- """
- Rewrite references to `local_path` with `remote_path` in job inputs.
- """
- self.__rewrite_command_line(local_path, remote_path)
- self.__rewrite_config_files(local_path, remote_path)
-
- def __rewrite_command_line(self, local_path, remote_path):
- self.command_line = self.command_line.replace(local_path, remote_path)
-
- def __rewrite_config_files(self, local_path, remote_path):
- for config_file, contents in self.config_files.items():
- self.config_files[config_file] = contents.replace(local_path, remote_path)
-
- def __items(self):
- items = [self.command_line]
- items.extend(self.config_files.values())
- return items
-
-
-class TransferTracker(object):
-
- def __init__(self, client, path_helper, action_mapper, job_inputs, rewrite_paths):
- self.client = client
- self.path_helper = path_helper
- self.action_mapper = action_mapper
-
- self.job_inputs = job_inputs
- self.rewrite_paths = rewrite_paths
- self.file_renames = {}
- self.remote_staging_actions = []
-
- def handle_transfer(self, path, type, name=None, contents=None):
- action = self.__action_for_transfer(path, type, contents)
-
- if action.staging_needed:
- local_action = action.staging_action_local
- if local_action:
- response = self.client.put_file(path, type, name=name, contents=contents)
- get_path = lambda: response['path']
- else:
- job_directory = self.client.job_directory
- assert job_directory, "job directory required for action %s" % action
- if not name:
- name = basename(path)
- self.__add_remote_staging_input(action, name, type)
- get_path = lambda: job_directory.calculate_path(name, type)
- register = self.rewrite_paths or type == 'tool' # Even if inputs not rewritten, tool must be.
- if register:
- self.register_rewrite(path, get_path(), type, force=True)
- elif self.rewrite_paths:
- path_rewrite = action.path_rewrite(self.path_helper)
- if path_rewrite:
- self.register_rewrite(path, path_rewrite, type, force=True)
-
- # else: # No action for this file
-
- def __add_remote_staging_input(self, action, name, type):
- input_dict = dict(
- name=name,
- type=type,
- action=action.to_dict(),
- )
- self.remote_staging_actions.append(input_dict)
-
- def __action_for_transfer(self, path, type, contents):
- if contents:
- # If contents loaded in memory, no need to write out file and copy,
- # just transfer.
- action = MessageAction(contents=contents, client=self.client)
- else:
- if not exists(path):
- message = "handle_tranfer called on non-existent file - [%s]" % path
- log.warn(message)
- raise Exception(message)
- action = self.__action(path, type)
- return action
-
- def register_rewrite(self, local_path, remote_path, type, force=False):
- action = self.__action(local_path, type)
- if action.staging_needed or force:
- self.file_renames[local_path] = remote_path
-
- def rewrite_input_paths(self):
- """
- For each file that has been transferred and renamed, updated
- command_line and configfiles to reflect that rewrite.
- """
- for local_path, remote_path in self.file_renames.items():
- self.job_inputs.rewrite_paths(local_path, remote_path)
-
- def __action(self, path, type):
- return self.action_mapper.action(path, type)
-
-
-def _read(path):
- """
- Utility method to quickly read small files (config files and tool
- wrappers) into memory as bytes.
- """
- input = open(path, "r", encoding="utf-8")
- try:
- return input.read()
- finally:
- input.close()
-
-
-__all__ = [submit_job]
diff --git a/lib/galaxy/jobs/runners/lwr_client/transport/__init__.py b/lib/galaxy/jobs/runners/lwr_client/transport/__init__.py
deleted file mode 100644
index f7119bbdc93..00000000000
--- a/lib/galaxy/jobs/runners/lwr_client/transport/__init__.py
+++ /dev/null
@@ -1,31 +0,0 @@
-from .standard import Urllib2Transport
-from .curl import PycurlTransport
-import os
-
-
-def get_transport(transport_type=None, os_module=os):
- transport_type = __get_transport_type(transport_type, os_module)
- if transport_type == 'urllib':
- transport = Urllib2Transport()
- else:
- transport = PycurlTransport()
- return transport
-
-
-def __get_transport_type(transport_type, os_module):
- if not transport_type:
- use_curl = os_module.getenv('LWR_CURL_TRANSPORT', "0")
- # If LWR_CURL_TRANSPORT is unset or set to 0, use default,
- # else use curl.
- if use_curl.isdigit() and not int(use_curl):
- transport_type = 'urllib'
- else:
- transport_type = 'curl'
- return transport_type
-
-# TODO: Provide urllib implementation if these unavailable,
-# also explore a requests+poster option.
-from .curl import get_file
-from .curl import post_file
-
-__all__ = [get_transport, get_file, post_file]
diff --git a/lib/galaxy/jobs/runners/lwr_client/transport/curl.py b/lib/galaxy/jobs/runners/lwr_client/transport/curl.py
deleted file mode 100644
index 307af0b1fce..00000000000
--- a/lib/galaxy/jobs/runners/lwr_client/transport/curl.py
+++ /dev/null
@@ -1,69 +0,0 @@
-try:
- from cStringIO import StringIO
-except ImportError:
- from io import StringIO
-try:
- from pycurl import Curl
-except ImportError:
- pass
-from os.path import getsize
-
-
-PYCURL_UNAVAILABLE_MESSAGE = \
- "You are attempting to use the Pycurl version of the LWR client but pycurl is unavailable."
-
-
-class PycurlTransport(object):
-
- def execute(self, url, data=None, input_path=None, output_path=None):
- buf = _open_output(output_path)
- try:
- c = _new_curl_object()
- c.setopt(c.URL, url.encode('ascii'))
- c.setopt(c.WRITEFUNCTION, buf.write)
- if input_path:
- c.setopt(c.UPLOAD, 1)
- c.setopt(c.READFUNCTION, open(input_path, 'rb').read)
- filesize = getsize(input_path)
- c.setopt(c.INFILESIZE, filesize)
- if data:
- c.setopt(c.POST, 1)
- if type(data).__name__ == 'unicode':
- data = data.encode('UTF-8')
- c.setopt(c.POSTFIELDS, data)
- c.perform()
- if not output_path:
- return buf.getvalue()
- finally:
- buf.close()
-
-
-def post_file(url, path):
- c = _new_curl_object()
- c.setopt(c.URL, url.encode('ascii'))
- c.setopt(c.HTTPPOST, [("file", (c.FORM_FILE, path.encode('ascii')))])
- c.perform()
-
-
-def get_file(url, path):
- buf = _open_output(path)
- try:
- c = _new_curl_object()
- c.setopt(c.URL, url.encode('ascii'))
- c.setopt(c.WRITEFUNCTION, buf.write)
- c.perform()
- finally:
- buf.close()
-
-
-def _open_output(output_path):
- return open(output_path, 'wb') if output_path else StringIO()
-
-
-def _new_curl_object():
- try:
- return Curl()
- except NameError:
- raise ImportError(PYCURL_UNAVAILABLE_MESSAGE)
-
-___all__ = [PycurlTransport, post_file, get_file]
diff --git a/lib/galaxy/jobs/runners/lwr_client/transport/standard.py b/lib/galaxy/jobs/runners/lwr_client/transport/standard.py
deleted file mode 100644
index 1bd504ace40..00000000000
--- a/lib/galaxy/jobs/runners/lwr_client/transport/standard.py
+++ /dev/null
@@ -1,45 +0,0 @@
-"""
-LWR HTTP Client layer based on Python Standard Library (urllib2)
-"""
-from __future__ import with_statement
-from os.path import getsize
-import mmap
-try:
- from urllib2 import urlopen
-except ImportError:
- from urllib.request import urlopen
-try:
- from urllib2 import Request
-except ImportError:
- from urllib.request import Request
-
-
-class Urllib2Transport(object):
-
- def _url_open(self, request, data):
- return urlopen(request, data)
-
- def execute(self, url, data=None, input_path=None, output_path=None):
- request = Request(url=url, data=data)
- input = None
- try:
- if input_path:
- if getsize(input_path):
- input = open(input_path, 'rb')
- data = mmap.mmap(input.fileno(), 0, access=mmap.ACCESS_READ)
- else:
- data = b""
- response = self._url_open(request, data)
- finally:
- if input:
- input.close()
- if output_path:
- with open(output_path, 'wb') as output:
- while True:
- buffer = response.read(1024)
- if not buffer:
- break
- output.write(buffer)
- return response
- else:
- return response.read()
diff --git a/lib/galaxy/jobs/runners/lwr_client/util.py b/lib/galaxy/jobs/runners/lwr_client/util.py
deleted file mode 100644
index 62f5aba3ff9..00000000000
--- a/lib/galaxy/jobs/runners/lwr_client/util.py
+++ /dev/null
@@ -1,177 +0,0 @@
-from threading import Lock, Event
-from weakref import WeakValueDictionary
-from os import walk
-from os import curdir
-from os.path import relpath
-from os.path import join
-import os.path
-import hashlib
-import shutil
-import json
-import base64
-
-
-def unique_path_prefix(path):
- m = hashlib.md5()
- m.update(path)
- return m.hexdigest()
-
-
-def copy(source, destination):
- """ Copy file from source to destination if needed (skip if source
- is destination).
- """
- source = os.path.abspath(source)
- destination = os.path.abspath(destination)
- if source != destination:
- shutil.copyfile(source, destination)
-
-
-def ensure_directory(file_path):
- directory = os.path.dirname(file_path)
- if not os.path.exists(directory):
- os.makedirs(directory)
-
-
-def directory_files(directory):
- """
-
- >>> from tempfile import mkdtemp
- >>> from shutil import rmtree
- >>> from os.path import join
- >>> from os import makedirs
- >>> tempdir = mkdtemp()
- >>> with open(join(tempdir, "moo"), "w") as f: pass
- >>> directory_files(tempdir)
- ['moo']
- >>> subdir = join(tempdir, "cow", "sub1")
- >>> makedirs(subdir)
- >>> with open(join(subdir, "subfile1"), "w") as f: pass
- >>> with open(join(subdir, "subfile2"), "w") as f: pass
- >>> sorted(directory_files(tempdir))
- ['cow/sub1/subfile1', 'cow/sub1/subfile2', 'moo']
- >>> rmtree(tempdir)
- """
- contents = []
- for path, _, files in walk(directory):
- relative_path = relpath(path, directory)
- for name in files:
- # Return file1.txt, dataset_1_files/image.png, etc... don't
- # include . in path.
- if relative_path != curdir:
- contents.append(join(relative_path, name))
- else:
- contents.append(name)
- return contents
-
-
-def filter_destination_params(destination_params, prefix):
- destination_params = destination_params or {}
- return dict([(key[len(prefix):], destination_params[key])
- for key in destination_params
- if key.startswith(prefix)])
-
-
-def to_base64_json(data):
- """
-
- >>> x = from_base64_json(to_base64_json(dict(a=5)))
- >>> x["a"]
- 5
- """
- return base64.b64encode(json.dumps(data))
-
-
-def from_base64_json(data):
- return json.loads(base64.b64decode(data))
-
-
-class PathHelper(object):
- '''
-
- >>> import posixpath
- >>> # Forcing local path to posixpath because LWR designed to be used with
- >>> # posix client.
- >>> posix_path_helper = PathHelper("/", local_path_module=posixpath)
- >>> windows_slash = "\\\\"
- >>> len(windows_slash)
- 1
- >>> nt_path_helper = PathHelper(windows_slash, local_path_module=posixpath)
- >>> posix_path_helper.remote_name("moo/cow")
- 'moo/cow'
- >>> nt_path_helper.remote_name("moo/cow")
- 'moo\\\\cow'
- >>> posix_path_helper.local_name("moo/cow")
- 'moo/cow'
- >>> nt_path_helper.local_name("moo\\\\cow")
- 'moo/cow'
- >>> posix_path_helper.from_posix_with_new_base("/galaxy/data/bowtie/hg19.fa", "/galaxy/data/", "/work/galaxy/data")
- '/work/galaxy/data/bowtie/hg19.fa'
- >>> posix_path_helper.from_posix_with_new_base("/galaxy/data/bowtie/hg19.fa", "/galaxy/data", "/work/galaxy/data")
- '/work/galaxy/data/bowtie/hg19.fa'
- >>> posix_path_helper.from_posix_with_new_base("/galaxy/data/bowtie/hg19.fa", "/galaxy/data", "/work/galaxy/data/")
- '/work/galaxy/data/bowtie/hg19.fa'
- '''
-
- def __init__(self, separator, local_path_module=os.path):
- self.separator = separator
- self.local_join = local_path_module.join
- self.local_sep = local_path_module.sep
-
- def remote_name(self, local_name):
- return self.remote_join(*local_name.split(self.local_sep))
-
- def local_name(self, remote_name):
- return self.local_join(*remote_name.split(self.separator))
-
- def remote_join(self, *args):
- return self.separator.join(args)
-
- def from_posix_with_new_base(self, posix_path, old_base, new_base):
- # TODO: Test with new_base as a windows path against nt_path_helper.
- if old_base.endswith("/"):
- old_base = old_base[:-1]
- if not posix_path.startswith(old_base):
- message_template = "Cannot compute new path for file %s, does not start with %s."
- message = message_template % (posix_path, old_base)
- raise Exception(message)
- stripped_path = posix_path[len(old_base):]
- while stripped_path.startswith("/"):
- stripped_path = stripped_path[1:]
- path_parts = stripped_path.split(self.separator)
- if new_base.endswith(self.separator):
- new_base = new_base[:-len(self.separator)]
- return self.remote_join(new_base, *path_parts)
-
-
-class TransferEventManager(object):
-
- def __init__(self):
- self.events = WeakValueDictionary(dict())
- self.events_lock = Lock()
-
- def acquire_event(self, path, force_clear=False):
- with self.events_lock:
- if path in self.events:
- event_holder = self.events[path]
- else:
- event_holder = EventHolder(Event(), path, self)
- self.events[path] = event_holder
- if force_clear:
- event_holder.event.clear()
- return event_holder
-
-
-class EventHolder(object):
-
- def __init__(self, event, path, condition_manager):
- self.event = event
- self.path = path
- self.condition_manager = condition_manager
- self.failed = False
-
- def release(self):
- self.event.set()
-
- def fail(self):
- self.failed = True
diff --git a/lib/galaxy/jobs/runners/pbs.py b/lib/galaxy/jobs/runners/pbs.py
index e7adfebb6f2..e4bd3432325 100644
--- a/lib/galaxy/jobs/runners/pbs.py
+++ b/lib/galaxy/jobs/runners/pbs.py
@@ -287,10 +287,7 @@ class PBSJobRunner( AsynchronousJobRunner ):
env_setup_commands = [ stage_commands ]
script = self.get_job_file(job_wrapper, exit_code_path=ecfile, env_setup_commands=env_setup_commands)
job_file = "%s/%s.sh" % (self.app.config.cluster_files_directory, job_wrapper.job_id)
- fh = file(job_file, "w")
- fh.write(script)
- fh.close()
-
+ self.write_executable_script( job_file, script )
# job was deleted while we were preparing it
if job_wrapper.get_state() == model.Job.states.DELETED:
log.debug( "Job %s deleted by user before it entered the PBS queue" % job_wrapper.job_id )
diff --git a/lib/galaxy/jobs/runners/util/job_script/DEFAULT_JOB_FILE_TEMPLATE.sh b/lib/galaxy/jobs/runners/util/job_script/DEFAULT_JOB_FILE_TEMPLATE.sh
index 1ad0d26c74c..5820de9e4a1 100644
--- a/lib/galaxy/jobs/runners/util/job_script/DEFAULT_JOB_FILE_TEMPLATE.sh
+++ b/lib/galaxy/jobs/runners/util/job_script/DEFAULT_JOB_FILE_TEMPLATE.sh
@@ -1,4 +1,12 @@
#!/bin/sh
+
+# The following block can be used by the job creation system
+# to ensure this script is runnable before running it directly
+# or submitting it to a cluster manager.
+if [ -n "$ABC_TEST_JOB_SCRIPT_INTEGRITY_XYZ" ]; then
+ exit 42
+fi
+
$headers
$slots_statement
export GALAXY_SLOTS
diff --git a/lib/galaxy/managers/datasets.py b/lib/galaxy/managers/datasets.py
index 6480a990e1c..44e5dbe4b21 100644
--- a/lib/galaxy/managers/datasets.py
+++ b/lib/galaxy/managers/datasets.py
@@ -383,6 +383,9 @@ class _UnflattenedMetadataDatasetAssociationSerializer( base.ModelSerializer,
'meta_files' : self.serialize_meta_files,
'metadata' : self.serialize_metadata,
+ 'creating_job' : self.serialize_creating_job,
+ 'rerunnable' : self.serialize_rerunnable,
+
'parent_id' : self.serialize_id,
'designation' : lambda i, k, **c: i.designation,
@@ -446,6 +449,27 @@ class _UnflattenedMetadataDatasetAssociationSerializer( base.ModelSerializer,
return metadata
+ def serialize_creating_job( self, dataset, key, **context ):
+ """
+ Return the id of the Job that created this dataset (or its original)
+ or None if no `creating_job` is found.
+ """
+ if dataset.creating_job:
+ return self.serialize_id( dataset.creating_job, 'id' )
+ else:
+ return None
+
+ def serialize_rerunnable( self, dataset, key, **context ):
+ """
+ Return False if this tool that created this dataset can't be re-run
+ (e.g. upload).
+ """
+ if dataset.creating_job:
+ tool = self.app.toolbox.get_tool( dataset.creating_job.tool_id, dataset.creating_job.tool_version )
+ if tool and tool.is_workflow_compatible:
+ return True
+ return False
+
def serialize_converted_datasets( self, dataset_assoc, key, **context ):
"""
Return a file extension -> converted dataset encoded id map with all
diff --git a/lib/galaxy/managers/folders.py b/lib/galaxy/managers/folders.py
index 7968048bdcd..0f76208be47 100644
--- a/lib/galaxy/managers/folders.py
+++ b/lib/galaxy/managers/folders.py
@@ -2,7 +2,13 @@
Manager and Serializer for Library Folders.
"""
-import galaxy.exceptions
+from galaxy.exceptions import ItemAccessibilityException
+from galaxy.exceptions import InconsistentDatabase
+from galaxy.exceptions import RequestParameterInvalidException
+from galaxy.exceptions import InternalServerError
+from galaxy.exceptions import AuthenticationRequired
+from galaxy.exceptions import InsufficientPermissionsException
+from galaxy.exceptions import MalformedId
from sqlalchemy.orm.exc import MultipleResultsFound
from sqlalchemy.orm.exc import NoResultFound
import logging
@@ -34,11 +40,11 @@ class FolderManager( object ):
try:
folder = trans.sa_session.query( trans.app.model.LibraryFolder ).filter( trans.app.model.LibraryFolder.table.c.id == decoded_folder_id ).one()
except MultipleResultsFound:
- raise galaxy.exceptions.InconsistentDatabase( 'Multiple folders found with the same id.' )
+ raise InconsistentDatabase( 'Multiple folders found with the same id.' )
except NoResultFound:
- raise galaxy.exceptions.RequestParameterInvalidException( 'No folder found with the id provided.' )
+ raise RequestParameterInvalidException( 'No folder found with the id provided.' )
except Exception, e:
- raise galaxy.exceptions.InternalServerError( 'Error loading from the database.' + str( e ) )
+ raise InternalServerError( 'Error loading from the database.' + str( e ) )
folder = self.secure( trans, folder, check_manageable, check_accessible )
return folder
@@ -75,10 +81,10 @@ class FolderManager( object ):
:raises: AuthenticationRequired, InsufficientPermissionsException
"""
if not trans.user:
- raise galaxy.exceptions.AuthenticationRequired( "Must be logged in to manage Galaxy items.", type='error' )
+ raise AuthenticationRequired( "Must be logged in to manage Galaxy items.", type='error' )
current_user_roles = trans.get_current_user_roles()
if not trans.app.security_agent.can_manage_library_item( current_user_roles, folder ):
- raise galaxy.exceptions.InsufficientPermissionsException( "You don't have permissions to manage this folder.", type='error' )
+ raise InsufficientPermissionsException( "You don't have permissions to manage this folder.", type='error' )
else:
return folder
@@ -105,6 +111,7 @@ class FolderManager( object ):
folder_dict[ 'id' ] = 'F' + folder_dict[ 'id' ]
if folder_dict[ 'parent_id' ] is not None:
folder_dict[ 'parent_id' ] = 'F' + folder_dict[ 'parent_id' ]
+ folder_dict['update_time'] = folder.update_time.strftime( "%Y-%m-%d %I:%M %p" )
return folder_dict
def create( self, trans, parent_folder_id, new_folder_name, new_folder_description='' ):
@@ -126,7 +133,7 @@ class FolderManager( object ):
parent_folder = self.get( trans, parent_folder_id )
current_user_roles = trans.get_current_user_roles()
if not ( trans.user_is_admin or trans.app.security_agent.can_add_library_item( current_user_roles, parent_folder ) ):
- raise galaxy.exceptions.InsufficientPermissionsException( 'You do not have proper permission to create folders under given folder.' )
+ raise InsufficientPermissionsException( 'You do not have proper permission to create folders under given folder.' )
new_folder = trans.app.model.LibraryFolder( name=new_folder_name, description=new_folder_description )
# We are associating the last used genome build with folders, so we will always
# initialize a new folder with the first dbkey in genome builds list which is currently
@@ -139,12 +146,45 @@ class FolderManager( object ):
trans.app.security_agent.copy_library_permissions( trans, parent_folder, new_folder )
return new_folder
+ def update( self, trans, folder, name=None, description=None):
+ """
+ Update the given folder's name or description.
+
+ :param folder: the model object
+ :type folder: LibraryFolder
+ :param name: new name for the library folder
+ :type name: str
+ :param description: new description for the library folder
+ :type description: str
+
+ :returns: the folder
+ :rtype: LibraryFolder
+
+ :raises: ItemAccessibilityException, InsufficientPermissionsException
+ """
+ changed = False
+ if not trans.user_is_admin():
+ if not self.check_manageable( trans, folder ):
+ raise InsufficientPermissionsException( "You do not have proper permission to update the library folder." )
+ if folder.deleted is True:
+ raise ItemAccessibilityException( "You cannot update a deleted library folder. Undelete it first." )
+ if name is not None and name != folder.name:
+ folder.name = name
+ changed = True
+ if description is not None and description != folder.description:
+ folder.description = description
+ changed = True
+ if changed:
+ trans.sa_session.add( folder )
+ trans.sa_session.flush()
+ return folder
+
def delete( self, trans, folder, undelete=False ):
"""
Mark given folder deleted/undeleted based on the flag.
- :param folder: the model object
- :type folder: LibraryFolder
+ :param folder: the model object
+ :type folder: LibraryFolder
:param undelete: flag whether to delete (when False) or undelete
:type undelete: Bool
@@ -214,7 +254,7 @@ class FolderManager( object ):
if ( ( len( encoded_folder_id ) % 16 == 1 ) and encoded_folder_id.startswith( 'F' ) ):
cut_id = encoded_folder_id[ 1: ]
else:
- raise galaxy.exceptions.MalformedId( 'Malformed folder id ( %s ) specified, unable to decode.' % str( encoded_folder_id ) )
+ raise MalformedId( 'Malformed folder id ( %s ) specified, unable to decode.' % str( encoded_folder_id ) )
return cut_id
def decode_folder_id( self, trans, encoded_folder_id ):
@@ -232,7 +272,7 @@ class FolderManager( object ):
try:
decoded_id = trans.security.decode_id( encoded_folder_id )
except ValueError:
- raise galaxy.exceptions.MalformedId( "Malformed folder id ( %s ) specified, unable to decode" % ( str( encoded_folder_id ) ) )
+ raise MalformedId( "Malformed folder id ( %s ) specified, unable to decode" % ( str( encoded_folder_id ) ) )
return decoded_id
def cut_and_decode( self, trans, encoded_folder_id ):
diff --git a/lib/galaxy/managers/hdas.py b/lib/galaxy/managers/hdas.py
index db6e767d442..1ff4f98ff0f 100644
--- a/lib/galaxy/managers/hdas.py
+++ b/lib/galaxy/managers/hdas.py
@@ -280,6 +280,7 @@ class HDASerializer( # datasets._UnflattenedMetadataDatasetAssociationSerialize
'peek',
'creating_job',
+ 'rerunnable',
'uuid',
'permissions',
@@ -327,8 +328,6 @@ class HDASerializer( # datasets._UnflattenedMetadataDatasetAssociationSerialize
'file_path' : self._remap_from( 'file_name' ),
'resubmitted' : lambda i, k, **c: self.hda_manager.has_been_resubmitted( i ),
- 'creating_job' : self.serialize_creating_job,
-
'display_apps' : self.serialize_display_apps,
'display_types' : self.serialize_old_display_applications,
'visualizations': self.serialize_visualization_links,
@@ -433,12 +432,6 @@ class HDASerializer( # datasets._UnflattenedMetadataDatasetAssociationSerialize
}
return urls
- def serialize_creating_job( self, hda, key, **context ):
- if hda.creating_job:
- return self.app.security.encode_id(hda.creating_job.id),
- else:
- return None
-
class HDADeserializer( datasets.DatasetAssociationDeserializer,
taggable.TaggableDeserializerMixin,
diff --git a/lib/galaxy/managers/libraries.py b/lib/galaxy/managers/libraries.py
index 5eba473633a..3cd940d3718 100644
--- a/lib/galaxy/managers/libraries.py
+++ b/lib/galaxy/managers/libraries.py
@@ -8,6 +8,8 @@ from sqlalchemy.orm.exc import MultipleResultsFound
from sqlalchemy.orm.exc import NoResultFound
from galaxy import exceptions
+from galaxy.managers import folders
+from galaxy.util import pretty_print_time_interval
log = logging.getLogger( __name__ )
@@ -27,8 +29,8 @@ class LibraryManager( object ):
:param decoded_library_id: decoded library id
:type decoded_library_id: int
- :param check_accessible: flag whether to check that user can access item
- :type check_accessible: bool
+ :param check_accessible: flag whether to check that user can access item
+ :type check_accessible: bool
:returns: the requested library
:rtype: Library
@@ -70,6 +72,9 @@ class LibraryManager( object ):
if name is not None:
library.name = name
changed = True
+ # When library is renamed the root folder has to be renamed too.
+ folder_manager = folders.FolderManager()
+ folder_manager.update( trans, library.root_folder, name=name )
if description is not None:
library.description = description
changed = True
@@ -179,6 +184,7 @@ class LibraryManager( object ):
library_dict = library.to_dict( view='element', value_mapper={ 'id': trans.security.encode_id, 'root_folder_id': trans.security.encode_id } )
if trans.app.security_agent.library_is_public( library, contents=False ):
library_dict[ 'public' ] = True
+ library_dict[ 'create_time_pretty'] = pretty_print_time_interval( library_dict[ 'create_time' ], precise=True )
current_user_roles = trans.get_current_user_roles()
if not trans.user_is_admin():
library_dict[ 'can_user_add' ] = trans.app.security_agent.can_add_library_item( current_user_roles, library )
diff --git a/lib/galaxy/model/__init__.py b/lib/galaxy/model/__init__.py
index 2ecdaec2873..d676cbe6ee6 100644
--- a/lib/galaxy/model/__init__.py
+++ b/lib/galaxy/model/__init__.py
@@ -2373,7 +2373,7 @@ class HistoryDatasetAssociationSubset( object ):
class Library( object, Dictifiable, HasName ):
permitted_actions = get_permitted_actions( filter='LIBRARY' )
dict_collection_visible_keys = ( 'id', 'name' )
- dict_element_visible_keys = ( 'id', 'deleted', 'name', 'description', 'synopsis', 'root_folder_id' )
+ dict_element_visible_keys = ( 'id', 'deleted', 'name', 'description', 'synopsis', 'root_folder_id', 'create_time' )
def __init__( self, name=None, description=None, synopsis=None, root_folder=None ):
self.name = name or "Unnamed library"
@@ -3045,6 +3045,9 @@ class DatasetCollectionInstance( object, HasName ):
id=self.id,
name=self.name,
collection_type=self.collection.collection_type,
+ populated=self.collection.populated,
+ populated_state=self.collection.populated_state,
+ populated_state_message=self.collection.populated_state_message,
type="collection", # contents type (distinguished from file or folder (in case of library))
)
diff --git a/lib/galaxy/model/migrate/versions/0001_initial_tables.py b/lib/galaxy/model/migrate/versions/0001_initial_tables.py
index b3371ecb36a..50eec0e8f90 100644
--- a/lib/galaxy/model/migrate/versions/0001_initial_tables.py
+++ b/lib/galaxy/model/migrate/versions/0001_initial_tables.py
@@ -1,15 +1,13 @@
-from sqlalchemy import *
-from migrate import *
-
import datetime
-now = datetime.datetime.utcnow
+import logging
+
+from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, MetaData, Numeric, String, Table, TEXT
# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
+from galaxy.model.custom_types import JSONType, MetadataType, TrimmedString
-import logging
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
-
metadata = MetaData()
# Tables as of changeset 1464:c7acaa1bb88f
@@ -138,8 +136,7 @@ GalaxySession_table = Table( "galaxy_session", metadata,
Column( "current_history_id", Integer, ForeignKey( "history.id" ), nullable=True ),
Column( "session_key", TrimmedString( 255 ), index=True, unique=True ),
Column( "is_valid", Boolean, default=False ),
- Column( "prev_session_id", Integer )
- )
+ Column( "prev_session_id", Integer ) )
GalaxySessionToHistoryAssociation_table = Table( "galaxy_session_to_history", metadata,
Column( "id", Integer, primary_key=True ),
@@ -155,8 +152,7 @@ StoredWorkflow_table = Table( "stored_workflow", metadata,
Column( "latest_workflow_id", Integer,
ForeignKey( "workflow.id", use_alter=True, name='stored_workflow_latest_workflow_id_fk' ), index=True ),
Column( "name", TEXT ),
- Column( "deleted", Boolean, default=False ),
- )
+ Column( "deleted", Boolean, default=False ) )
Workflow_table = Table( "workflow", metadata,
Column( "id", Integer, primary_key=True ),
@@ -165,8 +161,7 @@ Workflow_table = Table( "workflow", metadata,
Column( "stored_workflow_id", Integer, ForeignKey( "stored_workflow.id" ), index=True, nullable=False ),
Column( "name", TEXT ),
Column( "has_cycles", Boolean ),
- Column( "has_errors", Boolean )
- )
+ Column( "has_errors", Boolean ) )
WorkflowStep_table = Table( "workflow_step", metadata,
Column( "id", Integer, primary_key=True ),
@@ -180,22 +175,19 @@ WorkflowStep_table = Table( "workflow_step", metadata,
Column( "tool_errors", JSONType ),
Column( "position", JSONType ),
Column( "config", JSONType ),
- Column( "order_index", Integer )
- )
+ Column( "order_index", Integer ) )
WorkflowStepConnection_table = Table( "workflow_step_connection", metadata,
Column( "id", Integer, primary_key=True ),
Column( "output_step_id", Integer, ForeignKey( "workflow_step.id" ), index=True ),
Column( "input_step_id", Integer, ForeignKey( "workflow_step.id" ), index=True ),
Column( "output_name", TEXT ),
- Column( "input_name", TEXT)
- )
+ Column( "input_name", TEXT) )
StoredWorkflowUserShareAssociation_table = Table( "stored_workflow_user_share_connection", metadata,
Column( "id", Integer, primary_key=True ),
Column( "stored_workflow_id", Integer, ForeignKey( "stored_workflow.id" ), index=True ),
- Column( "user_id", Integer, ForeignKey( "galaxy_user.id" ), index=True )
- )
+ Column( "user_id", Integer, ForeignKey( "galaxy_user.id" ), index=True ) )
StoredWorkflowMenuEntry_table = Table( "stored_workflow_menu_entry", metadata,
Column( "id", Integer, primary_key=True ),
@@ -203,11 +195,7 @@ StoredWorkflowMenuEntry_table = Table( "stored_workflow_menu_entry", metadata,
Column( "user_id", Integer, ForeignKey( "galaxy_user.id" ), index=True ),
Column( "order_index", Integer ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.create_all()
-
-## def downgrade(migrate_engine):
-## metadata.bind = migrate_engine
-## # Operations to reverse the above upgrade go here.
-## pass
diff --git a/lib/galaxy/model/migrate/versions/0002_metadata_file_table.py b/lib/galaxy/model/migrate/versions/0002_metadata_file_table.py
index aef94d00334..d1b20f0f2bf 100644
--- a/lib/galaxy/model/migrate/versions/0002_metadata_file_table.py
+++ b/lib/galaxy/model/migrate/versions/0002_metadata_file_table.py
@@ -1,15 +1,10 @@
-from sqlalchemy import *
-from migrate import *
-
import datetime
-now = datetime.datetime.utcnow
-
-# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, MetaData, Table, TEXT
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
# New table in changeset 1568:0b022adfdc34
@@ -22,11 +17,13 @@ MetadataFile_table = Table( "metadata_file", metadata,
Column( "deleted", Boolean, index=True, default=False ),
Column( "purged", Boolean, index=True, default=False ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
MetadataFile_table.create()
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0003_security_and_libraries.py b/lib/galaxy/model/migrate/versions/0003_security_and_libraries.py
index 3e3597a51eb..8bb6ff0366d 100644
--- a/lib/galaxy/model/migrate/versions/0003_security_and_libraries.py
+++ b/lib/galaxy/model/migrate/versions/0003_security_and_libraries.py
@@ -1,13 +1,15 @@
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from sqlalchemy.exc import *
-from migrate import *
-from migrate.changeset import *
-
import datetime
-now = datetime.datetime.utcnow
+import logging
+import sys
-import sys, logging
+from migrate import ForeignKeyConstraint
+from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Index, Integer, MetaData, String, Table, TEXT
+from sqlalchemy.exc import NoSuchTableError
+
+# Need our custom types, but don't import anything else from model
+from galaxy.model.custom_types import JSONType, MetadataType, TrimmedString
+
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler( sys.stdout )
@@ -16,9 +18,6 @@ formatter = logging.Formatter( format )
handler.setFormatter( formatter )
log.addHandler( handler )
-# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
-
metadata = MetaData()
# New tables as of changeset 2341:5498ac35eedd
@@ -131,13 +130,13 @@ DefaultHistoryPermissions_table = Table( "default_history_permissions", metadata
LibraryDataset_table = Table( "library_dataset", metadata,
Column( "id", Integer, primary_key=True ),
- Column( "library_dataset_dataset_association_id", Integer, ForeignKey( "library_dataset_dataset_association.id", use_alter=True, name="library_dataset_dataset_association_id_fk" ), nullable=True, index=True ),#current version of dataset, if null, there is not a current version selected
+ Column( "library_dataset_dataset_association_id", Integer, ForeignKey( "library_dataset_dataset_association.id", use_alter=True, name="library_dataset_dataset_association_id_fk" ), nullable=True, index=True ), # current version of dataset, if null, there is not a current version selected
Column( "folder_id", Integer, ForeignKey( "library_folder.id" ), index=True ),
Column( "order_id", Integer ),
Column( "create_time", DateTime, default=now ),
Column( "update_time", DateTime, default=now, onupdate=now ),
- Column( "name", TrimmedString( 255 ), key="_name" ), #when not None/null this will supercede display in library (but not when imported into user's history?)
- Column( "info", TrimmedString( 255 ), key="_info" ), #when not None/null this will supercede display in library (but not when imported into user's history?)
+ Column( "name", TrimmedString( 255 ), key="_name" ), # when not None/null this will supercede display in library (but not when imported into user's history?)
+ Column( "info", TrimmedString( 255 ), key="_info" ), # when not None/null this will supercede display in library (but not when imported into user's history?)
Column( "deleted", Boolean, index=True, default=False ) )
LibraryDatasetDatasetAssociation_table = Table( "library_dataset_dataset_association", metadata,
@@ -173,7 +172,7 @@ Library_table = Table( "library", metadata,
LibraryFolder_table = Table( "library_folder", metadata,
Column( "id", Integer, primary_key=True ),
- Column( "parent_id", Integer, ForeignKey( "library_folder.id" ), nullable = True, index=True ),
+ Column( "parent_id", Integer, ForeignKey( "library_folder.id" ), nullable=True, index=True ),
Column( "create_time", DateTime, default=now ),
Column( "update_time", DateTime, default=now, onupdate=now ),
Column( "name", TEXT ),
@@ -304,35 +303,35 @@ JobExternalOutputMetadata_table = Table( "job_external_output_metadata", metadat
Column( "job_runner_external_pid", String( 255 ) ) )
Index( "ix_jeom_library_dataset_dataset_association_id", JobExternalOutputMetadata_table.c.library_dataset_dataset_association_id )
+
def upgrade(migrate_engine):
- db_session = scoped_session( sessionmaker( bind=migrate_engine, autoflush=False, autocommit=True ) )
metadata.bind = migrate_engine
# Load existing tables
metadata.reflect()
def nextval( table, col='id' ):
- if migrate_engine.name == 'postgres':
+ if migrate_engine.name in ['postgres', 'postgresql']:
return "nextval('%s_%s_seq')" % ( table, col )
- elif migrate_engine.name == 'mysql' or migrate_engine.name == 'sqlite':
+ elif migrate_engine.name in ['mysql', 'sqlite']:
return "null"
else:
raise Exception( 'Unable to convert data for unknown database type: %s' % migrate_engine.name )
def localtimestamp():
- if migrate_engine.name == 'postgres' or migrate_engine.name == 'mysql':
- return "LOCALTIMESTAMP"
- elif migrate_engine.name == 'sqlite':
- return "current_date || ' ' || current_time"
- else:
- raise Exception( 'Unable to convert data for unknown database type: %s' % db )
+ if migrate_engine.name in ['mysql', 'postgres', 'postgresql']:
+ return "LOCALTIMESTAMP"
+ elif migrate_engine.name == 'sqlite':
+ return "current_date || ' ' || current_time"
+ else:
+ raise Exception( 'Unable to convert data for unknown database type: %s' % migrate_engine.name )
def boolean_false():
- if migrate_engine.name == 'postgres' or migrate_engine.name == 'mysql':
- return False
- elif migrate_engine.name == 'sqlite':
- return 0
- else:
- raise Exception( 'Unable to convert data for unknown database type: %s' % db )
+ if migrate_engine.name in ['mysql', 'postgres', 'postgresql']:
+ return False
+ elif migrate_engine.name == 'sqlite':
+ return 0
+ else:
+ raise Exception( 'Unable to convert data for unknown database type: %s' % migrate_engine.name )
# Add 2 new columns to the galaxy_user table
try:
@@ -441,7 +440,7 @@ def upgrade(migrate_engine):
LibraryDatasetDatasetAssociation_table = None
log.debug( "Failed loading table library_dataset_dataset_association" )
if migrate_engine.name != 'sqlite':
- #Sqlite can't alter table add foreign key.
+ # Sqlite can't alter table add foreign key.
if MetadataFile_table is not None and LibraryDatasetDatasetAssociation_table is not None:
try:
cons = ForeignKeyConstraint( [MetadataFile_table.c.lda_id],
@@ -453,10 +452,10 @@ def upgrade(migrate_engine):
log.debug( "Adding foreign key constraint 'metadata_file_lda_id_fkey' to table 'metadata_file' failed: %s" % ( str( e ) ) )
# Make sure we have at least 1 user
cmd = "SELECT * FROM galaxy_user;"
- users = db_session.execute( cmd ).fetchall()
+ users = migrate_engine.execute( cmd ).fetchall()
if users:
cmd = "SELECT * FROM role;"
- roles = db_session.execute( cmd ).fetchall()
+ roles = migrate_engine.execute( cmd ).fetchall()
if not roles:
# Create private roles for each user - pass 1
cmd = \
@@ -471,13 +470,13 @@ def upgrade(migrate_engine):
"FROM galaxy_user " + \
"ORDER BY id;"
cmd = cmd % ( nextval('role'), localtimestamp(), localtimestamp(), boolean_false() )
- db_session.execute( cmd )
+ migrate_engine.execute( cmd )
# Create private roles for each user - pass 2
- if migrate_engine.name == 'postgres' or migrate_engine.name == 'sqlite':
+ if migrate_engine.name in ['postgres', 'postgresql', 'sqlite']:
cmd = "UPDATE role SET description = 'Private role for ' || description;"
elif migrate_engine.name == 'mysql':
cmd = "UPDATE role SET description = CONCAT( 'Private role for ', description );"
- db_session.execute( cmd )
+ migrate_engine.execute( cmd )
# Create private roles for each user - pass 3
cmd = \
"INSERT INTO user_role_association " + \
@@ -490,7 +489,7 @@ def upgrade(migrate_engine):
"WHERE galaxy_user.email = role.name " + \
"ORDER BY galaxy_user.id;"
cmd = cmd % ( nextval('user_role_association'), localtimestamp(), localtimestamp() )
- db_session.execute( cmd )
+ migrate_engine.execute( cmd )
# Create default permissions for each user
cmd = \
"INSERT INTO default_user_permissions " + \
@@ -502,7 +501,7 @@ def upgrade(migrate_engine):
"JOIN user_role_association ON user_role_association.user_id = galaxy_user.id " + \
"ORDER BY galaxy_user.id;"
cmd = cmd % nextval('default_user_permissions')
- db_session.execute( cmd )
+ migrate_engine.execute( cmd )
# Create default history permissions for each active history associated with a user
cmd = \
@@ -515,7 +514,7 @@ def upgrade(migrate_engine):
"JOIN user_role_association ON user_role_association.user_id = history.user_id " + \
"WHERE history.purged = %s AND history.user_id IS NOT NULL;"
cmd = cmd % ( nextval('default_history_permissions'), boolean_false() )
- db_session.execute( cmd )
+ migrate_engine.execute( cmd )
# Create "manage permissions" dataset_permissions for all activate-able datasets
cmd = \
"INSERT INTO dataset_permissions " + \
@@ -531,23 +530,11 @@ def upgrade(migrate_engine):
"JOIN user_role_association ON user_role_association.user_id = history.user_id " + \
"WHERE dataset.purged = %s AND history.user_id IS NOT NULL;"
cmd = cmd % ( nextval('dataset_permissions'), localtimestamp(), localtimestamp(), boolean_false() )
- db_session.execute( cmd )
+ migrate_engine.execute( cmd )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
- if migrate_engine.name == 'postgres':
- # http://blog.pythonisito.com/2008/01/cascading-drop-table-with-sqlalchemy.html
- from sqlalchemy.databases import postgres
- class PGCascadeSchemaDropper(postgres.PGSchemaDropper):
- def visit_table(self, table):
- for column in table.columns:
- if column.default is not None:
- self.traverse_single(column.default)
- self.append("\nDROP TABLE " +
- self.preparer.format_table(table) +
- " CASCADE")
- self.execute()
- postgres.dialect.schemadropper = PGCascadeSchemaDropper
# Load existing tables
metadata.reflect()
# NOTE: all new data added in the upgrade method is eliminated here via table drops
diff --git a/lib/galaxy/model/migrate/versions/0004_indexes_and_defaults.py b/lib/galaxy/model/migrate/versions/0004_indexes_and_defaults.py
index 907f46410b1..1039c36fad8 100644
--- a/lib/galaxy/model/migrate/versions/0004_indexes_and_defaults.py
+++ b/lib/galaxy/model/migrate/versions/0004_indexes_and_defaults.py
@@ -1,7 +1,7 @@
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-import sys, logging
+import logging
+import sys
+
+from sqlalchemy import Index, MetaData, Table
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
@@ -13,18 +13,19 @@ log.addHandler( handler )
metadata = MetaData()
+
def upgrade(migrate_engine):
- db_session = scoped_session( sessionmaker( bind=migrate_engine, autoflush=False, autocommit=True ) )
metadata.bind = migrate_engine
User_table = Table( "galaxy_user", metadata, autoload=True )
HistoryDatasetAssociation_table = Table( "history_dataset_association", metadata, autoload=True )
+
def boolean_false():
- if migrate_engine.name == 'postgresql' or migrate_engine.name == 'mysql':
- return False
- elif migrate_engine.name == 'sqlite':
- return 0
- else:
- raise Exception( 'Unable to convert data for unknown database type: %s' % migrate_engine.name)
+ if migrate_engine.name in ['mysql', 'postgres', 'postgresql']:
+ return False
+ elif migrate_engine.name == 'sqlite':
+ return 0
+ else:
+ raise Exception( 'Unable to convert data for unknown database type: %s' % migrate_engine.name)
# Load existing tables
metadata.reflect()
# Add 2 indexes to the galaxy_user table
@@ -42,13 +43,13 @@ def upgrade(migrate_engine):
cmd = "UPDATE galaxy_user SET deleted = %s WHERE deleted is null"
cmd = cmd % boolean_false()
try:
- db_session.execute( cmd )
+ migrate_engine.execute( cmd )
except Exception, e:
log.debug( "Setting default data for galaxy_user.deleted column failed: %s" % ( str( e ) ) )
cmd = "UPDATE galaxy_user SET purged = %s WHERE purged is null"
cmd = cmd % boolean_false()
try:
- db_session.execute( cmd )
+ migrate_engine.execute( cmd )
except Exception, e:
log.debug( "Setting default data for galaxy_user.purged column failed: %s" % ( str( e ) ) )
# Add 1 index to the history_dataset_association table
@@ -57,6 +58,8 @@ def upgrade(migrate_engine):
i.create()
except Exception, e:
log.debug( "Adding index 'ix_hda_copied_from_library_dataset_dataset_association_id' to history_dataset_association table failed: %s" % ( str( e ) ) )
+
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
pass
diff --git a/lib/galaxy/model/migrate/versions/0005_cleanup_datasets_fix.py b/lib/galaxy/model/migrate/versions/0005_cleanup_datasets_fix.py
index ea62c5b5c9f..2a8042eb25e 100644
--- a/lib/galaxy/model/migrate/versions/0005_cleanup_datasets_fix.py
+++ b/lib/galaxy/model/migrate/versions/0005_cleanup_datasets_fix.py
@@ -2,31 +2,18 @@ import datetime
import errno
import logging
import os
-import sys
import time
+from sqlalchemy import and_, Boolean, Column, DateTime, false, ForeignKey, Integer, MetaData, not_, Numeric, Table, TEXT, true
+from sqlalchemy.orm import backref, mapper, relation, scoped_session, sessionmaker
+
+from galaxy.datatypes.metadata import MetadataCollection
+from galaxy.model.custom_types import MetadataType, TrimmedString
+from galaxy.util.bunch import Bunch
+
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
-handler = logging.StreamHandler( sys.stdout )
-format = "%(name)s %(levelname)s %(asctime)s %(message)s"
-formatter = logging.Formatter( format )
-handler.setFormatter( formatter )
-log.addHandler( handler )
-
-from sqlalchemy import and_, false, true
-
-from sqlalchemy import *
now = datetime.datetime.utcnow
-from sqlalchemy.orm import *
-
-from migrate import *
-from migrate.changeset import *
-
-from galaxy.model.custom_types import *
-
-from galaxy.util.bunch import Bunch
-
-
metadata = MetaData()
context = scoped_session( sessionmaker( autoflush=False, autocommit=True ) )
@@ -35,6 +22,7 @@ context = scoped_session( sessionmaker( autoflush=False, autocommit=True ) )
def get_permitted_actions( **kwds ):
return Bunch()
+
def directory_hash_id( id ):
s = str( id )
l = len( s )
@@ -46,21 +34,22 @@ def directory_hash_id( id ):
# Drop the last three digits -- 1000 files per directory
padded = padded[:-3]
# Break into chunks of three
- return [ padded[i*3:(i+1)*3] for i in range( len( padded ) // 3 ) ]
+ return [ padded[i * 3:(i + 1) * 3] for i in range( len( padded ) // 3 ) ]
class Dataset( object ):
- states = Bunch( NEW = 'new',
- UPLOAD = 'upload',
- QUEUED = 'queued',
- RUNNING = 'running',
- OK = 'ok',
- EMPTY = 'empty',
- ERROR = 'error',
- DISCARDED = 'discarded' )
+ states = Bunch( NEW='new',
+ UPLOAD='upload',
+ QUEUED='queued',
+ RUNNING='running',
+ OK='ok',
+ EMPTY='empty',
+ ERROR='error',
+ DISCARDED='discarded' )
permitted_actions = get_permitted_actions( filter='DATASET' )
file_path = "/tmp/"
engine = None
+
def __init__( self, id=None, state=None, external_filename=None, extra_files_path=None, file_size=None, purgable=True ):
self.id = id
self.state = state
@@ -70,6 +59,7 @@ class Dataset( object ):
self.external_filename = external_filename
self._extra_files_path = extra_files_path
self.file_size = file_size
+
def get_file_name( self ):
if not self.external_filename:
assert self.id is not None, "ID must be set before filename used (commit the object)"
@@ -92,23 +82,26 @@ class Dataset( object ):
filename = self.external_filename
# Make filename absolute
return os.path.abspath( filename )
- def set_file_name ( self, filename ):
+
+ def set_file_name( self, filename ):
if not filename:
self.external_filename = None
else:
self.external_filename = filename
file_name = property( get_file_name, set_file_name )
+
@property
def extra_files_path( self ):
if self._extra_files_path:
path = self._extra_files_path
else:
path = os.path.join( self.file_path, "dataset_%d_files" % self.id )
- #only use path directly under self.file_path if it exists
+ # only use path directly under self.file_path if it exists
if not os.path.exists( path ):
path = os.path.join( os.path.join( self.file_path, *directory_hash_id( self.id ) ), "dataset_%d_files" % self.id )
# Make path absolute
return os.path.abspath( path )
+
def get_size( self ):
"""Returns the size of the data on disk"""
if self.file_size:
@@ -118,6 +111,7 @@ class Dataset( object ):
return os.path.getsize( self.file_name )
except OSError:
return 0
+
def set_size( self ):
"""Returns the size of the data on disk"""
try:
@@ -125,12 +119,15 @@ class Dataset( object ):
self.file_size = os.path.getsize( self.file_name )
except OSError:
self.file_size = 0
+
def has_data( self ):
"""Detects whether there is any data"""
return self.get_size() > 0
+
def mark_deleted( self, include_children=True ):
self.deleted = True
# FIXME: sqlalchemy will replace this
+
def _delete(self):
"""Remove the file that corresponds to this data"""
try:
@@ -138,13 +135,15 @@ class Dataset( object ):
except OSError, e:
log.critical('%s delete error %s' % (self.__class__.__name__, e))
+
class DatasetInstance( object ):
"""A base class for all 'dataset instances', HDAs, LDAs, etc"""
states = Dataset.states
permitted_actions = Dataset.permitted_actions
+
def __init__( self, id=None, hid=None, name=None, info=None, blurb=None, peek=None, extension=None,
dbkey=None, metadata=None, history=None, dataset=None, deleted=False, designation=None,
- parent_id=None, validation_errors=None, visible=True, create_dataset = False ):
+ parent_id=None, validation_errors=None, visible=True, create_dataset=False ):
self.name = name or "Unnamed dataset"
self.id = id
self.info = info
@@ -153,7 +152,7 @@ class DatasetInstance( object ):
self.extension = extension
self.designation = designation
self.metadata = metadata or dict()
- if dbkey: #dbkey is stored in metadata, only set if non-zero, or else we could clobber one supplied by input 'metadata'
+ if dbkey: # dbkey is stored in metadata, only set if non-zero, or else we could clobber one supplied by input 'metadata'
self.dbkey = dbkey
self.deleted = deleted
self.visible = visible
@@ -165,42 +164,51 @@ class DatasetInstance( object ):
self.dataset = dataset
self.parent_id = parent_id
self.validation_errors = validation_errors
+
@property
def ext( self ):
return self.extension
+
def get_dataset_state( self ):
return self.dataset.state
- def set_dataset_state ( self, state ):
+
+ def set_dataset_state( self, state ):
self.dataset.state = state
context.add( self.dataset )
- context.flush() #flush here, because hda.flush() won't flush the Dataset object
+ context.flush() # flush here, because hda.flush() won't flush the Dataset object
state = property( get_dataset_state, set_dataset_state )
+
def get_file_name( self ):
return self.dataset.get_file_name()
- def set_file_name (self, filename):
+
+ def set_file_name(self, filename):
return self.dataset.set_file_name( filename )
file_name = property( get_file_name, set_file_name )
+
@property
def extra_files_path( self ):
return self.dataset.extra_files_path
- @property
- def datatype( self ):
- return datatypes_registry.get_datatype_by_extension( self.extension )
+
def get_metadata( self ):
- if not hasattr( self, '_metadata_collection' ) or self._metadata_collection.parent != self: #using weakref to store parent (to prevent circ ref), does a context.clear() cause parent to be invalidated, while still copying over this non-database attribute?
+ if not hasattr( self, '_metadata_collection' ) or self._metadata_collection.parent != self: # using weakref to store parent (to prevent circ ref), does a context.clear() cause parent to be invalidated, while still copying over this non-database attribute?
self._metadata_collection = MetadataCollection( self )
return self._metadata_collection
+
def set_metadata( self, bunch ):
# Needs to accept a MetadataCollection, a bunch, or a dict
self._metadata = self.metadata.make_dict_copy( bunch )
metadata = property( get_metadata, set_metadata )
# This provide backwards compatibility with using the old dbkey
# field in the database. That field now maps to "old_dbkey" (see mapping.py).
+
def get_dbkey( self ):
dbkey = self.metadata.dbkey
- if not isinstance(dbkey, list): dbkey = [dbkey]
- if dbkey in [[None], []]: return "?"
+ if not isinstance(dbkey, list):
+ dbkey = [dbkey]
+ if dbkey in [[None], []]:
+ return "?"
return dbkey[0]
+
def set_dbkey( self, value ):
if "dbkey" in self.datatype.metadata_spec:
if not isinstance(value, list):
@@ -208,83 +216,95 @@ class DatasetInstance( object ):
else:
self.metadata.dbkey = value
dbkey = property( get_dbkey, set_dbkey )
- def change_datatype( self, new_ext ):
- self.clear_associated_files()
- datatypes_registry.change_datatype( self, new_ext )
+
def get_size( self ):
"""Returns the size of the data on disk"""
return self.dataset.get_size()
+
def set_size( self ):
"""Returns the size of the data on disk"""
return self.dataset.set_size()
+
def has_data( self ):
"""Detects whether there is any data"""
return self.dataset.has_data()
+
def get_raw_data( self ):
"""Returns the full data. To stream it open the file_name and read/write as needed"""
return self.datatype.get_raw_data( self )
+
def write_from_stream( self, stream ):
"""Writes data from a stream"""
self.datatype.write_from_stream(self, stream)
+
def set_raw_data( self, data ):
"""Saves the data on the disc"""
self.datatype.set_raw_data(self, data)
- def get_mime( self ):
- """Returns the mime type of the data"""
- return datatypes_registry.get_mimetype_by_extension( self.extension.lower() )
+
def set_peek( self, is_multi_byte=False ):
return self.datatype.set_peek( self, is_multi_byte=is_multi_byte )
+
def init_meta( self, copy_from=None ):
return self.datatype.init_meta( self, copy_from=copy_from )
+
def set_meta( self, **kwd ):
- self.clear_associated_files( metadata_safe = True )
+ self.clear_associated_files( metadata_safe=True )
return self.datatype.set_meta( self, **kwd )
+
def missing_meta( self, **kwd ):
return self.datatype.missing_meta( self, **kwd )
+
def as_display_type( self, type, **kwd ):
return self.datatype.as_display_type( self, type, **kwd )
+
def display_peek( self ):
return self.datatype.display_peek( self )
+
def display_name( self ):
return self.datatype.display_name( self )
+
def display_info( self ):
return self.datatype.display_info( self )
+
def get_converted_files_by_type( self, file_type ):
valid = []
for assoc in self.implicitly_converted_datasets:
if not assoc.deleted and assoc.type == file_type:
valid.append( assoc.dataset )
return valid
- def clear_associated_files( self, metadata_safe = False, purge = False ):
+
+ def clear_associated_files( self, metadata_safe=False, purge=False ):
raise 'Unimplemented'
+
def get_child_by_designation(self, designation):
for child in self.children:
if child.designation == designation:
return child
return None
- def get_converter_types(self):
- return self.datatype.get_converter_types( self, datatypes_registry)
- def find_conversion_destination( self, accepted_formats, **kwd ):
- """Returns ( target_ext, exisiting converted dataset )"""
- return self.datatype.find_conversion_destination( self, accepted_formats, datatypes_registry, **kwd )
+
def add_validation_error( self, validation_error ):
self.validation_errors.append( validation_error )
+
def extend_validation_errors( self, validation_errors ):
self.validation_errors.extend(validation_errors)
+
def mark_deleted( self, include_children=True ):
self.deleted = True
if include_children:
for child in self.children:
child.mark_deleted()
+
def mark_undeleted( self, include_children=True ):
self.deleted = False
if include_children:
for child in self.children:
child.mark_undeleted()
+
def undeletable( self ):
if self.purged:
return False
return True
+
@property
def source_library_dataset( self ):
def get_source( dataset ):
@@ -305,10 +325,10 @@ class DatasetInstance( object ):
class HistoryDatasetAssociation( DatasetInstance ):
def __init__( self,
- hid = None,
- history = None,
- copied_from_history_dataset_association = None,
- copied_from_library_dataset_dataset_association = None,
+ hid=None,
+ history=None,
+ copied_from_history_dataset_association=None,
+ copied_from_library_dataset_dataset_association=None,
**kwd ):
DatasetInstance.__init__( self, **kwd )
self.hid = hid
@@ -316,7 +336,8 @@ class HistoryDatasetAssociation( DatasetInstance ):
self.history = history
self.copied_from_history_dataset_association = copied_from_history_dataset_association
self.copied_from_library_dataset_dataset_association = copied_from_library_dataset_dataset_association
- def copy( self, copy_children = False, parent_id = None, target_history = None ):
+
+ def copy( self, copy_children=False, parent_id=None, target_history=None ):
hda = HistoryDatasetAssociation( hid=self.hid,
name=self.name,
info=self.info,
@@ -324,12 +345,12 @@ class HistoryDatasetAssociation( DatasetInstance ):
peek=self.peek,
extension=self.extension,
dbkey=self.dbkey,
- dataset = self.dataset,
+ dataset=self.dataset,
visible=self.visible,
deleted=self.deleted,
parent_id=parent_id,
copied_from_history_dataset_association=self,
- history = target_history )
+ history=target_history )
context.add( hda )
context.flush()
hda.set_size()
@@ -337,12 +358,13 @@ class HistoryDatasetAssociation( DatasetInstance ):
hda.metadata = self.metadata
if copy_children:
for child in self.children:
- child_copy = child.copy( copy_children = copy_children, parent_id = hda.id )
+ child.copy( copy_children=copy_children, parent_id=hda.id )
if not self.datatype.copy_safe_peek:
# In some instances peek relies on dataset_id, i.e. gmaj.zip for viewing MAFs
hda.set_peek()
context.flush()
return hda
+
def to_library_dataset_dataset_association( self, target_folder, replace_dataset=None, parent_id=None ):
if replace_dataset:
# The replace_dataset param ( when not None ) refers to a LibraryDataset that is being replaced with a new version.
@@ -379,18 +401,18 @@ class HistoryDatasetAssociation( DatasetInstance ):
context.add( library_dataset )
context.flush()
for child in self.children:
- child_copy = child.to_library_dataset_dataset_association( target_folder=target_folder, replace_dataset=replace_dataset, parent_id=ldda.id )
+ child.to_library_dataset_dataset_association( target_folder=target_folder, replace_dataset=replace_dataset, parent_id=ldda.id )
if not self.datatype.copy_safe_peek:
# In some instances peek relies on dataset_id, i.e. gmaj.zip for viewing MAFs
ldda.set_peek()
context.flush()
return ldda
- def clear_associated_files( self, metadata_safe = False, purge = False ):
+
+ def clear_associated_files( self, metadata_safe=False, purge=False ):
# metadata_safe = True means to only clear when assoc.metadata_safe == False
for assoc in self.implicitly_converted_datasets:
if not metadata_safe or not assoc.metadata_safe:
- assoc.clear( purge = purge )
-
+ assoc.clear( purge=purge )
class LibraryDatasetDatasetAssociation( DatasetInstance ):
@@ -405,6 +427,7 @@ class LibraryDatasetDatasetAssociation( DatasetInstance ):
self.copied_from_library_dataset_dataset_association = copied_from_library_dataset_dataset_association
self.library_dataset = library_dataset
self.user = user
+
def to_history_dataset_association( self, target_history, parent_id=None ):
hid = target_history._next_hid()
hda = HistoryDatasetAssociation( name=self.name,
@@ -420,16 +443,17 @@ class LibraryDatasetDatasetAssociation( DatasetInstance ):
copied_from_library_dataset_dataset_association=self,
history=target_history,
hid=hid )
- context.flush()
- hda.metadata = self.metadata #need to set after flushed, as MetadataFiles require dataset.id
- for child in self.children:
- child_copy = child.to_history_dataset_association( target_history=target_history, parent_id=hda.id )
- if not self.datatype.copy_safe_peek:
- hda.set_peek() #in some instances peek relies on dataset_id, i.e. gmaj.zip for viewing MAFs
context.add( hda )
context.flush()
+ hda.metadata = self.metadata # need to set after flushed, as MetadataFiles require dataset.id
+ for child in self.children:
+ child.to_history_dataset_association( target_history=target_history, parent_id=hda.id )
+ if not self.datatype.copy_safe_peek:
+ hda.set_peek() # in some instances peek relies on dataset_id, i.e. gmaj.zip for viewing MAFs
+ context.flush()
return hda
- def copy( self, copy_children = False, parent_id = None, target_folder = None ):
+
+ def copy( self, copy_children=False, parent_id=None, target_folder=None ):
ldda = LibraryDatasetDatasetAssociation( name=self.name,
info=self.info,
blurb=self.blurb,
@@ -444,18 +468,20 @@ class LibraryDatasetDatasetAssociation( DatasetInstance ):
folder=target_folder )
context.add( ldda )
context.flush()
- # Need to set after flushed, as MetadataFiles require dataset.id
+ # Need to set after flushed, as MetadataFiles require dataset.id
ldda.metadata = self.metadata
if copy_children:
for child in self.children:
- child_copy = child.copy( copy_children = copy_children, parent_id = ldda.id )
+ child.copy( copy_children=copy_children, parent_id=ldda.id )
if not self.datatype.copy_safe_peek:
- # In some instances peek relies on dataset_id, i.e. gmaj.zip for viewing MAFs
+ # In some instances peek relies on dataset_id, i.e. gmaj.zip for viewing MAFs
ldda.set_peek()
context.flush()
return ldda
- def clear_associated_files( self, metadata_safe = False, purge = False ):
+
+ def clear_associated_files( self, metadata_safe=False, purge=False ):
return
+
def get_library_item_info_templates( self, template_list=[], restrict=False ):
# If restrict is True, we'll return only those templates directly associated with this LibraryDatasetDatasetAssociation
if self.library_dataset_dataset_info_template_associations:
@@ -464,7 +490,6 @@ class LibraryDatasetDatasetAssociation( DatasetInstance ):
return template_list
-
class LibraryDataset( object ):
# This class acts as a proxy to the currently selected LDDA
def __init__( self, folder=None, order_id=None, name=None, info=None, library_dataset_dataset_association=None, **kwd ):
@@ -473,11 +498,13 @@ class LibraryDataset( object ):
self.name = name
self.info = info
self.library_dataset_dataset_association = library_dataset_dataset_association
+
def set_library_dataset_dataset_association( self, ldda ):
self.library_dataset_dataset_association = ldda
ldda.library_dataset = self
context.add_all( ( self, ldda ) )
context.flush()
+
def get_info( self ):
if self.library_dataset_dataset_association:
return self.library_dataset_dataset_association.info
@@ -485,9 +512,11 @@ class LibraryDataset( object ):
return self._info
else:
return 'no info'
+
def set_info( self, info ):
self._info = info
info = property( get_info, set_info )
+
def get_name( self ):
if self.library_dataset_dataset_association:
return self.library_dataset_dataset_association.name
@@ -495,19 +524,24 @@ class LibraryDataset( object ):
return self._name
else:
return 'Unnamed dataset'
+
def set_name( self, name ):
self._name = name
name = property( get_name, set_name )
+
def display_name( self ):
self.library_dataset_dataset_association.display_name()
+
def get_purged( self ):
return self.library_dataset_dataset_association.dataset.purged
+
def set_purged( self, purged ):
if purged:
raise Exception( "Not implemented" )
if not purged and self.purged:
raise Exception( "Cannot unpurge once purged" )
purged = property( get_purged, set_purged )
+
def get_library_item_info_templates( self, template_list=[], restrict=False ):
# If restrict is True, we'll return only those templates directly associated with this LibraryDataset
if self.library_dataset_info_template_associations:
@@ -516,7 +550,7 @@ class LibraryDataset( object ):
self.folder.get_library_item_info_templates( template_list, restrict )
return template_list
-##tables
+# tables
Dataset.table = Table( "dataset", metadata,
@@ -532,7 +566,6 @@ Dataset.table = Table( "dataset", metadata,
Column( 'file_size', Numeric( 15, 0 ) ) )
-
HistoryDatasetAssociation.table = Table( "history_dataset_association", metadata,
Column( "id", Integer, primary_key=True ),
Column( "dataset_id", Integer, ForeignKey( "dataset.id" ), index=True ),
@@ -575,17 +608,16 @@ LibraryDatasetDatasetAssociation.table = Table( "library_dataset_dataset_associa
LibraryDataset.table = Table( "library_dataset", metadata,
Column( "id", Integer, primary_key=True ),
- Column( "library_dataset_dataset_association_id", Integer, ForeignKey( "library_dataset_dataset_association.id", use_alter=True, name="library_dataset_dataset_association_id_fk" ), nullable=True, index=True ),#current version of dataset, if null, there is not a current version selected
+ Column( "library_dataset_dataset_association_id", Integer, ForeignKey( "library_dataset_dataset_association.id", use_alter=True, name="library_dataset_dataset_association_id_fk" ), nullable=True, index=True ), # current version of dataset, if null, there is not a current version selected
Column( "order_id", Integer ),
Column( "create_time", DateTime, default=now ),
Column( "update_time", DateTime, default=now, onupdate=now ),
- Column( "name", TrimmedString( 255 ), key="_name" ), #when not None/null this will supercede display in library (but not when imported into user's history?)
- Column( "info", TrimmedString( 255 ), key="_info" ), #when not None/null this will supercede display in library (but not when imported into user's history?)
+ Column( "name", TrimmedString( 255 ), key="_name" ), # when not None/null this will supercede display in library (but not when imported into user's history?)
+ Column( "info", TrimmedString( 255 ), key="_info" ), # when not None/null this will supercede display in library (but not when imported into user's history?)
Column( "deleted", Boolean, index=True, default=False ) )
-
-##mappers
+# mappers
mapper( Dataset, Dataset.table,
@@ -601,8 +633,7 @@ mapper( Dataset, Dataset.table,
primaryjoin=( Dataset.table.c.id == LibraryDatasetDatasetAssociation.table.c.dataset_id ) ),
active_library_associations=relation(
LibraryDatasetDatasetAssociation,
- primaryjoin=( ( Dataset.table.c.id == LibraryDatasetDatasetAssociation.table.c.dataset_id ) & ( LibraryDatasetDatasetAssociation.table.c.deleted == false() ) ) )
- ) )
+ primaryjoin=( ( Dataset.table.c.id == LibraryDatasetDatasetAssociation.table.c.dataset_id ) & ( LibraryDatasetDatasetAssociation.table.c.deleted == false() ) ) ) ) )
mapper( HistoryDatasetAssociation, HistoryDatasetAssociation.table,
@@ -625,13 +656,12 @@ mapper( HistoryDatasetAssociation, HistoryDatasetAssociation.table,
backref=backref( "parent", primaryjoin=( HistoryDatasetAssociation.table.c.parent_id == HistoryDatasetAssociation.table.c.id ), remote_side=[HistoryDatasetAssociation.table.c.id], uselist=False ) ),
visible_children=relation(
HistoryDatasetAssociation,
- primaryjoin=( ( HistoryDatasetAssociation.table.c.parent_id == HistoryDatasetAssociation.table.c.id ) & ( HistoryDatasetAssociation.table.c.visible == true() ) ) )
- ) )
+ primaryjoin=( ( HistoryDatasetAssociation.table.c.parent_id == HistoryDatasetAssociation.table.c.id ) & ( HistoryDatasetAssociation.table.c.visible == true() ) ) ) ) )
mapper( LibraryDatasetDatasetAssociation, LibraryDatasetDatasetAssociation.table,
properties=dict(
dataset=relation( Dataset ),
- library_dataset = relation( LibraryDataset,
+ library_dataset=relation( LibraryDataset,
primaryjoin=( LibraryDatasetDatasetAssociation.table.c.library_dataset_id == LibraryDataset.table.c.id ) ),
copied_to_library_dataset_dataset_associations=relation(
LibraryDatasetDatasetAssociation,
@@ -647,14 +677,12 @@ mapper( LibraryDatasetDatasetAssociation, LibraryDatasetDatasetAssociation.table
backref=backref( "parent", primaryjoin=( LibraryDatasetDatasetAssociation.table.c.parent_id == LibraryDatasetDatasetAssociation.table.c.id ), remote_side=[LibraryDatasetDatasetAssociation.table.c.id] ) ),
visible_children=relation(
LibraryDatasetDatasetAssociation,
- primaryjoin=( ( LibraryDatasetDatasetAssociation.table.c.parent_id == LibraryDatasetDatasetAssociation.table.c.id ) & ( LibraryDatasetDatasetAssociation.table.c.visible == true() ) ) )
- ) )
+ primaryjoin=( ( LibraryDatasetDatasetAssociation.table.c.parent_id == LibraryDatasetDatasetAssociation.table.c.id ) & ( LibraryDatasetDatasetAssociation.table.c.visible == true() ) ) ) ) )
mapper( LibraryDataset, LibraryDataset.table,
properties=dict(
library_dataset_dataset_association=relation( LibraryDatasetDatasetAssociation, primaryjoin=( LibraryDataset.table.c.library_dataset_dataset_association_id == LibraryDatasetDatasetAssociation.table.c.id ) ),
- expired_datasets = relation( LibraryDatasetDatasetAssociation, foreign_keys=[LibraryDataset.table.c.id,LibraryDataset.table.c.library_dataset_dataset_association_id ], primaryjoin=( ( LibraryDataset.table.c.id == LibraryDatasetDatasetAssociation.table.c.library_dataset_id ) & ( not_( LibraryDataset.table.c.library_dataset_dataset_association_id == LibraryDatasetDatasetAssociation.table.c.id ) ) ), viewonly=True, uselist=True )
- ) )
+ expired_datasets=relation( LibraryDatasetDatasetAssociation, foreign_keys=[LibraryDataset.table.c.id, LibraryDataset.table.c.library_dataset_dataset_association_id ], primaryjoin=( ( LibraryDataset.table.c.id == LibraryDatasetDatasetAssociation.table.c.library_dataset_id ) & ( not_( LibraryDataset.table.c.library_dataset_dataset_association_id == LibraryDatasetDatasetAssociation.table.c.id ) ) ), viewonly=True, uselist=True ) ) )
def __guess_dataset_by_filename( filename ):
@@ -662,12 +690,13 @@ def __guess_dataset_by_filename( filename ):
try:
fields = os.path.split( filename )
if fields:
- if fields[-1].startswith( 'dataset_' ) and fields[-1].endswith( '.dat' ): #dataset_%d.dat
+ if fields[-1].startswith( 'dataset_' ) and fields[-1].endswith( '.dat' ): # dataset_%d.dat
return Dataset.get( int( fields[-1][ len( 'dataset_' ): -len( '.dat' ) ] ) )
except:
- pass #some parsing error, we can't guess Dataset
+ pass # some parsing error, we can't guess Dataset
return None
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
log.debug( "Fixing a discrepancy concerning deleted shared history items." )
@@ -678,14 +707,14 @@ def upgrade(migrate_engine):
if not dataset_instance.deleted:
dataset.deleted = False
if dataset.file_size in [ None, 0 ]:
- dataset.set_size() #Restore filesize
+ dataset.set_size() # Restore filesize
affected_items += 1
break
context.flush()
log.debug( "%i items affected, and restored." % ( affected_items ) )
log.debug( "Time elapsed: %s" % ( time.time() - start_time ) )
- #fix share before hda
+ # fix share before hda
log.debug( "Fixing a discrepancy concerning cleaning up deleted history items shared before HDAs." )
dataset_by_filename = {}
changed_associations = 0
@@ -695,23 +724,23 @@ def upgrade(migrate_engine):
guessed_dataset = dataset_by_filename[ dataset.file_name ]
else:
guessed_dataset = __guess_dataset_by_filename( dataset.file_name )
- if guessed_dataset and dataset.file_name != guessed_dataset.file_name:#not os.path.samefile( dataset.file_name, guessed_dataset.file_name ):
+ if guessed_dataset and dataset.file_name != guessed_dataset.file_name: # not os.path.samefile( dataset.file_name, guessed_dataset.file_name ):
guessed_dataset = None
dataset_by_filename[ dataset.file_name ] = guessed_dataset
- if guessed_dataset is not None and guessed_dataset.id != dataset.id: #could we have a self referential dataset?
+ if guessed_dataset is not None and guessed_dataset.id != dataset.id: # could we have a self referential dataset?
for dataset_instance in dataset.history_associations + dataset.library_associations:
dataset_instance.dataset = guessed_dataset
changed_associations += 1
- #mark original Dataset as deleted and purged, it is no longer in use, but do not delete file_name contents
+ # mark original Dataset as deleted and purged, it is no longer in use, but do not delete file_name contents
dataset.deleted = True
dataset.external_filename = "Dataset was result of share before HDA, and has been replaced: %s mapped to Dataset %s" % ( dataset.external_filename, guessed_dataset.id )
- dataset.purged = True #we don't really purge the file here, but we mark it as purged, since this dataset is now defunct
+ dataset.purged = True # we don't really purge the file here, but we mark it as purged, since this dataset is now defunct
context.flush()
log.debug( "%i items affected, and restored." % ( changed_associations ) )
log.debug( "Time elapsed: %s" % ( time.time() - start_time ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
log.debug( "Downgrade is not possible." )
-
diff --git a/lib/galaxy/model/migrate/versions/0006_change_qual_datatype.py b/lib/galaxy/model/migrate/versions/0006_change_qual_datatype.py
index ff4521ca60f..3a84009d96e 100644
--- a/lib/galaxy/model/migrate/versions/0006_change_qual_datatype.py
+++ b/lib/galaxy/model/migrate/versions/0006_change_qual_datatype.py
@@ -2,10 +2,11 @@
This migration script changes certain values in the history_dataset_association.extension
column, specifically 'qual' is chaged to be 'qual454'.
"""
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-import sys, logging
+import logging
+import sys
+
+from sqlalchemy import Index, MetaData, Table
+from sqlalchemy.orm import scoped_session, sessionmaker
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
@@ -17,6 +18,7 @@ log.addHandler( handler )
metadata = MetaData()
+
def display_migration_details():
print "========================================"
print "This migration script changes certain values in the history_dataset_association.extension"
@@ -55,6 +57,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "Dropping index 'ix_hda_extension' to history_dataset_association table failed: %s" % ( str( e ) ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
pass
diff --git a/lib/galaxy/model/migrate/versions/0007_sharing_histories.py b/lib/galaxy/model/migrate/versions/0007_sharing_histories.py
index 6fadb323ae4..c83bd51ad4d 100644
--- a/lib/galaxy/model/migrate/versions/0007_sharing_histories.py
+++ b/lib/galaxy/model/migrate/versions/0007_sharing_histories.py
@@ -3,11 +3,11 @@ This migration script creates the new history_user_share_association table, and
a new boolean type column to the history table. This provides support for sharing
histories in the same way that workflows are shared.
"""
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-import sys, logging
+import logging
+import sys
+
+from sqlalchemy import Boolean, Column, ForeignKey, Integer, MetaData, Table
+from sqlalchemy.exc import NoSuchTableError
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
@@ -19,6 +19,7 @@ log.addHandler( handler )
metadata = MetaData()
+
def display_migration_details():
print "========================================"
print "This migration script creates the new history_user_share_association table, and adds"
@@ -29,8 +30,8 @@ def display_migration_details():
HistoryUserShareAssociation_table = Table( "history_user_share_association", metadata,
Column( "id", Integer, primary_key=True ),
Column( "history_id", Integer, ForeignKey( "history.id" ), index=True ),
- Column( "user_id", Integer, ForeignKey( "galaxy_user.id" ), index=True )
- )
+ Column( "user_id", Integer, ForeignKey( "galaxy_user.id" ), index=True ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
@@ -56,6 +57,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "Adding column 'importable' to history table failed: %s" % ( str( e ) ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
# Load existing tables
diff --git a/lib/galaxy/model/migrate/versions/0008_galaxy_forms.py b/lib/galaxy/model/migrate/versions/0008_galaxy_forms.py
index 9128370639b..4cc3e1cbbcf 100644
--- a/lib/galaxy/model/migrate/versions/0008_galaxy_forms.py
+++ b/lib/galaxy/model/migrate/versions/0008_galaxy_forms.py
@@ -9,16 +9,17 @@ This migration script adds the following new tables for supporting Galaxy forms:
7) sample_state
8) sample_event
"""
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from sqlalchemy.exc import *
-from migrate import *
-from migrate.changeset import *
-
import datetime
-now = datetime.datetime.utcnow
+import logging
+import sys
-import sys, logging
+from migrate import ForeignKeyConstraint
+from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, MetaData, Table, TEXT
+
+# Need our custom types, but don't import anything else from model
+from galaxy.model.custom_types import JSONType, TrimmedString
+
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler( sys.stdout )
@@ -26,12 +27,9 @@ format = "%(name)s %(levelname)s %(asctime)s %(message)s"
formatter = logging.Formatter( format )
handler.setFormatter( formatter )
log.addHandler( handler )
-
-# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
-
metadata = MetaData()
+
def display_migration_details():
print "========================================"
print "This migration script adds the following new tables for supporting Galaxy forms:"
@@ -115,8 +113,8 @@ SampleEvent_table = Table('sample_event', metadata,
Column( "sample_state_id", Integer, ForeignKey( "sample_state.id" ), index=True ),
Column( "comment", TEXT ) )
+
def upgrade(migrate_engine):
- db_session = scoped_session( sessionmaker( bind=migrate_engine, autoflush=False, autocommit=True ) )
metadata.bind = migrate_engine
display_migration_details()
# Load existing tables
@@ -166,8 +164,8 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "Creating sample_event table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
- db_session = scoped_session( sessionmaker( bind=migrate_engine, autoflush=False, autocommit=True ) )
metadata.bind = migrate_engine
# Load existing tables
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0009_request_table.py b/lib/galaxy/model/migrate/versions/0009_request_table.py
index d89a28bb060..c51375f1c3c 100644
--- a/lib/galaxy/model/migrate/versions/0009_request_table.py
+++ b/lib/galaxy/model/migrate/versions/0009_request_table.py
@@ -3,13 +3,12 @@ This migration script adds a new column to 2 tables:
1) a new boolean type column named 'submitted' to the 'request' table
2) a new string type column named 'bar_code' to the 'sample' table
"""
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-import sys, logging
-from galaxy.model.custom_types import *
-from sqlalchemy.exc import *
+import logging
+import sys
+
+from sqlalchemy import Boolean, Column, MetaData, Table
+
+from galaxy.model.custom_types import TrimmedString
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
@@ -21,6 +20,7 @@ log.addHandler( handler )
metadata = MetaData()
+
def display_migration_details():
print "========================================"
print "This migration script adds a new column to 2 tables:"
@@ -28,6 +28,7 @@ def display_migration_details():
print "2) a new string type column named 'bar_code' to the 'sample' table"
print "========================================"
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
display_migration_details()
@@ -40,7 +41,6 @@ def upgrade(migrate_engine):
try:
col = Column( 'submitted', Boolean, default=False )
col.create( Request_table)
- #col.create( Request_table, index_name='ix_request_submitted')
assert col is Request_table.c.submitted
except Exception, e:
log.debug( "Adding column 'submitted' to request table failed: %s" % ( str( e ) ) )
@@ -54,6 +54,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "Adding column 'bar_code' to sample table failed: %s" % ( str( e ) ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
pass
diff --git a/lib/galaxy/model/migrate/versions/0010_hda_display_at_authz_table.py b/lib/galaxy/model/migrate/versions/0010_hda_display_at_authz_table.py
index 9234e73185f..453dbfd36e0 100644
--- a/lib/galaxy/model/migrate/versions/0010_hda_display_at_authz_table.py
+++ b/lib/galaxy/model/migrate/versions/0010_hda_display_at_authz_table.py
@@ -8,16 +8,16 @@ history_dataset_association_display_at_authorization table failed: (Operational
(1059, "Identifier name 'ix_history_dataset_association_display_at_authorization_update_time'
is too long
"""
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from sqlalchemy.exc import *
-from migrate import *
-from migrate.changeset import *
-
import datetime
-now = datetime.datetime.utcnow
+import logging
+import sys
-import sys, logging
+from sqlalchemy import Column, DateTime, ForeignKey, Integer, MetaData, Table
+
+# Need our custom types, but don't import anything else from model
+from galaxy.model.custom_types import TrimmedString
+
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler( sys.stdout )
@@ -25,12 +25,9 @@ format = "%(name)s %(levelname)s %(asctime)s %(message)s"
formatter = logging.Formatter( format )
handler.setFormatter( formatter )
log.addHandler( handler )
-
-# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
-
metadata = MetaData()
+
def display_migration_details():
print "========================================"
print "This migration script adds the history_dataset_association_display_at_authorization table, which"
@@ -50,6 +47,7 @@ HistoryDatasetAssociationDisplayAtAuthorization_table = Table( "history_dataset_
Column( "user_id", Integer, ForeignKey( "galaxy_user.id" ), index=True ),
Column( "site", TrimmedString( 255 ) ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
display_migration_details()
@@ -60,6 +58,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "Creating history_dataset_association_display_at_authorization table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
# Load existing tables
diff --git a/lib/galaxy/model/migrate/versions/0011_v0010_mysql_index_fix.py b/lib/galaxy/model/migrate/versions/0011_v0010_mysql_index_fix.py
index 580f2b1ddf9..964c9685caa 100644
--- a/lib/galaxy/model/migrate/versions/0011_v0010_mysql_index_fix.py
+++ b/lib/galaxy/model/migrate/versions/0011_v0010_mysql_index_fix.py
@@ -3,16 +3,16 @@ This script fixes a problem introduced in 0010_hda_display_at_atuhz_table.py. M
name length limit and thus the index "ix_hdadaa_history_dataset_association_id" has to be
manually created.
"""
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from sqlalchemy.exc import *
-from migrate import *
-from migrate.changeset import *
-
import datetime
-now = datetime.datetime.utcnow
+import logging
+import sys
-import sys, logging
+from sqlalchemy import Column, DateTime, ForeignKey, Index, Integer, MetaData, Table
+
+# Need our custom types, but don't import anything else from model
+from galaxy.model.custom_types import TrimmedString
+
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler( sys.stdout )
@@ -20,12 +20,9 @@ format = "%(name)s %(levelname)s %(asctime)s %(message)s"
formatter = logging.Formatter( format )
handler.setFormatter( formatter )
log.addHandler( handler )
-
-# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
-
metadata = MetaData()
+
def display_migration_details():
print "========================================"
print "This script fixes a problem introduced in the previous migration script ( 9->10 ). MySQL"
@@ -41,6 +38,7 @@ HistoryDatasetAssociationDisplayAtAuthorization_table = Table( "history_dataset_
Column( "user_id", Integer, ForeignKey( "galaxy_user.id" ), index=True ),
Column( "site", TrimmedString( 255 ) ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
display_migration_details()
@@ -53,6 +51,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "Adding index 'ix_hdadaa_history_dataset_association_id' to table 'history_dataset_association_display_at_authorization' table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
if migrate_engine.name == 'mysql':
diff --git a/lib/galaxy/model/migrate/versions/0012_user_address.py b/lib/galaxy/model/migrate/versions/0012_user_address.py
index 4f0250a0ec1..31095ee6850 100644
--- a/lib/galaxy/model/migrate/versions/0012_user_address.py
+++ b/lib/galaxy/model/migrate/versions/0012_user_address.py
@@ -4,17 +4,17 @@ a user can select from a list of his addresses to associate with the request. T
drops the request.submitted column which was boolean and replaces it with a request.state column
which is a string, allowing for more flexibility with request states.
"""
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from sqlalchemy.exc import *
-from galaxy.model.custom_types import *
-from migrate import *
-from migrate.changeset import *
import datetime
-now = datetime.datetime.utcnow
-import sys, logging
-# Need our custom types, but don't import anything else from model
+import logging
+import sys
+from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, MetaData, Table, TEXT
+from sqlalchemy.exc import NoSuchTableError
+
+# Need our custom types, but don't import anything else from model
+from galaxy.model.custom_types import TrimmedString
+
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler( sys.stdout )
@@ -22,9 +22,9 @@ format = "%(name)s %(levelname)s %(asctime)s %(message)s"
formatter = logging.Formatter( format )
handler.setFormatter( formatter )
log.addHandler( handler )
-
metadata = MetaData()
+
def display_migration_details():
print "========================================"
print "This script adds a new user_address table that is currently only used with sample requests, where"
@@ -50,8 +50,8 @@ UserAddress_table = Table( "user_address", metadata,
Column( "deleted", Boolean, index=True, default=False ),
Column( "purged", Boolean, index=True, default=False ) )
+
def upgrade(migrate_engine):
- #raise Exception
metadata.bind = migrate_engine
display_migration_details()
# Load existing tables
@@ -84,12 +84,13 @@ def upgrade(migrate_engine):
log.debug( "Failed loading table request" )
if Request_table is not None:
if migrate_engine.name != 'sqlite':
- #DBTODO drop from table doesn't work in sqlite w/ sqlalchemy-migrate .6+
+ # DBTODO drop from table doesn't work in sqlite w/ sqlalchemy-migrate .6+
Request_table.c.submitted.drop()
col = Column( "state", TrimmedString( 255 ), index=True )
col.create( Request_table, index_name='ix_request_state')
assert col is Request_table.c.state
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
pass
diff --git a/lib/galaxy/model/migrate/versions/0013_change_lib_item_templates_to_forms.py b/lib/galaxy/model/migrate/versions/0013_change_lib_item_templates_to_forms.py
index 1b9a127e4d6..bfefeb07ba6 100644
--- a/lib/galaxy/model/migrate/versions/0013_change_lib_item_templates_to_forms.py
+++ b/lib/galaxy/model/migrate/versions/0013_change_lib_item_templates_to_forms.py
@@ -15,12 +15,11 @@ If using mysql, this script will throw an (OperationalError) exception due to a
the library_dataset_dataset_info_association table, which is OK because the script creates an index
with a shortened name.
"""
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from sqlalchemy.exc import *
-from migrate import *
-from migrate.changeset import *
-import sys, logging
+import logging
+import sys
+
+from sqlalchemy import Column, ForeignKey, Index, Integer, MetaData, Table
+from sqlalchemy.exc import NoSuchTableError
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
@@ -29,9 +28,9 @@ format = "%(name)s %(levelname)s %(asctime)s %(message)s"
formatter = logging.Formatter( format )
handler.setFormatter( formatter )
log.addHandler( handler )
-
metadata = MetaData()
+
def display_migration_details():
print "========================================"
print "This migration script eliminates all of the tables that were used for the 1st version of the"
@@ -68,24 +67,12 @@ LibraryDatasetDatasetInfoAssociation_table = Table( 'library_dataset_dataset_inf
Column( "form_definition_id", Integer, ForeignKey( "form_definition.id" ), index=True ),
Column( "form_values_id", Integer, ForeignKey( "form_values.id" ), index=True ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
display_migration_details()
# Load existing tables
metadata.reflect()
- if migrate_engine.name == 'postgres':
- # http://blog.pythonisito.com/2008/01/cascading-drop-table-with-sqlalchemy.html
- from sqlalchemy.databases import postgres
- class PGCascadeSchemaDropper(postgres.PGSchemaDropper):
- def visit_table(self, table):
- for column in table.columns:
- if column.default is not None:
- self.traverse_single(column.default)
- self.append("\nDROP TABLE " +
- self.preparer.format_table(table) +
- " CASCADE")
- self.execute()
- postgres.dialect.schemadropper = PGCascadeSchemaDropper
# Drop all of the original library_item_info tables
# NOTE: all existing library item into template data is eliminated here via table drops
try:
@@ -251,6 +238,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "Adding index 'ix_lddaia_ldda_id' to table 'library_dataset_dataset_info_association' table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
log.debug( "Downgrade is not possible." )
diff --git a/lib/galaxy/model/migrate/versions/0014_pages.py b/lib/galaxy/model/migrate/versions/0014_pages.py
index c7fb1db2295..42fa44f56af 100644
--- a/lib/galaxy/model/migrate/versions/0014_pages.py
+++ b/lib/galaxy/model/migrate/versions/0014_pages.py
@@ -3,17 +3,13 @@ Migration script to add support for "Pages".
1) Creates Page and PageRevision tables
2) Adds username column to User table
"""
-
-from sqlalchemy import *
-from migrate import *
-from migrate.changeset import *
-
import datetime
-now = datetime.datetime.utcnow
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, DateTime, ForeignKey, Index, Integer, MetaData, String, Table, TEXT
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
Page_table = Table( "page", metadata,
@@ -24,8 +20,7 @@ Page_table = Table( "page", metadata,
Column( "latest_revision_id", Integer,
ForeignKey( "page_revision.id", use_alter=True, name='page_latest_revision_id_fk' ), index=True ),
Column( "title", TEXT ),
- Column( "slug", TEXT, unique=True, index=True ),
- )
+ Column( "slug", TEXT, unique=True, index=True ) )
PageRevision_table = Table( "page_revision", metadata,
Column( "id", Integer, primary_key=True ),
@@ -33,8 +28,8 @@ PageRevision_table = Table( "page_revision", metadata,
Column( "update_time", DateTime, default=now, onupdate=now ),
Column( "page_id", Integer, ForeignKey( "page.id" ), index=True, nullable=False ),
Column( "title", TEXT ),
- Column( "content", TEXT )
- )
+ Column( "content", TEXT ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
@@ -51,7 +46,7 @@ def upgrade(migrate_engine):
Page_table.create()
if migrate_engine.name == 'mysql':
# Create slug index manually afterward.
- i = Index( "ix_page_slug", Page_table.c.slug, mysql_length = 200)
+ i = Index( "ix_page_slug", Page_table.c.slug, mysql_length=200)
i.create()
except Exception, ex:
log.debug(ex)
@@ -67,6 +62,7 @@ def upgrade(migrate_engine):
col.create( User_table, index_name='ix_user_username', unique_name='username' )
assert col is User_table.c.username
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0015_tagging.py b/lib/galaxy/model/migrate/versions/0015_tagging.py
index 10f682ec0de..6eafbf9c57e 100644
--- a/lib/galaxy/model/migrate/versions/0015_tagging.py
+++ b/lib/galaxy/model/migrate/versions/0015_tagging.py
@@ -9,21 +9,20 @@ history_dataset_association_tag_association table failed: (OperationalError)
(1059, "Identifier name 'ix_history_dataset_association_tag_association_history_dataset_association_id'
is too long)
"""
-
-from sqlalchemy import *
-from migrate import *
-
import datetime
-now = datetime.datetime.utcnow
+import logging
+
+from migrate import UniqueConstraint
+from sqlalchemy import Column, ForeignKey, Integer, MetaData, Table
# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
+from galaxy.model.custom_types import TrimmedString
-import logging
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
-
metadata = MetaData()
+
def display_migration_details():
print ""
print "This migration script adds the tables necessary to support tagging of histories,"
@@ -66,6 +65,7 @@ HistoryDatasetAssociationTagAssociation_table = Table( "history_dataset_associat
Column( "value", TrimmedString(255), index=True),
Column( "user_value", TrimmedString(255), index=True) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
display_migration_details()
@@ -91,6 +91,7 @@ def upgrade(migrate_engine):
print str(e)
log.debug( "Creating history_dataset_association_tag_association table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0016_v0015_mysql_index_fix.py b/lib/galaxy/model/migrate/versions/0016_v0015_mysql_index_fix.py
index 1b9b9deec8d..742ce73571d 100644
--- a/lib/galaxy/model/migrate/versions/0016_v0015_mysql_index_fix.py
+++ b/lib/galaxy/model/migrate/versions/0016_v0015_mysql_index_fix.py
@@ -3,21 +3,19 @@ This script fixes a problem introduced in 0015_tagging.py. MySQL has a name leng
limit and thus the index "ix_hda_ta_history_dataset_association_id" has to be
manually created.
"""
-
-from sqlalchemy import *
-from migrate import *
-
import datetime
-now = datetime.datetime.utcnow
+import logging
+
+from sqlalchemy import Column, ForeignKey, Index, Integer, MetaData, Table
# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
+from galaxy.model.custom_types import TrimmedString
-import logging
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
-
metadata = MetaData()
+
def display_migration_details():
print ""
print "This script fixes a problem introduced in 0015_tagging.py. MySQL has a"
@@ -31,6 +29,7 @@ HistoryDatasetAssociationTagAssociation_table = Table( "history_dataset_associat
Column( "value", TrimmedString(255), index=True),
Column( "user_value", TrimmedString(255), index=True) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
display_migration_details()
@@ -42,6 +41,7 @@ def upgrade(migrate_engine):
print str(e)
log.debug( "Adding index 'ix_hdata_history_dataset_association_id' to table 'history_dataset_association_tag_association' table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0017_library_item_indexes.py b/lib/galaxy/model/migrate/versions/0017_library_item_indexes.py
index a1733fcd17d..4e6174dcd43 100644
--- a/lib/galaxy/model/migrate/versions/0017_library_item_indexes.py
+++ b/lib/galaxy/model/migrate/versions/0017_library_item_indexes.py
@@ -2,10 +2,10 @@
This script adds 3 indexes to table columns: library_folder.name,
library_dataset.name, library_dataset_dataset_association.name.
"""
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-import sys, logging
+import logging
+import sys
+
+from sqlalchemy import Index, MetaData, Table
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
@@ -14,15 +14,16 @@ format = "%(name)s %(levelname)s %(asctime)s %(message)s"
formatter = logging.Formatter( format )
handler.setFormatter( formatter )
log.addHandler( handler )
-
metadata = MetaData()
+
def display_migration_details():
print "========================================"
print "This script adds 3 indexes to table columns: library_folder.name,"
print "library_dataset.name, library_dataset_dataset_association.name."
print "========================================"
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
LibraryFolder_table = Table( "library_folder", metadata, autoload=True )
@@ -49,6 +50,8 @@ def upgrade(migrate_engine):
i.create()
except Exception, e:
log.debug( "Adding index 'ix_library_dataset_name' to library_dataset table failed: %s" % ( str( e ) ) )
+
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
log.debug( "Downgrade is not possible." )
diff --git a/lib/galaxy/model/migrate/versions/0018_ordered_tags_and_page_tags.py b/lib/galaxy/model/migrate/versions/0018_ordered_tags_and_page_tags.py
index 6ffa9351ab9..062c77b8283 100644
--- a/lib/galaxy/model/migrate/versions/0018_ordered_tags_and_page_tags.py
+++ b/lib/galaxy/model/migrate/versions/0018_ordered_tags_and_page_tags.py
@@ -2,24 +2,20 @@
This migration script provides support for (a) ordering tags by recency and
(b) tagging pages. This script deletes all existing tags.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from sqlalchemy.exc import *
-from migrate import *
-import migrate.changeset
-
import datetime
-now = datetime.datetime.utcnow
+import logging
+
+from sqlalchemy import Column, ForeignKey, Index, Integer, MetaData, Table
+from sqlalchemy.exc import OperationalError
# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
+from galaxy.model.custom_types import TrimmedString
-import logging
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
-
metadata = MetaData()
+
def display_migration_details():
print ""
print "This migration script provides support for (a) ordering tags by recency and"
@@ -57,6 +53,7 @@ PageTagAssociation_table = Table( "page_tag_association", metadata,
Column( "value", TrimmedString(255), index=True),
Column( "user_value", TrimmedString(255), index=True) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
display_migration_details()
@@ -104,6 +101,7 @@ def upgrade(migrate_engine):
print str(e)
log.debug( "Creating page_tag_association table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0019_request_library_folder.py b/lib/galaxy/model/migrate/versions/0019_request_library_folder.py
index 2e596c30ca8..b60b265ee13 100644
--- a/lib/galaxy/model/migrate/versions/0019_request_library_folder.py
+++ b/lib/galaxy/model/migrate/versions/0019_request_library_folder.py
@@ -1,14 +1,15 @@
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from sqlalchemy.exc import *
-from migrate import *
-from migrate.changeset import *
import datetime
-now = datetime.datetime.utcnow
-import sys, logging
-# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
+import logging
+import sys
+from migrate import ForeignKeyConstraint
+from sqlalchemy import Column, Integer, MetaData, Table
+from sqlalchemy.exc import NoSuchTableError
+
+# Need our custom types, but don't import anything else from model
+from galaxy.model.custom_types import JSONType, TrimmedString
+
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler( sys.stdout )
@@ -16,9 +17,9 @@ format = "%(name)s %(levelname)s %(asctime)s %(message)s"
formatter = logging.Formatter( format )
handler.setFormatter( formatter )
log.addHandler( handler )
-
metadata = MetaData()
+
def display_migration_details():
print "========================================"
print """This script creates a request.folder_id column which is a foreign
@@ -26,6 +27,7 @@ key to the library_folder table. This also adds a 'type' and 'layout' column
to the form_definition table."""
print "========================================"
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
display_migration_details()
diff --git a/lib/galaxy/model/migrate/versions/0020_library_upload_job.py b/lib/galaxy/model/migrate/versions/0020_library_upload_job.py
index 22fd33b2d8b..a68950b569a 100644
--- a/lib/galaxy/model/migrate/versions/0020_library_upload_job.py
+++ b/lib/galaxy/model/migrate/versions/0020_library_upload_job.py
@@ -1,14 +1,12 @@
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from sqlalchemy.exc import *
-from migrate import *
-from migrate.changeset import *
import datetime
-now = datetime.datetime.utcnow
-import sys, logging
-# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
+import logging
+import sys
+from migrate import ForeignKeyConstraint
+from sqlalchemy import Column, ForeignKey, Index, Integer, MetaData, String, Table
+from sqlalchemy.exc import NoSuchTableError
+
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler( sys.stdout )
@@ -19,6 +17,7 @@ log.addHandler( handler )
metadata = MetaData()
+
def display_migration_details():
print ""
print "========================================"
@@ -35,6 +34,7 @@ JobToOutputLibraryDatasetAssociation_table = Table( "job_to_output_library_datas
Column( "ldda_id", Integer, ForeignKey( "library_dataset_dataset_association.id" ), index=True ),
Column( "name", String(255) ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
display_migration_details()
@@ -66,7 +66,7 @@ def upgrade(migrate_engine):
log.debug( "Failed loading table library_folder" )
# Add 1 foreign key constraint to the job table
if migrate_engine.name != 'sqlite':
- #Sqlite can't alter-table-add-foreign-key
+ # Sqlite can't alter-table-add-foreign-key
if Job_table is not None and LibraryFolder_table is not None:
try:
cons = ForeignKeyConstraint( [Job_table.c.library_folder_id],
@@ -89,6 +89,7 @@ def upgrade(migrate_engine):
print str(e)
log.debug( "Adding index 'ix_dataset_state' to dataset table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0021_user_prefs.py b/lib/galaxy/model/migrate/versions/0021_user_prefs.py
index 192c89d04ea..8f4a34e2ae6 100644
--- a/lib/galaxy/model/migrate/versions/0021_user_prefs.py
+++ b/lib/galaxy/model/migrate/versions/0021_user_prefs.py
@@ -1,18 +1,16 @@
"""
This migration script adds a user preferences table to Galaxy.
"""
-
-from sqlalchemy import *
-from migrate import *
-
import datetime
-now = datetime.datetime.utcnow
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, ForeignKey, Integer, MetaData, Table, Unicode
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
+
def display_migration_details():
print ""
print "This migration script adds a user preferences table to Galaxy."
@@ -27,6 +25,7 @@ UserPreference_table = Table( "user_preference", metadata,
Column( "name", Unicode( 255 ), index=True),
Column( "value", Unicode( 1024 ) ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
display_migration_details()
@@ -37,6 +36,7 @@ def upgrade(migrate_engine):
print str(e)
log.debug( "Creating user_preference table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0022_visualization_tables.py b/lib/galaxy/model/migrate/versions/0022_visualization_tables.py
index ebd0e78ba1c..47191e43dad 100644
--- a/lib/galaxy/model/migrate/versions/0022_visualization_tables.py
+++ b/lib/galaxy/model/migrate/versions/0022_visualization_tables.py
@@ -2,17 +2,13 @@
Migration script to add support for storing visualizations.
1) Creates Visualization and VisualizationRevision tables
"""
-
-from sqlalchemy import *
-from migrate import *
-from migrate.changeset import *
-
import datetime
-now = datetime.datetime.utcnow
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, DateTime, ForeignKey, Integer, MetaData, Table, TEXT
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
Visualization_table = Table( "visualization", metadata,
@@ -23,8 +19,7 @@ Visualization_table = Table( "visualization", metadata,
Column( "latest_revision_id", Integer,
ForeignKey( "visualization_revision.id", use_alter=True, name='visualization_latest_revision_id_fk' ), index=True ),
Column( "title", TEXT ),
- Column( "type", TEXT )
- )
+ Column( "type", TEXT ) )
VisualizationRevision_table = Table( "visualization_revision", metadata,
Column( "id", Integer, primary_key=True ),
@@ -32,8 +27,8 @@ VisualizationRevision_table = Table( "visualization_revision", metadata,
Column( "update_time", DateTime, default=now, onupdate=now ),
Column( "visualization_id", Integer, ForeignKey( "visualization.id" ), index=True, nullable=False ),
Column( "title", TEXT ),
- Column( "config", TEXT )
- )
+ Column( "config", TEXT ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
@@ -54,4 +49,3 @@ def downgrade(migrate_engine):
metadata.reflect()
Visualization_table.drop()
VisualizationRevision_table.drop()
-
diff --git a/lib/galaxy/model/migrate/versions/0023_page_published_and_deleted_columns.py b/lib/galaxy/model/migrate/versions/0023_page_published_and_deleted_columns.py
index 8de454e4a82..f5edaa28cb5 100644
--- a/lib/galaxy/model/migrate/versions/0023_page_published_and_deleted_columns.py
+++ b/lib/galaxy/model/migrate/versions/0023_page_published_and_deleted_columns.py
@@ -2,16 +2,14 @@
Migration script to add columns for tracking whether pages are deleted and
publicly accessible.
"""
-
-from sqlalchemy import *
-from migrate import *
-from migrate.changeset import *
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Boolean, Column, MetaData, Table
+
+log = logging.getLogger( __name__ )
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
@@ -21,13 +19,14 @@ def upgrade(migrate_engine):
Page_table = Table( "page", metadata, autoload=True )
c = Column( "published", Boolean, index=True, default=False )
- c.create( Page_table, index_name = 'ix_page_published' )
+ c.create( Page_table, index_name='ix_page_published' )
assert c is Page_table.c.published
c = Column( "deleted", Boolean, index=True, default=False )
c.create( Page_table, index_name='ix_page_deleted')
assert c is Page_table.c.deleted
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0024_page_slug_unique_constraint.py b/lib/galaxy/model/migrate/versions/0024_page_slug_unique_constraint.py
index 861d2a3e45d..272e6b29f65 100644
--- a/lib/galaxy/model/migrate/versions/0024_page_slug_unique_constraint.py
+++ b/lib/galaxy/model/migrate/versions/0024_page_slug_unique_constraint.py
@@ -2,19 +2,16 @@
Remove unique constraint from page slugs to allow creating a page with
the same slug as a deleted page.
"""
-
-from sqlalchemy import *
-from migrate import *
-from migrate.changeset import *
-
import datetime
-now = datetime.datetime.utcnow
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Index, MetaData, Table
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -38,8 +35,7 @@ def upgrade(migrate_engine):
Page_table.c.slug.alter( unique=False )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
- #Page_table = Table( "page", metadata, autoload=True )
- #Page_table.c.slug.alter( unique=True )
diff --git a/lib/galaxy/model/migrate/versions/0025_user_info.py b/lib/galaxy/model/migrate/versions/0025_user_info.py
index 633e6bdc1da..c687e06ed67 100644
--- a/lib/galaxy/model/migrate/versions/0025_user_info.py
+++ b/lib/galaxy/model/migrate/versions/0025_user_info.py
@@ -1,16 +1,15 @@
"""
This script adds a foreign key to the form_values table in the galaxy_user table
"""
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
import datetime
-now = datetime.datetime.utcnow
-import sys, logging
-# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
+import logging
+import sys
+from migrate import ForeignKeyConstraint
+from sqlalchemy import Column, Integer, MetaData, Table
+from sqlalchemy.exc import NoSuchTableError
+
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler( sys.stdout )
@@ -21,10 +20,13 @@ log.addHandler( handler )
metadata = MetaData()
+
def display_migration_details():
print "========================================"
print "This script adds a foreign key to the form_values table in the galaxy_user table"
print "========================================"
+
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
display_migration_details()
@@ -58,6 +60,8 @@ def upgrade(migrate_engine):
cons.create()
except Exception, e:
log.debug( "Adding foreign key constraint 'user_form_values_id_fk' to table 'galaxy_user' failed: %s" % ( str( e ) ) )
+
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
pass
diff --git a/lib/galaxy/model/migrate/versions/0026_cloud_tables.py b/lib/galaxy/model/migrate/versions/0026_cloud_tables.py
index 8c44aa2d1f6..816c1c00ef9 100644
--- a/lib/galaxy/model/migrate/versions/0026_cloud_tables.py
+++ b/lib/galaxy/model/migrate/versions/0026_cloud_tables.py
@@ -1,17 +1,13 @@
-from sqlalchemy import *
-from migrate import *
-
import datetime
-now = datetime.datetime.utcnow
-
-# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, MetaData, Table, TEXT
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
+
def display_migration_details():
print
print "========================================"
@@ -127,6 +123,7 @@ CloudProvider_table = Table( "cloud_provider", metadata,
Column( "path", TEXT ),
Column( "deleted", Boolean, default=False ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
display_migration_details()
@@ -143,6 +140,7 @@ def upgrade(migrate_engine):
CloudStore_table.create()
CloudSnapshot_table.create()
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0027_request_events.py b/lib/galaxy/model/migrate/versions/0027_request_events.py
index dc683f6e337..5ab18f00b75 100644
--- a/lib/galaxy/model/migrate/versions/0027_request_events.py
+++ b/lib/galaxy/model/migrate/versions/0027_request_events.py
@@ -2,21 +2,17 @@
This migration script adds the request_event table and
removes the state field in the request table
"""
-# Need our custom types, but don't import anything else from model
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from sqlalchemy.exc import *
-from migrate import *
-from migrate.changeset import *
-
import datetime
-import sys
import logging
+import sys
-now = datetime.datetime.utcnow
+from sqlalchemy import Column, DateTime, ForeignKey, Integer, MetaData, Table, TEXT
+from sqlalchemy.exc import NoSuchTableError
+# Need our custom types, but don't import anything else from model
from galaxy.model.custom_types import TrimmedString
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler( sys.stdout )
@@ -24,10 +20,9 @@ format = "%(name)s %(levelname)s %(asctime)s %(message)s"
formatter = logging.Formatter( format )
handler.setFormatter( formatter )
log.addHandler( handler )
-
-
metadata = MetaData()
+
def display_migration_details():
print "========================================"
print "This migration script adds the request_event table and"
@@ -40,23 +35,26 @@ RequestEvent_table = Table('request_event', metadata,
Column( "create_time", DateTime, default=now ),
Column( "update_time", DateTime, default=now, onupdate=now ),
Column( "request_id", Integer, ForeignKey( "request.id" ), index=True ),
- Column( "state", TrimmedString( 255 ), index=True ),
+ Column( "state", TrimmedString( 255 ), index=True ),
Column( "comment", TEXT ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
display_migration_details()
+
def localtimestamp():
- if migrate_engine.name == 'postgresql' or migrate_engine.name == 'mysql':
- return "LOCALTIMESTAMP"
- elif migrate_engine.name == 'sqlite':
- return "current_date || ' ' || current_time"
- else:
- raise Exception( 'Unable to convert data for unknown database type: %s' % migrate_engine.name )
+ if migrate_engine.name in ['mysql', 'postgres', 'postgresql']:
+ return "LOCALTIMESTAMP"
+ elif migrate_engine.name == 'sqlite':
+ return "current_date || ' ' || current_time"
+ else:
+ raise Exception( 'Unable to convert data for unknown database type: %s' % migrate_engine.name )
+
def nextval( table, col='id' ):
- if migrate_engine.name == 'postgresql':
+ if migrate_engine.name in ['postgres', 'postgresql']:
return "nextval('%s_%s_seq')" % ( table, col )
- elif migrate_engine.name == 'mysql' or migrate_engine.name == 'sqlite':
+ elif migrate_engine.name in ['mysql', 'sqlite']:
return "null"
else:
raise Exception( 'Unable to convert data for unknown database type: %s' % migrate_engine.name )
@@ -94,6 +92,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "Deleting column 'state' to request table failed: %s" % ( str( e ) ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
pass
diff --git a/lib/galaxy/model/migrate/versions/0028_external_metadata_file_override.py b/lib/galaxy/model/migrate/versions/0028_external_metadata_file_override.py
index fbcce08f7d2..096a4a4c786 100644
--- a/lib/galaxy/model/migrate/versions/0028_external_metadata_file_override.py
+++ b/lib/galaxy/model/migrate/versions/0028_external_metadata_file_override.py
@@ -3,16 +3,13 @@ This script adds the filename_override_metadata column to the JobExternalOutputM
allowing existing metadata files to be written when using external metadata and a cluster
set up with read-only access to database/files
"""
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
import datetime
-now = datetime.datetime.utcnow
-import sys, logging
-# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
+import logging
+import sys
+from sqlalchemy import Column, MetaData, String, Table
+
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler( sys.stdout )
@@ -23,6 +20,7 @@ log.addHandler( handler )
metadata = MetaData()
+
def display_migration_details():
print "========================================"
print "This script adds the filename_override_metadata column to the JobExternalOutputMetadata table,"
@@ -30,6 +28,7 @@ def display_migration_details():
print "set up with read-only access to database/files"
print "========================================"
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
display_migration_details()
@@ -43,6 +42,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "Adding column 'filename_override_metadata' to job_external_output_metadata table failed: %s" % ( str( e ) ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
pass
diff --git a/lib/galaxy/model/migrate/versions/0029_user_actions.py b/lib/galaxy/model/migrate/versions/0029_user_actions.py
index 3504fb407fb..49d8a8143ab 100644
--- a/lib/galaxy/model/migrate/versions/0029_user_actions.py
+++ b/lib/galaxy/model/migrate/versions/0029_user_actions.py
@@ -1,18 +1,16 @@
"""
This migration script adds a user actions table to Galaxy.
"""
-
-from sqlalchemy import *
-from migrate import *
-
import datetime
-now = datetime.datetime.utcnow
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, DateTime, ForeignKey, Integer, MetaData, Table, Unicode
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
+
def display_migration_details():
print ""
print "This migration script adds a user actions table to Galaxy."
@@ -29,6 +27,7 @@ UserAction_table = Table( "user_action", metadata,
Column( "context", Unicode( 512 ) ),
Column( "params", Unicode( 1024 ) ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
display_migration_details()
@@ -39,6 +38,7 @@ def upgrade(migrate_engine):
print str(e)
log.debug( "Creating user_action table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0030_history_slug_column.py b/lib/galaxy/model/migrate/versions/0030_history_slug_column.py
index 3486df23c4d..5e180c82ef3 100644
--- a/lib/galaxy/model/migrate/versions/0030_history_slug_column.py
+++ b/lib/galaxy/model/migrate/versions/0030_history_slug_column.py
@@ -1,16 +1,14 @@
"""
Migration script to add column for a history slug.
"""
-
-from sqlalchemy import *
-from migrate import *
-from migrate.changeset import *
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, Index, MetaData, Table, TEXT
+
+log = logging.getLogger( __name__ )
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
@@ -19,7 +17,6 @@ def upgrade(migrate_engine):
History_table = Table( "history", metadata, autoload=True )
-
# Mysql needs manual index creation because of max length index.
if migrate_engine.name != 'mysql':
# Create slug column.
@@ -28,20 +25,11 @@ def upgrade(migrate_engine):
else:
c = Column( "slug", TEXT )
c.create( History_table , index_name='')
- i = Index( "ix_history_slug", History_table.c.slug, mysql_length = 200)
+ i = Index( "ix_history_slug", History_table.c.slug, mysql_length=200)
i.create()
assert c is History_table.c.slug
-
- ## Create slug index.
- #try:
- #i = Index( "ix_history_slug", History_table.c.slug )
- #i.create()
- #except:
- ## Mysql doesn't have a named index, but alter should work
- #History_table.c.slug.alter( unique=False )
-
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0031_community_and_workflow_tags.py b/lib/galaxy/model/migrate/versions/0031_community_and_workflow_tags.py
index 1ed576e1ec9..f483919bb02 100644
--- a/lib/galaxy/model/migrate/versions/0031_community_and_workflow_tags.py
+++ b/lib/galaxy/model/migrate/versions/0031_community_and_workflow_tags.py
@@ -5,15 +5,11 @@ Migration script to (a) add and populate necessary columns for doing community t
SQLite does not support 'ALTER TABLE ADD FOREIGN KEY', so this script will generate error messages when run against \
SQLite; however, script does execute successfully against SQLite.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, ForeignKey, Integer, MetaData, Table, Unicode
+
+log = logging.getLogger( __name__ )
metadata = MetaData()
StoredWorkflowTagAssociation_table = Table( "stored_workflow_tag_association", metadata,
@@ -34,6 +30,7 @@ WorkflowTagAssociation_table = Table( "workflow_tag_association", metadata,
Column( "value", Unicode(255), index=True),
Column( "user_value", Unicode(255), index=True) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -63,8 +60,7 @@ def upgrade(migrate_engine):
# Populate column so that user_id is the id of the user who owns the history (and, up to now, was the only person able to tag the history).
if c is HistoryTagAssociation_table.c.user_id:
migrate_engine.execute(
- "UPDATE history_tag_association SET user_id=( SELECT user_id FROM history WHERE history_tag_association.history_id = history.id )"
- )
+ "UPDATE history_tag_association SET user_id=( SELECT user_id FROM history WHERE history_tag_association.history_id = history.id )" )
if migrate_engine.name != 'sqlite':
# Create user_id column in history_dataset_association_tag_association table.
@@ -78,7 +74,7 @@ def upgrade(migrate_engine):
print str(e)
log.debug( "Adding user_id column to history_dataset_association_tag_association table failed: %s" % str( e ) )
else:
- #In sqlite, we can no longer quietly fail to add foreign key.
+ # In sqlite, we can no longer quietly fail to add foreign key.
# Create user_id column in history_dataset_association_tag_association table.
HistoryDatasetAssociationTagAssociation_table = Table( "history_dataset_association_tag_association", metadata, autoload=True )
c = Column( "user_id", Integer)
@@ -93,8 +89,7 @@ def upgrade(migrate_engine):
# Populate column so that user_id is the id of the user who owns the history_dataset_association (and, up to now, was the only person able to tag the page).
if c is HistoryDatasetAssociationTagAssociation_table.c.user_id:
migrate_engine.execute(
- "UPDATE history_dataset_association_tag_association SET user_id=( SELECT history.user_id FROM history, history_dataset_association WHERE history_dataset_association.history_id = history.id AND history_dataset_association.id = history_dataset_association_tag_association.history_dataset_association_id)"
- )
+ "UPDATE history_dataset_association_tag_association SET user_id=( SELECT history.user_id FROM history, history_dataset_association WHERE history_dataset_association.history_id = history.id AND history_dataset_association.id = history_dataset_association_tag_association.history_dataset_association_id)" )
if migrate_engine.name != 'sqlite':
# Create user_id column in page_tag_association table.
PageTagAssociation_table = Table( "page_tag_association", metadata, autoload=True )
@@ -121,8 +116,7 @@ def upgrade(migrate_engine):
# Populate column so that user_id is the id of the user who owns the page (and, up to now, was the only person able to tag the page).
if c is PageTagAssociation_table.c.user_id:
migrate_engine.execute(
- "UPDATE page_tag_association SET user_id=( SELECT user_id FROM page WHERE page_tag_association.page_id = page.id )"
- )
+ "UPDATE page_tag_association SET user_id=( SELECT user_id FROM page WHERE page_tag_association.page_id = page.id )" )
# Create stored_workflow_tag_association table.
try:
@@ -138,6 +132,7 @@ def upgrade(migrate_engine):
print str(e)
log.debug( "Creating workflow_tag_association table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
@@ -150,7 +145,6 @@ def downgrade(migrate_engine):
print str(e)
log.debug( "Dropping column user_id from history_tag_association table failed: %s" % str( e ) )
-
# Drop user_id column from history_dataset_association_tag_association table.
HistoryDatasetAssociationTagAssociation_table = Table( "history_dataset_association_tag_association", metadata, autoload=True )
try:
diff --git a/lib/galaxy/model/migrate/versions/0032_stored_workflow_slug_column.py b/lib/galaxy/model/migrate/versions/0032_stored_workflow_slug_column.py
index 4f4cac3efdf..331e61189ea 100644
--- a/lib/galaxy/model/migrate/versions/0032_stored_workflow_slug_column.py
+++ b/lib/galaxy/model/migrate/versions/0032_stored_workflow_slug_column.py
@@ -1,16 +1,14 @@
"""
Migration script to add slug column for stored workflow.
"""
-
-from sqlalchemy import *
-from migrate import *
-from migrate.changeset import *
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, Index, MetaData, Table, TEXT
+
+log = logging.getLogger( __name__ )
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
@@ -28,12 +26,13 @@ def upgrade(migrate_engine):
# Create slug index.
if migrate_engine.name != 'sqlite':
try:
- i = Index( "ix_stored_workflow_slug", StoredWorkflow_table.c.slug, mysql_length = 200 )
+ i = Index( "ix_stored_workflow_slug", StoredWorkflow_table.c.slug, mysql_length=200 )
i.create()
except:
# Mysql doesn't have a named index, but alter should work
StoredWorkflow_table.c.slug.alter( unique=False )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0033_published_cols_for_histories_and_workflows.py b/lib/galaxy/model/migrate/versions/0033_published_cols_for_histories_and_workflows.py
index dc580d67140..9ed0e866b69 100644
--- a/lib/galaxy/model/migrate/versions/0033_published_cols_for_histories_and_workflows.py
+++ b/lib/galaxy/model/migrate/versions/0033_published_cols_for_histories_and_workflows.py
@@ -2,17 +2,14 @@
Migration script to add necessary columns for distinguishing between viewing/importing and publishing histories, \
workflows, and pages. Script adds published column to histories and workflows and importable column to pages.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Boolean, Column, Index, MetaData, Table
+
+log = logging.getLogger( __name__ )
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -28,7 +25,6 @@ def upgrade(migrate_engine):
print "Adding published column to history table failed: %s" % str( e )
log.debug( "Adding published column to history table failed: %s" % str( e ) )
-
if migrate_engine.name != 'sqlite':
# Create index for published column in history table.
try:
@@ -76,6 +72,7 @@ def upgrade(migrate_engine):
# Mysql doesn't have a named index, but alter should work
Page_table.c.importable.alter( unique=False )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0034_page_user_share_association.py b/lib/galaxy/model/migrate/versions/0034_page_user_share_association.py
index 306a933edba..8ef1b460345 100644
--- a/lib/galaxy/model/migrate/versions/0034_page_user_share_association.py
+++ b/lib/galaxy/model/migrate/versions/0034_page_user_share_association.py
@@ -1,22 +1,18 @@
"""
Migration script to create a table for page-user share association.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, ForeignKey, Integer, MetaData, Table
+
+log = logging.getLogger( __name__ )
metadata = MetaData()
PageUserShareAssociation_table = Table( "page_user_share_association", metadata,
Column( "id", Integer, primary_key=True ),
Column( "page_id", Integer, ForeignKey( "page.id" ), index=True ),
- Column( "user_id", Integer, ForeignKey( "galaxy_user.id" ), index=True )
- )
+ Column( "user_id", Integer, ForeignKey( "galaxy_user.id" ), index=True ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
@@ -30,6 +26,7 @@ def upgrade(migrate_engine):
print str(e)
log.debug( "Creating page_user_share_association table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0035_item_annotations_and_workflow_step_tags.py b/lib/galaxy/model/migrate/versions/0035_item_annotations_and_workflow_step_tags.py
index c95909c8b39..7a5b84499f2 100644
--- a/lib/galaxy/model/migrate/versions/0035_item_annotations_and_workflow_step_tags.py
+++ b/lib/galaxy/model/migrate/versions/0035_item_annotations_and_workflow_step_tags.py
@@ -1,15 +1,11 @@
"""
Migration script to (a) create tables for annotating objects and (b) create tags for workflow steps.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, ForeignKey, Index, Integer, MetaData, Table, TEXT, Unicode
+
+log = logging.getLogger( __name__ )
metadata = MetaData()
# Annotation tables.
@@ -90,10 +86,10 @@ def upgrade(migrate_engine):
print str(e)
log.debug( "Creating workflow_step_tag_association table failed: %s" % str( e ) )
- haaa = Index( "ix_history_anno_assoc_annotation", HistoryAnnotationAssociation_table.c.annotation, mysql_length = 200)
- hdaaa = Index( "ix_history_dataset_anno_assoc_annotation", HistoryDatasetAssociationAnnotationAssociation_table.c.annotation, mysql_length = 200)
- swaaa = Index( "ix_stored_workflow_ann_assoc_annotation", StoredWorkflowAnnotationAssociation_table.c.annotation, mysql_length = 200)
- wsaaa = Index( "ix_workflow_step_ann_assoc_annotation", WorkflowStepAnnotationAssociation_table.c.annotation, mysql_length = 200)
+ haaa = Index( "ix_history_anno_assoc_annotation", HistoryAnnotationAssociation_table.c.annotation, mysql_length=200)
+ hdaaa = Index( "ix_history_dataset_anno_assoc_annotation", HistoryDatasetAssociationAnnotationAssociation_table.c.annotation, mysql_length=200)
+ swaaa = Index( "ix_stored_workflow_ann_assoc_annotation", StoredWorkflowAnnotationAssociation_table.c.annotation, mysql_length=200)
+ wsaaa = Index( "ix_workflow_step_ann_assoc_annotation", WorkflowStepAnnotationAssociation_table.c.annotation, mysql_length=200)
try:
haaa.create()
@@ -104,42 +100,42 @@ def upgrade(migrate_engine):
print str(e)
log.debug( "Creating annotation indices failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
# Drop history_annotation_association table.
try:
- HistoryAnnotationAssociation_table.drop()
+ HistoryAnnotationAssociation_table.drop()
except Exception, e:
- print str(e)
- log.debug( "Dropping history_annotation_association table failed: %s" % str( e ) )
+ print str(e)
+ log.debug( "Dropping history_annotation_association table failed: %s" % str( e ) )
# Drop history_dataset_association_annotation_association table.
try:
- HistoryDatasetAssociationAnnotationAssociation_table.drop()
+ HistoryDatasetAssociationAnnotationAssociation_table.drop()
except Exception, e:
- print str(e)
- log.debug( "Dropping history_dataset_association_annotation_association table failed: %s" % str( e ) )
+ print str(e)
+ log.debug( "Dropping history_dataset_association_annotation_association table failed: %s" % str( e ) )
# Drop stored_workflow_annotation_association table.
try:
- StoredWorkflowAnnotationAssociation_table.drop()
+ StoredWorkflowAnnotationAssociation_table.drop()
except Exception, e:
- print str(e)
- log.debug( "Dropping stored_workflow_annotation_association table failed: %s" % str( e ) )
+ print str(e)
+ log.debug( "Dropping stored_workflow_annotation_association table failed: %s" % str( e ) )
# Drop workflow_step_annotation_association table.
try:
- WorkflowStepAnnotationAssociation_table.drop()
+ WorkflowStepAnnotationAssociation_table.drop()
except Exception, e:
- print str(e)
- log.debug( "Dropping workflow_step_annotation_association table failed: %s" % str( e ) )
+ print str(e)
+ log.debug( "Dropping workflow_step_annotation_association table failed: %s" % str( e ) )
# Drop workflow_step_tag_association table.
try:
- WorkflowStepTagAssociation_table.drop()
+ WorkflowStepTagAssociation_table.drop()
except Exception, e:
- print str(e)
- log.debug( "Dropping workflow_step_tag_association table failed: %s" % str( e ) )
-
+ print str(e)
+ log.debug( "Dropping workflow_step_tag_association table failed: %s" % str( e ) )
diff --git a/lib/galaxy/model/migrate/versions/0036_add_deleted_column_to_library_template_assoc_tables.py b/lib/galaxy/model/migrate/versions/0036_add_deleted_column_to_library_template_assoc_tables.py
index a4aafd55ba0..2863b068695 100644
--- a/lib/galaxy/model/migrate/versions/0036_add_deleted_column_to_library_template_assoc_tables.py
+++ b/lib/galaxy/model/migrate/versions/0036_add_deleted_column_to_library_template_assoc_tables.py
@@ -2,22 +2,21 @@
Migration script to add a deleted column to the following tables:
library_info_association, library_folder_info_association, library_dataset_dataset_info_association.
"""
-
-from sqlalchemy import *
-from migrate import *
-from migrate.changeset import *
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Boolean, Column, MetaData, Table
+
+log = logging.getLogger( __name__ )
metadata = MetaData()
+
def get_false_value(migrate_engine):
if migrate_engine.name == 'sqlite':
return '0'
else:
return 'false'
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -53,6 +52,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "deleted to false in library_dataset_dataset_info_association failed: %s" % ( str( e ) ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
pass
diff --git a/lib/galaxy/model/migrate/versions/0037_samples_library.py b/lib/galaxy/model/migrate/versions/0037_samples_library.py
index 329837b197a..c41c0f2eb86 100644
--- a/lib/galaxy/model/migrate/versions/0037_samples_library.py
+++ b/lib/galaxy/model/migrate/versions/0037_samples_library.py
@@ -4,16 +4,16 @@ adds the same to the 'sample' table. This also adds a 'datatx' column to request
to store the sequencer login information. Finally, this adds a 'dataset_files' column to
the sample table.
"""
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-import sys, logging
-from galaxy.model.custom_types import *
-from sqlalchemy.exc import *
import datetime
-now = datetime.datetime.utcnow
+import logging
+import sys
+from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, MetaData, Table, TEXT
+from sqlalchemy.exc import NoSuchTableError
+
+from galaxy.model.custom_types import JSONType, TrimmedString
+
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler( sys.stdout )
@@ -24,6 +24,7 @@ log.addHandler( handler )
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -56,32 +57,23 @@ def upgrade(migrate_engine):
if migrate_engine.name == 'sqlite':
# create a temporary table
RequestTemp_table = Table( 'request_temp', metadata,
- Column( "id", Integer, primary_key=True),
- Column( "create_time", DateTime, default=now ),
- Column( "update_time", DateTime, default=now, onupdate=now ),
- Column( "name", TrimmedString( 255 ), nullable=False ),
- Column( "desc", TEXT ),
- Column( "form_values_id", Integer, ForeignKey( "form_values.id" ), index=True ),
- Column( "request_type_id", Integer, ForeignKey( "request_type.id" ), index=True ),
- Column( "user_id", Integer, ForeignKey( "galaxy_user.id" ), index=True ),
- Column( "deleted", Boolean, index=True, default=False ) )
+ Column( "id", Integer, primary_key=True),
+ Column( "create_time", DateTime, default=now ),
+ Column( "update_time", DateTime, default=now, onupdate=now ),
+ Column( "name", TrimmedString( 255 ), nullable=False ),
+ Column( "desc", TEXT ),
+ Column( "form_values_id", Integer, ForeignKey( "form_values.id" ), index=True ),
+ Column( "request_type_id", Integer, ForeignKey( "request_type.id" ), index=True ),
+ Column( "user_id", Integer, ForeignKey( "galaxy_user.id" ), index=True ),
+ Column( "deleted", Boolean, index=True, default=False ) )
try:
RequestTemp_table.create()
except Exception, e:
log.debug( "Creating request_temp table failed: %s" % str( e ) )
# insert all the rows from the request table to the request_temp table
- cmd = \
- "INSERT INTO request_temp " + \
- "SELECT id," + \
- "create_time," + \
- "update_time," + \
- "name," + \
- "desc," + \
- "form_values_id," + \
- "request_type_id," + \
- "user_id," + \
- "deleted " + \
- "FROM request;"
+ cmd = "INSERT INTO request_temp SELECT id, create_time, " + \
+ "update_time, name, desc, form_values_id, request_type_id, " + \
+ "user_id, deleted FROM request;"
migrate_engine.execute( cmd )
# delete the 'request' table
try:
@@ -151,6 +143,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "Adding column 'folder_id' to sample table failed: %s" % ( str( e ) ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
pass
diff --git a/lib/galaxy/model/migrate/versions/0038_add_inheritable_column_to_library_template_assoc_tables.py b/lib/galaxy/model/migrate/versions/0038_add_inheritable_column_to_library_template_assoc_tables.py
index 0c189ed970e..73caa15fd63 100644
--- a/lib/galaxy/model/migrate/versions/0038_add_inheritable_column_to_library_template_assoc_tables.py
+++ b/lib/galaxy/model/migrate/versions/0038_add_inheritable_column_to_library_template_assoc_tables.py
@@ -4,24 +4,25 @@ library_info_association, library_folder_info_association.
Also, in case of sqlite check if the previous migration script deleted the
request table and if so, restore the table.
"""
-
-from sqlalchemy import *
-from migrate import *
-from migrate.changeset import *
-from galaxy.model.custom_types import *
import datetime
-now = datetime.datetime.utcnow
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, MetaData, Table, TEXT
+
+from galaxy.model.custom_types import TrimmedString
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
+
def get_false_value(migrate_engine):
if migrate_engine.name == 'sqlite':
return '0'
else:
return 'false'
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -36,15 +37,15 @@ def upgrade(migrate_engine):
metadata.reflect(only=['form_values', 'request_type', 'galaxy_user'])
# create a temporary table
Request_table = Table( 'request', metadata,
- Column( "id", Integer, primary_key=True),
- Column( "create_time", DateTime, default=now ),
- Column( "update_time", DateTime, default=now, onupdate=now ),
- Column( "name", TrimmedString( 255 ), nullable=False ),
- Column( "desc", TEXT ),
- Column( "form_values_id", Integer, ForeignKey( "form_values.id" ), index=True ),
- Column( "request_type_id", Integer, ForeignKey( "request_type.id" ), index=True ),
- Column( "user_id", Integer, ForeignKey( "galaxy_user.id" ), index=True ),
- Column( "deleted", Boolean, index=True, default=False ) )
+ Column( "id", Integer, primary_key=True),
+ Column( "create_time", DateTime, default=now ),
+ Column( "update_time", DateTime, default=now, onupdate=now ),
+ Column( "name", TrimmedString( 255 ), nullable=False ),
+ Column( "desc", TEXT ),
+ Column( "form_values_id", Integer, ForeignKey( "form_values.id" ), index=True ),
+ Column( "request_type_id", Integer, ForeignKey( "request_type.id" ), index=True ),
+ Column( "user_id", Integer, ForeignKey( "galaxy_user.id" ), index=True ),
+ Column( "deleted", Boolean, index=True, default=False ) )
try:
Request_table.create()
except Exception, e:
@@ -72,6 +73,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "Setting value of column inheritable to false in library_folder_info_association failed: %s" % ( str( e ) ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
pass
diff --git a/lib/galaxy/model/migrate/versions/0039_add_synopsis_column_to_library_table.py b/lib/galaxy/model/migrate/versions/0039_add_synopsis_column_to_library_table.py
index 538fbc3e2c8..aafe6500feb 100644
--- a/lib/galaxy/model/migrate/versions/0039_add_synopsis_column_to_library_table.py
+++ b/lib/galaxy/model/migrate/versions/0039_add_synopsis_column_to_library_table.py
@@ -1,16 +1,14 @@
"""
Migration script to add a synopsis column to the library table.
"""
-
-from sqlalchemy import *
-from migrate import *
-from migrate.changeset import *
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, MetaData, Table, TEXT
+
+log = logging.getLogger( __name__ )
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -21,6 +19,7 @@ def upgrade(migrate_engine):
c.create( Library_table )
assert c is Library_table.c.synopsis
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
pass
diff --git a/lib/galaxy/model/migrate/versions/0040_page_annotations.py b/lib/galaxy/model/migrate/versions/0040_page_annotations.py
index d54b2af32f4..d4ca1803617 100644
--- a/lib/galaxy/model/migrate/versions/0040_page_annotations.py
+++ b/lib/galaxy/model/migrate/versions/0040_page_annotations.py
@@ -1,15 +1,11 @@
"""
Migration script to (a) create tables for annotating pages.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, ForeignKey, Integer, MetaData, Table, TEXT
+
+log = logging.getLogger( __name__ )
metadata = MetaData()
PageAnnotationAssociation_table = Table( "page_annotation_association", metadata,
@@ -18,6 +14,7 @@ PageAnnotationAssociation_table = Table( "page_annotation_association", metadata
Column( "user_id", Integer, ForeignKey( "galaxy_user.id" ), index=True ),
Column( "annotation", TEXT, index=True) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -30,6 +27,7 @@ def upgrade(migrate_engine):
print str(e)
log.debug( "Creating page_annotation_association table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
@@ -38,6 +36,5 @@ def downgrade(migrate_engine):
try:
PageAnnotationAssociation_table.drop()
except Exception, e:
- print str(e)
- log.debug( "Dropping page_annotation_association table failed: %s" % str( e ) )
-
+ print str(e)
+ log.debug( "Dropping page_annotation_association table failed: %s" % str( e ) )
diff --git a/lib/galaxy/model/migrate/versions/0041_workflow_invocation.py b/lib/galaxy/model/migrate/versions/0041_workflow_invocation.py
index 6118d9d7318..57699db3ecf 100644
--- a/lib/galaxy/model/migrate/versions/0041_workflow_invocation.py
+++ b/lib/galaxy/model/migrate/versions/0041_workflow_invocation.py
@@ -1,39 +1,33 @@
"""
Migration script to create tables for tracking workflow invocations.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
+import datetime
import logging
+
+from sqlalchemy import Column, DateTime, ForeignKey, Integer, MetaData, Table
+
logging.basicConfig( level=logging.DEBUG )
log = logging.getLogger( __name__ )
-
-import datetime
now = datetime.datetime.utcnow
-
metadata = MetaData()
WorkflowInvocation_table = Table( "workflow_invocation", metadata,
Column( "id", Integer, primary_key=True ),
Column( "create_time", DateTime, default=now ),
Column( "update_time", DateTime, default=now, onupdate=now ),
- Column( "workflow_id", Integer, ForeignKey( "workflow.id" ), index=True, nullable=False )
- )
+ Column( "workflow_id", Integer, ForeignKey( "workflow.id" ), index=True, nullable=False ) )
WorkflowInvocationStep_table = Table( "workflow_invocation_step", metadata,
Column( "id", Integer, primary_key=True ),
Column( "create_time", DateTime, default=now ),
Column( "update_time", DateTime, default=now, onupdate=now ),
Column( "workflow_invocation_id", Integer, ForeignKey( "workflow_invocation.id" ), index=True, nullable=False ),
- Column( "workflow_step_id", Integer, ForeignKey( "workflow_step.id" ), index=True, nullable=False ),
- Column( "job_id", Integer, ForeignKey( "job.id" ), index=True, nullable=False )
- )
+ Column( "workflow_step_id", Integer, ForeignKey( "workflow_step.id" ), index=True, nullable=False ),
+ Column( "job_id", Integer, ForeignKey( "job.id" ), index=True, nullable=False ) )
tables = [ WorkflowInvocation_table, WorkflowInvocationStep_table ]
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -45,6 +39,7 @@ def upgrade(migrate_engine):
except:
log.warn( "Failed to create table '%s', ignoring (might result in wrong schema)" % table.name )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0042_workflow_invocation_fix.py b/lib/galaxy/model/migrate/versions/0042_workflow_invocation_fix.py
index 50ca9969b03..a19598ef210 100644
--- a/lib/galaxy/model/migrate/versions/0042_workflow_invocation_fix.py
+++ b/lib/galaxy/model/migrate/versions/0042_workflow_invocation_fix.py
@@ -1,20 +1,17 @@
"""
Drop and readd workflow invocation tables, allowing null jobs
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
+import datetime
import logging
+
+from sqlalchemy import Column, DateTime, ForeignKey, Integer, MetaData, Table
+
logging.basicConfig( level=logging.DEBUG )
log = logging.getLogger( __name__ )
-
-import datetime
now = datetime.datetime.utcnow
-
metadata = MetaData()
+
+
def upgrade(migrate_engine):
print __doc__
metadata.bind = migrate_engine
@@ -34,17 +31,15 @@ def upgrade(migrate_engine):
Column( "id", Integer, primary_key=True ),
Column( "create_time", DateTime, default=now ),
Column( "update_time", DateTime, default=now, onupdate=now ),
- Column( "workflow_id", Integer, ForeignKey( "workflow.id" ), index=True, nullable=False ),
- )
+ Column( "workflow_id", Integer, ForeignKey( "workflow.id" ), index=True, nullable=False ) )
WorkflowInvocationStep_table = Table( "workflow_invocation_step", metadata,
Column( "id", Integer, primary_key=True ),
Column( "create_time", DateTime, default=now ),
Column( "update_time", DateTime, default=now, onupdate=now ),
Column( "workflow_invocation_id", Integer, ForeignKey( "workflow_invocation.id" ), index=True, nullable=False ),
- Column( "workflow_step_id", Integer, ForeignKey( "workflow_step.id" ), index=True, nullable=False ),
- Column( "job_id", Integer, ForeignKey( "job.id" ), index=True, nullable=True ),
- )
+ Column( "workflow_step_id", Integer, ForeignKey( "workflow_step.id" ), index=True, nullable=False ),
+ Column( "job_id", Integer, ForeignKey( "job.id" ), index=True, nullable=True ) )
for table in [ WorkflowInvocation_table, WorkflowInvocationStep_table ]:
try:
@@ -52,6 +47,7 @@ def upgrade(migrate_engine):
except:
log.exception( "Failed to create table '%s', ignoring (might result in wrong schema)" % table.name )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
# No downgrade
diff --git a/lib/galaxy/model/migrate/versions/0043_visualization_sharing_tagging_annotating.py b/lib/galaxy/model/migrate/versions/0043_visualization_sharing_tagging_annotating.py
index ed677bf70d9..a8ebaa58b82 100644
--- a/lib/galaxy/model/migrate/versions/0043_visualization_sharing_tagging_annotating.py
+++ b/lib/galaxy/model/migrate/versions/0043_visualization_sharing_tagging_annotating.py
@@ -1,15 +1,11 @@
"""
Migration script to create tables and columns for sharing visualizations.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Boolean, Column, ForeignKey, Index, Integer, MetaData, Table, TEXT, Unicode
+
+log = logging.getLogger( __name__ )
metadata = MetaData()
# Sharing visualizations.
@@ -17,8 +13,7 @@ metadata = MetaData()
VisualizationUserShareAssociation_table = Table( "visualization_user_share_association", metadata,
Column( "id", Integer, primary_key=True ),
Column( "visualization_id", Integer, ForeignKey( "visualization.id" ), index=True ),
- Column( "user_id", Integer, ForeignKey( "galaxy_user.id" ), index=True )
- )
+ Column( "user_id", Integer, ForeignKey( "galaxy_user.id" ), index=True ) )
# Tagging visualizations.
@@ -43,7 +38,6 @@ VisualizationAnnotationAssociation_table = Table( "visualization_annotation_asso
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
- db_session = scoped_session( sessionmaker( bind=migrate_engine, autoflush=False, autocommit=True ) )
metadata.reflect()
Visualiation_table = Table( "visualization", metadata, autoload=True )
@@ -55,9 +49,9 @@ def upgrade(migrate_engine):
log.debug( "Creating visualization_user_share_association table failed: %s" % str( e ) )
# Get default boolean value 'false' so that columns can be initialized.
- if migrate_engine.name == 'mysql' or migrate_engine.name == 'sqlite':
+ if migrate_engine.name in ['mysql', 'sqlite']:
default_false = "0"
- elif migrate_engine.name == 'postgresql':
+ elif migrate_engine.name in ['postgres', 'postgresql']:
default_false = "false"
# Add columns & create indices for supporting sharing to visualization table.
@@ -68,12 +62,12 @@ def upgrade(migrate_engine):
try:
# Add column.
- deleted_column.create( Visualiation_table, index_name = "ix_visualization_deleted")
+ deleted_column.create( Visualiation_table, index_name="ix_visualization_deleted")
assert deleted_column is Visualiation_table.c.deleted
# Fill column with default value.
cmd = "UPDATE visualization SET deleted = %s" % default_false
- db_session.execute( cmd )
+ migrate_engine.execute( cmd )
except Exception, e:
print "Adding deleted column to visualization table failed: %s" % str( e )
log.debug( "Adding deleted column to visualization table failed: %s" % str( e ) )
@@ -85,7 +79,7 @@ def upgrade(migrate_engine):
# Fill column with default value.
cmd = "UPDATE visualization SET importable = %s" % default_false
- db_session.execute( cmd )
+ migrate_engine.execute( cmd )
except Exception, e:
print "Adding importable column to visualization table failed: %s" % str( e )
log.debug( "Adding importable column to visualization table failed: %s" % str( e ) )
@@ -101,7 +95,7 @@ def upgrade(migrate_engine):
if migrate_engine.name == 'mysql':
# Have to create index manually.
cmd = "CREATE INDEX ix_visualization_slug ON visualization ( slug ( 100 ) )"
- db_session.execute( cmd )
+ migrate_engine.execute( cmd )
else:
i = Index( "ix_visualization_slug", Visualiation_table.c.slug )
i.create()
@@ -116,7 +110,7 @@ def upgrade(migrate_engine):
# Fill column with default value.
cmd = "UPDATE visualization SET published = %s" % default_false
- db_session.execute( cmd )
+ migrate_engine.execute( cmd )
except Exception, e:
print "Adding published column to visualization table failed: %s" % str( e )
log.debug( "Adding published column to visualization table failed: %s" % str( e ) )
@@ -137,16 +131,17 @@ def upgrade(migrate_engine):
# Need to create index for visualization annotation manually to deal with errors.
try:
- if migrate_engine.name == 'mysql':
- # Have to create index manually.
- cmd = "CREATE INDEX ix_visualization_annotation_association_annotation ON visualization_annotation_association ( annotation ( 100 ) )"
- db_session.execute( cmd )
- else:
- i = Index( "ix_visualization_annotation_association_annotation", VisualizationAnnotationAssociation_table.c.annotation )
- i.create()
+ if migrate_engine.name == 'mysql':
+ # Have to create index manually.
+ cmd = "CREATE INDEX ix_visualization_annotation_association_annotation ON visualization_annotation_association ( annotation ( 100 ) )"
+ migrate_engine.execute( cmd )
+ else:
+ i = Index( "ix_visualization_annotation_association_annotation", VisualizationAnnotationAssociation_table.c.annotation )
+ i.create()
except Exception, e:
- print "Adding index 'ix_visualization_annotation_association_annotation' failed: %s" % str( e )
- log.debug( "Adding index 'ix_visualization_annotation_association_annotation' failed: %s" % str( e ) )
+ print "Adding index 'ix_visualization_annotation_association_annotation' failed: %s" % str( e )
+ log.debug( "Adding index 'ix_visualization_annotation_association_annotation' failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
diff --git a/lib/galaxy/model/migrate/versions/0044_add_notify_column_to_request_table.py b/lib/galaxy/model/migrate/versions/0044_add_notify_column_to_request_table.py
index 26e8a086874..e887f69666f 100644
--- a/lib/galaxy/model/migrate/versions/0044_add_notify_column_to_request_table.py
+++ b/lib/galaxy/model/migrate/versions/0044_add_notify_column_to_request_table.py
@@ -1,16 +1,14 @@
"""
Migration script to add a notify column to the request table.
"""
-
-from sqlalchemy import *
-from migrate import *
-from migrate.changeset import *
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Boolean, Column, MetaData, Table
+
+log = logging.getLogger( __name__ )
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -21,6 +19,7 @@ def upgrade(migrate_engine):
c.create( Request_table )
assert c is Request_table.c.notify
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
pass
diff --git a/lib/galaxy/model/migrate/versions/0045_request_type_permissions_table.py b/lib/galaxy/model/migrate/versions/0045_request_type_permissions_table.py
index 773b02cf988..267f0efb85d 100644
--- a/lib/galaxy/model/migrate/versions/0045_request_type_permissions_table.py
+++ b/lib/galaxy/model/migrate/versions/0045_request_type_permissions_table.py
@@ -1,17 +1,13 @@
"""
Migration script to add the request_type_permissions table.
"""
-
-from sqlalchemy import *
-from migrate import *
-from migrate.changeset import *
-
import datetime
-now = datetime.datetime.utcnow
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, DateTime, ForeignKey, Integer, MetaData, Table, TEXT
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
RequestTypePermissions_table = Table( "request_type_permissions", metadata,
@@ -22,6 +18,7 @@ RequestTypePermissions_table = Table( "request_type_permissions", metadata,
Column( "request_type_id", Integer, ForeignKey( "request_type.id" ), nullable=True, index=True ),
Column( "role_id", Integer, ForeignKey( "role.id" ), index=True ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -31,6 +28,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "Creating request_type_permissions table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
pass
diff --git a/lib/galaxy/model/migrate/versions/0046_post_job_actions.py b/lib/galaxy/model/migrate/versions/0046_post_job_actions.py
index 20e8eeee480..e9d1c170f40 100644
--- a/lib/galaxy/model/migrate/versions/0046_post_job_actions.py
+++ b/lib/galaxy/model/migrate/versions/0046_post_job_actions.py
@@ -1,22 +1,17 @@
"""
Migration script to create tables for handling post-job actions.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
+import datetime
import logging
-logging.basicConfig( level=logging.DEBUG )
-log = logging.getLogger( __name__ )
+
+from sqlalchemy import Column, ForeignKey, Integer, MetaData, String, Table
# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
+from galaxy.model.custom_types import JSONType
-import datetime
+logging.basicConfig( level=logging.DEBUG )
+log = logging.getLogger( __name__ )
now = datetime.datetime.utcnow
-
metadata = MetaData()
PostJobAction_table = Table("post_job_action", metadata,
@@ -31,7 +26,8 @@ PostJobAction_table = Table("post_job_action", metadata,
# Column("post_job_action_id", Integer, ForeignKey("post_job_action.id"), index=True, nullable=False),
# Column("job_id", Integer, ForeignKey("job.id"), index=True, nullable=False))
-tables = [PostJobAction_table]#, PostJobActionAssociation_table]
+tables = [PostJobAction_table] # , PostJobActionAssociation_table]
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
@@ -43,6 +39,7 @@ def upgrade(migrate_engine):
except:
log.warn( "Failed to create table '%s', ignoring (might result in wrong schema)" % table.name )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0047_job_table_user_id_column.py b/lib/galaxy/model/migrate/versions/0047_job_table_user_id_column.py
index 35fb6a8c018..9cf9cc7022b 100644
--- a/lib/galaxy/model/migrate/versions/0047_job_table_user_id_column.py
+++ b/lib/galaxy/model/migrate/versions/0047_job_table_user_id_column.py
@@ -1,18 +1,14 @@
"""
Add a user_id column to the job table.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from sqlalchemy.exc import *
-from migrate import *
-from migrate.changeset import *
-from galaxy.model.custom_types import *
-
import datetime
-now = datetime.datetime.utcnow
+import logging
+import sys
-import sys, logging
+from sqlalchemy import Column, ForeignKey, Integer, MetaData, Table
+from sqlalchemy.exc import NoSuchTableError
+
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler( sys.stdout )
@@ -23,6 +19,7 @@ log.addHandler( handler )
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -68,6 +65,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "Updating job.user_id column failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0048_dataset_instance_state_column.py b/lib/galaxy/model/migrate/versions/0048_dataset_instance_state_column.py
index b6c858737ac..8136a6c0065 100644
--- a/lib/galaxy/model/migrate/versions/0048_dataset_instance_state_column.py
+++ b/lib/galaxy/model/migrate/versions/0048_dataset_instance_state_column.py
@@ -1,18 +1,16 @@
"""
Add a state column to the history_dataset_association and library_dataset_dataset_association table.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from sqlalchemy.exc import *
-from migrate import *
-from migrate.changeset import *
-from galaxy.model.custom_types import *
-
import datetime
-now = datetime.datetime.utcnow
+import logging
+import sys
-import sys, logging
+from sqlalchemy import Column, MetaData, Table
+from sqlalchemy.exc import NoSuchTableError
+
+from galaxy.model.custom_types import TrimmedString
+
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler( sys.stdout )
@@ -25,6 +23,7 @@ metadata = MetaData()
DATASET_INSTANCE_TABLE_NAMES = [ 'history_dataset_association', 'library_dataset_dataset_association' ]
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -40,11 +39,12 @@ def upgrade(migrate_engine):
index_name = "ix_%s_state" % table_name
try:
col = Column( "state", TrimmedString( 64 ), index=True, nullable=True )
- col.create( dataset_instance_table, index_name = index_name)
+ col.create( dataset_instance_table, index_name=index_name)
assert col is dataset_instance_table.c.state
except Exception, e:
log.debug( "Adding column 'state' to %s table failed: %s" % ( table_name, str( e ) ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0049_api_keys_table.py b/lib/galaxy/model/migrate/versions/0049_api_keys_table.py
index 4fcb1f5d289..66c956a5a73 100644
--- a/lib/galaxy/model/migrate/versions/0049_api_keys_table.py
+++ b/lib/galaxy/model/migrate/versions/0049_api_keys_table.py
@@ -1,18 +1,15 @@
"""
Migration script to add the api_keys table.
"""
-
-from sqlalchemy import *
-from migrate import *
-from migrate.changeset import *
-from galaxy.model.custom_types import *
-
import datetime
-now = datetime.datetime.utcnow
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, DateTime, ForeignKey, Integer, MetaData, Table
+
+from galaxy.model.custom_types import TrimmedString
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
APIKeys_table = Table( "api_keys", metadata,
@@ -21,6 +18,7 @@ APIKeys_table = Table( "api_keys", metadata,
Column( "user_id", Integer, ForeignKey( "galaxy_user.id" ), index=True ),
Column( "key", TrimmedString( 32 ), index=True, unique=True ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -30,6 +28,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "Creating api_keys table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
# Load existing tables
diff --git a/lib/galaxy/model/migrate/versions/0050_drop_cloud_tables.py b/lib/galaxy/model/migrate/versions/0050_drop_cloud_tables.py
index df9264a0af6..a825555ca7d 100644
--- a/lib/galaxy/model/migrate/versions/0050_drop_cloud_tables.py
+++ b/lib/galaxy/model/migrate/versions/0050_drop_cloud_tables.py
@@ -1,17 +1,13 @@
-from sqlalchemy import *
-from migrate import *
-
import datetime
-now = datetime.datetime.utcnow
-
-# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, MetaData, Table, TEXT
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
+
def display_migration_details():
print
print "========================================"
@@ -127,6 +123,7 @@ CloudProvider_table = Table( "cloud_provider", metadata,
Column( "path", TEXT ),
Column( "deleted", Boolean, default=False ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
display_migration_details()
diff --git a/lib/galaxy/model/migrate/versions/0051_imported_col_for_jobs_table.py b/lib/galaxy/model/migrate/versions/0051_imported_col_for_jobs_table.py
index f9654cf63d4..fc40791b081 100644
--- a/lib/galaxy/model/migrate/versions/0051_imported_col_for_jobs_table.py
+++ b/lib/galaxy/model/migrate/versions/0051_imported_col_for_jobs_table.py
@@ -1,17 +1,14 @@
"""
Migration script to add imported column for jobs table.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Boolean, Column, MetaData, Table
+
+log = logging.getLogger( __name__ )
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -26,7 +23,7 @@ def upgrade(migrate_engine):
assert c is Jobs_table.c.imported
# Initialize.
- if migrate_engine.name == 'mysql' or migrate_engine.name == 'sqlite':
+ if migrate_engine.name in ['mysql', 'sqlite']:
default_false = "0"
elif migrate_engine.name in ['postgres', 'postgresql']:
default_false = "false"
@@ -36,6 +33,7 @@ def upgrade(migrate_engine):
print "Adding imported column to job table failed: %s" % str( e )
log.debug( "Adding imported column to job table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0052_sample_dataset_table.py b/lib/galaxy/model/migrate/versions/0052_sample_dataset_table.py
index 33a18b1520a..098ac204bb8 100644
--- a/lib/galaxy/model/migrate/versions/0052_sample_dataset_table.py
+++ b/lib/galaxy/model/migrate/versions/0052_sample_dataset_table.py
@@ -2,40 +2,36 @@
Migration script to add the sample_dataset table and remove the 'dataset_files' column
from the 'sample' table
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-from sqlalchemy.exc import *
-
-from galaxy.model.custom_types import *
-from galaxy.util.json import loads, dumps
-
import datetime
-now = datetime.datetime.utcnow
-
import logging
-log = logging.getLogger( __name__ )
+from json import loads
+from sqlalchemy import Column, DateTime, ForeignKey, Integer, MetaData, Table, TEXT
+from sqlalchemy.exc import NoSuchTableError
+
+from galaxy.model.custom_types import TrimmedString
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
-def nextval( table, col='id' ):
- if migrate_engine.name == 'postgres':
+def nextval( migrate_engine, table, col='id' ):
+ if migrate_engine.name in ['postgres', 'postgresql']:
return "nextval('%s_%s_seq')" % ( table, col )
- elif migrate_engine.name == 'mysql' or migrate_engine.name == 'sqlite':
+ elif migrate_engine.name in ['mysql', 'sqlite']:
return "null"
else:
raise Exception( 'Unable to convert data for unknown database type: %s' % migrate_engine.name )
-def localtimestamp():
- if migrate_engine.name == 'postgres' or migrate_engine.name == 'mysql':
- return "LOCALTIMESTAMP"
- elif migrate_engine.name == 'sqlite':
- return "current_date || ' ' || current_time"
- else:
- raise Exception( 'Unable to convert data for unknown database type: %s' % db )
+
+def localtimestamp(migrate_engine):
+ if migrate_engine.name in ['mysql', 'postgres', 'postgresql']:
+ return "LOCALTIMESTAMP"
+ elif migrate_engine.name == 'sqlite':
+ return "current_date || ' ' || current_time"
+ else:
+ raise Exception( 'Unable to convert data for unknown database type: %s' % migrate_engine.name )
SampleDataset_table = Table('sample_dataset', metadata,
Column( "id", Integer, primary_key=True ),
@@ -48,6 +44,7 @@ SampleDataset_table = Table('sample_dataset', metadata,
Column( "error_msg", TEXT ),
Column( "size", TrimmedString( 255 ) ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -66,9 +63,9 @@ def upgrade(migrate_engine):
for df in dataset_files:
if type(df) == type(dict()):
cmd = "INSERT INTO sample_dataset VALUES (%s, %s, %s, %s, '%s', '%s', '%s', '%s', '%s')"
- cmd = cmd % ( nextval('sample_dataset'),
- localtimestamp(),
- localtimestamp(),
+ cmd = cmd % ( nextval(migrate_engine, 'sample_dataset'),
+ localtimestamp(migrate_engine),
+ localtimestamp(migrate_engine),
str(sample_id),
df.get('name', ''),
df.get('filepath', ''),
diff --git a/lib/galaxy/model/migrate/versions/0053_item_ratings.py b/lib/galaxy/model/migrate/versions/0053_item_ratings.py
index 4f379ffef97..83b2df0d9b4 100644
--- a/lib/galaxy/model/migrate/versions/0053_item_ratings.py
+++ b/lib/galaxy/model/migrate/versions/0053_item_ratings.py
@@ -1,15 +1,11 @@
"""
Migration script to create tables for rating histories, datasets, workflows, pages, and visualizations.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, ForeignKey, Index, Integer, MetaData, Table
+
+log = logging.getLogger( __name__ )
metadata = MetaData()
# Rating tables.
@@ -44,6 +40,7 @@ VisualizationRatingAssociation_table = Table( "visualization_rating_association"
Column( "user_id", Integer, ForeignKey( "galaxy_user.id" ), index=True ),
Column( "rating", Integer, index=True) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -62,7 +59,7 @@ def upgrade(migrate_engine):
except Exception, e:
# MySQL cannot handle long index names; when we see this error, create the index name manually.
if migrate_engine.name == 'mysql' and \
- str(e).lower().find("identifier name 'ix_history_dataset_association_rating_association_history_dataset_association_id' is too long"):
+ str(e).lower().find("identifier name 'ix_history_dataset_association_rating_association_history_dataset_association_id' is too long"):
i = Index( "ix_hda_rating_association_hda_id", HistoryDatasetAssociationRatingAssociation_table.c.history_dataset_association_id )
try:
i.create()
@@ -94,41 +91,42 @@ def upgrade(migrate_engine):
print str(e)
log.debug( "Creating visualization_rating_association table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
# Drop history_rating_association table.
try:
- HistoryRatingAssociation_table.drop()
+ HistoryRatingAssociation_table.drop()
except Exception, e:
- print str(e)
- log.debug( "Dropping history_rating_association table failed: %s" % str( e ) )
+ print str(e)
+ log.debug( "Dropping history_rating_association table failed: %s" % str( e ) )
# Drop history_dataset_association_rating_association table.
try:
- HistoryDatasetAssociationRatingAssociation_table.drop()
+ HistoryDatasetAssociationRatingAssociation_table.drop()
except Exception, e:
- print str(e)
- log.debug( "Dropping history_dataset_association_rating_association table failed: %s" % str( e ) )
+ print str(e)
+ log.debug( "Dropping history_dataset_association_rating_association table failed: %s" % str( e ) )
# Drop stored_workflow_rating_association table.
try:
- StoredWorkflowRatingAssociation_table.drop()
+ StoredWorkflowRatingAssociation_table.drop()
except Exception, e:
- print str(e)
- log.debug( "Dropping stored_workflow_rating_association table failed: %s" % str( e ) )
+ print str(e)
+ log.debug( "Dropping stored_workflow_rating_association table failed: %s" % str( e ) )
# Drop page_rating_association table.
try:
- PageRatingAssociation_table.drop()
+ PageRatingAssociation_table.drop()
except Exception, e:
- print str(e)
- log.debug( "Dropping page_rating_association table failed: %s" % str( e ) )
+ print str(e)
+ log.debug( "Dropping page_rating_association table failed: %s" % str( e ) )
# Drop visualization_rating_association table.
try:
- VisualizationRatingAssociation_table.drop()
+ VisualizationRatingAssociation_table.drop()
except Exception, e:
- print str(e)
- log.debug( "Dropping visualization_rating_association table failed: %s" % str( e ) )
+ print str(e)
+ log.debug( "Dropping visualization_rating_association table failed: %s" % str( e ) )
diff --git a/lib/galaxy/model/migrate/versions/0054_visualization_dbkey.py b/lib/galaxy/model/migrate/versions/0054_visualization_dbkey.py
index 9cac2d61df5..a60768336f5 100644
--- a/lib/galaxy/model/migrate/versions/0054_visualization_dbkey.py
+++ b/lib/galaxy/model/migrate/versions/0054_visualization_dbkey.py
@@ -1,18 +1,15 @@
"""
Migration script to add dbkey column for visualization.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-from galaxy.util.json import loads
-
import logging
-log = logging.getLogger( __name__ )
+from json import loads
+from sqlalchemy import Column, Index, MetaData, Table, TEXT
+
+log = logging.getLogger( __name__ )
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
@@ -28,9 +25,9 @@ def upgrade(migrate_engine):
x.create( Visualization_table )
y.create( Visualization_revision_table )
# Manually create indexes for compatability w/ mysql_length.
- xi = Index( "ix_visualization_dbkey", Visualization_table.c.dbkey, mysql_length = 200)
+ xi = Index( "ix_visualization_dbkey", Visualization_table.c.dbkey, mysql_length=200)
xi.create()
- yi = Index( "ix_visualization_revision_dbkey", Visualization_revision_table.c.dbkey, mysql_length = 200)
+ yi = Index( "ix_visualization_revision_dbkey", Visualization_revision_table.c.dbkey, mysql_length=200)
yi.create()
assert x is Visualization_table.c.dbkey
assert y is Visualization_revision_table.c.dbkey
@@ -45,6 +42,7 @@ def upgrade(migrate_engine):
migrate_engine.execute("UPDATE visualization_revision SET dbkey='%s' WHERE id=%s" % (dbkey, viz_rev_id))
migrate_engine.execute("UPDATE visualization SET dbkey='%s' WHERE id=%s" % (dbkey, viz_id))
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0055_add_pja_assoc_for_jobs.py b/lib/galaxy/model/migrate/versions/0055_add_pja_assoc_for_jobs.py
index ce49bf86fa0..777d913780a 100644
--- a/lib/galaxy/model/migrate/versions/0055_add_pja_assoc_for_jobs.py
+++ b/lib/galaxy/model/migrate/versions/0055_add_pja_assoc_for_jobs.py
@@ -1,18 +1,13 @@
"""
Migration script to add the post_job_action_association table.
"""
-
-from sqlalchemy import *
-from migrate import *
-from migrate.changeset import *
-from galaxy.model.custom_types import *
-
import datetime
-now = datetime.datetime.utcnow
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, ForeignKey, Integer, MetaData, Table
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
PostJobActionAssociation_table = Table("post_job_action_association", metadata,
@@ -20,6 +15,7 @@ PostJobActionAssociation_table = Table("post_job_action_association", metadata,
Column("post_job_action_id", Integer, ForeignKey("post_job_action.id"), index=True, nullable=False),
Column("job_id", Integer, ForeignKey("job.id"), index=True, nullable=False))
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -29,6 +25,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "Creating PostJobActionAssociation table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
# Load existing tables
diff --git a/lib/galaxy/model/migrate/versions/0056_workflow_outputs.py b/lib/galaxy/model/migrate/versions/0056_workflow_outputs.py
index a495c23d7a9..2e52a773295 100644
--- a/lib/galaxy/model/migrate/versions/0056_workflow_outputs.py
+++ b/lib/galaxy/model/migrate/versions/0056_workflow_outputs.py
@@ -1,19 +1,13 @@
"""
Migration script to create tables for adding explicit workflow outputs.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import logging
+
+from sqlalchemy import Column, ForeignKey, Integer, MetaData, String, Table
+
logging.basicConfig( level=logging.DEBUG )
log = logging.getLogger( __name__ )
-# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
-
metadata = MetaData()
WorkflowOutput_table = Table( "workflow_output", metadata,
@@ -23,6 +17,7 @@ WorkflowOutput_table = Table( "workflow_output", metadata,
tables = [WorkflowOutput_table]
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -33,6 +28,7 @@ def upgrade(migrate_engine):
except:
log.warn( "Failed to create table '%s', ignoring (might result in wrong schema)" % table.name )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0057_request_notify.py b/lib/galaxy/model/migrate/versions/0057_request_notify.py
index e763f524d96..2ee057e3e86 100644
--- a/lib/galaxy/model/migrate/versions/0057_request_notify.py
+++ b/lib/galaxy/model/migrate/versions/0057_request_notify.py
@@ -2,24 +2,20 @@
Migration script to modify the 'notify' field in the 'request' table from a boolean
to a JSONType
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-from sqlalchemy.exc import *
-
-from galaxy.model.custom_types import *
-from galaxy.util.json import loads, dumps
-
import datetime
-now = datetime.datetime.utcnow
-
import logging
-log = logging.getLogger( __name__ )
+from json import dumps
+from sqlalchemy import Column, MetaData, Table
+from sqlalchemy.exc import NoSuchTableError
+
+from galaxy.model.custom_types import JSONType
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -30,7 +26,6 @@ def upgrade(migrate_engine):
Request_table = None
log.debug( "Failed loading table 'request'" )
-
if Request_table is not None:
# create the column again as JSONType
try:
@@ -44,16 +39,10 @@ def upgrade(migrate_engine):
result = migrate_engine.execute( cmd )
for r in result:
id = int(r[0])
- notify_old = r[1]
notify_new = dict(email=[], sample_states=[], body='', subject='')
- cmd = "update request set notification='%s' where id=%i" % (dumps(notify_new), id)
+ cmd = "UPDATE request SET notification='%s' WHERE id=%i" % (dumps(notify_new), id)
migrate_engine.execute( cmd )
- cmd = "SELECT id, notification FROM request"
- result = migrate_engine.execute( cmd )
- for r in result:
- rr = loads(str(r[1]))
-
# remove the 'notify' column for non-sqlite databases.
if migrate_engine.name != 'sqlite':
try:
@@ -62,7 +51,6 @@ def upgrade(migrate_engine):
log.debug( "Deleting column 'notify' from the 'request' table failed: %s" % ( str( e ) ) )
-
def downgrade(migrate_engine):
metadata.bind = migrate_engine
pass
diff --git a/lib/galaxy/model/migrate/versions/0058_history_import_export.py b/lib/galaxy/model/migrate/versions/0058_history_import_export.py
index e430eb65f13..070bfc7474a 100644
--- a/lib/galaxy/model/migrate/versions/0058_history_import_export.py
+++ b/lib/galaxy/model/migrate/versions/0058_history_import_export.py
@@ -1,15 +1,11 @@
"""
Migration script to create table for exporting histories to archives.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Boolean, Column, ForeignKey, Integer, MetaData, Table, TEXT
+
+log = logging.getLogger( __name__ )
metadata = MetaData()
# Table to add.
@@ -22,8 +18,8 @@ JobExportHistoryArchive_table = Table( "job_export_history_archive", metadata,
Column( "compressed", Boolean, index=True, default=False ),
Column( "history_attrs_filename", TEXT ),
Column( "datasets_attrs_filename", TEXT ),
- Column( "jobs_attrs_filename", TEXT )
- )
+ Column( "jobs_attrs_filename", TEXT ) )
+
def upgrade(migrate_engine):
print __doc__
@@ -37,6 +33,7 @@ def upgrade(migrate_engine):
print str(e)
log.debug( "Creating job_export_history_archive table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0059_sample_dataset_file_path.py b/lib/galaxy/model/migrate/versions/0059_sample_dataset_file_path.py
index 53ee2ef8840..aa78fa5c71f 100644
--- a/lib/galaxy/model/migrate/versions/0059_sample_dataset_file_path.py
+++ b/lib/galaxy/model/migrate/versions/0059_sample_dataset_file_path.py
@@ -2,24 +2,17 @@
Migration script to modify the 'file_path' field type in 'sample_dataset' table
to 'TEXT' so that it can support large file paths exceeding 255 characters
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-from sqlalchemy.exc import *
-
-from galaxy.model.custom_types import *
-from galaxy.util.json import loads, dumps
-
import datetime
-now = datetime.datetime.utcnow
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, MetaData, Table, TEXT
+from sqlalchemy.exc import NoSuchTableError
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -54,6 +47,7 @@ def upgrade(migrate_engine):
cmd = "update sample_dataset set file_path='%s' where id=%i" % (file_path, id)
migrate_engine.execute( cmd )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
pass
diff --git a/lib/galaxy/model/migrate/versions/0060_history_archive_import.py b/lib/galaxy/model/migrate/versions/0060_history_archive_import.py
index c9576429646..6d6c17190af 100644
--- a/lib/galaxy/model/migrate/versions/0060_history_archive_import.py
+++ b/lib/galaxy/model/migrate/versions/0060_history_archive_import.py
@@ -2,15 +2,11 @@
Migration script to create column and table for importing histories from
file archives.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Boolean, Column, ForeignKey, Integer, MetaData, Table, TEXT
+
+log = logging.getLogger( __name__ )
metadata = MetaData()
# Columns to add.
@@ -24,8 +20,8 @@ JobImportHistoryArchive_table = Table( "job_import_history_archive", metadata,
Column( "id", Integer, primary_key=True ),
Column( "job_id", Integer, ForeignKey( "job.id" ), index=True ),
Column( "history_id", Integer, ForeignKey( "history.id" ), index=True ),
- Column( "archive_dir", TEXT )
- )
+ Column( "archive_dir", TEXT ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
@@ -39,7 +35,7 @@ def upgrade(migrate_engine):
assert importing_col is History_table.c.importing
# Initialize column to false.
- if migrate_engine.name == 'mysql' or migrate_engine.name == 'sqlite':
+ if migrate_engine.name in ['mysql', 'sqlite']:
default_false = "0"
elif migrate_engine.name in ['postgres', 'postgresql']:
default_false = "false"
@@ -54,6 +50,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "Creating job_import_history_archive table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0061_tasks.py b/lib/galaxy/model/migrate/versions/0061_tasks.py
index 549c84fa979..cf834eedb85 100644
--- a/lib/galaxy/model/migrate/versions/0061_tasks.py
+++ b/lib/galaxy/model/migrate/versions/0061_tasks.py
@@ -1,16 +1,12 @@
"""
Migration script to create tables task management.
"""
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import datetime
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, DateTime, ForeignKey, Integer, MetaData, String, Table, TEXT
+
+log = logging.getLogger( __name__ )
metadata = MetaData()
now = datetime.datetime.utcnow
@@ -33,6 +29,7 @@ Task_table = Table( "task", metadata,
tables = [Task_table]
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -43,6 +40,7 @@ def upgrade(migrate_engine):
except:
log.warn( "Failed to create table '%s', ignoring (might result in wrong schema)" % table.name )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0062_user_openid_table.py b/lib/galaxy/model/migrate/versions/0062_user_openid_table.py
index fdc58e4736c..1430645880c 100644
--- a/lib/galaxy/model/migrate/versions/0062_user_openid_table.py
+++ b/lib/galaxy/model/migrate/versions/0062_user_openid_table.py
@@ -2,18 +2,13 @@
Migration script to create table for associating sessions and users with
OpenIDs.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import datetime
-now = datetime.datetime.utcnow
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, DateTime, ForeignKey, Index, Integer, MetaData, Table, TEXT
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
# Table to add
@@ -24,8 +19,8 @@ UserOpenID_table = Table( "galaxy_user_openid", metadata,
Column( "update_time", DateTime, index=True, default=now, onupdate=now ),
Column( "session_id", Integer, ForeignKey( "galaxy_session.id" ), index=True ),
Column( "user_id", Integer, ForeignKey( "galaxy_user.id" ), index=True ),
- Column( "openid", TEXT ),
- )
+ Column( "openid", TEXT ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
@@ -49,6 +44,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "Adding index '%s' failed: %s" % ( ix_name, str( e ) ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0063_sequencer_table.py b/lib/galaxy/model/migrate/versions/0063_sequencer_table.py
index 3270649160d..85ea0acebb5 100644
--- a/lib/galaxy/model/migrate/versions/0063_sequencer_table.py
+++ b/lib/galaxy/model/migrate/versions/0063_sequencer_table.py
@@ -1,21 +1,16 @@
"""
Migration script to create a new 'sequencer' table
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-from sqlalchemy.exc import *
-
-from galaxy.model.custom_types import *
-
import datetime
-now = datetime.datetime.utcnow
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, MetaData, Table, TEXT
+from sqlalchemy.exc import NoSuchTableError
+
+from galaxy.model.custom_types import TrimmedString
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
# Table to add
@@ -31,6 +26,7 @@ Sequencer_table = Table( 'sequencer', metadata,
Column( "form_values_id", Integer, ForeignKey( "form_values.id" ), index=True ),
Column( "deleted", Boolean, index=True, default=False ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -56,4 +52,3 @@ def downgrade(migrate_engine):
Sequencer_table.drop()
except Exception, e:
log.debug( "Deleting 'sequencer' table failed: %s" % str( e ) )
-
diff --git a/lib/galaxy/model/migrate/versions/0064_add_run_and_sample_run_association_tables.py b/lib/galaxy/model/migrate/versions/0064_add_run_and_sample_run_association_tables.py
index 41fd3e92cbc..efdbe4306c9 100644
--- a/lib/galaxy/model/migrate/versions/0064_add_run_and_sample_run_association_tables.py
+++ b/lib/galaxy/model/migrate/versions/0064_add_run_and_sample_run_association_tables.py
@@ -1,18 +1,13 @@
"""
Migration script to add the run and sample_run_association tables.
"""
-
-from sqlalchemy import *
-from migrate import *
-from migrate.changeset import *
-from galaxy.model.custom_types import *
-
import datetime
-now = datetime.datetime.utcnow
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, MetaData, Table
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
Run_table = Table( "run", metadata,
@@ -33,6 +28,7 @@ SampleRunAssociation_table = Table( "sample_run_association", metadata,
Column( "sample_id", Integer, ForeignKey( "sample.id" ), index=True, nullable=False ),
Column( "run_id", Integer, ForeignKey( "run.id" ), index=True, nullable=False ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -50,6 +46,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "Creating SampleRunAssociation table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
# Load existing tables
diff --git a/lib/galaxy/model/migrate/versions/0065_add_name_to_form_fields_and_values.py b/lib/galaxy/model/migrate/versions/0065_add_name_to_form_fields_and_values.py
index 2776d859fd1..78de6e579d2 100644
--- a/lib/galaxy/model/migrate/versions/0065_add_name_to_form_fields_and_values.py
+++ b/lib/galaxy/model/migrate/versions/0065_add_name_to_form_fields_and_values.py
@@ -3,19 +3,16 @@ Migration script to add 'name' attribute to the JSON dict which describes
a form definition field and the form values in the database. In the 'form_values'
table, the 'content' column is now a JSON dict instead of a list.
"""
+import datetime
+import logging
+import sys
+from json import loads, dumps
+
+from sqlalchemy import MetaData, Table
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-from sqlalchemy.exc import *
-from galaxy.util.json import loads, dumps
from galaxy.model.custom_types import _sniffnfix_pg9_hex
-import datetime
now = datetime.datetime.utcnow
-
-import sys, logging
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler( sys.stdout )
@@ -23,7 +20,6 @@ format = "%(name)s %(levelname)s %(asctime)s %(message)s"
formatter = logging.Formatter( format )
handler.setFormatter( formatter )
log.addHandler( handler )
-
metadata = MetaData()
@@ -32,17 +28,18 @@ def upgrade(migrate_engine):
print __doc__
metadata.reflect()
try:
- FormDefinition_table = Table( "form_definition", metadata, autoload=True )
+ Table( "form_definition", metadata, autoload=True )
except Exception, e:
log.debug( "Loading 'form_definition' table failed: %s" % str( e ) )
try:
- FormValues_table = Table( "form_values", metadata, autoload=True )
+ Table( "form_values", metadata, autoload=True )
except Exception, e:
log.debug( "Loading 'form_values' table failed: %s" % str( e ) )
+
def get_value(lst, index):
try:
return str(lst[index]).replace("'", "''")
- except IndexError,e:
+ except IndexError:
return ''
# Go through the entire table and add a 'name' attribute for each field
# in the list of fields for each form definition
@@ -61,9 +58,9 @@ def upgrade(migrate_engine):
field[ 'label' ] = field[ 'label' ].replace("'", "''")
fields_json = dumps( fields_list )
if migrate_engine.name == 'mysql':
- cmd = "UPDATE form_definition AS f SET f.fields='%s' WHERE f.id=%i" %( fields_json, form_definition_id )
+ cmd = "UPDATE form_definition AS f SET f.fields='%s' WHERE f.id=%i" % ( fields_json, form_definition_id )
else:
- cmd = "UPDATE form_definition SET fields='%s' WHERE id=%i" %( fields_json, form_definition_id )
+ cmd = "UPDATE form_definition SET fields='%s' WHERE id=%i" % ( fields_json, form_definition_id )
migrate_engine.execute( cmd )
# replace the values list in the content field of the form_values table with a name:value dict
cmd = "SELECT form_values.id, form_values.content, form_definition.fields" \
@@ -85,18 +82,19 @@ def upgrade(migrate_engine):
for field_index, field in enumerate( fields_list ):
field_name = field[ 'name' ]
values_dict[ field_name ] = get_value(values_list, field_index )
- cmd = "UPDATE form_values SET content='%s' WHERE id=%i" %( dumps( values_dict ), form_values_id )
+ cmd = "UPDATE form_values SET content='%s' WHERE id=%i" % ( dumps( values_dict ), form_values_id )
migrate_engine.execute( cmd )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
try:
- FormDefinition_table = Table( "form_definition", metadata, autoload=True )
+ Table( "form_definition", metadata, autoload=True )
except Exception, e:
log.debug( "Loading 'form_definition' table failed: %s" % str( e ) )
try:
- FormValues_table = Table( "form_values", metadata, autoload=True )
+ Table( "form_values", metadata, autoload=True )
except Exception, e:
log.debug( "Loading 'form_values' table failed: %s" % str( e ) )
# remove the name attribute in the content column JSON dict in the form_values table
@@ -120,7 +118,7 @@ def downgrade(migrate_engine):
field_name = field[ 'name' ]
field_value = values_dict[ field_name ]
values_list.append( field_value )
- cmd = "UPDATE form_values SET content='%s' WHERE id=%i" %( dumps( values_list ), form_values_id )
+ cmd = "UPDATE form_values SET content='%s' WHERE id=%i" % ( dumps( values_list ), form_values_id )
migrate_engine.execute( cmd )
# remove name attribute from the field column of the form_definition table
cmd = "SELECT f.id, f.fields FROM form_definition AS f"
@@ -133,10 +131,10 @@ def downgrade(migrate_engine):
fields_list = loads( _sniffnfix_pg9_hex( fields ) )
if len( fields_list ):
for index, field in enumerate( fields_list ):
- if field.has_key( 'name' ):
+ if 'name' in field:
del field[ 'name' ]
if migrate_engine.name == 'mysql':
- cmd = "UPDATE form_definition AS f SET f.fields='%s' WHERE f.id=%i" %( dumps( fields_list ), form_definition_id )
+ cmd = "UPDATE form_definition AS f SET f.fields='%s' WHERE f.id=%i" % ( dumps( fields_list ), form_definition_id )
else:
- cmd = "UPDATE form_definition SET fields='%s' WHERE id=%i" %( dumps( fields_list ), form_definition_id )
+ cmd = "UPDATE form_definition SET fields='%s' WHERE id=%i" % ( dumps( fields_list ), form_definition_id )
migrate_engine.execute( cmd )
diff --git a/lib/galaxy/model/migrate/versions/0066_deferred_job_and_transfer_job_tables.py b/lib/galaxy/model/migrate/versions/0066_deferred_job_and_transfer_job_tables.py
index d6fe7eb1dec..d731aec8ecc 100644
--- a/lib/galaxy/model/migrate/versions/0066_deferred_job_and_transfer_job_tables.py
+++ b/lib/galaxy/model/migrate/versions/0066_deferred_job_and_transfer_job_tables.py
@@ -2,20 +2,15 @@
Migration script to create table for storing deferred job and managed transfer
information.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
-from galaxy.model.custom_types import *
-
import datetime
-now = datetime.datetime.utcnow
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, DateTime, Integer, MetaData, String, Table
+
+from galaxy.model.custom_types import JSONType
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
# Table to add
@@ -36,6 +31,7 @@ TransferJob_table = Table( "transfer_job", metadata,
Column( "path", String( 1024 ) ),
Column( "params", JSONType ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -53,6 +49,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.error( "Creating transfer_job table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0067_populate_sequencer_table.py b/lib/galaxy/model/migrate/versions/0067_populate_sequencer_table.py
index 3c790330a24..5083dc80e81 100644
--- a/lib/galaxy/model/migrate/versions/0067_populate_sequencer_table.py
+++ b/lib/galaxy/model/migrate/versions/0067_populate_sequencer_table.py
@@ -4,42 +4,39 @@ entries in the 'datatx_info' column in the 'request_type' table. It also deletes
column in the 'request_type' table and adds a foreign key to the 'sequencer' table. The
actual contents of the datatx_info column are stored as form_values.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-from sqlalchemy.exc import *
-
-from galaxy.model.custom_types import *
-
-from galaxy.util.json import loads, dumps
-
import datetime
-now = datetime.datetime.utcnow
-
import logging
-log = logging.getLogger( __name__ )
+from json import dumps, loads
+from sqlalchemy import Column, ForeignKey, Integer, MetaData, Table
+from sqlalchemy.exc import NoSuchTableError
+
+from galaxy.model.custom_types import JSONType
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
-def nextval( table, col='id' ):
- if migrate_engine.name == 'postgres':
+
+def nextval( migrate_engine, table, col='id' ):
+ if migrate_engine.name in ['postgres', 'postgresql']:
return "nextval('%s_%s_seq')" % ( table, col )
- elif migrate_engine.name == 'mysql' or migrate_engine.name == 'sqlite':
+ elif migrate_engine.name in ['mysql', 'sqlite']:
return "null"
else:
raise Exception( 'Unable to convert data for unknown database type: %s' % migrate_engine.name )
-def localtimestamp():
- if migrate_engine.name == 'postgres' or migrate_engine.name == 'mysql':
- return "LOCALTIMESTAMP"
- elif migrate_engine.name == 'sqlite':
- return "current_date || ' ' || current_time"
- else:
- raise Exception( 'Unable to convert data for unknown database type: %s' % db )
-def get_latest_id( table ):
+def localtimestamp( migrate_engine ):
+ if migrate_engine.name in ['mysql', 'postgres', 'postgresql']:
+ return "LOCALTIMESTAMP"
+ elif migrate_engine.name == 'sqlite':
+ return "current_date || ' ' || current_time"
+ else:
+ raise Exception( 'Unable to convert data for unknown database type: %s' % migrate_engine.name )
+
+
+def get_latest_id( migrate_engine, table ):
result = migrate_engine.execute( "select id from %s order by id desc" % table )
row = result.fetchone()
if row:
@@ -47,17 +44,19 @@ def get_latest_id( table ):
else:
raise Exception( 'Unable to get the latest id in the %s table.' % table )
-def boolean( value ):
- if migrate_engine.name == 'postgres' or migrate_engine.name == 'mysql':
- return value
- elif migrate_engine.name == 'sqlite':
- if value in [ 'True, true' ]:
- return 1
- return 0
- else:
- raise Exception( 'Unable to convert data for unknown database type: %s' % db )
-def create_sequencer_form_definition():
+def boolean( migrate_engine, value ):
+ if migrate_engine.name in ['mysql', 'postgres', 'postgresql']:
+ return value
+ elif migrate_engine.name == 'sqlite':
+ if value in [ 'True', 'true' ]:
+ return 1
+ return 0
+ else:
+ raise Exception( 'Unable to convert data for unknown database type: %s' % migrate_engine.name )
+
+
+def create_sequencer_form_definition( migrate_engine ):
'''
Create a new form_definition containing 5 fields (host, username, password,
data_dir & rename_datasets) which described the existing datatx_info json
@@ -65,14 +64,14 @@ def create_sequencer_form_definition():
'''
# create new form_definition_current in the db
cmd = "INSERT INTO form_definition_current VALUES ( %s, %s, %s, %s, %s )"
- cmd = cmd % ( nextval( 'form_definition_current' ),
- localtimestamp(),
- localtimestamp(),
+ cmd = cmd % ( nextval( migrate_engine, 'form_definition_current' ),
+ localtimestamp( migrate_engine ),
+ localtimestamp( migrate_engine ),
'NULL',
- boolean( 'false' ) )
+ boolean( migrate_engine, 'false' ) )
migrate_engine.execute( cmd )
# get this form_definition_current id
- form_definition_current_id = get_latest_id( 'form_definition_current' )
+ form_definition_current_id = get_latest_id( migrate_engine, 'form_definition_current' )
# create new form_definition in the db
form_definition_name = 'Generic sequencer form'
form_definition_desc = ''
@@ -106,9 +105,9 @@ def create_sequencer_form_definition():
form_definition_type = 'Sequencer Information Form'
form_definition_layout = dumps('[]')
cmd = "INSERT INTO form_definition VALUES ( %s, %s, %s, '%s', '%s', %s, '%s', '%s', '%s' )"
- cmd = cmd % ( nextval( 'form_definition' ),
- localtimestamp(),
- localtimestamp(),
+ cmd = cmd % ( nextval( migrate_engine, 'form_definition' ),
+ localtimestamp( migrate_engine ),
+ localtimestamp( migrate_engine ),
form_definition_name,
form_definition_desc,
form_definition_current_id,
@@ -117,13 +116,14 @@ def create_sequencer_form_definition():
form_definition_layout )
migrate_engine.execute( cmd )
# get this form_definition id
- form_definition_id = get_latest_id( 'form_definition' )
+ form_definition_id = get_latest_id( migrate_engine, 'form_definition' )
# update the form_definition_id column in form_definition_current
- cmd = "UPDATE form_definition_current SET latest_form_id=%i WHERE id=%i" %( form_definition_id, form_definition_current_id )
+ cmd = "UPDATE form_definition_current SET latest_form_id=%i WHERE id=%i" % ( form_definition_id, form_definition_current_id )
migrate_engine.execute( cmd )
return form_definition_id
-def get_sequencer_id( sequencer_info ):
+
+def get_sequencer_id( migrate_engine, sequencer_info ):
'''Get the sequencer id corresponding to the sequencer information'''
# Check if there is any existing sequencer which have the same sequencer
# information fields & values
@@ -145,46 +145,48 @@ def get_sequencer_id( sequencer_info ):
return sequencer_id
return None
-def add_sequencer( sequencer_index, sequencer_form_definition_id, sequencer_info ):
+
+def add_sequencer( migrate_engine, sequencer_index, sequencer_form_definition_id, sequencer_info ):
'''Adds a new sequencer to the sequencer table along with its form values.'''
# Create a new form values record with the supplied sequencer information
values = dumps( { 'field_0': sequencer_info.get( 'host', '' ),
- 'field_1': sequencer_info.get( 'username', '' ),
- 'field_2': sequencer_info.get( 'password', '' ),
- 'field_3': sequencer_info.get( 'data_dir', '' ),
- 'field_4': sequencer_info.get( 'rename_dataset', '' ) } )
- cmd = "INSERT INTO form_values VALUES ( %s, %s, %s, %s, '%s' )" % ( nextval( 'form_values' ),
- localtimestamp(),
- localtimestamp(),
+ 'field_1': sequencer_info.get( 'username', '' ),
+ 'field_2': sequencer_info.get( 'password', '' ),
+ 'field_3': sequencer_info.get( 'data_dir', '' ),
+ 'field_4': sequencer_info.get( 'rename_dataset', '' ) } )
+ cmd = "INSERT INTO form_values VALUES ( %s, %s, %s, %s, '%s' )" % ( nextval( migrate_engine, 'form_values' ),
+ localtimestamp( migrate_engine ),
+ localtimestamp( migrate_engine ),
sequencer_form_definition_id,
values )
migrate_engine.execute(cmd)
- sequencer_form_values_id = get_latest_id( 'form_values' )
+ sequencer_form_values_id = get_latest_id( migrate_engine, 'form_values' )
# Create a new sequencer record with reference to the form value created above.
name = 'Sequencer_%i' % sequencer_index
desc = ''
version = ''
- result_datasets = dict()
sequencer_type_id = 'simple_unknown_sequencer'
cmd = "INSERT INTO sequencer VALUES ( %s, %s, %s, '%s', '%s', '%s', '%s', %s, %s, %s )"
- cmd = cmd % ( nextval('sequencer'),
- localtimestamp(),
- localtimestamp(),
+ cmd = cmd % ( nextval( migrate_engine, 'sequencer'),
+ localtimestamp( migrate_engine ),
+ localtimestamp( migrate_engine ),
name,
desc,
sequencer_type_id,
version,
sequencer_form_definition_id,
sequencer_form_values_id,
- boolean( 'false' ) )
+ boolean( migrate_engine, 'false' ) )
migrate_engine.execute(cmd)
- return get_latest_id( 'sequencer' )
+ return get_latest_id( migrate_engine, 'sequencer' )
-def update_sequencer_id_in_request_type( request_type_id, sequencer_id ):
+
+def update_sequencer_id_in_request_type( migrate_engine, request_type_id, sequencer_id ):
'''Update the foreign key to the sequencer table in the request_type table'''
- cmd = "UPDATE request_type SET sequencer_id=%i WHERE id=%i" %( sequencer_id, request_type_id )
+ cmd = "UPDATE request_type SET sequencer_id=%i WHERE id=%i" % ( sequencer_id, request_type_id )
migrate_engine.execute( cmd )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -224,12 +226,11 @@ def upgrade(migrate_engine):
# For the existing request_types in the database, we add a new form_definition
# with these 4 fields. Then we populate the sequencer table with unique datatx_info
# column from the existing request_types.
- sequencer_form_definition_id = create_sequencer_form_definition()
+ sequencer_form_definition_id = create_sequencer_form_definition( migrate_engine )
sequencer_index = 1
for row in results_list:
request_type_id = row[0]
- request_type_name = row[1]
- sequencer_info = str( row[2] ) # datatx_info column
+ sequencer_info = str( row[2] ) # datatx_info column
# skip if sequencer_info is empty
if not sequencer_info.strip() or sequencer_info in ['None', 'null']:
continue
@@ -237,12 +238,12 @@ def upgrade(migrate_engine):
# proceed only if sequencer_info is a valid dict
if sequencer_info and type( sequencer_info ) == type( dict() ):
# check if this sequencer has already been added to the sequencer table
- sequencer_id = get_sequencer_id( sequencer_info )
+ sequencer_id = get_sequencer_id( migrate_engine, sequencer_info )
if not sequencer_id:
# add to the sequencer table
- sequencer_id = add_sequencer( sequencer_index, sequencer_form_definition_id, sequencer_info )
+ sequencer_id = add_sequencer( migrate_engine, sequencer_index, sequencer_form_definition_id, sequencer_info )
# now update the sequencer_id column in request_type table
- update_sequencer_id_in_request_type( request_type_id, sequencer_id )
+ update_sequencer_id_in_request_type( migrate_engine, request_type_id, sequencer_id )
sequencer_index = sequencer_index + 1
# Finally delete the 'datatx_info' column from the request_type table
@@ -279,34 +280,16 @@ def downgrade(migrate_engine):
request_type_id = row[0]
seq_values = loads( str( row[1] ) )
# create the datatx_info json dict
- datatx_info = dumps( dict( host = seq_values.get( 'field_0', '' ),
- username = seq_values.get( 'field_1', '' ),
- password = seq_values.get( 'field_2', '' ),
- data_dir = seq_values.get( 'field_3', '' ),
- rename_dataset = seq_values.get( 'field_4', '' ) ) )
+ datatx_info = dumps( dict( host=seq_values.get( 'field_0', '' ),
+ username=seq_values.get( 'field_1', '' ),
+ password=seq_values.get( 'field_2', '' ),
+ data_dir=seq_values.get( 'field_3', '' ),
+ rename_dataset=seq_values.get( 'field_4', '' ) ) )
# update the column
- cmd = "UPDATE request_type SET datatx_info='%s' WHERE id=%i" %( datatx_info, request_type_id )
+ cmd = "UPDATE request_type SET datatx_info='%s' WHERE id=%i" % ( datatx_info, request_type_id )
migrate_engine.execute( cmd )
# delete foreign key field to the sequencer table in the request_type table
try:
RequestType_table.c.sequencer_id.drop()
except Exception, e:
log.debug( "Deleting column 'sequencer_id' in the 'request_type' table failed: %s" % ( str( e ) ) )
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
diff --git a/lib/galaxy/model/migrate/versions/0068_rename_sequencer_to_external_services.py b/lib/galaxy/model/migrate/versions/0068_rename_sequencer_to_external_services.py
index ec05e516239..f9ea4312457 100644
--- a/lib/galaxy/model/migrate/versions/0068_rename_sequencer_to_external_services.py
+++ b/lib/galaxy/model/migrate/versions/0068_rename_sequencer_to_external_services.py
@@ -6,30 +6,25 @@ The 'sequencer_type_id' column is renamed to 'external_service_type_id' in the r
table 'external_service'. Finally, adds a foreign key to the external_service table in the
sample_dataset table and populates it.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-from sqlalchemy.exc import *
-
-from galaxy.model.custom_types import *
-
-from galaxy.util.json import loads, dumps
-
import datetime
-now = datetime.datetime.utcnow
-
import logging
+
+from migrate import ForeignKeyConstraint
+from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, MetaData, Table, TEXT
+from sqlalchemy.exc import NoSuchTableError
+
+# Need our custom types, but don't import anything else from model
+from galaxy.model.custom_types import TrimmedString
+
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
-
metadata = MetaData()
-#migrate_engine = scoped_session( sessionmaker( bind=migrate_engine, autoflush=False, autocommit=True ) )
-def nextval( table, col='id' ):
- if migrate_engine.name == 'postgres':
+
+def nextval( migrate_engine, table, col='id' ):
+ if migrate_engine.name in ['postgres', 'postgresql']:
return "nextval('%s_%s_seq')" % ( table, col )
- elif migrate_engine.name == 'mysql' or migrate_engine.name == 'sqlite':
+ elif migrate_engine.name in ['mysql', 'sqlite']:
return "null"
else:
raise Exception( 'Unable to convert data for unknown database type: %s' % migrate_engine.name )
@@ -150,7 +145,7 @@ def upgrade(migrate_engine):
if not sequencer_id:
sequencer_id = 'null'
cmd = "INSERT INTO request_type_external_service_association VALUES ( %s, %s, %s )"
- cmd = cmd % ( nextval( 'request_type_external_service_association' ),
+ cmd = cmd % ( nextval( migrate_engine, 'request_type_external_service_association' ),
request_type_id,
sequencer_id )
migrate_engine.execute( cmd )
@@ -160,30 +155,22 @@ def upgrade(migrate_engine):
# In sqlite, create a temp table without the column that needs to be removed.
# then copy all the rows from the original table and finally rename the temp table
RequestTypeTemp_table = Table( 'request_type_temp', metadata,
- Column( "id", Integer, primary_key=True),
- Column( "create_time", DateTime, default=now ),
- Column( "update_time", DateTime, default=now, onupdate=now ),
- Column( "name", TrimmedString( 255 ), nullable=False ),
- Column( "desc", TEXT ),
- Column( "request_form_id", Integer, ForeignKey( "form_definition.id" ), index=True ),
- Column( "sample_form_id", Integer, ForeignKey( "form_definition.id" ), index=True ),
- Column( "deleted", Boolean, index=True, default=False ) )
+ Column( "id", Integer, primary_key=True),
+ Column( "create_time", DateTime, default=now ),
+ Column( "update_time", DateTime, default=now, onupdate=now ),
+ Column( "name", TrimmedString( 255 ), nullable=False ),
+ Column( "desc", TEXT ),
+ Column( "request_form_id", Integer, ForeignKey( "form_definition.id" ), index=True ),
+ Column( "sample_form_id", Integer, ForeignKey( "form_definition.id" ), index=True ),
+ Column( "deleted", Boolean, index=True, default=False ) )
try:
RequestTypeTemp_table.create()
except Exception, e:
log.debug( "Creating request_type_temp table failed: %s" % str( e ) )
# insert all the rows from the request table to the request_temp table
- cmd = \
- "INSERT INTO request_type_temp " + \
- "SELECT id," + \
- "create_time," + \
- "update_time," + \
- "name," + \
- "desc," + \
- "request_form_id," + \
- "sample_form_id," + \
- "deleted " + \
- "FROM request_type;"
+ cmd = "INSERT INTO request_type_temp SELECT id, create_time," + \
+ "update_time, name, desc, request_form_id, sample_form_id," + \
+ "deleted FROM request_type;"
migrate_engine.execute( cmd )
# delete the 'request_type' table
try:
@@ -269,7 +256,7 @@ def downgrade(migrate_engine):
cmd = "ALTER TABLE external_service RENAME TO sequencer"
migrate_engine.execute( cmd )
# if running postgres then rename the primary key sequence too
- if migrate_engine.name == 'postgres':
+ if migrate_engine.name in ['postgres', 'postgresql']:
cmd = "ALTER SEQUENCE external_service_id_seq RENAME TO sequencer_id_seq"
migrate_engine.execute( cmd )
# drop the 'external_service_id' column in the 'sample_dataset' table
@@ -284,6 +271,3 @@ def downgrade(migrate_engine):
SampleDataset_table.c.external_service_id.drop()
except Exception, e:
log.debug( "Deleting column 'external_service_id' from the 'sample_dataset' table failed: %s" % ( str( e ) ) )
-
-
-
diff --git a/lib/galaxy/model/migrate/versions/0069_rename_sequencer_form_type.py b/lib/galaxy/model/migrate/versions/0069_rename_sequencer_form_type.py
index 337b3cd1742..c92c7a554bb 100644
--- a/lib/galaxy/model/migrate/versions/0069_rename_sequencer_form_type.py
+++ b/lib/galaxy/model/migrate/versions/0069_rename_sequencer_form_type.py
@@ -1,23 +1,16 @@
"""
Migration script to rename the sequencer information form type to external service information form
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-from sqlalchemy.exc import *
-
-from galaxy.model.custom_types import *
-
import datetime
-now = datetime.datetime.utcnow
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import MetaData
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -28,7 +21,6 @@ def upgrade(migrate_engine):
migrate_engine.execute( cmd )
-
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0070_add_info_column_to_deferred_job_table.py b/lib/galaxy/model/migrate/versions/0070_add_info_column_to_deferred_job_table.py
index 3f7f72d243d..edaa0c7f554 100644
--- a/lib/galaxy/model/migrate/versions/0070_add_info_column_to_deferred_job_table.py
+++ b/lib/galaxy/model/migrate/versions/0070_add_info_column_to_deferred_job_table.py
@@ -1,17 +1,14 @@
"""
Migration script to add 'info' column to the transfer_job table.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, MetaData, Table, TEXT
+
+log = logging.getLogger( __name__ )
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -25,6 +22,7 @@ def upgrade(migrate_engine):
print "Adding info column to transfer_job table failed: %s" % str( e )
log.debug( "Adding info column to transfer_job table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0071_add_history_and_workflow_to_sample.py b/lib/galaxy/model/migrate/versions/0071_add_history_and_workflow_to_sample.py
index 710f10e6bae..a8f2db7b0e2 100644
--- a/lib/galaxy/model/migrate/versions/0071_add_history_and_workflow_to_sample.py
+++ b/lib/galaxy/model/migrate/versions/0071_add_history_and_workflow_to_sample.py
@@ -1,18 +1,16 @@
"""
Migration script to add 'workflow' and 'history' columns for a sample.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-from galaxy.model.custom_types import *
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, ForeignKey, Integer, MetaData, Table
+
+from galaxy.model.custom_types import JSONType
+
+log = logging.getLogger( __name__ )
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -29,6 +27,7 @@ def upgrade(migrate_engine):
print "Adding history and workflow columns to sample table failed: %s" % str( e )
log.debug( "Adding history and workflow columns to sample table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0072_add_pid_and_socket_columns_to_transfer_job_table.py b/lib/galaxy/model/migrate/versions/0072_add_pid_and_socket_columns_to_transfer_job_table.py
index 9dbeea7d91d..ef05650c0ba 100644
--- a/lib/galaxy/model/migrate/versions/0072_add_pid_and_socket_columns_to_transfer_job_table.py
+++ b/lib/galaxy/model/migrate/versions/0072_add_pid_and_socket_columns_to_transfer_job_table.py
@@ -1,17 +1,14 @@
"""
Migration script to add 'pid' and 'socket' columns to the transfer_job table.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, Integer, MetaData, Table
+
+log = logging.getLogger( __name__ )
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -28,6 +25,7 @@ def upgrade(migrate_engine):
print "Adding columns to transfer_job table failed: %s" % str( e )
log.debug( "Adding columns to transfer_job table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0073_add_ldda_to_implicit_conversion_table.py b/lib/galaxy/model/migrate/versions/0073_add_ldda_to_implicit_conversion_table.py
index e6ba0ea03b3..4d599172ad1 100644
--- a/lib/galaxy/model/migrate/versions/0073_add_ldda_to_implicit_conversion_table.py
+++ b/lib/galaxy/model/migrate/versions/0073_add_ldda_to_implicit_conversion_table.py
@@ -1,17 +1,14 @@
"""
Migration script to add 'ldda_parent_id' column to the implicitly_converted_dataset_association table.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, ForeignKey, Integer, MetaData, Table
+
+log = logging.getLogger( __name__ )
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -21,7 +18,7 @@ def upgrade(migrate_engine):
if migrate_engine.name != 'sqlite':
c = Column( "ldda_parent_id", Integer, ForeignKey( "library_dataset_dataset_association.id" ), index=True, nullable=True )
else:
- #Can't use the ForeignKey in sqlite.
+ # Can't use the ForeignKey in sqlite.
c = Column( "ldda_parent_id", Integer, index=True, nullable=True )
c.create( Implicitly_converted_table, index_name="ix_implicitly_converted_dataset_assoc_ldda_parent_id")
assert c is Implicitly_converted_table.c.ldda_parent_id
@@ -29,6 +26,7 @@ def upgrade(migrate_engine):
print "Adding ldda_parent_id column to implicitly_converted_dataset_association table failed: %s" % str( e )
log.debug( "Adding ldda_parent_id column to implicitly_converted_dataset_association table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0074_add_purged_column_to_library_dataset_table.py b/lib/galaxy/model/migrate/versions/0074_add_purged_column_to_library_dataset_table.py
index 16bbb6f088f..f2168d3cb0f 100644
--- a/lib/galaxy/model/migrate/versions/0074_add_purged_column_to_library_dataset_table.py
+++ b/lib/galaxy/model/migrate/versions/0074_add_purged_column_to_library_dataset_table.py
@@ -1,32 +1,31 @@
"""
Migration script to add 'purged' column to the library_dataset table.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Boolean, Column, MetaData, Table
+
+log = logging.getLogger( __name__ )
metadata = MetaData()
+
def boolean_false(migrate_engine):
- if migrate_engine.name in ['postgres','mysql', 'postgresql']:
- return False
- elif migrate_engine.name == 'sqlite':
- return 0
- else:
- raise Exception( 'Unable to set True data value for unknown database type: %s' % str( migrate_engine.name ) )
+ if migrate_engine.name in ['mysql', 'postgres', 'postgresql']:
+ return False
+ elif migrate_engine.name == 'sqlite':
+ return 0
+ else:
+ raise Exception( 'Unable to set True data value for unknown database type: %s' % str( migrate_engine.name ) )
+
def boolean_true(migrate_engine):
- if migrate_engine.name in ['postgres','mysql', 'postgresql']:
- return True
- elif migrate_engine.name == 'sqlite':
- return 1
- else:
- raise Exception( 'Unable to set False data value for unknown database type: %s' % str( migrate_engine.name ) )
+ if migrate_engine.name in ['mysql', 'postgres', 'postgresql']:
+ return True
+ elif migrate_engine.name == 'sqlite':
+ return 1
+ else:
+ raise Exception( 'Unable to set False data value for unknown database type: %s' % str( migrate_engine.name ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
@@ -58,6 +57,7 @@ def upgrade(migrate_engine):
cmd = "UPDATE library_dataset SET purged = %s WHERE id = %d;" % ( boolean_true(migrate_engine), int( row.id ) )
migrate_engine.execute( cmd )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0075_add_subindex_column_to_run_table.py b/lib/galaxy/model/migrate/versions/0075_add_subindex_column_to_run_table.py
index 172181ffbef..651246407ea 100644
--- a/lib/galaxy/model/migrate/versions/0075_add_subindex_column_to_run_table.py
+++ b/lib/galaxy/model/migrate/versions/0075_add_subindex_column_to_run_table.py
@@ -1,16 +1,13 @@
"""
Migration script to add a 'subindex' column to the run table.
"""
+from sqlalchemy import Column, MetaData, Table
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
-from galaxy.model.custom_types import *
+from galaxy.model.custom_types import TrimmedString
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -23,6 +20,7 @@ def upgrade(migrate_engine):
except Exception, e:
print "Adding the subindex column to the run table failed: ", str( e )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0076_fix_form_values_data_corruption.py b/lib/galaxy/model/migrate/versions/0076_fix_form_values_data_corruption.py
index 84202ca2a4e..bc90ec4769a 100644
--- a/lib/galaxy/model/migrate/versions/0076_fix_form_values_data_corruption.py
+++ b/lib/galaxy/model/migrate/versions/0076_fix_form_values_data_corruption.py
@@ -2,31 +2,16 @@
This migration script fixes the data corruption caused in the form_values
table (content json field) by migrate script 65.
'''
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-from sqlalchemy.exc import *
-import binascii
-
-from galaxy.util.json import loads, dumps
-
import logging
-log = logging.getLogger( __name__ )
+from json import dumps, loads
+from sqlalchemy import MetaData
+
+from galaxy.model.custom_types import _sniffnfix_pg9_hex
+
+log = logging.getLogger( __name__ )
metadata = MetaData()
-def _sniffnfix_pg9_hex(value):
- """
- Sniff for and fix postgres 9 hex decoding issue
- """
- try:
- if value[0] == 'x':
- return binascii.unhexlify(value[1:])
- else:
- return value
- except Exception, ex:
- return value
def upgrade(migrate_engine):
metadata.bind = migrate_engine
@@ -43,13 +28,13 @@ def upgrade(migrate_engine):
# if that fails, it means that the content field is corrupted.
try:
field_values_dict = loads( _sniffnfix_pg9_hex( str( row['field_values'] ) ) )
- except Exception, e:
+ except Exception:
corrupted_rows = corrupted_rows + 1
# content field is corrupted
fields_list = loads( _sniffnfix_pg9_hex( str( row['fdfields'] ) ) )
field_values_str = _sniffnfix_pg9_hex( str( row['field_values'] ) )
try:
- #Encoding errors? Just to be safe.
+ # Encoding errors? Just to be safe.
print "Attempting to fix row %s" % row['id']
print "Prior to replacement: %s" % field_values_str
except:
@@ -66,16 +51,16 @@ def upgrade(migrate_engine):
print "The 'content' field of row 'id' %i does not have the field '%s' in the 'form_values' table and could not be fixed by this migration script." % ( int( field['id'] ), field['name'] )
else:
# check if this is the last field
- if index == len( fields_list )-1:
+ if index == len( fields_list ) - 1:
# since this is the last field, the value string lies between the
- # field name and the '"}' string at the end, hence len(field_values_str)-2
- value = field_values_str[ field_index+len( field_name_key ):len( field_values_str )-2 ]
+ # field name and the '"}' string at the end, hence len(field_values_str) - 2
+ value = field_values_str[ field_index + len( field_name_key ):len( field_values_str ) - 2 ]
else:
# if this is not the last field then the value string lies between
# this field name and the next field name
- next_field = fields_list[index+1]
+ next_field = fields_list[index + 1]
next_field_index = field_values_str.find( '", "%s": "' % next_field['name'] )
- value = field_values_str[ field_index+len( field_name_key ):next_field_index ]
+ value = field_values_str[ field_index + len( field_name_key ):next_field_index ]
# clean up the value string, escape the required quoutes and newline characters
value = value.replace( "'", "\''" )\
.replace( '"', '\\\\"' )\
@@ -86,7 +71,7 @@ def upgrade(migrate_engine):
field_values_dict[ field['name'] ] = value
# update the db
json_values = dumps(field_values_dict)
- cmd = "UPDATE form_values SET content='%s' WHERE id=%i" %( json_values, int( row['id'] ) )
+ cmd = "UPDATE form_values SET content='%s' WHERE id=%i" % ( json_values, int( row['id'] ) )
migrate_engine.execute( cmd )
try:
print "Post replacement: %s" % json_values
@@ -97,7 +82,7 @@ def upgrade(migrate_engine):
else:
print 'No corrupted rows found.'
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
pass
-
diff --git a/lib/galaxy/model/migrate/versions/0077_create_tool_tag_association_table.py b/lib/galaxy/model/migrate/versions/0077_create_tool_tag_association_table.py
index fbb8b873600..c85a080d362 100644
--- a/lib/galaxy/model/migrate/versions/0077_create_tool_tag_association_table.py
+++ b/lib/galaxy/model/migrate/versions/0077_create_tool_tag_association_table.py
@@ -1,20 +1,15 @@
"""
Migration script to create table for storing tool tag associations.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
-from galaxy.model.custom_types import *
-
import datetime
-now = datetime.datetime.utcnow
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, ForeignKey, Integer, MetaData, Table
+
+from galaxy.model.custom_types import TrimmedString
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
# Table to add
@@ -28,6 +23,7 @@ ToolTagAssociation_table = Table( "tool_tag_association", metadata,
Column( "value", TrimmedString(255), index=True),
Column( "user_value", TrimmedString(255), index=True) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -39,6 +35,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.error( "Creating tool_tag_association table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0078_add_columns_for_disk_usage_accounting.py b/lib/galaxy/model/migrate/versions/0078_add_columns_for_disk_usage_accounting.py
index babff8d59c9..cd780cc984d 100644
--- a/lib/galaxy/model/migrate/versions/0078_add_columns_for_disk_usage_accounting.py
+++ b/lib/galaxy/model/migrate/versions/0078_add_columns_for_disk_usage_accounting.py
@@ -3,17 +3,14 @@ Migration script to add 'total_size' column to the dataset table, 'purged'
column to the HDA table, and 'disk_usage' column to the User and GalaxySession
tables.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Boolean, Column, MetaData, Numeric, Table
+
+log = logging.getLogger( __name__ )
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -56,6 +53,7 @@ def upgrade(migrate_engine):
print "Adding disk_usage column to galaxy_session table failed: %s" % str( e )
log.debug( "Adding disk_usage column to galaxy_session table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0079_input_library_to_job_table.py b/lib/galaxy/model/migrate/versions/0079_input_library_to_job_table.py
index b4c43290c20..d8dd928cde7 100644
--- a/lib/galaxy/model/migrate/versions/0079_input_library_to_job_table.py
+++ b/lib/galaxy/model/migrate/versions/0079_input_library_to_job_table.py
@@ -1,13 +1,10 @@
"""
Migration script to add the job_to_input_library_dataset table.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import logging
+
+from sqlalchemy import Column, ForeignKey, Integer, MetaData, String, Table
+
log = logging.getLogger( __name__ )
metadata = MetaData()
@@ -18,6 +15,7 @@ JobToInputLibraryDatasetAssociation_table = Table( "job_to_input_library_dataset
Column( "ldda_id", Integer, ForeignKey( "library_dataset_dataset_association.id" ), index=True ),
Column( "name", String(255) ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -30,6 +28,7 @@ def upgrade(migrate_engine):
print "Creating job_to_input_library_dataset table failed: %s" % str( e )
log.debug( "Creating job_to_input_library_dataset table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0080_quota_tables.py b/lib/galaxy/model/migrate/versions/0080_quota_tables.py
index 4559f774f5b..352821797fe 100644
--- a/lib/galaxy/model/migrate/versions/0080_quota_tables.py
+++ b/lib/galaxy/model/migrate/versions/0080_quota_tables.py
@@ -1,20 +1,13 @@
"""
Migration script to create tables for disk quotas.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-# from galaxy.model.orm.ext.assignmapper import *
-# from galaxy.model.custom_types import *
-
import datetime
-now = datetime.datetime.utcnow
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import BigInteger, Boolean, Column, DateTime, ForeignKey, Integer, MetaData, String, Table, TEXT
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
# Tables to add
@@ -50,6 +43,7 @@ DefaultQuotaAssociation_table = Table( "default_quota_association", metadata,
Column( "type", String( 32 ), index=True, unique=True ),
Column( "quota_id", Integer, ForeignKey( "quota.id" ), index=True ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -79,18 +73,6 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "Creating default_quota_association table failed: %s" % str( e ) )
- # Create the default quota record
- #class Quota( object ):
- # def __init__( self, name, description, bytes, operation ):
- # self.name = name
- # self.description = description
- # self.bytes = bytes
- # self.operation = operation
- #assign_mapper( db_session, Quota, Quota_table )
- #default_quota = Quota( 'Default Quota', 'The base quota applied to all users', -1, '=' )
- #db_session.add( default_quota )
- #db_session.flush()
-
def downgrade(migrate_engine):
metadata.bind = migrate_engine
diff --git a/lib/galaxy/model/migrate/versions/0081_add_tool_version_to_hda_ldda.py b/lib/galaxy/model/migrate/versions/0081_add_tool_version_to_hda_ldda.py
index 795cfa7a073..d1d8bea507d 100644
--- a/lib/galaxy/model/migrate/versions/0081_add_tool_version_to_hda_ldda.py
+++ b/lib/galaxy/model/migrate/versions/0081_add_tool_version_to_hda_ldda.py
@@ -1,16 +1,11 @@
"""
Migration script to add a 'tool_version' column to the hda/ldda tables.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
-from galaxy.model.custom_types import *
+from sqlalchemy import Column, MetaData, Table, TEXT
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -29,6 +24,7 @@ def upgrade(migrate_engine):
except Exception, e:
print "Adding the tool_version column to the hda/ldda tables failed: ", str( e )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0082_add_tool_shed_repository_table.py b/lib/galaxy/model/migrate/versions/0082_add_tool_shed_repository_table.py
index dc4dead2198..2b2bac72d91 100644
--- a/lib/galaxy/model/migrate/versions/0082_add_tool_shed_repository_table.py
+++ b/lib/galaxy/model/migrate/versions/0082_add_tool_shed_repository_table.py
@@ -1,16 +1,16 @@
"""
Migration script to add the tool_shed_repository table.
"""
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-import sys, logging
-from galaxy.model.custom_types import *
-from sqlalchemy.exc import *
import datetime
-now = datetime.datetime.utcnow
+import logging
+import sys
+from sqlalchemy import Boolean, Column, DateTime, Integer, MetaData, Table, TEXT
+
+# Need our custom types, but don't import anything else from model
+from galaxy.model.custom_types import TrimmedString
+
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler( sys.stdout )
@@ -33,6 +33,7 @@ ToolShedRepository_table = Table( "tool_shed_repository", metadata,
Column( "changeset_revision", TrimmedString( 255 ), index=True ),
Column( "deleted", Boolean, index=True, default=False ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -42,6 +43,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "Creating tool_shed_repository table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0083_add_prepare_files_to_task.py b/lib/galaxy/model/migrate/versions/0083_add_prepare_files_to_task.py
index 0a0b75f293e..5c0693a3860 100644
--- a/lib/galaxy/model/migrate/versions/0083_add_prepare_files_to_task.py
+++ b/lib/galaxy/model/migrate/versions/0083_add_prepare_files_to_task.py
@@ -1,17 +1,14 @@
"""
Migration script to add 'prepare_input_files_cmd' column to the task table and to rename a column.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, MetaData, String, Table, TEXT
+
+log = logging.getLogger( __name__ )
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -26,7 +23,7 @@ def upgrade(migrate_engine):
log.debug( "Adding prepare_input_files_cmd column to task table failed: %s" % str( e ) )
try:
task_table = Table( "task", metadata, autoload=True )
- c = Column( "working_directory", String ( 1024 ), nullable=True )
+ c = Column( "working_directory", String( 1024 ), nullable=True )
c.create( task_table )
assert c is task_table.c.working_directory
except Exception, e:
@@ -39,6 +36,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "Deleting column 'part_file' from the 'task' table failed: %s" % ( str( e ) ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
@@ -56,7 +54,7 @@ def downgrade(migrate_engine):
log.debug( "Dropping working_directory column from task table failed: %s" % str( e ) )
try:
task_table = Table( "task", metadata, autoload=True )
- c = Column( "part_file", String ( 1024 ), nullable=True )
+ c = Column( "part_file", String( 1024 ), nullable=True )
c.create( task_table )
assert c is task_table.c.part_file
except Exception, e:
diff --git a/lib/galaxy/model/migrate/versions/0084_add_ldda_id_to_implicit_conversion_table.py b/lib/galaxy/model/migrate/versions/0084_add_ldda_id_to_implicit_conversion_table.py
index f3ae10e6a20..29f2d590688 100644
--- a/lib/galaxy/model/migrate/versions/0084_add_ldda_id_to_implicit_conversion_table.py
+++ b/lib/galaxy/model/migrate/versions/0084_add_ldda_id_to_implicit_conversion_table.py
@@ -1,17 +1,14 @@
"""
Migration script to add 'ldda_id' column to the implicitly_converted_dataset_association table.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, ForeignKey, Integer, MetaData, Table
+
+log = logging.getLogger( __name__ )
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -28,6 +25,7 @@ def upgrade(migrate_engine):
print "Adding ldda_id column to implicitly_converted_dataset_association table failed: %s" % str( e )
log.debug( "Adding ldda_id column to implicitly_converted_dataset_association table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0085_add_task_info.py b/lib/galaxy/model/migrate/versions/0085_add_task_info.py
index 92ccc6d2d78..2a39dae269e 100644
--- a/lib/galaxy/model/migrate/versions/0085_add_task_info.py
+++ b/lib/galaxy/model/migrate/versions/0085_add_task_info.py
@@ -1,31 +1,30 @@
"""
Migration script to add 'info' column to the task table.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import logging
-log = logging.getLogger( __name__ )
+
+from sqlalchemy import Column, MetaData, Table
+
from galaxy.model.custom_types import TrimmedString
+log = logging.getLogger( __name__ )
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
metadata.reflect()
try:
task_table = Table( "task", metadata, autoload=True )
- c = Column( "info", TrimmedString (255) , nullable=True )
+ c = Column( "info", TrimmedString(255) , nullable=True )
c.create( task_table )
assert c is task_table.c.info
except Exception, e:
print "Adding info column to table table failed: %s" % str( e )
log.debug( "Adding info column to task table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0086_add_tool_shed_repository_table_columns.py b/lib/galaxy/model/migrate/versions/0086_add_tool_shed_repository_table_columns.py
index 828bacd910b..da2ffda0802 100644
--- a/lib/galaxy/model/migrate/versions/0086_add_tool_shed_repository_table_columns.py
+++ b/lib/galaxy/model/migrate/versions/0086_add_tool_shed_repository_table_columns.py
@@ -1,18 +1,16 @@
"""
Migration script to add the metadata, update_available and includes_datatypes columns to the tool_shed_repository table.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import datetime
-now = datetime.datetime.utcnow
-# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
+import logging
+import sys
-import sys, logging
+from sqlalchemy import Boolean, Column, MetaData, Table
+
+# Need our custom types, but don't import anything else from model
+from galaxy.model.custom_types import JSONType
+
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler( sys.stdout )
@@ -23,12 +21,14 @@ log.addHandler( handler )
metadata = MetaData()
+
def get_default_false(migrate_engine):
- if migrate_engine.name == 'mysql' or migrate_engine.name == 'sqlite':
+ if migrate_engine.name in ['mysql', 'sqlite']:
return "0"
- elif migrate_engine.name in ['postgresql', 'postgres']:
+ elif migrate_engine.name in ['postgres', 'postgresql']:
return "false"
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -58,6 +58,7 @@ def upgrade(migrate_engine):
print "Adding update_available column to the tool_shed_repository table failed: %s" % str( e )
log.debug( "Adding update_available column to the tool_shed_repository table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0087_tool_id_guid_map_table.py b/lib/galaxy/model/migrate/versions/0087_tool_id_guid_map_table.py
index 7395772f03f..bfba9e0c6b8 100644
--- a/lib/galaxy/model/migrate/versions/0087_tool_id_guid_map_table.py
+++ b/lib/galaxy/model/migrate/versions/0087_tool_id_guid_map_table.py
@@ -1,18 +1,16 @@
"""
Migration script to create the tool_id_guid_map table.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import datetime
-now = datetime.datetime.utcnow
-# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
+import logging
+import sys
-import sys, logging
+from sqlalchemy import Column, DateTime, Integer, MetaData, String, Table, TEXT
+
+# Need our custom types, but don't import anything else from model
+from galaxy.model.custom_types import TrimmedString
+
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler( sys.stdout )
@@ -34,6 +32,7 @@ ToolIdGuidMap_table = Table( "tool_id_guid_map", metadata,
Column( "repository_name", TrimmedString( 255 ) ),
Column( "guid", TEXT, index=True, unique=True ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -43,6 +42,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "Creating tool_id_guid_map table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0088_add_installed_changeset_revison_column.py b/lib/galaxy/model/migrate/versions/0088_add_installed_changeset_revison_column.py
index eff95ca0076..9e4641ee439 100644
--- a/lib/galaxy/model/migrate/versions/0088_add_installed_changeset_revison_column.py
+++ b/lib/galaxy/model/migrate/versions/0088_add_installed_changeset_revison_column.py
@@ -1,18 +1,16 @@
"""
Migration script to add the installed_changeset_revision column to the tool_shed_repository table.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import datetime
-now = datetime.datetime.utcnow
-# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
+import logging
+import sys
-import sys, logging
+from sqlalchemy import Column, MetaData, Table
+
+# Need our custom types, but don't import anything else from model
+from galaxy.model.custom_types import TrimmedString
+
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler( sys.stdout )
@@ -23,6 +21,7 @@ log.addHandler( handler )
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -48,11 +47,13 @@ def upgrade(migrate_engine):
update_count = 0
for row in tool_shed_repositories:
cmd = "UPDATE tool_shed_repository " \
- + "SET installed_changeset_revision = '%s' " % row.changeset_revision \
+ + "SET installed_changeset_revision = '%s' " % row.changeset_revision \
+ "WHERE changeset_revision = '%s';" % row.changeset_revision
migrate_engine.execute( cmd )
update_count += 1
print "Updated the installed_changeset_revision column for ", update_count, " rows in the tool_shed_repository table. "
+
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0089_add_object_store_id_columns.py b/lib/galaxy/model/migrate/versions/0089_add_object_store_id_columns.py
index bd905bc648b..e146a126538 100644
--- a/lib/galaxy/model/migrate/versions/0089_add_object_store_id_columns.py
+++ b/lib/galaxy/model/migrate/versions/0089_add_object_store_id_columns.py
@@ -1,18 +1,16 @@
"""
Migration script to add 'object_store_id' column to various tables
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import logging
-log = logging.getLogger( __name__ )
+
+from sqlalchemy import Column, MetaData, Table
+
from galaxy.model.custom_types import TrimmedString
+log = logging.getLogger( __name__ )
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -27,6 +25,7 @@ def upgrade(migrate_engine):
print "Adding object_store_id column to %s table failed: %s" % ( t_name, str( e ) )
log.debug( "Adding object_store_id column to %s table failed: %s" % ( t_name, str( e ) ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0090_add_tool_shed_repository_table_columns.py b/lib/galaxy/model/migrate/versions/0090_add_tool_shed_repository_table_columns.py
index 5f412cb5a40..14d13bbf8ba 100644
--- a/lib/galaxy/model/migrate/versions/0090_add_tool_shed_repository_table_columns.py
+++ b/lib/galaxy/model/migrate/versions/0090_add_tool_shed_repository_table_columns.py
@@ -1,18 +1,13 @@
"""
Migration script to add the uninstalled and dist_to_shed columns to the tool_shed_repository table.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import datetime
-now = datetime.datetime.utcnow
-# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
+import logging
+import sys
-import sys, logging
+from sqlalchemy import Boolean, Column, MetaData, Table
+
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler( sys.stdout )
@@ -23,12 +18,14 @@ log.addHandler( handler )
metadata = MetaData()
+
def default_false(migrate_engine):
- if migrate_engine.name == 'mysql' or migrate_engine.name == 'sqlite':
+ if migrate_engine.name in ['mysql', 'sqlite']:
return "0"
- elif migrate_engine.name in ['postgresql', 'postgres']:
+ elif migrate_engine.name in ['postgres', 'postgresql']:
return "false"
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -49,6 +46,7 @@ def upgrade(migrate_engine):
except Exception, e:
print "Adding dist_to_shed column to the tool_shed_repository table failed: %s" % str( e )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0091_add_tool_version_tables.py b/lib/galaxy/model/migrate/versions/0091_add_tool_version_tables.py
index 5d43fdc509a..50ed32c10a2 100644
--- a/lib/galaxy/model/migrate/versions/0091_add_tool_version_tables.py
+++ b/lib/galaxy/model/migrate/versions/0091_add_tool_version_tables.py
@@ -1,20 +1,17 @@
"""
Migration script to create the tool_version and tool_version_association tables and drop the tool_id_guid_map table.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import datetime
-now = datetime.datetime.utcnow
-# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
-from galaxy.model.custom_types import _sniffnfix_pg9_hex
-from galaxy.util.json import loads, dumps
+import logging
+import sys
+from json import loads
-import sys, logging
+from sqlalchemy import Column, DateTime, ForeignKey, Integer, MetaData, String, Table, TEXT
+
+# Need our custom types, but don't import anything else from model
+from galaxy.model.custom_types import _sniffnfix_pg9_hex, TrimmedString
+
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler( sys.stdout )
@@ -24,23 +21,24 @@ handler.setFormatter( formatter )
log.addHandler( handler )
metadata = MetaData()
-#migrate_engine = scoped_session( sessionmaker( bind=migrate_engine, autoflush=False, autocommit=True ) )
-def nextval( table, col='id' ):
- if migrate_engine.name == 'postgres':
+
+def nextval( migrate_engine, table, col='id' ):
+ if migrate_engine.name in ['postgres', 'postgresql']:
return "nextval('%s_%s_seq')" % ( table, col )
- elif migrate_engine.name == 'mysql' or migrate_engine.name == 'sqlite':
+ elif migrate_engine.name in ['mysql', 'sqlite']:
return "null"
else:
raise Exception( 'Unable to convert data for unknown database type: %s' % migrate_engine.name )
-def localtimestamp():
- if migrate_engine.name == 'postgres' or migrate_engine.name == 'mysql':
- return "LOCALTIMESTAMP"
- elif migrate_engine.name == 'sqlite':
- return "current_date || ' ' || current_time"
- else:
- raise Exception( 'Unable to convert data for unknown database type: %s' % db )
+
+def localtimestamp(migrate_engine):
+ if migrate_engine.name in ['mysql', 'postgres', 'postgresql']:
+ return "LOCALTIMESTAMP"
+ elif migrate_engine.name == 'sqlite':
+ return "current_date || ' ' || current_time"
+ else:
+ raise Exception( 'Unable to convert data for unknown database type: %s' % migrate_engine.name )
ToolVersion_table = Table( "tool_version", metadata,
Column( "id", Integer, primary_key=True ),
@@ -54,6 +52,7 @@ ToolVersionAssociation_table = Table( "tool_version_association", metadata,
Column( "tool_id", Integer, ForeignKey( "tool_version.id" ), index=True, nullable=False ),
Column( "parent_id", Integer, ForeignKey( "tool_version.id" ), index=True, nullable=False ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -83,7 +82,7 @@ def upgrade(migrate_engine):
tools = repository_metadata.get( 'tools', [] )
for tool_dict in tools:
cmd = "INSERT INTO tool_version VALUES (%s, %s, %s, '%s', %s)" % \
- ( nextval( 'tool_version' ), localtimestamp(), localtimestamp(), tool_dict[ 'guid' ], tool_shed_repository_id )
+ ( nextval( migrate_engine, 'tool_version' ), localtimestamp( migrate_engine ), localtimestamp( migrate_engine ), tool_dict[ 'guid' ], tool_shed_repository_id )
migrate_engine.execute( cmd )
count += 1
print "Added %d rows to the new tool_version table." % count
@@ -93,6 +92,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "Dropping tool_id_guid_map table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
diff --git a/lib/galaxy/model/migrate/versions/0092_add_migrate_tools_table.py b/lib/galaxy/model/migrate/versions/0092_add_migrate_tools_table.py
index 2615ed56a26..b8b00eafff0 100644
--- a/lib/galaxy/model/migrate/versions/0092_add_migrate_tools_table.py
+++ b/lib/galaxy/model/migrate/versions/0092_add_migrate_tools_table.py
@@ -1,18 +1,16 @@
"""
Migration script to create the migrate_tools table.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import datetime
-now = datetime.datetime.utcnow
-# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
+import logging
+import sys
-import sys, logging
+from sqlalchemy import Column, Integer, MetaData, Table, TEXT
+
+# Need our custom types, but don't import anything else from model
+from galaxy.model.custom_types import TrimmedString
+
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler( sys.stdout )
@@ -28,6 +26,7 @@ MigrateTools_table = Table( "migrate_tools", metadata,
Column( "repository_path", TEXT ),
Column( "version", Integer ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -41,6 +40,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "Creating migrate_tools table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0093_add_job_params_col.py b/lib/galaxy/model/migrate/versions/0093_add_job_params_col.py
index 20912f44323..34ae580da50 100644
--- a/lib/galaxy/model/migrate/versions/0093_add_job_params_col.py
+++ b/lib/galaxy/model/migrate/versions/0093_add_job_params_col.py
@@ -1,27 +1,25 @@
"""
Migration script to create "params" column in job table.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import logging
-log = logging.getLogger( __name__ )
+
+from sqlalchemy import Column, MetaData, Table
# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
+from galaxy.model.custom_types import TrimmedString
+log = logging.getLogger( __name__ )
metadata = MetaData()
# Column to add.
params_col = Column( "params", TrimmedString(255), index=True )
+
def display_migration_details():
print ""
print "This migration script adds a 'params' column to the Job table."
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -37,6 +35,7 @@ def upgrade(migrate_engine):
print str(e)
log.debug( "Adding column 'params' to job table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0094_add_job_handler_col.py b/lib/galaxy/model/migrate/versions/0094_add_job_handler_col.py
index f8ad97dadd6..ba074730f5d 100644
--- a/lib/galaxy/model/migrate/versions/0094_add_job_handler_col.py
+++ b/lib/galaxy/model/migrate/versions/0094_add_job_handler_col.py
@@ -1,27 +1,25 @@
"""
Migration script to create "handler" column in job table.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import logging
-log = logging.getLogger( __name__ )
+
+from sqlalchemy import Column, MetaData, Table
# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
+from galaxy.model.custom_types import TrimmedString
+log = logging.getLogger( __name__ )
metadata = MetaData()
# Column to add.
handler_col = Column( "handler", TrimmedString(255), index=True )
+
def display_migration_details():
print ""
print "This migration script adds a 'handler' column to the Job table."
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -37,6 +35,7 @@ def upgrade(migrate_engine):
print str(e)
log.debug( "Adding column 'handler' to job table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0095_hda_subsets.py b/lib/galaxy/model/migrate/versions/0095_hda_subsets.py
index 58069cde418..8dadc1dcc08 100644
--- a/lib/galaxy/model/migrate/versions/0095_hda_subsets.py
+++ b/lib/galaxy/model/migrate/versions/0095_hda_subsets.py
@@ -1,15 +1,11 @@
"""
Migration script to create table for tracking history_dataset_association subsets.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, ForeignKey, Index, Integer, MetaData, Table, Unicode
+
+log = logging.getLogger( __name__ )
metadata = MetaData()
# Table to add.
@@ -18,8 +14,8 @@ HistoryDatasetAssociationSubset_table = Table( "history_dataset_association_subs
Column( "id", Integer, primary_key=True ),
Column( "history_dataset_association_id", Integer, ForeignKey( "history_dataset_association.id" ) ),
Column( "history_dataset_association_subset_id", Integer, ForeignKey( "history_dataset_association.id" ) ),
- Column( "location", Unicode(255), index=True)
-)
+ Column( "location", Unicode(255), index=True) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
@@ -43,6 +39,7 @@ def upgrade(migrate_engine):
print str(e)
log.debug( "Adding indices to table 'history_dataset_association_subset' table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0096_openid_provider.py b/lib/galaxy/model/migrate/versions/0096_openid_provider.py
index 15c3d993a76..52d7a1df8b2 100644
--- a/lib/galaxy/model/migrate/versions/0096_openid_provider.py
+++ b/lib/galaxy/model/migrate/versions/0096_openid_provider.py
@@ -2,19 +2,17 @@
Migration script to add column to openid table for provider.
Remove any OpenID entries with nonunique GenomeSpace Identifier
"""
+import logging
+
+from sqlalchemy import Column, MetaData, Table
-BAD_IDENTIFIER = 'https://identity.genomespace.org/identityServer/xrd.jsp'
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
from galaxy.model.custom_types import TrimmedString
-import logging
log = logging.getLogger( __name__ )
-
+BAD_IDENTIFIER = 'https://identity.genomespace.org/identityServer/xrd.jsp'
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -35,6 +33,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "Deleting bad Identifiers from galaxy_user_openid failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0097_add_ctx_rev_column.py b/lib/galaxy/model/migrate/versions/0097_add_ctx_rev_column.py
index 4d2acc5551a..b2cf86cd98f 100644
--- a/lib/galaxy/model/migrate/versions/0097_add_ctx_rev_column.py
+++ b/lib/galaxy/model/migrate/versions/0097_add_ctx_rev_column.py
@@ -1,18 +1,16 @@
"""
Migration script to add the ctx_rev column to the tool_shed_repository table.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import datetime
-now = datetime.datetime.utcnow
-# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
+import logging
+import sys
-import sys, logging
+from sqlalchemy import Column, MetaData, Table
+
+# Need our custom types, but don't import anything else from model
+from galaxy.model.custom_types import TrimmedString
+
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler( sys.stdout )
@@ -23,6 +21,7 @@ log.addHandler( handler )
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -34,6 +33,8 @@ def upgrade(migrate_engine):
assert col is ToolShedRepository_table.c.ctx_rev
except Exception, e:
print "Adding ctx_rev column to the tool_shed_repository table failed: %s" % str( e )
+
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0098_genome_index_tool_data_table.py b/lib/galaxy/model/migrate/versions/0098_genome_index_tool_data_table.py
index 826fcd612d0..8becd71717c 100644
--- a/lib/galaxy/model/migrate/versions/0098_genome_index_tool_data_table.py
+++ b/lib/galaxy/model/migrate/versions/0098_genome_index_tool_data_table.py
@@ -1,17 +1,13 @@
"""
Migration script to create the genome_index_tool_data table.
"""
-
-from sqlalchemy import *
-from migrate import *
-
import datetime
+import logging
+import sys
+
+from sqlalchemy import Column, DateTime, ForeignKey, Integer, MetaData, String, Table
+
now = datetime.datetime.utcnow
-
-# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
-
-import sys, logging
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler( sys.stdout )
@@ -33,8 +29,8 @@ GenomeIndexToolData_table = Table( "genome_index_tool_data", metadata,
Column( "created_time", DateTime, default=now ),
Column( "modified_time", DateTime, default=now, onupdate=now ),
Column( "indexer", String( 64 ) ),
- Column( "user_id", Integer, ForeignKey( "galaxy_user.id" ), index=True ),
- )
+ Column( "user_id", Integer, ForeignKey( "galaxy_user.id" ), index=True ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
@@ -46,6 +42,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "Creating genome_index_tool_data table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0099_add_tool_dependency_table.py b/lib/galaxy/model/migrate/versions/0099_add_tool_dependency_table.py
index 3567b41513f..c36fd5856f0 100644
--- a/lib/galaxy/model/migrate/versions/0099_add_tool_dependency_table.py
+++ b/lib/galaxy/model/migrate/versions/0099_add_tool_dependency_table.py
@@ -1,16 +1,15 @@
"""
Migration script to add the tool_dependency table.
"""
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-import sys, logging
-from galaxy.model.custom_types import *
-from sqlalchemy.exc import *
import datetime
-now = datetime.datetime.utcnow
+import logging
+import sys
+from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, MetaData, Table
+
+from galaxy.model.custom_types import TrimmedString
+
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
log.setLevel( logging.DEBUG )
handler = logging.StreamHandler( sys.stdout )
@@ -33,6 +32,7 @@ ToolDependency_table = Table( "tool_dependency", metadata,
Column( "type", TrimmedString( 40 ) ),
Column( "uninstalled", Boolean, default=False ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -42,6 +42,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "Creating tool_dependency table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0100_alter_tool_dependency_table_version_column.py b/lib/galaxy/model/migrate/versions/0100_alter_tool_dependency_table_version_column.py
index 95554ea2791..a79315c2778 100644
--- a/lib/galaxy/model/migrate/versions/0100_alter_tool_dependency_table_version_column.py
+++ b/lib/galaxy/model/migrate/versions/0100_alter_tool_dependency_table_version_column.py
@@ -1,18 +1,13 @@
"""
Migration script to alter the type of the tool_dependency.version column from TrimmedString(40) to Text.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import datetime
-now = datetime.datetime.utcnow
-# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
+import logging
+import sys
-import sys, logging
+from sqlalchemy import MetaData, Table
+
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler( sys.stdout )
@@ -20,16 +15,16 @@ format = "%(name)s %(levelname)s %(asctime)s %(message)s"
formatter = logging.Formatter( format )
handler.setFormatter( formatter )
log.addHandler( handler )
-
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
metadata.reflect()
- ToolDependency_table = Table( "tool_dependency", metadata, autoload=True )
+ Table( "tool_dependency", metadata, autoload=True )
# Change the tool_dependency table's version column from TrimmedString to Text.
- if migrate_engine.name in ['postgresql', 'postgres']:
+ if migrate_engine.name in ['postgres', 'postgresql']:
cmd = "ALTER TABLE tool_dependency ALTER COLUMN version TYPE Text;"
elif migrate_engine.name == 'mysql':
cmd = "ALTER TABLE tool_dependency MODIFY COLUMN version Text;"
@@ -48,6 +43,8 @@ def upgrade(migrate_engine):
migrate_engine.execute( cmd )
except Exception, e:
log.debug( "Altering tool_dependency.version column from TrimmedString(40) to Text failed: %s" % str( e ) )
+
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
# Not necessary to change column type Text to TrimmedString(40).
diff --git a/lib/galaxy/model/migrate/versions/0101_drop_installed_changeset_revision_column.py b/lib/galaxy/model/migrate/versions/0101_drop_installed_changeset_revision_column.py
index 9fc373d4560..982065ffb26 100644
--- a/lib/galaxy/model/migrate/versions/0101_drop_installed_changeset_revision_column.py
+++ b/lib/galaxy/model/migrate/versions/0101_drop_installed_changeset_revision_column.py
@@ -1,16 +1,14 @@
"""
Migration script to drop the installed_changeset_revision column from the tool_dependency table.
"""
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-import sys, logging
-from galaxy.model.custom_types import *
-from sqlalchemy.exc import *
import datetime
-now = datetime.datetime.utcnow
+import logging
+import sys
+from sqlalchemy import MetaData, Table
+from sqlalchemy.exc import NoSuchTableError
+
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
log.setLevel( logging.DEBUG )
handler = logging.StreamHandler( sys.stdout )
@@ -21,6 +19,7 @@ log.addHandler( handler )
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -36,6 +35,8 @@ def upgrade(migrate_engine):
col.drop()
except Exception, e:
log.debug( "Dropping column 'installed_changeset_revision' from tool_dependency table failed: %s" % ( str( e ) ) )
+
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
pass
diff --git a/lib/galaxy/model/migrate/versions/0102_add_tool_dependency_status_columns.py b/lib/galaxy/model/migrate/versions/0102_add_tool_dependency_status_columns.py
index 54eb23d2fb0..24b3802c597 100644
--- a/lib/galaxy/model/migrate/versions/0102_add_tool_dependency_status_columns.py
+++ b/lib/galaxy/model/migrate/versions/0102_add_tool_dependency_status_columns.py
@@ -1,18 +1,16 @@
"""
Migration script to add status and error_message columns to the tool_dependency table and drop the uninstalled column from the tool_dependency table.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import datetime
-now = datetime.datetime.utcnow
-# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
+import logging
+import sys
-import sys, logging
+from sqlalchemy import Boolean, Column, MetaData, Table, TEXT
+
+# Need our custom types, but don't import anything else from model
+from galaxy.model.custom_types import TrimmedString
+
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler( sys.stdout )
@@ -23,6 +21,7 @@ log.addHandler( handler )
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -45,12 +44,14 @@ def upgrade(migrate_engine):
print "Adding error_message column to the tool_dependency table failed: %s" % str( e )
if migrate_engine.name != 'sqlite':
- #This breaks in sqlite due to failure to drop check constraint.
+ # This breaks in sqlite due to failure to drop check constraint.
# TODO move to alembic.
try:
ToolDependency_table.c.uninstalled.drop()
except Exception, e:
print "Dropping uninstalled column from the tool_dependency table failed: %s" % str( e )
+
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0103_add_tool_shed_repository_status_columns.py b/lib/galaxy/model/migrate/versions/0103_add_tool_shed_repository_status_columns.py
index ff45a2426b7..4e8a0e7dde1 100644
--- a/lib/galaxy/model/migrate/versions/0103_add_tool_shed_repository_status_columns.py
+++ b/lib/galaxy/model/migrate/versions/0103_add_tool_shed_repository_status_columns.py
@@ -1,17 +1,15 @@
"""Migration script to add status and error_message columns to the tool_shed_repository table."""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import datetime
-now = datetime.datetime.utcnow
-# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
+from sqlalchemy import Column, MetaData, Table, TEXT
+
+# Need our custom types, but don't import anything else from model
+from galaxy.model.custom_types import TrimmedString
+
+now = datetime.datetime.utcnow
metadata = MetaData()
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -55,6 +53,8 @@ def upgrade(migrate_engine):
print "Exception executing sql command: "
print cmd
print str( e )
+
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0104_update_genome_downloader_job_parameters.py b/lib/galaxy/model/migrate/versions/0104_update_genome_downloader_job_parameters.py
index 9a713781018..11e2ae94f92 100644
--- a/lib/galaxy/model/migrate/versions/0104_update_genome_downloader_job_parameters.py
+++ b/lib/galaxy/model/migrate/versions/0104_update_genome_downloader_job_parameters.py
@@ -1,18 +1,16 @@
"""
Migration script to update the deferred job parameters for liftover transfer jobs.
"""
-
import datetime
-import logging, sys
+import logging
+import sys
+
+from sqlalchemy import Column, DateTime, Integer, MetaData, String, Table
+from sqlalchemy.orm import mapper, scoped_session, sessionmaker
+
+# Need our custom types, but don't import anything else from model
from galaxy.model.custom_types import JSONType
from galaxy.util.bunch import Bunch
-from sqlalchemy import *
-from sqlalchemy import Integer, Table, MetaData, Column
-from sqlalchemy.orm import *
-from sqlalchemy.orm import scoped_session, sessionmaker
-from migrate import *
-# Need our custom types, but don't import anything else from model
-
now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
@@ -26,18 +24,21 @@ log.addHandler( handler )
metadata = MetaData()
context = scoped_session( sessionmaker( autoflush=False, autocommit=True ) )
+
class DeferredJob( object ):
- states = Bunch( NEW = 'new',
- WAITING = 'waiting',
- QUEUED = 'queued',
- RUNNING = 'running',
- OK = 'ok',
- ERROR = 'error' )
+ states = Bunch( NEW='new',
+ WAITING='waiting',
+ QUEUED='queued',
+ RUNNING='running',
+ OK='ok',
+ ERROR='error' )
+
def __init__( self, state=None, plugin=None, params=None ):
self.state = state
self.plugin = plugin
self.params = params
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
@@ -49,7 +50,7 @@ def upgrade(migrate_engine):
Column( "plugin", String( 128 ), index=True ),
Column( "params", JSONType ) )
- mapper( DeferredJob, DeferredJob.table, properties = {} )
+ mapper( DeferredJob, DeferredJob.table, properties={} )
liftoverjobs = dict()
@@ -67,6 +68,7 @@ def upgrade(migrate_engine):
context.flush()
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
diff --git a/lib/galaxy/model/migrate/versions/0105_add_cleanup_event_table.py b/lib/galaxy/model/migrate/versions/0105_add_cleanup_event_table.py
index 0216567cca2..0a99a18c6f2 100644
--- a/lib/galaxy/model/migrate/versions/0105_add_cleanup_event_table.py
+++ b/lib/galaxy/model/migrate/versions/0105_add_cleanup_event_table.py
@@ -1,16 +1,15 @@
"""
Migration script to add the cleanup_event* tables.
"""
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-import sys, logging
-from galaxy.model.custom_types import *
-from sqlalchemy.exc import *
import datetime
-now = datetime.datetime.utcnow
+import logging
+import sys
+from sqlalchemy import Column, DateTime, ForeignKey, Integer, MetaData, Table
+
+from galaxy.model.custom_types import TrimmedString
+
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
log.setLevel( logging.DEBUG )
handler = logging.StreamHandler( sys.stdout )
@@ -81,6 +80,7 @@ CleanupEventImplicitlyConvertedDatasetAssociationAssociation_table = Table( "cle
Column( "cleanup_event_id", Integer, ForeignKey( "cleanup_event.id" ), index=True, nullable=True ),
Column( "icda_id", Integer, ForeignKey( "implicitly_converted_dataset_association.id" ), index=True ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
print __doc__
@@ -99,6 +99,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "Creating table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0106_add_missing_indexes.py b/lib/galaxy/model/migrate/versions/0106_add_missing_indexes.py
index 529eb93dbe9..0bf4922b4c4 100644
--- a/lib/galaxy/model/migrate/versions/0106_add_missing_indexes.py
+++ b/lib/galaxy/model/migrate/versions/0106_add_missing_indexes.py
@@ -1,23 +1,13 @@
"""
Migration script to create missing indexes. Adding new columns to existing tables via SQLAlchemy does not create the index, even if the column definition includes index=True.
"""
+import logging
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from sqlalchemy.exc import ProgrammingError
+from sqlalchemy import Index, MetaData, Table
from sqlalchemy.engine import reflection
-from migrate import *
-from migrate.changeset import *
-import sys, logging
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
-handler = logging.StreamHandler( sys.stdout )
-format = "%(name)s %(levelname)s %(asctime)s %(message)s"
-formatter = logging.Formatter( format )
-handler.setFormatter( formatter )
-log.addHandler( handler )
-
metadata = MetaData()
indexes = (
@@ -42,7 +32,7 @@ indexes = (
( "ix_job_imported", 'job', 'imported' ), # 0051
( "ix_request_notification", 'request', 'notification' ), # 0057
( "ix_sd_external_service_id", 'sample_dataset', 'external_service_id' ), # 0068
- ( "ix_icda_ldda_parent_id", 'implicitly_converted_dataset_association', 'ldda_parent_id' ), # 0073
+ ( "ix_icda_ldda_parent_id", 'implicitly_converted_dataset_association', 'ldda_parent_id' ), # 0073
( "ix_library_dataset_purged", 'library_dataset', 'purged' ), # 0074
( "ix_run_subindex", 'run', 'subindex' ), # 0075
( "ix_history_dataset_association_purged", 'history_dataset_association', 'purged' ), # 0078
@@ -57,6 +47,7 @@ indexes = (
( "ix_galaxy_user_email", 'galaxy_user', 'email' ) # 0106
)
+
def upgrade(migrate_engine):
print __doc__
metadata.bind = migrate_engine
@@ -70,10 +61,11 @@ def upgrade(migrate_engine):
if ix not in [ins_ix.get('name', None) for ins_ix in insp.get_indexes(table)]:
Index( ix, t.c[col] ).create()
else:
- pass #Index already exists, don't recreate.
+ pass # Index already exists, don't recreate.
except Exception, e:
log.error("Unable to create index '%s': %s" % (ix, str(e)))
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0107_add_exit_code_to_job_and_task.py b/lib/galaxy/model/migrate/versions/0107_add_exit_code_to_job_and_task.py
index 6c9b73b55b1..e68f355be30 100644
--- a/lib/galaxy/model/migrate/versions/0107_add_exit_code_to_job_and_task.py
+++ b/lib/galaxy/model/migrate/versions/0107_add_exit_code_to_job_and_task.py
@@ -1,18 +1,11 @@
"""
Add the exit_code column to the Job and Task tables.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import logging
+
+from sqlalchemy import Column, Integer, MetaData, Table
+
log = logging.getLogger( __name__ )
-
-# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
-
metadata = MetaData()
# There was a bug when only one column was used for both tables,
@@ -20,10 +13,12 @@ metadata = MetaData()
exit_code_job_col = Column( "exit_code", Integer, nullable=True )
exit_code_task_col = Column( "exit_code", Integer, nullable=True )
+
def display_migration_details():
print ""
print "This migration script adds a 'handler' column to the Job table."
+
def upgrade(migrate_engine):
print __doc__
metadata.bind = migrate_engine
@@ -49,6 +44,7 @@ def upgrade(migrate_engine):
log.error( "Adding column 'exit_code' to task table failed: %s" % str( e ) )
return
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
@@ -68,4 +64,3 @@ def downgrade(migrate_engine):
exit_code_col.drop()
except Exception, e:
log.debug( "Dropping 'exit_code' column from task table failed: %s" % ( str( e ) ) )
-
diff --git a/lib/galaxy/model/migrate/versions/0108_add_extended_metadata.py b/lib/galaxy/model/migrate/versions/0108_add_extended_metadata.py
index f9d16dcb21a..5da8e18e51f 100644
--- a/lib/galaxy/model/migrate/versions/0108_add_extended_metadata.py
+++ b/lib/galaxy/model/migrate/versions/0108_add_extended_metadata.py
@@ -1,16 +1,13 @@
"""
Add the ExtendedMetadata and ExtendedMetadataIndex tables
"""
+import logging
+
+from sqlalchemy import Column, ForeignKey, Integer, MetaData, String, Table, TEXT
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
from galaxy.model.custom_types import JSONType
-import logging
log = logging.getLogger( __name__ )
-
metadata = MetaData()
ExtendedMetadata_table = Table("extended_metadata", metadata,
@@ -22,7 +19,7 @@ ExtendedMetadataIndex_table = Table("extended_metadata_index", metadata,
Column( "extended_metadata_id", Integer, ForeignKey("extended_metadata.id",
onupdate="CASCADE",
ondelete="CASCADE" ),
- index=True ),
+ index=True ),
Column( "path", String( 255 )),
Column( "value", TEXT))
@@ -32,6 +29,7 @@ extended_metadata_ldda_col = Column( "extended_metadata_id", Integer, ForeignKey
def display_migration_details():
print "This migration script adds a ExtendedMetadata tables"
+
def upgrade(migrate_engine):
print __doc__
metadata.bind = migrate_engine
@@ -74,5 +72,3 @@ def downgrade(migrate_engine):
extended_metadata_id.drop()
except Exception, e:
log.debug( "Dropping 'extended_metadata_id' column from library_dataset_dataset_association table failed: %s" % ( str( e ) ) )
-
-
diff --git a/lib/galaxy/model/migrate/versions/0109_add_repository_dependency_tables.py b/lib/galaxy/model/migrate/versions/0109_add_repository_dependency_tables.py
index 3181648f9d1..32138cb8b12 100644
--- a/lib/galaxy/model/migrate/versions/0109_add_repository_dependency_tables.py
+++ b/lib/galaxy/model/migrate/versions/0109_add_repository_dependency_tables.py
@@ -1,16 +1,13 @@
"""
Migration script to add the repository_dependency and repository_repository_dependency_association tables.
"""
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-import sys, logging
-from galaxy.model.custom_types import *
-from sqlalchemy.exc import *
import datetime
-now = datetime.datetime.utcnow
+import logging
+import sys
+from sqlalchemy import Column, DateTime, ForeignKey, Integer, MetaData, Table
+
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
log.setLevel( logging.DEBUG )
handler = logging.StreamHandler( sys.stdout )
@@ -34,6 +31,7 @@ RepositoryRepositoryDependencyAssociation_table = Table( "repository_repository_
Column( "tool_shed_repository_id", Integer, ForeignKey( "tool_shed_repository.id" ), index=True ),
Column( "repository_dependency_id", Integer, ForeignKey( "repository_dependency.id" ), index=True ) )
+
def upgrade(migrate_engine):
print __doc__
metadata.bind = migrate_engine
@@ -47,6 +45,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "Creating repository_repository_dependency_association table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0110_add_dataset_uuid.py b/lib/galaxy/model/migrate/versions/0110_add_dataset_uuid.py
index 366419e1b55..d6eed4e5413 100644
--- a/lib/galaxy/model/migrate/versions/0110_add_dataset_uuid.py
+++ b/lib/galaxy/model/migrate/versions/0110_add_dataset_uuid.py
@@ -1,16 +1,13 @@
"""
Add UUID column to dataset table
"""
+import logging
+
+from sqlalchemy import Column, MetaData, Table
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
from galaxy.model.custom_types import UUIDType
-import logging
log = logging.getLogger( __name__ )
-
dataset_uuid_column = Column( "uuid", UUIDType, nullable=True )
@@ -18,6 +15,7 @@ def display_migration_details():
print ""
print "This migration adds uuid column to dataset table"
+
def upgrade(migrate_engine):
print __doc__
metadata = MetaData()
@@ -47,5 +45,3 @@ def downgrade(migrate_engine):
dataset_uuid.drop()
except Exception, e:
log.debug( "Dropping 'uuid' column from dataset table failed: %s" % ( str( e ) ) )
-
-
diff --git a/lib/galaxy/model/migrate/versions/0111_add_job_destinations.py b/lib/galaxy/model/migrate/versions/0111_add_job_destinations.py
index 0a28694bc6c..af2196a86f7 100644
--- a/lib/galaxy/model/migrate/versions/0111_add_job_destinations.py
+++ b/lib/galaxy/model/migrate/versions/0111_add_job_destinations.py
@@ -1,20 +1,20 @@
"""
Add support for job destinations to the job table
"""
+import logging
+
+from sqlalchemy import Column, MetaData, String, Table
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
from galaxy.model.custom_types import JSONType
-import logging
log = logging.getLogger( __name__ )
+
def display_migration_details():
print ""
print "This migration script adds 'destination_id' and 'destination_params' columns to the Job table."
+
def upgrade(migrate_engine):
print __doc__
metadata = MetaData()
@@ -36,6 +36,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.error( "Adding column 'destination_params' to job table failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
metadata = MetaData()
metadata.bind = migrate_engine
diff --git a/lib/galaxy/model/migrate/versions/0112_add_data_manager_history_association_and_data_manager_job_association_tables.py b/lib/galaxy/model/migrate/versions/0112_add_data_manager_history_association_and_data_manager_job_association_tables.py
index b16ef973dd1..9a37fdb5bb7 100644
--- a/lib/galaxy/model/migrate/versions/0112_add_data_manager_history_association_and_data_manager_job_association_tables.py
+++ b/lib/galaxy/model/migrate/versions/0112_add_data_manager_history_association_and_data_manager_job_association_tables.py
@@ -1,25 +1,14 @@
"""
Migration script to add the data_manager_history_association table and data_manager_job_association.
"""
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-import sys, logging
-from galaxy.model.custom_types import *
-from sqlalchemy.exc import *
import datetime
-now = datetime.datetime.utcnow
+import logging
+from sqlalchemy import Column, DateTime, ForeignKey, Integer, MetaData, Table, TEXT
+
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
log.setLevel( logging.DEBUG )
-handler = logging.StreamHandler( sys.stdout )
-format = "%(name)s %(levelname)s %(asctime)s %(message)s"
-formatter = logging.Formatter( format )
-handler.setFormatter( formatter )
-log.addHandler( handler )
-
-
metadata = MetaData()
DataManagerHistoryAssociation_table = Table( "data_manager_history_association", metadata,
@@ -27,16 +16,15 @@ DataManagerHistoryAssociation_table = Table( "data_manager_history_association",
Column( "create_time", DateTime, default=now ),
Column( "update_time", DateTime, index=True, default=now, onupdate=now ),
Column( "history_id", Integer, ForeignKey( "history.id" ), index=True ),
- Column( "user_id", Integer, ForeignKey( "galaxy_user.id" ), index=True )
- )
+ Column( "user_id", Integer, ForeignKey( "galaxy_user.id" ), index=True ) )
DataManagerJobAssociation_table = Table( "data_manager_job_association", metadata,
Column( "id", Integer, primary_key=True),
Column( "create_time", DateTime, default=now ),
Column( "update_time", DateTime, index=True, default=now, onupdate=now ),
Column( "job_id", Integer, ForeignKey( "job.id" ), index=True ),
- Column( "data_manager_id", TEXT, index=True )
- )
+ Column( "data_manager_id", TEXT, index=True ) )
+
def upgrade(migrate_engine):
print __doc__
diff --git a/lib/galaxy/model/migrate/versions/0113_update_migrate_tools_table.py b/lib/galaxy/model/migrate/versions/0113_update_migrate_tools_table.py
index bc423a0a93c..e8835131667 100644
--- a/lib/galaxy/model/migrate/versions/0113_update_migrate_tools_table.py
+++ b/lib/galaxy/model/migrate/versions/0113_update_migrate_tools_table.py
@@ -1,18 +1,11 @@
"""
Migration script to update the migrate_tools.repository_path column to point to the new location lib/tool_shed/galaxy_install/migrate.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import datetime
-now = datetime.datetime.utcnow
-# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
+import logging
+import sys
-import sys, logging
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler( sys.stdout )
@@ -21,6 +14,7 @@ formatter = logging.Formatter( format )
handler.setFormatter( formatter )
log.addHandler( handler )
+
def upgrade(migrate_engine):
print __doc__
# Create the table.
@@ -30,6 +24,7 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "Updating migrate_tools.repository_path column to point to the new location lib/tool_shed/galaxy_install/migrate failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
try:
cmd = "UPDATE migrate_tools set repository_path='lib/galaxy/tool_shed/migrate';"
diff --git a/lib/galaxy/model/migrate/versions/0114_update_migrate_tools_table_again.py b/lib/galaxy/model/migrate/versions/0114_update_migrate_tools_table_again.py
index aab8bbfc55d..55902c1c816 100644
--- a/lib/galaxy/model/migrate/versions/0114_update_migrate_tools_table_again.py
+++ b/lib/galaxy/model/migrate/versions/0114_update_migrate_tools_table_again.py
@@ -1,18 +1,11 @@
"""
Migration script to update the migrate_tools.repository_path column to point to the new location lib/tool_shed/galaxy_install/migrate.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import datetime
-now = datetime.datetime.utcnow
-# Need our custom types, but don't import anything else from model
-from galaxy.model.custom_types import *
+import logging
+import sys
-import sys, logging
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler( sys.stdout )
@@ -31,10 +24,10 @@ def upgrade(migrate_engine):
except Exception, e:
log.debug( "Updating migrate_tools.repository_path column to point to the new location lib/tool_shed/galaxy_install/migrate failed: %s" % str( e ) )
+
def downgrade(migrate_engine):
try:
cmd = "UPDATE migrate_tools set repository_path='lib/galaxy/tool_shed/migrate';"
migrate_engine.execute( cmd )
except Exception, e:
log.debug( "Updating migrate_tools.repository_path column to point to the old location lib/galaxy/tool_shed/migrate failed: %s" % str( e ) )
-
diff --git a/lib/galaxy/model/migrate/versions/0115_longer_user_password_field.py b/lib/galaxy/model/migrate/versions/0115_longer_user_password_field.py
index 0efb653eb7c..c981e19ec6c 100644
--- a/lib/galaxy/model/migrate/versions/0115_longer_user_password_field.py
+++ b/lib/galaxy/model/migrate/versions/0115_longer_user_password_field.py
@@ -1,13 +1,13 @@
"""
Expand the length of the password fields in the galaxy_user table to allow for other hasing schemes
"""
-
-from sqlalchemy import *
-from migrate import *
-
import logging
+
+from sqlalchemy import MetaData, String, Table
+
log = logging.getLogger( __name__ )
+
def upgrade( migrate_engine ):
meta = MetaData( bind=migrate_engine )
user = Table( 'galaxy_user', meta, autoload=True )
@@ -16,6 +16,7 @@ def upgrade( migrate_engine ):
except:
log.exception( "Altering password column failed" )
+
def downgrade(migrate_engine):
meta = MetaData( bind=migrate_engine )
user = Table( 'galaxy_user', meta, autoload=True )
diff --git a/lib/galaxy/model/migrate/versions/0116_drop_update_available_col_add_tool_shed_status_col.py b/lib/galaxy/model/migrate/versions/0116_drop_update_available_col_add_tool_shed_status_col.py
index 5cec56f23cd..7d3017b0c07 100644
--- a/lib/galaxy/model/migrate/versions/0116_drop_update_available_col_add_tool_shed_status_col.py
+++ b/lib/galaxy/model/migrate/versions/0116_drop_update_available_col_add_tool_shed_status_col.py
@@ -1,17 +1,16 @@
"""
Migration script to drop the update_available Boolean column and replace it with the tool_shed_status JSONType column in the tool_shed_repository table.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-import sys, logging
-from galaxy.model.custom_types import *
-from sqlalchemy.exc import *
import datetime
-now = datetime.datetime.utcnow
+import logging
+import sys
+from sqlalchemy import Boolean, Column, MetaData, Table
+from sqlalchemy.exc import NoSuchTableError
+
+from galaxy.model.custom_types import JSONType
+
+now = datetime.datetime.utcnow
log = logging.getLogger( __name__ )
log.setLevel( logging.DEBUG )
handler = logging.StreamHandler( sys.stdout )
@@ -22,12 +21,14 @@ log.addHandler( handler )
metadata = MetaData()
+
def default_false( migrate_engine ):
- if migrate_engine.name == 'mysql' or migrate_engine.name == 'sqlite':
+ if migrate_engine.name in ['mysql', 'sqlite']:
return "0"
- elif migrate_engine.name in [ 'postgresql', 'postgres' ]:
+ elif migrate_engine.name in [ 'postgres', 'postgresql' ]:
return "false"
+
def upgrade( migrate_engine ):
metadata.bind = migrate_engine
print __doc__
@@ -52,6 +53,7 @@ def upgrade( migrate_engine ):
except Exception, e:
print "Adding tool_shed_status column to the tool_shed_repository table failed: %s" % str( e )
+
def downgrade( migrate_engine ):
metadata.bind = migrate_engine
metadata.reflect()
diff --git a/lib/galaxy/model/migrate/versions/0117_add_user_activation.py b/lib/galaxy/model/migrate/versions/0117_add_user_activation.py
index a55d26531ef..2b7b3c48f55 100644
--- a/lib/galaxy/model/migrate/versions/0117_add_user_activation.py
+++ b/lib/galaxy/model/migrate/versions/0117_add_user_activation.py
@@ -5,16 +5,13 @@ Created on Sep 10, 2013
Adds 'active' and 'activation_token' columns to the galaxy_user table.
'''
+import logging
+
+from sqlalchemy import Boolean, Column, MetaData, Table
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
from galaxy.model.custom_types import TrimmedString
-import logging
log = logging.getLogger( __name__ )
-
user_active_column = Column( "active", Boolean, default=True, nullable=True )
user_activation_token_column = Column( "activation_token", TrimmedString( 64 ), nullable=True )
diff --git a/lib/galaxy/model/migrate/versions/0118_add_hda_extended_metadata.py b/lib/galaxy/model/migrate/versions/0118_add_hda_extended_metadata.py
index 879f5ad8a41..3642fe41810 100644
--- a/lib/galaxy/model/migrate/versions/0118_add_hda_extended_metadata.py
+++ b/lib/galaxy/model/migrate/versions/0118_add_hda_extended_metadata.py
@@ -1,29 +1,24 @@
"""
Add link from history_dataset_association to the extended_metadata table
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-from galaxy.model.custom_types import JSONType
-
import logging
+
+from sqlalchemy import Column, ForeignKey, Integer, MetaData, Table
+
log = logging.getLogger( __name__ )
-
metadata = MetaData()
-
extended_metadata_hda_col = Column( "extended_metadata_id", Integer, ForeignKey("extended_metadata.id"), nullable=True )
def display_migration_details():
print "This migration script adds a ExtendedMetadata links to HistoryDatasetAssociation tables"
+
def upgrade(migrate_engine):
print __doc__
metadata.bind = migrate_engine
metadata.reflect()
-
+
try:
hda_table = Table( "history_dataset_association", metadata, autoload=True )
extended_metadata_hda_col.create( hda_table )
@@ -36,7 +31,7 @@ def upgrade(migrate_engine):
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
-
+
# Drop the HDA table's extended metadata ID column.
try:
hda_table = Table( "history_dataset_association", metadata, autoload=True )
@@ -44,5 +39,3 @@ def downgrade(migrate_engine):
extended_metadata_id.drop()
except Exception, e:
log.debug( "Dropping 'extended_metadata_id' column from history_dataset_association table failed: %s" % ( str( e ) ) )
-
-
diff --git a/lib/galaxy/model/migrate/versions/0119_job_metrics.py b/lib/galaxy/model/migrate/versions/0119_job_metrics.py
index 8a34975ecc6..94607d1e0ba 100644
--- a/lib/galaxy/model/migrate/versions/0119_job_metrics.py
+++ b/lib/galaxy/model/migrate/versions/0119_job_metrics.py
@@ -1,18 +1,13 @@
"""
Migration script for job metric plugins.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-
import datetime
-now = datetime.datetime.utcnow
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, ForeignKey, Integer, MetaData, Numeric, Table, Unicode
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
TEXT_METRIC_MAX_LENGTH = 1023
diff --git a/lib/galaxy/model/migrate/versions/0120_dataset_collections.py b/lib/galaxy/model/migrate/versions/0120_dataset_collections.py
index 8d4d89c1f7e..b6c0b533a0f 100644
--- a/lib/galaxy/model/migrate/versions/0120_dataset_collections.py
+++ b/lib/galaxy/model/migrate/versions/0120_dataset_collections.py
@@ -1,27 +1,22 @@
"""
Migration script for tables related to dataset collections.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-from galaxy.model.custom_types import *
-
import datetime
-now = datetime.datetime.utcnow
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, MetaData, Table, TEXT, Unicode
+
+from galaxy.model.custom_types import TrimmedString
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
DatasetCollection_table = Table( "dataset_collection", metadata,
Column( "id", Integer, primary_key=True ),
Column( "collection_type", Unicode(255), nullable=False, ),
Column( "create_time", DateTime, default=now ),
- Column( "update_time", DateTime, default=now, onupdate=now ),
-)
+ Column( "update_time", DateTime, default=now, onupdate=now ) )
HistoryDatasetCollectionAssociation_table = Table( "history_dataset_collection_association", metadata,
Column( "id", Integer, primary_key=True ),
@@ -32,16 +27,14 @@ HistoryDatasetCollectionAssociation_table = Table( "history_dataset_collection_a
Column( "deleted", Boolean, default=False ),
Column( "visible", Boolean, default=True ),
Column( "copied_from_history_dataset_collection_association_id", Integer, ForeignKey( "history_dataset_collection_association.id" ), nullable=True ),
- Column( "implicit_output_name", Unicode(255), nullable=True ),
-)
+ Column( "implicit_output_name", Unicode(255), nullable=True ) )
LibraryDatasetCollectionAssociation_table = Table( "library_dataset_collection_association", metadata,
Column( "id", Integer, primary_key=True ),
Column( "collection_id", Integer, ForeignKey( "dataset_collection.id" ), index=True ),
Column( "name", TrimmedString( 255 ) ),
Column( "deleted", Boolean, default=False ),
- Column( "folder_id", Integer, ForeignKey( "library_folder.id" ), index=True ),
-)
+ Column( "folder_id", Integer, ForeignKey( "library_folder.id" ), index=True ) )
DatasetCollectionElement_table = Table( "dataset_collection_element", metadata,
Column( "id", Integer, primary_key=True ),
@@ -50,36 +43,31 @@ DatasetCollectionElement_table = Table( "dataset_collection_element", metadata,
Column( "ldda_id", Integer, ForeignKey( "library_dataset_dataset_association.id" ), index=True, nullable=True ),
Column( "child_collection_id", Integer, ForeignKey( "dataset_collection.id" ), index=True, nullable=True ),
Column( "element_index", Integer, nullable=False ),
- Column( "element_identifier", Unicode(255), nullable=False ),
-)
+ Column( "element_identifier", Unicode(255), nullable=False ) )
HistoryDatasetCollectionAnnotationAssociation_table = Table( "history_dataset_collection_annotation_association", metadata,
Column( "id", Integer, primary_key=True ),
Column( "history_dataset_collection_id", Integer, ForeignKey( "history_dataset_collection_association.id" ), index=True ),
Column( "user_id", Integer, ForeignKey( "galaxy_user.id" ), index=True ),
- Column( "annotation", TEXT )
-)
+ Column( "annotation", TEXT ) )
LibraryDatasetCollectionAnnotationAssociation_table = Table( "library_dataset_collection_annotation_association", metadata,
Column( "id", Integer, primary_key=True ),
Column( "library_dataset_collection_id", Integer, ForeignKey( "library_dataset_collection_association.id" ), index=True ),
Column( "user_id", Integer, ForeignKey( "galaxy_user.id" ), index=True ),
- Column( "annotation", TEXT )
-)
+ Column( "annotation", TEXT ) )
HistoryDatasetCollectionRatingAssociation_table = Table( "history_dataset_collection_rating_association", metadata,
Column( "id", Integer, primary_key=True ),
Column( "history_dataset_collection_id", Integer, ForeignKey( "history_dataset_collection_association.id" ), index=True ),
Column( "user_id", Integer, ForeignKey( "galaxy_user.id" ), index=True ),
- Column( "rating", Integer, index=True)
-)
+ Column( "rating", Integer, index=True) )
LibraryDatasetCollectionRatingAssociation_table = Table( "library_dataset_collection_rating_association", metadata,
Column( "id", Integer, primary_key=True ),
Column( "library_dataset_collection_id", Integer, ForeignKey( "library_dataset_collection_association.id" ), index=True ),
Column( "user_id", Integer, ForeignKey( "galaxy_user.id" ), index=True ),
- Column( "rating", Integer, index=True)
-)
+ Column( "rating", Integer, index=True) )
HistoryDatasetCollectionTagAssociation_table = Table( "history_dataset_collection_tag_association", metadata,
Column( "id", Integer, primary_key=True ),
@@ -88,8 +76,7 @@ HistoryDatasetCollectionTagAssociation_table = Table( "history_dataset_collectio
Column( "user_id", Integer, ForeignKey( "galaxy_user.id" ), index=True ),
Column( "user_tname", Unicode(255), index=True),
Column( "value", Unicode(255), index=True),
- Column( "user_value", Unicode(255), index=True)
-)
+ Column( "user_value", Unicode(255), index=True) )
LibraryDatasetCollectionTagAssociation_table = Table( "library_dataset_collection_tag_association", metadata,
Column( "id", Integer, primary_key=True ),
@@ -98,22 +85,19 @@ LibraryDatasetCollectionTagAssociation_table = Table( "library_dataset_collectio
Column( "user_id", Integer, ForeignKey( "galaxy_user.id" ), index=True ),
Column( "user_tname", Unicode(255), index=True),
Column( "value", Unicode(255), index=True),
- Column( "user_value", Unicode(255), index=True)
-)
+ Column( "user_value", Unicode(255), index=True) )
JobToInputDatasetCollectionAssociation_table = Table( "job_to_input_dataset_collection", metadata,
Column( "id", Integer, primary_key=True ),
Column( "job_id", Integer, ForeignKey( "job.id" ), index=True ),
Column( "dataset_collection_id", Integer, ForeignKey( "history_dataset_collection_association.id" ), index=True ),
- Column( "name", Unicode(255) )
-)
+ Column( "name", Unicode(255) ) )
JobToOutputDatasetCollectionAssociation_table = Table( "job_to_output_dataset_collection", metadata,
Column( "id", Integer, primary_key=True ),
Column( "job_id", Integer, ForeignKey( "job.id" ), index=True ),
Column( "dataset_collection_id", Integer, ForeignKey( "history_dataset_collection_association.id" ), index=True ),
- Column( "name", Unicode(255) )
-)
+ Column( "name", Unicode(255) ) )
ImplicitlyCreatedDatasetCollectionInput_table = Table( "implicitly_created_dataset_collection_inputs", metadata,
Column( "id", Integer, primary_key=True ),
diff --git a/lib/galaxy/model/migrate/versions/0121_workflow_uuids.py b/lib/galaxy/model/migrate/versions/0121_workflow_uuids.py
index 19f0c8e39f8..86df373d47d 100644
--- a/lib/galaxy/model/migrate/versions/0121_workflow_uuids.py
+++ b/lib/galaxy/model/migrate/versions/0121_workflow_uuids.py
@@ -1,24 +1,20 @@
"""
Add UUIDs to workflows
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-from galaxy.model.custom_types import UUIDType, TrimmedString
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, MetaData, Table
+
+from galaxy.model.custom_types import UUIDType
+
+log = logging.getLogger( __name__ )
metadata = MetaData()
-
"""
-Because both workflow and job requests can be determined
+Because both workflow and job requests can be determined
based the a fixed data structure, their IDs are based on
-hashing the data structure
+hashing the data structure
"""
workflow_uuid_column = Column( "uuid", UUIDType, nullable=True )
@@ -26,6 +22,7 @@ workflow_uuid_column = Column( "uuid", UUIDType, nullable=True )
def display_migration_details():
print "This migration script adds a UUID column to workflows"
+
def upgrade(migrate_engine):
print __doc__
metadata.bind = migrate_engine
@@ -41,6 +38,7 @@ def upgrade(migrate_engine):
log.error( "Adding column 'uuid' to workflow table failed: %s" % str( e ) )
return
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
@@ -52,4 +50,3 @@ def downgrade(migrate_engine):
workflow_uuid.drop()
except Exception, e:
log.debug( "Dropping 'uuid' column from workflow table failed: %s" % ( str( e ) ) )
-
diff --git a/lib/galaxy/model/migrate/versions/0123_add_workflow_request_tables.py b/lib/galaxy/model/migrate/versions/0123_add_workflow_request_tables.py
index 653220fad22..5dbc356387e 100644
--- a/lib/galaxy/model/migrate/versions/0123_add_workflow_request_tables.py
+++ b/lib/galaxy/model/migrate/versions/0123_add_workflow_request_tables.py
@@ -1,18 +1,15 @@
"""
Migration script for workflow request tables.
"""
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-from galaxy.model.custom_types import *
-
import datetime
-now = datetime.datetime.utcnow
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, ForeignKey, Integer, MetaData, String, Table, TEXT, Unicode
+
+from galaxy.model.custom_types import JSONType, TrimmedString, UUIDType
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
@@ -93,7 +90,7 @@ def upgrade(migrate_engine):
WorkflowInvocationStepAction_column = Column( "action", JSONType, nullable=True )
__add_column( WorkflowInvocationStepAction_column, "workflow_invocation_step", metadata )
-
+
def downgrade(migrate_engine):
metadata.bind = migrate_engine
diff --git a/lib/galaxy/model/migrate/versions/0124_job_state_history.py b/lib/galaxy/model/migrate/versions/0124_job_state_history.py
index 38796fac69d..35189f61f59 100644
--- a/lib/galaxy/model/migrate/versions/0124_job_state_history.py
+++ b/lib/galaxy/model/migrate/versions/0124_job_state_history.py
@@ -1,19 +1,15 @@
"""
Migration script for the job state history table
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-from galaxy.model.custom_types import *
-
import datetime
-now = datetime.datetime.utcnow
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, DateTime, ForeignKey, Integer, MetaData, String, Table
+
+from galaxy.model.custom_types import TrimmedString
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
JobStateHistory_table = Table( "job_state_history", metadata,
@@ -22,8 +18,8 @@ JobStateHistory_table = Table( "job_state_history", metadata,
Column( "update_time", DateTime, default=now, onupdate=now ),
Column( "job_id", Integer, ForeignKey( "job.id" ), index=True ),
Column( "state", String( 64 ), index=True ),
- Column( "info", TrimmedString( 255 ) )
-)
+ Column( "info", TrimmedString( 255 ) ) )
+
def upgrade(migrate_engine):
metadata.bind = migrate_engine
diff --git a/lib/galaxy/model/migrate/versions/0125_workflow_step_tracking.py b/lib/galaxy/model/migrate/versions/0125_workflow_step_tracking.py
index 5564c00ce78..2aebdc78c09 100644
--- a/lib/galaxy/model/migrate/versions/0125_workflow_step_tracking.py
+++ b/lib/galaxy/model/migrate/versions/0125_workflow_step_tracking.py
@@ -1,18 +1,15 @@
"""
Migration script to enhance workflow step usability by adding labels and UUIDs.
"""
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-from galaxy.model.custom_types import *
-
import datetime
-now = datetime.datetime.utcnow
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, MetaData, Table
+
+from galaxy.model.custom_types import TrimmedString, UUIDType
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
diff --git a/lib/galaxy/model/migrate/versions/0126_password_reset.py b/lib/galaxy/model/migrate/versions/0126_password_reset.py
index 96be92b0258..63cacb803a3 100644
--- a/lib/galaxy/model/migrate/versions/0126_password_reset.py
+++ b/lib/galaxy/model/migrate/versions/0126_password_reset.py
@@ -1,19 +1,13 @@
"""
Migration script for the password reset table
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-from galaxy.model.custom_types import *
-
import datetime
-now = datetime.datetime.utcnow
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, DateTime, ForeignKey, Integer, MetaData, String, Table
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
PasswordResetToken_table = Table("password_reset_token", metadata,
diff --git a/lib/galaxy/model/migrate/versions/0127_output_collection_adjustments.py b/lib/galaxy/model/migrate/versions/0127_output_collection_adjustments.py
index 76673edeafc..65355aba28d 100644
--- a/lib/galaxy/model/migrate/versions/0127_output_collection_adjustments.py
+++ b/lib/galaxy/model/migrate/versions/0127_output_collection_adjustments.py
@@ -1,19 +1,15 @@
"""
Migration script updating collections tables for output collections.
"""
-
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-from galaxy.model.custom_types import *
-
import datetime
-now = datetime.datetime.utcnow
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, ForeignKey, Integer, MetaData, Table, TEXT, Unicode
+
+from galaxy.model.custom_types import TrimmedString
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
JobToImplicitOutputDatasetCollectionAssociation_table = Table(
diff --git a/lib/galaxy/model/migrate/versions/0128_session_timeout.py b/lib/galaxy/model/migrate/versions/0128_session_timeout.py
index cf92c0f0e27..fc63674a88b 100644
--- a/lib/galaxy/model/migrate/versions/0128_session_timeout.py
+++ b/lib/galaxy/model/migrate/versions/0128_session_timeout.py
@@ -1,18 +1,13 @@
"""
Migration script to add session update time (used for timeouts)
"""
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-from galaxy.model.custom_types import *
-
import datetime
-now = datetime.datetime.utcnow
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Column, DateTime, MetaData, Table
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
diff --git a/lib/galaxy/model/migrate/versions/0129_job_external_output_metadata_validity.py b/lib/galaxy/model/migrate/versions/0129_job_external_output_metadata_validity.py
index f24a99f61b2..b158ee82224 100644
--- a/lib/galaxy/model/migrate/versions/0129_job_external_output_metadata_validity.py
+++ b/lib/galaxy/model/migrate/versions/0129_job_external_output_metadata_validity.py
@@ -1,18 +1,13 @@
"""
Migration script to allow invalidation of job external output metadata temp files
"""
-from sqlalchemy import *
-from sqlalchemy.orm import *
-from migrate import *
-from migrate.changeset import *
-from galaxy.model.custom_types import *
-
import datetime
-now = datetime.datetime.utcnow
-
import logging
-log = logging.getLogger( __name__ )
+from sqlalchemy import Boolean, Column, MetaData, Table
+
+now = datetime.datetime.utcnow
+log = logging.getLogger( __name__ )
metadata = MetaData()
@@ -29,7 +24,7 @@ def downgrade(migrate_engine):
metadata.bind = migrate_engine
metadata.reflect()
- __drop_column( isvalid_column, "job_external_output_metadata", metadata )
+ __drop_column( "is_valid", "job_external_output_metadata", metadata )
def __add_column(column, table_name, metadata, **kwds):
diff --git a/lib/galaxy/model/orm/scripts.py b/lib/galaxy/model/orm/scripts.py
index 56258e002a4..86d5dc02852 100644
--- a/lib/galaxy/model/orm/scripts.py
+++ b/lib/galaxy/model/orm/scripts.py
@@ -45,8 +45,10 @@ def read_config_file_arg( argv, default, old_default ):
else:
if not os.path.exists( default ) and os.path.exists( old_default ):
config_file = old_default
- else:
+ elif os.path.exists( default ):
config_file = default
+ else:
+ config_file = default + ".sample"
return config_file
diff --git a/lib/galaxy/tools/__init__.py b/lib/galaxy/tools/__init__.py
index 5ed39f44605..cf1ee9eff5d 100755
--- a/lib/galaxy/tools/__init__.py
+++ b/lib/galaxy/tools/__init__.py
@@ -38,7 +38,7 @@ from galaxy.tools.test import parse_tests
from galaxy.tools.parser import get_tool_source
from galaxy.tools.parser.xml import XmlPageSource
from galaxy.tools.toolbox import AbstractToolBox
-from galaxy.util import rst_to_html, string_as_bool, string_to_object
+from galaxy.util import rst_to_html, string_as_bool
from galaxy.tools.parameters.meta import expand_meta_parameters
from galaxy.util.bunch import Bunch
from galaxy.util.expressions import ExpressionContext
@@ -444,7 +444,6 @@ class Tool( object, Dictifiable ):
requires_setting_metadata = True
default_tool_action = DefaultToolAction
dict_collection_visible_keys = ( 'id', 'name', 'version', 'description' )
- default_template = 'tool_form.mako'
def __init__( self, config_file, tool_source, app, guid=None, repository_id=None, allow_code_files=True ):
"""Load a tool from the config named by `config_file`"""
@@ -1188,21 +1187,14 @@ class Tool( object, Dictifiable ):
# outputs?
return True
- def new_state( self, trans, all_pages=False, history=None ):
+ def new_state( self, trans, history=None ):
"""
Create a new `DefaultToolState` for this tool. It will be initialized
with default values for inputs.
-
- Only inputs on the first page will be initialized unless `all_pages` is
- True, in which case all inputs regardless of page are initialized.
"""
state = DefaultToolState()
state.inputs = {}
- if all_pages:
- inputs = self.inputs
- else:
- inputs = self.inputs_by_page[ 0 ]
- self.fill_in_new_state( trans, inputs, state.inputs, history=history )
+ self.fill_in_new_state( trans, self.inputs, state.inputs, history=history )
return state
def fill_in_new_state( self, trans, inputs, state, context=None, history=None ):
@@ -1267,101 +1259,70 @@ class Tool( object, Dictifiable ):
else:
input.visit_inputs( "", value[input.name], callback )
- def handle_input( self, trans, incoming, history=None, old_errors=None, process_state='update', source='html' ):
+ def handle_input( self, trans, incoming, history=None, source='html' ):
"""
Process incoming parameters for this tool from the dict `incoming`,
update the tool state (or create if none existed), and either return
to the form or execute the tool (only if 'execute' was clicked and
there were no errors).
-
- process_state can be either 'update' (to incrementally build up the state
- over several calls - one repeat per handle for instance) or 'populate'
- force a complete build of the state and submission all at once (like
- from API). May want an incremental version of the API also at some point,
- that is why this is not just called for_api.
"""
- all_pages = ( process_state == "populate" ) # If process_state = update, handle all pages at once.
rerun_remap_job_id = None
if 'rerun_remap_job_id' in incoming:
try:
rerun_remap_job_id = trans.app.security.decode_id( incoming[ 'rerun_remap_job_id' ] )
except Exception, exception:
log.error( str( exception ) )
- message = 'Failure executing tool (attempting to rerun invalid job).'
- return 'message.mako', dict( status='error', message=message, refresh_frames=[] )
+ raise exceptions.MessageException( 'Failure executing tool (attempting to rerun invalid job).' )
# Fixed set of input parameters may correspond to any number of jobs.
- # Expand these out to individual parameters for given jobs (tool
- # executions).
+ # Expand these out to individual parameters for given jobs (tool executions).
expanded_incomings, collection_info = expand_meta_parameters( trans, self, incoming )
-
if not expanded_incomings:
- raise exceptions.MessageException( "Tool execution failed, trying to run a tool over an empty collection." )
+ raise exceptions.MessageException( 'Tool execution failed, trying to run a tool over an empty collection.' )
# Remapping a single job to many jobs doesn't make sense, so disable
# remap if multi-runs of tools are being used.
if rerun_remap_job_id and len( expanded_incomings ) > 1:
- message = 'Failure executing tool (cannot create multiple jobs when remapping existing job).'
- return 'message.mako', dict( status='error', message=message, refresh_frames=[] )
-
- all_states = []
- for expanded_incoming in expanded_incomings:
- state, state_new = self.__fetch_state( trans, expanded_incoming, history, all_pages=all_pages )
- all_states.append( state )
- if state_new:
- # This feels a bit like a hack. It allows forcing full processing
- # of inputs even when there is no state in the incoming dictionary
- # by providing either 'runtool_btn' (the name of the submit button
- # on the standard run form) or "URL" (a parameter provided by
- # external data source tools).
- if "runtool_btn" not in incoming and "URL" not in incoming:
- if not self.display_interface:
- return self.__no_display_interface_response()
- if len(incoming):
- self.update_state( trans, self.inputs_by_page[state.page], state.inputs, incoming, old_errors=old_errors or {}, source=source )
- return self.default_template, dict( errors={}, tool_state=state, param_values={}, incoming={} )
+ raise exceptions.MessageException( 'Failure executing tool (cannot create multiple jobs when remapping existing job).' )
all_errors = []
all_params = []
- for expanded_incoming, expanded_state in zip(expanded_incomings, all_states):
- errors, params = self.__check_param_values( trans, expanded_incoming, expanded_state, old_errors, process_state, history=history, source=source )
+ for expanded_incoming in expanded_incomings:
+ expanded_state = self.new_state( trans, history=history )
+ # Process incoming data
+ if not self.check_values:
+ # If `self.check_values` is false we don't do any checking or
+ # processing on input This is used to pass raw values
+ # through to/from external sites.
+ errors = {}
+ params = expanded_incoming
+ else:
+ # Update state for all inputs on the current page taking new
+ # values from `incoming`.
+ errors = self.populate_state( trans, self.inputs, expanded_state.inputs, expanded_incoming, history, source=source )
+ # If the tool provides a `validate_input` hook, call it.
+ validate_input = self.get_hook( 'validate_input' )
+ if validate_input:
+ validate_input( trans, errors, expanded_state.inputs, self.inputs )
+ params = expanded_state.inputs
all_errors.append( errors )
all_params.append( params )
- if self.__should_refresh_state( incoming ):
- template, template_vars = self.__handle_state_refresh( trans, state, errors )
+ # If there were errors, we stay on the same page and display
+ # error messages
+ if any( all_errors ):
+ raise exceptions.MessageException( err_data=all_errors[ 0 ] )
else:
- # User actually clicked next or execute.
-
- # If there were errors, we stay on the same page and display
- # error messages
- if any( all_errors ):
- error_message = "One or more errors were found in the input you provided. The specific errors are marked below."
- template = self.default_template
- template_vars = dict( errors=errors, tool_state=state, incoming=incoming, error_message=error_message )
- # If we've completed the last page we can execute the tool
- elif all_pages or state.page == self.last_page:
- execution_tracker = execute_job( trans, self, all_params, history=history, rerun_remap_job_id=rerun_remap_job_id, collection_info=collection_info )
- if execution_tracker.successful_jobs:
- template = 'tool_executed.mako'
- template_vars = dict(
- out_data=execution_tracker.output_datasets,
- num_jobs=len( execution_tracker.successful_jobs ),
- job_errors=execution_tracker.execution_errors,
- jobs=execution_tracker.successful_jobs,
- output_collections=execution_tracker.output_collections,
- implicit_collections=execution_tracker.implicit_collections,
- )
- else:
- template = 'message.mako'
- template_vars = dict( status='error', message=execution_tracker.execution_errors[0], refresh_frames=[] )
- # Otherwise move on to the next page
+ execution_tracker = execute_job( trans, self, all_params, history=history, rerun_remap_job_id=rerun_remap_job_id, collection_info=collection_info )
+ if execution_tracker.successful_jobs:
+ return dict( out_data=execution_tracker.output_datasets,
+ num_jobs=len( execution_tracker.successful_jobs ),
+ job_errors=execution_tracker.execution_errors,
+ jobs=execution_tracker.successful_jobs,
+ output_collections=execution_tracker.output_collections,
+ implicit_collections=execution_tracker.implicit_collections )
else:
- template, template_vars = self.__handle_page_advance( trans, state, errors )
- return template, template_vars
-
- def __should_refresh_state( self, incoming ):
- return not( 'runtool_btn' in incoming or 'URL' in incoming or 'ajax_upload' in incoming )
+ raise exceptions.MessageException( execution_tracker.execution_errors[ 0 ] )
def handle_single_execution( self, trans, rerun_remap_job_id, params, history, mapping_over_collection ):
"""
@@ -1387,72 +1348,6 @@ class Tool( object, Dictifiable ):
message = 'Failure executing tool (invalid data returned from tool execution)'
return False, message
- def __handle_state_refresh( self, trans, state, errors ):
- try:
- self.find_fieldstorage( state.inputs )
- except InterruptedUpload:
- # If inputs contain a file it won't persist. Most likely this
- # is an interrupted upload. We should probably find a more
- # standard method of determining an incomplete POST.
- return self.handle_interrupted( trans, state.inputs )
- except:
- pass
- # Just a refresh, render the form with updated state and errors.
- if not self.display_interface:
- return self.__no_display_interface_response()
- return self.default_template, dict( errors=errors, tool_state=state )
-
- def __handle_page_advance( self, trans, state, errors ):
- state.page += 1
- # Fill in the default values for the next page
- self.fill_in_new_state( trans, self.inputs_by_page[ state.page ], state.inputs )
- if not self.display_interface:
- return self.__no_display_interface_response()
- return self.default_template, dict( errors=errors, tool_state=state )
-
- def __no_display_interface_response( self ):
- return 'message.mako', dict( status='info', message="The interface for this tool cannot be displayed", refresh_frames=['everything'] )
-
- def __fetch_state( self, trans, incoming, history, all_pages ):
- # Get the state or create if not found
- if "tool_state" in incoming:
- encoded_state = string_to_object( incoming["tool_state"] )
- state = DefaultToolState()
- state.decode( encoded_state, self, trans.app )
- new = False
- else:
- state = self.new_state( trans, history=history, all_pages=all_pages )
- new = True
- return state, new
-
- def __check_param_values( self, trans, incoming, state, old_errors, process_state, history, source ):
- # Process incoming data
- if not self.check_values:
- # If `self.check_values` is false we don't do any checking or
- # processing on input This is used to pass raw values
- # through to/from external sites. FIXME: This should be handled
- # more cleanly, there is no reason why external sites need to
- # post back to the same URL that the tool interface uses.
- errors = {}
- params = incoming
- else:
- # Update state for all inputs on the current page taking new
- # values from `incoming`.
- if process_state == "update":
- inputs = self.inputs_by_page[state.page]
- errors = self.update_state( trans, inputs, state.inputs, incoming, old_errors=old_errors or {}, source=source )
- elif process_state == "populate":
- inputs = self.inputs
- errors = self.populate_state( trans, inputs, state.inputs, incoming, history, source=source )
- else:
- raise Exception("Unknown process_state type %s" % process_state)
- # If the tool provides a `validate_input` hook, call it.
- validate_input = self.get_hook( 'validate_input' )
- if validate_input:
- validate_input( trans, errors, state.inputs, inputs )
- params = state.inputs
- return errors, params
-
def find_fieldstorage( self, x ):
if isinstance( x, FieldStorage ):
raise InterruptedUpload( None )
@@ -1461,37 +1356,6 @@ class Tool( object, Dictifiable ):
elif isinstance(x, list):
[ self.find_fieldstorage( y ) for y in x ]
- def handle_interrupted( self, trans, inputs ):
- """
- Upon handling inputs, if it appears that we have received an incomplete
- form, do some cleanup or anything else deemed necessary. Currently
- this is only likely during file uploads, but this method could be
- generalized and a method standardized for handling other tools.
- """
- # If the async upload tool has uploading datasets, we need to error them.
- if 'async_datasets' in inputs and inputs['async_datasets'] not in [ 'None', '', None ]:
- for id in inputs['async_datasets'].split(','):
- try:
- data = self.sa_session.query( trans.model.HistoryDatasetAssociation ).get( int( id ) )
- except:
- log.exception( 'Unable to load precreated dataset (%s) sent in upload form' % id )
- continue
- if trans.user is None and trans.galaxy_session.current_history != data.history:
- log.error( 'Got a precreated dataset (%s) but it does not belong to anonymous user\'s current session (%s)'
- % ( data.id, trans.galaxy_session.id ) )
- elif data.history.user != trans.user:
- log.error( 'Got a precreated dataset (%s) but it does not belong to current user (%s)'
- % ( data.id, trans.user.id ) )
- else:
- data.state = data.states.ERROR
- data.info = 'Upload of this dataset was interrupted. Please try uploading again or'
- self.sa_session.add( data )
- self.sa_session.flush()
- # It's unlikely the user will ever see this.
- return 'message.mako', dict( status='error',
- message='Your upload was interrupted. If this was uninentional, please retry it.',
- refresh_frames=[], cont=None )
-
def populate_state( self, trans, inputs, state, incoming, history=None, source="html", prefix="", context=None ):
errors = dict()
# Push this level onto the context stack
@@ -1531,6 +1395,8 @@ class Tool( object, Dictifiable ):
group_errors[-1] = { '__index__': 'Cannot add repeat (max size=%i).' % input.max }
any_group_errors = True
rep_index += 1
+ if any_group_errors:
+ errors[ input.name ] = group_errors
elif isinstance( input, Conditional ):
group_state = state[input.name]
group_prefix = "%s|" % ( key )
@@ -1813,56 +1679,6 @@ class Tool( object, Dictifiable ):
item_callback=item_callback )
if group_errors:
errors[ input.name ] = group_errors
- elif isinstance( input, UploadDataset ):
- group_state = state[input.name]
- group_errors = []
- group_old_errors = old_errors.get( input.name, None )
- any_group_errors = False
- d_type = input.get_datatype( trans, context )
- writable_files = d_type.writable_files
- # remove extra files
- while len( group_state ) > len( writable_files ):
- del group_state[-1]
- if group_old_errors:
- del group_old_errors[-1]
- # Update state
- max_index = -1
- for i, rep_state in enumerate( group_state ):
- rep_index = rep_state['__index__']
- max_index = max( max_index, rep_index )
- rep_prefix = "%s_%d|" % ( key, rep_index )
- if group_old_errors:
- rep_old_errors = group_old_errors[i]
- else:
- rep_old_errors = {}
- rep_errors = self.update_state( trans,
- input.inputs,
- rep_state,
- incoming,
- prefix=rep_prefix,
- context=context,
- source=source,
- update_only=update_only,
- old_errors=rep_old_errors,
- item_callback=item_callback )
- if rep_errors:
- any_group_errors = True
- group_errors.append( rep_errors )
- else:
- group_errors.append( {} )
- # Add new fileupload as needed
- offset = 1
- while len( writable_files ) > len( group_state ):
- new_state = {}
- new_state['__index__'] = max_index + offset
- offset += 1
- self.fill_in_new_state( trans, input.inputs, new_state, context )
- group_state.append( new_state )
- if any_group_errors:
- group_errors.append( {} )
- # Were there *any* errors for any repetition?
- if any_group_errors:
- errors[input.name] = group_errors
else:
if key not in incoming \
and "__force_update__" + key not in incoming \
@@ -2321,15 +2137,11 @@ class Tool( object, Dictifiable ):
return tool_dict
- def to_json(self, trans, kwd={}, is_workflow=False):
+ def to_json(self, trans, kwd={}, job=None, is_workflow=False):
"""
Recursively creates a tool dictionary containing repeats, dynamic options and updated states.
"""
- job_id = kwd.get('__job_id__', None)
- dataset_id = kwd.get('__dataset_id__', None)
history_id = kwd.get('history_id', None)
-
- # history id
history = None
try:
if history_id is not None:
@@ -2337,40 +2149,9 @@ class Tool( object, Dictifiable ):
else:
history = trans.get_history()
if history is None:
- raise Exception('History unavailable. Please specify a valid history id')
+ raise exceptions.MessageException( 'History unavailable. Please specify a valid history id' )
except Exception, e:
- trans.response.status = 500
- error = '[history_id=%s] Failed to retrieve history. %s.' % (history_id, str(e))
- log.exception('tools::to_json - %s.' % error)
- return { 'error': error }
-
- # load job details if provided
- job = None
- if job_id:
- try:
- job_id = trans.security.decode_id( job_id )
- job = trans.sa_session.query( trans.app.model.Job ).get( job_id )
- except Exception, exception:
- trans.response.status = 500
- log.error('Failed to retrieve job.')
- return { 'error': 'Failed to retrieve job.' }
- elif dataset_id:
- try:
- dataset_id = trans.security.decode_id( dataset_id )
- data = trans.sa_session.query( trans.app.model.HistoryDatasetAssociation ).get( dataset_id )
- if not ( trans.user_is_admin() or trans.app.security_agent.can_access_dataset( trans.get_current_user_roles(), data.dataset ) ):
- trans.response.status = 500
- log.error('User has no access to dataset.')
- return { 'error': 'User has no access to dataset.' }
- job = data.creating_job
- if not job:
- trans.response.status = 500
- log.error('Creating job not found.')
- return { 'error': 'Creating job not found.' }
- except Exception, exception:
- trans.response.status = 500
- log.error('Failed to get job information.')
- return { 'error': 'Failed to get job information.' }
+ raise exceptions.MessageException( '[history_id=%s] Failed to retrieve history. %s.' % ( history_id, str( e ) ) )
# load job parameters into incoming
tool_message = ''
@@ -2381,10 +2162,8 @@ class Tool( object, Dictifiable ):
self._map_source_to_history( trans, self.inputs, job_params, history )
tool_message = self._compare_tool_version(trans, job)
params_to_incoming( kwd, self.inputs, job_params, trans.app, to_html=False )
- except Exception, exception:
- trans.response.status = 500
- log.error( str( exception ) )
- return { 'error': str( exception ) }
+ except Exception, e:
+ raise exceptions.MessageException( str( e ) )
# create parameter object
params = galaxy.util.Params( kwd, sanitize=False )
@@ -2531,21 +2310,16 @@ class Tool( object, Dictifiable ):
def iterate(group_inputs, inputs, state_inputs, other_values=None):
other_values = ExpressionContext( state_inputs, other_values )
for input_index, input in enumerate( inputs.itervalues() ):
- # create model dictionary
- tool_dict = input.to_dict(trans)
- if tool_dict is None:
- continue
-
- # state for subsection/group
+ tool_dict = None
group_state = state_inputs.get(input.name, {})
-
- # iterate and update values
if input.type == 'repeat':
+ tool_dict = input.to_dict(trans)
group_cache = tool_dict['cache'] = {}
for i in range( len( group_state ) ):
group_cache[i] = {}
iterate( group_cache[i], input.inputs, group_state[i], other_values )
elif input.type == 'conditional':
+ tool_dict = input.to_dict(trans)
if 'test_param' in tool_dict:
test_param = tool_dict['test_param']
test_param['default_value'] = jsonify(input.test_param.get_initial_value(trans, other_values, history=history))
@@ -2556,29 +2330,27 @@ class Tool( object, Dictifiable ):
current_state = group_state
iterate(tool_dict['cases'][i]['inputs'], input.cases[i].inputs, current_state, other_values)
elif input.type == 'section':
+ tool_dict = input.to_dict(trans)
iterate( tool_dict['inputs'], input.inputs, group_state, other_values )
else:
- # identify name
- input_name = tool_dict.get('name')
-
- # expand input dictionary incl. repeats and dynamic_parameters
+ # expand input dictionary, resolve dynamic parameters
try:
tool_dict = input.to_dict(trans, other_values=other_values)
except Exception:
- log.exception('tools::to_json() - Skipping parameter expansion for %s.' % input_name)
+ tool_dict = input.to_dict(trans)
+ log.exception('tools::to_json() - Skipping parameter expansion for %s.' % input.name)
pass
# backup default value
try:
tool_dict['default_value'] = input.get_initial_value(trans, other_values, history=history)
except Exception:
- log.exception('tools::to_json() - Getting initial value failed %s.' % input_name)
- # get initial value failed due to improper late validation
tool_dict['default_value'] = None
+ log.exception('tools::to_json() - Getting initial value failed %s.' % input.name)
pass
# update input value from tool state
- tool_dict['value'] = state_inputs.get(input_name, tool_dict['default_value'])
+ tool_dict['value'] = state_inputs.get(input.name, tool_dict['default_value'])
# sanitize values
sanitize(tool_dict, 'value')
@@ -2661,6 +2433,7 @@ class Tool( object, Dictifiable ):
# add additional properties
tool_model.update({
+ 'id' : self.id,
'help' : tool_help,
'citations' : tool_citations,
'biostar_url' : trans.app.config.biostar_url,
@@ -2670,13 +2443,11 @@ class Tool( object, Dictifiable ):
'requirements' : tool_requirements,
'errors' : state_errors,
'state_inputs' : state_inputs,
- 'job_remap' : self._get_job_remap(job)
+ 'job_id' : trans.security.encode_id( job.id ) if job else None,
+ 'job_remap' : self._get_job_remap( job ),
+ 'history_id' : trans.security.encode_id( history.id )
})
- # check for errors
- if 'error' in tool_message:
- return tool_message
-
# return enriched tool model
return tool_model
@@ -2766,8 +2537,7 @@ class Tool( object, Dictifiable ):
try:
select_field, tools, tool = self.app.toolbox.get_tool_components( tool_id, tool_version=tool_version, get_loaded_tools_by_lineage=False, set_selected=True )
if tool is None:
- trans.response.status = 500
- return { 'error': 'This dataset was created by an obsolete tool (%s). Can\'t re-run.' % tool_id }
+ raise exceptions.MessageException( 'This dataset was created by an obsolete tool (%s). Can\'t re-run.' % tool_id )
if ( self.id != tool_id and self.old_id != tool_id ) or self.version != tool_version:
if self.id == tool_id:
if tool_version is None:
@@ -2786,14 +2556,8 @@ class Tool( object, Dictifiable ):
else:
message = 'This job was initially run with tool id "%s", version "%s", which is ' % ( tool_id, tool_version )
message += 'currently not available. You can re-run the job with this tool, which is a derivation of the original tool.'
- except Exception, error:
- trans.response.status = 500
- return { 'error': str(error) }
-
- # can't rerun upload, external data sources, et cetera. workflow compatible will proxy this for now
- # if not self.is_workflow_compatible:
- # trans.response.status = 500
- # return { 'error': 'The \'%s\' tool does currently not support re-running.' % self.name }
+ except Exception, e:
+ raise exceptions.MessageException( str( e ) )
return message
def get_default_history_by_trans( self, trans, create=False ):
@@ -2999,10 +2763,6 @@ class ImportHistoryTool( Tool ):
tool_type = 'import_history'
-class GenomeIndexTool( Tool ):
- tool_type = 'index_genome'
-
-
class DataManagerTool( OutputParameterJSONTool ):
tool_type = 'manage_data'
default_tool_action = DataManagerToolAction
diff --git a/lib/galaxy/tools/errors.py b/lib/galaxy/tools/errors.py
index c9b8f40feb5..251e1e894db 100644
--- a/lib/galaxy/tools/errors.py
+++ b/lib/galaxy/tools/errors.py
@@ -11,9 +11,9 @@ GALAXY TOOL ERROR REPORT
This error report was sent from the Galaxy instance hosted on the server
"${host}"
-----------------------------------------------------------------------------
-This is in reference to dataset id ${dataset_id} from history id ${history_id}
+This is in reference to dataset id ${dataset_id} (${dataset_id_encoded}) from history id ${history_id} (${history_id_encoded})
-----------------------------------------------------------------------------
-You should be able to view the history containing the related history item
+You should be able to view the history containing the related history item (${hda_id_encoded})
${hid}: ${history_item_name}
@@ -26,7 +26,8 @@ The user ${email_str} provided the following information:
${message}
-----------------------------------------------------------------------------
-job id: ${job_id}
+info url: ${hda_show_params_link}
+job id: ${job_id} (${job_id_encoded})
tool id: ${job_tool_id}
tool version: ${tool_version}
job pid or drm id: ${job_runner_external_id}
@@ -60,7 +61,7 @@ class ErrorReporter( object ):
try:
hda = sa_session.query( model.HistoryDatasetAssociation ).get( hda_id )
assert hda is not None, ValueError( "No HDA yet" )
- except:
+ except Exception:
hda = sa_session.query( model.HistoryDatasetAssociation ).get( app.security.decode_id( hda_id ) )
assert isinstance( hda, model.HistoryDatasetAssociation ), ValueError( "Bad value provided for HDA (%s)." % ( hda ) )
self.hda = hda
@@ -81,7 +82,10 @@ class ErrorReporter( object ):
hda = self.hda
job = self.job
host = web.url_for( '/', qualified=True )
- history_view_link = web.url_for( controller="history", action="view", id=self.app.security.encode_id( hda.history_id ), qualified=True )
+ history_id_encoded = self.app.security.encode_id( hda.history_id )
+ history_view_link = web.url_for( controller="history", action="view", id=history_id_encoded, qualified=True )
+ hda_id_encoded = self.app.security.encode_id( hda.id )
+ hda_show_params_link = web.url_for( controller="dataset", action="show_params", dataset_id=hda_id_encoded, qualified=True )
# Build the email message
if user and user.email != email:
email_str = "'%s' (providing preferred contact email '%s')" % (user.email, email)
@@ -89,11 +93,16 @@ class ErrorReporter( object ):
email_str = "'%s'" % (email or 'anonymously')
self.report = string.Template( error_report_template ) \
.safe_substitute( host=host,
+ dataset_id_encoded=self.app.security.encode_id( hda.dataset_id ),
dataset_id=hda.dataset_id,
+ history_id_encoded=history_id_encoded,
history_id=hda.history_id,
+ hda_id_encoded=hda_id_encoded,
hid=hda.hid,
history_item_name=hda.get_display_name(),
history_view_link=history_view_link,
+ hda_show_params_link=hda_show_params_link,
+ job_id_encoded=self.app.security.encode_id( job.id ),
job_id=job.id,
tool_version=job.tool_version,
job_tool_id=job.tool_id,
@@ -135,7 +144,7 @@ class EmailErrorReporter( ErrorReporter ):
subject = "Galaxy tool error report from %s" % email
try:
subject = "%s (%s)" % ( subject, self.app.toolbox.get_tool( self.job.tool_id, self.job.tool_version ).old_id )
- except:
+ except Exception:
pass
# Send it
return util.send_mail( frm, to, subject, self.report, self.app.config )
diff --git a/lib/galaxy/tools/evaluation.py b/lib/galaxy/tools/evaluation.py
index 4b00d004363..3a9e4460d77 100644
--- a/lib/galaxy/tools/evaluation.py
+++ b/lib/galaxy/tools/evaluation.py
@@ -316,7 +316,7 @@ class ToolEvaluator( object ):
try:
open( dataset_path.false_path, 'w' ).close()
except EnvironmentError:
- pass # May well not exist - e.g. LWR.
+ pass # May well not exist - e.g. Pulsar.
else:
param_dict[name] = DatasetFilenameWrapper( hda )
# Provide access to a path to store additional files
diff --git a/lib/galaxy/tools/imp_exp/__init__.py b/lib/galaxy/tools/imp_exp/__init__.py
index 9574541d2aa..ec2aef117ff 100644
--- a/lib/galaxy/tools/imp_exp/__init__.py
+++ b/lib/galaxy/tools/imp_exp/__init__.py
@@ -17,42 +17,16 @@ from sqlalchemy.sql import expression
log = logging.getLogger(__name__)
-EXPORT_HISTORY_TEXT = """
-
-
-
- python $export_history $__EXPORT_HISTORY_COMMAND_INPUTS_OPTIONS__ $output_file
-
-
-
-
-
-
- from galaxy.tools.imp_exp.export_history import main; main()
-
-
-
-
-
-"""
-
def load_history_imp_exp_tools( toolbox ):
""" Adds tools for importing/exporting histories to archives. """
- # Use same process as that used in load_external_metadata_tool; see that
- # method for why create tool description files on the fly.
- tool_xml_text = EXPORT_HISTORY_TEXT
# Load export tool.
- tmp_name = tempfile.NamedTemporaryFile()
- tmp_name.write( tool_xml_text )
- tmp_name.flush()
- history_exp_tool = toolbox.load_hidden_tool( tmp_name.name )
+ history_exp_tool = toolbox.load_hidden_lib_tool( "galaxy/tools/imp_exp/exp_history_to_archive.xml" )
log.debug( "Loaded history export tool: %s", history_exp_tool.id )
# Load import tool.
- tool_xml = os.path.join( os.getcwd(), "lib/galaxy/tools/imp_exp/imp_history_from_archive.xml" )
- history_imp_tool = toolbox.load_hidden_tool( tool_xml )
+ history_imp_tool = toolbox.load_hidden_lib_tool( "galaxy/tools/imp_exp/imp_history_from_archive.xml" )
log.debug( "Loaded history import tool: %s", history_imp_tool.id )
diff --git a/lib/galaxy/tools/imp_exp/exp_history_to_archive.xml b/lib/galaxy/tools/imp_exp/exp_history_to_archive.xml
new file mode 100644
index 00000000000..9b2454150ab
--- /dev/null
+++ b/lib/galaxy/tools/imp_exp/exp_history_to_archive.xml
@@ -0,0 +1,16 @@
+
+
+
+ python $export_history $__EXPORT_HISTORY_COMMAND_INPUTS_OPTIONS__ $output_file
+
+
+
+
+
+
+ from galaxy.tools.imp_exp.export_history import main; main()
+
+
+
+
+
diff --git a/lib/galaxy/tools/parameters/basic.py b/lib/galaxy/tools/parameters/basic.py
index b00cc86c828..51cda184de6 100644
--- a/lib/galaxy/tools/parameters/basic.py
+++ b/lib/galaxy/tools/parameters/basic.py
@@ -816,7 +816,6 @@ class SelectToolParameter( ToolParameter ):
self.display = input_source.get( 'display', None )
self.separator = input_source.get( 'separator', ',' )
self.legal_values = set()
- # TODO: the tag is deprecated and should be replaced with the tag.
self.dynamic_options = input_source.get( "dynamic_options", None )
self.options = input_source.parse_dynamic_options(self)
if self.options is not None:
@@ -979,17 +978,14 @@ class SelectToolParameter( ToolParameter ):
return super( SelectToolParameter, self ).value_from_basic( value, app, ignore_errors=ignore_errors )
def get_initial_value( self, trans, context, history=None ):
- # More working around dynamic options for workflow
options = list( self.get_options( trans, context ) )
if len(options) == 0 and (trans is None or trans.workflow_building_mode):
- # Really the best we can do?
return UnvalidatedValue( None )
value = [ optval for _, optval, selected in options if selected ]
if len( value ) == 0:
- if not self.multiple and options:
- # Nothing selected, but not a multiple select, with some values,
+ if not self.optional and not self.multiple and options:
+ # Nothing selected, but not optional and not a multiple select, with some values,
# so we have to default to something (the HTML form will anyway)
- # TODO: deal with optional parameters in a better way
value = options[0][1]
else:
value = None
@@ -1764,6 +1760,22 @@ class DataToolParameter( BaseDataToolParameter ):
self.validators.append( validation.MetadataValidator() )
self._parse_formats( trans, tool, input_source )
self.multiple = input_source.get_bool('multiple', False)
+ self.min = input_source.get( 'min' )
+ self.max = input_source.get( 'max' )
+ if self.min:
+ try:
+ self.min = int( self.min )
+ except:
+ raise ValueError( "An integer is required for min property." )
+ if self.max:
+ try:
+ self.max = int( self.max )
+ except:
+ raise ValueError( "An integer is required for max property." )
+ if not self.multiple and (self.min is not None):
+ raise ValueError( "Cannot specify min property on single data parameter '%s'. Set multiple=\"true\" to enable this option." % self.name )
+ if not self.multiple and (self.max is not None):
+ raise ValueError( "Cannot specify max property on single data parameter '%s'. Set multiple=\"true\" to enable this option." % self.name )
self.is_dynamic = True
self._parse_options( input_source )
# Load conversions required for the dataset input
@@ -2052,17 +2064,31 @@ class DataToolParameter( BaseDataToolParameter ):
return "No dataset"
def validate( self, value, history=None ):
+ dataset_count = 0
for validator in self.validators:
if value and self.multiple:
- if isinstance(value, list):
- for v in value:
- validator.validate( v, history )
- elif isinstance(value, galaxy.model.HistoryDatasetCollectionAssociation):
- for v in value.collection.dataset_instances:
+ if not isinstance( value, list ):
+ value = [ value ]
+ for v in value:
+ if isinstance(v, galaxy.model.HistoryDatasetCollectionAssociation):
+ for dataset_instance in v.collection.dataset_instances:
+ dataset_count += 1
+ validator.validate( dataset_instance, history )
+ else:
+ dataset_count += 1
validator.validate( v, history )
else:
+ if value:
+ dataset_count += 1
validator.validate( value, history )
+ if self.min is not None:
+ if self.min > dataset_count:
+ raise ValueError( "At least %d datasets are required." % self.min )
+ if self.max is not None:
+ if self.max < dataset_count:
+ raise ValueError( "At most %d datasets are required." % self.max )
+
def get_dependencies( self ):
"""
Get the *names* of the other params this param depends on.
@@ -2124,6 +2150,10 @@ class DataToolParameter( BaseDataToolParameter ):
d['extensions'] = extensions
d['edam_formats'] = edam_formats
d['multiple'] = self.multiple
+ if self.multiple:
+ # For consistency, should these just always be in the dict?
+ d['min'] = self.min
+ d['max'] = self.max
d['options'] = {'hda': [], 'hdca': []}
# return default content if context is not available
diff --git a/lib/galaxy/tools/parameters/grouping.py b/lib/galaxy/tools/parameters/grouping.py
index 6c238fcb406..f99c48ae023 100644
--- a/lib/galaxy/tools/parameters/grouping.py
+++ b/lib/galaxy/tools/parameters/grouping.py
@@ -573,7 +573,8 @@ class Conditional( Group ):
current_case = rval['__current_case__'] = value['__current_case__']
rval[ self.test_param.name ] = self.test_param.value_to_basic( value[ self.test_param.name ], app )
for input in self.cases[current_case].inputs.itervalues():
- rval[ input.name ] = input.value_to_basic( value[ input.name ], app )
+ if input.name in value: # parameter might be absent in unverified workflow
+ rval[ input.name ] = input.value_to_basic( value[ input.name ], app )
return rval
def value_from_basic( self, value, app, ignore_errors=False ):
diff --git a/lib/galaxy/tools/parameters/meta.py b/lib/galaxy/tools/parameters/meta.py
index 2c91a702224..77076a76bae 100644
--- a/lib/galaxy/tools/parameters/meta.py
+++ b/lib/galaxy/tools/parameters/meta.py
@@ -35,7 +35,7 @@ def expand_meta_parameters( trans, tool, incoming ):
classification = permutations.input_classification.SINGLE
if __collection_multirun_parameter( value ):
collection_value = value[ 'values' ][ 0 ]
- values = __expand_collection_parameter( trans, input_key, collection_value, collections_to_match )
+ values = __expand_collection_parameter( trans, input_key, collection_value, collections_to_match, linked=is_linked )
else:
values = value[ 'values' ]
else:
@@ -102,7 +102,7 @@ def expand_meta_parameters( trans, tool, incoming ):
return expanded_incomings, collection_info
-def __expand_collection_parameter( trans, input_key, incoming_val, collections_to_match ):
+def __expand_collection_parameter( trans, input_key, incoming_val, collections_to_match, linked=False ):
# If subcollectin multirun of data_collection param - value will
# be "hdca_id|subcollection_type" else it will just be hdca_id
if "|" in incoming_val:
@@ -119,7 +119,7 @@ def __expand_collection_parameter( trans, input_key, incoming_val, collections_t
subcollection_type = None
hdc_id = trans.app.security.decode_id( encoded_hdc_id )
hdc = trans.sa_session.query( model.HistoryDatasetCollectionAssociation ).get( hdc_id )
- collections_to_match.add( input_key, hdc, subcollection_type=subcollection_type )
+ collections_to_match.add( input_key, hdc, subcollection_type=subcollection_type, linked=linked )
if subcollection_type is not None:
from galaxy.dataset_collections import subcollections
subcollection_elements = subcollections.split_dataset_collection_instance( hdc, subcollection_type )
diff --git a/lib/galaxy/tools/parameters/output_collect.py b/lib/galaxy/tools/parameters/output_collect.py
index b673e1f787c..7403705d64e 100644
--- a/lib/galaxy/tools/parameters/output_collect.py
+++ b/lib/galaxy/tools/parameters/output_collect.py
@@ -355,6 +355,8 @@ class DatasetCollector( object ):
self.pattern = pattern
self.default_dbkey = kwargs.get( "dbkey", None )
self.default_ext = kwargs.get( "ext", None )
+ if self.default_ext is None and "format" in kwargs:
+ self.default_ext = kwargs.get( "format" )
self.default_visible = util.asbool( kwargs.get( "visible", None ) )
self.directory = kwargs.get( "directory", None )
self.assign_primary_output = util.asbool( kwargs.get( 'assign_primary_output', False ) )
diff --git a/lib/galaxy/tools/parser/yaml.py b/lib/galaxy/tools/parser/yaml.py
index f4f05a15c9e..9c59b623c99 100644
--- a/lib/galaxy/tools/parser/yaml.py
+++ b/lib/galaxy/tools/parser/yaml.py
@@ -75,7 +75,7 @@ class YamlToolSource(ToolSource):
for output in output_defs:
outputs[output.name] = output
# TODO: parse outputs collections
- return output_defs, odict()
+ return outputs, odict()
def _parse_output(self, tool, name, output_dict):
# TODO: handle filters, actions, change_format
@@ -156,6 +156,8 @@ def _parse_test(i, test_dict):
_ensure_has(attributes, defaults)
test_dict["outputs"] = new_outputs
+ # TODO: implement output collections for YAML tools.
+ test_dict["output_collections"] = []
test_dict["command"] = __to_test_assert_list( test_dict.get( "command", [] ) )
test_dict["stdout"] = __to_test_assert_list( test_dict.get( "stdout", [] ) )
test_dict["stderr"] = __to_test_assert_list( test_dict.get( "stderr", [] ) )
diff --git a/lib/galaxy/tools/test.py b/lib/galaxy/tools/test.py
index e98527c1d0e..5e34e4235e0 100644
--- a/lib/galaxy/tools/test.py
+++ b/lib/galaxy/tools/test.py
@@ -157,7 +157,8 @@ class ToolTestBuilder( object ):
for input_name, input_value in case.inputs.items():
case_inputs = self.__process_raw_inputs( { input_name: input_value }, raw_inputs, parent_context=cond_context )
expanded_inputs.update( case_inputs )
- expanded_case_value = self.__split_if_str( case.value )
+ if not value.type == "text":
+ expanded_case_value = self.__split_if_str( case.value )
if case_value is not None:
# A bit tricky here - we are growing inputs with value
# that may be implicit (i.e. not defined by user just
@@ -192,7 +193,8 @@ class ToolTestBuilder( object ):
raw_input = context.extract_value( raw_inputs )
if raw_input:
(name, param_value, param_extra) = raw_input
- param_value = self.__split_if_str( param_value )
+ if not value.type == "text":
+ param_value = self.__split_if_str( param_value )
if isinstance( value, galaxy.tools.parameters.basic.DataToolParameter ):
if not isinstance(param_value, list):
param_value = [ param_value ]
diff --git a/lib/galaxy/tools/toolbox/base.py b/lib/galaxy/tools/toolbox/base.py
index 2b8a0df45aa..1238cb75f11 100644
--- a/lib/galaxy/tools/toolbox/base.py
+++ b/lib/galaxy/tools/toolbox/base.py
@@ -738,6 +738,10 @@ class AbstractToolBox( object, Dictifiable, ManagesIntegratedToolPanelMixin ):
self._tool_watcher.watch_file( config_file, tool.id )
return tool
+ def load_hidden_lib_tool( self, path ):
+ tool_xml = os.path.join( os.getcwd(), "lib", path )
+ return self.load_hidden_tool( tool_xml )
+
def load_hidden_tool( self, config_file, **kwds ):
""" Load a hidden tool (in this context meaning one that does not
appear in the tool panel) and register it in _tools_by_id.
diff --git a/lib/galaxy/tools/wrappers.py b/lib/galaxy/tools/wrappers.py
index 1fc5b75b97a..43583842008 100644
--- a/lib/galaxy/tools/wrappers.py
+++ b/lib/galaxy/tools/wrappers.py
@@ -9,7 +9,7 @@ log = getLogger( __name__ )
# Fields in .log files corresponding to paths, must have one of the following
# field names and all such fields are assumed to be paths. This is to allow
-# remote ComputeEnvironments (such as one used by LWR) determine what values to
+# remote ComputeEnvironments (such as one used by Pulsar) determine what values to
# rewrite or transfer...
PATH_ATTRIBUTES = [ "path" ]
# ... by default though - don't rewrite anything (if no ComputeEnviornment
diff --git a/lib/galaxy/util/__init__.py b/lib/galaxy/util/__init__.py
index b6a3d426711..e04c973f07c 100644
--- a/lib/galaxy/util/__init__.py
+++ b/lib/galaxy/util/__init__.py
@@ -347,7 +347,10 @@ def pretty_print_time_interval( time=False, precise=False ):
diff = now - datetime.fromtimestamp( time )
elif isinstance( time, datetime ):
diff = now - time
- elif not time:
+ elif isinstance( time, basestring ):
+ time = datetime.strptime( time, "%Y-%m-%dT%H:%M:%S.%f" )
+ diff = now - time
+ else:
diff = now - now
second_diff = diff.seconds
day_diff = diff.days
diff --git a/lib/galaxy/visualization/plugins/config_parser.py b/lib/galaxy/visualization/plugins/config_parser.py
index df1a5c52607..eb79095d362 100644
--- a/lib/galaxy/visualization/plugins/config_parser.py
+++ b/lib/galaxy/visualization/plugins/config_parser.py
@@ -217,10 +217,8 @@ class DataSourceParser( object ):
returned[ 'model_class' ] = model_class
# tests (optional, 0 or more) - data for boolean test: 'is the visualization usable by this object?'
- tests = self.parse_tests( xml_tree.findall( 'test' ) )
# when no tests are given, default to isinstance( object, model_class )
- if tests:
- returned[ 'tests' ] = tests
+ returned[ 'tests' ] = self.parse_tests( xml_tree.findall( 'test' ) )
# to_params (optional, 0 or more) - tells the registry to set certain params based on the model_clas, tests
returned[ 'to_params' ] = {}
diff --git a/lib/galaxy/web/base/interactive_environments.py b/lib/galaxy/web/base/interactive_environments.py
index e3650992944..52c5886f569 100644
--- a/lib/galaxy/web/base/interactive_environments.py
+++ b/lib/galaxy/web/base/interactive_environments.py
@@ -1,6 +1,7 @@
import ConfigParser
import os
+import json
import stat
import random
import tempfile
@@ -40,18 +41,15 @@ class InteractiveEnviornmentRequest(object):
self.load_deploy_config()
self.attr.docker_hostname = self.attr.viz_config.get("docker", "docker_hostname")
- self.attr.proxy_request = trans.app.proxy_manager.setup_proxy(
- trans, host=self.attr.docker_hostname
- )
- self.attr.proxy_url = self.attr.proxy_request[ 'proxy_url' ]
- self.attr.PORT = self.attr.proxy_request[ 'proxied_port' ]
# Generate per-request passwords the IE plugin can use to configure
# the destination container.
self.notebook_pw_salt = self.generate_password(length=12)
self.notebook_pw = self.generate_password(length=24)
- self.temp_dir = os.path.abspath( tempfile.mkdtemp() )
+ ie_parent_temp_dir = self.attr.viz_config.get("docker", "docker_galaxy_temp_dir") or None
+ self.temp_dir = os.path.abspath( tempfile.mkdtemp( dir=ie_parent_temp_dir ) )
+
if self.attr.viz_config.getboolean("docker", "wx_tempdir"):
# Ensure permissions are set
try:
@@ -60,14 +58,26 @@ class InteractiveEnviornmentRequest(object):
log.error( "Could not change permissions of tmpdir %s" % self.temp_dir )
# continue anyway
+ # This duplicates the logic in the proxy manager
+ if self.attr.galaxy_config.dynamic_proxy_external_proxy:
+ self.attr.proxy_prefix = '%s/%s' % (
+ self.attr.galaxy_config.cookie_path,
+ self.attr.galaxy_config.dynamic_proxy_prefix)
+ else:
+ self.attr.proxy_prefix = ''
+
def load_deploy_config(self, default_dict={}):
# For backwards compat, any new variables added to the base .ini file
# will need to be recorded here. The ConfigParser doesn't provide a
# .get() that will ignore missing sections, so we must make use of
# their defaults dictionary instead.
- default_dict['command_inject'] = '--sig-proxy=true'
- default_dict['docker_hostname'] = 'localhost'
- default_dict['wx_tempdir'] = False
+ default_dict = {
+ 'command': 'docker {docker_args}',
+ 'command_inject': '--sig-proxy=true -e DEBUG=false',
+ 'docker_hostname': 'localhost',
+ 'wx_tempdir': 'False',
+ 'docker_galaxy_temp_dir': None
+ }
viz_config = ConfigParser.SafeConfigParser(default_dict)
conf_path = os.path.join( self.attr.our_config_dir, self.attr.viz_id + ".ini" )
if not os.path.exists( conf_path ):
@@ -101,8 +111,10 @@ class InteractiveEnviornmentRequest(object):
'history_id': self.attr.history_id,
'api_key': api_key,
'remote_host': request.remote_addr,
- 'docker_port': self.attr.PORT,
+ # DOCKER_PORT is NO LONGER AVAILABLE. All IEs must update.
'cors_origin': request.host_url,
+ 'user_email': self.trans.user.email,
+ 'proxy_prefix': self.attr.proxy_prefix,
}
if self.attr.viz_config.has_option("docker", "galaxy_url"):
@@ -137,36 +149,27 @@ class InteractiveEnviornmentRequest(object):
return "false"
def url_template(self, url_template):
+ """Process a URL template
+
+ There are several variables accessible to the user:
+
+ - ${PROXY_URL} will be replaced with the dynamically create proxy's url
+ - ${PROXY_PREFIX} will be replaced with the prefix that may occur
"""
- Process a URL template
-
- There are several variables accessible to the user:
-
- - ${PROXY_URL} will be replaced with dynamically create proxy
- - ${PORT} will be replaced with the port the docker image is attached to
- """
- # Figure out our substitutions
-
# Next several lines for older style replacements (not used with Galaxy dynamic
# proxy)
-
if self.attr.SSL_URLS:
protocol = 'https'
else:
protocol = 'http'
- if not self.attr.APACHE_URLS:
- # If they are not using apache URLs, that implies there's a port attached to the host
- # string, thus we replace just the first instance of host that we see.
- url_template = url_template.replace('${HOST}', '${HOST}:${PORT}', 1)
-
url_template = url_template.replace('${PROTO}', protocol) \
.replace('${HOST}', self.attr.HOST)
# Only the following replacements are used with Galaxy dynamic proxy
# URLs
url = url_template.replace('${PROXY_URL}', str(self.attr.proxy_url)) \
- .replace('${PORT}', str(self.attr.PORT))
+ .replace('${PROXY_PREFIX}', str(self.attr.proxy_prefix.replace('/', '%2F')))
return url
def volume(self, host_path, container_path, **kwds):
@@ -181,14 +184,22 @@ class InteractiveEnviornmentRequest(object):
conf.update(env_override)
env_str = ' '.join(['-e "%s=%s"' % (key.upper(), item) for key, item in conf.items()])
volume_str = ' '.join(['-v "%s"' % volume for volume in volumes])
- return '%s run %s -d %s -p %s:%s -v "%s:/import/" %s %s' % \
- (self.attr.viz_config.get("docker", "command"),
- env_str,
- self.attr.viz_config.get("docker", "command_inject"),
- self.attr.PORT, self.attr.docker_port,
- temp_dir,
- volume_str,
- self.attr.viz_config.get("docker", "image"))
+ # This is the basic docker command such as "sudo -u docker docker {docker_args}"
+ # or just "docker {docker_args}"
+ command = self.attr.viz_config.get("docker", "command")
+ # Then we format in the entire docker command in place of
+ # {docker_args}, so as to let the admin not worry about which args are
+ # getting passed
+ command = command.format(docker_args='run {command_inject} {environment} -d -P -v "{temp_dir}:/import/" {volume_str} {image}')
+ # Once that's available, we format again with all of our arguments
+ command = command.format(
+ command_inject=self.attr.viz_config.get("docker", "command_inject"),
+ environment=env_str,
+ temp_dir=temp_dir,
+ volume_str=volume_str,
+ image=self.attr.viz_config.get("docker", "image")
+ )
+ return command
def launch(self, raw_cmd=None, env_override={}, volumes=[]):
if raw_cmd is None:
@@ -201,5 +212,81 @@ class InteractiveEnviornmentRequest(object):
stdout, stderr = p.communicate()
if p.returncode != 0 or len(stderr):
log.error( "%s\n%s" % (stdout, stderr) )
+ return None
else:
log.debug( "Container id: %s" % stdout)
+ port_mappings = self.get_proxied_ports(stdout)
+ if len(port_mappings) > 1:
+ log.warning("Don't know how to handle proxies to containers with multiple exposed ports. Arbitrarily choosing first")
+ elif len(port_mappings) == 0:
+ log.warning("No exposed ports to map! Images MUST EXPOSE")
+ return None
+ # Fetch the first port_mapping
+ (service, host_ip, host_port) = port_mappings[0]
+
+ # Now we configure our proxy_requst object and we manually specify
+ # the port to map to and ensure the proxy is available.
+ self.attr.proxy_request = self.trans.app.proxy_manager.setup_proxy(
+ self.trans,
+ host=self.attr.docker_hostname,
+ port=host_port,
+ proxy_prefix=self.attr.proxy_prefix,
+ )
+ # These variables then become available for use in templating URLs
+ self.attr.proxy_url = self.attr.proxy_request[ 'proxy_url' ]
+ # Commented out because it needs to be documented and visible that
+ # this variable was moved here. Usually would remove commented
+ # code, but again, needs to be clear where this went. Remove at a
+ # later time.
+ #
+ # PORT is no longer exposed internally. All requests are forced to
+ # go through the proxy we ship.
+ # self.attr.PORT = self.attr.proxy_request[ 'proxied_port' ]
+
+ def get_proxied_ports(self, container_id):
+ """Run docker inspect on a container to figure out which ports were
+ mapped where.
+
+ :type container_id: str
+ :param container_id: a docker container ID
+
+ :returns: a list of triples containing (internal_port, external_ip,
+ external_port), of which the ports are probably the only
+ useful information.
+
+ Someday code that calls this should be refactored whenever we get
+ containers with multiple ports working.
+ """
+ command = self.attr.viz_config.get("docker", "command")
+ command = command.format(docker_args="inspect %s" % container_id)
+ log.info("Inspecting docker container {0} with command [{1}]".format(
+ container_id,
+ command
+ ))
+
+ p = Popen(command, stdout=PIPE, stderr=PIPE, close_fds=True, shell=True)
+ stdout, stderr = p.communicate()
+ if p.returncode != 0 or len(stderr):
+ log.error( "%s\n%s" % (stdout, stderr) )
+ return None
+
+ inspect_data = json.loads(stdout)
+ # [{
+ # "NetworkSettings" : {
+ # "Ports" : {
+ # "3306/tcp" : [
+ # {
+ # "HostIp" : "127.0.0.1",
+ # "HostPort" : "3306"
+ # }
+ # ]
+ mappings = []
+ port_mappings = inspect_data[0]['NetworkSettings']['Ports']
+ for port_name in port_mappings:
+ for binding in port_mappings[port_name]:
+ mappings.append((
+ port_name.replace('/tcp', '').replace('/udp', ''),
+ binding['HostIp'],
+ binding['HostPort']
+ ))
+ return mappings
diff --git a/lib/galaxy/web/framework/middleware/remoteuser.py b/lib/galaxy/web/framework/middleware/remoteuser.py
index 10bd432df71..4dbc1d7f8ba 100644
--- a/lib/galaxy/web/framework/middleware/remoteuser.py
+++ b/lib/galaxy/web/framework/middleware/remoteuser.py
@@ -106,28 +106,34 @@ class RemoteUser( object ):
before you may access Galaxy.
"""
return self.error( start_response, title, message )
+
+ user_accessible_paths = (
+ '/user/api_keys',
+ '/user/edit_username',
+ '/user/dbkeys',
+ '/user/toolbox_filters',
+ '/user/set_default_permissions',
+ )
+
+ admin_accessible_paths = (
+ '/user/create',
+ '/user/logout',
+ '/user/manage_user_info',
+ '/user/edit_info',
+ '/userskeys/all_users',
+ )
+
if not path_info.startswith('/user'):
# shortcut the following whitelist for non-user-controller
# requests.
pass
- elif path_info.startswith( '/user/create' ) and environ[ self.remote_user_header ] in self.admin_users:
- pass # admins can create users
- elif path_info.startswith( '/user/logout' ) and environ[ self.remote_user_header ] in self.admin_users:
- pass # Admin users may be impersonating, allow logout.
- elif path_info.startswith( '/user/manage_user_info' ) and environ[ self.remote_user_header ] in self.admin_users:
- pass # Admin users need to be able to change user information
- elif path_info.startswith( '/user/edit_info' ) and environ[ self.remote_user_header ] in self.admin_users:
- pass # Admin users need to be able to change user information
- elif path_info.startswith( '/user/api_keys' ):
- pass # api keys can be managed when remote_user is in use
- elif path_info.startswith( '/user/edit_username' ):
- pass # username can be managed when remote_user is in use
- elif path_info.startswith( '/user/dbkeys' ):
- pass # dbkeys can be managed when remote_user is in use
- elif path_info.startswith( '/user/toolbox_filters' ):
- pass # toolbox filters can be managed when remote_user is in use
- elif path_info.startswith( '/user/set_default_permissions' ):
- pass # default permissions can be managed when remote_user is in use
+ elif environ[self.remote_user_header] in self.admin_users and \
+ any([path_info.startswith(prefix) for prefix in admin_accessible_paths]):
+ # If the user is an admin user, and any of the admin accessible paths match..., allow them to execute that action.
+ pass
+ elif any([path_info.startswith(prefix) for prefix in user_accessible_paths]):
+ # If the user is allowed to access the path, pass
+ pass
elif path_info == '/user' or path_info == '/user/':
pass # We do allow access to the root user preferences page.
elif path_info.startswith( '/user' ):
diff --git a/lib/galaxy/web/framework/middleware/statsd.py b/lib/galaxy/web/framework/middleware/statsd.py
new file mode 100644
index 00000000000..eb33d43ae2a
--- /dev/null
+++ b/lib/galaxy/web/framework/middleware/statsd.py
@@ -0,0 +1,36 @@
+"""
+Middleware for sending request statistics to statsd
+"""
+from __future__ import absolute_import
+import time
+
+try:
+ import statsd
+except ImportError:
+ # This middleware will never be used without statsd. This block allows
+ # unit tests pass on systems without it.
+ statsd = None
+
+
+class StatsdMiddleware(object):
+ """
+ This middleware will log request durations to the configured statsd
+ instance.
+ """
+
+ def __init__(self,
+ application,
+ statsd_host,
+ statsd_port):
+ if not statsd:
+ raise ImportError( "Statsd middleware configured, but no statsd python module found. "
+ "Please install the python statsd module to use this functionality." )
+ self.application = application
+ self.statsd_client = statsd.StatsClient(statsd_host, statsd_port, prefix='galaxy')
+
+ def __call__(self, environ, start_response):
+ start_time = time.time()
+ req = self.application(environ, start_response)
+ dt = int((time.time() - start_time) * 1000)
+ self.statsd_client.timing(environ.get('PATH_INFO', "NOPATH").strip('/').replace('/', '.'), dt)
+ return req
diff --git a/lib/galaxy/web/framework/webapp.py b/lib/galaxy/web/framework/webapp.py
index cc4014e339e..f2b5a59618b 100644
--- a/lib/galaxy/web/framework/webapp.py
+++ b/lib/galaxy/web/framework/webapp.py
@@ -458,6 +458,7 @@ class GalaxyWebTransaction( base.DefaultWebTransaction,
url_for( controller='user', action='manage_user_info' ),
url_for( controller='user', action='set_default_permissions' ),
url_for( controller='user', action='reset_password' ),
+ url_for( controller='user', action='change_password' ),
url_for( controller='user', action='openid_auth' ),
url_for( controller='user', action='openid_process' ),
url_for( controller='user', action='openid_associate' ),
diff --git a/lib/galaxy/web/proxy/__init__.py b/lib/galaxy/web/proxy/__init__.py
index 30738ad6c11..a23b42a285c 100644
--- a/lib/galaxy/web/proxy/__init__.py
+++ b/lib/galaxy/web/proxy/__init__.py
@@ -17,7 +17,9 @@ SECURE_COOKIE = "galaxysession"
class ProxyManager(object):
def __init__( self, config ):
- for option in [ "manage_dynamic_proxy", "dynamic_proxy_bind_port", "dynamic_proxy_bind_ip", "dynamic_proxy_debug", "dynamic_proxy_external_proxy" ]:
+ for option in ["manage_dynamic_proxy", "dynamic_proxy_bind_port",
+ "dynamic_proxy_bind_ip", "dynamic_proxy_debug",
+ "dynamic_proxy_external_proxy", "dynamic_proxy_prefix"]:
setattr( self, option, getattr( config, option ) )
self.launch_by = "node" # TODO: Support docker
if self.manage_dynamic_proxy:
@@ -29,7 +31,7 @@ class ProxyManager(object):
def shutdown( self ):
self.lazy_process.shutdown()
- def setup_proxy( self, trans, host=DEFAULT_PROXY_TO_HOST, port=None ):
+ def setup_proxy( self, trans, host=DEFAULT_PROXY_TO_HOST, port=None, proxy_prefix="" ):
if self.manage_dynamic_proxy:
log.info("Attempting to start dynamic proxy process")
self.lazy_process.start_process()
@@ -46,7 +48,7 @@ class ProxyManager(object):
if not self.dynamic_proxy_external_proxy:
proxy_url = '%s://%s:%d' % (scheme, host, self.dynamic_proxy_bind_port)
else:
- proxy_url = '%s://%s' % (scheme, host)
+ proxy_url = '%s://%s%s' % (scheme, host, proxy_prefix)
return {
'proxy_url': proxy_url,
'proxied_port': proxy_requests.port,
diff --git a/lib/galaxy/web/proxy/js/package.json b/lib/galaxy/web/proxy/js/package.json
index 2cd17edacbf..f97f0cb1941 100644
--- a/lib/galaxy/web/proxy/js/package.json
+++ b/lib/galaxy/web/proxy/js/package.json
@@ -14,6 +14,6 @@
"eventemitter3": "0.1.6",
"http-proxy": "1.6.0",
"commander": "~2.2",
- "sqlite3": "3.0.2"
+ "sqlite3": "3.0.5"
}
}
diff --git a/lib/galaxy/webapps/galaxy/api/folders.py b/lib/galaxy/webapps/galaxy/api/folders.py
index 8671bdd336f..a223b96cc4e 100644
--- a/lib/galaxy/webapps/galaxy/api/folders.py
+++ b/lib/galaxy/webapps/galaxy/api/folders.py
@@ -229,16 +229,16 @@ class FoldersController( BaseAPIController, UsesLibraryMixin, UsesLibraryMixinIt
return self.folder_manager.get_current_roles( trans, folder )
@expose_api
- def delete( self, trans, id, **kwd ):
+ def delete( self, trans, encoded_folder_id, **kwd ):
"""
- delete( self, trans, id, **kwd )
- * DELETE /api/folders/{id}
- marks the folder with the given ``id`` as `deleted` (or removes the `deleted` mark if the `undelete` param is true)
+ delete( self, trans, encoded_folder_id, **kwd )
+ * DELETE /api/folders/{encoded_folder_id}
+ marks the folder with the given ``encoded_folder_id`` as `deleted` (or removes the `deleted` mark if the `undelete` param is true)
.. note:: Currently, only admin users can un/delete folders.
- :param id: the encoded id of the folder to un/delete
- :type id: an encoded id string
+ :param encoded_folder_id: the encoded id of the folder to un/delete
+ :type encoded_folder_id: an encoded id string
:param undelete: (optional) flag specifying whether the item should be deleted or undeleted, defaults to false:
:type undelete: bool
@@ -246,21 +246,47 @@ class FoldersController( BaseAPIController, UsesLibraryMixin, UsesLibraryMixinIt
:returns: detailed folder information
:rtype: dictionary
- :raises: ItemAccessibilityException, MalformedId, ObjectNotFound
"""
- folder = self.folder_manager.get( trans, self.folder_manager.cut_and_decode( trans, id ), True )
+ folder = self.folder_manager.get( trans, self.folder_manager.cut_and_decode( trans, encoded_folder_id ), True )
undelete = util.string_as_bool( kwd.get( 'undelete', False ) )
folder = self.folder_manager.delete( trans, folder, undelete )
folder_dict = self.folder_manager.get_folder_dict( trans, folder )
return folder_dict
@expose_api
- def update( self, trans, id, library_id, payload, **kwd ):
+ def update( self, trans, encoded_folder_id, **kwd ):
"""
- PUT /api/folders/{encoded_folder_id}
+ * PATCH /api/folders/{encoded_folder_id}
+ Updates the folder defined by an ``encoded_folder_id`` with the data in the payload.
+ .. note:: Currently, only admin users can update library folders. Also the folder must not be `deleted`.
+
+ :param id: the encoded id of the folder
+ :type id: an encoded id string
+
+ :param payload: (required) dictionary structure containing::
+ 'name': new folder's name, cannot be empty
+ 'description': new folder's description
+ :type payload: dict
+
+ :returns: detailed folder information
+ :rtype: dict
+
+ :raises: RequestParameterMissingException
"""
- raise exceptions.NotImplemented( 'Updating folder through this endpoint is not implemented yet.' )
+ decoded_folder_id = self.folder_manager.cut_and_decode( trans, encoded_folder_id )
+ folder = self.folder_manager.get( trans, decoded_folder_id )
+ payload = kwd.get( 'payload', None )
+ if payload:
+ name = payload.get( 'name', None )
+ if not name:
+ raise exceptions.RequestParameterMissingException( "Parameter 'name' of library folder is required. You cannot remove it." )
+ description = payload.get( 'description', None )
+ else:
+ raise exceptions.RequestParameterMissingException( "You did not specify any payload." )
+ updated_folder = self.folder_manager.update( trans, folder, name, description )
+ folder_dict = self.folder_manager.get_folder_dict( trans, updated_folder )
+ return folder_dict
# TODO move to Role manager
def _load_role( self, trans, role_name ):
diff --git a/lib/galaxy/webapps/galaxy/api/jobs.py b/lib/galaxy/webapps/galaxy/api/jobs.py
index a2839099f5f..db5272d6038 100644
--- a/lib/galaxy/webapps/galaxy/api/jobs.py
+++ b/lib/galaxy/webapps/galaxy/api/jobs.py
@@ -202,10 +202,10 @@ class JobController( BaseAPIController, UsesLibraryMixinItems ):
job = self.__get_job(trans, id)
if not job:
raise exceptions.ObjectNotFound("Could not access job with id '%s'" % id)
- tool_id = job.tool_id
- tool_version = job.tool_version
- tool = self.app.toolbox.get_tool( tool_id, tool_version )
- return tool.to_json(trans, kwd={'__job_id__' : id})
+ tool = self.app.toolbox.get_tool( job.tool_id, job.tool_version )
+ if not tool.is_workflow_compatible:
+ raise exceptions.ConfigDoesNotAllowException( "Tool '%s' cannot be rerun." % ( job.tool_id ) )
+ return tool.to_json(trans, {}, job=job)
def __dictify_associations( self, trans, *association_lists ):
rval = []
diff --git a/lib/galaxy/webapps/galaxy/api/lda_datasets.py b/lib/galaxy/webapps/galaxy/api/lda_datasets.py
index 439d404d5c0..7dcab2df335 100644
--- a/lib/galaxy/webapps/galaxy/api/lda_datasets.py
+++ b/lib/galaxy/webapps/galaxy/api/lda_datasets.py
@@ -477,7 +477,7 @@ class LibraryDatasetsController( BaseAPIController, UsesVisualizationMixin ):
tool_id = 'upload1'
tool = trans.app.toolbox.get_tool( tool_id )
state = tool.new_state( trans )
- tool.update_state( trans, tool.inputs_by_page[ 0 ], state.inputs, kwd )
+ tool.populate_state( trans, tool.inputs, state.inputs, kwd )
tool_params = state.inputs
dataset_upload_inputs = []
for input in tool.inputs.itervalues():
diff --git a/lib/galaxy/webapps/galaxy/api/tools.py b/lib/galaxy/webapps/galaxy/api/tools.py
index e498ee3ee9e..e4a3112d07e 100644
--- a/lib/galaxy/webapps/galaxy/api/tools.py
+++ b/lib/galaxy/webapps/galaxy/api/tools.py
@@ -1,7 +1,6 @@
import urllib
-from galaxy.exceptions import ObjectNotFound
-from galaxy.exceptions import InternalServerError
+from galaxy import exceptions
from galaxy import web, util
from galaxy import managers
from galaxy.web import _future_expose_api_anonymous_and_sessionless as expose_api_anonymous_and_sessionless
@@ -72,7 +71,7 @@ class ToolsController( BaseAPIController, UsesVisualizationMixin ):
try:
return self.app.toolbox.to_dict( trans, in_panel=in_panel, trackster=trackster)
except Exception:
- raise InternalServerError( "Error: Could not convert toolbox to dictionary" )
+ raise exceptions.InternalServerError( "Error: Could not convert toolbox to dictionary" )
@expose_api_anonymous_and_sessionless
def show( self, trans, id, **kwd ):
@@ -206,7 +205,7 @@ class ToolsController( BaseAPIController, UsesVisualizationMixin ):
trans.response.headers[ "Content-Disposition" ] = 'attachment; filename="%s.tgz"' % ( id )
return download_file
- @web.expose_api_anonymous
+ @expose_api_anonymous
def create( self, trans, payload, **kwd ):
"""
POST /api/tools
@@ -223,13 +222,12 @@ class ToolsController( BaseAPIController, UsesVisualizationMixin ):
tool_version = payload.get( 'tool_version', None )
tool = trans.app.toolbox.get_tool( payload[ 'tool_id' ] , tool_version ) if 'tool_id' in payload else None
if not tool or not tool.allow_user_access( trans.user ):
- trans.response.status = 404
- return { "message": { "type": "error", "text" : trans.app.model.Dataset.conversion_messages.NO_TOOL } }
+ raise exceptions.MessageException( 'Tool not found or not accessible.' )
# Set running history from payload parameters.
# History not set correctly as part of this API call for
# dataset upload.
- history_id = payload.get("history_id", None)
+ history_id = payload.get('history_id', None)
if history_id:
decoded_id = self.decode_id( history_id )
target_history = self.history_manager.get_owned( decoded_id, trans.user, current_history=trans.history )
@@ -240,7 +238,7 @@ class ToolsController( BaseAPIController, UsesVisualizationMixin ):
inputs = payload.get( 'inputs', {} )
# Find files coming in as multipart file data and add to inputs.
for k, v in payload.iteritems():
- if k.startswith("files_") or k.startswith("__files_"):
+ if k.startswith('files_') or k.startswith('__files_'):
inputs[k] = v
# for inputs that are coming from the Library, copy them into the history
@@ -254,43 +252,22 @@ class ToolsController( BaseAPIController, UsesVisualizationMixin ):
for k, v in input_patch.iteritems():
inputs[k] = v
- # HACK: add run button so that tool.handle_input will run tool.
- inputs['runtool_btn'] = 'Execute'
# TODO: encode data ids and decode ids.
# TODO: handle dbkeys
params = util.Params( inputs, sanitize=False )
- # process_state will be 'populate' or 'update'. When no tool
- # state is specified in input - it will be 'populate', and
- # tool will fully expand repeat and conditionals when building
- # up state. If tool state is found in input
- # parameters,process_state will be 'update' and complex
- # submissions (with repeats and conditionals) must be built up
- # over several iterative calls to the API - mimicing behavior
- # of web controller (though frankly API never returns
- # tool_state so this "legacy" behavior is probably impossible
- # through API currently).
incoming = params.__dict__
- process_state = "update" if "tool_state" in incoming else "populate"
- template, vars = tool.handle_input( trans, incoming, history=target_history, process_state=process_state, source="json" )
- if 'errors' in vars:
- trans.response.status = 400
- return { "message": { "type": "error", "data" : vars[ 'errors' ] } }
+ vars = tool.handle_input( trans, incoming, history=target_history, source='json' )
# TODO: check for errors and ensure that output dataset(s) are available.
output_datasets = vars.get( 'out_data', [] )
- rval = {
- "outputs": [],
- "output_collections": [],
- "jobs": [],
- "implicit_collections": [],
- }
+ rval = { 'outputs': [], 'output_collections': [], 'jobs': [], 'implicit_collections': [] }
job_errors = vars.get( 'job_errors', [] )
if job_errors:
# If we are here - some jobs were successfully executed but some failed.
- rval[ "errors" ] = job_errors
+ rval[ 'errors' ] = job_errors
- outputs = rval[ "outputs" ]
+ outputs = rval[ 'outputs' ]
# TODO:?? poss. only return ids?
for output_name, output in output_datasets:
output_dict = output.to_dict()
@@ -324,7 +301,7 @@ class ToolsController( BaseAPIController, UsesVisualizationMixin ):
id = urllib.unquote_plus( id )
tool = self.app.toolbox.get_tool( id, tool_version )
if not tool or not tool.allow_user_access( user ):
- raise ObjectNotFound("Could not find tool with id '%s'" % id)
+ raise exceptions.ObjectNotFound("Could not find tool with id '%s'" % id)
return tool
def _rerun_tool( self, trans, payload, **kwargs ):
diff --git a/lib/galaxy/webapps/galaxy/buildapp.py b/lib/galaxy/webapps/galaxy/buildapp.py
index fd9c42c951b..b922a30fa29 100644
--- a/lib/galaxy/webapps/galaxy/buildapp.py
+++ b/lib/galaxy/webapps/galaxy/buildapp.py
@@ -485,6 +485,18 @@ def populate_api_routes( webapp, app ):
action='create',
conditions=dict( method=[ "POST" ] ) )
+ webapp.mapper.connect( 'delete_folder',
+ '/api/folders/{encoded_folder_id}',
+ controller='folders',
+ action='delete',
+ conditions=dict( method=[ "DELETE" ] ) )
+
+ webapp.mapper.connect( 'update_folder',
+ '/api/folders/{encoded_folder_id}',
+ controller='folders',
+ action='update',
+ conditions=dict( method=[ "PATCH", "PUT" ] ) )
+
webapp.mapper.resource( 'folder',
'folders',
path_prefix='/api' )
@@ -686,6 +698,13 @@ def wrap_in_middleware( app, global_conf, **local_conf ):
from galaxy.web.framework.middleware.translogger import TransLogger
app = TransLogger( app )
log.debug( "Enabling 'trans logger' middleware" )
+ # Statsd request timing and profiling
+ statsd_host = conf.get('statsd_host', None)
+ if statsd_host:
+ from galaxy.web.framework.middleware.statsd import StatsdMiddleware
+ app = StatsdMiddleware( app, statsd_host, conf.get('statsd_port'))
+ log.debug( "Enabling 'statsd' middleware" )
+
# X-Forwarded-Host handling
from galaxy.web.framework.middleware.xforwardedhost import XForwardedHostMiddleware
app = XForwardedHostMiddleware( app )
diff --git a/lib/galaxy/webapps/galaxy/controllers/async.py b/lib/galaxy/webapps/galaxy/controllers/async.py
index 163dd0504f2..8aa3d46d78e 100644
--- a/lib/galaxy/webapps/galaxy/controllers/async.py
+++ b/lib/galaxy/webapps/galaxy/controllers/async.py
@@ -5,7 +5,7 @@ Upload class
import logging
import urllib
-from galaxy import jobs, util, web
+from galaxy import jobs, web
from galaxy.util import Params
from galaxy.util.hash_util import hmac_new
from galaxy.web.base.controller import BaseUIController
@@ -32,7 +32,6 @@ class ASync( BaseUIController ):
if not kwd:
return trans.response.send_redirect( "/index" )
- history = trans.get_history( create=True )
params = Params(kwd, sanitize=False)
STATUS = params.STATUS
URL = params.URL
@@ -143,4 +142,4 @@ class ASync( BaseUIController ):
trans.sa_session.flush()
- return trans.fill_template( 'tool_executed.mako', history=history, toolbox=toolbox, tool=tool, util=util, out_data={}, num_jobs=1, job_errors=[] )
+ return trans.fill_template( 'root/tool_runner.mako', out_data={}, num_jobs=1, job_errors=[] )
diff --git a/lib/galaxy/webapps/galaxy/controllers/library_common.py b/lib/galaxy/webapps/galaxy/controllers/library_common.py
index 83e588b4246..e2916a98a5c 100644
--- a/lib/galaxy/webapps/galaxy/controllers/library_common.py
+++ b/lib/galaxy/webapps/galaxy/controllers/library_common.py
@@ -1039,7 +1039,7 @@ class LibraryCommon( BaseUIController, UsesFormDefinitionsMixin, UsesExtendedMet
tool_id = 'upload1'
tool = trans.app.toolbox.get_tool( tool_id )
state = tool.new_state( trans )
- tool.update_state( trans, tool.inputs_by_page[0], state.inputs, kwd )
+ tool.populate_state( trans, tool.inputs, state.inputs, kwd )
tool_params = state.inputs
dataset_upload_inputs = []
for input_name, input in tool.inputs.iteritems():
diff --git a/lib/galaxy/webapps/galaxy/controllers/tool_runner.py b/lib/galaxy/webapps/galaxy/controllers/tool_runner.py
index 70b36457005..806c940923c 100644
--- a/lib/galaxy/webapps/galaxy/controllers/tool_runner.py
+++ b/lib/galaxy/webapps/galaxy/controllers/tool_runner.py
@@ -1,5 +1,5 @@
"""
-Upload class
+Controller handles external tool related requests
"""
import logging
@@ -7,29 +7,13 @@ from markupsafe import escape
import galaxy.util
from galaxy import web
-from galaxy.tools import DefaultToolState
from galaxy.tools import DataSourceTool
-from galaxy.tools.actions import upload_common
-from galaxy.tools.parameters import params_to_incoming
-from galaxy.tools.parameters import visit_input_values
-from galaxy.tools.parameters.basic import DataToolParameter
-from galaxy.tools.parameters.basic import DataCollectionToolParameter
-from galaxy.tools.parameters.basic import UnvalidatedValue
-from galaxy.util.bunch import Bunch
-from galaxy.util.hash_util import is_hashable
from galaxy.web import error, url_for
from galaxy.web.base.controller import BaseUIController
log = logging.getLogger( __name__ )
-class AddFrameData:
- def __init__( self ):
- self.wiki_url = None
- self.debug = None
- self.from_noframe = None
-
-
class ToolRunner( BaseUIController ):
# Hack to get biomart to work, ideally, we could pass tool_id to biomart and receive it back
@@ -49,98 +33,57 @@ class ToolRunner( BaseUIController ):
"""Catches the tool id and redirects as needed"""
return self.index(trans, tool_id=tool_id, **kwd)
- def __get_tool_components( self, tool_id, tool_version=None, get_loaded_tools_by_lineage=False, set_selected=False ):
- return self.get_toolbox().get_tool_components( tool_id, tool_version, get_loaded_tools_by_lineage, set_selected )
+ def __get_tool( self, tool_id, tool_version=None, get_loaded_tools_by_lineage=False, set_selected=False ):
+ tool_version_select_field, tools, tool = self.get_toolbox().get_tool_components( tool_id, tool_version, get_loaded_tools_by_lineage, set_selected )
+ return tool
@web.expose
def index( self, trans, tool_id=None, from_noframe=None, **kwd ):
- # No tool id passed, redirect to main page
+ # tool id not available, redirect to main page
if tool_id is None:
- return trans.response.send_redirect( url_for( controller="root", action="welcome" ) )
- # When the tool form is initially loaded, the received kwd will not include a 'refresh'
- # entry (which only is included when another option is selected in the tool_version_select_field),
- # so the default selected option should be the most recent version of the tool. The following
- # check will mae sure this occurs.
- refreshed_on_change = kwd.get( 'refresh', False )
- tool_version_select_field, tools, tool = self.__get_tool_components( tool_id,
- tool_version=None,
- get_loaded_tools_by_lineage=False,
- set_selected=refreshed_on_change )
- # No tool matching the tool id, display an error (shouldn't happen)
+ return trans.response.send_redirect( url_for( controller='root', action='welcome' ) )
+ tool = self.__get_tool( tool_id )
+ # tool id is not matching, display an error
if not tool or not tool.allow_user_access( trans.user ):
- log.error( "index called with tool id '%s' but no such tool exists", tool_id )
- trans.log_event( "Tool id '%s' does not exist" % tool_id )
+ log.error( 'index called with tool id \'%s\' but no such tool exists', tool_id )
+ trans.log_event( 'Tool id \'%s\' does not exist' % tool_id )
trans.response.status = 404
- return trans.show_error_message("Tool '%s' does not exist." % ( escape(tool_id) ))
+ return trans.show_error_message('Tool \'%s\' does not exist.' % ( escape(tool_id) ))
if tool.require_login and not trans.user:
- message = "You must be logged in to use this tool."
- status = "info"
redirect = url_for( controller='tool_runner', action='index', tool_id=tool_id, **kwd )
return trans.response.send_redirect( url_for( controller='user',
action='login',
cntrller='user',
- message=message,
- status=status,
+ status='info',
+ message='You must be logged in to use this tool.',
redirect=redirect ) )
+ if tool.tool_type == 'default':
+ return trans.response.send_redirect( url_for( controller='root', tool_id=tool_id ) )
- def _validated_params_for( kwd ):
- params = galaxy.util.Params( kwd, sanitize=False ) # Sanitize parameters when substituting into command line via input wrappers
- # do param translation here, used by datasource tools
- if tool.input_translator:
- tool.input_translator.translate( params )
- return params
-
- params = _validated_params_for( kwd )
+ # execute tool without displaying form (used for datasource tools)
+ params = galaxy.util.Params( kwd, sanitize=False )
+ # do param translation here, used by datasource tools
+ if tool.input_translator:
+ tool.input_translator.translate( params )
# We may be visiting Galaxy for the first time ( e.g., sending data from UCSC ),
# so make sure to create a new history if we've never had one before.
history = tool.get_default_history_by_trans( trans, create=True )
try:
- template, vars = tool.handle_input( trans, params.__dict__ )
- except KeyError:
- # This error indicates (or at least can indicate) there was a
- # problem with the stored tool_state - it is incompatible with
- # this variant of the tool - possibly because the tool changed
- # or because the tool version changed.
- del kwd[ "tool_state" ]
- params = _validated_params_for( kwd )
- template, vars = tool.handle_input( trans, params.__dict__ )
+ vars = tool.handle_input( trans, params.__dict__, history=history )
+ except Exception, e:
+ error( str( e ) )
if len( params ) > 0:
- trans.log_event( "Tool params: %s" % ( str( params ) ), tool_id=tool_id )
- add_frame = AddFrameData()
- add_frame.debug = trans.debug
- if from_noframe is not None:
- add_frame.wiki_url = trans.app.config.wiki_url
- add_frame.from_noframe = True
- return trans.fill_template( template,
- history=history,
- toolbox=self.get_toolbox(),
- tool_version_select_field=tool_version_select_field,
- tool=tool,
- util=galaxy.util,
- add_frame=add_frame,
- form_input_auto_focus=True,
- **vars )
+ trans.log_event( 'Tool params: %s' % ( str( params ) ), tool_id=tool_id )
+ return trans.fill_template( 'root/tool_runner.mako', **vars )
@web.expose
- def rerun( self, trans, id=None, from_noframe=None, job_id=None, **kwd ):
+ def rerun( self, trans, id=None, job_id=None, **kwd ):
"""
Given a HistoryDatasetAssociation id, find the job and that created
the dataset, extract the parameters, and display the appropriate tool
form with parameters already filled in.
"""
- if job_id:
- try:
- job_id = trans.security.decode_id( job_id )
- job = trans.sa_session.query( trans.app.model.Job ).get( job_id )
- except:
- error( "Invalid value for 'job_id' parameter" )
- if not trans.user_is_admin():
- for data_assoc in job.output_datasets:
- # only allow rerunning if user is allowed access to the dataset.
- if not trans.app.security_agent.can_access_dataset( trans.get_current_user_roles(), data_assoc.dataset.dataset ):
- error( "You are not allowed to rerun this job" )
- param_error_text = "Failed to get parameters for job id %d " % job_id
- else:
+ if job_id is None:
if not id:
error( "'id' parameter is required" )
try:
@@ -158,135 +101,11 @@ class ToolRunner( BaseUIController ):
error( "You are not allowed to access this dataset" )
# Get the associated job, if any.
job = data.creating_job
- if not job:
- raise Exception("Failed to get job information for dataset hid %d" % data.hid)
- param_error_text = "Failed to get parameters for dataset id %d " % data.id
- # Get the tool object
- tool_id = job.tool_id
- tool_version = job.tool_version
- try:
- tool_version_select_field, tools, tool = self.__get_tool_components( tool_id,
- tool_version=tool_version,
- get_loaded_tools_by_lineage=False,
- set_selected=True )
- if ( tool.id == job.tool_id or tool.old_id == job.tool_id ) and tool.version == job.tool_version:
- tool_id_version_message = ''
- elif tool.id == job.tool_id:
- if job.tool_version is None:
- # For some reason jobs don't always keep track of the tool version.
- tool_id_version_message = ''
- else:
- tool_id_version_message = 'This job was initially run with tool version "%s", which is not currently available. ' % job.tool_version
- if len( tools ) > 1:
- tool_id_version_message += 'You can rerun the job with the selected tool or choose another derivation of the tool.'
- else:
- tool_id_version_message += 'You can rerun the job with this tool version, which is a derivation of the original tool.'
+ if job:
+ job_id = trans.security.encode_id( job.id )
else:
- if len( tools ) > 1:
- tool_id_version_message = 'This job was initially run with tool version "%s", which is not currently available. ' % job.tool_version
- tool_id_version_message += 'You can rerun the job with the selected tool or choose another derivation of the tool.'
- else:
- tool_id_version_message = 'This job was initially run with tool id "%s", version "%s", which is not ' % ( job.tool_id, job.tool_version )
- tool_id_version_message += 'currently available. You can rerun the job with this tool, which is a derivation of the original tool.'
- assert tool is not None, 'Requested tool has not been loaded.'
- except:
- # This is expected so not an exception.
- tool_id_version_message = ''
- error( "This dataset was created by an obsolete tool (%s). Can't re-run." % tool_id )
- if not tool.allow_user_access( trans.user ):
- error( "The requested tool is unknown." )
- # Can't rerun upload, external data sources, et cetera. Workflow compatible will proxy this for now
- if not tool.is_workflow_compatible:
- error( "The '%s' tool does not currently support rerunning." % tool.name )
- # Get the job's parameters
- try:
- params_objects = job.get_param_values( trans.app, ignore_errors=True )
- except:
- raise Exception( param_error_text )
- upgrade_messages = tool.check_and_update_param_values( params_objects, trans, update_values=False )
- # Need to remap dataset parameters. Job parameters point to original
- # dataset used; parameter should be the analygous dataset in the
- # current history.
- history = trans.get_history()
- hda_source_dict = {} # Mapping from HDA in history to source HDAs.
- for hda in history.datasets:
- source_hda = hda.copied_from_history_dataset_association
- while source_hda: # should this check library datasets as well?
- # FIXME: could be multiple copies of a hda in a single history, this does a better job of matching on cloned histories,
- # but is still less than perfect when eg individual datasets are copied between histories
- if source_hda not in hda_source_dict or source_hda.hid == hda.hid:
- hda_source_dict[ source_hda ] = hda
- source_hda = source_hda.copied_from_history_dataset_association
- # Ditto for dataset collections.
- hdca_source_dict = {}
- for hdca in history.dataset_collections:
- source_hdca = hdca.copied_from_history_dataset_collection_association
- while source_hdca:
- if source_hdca not in hdca_source_dict or source_hdca.hid == hdca.hid:
- hdca_source_dict[ source_hdca ] = hdca
- source_hdca = source_hdca.copied_from_history_dataset_collection_association
-
- # Unpack unvalidated values to strings, they'll be validated when the
- # form is submitted (this happens when re-running a job that was
- # initially run by a workflow)
- # This needs to be done recursively through grouping parameters
- def rerun_callback( input, value, prefixed_name, prefixed_label ):
- if isinstance( value, UnvalidatedValue ):
- try:
- return input.to_html_value( value.value, trans.app )
- except Exception, e:
- # Need to determine when (if ever) the to_html_value call could fail.
- log.debug( "Failed to use input.to_html_value to determine value of unvalidated parameter, defaulting to string: %s" % ( e ) )
- return str( value )
- if isinstance( input, DataToolParameter ):
- if isinstance(value, list):
- values = []
- for val in value:
- if is_hashable( val ):
- if val in history.datasets:
- values.append( val )
- elif val in hda_source_dict:
- values.append( hda_source_dict[ val ])
- return values
- if is_hashable( value ) and value not in history.datasets and value in hda_source_dict:
- return hda_source_dict[ value ]
- elif isinstance( input, DataCollectionToolParameter ):
- if is_hashable( value ) and value not in history.dataset_collections and value in hdca_source_dict:
- return hdca_source_dict[ value ]
-
- visit_input_values( tool.inputs, params_objects, rerun_callback )
- # Create a fake tool_state for the tool, with the parameters values
- state = tool.new_state( trans )
- state.inputs = params_objects
- # If the job failed and has dependencies, allow dependency remap
- if job.state == job.states.ERROR:
- try:
- if [ hda.dependent_jobs for hda in [ jtod.dataset for jtod in job.output_datasets ] if hda.dependent_jobs ]:
- state.rerun_remap_job_id = trans.app.security.encode_id(job.id)
- except:
- # Job has no outputs?
- pass
- # create an incoming object from the original job's dataset-modified param objects
- incoming = {}
- params_to_incoming( incoming, tool.inputs, params_objects, trans.app )
- incoming[ "tool_state" ] = galaxy.util.object_to_string( state.encode( tool, trans.app ) )
- template, vars = tool.handle_input( trans, incoming, old_errors=upgrade_messages ) # update new state with old parameters
- # Is the "add frame" stuff neccesary here?
- add_frame = AddFrameData()
- add_frame.debug = trans.debug
- if from_noframe is not None:
- add_frame.wiki_url = trans.app.config.wiki_url
- add_frame.from_noframe = True
- return trans.fill_template( template,
- history=history,
- toolbox=self.get_toolbox(),
- tool_version_select_field=tool_version_select_field,
- tool=tool,
- job=job,
- util=galaxy.util,
- add_frame=add_frame,
- tool_id_version_message=tool_id_version_message,
- **vars )
+ raise Exception("Failed to get job information for dataset hid %d" % data.hid)
+ return trans.response.send_redirect( url_for( controller="root", job_id=job_id ) )
@web.expose
def data_source_redirect( self, trans, tool_id=None ):
@@ -300,10 +119,7 @@ class ToolRunner( BaseUIController ):
"""
if tool_id is None:
return trans.response.send_redirect( url_for( controller="root", action="welcome" ) )
- tool_version_select_field, tools, tool = self.__get_tool_components( tool_id,
- tool_version=None,
- get_loaded_tools_by_lineage=False,
- set_selected=False )
+ tool = self.__get_tool( tool_id )
# No tool matching the tool id, display an error (shouldn't happen)
if not tool:
log.error( "data_source_redirect called with tool id '%s' but no such tool exists", tool_id )
@@ -323,95 +139,3 @@ class ToolRunner( BaseUIController ):
return trans.show_error_message( "Required URL for redirection missing" )
trans.log_event( "Redirecting to: %s" % redirect_url )
return trans.fill_template( 'root/redirect.mako', redirect_url=redirect_url )
-
- @web.json
- def upload_async_create( self, trans, tool_id=None, **kwd ):
- """
- Precreate datasets for asynchronous uploading.
- """
- cntrller = kwd.get( 'cntrller', '' )
- roles = kwd.get( 'roles', False )
- if roles:
- # The user associated the DATASET_ACCESS permission on the uploaded datasets with 1 or more roles.
- # We need to ensure that the roles are legitimately derived from the roles associated with the LIBRARY_ACCESS
- # permission if the library is not public ( this should always be the case since any ill-legitimate roles
- # were filtered out of the roles displayed on the upload form. In addition, we need to ensure that the user
- # did not associated roles that would make the dataset in-accessible by everyone.
- library_id = trans.app.security.decode_id( kwd.get( 'library_id', '' ) )
- vars = dict( DATASET_ACCESS_in=roles )
- permissions, in_roles, error, msg = trans.app.security_agent.derive_roles_from_access( trans, library_id, cntrller, library=True, **vars )
- if error:
- return [ 'error', msg ]
-
- def create_dataset( name ):
- ud = Bunch( name=name, file_type=None, dbkey=None )
- if nonfile_params.get( 'folder_id', False ):
- replace_id = nonfile_params.get( 'replace_id', None )
- if replace_id not in [ None, 'None' ]:
- replace_dataset = trans.sa_session.query( trans.app.model.LibraryDataset ).get( trans.security.decode_id( replace_id ) )
- else:
- replace_dataset = None
- # FIXME: instead of passing params here ( chiech have been process by util.Params(), the original kwd
- # should be passed so that complex objects that may have been included in the initial request remain.
- library_bunch = upload_common.handle_library_params( trans, nonfile_params, nonfile_params.folder_id, replace_dataset )
- else:
- library_bunch = None
- return upload_common.new_upload( trans, cntrller, ud, library_bunch=library_bunch, state=trans.app.model.HistoryDatasetAssociation.states.UPLOAD )
- tool = self.get_toolbox().get_tool( tool_id )
- if not tool:
- return False # bad tool_id
- nonfile_params = galaxy.util.Params( kwd, sanitize=False )
- if kwd.get( 'tool_state', None ) not in ( None, 'None' ):
- encoded_state = galaxy.util.string_to_object( kwd["tool_state"] )
- tool_state = DefaultToolState()
- tool_state.decode( encoded_state, tool, trans.app )
- else:
- tool_state = tool.new_state( trans )
- tool.update_state( trans, tool.inputs, tool_state.inputs, kwd, update_only=True )
- datasets = []
- dataset_upload_inputs = []
- for input_name, input in tool.inputs.iteritems():
- if input.type == "upload_dataset":
- dataset_upload_inputs.append( input )
- assert dataset_upload_inputs, Exception( "No dataset upload groups were found." )
- for dataset_upload_input in dataset_upload_inputs:
- d_type = dataset_upload_input.get_datatype( trans, kwd )
- if d_type.composite_type is not None:
- datasets.append( create_dataset( dataset_upload_input.get_composite_dataset_name( kwd ) ) )
- else:
- params = Bunch( ** tool_state.inputs[dataset_upload_input.name][0] )
- if params.file_data not in [ None, "" ]:
- name = params.file_data
- if name.count('/'):
- name = name.rsplit('/', 1)[1]
- if name.count('\\'):
- name = name.rsplit('\\', 1)[1]
- datasets.append( create_dataset( name ) )
- if params.url_paste not in [ None, "" ]:
- url_paste = params.url_paste.replace( '\r', '' ).split( '\n' )
- url = False
- for line in url_paste:
- line = line.rstrip( '\r\n' ).strip()
- if not line:
- continue
- elif line.lower().startswith( 'http://' ) or line.lower().startswith( 'ftp://' ) or line.lower().startswith( 'https://' ):
- url = True
- datasets.append( create_dataset( line ) )
- else:
- if url:
- continue # non-url when we've already processed some urls
- else:
- # pasted data
- datasets.append( create_dataset( 'Pasted Entry' ) )
- break
- return [ d.id for d in datasets ]
-
- @web.expose
- def upload_async_message( self, trans, **kwd ):
- # might be more appropriate in a different controller
- msg = """
Your upload has been queued. History entries that are still uploading will be blue, and turn green upon completion.
-
Please do not use your browser\'s "stop" or "reload" buttons until the upload is complete, or it may be interrupted.
-
You may safely continue to use Galaxy while the upload is in progress. Using "stop" and "reload" on pages other than Galaxy is also safe.
- """
- # return trans.show_message( msg, refresh_frames=[ 'history' ] )
- return trans.show_message( msg )
diff --git a/lib/galaxy/webapps/galaxy/controllers/ucsc_proxy.py b/lib/galaxy/webapps/galaxy/controllers/ucsc_proxy.py
deleted file mode 100644
index a2ea97d6fb6..00000000000
--- a/lib/galaxy/webapps/galaxy/controllers/ucsc_proxy.py
+++ /dev/null
@@ -1,165 +0,0 @@
-"""
-Contains the UCSC proxy
-"""
-
-from galaxy.web.base.controller import BaseUIController
-
-import json
-from galaxy import web, util
-
-import re
-import urllib
-import logging
-
-log = logging.getLogger( __name__ )
-
-
-class UCSCProxy( BaseUIController ):
-
- def create_display(self, store):
- """Creates a more meaningulf display name"""
- track = store.get('hgta_track', 'no track')
- table = store.get('hgta_table', 'no table')
- region = store.get('hgta_regionType', '')
- if region not in [ 'genome', 'encode']:
- region = store.get('position', '')
- if track == table:
- display = 'UCSC: %s (%s)' % (track, region)
- else:
- display = 'UCSC: %s, %s (%s)' % (track, table, region)
- return display
-
- @web.expose
- def index(self, trans, init=False, **kwd):
- base_url = None
- params = dict(kwd)
- try:
- store = params.get("__GALAXY__", None)
- if store:
- store = json.loads(util.string_to_object(store))
- else:
- store = {}
- UCSC_URL = 'UCSC_URL'
- base_url = store.get(UCSC_URL, "http://genome.ucsc.edu/cgi-bin/hgTables?")
- params = dict(kwd)
- params['init'] = init
-
- if not init:
- for key, value in kwd.items():
- store[key] = value
- try:
- del store["__GALAXY__"]
- except:
- pass
- else:
- store = {}
-
- if init == "1":
- base_url = "http://genome.ucsc.edu/cgi-bin/hgTables?"
- params['db'] = 'hg17'
- if init == "2":
- base_url = "http://genome-test.cse.ucsc.edu/cgi-bin/hgTables?"
- params['db'] = 'hg17'
- if init == "3":
- base_url = "http://archaea.ucsc.edu/cgi-bin/hgTables?"
-
- store[UCSC_URL] = base_url
-
- try:
- del params["__GALAXY__"]
- except:
- pass
- url = base_url + urllib.urlencode(params)
-
- page = urllib.urlopen(url)
- content = page.info().get('Content-type', '')
- except Exception, exc:
- trans.log_event( "Proxy Error -> %s" % str(exc) )
- msg = 'There has been a problem connecting to %s
%s' % (base_url, exc)
- return msg
-
- if content.startswith('text/plain'):
- params['display'] = self.create_display(store)
- params['dbkey'] = store.get('db', '*')
- params['tool_id'] = 'ucsc_proxy'
- params['proxy_url'] = base_url
- params['runtool_btn'] = 'T'
- url = "/tool_runner/index?" + urllib.urlencode(params)
- trans.response.send_redirect(url)
- else:
- try:
- text = page.read()
-
- # Serialize store into a form element
- store_text = ""
-
- # Remove text regions that should not be exposed
- for key, value in altered_regions.items():
- text = text.replace(key, value)
- # Capture only the forms
- newtext = beginning
- for form in re.finditer("(?s)()", text):
- newtext = newtext + form.group(1) + store_text + form.group(2)
- if 'hgta_doLookupPosition' in params:
- lookup = re.search("(?s).*?(
.*)", text)
- if lookup:
- newtext = newtext + lookup.group(1)
- # if these keys are in the params, then pass the content through
- passthruContent = ['hgta_doSummaryStats', 'hgta_doSchema', 'hgta_doSchemaDb']
- for k in passthruContent:
- if k in params:
- content = re.search("(?s)CONTENT TABLES.*?-->(.*/TABLE>)", text)
- if content:
- newtext = newtext + "
" + content.group(1)
-
- newtext = newtext + ending
- return newtext
- except KeyError, exc:
- log.error(str(exc))
- trans.log_event( "Proxy Error -> %s" % str(exc) )
- msg = 'There has been a problem connecting to %s
%s' % (base_url, exc)
- return msg
-
-# HTML for generating the proxy page.
-beginning = '''
-
-
-
-Galaxy
-
-
-
-
-
-
-
-
-'''
-
-# This is a mess of mappings of text to make the proxy friendlier to
-# galaxy users.
-altered_regions = {
- '"../cgi-bin/hgTables' : '"/ucsc_proxy/index',
- '
\noutput file: (leave blank to keep output in browser)