mirror of
https://github.com/galaxyproject/galaxy.git
synced 2026-10-25 00:12:08 +08:00
226 lines
11 KiB
ReStructuredText
226 lines
11 KiB
ReStructuredText
|
|
===========================================================
|
|
January 2018 Galaxy Release (v 18.01)
|
|
===========================================================
|
|
|
|
.. include:: _header.rst
|
|
|
|
Highlights
|
|
===========================================================
|
|
|
|
**Performance and User Experience Improvements**
|
|
We made Galaxy more lively and responsive. Homepage, published workflows, published/saved histories, and data libraries should all load much faster now. Importing data from FTP will also take less of your time.
|
|
We achieved this by optimizing database queries, implementing cache, rethinking presentation, and adding progress bars and loading indicators where needed. See the `list of related performance pull requests`_.
|
|
|
|
**Web Server and Configuration**
|
|
The default web server used by Galaxy has changed from Paste to `uWSGI <https://uwsgi-docs.readthedocs.io/>`__ and the default configuration file for Galaxy is now ``config/galaxy.yml`` instead of ``config/galaxy.ini``.
|
|
To minimize the impact of this change on existing Galaxy instances, if a Galaxy has a ``galaxy.ini`` file configured, it will continue to use Paste by default unless additional steps are taken by the administrator.
|
|
uWSGI is more production ready and allows Galaxy to scale better in its default configuration. Read more about the server, configuration, and documentation changes in the `uWSGI details`_ section of this document.
|
|
|
|
**Dataset Collection Usability**
|
|
This release has significantly improved the usability of Galaxy dataset collections. Dozens of improvements
|
|
to collections have been made, some of the key highlights include:
|
|
|
|
- Data library folders can now be sent to histories as a dataset collection. `Pull Request 4998`_, `Pull Request 5080`_
|
|
- Failed dataset collection elements can now be fixed using job re-running (thanks to `@mvdbeek <https://github.com/mvdbeek>`__). `Pull Request 5321`_, `Pull Request 5247`_, `Pull Request 5270`_
|
|
- Collections now appear with state and progress bars in the history panel and contained datasets
|
|
are hidden by default (with help from `@mvdbeek <https://github.com/mvdbeek>`__).
|
|
`Pull Request 5013`_, `Pull Request 5078`_
|
|
- We added intuitive workflow post job actions for dataset collections. `Pull Request 5416`_, `Pull Request 5418`_, `Pull Request 5414`_
|
|
- The web interface now supports collections with arbitrary nesting and size (with help
|
|
from `@mvdbeek <https://github.com/mvdbeek>`__). `Pull Request 4942`_, `Pull Request 4934`_, `Pull Request 5091`_
|
|
- More robust nametag discovery and propagation when using collections. `Pull Request 5403`_
|
|
|
|
**Client Architecture**
|
|
The architecture for the client code that powers the Galaxy user interface has been significantly
|
|
overhauled. The code base has been converted to `ES6 <http://es6-features.org/>`__,
|
|
`Yarn <https://github.com/yarnpkg/yarn>`__ now powers the build and dependency management of the code,
|
|
`Prettier <https://prettier.io/>`__ is now used to ensure consistent code formatting,
|
|
and the `VueJS <https://vuejs.org/>`__ framework has been integrated. Taken together, these changes are
|
|
enabling Galaxy developers to write usable, responsive client code more quickly and concisely than
|
|
previously possible. A big thanks goes out to community contributions from
|
|
`@anuprulez <https://github.com/anuprulez>`__ and `@anatskiy <https://github.com/anatskiy>`__ that are
|
|
already converting existing Galaxy components to reactive VueJS ones. See the `list of related client pull requests`_.
|
|
|
|
**New BAM datatypes**
|
|
Previously Galaxy only supported coordinate sorted BAM files by default (the ``bam`` datatype). In addition, this
|
|
release of Galaxy now supports three new types of BAM:
|
|
|
|
- ``qname_sorted.bam``, that ensures that the file is queryname sorted (e.g. ``SO:queryname``);
|
|
- ``qname_input_sorted.bam``, that can be used to describe the output of aligners which generally keep mate pairs adjacent
|
|
- ``unsorted.bam``, that makes no assumptions about the sort order of the file.
|
|
|
|
A huge thanks goes out to `@bgruening <https://github.com/bgruening>`__ and `@mvdbeek <https://github.com/mvdbeek>`__ who
|
|
implemented these datatypes.
|
|
|
|
`Pull Request 5180`_, `Pull Request 5589`_, `Pull Request 5532`_, `Pull Request 5644`_, `Pull Request 5674`_
|
|
|
|
**Experimental Job Caching**
|
|
Galaxy can now be configured to allow users the option of skipping duplicated jobs if one with
|
|
identical parameters has been previously executed and simply reuse the previously generated outputs.
|
|
This contribution is thanks to `@mvdbeek <https://github.com/mvdbeek>`__.
|
|
|
|
`Pull Request 4690`_
|
|
|
|
|
|
Get Galaxy
|
|
==========
|
|
|
|
The code lives at `Github <https://github.com/galaxyproject/galaxy>`__ and you should have `Git <https://git-scm.com/>`__ to obtain it.
|
|
|
|
To get a new Galaxy repository run:
|
|
.. code-block:: shell
|
|
|
|
$ git clone -b release_18.01 https://github.com/galaxyproject/galaxy.git
|
|
|
|
To update an existing Galaxy repository run:
|
|
.. code-block:: shell
|
|
|
|
$ git fetch origin && git checkout release_18.01 && git pull --ff-only origin release_18.01
|
|
|
|
See the `community hub <https://galaxyproject.org/develop/source-code/>`__ for additional details regarding the source code locations.
|
|
|
|
|
|
Security
|
|
========
|
|
|
|
Unsecure GenomeSpace token exposure
|
|
-----------------------------------
|
|
Tracked as ``GX-2018-0002``.
|
|
|
|
We have found and fixed a medium-level security issue concering the GenomeSpace importer/exporter tools that were updated in the Galaxy release 17.09. These tools did not handle the GenomeSpace access token securely and stored it as a job parameter which made it accessible to anybody with access to the datasets created by these tools.
|
|
This means that any user that used a GenomeSpace token to access these tools and subsequently shared the output dataset (or history that contains it) with another user shared their GenomeSpace token also.
|
|
|
|
These tools are both included in the ``tool_conf.xml.sample`` and are therefore *enabled on every new Galaxy by default*.
|
|
|
|
Administrators please see the `GenomeSpace security sanitization`_ section of this document for the details on how to sanitize the tokens stored in the Galaxy database created prior to this fix.
|
|
|
|
The vulnerability has been resolved by removing the token functionality until a proper implementation is in place. The GenomeSpace tools continue to work using the OpenID authentication as before.
|
|
|
|
The fix for this issue has been applied back to Galaxy release 17.09 and can be found in this `pull request <https://github.com/galaxyproject/galaxy/pull/5631>`__.
|
|
|
|
Breaking Changes
|
|
================
|
|
|
|
We have reworked the **Galaxy Webhook** interface so if you have custom webhooks at your instance
|
|
you need to take the following steps in order to make them work:
|
|
|
|
1. Rename the main config file: <name>.yml -> config.yml
|
|
2. In the config file, rename the name attribute: name -> id
|
|
3. Put all files into the plugin's root folder, which should contain only four files: ``config.yml`` (mandatory), ``__init__.py`` (optional), ``script.js`` (optional), ``styles.css`` (optional)
|
|
|
|
Deprecation Notices
|
|
===================
|
|
|
|
* This is the last release we are shipping JavaScript source maps for Galaxy client.
|
|
|
|
Removal Notices
|
|
===============
|
|
The following features that have been deprecated in the past releases are being removed in 18.01:
|
|
|
|
* Sample tracking
|
|
* Sample request and external services management
|
|
* Legacy library interface for Administrators
|
|
|
|
In addition to that, the PlantTribes datatypes have been commented out in the source code. Uncomment them if you want to re-enable it.
|
|
|
|
uWSGI details
|
|
=============
|
|
Galaxy can be forced to start under uWSGI even with an older configuration file
|
|
by setting ``APP_WEBSERVER=uwsgi`` in the environment. As part of the transition to YAML-based
|
|
configuration files, we have implemented a schema to validate Galaxy configuration files. Run
|
|
``make config-validate`` from Galaxy's root directory to validate a schema and ``make config-lint`` to
|
|
check for best practices. While there is no need to convert your configuration file (``galaxy.ini`` hasn't
|
|
been deprecated), you can run ``make config-convert-dry-run`` and ``make config-convert``
|
|
to respectively test and perform the conversion of an ``ini`` configuration file to a YAML one.
|
|
|
|
In the future uWSGI will allow Galaxy to setup GIE proxies without additional configuration and use modern web technologies such as web sockets.
|
|
|
|
These are big changes that affect many parts of Galaxy's administration documentation and makes
|
|
this documentation very dependent on which Galaxy version they are targeting. To address this, we have
|
|
moved a significant amount of administration documentation into Galaxy's code
|
|
base and made it available on a per-release basis. The latest administration documentation for
|
|
the previous release of Galaxy (17.09) which reference ``galaxy.ini`` files and Paste servers can
|
|
be found `here <https://docs.galaxyproject.org/en/release_17.09/admin/index.html>`__, while documentation for this release can be found `here <https://docs.galaxyproject.org/en/release_18.01/admin/index.html>`__.
|
|
|
|
`Pull Request 4475`_, `Pull Request 5135`_, `Pull Request 5390`_, `Pull Request 5373`_,
|
|
`Pull Request 5105`_, `Pull Request 5441`_
|
|
|
|
Release Notes
|
|
=============
|
|
|
|
.. include:: 18.01.rst
|
|
:start-after: announce_start
|
|
|
|
GenomeSpace security sanitization
|
|
=================================
|
|
|
|
Outputs of these tools may require sanitization: ``genomespace_importer`` and ``genomespace_exporter``
|
|
|
|
The following SQL commands will help you **identify** the datasets in your Galaxy's database.
|
|
|
|
Finding bad GenomeSpace importer params:
|
|
|
|
.. code-block:: sql
|
|
|
|
SELECT j.id,
|
|
j.create_time,
|
|
j.user_id,
|
|
jp.value
|
|
FROM job_parameter jp
|
|
JOIN job j ON j.id = jp.job_id
|
|
WHERE jp.job_id IN
|
|
(SELECT id
|
|
FROM job
|
|
WHERE tool_id = 'genomespace_importer')
|
|
AND jp.name = 'URL'
|
|
AND jp.value LIKE '%^%'
|
|
ORDER BY j.id DESC;
|
|
|
|
Finding bad GenomeSpace exporter params:
|
|
|
|
.. code-block:: sql
|
|
|
|
SELECT j.id,
|
|
j.create_time,
|
|
j.user_id,
|
|
jp.value
|
|
FROM job_parameter jp
|
|
JOIN job j ON j.id = jp.job_id
|
|
WHERE jp.job_id IN
|
|
(SELECT id
|
|
FROM job
|
|
WHERE tool_id = 'genomespace_exporter')
|
|
AND jp.name = 'genomespace_browser'
|
|
AND jp.value LIKE '%^%'
|
|
ORDER BY j.id DESC;
|
|
|
|
The following SQL commands will help you **sanitize** the datasets in your Galaxy's database.
|
|
|
|
Sanitizing GenomeSpace importer params:
|
|
|
|
.. code-block:: sql
|
|
|
|
UPDATE job_parameter jp
|
|
SET value = split_part(jp.value, '^', 1) || '"'
|
|
FROM job j
|
|
WHERE jp.job_id = j.id
|
|
AND j.tool_id = 'genomespace_importer'
|
|
AND jp.name = 'URL'
|
|
AND jp.value LIKE '%^%';
|
|
|
|
Sanitizing GenomeSpace exporter params:
|
|
|
|
.. code-block:: sql
|
|
|
|
UPDATE job_parameter jp
|
|
SET value = split_part(jp.value, '^', 1) || '"'
|
|
FROM job j
|
|
WHERE jp.job_id = j.id
|
|
AND j.tool_id = 'genomespace_exporter'
|
|
AND jp.name = 'genomespace_browser'
|
|
AND jp.value LIKE '%^%';
|
|
|
|
.. include:: _thanks.rst
|
|
|