diff --git a/doc/source/admin/config_logging.rst b/doc/source/admin/config_logging.rst index a64f9ff29ec..db3e9fc5073 100644 --- a/doc/source/admin/config_logging.rst +++ b/doc/source/admin/config_logging.rst @@ -238,5 +238,5 @@ advanced logging configuration. For example: While Galaxy's custom log format fields can be used (as seen in the example), the ``filename_template`` handler configuration extension is only available in the YAML format configuration file. -.. _logging levels: https://docs.python.org/2/library/logging.html#logging-levels -.. _fileConfig file format: https://docs.python.org/2/library/logging.config.html#configuration-file-format +.. _logging levels: https://docs.python.org/library/logging.html#logging-levels +.. _fileConfig file format: https://docs.python.org/library/logging.config.html#configuration-file-format diff --git a/doc/source/admin/framework_dependencies.rst b/doc/source/admin/framework_dependencies.rst index 8f9285d172f..fa1c0e9f8f6 100644 --- a/doc/source/admin/framework_dependencies.rst +++ b/doc/source/admin/framework_dependencies.rst @@ -96,18 +96,6 @@ this script yourself to set up Galaxy pip and the dependencies without creating .. code-block:: console (venv)$ PYTHONPATH= sh /srv/galaxy/server/scripts/common_startup.sh --no-create-venv - Requirement already satisfied: pip>=8.1 in /home/nate/.virtualenvs/test/lib/python2.7/site-packages - Collecting numpy==1.9.2 (from -r requirements.txt (line 4)) - Downloading https://wheels.galaxyproject.org/packages/numpy-1.9.2-cp27-cp27mu-manylinux1_x86_64.whl (10.2MB) - 100% |████████████████████████████████| 10.2MB 21.7MB/s - Collecting bx-python==0.7.3 (from -r requirements.txt (line 5)) - Downloading https://wheels.galaxyproject.org/packages/bx_python-0.7.3-cp27-cp27mu-manylinux1_x86_64.whl (2.1MB) - 100% |████████████████████████████████| 2.2MB 97.2MB/s - - ... - - Installing collected packages: numpy, bx-python, ... - Successfully installed numpy-1.9.2 bx-python-0.7.3 ... .. warning:: @@ -195,7 +183,7 @@ Galaxy can create a virtualenv using the adapted virtualenv package. Once a vali A Conda environment named ``_galaxy_`` will be created using python 2 and the appropriate virtualenv package will be installed into this environment. Using this environment a ``.venv`` is initialized. This is a one-time setup, and all other activation and dependency - management happens exactly as if a system python was used for creating ``.venv``. + management happens exactly as if a system Python was used for creating ``.venv``. .. _Conda: https://conda.io/ .. _Conda environments: https://conda.io/docs/user-guide/tasks/manage-environments.html @@ -255,12 +243,6 @@ Adding additional Galaxy dependencies New packages can be added to Galaxy, or the versions of existing packages can be updated, using `pipenv`_ and `Starforge`_, Galaxy's Docker-based build system. -.. note:: - - Dependency pinning management is being migrated to pipenv_. As of this release, pinning for packages used for Galaxy - development are managed by pipenv_, but pinning for regular runtime packages are still managed with manual changes - to ``pinned-requirements.txt``. See `Pull Request #4891`_ for details. - The process is still under development and will be streamlined and automated over time. For the time being, please use the following process to add new packages and have their wheels built: diff --git a/doc/source/admin/nginx.md b/doc/source/admin/nginx.md index ff204970d5e..fa1ec7561ed 100644 --- a/doc/source/admin/nginx.md +++ b/doc/source/admin/nginx.md @@ -349,11 +349,15 @@ galaxy: nginx_upload_path: '/_upload' ``` +```eval_rst +.. _protect-reports: +``` + ### Use Galaxy Authentication to Protect Custom Paths You may find it useful to require authentication for access to certain paths on your server. For example, Galaxy can run a separate reports app which gives useful information about your Galaxy instance. See the [Reports Configuration -documentation](reports.md) and [Peter Briggs' blog post on the +documentation](reports) and [Peter Briggs' blog post on the subject](http://galacticengineer.blogspot.com/2015/06/exposing-galaxy-reports-via-nginx-in.html) for more. After successfully following the blog post, Galaxy reports should be available at e.g. `https://galaxy.example.org/reports`. diff --git a/doc/source/admin/reports.md b/doc/source/admin/reports.md deleted file mode 100644 index 71cb4924609..00000000000 --- a/doc/source/admin/reports.md +++ /dev/null @@ -1,21 +0,0 @@ -# Galaxy Reports - -Galaxy includes a report tool that is separate from the main process but which gives lots of potentially useful information about the usage of a Galaxy instance, for example the numbers of jobs that have been run each month, how much disk space each user is currently consuming and so on. - -## Setup on localhost - -The report tool takes its configuration settings from a file called `reports.yml`, which is located in the ``config/` subdirectory of the Galaxy distribution. - -Configuring the reports for your local setup is a case of: - -* Making a copy of ``reports.yml.sample`` called `reports.yml`. -* Editing the `database_connection` parameter to match the one in your `galaxy.yml` file -* Optionally, editing the `port` parameter (by default the tool uses port 9001) - -You should also set the 'salt' parameter `session_secret` if you intend to expose the reports via the web proxy (see below). - -Then you can start the report server using `sh run_reports.sh` and view the reports by pointing a web browser running on the same server to http://127.0.0.1:9001. If you'd like the report tool to persist between sessions then use `sh run_reports.sh --daemon` to run it as a background process. As with Galaxy itself, use `--stop-daemon` to halt the background process. (The log file is written to `reports_webapp.log` if you need to try and debug a problem.) - -## Expose Outside - -To make your reports available from outside of the localhost using NGINX proxy server you can check out the [blogpost](http://galacticengineer.blogspot.co.uk/2015/06/exposing-galaxy-reports-via-nginx-in.html) by Peter Briggs and the [Protect Galaxy Reports](https://galaxyproject.org/admin/config/nginx-proxy/#protect-galaxy-reports) section at the [Community Hub](https://galaxyproject.org). diff --git a/doc/source/admin/reports.rst b/doc/source/admin/reports.rst index 87d5c763009..91d10028d6a 100644 --- a/doc/source/admin/reports.rst +++ b/doc/source/admin/reports.rst @@ -45,10 +45,20 @@ Configuration - The default port for the reports application is ``9001``, and like Galaxy it only binds to localhost by default. - ``database_connection`` should match the value used in your Galaxy configuration - - ``database_connection`` should point at a Postgres database, experimental support for MySQL is available but sqlite is not supported at all. + - ``database_connection`` should point at a PostgreSQL database, + experimental support for MySQL is available but SQLite is not supported + at all. - Run reports in a uWSGI server with ``sh run_reports.sh`` - Use a web browser and go to the address you configured in ``reports.yml`` (defaults to http://localhost:9001/) +- If you'd like the report tool to persist between sessions then use + ``sh run_reports.sh --daemon`` to run it as a background process. As with + Galaxy itself, use the ``--stop-daemon`` option to halt the background + process. (The process output is written to ``reports_webapp.log`` if you have + to debug a problem.) +- To make your reports available from outside of the localhost using the NGINX + proxy server, you can check out the documentation on how to + :ref:`protect Galaxy reports ` using authentication. ---------------------------- Configuration Options diff --git a/doc/source/admin/reports_options.rst b/doc/source/admin/reports_options.rst index c41634b67eb..e60fb645988 100644 --- a/doc/source/admin/reports_options.rst +++ b/doc/source/admin/reports_options.rst @@ -4,8 +4,7 @@ :Description: Verbosity of console log messages. Acceptable values can be found - here: - https://docs.python.org/2/library/logging.html#logging-levels + here: https://docs.python.org/library/logging.html#logging-levels :Default: ``DEBUG`` :Type: str diff --git a/doc/source/admin/special_topics/index.rst b/doc/source/admin/special_topics/index.rst index ce7894d34cc..3a4d1801570 100644 --- a/doc/source/admin/special_topics/index.rst +++ b/doc/source/admin/special_topics/index.rst @@ -6,6 +6,7 @@ Special Topics :maxdepth: 2 ftp + interactivetools interactive_environments mulled_containers grt diff --git a/doc/source/admin/special_topics/interactivetools.rst b/doc/source/admin/special_topics/interactivetools.rst index 9e9fcfa156d..d08e4d7a968 100644 --- a/doc/source/admin/special_topics/interactivetools.rst +++ b/doc/source/admin/special_topics/interactivetools.rst @@ -40,7 +40,7 @@ Some important benefits of using Galaxy InteractiveTools Server-side configuration of Galaxy InteractiveTools -------------------------------------------------- +---------------------------------------------------- The **galaxy.yml** file will need to be populated as seen in **config/galaxy.yml.interactivetools**. diff --git a/doc/source/conf.py b/doc/source/conf.py index 58de04df014..2c57af0c459 100644 --- a/doc/source/conf.py +++ b/doc/source/conf.py @@ -129,8 +129,8 @@ pygments_style = 'sphinx' # A list of ignored prefixes for module index sorting. #modindex_common_prefix = [] -# Intersphinx mapping to Python 2.7 documentation -intersphinx_mapping = {'python': ('https://docs.python.org/2.7', None)} +# Intersphinx mapping to Python documentation +intersphinx_mapping = {'python': ('https://docs.python.org/3', None)} # -- Options for HTML output --------------------------------------------------- diff --git a/doc/source/dev/build_a_job_runner.rst b/doc/source/dev/build_a_job_runner.rst index d0ccaefff86..f160a0c4c6a 100644 --- a/doc/source/dev/build_a_job_runner.rst +++ b/doc/source/dev/build_a_job_runner.rst @@ -234,7 +234,7 @@ Updated result after an iteration (after invocation of ``check_watched_item`` 6 Note: Iterating through the queue is already taken care by the framework. -To inform galaxy about the status of the job: +To inform Galaxy about the status of the job: - Get the job status from external runner using the ``job_id``. @@ -249,38 +249,38 @@ To inform galaxy about the status of the job: .. code-block:: python def check_watched_item(self, job_state): - !job_status = get_task_from_external_runner(job_state.job_id) - if job_status == "over_with_success": - job_state.running = False - job_state.job_wrapper.change_state(model.Job.states.OK) - !create_log_file() - self.mark_as_finished(job_state) - return None + job_status = get_task_from_external_runner(job_state.job_id) + if job_status == "over_with_success": + job_state.running = False + job_state.job_wrapper.change_state(model.Job.states.OK) + create_log_files() + self.mark_as_finished(job_state) + return None - elif job_status == "running": - job_state.running = True - job_state.job_wrapper.change_state(model.Job.states.RUNNING) - return job_state + elif job_status == "running": + job_state.running = True + job_state.job_wrapper.change_state(model.Job.states.RUNNING) + return job_state - elif job_status == "pending": - return job_state + elif job_status == "pending": + return job_state - elif job_status == "over_with_error": - job_state.running = False - job_state.job_wrapper.change_state(model.Job.states.ERROR) - !create_log_file() - self.mark_as_failed(job_state) - return None + elif job_status == "over_with_error": + job_state.running = False + job_state.job_wrapper.change_state(model.Job.states.ERROR) + create_log_files() + self.mark_as_failed(job_state) + return None Note: -- Methods prefixed with ! are user-defined methods. +- ``get_task_from_external_runner`` and ``create_log_files`` are user-defined methods. - Return value is ``job_state`` for running, pending jobs and None for rest of the states of jobs. ``create_log_files()`` are nothing but copying the files (``error_file``, -``output_file``, ``exit_code_file``) from external runner's directory to -working directory of Galaxy. +``output_file``, ``exit_code_file``) from the external runner's directory to +the working directory of Galaxy. Source of the files are from the output directory of your external runner. Destination of the files will be: diff --git a/doc/source/dev/data_managers.rst b/doc/source/dev/data_managers.rst index acca5695035..b3c35839014 100644 --- a/doc/source/dev/data_managers.rst +++ b/doc/source/dev/data_managers.rst @@ -365,9 +365,7 @@ Example 1 JSON Output from Data Manager Tool to Galaxy } } -This creates a new entry in the Tool Data Table: - -.. code-block:: +This creates a new entry in the Tool Data Table:: # sacCer2 sacCer2 S. cerevisiae June 2008 (SGD/sacCer2) (sacCer2) /Users/dan/galaxy-central/tool-data/sacCer2/seq/sacCer2.fa @@ -881,10 +879,7 @@ Example JSON Output from tool to galaxy, dbkey is sacCer2 } } -New Entry in Data Table, dbkey is sacCer2 -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -.. code-block:: +This creates a new entry in the Tool Data Table:: # sacCer2 sacCer2 S. cerevisiae June 2008 (SGD/sacCer2) (sacCer2) /Users/dan/galaxy-central/tool-data/sacCer2/seq/sacCer2.fa diff --git a/doc/source/dev/data_types.md b/doc/source/dev/data_types.md index 3fa19b246f8..ce1bc146b1b 100644 --- a/doc/source/dev/data_types.md +++ b/doc/source/dev/data_types.md @@ -35,27 +35,17 @@ tag section of the `datatypes_conf.xml` file. Sample where -- `extension` - the data type's Dataset file - extension (e.g., `ab1`, `bed`, `gff`, `qual`, - etc.) `type` - the path to the class for that - data type. +- `extension` - the data type's Dataset file extension (e.g., `ab1`, `bed`, + `gff`, `qual`, etc.) +- `type` - the path to the class for that data type. +- `mimetype` - if present (it's optional), the data type's mime type +- `display_in_upload` - if present (it's optional and defaults to False), the + associated file extension will be displayed in the "File Format" select list + in the "Upload File from your computer" tool in the "Get Data" tool section of + the tool panel. -- `mimetype` - if present (it's optional), the data - type's mime type `display_in_upload` - if present - (it's optional and defaults to False), the - associated file extension will be displayed in - the "File Format" select list in the "Upload File - from your computer" tool in the "Get Data" tool - section of the tool panel. -```xml - - - - - ... -``` **Note:** If you do not wish to add extended -functionality to for a new datatype, but simply want +functionality to a new datatype, but simply want to restrict the output of a set of tools to be used in another set of tools, you can add the flag `subclass="True"` to the datatype definition line. @@ -560,7 +550,7 @@ from galaxy.datatypes.metadata import MetadataElement The use of `` tags contained within `` tags is supported in the same way they are supported within the `datatypes_conf.xml.sample` file in the Galaxy distribution. ```xml - ``` diff --git a/lib/galaxy/config/sample/reports.yml.sample b/lib/galaxy/config/sample/reports.yml.sample index b1e950bee6a..e20f067bd7e 100644 --- a/lib/galaxy/config/sample/reports.yml.sample +++ b/lib/galaxy/config/sample/reports.yml.sample @@ -90,7 +90,7 @@ uwsgi: reports: # Verbosity of console log messages. Acceptable values can be found - # here: https://docs.python.org/2/library/logging.html#logging-levels + # here: https://docs.python.org/library/logging.html#logging-levels #log_level: DEBUG # Database connection. Galaxy Reports are intended for production diff --git a/lib/galaxy/webapps/reports/config_schema.yml b/lib/galaxy/webapps/reports/config_schema.yml index 0a78e606276..41ff5ba4f1d 100644 --- a/lib/galaxy/webapps/reports/config_schema.yml +++ b/lib/galaxy/webapps/reports/config_schema.yml @@ -13,7 +13,7 @@ mapping: default: DEBUG desc: | Verbosity of console log messages. Acceptable values can be found here: - https://docs.python.org/2/library/logging.html#logging-levels + https://docs.python.org/library/logging.html#logging-levels database_connection: type: str