From e725f8423a68c5fb802cdbe27c17d30f01272c3c Mon Sep 17 00:00:00 2001 From: Nicola Soranzo Date: Tue, 5 Feb 2019 17:53:57 +0000 Subject: [PATCH] Disable auto toctree for Markdown docs. Doc syntax fixes recommonmark allows now to use the `.md` extension for cross referencing other documents, which also works on GitHub. --- doc/source/admin/apache.md | 12 ++++---- doc/source/admin/cluster.md | 6 ++-- doc/source/admin/config.rst | 13 ++++----- doc/source/admin/jobs.md | 39 +++++++++++++------------- doc/source/admin/nginx.md | 14 ++++----- doc/source/admin/production.md | 24 ++++++++-------- doc/source/admin/scaling.md | 16 +++++------ doc/source/conf.py | 1 + doc/source/releases/16.10.rst | 6 ++-- doc/source/releases/17.09_announce.rst | 8 +++--- lib/galaxy/tools/xsd/galaxy.xsd | 2 +- 11 files changed, 70 insertions(+), 71 deletions(-) diff --git a/doc/source/admin/apache.md b/doc/source/admin/apache.md index 3305c4952cb..89151ba11f5 100644 --- a/doc/source/admin/apache.md +++ b/doc/source/admin/apache.md @@ -10,7 +10,7 @@ some of the more menial and resource-intensive tasks. [The Apache HTTP Server][apache] is a widely deployed and very featureful general purpose web server with mature proxying capabilities. -Instructions for [proxying with NGINX](nginx.html), which is the proxy server used by The Galaxy Project's public +Instructions for [proxying with NGINX](nginx.md), which is the proxy server used by The Galaxy Project's public servers, [usegalaxy.org][main] ("Main") and [Test][test], as well as the [Docker Galaxy project][docker-galaxy], are also available. @@ -89,7 +89,7 @@ And on EL: The following configuration is not exhaustive, only the portions most relevant to serving Galaxy are shown, these should be incorporated with your existing/default Apache config as is appropriate for your server. Notably, the Apache package you installed most likely has a multi-file config layout. If you are not already familiar with that layout and where -best to place your configuration, you can learn more in the [Proxy Package Layouts](proxy_package_layout.html) +best to place your configuration, you can learn more in the [Proxy Package Layouts](proxy_package_layout) documentation. ```apache @@ -172,7 +172,7 @@ SSLStaplingCache shmcb:/var/run/ocsp(128000) Be sure to set `galaxy_root` to the path to your copy of Galaxy and modify the value of `ProxyPass /` to match your uWSGI socket path. With the default configuration, uWSGI will bind to a random TCP socket, so you will need to set it to -a fixed value as described in the [Scaling and Load Balancing](scaling.html) documentation. If using a UNIX domain +a fixed value as described in the [Scaling and Load Balancing](scaling.md) documentation. If using a UNIX domain socket, be sure to pay particular attention to the discussion of users and permissions. ### Additional Notes @@ -243,7 +243,7 @@ previous section: `cookie_path` should be set to prevent Galaxy's session cookies from clobbering each other if you are running more than one instance of Galaxy under different URL prefixes on the same hostname. - Be sure to consult the [Scaling and Load Balancing](scaling.html) documentation, other options unrelated to proxying + Be sure to consult the [Scaling and Load Balancing](scaling.md) documentation, other options unrelated to proxying should also be set in the `uwsgi` section of the config. ## Advanced Configuration Topics @@ -252,7 +252,7 @@ previous section: Galaxy sends files (e.g. dataset downloads) by opening the file and streaming it in chunks through the proxy server. However, this ties up the Galaxy process, which can impact the performance of other operations (see [Production Server -Configuration](production.html) for a more in-depth explanation). +Configuration](production.md) for a more in-depth explanation). Apache can assume this task instead and as an added benefit, speed up downloads. This is accomplished through the use of `mod_xsendfile`, a 3rd-party Apache module. Dataset security is maintained in this configuration because Apache will @@ -299,7 +299,7 @@ group as shown above. ### External user authentication - [Apache for External Authentication](https://galaxyproject.org/admin/config/apache-external-user-auth/) -- [Built-in Galaxy External Authentication](authentication.html) +- [Built-in Galaxy External Authentication](authentication.md) #### Display Sites diff --git a/doc/source/admin/cluster.md b/doc/source/admin/cluster.md index 307bfb19686..9d76fd5f459 100644 --- a/doc/source/admin/cluster.md +++ b/doc/source/admin/cluster.md @@ -2,7 +2,7 @@ Galaxy is designed to run jobs on your local system by default, but it can be configured to run jobs on a cluster. The front-end Galaxy application runs on a single server as usual, but tools are run on cluster nodes instead. -A [general reference for the job configuration file](jobs.html) is also available. +A [general reference for the job configuration file](jobs.md) is also available. ## Distributed Resources Managers @@ -70,7 +70,7 @@ You may also find that attribute caching in your filesystem causes problems with ## Runner Configuration -**This documentation covers configuration of the various runner plugins, not how to distribute jobs to the various plugins.** Consult the [job configuration file documentation](jobs.html) for full details on the correct syntax, and for instructions on how to configure tools to actually use the runners explained below. +**This documentation covers configuration of the various runner plugins, not how to distribute jobs to the various plugins.** Consult the [job configuration file documentation](jobs.md) for full details on the correct syntax, and for instructions on how to configure tools to actually use the runners explained below. ### Local @@ -121,7 +121,7 @@ galaxy_server% export DRMAA_LIBRARY_PATH=/galaxy/sge/lib/lx24-amd64/libdrmaa.so **Limitations**: The DRMAA runner does not work if Galaxy is configured to run jobs as real user, because in this setting jobs are submitted with an external script, i.e. in an extra DRMAA session, and the session based (python) DRMAA library can only query jobs within the session in which started them. Furthermore, the DRMAA job runner only distinguishes successful and failed jobs and ignores information about possible failure sources, e.g. runtime / memory violation, which could be used for job resubmission. Specialized job runners are abvailable that are not affected by these limitations, e.g. univa and slurm runners. -**TORQUE**: The DRMAA runner can also be used (instead of the [PBS](cluster.html#pbs) runner) to submit jobs to TORQUE, however, problems have been reported when using the `libdrmaa.so` provided with TORQUE. Using this library will result in a segmentation fault when the drmaa runner attempts to write the job template, and any native job runner options will not be passed to the DRM. Instead, you should compile the [pbs-drmaa](http://apps.man.poznan.pl/trac/pbs-drmaa/wiki) library and use this as the value for `$DRMAA_LIBRARY_PATH`. +**TORQUE**: The DRMAA runner can also be used (instead of the [PBS](#pbs) runner) to submit jobs to TORQUE, however, problems have been reported when using the `libdrmaa.so` provided with TORQUE. Using this library will result in a segmentation fault when the drmaa runner attempts to write the job template, and any native job runner options will not be passed to the DRM. Instead, you should compile the [pbs-drmaa](http://apps.man.poznan.pl/trac/pbs-drmaa/wiki) library and use this as the value for `$DRMAA_LIBRARY_PATH`. **Slurm**: You will need to install [slurm-drmaa](https://github.com/natefoo/slurm-drmaa/). In production on [usegalaxy.org](https://usegalaxy.org) we observed pthread deadlocks in slurm-drmaa that would cause Galaxy job handlers to eventually stop processing jobs until the handler was restarted. Compiling slurm-drmaa using the compiler flags `-g -O0` (keep debugging symbols, disable optimization) caused the deadlock to disappear. diff --git a/doc/source/admin/config.rst b/doc/source/admin/config.rst index b10be4dc053..dd628554a2d 100644 --- a/doc/source/admin/config.rst +++ b/doc/source/admin/config.rst @@ -31,12 +31,12 @@ The most commonly modified configuration files include: - ``galaxy.yml``: Core Galaxy configuration file. - ``tool_conf.xml``: Describes the paths to local tool configurations that Galaxy should attempt to load. Tools that - are installed via the Tool Shed are configured to load in the ``shed_tool_conf.xml`` file. See the :ref:`Tool Panel + are installed via the Tool Shed are configured to load in the ``shed_tool_conf.xml`` file. See the :doc:`Tool Panel Administration ` and `Installing Tools into Galaxy`_ documentation for more. - ``datatypes_conf.xml``: Describes the file formats that are supported in Galaxy. See the `Datatypes documentation`_ for more. -- ``job_conf.xml``: Controls how Galaxy runs tools, e.g. to run them on a compute cluster. See the `Cluster - documentation`_ for more. +- ``job_conf.xml``: Controls how Galaxy runs tools, e.g. to run them on a compute cluster. See the :doc:`Cluster + documentation ` for more. Some configuration files are only used when adding local components, rather than ones installed from the Tool Shed: @@ -50,7 +50,7 @@ Some configuration files are only used when adding local components, rather than Managers documentation`_ for more. - ``local_conda_mapping.yml``: Define mappings between the names specified in the tool configuration (```` tags) and the conda resolver's names (conda package name). -- ``lmod_modules_mapping.yml``: Define mappings between the names specified in the tool configuration (```` +- ``lmod_modules_mapping.yml``: Define mappings between the names specified in the tool configuration (```` tags) and the Lmod system. Some configuration files are used to control the way that Galaxy resolves tool dependencies. Most Galaxy tools are only @@ -69,7 +69,7 @@ Additional configuration files and their purposes are: particular command line tool) should locate their dependencies (the command line tool) that are not part of the tool. See the `Dependency Resolvers documentation ` for more. - ``error_report.yml``: Controls how user-initiated error reporting (e.g. due to tool failure) is performed. See the - :ref:`Bug Reports documentation ` for more. + :doc:`Bug Reports documentation ` for more. - ``job_metrics_conf.xml``: Enables reporting of certain conditions and collection of metrics when jobs run. - ``job_resource_params_conf.xml``: Describes tool form elements that should be inserted into tool forms that can be used by users to control runtime parameters such as memory allocations, cluster selection, and so forth. @@ -85,7 +85,6 @@ Additional configuration files and their purposes are: .. _standardize and unify configuration formats: https://github.com/galaxyproject/galaxy/issues/5148 .. _Installing Tools into Galaxy: https://galaxyproject.org/admin/tools/add-tool-from-toolshed-tutorial/ .. _Datatypes documentation: https://galaxyproject.org/learn/datatypes/ -.. _Cluster documentation: cluster.html .. _Data Preparation documentation: https://galaxyproject.org/admin/data-preparation/ .. _Data Managers documentation: https://galaxyproject.org/admin/tools/data-managers/ @@ -110,7 +109,7 @@ Configuration Basics .. _uWSGI YAML configuration file: https://uwsgi-docs.readthedocs.io/en/latest/Configuration.html .. _large number of options: https://uwsgi-docs.readthedocs.io/en/latest/Options.html - + ---------------------------- Configuration Options ---------------------------- diff --git a/doc/source/admin/jobs.md b/doc/source/admin/jobs.md index a925afbc2d9..cb86595e05a 100644 --- a/doc/source/admin/jobs.md +++ b/doc/source/admin/jobs.md @@ -2,7 +2,7 @@ By default, jobs in Galaxy are run locally on the server on which the Galaxy application was started. Many options are available for running Galaxy jobs on other systems, including clusters and other remote resources. -This document is a reference for the job configuration file. [Detailed documentation](cluster.html) is provided for configuring Galaxy to work with a variety of Distributed Resource Managers (DRMs) such as TORQUE, Grid Engine, LSF, and HTCondor. Additionally, a wide range of infrastructure decisions and configuration changes should be made when running Galaxy as a production service, as one is likely doing if using a cluster. It is highly recommended that the [production server documentation](production.html) and [cluster configuration documentation](cluster.html) be read before making changes to the job configuration. +This document is a reference for the job configuration file. [Detailed documentation](cluster.md) is provided for configuring Galaxy to work with a variety of Distributed Resource Managers (DRMs) such as TORQUE, Grid Engine, LSF, and HTCondor. Additionally, a wide range of infrastructure decisions and configuration changes should be made when running Galaxy as a production service, as one is likely doing if using a cluster. It is highly recommended that the [production server documentation](production.md) and [cluster configuration documentation](cluster.md) be read before making changes to the job configuration. **The most up-to-date details of advanced job configuration features can be found in the [sample job_conf.xml](https://github.com/galaxyproject/galaxy/blob/dev/config/job_conf.xml.sample_advanced) found in the Galaxy distribution.** @@ -38,7 +38,7 @@ workers ### Job Handlers -The `` configuration elements defines which Galaxy server processes (when [running multiple server processes](scaling.html)) should be used for running jobs, and how to group those processes. +The `` configuration elements defines which Galaxy server processes (when [running multiple server processes](scaling.md)) should be used for running jobs, and how to group those processes. The handlers configuration may define a ``default`` attribute. This is the the handler(s) that should be used if no explicit handler is defined for a job. If unset, any untagged handlers will be used by default. @@ -49,7 +49,7 @@ id A server name that should be used to run jobs. Server names are dependent on your application server deployment scenario and are explained in the :ref:`configuration section of the scaling documentation `. tags - A comma-separated set of strings that optional define tags to which this handler belongs. + A comma-separated set of strings that optional define tags to which this handler belongs. ``` ### Job Destinations @@ -70,7 +70,7 @@ tags Tags to which this destination belongs (for example `tags="longwalltime,bigcluster"`). ``` -``destination`` elements may contain zero or more ````s, which are passed to the destination's defined runner plugin and interpreted in a way native to that plugin. For details on the parameter specification, see the documentation on [Cluster configuration](cluster.html). +``destination`` elements may contain zero or more ````s, which are passed to the destination's defined runner plugin and interpreted in a way native to that plugin. For details on the parameter specification, see the documentation on [Cluster configuration](cluster.md). ### Environment Modifications @@ -109,7 +109,7 @@ destination Job destination(s) that should be used to run jobs for this tool after resubmission. ``` -**Note:** Currently, failure conditions for memory limits and walltime are only implemented for the [Slurm](cluster.html) job runner plugin. Contributions for other implementations would be greatly appreciated! An example job configuration and an always-fail job runner plugin for development [can be found in this gist](https://gist.github.com/natefoo/361414fbca3c0ea63aa5). +**Note:** Currently, failure conditions for memory limits and walltime are only implemented for the [Slurm](cluster.md) job runner plugin. Contributions for other implementations would be greatly appreciated! An example job configuration and an always-fail job runner plugin for development [can be found in this gist](https://gist.github.com/natefoo/361414fbca3c0ea63aa5). ### Running jobs in containers @@ -124,7 +124,7 @@ work. The images used for containers can either be specified explicitely in the ```` using the *docker_default_container_id*, *docker_container_id_override*, *singularity_default_container_id* and *singularity_container_id_override* parameters, but (perhaps more commonly) the image to use can be derived from the tool requirements of the Galaxy tool being executed. In this latter case the image is specified by the -tool using a ```` tag in the ```` section. +tool using a ```` tag in the ```` section. ### Macros @@ -184,7 +184,6 @@ To define and use rules, copy this sample file to `config/tool_destinations.yml` ``` - #### Dynamic Destination Mapping (Python method) The simplest way to get started with dynamic job destinations is to first create a dynamic job destination in `job_conf.xml`'s `` section: @@ -205,7 +204,6 @@ Next for any tool one wants to dynamically assign job destinations for, this `bl ``` - Finally, you will need to define a function that describes how `ncbi_blastn_wrapper` should be executed. To do this, one must create a python source file in `lib/galaxy/jobs/rules`, for instance `destinations.py` (though the name of this file is largely unimportant, one can distribute any number of functions across any number of files and they will be automatically detected by Galaxy). So open `lib/galaxy/jobs/rules/destinations.py` and define a `ncbi_blastn_wrapper` function. A couple possible examples may be: @@ -228,8 +226,7 @@ def ncbi_blastn_wrapper(job): ``` - -or +or ```python from galaxy.jobs import JobDestination @@ -278,7 +275,7 @@ The above examples demonstrate that the dynamic job destination framework will p A dictionary of parameters specified by the user using ``job_resource_params_conf.xml`` (if configured). ``workflow_invocation_uuid`` - A randomly generated UUID for the workflow invocation generating this job - this can be + A randomly generated UUID for the workflow invocation generating this job - this can be useful for instance in routing all the jobs in the same workflow to one resource. ``` @@ -356,7 +353,7 @@ def dev_only(user_email): if user_email in DEV_EMAILS return JobDestination(runner="drmaa") else: - raise JobMappingException("This tool is under development and you are not authorized to it.") + raise JobMappingException("This tool is under development and you are not authorized to it.") ``` @@ -377,39 +374,41 @@ The `` collection has no attributes. The collection contains ``s, which have different meanings based on their required `type` attribute: ```eval_rst -type +``type`` Type of limit to define - one of ``registered_user_concurrent_jobs``, ``anonymous_user_concurrent_jobs``, ``destination_user_concurrent_jobs``, ``destination_total_concurrent_jobs``, ``walltime``, and ``output_size``. -id +``id`` Optional destination on which to apply limit (for ``destination_user_concurrent_jobs`` and ``destination_total_concurrent_jobs`` types only) (e.g. ``id="galaxy_cluster"``). -tag +``tag`` Optional destinations on which to apply limit (for ``destination_user_concurrent_jobs`` and ``destination_total_concurrent_jobs`` types only). +``` If a limit tag is defined, its value must be set. If the limit tag is not defined, the default for each type is unlimited. The syntax for the available `type`s are: +```eval_rst ``registered_user_concurrent_jobs`` Limit on the number of jobs a user with a registered Galaxy account can have active across all destinations. ``anonymous_user_concurrent_jobs`` Limit on the number of jobs an unregistered/anonymous user can have active across all destinations. -``destination_user_concurrent_jobs`` +``destination_user_concurrent_jobs`` The number of jobs a user can have active in the specified destination, or across all destinations identified by the specified tag. ``destination_total_concurrent_jobs`` The number of jobs that can be active in the specified destination (or across all destinations identified by the specified tag) by any/all users. ``walltime`` - Amount of time a job can run (in any destination) before it will be terminated by Galaxy. + Amount of time a job can run (in any destination) before it will be terminated by Galaxy. ``total_walltime`` - Total walltime that jobs may not exceed during a set period. If total walltime of finished - jobs exceeds this value, any new jobs are paused. This limit should include a `window` + Total walltime that jobs may not exceed during a set period. If total walltime of finished + jobs exceeds this value, any new jobs are paused. This limit should include a ``window`` attribute that is the number in days representing the period. ``output_size`` - Size that any defined tool output can grow to before the job will be terminated. This does not include temporary files created by the job (e.g. ``53687091200`` for (50 GB)). + Size that any defined tool output can grow to before the job will be terminated. This does not include temporary files created by the job (e.g. ``53687091200`` for 50 GB). ``` The concept of "across all destinations" is used because Galaxy allows users to run jobs across any number of local or remote (cluster) resources. A user may always queue an unlimited number of jobs in Galaxy's internal job queue. The concurrency limits apply to jobs that have been dispatched and are in the `queued` or `running` states. These limits prevent users from monopolizing the resources Galaxy runs on by, for example, preventing a single user from submitting more long-running jobs than Galaxy has cluster slots to run and subsequently blocking all Galaxy jobs from running for any other user. diff --git a/doc/source/admin/nginx.md b/doc/source/admin/nginx.md index ebd6960c462..1283cf5dec3 100644 --- a/doc/source/admin/nginx.md +++ b/doc/source/admin/nginx.md @@ -13,7 +13,7 @@ servers, [usegalaxy.org][main] ("Main") and [Test][test], as well as the [Docker NGINX, rather than Apache, to proxy Galaxy. NGINX was chosen for its simple, fast load balancing and other proxy-oriented features. -Instructions for [proxying with Apache](apache.html) are also available. +Instructions for [proxying with Apache](apache.md) are also available. [nginx]: http://nginx.org/en/ [main]: https://galaxyproject.org/main/ @@ -57,7 +57,7 @@ uWSGI protocol support is built in to nginx, so (unlike Apache) no extra modules The following configuration is not exhaustive, only the portions most relevant to serving Galaxy are shown, these should be incorporated with your existing/default nginx config as is appropriate for your server. Notably, the nginx package you installed most likely has a multi-file config layout. If you are not already familiar with that layout and where -best to place your configuration, you can learn more in the [Proxy Package Layouts](proxy_package_layout.html) +best to place your configuration, you can learn more in the [Proxy Package Layouts](proxy_package_layout) documentation. ```nginx @@ -158,7 +158,7 @@ http { Be sure to set `$galaxy_root` to the path to your copy of Galaxy and modify the value of `uwsgi_pass` to match your uWSGI socket path. With the default configuration, uWSGI will bind to a random TCP socket, so you will need to set it to -a fixed value as described in the [Scaling and Load Balancing](scaling.html) documentation. If using a UNIX domain +a fixed value as described in the [Scaling and Load Balancing](scaling.md) documentation. If using a UNIX domain socket, be sure to pay particular attention to the discussion of users and permissions. ### Additional Notes @@ -235,7 +235,7 @@ previous section: `cookie_path` should be set to prevent Galaxy's session cookies from clobbering each other if you are running more than one instance of Galaxy under different URL prefixes on the same hostname. - Be sure to consult the [Scaling and Load Balancing](scaling.html) documentation, other options unrelated to proxying + Be sure to consult the [Scaling and Load Balancing](scaling.md) documentation, other options unrelated to proxying should also be set in the `uwsgi` section of the config. ## Advanced Configuration Topics @@ -244,7 +244,7 @@ previous section: Galaxy sends files (e.g. dataset downloads) by opening the file and streaming it in chunks through the proxy server. However, this ties up the Galaxy process, which can impact the performance of other operations (see [Production Server -Configuration](production.html) for a more in-depth explanation). +Configuration](production.md) for a more in-depth explanation). Nginx can assume this task instead and as an added benefit, speed up downloads. This is accomplished through the use of the special `X-Accel-Redirect` header. Dataset security is maintained in this configuration because nginx will still @@ -360,7 +360,7 @@ galaxy: You may find it useful to require authentication for access to certain paths on your server. For example, Galaxy can run a separate reports app which gives useful information about your Galaxy instance. See the [Reports Configuration -documentation](reports.html) and [Peter Briggs' blog post on the +documentation](reports.md) and [Peter Briggs' blog post on the subject](http://galacticengineer.blogspot.com/2015/06/exposing-galaxy-reports-via-nginx-in.html) for more. After successfully following the blog post, Galaxy reports should be available at e.g. `https://galaxy.example.org/reports`. @@ -390,4 +390,4 @@ To secure this page to only Galaxy administrators, adjust your nginx config acco ### External User Authentication - [Nginx for External Authentication](https://galaxyproject.org/admin/config/nginx-external-user-auth/) -- [Built-in Galaxy External Authentication](authentication.html) +- [Built-in Galaxy External Authentication](authentication.md) diff --git a/doc/source/admin/production.md b/doc/source/admin/production.md index 5d4a04c7489..6234953f783 100644 --- a/doc/source/admin/production.md +++ b/doc/source/admin/production.md @@ -7,8 +7,8 @@ The [basic installation instructions](https://getgalaxy.org) are suitable for de By default, Galaxy: * Uses [SQLite](https://www.sqlite.org/) (a serverless database), so you don't have to run/configure a database server for quick or basic development. However, while SQLite [supports concurrent access](https://sqlite.org/lockingv3.html) it does not support multiple concurrent writes, which can reduce system throughput. -* Uses a built-in HTTP server, written in Python. Much of the work performed by this server can be moved to [nginx](nginx.html) or Apache, which will increase performance. -* Runs all tools locally. Moving to a [cluster](cluster.html) will greatly increase capacity. +* Uses a built-in HTTP server, written in Python. Much of the work performed by this server can be moved to [nginx](nginx.md) or [Apache](apache.md), which will increase performance. +* Runs all tools locally. Moving to a [cluster](cluster.md) will greatly increase capacity. * Runs in a single process, which is a performance problem in [CPython](http://en.wikipedia.org/wiki/CPython). Galaxy ships with this default configuration to ensure the simplest, most error-proof configuration possible when doing basic development. As you'll soon see, the goal is to remove as much work as possible from the Galaxy process, since doing so will greatly speed up the performance of its remaining duties. This is due to the Python Global Interpreter Lock (GIL), which is explained in detail in the [Advanced Configuration](#advanced-configuration) section. @@ -33,7 +33,7 @@ nate@weyerbacher% cd galaxy-dist nate@weyerbacher% sh run.sh ``` -* Galaxy can be housed in a cluster/network filesystem (it's been tested with NFS and GPFS), and you'll want to do this if you'll be running it on a [cluster](cluster.html). +* Galaxy can be housed in a cluster/network filesystem (it's been tested with NFS and GPFS), and you'll want to do this if you'll be running it on a [cluster](cluster.md). ## Basic configuration @@ -93,16 +93,16 @@ Downloading and uploading data can also be moved to the proxy server. This is e Virtually any server that proxies HTTP should work, although we provide configuration examples for: -* [Apache](apache.html), and -* [nginx](nginx.html), a high performance reverse proxy, used by our public Galaxy sites +* [Apache](apache.md), and +* [nginx](nginx.md), a high performance reverse proxy, used by our public Galaxy sites ### Using a compute cluster -Galaxy is a framework that runs command-line tools, and if properly configured, can run these tools on a compute [cluster](cluster.html). Without a cluster, you'll be limited to the number of cores in your server, minus those needed to run Galaxy itself. Galaxy currently supports TORQUE PBS, PBS Pro, Platform LSF, and Sun Grid Engine clusters, and does not require a dedicated or special cluster configuration. Tools can even run on heterogeneous cluster nodes (differing operating systems), as long as any dependencies necessary to run the tool are available on that platform. +Galaxy is a framework that runs command-line tools, and if properly configured, can run these tools on a compute [cluster](cluster.md). Without a cluster, you'll be limited to the number of cores in your server, minus those needed to run Galaxy itself. Galaxy currently supports TORQUE PBS, PBS Pro, Platform LSF, and Sun Grid Engine clusters, and does not require a dedicated or special cluster configuration. Tools can even run on heterogeneous cluster nodes (differing operating systems), as long as any dependencies necessary to run the tool are available on that platform. Using a cluster will also net you a fringe benefit: When running tools locally, they are child processes of the Galaxy server. This means that if you restart the server, you lose contact with those jobs, and they must be restarted. However on the cluster, if the Galaxy server restarts, the jobs will continue to run and finish. Once the Galaxy job manager starts up, it'll resume tracking and finishing jobs as if nothing had happened. -Configuration is not difficult once your cluster is set up. Details can be found on the [cluster](cluster.html) page. +Configuration is not difficult once your cluster is set up. Details can be found on the [cluster](cluster.md) page. ### Cleaning up datasets @@ -110,7 +110,7 @@ When datasets are deleted from a history or library, it is simply marked as dele ### Rotate log files -To use logrotate to rotate Galaxy log files, add a new file named "galaxy" to /etc/logrotate.d/ directory with something like: +To use logrotate to rotate Galaxy log files, add a new file named `galaxy` to `/etc/logrotate.d/` directory with something like: ``` PATH_TO_GALAXY_LOG_FILES { @@ -137,7 +137,7 @@ To get started with setting up local data, please see [Data Integration](https:/ ### Enable upload via FTP -File sizes have grown very large thanks to rapidly advancing sequencer technology, and it is not always practical to upload these files through the browser. Thankfully, a simple solution is to allow Galaxy users to upload them via FTP and import those files in to their histories. Configuration for FTP is explained on the [File Upload via FTP](special_topics/ftp.html) page. +File sizes have grown very large thanks to rapidly advancing sequencer technology, and it is not always practical to upload these files through the browser. Thankfully, a simple solution is to allow Galaxy users to upload them via FTP and import those files in to their histories. Configuration for FTP is explained on the [File Upload via FTP](special_topics/ftp.md) page. ## Advanced configuration @@ -145,11 +145,11 @@ File sizes have grown very large thanks to rapidly advancing sequencer technolog As already mentioned, unloading work from the Galaxy process is important due to the Python [Global Interpreter Lock](https://docs.python.org/c-api/init.html#thread-state-and-the-global-interpreter-lock) (GIL). The GIL is how Python ensures thread safety, and it accomplishes this by only allowing one thread to control execution at a time. This means that regardless of the number of cores in your server, Galaxy can only use one. However, there's a solution: run multiple Galaxy processes and use the proxy server to balance across all of these processes. In practice, Galaxy is split into job handler and web server processes. Job handlers do not service any user requests directly via the web. Instead, they watch the database for new jobs, and upon finding them, handle the preparation, monitoring, running, and completion of them. Likewise, the web server processes are free to deal only with serving content and files to web clients. -Full details on how to configure scaling and load balancing can be found in the [scaling](scaling.html) documentation. +Full details on how to configure scaling and load balancing can be found in the [scaling](scaling.md) documentation. ### Unloading even more work -For those readers who've already been running Galaxy on a cluster, a bit of information was recently added to the [cluster](cluster.html) documentation regarding running the data source tools on the cluster (contrary to the default configuration). Running all tools on the cluster is strongly encouraged, so if you have not done this, please check out the new information. +For those readers who've already been running Galaxy on a cluster, a bit of information was recently added to the [cluster](cluster.md) documentation regarding running the data source tools on the cluster (contrary to the default configuration). Running all tools on the cluster is strongly encouraged, so if you have not done this, please check out the new information. ### Tune the database @@ -161,4 +161,4 @@ Finally, if you are using Galaxy <= release_2014.06.02, we recommend that you in ### Make the proxy handle uploads and downloads -By default, Galaxy receives file uploads as a stream from the proxy server and then writes this file to disk. Likewise, it sends files as a stream to the proxy server. This occupies the GIL in that Galaxy process and will decrease responsiveness for other operations in that process. To solve this problem, you can configure your proxy server to serve downloads directly, involving Galaxy only for the task of authorizing that the user has permission to read the dataset. If using nginx as the proxy, you can configure it to receive uploaded files and write them to disk itself, only notifying Galaxy of the upload once it's completed. All the details on how to configure these can be found on the [Apache](apache.html) and [nginx](nginx.html) proxy instruction pages. +By default, Galaxy receives file uploads as a stream from the proxy server and then writes this file to disk. Likewise, it sends files as a stream to the proxy server. This occupies the GIL in that Galaxy process and will decrease responsiveness for other operations in that process. To solve this problem, you can configure your proxy server to serve downloads directly, involving Galaxy only for the task of authorizing that the user has permission to read the dataset. If using nginx as the proxy, you can configure it to receive uploaded files and write them to disk itself, only notifying Galaxy of the upload once it's completed. All the details on how to configure these can be found on the [Apache](apache.md) and [nginx](nginx.md) proxy instruction pages. diff --git a/doc/source/admin/scaling.md b/doc/source/admin/scaling.md index 0507dbd98b4..be3ef0a2cd8 100644 --- a/doc/source/admin/scaling.md +++ b/doc/source/admin/scaling.md @@ -6,7 +6,7 @@ which prevents more than one thread from being on CPU at a time. Because of thi improve the Galaxy framework's performance out of the box since Galaxy can use (at most) one core at a time in its default configuration. However, Galaxy can easily run in multiple separate processes, which solves this problem. For a more thorough explanation of this problem and why you will almost surely want to switch to the multiprocess -configuration if running for more than a small handful of users, see the [production configuration](production.html) +configuration if running for more than a small handful of users, see the [production configuration](production.md) page. Just to be clear: increasing the values of `threadpool_workers` in `galaxy.yml` or the number of plugin workers in @@ -43,7 +43,7 @@ Beginning with Galaxy release 18.01, the default application server for new inst Prior to 18.01, it was possible (and indeed, recommended for production Galaxy servers) to run Galaxy under uWSGI, but it was necessary to install and configure uWSGI separately from Galaxy. uWSGI is now provided with Galaxy as a Python Wheel and installed in to its virtualenv, as described in detail in the [Framework -Dependencies](framework_dependencies.html) documentation. +Dependencies](framework_dependencies) documentation. uWSGI has numerous benefits over Python Paste for our purposes: @@ -62,15 +62,15 @@ uWSGI has numerous benefits over Python Paste for our purposes: There are multiple deployment strategies for the Galaxy application that you can choose from. The right one depends on the configuration of the infrastructure on which you are deploying. In all cases, all Galaxy job features such as -[running on a cluster](cluster.html) are supported. +[running on a cluster](cluster.md) are supported. Although uWSGI implements nearly all the features that were previously the responsibility of an upstream proxy server, at this time, it is still recomended to place a proxy server in front of uWSGI and utilize it for all of its traditional roles (serving static content, serving dataset downloads, etc.) as described in the [production -configuration](production.html) documentation. +configuration](production.md) documentation. When using uWSGI with a proxy server, it is recommended that you use the native high performance uWSGI protocol -(supported by both [Apache](apache.html) and [nginx](nginx.html)) between uWSGI and the +(supported by both [Apache](apache.md) and [nginx](nginx.md)) between uWSGI and the proxy server, rather than HTTP. ### uWSGI with jobs handled by web workers (default configuration) @@ -340,7 +340,7 @@ permission on the socket. Because Galaxy and the proxy server most likely run as be the case by default. One common solution is to add the proxy server's user to the Galaxy user's primary group. uWSGI's `chmod-socket` option can also help here. -You can consult the Galaxy documentation for [Apache](apache.html) or [nginx](nginx.html) +You can consult the Galaxy documentation for [Apache](apache.md) or [nginx](nginx.md) for help with the proxy-side configuration. By setting the `socket` option, `run.sh` will no longer automatically serve Galaxy via HTTP (since it is assumed that @@ -440,7 +440,7 @@ separated, to the `job-handlers` farm. For example, 3 handlers are defined like By default, a job will be handled by whatever mule currently has the lock on the mule message queue. After receiving a message, it will release the lock, giving other mules a chance to handle future jobs. Jobs can be explicitly mapped to -specific mules as described in the [Job configuration documentation](jobs.html) by using the handler IDs +specific mules as described in the [Job configuration documentation](jobs.md) by using the handler IDs `main.job-handlers.N`, where `N` is the mule's position in the farm, starting at 1 and incrementing for each mule in the farm (this is not necessarily the mule ID, but it will be if you only define one farm and you add mules to that farm in sequential order). Each worker that you wish to explicitly map jobs to should be defined in the `` section @@ -624,7 +624,7 @@ More details on the `unix_signal` hook can be found in [uWSGI Issue #849](https: It's possible to configure uWSGI to log to a file with the `logto` or `logto2` options (when running in the foreground, the default), but more advanced logging options that split log files for each process are possible and described in the -Galaxy [Logging Configuration documentation](config_logging.html) +Galaxy [Logging Configuration documentation](config_logging) When running as a daemon with `run.sh --daemon`, output is logged to `galaxy.log` and the pid is written to `galaxy.pid`. These can be controlled with the `daemonize` and `pidfile` arguments (their `daemonize2` and `pidfile2` diff --git a/doc/source/conf.py b/doc/source/conf.py index 49173724736..94be863691c 100644 --- a/doc/source/conf.py +++ b/doc/source/conf.py @@ -64,6 +64,7 @@ def setup(app): app.connect("autodoc-skip-member", dont_skip_init) app.add_config_value('recommonmark_config', { 'enable_auto_doc_ref': False, + 'enable_auto_toc_tree': False, }, True) app.add_transform(AutoStructify) diff --git a/doc/source/releases/16.10.rst b/doc/source/releases/16.10.rst index 14fbd984870..7ffb4f35fc1 100644 --- a/doc/source/releases/16.10.rst +++ b/doc/source/releases/16.10.rst @@ -285,7 +285,7 @@ Fixes * Remove unnecessary ``set_output_history`` parameter (thanks to `@nsoranzo `__). `Pull Request 3155`_ -* Fix BLAST database *.loc files inconsistency +* Fix BLAST database ``*.loc`` files inconsistency (thanks to `@peterjc `__). `Pull Request 3098`_ * Log invalid XML filename @@ -387,7 +387,7 @@ Fixes `Pull Request 3049`_ * Remove incorrect communication server check. `Pull Request 3053`_ -* Fix tool XSD to accept a help attribute for ``section``s +* Fix tool XSD to accept a help attribute for ``section``\ s (thanks to `@joachimwolff `__). `Pull Request 3131`_ * Fix import orders for updates to flake8_import_order. @@ -445,8 +445,8 @@ Fixes * Fix for ToolShed install when copied sample data target exists, but is broken symlink. `Pull Request 3279`_ -.. _Issue 2375: https://github.com/galaxyproject/galaxy/issues/2375 .. github_links +.. _Issue 2375: https://github.com/galaxyproject/galaxy/issues/2375 .. _Pull Request 1768: https://github.com/galaxyproject/galaxy/pull/1768 .. _Pull Request 2588: https://github.com/galaxyproject/galaxy/pull/2588 .. _Pull Request 2653: https://github.com/galaxyproject/galaxy/pull/2653 diff --git a/doc/source/releases/17.09_announce.rst b/doc/source/releases/17.09_announce.rst index b0314cfbef6..cb9e14e9b29 100644 --- a/doc/source/releases/17.09_announce.rst +++ b/doc/source/releases/17.09_announce.rst @@ -62,7 +62,7 @@ Deprecation Notices * The Galaxy Sample Tracking and External Services functionality is now considered deprecated. In the next releases we will remove it completely. Related PRs:`#4526 `__ `#4872 `__ . * The deprecated admin-only interface for Galaxy Data Libraries is staged to be removed in the next release. * Workflows API: When exposing WorkflowInvocationSteps ``state`` will no longer be available. -* The ``refresh_on_change`` attribute of a ```` tag in the tool syntax can no longer be set to a value of another parameter. Use boolean instead (e.g.``refresh_on_change="True"``). `Details `__ +* The ``refresh_on_change`` attribute of a ```` tag in the tool syntax can no longer be set to a value of another parameter. Use boolean instead (e.g. ``refresh_on_change="True"``). `Details `__ * The endpoint ``/api/configuration/toolbox`` is now deprecated and will be removed in the future. All tools are now watched for changes and this feature became obsolete. @@ -105,8 +105,8 @@ on the Galaxy server as the user running the Galaxy server process. The vulnerability only affects Galaxy servers on which Galaxy Interactive Environments are enabled (by setting the -`interactive_environment_plugins_directory` -option in `galaxy.ini`). Because the vulnerability can be exploited to +``interactive_environment_plugins_directory`` +option in ``galaxy.ini``). Because the vulnerability can be exploited to execute arbitrary code, the impact for affected servers is severe. Administrators of Galaxy servers where GIEs *are* enabled should update @@ -130,7 +130,7 @@ distribution and are enabled by default (most tools under the "Get Data" section of the tool panel), meaning that its exploitability is fairly high, as only one such tool needs to be enabled to be vulnerable, including any custom data source tools (any tool that uses -`tools/data_source/data_source.py`). +``tools/data_source/data_source.py``). What files are readable depends entirely upon what the job's user has access to read on the host(s) where jobs run. diff --git a/lib/galaxy/tools/xsd/galaxy.xsd b/lib/galaxy/tools/xsd/galaxy.xsd index 4fa7f826a7e..24200a149fa 100644 --- a/lib/galaxy/tools/xsd/galaxy.xsd +++ b/lib/galaxy/tools/xsd/galaxy.xsd @@ -279,7 +279,7 @@ and can be configured locally to adapt to any other package management system. ``` This older example shows a tool that requires R version 2.15.1. The -``tool_depensencies.xml`` should contain matching declarations for Galaxy to +``tool_dependencies.xml`` should contain matching declarations for Galaxy to actually install the R runtime. The ``set_envirornment`` type is only respected by the tool shed and is ignored by the newer and preferred conda dependency resolver.