diff --git a/doc/source/admin/application_server.rst b/doc/source/admin/application_server.rst new file mode 100644 index 00000000000..c2b39bcfb29 --- /dev/null +++ b/doc/source/admin/application_server.rst @@ -0,0 +1,57 @@ +Application Server +======================================== + +Overview +---------------------------- + +It is possible to run the Galaxy server in many different ways, including under different web application frameworks, or +as a standalone server with no web stack. For most of its modern life, prior to the 18.01 release, Galaxy used the +`Python Paste`_ web stack, and ran in a single process. + +Beginning with Galaxy release 18.01, the default application server for new installations of Galaxy is `uWSGI`_, which +has numerous benefits: + +- Written in C and designed to be high performance. +- Internal support for serving static content. +- Supports WebSockets, which allows for support of Galaxy Interactive Environments out of the box, removing the + dependency on Node.js. +- ... + +In addition, uWSGI has been the recommended application server for production Galaxy servers for many years, but was not +previously provided with Galaxy. + +What server am I running? +---------------------------- + +**Galaxy releases older than 18.01** + +- If you start your server with the provided ``run.sh`` script, it is using Paste. +- + +Unless it has been manually configured to run under uWSGI, your server is using Paste. + +**Galaxy releases 18.01 and newer** + +If you upgraded from + +In any case, ``pgrep(1)`` or ``ps(1)`` should provide the answer, for example: + +```sh-session +galaxy@server$ pgrep -af -u $EUID galaxy +16232 uwsgi --ini-paste config/galaxy.ini +16233 uwsgi --ini-paste config/galaxy.ini +17406 /srv/galaxy/venv/bin/python2.7 /srv/galaxy/venv/bin/uwsgi --yaml config/galaxy.yml +17419 /srv/galaxy/venv/bin/python2.7 /srv/galaxy/venv/bin/uwsgi --yaml config/galaxy.yml +17172 python ./scripts/paster.py serve config/galaxy.ini +``` + +The first 4 processes listed are uWSGI + + + +If you are upgrading to Galaxy 18.01 +from an older release, still have your ``galaxy.ini`` file in place, and have not manually switched to another stack +(such as uWSGI), Galaxy will continue to run under Paste. + +.. _Python Paste: http://paste.readthedocs.io/ +.. _uWSGI: https://uwsgi-docs.readthedocs.io/ diff --git a/doc/source/admin/config.rst b/doc/source/admin/config.rst new file mode 100644 index 00000000000..2b77488aa6f --- /dev/null +++ b/doc/source/admin/config.rst @@ -0,0 +1,111 @@ +Galaxy Configuration +======================================== + +Overview +---------------------------- + +Galaxy has a large number of configuration files in an array of formats. Although it is not required to modify *any* +Galaxy configuration files to run the server, most Galaxy servers will modify at least the core configuration file. +These have grown organically over time as new features with the needs for advanced syntaxes and config isolation have +been added. An effort is under way to `standardize and unify configuration formats`_. + +Configuration files can be found underneath the ``config/`` subdirectory, wherein you can find ``.sample`` files +corresponding to configuration files that you can modify by copying the ``.sample`` to ````. In many cases, +you will find that the sample configuration provides the most up-to-date and detailed documentation about the features +configured therein. + +Most config files are administered by hand, although a few (ones which begin with ``shed_*``) are modified by Galaxy +when installing from the Galaxy Tool Shed. When starting Galaxy for the first time, these files are copied from their +sample files automatically. You should not need to modify these unless you want to change the directory in to which Tool +Shed tools are installed. + + +Configuration Files +---------------------------- + +The primary Galaxy configuration file is ``galaxy.yml``. You will need to use this file to modify core functionality +such as the port on which Galaxy listens, the directory in which Galaxy datasets are stored, the database connection +options, and so forth. + +The most commonly modified configuration files include: + +- ``galaxy.yml``: Core Galaxy configuration file. +- ``tool_conf.xml``: Describes the paths to local tool configurations that Galaxy should attempt to load. Tools that + are installed via the Tool Shed are configured to load in the ``shed_tool_conf.xml`` file. See the :ref:`Tool Panel + Administration ` and `Installing Tools into Galaxy`_ documentation for more. +- ``datatypes_conf.xml``: Describes the file formats that are supported in Galaxy. See the `Datatypes documentation`_ + for more. +- ``job_conf.xml``: Controls how Galaxy runs tools, e.g. to run them on a compute cluster. See the `Cluster + documentation`_ for more. + +Some configuration files are only used when adding local components, rather than ones installed from the Tool Shed: + +- ``tool_conf.xml``: As described above. +- ``tool_data_table_conf.xml``: Describes the mapping between Data Tables - the structured format that allow tools to + work with locally cached reference data - and the location files that describe the actual data that is available (e.g. + paths, genome builds, etc.). Data table configurations are also provided by tools in the Tool Shed, those are + configured in ``shed_tool_data_table_conf.xml``. See the `Data Preparation documentation`_ for more. +- ``data_manager_conf.xml``: Describes the paths to local Data Managers, special Galaxy tools that automatically fetch + or create data for Data Tables and manage the corresponding data table and location configuration files. See the `Data + Managers documentation`_ for more. +- ``local_conda_mapping.yml``: Define mappings between the names specified in the tool configuration (```` + tags) and the conda resolver's names (conda package name). +- ``lmod_modules_mapping.yml``: Define mappings between the names specified in the tool configuration (```` + tags) and the Lmod system. + +Some configuration files are used to control the way that Galaxy resolves tool dependencies. Most Galaxy tools are only +descriptions of how to run a particular command line tool, and they do not contain the dependent command line tool. The +task of locating and making available these command line tools is performed by the Galaxy tool dependencies system, +which has configuration files of its own: + +Additional configuration files and their purposes are: + +- ``auth_conf.xml``: Configures the pluggable authentication service. By default, Galaxy users are created and managed + internally. +- ``build_sites.yml``: Controls which display applications are available and their configuration paths +- ``containers_conf.yml``: Configures the beta Galaxy containers interface, currently only used by Galaxy Interactive + Environments, and only neccesary for Docker Swarm support. +- ``dependency_resolvers_conf.xml``: Describes how Galaxy tools (which are typically just descriptions of how to run a + particular command line tool) should locate their dependencies (the command line tool) that are not part of the tool. + See the `Dependency Resolvers documentation ` for more. +- ``error_report.yml``: Controls how user-initiated error reporting (e.g. due to tool failure) is performed. See the + :ref:`Bug Reports documentation ` for more. +- ``job_metrics_conf.xml``: Enables reporting of certain conditions and collection of metrics when jobs run. +- ``job_resource_params_conf.xml``: Describes tool form elements that should be inserted into tool forms that can be + used by users to control runtime parameters such as memory allocations, cluster selection, and so forth. +- ``object_store_conf.xml``: Configures more advanced storage paradigms for Galaxy datasets, including layout across + multiple filesystems, or in object storage systems such as Swift or Amazon S3. +- ``openid_conf.xml``: Controls which OpenID (if enabled) providers should be presented as options to the user on the + login form. +- ``swarm_manager_conf.yml``: Configures the experimental Docker Swarm manager. +- ``tool_destinations.yml``: Configures dynamic tool destinations, which allow for mapping tools to job destinations + based on certain runtime job properties, such as the user submitting it, input sizes, and so forth. +- ``tool_sheds_conf.xml``: Defines the list of Tool Shed servers that should appear in the Galaxy Administration + interface when searching for new tools. +- ``workflow_schedulers_conf.xml``: Similar to the job configuration, controls the scheduling of workflows as jobs. + +.. _standardize and unify configuration formats: https://github.com/galaxyproject/galaxy/issues/... +.. _Installing Tools into Galaxy: https://galaxyproject.org/admin/tools/add-tool-from-toolshed-tutorial/ +.. _Datatypes documentation: https://galaxyproject.org/learn/datatypes/ +.. _Cluster documentation: https://galaxyproject.org/admin/config/performance/cluster/ +.. _Data Preparation documentation: https://galaxyproject.org/admin/data-preparation/ +.. _Data Managers documentation: https://galaxyproject.org/admin/tools/data-managers/ + + +Configuration +---------------------------- + +- Configure ``config/reports.yml`` in the same manner as your main galaxy instance (i.e., same database connection, but different port). This is a uwsgi YAML configuration file and should contain a reports section with app-specific configuration (options described below). + + - The default port for the reports application is ``9001``, and like Galaxy it only binds to localhost by default. + - ``database_connection`` should match the value used in your Galaxy configuration + - ``database_connection`` should point at a Postgres database, experimental support for MySQL is available but sqlite is not supported at all. + +- Run reports in a uWSGI server with ``sh run_reports.sh`` +- Use a web browser and go to the address you configured in ``reports.yml`` (defaults to http://localhost:9001/) + +.. ---------------------------- +.. Configuration Options +.. ---------------------------- +.. +.. .. include:: galaxy_options.rst diff --git a/doc/source/admin/config_logging.rst b/doc/source/admin/config_logging.rst new file mode 100644 index 00000000000..43da5f85f98 --- /dev/null +++ b/doc/source/admin/config_logging.rst @@ -0,0 +1,99 @@ +Logging Configuration +======================================== + +Overview +---------------------------- + +Logging in Galaxy is performed through the Python `logging`_ module. Traditionally, and on Galaxy releases prior to +18.01, customization of the logging could be performed in two ways: + +1. Basic/automatic configuration with control over log level and log destination (standard output or a named log file). +2. More complex configuration using `logging.config.fileConfig`_. + +Starting with release 18.01, it's possible to use the more featureful `logging.config.dictConfig`_ format, with a few +Galaxy-specific extensions described below. + +By default, Galaxy logs all messages to standard output at the ``DEBUG`` logging level. + +.. .. The way in which logging is configured depends on whether you are using Python Paste or uWSGI. See :ref:`Application +.. .. Server ` for help figuring out what you're using. If using uWSGI, it also depends on whether you are +.. .. using an INI or YAML configuration file. + +The way in which logging is configured depends on whether you are using an YAML or INI configuration file. Galaxy +servers that were created starting with Galaxy Release 18.01 or later use a YAML configuration file. Galaxy servers that +were created with 17.09 or older use an INI configuration file. If you upgrade a pre-18.01 server to 18.01 or later but +do not convert your INI config (``galaxy.ini``) to a YAML config (``galaxy.yml``), the INI config will still be used. + +Basic Configuration +---------------------------- + +YAML +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**Setting the log level:** + +```yaml +galaxy: + log_level: LEVEL +``` + +Where ``LEVEL`` is one of the `logging levels`_ documented in the `logging`_ module. + +**Logging to a file:** + +When using ``run.sh --daemon``, the log is written to ``galaxy.log`` in the current directory. To change this: + +INI +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Setting the log level: + + +```ini +[app:main] +log_level = LEVEL +``` + +**Logging to a file:** + +Where ``LEVEL`` is one of the `logging levels`_ documented in the `logging`_ module. + +When using ``run.sh --daemon``, the log is written to ``paster.log`` in the current directory. To change this: + +Advanced Configuration +---------------------------- + +YAML +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +INI +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + + + + + + +On Galaxy servers older than 18.01, or newer servers still running under Paste (see :ref:`Application Server +