mirror of
https://github.com/galaxyproject/galaxy.git
synced 2026-10-28 02:58:18 +08:00
172 lines
5.0 KiB
ReStructuredText
172 lines
5.0 KiB
ReStructuredText
.. _job_metrics:
|
|
|
|
|
|
Collecting Job Metrics
|
|
======================
|
|
|
|
Galaxy can collect various metrics about jobs that it runs. The metrics that can be collected depend on which plugins
|
|
(described in this section) are enabled. Two ``galaxy.yml`` configuration options control the job metrics plugin
|
|
configuration:
|
|
|
|
1. ``job_metrics``: Inline global configuration of job metrics plugins
|
|
2. ``job_metrics_config_file``: Path to a standalone metrics configuration file. Prior to Galaxy 23.2, this was the only
|
|
way to configure job metrics plugins. It defaults to ``<config_dir>/job_metrics_conf.xml`` for legacy reasons, but
|
|
using the XML syntax is discouraged, YAML (the syntax is the same as ``job_metrics``) is preferred.
|
|
|
|
If the ``job_metrics_config_file`` exists, it overrides anything configured in ``job_metrics``.
|
|
|
|
Default Job Metrics Configuration
|
|
---------------------------------
|
|
|
|
If no configuration is specified, the default is to load only the ``core`` plugin:
|
|
|
|
.. code-block:: yaml
|
|
|
|
- type: core
|
|
|
|
Available Job Metrics Plugins
|
|
-----------------------------
|
|
|
|
The list of metrics plugins implemented in the code can be found at ``lib/galaxy/job_metrics/instrumenters``.
|
|
|
|
|
|
core
|
|
~~~~
|
|
|
|
The core plugin captures the number of cores allocated to the job (``$GALAXY_SLOTS``), the start and end time of job (in
|
|
seconds since epoch) and computes the runtime in seconds.
|
|
|
|
It has no options.
|
|
|
|
.. code-block:: yaml
|
|
|
|
- type: core
|
|
|
|
cpuinfo
|
|
~~~~~~~
|
|
|
|
The cpuinfo plugin captures the processor count on the system that that job ran on (note that this may differ from the
|
|
number of CPUs actually allocated to the job).
|
|
|
|
The optional ``verbose`` option (default: ``false``) captures details (likely far too much) about each CPU, as found in
|
|
``/proc/cpuinfo``.
|
|
|
|
The cpuinfo plugin works on Linux only.
|
|
|
|
.. code-block:: yaml
|
|
|
|
- type: cpuinfo
|
|
verbose: false
|
|
|
|
meminfo
|
|
~~~~~~~
|
|
|
|
The meminfo plugin captures the memory information on the system that the job ran on (note that this may differ from the
|
|
amount of memory actually allocated to the job).
|
|
|
|
It has no options.
|
|
|
|
.. code-block:: yaml
|
|
|
|
- type: meminfo
|
|
|
|
hostname
|
|
~~~~~~~~
|
|
|
|
The hostname plugin captures the output of ``hostname`` on the system that the job ran on.
|
|
|
|
It has no options.
|
|
|
|
.. code-block:: yaml
|
|
|
|
- type: hostname
|
|
|
|
uname
|
|
~~~~~
|
|
|
|
The uname plugin captures the output of ``uname -a`` on the system that the job ran on.
|
|
|
|
It has no options.
|
|
|
|
.. code-block:: yaml
|
|
|
|
- type: uname
|
|
|
|
env
|
|
~~~
|
|
|
|
The env plugin captures environment variables set in the job's executing environment.
|
|
|
|
By default, it captures **all** environment variables, which is likely excessive but may be useful for debugging. The
|
|
optional ``variables`` option can be set to a list of variables to capture (if set). For legacy purposes, this can also
|
|
be a comma-separated string of variable names.
|
|
|
|
.. code-block:: yaml
|
|
|
|
- type: env
|
|
variables:
|
|
- HOSTNAME
|
|
- SLURM_CPUS_ON_NODE
|
|
- SLURM_JOBID
|
|
|
|
cgroup
|
|
~~~~~~
|
|
|
|
The cgroup plugin captures values set by `Linux Control Groups (cgroups)
|
|
<https://docs.kernel.org/admin-guide/cgroup-v2.html>`_. This is most useful if your jobs run in unique per-job Cgroups
|
|
(as Slurm does `if so configured <https://slurm.schedmd.com/cgroups.html>`_).
|
|
|
|
Both cgroups version 1 (cgroupsv1) and cgroups version 2 (cgroupsv2) are supported, by default metrics will be collected
|
|
for whichever version is mounted on the system where the job ran. The optional ``version`` option (default: ``auto``)
|
|
can be used to only generate metrics capture commands in the job script for the specified cgroups version (``1`` or
|
|
``2``).
|
|
|
|
By default, only a small set of cgroup parameters will be recorded, the list of which can be found in
|
|
``lib/galaxy/job_metrics/instrumenters/cgroup.py`` in the Galaxy code. The optional ``verbose`` option (default:
|
|
``false``) can be set to capture all parameters in the ``cpu``, ``cpuacct``, and ``memory`` controllers (cgroups version
|
|
1) or ``cpu`` and ``memory`` controllers (cgroups version 2).
|
|
|
|
It is also possible to specify exactly which cgroup parameters to capture by setting the optional ``params`` option to a
|
|
list of parameter names (files in the controller directory) to capture. For legacy purposes, this can also be a
|
|
comma-separated string of cgroup parameter names.
|
|
|
|
The cgroup plugin works on Linux only.
|
|
|
|
.. code-block:: yaml
|
|
|
|
- type: cgroup
|
|
verbose: false
|
|
version: 2
|
|
params:
|
|
- cpu.stat
|
|
- memory.peak
|
|
|
|
Overriding the Global Job Metrics Configuration
|
|
-----------------------------------------------
|
|
|
|
Individual Galaxy job config environments (destinations) can disable metric collection by setting the ``metrics`` parameter on that environment:
|
|
|
|
|
|
.. code-block:: yaml
|
|
|
|
execution:
|
|
environments:
|
|
example:
|
|
metrics:
|
|
- type: core
|
|
- type: cpuinfo
|
|
- type: meminfo
|
|
|
|
Alternatively, a file can be specified:
|
|
|
|
.. code-block:: yaml
|
|
|
|
execution:
|
|
environments:
|
|
example:
|
|
metrics:
|
|
src: path
|
|
path: /srv/galaxy/config/metrics_override.yml
|
|
|
|
Additional accepted values for ``src`` include ``default`` and ``disabled``.
|