diff --git a/doc/source/admin/cluster.md b/doc/source/admin/cluster.md index b1db9437b54..b737bf64345 100644 --- a/doc/source/admin/cluster.md +++ b/doc/source/admin/cluster.md @@ -146,31 +146,15 @@ Runs jobs via the [TORQUE Resource Manager](http://www.adaptivecomputing.com/pro #### Dependencies -Galaxy uses the [pbs_python](https://oss.trac.surfsara.nl/pbs_python/) module to interface with TORQUE. pbs_python must be compiled against your TORQUE installation, so it cannot be provided with Galaxy. However, we provide all the necessary automation to compile it - ([more about Galaxy's Framework dependencies](framework_dependencies.html)): +Galaxy uses the [pbs_python](https://github.com/ehiggs/pbs-python) module to interface with TORQUE. pbs_python must be compiled against your TORQUE installation, so it cannot be provided with Galaxy. You can install the package as follows: ```console -galaxy_user@galaxy_server% cd /clusterfs/galaxy/galaxy-app -galaxy_user@galaxy_server% LIBTORQUE_DIR=/path/to/libtorque python scripts/scramble.py -e pbs_python +galaxy_user@galaxy_server% git clone https://github.com/ehiggs/pbs-python +galaxy_user@galaxy_server% cd pbs-python +galaxy_user@galaxy_server% source /clusterfs/galaxy/galaxy-app/.venv/bin/activate +galaxy_user@galaxy_server% python setup.py install ``` - -#### Newer versions of TORQUE (>4.2) - -Galaxy is compatible with newer versions of TORQUE now that pbs_python has been updated to add support for TORQUE >= v4.2. Scrambling the new version of pbs_python is accomplished simply by modifying eggs.ini to use the new version of pbs_python: - -```ini -;pbs_python = 4.3.5 -pbs_python = 4.4.0 -``` - - -Then scramble as normal: -```console -galaxy_user@galaxy_server% cd /clusterfs/galaxy/galaxy-app -galaxy_user@galaxy_server% LIBTORQUE_DIR=/path/to/libtorque python scripts/scramble.py -e pbs_python -``` - - As of May 2014 there are still some outstanding bugs in pbs_python. Notably, error code translation is out of alignment. For example if you get error #15025 "Bad UID for Job Execution" pbs_python will report this error incorrectly as "Queue already exists". You may consult the [TORQUE source code](https://github.com/adaptivecomputing/torque/blob/4.2.7/src/include/pbs_error_db.h) for the proper error message that corresponds with a given error number. diff --git a/lib/galaxy/tools/xsd/galaxy.xsd b/lib/galaxy/tools/xsd/galaxy.xsd index 2ccde6d64e9..5fa511457b6 100644 --- a/lib/galaxy/tools/xsd/galaxy.xsd +++ b/lib/galaxy/tools/xsd/galaxy.xsd @@ -4849,7 +4849,7 @@ checked.]]> tag defines a range of exit codes, and each range can be associated with a description of the error (e.g., "Out of Memory", "Invalid Sequence File") and an error level. The description just describes the condition and can be anything. The error level is either a warning or a fatal error. A warning means that stderr will be updated with the error's description. A fatal error means that the tool's execution will be marked as having an error and the workflow will stop. Note that, if the error level is not supplied, then a fatal error is assumed to have occurred. +Tools may use exit codes to indicate specific execution errors. Many programs use 0 to indicate success and non-zero exit codes to indicate errors. Galaxy allows each tool to specify exit codes that indicate errors. Each tag defines a range of exit codes, and each range can be associated with a description of the error (e.g., "Out of Memory", "Invalid Sequence File") and an error level. The description just describes the condition and can be anything. The error level is either log, warning, fatal error, or fatal_oom. A warning means that stderr will be updated with the error's description. A fatal error means that the tool's execution will be marked as having an error and the workflow will stop. A fatal_oom indicates an out of memory condition and the job might be resubmitted if Galaxy is configured appropriately. Note that, if the error level is not supplied, then a fatal error is assumed to have occurred. The exit code's range can be any consecutive group of integers. More advanced ranges, such as noncontiguous ranges, are currently not supported. Ranges can be specified in the form "m:n", where m is the start integer and n is the end integer. If ":n" is specified, then the exit code will be compared against all integers less than or equal to n. If "m:" is used, then the exit code will be compared against all integers greater than or equal to m. If the exit code matches, then the error level is applied and the error's description is added to stderr. If a tool's exit code does not match any of the supplied tags' ranges, then no errors are applied to the tool's execution. @@ -4862,9 +4862,10 @@ The tag's supported attributes are as follows: * ``[m:n]``: the exit code must be greater than or equal to m and less than or equal to n; * ``[m:]``: the exit code must be greater than or equal to m; * ``[:n]``: the exit code must be less than or equal to n. -* ``level``: This indicates the error level of the exit code. The level can have one of two values: - * ``warning``: If an exit code falls in the given range, then a description of the error will be added to the beginning of stderr. A warning-level error will not cause the tool to fail. - * ``fatal``: If an exit code falls in the given range, then a description of the error will be added to the beginning of stderr. A fatal-level error will cause the tool to fail. If no level is specified, then the fatal error level will be assumed to have occurred. +* ``level``: This indicates the error level of the exit code. If no level is specified, then the fatal error level will be assumed to have occurred. The level can have one of following values: + * ``log`` and ``warning``: If an exit code falls in the given range, then a description of the error will be added to the beginning of the source, prepended with either 'Log:' or 'Warning:'. A log-level/warning-level error will not cause the tool to fail. + * ``fatal``: If an exit code falls in the given range, then a description of the error will be added to the beginning of stderr. A fatal-level error will cause the tool to fail. + * ``fatal_oom``: If an exit code falls in the given range, then a description of the error will be added to the beginning of stderr. Depending on the job configuration, a fatal_oom-level error will cause the tool to be resubmitted or fail. * ``description``: This is an optional description of the error that corresponds to the exit code. The following is an example of the tag: @@ -4872,10 +4873,10 @@ The following is an example of the tag: ```xml - + - + ``` @@ -4930,9 +4931,10 @@ A regular expression includes the following attributes: * ``stderr``: the regular expression will be applied to stderr; * ``both``: the regular expression will be applied to both stderr and stdout (which is the default case). * ``match``: This is the regular expression that will be used to match against stdout and/or stderr. If the tag does not contain the match attribute, then the tag will be ignored. The regular expression can be any valid Python regular expression. All regular expressions are performed case insensitively. For example, if match contains the regular expression "actg", then the regular expression will match against "actg", "ACTG", "AcTg", and so on. Also note that, if double quotes (") are to be used in the match attribute, then the value " can be used in place of double quotes. Likewise, if single quotes (') are to be used in the match attribute, then the value ' can be used if necessary. -* ``level``: This works very similarly to the tag, except that, when a regular expression matches against its source, the description is added to the beginning of the source. For example, if stdout matches on a regular expression, then the regular expression's description is added to the beginning of stdout (instead of stderr). The level can be log, warning or fatal as described below. +* ``level``: This works very similarly to the tag, except that, when a regular expression matches against its source, the description is added to the beginning of the source. For example, if stdout matches on a regular expression, then the regular expression's description is added to the beginning of stdout (instead of stderr). If no level is specified, then the fatal error level will be assumed to have occurred. The level can have one of following values: * ``log`` and ``warning``: If the regular expression matches against its source input (i.e., stdout and/or stderr), then a description of the error will be added to the beginning of the source, prepended with either 'Log:' or 'Warning:'. A log-level/warning-level error will not cause the tool to fail. - * ``fatal``: If the regular expression matches against its source input, then a description of the error will be added to the beginning of the source. A fatal-level error will cause the tool to fail. If no level is specified, then the fatal error level will be assumed to have occurred. + * ``fatal``: If the regular expression matches against its source input, then a description of the error will be added to the beginning of the source. A fatal-level error will cause the tool to fail. + * ``fatal_oom``: In contrast to fatal the job might be resubmitted if possible according to the job configuration. * ``description``: Just like its ``exit_code`` counterpart, this is an optional description of the regular expression that has matched. The following is an example of regular expressions that may be used: @@ -4975,10 +4977,14 @@ prepended with the string ``Fatal: Unknown error encountered``. Note that, if stderr contained ``error``, ``ERROR``, or ``ErRor`` then it would not matter - stderr was not being scanned. -If the second regular expression did not match, then the third regular -expression is checked. The third regular expression does not contain an error -level, so an error level of ``fatal`` is assumed. The third regular expression -also does not contain a source, so both stdout and stderr are checked. The third +If the second regular expression does not match, the regular expression "out of memory" +is checked on stdout. If found, Galaxy tries to resubmit the job with more memory +if configured correctly, otherwise the job fails. + +If the previous regular expressions does not match, then the fourth regular +expression is checked. The fourth regular expression does not contain an error +level, so an error level of ``fatal`` is assumed. The fourth regular expression +also does not contain a source, so both stdout and stderr are checked. The fourth regular expression looks for 12 consecutive "C"s or "G"s in any order and in uppercase or lowercase. If stdout contained ``cgccGGCCcGGcG`` or stderr contained ``CCCCCCgggGGG``, then the regular expression would match, the tool @@ -4986,7 +4992,7 @@ would be marked with a fatal error, and the stream that contained the 12-nucleotide CG island would be prepended with ``Fatal: Fatal error - CG island 12 nts long found``. -Finally, if the tool did not match any of the fatal errors, then the fourth +Finally, if the tool did not match any of the fatal errors, then the fifth regular expression is checked. Since no source is specified, both stdout and stderr are checked. If ``Branch A`` is at the beginning of stdout or stderr, then a warning will be registered and the source that contained ``Branch A`` will be @@ -5340,6 +5346,7 @@ and ``bibtex`` are the only supported options. Documentation for LevelType + diff --git a/scripts/common_startup.sh b/scripts/common_startup.sh index 2ca25478acc..c7321a8e854 100755 --- a/scripts/common_startup.sh +++ b/scripts/common_startup.sh @@ -246,4 +246,7 @@ if [ $SKIP_CLIENT_BUILD -eq 0 ]; then echo "ERROR: Galaxy client build failed. See ./client/README.md for more information, including how to get help." exit 1 fi +else + echo "Regenerating static plugin directories." + python ./scripts/plugin_staging.py fi diff --git a/scripts/get_uwsgi_args.py b/scripts/get_uwsgi_args.py index 03f7ab3d9eb..a915741e41f 100644 --- a/scripts/get_uwsgi_args.py +++ b/scripts/get_uwsgi_args.py @@ -23,7 +23,7 @@ ALIASES = { 'module': ('mount',), # mount is not actually an alias for module, but we don't want to set module if mount is set } DEFAULT_ARGS = { - '_all_': ('pythonpath', 'threads', 'buffer-size', 'http', 'static-map', 'die-on-term', 'hook-master-start', 'enable-threads'), + '_all_': ('pythonpath', 'threads', 'buffer-size', 'http', 'static-map', 'static-safe', 'die-on-term', 'hook-master-start', 'enable-threads'), 'galaxy': ('py-call-osafterfork',), 'reports': (), 'tool_shed': (), @@ -92,6 +92,8 @@ def _get_uwsgi_args(cliargs, kwargs): 'http': 'localhost:{port}'.format(port=DEFAULT_PORTS[cliargs.app]), 'static-map': ('/static/style={here}/static/style/blue'.format(here=os.getcwd()), '/static={here}/static'.format(here=os.getcwd())), + 'static-safe': ('{here}/config/plugins/visualizations'.format(here=os.getcwd()), + '{here}/config/plugins/interactive_environments'.format(here=os.getcwd())), 'die-on-term': True, 'enable-threads': True, 'hook-master-start': ('unix_signal:2 gracefully_kill_them_all', diff --git a/scripts/plugin_staging.py b/scripts/plugin_staging.py new file mode 100644 index 00000000000..1c21cc13614 --- /dev/null +++ b/scripts/plugin_staging.py @@ -0,0 +1,28 @@ +import glob +import os +import shutil + +GXY_ROOT = os.path.abspath(os.path.join(os.path.dirname(os.path.realpath(__file__)), os.pardir)) + + +def link_up_static(f): + src = os.path.join(GXY_ROOT, 'config/plugins', f) + dest = os.path.join(GXY_ROOT, 'static/plugins', f) + dest_parent = os.path.abspath(os.path.join(dest, os.pardir)) + if os.path.exists(dest): + # We have to clear out the old staged or linked static to relink. + if os.path.islink(dest): + os.unlink(dest) + else: + shutil.rmtree(dest) + elif not os.path.exists(dest_parent): + # Create parent dir structure to symlink directly to static. + os.makedirs(dest_parent) + os.symlink(src, dest) + + +if __name__ == "__main__": + # This is not awesome, but it's temporary, and it supports two-tier plugin static. + for f in glob.glob(os.path.join(GXY_ROOT, 'config/plugins/*/*/static')) + glob.glob(os.path.join(GXY_ROOT, 'config/plugins/*/*/*/static')): + f = os.path.relpath(f, os.path.join(GXY_ROOT, 'config/plugins')) + link_up_static(f)