Improvements to workflow scaling performance testing scripts.

- Implement a wrapper script for launching a docker-galaxy-stable setup a many ways for this task including:
    - Configure container's Galaxy with outer Galaxy's lib and test directory.
    - Collect log files.
    - Log a statement for interacting with the container easily while the test is running.
    - Disable unneeded services.
 - Implement a fan-in-fan-out style workflow for performance testing.
 - Add an option to test performance of scheduling workflows without actually running jobs.
 - Sample tool for creating large collections easily at beginning of tests (much faster than uploads).
 - Sample tool for splitting a file into a collection for fan in and out workflow test.
 - Assume Python 2.7 in scripts to simplify things.
 - Add documentation.
This commit is contained in:
John Chilton
2016-03-21 01:39:45 +00:00
parent d0ceb29bd8
commit 85ef86ab2e
8 changed files with 266 additions and 30 deletions
+7 -9
View File
@@ -1,9 +1,7 @@
"""Script to parse timings out of a Galaxy log and summarize."""
from __future__ import print_function
try:
from argparse import ArgumentParser
except ImportError:
ArgumentParser = None
from argparse import ArgumentParser
import re
import numpy
@@ -15,19 +13,19 @@ TIMING_LINE_PATTERN = re.compile("\((\d+.\d+) ms\)")
def main(argv=None):
if ArgumentParser is None:
raise Exception("Test requires Python 2.7")
"""Entry point for script."""
arg_parser = ArgumentParser(description=DESCRIPTION)
arg_parser.add_argument("--file", default="paster.log")
arg_parser.add_argument("--print_lines", default=False, action="store_true")
arg_parser.add_argument("--pattern")
arg_parser.add_argument("--pattern", default=None)
args = arg_parser.parse_args(argv)
print_lines = args.print_lines
filter_pattern = re.compile(args.pattern)
pattern_str = args.pattern
filter_pattern = re.compile(pattern_str) if pattern_str is not None else None
times = []
for line in open(args.file, "r"):
if not filter_pattern.search(line):
if filter_pattern and not filter_pattern.search(line):
continue
match = TIMING_LINE_PATTERN.search(line)