John flagged in PR review that big connected Python packages should live
under lib/ rather than at the repo root. test/ is the lighter-weight
neighbor and slots in next to test/unit/, test/integration/,
test/functional/ -- and the live runner already lives at
test/integration/test_live_evals.py.
The package isn't installable (test/ has no __init__.py by design), so
the CLI invocation changes to `cd test && python -m evals.run_evals`,
and test_live_evals.py puts test/ on sys.path explicitly.
scripts/seed_live26_demo_history.py stays at the repo root for now and
moves in a follow-up.
Caught in self-review -- the eval-results gitignore block still
pointed at ~/work/brain/plans/galaxy-agent-evals-baselines/ as the
home for baselines. Same scrub as the README pass: replace with
generic "keep baselines you care about somewhere outside the repo"
guidance.
Eval result artifacts -- particularly the JSON sidecars with full per-case
prompts and responses -- balloon to multi-thousand-line files and don't
diff usefully. The most recent baseline alone added 4797 lines. Keeping
them committed was making the repo noisy without much payoff: the harness
already encodes the repo short-SHA into every result filename, so we
don't need git history to reconstruct which commit a baseline came from.
Adds evals/results/* to .gitignore (preserving .gitkeep so the directory
still exists for local writes) and untracks the existing committed
baselines. Updates evals/README.md to point at the new home:
~/work/brain/plans/galaxy-agent-evals-baselines/
That directory has its own README.md indexing every past run with its
date, datasets, models, repo SHA, and a one-line note. The committed
baselines have been copied there so nothing is lost; --baseline still
works the same, just point it at the brain-vault path.
Scaffolds evals/ as a top-level package built on pydantic-evals so we can
actually measure agent behaviour against real LLMs and weigh models against
each other. First dataset is routing -- (query, expected handoff target)
pairs lifted from the standalone test_agents_live.py script and the existing
TEST_QUERIES fixture. CLI runs the dataset across one or more models through
the LiteLLM proxy (with --model-config for models on different backends like
TACC SambaNova) and emits a markdown comparison table to stdout and to
evals/results/. Not run in CI -- real LLMs are slow, flaky, and cost money.
The first checked-in result file is gpt-oss-120b at 6/10, which already
surfaces two real findings: the model can't satisfy the error_analysis
handoff schema, and tool_discovery_rnaseq probably wants to route to
tool_recommendation rather than a direct router answer.
Context managers for all SQLite connections in build_database.py to prevent
leaks on exceptions. Narrowed bare except Exception to specific types.
Switched FTS5 tables to content=/content_rowid= so rowid alignment is
guaranteed by SQLite rather than assumed. Added re.escape for regex safety
in extract_section, deduplicated version into DB_VERSION constant.
GTNSearchDB now downloads the database from a configurable URL when the
local file is missing, so the 25MB .db no longer needs to live in git.
Removed it from tracking and added .gitignore entry.
Consolidated dead FileNotFoundError/OSError branches in gtn_training.py,
removed redundant safety checks, replaced IMPORTANT directive docstrings
with concise descriptions throughout.
Useful when creating virtual environments for several Python versions
(e.g. `.venv38`, `.venv312`), which otherwise are linted by tools like
black, isort and flake8.
Store webpack build hash in meta.json file, and use that to setup
the cache busting.
This should prevent clients from unnecessarily reloading client bundles
when the server restarts and/or when individual workers finish booting
at staggered intervals.
This became more important with usegalaxy.org not forking off workers
from a preloaded master (so all workers would have the same starttime).
1) Download conformance tests to `test/functional/tools/cwl_tools` where
they can be used also by framework and unit tests in the future.
2) Add populators to enable executing CWL conformance tests
3) Generate CWL conformance API tests only when needed for tests
make runnable from pr. Still need to debug
divs out
tree
broken into components
cleaning
formatting, file names, jest tests
jest
removed image files, will be retrieved via npm or separate repo
update
plugin via webpack
webpack plugin
change to plugin to prevent repeated image downloading
update
add actual page accessible from help and registration
Full draft
jest fix, next to make package customizeable
yarn lock
fix
update
Update
Refactor from here for the confiuration stuff -- this displays the bundle
drop plugin_build_hash -- these should never be committed
Update plugin gitignore
Oops, left in a debug
The new Makefile-based command ``make serve-selenium-notebooks`` will serve notebooks in lib/galaxy_test/selenium/jupyter.
The notebooks will target a Galaxy host running on port 8080 - but are parameterized to be run with papermill (https://papermill.readthedocs.io/) for automate targeting different Galaxy servers, remote Selenium servers, etc..
Screenshots will be embedded right into the notebook.
Example notebooks setup to target just the library code in ``galaxy-selenium`` as well as one appropriate to leverage the Galaxy testing framework are included and documented in depth via modules docstring for ``galaxy_test.selenium.jupyter``.
Dannon Baker