opensource/galaxy
3
0
Fork 0
mirror of https://github.com/galaxyproject/galaxy.git synced 2026-09-01 15:37:32 +08:00
Code Issues Packages Projects Releases Wiki Activity

Strip the provider prefix when building the eval judge model

Browse Source 
--judge-model values are provider-qualified (e.g. `openai:gpt-5`), but the OpenAI API
expects the bare model id. build_judge_model passed the qualified name straight to
OpenAIChatModel, so any judge model failed. Strip the `provider:` prefix the same way the
agents under test already do.
This commit is contained in:
mvdbeek
2026-06-20 16:28:58 +02:00
parent 8829aa8cc2
commit 806d7c22ec
1 changed files with 4 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/evals/judge.py
+4 -1
View File
@@ -14,5 +14,8 @@ def build_judge_model(
api_key: str,
) -> OpenAIChatModel:
"""Build an OpenAI-compatible pydantic-ai model for use as an LLMJudge."""
# Strip the `provider:` prefix (e.g. `openai:gpt-5-mini`) the same way the
# agents under test do; the bare model name is what the API expects.
bare_name = model_name.split(":", 1)[1] if ":" in model_name else model_name
provider = OpenAIProvider(base_url=base_url, api_key=api_key)
return OpenAIChatModel(model_name, provider=provider)
return OpenAIChatModel(bare_name, provider=provider)