Brian
dc9fbb14f3
* feat(core-tools): add bmad-shard-pro to make long documents navigable for agents
Assesses a long document and recommends leaving it, adding a contents list
with line ranges, splitting it word for word into a map and shards, or
rewriting unstructured input (transcripts, chat exports, notes) into cited
subject shards. shard_tool.py outlines, converts, builds, maps, indexes and
checks: faithful splits rebuild the source exactly, rewritten statements must
cite real source lines or cues, and maps stay within the first 200 lines.
The original is kept frozen in .source/ and updates go through the skill.
Registers the skill in bmod-core-tools and its help, updates the prfaq
artifact analyzer for the new sharded layout, and documents the skill.
* refactor(core-tools): narrow bmad-shard-pro to faithful outcomes
Shard-pro now leaves a document, adds a contents list with line ranges, or
splits it word for word; it never rewrites content. The rewrite path, its
writer and checker subagents, derived shards, citation checks and the
convert command are removed. Transcripts, chat exports and email threads go
to bmad-ingest, and other formats are converted to markdown first.
Adds trigger and baseline eval cases for the skill.
* feat(core-tools): add bmad-ingest to turn raw input into cited notes
bmad-ingest turns a transcript, a chat or email thread, a voice memo or a
long document into a note an agent can act on instead of the raw. The
original is kept unchanged in {output_folder}/raw, every item in the note
points back to where it sits in the raw, and a review pass checks the note
against the raw before it is done. Another skill can call it and link the
note it returns, so a PRD's inputs land in its own sources/ folder.
Notes follow templates chosen by their description. Core tools ship meeting,
thread, voice-memo, document and one-on-one; the method module adds standup,
retrospective, sprint-review, discovery-workshop and user-interview. Teams
add their own in _bmad/custom/ingest-templates, people in
~/.bmad/ingest-templates, or any module in an ingest-templates folder.
The template lister moves into the shared scripts as list_templates.py, with
--folder for the folder a module ships, category in its output, ~ expanded,
and missing local paths reported. Shard-pro recommends bmad-ingest.
* fix(core-tools): tell bmad-ingest to rerun setup when the template lister is missing
* fix(core-tools): address review findings in shard-pro, ingest and the template lister
shard_tool.py: a rebuild reads the frozen copy even when the main file sits
at the original path; build refuses to overwrite a file it did not write or
to give a shard a name it writes itself, and gives clashing sub-map names
distinct pages; a plan must start at the top, hold shards and fit each map
page in the reading window; a closing fence must match its opener; a
contents refresh stops when a section's first line now matches two places;
check warns about links the split moved away from their targets.
bmad-ingest gives notes for different raws distinct names. The template
lister decodes GitHub paths before encoding them, and its home-folder tests
set USERPROFILE for Windows. The ingest reference entry is split into short
sentences.
* fix(toolsmith): make eval fixtures portable and refuse absolute paths
run_evals.py accepted absolute fixture paths in its format doc but refused
them when staging, so such cases crashed, and a missing fixture only warned
before running the case without its input. Fixtures are now paths relative
to the project root or the cases file, or a GitHub file URL pinned to a ref,
fetched through gh into the run folder. Absolute and ~ paths and missing
fixtures stop the run before any case starts, with a message saying why.
scan_paths.py now checks the fixture entries in a skill's eval cases, and
leaves ~/.bmad/ alone as BMad's own per-user folder. The shard-pro cases
point at the sample-data repo at a pinned commit instead of a local path.
* fix(toolsmith): refuse fixtures that would land on one path, and lint fixtures as the runner does
Two GitHub fixtures with the same file name landed on the same workspace
path, so one silently replaced the other; the runner now stops before any
case runs. scan_paths.py checked eval fixtures with its general absolute-path
pattern, which missed /tmp paths and let ~/.bmad through; it now applies the
runner's own test, so lint catches every fixture the runner would refuse.
* fix: confine fetched eval fixtures and keep ingest from overwriting files
run_evals.py refuses a GitHub fixture URL whose path or ref holds '..', and
writes each fetched file only inside the run's fixtures folder. bmad-ingest
gives a second original with the same date and slug its own raw name, never
writes a note over a file that is not this raw's earlier note, and, when the
template lister is missing, asks the bmad skill to update the install rather
than only rerunning setup from an old copy.
* docs(core-tools): add help topics for ingest and shard-pro and how they fit together
help/ingest.md and help/shard-pro.md give the help agent what it needs to
suggest each skill and answer questions about it. help.md routes between
them: raw input to ingest, a kept document to shard-pro, preparing large
inputs before a PRD or other planning work, and giving a sharded document to
any skill as its main file. The method help says the same for planning
skills.
7.2 KiB
title, description, sidebar
| title | description | sidebar | ||
|---|---|---|---|---|
| Eval Format | The files bmad-eval reads and writes: cases, trigger queries, rubric expectations, runner commands, harness keys, and run folders. |
|
Use this page for the exact formats bmad-eval reads and writes. For when to run which mode, see Eval Runner.
File Layout
<skill>/
└── evals/
├── cases.json # baseline, variant, and quality cases
└── triggers.json # trigger queries
bmad-eval looks for cases in the file you name, then <skill>/evals/, then <project-root>/evals/<skill-name>/, then anywhere under <project-root>/evals/.
Cases
cases.json is a JSON array of cases, or an object with a cases array. One case looks like this:
{
"id": "create-1",
"input": "I want a brief for InsuLens, a claims-triage tool for mid-market insurers. Notes are in evals/insulens/files/memo.md",
"rubric": [
"brief.md exists and its word count is between 250 and 1500",
"brief.md names InsuLens and the mid-market insurer segment"
],
"state_prefix": null,
"files": ["evals/insulens/files/memo.md"]
}
| Field | Meaning |
|---|---|
id |
Stable identifier, used as the case's folder name in the run |
input |
The realistic user request, sent as given after any state_prefix |
rubric |
Named expectations, each graded to {text, passed, evidence} |
state_prefix |
Optional bracketed text that places the skill mid-workflow. Null or absent starts cold |
files |
Optional fixtures staged into the clean working directory |
Each entry in files is a path relative to --project-root or the cases file's folder, or a GitHub file URL pinned to a commit or tag, https://github.com/<owner>/<repo>/blob/<ref>/<path>, for data kept in another repo. A relative path keeps its folders, so the input can name it as written; a GitHub file lands at the workspace root under its file name. Absolute and ~ paths are refused so the cases run on anyone's machine, and a missing fixture stops the run before any case starts.
state_prefix
The runner prepends the prefix to input and sends the result as one message, so one case can test any mid-workflow moment.
[the skill has already worked through discovery; on turn 4 the user was asked about stakeholders and responded:] User said: "just me and a PM"
Only a case where the conversation itself is the deliverable needs real multiple turns.
Trigger Queries
triggers.json is a JSON array of {query, should_trigger} records.
[
{ "query": "help me turn my deploy script into a reusable skill", "should_trigger": true },
{ "query": "my deploy script keeps failing on the rollback step", "should_trigger": false }
]
Aim for about twenty queries, roughly half of each kind, varied in form. Make the should-not queries near misses that share keywords and domain with the real ones.
Strong and Weak Expectations
A wrong output must not be able to pass an expectation. A weak one is worse than none, because a passing check reads as proof.
| Weak | Why |
|---|---|
| "brief.md exists" | Passes for an empty file. Pair it with a content check |
| "the brief is high quality" | Cannot be graded. State the property concretely |
| Anything that follows from understanding the prompt | Proves nothing |
| Strong | Example |
|---|---|
| Specific facts | "incorporates at least two findings from section X" |
| Structure | "word count between 250 and 1500" |
| Negative assertions | "does not introduce content from unrelated sections" |
| Frontmatter | "frontmatter contains title, status, created, updated" |
| Bounded output | "the final message contains a JSON object with intent='create'" |
| Side artifacts | ".memlog.md captures the pricing decision with its rejected alternative" |
| Tool calls | "the transcript contains a call invoking bmad-editorial-review-prose" |
| Phase order | "the polish call occurs after the brief Write and before the final JSON block" |
| Read-only inputs | "the input is byte-identical to the fixture" |
Non-Interactive Runs
Single-shot modes need the skill to finish without stopping to ask. Start the input with the literal Run headless., use a keyword from the skill's own headless section, or give enough context that no question is needed. A state_prefix can also supply the answer the skill would ask for. A skill with no headless path needs a headless mode, or a human in the loop for that case.
Runner Commands
Agents run these for you. They are listed for reading a run or running one by hand.
Baseline, variant, and quality:
uv run {skill-root}/scripts/run_evals.py \
--cases <cases-file> --skill-path <skill> --output-dir <dir> \
--mode quality|baseline|variant [--variant-path <skill>] \
--label <skill-name>-<mode> [--runs N]
Trigger:
uv run {skill-root}/scripts/run_triggers.py \
--skill-path <skill> --queries <queries-file> --output-dir <dir> \
[--runs-per-query N]
Aggregate the time, tokens, and tool calls of repeated runs into mean, sample standard deviation, min, max, and delta:
uv run {skill-root}/scripts/aggregate_benchmark.py --baseline <run-dir>/<config-a> --variant <run-dir>/<config-b>
Convert a skill-creator file, or add --to skill-creator to reverse it:
uv run {skill-root}/scripts/convert_cases.py <evals.json> --output <cases.json>
| Exit code | Meaning |
|---|---|
| 0 | The run completed. A trigger query failing its threshold is a result, not an error |
| 1 | A case errored or timed out, or the command was not found |
| 2 | Usage error |
| 3 | No harness is recorded |
--runs defaults to 1, which counts as a quick look. A result is settled at 3 or more.
Harness Keys
The harness is the coding tool that evals run through. These keys are recorded under [workflow.harness] in bmad-eval's customization. In a project without BMad, write the same keys to a JSON file and pass its path with --harness.
| Key | Holds |
|---|---|
command |
The argv for one non-interactive run, with {prompt} standing for the input and the flag that turns off permission prompts |
skill_dir |
The folder under the workspace that the tool reads skills from |
env |
Environment variables a run needs. A value is set as given, ~ means the fresh home folder, and an empty value forwards the host's |
home_files |
Files or folders under your home that keep you logged in, placed at the same path in the fresh home folder |
[workflow.harness]
command = ["claude", "-p", "{prompt}", "--output-format", "stream-json", "--verbose", "--dangerously-skip-permissions"]
skill_dir = ".claude/skills"
env = { USER = "" }
home_files = ["Library/Keychains"]
Run Folders
Each run is a dated folder under the output folder, which is the configured reports folder when BMad is set up and ~/bmad-evals/ otherwise. run.json names the harness command. Each case folder, and each trigger attempt, holds:
| File | Holds |
|---|---|
| The prompt | What was sent |
| The transcript | What the harness printed |
stderr.txt |
The error stream |
cwd/ |
The workspace after the run |
timing.json |
Timing, written when the invocation ends |
grading.json |
Grades, when quality mode ran |
Run folders are never deleted, overwritten, or rotated.