Files
BMAD-METHOD/tools/validate-skills.js
T
Alex Verkhovsky 49069b8b52 feat(quick-dev): render templates via stdlib Python at skill entry (#2281)
* feat(quick-dev): render templates via stdlib Python at skill entry

Move compile-time variable substitution out of the LLM and into a
deterministic Python step. SKILL.md becomes a two-line stdout-dispatch
shim that runs render.py and follows the instruction it prints. The
renderer reads BMad configuration from the central four-layer TOML
surface introduced in #2285 (_bmad/config.toml plus config.user.toml
and the two _bmad/custom/ overrides), with a fallback to the legacy
per-module _bmad/bmm/config.yaml for pre-#2285 installs.

Compile-time refs ({{.var}}) get substituted at render time. LLM-runtime
refs ({var}) pass through untouched.

Renderer (render.py)
- Python 3 stdlib only (tomllib, already bundled since 3.11). UTF-8 I/O.
  Every invocation rebuilds from scratch — no hash, no cache.
- find_project_root walks up from cwd; HALT to stdout if no _bmad/
  is found anywhere on the path.
- load_central_config deep-merges the four TOML layers in priority
  order (base-team → base-user → custom-team → custom-user) so user
  overrides in _bmad/custom/config.user.toml win over installer-
  regenerated base values. flatten_central_config lifts scalar keys
  from [core] and [modules.bmm] into the renderer's flat namespace;
  module keys beat core on collision (matches the installer's own
  core-key-stripping behavior).
- When _bmad/config.toml is absent, falls through to the legacy
  flat-YAML parser for _bmad/bmm/config.yaml — the renderer keeps
  working across the #2285 transition.
- {{.var}} substitution; unresolved refs emit empty string (Go
  missingkey=zero semantics).
- Smart defaults for planning_artifacts / implementation_artifacts /
  communication_language applied after config load. Derives
  sprint_status / deferred_work_file from implementation_artifacts.
  {{.main_config}} points at whichever surface was actually read.
- Renders every .md in the skill dir except SKILL.md to
  {project-root}/_bmad/render/bmad-quick-dev/.
- On success, stderr summary plus a single stdout line:
  "read and follow {workflow_md}". On failure, stdout HALT directive —
  per the Anthropic skills spec, script stdout is the defined agent-
  communication channel.

Skill entry (SKILL.md)
- Two-line shim: run python render.py, follow stdout. No template
  tokens in SKILL.md itself.

Template conversions
- workflow.md, step-01..05, step-oneshot, sync-sprint-status: convert
  every compile-time {var} reference to {{.var}}. Runtime refs
  preserved.
- spec-template.md untouched (single-curly comment hint stays as
  documentation).

Skill-prose cleanups bundled in
- Remove dead step-file frontmatter: empty-string variable declarations
  (spec_file, story_key, diff_output, review_mode) in quick-dev step-01
  and code-review step-01; empty --- --- blocks in step-03 and step-05;
  the specLoopIteration counter init moved from step-04 frontmatter into
  the step body where first-entry vs loopback semantics are explicit.
- Unify the language rule across all six quick-dev step files plus
  workflow.md.

Tooling
- tools/validate-skills.js: add TPL-01 rule. Files whose name contains
  "template" must not contain compile-time {{.var}} substitutions.
  Template files seed durable, version-controlled artifacts that
  execute on other machines; baking a value at render time would
  freeze a machine-local path into every downstream artifact.
- tools/validate-file-refs.js: add render/ to INSTALL_ONLY_PATHS so
  the validator recognizes the runtime-generated buffer.
- tools/skill-validator.md: document TPL-01; deterministic rule count
  bumped from 14 to 15.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor(quick-dev): drop render.py YAML fallback and smart defaults

Single happy path: central _bmad/config.toml with four-layer merge,
Python 3.11+ required (no ImportError guard), HALT if config missing.
Deletes load_flat_yaml, the YAML fallback branch, the setdefault block
for planning_artifacts/implementation_artifacts/communication_language,
and the tomllib ImportError fallback.

Part of plan-quick-dev-python-config-hardening.md (F0).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(quick-dev): normalize render.py paths to forward slashes

On Windows, os.path.join returns backslash-separated paths that can
misrender as escape sequences when later concatenated into POSIX
shell strings or regexes. Normalize the project root to forward
slashes after find_project_root, and use posixpath.join for every
path that gets baked into rendered .md files or joined into config
values. os.makedirs and os.listdir accept forward-slash paths on
Windows, so their call sites stay as-is.

Part of plan-quick-dev-python-config-hardening.md (F3).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(quick-dev): preserve source line endings in render.py

Python text-mode open() with the platform default performs universal-
newline translation: on Windows, LF source files get written as CRLF,
producing spurious diffs when rendered output is compared against
source. Pass newline="" on both the source read and the rendered
write so line endings pass through verbatim.

Part of plan-quick-dev-python-config-hardening.md (F4).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(quick-dev): delete stale .md renders before rebuilding

render.py rebuilds from scratch per the docstring, but
makedirs(exist_ok=True) only overwrites files that still exist in
the source — stale outputs from renamed/deleted source files linger
in _bmad/render/bmad-quick-dev/ forever. Remove every .md in the
render dir before the render loop; keep the dir itself and any
non-.md files.

Part of plan-quick-dev-python-config-hardening.md (F5).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(quick-dev): scope render/ whitelist to bmad-quick-dev

The previous INSTALL_ONLY_PATHS entry 'render/' was a blanket prefix
that let every {project-root}/_bmad/render/... reference in any skill
slip past validation. Narrow to 'render/bmad-quick-dev/' so only this
skill's render buffer is whitelisted. Future skills adopting the
stdout-dispatch renderer pattern add their own entries explicitly.

Part of plan-quick-dev-python-config-hardening.md (F6).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(quick-dev): add renderer smoke test with TOML override

New test/test-quick-dev-renderer.js spins up a temp project with
base _bmad/config.toml and a _bmad/custom/config.user.toml override,
runs render.py, and asserts the override wins in rendered workflow.md
and that sprint_status is rooted at an absolute path in the temp
project. Registered as test:renderer in package.json and chained
into the npm test script.

Part of plan-quick-dev-python-config-hardening.md (F7).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(quick-dev): HALT cleanly when base config.toml is unparseable

Load the four config layers through a load_toml helper that marks the
base _bmad/config.toml as required. A missing, unparseable, or unreadable
base now prints a HALT directive to stdout and exits, instead of being
silently skipped and then crashing downstream with a KeyError when a
derived value (e.g. implementation_artifacts) is absent. Optional layers
still warn on stderr and fall back to empty. Merge semantics are
unchanged (dict-aware deep merge, override wins for lists and scalars).

* fix(quick-dev): resolve render.py via {skill-root} in skill entry shim

The bare `python render.py` shim assumes the agent's working directory is
the skill directory, but agents run from the project root, so the script
is not found. Reference it as `{skill-root}/render.py` — BMAD's standard
token for a skill's installed directory, already used by every other
skill's resolve_customization.py invocation — and add the one-line
`{skill-root}` explainer so the model resolves it from an instruction
rather than guessing. Interpreter stays `python`; the python vs python3
choice is a separate cross-platform concern.

* refactor(quick-dev): resolve [workflow] customization in render.py

render.py now merges the three customize layers (customize.toml ->
custom/bmad-quick-dev.toml -> .user.toml) with the same structural rules as
resolve_customization.py and inlines the resolved [workflow] values, so no
{workflow.*} placeholder survives. workflow.md drops its Step 1 runtime
resolver + manual-merge fallback; step-05 and step-oneshot drop their runtime
workflow.on_complete calls. The shared resolve_customization.py and every
other skill are untouched. Smoke test extended with a [workflow] override
fixture covering inlining, array append, and no-leak assertions.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(quick-dev): harden render.py invocation in the SKILL.md shim

The shim called bare `python`, which can resolve to Python 2 or be absent;
render.py needs 3.11+ for tomllib. Spell out python3 and the version
requirement. Also make the exit code authoritative: on a non-zero exit
(including an uncaught crash that writes only to stderr), do not proceed --
report what was printed and stop.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(quick-dev): drop the render.py success stderr line

The "rendered N files" progress line was pure diagnostic noise. The shim
already tells the LLM to ignore stderr and follow the stdout instruction, so
on success render.py now prints only the "read and follow ..." line.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(quick-dev): drop the activation gate sentence from the rendered workflow

The gate ported from #2398 defended against runtime customization
indirection: agents guessed resolver outputs instead of executing them,
silently skipping append steps. render.py inlines the prepend/append
entries into the rendered workflow.md, so there is nothing left to
short-circuit, and each inlined list already carries its own execute-
in-order imperative. In the default install both lists render as
_None._ and the gate is pure noise.

* feat(quick-dev): materialize review layers into invocation blocks

Reconcile #2550 with render-time [workflow] resolution. Main made review
layers configurable as [[workflow.review_layers]] arrays of tables and
had the LLM resolve them during activation; this branch resolves the
[workflow] block in render.py instead, so activation-time resolution no
longer exists and the layer refs must be materialized at render time.

Rather than inlining the layer tables as data plus interpretation rules,
render.py now knows this skill's customization schema outright and
renders review_layers/oneshot_review_layers as direct invocation blocks:
disabled layers (empty instruction) drop out, each active layer becomes
a #### section holding its instruction verbatim, zero active layers
renders the HALT instruction, and runtime placeholders like
{diff_output} pass through. The only judgment left to the LLM is the
optional `when` condition, which renders as a run-time guard line.
The step-04/step-oneshot review intros collapse to a single execute-in-
parallel imperative. Smoke test covers default rendering, replace-by-id,
disable-by-empty-instruction, when-guards, and the all-disabled HALT.

* fix(quick-dev): invoke render.py via uv run per house standard

The SKILL.md shim launched render.py with bare `python3`, which the rest
of BMAD is migrating away from: the customize-bmad docs and the
installer's uv-check standardize on `uv run` (uv provisions a suitable
3.11+ interpreter on demand). Bare `python3` is also fragile on Windows,
where python.org installs expose `python`/`py` rather than `python3`.

Make `uv run` the primary invocation and demote `python3` to the
documented fallback, spelling out `python`/`py -3` for Windows and the
3.11+ tomllib requirement. render.py itself is unchanged; the renderer
test drives it directly and is unaffected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(quick-dev): HALT cleanly on missing or malformed config

render.py derived sprint_status/deferred_work_file from an unconditional
vars_["implementation_artifacts"] subscript, so a config lacking that key
raised a raw KeyError instead of the stdout HALT the rest of the script
uses on bad input. flatten_central_config likewise called .get("bmm") on
merged["modules"] without checking it was a table, so a non-table
[modules] crashed with an AttributeError.

Guard both: HALT with a clear stdout directive when implementation_artifacts
is missing or blank, and coerce a non-dict modules to {} before indexing.
Add renderer regression tests asserting each path exits without a Python
traceback.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(quick-dev): resolve config at compile time, drop the runtime re-read

The activation "Load Config" step told the LLM to open {{.main_config}}
and re-resolve project_name, communication_language, sprint_status, etc.
at run time -- but render.py already bakes those from the full four-layer
config merge. main_config pointed at only the base _bmad/config.toml, so
on installs with override layers (config.user.toml / custom/*) the runtime
re-read saw stale values that could contradict the baked {{.var}} in the
same rendered file. It also handed resolution back to the LLM: the drift
this skill's renderer exists to remove.

Delete the ceremony and wire each value where it is actually used:

- Every value the step resolved is already inlined at its point of use
  (planning/implementation_artifacts, sprint_status, communication_language)
  or loaded via persistent_facts (project-context.md), so the central
  block was pure redundancy.
- Fold document_output_language into the per-step language rule, adopting
  the house-canonical form ("Speak in X. Write any file output in Y.")
  already used by bmad-checkpoint-preview.
- Move the {date} = current-datetime definition to step-02, where the
  spec template's {date} field is filled.
- Drop the user greeting (user_name) and user_skill_level tailoring:
  quick-dev is not a conversational skill and neither was load-bearing.
- Remove main_config from render.py; it had no remaining consumer.

Renderer tests repointed at the files that now carry these values, plus
coverage for document_output_language baking and main_config removal.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(quick-dev): reference variables by bare name, not placeholder curlies

Curlies mean "expand this to the value"; a bare backticked name means
"this is the variable/field I'm talking about". Several step files wrapped
a variable in curlies where they were only naming, assigning, passing, or
testing it -- so the notation implied an expansion that never happens:

- step-01: identify `epic_num`/`story_num`, set/leave `story_key` unset
- step-02: test `preserved_intent`; and resolve the template's `date` field
  (was `{date}`, which read as "expand date here" rather than naming it)
- step-03/step-05/step-oneshot: pass `target_status` to sync-sprint-status,
  set `title`
- sync-sprint-status: the `target_status` parameter, `story_key` precondition,
  and both `target_status` conditionals

Value tokens that are genuinely materialized in place -- `{spec_file}` paths,
`development_status[{story_key}]`, "set ... to `{target_status}`" -- stay
curly. Also reword step-02's frozen-block instruction from the ambiguous
"substitute it for the `<frozen-after-approval>` block" to "replace the
`<frozen-after-approval>` block in the spec you just filled out with
`preserved_intent`" so it's clear the replacement happens in the artifact.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(quick-dev): require uv, drop the python3 interpreter fallback

The SKILL.md shim tried `uv run render.py` and, if uv was missing, retried
with a bare `python3`/`py -3` interpreter. Nothing else in the codebase
does that interpreter fallback: the uv-based skills (bmad-prd, bmad-ux,
bmad-architecture, bmad-product-brief) fall back to reading customize.toml
and using defaults -- graceful feature degradation, never a different
runner -- and the legacy skills just call python3 outright. uv is the
established house runner (memlog.py, resolve_customization.py, lint_spine.py
all invoke it).

That graceful-degrade path does not exist here: render.py is the entry
dispatch that produces the workflow.md the LLM then follows, so there is
nothing to fall back to. The only honest outcomes are "uv runs it" or
"HALT". Make uv the floor and drop the fallback.

Pin the interpreter the house way -- a PEP 723 `requires-python = ">=3.11"`
block, matching memlog.py/lint_spine.py -- so `uv run` provisions a 3.11+
interpreter and the tomllib requirement is guaranteed rather than hoped for.
This replaces the prose "needs 3.11+" hedge the shim used to carry.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-07-07 19:31:55 -07:00

736 lines
24 KiB
JavaScript

/**
* Deterministic Skill Validator
*
* Validates 12 deterministic rules across all skill directories.
* Acts as a fast first-pass complement to the inference-based skill validator.
*
* What it checks:
* - SKILL-01: SKILL.md exists
* - SKILL-02: SKILL.md frontmatter has name
* - SKILL-03: SKILL.md frontmatter has description
* - SKILL-04: name format (lowercase, hyphens, no forbidden substrings)
* - SKILL-05: name matches directory basename
* - SKILL-06: description quality (length, "Use when"/"Use if")
* - SKILL-07: SKILL.md has body content after frontmatter
* - PATH-02: no installed_path variable
* - STEP-01: step filename format
* - STEP-06: step frontmatter has no name/description
* - STEP-07: step count 2-10
* - SEQ-02: no time estimates
* - TPL-01: template files must not contain compile-time {{.var}} substitutions
*
* Usage:
* node tools/validate-skills.js # All skills, human-readable
* node tools/validate-skills.js path/to/skill-dir # Single skill
* node tools/validate-skills.js --strict # Exit 1 on HIGH+ findings
* node tools/validate-skills.js --json # JSON output
*/
const fs = require('node:fs');
const path = require('node:path');
const PROJECT_ROOT = path.resolve(__dirname, '..');
const SRC_DIR = path.join(PROJECT_ROOT, 'src');
// --- CLI Parsing ---
const args = process.argv.slice(2);
const STRICT = args.includes('--strict');
const JSON_OUTPUT = args.includes('--json');
const positionalArgs = args.filter((a) => !a.startsWith('--'));
// --- Constants ---
const NAME_REGEX = /^bmad-[a-z0-9]+(-[a-z0-9]+)*$/;
const STEP_FILENAME_REGEX = /^step-\d{2}[a-z]?-[a-z0-9-]+\.md$/;
const TIME_ESTIMATE_PATTERNS = [/takes?\s+\d+\s*min/i, /~\s*\d+\s*min/i, /estimated\s+time/i, /\bETA\b/];
const TEMPLATE_FILENAME_REGEX = /template/i;
const COMPILE_TIME_SUB_REGEX = /\{\{\.\w+\}\}/;
const SEVERITY_ORDER = { CRITICAL: 0, HIGH: 1, MEDIUM: 2, LOW: 3 };
// --- Output Escaping ---
function escapeAnnotation(str) {
return str.replaceAll('%', '%25').replaceAll('\r', '%0D').replaceAll('\n', '%0A');
}
function escapeTableCell(str) {
return String(str).replaceAll('|', String.raw`\|`);
}
// --- Frontmatter Parsing ---
/**
* Parse YAML frontmatter from a markdown file.
* Returns an object with key-value pairs, or null if no frontmatter.
*/
function parseFrontmatter(content) {
const trimmed = content.trimStart();
if (!trimmed.startsWith('---')) return null;
let endIndex = trimmed.indexOf('\n---\n', 3);
if (endIndex === -1) {
// Handle file ending with \n---
if (trimmed.endsWith('\n---')) {
endIndex = trimmed.length - 4;
} else {
return null;
}
}
const fmBlock = trimmed.slice(3, endIndex).trim();
if (fmBlock === '') return {};
const result = {};
for (const line of fmBlock.split('\n')) {
const colonIndex = line.indexOf(':');
if (colonIndex === -1) continue;
// Skip indented lines (nested YAML values)
if (line[0] === ' ' || line[0] === '\t') continue;
const key = line.slice(0, colonIndex).trim();
let value = line.slice(colonIndex + 1).trim();
// Strip surrounding quotes (single or double)
if ((value.startsWith("'") && value.endsWith("'")) || (value.startsWith('"') && value.endsWith('"'))) {
value = value.slice(1, -1);
}
result[key] = value;
}
return result;
}
/**
* Parse YAML frontmatter, handling multiline values (description often spans lines).
* Returns an object with key-value pairs, or null if no frontmatter.
*/
function parseFrontmatterMultiline(content) {
const trimmed = content.trimStart();
if (!trimmed.startsWith('---')) return null;
let endIndex = trimmed.indexOf('\n---\n', 3);
if (endIndex === -1) {
// Handle file ending with \n---
if (trimmed.endsWith('\n---')) {
endIndex = trimmed.length - 4;
} else {
return null;
}
}
const fmBlock = trimmed.slice(3, endIndex).trim();
if (fmBlock === '') return {};
const result = {};
let currentKey = null;
let currentValue = '';
for (const line of fmBlock.split('\n')) {
const colonIndex = line.indexOf(':');
// New key-value pair: must start at column 0 (no leading whitespace) and have a colon
if (colonIndex > 0 && line[0] !== ' ' && line[0] !== '\t') {
// Save previous key
if (currentKey !== null) {
result[currentKey] = stripQuotes(currentValue.trim());
}
currentKey = line.slice(0, colonIndex).trim();
currentValue = line.slice(colonIndex + 1);
} else if (currentKey !== null) {
// Skip YAML comment lines
if (line.trimStart().startsWith('#')) continue;
// Continuation of multiline value
currentValue += '\n' + line;
}
}
// Save last key
if (currentKey !== null) {
result[currentKey] = stripQuotes(currentValue.trim());
}
return result;
}
function stripQuotes(value) {
if ((value.startsWith("'") && value.endsWith("'")) || (value.startsWith('"') && value.endsWith('"'))) {
return value.slice(1, -1);
}
return value;
}
// --- Safe File Reading ---
/**
* Read a file safely, returning null on error.
* Pushes a warning finding if the file cannot be read.
*/
function safeReadFile(filePath, findings, relFile) {
try {
return fs.readFileSync(filePath, 'utf-8');
} catch (error) {
findings.push({
rule: 'READ-ERR',
title: 'File Read Error',
severity: 'MEDIUM',
file: relFile || path.basename(filePath),
detail: `Cannot read file: ${error.message}`,
fix: 'Check file permissions and ensure the file exists.',
});
return null;
}
}
// --- Code Block Stripping ---
function stripCodeBlocks(content) {
return content.replaceAll(/```[\s\S]*?```/g, (m) => m.replaceAll(/[^\n]/g, ''));
}
// --- Skill Discovery ---
function discoverSkillDirs(rootDirs) {
const skillDirs = [];
function walk(dir) {
if (!fs.existsSync(dir)) return;
const entries = fs.readdirSync(dir, { withFileTypes: true });
for (const entry of entries) {
if (!entry.isDirectory()) continue;
if (entry.name === 'node_modules' || entry.name === '.git') continue;
const fullPath = path.join(dir, entry.name);
const skillMd = path.join(fullPath, 'SKILL.md');
if (fs.existsSync(skillMd)) {
skillDirs.push(fullPath);
}
// Keep walking into subdirectories to find nested skills
walk(fullPath);
}
}
for (const rootDir of rootDirs) {
walk(rootDir);
}
return skillDirs.sort();
}
// --- File Collection ---
function collectSkillFiles(skillDir) {
const files = [];
function walk(dir) {
const entries = fs.readdirSync(dir, { withFileTypes: true });
for (const entry of entries) {
if (entry.name === 'node_modules' || entry.name === '.git') continue;
const fullPath = path.join(dir, entry.name);
if (entry.isDirectory()) {
walk(fullPath);
} else if (entry.isFile()) {
files.push(fullPath);
}
}
}
walk(skillDir);
return files;
}
// --- Rule Checks ---
function validateSkill(skillDir) {
const findings = [];
const dirName = path.basename(skillDir);
const skillMdPath = path.join(skillDir, 'SKILL.md');
const workflowMdPath = path.join(skillDir, 'workflow.md');
const stepsDir = path.join(skillDir, 'steps');
// Collect all files in the skill for PATH-02 and SEQ-02
const allFiles = collectSkillFiles(skillDir);
// --- SKILL-01: SKILL.md must exist ---
if (!fs.existsSync(skillMdPath)) {
findings.push({
rule: 'SKILL-01',
title: 'SKILL.md Must Exist',
severity: 'CRITICAL',
file: 'SKILL.md',
detail: 'SKILL.md not found in skill directory.',
fix: 'Create SKILL.md as the skill entrypoint.',
});
// Cannot check SKILL-02 through SKILL-07 without SKILL.md
return findings;
}
const skillContent = safeReadFile(skillMdPath, findings, 'SKILL.md');
if (skillContent === null) return findings;
const skillFm = parseFrontmatterMultiline(skillContent);
// --- SKILL-02: frontmatter has name ---
if (!skillFm || !('name' in skillFm)) {
findings.push({
rule: 'SKILL-02',
title: 'SKILL.md Must Have name in Frontmatter',
severity: 'CRITICAL',
file: 'SKILL.md',
detail: 'Frontmatter is missing the `name` field.',
fix: 'Add `name: <skill-name>` to the frontmatter.',
});
} else if (skillFm.name === '') {
findings.push({
rule: 'SKILL-02',
title: 'SKILL.md Must Have name in Frontmatter',
severity: 'CRITICAL',
file: 'SKILL.md',
detail: 'Frontmatter `name` field is empty.',
fix: 'Set `name` to the skill directory name (kebab-case).',
});
}
// --- SKILL-03: frontmatter has description ---
if (!skillFm || !('description' in skillFm)) {
findings.push({
rule: 'SKILL-03',
title: 'SKILL.md Must Have description in Frontmatter',
severity: 'CRITICAL',
file: 'SKILL.md',
detail: 'Frontmatter is missing the `description` field.',
fix: 'Add `description: <what it does and when to use it>` to the frontmatter.',
});
} else if (skillFm.description === '') {
findings.push({
rule: 'SKILL-03',
title: 'SKILL.md Must Have description in Frontmatter',
severity: 'CRITICAL',
file: 'SKILL.md',
detail: 'Frontmatter `description` field is empty.',
fix: 'Add a description stating what the skill does and when to use it.',
});
}
const name = skillFm && skillFm.name;
const description = skillFm && skillFm.description;
// Deprecated skills are thin compatibility shims that forward to a replacement.
// They intentionally omit a "Use when" trigger so users are steered to the new
// skill instead, so exempt them from the SKILL-06 trigger-phrase requirement.
const isDeprecated = typeof description === 'string' && /^\s*deprecated\b/i.test(description);
// --- SKILL-04: name format ---
if (name && !NAME_REGEX.test(name)) {
findings.push({
rule: 'SKILL-04',
title: 'name Format',
severity: 'HIGH',
file: 'SKILL.md',
detail: `name "${name}" does not match pattern: ${NAME_REGEX}`,
fix: 'Rename to comply with lowercase letters, numbers, and hyphens only (max 64 chars).',
});
}
// --- SKILL-05: name matches directory ---
if (name && name !== dirName) {
findings.push({
rule: 'SKILL-05',
title: 'name Must Match Directory Name',
severity: 'HIGH',
file: 'SKILL.md',
detail: `name "${name}" does not match directory name "${dirName}".`,
fix: `Change name to "${dirName}" or rename the directory.`,
});
}
// --- SKILL-06: description quality ---
if (description) {
if (description.length > 1024) {
findings.push({
rule: 'SKILL-06',
title: 'description Quality',
severity: 'MEDIUM',
file: 'SKILL.md',
detail: `description is ${description.length} characters (max 1024).`,
fix: 'Shorten the description to 1024 characters or less.',
});
}
if (!isDeprecated && !/use\s+when\b/i.test(description) && !/use\s+if\b/i.test(description)) {
findings.push({
rule: 'SKILL-06',
title: 'description Quality',
severity: 'MEDIUM',
file: 'SKILL.md',
detail: 'description does not contain "Use when" or "Use if" trigger phrase.',
fix: 'Append a "Use when..." clause to explain when to invoke this skill.',
});
}
}
// --- SKILL-07: SKILL.md must have body content after frontmatter ---
{
const trimmed = skillContent.trimStart();
let bodyStart = -1;
if (trimmed.startsWith('---')) {
let endIdx = trimmed.indexOf('\n---\n', 3);
if (endIdx !== -1) {
bodyStart = endIdx + 4;
} else if (trimmed.endsWith('\n---')) {
bodyStart = trimmed.length; // no body at all
}
} else {
bodyStart = 0; // no frontmatter, entire file is body
}
const body = bodyStart >= 0 ? trimmed.slice(bodyStart).trim() : '';
if (body === '') {
findings.push({
rule: 'SKILL-07',
title: 'SKILL.md Must Have Body Content',
severity: 'HIGH',
file: 'SKILL.md',
detail: 'SKILL.md has no content after frontmatter. L2 instructions are required.',
fix: 'Add markdown body with skill instructions after the closing ---.',
});
}
}
// --- PATH-02: no installed_path ---
for (const filePath of allFiles) {
// Only check markdown and yaml files
const ext = path.extname(filePath);
if (!['.md', '.yaml', '.yml'].includes(ext)) continue;
const relFile = path.relative(skillDir, filePath);
const content = safeReadFile(filePath, findings, relFile);
if (content === null) continue;
// Check frontmatter for installed_path key
const fm = parseFrontmatter(content);
if (fm && 'installed_path' in fm) {
findings.push({
rule: 'PATH-02',
title: 'No installed_path Variable',
severity: 'HIGH',
file: relFile,
detail: 'Frontmatter contains `installed_path:` key.',
fix: 'Remove `installed_path` from frontmatter. Use relative paths instead.',
});
}
// Check content for any mention of installed_path (variable ref, prose, bare text)
const stripped = stripCodeBlocks(content);
const lines = stripped.split('\n');
for (const [i, line] of lines.entries()) {
if (/installed_path/i.test(line)) {
findings.push({
rule: 'PATH-02',
title: 'No installed_path Variable',
severity: 'HIGH',
file: relFile,
line: i + 1,
detail: '`installed_path` reference found in content.',
fix: 'Remove all installed_path usage. Use relative paths (`./path` or `../path`) instead.',
});
}
}
}
// --- STEP-01: step filename format ---
// --- STEP-06: step frontmatter no name/description ---
// --- STEP-07: step count ---
// Only check the literal steps/ directory (variant directories like steps-c, steps-v
// use different naming conventions and are excluded per the rule specification)
if (fs.existsSync(stepsDir) && fs.statSync(stepsDir).isDirectory()) {
const stepDirName = 'steps';
const stepFiles = fs.readdirSync(stepsDir).filter((f) => f.endsWith('.md'));
// STEP-01: filename format
for (const stepFile of stepFiles) {
if (!STEP_FILENAME_REGEX.test(stepFile)) {
findings.push({
rule: 'STEP-01',
title: 'Step File Naming',
severity: 'MEDIUM',
file: path.join(stepDirName, stepFile),
detail: `Filename "${stepFile}" does not match pattern: ${STEP_FILENAME_REGEX}`,
fix: 'Rename to step-NN-description.md (NN = zero-padded number, optional letter suffix).',
});
}
}
// STEP-06: step frontmatter has no name/description
for (const stepFile of stepFiles) {
const stepPath = path.join(stepsDir, stepFile);
const stepContent = safeReadFile(stepPath, findings, path.join(stepDirName, stepFile));
if (stepContent === null) continue;
const stepFm = parseFrontmatter(stepContent);
if (stepFm) {
if ('name' in stepFm) {
findings.push({
rule: 'STEP-06',
title: 'Step File Frontmatter: No name or description',
severity: 'MEDIUM',
file: path.join(stepDirName, stepFile),
detail: 'Step file frontmatter contains `name:` — this is metadata noise.',
fix: 'Remove `name:` from step file frontmatter.',
});
}
if ('description' in stepFm) {
findings.push({
rule: 'STEP-06',
title: 'Step File Frontmatter: No name or description',
severity: 'MEDIUM',
file: path.join(stepDirName, stepFile),
detail: 'Step file frontmatter contains `description:` — this is metadata noise.',
fix: 'Remove `description:` from step file frontmatter.',
});
}
}
}
// STEP-07: step count 2-10
const stepCount = stepFiles.filter((f) => f.startsWith('step-')).length;
if (stepCount > 0 && (stepCount < 2 || stepCount > 10)) {
const detail =
stepCount < 2
? `Only ${stepCount} step file found — consider inlining into workflow.md.`
: `${stepCount} step files found — more than 10 risks LLM context degradation.`;
findings.push({
rule: 'STEP-07',
title: 'Step Count',
severity: 'LOW',
file: stepDirName + '/',
detail,
fix: stepCount > 10 ? 'Consider consolidating steps.' : 'Consider expanding or inlining.',
});
}
}
// --- SEQ-02: no time estimates ---
for (const filePath of allFiles) {
const ext = path.extname(filePath);
if (!['.md', '.yaml', '.yml'].includes(ext)) continue;
const relFile = path.relative(skillDir, filePath);
const content = safeReadFile(filePath, findings, relFile);
if (content === null) continue;
const stripped = stripCodeBlocks(content);
const lines = stripped.split('\n');
for (const [i, line] of lines.entries()) {
for (const pattern of TIME_ESTIMATE_PATTERNS) {
if (pattern.test(line)) {
findings.push({
rule: 'SEQ-02',
title: 'No Time Estimates',
severity: 'LOW',
file: relFile,
line: i + 1,
detail: `Time estimate pattern found: "${line.trim()}"`,
fix: 'Remove time estimates — AI execution speed varies too much.',
});
break; // Only report once per line
}
}
}
}
// --- TPL-01: template files must not contain compile-time {{.var}} substitutions ---
// Template files seed durable, version-controlled artifacts (spec files) that
// execute on other machines. Baking a {{.var}} at render time would freeze a
// machine-local value into every downstream artifact.
for (const filePath of allFiles) {
if (path.extname(filePath) !== '.md') continue;
const base = path.basename(filePath);
if (!TEMPLATE_FILENAME_REGEX.test(base)) continue;
const relFile = path.relative(skillDir, filePath);
const content = safeReadFile(filePath, findings, relFile);
if (content === null) continue;
const lines = content.split('\n');
for (const [i, line] of lines.entries()) {
const match = line.match(COMPILE_TIME_SUB_REGEX);
if (match) {
findings.push({
rule: 'TPL-01',
title: 'Template files must not contain compile-time substitutions',
severity: 'HIGH',
file: relFile,
line: i + 1,
detail: `Template file contains compile-time substitution \`${match[0]}\` — this would be baked at render time and leak a machine-local value into every spec produced from the template.`,
fix: 'Remove the `{{.var}}` reference. Use single-curly `{var}` if the value should be resolved at LLM runtime by the consumer of the generated spec.',
});
}
}
}
return findings;
}
// --- Output Formatting ---
function formatHumanReadable(results) {
const output = [];
let totalFindings = 0;
const severityCounts = { CRITICAL: 0, HIGH: 0, MEDIUM: 0, LOW: 0 };
output.push(
`\nValidating skills in: ${SRC_DIR}`,
`Mode: ${STRICT ? 'STRICT (exit 1 on HIGH+)' : 'WARNING (exit 0)'}${JSON_OUTPUT ? ' + JSON' : ''}\n`,
);
let totalSkills = 0;
let skillsWithFindings = 0;
for (const { skillDir, findings } of results) {
totalSkills++;
const relDir = path.relative(PROJECT_ROOT, skillDir);
if (findings.length > 0) {
skillsWithFindings++;
output.push(`\n${relDir}`);
for (const f of findings) {
totalFindings++;
severityCounts[f.severity]++;
const location = f.line ? ` (line ${f.line})` : '';
output.push(` [${f.severity}] ${f.rule}${f.title}`, ` File: ${f.file}${location}`, ` ${f.detail}`);
if (process.env.GITHUB_ACTIONS) {
const absFile = path.join(skillDir, f.file);
const ghFile = path.relative(PROJECT_ROOT, absFile);
const line = f.line || 1;
const level = f.severity === 'LOW' ? 'notice' : 'warning';
console.log(`::${level} file=${ghFile},line=${line}::${escapeAnnotation(`${f.rule}: ${f.detail}`)}`);
}
}
}
}
// Summary
output.push(
`\n${'─'.repeat(60)}`,
`\nSummary:`,
` Skills scanned: ${totalSkills}`,
` Skills with findings: ${skillsWithFindings}`,
` Total findings: ${totalFindings}`,
);
if (totalFindings > 0) {
output.push('', ` | Severity | Count |`, ` |----------|-------|`);
for (const sev of ['CRITICAL', 'HIGH', 'MEDIUM', 'LOW']) {
if (severityCounts[sev] > 0) {
output.push(` | ${sev.padEnd(8)} | ${String(severityCounts[sev]).padStart(5)} |`);
}
}
}
const hasHighPlus = severityCounts.CRITICAL > 0 || severityCounts.HIGH > 0;
if (totalFindings === 0) {
output.push(`\n All skills passed validation!`);
} else if (STRICT && hasHighPlus) {
output.push(`\n [STRICT MODE] HIGH+ findings found — exiting with failure.`);
} else if (STRICT) {
output.push(`\n [STRICT MODE] Only MEDIUM/LOW findings — pass.`);
} else {
output.push(`\n Run with --strict to treat HIGH+ findings as errors.`);
}
output.push('');
// Write GitHub Actions step summary
if (process.env.GITHUB_STEP_SUMMARY) {
let summary = '## Skill Validation\n\n';
if (totalFindings > 0) {
summary += '| Skill | Rule | Severity | File | Detail |\n';
summary += '|-------|------|----------|------|--------|\n';
for (const { skillDir, findings } of results) {
const relDir = path.relative(PROJECT_ROOT, skillDir);
for (const f of findings) {
summary += `| ${escapeTableCell(relDir)} | ${f.rule} | ${f.severity} | ${escapeTableCell(f.file)} | ${escapeTableCell(f.detail)} |\n`;
}
}
summary += '\n';
}
summary += `**${totalSkills} skills scanned, ${totalFindings} findings**\n`;
fs.appendFileSync(process.env.GITHUB_STEP_SUMMARY, summary);
}
return { output: output.join('\n'), hasHighPlus };
}
function formatJson(results) {
const allFindings = [];
for (const { skillDir, findings } of results) {
const relDir = path.relative(PROJECT_ROOT, skillDir);
for (const f of findings) {
allFindings.push({
skill: relDir,
rule: f.rule,
title: f.title,
severity: f.severity,
file: f.file,
line: f.line || null,
detail: f.detail,
fix: f.fix,
});
}
}
// Sort by severity
allFindings.sort((a, b) => SEVERITY_ORDER[a.severity] - SEVERITY_ORDER[b.severity]);
const hasHighPlus = allFindings.some((f) => f.severity === 'CRITICAL' || f.severity === 'HIGH');
return { output: JSON.stringify(allFindings, null, 2), hasHighPlus };
}
// --- Main ---
if (require.main === module) {
// Determine which skills to validate
let skillDirs;
if (positionalArgs.length > 0) {
// Single skill directory specified
const target = path.resolve(positionalArgs[0]);
if (!fs.existsSync(target) || !fs.statSync(target).isDirectory()) {
console.error(`Error: "${positionalArgs[0]}" is not a valid directory.`);
process.exit(2);
}
skillDirs = [target];
} else {
// Discover all skills
skillDirs = discoverSkillDirs([SRC_DIR]);
}
if (skillDirs.length === 0) {
console.error('No skill directories found.');
process.exit(2);
}
// Validate each skill
const results = [];
for (const skillDir of skillDirs) {
const findings = validateSkill(skillDir);
results.push({ skillDir, findings });
}
// Format output
const { output, hasHighPlus } = JSON_OUTPUT ? formatJson(results) : formatHumanReadable(results);
console.log(output);
// Exit code
if (STRICT && hasHighPlus) {
process.exit(1);
}
}
// --- Exports (for testing) ---
module.exports = { parseFrontmatter, parseFrontmatterMultiline, validateSkill, discoverSkillDirs };