mirror of
https://github.com/bmad-code-org/BMAD-METHOD.git
synced 2026-08-28 19:20:41 +08:00
49069b8b52
* feat(quick-dev): render templates via stdlib Python at skill entry Move compile-time variable substitution out of the LLM and into a deterministic Python step. SKILL.md becomes a two-line stdout-dispatch shim that runs render.py and follows the instruction it prints. The renderer reads BMad configuration from the central four-layer TOML surface introduced in #2285 (_bmad/config.toml plus config.user.toml and the two _bmad/custom/ overrides), with a fallback to the legacy per-module _bmad/bmm/config.yaml for pre-#2285 installs. Compile-time refs ({{.var}}) get substituted at render time. LLM-runtime refs ({var}) pass through untouched. Renderer (render.py) - Python 3 stdlib only (tomllib, already bundled since 3.11). UTF-8 I/O. Every invocation rebuilds from scratch — no hash, no cache. - find_project_root walks up from cwd; HALT to stdout if no _bmad/ is found anywhere on the path. - load_central_config deep-merges the four TOML layers in priority order (base-team → base-user → custom-team → custom-user) so user overrides in _bmad/custom/config.user.toml win over installer- regenerated base values. flatten_central_config lifts scalar keys from [core] and [modules.bmm] into the renderer's flat namespace; module keys beat core on collision (matches the installer's own core-key-stripping behavior). - When _bmad/config.toml is absent, falls through to the legacy flat-YAML parser for _bmad/bmm/config.yaml — the renderer keeps working across the #2285 transition. - {{.var}} substitution; unresolved refs emit empty string (Go missingkey=zero semantics). - Smart defaults for planning_artifacts / implementation_artifacts / communication_language applied after config load. Derives sprint_status / deferred_work_file from implementation_artifacts. {{.main_config}} points at whichever surface was actually read. - Renders every .md in the skill dir except SKILL.md to {project-root}/_bmad/render/bmad-quick-dev/. - On success, stderr summary plus a single stdout line: "read and follow {workflow_md}". On failure, stdout HALT directive — per the Anthropic skills spec, script stdout is the defined agent- communication channel. Skill entry (SKILL.md) - Two-line shim: run python render.py, follow stdout. No template tokens in SKILL.md itself. Template conversions - workflow.md, step-01..05, step-oneshot, sync-sprint-status: convert every compile-time {var} reference to {{.var}}. Runtime refs preserved. - spec-template.md untouched (single-curly comment hint stays as documentation). Skill-prose cleanups bundled in - Remove dead step-file frontmatter: empty-string variable declarations (spec_file, story_key, diff_output, review_mode) in quick-dev step-01 and code-review step-01; empty --- --- blocks in step-03 and step-05; the specLoopIteration counter init moved from step-04 frontmatter into the step body where first-entry vs loopback semantics are explicit. - Unify the language rule across all six quick-dev step files plus workflow.md. Tooling - tools/validate-skills.js: add TPL-01 rule. Files whose name contains "template" must not contain compile-time {{.var}} substitutions. Template files seed durable, version-controlled artifacts that execute on other machines; baking a value at render time would freeze a machine-local path into every downstream artifact. - tools/validate-file-refs.js: add render/ to INSTALL_ONLY_PATHS so the validator recognizes the runtime-generated buffer. - tools/skill-validator.md: document TPL-01; deterministic rule count bumped from 14 to 15. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(quick-dev): drop render.py YAML fallback and smart defaults Single happy path: central _bmad/config.toml with four-layer merge, Python 3.11+ required (no ImportError guard), HALT if config missing. Deletes load_flat_yaml, the YAML fallback branch, the setdefault block for planning_artifacts/implementation_artifacts/communication_language, and the tomllib ImportError fallback. Part of plan-quick-dev-python-config-hardening.md (F0). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(quick-dev): normalize render.py paths to forward slashes On Windows, os.path.join returns backslash-separated paths that can misrender as escape sequences when later concatenated into POSIX shell strings or regexes. Normalize the project root to forward slashes after find_project_root, and use posixpath.join for every path that gets baked into rendered .md files or joined into config values. os.makedirs and os.listdir accept forward-slash paths on Windows, so their call sites stay as-is. Part of plan-quick-dev-python-config-hardening.md (F3). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(quick-dev): preserve source line endings in render.py Python text-mode open() with the platform default performs universal- newline translation: on Windows, LF source files get written as CRLF, producing spurious diffs when rendered output is compared against source. Pass newline="" on both the source read and the rendered write so line endings pass through verbatim. Part of plan-quick-dev-python-config-hardening.md (F4). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(quick-dev): delete stale .md renders before rebuilding render.py rebuilds from scratch per the docstring, but makedirs(exist_ok=True) only overwrites files that still exist in the source — stale outputs from renamed/deleted source files linger in _bmad/render/bmad-quick-dev/ forever. Remove every .md in the render dir before the render loop; keep the dir itself and any non-.md files. Part of plan-quick-dev-python-config-hardening.md (F5). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(quick-dev): scope render/ whitelist to bmad-quick-dev The previous INSTALL_ONLY_PATHS entry 'render/' was a blanket prefix that let every {project-root}/_bmad/render/... reference in any skill slip past validation. Narrow to 'render/bmad-quick-dev/' so only this skill's render buffer is whitelisted. Future skills adopting the stdout-dispatch renderer pattern add their own entries explicitly. Part of plan-quick-dev-python-config-hardening.md (F6). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(quick-dev): add renderer smoke test with TOML override New test/test-quick-dev-renderer.js spins up a temp project with base _bmad/config.toml and a _bmad/custom/config.user.toml override, runs render.py, and asserts the override wins in rendered workflow.md and that sprint_status is rooted at an absolute path in the temp project. Registered as test:renderer in package.json and chained into the npm test script. Part of plan-quick-dev-python-config-hardening.md (F7). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(quick-dev): HALT cleanly when base config.toml is unparseable Load the four config layers through a load_toml helper that marks the base _bmad/config.toml as required. A missing, unparseable, or unreadable base now prints a HALT directive to stdout and exits, instead of being silently skipped and then crashing downstream with a KeyError when a derived value (e.g. implementation_artifacts) is absent. Optional layers still warn on stderr and fall back to empty. Merge semantics are unchanged (dict-aware deep merge, override wins for lists and scalars). * fix(quick-dev): resolve render.py via {skill-root} in skill entry shim The bare `python render.py` shim assumes the agent's working directory is the skill directory, but agents run from the project root, so the script is not found. Reference it as `{skill-root}/render.py` — BMAD's standard token for a skill's installed directory, already used by every other skill's resolve_customization.py invocation — and add the one-line `{skill-root}` explainer so the model resolves it from an instruction rather than guessing. Interpreter stays `python`; the python vs python3 choice is a separate cross-platform concern. * refactor(quick-dev): resolve [workflow] customization in render.py render.py now merges the three customize layers (customize.toml -> custom/bmad-quick-dev.toml -> .user.toml) with the same structural rules as resolve_customization.py and inlines the resolved [workflow] values, so no {workflow.*} placeholder survives. workflow.md drops its Step 1 runtime resolver + manual-merge fallback; step-05 and step-oneshot drop their runtime workflow.on_complete calls. The shared resolve_customization.py and every other skill are untouched. Smoke test extended with a [workflow] override fixture covering inlining, array append, and no-leak assertions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(quick-dev): harden render.py invocation in the SKILL.md shim The shim called bare `python`, which can resolve to Python 2 or be absent; render.py needs 3.11+ for tomllib. Spell out python3 and the version requirement. Also make the exit code authoritative: on a non-zero exit (including an uncaught crash that writes only to stderr), do not proceed -- report what was printed and stop. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(quick-dev): drop the render.py success stderr line The "rendered N files" progress line was pure diagnostic noise. The shim already tells the LLM to ignore stderr and follow the stdout instruction, so on success render.py now prints only the "read and follow ..." line. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(quick-dev): drop the activation gate sentence from the rendered workflow The gate ported from #2398 defended against runtime customization indirection: agents guessed resolver outputs instead of executing them, silently skipping append steps. render.py inlines the prepend/append entries into the rendered workflow.md, so there is nothing left to short-circuit, and each inlined list already carries its own execute- in-order imperative. In the default install both lists render as _None._ and the gate is pure noise. * feat(quick-dev): materialize review layers into invocation blocks Reconcile #2550 with render-time [workflow] resolution. Main made review layers configurable as [[workflow.review_layers]] arrays of tables and had the LLM resolve them during activation; this branch resolves the [workflow] block in render.py instead, so activation-time resolution no longer exists and the layer refs must be materialized at render time. Rather than inlining the layer tables as data plus interpretation rules, render.py now knows this skill's customization schema outright and renders review_layers/oneshot_review_layers as direct invocation blocks: disabled layers (empty instruction) drop out, each active layer becomes a #### section holding its instruction verbatim, zero active layers renders the HALT instruction, and runtime placeholders like {diff_output} pass through. The only judgment left to the LLM is the optional `when` condition, which renders as a run-time guard line. The step-04/step-oneshot review intros collapse to a single execute-in- parallel imperative. Smoke test covers default rendering, replace-by-id, disable-by-empty-instruction, when-guards, and the all-disabled HALT. * fix(quick-dev): invoke render.py via uv run per house standard The SKILL.md shim launched render.py with bare `python3`, which the rest of BMAD is migrating away from: the customize-bmad docs and the installer's uv-check standardize on `uv run` (uv provisions a suitable 3.11+ interpreter on demand). Bare `python3` is also fragile on Windows, where python.org installs expose `python`/`py` rather than `python3`. Make `uv run` the primary invocation and demote `python3` to the documented fallback, spelling out `python`/`py -3` for Windows and the 3.11+ tomllib requirement. render.py itself is unchanged; the renderer test drives it directly and is unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(quick-dev): HALT cleanly on missing or malformed config render.py derived sprint_status/deferred_work_file from an unconditional vars_["implementation_artifacts"] subscript, so a config lacking that key raised a raw KeyError instead of the stdout HALT the rest of the script uses on bad input. flatten_central_config likewise called .get("bmm") on merged["modules"] without checking it was a table, so a non-table [modules] crashed with an AttributeError. Guard both: HALT with a clear stdout directive when implementation_artifacts is missing or blank, and coerce a non-dict modules to {} before indexing. Add renderer regression tests asserting each path exits without a Python traceback. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(quick-dev): resolve config at compile time, drop the runtime re-read The activation "Load Config" step told the LLM to open {{.main_config}} and re-resolve project_name, communication_language, sprint_status, etc. at run time -- but render.py already bakes those from the full four-layer config merge. main_config pointed at only the base _bmad/config.toml, so on installs with override layers (config.user.toml / custom/*) the runtime re-read saw stale values that could contradict the baked {{.var}} in the same rendered file. It also handed resolution back to the LLM: the drift this skill's renderer exists to remove. Delete the ceremony and wire each value where it is actually used: - Every value the step resolved is already inlined at its point of use (planning/implementation_artifacts, sprint_status, communication_language) or loaded via persistent_facts (project-context.md), so the central block was pure redundancy. - Fold document_output_language into the per-step language rule, adopting the house-canonical form ("Speak in X. Write any file output in Y.") already used by bmad-checkpoint-preview. - Move the {date} = current-datetime definition to step-02, where the spec template's {date} field is filled. - Drop the user greeting (user_name) and user_skill_level tailoring: quick-dev is not a conversational skill and neither was load-bearing. - Remove main_config from render.py; it had no remaining consumer. Renderer tests repointed at the files that now carry these values, plus coverage for document_output_language baking and main_config removal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(quick-dev): reference variables by bare name, not placeholder curlies Curlies mean "expand this to the value"; a bare backticked name means "this is the variable/field I'm talking about". Several step files wrapped a variable in curlies where they were only naming, assigning, passing, or testing it -- so the notation implied an expansion that never happens: - step-01: identify `epic_num`/`story_num`, set/leave `story_key` unset - step-02: test `preserved_intent`; and resolve the template's `date` field (was `{date}`, which read as "expand date here" rather than naming it) - step-03/step-05/step-oneshot: pass `target_status` to sync-sprint-status, set `title` - sync-sprint-status: the `target_status` parameter, `story_key` precondition, and both `target_status` conditionals Value tokens that are genuinely materialized in place -- `{spec_file}` paths, `development_status[{story_key}]`, "set ... to `{target_status}`" -- stay curly. Also reword step-02's frozen-block instruction from the ambiguous "substitute it for the `<frozen-after-approval>` block" to "replace the `<frozen-after-approval>` block in the spec you just filled out with `preserved_intent`" so it's clear the replacement happens in the artifact. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(quick-dev): require uv, drop the python3 interpreter fallback The SKILL.md shim tried `uv run render.py` and, if uv was missing, retried with a bare `python3`/`py -3` interpreter. Nothing else in the codebase does that interpreter fallback: the uv-based skills (bmad-prd, bmad-ux, bmad-architecture, bmad-product-brief) fall back to reading customize.toml and using defaults -- graceful feature degradation, never a different runner -- and the legacy skills just call python3 outright. uv is the established house runner (memlog.py, resolve_customization.py, lint_spine.py all invoke it). That graceful-degrade path does not exist here: render.py is the entry dispatch that produces the workflow.md the LLM then follows, so there is nothing to fall back to. The only honest outcomes are "uv runs it" or "HALT". Make uv the floor and drop the fallback. Pin the interpreter the house way -- a PEP 723 `requires-python = ">=3.11"` block, matching memlog.py/lint_spine.py -- so `uv run` provisions a 3.11+ interpreter and the tomllib requirement is guaranteed rather than hoped for. This replaces the prose "needs 3.11+" hedge the shim used to carry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
736 lines
24 KiB
JavaScript
736 lines
24 KiB
JavaScript
/**
|
|
* Deterministic Skill Validator
|
|
*
|
|
* Validates 12 deterministic rules across all skill directories.
|
|
* Acts as a fast first-pass complement to the inference-based skill validator.
|
|
*
|
|
* What it checks:
|
|
* - SKILL-01: SKILL.md exists
|
|
* - SKILL-02: SKILL.md frontmatter has name
|
|
* - SKILL-03: SKILL.md frontmatter has description
|
|
* - SKILL-04: name format (lowercase, hyphens, no forbidden substrings)
|
|
* - SKILL-05: name matches directory basename
|
|
* - SKILL-06: description quality (length, "Use when"/"Use if")
|
|
* - SKILL-07: SKILL.md has body content after frontmatter
|
|
* - PATH-02: no installed_path variable
|
|
* - STEP-01: step filename format
|
|
* - STEP-06: step frontmatter has no name/description
|
|
* - STEP-07: step count 2-10
|
|
* - SEQ-02: no time estimates
|
|
* - TPL-01: template files must not contain compile-time {{.var}} substitutions
|
|
*
|
|
* Usage:
|
|
* node tools/validate-skills.js # All skills, human-readable
|
|
* node tools/validate-skills.js path/to/skill-dir # Single skill
|
|
* node tools/validate-skills.js --strict # Exit 1 on HIGH+ findings
|
|
* node tools/validate-skills.js --json # JSON output
|
|
*/
|
|
|
|
const fs = require('node:fs');
|
|
const path = require('node:path');
|
|
|
|
const PROJECT_ROOT = path.resolve(__dirname, '..');
|
|
const SRC_DIR = path.join(PROJECT_ROOT, 'src');
|
|
|
|
// --- CLI Parsing ---
|
|
|
|
const args = process.argv.slice(2);
|
|
const STRICT = args.includes('--strict');
|
|
const JSON_OUTPUT = args.includes('--json');
|
|
const positionalArgs = args.filter((a) => !a.startsWith('--'));
|
|
|
|
// --- Constants ---
|
|
|
|
const NAME_REGEX = /^bmad-[a-z0-9]+(-[a-z0-9]+)*$/;
|
|
const STEP_FILENAME_REGEX = /^step-\d{2}[a-z]?-[a-z0-9-]+\.md$/;
|
|
const TIME_ESTIMATE_PATTERNS = [/takes?\s+\d+\s*min/i, /~\s*\d+\s*min/i, /estimated\s+time/i, /\bETA\b/];
|
|
const TEMPLATE_FILENAME_REGEX = /template/i;
|
|
const COMPILE_TIME_SUB_REGEX = /\{\{\.\w+\}\}/;
|
|
|
|
const SEVERITY_ORDER = { CRITICAL: 0, HIGH: 1, MEDIUM: 2, LOW: 3 };
|
|
|
|
// --- Output Escaping ---
|
|
|
|
function escapeAnnotation(str) {
|
|
return str.replaceAll('%', '%25').replaceAll('\r', '%0D').replaceAll('\n', '%0A');
|
|
}
|
|
|
|
function escapeTableCell(str) {
|
|
return String(str).replaceAll('|', String.raw`\|`);
|
|
}
|
|
|
|
// --- Frontmatter Parsing ---
|
|
|
|
/**
|
|
* Parse YAML frontmatter from a markdown file.
|
|
* Returns an object with key-value pairs, or null if no frontmatter.
|
|
*/
|
|
function parseFrontmatter(content) {
|
|
const trimmed = content.trimStart();
|
|
if (!trimmed.startsWith('---')) return null;
|
|
|
|
let endIndex = trimmed.indexOf('\n---\n', 3);
|
|
if (endIndex === -1) {
|
|
// Handle file ending with \n---
|
|
if (trimmed.endsWith('\n---')) {
|
|
endIndex = trimmed.length - 4;
|
|
} else {
|
|
return null;
|
|
}
|
|
}
|
|
|
|
const fmBlock = trimmed.slice(3, endIndex).trim();
|
|
if (fmBlock === '') return {};
|
|
|
|
const result = {};
|
|
for (const line of fmBlock.split('\n')) {
|
|
const colonIndex = line.indexOf(':');
|
|
if (colonIndex === -1) continue;
|
|
// Skip indented lines (nested YAML values)
|
|
if (line[0] === ' ' || line[0] === '\t') continue;
|
|
const key = line.slice(0, colonIndex).trim();
|
|
let value = line.slice(colonIndex + 1).trim();
|
|
// Strip surrounding quotes (single or double)
|
|
if ((value.startsWith("'") && value.endsWith("'")) || (value.startsWith('"') && value.endsWith('"'))) {
|
|
value = value.slice(1, -1);
|
|
}
|
|
result[key] = value;
|
|
}
|
|
|
|
return result;
|
|
}
|
|
|
|
/**
|
|
* Parse YAML frontmatter, handling multiline values (description often spans lines).
|
|
* Returns an object with key-value pairs, or null if no frontmatter.
|
|
*/
|
|
function parseFrontmatterMultiline(content) {
|
|
const trimmed = content.trimStart();
|
|
if (!trimmed.startsWith('---')) return null;
|
|
|
|
let endIndex = trimmed.indexOf('\n---\n', 3);
|
|
if (endIndex === -1) {
|
|
// Handle file ending with \n---
|
|
if (trimmed.endsWith('\n---')) {
|
|
endIndex = trimmed.length - 4;
|
|
} else {
|
|
return null;
|
|
}
|
|
}
|
|
|
|
const fmBlock = trimmed.slice(3, endIndex).trim();
|
|
if (fmBlock === '') return {};
|
|
|
|
const result = {};
|
|
let currentKey = null;
|
|
let currentValue = '';
|
|
|
|
for (const line of fmBlock.split('\n')) {
|
|
const colonIndex = line.indexOf(':');
|
|
// New key-value pair: must start at column 0 (no leading whitespace) and have a colon
|
|
if (colonIndex > 0 && line[0] !== ' ' && line[0] !== '\t') {
|
|
// Save previous key
|
|
if (currentKey !== null) {
|
|
result[currentKey] = stripQuotes(currentValue.trim());
|
|
}
|
|
currentKey = line.slice(0, colonIndex).trim();
|
|
currentValue = line.slice(colonIndex + 1);
|
|
} else if (currentKey !== null) {
|
|
// Skip YAML comment lines
|
|
if (line.trimStart().startsWith('#')) continue;
|
|
// Continuation of multiline value
|
|
currentValue += '\n' + line;
|
|
}
|
|
}
|
|
|
|
// Save last key
|
|
if (currentKey !== null) {
|
|
result[currentKey] = stripQuotes(currentValue.trim());
|
|
}
|
|
|
|
return result;
|
|
}
|
|
|
|
function stripQuotes(value) {
|
|
if ((value.startsWith("'") && value.endsWith("'")) || (value.startsWith('"') && value.endsWith('"'))) {
|
|
return value.slice(1, -1);
|
|
}
|
|
return value;
|
|
}
|
|
|
|
// --- Safe File Reading ---
|
|
|
|
/**
|
|
* Read a file safely, returning null on error.
|
|
* Pushes a warning finding if the file cannot be read.
|
|
*/
|
|
function safeReadFile(filePath, findings, relFile) {
|
|
try {
|
|
return fs.readFileSync(filePath, 'utf-8');
|
|
} catch (error) {
|
|
findings.push({
|
|
rule: 'READ-ERR',
|
|
title: 'File Read Error',
|
|
severity: 'MEDIUM',
|
|
file: relFile || path.basename(filePath),
|
|
detail: `Cannot read file: ${error.message}`,
|
|
fix: 'Check file permissions and ensure the file exists.',
|
|
});
|
|
return null;
|
|
}
|
|
}
|
|
|
|
// --- Code Block Stripping ---
|
|
|
|
function stripCodeBlocks(content) {
|
|
return content.replaceAll(/```[\s\S]*?```/g, (m) => m.replaceAll(/[^\n]/g, ''));
|
|
}
|
|
|
|
// --- Skill Discovery ---
|
|
|
|
function discoverSkillDirs(rootDirs) {
|
|
const skillDirs = [];
|
|
|
|
function walk(dir) {
|
|
if (!fs.existsSync(dir)) return;
|
|
const entries = fs.readdirSync(dir, { withFileTypes: true });
|
|
|
|
for (const entry of entries) {
|
|
if (!entry.isDirectory()) continue;
|
|
if (entry.name === 'node_modules' || entry.name === '.git') continue;
|
|
|
|
const fullPath = path.join(dir, entry.name);
|
|
const skillMd = path.join(fullPath, 'SKILL.md');
|
|
|
|
if (fs.existsSync(skillMd)) {
|
|
skillDirs.push(fullPath);
|
|
}
|
|
|
|
// Keep walking into subdirectories to find nested skills
|
|
walk(fullPath);
|
|
}
|
|
}
|
|
|
|
for (const rootDir of rootDirs) {
|
|
walk(rootDir);
|
|
}
|
|
|
|
return skillDirs.sort();
|
|
}
|
|
|
|
// --- File Collection ---
|
|
|
|
function collectSkillFiles(skillDir) {
|
|
const files = [];
|
|
|
|
function walk(dir) {
|
|
const entries = fs.readdirSync(dir, { withFileTypes: true });
|
|
for (const entry of entries) {
|
|
if (entry.name === 'node_modules' || entry.name === '.git') continue;
|
|
const fullPath = path.join(dir, entry.name);
|
|
if (entry.isDirectory()) {
|
|
walk(fullPath);
|
|
} else if (entry.isFile()) {
|
|
files.push(fullPath);
|
|
}
|
|
}
|
|
}
|
|
|
|
walk(skillDir);
|
|
return files;
|
|
}
|
|
|
|
// --- Rule Checks ---
|
|
|
|
function validateSkill(skillDir) {
|
|
const findings = [];
|
|
const dirName = path.basename(skillDir);
|
|
const skillMdPath = path.join(skillDir, 'SKILL.md');
|
|
const workflowMdPath = path.join(skillDir, 'workflow.md');
|
|
const stepsDir = path.join(skillDir, 'steps');
|
|
|
|
// Collect all files in the skill for PATH-02 and SEQ-02
|
|
const allFiles = collectSkillFiles(skillDir);
|
|
|
|
// --- SKILL-01: SKILL.md must exist ---
|
|
if (!fs.existsSync(skillMdPath)) {
|
|
findings.push({
|
|
rule: 'SKILL-01',
|
|
title: 'SKILL.md Must Exist',
|
|
severity: 'CRITICAL',
|
|
file: 'SKILL.md',
|
|
detail: 'SKILL.md not found in skill directory.',
|
|
fix: 'Create SKILL.md as the skill entrypoint.',
|
|
});
|
|
// Cannot check SKILL-02 through SKILL-07 without SKILL.md
|
|
return findings;
|
|
}
|
|
|
|
const skillContent = safeReadFile(skillMdPath, findings, 'SKILL.md');
|
|
if (skillContent === null) return findings;
|
|
const skillFm = parseFrontmatterMultiline(skillContent);
|
|
|
|
// --- SKILL-02: frontmatter has name ---
|
|
if (!skillFm || !('name' in skillFm)) {
|
|
findings.push({
|
|
rule: 'SKILL-02',
|
|
title: 'SKILL.md Must Have name in Frontmatter',
|
|
severity: 'CRITICAL',
|
|
file: 'SKILL.md',
|
|
detail: 'Frontmatter is missing the `name` field.',
|
|
fix: 'Add `name: <skill-name>` to the frontmatter.',
|
|
});
|
|
} else if (skillFm.name === '') {
|
|
findings.push({
|
|
rule: 'SKILL-02',
|
|
title: 'SKILL.md Must Have name in Frontmatter',
|
|
severity: 'CRITICAL',
|
|
file: 'SKILL.md',
|
|
detail: 'Frontmatter `name` field is empty.',
|
|
fix: 'Set `name` to the skill directory name (kebab-case).',
|
|
});
|
|
}
|
|
|
|
// --- SKILL-03: frontmatter has description ---
|
|
if (!skillFm || !('description' in skillFm)) {
|
|
findings.push({
|
|
rule: 'SKILL-03',
|
|
title: 'SKILL.md Must Have description in Frontmatter',
|
|
severity: 'CRITICAL',
|
|
file: 'SKILL.md',
|
|
detail: 'Frontmatter is missing the `description` field.',
|
|
fix: 'Add `description: <what it does and when to use it>` to the frontmatter.',
|
|
});
|
|
} else if (skillFm.description === '') {
|
|
findings.push({
|
|
rule: 'SKILL-03',
|
|
title: 'SKILL.md Must Have description in Frontmatter',
|
|
severity: 'CRITICAL',
|
|
file: 'SKILL.md',
|
|
detail: 'Frontmatter `description` field is empty.',
|
|
fix: 'Add a description stating what the skill does and when to use it.',
|
|
});
|
|
}
|
|
|
|
const name = skillFm && skillFm.name;
|
|
const description = skillFm && skillFm.description;
|
|
|
|
// Deprecated skills are thin compatibility shims that forward to a replacement.
|
|
// They intentionally omit a "Use when" trigger so users are steered to the new
|
|
// skill instead, so exempt them from the SKILL-06 trigger-phrase requirement.
|
|
const isDeprecated = typeof description === 'string' && /^\s*deprecated\b/i.test(description);
|
|
|
|
// --- SKILL-04: name format ---
|
|
if (name && !NAME_REGEX.test(name)) {
|
|
findings.push({
|
|
rule: 'SKILL-04',
|
|
title: 'name Format',
|
|
severity: 'HIGH',
|
|
file: 'SKILL.md',
|
|
detail: `name "${name}" does not match pattern: ${NAME_REGEX}`,
|
|
fix: 'Rename to comply with lowercase letters, numbers, and hyphens only (max 64 chars).',
|
|
});
|
|
}
|
|
|
|
// --- SKILL-05: name matches directory ---
|
|
if (name && name !== dirName) {
|
|
findings.push({
|
|
rule: 'SKILL-05',
|
|
title: 'name Must Match Directory Name',
|
|
severity: 'HIGH',
|
|
file: 'SKILL.md',
|
|
detail: `name "${name}" does not match directory name "${dirName}".`,
|
|
fix: `Change name to "${dirName}" or rename the directory.`,
|
|
});
|
|
}
|
|
|
|
// --- SKILL-06: description quality ---
|
|
if (description) {
|
|
if (description.length > 1024) {
|
|
findings.push({
|
|
rule: 'SKILL-06',
|
|
title: 'description Quality',
|
|
severity: 'MEDIUM',
|
|
file: 'SKILL.md',
|
|
detail: `description is ${description.length} characters (max 1024).`,
|
|
fix: 'Shorten the description to 1024 characters or less.',
|
|
});
|
|
}
|
|
|
|
if (!isDeprecated && !/use\s+when\b/i.test(description) && !/use\s+if\b/i.test(description)) {
|
|
findings.push({
|
|
rule: 'SKILL-06',
|
|
title: 'description Quality',
|
|
severity: 'MEDIUM',
|
|
file: 'SKILL.md',
|
|
detail: 'description does not contain "Use when" or "Use if" trigger phrase.',
|
|
fix: 'Append a "Use when..." clause to explain when to invoke this skill.',
|
|
});
|
|
}
|
|
}
|
|
|
|
// --- SKILL-07: SKILL.md must have body content after frontmatter ---
|
|
{
|
|
const trimmed = skillContent.trimStart();
|
|
let bodyStart = -1;
|
|
if (trimmed.startsWith('---')) {
|
|
let endIdx = trimmed.indexOf('\n---\n', 3);
|
|
if (endIdx !== -1) {
|
|
bodyStart = endIdx + 4;
|
|
} else if (trimmed.endsWith('\n---')) {
|
|
bodyStart = trimmed.length; // no body at all
|
|
}
|
|
} else {
|
|
bodyStart = 0; // no frontmatter, entire file is body
|
|
}
|
|
const body = bodyStart >= 0 ? trimmed.slice(bodyStart).trim() : '';
|
|
if (body === '') {
|
|
findings.push({
|
|
rule: 'SKILL-07',
|
|
title: 'SKILL.md Must Have Body Content',
|
|
severity: 'HIGH',
|
|
file: 'SKILL.md',
|
|
detail: 'SKILL.md has no content after frontmatter. L2 instructions are required.',
|
|
fix: 'Add markdown body with skill instructions after the closing ---.',
|
|
});
|
|
}
|
|
}
|
|
|
|
// --- PATH-02: no installed_path ---
|
|
for (const filePath of allFiles) {
|
|
// Only check markdown and yaml files
|
|
const ext = path.extname(filePath);
|
|
if (!['.md', '.yaml', '.yml'].includes(ext)) continue;
|
|
|
|
const relFile = path.relative(skillDir, filePath);
|
|
const content = safeReadFile(filePath, findings, relFile);
|
|
if (content === null) continue;
|
|
|
|
// Check frontmatter for installed_path key
|
|
const fm = parseFrontmatter(content);
|
|
if (fm && 'installed_path' in fm) {
|
|
findings.push({
|
|
rule: 'PATH-02',
|
|
title: 'No installed_path Variable',
|
|
severity: 'HIGH',
|
|
file: relFile,
|
|
detail: 'Frontmatter contains `installed_path:` key.',
|
|
fix: 'Remove `installed_path` from frontmatter. Use relative paths instead.',
|
|
});
|
|
}
|
|
|
|
// Check content for any mention of installed_path (variable ref, prose, bare text)
|
|
const stripped = stripCodeBlocks(content);
|
|
const lines = stripped.split('\n');
|
|
for (const [i, line] of lines.entries()) {
|
|
if (/installed_path/i.test(line)) {
|
|
findings.push({
|
|
rule: 'PATH-02',
|
|
title: 'No installed_path Variable',
|
|
severity: 'HIGH',
|
|
file: relFile,
|
|
line: i + 1,
|
|
detail: '`installed_path` reference found in content.',
|
|
fix: 'Remove all installed_path usage. Use relative paths (`./path` or `../path`) instead.',
|
|
});
|
|
}
|
|
}
|
|
}
|
|
|
|
// --- STEP-01: step filename format ---
|
|
// --- STEP-06: step frontmatter no name/description ---
|
|
// --- STEP-07: step count ---
|
|
// Only check the literal steps/ directory (variant directories like steps-c, steps-v
|
|
// use different naming conventions and are excluded per the rule specification)
|
|
if (fs.existsSync(stepsDir) && fs.statSync(stepsDir).isDirectory()) {
|
|
const stepDirName = 'steps';
|
|
const stepFiles = fs.readdirSync(stepsDir).filter((f) => f.endsWith('.md'));
|
|
|
|
// STEP-01: filename format
|
|
for (const stepFile of stepFiles) {
|
|
if (!STEP_FILENAME_REGEX.test(stepFile)) {
|
|
findings.push({
|
|
rule: 'STEP-01',
|
|
title: 'Step File Naming',
|
|
severity: 'MEDIUM',
|
|
file: path.join(stepDirName, stepFile),
|
|
detail: `Filename "${stepFile}" does not match pattern: ${STEP_FILENAME_REGEX}`,
|
|
fix: 'Rename to step-NN-description.md (NN = zero-padded number, optional letter suffix).',
|
|
});
|
|
}
|
|
}
|
|
|
|
// STEP-06: step frontmatter has no name/description
|
|
for (const stepFile of stepFiles) {
|
|
const stepPath = path.join(stepsDir, stepFile);
|
|
const stepContent = safeReadFile(stepPath, findings, path.join(stepDirName, stepFile));
|
|
if (stepContent === null) continue;
|
|
const stepFm = parseFrontmatter(stepContent);
|
|
|
|
if (stepFm) {
|
|
if ('name' in stepFm) {
|
|
findings.push({
|
|
rule: 'STEP-06',
|
|
title: 'Step File Frontmatter: No name or description',
|
|
severity: 'MEDIUM',
|
|
file: path.join(stepDirName, stepFile),
|
|
detail: 'Step file frontmatter contains `name:` — this is metadata noise.',
|
|
fix: 'Remove `name:` from step file frontmatter.',
|
|
});
|
|
}
|
|
if ('description' in stepFm) {
|
|
findings.push({
|
|
rule: 'STEP-06',
|
|
title: 'Step File Frontmatter: No name or description',
|
|
severity: 'MEDIUM',
|
|
file: path.join(stepDirName, stepFile),
|
|
detail: 'Step file frontmatter contains `description:` — this is metadata noise.',
|
|
fix: 'Remove `description:` from step file frontmatter.',
|
|
});
|
|
}
|
|
}
|
|
}
|
|
|
|
// STEP-07: step count 2-10
|
|
const stepCount = stepFiles.filter((f) => f.startsWith('step-')).length;
|
|
if (stepCount > 0 && (stepCount < 2 || stepCount > 10)) {
|
|
const detail =
|
|
stepCount < 2
|
|
? `Only ${stepCount} step file found — consider inlining into workflow.md.`
|
|
: `${stepCount} step files found — more than 10 risks LLM context degradation.`;
|
|
findings.push({
|
|
rule: 'STEP-07',
|
|
title: 'Step Count',
|
|
severity: 'LOW',
|
|
file: stepDirName + '/',
|
|
detail,
|
|
fix: stepCount > 10 ? 'Consider consolidating steps.' : 'Consider expanding or inlining.',
|
|
});
|
|
}
|
|
}
|
|
|
|
// --- SEQ-02: no time estimates ---
|
|
for (const filePath of allFiles) {
|
|
const ext = path.extname(filePath);
|
|
if (!['.md', '.yaml', '.yml'].includes(ext)) continue;
|
|
|
|
const relFile = path.relative(skillDir, filePath);
|
|
const content = safeReadFile(filePath, findings, relFile);
|
|
if (content === null) continue;
|
|
const stripped = stripCodeBlocks(content);
|
|
const lines = stripped.split('\n');
|
|
|
|
for (const [i, line] of lines.entries()) {
|
|
for (const pattern of TIME_ESTIMATE_PATTERNS) {
|
|
if (pattern.test(line)) {
|
|
findings.push({
|
|
rule: 'SEQ-02',
|
|
title: 'No Time Estimates',
|
|
severity: 'LOW',
|
|
file: relFile,
|
|
line: i + 1,
|
|
detail: `Time estimate pattern found: "${line.trim()}"`,
|
|
fix: 'Remove time estimates — AI execution speed varies too much.',
|
|
});
|
|
break; // Only report once per line
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
// --- TPL-01: template files must not contain compile-time {{.var}} substitutions ---
|
|
// Template files seed durable, version-controlled artifacts (spec files) that
|
|
// execute on other machines. Baking a {{.var}} at render time would freeze a
|
|
// machine-local value into every downstream artifact.
|
|
for (const filePath of allFiles) {
|
|
if (path.extname(filePath) !== '.md') continue;
|
|
const base = path.basename(filePath);
|
|
if (!TEMPLATE_FILENAME_REGEX.test(base)) continue;
|
|
|
|
const relFile = path.relative(skillDir, filePath);
|
|
const content = safeReadFile(filePath, findings, relFile);
|
|
if (content === null) continue;
|
|
|
|
const lines = content.split('\n');
|
|
for (const [i, line] of lines.entries()) {
|
|
const match = line.match(COMPILE_TIME_SUB_REGEX);
|
|
if (match) {
|
|
findings.push({
|
|
rule: 'TPL-01',
|
|
title: 'Template files must not contain compile-time substitutions',
|
|
severity: 'HIGH',
|
|
file: relFile,
|
|
line: i + 1,
|
|
detail: `Template file contains compile-time substitution \`${match[0]}\` — this would be baked at render time and leak a machine-local value into every spec produced from the template.`,
|
|
fix: 'Remove the `{{.var}}` reference. Use single-curly `{var}` if the value should be resolved at LLM runtime by the consumer of the generated spec.',
|
|
});
|
|
}
|
|
}
|
|
}
|
|
|
|
return findings;
|
|
}
|
|
|
|
// --- Output Formatting ---
|
|
|
|
function formatHumanReadable(results) {
|
|
const output = [];
|
|
let totalFindings = 0;
|
|
const severityCounts = { CRITICAL: 0, HIGH: 0, MEDIUM: 0, LOW: 0 };
|
|
|
|
output.push(
|
|
`\nValidating skills in: ${SRC_DIR}`,
|
|
`Mode: ${STRICT ? 'STRICT (exit 1 on HIGH+)' : 'WARNING (exit 0)'}${JSON_OUTPUT ? ' + JSON' : ''}\n`,
|
|
);
|
|
|
|
let totalSkills = 0;
|
|
let skillsWithFindings = 0;
|
|
|
|
for (const { skillDir, findings } of results) {
|
|
totalSkills++;
|
|
const relDir = path.relative(PROJECT_ROOT, skillDir);
|
|
|
|
if (findings.length > 0) {
|
|
skillsWithFindings++;
|
|
output.push(`\n${relDir}`);
|
|
|
|
for (const f of findings) {
|
|
totalFindings++;
|
|
severityCounts[f.severity]++;
|
|
const location = f.line ? ` (line ${f.line})` : '';
|
|
output.push(` [${f.severity}] ${f.rule} — ${f.title}`, ` File: ${f.file}${location}`, ` ${f.detail}`);
|
|
|
|
if (process.env.GITHUB_ACTIONS) {
|
|
const absFile = path.join(skillDir, f.file);
|
|
const ghFile = path.relative(PROJECT_ROOT, absFile);
|
|
const line = f.line || 1;
|
|
const level = f.severity === 'LOW' ? 'notice' : 'warning';
|
|
console.log(`::${level} file=${ghFile},line=${line}::${escapeAnnotation(`${f.rule}: ${f.detail}`)}`);
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
// Summary
|
|
output.push(
|
|
`\n${'─'.repeat(60)}`,
|
|
`\nSummary:`,
|
|
` Skills scanned: ${totalSkills}`,
|
|
` Skills with findings: ${skillsWithFindings}`,
|
|
` Total findings: ${totalFindings}`,
|
|
);
|
|
|
|
if (totalFindings > 0) {
|
|
output.push('', ` | Severity | Count |`, ` |----------|-------|`);
|
|
for (const sev of ['CRITICAL', 'HIGH', 'MEDIUM', 'LOW']) {
|
|
if (severityCounts[sev] > 0) {
|
|
output.push(` | ${sev.padEnd(8)} | ${String(severityCounts[sev]).padStart(5)} |`);
|
|
}
|
|
}
|
|
}
|
|
|
|
const hasHighPlus = severityCounts.CRITICAL > 0 || severityCounts.HIGH > 0;
|
|
|
|
if (totalFindings === 0) {
|
|
output.push(`\n All skills passed validation!`);
|
|
} else if (STRICT && hasHighPlus) {
|
|
output.push(`\n [STRICT MODE] HIGH+ findings found — exiting with failure.`);
|
|
} else if (STRICT) {
|
|
output.push(`\n [STRICT MODE] Only MEDIUM/LOW findings — pass.`);
|
|
} else {
|
|
output.push(`\n Run with --strict to treat HIGH+ findings as errors.`);
|
|
}
|
|
|
|
output.push('');
|
|
|
|
// Write GitHub Actions step summary
|
|
if (process.env.GITHUB_STEP_SUMMARY) {
|
|
let summary = '## Skill Validation\n\n';
|
|
if (totalFindings > 0) {
|
|
summary += '| Skill | Rule | Severity | File | Detail |\n';
|
|
summary += '|-------|------|----------|------|--------|\n';
|
|
for (const { skillDir, findings } of results) {
|
|
const relDir = path.relative(PROJECT_ROOT, skillDir);
|
|
for (const f of findings) {
|
|
summary += `| ${escapeTableCell(relDir)} | ${f.rule} | ${f.severity} | ${escapeTableCell(f.file)} | ${escapeTableCell(f.detail)} |\n`;
|
|
}
|
|
}
|
|
summary += '\n';
|
|
}
|
|
summary += `**${totalSkills} skills scanned, ${totalFindings} findings**\n`;
|
|
fs.appendFileSync(process.env.GITHUB_STEP_SUMMARY, summary);
|
|
}
|
|
|
|
return { output: output.join('\n'), hasHighPlus };
|
|
}
|
|
|
|
function formatJson(results) {
|
|
const allFindings = [];
|
|
for (const { skillDir, findings } of results) {
|
|
const relDir = path.relative(PROJECT_ROOT, skillDir);
|
|
for (const f of findings) {
|
|
allFindings.push({
|
|
skill: relDir,
|
|
rule: f.rule,
|
|
title: f.title,
|
|
severity: f.severity,
|
|
file: f.file,
|
|
line: f.line || null,
|
|
detail: f.detail,
|
|
fix: f.fix,
|
|
});
|
|
}
|
|
}
|
|
|
|
// Sort by severity
|
|
allFindings.sort((a, b) => SEVERITY_ORDER[a.severity] - SEVERITY_ORDER[b.severity]);
|
|
|
|
const hasHighPlus = allFindings.some((f) => f.severity === 'CRITICAL' || f.severity === 'HIGH');
|
|
|
|
return { output: JSON.stringify(allFindings, null, 2), hasHighPlus };
|
|
}
|
|
|
|
// --- Main ---
|
|
|
|
if (require.main === module) {
|
|
// Determine which skills to validate
|
|
let skillDirs;
|
|
|
|
if (positionalArgs.length > 0) {
|
|
// Single skill directory specified
|
|
const target = path.resolve(positionalArgs[0]);
|
|
if (!fs.existsSync(target) || !fs.statSync(target).isDirectory()) {
|
|
console.error(`Error: "${positionalArgs[0]}" is not a valid directory.`);
|
|
process.exit(2);
|
|
}
|
|
skillDirs = [target];
|
|
} else {
|
|
// Discover all skills
|
|
skillDirs = discoverSkillDirs([SRC_DIR]);
|
|
}
|
|
|
|
if (skillDirs.length === 0) {
|
|
console.error('No skill directories found.');
|
|
process.exit(2);
|
|
}
|
|
|
|
// Validate each skill
|
|
const results = [];
|
|
for (const skillDir of skillDirs) {
|
|
const findings = validateSkill(skillDir);
|
|
results.push({ skillDir, findings });
|
|
}
|
|
|
|
// Format output
|
|
const { output, hasHighPlus } = JSON_OUTPUT ? formatJson(results) : formatHumanReadable(results);
|
|
console.log(output);
|
|
|
|
// Exit code
|
|
if (STRICT && hasHighPlus) {
|
|
process.exit(1);
|
|
}
|
|
}
|
|
|
|
// --- Exports (for testing) ---
|
|
module.exports = { parseFrontmatter, parseFrontmatterMultiline, validateSkill, discoverSkillDirs };
|