opensource/awesome-llm-apps
3
0
Fork 0
mirror of https://github.com/Shubhamsaboo/awesome-llm-apps.git synced 2026-08-28 19:11:53 +08:00
Code Issues Packages Projects Releases Wiki Activity

ci: adopt addyosmani/agent-skills' full tier model — no reinvented wheels

Browse Source 
Tier 2 (trigger & routing) is now automated in CI instead of a manual
protocol: run_trigger_evals.py checks that positive prompts clear the
near-miss negatives on description vocabulary alone, and — once the
catalog has 2+ skills — that positives rank their own skill first and
no two descriptions near-collide. Reasoning-triggered cases (necromancer
mode) are marked lexical:false and covered by tier 3.

Tier 3 (behavioral) adopts skill-creator's evals.json schema verbatim,
so Anthropic's run_eval.py, benchmarking, and eval viewer work against
our files unmodified. project-graveyard's four behavioral cases written
in that schema.

The evals README tier table now uses Addy's names for Addy's tiers, and
marks the two tiers that are ours because our skills ship executable
code and his don't (security scan, deterministic script evals).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Shubhamsaboo
2026-07-08 00:20:22 -07:00
parent c2ad7a4603
commit 0aa5126bf2
5 changed files with 215 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/skill-evals.yml
+11
View File
@@ -65,3 +65,14 @@ jobs:
python-version: "3.11"
- name: Scan all skills for supply-chain and injection patterns
run: python3 awesome_agent_skills/evals/tools/skill_scanner.py awesome_agent_skills
trigger-routing:
name: trigger & routing
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Positives must clear near-misses on description vocabulary
run: python3 awesome_agent_skills/evals/tools/run_trigger_evals.py
awesome_agent_skills/evals/README.md
+20 -16
View File
@@ -5,32 +5,36 @@ every change after. Layout: one folder per skill, `evals/<skill-name>/`,
mirroring the skill's name. These files never ship in an install; the skill
folders contain only what runs at runtime.
Two tiers (vocabulary borrowed from
[addyosmani/agent-skills](https://github.com/addyosmani/agent-skills/tree/main/evals),
which formalized the pattern):
The tier model follows
[addyosmani/agent-skills](https://github.com/addyosmani/agent-skills/tree/main/evals)
— same names, same jobs — plus two tiers of our own, because skills here ship
executable code and his don't:
| Tier | What it checks | Runs | Cost |
|---|---|---|---|
| 0. Structural | Every SKILL.md obeys the spec: frontmatter rules, name==dir, no unfilled placeholders, no text-only prompt dumps (`tools/skill_lint.py --strict`) | CI on every PR | Free, ~1s |
| 0b. Security | No install lures, undeclared network calls, credential access, or obfuscated payloads in any skill (`tools/skill_scanner.py`) | CI on every PR | Free, ~1s |
| 1. Deterministic | The skill's scripts do what they claim — every classifier, edge case, and output shape, asserted against synthetic fixtures | CI on every PR, and by hand before installing | Free, ~10s |
| 2. Trigger | The skill activates on the prompts it should and stays quiet on near-misses | By hand, in an agent session (needs a model) | Tokens |
The tier 0 tools live in [`tools/`](tools/) — repo-side quality tooling, not
installable skills.
| 1. Structural | Frontmatter, naming, name==dir, unfilled placeholders, text-only prompt dumps (`tools/skill_lint.py --strict`) | CI | Free |
| 1b. Security *(ours)* | Install lures, undeclared network calls, credential access, obfuscated payloads (`tools/skill_scanner.py`) | CI | Free |
| 2. Trigger & routing | Positive prompts clear near-miss negatives on description vocabulary; with 2+ skills, positives rank their own skill first and no two descriptions near-collide (`tools/run_trigger_evals.py`) | CI | Free |
| 2b. Deterministic scripts *(ours)* | The skill's bundled scripts do what they claim — every classifier, edge case, and output shape against synthetic fixtures (`<skill>/test_*.py`) | CI | Free, ~10s |
| 3. Behavioral | An agent following the skill satisfies its `expectations[]` — `evals.json` uses [skill-creator's schema](https://github.com/anthropics/skills/tree/main/skills/skill-creator) verbatim, so its `run_eval.py`, benchmarking, and eval viewer work against our files unmodified | On demand | Tokens |
## Running
```bash
# Tier 1 — deterministic, no dependencies beyond git + Python
# Tiers 1–2b, exactly what CI runs — deterministic, git + Python only
python3 awesome_agent_skills/evals/tools/skill_lint.py awesome_agent_skills/project-graveyard --strict
python3 awesome_agent_skills/evals/tools/skill_scanner.py awesome_agent_skills
python3 awesome_agent_skills/evals/tools/run_trigger_evals.py
python3 awesome_agent_skills/evals/project-graveyard/test_graveyard.py
```
Tier 2 is a manual protocol for now: each skill's `trigger-cases.json` lists
the prompts that must activate it (and the near-misses that must not). Install
the skill in a fresh agent session, paste each prompt, check activation
matches `should_trigger`. Re-run whenever a skill's `description` changes —
that field is the entire trigger surface.
Tier 3 is on demand and spends tokens: each skill's `evals.json` is in
skill-creator's schema, so run it with Anthropic's own tooling (install the
skill-creator plugin and point `run_eval.py` at the file), or by hand — fresh
agent session, paste each prompt, grade against `expectations[]`. Cases marked
`lexical: false` in `trigger-cases.json` (reasoning-triggered, e.g. necromancer
mode) are only covered here. Re-run tier 3 whenever `SKILL.md` behavior
changes; re-run tier 2 whenever a `description` changes.
## Track record
awesome_agent_skills/evals/project-graveyard/evals.json
+48
View File
@@ -0,0 +1,48 @@
{
"skill_name": "project-graveyard",
"evals": [
{
"id": 1,
"prompt": "run the graveyard on my machine, my projects are in ~/dev and ~/hacks",
"expected_output": "A tombstone report over ~/dev and ~/hacks with census, causes of death, patterns, and exactly one resurrection pick.",
"expectations": [
"The skill ran scripts/graveyard.py with the user-provided roots and no others",
"The report includes a census, per-corpse causes with evidence, and a patterns section",
"Exactly one project is proposed for resurrection, with a plan of at most 7 steps ending at shipped",
"The agent offered to start step 1 in the current session"
]
},
{
"id": 2,
"prompt": "I have like 30 half-finished side projects and it's honestly depressing. why do I never finish anything?",
"expected_output": "A scan followed by a patterns-led answer (death timing, shiny-object chains), empathetic but evidence-based, with one resurrection offer.",
"expectations": [
"The agent asked where projects live (or confirmed roots) before scanning broadly",
"The answer leads with measured patterns, not generic productivity advice",
"No shaming language; epitaphs punch at the pattern, not the person",
"One resurrection candidate offered, not a list"
]
},
{
"id": 3,
"prompt": "of all my old abandoned repos, which one is actually worth finishing?",
"expected_output": "Pulse-ranked recommendation of a single project, justified by evidence and a world-check.",
"expectations": [
"The skill used the pulse ranking from the scanner rather than guessing",
"The agent read the top candidate's README/code before recommending",
"The recommendation cites what got easier since the project died (world-check)",
"Exactly one project is recommended"
]
},
{
"id": 4,
"prompt": "I want to build a newsletter analytics dashboard this weekend, can you scaffold it?",
"expected_output": "Before scaffolding, the agent checks the graveyard for a prior attempt at the same idea and mentions it once if found; never blocks the new project.",
"expectations": [
"The graveyard (state file or fresh scan) was consulted before any scaffolding",
"If a prior attempt exists, it is mentioned exactly once with its state and a resurrect-or-restart choice",
"The user's new project is not blocked or argued against"
]
}
]
}
awesome_agent_skills/evals/project-graveyard/trigger-cases.json
+4 -3
View File
@@ -1,6 +1,6 @@
{
"skill": "project-graveyard",
"purpose": "Trigger-behavior spec — the half of testing that needs an agent. If you change the SKILL.md description, install the skill in a fresh session, paste each prompt, and check activation matches should_trigger. The other half is executable: `python3 test_graveyard.py` (this directory) builds a synthetic graveyard and asserts every classifier, redaction, state, and relapse detection deterministically.",
"purpose": "Trigger-behavior spec, two consumers: (1) CI runs tools/run_trigger_evals.py against these cases lexically \u2014 positives must clear near-misses on description vocabulary alone; (2) after any description change, also verify by hand in a fresh agent session. Cases with lexical:false trigger via reasoning, not vocabulary, and are covered by the behavioral tier (evals.json).",
"cases": [
{
"id": "direct-invocation",
@@ -24,7 +24,8 @@
"id": "necromancer-mode",
"prompt": "I want to build a newsletter analytics dashboard this weekend, can you scaffold it?",
"should_trigger": true,
"assert": "Before scaffolding, checks the graveyard for prior attempts at the same idea; if found, mentions it once and lets the user choose; never blocks the new project."
"assert": "Before scaffolding, checks the graveyard for prior attempts at the same idea; if found, mentions it once and lets the user choose; never blocks the new project.",
"lexical": false
},
{
"id": "near-miss-disk-cleanup",
@@ -45,4 +46,4 @@
"assert": "Does not trigger; single-repo history analysis is plain git log reading, no scan needed."
}
]
}
}
awesome_agent_skills/evals/tools/run_trigger_evals.py
+132
View File
@@ -0,0 +1,132 @@
#!/usr/bin/env python3
"""
Tier 2 — trigger & routing. Deterministic, CI-safe, no model.
Pattern from addyosmani/agent-skills (evals/README.md there): a skill's
description is its entire trigger surface, so check it lexically —
positive prompts must score against their skill's description clearly
above the near-miss negatives, and (once the catalog has 2+ skills) must
rank their own skill first, with no two descriptions near-colliding.
This is a lexical approximation of routing. It cannot judge semantics —
that's the behavioral tier's job — but it catches the two failure modes
that dominate real trigger bugs: a description missing the vocabulary
users actually say, and an over-broad description that outranks the
right skill. A failure here usually means *fix the description*.
Cases live in evals/<skill>/trigger-cases.json. A case marked
"lexical": false is skipped here (it triggers via reasoning, e.g.
necromancer mode, not via description vocabulary) and is covered by the
behavioral tier instead.
python3 run_trigger_evals.py # checks every skill in the catalog
"""
import json
import math
import os
import re
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
SKILLS_ROOT = os.path.abspath(os.path.join(HERE, "..", ".."))
EVALS_ROOT = os.path.abspath(os.path.join(HERE, ".."))
STOP = set("""a an the and or of to in on for with is are was were be been it its this
that those these you your i me my we our they their he she his her do does did done
can could should would will just very really some any all not no yes if then than as
at by from into out up down over under again more most other own same so too s t don
""".split())
MARGIN = 1.15 # min positive score must beat max negative score by this factor
def tokens(text):
out = set()
for w in re.findall(r"[a-z0-9']+", text.lower()):
if w in STOP or len(w) < 3:
continue
# crude stemmer: enough to match "projects/project", "finished/finish"
for suf in ("ing", "ed", "es", "s"):
if w.endswith(suf) and len(w) - len(suf) >= 3:
w = w[: -len(suf)]
break
out.add(w)
return out
def description_of(skill_dir):
text = open(os.path.join(skill_dir, "SKILL.md")).read()
m = re.search(r"^description:\s*(.+?)^(?=[a-zA-Z-]+:|---)", text, re.S | re.M)
return m.group(1) if m else ""
def score(prompt_toks, desc_toks):
if not prompt_toks:
return 0.0
return len(prompt_toks & desc_toks) / math.sqrt(len(prompt_toks))
def main():
skills = {} # name -> description tokens
for entry in sorted(os.listdir(SKILLS_ROOT)):
if os.path.exists(os.path.join(SKILLS_ROOT, entry, "SKILL.md")):
skills[entry] = tokens(description_of(os.path.join(SKILLS_ROOT, entry)))
if not skills:
print("no skills found under %s" % SKILLS_ROOT)
return 1
failures = 0
# routing sanity across the catalog: no two descriptions near-collide
names = sorted(skills)
for i in range(len(names)):
for j in range(i + 1, len(names)):
a, b = skills[names[i]], skills[names[j]]
overlap = len(a & b) / max(1, min(len(a), len(b)))
if overlap > 0.5:
print("FAIL descriptions near-collide: %s vs %s (%.0f%% shared vocabulary)"
% (names[i], names[j], overlap * 100))
failures += 1
for name in names:
case_file = os.path.join(EVALS_ROOT, name, "trigger-cases.json")
if not os.path.exists(case_file):
print("WARN %s has no trigger-cases.json — add one" % name)
continue
cases = json.load(open(case_file))["cases"]
pos, neg = [], []
for c in cases:
if c.get("lexical") is False:
continue
s = score(tokens(c["prompt"]), skills[name])
(pos if c["should_trigger"] else neg).append((s, c["id"]))
# routing: with 2+ skills, a positive must rank its own skill first
if c["should_trigger"] and len(skills) > 1:
best = max(skills, key=lambda k: score(tokens(c["prompt"]), skills[k]))
if best != name:
print("FAIL %s: %r routes to %s instead" % (name, c["id"], best))
failures += 1
if pos and neg:
worst_pos, wp_id = min(pos)
best_neg, bn_id = max(neg)
if worst_pos <= best_neg * MARGIN:
print("FAIL %s: weakest positive %r (%.2f) does not clear strongest "
"near-miss %r (%.2f) — the description is missing vocabulary "
"users say, or a negative shares too much of it"
% (name, wp_id, worst_pos, bn_id, best_neg))
failures += 1
else:
print("PASS %s: %d positives clear %d near-misses "
"(weakest %.2f vs strongest %.2f)"
% (name, len(pos), len(neg), worst_pos, best_neg))
if failures:
print("\n%d failure(s)" % failures)
return 1
print("\ntrigger & routing: all clear (%d skill%s)" % (len(skills), "" if len(skills) == 1 else "s"))
return 0
if __name__ == "__main__":
sys.exit(main())