mirror of
https://github.com/HKUDS/CLI-Anything.git
synced 2026-08-29 07:30:51 +08:00
feat(cli-matrix): eco-first capability-based matrix, v2 schema + S2-S5 SKILLs
- Add docs/cli-matrix/matrix_registry.schema.md describing v2 capability-based registry shape (capabilities[], providers with kind/requires/cost/quality/ offline, recipes[], known_gaps[], decision rubric, suggest-to-user template). - Rewrite cli-hub-matrix/video-creation/SKILL.md and matrix_registry.json (S1) around capabilities + providers + recipes instead of linear stages. - Rename Vn -> Sn across cli-matrix-plan.md and test fixtures. - Reorder scenarios by current completeness; rewrite S2 knowledge-research, S3 3d-cad, S4 game-development, S5 image-design in v2 capability form with full SKILL.md files. - Add docs/cli-matrix/test-plans/video-creation.md with 13 long realistic end-to-end tasks as checkable todo lists, each exercising 5-9 capabilities. - Move cli-matrix-plan.md and matrix_registry.schema.md under docs/cli-matrix/. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -313,6 +313,8 @@ assets/gen_typing_gif.py
|
||||
/docs/scripts/**/*.pyc
|
||||
!/docs/hub/
|
||||
/docs/hub/registry-dates.json
|
||||
!/docs/cli-matrix/
|
||||
!/docs/cli-matrix/**
|
||||
!/notebooklm/
|
||||
/notebooklm/*
|
||||
/notebooklm/.*
|
||||
|
||||
@@ -0,0 +1,229 @@
|
||||
---
|
||||
name: cli-hub-matrix-3d-cad
|
||||
description: >-
|
||||
Capability-based multi-tool matrix for 3D modeling, CAD, point clouds, rendering,
|
||||
GPU debugging, and fabrication. Covers mesh/parametric/photogrammetry and the
|
||||
path from idea to printed part or game-ready asset.
|
||||
---
|
||||
|
||||
# 3D & CAD Matrix (S3 — v2 capability-based)
|
||||
|
||||
Scenario **S3**. Strongest technical scenario in the matrix by depth — Blender + FreeCAD + CloudCompare + RenderDoc is rare coverage. Structural gaps: **texturing** and **fabrication slicing**.
|
||||
|
||||
Schema: [`docs/cli-matrix/matrix_registry.schema.md`](../../docs/cli-matrix/matrix_registry.schema.md). Matrix plan: [`docs/cli-matrix/cli-matrix-plan.md`](../../docs/cli-matrix/cli-matrix-plan.md).
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
cli-hub matrix install 3d-cad
|
||||
cli-hub matrix info 3d-cad
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decision rubric
|
||||
|
||||
1. **Available & adequate** — ranked by quality then inverse cost.
|
||||
2. **Free-to-install** — Python libs / native binaries without credentials.
|
||||
3. **Harness / public CLI install** — when warranted.
|
||||
4. **Paid API escalation** — only when lower tiers can't meet the bar, env holds the key, or the user consents.
|
||||
|
||||
Offline context? Most of this matrix is offline-first; rendering and photogrammetry are the exceptions.
|
||||
|
||||
---
|
||||
|
||||
## Preflight
|
||||
|
||||
```bash
|
||||
cli-hub list --json
|
||||
python - <<'PY'
|
||||
import importlib.util
|
||||
for m in ("trimesh","pymeshlab","open3d","cadquery","build123d","solid",
|
||||
"pygltflib","laspy","pyvista","bpy","rawpy","pycam"):
|
||||
print(m, importlib.util.find_spec(m) is not None)
|
||||
PY
|
||||
for b in blender freecad meshroom colmap openmvg openmvs prusaslicer slic3r \
|
||||
cura CuraEngine apitrace vulkaninfo renderdoc OpenSCAD; do
|
||||
command -v "$b" >/dev/null && echo "$b: yes" || echo "$b: no"
|
||||
done
|
||||
for e in POLYCAM_API_KEY LUMA_API_KEY ONSHAPE_ACCESS_KEY FUSION360_TOKEN \
|
||||
OCTANE_TOKEN; do
|
||||
[ -n "${!e}" ] && echo "$e: set" || echo "$e: unset"
|
||||
done
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Suggest-to-user template
|
||||
|
||||
```
|
||||
To enable <capability> via <provider>, please set <ENV_VAR>.
|
||||
Cost: <cost notes>
|
||||
Quality: <quality tier>
|
||||
Reply 'skip' to fall back to <next provider>.
|
||||
```
|
||||
|
||||
Example: *To enable cloud photogrammetry via Luma AI, please set `LUMA_API_KEY`. Cost: metered per-scan. Quality: sota. Reply 'skip' to fall back to local `Meshroom` (slower, free).*
|
||||
|
||||
---
|
||||
|
||||
## Capabilities
|
||||
|
||||
### `model.mesh` — mesh modeling / editing
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-blender` | harness-cli | Blender installed | free | sota | yes |
|
||||
| `trimesh` | python | pkg | free | high | yes |
|
||||
| `pymeshlab` | python | pkg | free | high | yes |
|
||||
| `open3d` | python | pkg | free | high | yes |
|
||||
| `pygalmesh` | python | pkg + CGAL | free | high | yes |
|
||||
|
||||
### `model.parametric` — parametric CAD
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-freecad` | harness-cli | FreeCAD installed | free | sota | yes |
|
||||
| `cadquery` | python | pkg | free | high | yes |
|
||||
| `build123d` | python | pkg | free | high | yes |
|
||||
| `OpenSCAD` | native | binary | free | high | yes |
|
||||
| `SolidPython` | python | pkg + OpenSCAD | free | good | yes |
|
||||
| Onshape API | api | `ONSHAPE_ACCESS_KEY` | paid | sota | no |
|
||||
| Fusion 360 API | api | `FUSION360_TOKEN` | paid | sota | no |
|
||||
|
||||
### `model.sculpt` — digital sculpting
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-blender` (sculpt mode, scripted) | harness-cli | Blender | free | good | yes |
|
||||
| ZBrush / Nomad | — | — | — | — | — |
|
||||
|
||||
Known gap — agent-driven sculpting is impractical beyond Blender scripts.
|
||||
|
||||
### `pointcloud.process` — point cloud ingest / clean / register
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-cloudcompare` | harness-cli | CloudCompare | free | sota | yes |
|
||||
| `cli-anything-cloudanalyzer` | harness-cli | harness installed | free | high | yes |
|
||||
| `open3d` | python | pkg | free | high | yes |
|
||||
| `laspy` | python | pkg | free | high | yes |
|
||||
| `pdal` | native | binary | free | sota | yes |
|
||||
| `pyvista` | python | pkg | free | high | yes |
|
||||
|
||||
### `photogrammetry.reconstruct` — images → 3D mesh
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `Meshroom` | native | binary | free | high | yes |
|
||||
| `colmap` | native | binary | free | high | yes |
|
||||
| `OpenMVG` + `OpenMVS` | native | binaries | free | high | yes |
|
||||
| RealityCapture | native | commercial license | paid | sota | yes |
|
||||
| Polycam API | api | `POLYCAM_API_KEY` | metered | sota | no |
|
||||
| Luma AI | api | `LUMA_API_KEY` | metered | sota | no |
|
||||
|
||||
### `material.texture` — PBR texturing
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-blender` (shader nodes, scripted) | harness-cli | Blender | free | good | yes |
|
||||
| `bpy` PBR synthesis | python | pkg | free | good | yes |
|
||||
| `Pillow` procedural maps | python | pkg | free | basic | yes |
|
||||
| Substance 3D / Polycam Materials / Adobe Sampler | — | — | — | — | — |
|
||||
|
||||
**Known gap** — no agent-native path approaches Substance quality.
|
||||
|
||||
### `render.preview` — fast viewport renders
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-blender` (Eevee) | harness-cli | Blender | free | high | yes |
|
||||
| `cli-anything-godot` (viewport screenshot) | harness-cli | Godot | free | good | yes |
|
||||
|
||||
### `render.offline` — path-traced / final renders
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-blender` (Cycles) | harness-cli | Blender + GPU | free | sota | yes |
|
||||
| `bpy` + Cycles | python | pkg + GPU | free | sota | yes |
|
||||
| Octane Cloud | api | `OCTANE_TOKEN` | paid | sota | no |
|
||||
| RenderMan on offline farm | — | license + farm | paid | sota | depends |
|
||||
|
||||
### `gpu.debug` — capture / inspect GPU frames
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-renderdoc` | harness-cli | RenderDoc installed | free | sota | yes |
|
||||
| `apitrace` | native | binary | free | high | yes |
|
||||
| `vulkan-tools` | native | binary | free | good | yes |
|
||||
|
||||
### `fabricate.slice` — 3D print slicer (mesh → gcode)
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `PrusaSlicer` CLI | native | binary | free | sota | yes |
|
||||
| `CuraEngine` | native | binary | free | high | yes |
|
||||
| `slic3r` | native | binary | free | good | yes |
|
||||
| Bambu Studio / OrcaSlicer (API-driven) | — | — | — | — | — |
|
||||
|
||||
**Known gap** — no agent-native harness, though the natives are scriptable.
|
||||
|
||||
### `fabricate.cam` — CAM / CNC toolpaths
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-freecad` (Path workbench, scripted) | harness-cli | FreeCAD | free | good | yes |
|
||||
| `pycam` | python | pkg | free | good | yes |
|
||||
| `kiri:moto` | native | binary | free | good | yes |
|
||||
| Fusion 360 CAM / Onshape CAM | api | tokens | paid | sota | no |
|
||||
|
||||
### `export.engine` — handoff to game / viz engine
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-godot` | harness-cli | Godot | free | high | yes |
|
||||
| `cli-anything-blender` (glTF/FBX export) | harness-cli | Blender | free | sota | yes |
|
||||
| `pygltflib` | python | pkg | free | high | yes |
|
||||
| `trimesh` glTF writer | python | pkg | free | good | yes |
|
||||
| Unity Asset Pipeline / Unreal Datasmith | — | — | — | — | — |
|
||||
|
||||
---
|
||||
|
||||
## Recipes
|
||||
|
||||
- **`printable-part`** — parametric CAD → mesh → slice → gcode.
|
||||
Uses: `model.parametric`, `model.mesh`, `fabricate.slice`.
|
||||
|
||||
- **`game-asset`** — model → texture → export to engine.
|
||||
Uses: `model.mesh`, `material.texture`, `export.engine`, (optional `render.preview`).
|
||||
|
||||
- **`reality-scan`** — photos → photogrammetry → clean mesh → export.
|
||||
Uses: `photogrammetry.reconstruct`, `pointcloud.process`, `model.mesh`, `export.engine`.
|
||||
|
||||
- **`cnc-part`** — parametric part → CAM toolpath → gcode.
|
||||
Uses: `model.parametric`, `fabricate.cam`.
|
||||
|
||||
- **`product-viz-still`** — CAD → import → scene → offline render.
|
||||
Uses: `model.parametric`, `model.mesh`, `material.texture`, `render.offline`.
|
||||
|
||||
- **`gpu-bug-repro`** — capture frame → analyze → patch shader.
|
||||
Uses: `gpu.debug`, `render.preview`.
|
||||
|
||||
---
|
||||
|
||||
## Known gaps
|
||||
|
||||
- **`material.texture`** — no open equivalent to Substance; Blender shader nodes are the current ceiling.
|
||||
- **`fabricate.slice`** — PrusaSlicer/Cura natives work but want a harness-level wrapper.
|
||||
- **`model.sculpt`** — impractical for agents today.
|
||||
- **Unity / Unreal export** — only Godot covered on the harness side.
|
||||
|
||||
---
|
||||
|
||||
## Agent guidance
|
||||
|
||||
- **Parametric first, mesh later.** Start in `model.parametric` when the object has dimensions; `model.mesh` when it's organic.
|
||||
- **Always bake before export.** When handing off to `export.engine`, apply modifiers and bake materials — runtime engines don't read Blender shader graphs.
|
||||
- **Check manifold before slicing.** `fabricate.slice` fails silently on non-manifold meshes; use `trimesh.Trimesh.is_volume` or pymeshlab's manifold check first.
|
||||
- **Offline by default.** This matrix is the most offline-capable — API escalation is rarely needed except for photogrammetry-at-scale.
|
||||
- **Workspace discipline.** Keep `.FCStd` / `.blend` / `.ply` / `.stl` / `.gcode` in one project folder so the chain of transformations is reproducible.
|
||||
@@ -0,0 +1,210 @@
|
||||
---
|
||||
name: cli-hub-matrix-game-development
|
||||
description: >-
|
||||
Capability-based multi-tool matrix for game development: engine, 3D/2D/audio assets,
|
||||
AI-generated assets, agent playtesting, packaging, store publishing, and telemetry.
|
||||
---
|
||||
|
||||
# Game Development Matrix (S4 — v2 capability-based)
|
||||
|
||||
Scenario **S4**. All asset-creation capabilities covered via the S3/S5 stacks; the structural gap is **distribution** (Steam/itch/console stores) and alternative engines (Unity/Unreal).
|
||||
|
||||
Schema: [`docs/cli-matrix/matrix_registry.schema.md`](../../docs/cli-matrix/matrix_registry.schema.md). Matrix plan: [`docs/cli-matrix/cli-matrix-plan.md`](../../docs/cli-matrix/cli-matrix-plan.md).
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
cli-hub matrix install game-development
|
||||
cli-hub matrix info game-development
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decision rubric
|
||||
|
||||
1. **Available & adequate** — ranked by quality then inverse cost.
|
||||
2. **Free-to-install** — Python libs / native binaries without credentials.
|
||||
3. **Harness / public CLI install** — when warranted.
|
||||
4. **Paid API escalation** — only when lower tiers can't meet the bar, env holds the key, or the user consents.
|
||||
|
||||
---
|
||||
|
||||
## Preflight
|
||||
|
||||
```bash
|
||||
cli-hub list --json
|
||||
python - <<'PY'
|
||||
import importlib.util
|
||||
for m in ("pygame","arcade","panda3d","ursina","trimesh","pygltflib","PIL",
|
||||
"pydub","librosa","music21","mido","abjad","diffusers","replicate",
|
||||
"posthog","gymnasium"):
|
||||
print(m, importlib.util.find_spec(m) is not None)
|
||||
PY
|
||||
for b in godot blender aseprite butler steamcmd ffmpeg pyinstaller; do
|
||||
command -v "$b" >/dev/null && echo "$b: yes" || echo "$b: no"
|
||||
done
|
||||
for e in STEAM_USERNAME STEAM_PASSWORD ITCH_API_KEY EPIC_DEV_TOKEN \
|
||||
PLAY_CONSOLE_KEY APP_STORE_CONNECT_KEY \
|
||||
REPLICATE_API_TOKEN OPENAI_API_KEY STABILITY_API_KEY IDEOGRAM_API_KEY \
|
||||
POSTHOG_API_KEY GAMEANALYTICS_KEY PLAYFAB_TITLE_KEY SENTRY_DSN; do
|
||||
[ -n "${!e}" ] && echo "$e: set" || echo "$e: unset"
|
||||
done
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Suggest-to-user template
|
||||
|
||||
```
|
||||
To enable <capability> via <provider>, please set <ENV_VAR>.
|
||||
Cost: <cost notes>
|
||||
Quality: <quality tier>
|
||||
Reply 'skip' to fall back to <next provider>.
|
||||
```
|
||||
|
||||
Example: *To publish to Steam, please install `steamcmd` and set `STEAM_USERNAME` + `STEAM_PASSWORD` (a dedicated build account is strongly recommended). Cost: one-time $100 Steamworks fee. Reply 'skip' to fall back to itch.io via `butler` which only needs `ITCH_API_KEY`.*
|
||||
|
||||
---
|
||||
|
||||
## Capabilities
|
||||
|
||||
### `game.engine` — engine authoring / project manipulation
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-godot` | harness-cli | Godot installed | free | sota | yes |
|
||||
| `pygame` | python | pkg | free | good | yes |
|
||||
| `arcade` | python | pkg | free | good | yes |
|
||||
| `panda3d` | python | pkg | free | high | yes |
|
||||
| `ursina` | python | pkg | free | good | yes |
|
||||
| Unity / Unreal | — | — | — | — | — |
|
||||
|
||||
### `asset.3d` — 3D game assets
|
||||
|
||||
See S3 matrix. Primary: `cli-anything-blender`, `cli-anything-freecad`, `trimesh`, `pygltflib`.
|
||||
|
||||
### `asset.2d` — sprites, tiles, UI art
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-krita` | harness-cli | Krita installed | free | sota | yes |
|
||||
| `cli-anything-inkscape` | harness-cli | Inkscape installed | free | sota | yes |
|
||||
| `Pillow` | python | pkg | free | good | yes |
|
||||
| `aseprite` | native | binary (commercial) | paid | sota | yes |
|
||||
| `pyxel` | python | pkg | free | good | yes |
|
||||
| OpenAI GPT-Image-1 | api | `OPENAI_API_KEY` | metered | sota | no |
|
||||
| Scenario GG | api | `SCENARIO_API_KEY` | paid | sota | no |
|
||||
| Leonardo AI | api | `LEONARDO_API_KEY` | paid | high | no |
|
||||
|
||||
### `asset.audio` — sfx and gameplay audio
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-audacity` | harness-cli | Audacity | free | high | yes |
|
||||
| `pydub` / `librosa` / `soundfile` | python | pkg | free | good | yes |
|
||||
| `sfxr` / `jsfxr` | native | binary | free | good | yes |
|
||||
| ElevenLabs SFX | api | `ELEVENLABS_API_KEY` | paid | sota | no |
|
||||
| Mubert | api | `MUBERT_API_KEY` | metered | high | no |
|
||||
|
||||
### `asset.notation` — music notation / score export
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-musescore` | harness-cli | MuseScore installed | free | sota | yes |
|
||||
| `music21` | python | pkg | free | high | yes |
|
||||
| `mido` | python | pkg | free | good | yes |
|
||||
| `abjad` | python | pkg | free | high | yes |
|
||||
|
||||
### `ai.gen-asset` — AI-generated art/audio for games
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-comfyui` | harness-cli | ComfyUI installed | free | high | yes |
|
||||
| `jimeng` | public-cli | bin | metered | high | no |
|
||||
| `diffusers` | python | pkg + weights | free | good | yes |
|
||||
| `replicate` | python | `REPLICATE_API_TOKEN` | metered | high | no |
|
||||
| Scenario GG | api | `SCENARIO_API_KEY` | paid | sota | no |
|
||||
| Leonardo AI | api | `LEONARDO_API_KEY` | paid | high | no |
|
||||
| OpenAI GPT-Image-1 | api | `OPENAI_API_KEY` | metered | sota | no |
|
||||
| Stability | api | `STABILITY_API_KEY` | metered | high | no |
|
||||
|
||||
### `playtest.agent` — agent-driven play / testing
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-slay-the-spire-ii` | harness-cli | StS II + bridge mod | free | sota | yes |
|
||||
| `gymnasium` envs | python | pkg | free | high | yes |
|
||||
| Headless game + scripted input (e.g. Godot `--headless` with GUT) | harness-cli | Godot | free | good | yes |
|
||||
|
||||
Distinctive capability — very few agent stacks can drive a live GUI game loop.
|
||||
|
||||
### `build.package` — package a game binary / bundle
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-godot` (export presets) | harness-cli | Godot export templates | free | sota | yes |
|
||||
| `pyinstaller` (pygame/arcade) | python | pkg | free | good | yes |
|
||||
| `butler` (itch packager) | native | binary | free | high | yes |
|
||||
|
||||
### `publish.store` — upload a build to a store
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `steamcmd` | native | binary + Steam account + app ID | paid (fee) | sota | no |
|
||||
| `butler` (itch.io) | native | binary + `ITCH_API_KEY` | free | high | no |
|
||||
| Epic Games Services | api | `EPIC_DEV_TOKEN` | paid | high | no |
|
||||
| Google Play Developer | api | `PLAY_CONSOLE_KEY` + service account | paid | high | no |
|
||||
| App Store Connect | api | `APP_STORE_CONNECT_KEY` | paid | high | no |
|
||||
|
||||
**Known gap** — no harness-level abstraction; escalate via the suggest-to-user template with platform-specific preflight.
|
||||
|
||||
### `telemetry.ingest` — gameplay analytics / crash reports
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `sentry` | public-cli | bin + `SENTRY_DSN` | free-paid | high | no |
|
||||
| `posthog` | python | pkg + `POSTHOG_API_KEY` | free-paid | high | no |
|
||||
| GameAnalytics | api | `GAMEANALYTICS_KEY` | free | good | no |
|
||||
| Unity Analytics | api | Unity project token | metered | high | no |
|
||||
| PlayFab | api | `PLAYFAB_TITLE_KEY` | metered | high | no |
|
||||
|
||||
---
|
||||
|
||||
## Recipes
|
||||
|
||||
- **`godot-jam-game`** — minimal end-to-end for a game jam.
|
||||
Uses: `game.engine`, `asset.2d`, `asset.audio`, `build.package`, (optional `publish.store`).
|
||||
|
||||
- **`agent-bot`** — train / evaluate an agent against an existing game.
|
||||
Uses: `playtest.agent`, `telemetry.ingest`.
|
||||
|
||||
- **`ai-indie-short`** — AI-generated art + simple engine loop.
|
||||
Uses: `ai.gen-asset`, `asset.2d`, `game.engine`, `asset.audio`, `build.package`.
|
||||
|
||||
- **`steam-release`** — ship to Steam with crash reporting.
|
||||
Uses: `build.package`, `publish.store`, `telemetry.ingest`.
|
||||
|
||||
- **`music-driven-game`** — composed score integrated with gameplay.
|
||||
Uses: `asset.notation`, `asset.audio`, `game.engine`.
|
||||
|
||||
- **`reality-to-game`** — photogrammetry asset → import → ship.
|
||||
Uses: (S3 `photogrammetry.reconstruct`), `asset.3d`, `game.engine`, `build.package`.
|
||||
|
||||
---
|
||||
|
||||
## Known gaps
|
||||
|
||||
- **`publish.store`** — Steam/itch/console wrappers are native-only; escalate with explicit credentials.
|
||||
- **`game.engine` for Unity/Unreal** — only Godot harness exists.
|
||||
- **`telemetry.ingest`** — Sentry works; no first-party for GameAnalytics/PlayFab.
|
||||
- **`asset.notation`** — uniquely strong (MuseScore) but rarely used in practice; keep as a differentiator.
|
||||
|
||||
---
|
||||
|
||||
## Agent guidance
|
||||
|
||||
- **Start in Godot** unless the user has explicitly asked for Unity/Unreal — it's the only engine we harness.
|
||||
- **Keep asset pipelines deterministic** — re-runnable seeds for `ai.gen-asset`, versioned source `.aseprite`/`.krita` files, scriptable Blender bakes.
|
||||
- **`playtest.agent` needs a bridge.** Most commercial games lack API access; agents should ask the user if modding is permitted before attempting.
|
||||
- **Version builds immediately.** `build.package` output goes into `builds/<version>/` with a manifest so `publish.store` has deterministic uploads.
|
||||
- **Don't auto-publish.** `publish.store` must always be gated behind explicit user confirmation — uploading a broken build is expensive to roll back.
|
||||
@@ -0,0 +1,215 @@
|
||||
---
|
||||
name: cli-hub-matrix-image-design
|
||||
description: >-
|
||||
Capability-based multi-tool matrix for image and graphic design: AI generation,
|
||||
raster/vector editing, UI mockups, diagrams, upscaling, photo library, and publishing.
|
||||
---
|
||||
|
||||
# Image & Graphic Design Matrix (S5 — v2 capability-based)
|
||||
|
||||
Scenario **S5**. Deepest creation coverage after S2 — raster (GIMP + Krita), vector (Inkscape), AI (ComfyUI + Jimeng + 7 APIs), diagram (drawio + mermaid). **Figma** is the headline gap (also affects S7).
|
||||
|
||||
Schema: [`docs/cli-matrix/matrix_registry.schema.md`](../../docs/cli-matrix/matrix_registry.schema.md). Matrix plan: [`docs/cli-matrix/cli-matrix-plan.md`](../../docs/cli-matrix/cli-matrix-plan.md).
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
cli-hub matrix install image-design
|
||||
cli-hub matrix info image-design
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decision rubric
|
||||
|
||||
1. **Available & adequate** — ranked by quality then inverse cost.
|
||||
2. **Free-to-install** — Python libs / native binaries without credentials.
|
||||
3. **Harness / public CLI install** — when warranted.
|
||||
4. **Paid API escalation** — only when lower tiers can't meet the bar, env holds the key, or the user consents. AI image is the most paid-API-diverse capability — ask which provider the user prefers.
|
||||
|
||||
Offline context? GIMP, Krita, Inkscape, and local `diffusers` cover almost everything offline.
|
||||
|
||||
---
|
||||
|
||||
## Preflight
|
||||
|
||||
```bash
|
||||
cli-hub list --json
|
||||
python - <<'PY'
|
||||
import importlib.util
|
||||
for m in ("PIL","cv2","skimage","rembg","svgwrite","cairosvg","svgpathtools","vpype",
|
||||
"diffusers","replicate","rawpy"):
|
||||
print(m, importlib.util.find_spec(m) is not None)
|
||||
PY
|
||||
for b in inkscape gimp krita drawio magick convert exiftool upscayl darktable-cli \
|
||||
rawtherapee-cli graphviz dot mermaid plantuml hugo mkdocs; do
|
||||
command -v "$b" >/dev/null && echo "$b: yes" || echo "$b: no"
|
||||
done
|
||||
for e in OPENAI_API_KEY STABILITY_API_KEY IDEOGRAM_API_KEY RECRAFT_API_KEY \
|
||||
GOOGLE_API_KEY MIDJOURNEY_TOKEN FIGMA_TOKEN CANVA_API_KEY PENPOT_TOKEN \
|
||||
REPLICATE_API_TOKEN TOPAZ_API_KEY REMOVE_BG_API_KEY FIREFLY_TOKEN; do
|
||||
[ -n "${!e}" ] && echo "$e: set" || echo "$e: unset"
|
||||
done
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Suggest-to-user template
|
||||
|
||||
```
|
||||
To enable <capability> via <provider>, please set <ENV_VAR>.
|
||||
Cost: <cost notes>
|
||||
Quality: <quality tier>
|
||||
Reply 'skip' to fall back to <next provider>.
|
||||
```
|
||||
|
||||
Examples:
|
||||
- *To enable sota AI image generation via OpenAI GPT-Image-1, please set `OPENAI_API_KEY`. Cost: metered per-image. Quality: sota for photoreal + text rendering. Reply 'skip' to fall back to local `comfyui` (FLUX/SDXL).*
|
||||
- *To round-trip UI mockups with Figma, please set `FIGMA_TOKEN`. Cost: free tier works for most users. Quality: sota (design system aware). Reply 'skip' to fall back to `sketch` or `drawio` wireframes.*
|
||||
|
||||
---
|
||||
|
||||
## Capabilities
|
||||
|
||||
### `visual.generate` — AI image generation
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-comfyui` | harness-cli | ComfyUI + weights | free | high | yes |
|
||||
| `jimeng` | public-cli | `dreamina` bin | metered | high | no |
|
||||
| `diffusers` (SD/SDXL/FLUX) | python | pkg + weights + GPU | free | good-high | yes |
|
||||
| `replicate` | python | `REPLICATE_API_TOKEN` | metered | high | no |
|
||||
| OpenAI GPT-Image-1 | api | `OPENAI_API_KEY` | metered | sota | no |
|
||||
| Ideogram | api | `IDEOGRAM_API_KEY` | metered | sota | no |
|
||||
| Stability AI | api | `STABILITY_API_KEY` | metered | high | no |
|
||||
| Recraft | api | `RECRAFT_API_KEY` | paid | sota | no |
|
||||
| Google Imagen / Nano Banana | api | `GOOGLE_API_KEY` | metered | high | no |
|
||||
| Midjourney (Discord bridge) | api | `MIDJOURNEY_TOKEN` | paid | sota | no |
|
||||
| Adobe Firefly | api | `FIREFLY_TOKEN` | paid | high | no |
|
||||
|
||||
### `visual.edit.raster` — raster editing
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-gimp` | harness-cli | GIMP installed | free | sota | yes |
|
||||
| `cli-anything-krita` | harness-cli | Krita installed | free | sota | yes |
|
||||
| `Pillow` | python | pkg | free | good | yes |
|
||||
| `opencv-python` | python | pkg | free | high | yes |
|
||||
| `scikit-image` | python | pkg | free | high | yes |
|
||||
| `rembg` (background removal) | python | pkg | free | high | yes |
|
||||
| `remove.bg` | api | `REMOVE_BG_API_KEY` | metered | sota | no |
|
||||
| Adobe Firefly | api | `FIREFLY_TOKEN` | paid | sota | no |
|
||||
| Cutout.pro | api | token | metered | high | no |
|
||||
|
||||
### `visual.edit.vector` — vector editing
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-inkscape` | harness-cli | Inkscape installed | free | sota | yes |
|
||||
| `svgwrite` | python | pkg | free | good | yes |
|
||||
| `cairosvg` | python | pkg | free | good | yes |
|
||||
| `svgpathtools` | python | pkg | free | high | yes |
|
||||
| `vpype` (vector post-processing) | python | pkg | free | high | yes |
|
||||
|
||||
### `visual.mockup` — UI / mockup / wireframe
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-sketch` | harness-cli | Sketch installed (macOS) | paid app | sota | yes |
|
||||
| `cli-anything-drawio` | harness-cli | drawio installed | free | high | yes |
|
||||
| `drawio-desktop` | native | binary | free | high | yes |
|
||||
| `penpot` | native | binary / self-host | free | high | yes |
|
||||
| Figma API | api | `FIGMA_TOKEN` | free tier | sota | no |
|
||||
| Canva API | api | `CANVA_API_KEY` | paid | high | no |
|
||||
| Penpot API | api | `PENPOT_TOKEN` | free | high | no |
|
||||
|
||||
**Known gap** — Figma is the cross-scenario gap (S5 and S7).
|
||||
|
||||
### `visual.diagram` — diagrams / flowcharts
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-drawio` | harness-cli | drawio | free | sota | yes |
|
||||
| `cli-anything-mermaid` | harness-cli | mermaid-cli | free | high | yes |
|
||||
| `graphviz` (`dot`) | native | binary | free | high | yes |
|
||||
| `plantuml` | native | binary + java | free | high | yes |
|
||||
|
||||
### `visual.upscale` — super-resolution
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `Real-ESRGAN` | python | pkg + weights | free | high | yes |
|
||||
| `GFPGAN` (faces) | python | pkg + weights | free | high | yes |
|
||||
| `upscayl` | native | binary | free | high | yes |
|
||||
| Topaz | api | `TOPAZ_API_KEY` | paid | sota | no |
|
||||
| Replicate upscalers | python | `REPLICATE_API_TOKEN` | metered | high | no |
|
||||
|
||||
### `photo.library` — DAM / photo organization
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `exiftool` + `Pillow` | native+python | binary+pkg | free | high | yes |
|
||||
| `digikam` | native | binary | free | high | yes |
|
||||
| `photoprism` | native | binary / self-host | free | high | yes |
|
||||
| Google Photos API | api | OAuth | free tier | high | no |
|
||||
|
||||
### `photo.develop` — RAW processing
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `rawpy` | python | pkg | free | high | yes |
|
||||
| `darktable-cli` | native | binary | free | sota | yes |
|
||||
| `rawtherapee-cli` | native | binary | free | sota | yes |
|
||||
| Adobe Lightroom API | api | Adobe creds | paid | sota | no |
|
||||
|
||||
### `publish.cms` — publish images / articles to a CMS
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `contentful` | public-cli | bin + token | metered | high | no |
|
||||
| `sanity` | public-cli | bin + token | metered | high | no |
|
||||
| `hugo` / `mkdocs` | native/python | binary/pkg | free | high | yes |
|
||||
| WordPress REST | api | app password | free | high | no |
|
||||
|
||||
---
|
||||
|
||||
## Recipes
|
||||
|
||||
- **`social-card`** — generate + compose a social sharing image.
|
||||
Uses: `visual.generate`, `visual.edit.raster`, `publish.cms`.
|
||||
|
||||
- **`logo-set`** — design a logo + size variants.
|
||||
Uses: `visual.edit.vector`, `visual.edit.raster` (rasterize), batch export.
|
||||
|
||||
- **`ui-wireframe`** — mockup a UI flow.
|
||||
Uses: `visual.mockup`, optional `visual.edit.vector` polish.
|
||||
|
||||
- **`photo-batch`** — RAW ingest → develop → edit → library → publish.
|
||||
Uses: `photo.develop`, `visual.edit.raster`, `photo.library`, `publish.cms`.
|
||||
|
||||
- **`icon-pack`** — systematic icon production.
|
||||
Uses: `visual.edit.vector`, batch rasterize via `cairosvg`.
|
||||
|
||||
- **`thumbnail-set`** — thumbnails for a video/article series.
|
||||
Uses: `visual.generate` (hero imagery), `visual.edit.raster` (layout, type), batch variant export.
|
||||
|
||||
- **`ai-concept-art`** — generate → upscale → touch up.
|
||||
Uses: `visual.generate`, `visual.upscale`, `visual.edit.raster`.
|
||||
|
||||
---
|
||||
|
||||
## Known gaps
|
||||
|
||||
- **Figma** — cross-scenario gap (UI/UX loop for S5 and S7). API escalation works but we lack a native harness.
|
||||
- **`photo.library` / `photo.develop`** — strong native CLIs, no harness.
|
||||
- **Midjourney** — Discord bridge only; no clean API.
|
||||
|
||||
---
|
||||
|
||||
## Agent guidance
|
||||
|
||||
- **Match generation provider to intent.** GPT-Image-1 for text-in-image and photoreal; Ideogram for poster/typography; FLUX for artistic; Midjourney for stylized beauty shots; Stability for control+speed.
|
||||
- **Vector-first for design systems** — logos, icons, illustrations start in `visual.edit.vector`, rasterize last.
|
||||
- **Preserve color profiles.** `exiftool` + `Pillow` handles ICC; when moving between `rawpy` and `Pillow`, convert to sRGB at the boundary.
|
||||
- **AI + edit chain.** Agents should plan: generate → background removal (`rembg`) → composite (`visual.edit.raster`) → export. Skipping cleanup is the most common visible defect.
|
||||
- **Workspace discipline.** Keep generation seeds + prompts in a manifest alongside output PNGs so variations are reproducible.
|
||||
@@ -0,0 +1,247 @@
|
||||
---
|
||||
name: cli-hub-matrix-knowledge-research
|
||||
description: >-
|
||||
Capability-based multi-tool matrix for research, note-taking, document authoring,
|
||||
and publishing. Agents compose providers (CLI-Anything harnesses, public CLIs,
|
||||
Python libs, native binaries, cloud APIs) per capability.
|
||||
---
|
||||
|
||||
# Knowledge / Office / Research Matrix (S2 — v2 capability-based)
|
||||
|
||||
Scenario **S2**. Strongest coverage in the current ecosystem: search, reference management, notes, synthesis, documents, and diagrams all have first-party harnesses; Python/native fallbacks fill the rest.
|
||||
|
||||
Schema: [`docs/cli-matrix/matrix_registry.schema.md`](../../docs/cli-matrix/matrix_registry.schema.md). Matrix plan: [`docs/cli-matrix/cli-matrix-plan.md`](../../docs/cli-matrix/cli-matrix-plan.md).
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
cli-hub matrix install knowledge-research
|
||||
cli-hub matrix info knowledge-research
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decision rubric
|
||||
|
||||
1. **Available & adequate** — ranked by quality then inverse cost.
|
||||
2. **Free-to-install** — Python libs / native binaries without credentials.
|
||||
3. **Harness / public CLI install** — when warranted.
|
||||
4. **Paid API escalation** — only when lower tiers can't meet the bar, env holds the key, or the user consents.
|
||||
|
||||
Offline context? Filter to `offline: true` providers.
|
||||
|
||||
---
|
||||
|
||||
## Preflight
|
||||
|
||||
```bash
|
||||
cli-hub list --json
|
||||
python - <<'PY'
|
||||
import importlib.util
|
||||
for m in ("pyzotero","bibtexparser","trafilatura","readability","httpx","playwright",
|
||||
"python_docx","openpyxl","pptx","reportlab","pypdf","pdfplumber","weasyprint",
|
||||
"langchain","llama_index","haystack","argostranslate"):
|
||||
print(m, importlib.util.find_spec(m) is not None)
|
||||
PY
|
||||
for b in pandoc qpdf pdftk exiftool latexmk tex dot mermaid plantuml hugo mkdocs jekyll; do
|
||||
command -v "$b" >/dev/null && echo "$b: yes" || echo "$b: no"
|
||||
done
|
||||
for e in OPENAI_API_KEY ANTHROPIC_API_KEY GOOGLE_API_KEY PERPLEXITY_API_KEY TAVILY_API_KEY \
|
||||
SERPAPI_API_KEY BRAVE_API_KEY FIRECRAWL_API_KEY DEEPL_API_KEY \
|
||||
NOTION_API_KEY SEMANTIC_SCHOLAR_API_KEY WORDPRESS_TOKEN GHOST_ADMIN_KEY; do
|
||||
[ -n "${!e}" ] && echo "$e: set" || echo "$e: unset"
|
||||
done
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Suggest-to-user template
|
||||
|
||||
```
|
||||
To enable <capability> via <provider>, please set <ENV_VAR>.
|
||||
Cost: <cost notes>
|
||||
Quality: <quality tier>
|
||||
Reply 'skip' to fall back to <next provider>.
|
||||
```
|
||||
|
||||
Example: *To enable live Notion sync via the Notion API, please set `NOTION_API_KEY`. Cost: free tier suffices for most users. Quality: sota (round-trips block/page IDs). Reply 'skip' to fall back to local Obsidian vault + manual paste.*
|
||||
|
||||
---
|
||||
|
||||
## Capabilities
|
||||
|
||||
### `research.search` — web / literature search
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-exa` | harness-cli | `EXA_API_KEY` | metered | sota | no |
|
||||
| `cli-anything-browser` / `safari` / `clibrowser` | harness-cli | browser installed | free | high | no |
|
||||
| `duckduckgo-search` | python | pkg | free | good | no |
|
||||
| `googlesearch-python` | python | pkg | free | good | no |
|
||||
| Perplexity API | api | `PERPLEXITY_API_KEY` | paid | sota | no |
|
||||
| Tavily | api | `TAVILY_API_KEY` | metered | high | no |
|
||||
| Brave Search | api | `BRAVE_API_KEY` | metered | good | no |
|
||||
| Google Custom Search | api | `GOOGLE_API_KEY` | metered | high | no |
|
||||
|
||||
### `research.retrieve` — fetch article text / scrape pages
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-browser` / `safari` / `clibrowser` | harness-cli | harness installed | free | high | no |
|
||||
| `trafilatura` | python | pkg | free | high | yes |
|
||||
| `readability-lxml` | python | pkg | free | good | yes |
|
||||
| `httpx` + manual parse | python | pkg | free | basic | yes |
|
||||
| `playwright` | python | pkg + browsers | free | sota | no |
|
||||
| Firecrawl | api | `FIRECRAWL_API_KEY` | metered | sota | no |
|
||||
| ScrapingBee | api | `SCRAPINGBEE_API_KEY` | paid | high | no |
|
||||
|
||||
### `research.reference` — citations, BibTeX, references
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-zotero` | harness-cli | Zotero installed | free | sota | yes |
|
||||
| `pyzotero` | python | pkg + Zotero key | free | high | no |
|
||||
| `habanero` (Crossref) | python | pkg | free | high | no |
|
||||
| `bibtexparser` | python | pkg | free | good | yes |
|
||||
| Crossref REST | api | none | free | high | no |
|
||||
| Semantic Scholar | api | `SEMANTIC_SCHOLAR_API_KEY` (optional) | free | high | no |
|
||||
|
||||
### `knowledge.note` — note-taking / PKM
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-obsidian` | harness-cli | Obsidian installed | free | sota | yes |
|
||||
| `cli-anything-mubu` | harness-cli | Mubu installed | free | high | no |
|
||||
| `obsidian-cli` | public-cli | bin + vault path | free | high | yes |
|
||||
| raw markdown + `git` | native | `git` | free | good | yes |
|
||||
| `mdformat` | python | pkg | free | good | yes |
|
||||
| Notion API | api | `NOTION_API_KEY` | free | sota | no |
|
||||
| Roam / Logseq sync | — | — | — | — | — |
|
||||
|
||||
### `knowledge.outline` — outlining
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-mubu` | harness-cli | Mubu installed | free | high | no |
|
||||
| OPML via `lxml` | python | pkg | free | good | yes |
|
||||
| YAML outlines + `yq` | native | binary | free | good | yes |
|
||||
|
||||
### `knowledge.synthesize` — multi-doc synthesis / summarization
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-notebooklm` | harness-cli | harness installed | metered | sota | no |
|
||||
| `cli-anything-intelwatch` | harness-cli | harness installed | free-metered | high | partial |
|
||||
| `langchain` / `llama_index` + local LLM | python | pkg + ollama | free | high | yes |
|
||||
| `haystack` | python | pkg | free | good | partial |
|
||||
| Claude / OpenAI / Gemini (long context) | api | model key | metered | sota | no |
|
||||
|
||||
### `document.author` — write docs (DOCX, XLSX, PPTX, PDF)
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-libreoffice` | harness-cli | LibreOffice installed | free | high | yes |
|
||||
| `cli-anything-anygen` | harness-cli | harness installed | free | high | yes |
|
||||
| `python-docx` / `openpyxl` / `python-pptx` / `reportlab` | python | pkg | free | good | yes |
|
||||
| Google Docs API | api | OAuth token | metered | sota | no |
|
||||
| Microsoft Graph | api | Azure app + token | metered | sota | no |
|
||||
|
||||
### `document.format` — convert / reformat
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `pandoc` | native | binary | free | sota | yes |
|
||||
| `cli-anything-libreoffice` (headless convert) | harness-cli | LibreOffice | free | high | yes |
|
||||
| `docx2pdf` | python | pkg + Word/LO | free | good | yes |
|
||||
| `weasyprint` | python | pkg | free | high | yes |
|
||||
| CloudConvert | api | `CLOUDCONVERT_API_KEY` | metered | sota | no |
|
||||
|
||||
### `document.pdf` — manipulate PDFs (split, merge, OCR, extract)
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `qpdf` / `pdftk` | native | binary | free | sota | yes |
|
||||
| `pypdf` | python | pkg | free | high | yes |
|
||||
| `pdfplumber` | python | pkg | free | high | yes |
|
||||
| `pdf2image` | python | pkg + poppler | free | good | yes |
|
||||
| `ocrmypdf` | native | binary + tesseract | free | high | yes |
|
||||
| Adobe PDF Services | api | Adobe creds | paid | sota | no |
|
||||
|
||||
### `diagram.create` — diagrams, flowcharts, architecture
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-drawio` | harness-cli | drawio installed | free | sota | yes |
|
||||
| `cli-anything-mermaid` | harness-cli | mermaid-cli | free | high | yes |
|
||||
| `graphviz` (`dot`) | native | binary | free | high | yes |
|
||||
| `plantuml` | native | binary | free | high | yes |
|
||||
|
||||
### `text.translate` — translation
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `argos-translate` | python | pkg + lang packs | free | good | yes |
|
||||
| DeepL | api | `DEEPL_API_KEY` | metered | sota | no |
|
||||
| Google Translate | api | `GOOGLE_API_KEY` | metered | high | no |
|
||||
| OpenAI / Claude | api | model key | metered | sota | no |
|
||||
|
||||
### `publish.web` — publish articles / blog posts
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `contentful` | public-cli | bin + space token | metered | high | no |
|
||||
| `sanity` | public-cli | bin + project token | metered | high | no |
|
||||
| `hugo` / `mkdocs` / `jekyll` / `pelican` | native/python | binary/pkg | free | high | yes |
|
||||
| WordPress REST | api | app password | free | high | no |
|
||||
| Ghost Admin | api | `GHOST_ADMIN_KEY` | free | high | no |
|
||||
| Medium (legacy) | api | token | free | good | no |
|
||||
|
||||
### `publish.latex` — build PDFs from LaTeX
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `latexmk` + `texlive` | native | binaries | free | sota | yes |
|
||||
| `pylatex` | python | pkg + texlive | free | high | yes |
|
||||
| Overleaf API | api | `OVERLEAF_API_KEY` | paid | sota | no |
|
||||
| Tectonic | native | binary | free | high | yes |
|
||||
|
||||
---
|
||||
|
||||
## Recipes
|
||||
|
||||
- **`literature-review`** — survey a topic, extract key papers, synthesize.
|
||||
Uses: `research.search`, `research.retrieve`, `research.reference`, `knowledge.synthesize`, `document.author`.
|
||||
|
||||
- **`meeting-to-doc`** — turn a recorded meeting into a structured document.
|
||||
Uses: `text.transcribe` (via S1 matrix), `knowledge.synthesize`, `document.author`, `document.format`.
|
||||
|
||||
- **`blog-post`** — research → write → publish.
|
||||
Uses: `research.search`, `research.retrieve`, `knowledge.synthesize`, `document.author`, `publish.web`.
|
||||
|
||||
- **`paper-draft`** — research + citations + LaTeX.
|
||||
Uses: `research.reference`, `knowledge.synthesize`, `document.author`, `publish.latex`.
|
||||
|
||||
- **`competitive-brief`** — fetch competitor pages, extract positioning, produce a briefing deck.
|
||||
Uses: `research.retrieve`, `knowledge.synthesize`, `document.author` (PPTX).
|
||||
|
||||
- **`pdf-book-digest`** — ingest a PDF book, OCR if needed, produce chapter summaries + quote file.
|
||||
Uses: `document.pdf`, `knowledge.synthesize`, `knowledge.note`.
|
||||
|
||||
---
|
||||
|
||||
## Known gaps
|
||||
|
||||
- **Notion** — Notion API exists as an escalation, but there is no harness for local-first Notion workflows. Recurs in S11 (team comms).
|
||||
- **Google Docs / Microsoft 365 live editing** — API path only.
|
||||
- **LaTeX harness** — `latexmk` + Tectonic cover the native path, but no agent-native wrapper yet.
|
||||
- **Translation** — no first-party CLI; Python fallback and APIs only.
|
||||
|
||||
---
|
||||
|
||||
## Agent guidance
|
||||
|
||||
- **Run preflight once per session.** Many capabilities have strong Python or native paths that bypass the need to install anything.
|
||||
- **Prefer local** for sensitive docs (legal, health, internal): use Obsidian + local LLM + `whisper` rather than cloud APIs.
|
||||
- **Citations are load-bearing** — for academic work always route through `research.reference` so bibliographies stay consistent.
|
||||
- **Chain `document.format` at the end**, not the middle: author in the richest source format (markdown or LaTeX) and convert last.
|
||||
- **Workspace discipline** — one directory per project; keep `refs.bib`, source markdown, figures, and output PDFs colocated.
|
||||
@@ -1,170 +1,224 @@
|
||||
---
|
||||
name: cli-hub-matrix-video-creation
|
||||
description: >-
|
||||
Curated multi-CLI matrix for generating, editing, captioning, and packaging
|
||||
videos across CLI-Anything harnesses, public CLIs, Python libraries, and cloud APIs.
|
||||
Capability-based multi-tool matrix for video production. Agents pick providers
|
||||
(CLI-Anything harnesses, public CLIs, Python libs, native binaries, cloud APIs)
|
||||
per capability rather than marching through fixed stages.
|
||||
---
|
||||
|
||||
# Video Creation Matrix
|
||||
# Video Creation Matrix (v3 — capability-based)
|
||||
|
||||
This matrix covers end-to-end video production. Each stage lists a **primary goal**, **CLI tools** from the matrix, and **alternative approaches** (Python libraries, cloud APIs, native commands). Pick the approach that fits your current environment and constraints.
|
||||
This matrix describes **capabilities** the agent can compose on demand — not a fixed pipeline. A "video creation" workflow picks a *recipe* (which capabilities it needs) and, per capability, picks a *provider* using the decision rubric below.
|
||||
|
||||
## Install
|
||||
Schema: [`docs/cli-matrix/matrix_registry.schema.md`](../../docs/cli-matrix/matrix_registry.schema.md).
|
||||
|
||||
## Install (installable portion)
|
||||
|
||||
```bash
|
||||
# Install the whole matrix
|
||||
cli-hub matrix install video-creation
|
||||
|
||||
# Inspect what it includes
|
||||
cli-hub matrix info video-creation
|
||||
cli-hub matrix install video-creation # installs the harness + public CLIs
|
||||
cli-hub matrix info video-creation # inspect providers & recipes
|
||||
```
|
||||
|
||||
## Stages
|
||||
Not everything in this matrix is a CLI. Cloud APIs, Python packages, and native binaries are first-class providers too.
|
||||
|
||||
### 1. AI Video Generation
|
||||
**Goal:** Generate video clips from text prompts or reference images
|
||||
---
|
||||
|
||||
| Approach | Options |
|
||||
|----------|---------|
|
||||
| CLI tools | `generate-veo-video` (Google Veo 3.1), `jimeng` (ByteDance Dreamina) |
|
||||
| Python libs | `replicate` (Runway, Kling), `diffusers` |
|
||||
| Cloud APIs | Runway API, Kling API, Pika API |
|
||||
## Decision rubric (agent: apply per capability)
|
||||
|
||||
### 2. Screen/Cam Capture
|
||||
**Goal:** Record screen activity, webcam, or application windows to video files
|
||||
1. **Available & adequate** — prefer providers whose `requires` is already satisfied, ranked by `quality_tier` then inverse `cost_tier`.
|
||||
2. **Free-to-install** — next prefer Python libs or native binaries that install in seconds without credentials.
|
||||
3. **Harness / public CLI install** — install a first-party or public CLI when the task warrants it.
|
||||
4. **Paid API escalation** — only when lower tiers can't meet quality, env already holds the key, **or** the user explicitly consents. Never silently call a paid API.
|
||||
|
||||
| Approach | Options |
|
||||
|----------|---------|
|
||||
| CLI tools | `cli-anything-openscreen` |
|
||||
| Python libs | `pyautogui` + `cv2`, `mss` |
|
||||
| Native cmds | `ffmpeg -f x11grab` (Linux), `screencapture` (macOS) |
|
||||
Offline context? Filter to `offline: true` providers only.
|
||||
|
||||
### 3. Audio Capture/Edit
|
||||
**Goal:** Record, clean, trim, normalize, and process audio tracks
|
||||
---
|
||||
|
||||
| Approach | Options |
|
||||
|----------|---------|
|
||||
| CLI tools | `cli-anything-audacity` |
|
||||
| Python libs | `pydub`, `soundfile`, `librosa`, `noisereduce` |
|
||||
| Native cmds | `sox`, `ffmpeg` |
|
||||
|
||||
### 4. Voice/TTS
|
||||
**Goal:** Generate speech audio from text for narration or voiceover
|
||||
|
||||
| Approach | Options |
|
||||
|----------|---------|
|
||||
| CLI tools | `minimax-cli`, `elevenlabs` |
|
||||
| Python libs | `TTS` (coqui-ai), `pyttsx3`, `edge-tts` |
|
||||
| Cloud APIs | Google Cloud TTS, Amazon Polly, Azure Speech |
|
||||
|
||||
### 5. Music/BGM Generation
|
||||
**Goal:** Generate background music or sound effects from prompts
|
||||
|
||||
| Approach | Options |
|
||||
|----------|---------|
|
||||
| CLI tools | `minimax-cli`, `suno` |
|
||||
| Python libs | `audiocraft` (Meta MusicGen), `stable-audio-tools` |
|
||||
| Cloud APIs | Suno API, Udio API |
|
||||
|
||||
### 6. NLE Editing
|
||||
**Goal:** Assemble video/audio clips on a timeline, add transitions, and export the final cut
|
||||
|
||||
| Approach | Options |
|
||||
|----------|---------|
|
||||
| CLI tools | `cli-anything-kdenlive`, `cli-anything-shotcut` |
|
||||
| Python libs | `moviepy`, `ffmpeg-python` |
|
||||
| Native cmds | `ffmpeg concat/filter_complex` |
|
||||
|
||||
### 7. Captions/Subtitles
|
||||
**Goal:** Transcribe speech, generate subtitles (SRT/ASS), optionally translate, and burn into video
|
||||
|
||||
| Approach | Options |
|
||||
|----------|---------|
|
||||
| CLI tools | `cli-anything-videocaptioner` |
|
||||
| Python libs | `whisper` (openai-whisper), `pysrt`, `stable-ts` |
|
||||
| Cloud APIs | Google Speech-to-Text, AssemblyAI, Deepgram |
|
||||
| Native cmds | `ffmpeg -vf subtitles=subs.srt` |
|
||||
|
||||
### 8. Thumbnail
|
||||
**Goal:** Create a thumbnail image or social card for the video
|
||||
|
||||
| Approach | Options |
|
||||
|----------|---------|
|
||||
| CLI tools | `cli-anything-gimp` |
|
||||
| Python libs | `Pillow`, `cairosvg`, `html2image` |
|
||||
| Cloud APIs | Google Nano Banana, OpenAI GPT-Image-1, Ideogram API, Stability AI image generation |
|
||||
| Native cmds | `ffmpeg -ss 00:01:00 -frames:v 1 thumb.png`, ImageMagick `convert` |
|
||||
|
||||
## Discovering Additional Tools
|
||||
|
||||
Before starting a stage, search for skills and tools that may already be available:
|
||||
## Preflight (run once per session, cache the result)
|
||||
|
||||
```bash
|
||||
# Search for Claude Code skills by keyword
|
||||
npx skills search "subtitle"
|
||||
npx skills search "video editing"
|
||||
npx skills search "text-to-speech"
|
||||
|
||||
# Check what CLI-Hub has available
|
||||
cli-hub search caption
|
||||
cli-hub search audio
|
||||
cli-hub search video
|
||||
|
||||
# Check for Python packages already installed
|
||||
pip list 2>/dev/null | grep -iE "whisper|moviepy|pydub|Pillow"
|
||||
|
||||
# Check for native tools on PATH
|
||||
which ffmpeg sox convert magick 2>/dev/null
|
||||
cli-hub list --json
|
||||
python - <<'PY'
|
||||
import importlib.util
|
||||
for m in ("moviepy","whisper","pydub","PIL","replicate","edge_tts","audiocraft","pysrt","TTS"):
|
||||
print(m, importlib.util.find_spec(m) is not None)
|
||||
PY
|
||||
for b in ffmpeg sox convert magick screencapture; do command -v "$b" >/dev/null && echo "$b: yes" || echo "$b: no"; done
|
||||
for e in RUNWAY_API_KEY KLING_API_KEY SEEDANCE_API_KEY ELEVENLABS_API_KEY MINIMAX_API_KEY \
|
||||
OPENAI_API_KEY GOOGLE_CLOUD_PROJECT ASSEMBLYAI_API_KEY DEEPGRAM_API_KEY \
|
||||
SUNO_API_KEY UDIO_API_KEY IDEOGRAM_API_KEY STABILITY_API_KEY; do
|
||||
[ -n "${!e}" ] && echo "$e: set" || echo "$e: unset"
|
||||
done
|
||||
```
|
||||
|
||||
When a stage's CLI tool is not installed and installing it is impractical, fall back to:
|
||||
1. A Python library you can `pip install` in seconds
|
||||
2. A native command already on the system (`ffmpeg`, `sox`, `ImageMagick`)
|
||||
3. A cloud API if credentials are available in the environment
|
||||
---
|
||||
|
||||
## Operating Pattern
|
||||
## Suggest-to-user template (agent uses verbatim when escalating)
|
||||
|
||||
1. The controller agent drafts the script, shot list, and prompt variants.
|
||||
2. For each stage, check what tools are available (installed CLIs, Python libs on PATH, API keys in env).
|
||||
3. Pick the simplest adequate tool for each stage -- a full CLI harness is not always needed.
|
||||
4. Generate/capture media, assemble the edit, add captions, and create the thumbnail.
|
||||
5. Keep all intermediate assets under a single workspace directory so sources stay linkable across tools.
|
||||
|
||||
## Example Flow
|
||||
|
||||
```bash
|
||||
# 1) Discover what is available
|
||||
cli-hub matrix info video-creation
|
||||
npx skills search "video"
|
||||
which ffmpeg && echo "ffmpeg available"
|
||||
|
||||
# 2) Generate hero clips
|
||||
cli-hub launch generate-veo-video --help
|
||||
# or: python -c "import replicate; ..."
|
||||
|
||||
# 3) Generate narration
|
||||
cli-hub launch elevenlabs --help
|
||||
# or: pip install edge-tts && edge-tts --text "Hello" --write-media narration.mp3
|
||||
|
||||
# 4) Assemble edit
|
||||
cli-anything-kdenlive --json project new -o edit.json
|
||||
cli-anything-kdenlive --json --project edit.json timeline add-track --type video
|
||||
# or: python -c "from moviepy.editor import *; ..."
|
||||
|
||||
# 5) Add subtitles
|
||||
cli-anything-videocaptioner process cut.mp4 --asr bijian --subtitle-mode hard -o captioned.mp4
|
||||
# or: pip install openai-whisper && python -c "import whisper; model = whisper.load_model('base'); ..."
|
||||
|
||||
# 6) Create thumbnail
|
||||
cli-anything-gimp --json project new -o thumb.json
|
||||
# or: python -c "from PIL import Image, ImageDraw, ImageFont; ..."
|
||||
```
|
||||
To enable <capability> via <provider>, please set <ENV_VAR>.
|
||||
Cost: <cost notes>
|
||||
Quality: <quality tier>
|
||||
Reply 'skip' to fall back to <next provider>.
|
||||
```
|
||||
|
||||
## Agent Guidance
|
||||
Examples:
|
||||
|
||||
- **Check before installing**: If `whisper` is already pip-installed, use it directly instead of installing `cli-anything-videocaptioner` for a simple transcription task.
|
||||
- **Match complexity to task**: A one-line `ffmpeg` command may beat a full CLI harness for simple operations like extracting a thumbnail or concatenating two clips.
|
||||
- **Search for skills first**: Before each stage, run `npx skills search "<keyword>"` with the stage's topic to discover relevant skills you may not know about.
|
||||
- **Prefer `--json` for harness CLIs**: When chaining harness CLI outputs to the next tool, use `--json` for structured data.
|
||||
- **Use env vars for API routing**: Check for `GOOGLE_CLOUD_PROJECT`, `ELEVENLABS_API_KEY`, `MINIMAX_API_KEY`, etc. before choosing between CLI and direct API approaches.
|
||||
- The matrix assumes a controller agent (Claude Code, Codex, OpenClaw, etc.) is doing the planning and orchestration.
|
||||
- *To enable cinematic AI video via Runway Gen-4, please set `RUNWAY_API_KEY`. Cost: ~$0.05/sec. Quality: sota. Reply 'skip' to fall back to `generate-veo-video` or local `diffusers`.*
|
||||
- *To enable ByteDance Seedance video generation, please set `SEEDANCE_API_KEY`. Cost: metered per-clip. Quality: sota for realistic motion. Reply 'skip' to fall back to `jimeng` (Dreamina) which shares the ByteDance model family.*
|
||||
|
||||
---
|
||||
|
||||
## Capabilities
|
||||
|
||||
### `visual.generate` — produce a video clip from prompt/reference
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `generate-veo-video` | public-cli | `generate-veo` bin + Google creds | metered | high | no |
|
||||
| `jimeng` | public-cli | `dreamina` bin + Dreamina login | metered | high | no |
|
||||
| Runway Gen-4 | api | `RUNWAY_API_KEY` | paid | sota | no |
|
||||
| Kling | api | `KLING_API_KEY` | paid | high | no |
|
||||
| Pika | api | `PIKA_API_KEY` | paid | good | no |
|
||||
| Seedance | api | `SEEDANCE_API_KEY` | paid | sota | no |
|
||||
| `replicate` | python | `REPLICATE_API_TOKEN` | metered | varies | no |
|
||||
| `diffusers` (SVD/AnimateDiff) | python | GPU + weights | free | basic-good | yes |
|
||||
|
||||
### `visual.capture` — record screen / webcam / window
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-openscreen` | harness-cli | harness installed | free | high | yes |
|
||||
| `cli-anything-obs-studio` | harness-cli | OBS installed | free | high | yes |
|
||||
| `ffmpeg -f x11grab` / `avfoundation` | native | `ffmpeg` | free | high | yes |
|
||||
| `screencapture` | native | macOS | free | high | yes |
|
||||
| `mss` / `pyautogui` + `cv2` | python | pkgs | free | good | yes |
|
||||
|
||||
### `audio.capture` — record and clean audio tracks
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-audacity` | harness-cli | Audacity installed | free | high | yes |
|
||||
| `sox` / `ffmpeg` | native | binary | free | high | yes |
|
||||
| `pydub` / `soundfile` / `librosa` / `noisereduce` | python | pkgs | free | good | yes |
|
||||
|
||||
### `audio.synthesize` — text-to-speech / voice
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `minimax-cli` | public-cli | bin + MiniMax key | metered | high | no |
|
||||
| `elevenlabs` | public-cli | bin + `ELEVENLABS_API_KEY` | paid | sota | no |
|
||||
| OpenAI TTS | api | `OPENAI_API_KEY` | metered | high | no |
|
||||
| Google Cloud TTS | api | `GOOGLE_CLOUD_PROJECT` | metered | high | no |
|
||||
| Amazon Polly / Azure Speech | api | AWS/Azure creds | metered | good | no |
|
||||
| `edge-tts` | python | pkg | free | good | no |
|
||||
| `TTS` (coqui) | python | pkg + weights | free | good | yes |
|
||||
| `pyttsx3` | python | pkg | free | basic | yes |
|
||||
|
||||
### `audio.music` — music / BGM generation
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `suno` | public-cli | bin + Suno account | metered | sota | no |
|
||||
| `minimax-cli` | public-cli | bin + MiniMax key | metered | high | no |
|
||||
| Udio | api | `UDIO_API_KEY` | paid | sota | no |
|
||||
| `audiocraft` (MusicGen) | python | pkg + weights | free | good | yes |
|
||||
| `stable-audio-tools` | python | pkg + weights | free | good | yes |
|
||||
|
||||
### `text.transcribe` — speech → text / subtitles
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-videocaptioner` | harness-cli | harness installed | free-metered | high | partial |
|
||||
| `openai-whisper` | python | pkg + weights | free | high | yes |
|
||||
| `stable-ts` / `faster-whisper` | python | pkg | free | high | yes |
|
||||
| AssemblyAI | api | `ASSEMBLYAI_API_KEY` | paid | sota | no |
|
||||
| Deepgram | api | `DEEPGRAM_API_KEY` | paid | sota | no |
|
||||
| Google Speech-to-Text | api | GCP creds | metered | high | no |
|
||||
|
||||
### `text.translate` — subtitle/caption translation
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| OpenAI / Claude | api | model key | metered | sota | no |
|
||||
| DeepL | api | `DEEPL_API_KEY` | metered | sota | no |
|
||||
| `argos-translate` | python | pkg + lang packs | free | good | yes |
|
||||
|
||||
### `composite.assemble` — timeline, cuts, transitions, export
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-kdenlive` | harness-cli | Kdenlive installed | free | high | yes |
|
||||
| `cli-anything-shotcut` | harness-cli | Shotcut installed | free | high | yes |
|
||||
| `moviepy` | python | pkg + `ffmpeg` | free | good | yes |
|
||||
| `ffmpeg-python` | python | pkg + `ffmpeg` | free | high | yes |
|
||||
| `ffmpeg concat/filter_complex` | native | `ffmpeg` | free | high | yes |
|
||||
|
||||
### `composite.overlay` — burn subs, watermark, picture-in-picture
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `ffmpeg -vf subtitles=...` | native | `ffmpeg` | free | high | yes |
|
||||
| `moviepy` (CompositeVideoClip) | python | pkg | free | good | yes |
|
||||
| NLE harnesses (via assemble) | harness-cli | installed | free | high | yes |
|
||||
|
||||
### `package.thumbnail` — thumbnail / social card
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `cli-anything-gimp` / `krita` / `inkscape` | harness-cli | installed | free | high | yes |
|
||||
| `Pillow` | python | pkg | free | good | yes |
|
||||
| `cairosvg` / `html2image` | python | pkg | free | good | yes |
|
||||
| OpenAI GPT-Image-1 | api | `OPENAI_API_KEY` | metered | sota | no |
|
||||
| Google Nano Banana | api | GCP creds | metered | high | no |
|
||||
| Ideogram | api | `IDEOGRAM_API_KEY` | metered | high | no |
|
||||
| Stability AI | api | `STABILITY_API_KEY` | metered | high | no |
|
||||
| `ffmpeg -ss ... -frames:v 1` | native | `ffmpeg` | free | basic | yes |
|
||||
| `convert` / `magick` | native | ImageMagick | free | good | yes |
|
||||
|
||||
### `package.encode` — final mux, codec, container
|
||||
|
||||
| Provider | Kind | Requires | Cost | Quality | Offline |
|
||||
|---|---|---|---|---|---|
|
||||
| `ffmpeg` | native | binary | free | sota | yes |
|
||||
| NLE export (via `composite.assemble`) | harness-cli | installed | free | high | yes |
|
||||
|
||||
### `publish.upload` — deliver to a platform
|
||||
|
||||
Currently a **known gap** (see below). Agents should surface this to the user.
|
||||
|
||||
---
|
||||
|
||||
## Recipes
|
||||
|
||||
Recipes declare *which capabilities a workflow needs* — not the order. Apply the decision rubric per capability.
|
||||
|
||||
- **`ai-short`** — fully-generative social short.
|
||||
Uses: `visual.generate`, `audio.synthesize`, `audio.music`, `composite.assemble`, `composite.overlay`, `package.thumbnail`, `package.encode`.
|
||||
|
||||
- **`screencast-tutorial`** — walkthrough with narration + subs.
|
||||
Uses: `visual.capture`, `audio.capture`, `text.transcribe`, `composite.overlay`, `package.thumbnail`, `package.encode`.
|
||||
|
||||
- **`talking-head-explainer`** — webcam + b-roll + captions.
|
||||
Uses: `visual.capture`, `visual.generate` (b-roll), `audio.capture`, `text.transcribe`, `composite.assemble`, `composite.overlay`, `package.thumbnail`, `package.encode`.
|
||||
|
||||
- **`podcast-to-video`** — audio-first, visualize + caption.
|
||||
Uses: `audio.capture`, `text.transcribe`, `package.thumbnail`, `composite.overlay`, `composite.assemble`, `package.encode`.
|
||||
|
||||
---
|
||||
|
||||
## Known gaps
|
||||
|
||||
- **`publish.upload`** — no first-party or public CLI for YouTube/TikTok/Bilibili/Instagram yet. *Workaround:* instruct the user to upload manually via the web UI, or escalate to a custom script using each platform's v3 API with an OAuth token the user supplies.
|
||||
- **`visual.generate` — top-tier "cinematic"** — available only via paid APIs (Runway, Kling, Seedance); local `diffusers` output is basic-to-good, not sota.
|
||||
|
||||
---
|
||||
|
||||
## Agent guidance
|
||||
|
||||
- **Run the preflight block once**, then consult the cached result when picking providers.
|
||||
- **Search for skills per capability** before starting: `npx skills search "<capability hints>"` (hints are in `matrix_registry.json`).
|
||||
- **Prefer `--json`** for harness CLI output when chaining tools.
|
||||
- **Escalate explicitly.** When a paid API would materially improve quality, use the suggest-to-user template. Do not silently burn credits.
|
||||
- **Recipes ≠ order.** A recipe says what's needed; pick a sensible order for the specific task. Most videos should transcribe *after* the final cut, not before; screencasts often capture audio + video simultaneously.
|
||||
- **Workspace discipline.** Keep all intermediate assets under one directory so cross-tool references stay stable.
|
||||
|
||||
@@ -102,7 +102,7 @@ SAMPLE_MATRIX_REGISTRY = {
|
||||
"description": "Curated video workflow matrix",
|
||||
"category": "video",
|
||||
"matrix": "cli-matrix",
|
||||
"matrix_id": "V1",
|
||||
"matrix_id": "S1",
|
||||
"skill_md": "cli-hub-matrix/video-creation/SKILL.md",
|
||||
"clis": ["gimp", "blender", "audacity"],
|
||||
"stages": [
|
||||
|
||||
@@ -0,0 +1,413 @@
|
||||
# CLI Matrix — Scenarios × Capabilities
|
||||
|
||||
> A living map of the agent tool ecosystem around each workflow: CLI-Anything harnesses, CLI-Hub public CLIs, cloud APIs, and traditional libraries/native tools, organized by **scenarios** (columns) × **capabilities** (rows), with explicit gap analysis to find where closed-loop agent workflows break.
|
||||
|
||||
**Last updated:** 2026-04-18
|
||||
|
||||
---
|
||||
|
||||
## Matrix framing (v2 — capability-based, eco-first)
|
||||
|
||||
- **Columns = Scenarios (`S1`…`S16`)** — closed-loop agent workflows (e.g. "video creation", "knowledge work", "game development").
|
||||
- **Rows = Capabilities** — verbs the agent can compose on demand (`visual.generate`, `model.mesh`, `knowledge.note`, `publish.upload`, …), **not** a fixed pipeline.
|
||||
- **Cells = providers** for that capability: first-party harness CLIs, public third-party CLIs, Python libs, native binaries, and direct cloud APIs. Each provider declares a `requires` preflight contract (env/binary/package), a `cost_tier`, a `quality_tier`, and whether it works `offline`.
|
||||
|
||||
> Stages silently impose a linear order that real workflows don't follow. A screencast tutorial, a generative short, a talking-head explainer, and a podcast-to-video each touch different subsets of the same tools in different orders. **Capabilities compose; stages march.** Workflows become *recipes* over capabilities.
|
||||
|
||||
Schema: [`matrix_registry.schema.md`](matrix_registry.schema.md). Reference implementation: [`cli-hub-matrix/video-creation/SKILL.md`](../cli-hub-matrix/video-creation/SKILL.md) and the v2 entry in `matrix_registry.json`.
|
||||
|
||||
### Eco-first direction
|
||||
|
||||
A matrix is "complete" when every capability has at least one practical agent path — CLI today, direct API tomorrow, Python package or native command as the fallback. Two quality axes:
|
||||
|
||||
1. **Coverage + depth** — every capability has multiple providers ranked by cost/quality/offline so the agent can always pick *something* adequate.
|
||||
2. **Full picture** — when our CLI fails or is inefficient, the matrix explicitly names the external tool/API/library the agent should reach for, including credential requirements. The agent uses a `suggest-to-user` template to escalate transparently rather than silently skipping or silently burning credits.
|
||||
|
||||
Provider layers (each capability should have some of each where possible):
|
||||
|
||||
- **First-party** — CLI-Anything harnesses; preferred default where available.
|
||||
- **Public CLI** — `cli-hub` third-party CLIs; broaden coverage fast.
|
||||
- **Python / native** — free fallbacks an agent can use immediately (`moviepy`, `ffmpeg`, `Pillow`, `whisper`).
|
||||
- **Cloud API** — when a hosted model is the fastest route to sota quality; escalated with an explicit suggest-to-user prompt.
|
||||
|
||||
### Decision rubric (canonical — SKILL.md files reference this)
|
||||
|
||||
Per capability, the agent walks this ladder:
|
||||
|
||||
1. **Available & adequate** — providers whose `requires` is already satisfied, ranked by quality then inverse cost.
|
||||
2. **Free-to-install** — Python libs / native binaries installable in seconds without credentials.
|
||||
3. **Harness / public CLI install** — install a first-party or public CLI when warranted.
|
||||
4. **Paid API escalation** — only when lower tiers can't meet the bar, the env already holds the key, **or** the user explicitly consents via the suggest-to-user prompt.
|
||||
|
||||
Offline context? Filter to `offline: true` providers only.
|
||||
|
||||
### Scenario ordering
|
||||
|
||||
Scenarios are ordered by **current completeness first** (migrated to v2 capability-form) then by rough strength of coverage, so readers hit the strongest, most-representative scenarios up front. Stage-form scenarios at the tail are still on the v1 framing and will be migrated opportunistically.
|
||||
|
||||
| # | Scenario | Form |
|
||||
|---|---|---|
|
||||
| S1 | 🎬 Video Creation & Editing | v2 capability (reference) |
|
||||
| S2 | 📚 Knowledge / Office / Research | v2 capability |
|
||||
| S3 | 📐 3D & CAD | v2 capability |
|
||||
| S4 | 🎮 Game Development | v2 capability |
|
||||
| S5 | 🖼️ Image & Graphic Design | v2 capability |
|
||||
| S6 | 🎵 Music & Audio Production | v1 stage |
|
||||
| S7 | 🌐 Web / App Development | v1 stage |
|
||||
| S8 | 📡 Creator Operations | v1 stage |
|
||||
| S9 | 🤖 AI/ML & Data Science | v1 stage |
|
||||
| S10 | 🛠️ DevOps / SRE | v1 stage |
|
||||
| S11 | 💬 Team Communication | v1 stage |
|
||||
| S12 | 🛒 E-Commerce | v1 stage |
|
||||
| S13 | 🔬 Science / Academic | v1 prose |
|
||||
| S14 | 🎯 Agentic Game Playing | v1 prose |
|
||||
| S15 | 🛡️ Security / OSINT | v1 prose |
|
||||
| S16 | 💰 Finance / Trading / Crypto | v1 prose |
|
||||
|
||||
---
|
||||
|
||||
## 🎬 S1 · Video Creation & Editing — **v2 reference**
|
||||
|
||||
See [`cli-hub-matrix/video-creation/SKILL.md`](../cli-hub-matrix/video-creation/SKILL.md) and the S1 entry in `matrix_registry.json` for the full shape.
|
||||
|
||||
| Capability | Harness CLI | Public CLI | Python / native | Paid API (escalate) |
|
||||
|---|---|---|---|---|
|
||||
| `visual.generate` | — | `generate-veo-video`, `jimeng` | `diffusers` (SVD/AnimateDiff), `replicate` | Runway Gen-4, Kling, Pika, **Seedance** |
|
||||
| `visual.capture` | `openscreen`, `obs-studio` | — | `ffmpeg x11grab`, `screencapture`, `mss+cv2` | — |
|
||||
| `audio.capture` | `audacity` | — | `ffmpeg`, `sox`, `pydub`, `noisereduce` | — |
|
||||
| `audio.synthesize` | — | `elevenlabs`, `minimax-cli` | `edge-tts`, `TTS` (coqui), `pyttsx3` | OpenAI TTS, Google TTS, Polly, Azure |
|
||||
| `audio.music` | — | `suno`, `minimax-cli` | `audiocraft` (MusicGen), `stable-audio-tools` | Udio |
|
||||
| `text.transcribe` | `videocaptioner` | — | `whisper`, `faster-whisper`, `stable-ts` | AssemblyAI, Deepgram, Google STT |
|
||||
| `text.translate` | — | — | `argos-translate` | OpenAI/Claude, DeepL |
|
||||
| `composite.assemble` | `kdenlive`, `shotcut` | — | `moviepy`, `ffmpeg-python`, `ffmpeg` | — |
|
||||
| `composite.overlay` | (via NLE) | — | `ffmpeg -vf subtitles/overlay`, `moviepy` | — |
|
||||
| `package.thumbnail` | `gimp`, `krita`, `inkscape` | — | `Pillow`, `cairosvg`, `html2image`, `ffmpeg`, ImageMagick | OpenAI GPT-Image-1, Nano Banana, Ideogram, Stability |
|
||||
| `package.encode` | (via NLE) | — | `ffmpeg` | — |
|
||||
| `publish.upload` | ❌ | ❌ | — | YouTube, TikTok, Bilibili, Instagram (**known gap**) |
|
||||
|
||||
**Recipes**: `ai-short`, `screencast-tutorial`, `talking-head-explainer`, `podcast-to-video`.
|
||||
**Known gaps**: `publish.upload` (no CLI anywhere); `visual.generate` sota tier is paid-API-only.
|
||||
**Verdict**: strong across generation/edit/package; distribution gap persists.
|
||||
|
||||
---
|
||||
|
||||
## 📚 S2 · Knowledge / Office / Research — **v2**
|
||||
|
||||
The strongest scenario in the matrix today. First-party harnesses cover search, notes, references, synthesis, office docs, and diagrams; Python fallbacks fill the rest.
|
||||
|
||||
| Capability | Harness CLI | Public CLI | Python / native | Paid API (escalate) |
|
||||
|---|---|---|---|---|
|
||||
| `research.search` | `exa`, `browser`, `safari`, `clibrowser` | — | `duckduckgo-search`, `googlesearch-python`, `serpapi` | Google Custom Search, Perplexity, Tavily, Brave Search |
|
||||
| `research.retrieve` | `browser`, `safari`, `clibrowser` | — | `trafilatura`, `readability-lxml`, `httpx`, `playwright` | Firecrawl, ScrapingBee |
|
||||
| `research.reference` | `zotero` | — | `pyzotero`, `bibtexparser`, `habanero` (Crossref) | Crossref API, Semantic Scholar API |
|
||||
| `knowledge.note` | `obsidian`, `mubu` | `obsidian-cli` | raw markdown + `git`, `mdformat` | **Notion API, Roam API, Logseq sync** (known gap) |
|
||||
| `knowledge.outline` | `mubu` | — | OPML via `lxml`, YAML outlines | — |
|
||||
| `knowledge.synthesize` | `notebooklm`, `intelwatch` | — | LangChain / LlamaIndex + local LLM, `haystack` | Claude, OpenAI, Gemini (long-context summarization) |
|
||||
| `document.author` | `libreoffice`, `anygen` | — | `python-docx`, `openpyxl`, `python-pptx`, `reportlab` | Google Docs API, Microsoft Graph |
|
||||
| `document.format` | `libreoffice` | — | `pandoc` (native), `docx2pdf`, `weasyprint` | CloudConvert |
|
||||
| `document.pdf` | partial (`libreoffice`) | — | `pypdf`, `pdfplumber`, `pdf2image`, native `qpdf`/`pdftk` | Adobe PDF Services, Smallpdf |
|
||||
| `diagram.create` | `drawio`, `mermaid` | — | `graphviz`, `mermaid-cli`, `plantuml` | — |
|
||||
| `text.translate` | — | — | `argos-translate` | DeepL, Google Translate, OpenAI/Claude |
|
||||
| `publish.web` | — | `contentful`, `sanity` | `hugo` (native), `mkdocs`, `jekyll`, `pelican` | WordPress REST, Ghost Admin API, Medium API |
|
||||
| `publish.latex` | — | — | `latexmk` + `texlive` (native), `pylatex` | Overleaf API |
|
||||
|
||||
**Recipes**: `literature-review` (search → retrieve → reference → synthesize → author), `meeting-to-doc` (transcribe via S1 `text.transcribe` → synthesize → author → format), `blog-post` (synthesize → author → publish.web), `paper-draft` (reference → author → publish.latex).
|
||||
|
||||
**Known gaps**: **Notion** (highest-leverage missing CLI — recurs in S11); **Google Docs live editing**; a maintained LaTeX harness. Translation has no first-party CLI yet.
|
||||
|
||||
**Verdict**: deepest coverage of any scenario. Closing Notion would also close S11's note-taking gap — highest cross-scenario ROI in this group.
|
||||
|
||||
---
|
||||
|
||||
## 📐 S3 · 3D & CAD — **v2**
|
||||
|
||||
Mid-pipe (modeling, point cloud, GPU debug, engine export) is strong. Texturing and fabrication are the two structural gaps — an agent can model but can't realistically texture or 3D-print.
|
||||
|
||||
| Capability | Harness CLI | Public CLI | Python / native | Paid API (escalate) |
|
||||
|---|---|---|---|---|
|
||||
| `model.mesh` | `blender` | — | `trimesh`, `pymeshlab`, `open3d`, `pygalmesh` | — |
|
||||
| `model.parametric` | `freecad` | — | `cadquery`, `build123d`, `OpenSCAD` (native), `SolidPython` | Fusion 360 API, Onshape API |
|
||||
| `model.sculpt` | partial (`blender` sculpt mode) | — | — | **ZBrush, Nomad Sculpt** (known gap) |
|
||||
| `pointcloud.process` | `cloudcompare`, `cloudanalyzer` | — | `open3d`, `laspy`, `pdal` (native), `pyvista` | — |
|
||||
| `photogrammetry.reconstruct` | ❌ | — | `Meshroom` (native), `OpenMVG`+`OpenMVS`, `colmap` (native) | RealityCapture, Polycam API, Luma AI |
|
||||
| `material.texture` | partial (`blender` shader nodes) | — | `Pillow`, `numpy`-based PBR synthesis, `bpy` (scripted) | **Substance 3D, Polycam Materials, Adobe Sampler** (known gap) |
|
||||
| `render.preview` | `blender` (eevee), `godot` | — | `bpy` background renders | — |
|
||||
| `render.offline` | `blender` (cycles) | — | `bpy` + `cycles` (native) | Octane Cloud, RenderMan (offline farms) |
|
||||
| `gpu.debug` | `renderdoc` | — | `apitrace` (native), `vulkan-tools` | — |
|
||||
| `fabricate.slice` | ❌ | — | `PrusaSlicer` (native CLI), `CuraEngine` (native), `slic3r` | **Bambu Studio, OrcaSlicer** (known gap for API-driven) |
|
||||
| `fabricate.cam` | partial (`freecad` Path workbench) | — | `pycam`, `kiri:moto` (native) | Fusion 360 CAM, OnShape CAM |
|
||||
| `export.engine` | `godot`, `blender` (glTF/FBX export) | — | `pygltflib`, `trimesh` glTF writer | Unity Asset Pipeline, Unreal Datasmith |
|
||||
|
||||
**Recipes**: `printable-part` (parametric → slice → fabricate), `game-asset` (mesh → texture → export.engine), `reality-scan` (photogrammetry.reconstruct → mesh clean → export), `cnc-part` (parametric → cam → gcode).
|
||||
|
||||
**Known gaps**: `material.texture` (no good CLI or open alternative approaches Substance quality); `fabricate.slice` (native slicers exist but need an agent-native wrapper); `model.sculpt`.
|
||||
|
||||
**Verdict**: strongest technical scenario by depth — Blender + FreeCAD + CloudCompare + RenderDoc is rare coverage. Closing **texture + slicer** makes the full "idea → CAD → printed part" loop agentic.
|
||||
|
||||
---
|
||||
|
||||
## 🎮 S4 · Game Development — **v2**
|
||||
|
||||
All asset-creation capabilities are covered via the S3 stack; the gap is publishing and alternative engines.
|
||||
|
||||
| Capability | Harness CLI | Public CLI | Python / native | Paid API (escalate) |
|
||||
|---|---|---|---|---|
|
||||
| `game.engine` | `godot` | — | `pygame`, `arcade`, `panda3d`, `ursina` | **Unity, Unreal** (known gap) |
|
||||
| `asset.3d` | `blender`, `freecad` | — | `trimesh`, `pygltflib` | — |
|
||||
| `asset.2d` | `krita`, `inkscape` | — | `Pillow`, `aseprite` (native CLI), `pyxel` | OpenAI GPT-Image-1, Scenario GG, Leonardo AI |
|
||||
| `asset.audio` | `audacity` | — | `pydub`, `librosa`, `sfxr`/`jsfxr` | ElevenLabs SFX, Mubert |
|
||||
| `asset.notation` | `musescore` | — | `music21`, `mido`, `abjad` | — |
|
||||
| `ai.gen-asset` | — | `comfyui`, `jimeng` | `diffusers`, `replicate` | Scenario GG, Leonardo, OpenAI Image, Stability |
|
||||
| `playtest.agent` | `slay_the_spire_ii` | — | `gym`/`gymnasium` wrappers for headless games | — |
|
||||
| `build.package` | partial (`godot` export) | — | `pyinstaller` (for pygame), `butler` (itch CLI, native) | — |
|
||||
| `publish.store` | ❌ | — | `steamcmd` (native), `butler` (itch, native) | **Steamworks Web API, Epic Games Services, Google Play, App Store Connect** (known gap) |
|
||||
| `telemetry.ingest` | ❌ | `sentry` | `posthog` SDK, raw HTTP analytics | **GameAnalytics, Unity Analytics, PlayFab** (known gap) |
|
||||
|
||||
**Recipes**: `godot-jam-game` (engine + asset.2d + asset.audio + build.package), `agent-bot` (playtest.agent on an existing game), `ai-indie-short` (ai.gen-asset → asset.2d → engine → build), `steam-release` (build.package → publish.store → telemetry.ingest).
|
||||
|
||||
**Known gaps**: `publish.store` (Steam/itch wrappers exist natively but need agent-native harnesses); `telemetry.ingest`; `game.engine` for Unity/Unreal. `asset.notation` via MuseScore is a surprisingly strong differentiator — very few agent stacks can write sheet music.
|
||||
|
||||
**Verdict**: all creation covered, same agent-can-ship gap as S1 and S8 — the missing piece is *distribution*. `playtest.agent` (S14 overlap) is unique among frameworks.
|
||||
|
||||
---
|
||||
|
||||
## 🖼️ S5 · Image & Graphic Design — **v2**
|
||||
|
||||
All creation capabilities are strongly covered; Figma remains the single highest-leverage missing CLI (also affects S7 web/app).
|
||||
|
||||
| Capability | Harness CLI | Public CLI | Python / native | Paid API (escalate) |
|
||||
|---|---|---|---|---|
|
||||
| `visual.generate` | — | `comfyui`, `jimeng` | `diffusers` (SD/SDXL/FLUX), `replicate` | Midjourney (via Discord bridge), OpenAI GPT-Image-1, Ideogram, Stability, Recraft, Google Imagen, Nano Banana |
|
||||
| `visual.edit.raster` | `gimp`, `krita` | — | `Pillow`, `opencv-python`, `scikit-image`, `rembg` (bg removal) | Adobe Firefly, Cutout.pro, remove.bg |
|
||||
| `visual.edit.vector` | `inkscape` | — | `svgwrite`, `cairosvg`, `svgpathtools`, `vpype` | — |
|
||||
| `visual.mockup` | `sketch`, `drawio` | — | `drawio-desktop` (native), `penpot` (native), `figma-plugin-sdk` (partial) | **Figma API, Canva API, Penpot API** (known gap — Figma is the big one) |
|
||||
| `visual.diagram` | `drawio`, `mermaid` | — | `graphviz`, `plantuml`, `mermaid-cli` | — |
|
||||
| `visual.upscale` | — | — | `Real-ESRGAN`, `GFPGAN`, `upscayl` (native) | Topaz, Replicate upscalers |
|
||||
| `photo.library` | ❌ | — | `Pillow` + `exiftool` (native), `digikam` (native), `photoprism` (native/self-host) | Google Photos API, Apple Photos (limited) |
|
||||
| `photo.develop` | ❌ | — | `rawpy`, `darktable-cli` (native), `rawtherapee-cli` (native) | Adobe Lightroom API |
|
||||
| `publish.cms` | — | `contentful`, `sanity` | `hugo`, `mkdocs` | WordPress, Ghost |
|
||||
|
||||
**Recipes**: `social-card` (visual.generate → edit.raster → publish), `logo-set` (visual.edit.vector → export variants), `ui-wireframe` (visual.mockup → export), `photo-batch` (photo.develop → edit.raster → photo.library → publish), `icon-pack` (visual.edit.vector → batch export).
|
||||
|
||||
**Known gaps**: **Figma** (also blocks S7 UI/UX); `photo.library` and `photo.develop` have strong native CLIs but no harness. AI image escalation is the most paid-API-diverse capability in the matrix — the agent should consult the user's preferred provider.
|
||||
|
||||
**Verdict**: creation side is the deepest in the matrix after S2 — raster (GIMP + Krita), vector (Inkscape), AI (ComfyUI + Jimeng + 7 APIs), diagram (drawio + mermaid). Figma is the headline gap.
|
||||
|
||||
---
|
||||
|
||||
## 🎵 S6 · Music & Audio Production
|
||||
|
||||
| Stage | CLI | Gap |
|
||||
|---|---|---|
|
||||
| Compose/notation | `musescore` | — |
|
||||
| Audio edit | `audacity` | |
|
||||
| Record | `obs-studio` | |
|
||||
| **DAW** | ❌ | **Reaper, Ableton, Logic, FL** |
|
||||
| **AI music** | `suno`, `minimax-cli` | Udio, MusicGen |
|
||||
| **Voice/TTS** | `elevenlabs`, `minimax-cli` | OpenAI TTS |
|
||||
| Distribution | ❌ | Spotify (track mgmt), SoundCloud, DistroKid |
|
||||
|
||||
**Verdict**: still one of the weakest full scenarios, but no longer empty on the AI side. The biggest remaining gaps are a DAW and distribution.
|
||||
|
||||
---
|
||||
|
||||
## 🌐 S7 · Web / App Development
|
||||
|
||||
| Stage | CLI | Gap |
|
||||
|---|---|---|
|
||||
| Code LLM | `ollama`, `novita`, `minimax-cli` | — |
|
||||
| CMS/content | `contentful`, `sanity` | WordPress, Strapi |
|
||||
| E-commerce | `shopify` | Stripe, WooCommerce |
|
||||
| Test mocks | `wiremock` | Postman/Insomnia, Playwright |
|
||||
| Browser automation | `browser`, `safari`, `clibrowser` | — |
|
||||
| Vector DB | `chromadb` | Pinecone, Weaviate, Qdrant |
|
||||
| Process mgmt | `pm2` | — |
|
||||
| Error tracking | `sentry` | — |
|
||||
| Secrets | `1password-cli` | Doppler, Vault |
|
||||
| Terminal | `iterm2` | — |
|
||||
| **Hosting/deploy** | ❌ | **Vercel, Netlify, Cloudflare, Fly.io** |
|
||||
| **Cloud provider** | ❌ | **AWS (aws-cli exists), GCP, Azure** |
|
||||
| **Database** | ❌ | **Postgres, MySQL, Redis, MongoDB** |
|
||||
| **CI/CD** | ❌ | GitHub Actions (gh), CircleCI |
|
||||
| **Feature flags** | ❌ | LaunchDarkly, Unleash |
|
||||
| Logging/APM | ❌ | Datadog, New Relic, Grafana |
|
||||
| Analytics | ❌ | PostHog, Amplitude, GA |
|
||||
|
||||
**Verdict**: building blocks there, **but the deploy→observe loop is broken**. Closing `hosting + DB + analytics` would make web-dev fully agentic end-to-end.
|
||||
|
||||
---
|
||||
|
||||
## 📡 S8 · Creator Operations (stream-to-audience)
|
||||
|
||||
| Stage | CLI | Gap |
|
||||
|---|---|---|
|
||||
| Plan | LLM stack | — |
|
||||
| Stream | `obs-studio`, `zoom` | Twitch/YouTube Live |
|
||||
| Record | `openscreen` | — |
|
||||
| Edit | `kdenlive`, `shotcut` | — |
|
||||
| Caption | `videocaptioner` | — |
|
||||
| Thumbnail | `gimp`, `krita` | — |
|
||||
| **Upload** | ❌ | **YouTube, TikTok, Bilibili, Instagram** |
|
||||
| **Social promo** | ❌ | **Twitter/X, LinkedIn, Threads** |
|
||||
| Livestream chat | ❌ | Twitch API, StreamElements |
|
||||
|
||||
**Verdict**: same gap as S1 — **no social/platform distribution**. Highest-impact gap across the whole matrix because it breaks 3+ scenarios.
|
||||
|
||||
---
|
||||
|
||||
## 🤖 S9 · AI/ML & Data Science
|
||||
|
||||
| Stage | CLI | Gap |
|
||||
|---|---|---|
|
||||
| LLM inference | `ollama`, `novita`, `minimax-cli`, `notebooklm` | — |
|
||||
| Image gen | `comfyui`, `jimeng` | — |
|
||||
| Video gen | `jimeng`, `generate-veo-video` | — |
|
||||
| Workflow | `dify-workflow`, `n8n` | LangGraph, Flowise |
|
||||
| Vector DB | `chromadb` | — |
|
||||
| Domain ML (chem) | `unimol_tools` | — |
|
||||
| **Dataset mgmt** | ❌ | **HuggingFace CLI, kaggle** |
|
||||
| **Experiment tracking** | ❌ | **W&B, MLflow** |
|
||||
| **Training** | ❌ | Axolotl, Unsloth, trl |
|
||||
| **Eval** | ❌ | lm-eval-harness, promptfoo |
|
||||
| **Serving** | ❌ | Modal, Replicate, BentoML |
|
||||
|
||||
**Verdict**: inference side rich; **training/eval/serving totally absent** — no way for an agent to fine-tune and ship a model today.
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ S10 · DevOps / SRE
|
||||
|
||||
| Stage | CLI | Gap |
|
||||
|---|---|---|
|
||||
| Terminal | `iterm2` | |
|
||||
| Process | `pm2` | |
|
||||
| Error/APM | `sentry` | Datadog, New Relic |
|
||||
| Secrets | `1password-cli` | Vault, Doppler |
|
||||
| Network/DNS | `adguardhome` | Cloudflare DNS |
|
||||
| IoT fleet | `rms` | |
|
||||
| Blockchain node | `eth2-quickstart` | |
|
||||
| **K8s/Docker** | ❌ | kubectl/helm (standard) |
|
||||
| **Cloud** | ❌ | **AWS, GCP, Azure** |
|
||||
| **IaC** | ❌ | **Terraform, Pulumi** |
|
||||
| **Incident** | ❌ | PagerDuty, Opsgenie |
|
||||
| **Uptime** | ❌ | Statuspage, Uptime Kuma |
|
||||
|
||||
---
|
||||
|
||||
## 💬 S11 · Team Communication & Productivity
|
||||
|
||||
| Stage | CLI | Gap |
|
||||
|---|---|---|
|
||||
| IM (APAC) | `feishu`, `wecom` | |
|
||||
| IM (global) | ❌ | **Slack, Teams, Discord** |
|
||||
| Meetings | `zoom` | Google Meet, Webex |
|
||||
| Kanban | `seaclip` | **Linear, Jira, Asana, Trello** |
|
||||
| Notes | `obsidian`, `mubu` | Notion |
|
||||
| Email | ❌ | **Gmail, Outlook** |
|
||||
| Calendar | partial | **Google Calendar, Cal.com** |
|
||||
| Automation | `n8n` | Zapier, Make |
|
||||
|
||||
**Verdict**: China/APAC side strong (feishu + wecom), **Western SaaS totally absent**. Slack + Linear + Gmail would unblock most Western-team workflows.
|
||||
|
||||
---
|
||||
|
||||
## 🛒 S12 · E-Commerce
|
||||
|
||||
| Stage | CLI | Gap |
|
||||
|---|---|---|
|
||||
| Storefront | `shopify` | WooCommerce |
|
||||
| Catalog/CMS | `contentful`, `sanity` | — |
|
||||
| Asset gen | `jimeng`, `comfyui`, `krita`, `inkscape` | — |
|
||||
| Video ads | `jimeng`, `generate-veo-video`, `kdenlive` | — |
|
||||
| **Payments** | ❌ | **Stripe, PayPal, Square** |
|
||||
| **Email marketing** | ❌ | **Resend, Mailchimp, Klaviyo** |
|
||||
| **Shipping** | ❌ | ShipStation, EasyPost |
|
||||
| **Analytics** | ❌ | GA, Shopify Analytics deep |
|
||||
|
||||
---
|
||||
|
||||
## 🔬 S13 · Science / Academic
|
||||
|
||||
Strong (`zotero` + `notebooklm` + `exa` + `unimol_tools` + `mubu` + `obsidian` + `mermaid`) — mostly **missing LaTeX, arXiv, Jupyter-as-CLI, R/stats, domain-specific (PyMol/Chimera/QGIS)**.
|
||||
|
||||
---
|
||||
|
||||
## 🎯 S14 · Agentic Game Playing
|
||||
|
||||
Single CLI: `slay_the_spire_ii` — niche but demonstrates the pattern (agent drives a real GUI game through a local bridge API). Future expansion: other single-player games with mod APIs.
|
||||
|
||||
---
|
||||
|
||||
## 🛡️ S15 · Security / OSINT / Competitive Intel
|
||||
|
||||
Current: `intelwatch` (OSINT), `1password-cli` (secrets), `exa`/`safari` (search).
|
||||
**Gaps**: Shodan, Censys, VirusTotal, pentesting frameworks, secrets-scanning, CVE feeds.
|
||||
|
||||
---
|
||||
|
||||
## 💰 S16 · Finance / Trading / Crypto
|
||||
|
||||
Current: `eth2-quickstart` (Ethereum staking).
|
||||
**Gaps**: exchange CLIs (Coinbase, Binance), DeFi protocols, wallets, accounting (QuickBooks), invoicing (Stripe Billing).
|
||||
|
||||
---
|
||||
|
||||
## Closed-loop gap heatmap (where to invest)
|
||||
|
||||
Ranked by **how many scenarios unblock**:
|
||||
|
||||
| Gap | Impact (# scenarios) | Examples |
|
||||
|---|---|---|
|
||||
| 🔥 **Social/video upload CLIs** | 3 (S1, S8, S12) | YouTube, TikTok, Bilibili, Twitter/X |
|
||||
| 🔥 **Cloud/hosting CLI harnesses** | 2 (S7, S10) | Vercel, Cloudflare, Fly.io |
|
||||
| 🔥 **Slack / Linear / Gmail** | 2 (S7, S11) | completes Western team-comms |
|
||||
| ⭐ **Notion** | 2 (S2, S11) | closes knowledge stack and team notes |
|
||||
| ⭐ **Figma** | 2 (S5, S7) | UI/UX agent loop |
|
||||
| ⭐ **Udio / MusicGen** | 2 (S1, S6) | deeper AI music coverage beyond Suno |
|
||||
| ⭐ **MiniMax Voice / OpenAI TTS** | 3 (S1, S6, S8) | broader TTS coverage beyond ElevenLabs |
|
||||
| ⭐ **HF + W&B + training harness** | 1 (S9) | full ML loop |
|
||||
| ⭐ **3D texturing (Substance) + slicer (Cura/Bambu)** | 1 (S3) | 3D-to-physical loop |
|
||||
| ⭐ **Unity / Steam / itch** | 1 (S4) | full gamedev ship loop |
|
||||
| ⭐ **Stripe + Resend** | 1 (S12) | full e-com loop |
|
||||
|
||||
---
|
||||
|
||||
## Recommendations — making the matrix a first-class concept
|
||||
|
||||
1. **Add a `scenarios` field** to each registry entry (one CLI can belong to several). Then `cli-hub` can offer `cli-hub scenarios list` and `cli-hub scenarios <name> --missing` to directly surface gaps to an agent.
|
||||
2. **Flip the category system** from taxonomy (`image`, `video`, `ai`) to **capability domain** (`visual`, `audio`, `text`, `composite`, `package`, `publish`). Capability reveals closed-loop gaps; category doesn't.
|
||||
3. **Define each matrix as an ecosystem bundle, not just an install bundle**: every capability should document the best available first-party CLI, adjacent public CLIs, relevant APIs, and traditional package/native fallbacks.
|
||||
4. **Seed matrix installs** in cli-hub for the installable portion only: `cli-hub matrix install <name>` grabs the CLIs, while the paired matrix docs also name non-installable APIs and library-level options.
|
||||
5. **Publish this matrix as a living doc** — gaps in the matrix *are* the contributor roadmap. Missing CLIs are one category of gap; sometimes the immediate improvement is documenting the right API or fallback package before we build a harness.
|
||||
|
||||
---
|
||||
|
||||
## Appendix — full CLI inventory (by current registry category)
|
||||
|
||||
| Category | Harness CLIs | Public CLIs |
|
||||
|---|---|---|
|
||||
| ai | `comfyui`, `notebooklm`, `ollama`, `novita`, `dify-workflow` | `minimax-cli`, `generate-veo-video`, `jimeng` |
|
||||
| audio | `audacity` | `elevenlabs` |
|
||||
| 3d | `blender`, `freecad` | — |
|
||||
| automation | `n8n` | — |
|
||||
| communication | `zoom` | `feishu`, `wecom` |
|
||||
| database | `chromadb` | — |
|
||||
| design | `sketch` | — |
|
||||
| devops | `eth2-quickstart`, `pm2`, `iterm2` | `sentry`, `1password-cli` |
|
||||
| diagrams | `drawio`, `mermaid` | — |
|
||||
| game / gamedev | `slay_the_spire_ii`, `godot` | — |
|
||||
| generation | `anygen` | — |
|
||||
| graphics | `renderdoc`, `cloudcompare`, `cloudanalyzer` | — |
|
||||
| image | `gimp`, `inkscape`, `krita` | — |
|
||||
| knowledge | `obsidian` | `obsidian-cli` |
|
||||
| music | `musescore` | `suno` |
|
||||
| network | `adguardhome`, `rms` | — |
|
||||
| office | `libreoffice`, `mubu`, `zotero` | — |
|
||||
| osint | `intelwatch` | — |
|
||||
| project-management | `seaclip` | — |
|
||||
| science | `unimol_tools` | — |
|
||||
| search | `exa` | — |
|
||||
| streaming | `obs-studio` | — |
|
||||
| testing | `wiremock` | — |
|
||||
| video | `kdenlive`, `shotcut`, `openscreen`, `videocaptioner` | — |
|
||||
| web | `browser`, `safari`, `clibrowser` | `contentful`, `sanity`, `shopify` |
|
||||
@@ -0,0 +1,182 @@
|
||||
# CLI Matrix Registry Schema (v2 — capability-based)
|
||||
|
||||
Status: **draft**. Applies to `matrix_registry.json` entries and the companion `cli-hub-matrix/<name>/SKILL.md` files.
|
||||
|
||||
The v1 layout organized each matrix by **pipeline stages** (Ideate → Acquire → Edit → ...). This silently imposed a linear ordering that most real workflows do not follow (a screencast tutorial, a generative short, and a podcast-to-video all touch different subsets of the same tools in different orders).
|
||||
|
||||
The v2 layout organizes each matrix around **capabilities** — verbs the agent can compose on demand. A workflow becomes a recipe over capabilities, not a march through stages.
|
||||
|
||||
---
|
||||
|
||||
## Top-level shape
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"meta": { "repo": "...", "description": "...", "updated": "YYYY-MM-DD" },
|
||||
"matrices": [
|
||||
{
|
||||
"name": "video-creation",
|
||||
"display_name": "Video Creation & Editing",
|
||||
"version": "3",
|
||||
"schema_version": "2",
|
||||
"matrix_id": "V1",
|
||||
"category": "video",
|
||||
"description": "...",
|
||||
"homepage": "...",
|
||||
"skill_md": "cli-hub-matrix/video-creation/SKILL.md",
|
||||
|
||||
"capabilities": [ /* see below */ ],
|
||||
"recipes": [ /* see below */ ],
|
||||
"known_gaps": [ /* see below */ ],
|
||||
|
||||
"clis": [ "...flat list preserved for cli-hub install..." ]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
`clis[]` is kept as a flat install list so `cli-hub matrix install <name>` continues to work unchanged. Everything else is new in v2.
|
||||
|
||||
---
|
||||
|
||||
## Capability
|
||||
|
||||
A capability is a single verb the agent can invoke — not a stage, not a tool. Capabilities are intentionally coarse; a capability has 1..N providers and an agent picks among them using the decision rubric in SKILL.md.
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"id": "visual.generate",
|
||||
"intent": "Produce a video clip from a text prompt or reference image.",
|
||||
"inputs": ["prompt:text", "ref_image?:path", "duration?:seconds"],
|
||||
"outputs": ["video_clip:path"],
|
||||
"providers": [ /* see below */ ],
|
||||
"skill_search_hints": ["text-to-video", "video generation"]
|
||||
}
|
||||
```
|
||||
|
||||
Naming: `<domain>.<verb>`. Domains in v2: `visual`, `audio`, `text`, `composite`, `package`, `publish`. Verbs are lowercase, imperative-ish nouns (`generate`, `capture`, `edit`, `synthesize`, `transcribe`, `translate`, `assemble`, `overlay`, `thumbnail`, `encode`, `upload`).
|
||||
|
||||
### Cross-matrix capabilities
|
||||
|
||||
Capabilities are shared across matrices by ID. `visual.generate` means the same thing in V1 (video) and V2 (image) only if the `outputs` differ — if they would collide, disambiguate (`visual.generate.video` vs `visual.generate.image`). Long-term, common capabilities move to a shared `capabilities.json`; for now each matrix redeclares them.
|
||||
|
||||
---
|
||||
|
||||
## Provider
|
||||
|
||||
A provider is one concrete way to satisfy a capability. Exactly one provider kind per entry.
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"kind": "api", // harness-cli | public-cli | python | native | api
|
||||
"name": "Runway Gen-4",
|
||||
"invocation_hint": "POST https://api.runway.ml/v1/...",
|
||||
"requires": {
|
||||
"env": ["RUNWAY_API_KEY"],
|
||||
"binary": [],
|
||||
"package": []
|
||||
},
|
||||
"cost_tier": "paid", // free | metered | paid | premium
|
||||
"quality_tier": "high", // basic | good | high | sota
|
||||
"offline": false,
|
||||
"notes": "Best for cinematic motion; ~$0.05/sec as of 2026-04."
|
||||
}
|
||||
```
|
||||
|
||||
| Kind | Meaning | Typical `requires` |
|
||||
|---|---|---|
|
||||
| `harness-cli` | A CLI-Anything harness (first-party) | `binary: ["cli-anything-<x>"]` |
|
||||
| `public-cli` | Third-party CLI from `cli-hub` | `binary: ["<entry>"]` |
|
||||
| `python` | An importable Python package | `package: ["..."]` |
|
||||
| `native` | A system binary / shell pipeline | `binary: ["ffmpeg", ...]` |
|
||||
| `api` | A hosted API the agent calls directly | `env: ["..._API_KEY"]` |
|
||||
|
||||
`requires` is a **preflight contract**: the agent can deterministically check it before choosing the provider. If any `env`/`binary`/`package` is missing, that provider is unavailable *unless* the agent explicitly escalates (see below).
|
||||
|
||||
---
|
||||
|
||||
## Decision rubric (canonical — SKILL.md references this)
|
||||
|
||||
When an agent needs to pick a provider for a capability, walk this ladder:
|
||||
|
||||
1. **Available & adequate.** Prefer providers whose `requires` is already satisfied in the environment, ranked by `quality_tier` then inverse `cost_tier`.
|
||||
2. **Free-to-install.** If none available, prefer `python` / `native` providers that can be installed in seconds (`pip install`, `apt`, single binary) without credentials.
|
||||
3. **Harness install.** If the task warrants it, install a `harness-cli` or `public-cli`.
|
||||
4. **Paid API escalation.** Only when (a) lower tiers can't meet the quality bar, (b) env already holds the key, or (c) the agent explicitly asks the user and the user consents. Never silently call a paid API without one of these conditions.
|
||||
|
||||
Providers marked `offline: true` are preferred when the user indicates an offline context.
|
||||
|
||||
---
|
||||
|
||||
## Suggest-to-user templates
|
||||
|
||||
When escalating to an API whose `requires.env` is missing, the agent uses a canned suggestion so the UX is consistent:
|
||||
|
||||
```text
|
||||
To enable <capability> via <provider.name>, set <env var>.
|
||||
Cost: <cost_tier notes>
|
||||
Quality: <quality_tier>
|
||||
Reply 'skip' to fall back to <next provider.name>.
|
||||
```
|
||||
|
||||
SKILL.md files may override the template per matrix but must keep the four slots (capability, provider, env, fallback).
|
||||
|
||||
---
|
||||
|
||||
## Recipe
|
||||
|
||||
A recipe is a named composition of capabilities — not a pipeline. It tells the agent *which capabilities this workflow needs*, not the order.
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"id": "screencast-tutorial",
|
||||
"description": "Record a screen walkthrough with narration and subtitles.",
|
||||
"capabilities_used": [
|
||||
"visual.capture", "audio.capture", "text.transcribe",
|
||||
"composite.overlay", "package.thumbnail", "package.encode"
|
||||
],
|
||||
"notes": "Order is flexible; capture first, transcribe from captured audio, overlay subs last."
|
||||
}
|
||||
```
|
||||
|
||||
Agents pick a recipe to narrow the capability set, then apply the decision rubric per capability.
|
||||
|
||||
---
|
||||
|
||||
## Known gaps
|
||||
|
||||
```jsonc
|
||||
"known_gaps": [
|
||||
{
|
||||
"capability": "publish.upload",
|
||||
"reason": "No first-party or public CLI for YouTube/TikTok/Bilibili yet.",
|
||||
"workaround": "Instruct user to upload manually, or escalate to a custom API script."
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
Surfacing gaps in-schema lets the agent tell the user where the ecosystem genuinely ends instead of flailing.
|
||||
|
||||
---
|
||||
|
||||
## Preflight block (SKILL.md convention)
|
||||
|
||||
Each SKILL.md declares a single preflight block the agent runs once per session:
|
||||
|
||||
```bash
|
||||
# Per-capability detection: CLI installed? package importable? binary on PATH? env var set?
|
||||
cli-hub list --json
|
||||
python -c "import importlib, sys; [print(m, importlib.util.find_spec(m) is not None) for m in ('moviepy','whisper','pydub','PIL')]"
|
||||
for b in ffmpeg sox convert magick; do command -v "$b" >/dev/null && echo "$b: yes" || echo "$b: no"; done
|
||||
for e in RUNWAY_API_KEY ELEVENLABS_API_KEY MINIMAX_API_KEY OPENAI_API_KEY; do [ -n "${!e}" ] && echo "$e: set" || echo "$e: unset"; done
|
||||
```
|
||||
|
||||
The agent caches the result and consults it when the decision rubric runs.
|
||||
|
||||
---
|
||||
|
||||
## Migration notes
|
||||
|
||||
- v1 `stages[]` → v2 `capabilities[]`. Stage name becomes capability `id`; `goal` becomes `intent`; `clis` + `alternatives` flatten into `providers[]` with an explicit `kind`.
|
||||
- v1 implied order → v2 `recipes[]`. If a matrix really does have a canonical order, express it as a recipe named `default`.
|
||||
- `cli-hub` should keep reading v1 entries (by absence of `schema_version`) until all matrices are migrated.
|
||||
@@ -0,0 +1,374 @@
|
||||
# S1 · Video Creation — End-to-End Test Plan
|
||||
|
||||
**Scenario**: `video-creation` · **Matrix**: [`cli-hub-matrix/video-creation/SKILL.md`](../../../cli-hub-matrix/video-creation/SKILL.md) · **Schema**: [`matrix_registry.schema.md`](../matrix_registry.schema.md)
|
||||
|
||||
Purpose: validate that the S1 capability matrix supports realistic, end-to-end agent workflows — not just single-capability demos. Each task below hits **3+ capabilities** and is written as a checkable todo list so an evaluator can step through it and mark progress.
|
||||
|
||||
**How to run a task**
|
||||
1. Pick a task, read its *setup* and *intent*.
|
||||
2. Run the preflight block from [`SKILL.md`](../../../cli-hub-matrix/video-creation/SKILL.md) to know what's available.
|
||||
3. Walk the todo list, picking providers via the decision rubric. When a provider's `requires` is unmet, either fall back or escalate via the suggest-to-user template — **do not silently skip**.
|
||||
4. At the end of each task, verify acceptance criteria and note which providers were actually chosen.
|
||||
|
||||
**Scoring** (optional, for evaluator): for each task, record per-capability the provider chosen, whether it succeeded, and whether the agent *suggested* rather than silently skipped when a paid API was the best fit.
|
||||
|
||||
---
|
||||
|
||||
## Task 1 — Product launch explainer (2-minute)
|
||||
|
||||
**Intent**: A SaaS startup wants a 90–120s launch video mixing generated hero shots, a real screen recording of the product, a scripted voiceover, light background music, burned captions, a thumbnail, and a social card. English only. Distribution is manual.
|
||||
|
||||
**Capabilities exercised**: `visual.generate`, `visual.capture`, `audio.synthesize`, `audio.music`, `composite.assemble`, `text.transcribe`, `composite.overlay`, `package.thumbnail`, `package.encode`. **Touches 9 capabilities.**
|
||||
**Recipe alignment**: blend of `ai-short` + `screencast-tutorial`.
|
||||
|
||||
- [ ] Preflight: list `cli-hub list --json`, check `ffmpeg`, `whisper`, TTS keys, any image-gen keys
|
||||
- [ ] Draft the 3-beat script (hook / demo / CTA), ~160 words total, save as `script.md`
|
||||
- [ ] Decide per-capability providers using the decision rubric; record the plan in `plan.json`
|
||||
- [ ] Generate 2 hero clips via `visual.generate` (veo or Seedance escalation or local diffusers fallback), 4–5s each
|
||||
- [ ] Record the product demo via `cli-anything-openscreen` at 1080p for ~45s (hide cursor during dead moments)
|
||||
- [ ] Generate narration via `audio.synthesize` (prefer `elevenlabs` if key present, else `edge-tts`)
|
||||
- [ ] Generate a loopable background bed via `audio.music` (`suno` or `audiocraft`) ~2 minutes, mood = "uplifting neutral"
|
||||
- [ ] Duck the music under narration (-12 dB) using `ffmpeg` sidechain or `pydub`
|
||||
- [ ] Assemble the timeline via `composite.assemble` (`cli-anything-kdenlive` preferred), order: hero → demo → hero → CTA
|
||||
- [ ] Transcribe the narration via `text.transcribe` (whisper locally) → `subs.srt`
|
||||
- [ ] Burn captions via `composite.overlay` (`ffmpeg -vf subtitles`) with a readable font + stroke
|
||||
- [ ] Create a 1280×720 thumbnail via `package.thumbnail` (`cli-anything-gimp` or Pillow) including product wordmark + tagline
|
||||
- [ ] Export a 1920×1080 H.264 master via `package.encode` (`ffmpeg -c:v libx264 -crf 18`)
|
||||
- [ ] Write `publish_notes.md` instructing the user how to upload to YouTube/LinkedIn (known gap)
|
||||
|
||||
**Acceptance criteria**
|
||||
- [ ] Final file is 90–120s, 1080p, audible voiceover + ducked music
|
||||
- [ ] Captions are legible and match the narration word-for-word (±5%)
|
||||
- [ ] Thumbnail is 1280×720, under 2MB, contains the product name
|
||||
- [ ] Every provider choice is recorded in `plan.json` with reason
|
||||
- [ ] Agent either used or *explicitly suggested* a paid provider when quality mattered; no silent skips
|
||||
|
||||
---
|
||||
|
||||
## Task 2 — Multilingual developer tutorial (EN → ES/JA/ZH)
|
||||
|
||||
**Intent**: Record a real screen tutorial explaining a CLI tool, then ship four language variants. Voiceover can be re-synthesized per language; subtitles must exist in all four.
|
||||
|
||||
**Capabilities exercised**: `visual.capture`, `audio.capture`, `text.transcribe`, `text.translate`, `audio.synthesize`, `composite.overlay`, `composite.assemble`, `package.thumbnail`, `package.encode`. **Touches 9 capabilities × 4 locales.**
|
||||
**Recipe alignment**: `screencast-tutorial` ×4.
|
||||
|
||||
- [ ] Preflight; check for `DEEPL_API_KEY`, `ELEVENLABS_API_KEY`, and `whisper` locally
|
||||
- [ ] Record screen + mic at 1440p via `cli-anything-openscreen` (~5–8 minutes)
|
||||
- [ ] Denoise captured audio via `cli-anything-audacity` or `noisereduce`
|
||||
- [ ] Transcribe EN via `text.transcribe` (whisper large-v3 or AssemblyAI if key) → `en.srt`
|
||||
- [ ] Translate `en.srt` to `es.srt`, `ja.srt`, `zh.srt` via `text.translate` (DeepL preferred; fall back to `argos-translate`)
|
||||
- [ ] Produce re-narrated voiceover for each language via `audio.synthesize` (ElevenLabs multilingual model or MiniMax)
|
||||
- [ ] For each locale: swap audio track via `composite.assemble`, align to video length (time-stretch narration ±5% if needed)
|
||||
- [ ] Burn captions for each locale via `composite.overlay` with appropriate font (CJK font for zh/ja)
|
||||
- [ ] Create 4 thumbnails (flag + title in localized text) via `package.thumbnail`
|
||||
- [ ] Encode H.264 MP4 per locale via `package.encode`
|
||||
- [ ] Produce `upload_matrix.md` listing which locale goes where (YT, B站, niconico, etc.)
|
||||
|
||||
**Acceptance criteria**
|
||||
- [ ] 4 final videos, synchronized visually to the same screen recording
|
||||
- [ ] Captions in each locale present and synced; CJK glyphs render correctly
|
||||
- [ ] Re-narrated audio is intelligible and roughly matches on-screen action
|
||||
- [ ] If DeepL key missing, agent offered escalation with cost note; otherwise used `argos-translate`
|
||||
|
||||
---
|
||||
|
||||
## Task 3 — Long-form podcast-to-video (60-minute episode)
|
||||
|
||||
**Intent**: A 60-minute audio-only podcast needs a YouTube video version: static host image, animated waveform/spectrogram, chapter markers derived from topic shifts, burned captions, thumbnail, and a highlight reel for socials.
|
||||
|
||||
**Capabilities exercised**: `audio.capture`, `text.transcribe`, `composite.overlay`, `composite.assemble`, `package.thumbnail`, `package.encode`. **Touches 6 capabilities, long-form stress test.**
|
||||
**Recipe alignment**: `podcast-to-video` + derived highlight reel.
|
||||
|
||||
- [ ] Preflight; verify disk space (≥20GB free for 60-min 1080p masters)
|
||||
- [ ] Clean source audio via `audio.capture` providers (`noisereduce` + EQ in Audacity if needed)
|
||||
- [ ] Transcribe full 60 min via `text.transcribe` (whisper large-v3 locally if GPU; else chunk into 10-min segments and stitch)
|
||||
- [ ] Detect chapter boundaries: cluster transcript by topic via `knowledge.synthesize` (S2 overlap) or prompt a local LLM; produce `chapters.json` with timestamps
|
||||
- [ ] Render animated waveform via `ffmpeg showwaves` or `moviepy`
|
||||
- [ ] Composite waveform onto host still via `composite.overlay`
|
||||
- [ ] Burn captions per chapter via `composite.overlay` (turn on/off per section if you prefer)
|
||||
- [ ] Assemble final via `composite.assemble` (native `ffmpeg concat` is fine; `moviepy` if you want per-chapter crossfades)
|
||||
- [ ] Encode 1080p master + 720p preview via `package.encode`
|
||||
- [ ] Create thumbnail via `package.thumbnail`: host portrait + episode title + # number
|
||||
- [ ] Extract 3× 60-second highlight clips based on `chapters.json` importance scores
|
||||
- [ ] For each highlight: re-crop 9:16, add kinetic captions, export separately
|
||||
- [ ] Produce `youtube_description.md` with timestamped chapter links from `chapters.json`
|
||||
|
||||
**Acceptance criteria**
|
||||
- [ ] Main video is ≈60 min, in sync with source audio throughout
|
||||
- [ ] Chapters in the YouTube description match visible chapter markers in the video
|
||||
- [ ] Highlights are 55–65s each, vertical (1080×1920), with readable kinetic captions
|
||||
- [ ] Transcription quality: spot-check at 5%/50%/95% marks — >95% word accuracy
|
||||
|
||||
---
|
||||
|
||||
## Task 4 — Talking-head YouTube essay with AI b-roll
|
||||
|
||||
**Intent**: A 6-minute personal essay: the creator on webcam delivering the script, cutaways of AI-generated b-roll illustrating key concepts, music bed under the voice, captions, thumbnail A/B variants.
|
||||
|
||||
**Capabilities exercised**: `visual.capture`, `visual.generate`, `audio.capture`, `text.transcribe`, `audio.music`, `composite.assemble`, `composite.overlay`, `package.thumbnail` (×2), `package.encode`. **Touches 9 capabilities, A/B variant testing.**
|
||||
**Recipe alignment**: `talking-head-explainer`.
|
||||
|
||||
- [ ] Preflight
|
||||
- [ ] Record webcam + lav mic via `visual.capture` + `audio.capture` at 1080p60
|
||||
- [ ] Transcribe raw take via `text.transcribe`; mark filler words / retakes in `edit_plan.json`
|
||||
- [ ] Cut silences and retakes per `edit_plan.json` via `composite.assemble` (`moviepy` is fine here)
|
||||
- [ ] Identify 6–10 "cutaway moments" from the transcript where b-roll should appear
|
||||
- [ ] Generate b-roll clips via `visual.generate` (prefer local `diffusers` SVD for cost; escalate to Runway/Kling/Seedance if quality bar demands it — ask user)
|
||||
- [ ] Add ducked music bed via `audio.music` (`audiocraft` local is fine for essay mood)
|
||||
- [ ] Overlay captions via `composite.overlay` (styled like a premium video essay, not burn-in ASS)
|
||||
- [ ] Create **two** thumbnail variants via `package.thumbnail`:
|
||||
- [ ] A: close-up face + bold question
|
||||
- [ ] B: abstract b-roll frame + short headline
|
||||
- [ ] Encode master + a 720p preview via `package.encode`
|
||||
- [ ] Write `ab_test_plan.md` describing which thumbnail to push first and how to swap after 48h
|
||||
|
||||
**Acceptance criteria**
|
||||
- [ ] Cutaways land on conceptual beats, not mid-sentence
|
||||
- [ ] Music never clashes with voice at crucial lines (check by RMS over the voiceover)
|
||||
- [ ] Both thumbnails exported at 1280×720, < 2MB each
|
||||
- [ ] Agent documented the cost/quality tradeoff if it escalated to a paid video API
|
||||
|
||||
---
|
||||
|
||||
## Task 5 — Beat-synced AI music video
|
||||
|
||||
**Intent**: Generate a 90-second song via AI music, then drive visual cuts to match the beat. Lyrics overlay. Thumbnail. No captions.
|
||||
|
||||
**Capabilities exercised**: `audio.music`, `visual.generate`, `composite.assemble`, `composite.overlay`, `package.thumbnail`, `package.encode`. **Touches 6 capabilities, timing-critical.**
|
||||
**Recipe alignment**: extension of `ai-short`.
|
||||
|
||||
- [ ] Preflight; confirm `suno`/Udio/minimax availability
|
||||
- [ ] Write lyrics (3 verses + chorus); save as `lyrics.txt`
|
||||
- [ ] Generate 90s song via `audio.music` (Suno preferred; fall back to `audiocraft`)
|
||||
- [ ] Beat-detect via `librosa.beat.beat_track` → `beats.json`
|
||||
- [ ] Plan ~30 short clips (~3s each) across the song, bias cuts to beat timestamps
|
||||
- [ ] Generate each clip via `visual.generate` (seed-locked per shot so re-rolls are deterministic)
|
||||
- [ ] Assemble on the beat via `composite.assemble` using `ffmpeg concat` with exact frame trims from `beats.json`
|
||||
- [ ] Overlay stylized lyrics via `composite.overlay` (word-by-word karaoke-style using ASS or ffmpeg drawtext+timecodes)
|
||||
- [ ] Create a striking thumbnail via `package.thumbnail` (pick best frame + title treatment)
|
||||
- [ ] Encode at 1080p60 via `package.encode`
|
||||
|
||||
**Acceptance criteria**
|
||||
- [ ] Cuts land within ±1 frame of detected beats (check via `ffprobe` packet timestamps vs `beats.json`)
|
||||
- [ ] Lyrics appear on the right word at the right time (sample 5 random lines)
|
||||
- [ ] No audio desync at end (lengths match within 50ms)
|
||||
|
||||
---
|
||||
|
||||
## Task 6 — Reaction video (imported clip + webcam + commentary)
|
||||
|
||||
**Intent**: Creator watches a 4-minute news clip and reacts. Final video has PiP webcam, full-screen source, running captions of the creator's commentary, plus bleeped moments.
|
||||
|
||||
**Capabilities exercised**: `visual.capture`, `audio.capture`, `text.transcribe`, `composite.assemble`, `composite.overlay`, `audio.capture` (bleep edit), `package.thumbnail`, `package.encode`. **Touches 8 capabilities.**
|
||||
|
||||
- [ ] Preflight
|
||||
- [ ] Load source clip (user-provided); verify copyright/fair-use notes with user before continuing
|
||||
- [ ] Record webcam + mic via `visual.capture` + `audio.capture` synced with source playback
|
||||
- [ ] Transcribe creator's reaction track via `text.transcribe` → `reaction.srt`
|
||||
- [ ] Identify bleep candidates (profanity list) in `reaction.srt`; edit audio to bleep those windows (`ffmpeg` sine tone overlay)
|
||||
- [ ] Composite: source full-screen, webcam PiP bottom-right, via `composite.assemble`
|
||||
- [ ] Overlay captions from `reaction.srt` via `composite.overlay` (styled distinct from any source subs)
|
||||
- [ ] Thumbnail via `package.thumbnail` with split-screen layout (source still + creator reaction still)
|
||||
- [ ] Encode via `package.encode`
|
||||
|
||||
**Acceptance criteria**
|
||||
- [ ] PiP never covers faces in the source clip (manual spot check)
|
||||
- [ ] Bleeps are audible and exactly cover flagged words (±100ms)
|
||||
- [ ] Agent asked the user about rights before proceeding
|
||||
|
||||
---
|
||||
|
||||
## Task 7 — Live-event recap with chapters and highlight grid
|
||||
|
||||
**Intent**: Concatenate footage from 3 camera angles of an event into a 7-minute recap with transitions, insert on-screen chapter titles, export a grid of thumbnails (one per chapter) as a single image.
|
||||
|
||||
**Capabilities exercised**: `composite.assemble`, `composite.overlay`, `text.transcribe` (for auto-chapter naming from any spoken words), `package.thumbnail`, `package.encode`. **Touches 5 capabilities, heavy on assembly.**
|
||||
|
||||
- [ ] Preflight
|
||||
- [ ] Receive 3 input files; probe via `ffprobe` to verify frame rates / resolutions; transcode to a common intermediate if needed
|
||||
- [ ] Detect scene cuts via `ffmpeg -vf "select='gt(scene,0.3)'"` per camera
|
||||
- [ ] Assemble multi-cam edit via `composite.assemble` (manual cut list is fine; rotate angles every 8–15s)
|
||||
- [ ] Transcribe any ambient speech via `text.transcribe` for chapter naming
|
||||
- [ ] Overlay chapter title cards via `composite.overlay` at 6 chapter boundaries
|
||||
- [ ] Generate chapter thumbnails (6 stills), compose into a 3×2 grid via `package.thumbnail` (Pillow or ImageMagick)
|
||||
- [ ] Encode final recap via `package.encode`
|
||||
|
||||
**Acceptance criteria**
|
||||
- [ ] Final duration = 7 min ±10s
|
||||
- [ ] Chapter cards fade in/out cleanly (no single-frame flash)
|
||||
- [ ] Grid image is 1920×1080, 6 panels evenly sized, labeled
|
||||
|
||||
---
|
||||
|
||||
## Task 8 — Offline-only documentary cut
|
||||
|
||||
**Intent**: Produce a 5-minute documentary-style piece without any cloud APIs (user is in a regulated / offline environment). All providers must have `offline: true`.
|
||||
|
||||
**Capabilities exercised**: `audio.capture`, `text.transcribe` (whisper local), `composite.assemble`, `composite.overlay`, `package.thumbnail`, `package.encode`. Explicitly **excludes** paid APIs and cloud TTS.
|
||||
|
||||
- [ ] Preflight; confirm network unreachable or flag "offline mode" for this task
|
||||
- [ ] Enumerate offline providers per capability in `offline_plan.json`
|
||||
- [ ] Record narration via `audio.capture` (mic) in Audacity
|
||||
- [ ] Assemble existing footage via `composite.assemble` (`kdenlive` or `ffmpeg`)
|
||||
- [ ] Transcribe via whisper local → `subs.srt`
|
||||
- [ ] Burn captions via `composite.overlay` (ffmpeg subtitles filter)
|
||||
- [ ] Generate thumbnail via `cli-anything-gimp` or Pillow (no cloud image gen)
|
||||
- [ ] Encode master via `package.encode`
|
||||
- [ ] If the agent at any point would benefit from an API, record *what* it would have escalated to in `escalations_declined.md`
|
||||
|
||||
**Acceptance criteria**
|
||||
- [ ] Network trace (or declared offline) shows zero external API calls
|
||||
- [ ] Final deliverable fully produced with offline providers
|
||||
- [ ] `escalations_declined.md` lists at least 2 spots where the agent would have preferred an API (honesty check)
|
||||
|
||||
---
|
||||
|
||||
## Task 9 — Short-form vertical cross-platform (TikTok / Reels / Shorts)
|
||||
|
||||
**Intent**: Take one long horizontal clip (90s) and produce 3 platform-specific vertical variants with kinetic captions, platform-safe aspect, watermark, and correct encoding profiles.
|
||||
|
||||
**Capabilities exercised**: `composite.assemble`, `composite.overlay`, `text.transcribe`, `package.thumbnail` (×3), `package.encode` (×3). **Touches 5 capabilities, strict format matrix.**
|
||||
|
||||
- [ ] Preflight
|
||||
- [ ] Transcribe source via `text.transcribe` with word-level timestamps
|
||||
- [ ] Reframe source to 9:16 via `composite.assemble`, tracking the subject (ffmpeg + opencv face/object center or manual crops per segment)
|
||||
- [ ] Generate kinetic captions (word-by-word pop) via `composite.overlay` with ASS karaoke or ffmpeg drawtext
|
||||
- [ ] Add platform watermark via `composite.overlay` (TikTok: bottom-left username; Reels: none, rely on caption; Shorts: bottom-right)
|
||||
- [ ] Thumbnail per platform via `package.thumbnail` (TikTok/Reels may auto-generate, still produce a cover still)
|
||||
- [ ] Encode three variants per platform guidelines:
|
||||
- [ ] TikTok: 1080×1920, H.264 High, ~16 Mbps, ≤60s trimmed version
|
||||
- [ ] Reels: 1080×1920, H.264, ≤90s
|
||||
- [ ] Shorts: 1080×1920, H.264, ≤60s
|
||||
- [ ] Produce `publish_checklist.md` with manual upload instructions + hashtags per platform (known gap)
|
||||
|
||||
**Acceptance criteria**
|
||||
- [ ] All three files open in their respective platform specs without re-encode warnings
|
||||
- [ ] Captions never cover the subject's face during close-ups
|
||||
- [ ] Agent noted `publish.upload` as a known gap and produced manual instructions rather than pretending to upload
|
||||
|
||||
---
|
||||
|
||||
## Task 10 — Localization re-cut with voice swap (ZH → EN)
|
||||
|
||||
**Intent**: The user supplies a finished Chinese video (with Chinese burn-in captions) and wants an English re-cut: strip the burn-ins, add English narration that matches the original pacing, add English captions, keep the same music bed.
|
||||
|
||||
**Capabilities exercised**: `composite.overlay` (inverse — crop/mask captions), `text.transcribe`, `text.translate`, `audio.synthesize`, `audio.capture` (isolate music bed), `composite.assemble`, `composite.overlay`, `package.thumbnail`, `package.encode`. **Touches 9 capabilities, advanced audio routing.**
|
||||
|
||||
- [ ] Preflight; confirm DeepL / OpenAI translate availability, multilingual TTS
|
||||
- [ ] Crop or blur the bottom-third caption band via `composite.overlay` (`ffmpeg` drawbox or delogo)
|
||||
- [ ] Isolate music bed: attempt vocal removal via `demucs` or spectral subtraction; verify clean bed at representative timecodes
|
||||
- [ ] Transcribe the original Chinese narration via `text.transcribe` (whisper zh)
|
||||
- [ ] Translate `zh.srt` → `en.srt` via `text.translate` (DeepL preferred; confirm translations make sense via a quick LLM review)
|
||||
- [ ] Generate English voiceover via `audio.synthesize` with pacing aligned to original SRT timecodes (time-stretch ±10% if needed)
|
||||
- [ ] Mix new narration over the isolated music bed via `ffmpeg` (ducking)
|
||||
- [ ] Rebuild final via `composite.assemble` (picture from masked source + new audio track)
|
||||
- [ ] Burn English captions via `composite.overlay`
|
||||
- [ ] Translate and rebuild thumbnail text via `package.thumbnail`
|
||||
- [ ] Encode via `package.encode`
|
||||
|
||||
**Acceptance criteria**
|
||||
- [ ] No Chinese text visible in final (spot-check 5 random frames)
|
||||
- [ ] English narration is synced to visible action within ±500ms
|
||||
- [ ] Music bed isn't noticeably degraded by vocal-removal artifacts
|
||||
- [ ] Agent flagged any section where vocal isolation failed rather than shipping garbled audio
|
||||
|
||||
---
|
||||
|
||||
## Task 11 — Meeting recording → executive explainer
|
||||
|
||||
**Intent**: A 45-minute Zoom-style recording needs to become a 3-minute executive-audience explainer: extract the key decisions, re-narrate over curated screen captures, add chapter markers, branded opening/closing.
|
||||
|
||||
**Capabilities exercised**: `audio.capture`, `text.transcribe`, `audio.synthesize` (re-narration), `visual.capture` (if re-shooting), `composite.assemble`, `composite.overlay`, `package.thumbnail`, `package.encode`. Overlaps with S2 `knowledge.synthesize`. **Touches 8 capabilities.**
|
||||
|
||||
- [ ] Preflight; confirm TTS, synthesis LLM, and `ffmpeg`
|
||||
- [ ] Transcribe full meeting via `text.transcribe`
|
||||
- [ ] Summarize into 3–5 decisions via LLM (S2 `knowledge.synthesize`); produce `summary.md`
|
||||
- [ ] Extract the original screen captures that best illustrate each decision (copy frames/segments from source video)
|
||||
- [ ] Generate exec-tone re-narration via `audio.synthesize` matching the summary
|
||||
- [ ] Assemble final via `composite.assemble`: branded intro (`visual.generate` or pre-made card), decision sections, closing CTA
|
||||
- [ ] Overlay chapter titles + lower-third decision labels via `composite.overlay`
|
||||
- [ ] Caption the new narration via `text.transcribe` + `composite.overlay`
|
||||
- [ ] Thumbnail with title + company logo via `package.thumbnail`
|
||||
- [ ] Encode 1080p via `package.encode`
|
||||
- [ ] Write `distribution_suggestion.md` — where this should go (internal Slack? shared drive?)
|
||||
|
||||
**Acceptance criteria**
|
||||
- [ ] Final is 2.5–3.5 min
|
||||
- [ ] Each on-screen decision is traceable to a timestamp in the source recording (record mapping in `trace.json`)
|
||||
- [ ] No PII from the meeting is surfaced that shouldn't be (agent flags risky moments)
|
||||
|
||||
---
|
||||
|
||||
## Task 12 — Gameplay "best moments" reel from raw footage
|
||||
|
||||
**Intent**: User provides 2 hours of raw gameplay recording. Produce a 5-minute "best moments" reel: auto-detect exciting moments (loud spikes, kill feed, chat excitement), add commentary via TTS using user-supplied notes, intro/outro, captions, thumbnail.
|
||||
|
||||
**Capabilities exercised**: `audio.capture` (analysis), `text.transcribe` (in-game voice/chat if present), `composite.assemble`, `audio.synthesize`, `composite.overlay`, `package.thumbnail`, `package.encode`. **Touches 7 capabilities, heavy on auto-detection.**
|
||||
|
||||
- [ ] Preflight; check disk space (2h raw ≈ 20–60GB)
|
||||
- [ ] Analyze source audio with `librosa` to find peaks/spikes; emit `candidates.json` with timestamps
|
||||
- [ ] Optionally OCR the kill feed via `tesseract` on sampled frames (not in matrix; document as cross-matrix use)
|
||||
- [ ] Merge peak candidates within 10s windows; cap at ~20 moments of 10–20s each
|
||||
- [ ] Generate commentary lines from user notes per moment via `audio.synthesize`
|
||||
- [ ] Assemble final via `composite.assemble`: intro card + moments + transitions + outro
|
||||
- [ ] Duck game audio under commentary
|
||||
- [ ] Caption commentary via `composite.overlay`
|
||||
- [ ] Thumbnail from best candidate frame via `package.thumbnail` (title + game logo)
|
||||
- [ ] Encode 1080p60 via `package.encode`
|
||||
|
||||
**Acceptance criteria**
|
||||
- [ ] Final duration 4.5–5.5 min
|
||||
- [ ] Each selected moment is genuinely "exciting" on manual review (false-positive rate < 25%)
|
||||
- [ ] Commentary is audible over game audio at every cut
|
||||
|
||||
---
|
||||
|
||||
## Task 13 — Ads variant generator (one brief → 5 creatives)
|
||||
|
||||
**Intent**: A brand brief and one stock clip should produce 5 ad creatives varying hook, CTA placement, and aspect ratio (16:9, 1:1, 9:16, 4:5, 9:16-short). Each includes AI-generated product shots, voiceover, music, captions.
|
||||
|
||||
**Capabilities exercised**: `visual.generate` (×many), `audio.synthesize`, `audio.music`, `composite.assemble` (×5), `composite.overlay` (×5), `package.thumbnail` (×5), `package.encode` (×5). **Touches 7 capabilities across 5 outputs — stress test for parallel execution.**
|
||||
|
||||
- [ ] Preflight
|
||||
- [ ] Parse brief.md → produce `creative_plan.json` with 5 variants (hook, CTA, aspect, length)
|
||||
- [ ] Generate shared music bed via `audio.music` (reuse across all 5)
|
||||
- [ ] Generate per-variant voiceovers via `audio.synthesize` (different voices per variant to test)
|
||||
- [ ] For each variant, generate 2 product shots via `visual.generate` (consistent seeds where useful)
|
||||
- [ ] For each variant, assemble via `composite.assemble` in the target aspect
|
||||
- [ ] For each variant, overlay headline/CTA + captions via `composite.overlay`
|
||||
- [ ] For each variant, thumbnail via `package.thumbnail`
|
||||
- [ ] Encode all 5 via `package.encode`
|
||||
- [ ] Write `ab_matrix.md` describing which combinations to test first and what metric
|
||||
|
||||
**Acceptance criteria**
|
||||
- [ ] 5 final videos in correct aspect ratios, lengths ≤ brief spec
|
||||
- [ ] Brand visual identity consistent (color palette, type, logo placement) across variants
|
||||
- [ ] Agent explicitly ranked providers by cost and gave the user an estimated total spend before generation
|
||||
|
||||
---
|
||||
|
||||
## Cross-task checks (run once per evaluation batch)
|
||||
|
||||
- [ ] No task left `publish.upload` as silently skipped; all 13 tasks either produced manual instructions or escalated with user-provided creds
|
||||
- [ ] Agent ran the preflight block at task start and didn't re-run unnecessarily across tasks within the same session
|
||||
- [ ] When paid APIs were used, the suggest-to-user template (capability / provider / env / cost / fallback) was used verbatim
|
||||
- [ ] Workspace hygiene: each task produced a single top-level directory with intermediates preserved, not scattered
|
||||
- [ ] Decision rubric order was followed: the agent didn't jump to paid APIs when a free provider would have been adequate (spot-audit 3 random tasks)
|
||||
|
||||
---
|
||||
|
||||
## Reporting template (per task)
|
||||
|
||||
```markdown
|
||||
### Task N — <title>
|
||||
|
||||
- **Status**: pass / partial / fail
|
||||
- **Providers chosen** (per capability):
|
||||
- visual.generate → <provider> (reason)
|
||||
- ...
|
||||
- **Escalations proposed**: <list>
|
||||
- **Escalations accepted by user**: <list>
|
||||
- **Unexpected blockers**: <text>
|
||||
- **Acceptance criteria met**: N of M
|
||||
- **Workspace**: <path>
|
||||
```
|
||||
+188
-95
@@ -1,24 +1,27 @@
|
||||
{
|
||||
"meta": {
|
||||
"repo": "https://github.com/HKUDS/CLI-Anything",
|
||||
"description": "Curated CLI Matrix registry for CLI-Hub multi-CLI workflows",
|
||||
"updated": "2026-04-17"
|
||||
"description": "Curated CLI Matrix registry for CLI-Hub multi-CLI workflows (capability-based, v2 schema)",
|
||||
"updated": "2026-04-18",
|
||||
"schema_doc": "docs/cli-matrix/matrix_registry.schema.md"
|
||||
},
|
||||
"matrices": [
|
||||
{
|
||||
"name": "video-creation",
|
||||
"display_name": "Video Creation & Editing",
|
||||
"version": "2",
|
||||
"description": "Curated end-to-end video production matrix spanning generation, capture, voice/music, editing, captioning, and thumbnailing for an agent-driven workflow.",
|
||||
"version": "3",
|
||||
"schema_version": "2",
|
||||
"description": "Capability-based matrix for end-to-end video production: generation, capture, voice/music, transcription, assembly, overlay, thumbnail, and encode. Providers include harness CLIs, public CLIs, Python libs, native binaries, and cloud APIs.",
|
||||
"category": "video",
|
||||
"matrix": "cli-matrix",
|
||||
"matrix_id": "V1",
|
||||
"homepage": "https://github.com/HKUDS/CLI-Anything/blob/main/docs/cli-matrix.md",
|
||||
"matrix_id": "S1",
|
||||
"homepage": "https://github.com/HKUDS/CLI-Anything/blob/main/docs/cli-matrix-plan.md",
|
||||
"skill_md": "cli-hub-matrix/video-creation/SKILL.md",
|
||||
"clis": [
|
||||
"generate-veo-video",
|
||||
"jimeng",
|
||||
"openscreen",
|
||||
"obs-studio",
|
||||
"audacity",
|
||||
"minimax-cli",
|
||||
"elevenlabs",
|
||||
@@ -26,112 +29,202 @@
|
||||
"kdenlive",
|
||||
"shotcut",
|
||||
"videocaptioner",
|
||||
"gimp"
|
||||
"gimp",
|
||||
"krita",
|
||||
"inkscape"
|
||||
],
|
||||
"stages": [
|
||||
"capabilities": [
|
||||
{
|
||||
"name": "AI video gen",
|
||||
"goal": "Generate video clips from text prompts or reference images",
|
||||
"clis": [
|
||||
"generate-veo-video",
|
||||
"jimeng"
|
||||
],
|
||||
"alternatives": {
|
||||
"python": ["replicate (Runway, Kling)", "diffusers"],
|
||||
"api": ["Runway API", "Kling API", "Pika API"]
|
||||
},
|
||||
"skill_search_hints": ["video generation", "text-to-video", "AI video"]
|
||||
"id": "visual.generate",
|
||||
"intent": "Produce a video clip from a text prompt or reference image.",
|
||||
"inputs": ["prompt:text", "ref_image?:path", "duration?:seconds"],
|
||||
"outputs": ["video_clip:path"],
|
||||
"skill_search_hints": ["text-to-video", "AI video", "video generation"],
|
||||
"providers": [
|
||||
{"kind": "public-cli", "name": "generate-veo-video", "requires": {"binary": ["generate-veo"], "env": ["GOOGLE_CLOUD_PROJECT"]}, "cost_tier": "metered", "quality_tier": "high", "offline": false},
|
||||
{"kind": "public-cli", "name": "jimeng", "requires": {"binary": ["dreamina"]}, "cost_tier": "metered", "quality_tier": "high", "offline": false, "notes": "ByteDance Dreamina; shares model family with Seedance."},
|
||||
{"kind": "api", "name": "Runway Gen-4", "requires": {"env": ["RUNWAY_API_KEY"]}, "cost_tier": "paid", "quality_tier": "sota", "offline": false, "notes": "~$0.05/sec, cinematic motion."},
|
||||
{"kind": "api", "name": "Kling", "requires": {"env": ["KLING_API_KEY"]}, "cost_tier": "paid", "quality_tier": "high", "offline": false},
|
||||
{"kind": "api", "name": "Pika", "requires": {"env": ["PIKA_API_KEY"]}, "cost_tier": "paid", "quality_tier": "good", "offline": false},
|
||||
{"kind": "api", "name": "Seedance", "requires": {"env": ["SEEDANCE_API_KEY"]}, "cost_tier": "paid", "quality_tier": "sota", "offline": false, "notes": "ByteDance Seedance; realistic motion."},
|
||||
{"kind": "python", "name": "replicate", "requires": {"package": ["replicate"], "env": ["REPLICATE_API_TOKEN"]}, "cost_tier": "metered", "quality_tier": "good", "offline": false},
|
||||
{"kind": "python", "name": "diffusers (SVD/AnimateDiff)", "requires": {"package": ["diffusers"]}, "cost_tier": "free", "quality_tier": "basic", "offline": true}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Screen/cam capture",
|
||||
"goal": "Record screen activity, webcam, or application windows to video files",
|
||||
"clis": [
|
||||
"openscreen"
|
||||
],
|
||||
"alternatives": {
|
||||
"python": ["pyautogui + cv2", "mss"],
|
||||
"native": ["ffmpeg -f x11grab (Linux)", "screencapture (macOS)"]
|
||||
},
|
||||
"skill_search_hints": ["screen recording", "capture", "screencast"]
|
||||
"id": "visual.capture",
|
||||
"intent": "Record screen, webcam, or application windows to a video file.",
|
||||
"inputs": ["source:screen|window|webcam", "duration?:seconds"],
|
||||
"outputs": ["video_clip:path"],
|
||||
"skill_search_hints": ["screen recording", "screencast", "webcam capture"],
|
||||
"providers": [
|
||||
{"kind": "harness-cli", "name": "cli-anything-openscreen", "requires": {"binary": ["cli-anything-openscreen"]}, "cost_tier": "free", "quality_tier": "high", "offline": true},
|
||||
{"kind": "harness-cli", "name": "cli-anything-obs-studio", "requires": {"binary": ["cli-anything-obs-studio"]}, "cost_tier": "free", "quality_tier": "high", "offline": true},
|
||||
{"kind": "native", "name": "ffmpeg x11grab/avfoundation", "requires": {"binary": ["ffmpeg"]}, "cost_tier": "free", "quality_tier": "high", "offline": true},
|
||||
{"kind": "native", "name": "screencapture (macOS)", "requires": {"binary": ["screencapture"]}, "cost_tier": "free", "quality_tier": "high", "offline": true},
|
||||
{"kind": "python", "name": "mss + cv2", "requires": {"package": ["mss", "opencv-python"]}, "cost_tier": "free", "quality_tier": "good", "offline": true}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Audio capture/edit",
|
||||
"goal": "Record, clean, trim, normalize, and process audio tracks",
|
||||
"clis": [
|
||||
"audacity"
|
||||
],
|
||||
"alternatives": {
|
||||
"python": ["pydub", "soundfile", "librosa", "noisereduce"],
|
||||
"native": ["sox", "ffmpeg"]
|
||||
},
|
||||
"skill_search_hints": ["audio editing", "noise reduction", "audio processing"]
|
||||
"id": "audio.capture",
|
||||
"intent": "Record, clean, trim, normalize, or denoise audio tracks.",
|
||||
"inputs": ["source:mic|file"],
|
||||
"outputs": ["audio_clip:path"],
|
||||
"skill_search_hints": ["audio editing", "noise reduction"],
|
||||
"providers": [
|
||||
{"kind": "harness-cli", "name": "cli-anything-audacity", "requires": {"binary": ["cli-anything-audacity"]}, "cost_tier": "free", "quality_tier": "high", "offline": true},
|
||||
{"kind": "native", "name": "sox/ffmpeg", "requires": {"binary": ["ffmpeg"]}, "cost_tier": "free", "quality_tier": "high", "offline": true},
|
||||
{"kind": "python", "name": "pydub/soundfile/librosa/noisereduce", "requires": {"package": ["pydub"]}, "cost_tier": "free", "quality_tier": "good", "offline": true}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Voice/TTS",
|
||||
"goal": "Generate speech audio from text for narration or voiceover",
|
||||
"clis": [
|
||||
"minimax-cli",
|
||||
"elevenlabs"
|
||||
],
|
||||
"alternatives": {
|
||||
"python": ["TTS (coqui-ai)", "pyttsx3", "edge-tts"],
|
||||
"api": ["Google Cloud TTS", "Amazon Polly", "Azure Speech"]
|
||||
},
|
||||
"skill_search_hints": ["text-to-speech", "TTS", "voice synthesis", "narration"]
|
||||
"id": "audio.synthesize",
|
||||
"intent": "Generate speech audio from text (narration / voiceover).",
|
||||
"inputs": ["text", "voice?:id", "lang?:code"],
|
||||
"outputs": ["audio_clip:path"],
|
||||
"skill_search_hints": ["text-to-speech", "TTS", "voice synthesis"],
|
||||
"providers": [
|
||||
{"kind": "public-cli", "name": "minimax-cli", "requires": {"binary": ["minimax-cli"], "env": ["MINIMAX_API_KEY"]}, "cost_tier": "metered", "quality_tier": "high", "offline": false},
|
||||
{"kind": "public-cli", "name": "elevenlabs", "requires": {"binary": ["elevenlabs"], "env": ["ELEVENLABS_API_KEY"]}, "cost_tier": "paid", "quality_tier": "sota", "offline": false},
|
||||
{"kind": "api", "name": "OpenAI TTS", "requires": {"env": ["OPENAI_API_KEY"]}, "cost_tier": "metered", "quality_tier": "high", "offline": false},
|
||||
{"kind": "api", "name": "Google Cloud TTS", "requires": {"env": ["GOOGLE_CLOUD_PROJECT"]}, "cost_tier": "metered", "quality_tier": "high", "offline": false},
|
||||
{"kind": "python", "name": "edge-tts", "requires": {"package": ["edge-tts"]}, "cost_tier": "free", "quality_tier": "good", "offline": false},
|
||||
{"kind": "python", "name": "TTS (coqui)", "requires": {"package": ["TTS"]}, "cost_tier": "free", "quality_tier": "good", "offline": true},
|
||||
{"kind": "python", "name": "pyttsx3", "requires": {"package": ["pyttsx3"]}, "cost_tier": "free", "quality_tier": "basic", "offline": true}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Music/BGM gen",
|
||||
"goal": "Generate background music or sound effects from prompts",
|
||||
"clis": [
|
||||
"minimax-cli",
|
||||
"suno"
|
||||
],
|
||||
"alternatives": {
|
||||
"python": ["audiocraft (Meta MusicGen)", "stable-audio-tools"],
|
||||
"api": ["Suno API", "Udio API"]
|
||||
},
|
||||
"skill_search_hints": ["music generation", "background music", "audio synthesis"]
|
||||
"id": "audio.music",
|
||||
"intent": "Generate background music or sound effects.",
|
||||
"inputs": ["prompt:text", "duration?:seconds"],
|
||||
"outputs": ["audio_clip:path"],
|
||||
"skill_search_hints": ["music generation", "background music"],
|
||||
"providers": [
|
||||
{"kind": "public-cli", "name": "suno", "requires": {"binary": ["suno"]}, "cost_tier": "metered", "quality_tier": "sota", "offline": false},
|
||||
{"kind": "public-cli", "name": "minimax-cli", "requires": {"binary": ["minimax-cli"], "env": ["MINIMAX_API_KEY"]}, "cost_tier": "metered", "quality_tier": "high", "offline": false},
|
||||
{"kind": "api", "name": "Udio", "requires": {"env": ["UDIO_API_KEY"]}, "cost_tier": "paid", "quality_tier": "sota", "offline": false},
|
||||
{"kind": "python", "name": "audiocraft (MusicGen)", "requires": {"package": ["audiocraft"]}, "cost_tier": "free", "quality_tier": "good", "offline": true},
|
||||
{"kind": "python", "name": "stable-audio-tools", "requires": {"package": ["stable-audio-tools"]}, "cost_tier": "free", "quality_tier": "good", "offline": true}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "NLE editing",
|
||||
"goal": "Assemble video/audio clips on a timeline, add transitions, and export the final cut",
|
||||
"clis": [
|
||||
"kdenlive",
|
||||
"shotcut"
|
||||
],
|
||||
"alternatives": {
|
||||
"python": ["moviepy", "ffmpeg-python"],
|
||||
"native": ["ffmpeg concat/filter_complex"]
|
||||
},
|
||||
"skill_search_hints": ["video editing", "timeline", "NLE", "video assembly"]
|
||||
"id": "text.transcribe",
|
||||
"intent": "Transcribe speech to text and produce subtitle files (SRT/ASS).",
|
||||
"inputs": ["audio_or_video:path", "lang?:code"],
|
||||
"outputs": ["subtitles:path", "transcript:text"],
|
||||
"skill_search_hints": ["subtitle", "transcribe", "speech-to-text"],
|
||||
"providers": [
|
||||
{"kind": "harness-cli", "name": "cli-anything-videocaptioner", "requires": {"binary": ["cli-anything-videocaptioner"]}, "cost_tier": "free", "quality_tier": "high", "offline": true, "notes": "Can use cloud ASR backends if configured."},
|
||||
{"kind": "python", "name": "openai-whisper", "requires": {"package": ["whisper"]}, "cost_tier": "free", "quality_tier": "high", "offline": true},
|
||||
{"kind": "python", "name": "faster-whisper/stable-ts", "requires": {"package": ["faster_whisper"]}, "cost_tier": "free", "quality_tier": "high", "offline": true},
|
||||
{"kind": "api", "name": "AssemblyAI", "requires": {"env": ["ASSEMBLYAI_API_KEY"]}, "cost_tier": "paid", "quality_tier": "sota", "offline": false},
|
||||
{"kind": "api", "name": "Deepgram", "requires": {"env": ["DEEPGRAM_API_KEY"]}, "cost_tier": "paid", "quality_tier": "sota", "offline": false},
|
||||
{"kind": "api", "name": "Google Speech-to-Text", "requires": {"env": ["GOOGLE_CLOUD_PROJECT"]}, "cost_tier": "metered", "quality_tier": "high", "offline": false}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Captions/subs",
|
||||
"goal": "Transcribe speech, generate subtitles (SRT/ASS), optionally translate, and burn into video",
|
||||
"clis": [
|
||||
"videocaptioner"
|
||||
],
|
||||
"alternatives": {
|
||||
"python": ["whisper (openai-whisper)", "pysrt", "stable-ts"],
|
||||
"api": ["Google Speech-to-Text", "AssemblyAI", "Deepgram"],
|
||||
"native": ["ffmpeg -vf subtitles=subs.srt"]
|
||||
},
|
||||
"skill_search_hints": ["subtitle", "transcribe", "caption", "speech-to-text"]
|
||||
"id": "text.translate",
|
||||
"intent": "Translate subtitles or script text across languages.",
|
||||
"inputs": ["text_or_srt:path", "target_lang:code"],
|
||||
"outputs": ["translated:path"],
|
||||
"skill_search_hints": ["translation", "subtitle translation"],
|
||||
"providers": [
|
||||
{"kind": "api", "name": "OpenAI/Claude", "requires": {"env": ["OPENAI_API_KEY"]}, "cost_tier": "metered", "quality_tier": "sota", "offline": false},
|
||||
{"kind": "api", "name": "DeepL", "requires": {"env": ["DEEPL_API_KEY"]}, "cost_tier": "metered", "quality_tier": "sota", "offline": false},
|
||||
{"kind": "python", "name": "argos-translate", "requires": {"package": ["argostranslate"]}, "cost_tier": "free", "quality_tier": "good", "offline": true}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Thumbnail",
|
||||
"goal": "Create a thumbnail image or social card for the video",
|
||||
"clis": [
|
||||
"gimp"
|
||||
],
|
||||
"alternatives": {
|
||||
"python": ["Pillow", "cairosvg", "html2image"],
|
||||
"api": ["Google Nano Banana", "OpenAI GPT-Image-1", "Ideogram API", "Stability AI image generation"],
|
||||
"native": ["ffmpeg -ss 00:01:00 -frames:v 1 thumb.png", "ImageMagick convert"]
|
||||
},
|
||||
"skill_search_hints": ["image editing", "thumbnail", "graphic design"]
|
||||
"id": "composite.assemble",
|
||||
"intent": "Assemble clips on a timeline with transitions and export a single cut.",
|
||||
"inputs": ["clips:path[]", "edit_plan?:json"],
|
||||
"outputs": ["video:path"],
|
||||
"skill_search_hints": ["video editing", "timeline", "NLE"],
|
||||
"providers": [
|
||||
{"kind": "harness-cli", "name": "cli-anything-kdenlive", "requires": {"binary": ["cli-anything-kdenlive"]}, "cost_tier": "free", "quality_tier": "high", "offline": true},
|
||||
{"kind": "harness-cli", "name": "cli-anything-shotcut", "requires": {"binary": ["cli-anything-shotcut"]}, "cost_tier": "free", "quality_tier": "high", "offline": true},
|
||||
{"kind": "python", "name": "moviepy", "requires": {"package": ["moviepy"], "binary": ["ffmpeg"]}, "cost_tier": "free", "quality_tier": "good", "offline": true},
|
||||
{"kind": "python", "name": "ffmpeg-python", "requires": {"package": ["ffmpeg-python"], "binary": ["ffmpeg"]}, "cost_tier": "free", "quality_tier": "high", "offline": true},
|
||||
{"kind": "native", "name": "ffmpeg concat/filter_complex", "requires": {"binary": ["ffmpeg"]}, "cost_tier": "free", "quality_tier": "high", "offline": true}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "composite.overlay",
|
||||
"intent": "Burn subtitles, watermark, or picture-in-picture onto a video.",
|
||||
"inputs": ["video:path", "overlay:path_or_srt"],
|
||||
"outputs": ["video:path"],
|
||||
"skill_search_hints": ["burn subtitles", "watermark", "overlay"],
|
||||
"providers": [
|
||||
{"kind": "native", "name": "ffmpeg -vf subtitles/overlay", "requires": {"binary": ["ffmpeg"]}, "cost_tier": "free", "quality_tier": "high", "offline": true},
|
||||
{"kind": "python", "name": "moviepy (CompositeVideoClip)", "requires": {"package": ["moviepy"]}, "cost_tier": "free", "quality_tier": "good", "offline": true}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "package.thumbnail",
|
||||
"intent": "Create a thumbnail image or social card.",
|
||||
"inputs": ["concept:text", "reference?:path"],
|
||||
"outputs": ["image:path"],
|
||||
"skill_search_hints": ["thumbnail", "social card", "image design"],
|
||||
"providers": [
|
||||
{"kind": "harness-cli", "name": "cli-anything-gimp", "requires": {"binary": ["cli-anything-gimp"]}, "cost_tier": "free", "quality_tier": "high", "offline": true},
|
||||
{"kind": "harness-cli", "name": "cli-anything-krita", "requires": {"binary": ["cli-anything-krita"]}, "cost_tier": "free", "quality_tier": "high", "offline": true},
|
||||
{"kind": "harness-cli", "name": "cli-anything-inkscape", "requires": {"binary": ["cli-anything-inkscape"]}, "cost_tier": "free", "quality_tier": "high", "offline": true},
|
||||
{"kind": "python", "name": "Pillow", "requires": {"package": ["PIL"]}, "cost_tier": "free", "quality_tier": "good", "offline": true},
|
||||
{"kind": "python", "name": "cairosvg/html2image", "requires": {"package": ["cairosvg"]}, "cost_tier": "free", "quality_tier": "good", "offline": true},
|
||||
{"kind": "api", "name": "OpenAI GPT-Image-1", "requires": {"env": ["OPENAI_API_KEY"]}, "cost_tier": "metered", "quality_tier": "sota", "offline": false},
|
||||
{"kind": "api", "name": "Google Nano Banana", "requires": {"env": ["GOOGLE_CLOUD_PROJECT"]}, "cost_tier": "metered", "quality_tier": "high", "offline": false},
|
||||
{"kind": "api", "name": "Ideogram", "requires": {"env": ["IDEOGRAM_API_KEY"]}, "cost_tier": "metered", "quality_tier": "high", "offline": false},
|
||||
{"kind": "api", "name": "Stability AI", "requires": {"env": ["STABILITY_API_KEY"]}, "cost_tier": "metered", "quality_tier": "high", "offline": false},
|
||||
{"kind": "native", "name": "ffmpeg frame-grab", "requires": {"binary": ["ffmpeg"]}, "cost_tier": "free", "quality_tier": "basic", "offline": true},
|
||||
{"kind": "native", "name": "ImageMagick convert/magick", "requires": {"binary": ["magick"]}, "cost_tier": "free", "quality_tier": "good", "offline": true}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "package.encode",
|
||||
"intent": "Final mux / codec / container for the output video.",
|
||||
"inputs": ["video:path", "target?:codec_or_container"],
|
||||
"outputs": ["video:path"],
|
||||
"skill_search_hints": ["encode", "transcode", "codec"],
|
||||
"providers": [
|
||||
{"kind": "native", "name": "ffmpeg", "requires": {"binary": ["ffmpeg"]}, "cost_tier": "free", "quality_tier": "sota", "offline": true}
|
||||
]
|
||||
}
|
||||
]
|
||||
],
|
||||
"recipes": [
|
||||
{
|
||||
"id": "ai-short",
|
||||
"description": "Fully-generative social short.",
|
||||
"capabilities_used": ["visual.generate", "audio.synthesize", "audio.music", "composite.assemble", "composite.overlay", "package.thumbnail", "package.encode"]
|
||||
},
|
||||
{
|
||||
"id": "screencast-tutorial",
|
||||
"description": "Record a screen walkthrough with narration and subtitles.",
|
||||
"capabilities_used": ["visual.capture", "audio.capture", "text.transcribe", "composite.overlay", "package.thumbnail", "package.encode"]
|
||||
},
|
||||
{
|
||||
"id": "talking-head-explainer",
|
||||
"description": "Webcam + b-roll + captions.",
|
||||
"capabilities_used": ["visual.capture", "visual.generate", "audio.capture", "text.transcribe", "composite.assemble", "composite.overlay", "package.thumbnail", "package.encode"]
|
||||
},
|
||||
{
|
||||
"id": "podcast-to-video",
|
||||
"description": "Audio-first, visualize and caption for video platforms.",
|
||||
"capabilities_used": ["audio.capture", "text.transcribe", "package.thumbnail", "composite.overlay", "composite.assemble", "package.encode"]
|
||||
}
|
||||
],
|
||||
"known_gaps": [
|
||||
{
|
||||
"capability": "publish.upload",
|
||||
"reason": "No first-party or public CLI for YouTube/TikTok/Bilibili/Instagram yet.",
|
||||
"workaround": "Instruct the user to upload manually via the web UI, or escalate to a custom script using each platform's v3 API with an OAuth token the user supplies."
|
||||
},
|
||||
{
|
||||
"capability": "visual.generate",
|
||||
"reason": "Sota-tier cinematic output is only available via paid APIs (Runway, Kling, Seedance).",
|
||||
"workaround": "Use generate-veo-video or jimeng for high-quality output; fall back to local diffusers (basic) for offline contexts."
|
||||
}
|
||||
],
|
||||
"suggest_to_user_template": "To enable <capability> via <provider>, please set <ENV_VAR>. Cost: <cost_tier>. Quality: <quality_tier>. Reply 'skip' to fall back to <next provider>."
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user