CLI-Anything/docs/PREVIEW_PROTOCOL.md

Preview Protocol

Last updated: 2026-04-23 UTC

This document proposes a minimal, cross-harness protocol for previewing intermediate results in CLI-Anything workflows.

The short version:

• Every participating harness emits a standard Preview Bundle.
• A bundle is just a directory with a manifest.json, a summary.json, and preview artifacts such as images, clips, JSON dumps, or model files.
• Harnesses still use the real software for rendering. The protocol standardizes the preview artifact contract, not the renderer.
• cli-hub becomes the generic viewer/inspector for bundles, instead of every harness inventing its own monitor UI.

This gives agents and humans one stable way to consume intermediate outputs across video, CAD, 3D, GPU debugging, and similar workflows.

Why This Exists

CLI-Anything already has strong rules for final rendering:

• use the real software
• manipulate native formats
• verify real output files

What it does not yet have is a cross-harness protocol for intermediate visual feedback.

Today, preview-related capability exists in many local forms:

• video thumbnail extraction
• screenshot export
• render presets
• capture thumbnails
• output-target dumps

But these are app-local commands. They do not produce a uniform artifact shape that an agent runtime, cli-hub, or a future host UI can consume in the same way across tools.

The missing abstraction is not "a universal monitor widget". The missing abstraction is "a universal preview artifact protocol".

Design Goals

1. Keep the rendering path honest. Previews must still come from the real software, its real backend, or its real project format. No toy reimplementation of the app's renderer.
2. Be generic across domains. The same protocol should handle video timelines, CAD views, Blender renders, RenderDoc outputs, and similar cases.
3. Be cheap enough to use often. Previews should be fast, low-resolution, cacheable, and small enough for agent loops.
4. Be simple enough to adopt incrementally. A harness should be able to add preview support with one command group and a small helper, not a new frontend app.
5. Work in headless environments. Bundle generation and validation must not require a GUI.

Non-Goals

• This is not a replacement for final render/export.
• This is not a live remote framebuffer or GUI streaming protocol.
• This does not require every harness to support interactive timeline scrubbing.
• This does not force one preview artifact type. Different apps may emit images, clips, JSON, or multiple artifacts together.

Core Model

The protocol has three layers:

1. Harness layer The harness generates real preview artifacts and writes a Preview Bundle.
2. Viewer layer cli-hub previews ... reads any compliant bundle or live session and renders a generic inspection view.
3. Host/runtime layer An agent host may decide to attach selected bundle artifacts back into model context, but this is outside the harness itself.

The central object is the Preview Bundle.

For live preview workflows, the stable object is the Live Session, with an append-only trajectory.json beside it.

Preview Bundle

A preview bundle is a directory with a small, stable layout:

<bundle_dir>/
  manifest.json
  summary.json
  artifacts/
    hero.png
    gallery_01.png
    gallery_02.png
    preview.mp4
    pipeline_diff.json

manifest.json is the machine contract.

summary.json is the human/agent summary.

artifacts/ contains preview outputs generated by the real software or by native tools that inspect those real outputs.

Default Bundle Location

When a project path is known:

<project_dir>/.cli-anything/previews/<software>/<recipe>/<bundle_id>/

When there is no stable project path yet:

~/.cli-anything/previews/<software>/<recipe>/<bundle_id>/

Rationale:

• preview artifacts stay near the project when possible
• bundles are easy to garbage-collect
• cli-hub can scan a predictable root

Bundle ID

Recommended bundle id:

<UTC timestamp>_<short fingerprint>_<recipe>

Example:

20260419T104530Z_9f0a2c4b_quick

The fingerprint should be derived from:

• project fingerprint or capture fingerprint
• recipe name
• normalized preview args
• harness version
• protocol version

This makes bundles cacheable and reproducible.

Manifest Schema

Required top-level fields:

Field	Type	Notes
protocol_version	string	Start with preview-bundle/v1
bundle_id	string	Stable id for this bundle
bundle_kind	string	capture or diff
software	string	Harness name, e.g. shotcut
recipe	string	Harness recipe name, e.g. quick, quad, turntable
status	string	ok, partial, or error
created_at	string	ISO-8601 UTC
generator	object	CLI command, harness version, backend info
source	object	Project/capture identity and fingerprint
artifacts	array	Preview artifact descriptors
summary_path	string	Relative path to summary.json

Recommended optional fields:

Field	Type	Notes
source_bundles	array	For diff bundles
context	object	App-specific context, e.g. event id, frame, camera
warnings	array	Non-fatal warnings
metrics	object	Cheap summary metrics
labels	array	Tags for host-side filtering

Manifest Example

{
  "protocol_version": "preview-bundle/v1",
  "bundle_id": "20260419T104530Z_9f0a2c4b_quick",
  "bundle_kind": "capture",
  "software": "shotcut",
  "recipe": "quick",
  "status": "ok",
  "created_at": "2026-04-19T10:45:30Z",
  "generator": {
    "entry_point": "cli-anything-shotcut",
    "harness_version": "1.0.0",
    "command": "cli-anything-shotcut --json -p demo.mlt preview capture --recipe quick"
  },
  "source": {
    "project_path": "/work/demo.mlt",
    "project_fingerprint": "sha256:3a0d6f...",
    "session_id": "abc123"
  },
  "context": {
    "timeline_duration_s": 42.1
  },
  "metrics": {
    "video_tracks": 2,
    "audio_tracks": 1,
    "filters": 7
  },
  "summary_path": "summary.json",
  "artifacts": [
    {
      "artifact_id": "hero",
      "role": "hero",
      "kind": "image",
      "label": "Midpoint frame",
      "media_type": "image/png",
      "path": "artifacts/hero.png",
      "width": 960,
      "height": 540,
      "bytes": 184220
    },
    {
      "artifact_id": "clip",
      "role": "preview-clip",
      "kind": "clip",
      "label": "Low-res preview render",
      "media_type": "video/mp4",
      "path": "artifacts/preview.mp4",
      "width": 640,
      "height": 360,
      "duration_s": 6.0,
      "bytes": 2419001
    }
  ]
}

Summary Schema

summary.json should stay compact and cheap to parse.

Recommended shape:

{
  "headline": "6-second quick preview rendered successfully",
  "facts": {
    "duration_s": 42.1,
    "resolution": "640x360",
    "recipe": "quick"
  },
  "warnings": [],
  "next_actions": [
    "Inspect the preview clip for pacing and transitions",
    "Use final export only after timeline review passes"
  ]
}

The viewer should render headline, facts, and warnings first.

Live Session

Live preview adds two more protocol objects:

1. session.json Mutable session head. Tracks the current bundle, viewer commands, and live poller state.
2. trajectory.json Append-only history of command-to-preview publishes for the session. This is the permanent replay object.

Recommended live session layout:

<session_dir>/
  session.json
  trajectory.json
  current -> ../quick/<bundle_id>/

Why trajectory.json exists

bundle_dir is a single immutable snapshot, not the stable history object. For the same working project:

• a bundle directory may be reused if the cache key is unchanged
• a new bundle directory is created whenever the source fingerprint, recipe, options, harness version, or protocol version changes

So agents and host UIs should not treat bundle_dir as the permanent handle for the whole build. The stable handle for live preview is:

• session_dir for the current channel
• trajectory.json for the full publish history

session.json responsibilities

Recommended fields:

Field	Type	Notes
protocol_version	string	Start with preview-live/v1
software	string	Harness name
recipe	string	Active live recipe
status	string	active or stopped
current_bundle_id	string	Current head bundle
current_bundle_dir	string	Absolute current bundle path
current_manifest_path	string	Absolute manifest path
current_summary_path	string	Absolute summary path
current_step_id	string	Current trajectory step id
latest_command	string	Latest publish command for agent/UI inspection
latest_publish_reason	string	Latest publish reason
trajectory_path	string	Relative path, usually trajectory.json
trajectory_step_count	integer	Total trajectory steps
history	array	Short recent-bundles list for compatibility

history remains useful for quick inspection, but it is not the authoritative replay log.

trajectory.json schema

Recommended top-level shape:

{
  "protocol_version": "preview-trajectory/v1",
  "software": "blender",
  "recipe": "quick",
  "session_name": "orbital-relay-drone-blend-cli-48261b51-quick",
  "project_path": "/work/scene.blend-cli.json",
  "project_name": "scene.blend-cli.json",
  "created_at": "2026-04-23T10:10:00Z",
  "updated_at": "2026-04-23T10:12:41Z",
  "step_count": 3,
  "current_step_id": "step-0003",
  "steps": [
    {
      "step_id": "step-0001",
      "step_index": 1,
      "command": "cli-anything-blender --project scene.blend-cli.json preview live start --recipe quick",
      "command_started_at": "2026-04-23T10:10:00Z",
      "command_finished_at": "2026-04-23T10:10:07Z",
      "publish_reason": "live-start",
      "source_fingerprint": "sha256:...",
      "bundle_id": "20260423T101007Z_deadbeef_quick",
      "bundle_dir": "/work/.cli-anything/previews/blender/quick/20260423T101007Z_deadbeef_quick",
      "manifest_path": "/work/.cli-anything/previews/blender/quick/20260423T101007Z_deadbeef_quick/manifest.json",
      "summary_path": "/work/.cli-anything/previews/blender/quick/20260423T101007Z_deadbeef_quick/summary.json",
      "status": "ok",
      "cached": false,
      "stage_label": "service-rig"
    }
  ]
}

Minimum recommended step fields:

• step_id
• command
• command_started_at
• command_finished_at
• publish_reason
• source_fingerprint
• bundle_id
• bundle_dir
• manifest_path
• summary_path

Optional but useful:

• stage_label
• note
• status
• cached

This gives viewers enough structure to render:

• command stream on the left
• matching preview state on the right
• arbitrary rewind to previous build checkpoints

Artifact Descriptor

Each artifact entry in manifest.json should use relative paths and carry enough metadata for a generic viewer to render a sensible page without probing files.

Required artifact fields:

Field	Type	Notes
artifact_id	string	Unique inside bundle
role	string	Semantic role
kind	string	image, clip, json, model, capture, text
media_type	string	MIME-like type
path	string	Relative path from bundle root
label	string	Short display label

Recommended fields:

Field	Type	Notes
width / height	int	For images and video
duration_s	number	For clips
bytes	int	For size display
view	string	iso, front, top, right, etc.
event_id	int	For RenderDoc-like outputs
tags	array	Search/filter tags

Standard Artifact Roles

The role vocabulary matters more than the exact file type.

Recommended roles:

Role	Meaning
hero	Best single preview artifact for quick inspection
gallery	One of several related visual previews
preview-clip	Short low-res video preview
before	Baseline artifact in a diff bundle
after	Target artifact in a diff bundle
diff-json	Machine diff payload
pipeline-json	Structured pipeline/debug state
viewport	A specific camera/view output
render-output	Output target or rendered frame
thumb	Capture or media thumbnail

The viewer should not assume a bundle has only one artifact.

Performance Budget

Previews should be cheap enough to generate inside an agent loop.

Recommended defaults:

• hero image: longest side <= 1280 px
• gallery images: 3 to 8 images
• preview clip: <= 8 seconds, <= 720p
• bundle size target: <= 25 MB

These are targets, not hard protocol limits. Larger bundles are allowed, but the default recipes should stay cheap.

Rendering Rule

The same rendering rule as the rest of CLI-Anything still applies:

• use the real software or real backend for rendering
• use native project formats
• do not quietly replace the app's renderer with a toy Python version

What is allowed:

• use the real app or backend to create a low-res preview
• use ffmpeg, RenderDoc APIs, or similar native tools to extract frames or dump outputs from a real render
• package those results into a bundle

What is not allowed:

• inventing a fake renderer to approximate the app's preview behavior
• screenshotting an unrelated GUI window as the only source of truth

Standard CLI Surface

Participating harnesses should expose a top-level preview group.

Recommended commands:

preview recipes

Lists supported recipes and what each emits.

Example:

cli-anything-shotcut preview recipes

preview capture

Generates a fresh or cached preview bundle.

Recommended options:

• --recipe <name>
• --output-dir <path>
• --force
• --label <text>

JSON mode should return at least:

• bundle_dir
• manifest_path
• status
• artifact_count

preview latest

Returns the latest successful bundle for the current project and optional recipe.

preview diff

Recommended for domains with meaningful before/after or event/event comparison. Not every harness must implement this in v1, but the protocol supports it.

preview live start

Starts a live preview session and publishes the initial bundle.

Recommended options:

• --recipe <name>
• --root-dir <path>
• --poll-ms <int>
• --mode manual|poll
• --source-poll-ms <int>
• --open

preview live push

Publishes a fresh bundle into an existing live session.

Use this when:

• the harness supports manual live refresh
• the agent wants an explicit command-to-preview checkpoint
• the source did not change enough to justify poll mode

preview live status

Returns the current live session state without rendering a new bundle.

This command exists for agents. It should answer:

• is there a live session for this project and recipe
• where is session.json
• what is the current bundle
• what is the latest publish reason
• where is trajectory.json

For --json, implementations SHOULD include:

• _session_dir
• _session_path
• _trajectory_path
• current_bundle_id
• current_bundle_dir
• current_step_id
• latest_publish_reason
• trajectory_summary

trajectory_summary should be a compact in-band summary so agents do not need a second file read just to understand the latest few steps.

preview live stop

Stops a live preview session without deleting published artifacts.

Caching Rules

preview capture may return a cached bundle if:

• the project/capture fingerprint is unchanged
• recipe and normalized preview args are unchanged
• protocol version is unchanged
• the bundle is still present on disk

--force must bypass the cache and generate a new bundle.

preview latest returns the newest successful bundle by timestamp.

Portability Rules

• artifact paths in manifest.json must be relative to bundle root
• bundle files must be immutable once written
• summary.json and manifest.json must be UTF-8 JSON
• absolute project paths are allowed in source, but viewer logic must not rely on them to render the bundle

Host/Agent Contract

The protocol solves the artifact contract. A host runtime may do more.

Recommended host behavior:

1. Read manifest.json first.
2. Render summary.json.
3. Prefer displaying, or attaching back into model context, artifacts in this order:
   • hero
   • gallery
   • preview-clip
   • before / after
   • diff-json / pipeline-json
4. If binary attachment is unavailable, at least expose file paths and summary.

This is the root simplification: the agent does not need app-specific preview logic. It only needs bundle-aware preview logic.

Generic Viewer

The generic viewer lives in cli-hub.

The split is intentional:

• cli-anything-<software> preview ... publishes preview state
• cli-hub previews ... inspects existing preview state

Canonical viewer commands:

• cli-hub previews inspect <bundle-or-session>
• cli-hub previews html <bundle-or-session> [-o output.html]
• cli-hub previews watch <session-dir> [--open] [--poll-ms 1500]
• cli-hub previews open <bundle-or-session>

inspect prints:

• bundle metadata
• source identity
• summary headline/facts/warnings
• artifact table
• trajectory summary when available

html writes a static HTML page that:

• shows the summary first
• renders image galleries from hero and gallery artifacts
• embeds playable <video> elements for preview-clip
• links to JSON artifacts for detailed inspection
• renders session history from trajectory.json when present

watch serves a live session over localhost and auto-refreshes the page.

open opens either:

• a generated HTML file for a bundle
• a browser-backed live session watcher for session_dir

This viewer is intentionally generic. It does not know what Shotcut, FreeCAD, Blender, or RenderDoc "mean". It only knows bundle roles and artifact kinds.

Testing Requirements

Any harness that adopts the protocol should add:

Unit Tests

• bundle writer path normalization
• manifest generation
• recipe listing
• cache key stability

E2E Tests

• generate a real preview bundle with the real backend
• verify manifest.json and summary.json exist
• verify every artifact path listed in the manifest exists
• verify at least one visual artifact exists
• verify JSON output includes manifest_path

For diff-capable harnesses:

• verify diff bundle references source bundles or source refs
• verify the diff JSON artifact exists

Implementation Pattern For Existing Harnesses

This repo already has one proven "copy canonical helper into harness" pattern: repl_skin.py.

The preview protocol should follow the same approach in v1:

• canonical helper lives in cli-anything-plugin/
• pilot harnesses vendor-copy a small helper as utils/preview_bundle.py
• the helper handles:
  • bundle directory creation
  • manifest writing
  • summary writing
  • relative artifact descriptors
  • cache key generation

This avoids introducing a new shared runtime dependency across all harness packages.

Initial 4-PR Plan

The first implementation should be split into four practical PRs.

PR1: Foundation - Protocol, Helper, and Generic Viewer

Goal:

• establish the protocol and a reusable bundle writer
• make cli-hub able to inspect any compliant bundle

Scope:

• add this document
• add canonical helper under cli-anything-plugin/
• update cli-anything-plugin/HARNESS.md with preview requirements
• add cli-hub preview inspection commands

Concrete file plan:

• docs/PREVIEW_PROTOCOL.md
• cli-anything-plugin/preview_bundle.py
• cli-anything-plugin/HARNESS.md
• cli-hub/cli_hub/preview.py
• cli-hub/cli_hub/cli.py
• cli-hub/tests/test_cli_hub.py

Concrete behavior:

• cli-hub previews inspect <bundle> prints manifest metadata, summary, and artifact inventory
• cli-hub previews html <bundle> -o page.html writes a static HTML page with summary + gallery + video embedding

Why this works:

• it does not depend on any specific harness
• it gives immediate value as soon as the first pilot bundle exists

Acceptance criteria:

• a sample bundle can be inspected by cli-hub
• the HTML viewer renders images and videos using only manifest metadata
• HARNESS.md tells future harnesses how to adopt preview support

Out of scope:

• no live preview streaming
• no harness-specific rendering logic here

PR2: Video Pilot - Shotcut and Openscreen

Goal:

• prove that video editors can produce standard preview bundles cheaply

Scope:

• add preview support to shotcut
• add preview support to openscreen

Concrete file plan:

• shotcut/agent-harness/cli_anything/shotcut/utils/preview_bundle.py
• shotcut/agent-harness/cli_anything/shotcut/core/preview.py
• shotcut/agent-harness/cli_anything/shotcut/shotcut_cli.py
• shotcut/agent-harness/cli_anything/shotcut/tests/test_core.py
• shotcut/agent-harness/cli_anything/shotcut/tests/test_full_e2e.py
• openscreen/agent-harness/cli_anything/openscreen/utils/preview_bundle.py
• openscreen/agent-harness/cli_anything/openscreen/core/preview.py
• openscreen/agent-harness/cli_anything/openscreen/openscreen_cli.py
• openscreen/agent-harness/cli_anything/openscreen/tests/test_core.py
• openscreen/agent-harness/cli_anything/openscreen/tests/test_full_e2e.py

Shotcut workable MVP design:

• command: cli-anything-shotcut preview capture --recipe quick
• recipe quick does:
  • low-res render using existing core/export.py
  • default preset: h264-fast
  • default size: 640x360
  • extract 5 PNG frames from the rendered clip at fixed percentages
  • choose the midpoint frame as hero
  • write summary with timeline duration, track counts, and filter counts
• command: preview recipes lists quick
• command: preview latest returns the latest bundle path

Existing code it reuses:

• render path in core/export.py
• thumbnail extraction in core/media.py

Openscreen workable MVP design:

• command: cli-anything-openscreen preview capture --recipe quick
• recipe quick does:
  • low-res render via existing core/export.render()
  • extract 5 PNG frames at 0%, 25%, 50%, 75%, and 95%
  • midpoint frame becomes hero
  • summary includes source duration, zoom region count, speed region count, and trim region count

Existing code it reuses:

• export path in core/export.py
• frame extraction in core/media.py

Why this works:

• both harnesses already have real render pipelines
• both already know how to extract thumbnails
• the only new layer is bundle assembly and command wiring

Acceptance criteria:

• both harnesses emit valid bundles for a real project
• cli-hub previews inspect can read them
• the HTML viewer shows the gallery and preview clip without app-specific code

Out of scope:

• no per-cut diffing in this PR
• no contact-sheet composition requirement

PR3: 3D/CAD Pilot - Blender and FreeCAD

Goal:

• prove the protocol works for still renders, multiple standard views, and 3D scene review

Scope:

• add preview support to blender
• add preview support to freecad

Concrete file plan:

• blender/agent-harness/cli_anything/blender/utils/preview_bundle.py
• blender/agent-harness/cli_anything/blender/core/preview.py
• blender/agent-harness/cli_anything/blender/blender_cli.py
• blender/agent-harness/cli_anything/blender/tests/test_core.py
• blender/agent-harness/cli_anything/blender/tests/test_full_e2e.py
• freecad/agent-harness/cli_anything/freecad/utils/preview_bundle.py
• freecad/agent-harness/cli_anything/freecad/core/preview.py
• freecad/agent-harness/cli_anything/freecad/utils/freecad_backend.py
• freecad/agent-harness/cli_anything/freecad/freecad_cli.py
• freecad/agent-harness/cli_anything/freecad/tests/test_core.py
• freecad/agent-harness/cli_anything/freecad/tests/test_full_e2e.py

Blender workable MVP design:

• command: cli-anything-blender preview capture --recipe quick
• recipe quick does:
  • apply eevee_preview or workbench style render settings
  • render one still image at low resolution
  • use the current frame or an explicit --frame
  • write summary with engine, resolution, active camera, object count

Preferred implementation detail:

• generate bpy script using existing render code
• execute it with the real Blender backend, not just script generation

Existing code it reuses:

• render presets documented in BLENDER.md
• render configuration in core/scene.py
• script generation in core/render.py
• real headless execution in utils/blender_backend.py

FreeCAD workable MVP design:

• command: cli-anything-freecad preview capture --recipe quick
• command: cli-anything-freecad preview capture --recipe quad
• recipe quick does:
  • export one low-res isometric PNG
• recipe quad does:
  • export four PNGs: iso, front, top, right
  • mark iso as hero
  • mark the four views as gallery
  • write summary with document name, object count, and bounding-box metrics if available

Implementation detail:

• extend the FreeCAD macro/backend path for png and jpg export with an explicit standard-view option
• do not add a GUI dependency; the macro should set the desired view itself

Existing code it reuses:

• image export support in core/export.py
• export CLI in freecad_cli.py

Why this works:

• Blender already has low-cost preview presets
• FreeCAD already exports rendered PNG/JPG
• the viewer can show multi-view galleries without requiring an app-specific viewport widget

Acceptance criteria:

• Blender quick preview emits a real PNG bundle
• FreeCAD quick and quad previews emit real PNG bundles
• the generic viewer shows Blender as a hero image and FreeCAD as a gallery

Out of scope:

• no fully interactive camera orbit viewer
• no mesh viewer in the browser for v1

PR4: GPU Debug Pilot - RenderDoc

Goal:

• prove the protocol also works for non-DCC visual inspection workflows

Scope:

• add preview capture and preview diff to renderdoc

Concrete file plan:

• renderdoc/agent-harness/cli_anything/renderdoc/utils/preview_bundle.py
• renderdoc/agent-harness/cli_anything/renderdoc/core/preview.py
• renderdoc/agent-harness/cli_anything/renderdoc/renderdoc_cli.py
• renderdoc/agent-harness/cli_anything/renderdoc/tests/test_core.py
• renderdoc/agent-harness/cli_anything/renderdoc/tests/test_full_e2e.py

RenderDoc workable MVP design:

• command: cli-anything-renderdoc -c frame.rdc preview capture --event-id 123

• capture recipe does:

  • extract capture thumbnail
  • save render target outputs at the target event
  • dump pipeline state JSON for the same event
  • write summary with API, event id, output count, viewport size

• command: cli-anything-renderdoc -c frame.rdc preview diff --event-a 100 --event-b 200

• diff recipe does:

  • run existing pipeline diff JSON
  • save output targets for event-a
  • save output targets for event-b
  • mark output images from A as before
  • mark output images from B as after
  • attach diff JSON as diff-json

Existing code it reuses:

• capture thumbnail in renderdoc_cli.py
• output target dumping in renderdoc_cli.py
• pipeline diff in renderdoc_cli.py

Why this works:

• RenderDoc already has the hard parts: thumbnail extraction, output saving, and structured diff
• the bundle simply packages these under one viewer-readable contract

Acceptance criteria:

• given a real .rdc file and event id, the harness emits a valid bundle
• the viewer shows the thumbnail and output galleries
• diff bundles include both sides plus the JSON diff payload

Out of scope:

• no shader-side visual heatmap generation in v1
• no browser-side replay

Recommended Adoption Order After The First 4 PRs

Once the four PRs above land, the next wave should target:

• kdenlive, because it can reuse most of the Shotcut preview logic
• godot, because it is already listed as a preview-capable provider in the matrix
• krita, because frame/image preview is already natural there

Decision Summary

The protocol is intentionally simple:

• standardize preview artifacts, not app-specific viewers
• use real software for preview generation
• make previews cheap, cacheable, and inspectable
• centralize preview consumption in cli-hub

That is the generic, compact, and practical path that should scale across the repo.