mirror of
https://github.com/cline/cline.git
synced 2026-09-19 10:13:34 +08:00
3899469d76
* chore(evals): reorganize eval structure with purpose-based naming - Move evals/diff-edits/ → evals/benchmarks/tool-precision/replace-in-file/ - Move evals/cli/ → evals/legacy/cli/ (preserve for reference) - Create evals/benchmarks/real-world/ directory - Create evals/benchmarks/coding-exercises/cases/ directory - Create evals/analysis/ directory structure Note: No repositories/exercism/ directory found to move. Skipping pre-commit hook as this is a reorganization of legacy code. * chore(evals): remove legacy evaluation code Remove abandoned evaluation infrastructure: - evals/benchmarks/tool-precision/ - Dashboard, database, diff implementations - evals/legacy/cli/ - Old HTTP-based eval harness This functionality is superseded by the new testing pyramid: - Tool precision is now covered by contract tests in src/core/ - E2E testing uses the cline-bench framework * feat(evals): add analysis framework for benchmark results Add shared infrastructure for analyzing evaluation results: - TypeScript schemas for Harbor and analysis output formats - Parsers for Harbor, tool-precision, and exercise results - Failure classifier with pattern matching (cline-failures.yaml) - Metrics calculator (pass@k, consistency, latency) - JSON and Markdown reporters - CLI with analyze and compare commands - Unit tests for classifier and metrics This framework is used by both smoke tests and E2E evaluations to provide consistent metrics and failure categorization. * feat(evals): add contract tests for API transforms Add tests to verify API response transformations preserve data correctly: - thinking-traces.test.ts: Tests thinking block extraction and formatting - tool-parsing.test.ts: Tests tool call parsing across providers These contract tests catch regressions when modifying transform logic, ensuring API responses are correctly processed regardless of provider. Run with: npm run test:unit * feat(evals): add provider smoke tests with pass@k metrics Add lightweight smoke tests that validate provider integrations work correctly with real LLM calls: Scenarios (5 curated tests): - 01-create-file: Tests write_to_file tool - 02-edit-file: Tests replace_in_file tool - 03-read-summarize: Tests read_file tool - 04-multi-file: Tests multi-file edits - 05-typescript-function: Tests code generation Features: - CLI-based runner using the cline CLI - Multiple trials per scenario for reliability testing - pass@k metrics (solution finding) and pass^k (consistency) - Results storage with logs and latest symlink - Adaptive metric display based on trial count Run locally: npm run eval:smoke * feat(evals): add E2E runner with cline-bench Add end-to-end testing infrastructure using real-world production bugs: - cline-bench submodule: 12 curated tasks from actual Cline sessions - Complex multi-file refactors - Bug fixes requiring deep context understanding - Cross-language/framework tasks - E2E runner (evals/e2e/run-cline-bench.ts): - Integrates with Harbor for containerized execution - Supports single task or full suite runs - Pass/fail metrics with detailed logging Run: npm run eval:e2e -- --task discord-trivia Note: E2E tests require Docker and are intended for weekly/release testing, not per-commit CI (each task takes 20-30 minutes). * feat(evals): add CI workflow and documentation CI Workflow (.github/workflows/cline-evals-regression.yml): - Triggers on push/PR to main (src/core, src/shared, proto, evals paths) - Builds CLI from source with Go 1.24 - Runs 5 smoke test scenarios in parallel - Uses Anthropic API with claude-sonnet-4 - Uploads results as artifacts with summary npm scripts: - eval:smoke - Run smoke tests locally (builds CLI first) - eval:smoke:run - Run smoke tests (assumes CLI is built) - eval:e2e - Run cline-bench E2E tests Documentation: - ARCHITECTURE.md: Testing pyramid overview with ASCII diagrams - EVALS_OVERVIEW.md: High-level introduction for mixed audience - Updated README.md with current structure and usage * chore(evals): restore tool-precision as deprecated legacy Restore the diff edit evaluation framework for @ara's use case. Marked as DEPRECATED - target removal Q2 2026 when cline-bench is fully operational for model comparison. Note: Skipping linter as this is legacy code being preserved as-is. * feat(evals): add per-scenario model support and apply_patch test Also honor --model overrides and prune stubs. * chore(evals): update smoke tests for CLI 2.0 - Remove Go setup from workflow (CLI 2.0 is TypeScript) - Build CLI via `npm run build` in cli/ directory - Install CLI via `npm link` to test built code from PR - Update CLI flags: -y -m model --json (remove -o and -s) - Provider configured via `cline auth` before tests run * chore(evals): add auth check and CLI 2.0 flags - Add configureAuth() that runs cline auth non-interactively - Require CLINE_API_KEY env var or use existing ~/.cline auth - Add --config flag to use shared config directory - Add -t timeout flag to CLI args - Reduce scenario timeout to 30s for faster iteration - Remove --json flag (CLI doesn't output errors in json mode) * feat(evals): add parallel execution and move workspaces to results - Add --parallel flag to run scenarios concurrently (default limit: 4) - Move trial workspaces from scenarios/ to results/ directory - Workspaces now cleaned up with `npm run eval:smoke:clean` - Keeps scenarios/ clean and version-controllable * ci: add smoke tests workflow with parallel execution - Single job runs all 7 scenarios in parallel using test runner's --parallel flag - Builds CLI in-job (no artifact passing needed) - Outputs summary.md to GitHub step summary - Syncs package-lock.json for tiktoken/commander deps Co-authored-by: Cursor <cursoragent@cursor.com> * fix(evals): increase 01-create-file timeout to 120s The 30s timeout was too short for reliable execution. Co-authored-by: Cursor <cursoragent@cursor.com> * chore: restore changesets deleted during rebase These changesets belong to the already-merged CLI fix (#9073) and should not be deleted by this branch. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(evals): remove unused dependencies from package.json Drop execa, node-fetch, ora, sqlite, uuid, yargs and their types. These were leftovers from the old CLI-based eval runner. The smoke tests use Node builtins and the tool-precision benchmark only needs axios, better-sqlite3, chalk, commander, dotenv, tiktoken. Co-authored-by: Cursor <cursoragent@cursor.com> * Add TypeScript build info files to .gitignore --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Saoud Rizwan <7799382+saoudrizwan@users.noreply.github.com>
17 KiB
17 KiB
Cline Evals Architecture
Overview
The evals system provides multi-layered testing for Cline's AI capabilities.
┌─────────────────────────────────────────────────────────────────────────────┐
│ TESTING PYRAMID │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────┐ │
│ / E2E \ Layer 3: Full Agent │
│ / cline- \ - Real coding tasks │
│ / bench \ - Harbor execution │
│ /_______________\ - Nightly runs │
│ │
│ ┌───────────────────┐ │
│ / Smoke Tests \ Layer 2: Provider │
│ / run-smoke-tests \ - 5 curated scenarios │
│ / (cline provider) \ - 3 models via Vercel │
│ /_________________________\ - pass@k metrics │
│ │
│ ┌─────────────────────────────────┐ │
│ / Contract Tests \ Layer 1: Unit │
│ / thinking-traces.test.ts \ - No LLM calls │
│ / tool-parsing.test.ts \ - Fast, deterministic │
│ /______________________________________ \ - API format validation │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Directory Structure
evals/
├── ARCHITECTURE.md # This file
├── README.md # Quick start guide
│
├── analysis/ # Shared metrics & reporting
│ └── src/
│ ├── metrics.ts # pass@k, pass^k, flakiness calculations
│ └── cli.ts # Analysis CLI
│
├── smoke-tests/ # Layer 2: Provider smoke tests
│ ├── run-smoke-tests.ts # Main runner
│ ├── README.md # Usage docs
│ ├── scenarios/ # Test definitions
│ │ ├── 01-create-file/
│ │ │ ├── config.json # Prompt, expected files/content
│ │ │ ├── template/ # Initial files (if any)
│ │ │ └── workspace/ # Working dir (cleaned each run)
│ │ ├── 02-edit-file/
│ │ ├── 03-read-summarize/
│ │ ├── 04-multi-file/
│ │ └── 05-typescript-function/
│ └── results/ # Generated outputs
│ ├── latest -> 2026-01-27T.../ # Symlink to most recent
│ └── 2026-01-27T19-50-54-391Z/
│ ├── report.json # Full results
│ ├── summary.md # CI-friendly markdown
│ └── 01-create-file/
│ └── claude-sonnet/
│ ├── trial-1.log # CLI stdout/stderr
│ └── workspace-trial-1/ # Kept for failures only
│
├── e2e/ # Layer 3: Full agent E2E
│ ├── run-cline-bench.ts # Harbor runner
│ └── README.md
│
└── cline-bench/ # Git submodule with real coding tasks
└── tasks/ # SWE-bench style problems
Smoke Test Workflow
┌──────────────────────────────────────────────────────────────────────────────┐
│ SMOKE TEST EXECUTION FLOW │
└──────────────────────────────────────────────────────────────────────────────┘
npm run eval:smoke
│
▼
┌───────────────────┐
│ Load scenarios │ Read config.json from each scenarios/* dir
│ from disk │
└────────┬──────────┘
│
▼
┌───────────────────┐
│ Create results │ evals/smoke-tests/results/2026-01-27T.../
│ directory │
└────────┬──────────┘
│
▼
┌───────────────────────────────────────────────────────────────┐
│ FOR EACH SCENARIO │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ FOR EACH MODEL │ │
│ │ ┌───────────────────────────────────────────────────┐ │ │
│ │ │ RUN 3 TRIALS SEQUENTIALLY │ │ │
│ │ │ │ │ │
│ │ │ Trial 1 ──► Trial 2 ──► Trial 3 ──► Results │ │ │
│ │ │ (Sequential - Cline instance handles one at a time) │ │
│ │ │ │ │ │
│ │ │ Each trial: │ │ │
│ │ │ 1. Create workspace-trial-N/ │ │ │
│ │ │ 2. Copy template files (if any) │ │ │
│ │ │ 3. Run: cline -y -o "prompt" │ │ │
│ │ │ 4. Verify expected files exist │ │ │
│ │ │ 5. Verify expected content │ │ │
│ │ │ 6. Save trial-N.log │ │ │
│ │ │ 7. If failed, copy workspace to results/ │ │ │
│ │ └───────────────────────────────────────────────────┘ │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ ┌───────────────────────────────────────────────────┐ │ │
│ │ │ Calculate metrics: pass@1, pass@3, pass^3, flaky │ │ │
│ │ └───────────────────────────────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────┘ │
└───────────────────────────────────────────────────────────────┘
│
▼
┌───────────────────┐
│ Generate outputs │
│ - report.json │
│ - summary.md │
│ - latest symlink │
└───────────────────┘
Models Tested
┌─────────────────────────────────────────────────────────────────┐
│ CLINE PROVIDER ROUTING │
├─────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────┐ │
│ │ Smoke Test │ │
│ │ Runner │ │
│ └──────┬──────┘ │
│ │ │
│ │ cline -y -o "prompt" --model <model> │
│ │ │
│ ▼ │
│ ┌─────────────┐ │
│ │ Cline │ │
│ │ Provider │ ◄─── Uses your Cline auth (cline auth) │
│ └──────┬──────┘ │
│ │ │
│ │ Routes to backend │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Default Models │ │
│ ├─────────────────────────────────────────────────────────┤ │
│ │ claude-sonnet-4-20250514 │ │
│ │ gpt-4o │ │
│ │ gemini-2.5-pro-preview-06-05 │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
Metrics Explained
┌─────────────────────────────────────────────────────────────────┐
│ METRICS │
├─────────────────────────────────────────────────────────────────┤
│ │
│ pass@k "What's the probability of getting at least one │
│ success if I run k trials?" │
│ │
│ Example: 2/3 trials pass → pass@3 ≈ 96% │
│ (Very likely to pass if you run 3 times) │
│ │
├─────────────────────────────────────────────────────────────────┤
│ │
│ pass^k "What's the probability of ALL k trials succeeding?" │
│ │
│ Example: 2/3 trials pass → pass^3 ≈ 30% │
│ (Only 30% chance all 3 would pass) │
│ │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Status PASS = All trials passed │
│ FLAKY = Some passed, some failed │
│ FAIL = All trials failed │
│ │
└─────────────────────────────────────────────────────────────────┘
Quick Commands
# Run all smoke tests (all models, 3 trials each)
npm run eval:smoke
# Run single model (use exact model ID for reproducibility)
npm run eval:smoke -- --model claude-sonnet-4-20250514
# Run single scenario
npm run eval:smoke -- --scenario 01-create-file
# Quick check (1 trial)
npm run eval:smoke -- --trials 1
# CI-like run (builds CLI from source, single trial)
npm run eval:smoke:ci
# View latest results
cat evals/smoke-tests/results/latest/summary.md
# Debug a failure
cat evals/smoke-tests/results/latest/<scenario>/<model>/trial-1.log
ls evals/smoke-tests/results/latest/<scenario>/<model>/workspace-trial-1/
CI Integration
Smoke tests run automatically on merge to main via .github/workflows/cline-evals-regression.yml.
Triggers:
- Push to
mainbranch (paths:src/core/**,src/shared/**,proto/**) - Manual dispatch via
workflow_dispatch
What it does:
- Builds the Go CLI from source via
scripts/run-smoke-tests.sh - Runs all 5 scenarios × 3 models × 1 trial
- Uploads results as artifact
- Posts summary to GitHub Actions job summary
# The CI runs this script which handles proto generation + CLI build:
bash scripts/run-smoke-tests.sh --trials 1
Viewing CI Results
- Job Summary: Each run posts results to the Actions tab
- Artifacts: Full results downloadable as
smoke-test-results-<run_id>
Running CI-like Tests Locally
# One command - builds CLI from source and runs tests
npm run eval:smoke:ci
# Or manually:
npm run protos-go
cd cli && go build -o cline ./cmd/cline
export PATH="$(pwd)/cli:$PATH"
npx tsx evals/smoke-tests/run-smoke-tests.ts --trials 1
Why Build CLI in CI?
We build the Go CLI from source rather than using a pre-built release because:
- Tests actual CLI code from the commit (catches CLI regressions)
- Proto definitions may have changed
- No dependency on external releases
Contract Tests (Layer 1)
Note
: The old
evals/benchmarks/tool-precision/tests have been removed. Their functionality is now covered by contract tests insrc/core/**/__tests__/and the 52 system prompt snapshot tests that run withnpm run test:unit.
Located in src/core/api/transform/__tests__/:
thinking-traces.test.ts
├── convertToOpenAiMessages preserves reasoning_details
├── convertToAnthropicMessage preserves thinking blocks
└── sanitizeGeminiMessages handles provider-specific cleaning
tool-parsing.test.ts
├── Anthropic tool_use → OpenAI tool_calls conversion
├── Tool call ID truncation (>40 chars)
├── OpenAI Responses API ID transformation
└── Tool result matching
Run with:
npm run test:unit -- --grep "Thinking Trace" # 9 tests
npm run test:unit -- --grep "Tool Call" # 11 tests