opensource/cline

Fork 0

mirror of https://github.com/cline/cline.git synced 2026-09-19 10:13:34 +08:00

Files

T

History

3899469d76 feat(evals): comprehensive LLM evaluation framework with CI (#8909 )

* chore(evals): reorganize eval structure with purpose-based naming

- Move evals/diff-edits/ → evals/benchmarks/tool-precision/replace-in-file/
- Move evals/cli/ → evals/legacy/cli/ (preserve for reference)
- Create evals/benchmarks/real-world/ directory
- Create evals/benchmarks/coding-exercises/cases/ directory
- Create evals/analysis/ directory structure

Note: No repositories/exercism/ directory found to move.
Skipping pre-commit hook as this is a reorganization of legacy code.

* chore(evals): remove legacy evaluation code

Remove abandoned evaluation infrastructure:
- evals/benchmarks/tool-precision/ - Dashboard, database, diff implementations
- evals/legacy/cli/ - Old HTTP-based eval harness

This functionality is superseded by the new testing pyramid:
- Tool precision is now covered by contract tests in src/core/
- E2E testing uses the cline-bench framework

* feat(evals): add analysis framework for benchmark results

Add shared infrastructure for analyzing evaluation results:
- TypeScript schemas for Harbor and analysis output formats
- Parsers for Harbor, tool-precision, and exercise results
- Failure classifier with pattern matching (cline-failures.yaml)
- Metrics calculator (pass@k, consistency, latency)
- JSON and Markdown reporters
- CLI with analyze and compare commands
- Unit tests for classifier and metrics

This framework is used by both smoke tests and E2E evaluations
to provide consistent metrics and failure categorization.

* feat(evals): add contract tests for API transforms

Add tests to verify API response transformations preserve data correctly:
- thinking-traces.test.ts: Tests thinking block extraction and formatting
- tool-parsing.test.ts: Tests tool call parsing across providers

These contract tests catch regressions when modifying transform logic,
ensuring API responses are correctly processed regardless of provider.
Run with: npm run test:unit

* feat(evals): add provider smoke tests with pass@k metrics

Add lightweight smoke tests that validate provider integrations work
correctly with real LLM calls:

Scenarios (5 curated tests):
- 01-create-file: Tests write_to_file tool
- 02-edit-file: Tests replace_in_file tool
- 03-read-summarize: Tests read_file tool
- 04-multi-file: Tests multi-file edits
- 05-typescript-function: Tests code generation

Features:
- CLI-based runner using the cline CLI
- Multiple trials per scenario for reliability testing
- pass@k metrics (solution finding) and pass^k (consistency)
- Results storage with logs and latest symlink
- Adaptive metric display based on trial count

Run locally: npm run eval:smoke

* feat(evals): add E2E runner with cline-bench

Add end-to-end testing infrastructure using real-world production bugs:

- cline-bench submodule: 12 curated tasks from actual Cline sessions
  - Complex multi-file refactors
  - Bug fixes requiring deep context understanding
  - Cross-language/framework tasks

- E2E runner (evals/e2e/run-cline-bench.ts):
  - Integrates with Harbor for containerized execution
  - Supports single task or full suite runs
  - Pass/fail metrics with detailed logging

Run: npm run eval:e2e -- --task discord-trivia

Note: E2E tests require Docker and are intended for weekly/release
testing, not per-commit CI (each task takes 20-30 minutes).

* feat(evals): add CI workflow and documentation

CI Workflow (.github/workflows/cline-evals-regression.yml):
- Triggers on push/PR to main (src/core, src/shared, proto, evals paths)
- Builds CLI from source with Go 1.24
- Runs 5 smoke test scenarios in parallel
- Uses Anthropic API with claude-sonnet-4
- Uploads results as artifacts with summary

npm scripts:
- eval:smoke - Run smoke tests locally (builds CLI first)
- eval:smoke:run - Run smoke tests (assumes CLI is built)
- eval:e2e - Run cline-bench E2E tests

Documentation:
- ARCHITECTURE.md: Testing pyramid overview with ASCII diagrams
- EVALS_OVERVIEW.md: High-level introduction for mixed audience
- Updated README.md with current structure and usage

* chore(evals): restore tool-precision as deprecated legacy

Restore the diff edit evaluation framework for @ara's use case.
Marked as DEPRECATED - target removal Q2 2026 when cline-bench
is fully operational for model comparison.

Note: Skipping linter as this is legacy code being preserved as-is.

* feat(evals): add per-scenario model support and apply_patch test

Also honor --model overrides and prune stubs.

* chore(evals): update smoke tests for CLI 2.0

- Remove Go setup from workflow (CLI 2.0 is TypeScript)
- Build CLI via `npm run build` in cli/ directory
- Install CLI via `npm link` to test built code from PR
- Update CLI flags: -y -m model --json (remove -o and -s)
- Provider configured via `cline auth` before tests run

* chore(evals): add auth check and CLI 2.0 flags

- Add configureAuth() that runs cline auth non-interactively
- Require CLINE_API_KEY env var or use existing ~/.cline auth
- Add --config flag to use shared config directory
- Add -t timeout flag to CLI args
- Reduce scenario timeout to 30s for faster iteration
- Remove --json flag (CLI doesn't output errors in json mode)

* feat(evals): add parallel execution and move workspaces to results

- Add --parallel flag to run scenarios concurrently (default limit: 4)
- Move trial workspaces from scenarios/ to results/ directory
- Workspaces now cleaned up with `npm run eval:smoke:clean`
- Keeps scenarios/ clean and version-controllable

* ci: add smoke tests workflow with parallel execution

- Single job runs all 7 scenarios in parallel using test runner's --parallel flag
- Builds CLI in-job (no artifact passing needed)
- Outputs summary.md to GitHub step summary
- Syncs package-lock.json for tiktoken/commander deps

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(evals): increase 01-create-file timeout to 120s

The 30s timeout was too short for reliable execution.

Co-authored-by: Cursor <cursoragent@cursor.com>

* chore: restore changesets deleted during rebase

These changesets belong to the already-merged CLI fix (#9073)
and should not be deleted by this branch.

Co-authored-by: Cursor <cursoragent@cursor.com>

* chore(evals): remove unused dependencies from package.json

Drop execa, node-fetch, ora, sqlite, uuid, yargs and their types.
These were leftovers from the old CLI-based eval runner. The smoke
tests use Node builtins and the tool-precision benchmark only needs
axios, better-sqlite3, chalk, commander, dotenv, tiktoken.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Add TypeScript build info files to .gitignore

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Saoud Rizwan <7799382+saoudrizwan@users.noreply.github.com>

2026-02-12 22:23:35 -06:00

README.md

feat(evals): comprehensive LLM evaluation framework with CI (#8909 )

2026-02-12 22:23:35 -06:00

run-cline-bench.ts

feat(evals): comprehensive LLM evaluation framework with CI (#8909 )

2026-02-12 22:23:35 -06:00

README.md

E2E Agent Tests

Full end-to-end tests using real-world tasks from cline-bench.

Overview

These tests run Cline against production-grade coding problems derived from actual user sessions. Each task:

Starts with a broken codebase in Docker
Gives Cline the task description
Verifies the fix with pytest

Prerequisites

Python 3.13 with uv

# macOS
brew install python@3.13
pip install uv

Harbor (benchmark execution framework)
```
uv tool install harbor
```
Docker (for local execution)
```
# Verify Docker is running
docker info
```

API Keys

export ANTHROPIC_API_KEY=sk-ant-...
# or
export API_KEY=sk-ant-...  # Generic fallback

Running Locally

# Run all tasks with default settings (Anthropic, Docker)
npx tsx evals/e2e/run-cline-bench.ts

# Run specific task
npx tsx evals/e2e/run-cline-bench.ts --tasks discord

# Use different provider/model
npx tsx evals/e2e/run-cline-bench.ts --provider openai --model gpt-4o

# Run on Daytona cloud (faster, parallel)
export DAYTONA_API_KEY=dtn_...
npx tsx evals/e2e/run-cline-bench.ts --env daytona

# Output to JSON
npx tsx evals/e2e/run-cline-bench.ts --output results.json

CLI Options

Option	Default	Description
`--env`	`docker`	Execution environment: `docker` or `daytona`
`--provider`	`anthropic`	Provider: `anthropic`, `openai`, `openrouter`, `gemini`
`--model`	`claude-sonnet-4-20250514`	Model ID
`--tasks`	`all`	Task filter pattern
`--trials`	`1`	Number of trials per task
`--output`	-	Write JSON results to file

Tasks

Current tasks from cline-bench (12 total):

every-plugin-api-migration - Migrate API calls in plugin
police-sync-segfault - Fix segmentation fault
intercept-axios-error-handling - Fix Axios error handling
telegram-plugin-refactor - Refactor Telegram plugin
discord-trivia-approval-keyerror - Fix KeyError in Discord bot
terraform-azurerm-deployment-stacks - Terraform provider fix
orpc-client-migration - Client migration task
v-edit-workspace-tests - Fix workspace tests
healthchain-prefetch-removal - Remove prefetch logic
aenet-pytorch-pbc-neighborlist - PyTorch PBC fix
suave-http-data-bleeding - Fix HTTP data bleeding
filmarchiver - Film archiver fixes

CI Integration

These tests run nightly (not on every PR) due to:

Long execution time (20-30 min per task)
API costs (~$1-5 per run depending on model)
Docker/Daytona infrastructure requirements

See .github/workflows/nightly-evals.yml for CI configuration.

Results

Results are written to evals/cline-bench/jobs/ directory by Harbor:

jobs/
└── 2025-01-25__10-00-00/
    ├── result.json              # Aggregate results
    └── <task-id>__<hash>/
        ├── result.json          # Trial result
        ├── agent/cline.txt      # Conversation log
        └── verifier/reward.txt  # 1 (pass) or 0 (fail)

Troubleshooting

"Harbor not found"

source .venv/bin/activate  # If using venv
uv tool install harbor

"Docker not available"

# Start Docker daemon
docker info  # Should show Docker info

Task timeouts

Some tasks (Qt WASM, Android) can take 20-30 minutes. If running locally, ensure Docker has sufficient resources (8GB+ RAM).