cline/evals/smoke-tests

Smoke Tests

Curated smoke tests that verify Cline works correctly with LLM providers.

Purpose

These tests catch regressions in:

• Tool execution (read, write, edit files)
• Provider response parsing
• Tool chaining (multiple operations)
• Basic code generation

Quick Start

# One-time auth setup
cline auth

# Run tests (3 trials by default)
npm run eval:smoke:run

Commands

Command	What it does
npm run eval:smoke:run	Run tests (uses installed CLI)

Options

# Run specific scenario
npm run eval:smoke:run -- --scenario 01-create-file

# Run with fewer trials (faster)
npm run eval:smoke:run -- --trials 1

# Run with specific model (overrides any per-scenario models)
npm run eval:smoke:run -- --model claude-sonnet-4-5-20250929

Authentication

Interactive (recommended for local dev)

cline auth

With API key (for automation)

cline auth -p cline -k "$CLINE_API_KEY" -m anthropic/claude-sonnet-4.5

Scenarios

ID	Name	What it tests
01-create-file	Create a simple file	write_to_file
02-edit-file	Edit existing file	replace_in_file
03-read-summarize	Read and summarize	read_file
04-multi-file	Create multiple files	Multiple tool calls
05-typescript-function	Generate TypeScript	Code generation
06-apply-patch	Edit file (GPT-5)	apply_patch tool, native tool calling
07-edit-gemini	Edit file (Gemini)	Gemini model variant

Per-Scenario Models

Scenarios can specify their own model(s) via the models field in config.json. This is useful for testing model-specific code paths like apply_patch (GPT-5 only).

If you pass --model, it overrides any per-scenario models list.

Examples:

# Run apply_patch scenario with its default model (GPT-5)
npm run eval:smoke:run -- --scenario 06-apply-patch

# Force that scenario to use a specific model
npm run eval:smoke:run -- --scenario 06-apply-patch --model openai/gpt-4o

Metrics

• pass@k: Probability at least 1 of k trials succeeds
• pass^k: Probability ALL k trials succeed (reliability)

Shows pass@1 when trials < 3, pass@3 otherwise.

Adding New Scenarios

1. Create directory: scenarios/<name>/
2. Add config.json:

   {
     "name": "Human-readable name",
     "description": "What this tests",
     "prompt": "The task prompt for Cline",
     "expectedFiles": ["file1.txt"],
     "expectedContent": [
       { "file": "file1.txt", "contains": "expected text" }
     ],
     "timeout": 60
   }
   

3. (Optional) Add template/ directory with starting files

CI Integration

Smoke test CI is temporarily disabled while the build step is repointed at the SDK CLI. The remaining root script runs the scenarios against whichever cline binary is already on $PATH.

Required Secrets

• CLINE_API_KEY - Cline API key, needed when CI is re-enabled

Viewing Results

• Local summaries are written under evals/smoke-tests/results/latest/

TODO

• Native tool calling tests: Add CLI support for native_tool_call_enabled setting, then create a scenario that tests Claude 4 with native tool calling enabled (currently only GPT-5 models automatically use native tools via the Responses API)