feat(task): add system prompt fingerprinting and artifact export

- Log system prompt variant, fingerprint, provider, and model details on the first API request to aid in debugging.
- Implement `writePromptArtifacts` to save the raw system prompt and a JSON manifest to disk.
- Enable artifact generation via the `CLINE_WRITE_PROMPT_ARTIFACTS` environment variable.
- Allow configuration of the output directory via `CLINE_PROMPT_ARTIFACT_DIR`.

.changeset/bright-rooms-wonder.md

@@ -0,0 +1,5 @@
+---
+"cline": patch
+---
+
+Replace the LiteLLM model list with a selector

.changeset/cli-stdin-hang.md

@@ -0,0 +1,5 @@
+---
+"cline": patch
+---
+
+fix(cli): prevent hang when spawned without TTY

.changeset/cold-opus-launch.md

@@ -0,0 +1,5 @@
+---
+"cline": patch
+---
+
+Add Claude Opus 4.6 model support

.changeset/dark-hotels-care.md

@@ -1,5 +0,0 @@
----
-"claude-dev": patch
----
-
-Add .agents/skills as default skill directory (global and local)

.changeset/evals-testing-framework.md

@@ -1,10 +0,0 @@
----
-"claude-dev": patch
----
-
-Add comprehensive LLM evaluation framework with CI integration
-
-- Smoke tests: 7 curated scenarios testing tools across providers (Claude, GPT-5, Gemini)
-- Analysis framework: pass@k metrics for measuring reliability
-- CI workflow: Parallel smoke tests on PRs with ~3min execution time
-- E2E runner: cline-bench integration for real-world task evaluation (local only for now)

.changeset/famous-poems-smell.md

@@ -0,0 +1,5 @@
+---
+"cline": patch
+---
+
+Supports rendering markdown table in chat view.

.changeset/fix-cli-stdin-ci.md

@@ -0,0 +1,5 @@
+---
+"cline": patch
+---
+
+Fix CLI crashing in CI environments and with stdin redirection (e.g., `cline "prompt" < /dev/null`). Now checks both stdin and stdout TTY status before using Ink, and only errors on empty stdin when no prompt is provided.

.changeset/fix-vscode-server-auth.md

@@ -0,0 +1,7 @@
+---
+"cline": patch
+---
+
+fix: use vscode.env.openExternal for auth in remote environments
+
+Fixes OAuth authentication in VS Code Server and remote environments by routing browser URL opening through VS Code's native openExternal API instead of the npm 'open' package.

.changeset/hungry-ants-pump.md

@@ -0,0 +1,5 @@
+---
+"claude-dev": patch
+---
+
+Updating script documentation and removing unnecessary continue on error

.changeset/plain-lies-repair.md

@@ -0,0 +1,5 @@
+---
+"cline": patch
+---
+
+Fix Bedrock model id

.changeset/swift-comics-hammer.md

@@ -1,5 +0,0 @@
----
-"cline": patch
----
-
-Allows users to enter custom aws region when selecting bedrock as a provider

.changeset/two-keys-create.md

@@ -0,0 +1,5 @@
+---
+"cline": minor
+---
+
+Add Generate API Key on Hicap Provider selection

.changeset/wicked-tables-sneeze.md

@@ -1,5 +0,0 @@
----
-"cline": patch
----
-
-Prevent Parent Container Scrolling In Dropdowns

.cline/skills/create-pull-request/SKILL.md

@@ -147,29 +147,14 @@ When filling out the template:
 
 ### Create PR with gh CLI
 
-**Use a temporary file for the PR body** to avoid shell escaping issues, newline problems, and other command-line flakiness:
-
-1. Write the PR body to a temporary file:
-   ```
-   /tmp/pr-body.md
-   ```
-
-2. Create the PR using the file:
-   ```bash
-   gh pr create --title "PR_TITLE" --body-file /tmp/pr-body.md --base main
-   ```
-
-3. Clean up the temporary file:
-   ```bash
-   rm /tmp/pr-body.md
-   ```
-
-For draft PRs:
 ```bash
-gh pr create --title "PR_TITLE" --body-file /tmp/pr-body.md --base main --draft
+gh pr create --title "PR_TITLE" --body "PR_BODY" --base main
 ```
 
-**Why use a file?** Passing complex markdown with newlines, special characters, and checkboxes directly via `--body` is error-prone. The `--body-file` flag handles all content reliably.
+Alternatively, create as draft if the user wants review before marking ready:
+```bash
+gh pr create --title "PR_TITLE" --body "PR_BODY" --base main --draft
+```
 
 ## Post-Creation
 

.clinerules/general.md

@@ -147,17 +147,6 @@ Required steps:
 
 Common mistake: Adding only the return value without the `context.globalState.get()` call. This compiles but the value is always `undefined` on load.
 
-Settings plumbing gotcha: if a key is user-toggleable from settings, wire both controller update paths:
-- `src/core/controller/state/updateSettings.ts` for webview `updateSetting(...)`
-- `src/core/controller/state/updateSettingsCli.ts` for CLI/ACP settings updates
-Missing one path causes a toggle to appear to change in one surface while the backend state stays unchanged.
-
-Webview toggle gotcha: settings changes must also round-trip back in state payloads.
-- Add the field to `UpdateSettingsRequest` in `proto/cline/state.proto` (for webview update requests), then run `npm run protos`
-- Include the key in `Controller.getStateToPostToWebview()` (`src/core/controller/index.ts`)
-- Ensure `ExtensionState` and webview defaults include the key (`src/shared/ExtensionMessage.ts`, `webview-ui/src/context/ExtensionStateContext.tsx`)
-If this round-trip wiring is missing, the backend value can update but the toggle in webview appears stuck or reverts.
-
 ## StateManager Cache vs Direct globalState Access
 StateManager uses an in-memory cache populated during `StateManager.initialize(context)` in `common.ts`. For most state, use `controller.stateManager.setGlobalState()`/`getGlobalStateKey()`.
 

.codex/environments/environment.toml

@@ -1,34 +0,0 @@
-# THIS IS AUTOGENERATED. DO NOT EDIT MANUALLY
-version = 1
-name = "cline"
-
-[setup]
-script = '''
-if [ ! -d "node_modules" ]; then
-  MAIN_WORKTREE="$(git worktree list | head -n1 | awk '{print $1}')"
-  ln -s "$MAIN_WORKTREE/node_modules" node_modules
-  ln -s "$MAIN_WORKTREE/webview-ui/node_modules" webview-ui/node_modules
-fi
-'''
-
-[[actions]]
-name = "VS Code"
-icon = "run"
-command = "chmod +x ./scripts/run-extension-host.sh && ./scripts/run-extension-host.sh production"
-
-[[actions]]
-name = "CLI"
-icon = "run"
-command = '''
-npm run cli:build
-npm run cli:run
-'''
-
-[[actions]]
-name = "npm install"
-icon = "tool"
-command = '''
-rm node_modules
-rm webview-ui/node_modules
-npm run install:all
-'''

.github/workflows/claude-issue-triage.yml

@@ -0,0 +1,173 @@
+name: Claude Issue Triage
+
+on:
+  issues:
+    types: [opened]
+  # Manual trigger for backfilling existing issues. Run from terminal:
+  #   gh workflow run claude-issue-triage.yml -f issue_number=1234
+  # Or batch process:
+  #   gh issue list --state open --limit 10 --json number --jq '.[].number' | while read num; do
+  #     gh workflow run claude-issue-triage.yml -f issue_number=$num
+  #     sleep 60
+  #   done
+  workflow_dispatch:
+    inputs:
+      issue_number:
+        description: 'Issue number to triage'
+        required: true
+        type: string
+
+jobs:
+  claude-issue-triage:
+
+    runs-on: ubuntu-latest
+    timeout-minutes: 120
+    # SECURITY: These permissions are intentionally restrictive.
+    # - contents: read  -> Claude can read the codebase but CANNOT write/push any code
+    # - issues: write   -> Claude can comment and add labels (the only write access needed)
+    # - pull-requests: read -> Claude can view PR context but CANNOT create PRs
+    # This ensures that even if a malicious user attempts prompt injection via issue content,
+    # Claude cannot modify repository code, create branches, or open PRs.
+    permissions:
+      contents: read
+      issues: write
+      pull-requests: read
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Run Issue Response & Triage
+        id: triage
+        uses: anthropics/claude-code-action@v1
+        with:
+          anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          allowed_non_write_users: "*"
+          # Allow all tools - security is enforced by GitHub permissions above (contents: read, issues: write)
+          claude_args: --model claude-opus-4-5-20251101 --allowedTools "Bash,Read,Write,Edit,Glob,Grep,WebFetch,WebSearch"
+          prompt: |
+            You're a GitHub issue first responder for the open source Cline repository.
+
+            **Issue:** #${{ github.event.issue.number || inputs.issue_number }}
+            **Title:** ${{ github.event.issue.title || 'See issue details below' }}
+            **Author:** @${{ github.event.issue.user.login || 'See issue details below' }}
+
+            ## Your job
+
+            Investigate this issue thoroughly, then post a single helpful comment that helps the user and gives maintainers the context they need.
+
+            ## Investigation
+
+            Start by reading the full issue:
+            gh issue view ${{ github.event.issue.number || inputs.issue_number }}
+
+            ### Search for duplicates and related issues
+
+            Search thoroughly for existing issues that match this one:
+            gh issue list --search "<keywords from the issue>" --state all --limit 30
+            gh issue list --search "<error messages>" --state all --limit 20
+            gh issue list --search "<affected feature/component>" --state all --limit 20
+
+            For each relevant issue you find, read it including its comments:
+            gh issue view <number> --comments
+
+            You're looking for:
+            - **Duplicates**: Issues describing the same problem. Link to them and explain why you think they're duplicates. If closed, check how they were resolved - the solution might apply here.
+            - **Related issues**: Similar problems or context that could help. Pull useful information from their comments (workarounds others found, debugging steps that helped, maintainer explanations). Link to them and explain the connection.
+
+            If there are closed issues with solutions, surface those solutions prominently - this might immediately solve the user's problem.
+
+            ### Analyze recent changes (ALWAYS DO THIS)
+
+            Many issues are regressions from recent releases. **Always** check what changed recently:
+            gh release list --limit 10
+            gh pr list --state merged --limit 50 --json number,title,mergedAt,author,body
+
+            Look for PRs merged in the last few weeks that might correlate with the issue. If you find a likely connection:
+            gh pr view <number>
+            gh pr diff <number>
+            git log --since="1 month ago" --oneline -- <relevant paths>
+            git show <commit>
+
+            **Always include your findings in your comment:**
+            - If you find a regression, call it out explicitly: which PR/commit likely caused it, who authored it, what changed, and suggest a fix direction if you can see one.
+            - If you don't find anything related, still mention it: "I analyzed recent PRs and releases but didn't find any changes that seem related to this issue."
+
+            ### Search the codebase
+
+            Find the relevant code:
+            - Use grep/find to locate code related to the issue
+            - Key areas: `src/api/` (providers/models), `src/core/prompts/` (tools/prompts), platform-specific code for VS Code vs JetBrains
+
+            ### Find documentation
+
+            Cline docs are at **https://docs.cline.bot/** and built with Mintlify from the `docs/` directory.
+
+            The URL structure maps directly to the file structure:
+            - `docs/getting-started/selecting-your-model.mdx` → https://docs.cline.bot/getting-started/selecting-your-model
+            - `docs/troubleshooting.mdx` → https://docs.cline.bot/troubleshooting
+            - Headings become anchors: `## Which Model` → `#which-model`
+
+            Search the `docs/` directory to find relevant documentation, then construct URLs to link users to:
+            ```bash
+            ls docs/
+            grep -r "keyword" docs/ --include="*.mdx" -l
+            ```
+
+            ### Identify subject matter experts
+
+            For issues that clearly need engineering attention:
+            git log --since="6 months ago" --format="%an" -- <relevant paths> | sort | uniq -c | sort -rn | head -5
+
+            Cross-reference with GitHub usernames. Include in your response (@mention, do NOT assign):
+
+            | SME | Reason |
+            |-----|--------|
+            | @username1 | Authored PR #X which modified this area |
+            | @username2 | Primary contributor to affected file |
+
+            ## Weak model detection
+
+            Many issues are caused by users running small or non-frontier models that don't tool-call reliably. Signs include:
+            - Model failing to use tools correctly
+            - Nonsensical or malformed responses
+            - User is running a small/local model or older model version
+
+            If this looks like a weak model issue, kindly suggest they try reproducing with Claude Sonnet and report back if it persists. Link to https://docs.cline.bot/getting-started/selecting-your-model if helpful. Still label and triage normally.
+
+            ## Your comment
+
+            Write a single comment as a helpful community member. Be conversational, not robotic. Include what's relevant:
+
+            - **Helpful response** - Answer their question, suggest a fix, provide a workaround. If you found solutions in related closed issues, surface those prominently.
+            - **Duplicates and related issues** - Link to any you found and explain why they're duplicates/related. Summarize useful context from their comments.
+            - **Regression analysis** - If this looks like a regression, explain what change likely caused it, link to the PR/commit, and tag the author.
+            - **Clarifying questions** - If you need more info, ask specific questions. Don't ask for things already provided.
+            - **SME table** - Include the table above if this needs engineering attention. Don't tag people for questions with obvious answers or weak-model issues.
+            - **Context for maintainers** - Relevant code paths, what you found. Keep it concise.
+            - **Docs links** - If there's relevant documentation, link to it naturally in your response as a recommendation (e.g., "For more details, check out [the Ollama setup guide](url)"). Do NOT add a "Sources" section at the end - integrate doc links into your response where they're helpful.
+            - **Possible Duplicates section** - ALWAYS include a "Possible Duplicates" section at the end of your comment listing issues that might be duplicates so maintainers can quickly close if appropriate. If none found, say "No obvious duplicates found."
+
+            ## Labels
+            First, retrieve all available labels and read their descriptions to understand what each is for:
+            gh label list --json name,description --limit 100
+
+            Then apply the appropriate labels based on your analysis. Only use labels from the list above—do not create new labels.
+            gh issue edit ${{ github.event.issue.number || inputs.issue_number }} --add-label "label1,label2"
+
+            If your regression analysis found a likely culprit (a recent PR/commit that probably caused this issue), add the "Regression" label:
+            gh issue edit ${{ github.event.issue.number || inputs.issue_number }} --add-label "Regression"
+
+            IMPORTANT: After posting your comment, add the "Bot Responded" label to indicate this issue has received an automated response:
+            gh issue edit ${{ github.event.issue.number || inputs.issue_number }} --add-label "Bot Responded"
+
+            ## Remember
+
+            - **This is a one-time automated response** - you will NOT see their reply or respond again. Never say things like "I can help you", "let me know", "once I have that info", or "I can give you more targeted help" - you won't be there to follow up. If you ask clarifying questions, frame them for the maintainers who will follow up, e.g., "If you can share X, that would help the maintainers diagnose this."
+            - Don't be formulaic. Respond to what the issue actually needs.
+            - Surface solutions from past issues - often the fastest path to helping.
+            - Connecting regressions to specific changes is extremely valuable.
+            - Link issues with #number so they're clickable.

.github/workflows/claude-pr-review.yml

@@ -0,0 +1,290 @@
+name: Claude PR Review
+
+on:
+  pull_request:
+    types: [opened, ready_for_review]
+  # Manual trigger for backfilling existing PRs. Run from terminal:
+  #   gh workflow run claude-pr-review.yml -f pr_number=1234
+  # Or batch process open PRs:
+  #   gh pr list --state open --limit 10 --json number --jq '.[].number' | while read num; do
+  #     gh workflow run claude-pr-review.yml -f pr_number=$num
+  #     sleep 60
+  #   done
+  workflow_dispatch:
+    inputs:
+      pr_number:
+        description: 'PR number to review'
+        required: true
+        type: string
+
+jobs:
+  claude-pr-review:
+    # Runs on PR opened/ready_for_review (skips drafts) or manual trigger for backfilling
+    if: |
+      (github.event_name == 'pull_request' && github.event.pull_request.draft == false) ||
+      github.event_name == 'workflow_dispatch'
+
+    runs-on: ubuntu-latest
+    timeout-minutes: 120
+
+    # SECURITY: These permissions are intentionally restrictive.
+    # - contents: read  -> Claude can read the codebase but CANNOT write/push any code
+    # - pull-requests: write -> Claude can post reviews and inline suggestions
+    # - issues: read -> Claude can search for related issues
+    # NOTE: Even with pull-requests: write, Claude CANNOT merge PRs because branch protection
+    # requires 1 approval from a Code Owner. The GITHUB_TOKEN cannot bypass this.
+    permissions:
+      contents: read
+      pull-requests: write
+      issues: read
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Print HEAD commit
+        run: |
+          echo "HEAD is at: $(git rev-parse HEAD)"
+          echo "Short: $(git rev-parse --short HEAD)"
+          git log -1 --format="Commit: %H%nAuthor: %an <%ae>%nDate: %ad%nMessage: %s"
+
+      - name: Get PR number
+        id: pr
+        run: |
+          if [ "${{ github.event_name }}" == "workflow_dispatch" ]; then
+            echo "number=${{ inputs.pr_number }}" >> $GITHUB_OUTPUT
+          else
+            echo "number=${{ github.event.pull_request.number }}" >> $GITHUB_OUTPUT
+          fi
+
+      - name: Run PR Review
+        id: review
+        uses: anthropics/claude-code-action@v1
+        with:
+          anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          allowed_non_write_users: "*"
+          claude_args: --model claude-opus-4-5-20251101 --allowedTools "Bash,Read,Write,Edit,Glob,Grep,WebFetch,WebSearch"
+          prompt: |
+            You're a GitHub PR reviewer for the open source Cline repository. Your goal is to give the PR author helpful feedback and give maintainers the context they need to review efficiently.
+
+            PR: #${{ steps.pr.outputs.number }}
+
+            ## Gather context
+
+            ```bash
+            # Get full PR details
+            gh pr view ${{ steps.pr.outputs.number }} --json number,title,body,author,createdAt,updatedAt,isDraft,labels,commits,files,additions,deletions,changedFiles,baseRefName,headRefName,mergeable,reviewDecision
+
+            # Get the diff
+            gh pr diff ${{ steps.pr.outputs.number }}
+
+            # Check CI status
+            gh pr checks ${{ steps.pr.outputs.number }}
+
+            # Get existing review comments (to understand context and your previous feedback)
+            gh api repos/${{ github.repository }}/pulls/${{ steps.pr.outputs.number }}/comments --jq '.[] | {user: .user.login, body: .body, path: .path, created_at: .created_at}'
+
+            # Get conversation comments
+            gh pr view ${{ steps.pr.outputs.number }} --comments
+            ```
+
+            If this is a re-review (workflow_dispatch event):
+            Read your previous comments carefully. Understand what you asked for before.
+            Check if new commits or comments address your previous feedback.
+
+            ## Check contributing guidelines
+
+            Flag (but don't block) if:
+            - Missing changeset - For user-facing changes, check if there's a `.changeset/` file:
+              ```bash
+              gh pr diff ${{ steps.pr.outputs.number }} --name-only | grep '.changeset/' || echo "No changeset found"
+              ```
+              If missing, ask them to run `npm run changeset`
+            - Missing tests - New features should have tests
+
+            ## Find related issues and PRs
+
+            Search thoroughly for context that might help with the review:
+
+            ```bash
+            # Find related issues for context
+            gh issue list --search "<keywords from the PR>" --state all --limit 30
+            gh issue list --search "<error messages or feature names>" --state all --limit 20
+
+            # Find similar PRs for reference
+            gh pr list --search "<keywords>" --state all --limit 30
+            ```
+
+            For each relevant issue or PR you find, read it including comments:
+            ```bash
+            gh issue view <number> --comments
+            gh pr view <number> --comments
+            ```
+
+            Look for:
+            - Open issues this PR might fix that weren't linked in the description
+            - Similar PRs that went through review - what feedback did they get? What patterns did they follow?
+            - Context from maintainer discussions that could inform your review
+
+            ## Find subject matter experts
+
+            For files changed in this PR, find who knows the code best:
+            ```bash
+            # Get files changed
+            gh pr diff ${{ steps.pr.outputs.number }} --name-only
+
+            # For each relevant path, find contributors
+            git log --since="6 months ago" --format="%an" -- <path> | sort | uniq -c | sort -rn | head -5
+            ```
+
+            Cross-reference git authors with GitHub usernames. Include an SME table in your response:
+
+            | SME | Reason |
+            |-----|--------|
+            | @username1 | Authored PR #X which modified this area |
+            | @username2 | Primary contributor to affected file (15 commits in 6 months) |
+            | @username3 | Reviewed similar PR #Y with extensive feedback |
+
+            ## Deep code review
+
+            This is the most important part. Don't just look for syntax issues - understand what the PR is trying to achieve and whether the implementation is the right approach.
+
+            Step 1: Understand the intent
+            Read the PR description and understand what the author is trying to accomplish. What problem are they solving? What feature are they adding?
+
+            Step 2: Form your own opinion first
+            Before analyzing their code, think about how YOU would implement this feature or fix. What files would you touch? What patterns would you follow? What edge cases would you handle?
+
+            Step 3: Compare approaches
+            Now look at their implementation. How does it compare to what you would have done?
+            - Is their approach better in some ways? Note what they did well.
+            - Is their approach missing something? Be specific about what and why.
+            - Are there edge cases they haven't considered?
+            - Does it follow the patterns established in similar parts of the codebase?
+
+            Step 4: Look at the bigger picture
+            - What other files or systems does this change interact with?
+            - Could this break anything else?
+            - Is there additional work needed beyond this PR to complete the feature?
+            - Does this fit well with the overall architecture?
+
+            Step 5: Find reference implementations
+            Look for similar changes in the codebase:
+            ```bash
+            git log --oneline --all --grep="<relevant keywords>" | head -20
+            git log --oneline -- <similar files> | head -20
+            ```
+
+            If this is adding a new API provider, look at how other providers are implemented.
+            If this is adding a new feature, look at how similar features were added.
+            Note where their implementation aligns with or diverges from established patterns.
+
+            Step 6: Standard code review checks
+            - DRY: Is there duplicated code that could be extracted?
+            - Error handling: Are errors handled appropriately?
+            - Security: Any injection risks, credential exposure, unsafe dependencies?
+            - Performance: Any obvious inefficiencies, memory leaks, N+1 patterns?
+            - Types: Is TypeScript used correctly? Any unsafe type assertions?
+            - Naming: Are variables and functions named clearly?
+            - Comments: Is complex logic explained? Are there outdated comments?
+
+            ## Inline code suggestions
+
+            For specific code improvements, use GitHub's suggestion syntax via `gh api`.
+            This creates suggestions the author can commit with one click.
+
+            Single-line suggestion:
+            ```bash
+            gh api repos/${{ github.repository }}/pulls/${{ steps.pr.outputs.number }}/reviews \
+              -X POST \
+              -f commit_id="$(gh pr view ${{ steps.pr.outputs.number }} --json headRefOid -q .headRefOid)" \
+              -f event="COMMENT" \
+              -f body="" \
+              -F comments='[
+                {
+                  "path": "src/example.ts",
+                  "line": 42,
+                  "body": "Consider simplifying:\n\n```suggestion\nconst result = items.filter(Boolean);\n```"
+                }
+              ]'
+            ```
+
+            Multi-line suggestion (replacing lines 40-45):
+            ```bash
+            gh api repos/${{ github.repository }}/pulls/${{ steps.pr.outputs.number }}/reviews \
+              -X POST \
+              -f commit_id="$(gh pr view ${{ steps.pr.outputs.number }} --json headRefOid -q .headRefOid)" \
+              -f event="COMMENT" \
+              -f body="" \
+              -F comments='[
+                {
+                  "path": "src/example.ts",
+                  "start_line": 40,
+                  "line": 45,
+                  "body": "This can be simplified:\n\n```suggestion\nconst simplified = doThing();\n```"
+                }
+              ]'
+            ```
+
+            Use inline suggestions for concrete improvements. Use regular comments for questions or broader feedback.
+
+            ## Post your review
+
+            After your investigation, post a single helpful comment that helps the author and gives maintainers context.
+
+            Start by noting the commit hash you reviewed:
+            ```bash
+            git rev-parse --short HEAD
+            ```
+            Include this at the top of your comment: "Reviewed at commit: <short hash>"
+
+            Then thank them for their contribution. Be conversational, not robotic.
+
+            Include what's relevant:
+            - In-depth explanation of what the PR does - Be comprehensive. A maintainer should be able to read this section and fully understand the author's intent, why they made the changes, how they implemented it, and what files/systems are affected. Don't just summarize - explain.
+            - Related issues/PRs you found that provide useful context (link to them)
+            - Your review findings (issues to address, suggestions, etc.)
+            - Clear next steps for the author
+
+            At the very bottom of your comment, append this exact footer:
+            ```text
+            ---
+            Generated by Claude PR Review
+            ```
+
+            Include a "For Maintainers" section with:
+            - Anything else useful to help the maintainer resolve this PR
+            - Related issues/PRs with context on why they're relevant
+            - Open issues this PR might fix that weren't linked in the description
+            - Your recommendation: merge as-is, needs changes, needs discussion, close, etc.
+            - SME table - who should review this and why
+
+            For the SME table:
+            | SME | Reason |
+            |-----|--------|
+            | @username | Primary contributor to affected files |
+
+            ## Update labels
+
+            Add appropriate labels based on your analysis:
+            ```bash
+            gh label list --json name,description --limit 100
+            gh pr edit ${{ steps.pr.outputs.number }} --add-label "label1,label2"
+            ```
+
+            When done, add the reviewed label:
+            ```bash
+            gh pr edit ${{ steps.pr.outputs.number }} --add-label "Bot Reviewed"
+            ```
+
+            ## Remember
+
+            - This is a one-time automated response - you will NOT see their reply or respond again. Never say things like "let me know if you have questions", "I can help you with", or "feel free to ask" - you won't be there to follow up. Frame any questions for the maintainers who will follow up.
+            - Be helpful and welcoming - Many contributors are new to the project
+            - Be specific - Point to exact lines and suggest fixes, don't give vague feedback
+            - Think deeply - Don't just surface-level review, understand the intent and evaluate the approach
+            - Use inline suggestions - Make it easy for authors to accept changes
+            - You're a first-pass reviewer - A human maintainer will do final approval

.github/workflows/cline-evals-regression.yml

@@ -1,70 +0,0 @@
-name: Smoke Tests
-
-on:
-  push:
-    branches: [main]
-    paths:
-      - 'src/core/**'
-      - 'src/shared/**'
-      - 'proto/**'
-      - 'evals/**'
-      - '.github/workflows/cline-evals-regression.yml'
-  pull_request:
-    paths:
-      - 'src/core/**'
-      - 'src/shared/**'
-      - 'proto/**'
-      - 'evals/**'
-      - '.github/workflows/cline-evals-regression.yml'
-  workflow_dispatch:
-
-permissions:
-  contents: read
-
-concurrency:
-  group: smoke-tests-${{ github.ref }}
-  cancel-in-progress: true
-
-jobs:
-  smoke-tests:
-    name: Smoke Tests
-    runs-on: ubuntu-latest
-    timeout-minutes: 15
-    steps:
-      - uses: actions/checkout@v4
-
-      - uses: actions/setup-node@v4
-        with:
-          node-version: '22'
-          cache: 'npm'
-
-      - name: Install dependencies
-        run: npm ci
-
-      - name: Build and install CLI
-        run: |
-          npm run protos
-          cd cli && npm install && npm run build && npm link
-          echo "$(npm config get prefix)/bin" >> $GITHUB_PATH
-
-      - name: Verify CLI
-        run: cline --version
-
-      - name: Run smoke tests
-        env:
-          CLINE_API_KEY: ${{ secrets.CLINE_API_KEY }}
-        run: |
-          cline auth -p cline -k "$CLINE_API_KEY" -m "anthropic/claude-sonnet-4.5"
-          npx tsx evals/smoke-tests/run-smoke-tests.ts --trials 1 --parallel
-
-      - name: Generate summary
-        if: always()
-        run: cat evals/smoke-tests/results/latest/summary.md >> $GITHUB_STEP_SUMMARY
-
-      - name: Upload results
-        uses: actions/upload-artifact@v4
-        if: always()
-        with:
-          name: smoke-test-results-${{ github.run_id }}
-          path: evals/smoke-tests/results/latest/
-          retention-days: 30

.github/workflows/cline-pr-review.yml

@@ -0,0 +1,331 @@
+name: Cline PR Code Review
+
+on:
+  pull_request:
+    types:
+      [opened, ready_for_review]
+      # Manual trigger for backfilling existing PRs. Run from terminal:
+      #   gh workflow run cline-pr-review.yml -f pr_number=1234
+      # Or batch process open PRs:
+      #   gh pr list --state open --limit 10 --json number --jq '.[].number' | while read num; do
+      #     gh workflow run cline-pr-review.yml -f pr_number=$num
+      #     sleep 60
+      #   done
+  workflow_dispatch:
+    inputs:
+      pr_number:
+        description: "PR number to review"
+        required: true
+        type: string
+
+concurrency:
+  group: pr-review-${{ github.event.pull_request.number || inputs.pr_number }}
+  cancel-in-progress: true
+
+jobs:
+  cline-pr-review:
+    # Runs on PR opened/ready_for_review (skips drafts) or manual trigger for backfilling
+    if: |
+      (github.event_name == 'pull_request' && github.event.pull_request.draft == false) ||
+      github.event_name == 'workflow_dispatch'
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+
+    # SECURITY: These permissions are intentionally restrictive.
+    # - contents: read  -> cline can read the codebase but CANNOT write/push any code
+    # - pull-requests: write -> cline can post reviews and inline suggestions
+    # - issues: read -> cline can search for related issues
+    # NOTE: Even with pull-requests: write, cline CANNOT merge PRs because branch protection
+    # requires 1 approval from a Code Owner. The GITHUB_TOKEN cannot bypass this.
+    permissions:
+      contents: read
+      pull-requests: write
+      issues: read
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Print HEAD commit
+        run: |
+          echo "HEAD is at: $(git rev-parse HEAD)"
+          echo "Short: $(git rev-parse --short HEAD)"
+          git log -1 --format="Commit: %H%nAuthor: %an <%ae>%nDate: %ad%nMessage: %s"
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: 22
+          cache: "npm"
+
+      - name: Install and Verify Cline CLI
+        run: |
+          npm install -g cline
+          cline version # verify installation
+
+      - name: Configure Cline with Anthropic
+        run: |
+          npx cline auth --provider anthropic \
+            --apikey "${{ secrets.ANTHROPIC_API_KEY }}" \
+            --modelid claude-opus-4-5-20251101
+
+      - name: Get PR number
+        id: pr
+        run: |
+          if [ "${{ github.event_name }}" == "workflow_dispatch" ]; then
+            echo "number=${{ inputs.pr_number }}" >> $GITHUB_OUTPUT
+          else
+            echo "number=${{ github.event.pull_request.number }}" >> $GITHUB_OUTPUT
+          fi
+
+      - name: Review PR with Cline
+        env:
+          PR_NUMBER: ${{ steps.pr.outputs.number }}
+          GITHUB_REPO: ${{ github.repository }}
+          GH_TOKEN: ${{ github.token }}
+          CLINE_COMMAND_PERMISSIONS: |
+            {
+              "allow": [
+                "gh pr diff *",
+                "gh pr view *",
+                "gh pr checks *",
+                "gh pr list *",
+                "gh label list *",
+                "gh issue list *",
+                "gh issue view *",
+                "git log *",
+                "gh pr comment ${{ steps.pr.outputs.number }} *",
+                "gh pr edit ${{ steps.pr.outputs.number }} *",
+                "gh api repos/${{ github.repository }}/pulls/${{ steps.pr.outputs.number }}/comments *",
+                "gh api repos/${{ github.repository }}/pulls/${{ steps.pr.outputs.number }}/reviews *"
+              ]
+            }
+        run: |
+          npx cline --yolo 'You'\''re a GitHub PR reviewer for the open source Cline repository. Your goal is to give the PR author helpful feedback and give maintainers the context they need to review efficiently.
+
+          PR: #'"${PR_NUMBER}"'
+
+          ## Gather context
+
+          ```bash
+          # Get full PR details
+          gh pr view '"${PR_NUMBER}"' --json number,title,body,author,createdAt,updatedAt,isDraft,labels,commits,files,additions,deletions,changedFiles,baseRefName,headRefName,mergeable,reviewDecision
+
+          # Get the diff
+          gh pr diff '"${PR_NUMBER}"'
+
+          # Check CI status
+          gh pr checks '"${PR_NUMBER}"'
+
+          # Get existing review comments (to understand context and your previous feedback)
+          gh api repos/'"${GITHUB_REPO}"'/pulls/'"${PR_NUMBER}"'/comments --jq '\''.[] | {user: .user.login, body: .body, path: .path, created_at: .created_at}'\''
+
+          # Get conversation comments
+          gh pr view '"${PR_NUMBER}"' --comments
+          ```
+
+          If this is a re-review (workflow_dispatch event):
+          Read your previous comments carefully. Understand what you asked for before.
+          Check if new commits or comments address your previous feedback.
+
+          ## Check contributing guidelines
+
+          Flag (but don'\''t block) if:
+          - Missing changeset - For user-facing changes, check if there'\''s a `.changeset/` file:
+            ```bash
+            gh pr diff '"${PR_NUMBER}"' --name-only | grep '\''.changeset/'\'' || echo '\''No changeset found'\''
+            ```
+            If missing, ask them to run `npm run changeset`
+          - Missing tests - New features should have tests
+
+          ## Find related issues and PRs
+
+          Search thoroughly for context that might help with the review:
+
+          ```bash
+          # Find related issues for context
+          gh issue list --search '\''<keywords from the PR>'\'' --state all --limit 30
+          gh issue list --search '\''<error messages or feature names>'\'' --state all --limit 20
+
+          # Find similar PRs for reference
+          gh pr list --search '\''<keywords>'\'' --state all --limit 30
+          ```
+
+          For each relevant issue or PR you find, read it including comments:
+          ```bash
+          gh issue view <number> --comments
+          gh pr view <number> --comments
+          ```
+
+          Look for:
+          - Open issues this PR might fix that weren'\''t linked in the description
+          - Similar PRs that went through review - what feedback did they get? What patterns did they follow?
+          - Context from maintainer discussions that could inform your review
+
+          ## Find subject matter experts
+
+          For files changed in this PR, find who knows the code best:
+          ```bash
+          # Get files changed
+          gh pr diff '"${PR_NUMBER}"' --name-only
+
+          # For each relevant path, find contributors
+          git log --since='\''6 months ago'\'' --format='\''%an'\'' -- <path> | sort | uniq -c | sort -rn | head -5
+          ```
+
+          Cross-reference git authors with GitHub usernames. Include an SME table in your response:
+
+          | SME | Reason |
+          |-----|--------|
+          | @username1 | Authored PR #X which modified this area |
+          | @username2 | Primary contributor to affected file (15 commits in 6 months) |
+          | @username3 | Reviewed similar PR #Y with extensive feedback |
+
+          ## Bash command usage
+
+          Don'\''t use operators like `|`, `&&`, or `;` - run each command separately and analyze the output.
+
+          When referencing command outputs, quote them properly to avoid formatting issues.
+
+          ## Deep code review
+
+          This is the most important part. Don'\''t just look for syntax issues - understand what the PR is trying to achieve and whether the implementation is the right approach.
+
+          Step 1: Understand the intent
+          Read the PR description and understand what the author is trying to accomplish. What problem are they solving? What feature are they adding?
+
+          Step 2: Form your own opinion first
+          Before analyzing their code, think about how YOU would implement this feature or fix. What files would you touch? What patterns would you follow? What edge cases would you handle?
+
+          Step 3: Compare approaches
+          Now look at their implementation. How does it compare to what you would have done?
+          - Is their approach better in some ways? Note what they did well.
+          - Is their approach missing something? Be specific about what and why.
+          - Are there edge cases they haven'\''t considered?
+          - Does it follow the patterns established in similar parts of the codebase?
+
+          Step 4: Look at the bigger picture
+          - What other files or systems does this change interact with?
+          - Could this break anything else?
+          - Is there additional work needed beyond this PR to complete the feature?
+          - Does this fit well with the overall architecture?
+
+          Step 5: Find reference implementations
+          Look for similar changes in the codebase:
+          ```bash
+          git log --oneline --all --grep='\''<relevant keywords>'\'' | head -20
+          git log --oneline -- <similar files> | head -20
+          ```
+
+          If this is adding a new API provider, look at how other providers are implemented.
+          If this is adding a new feature, look at how similar features were added.
+          Note where their implementation aligns with or diverges from established patterns.
+
+          Step 6: Standard code review checks
+          - DRY: Is there duplicated code that could be extracted?
+          - Error handling: Are errors handled appropriately?
+          - Security: Any injection risks, credential exposure, unsafe dependencies?
+          - Performance: Any obvious inefficiencies, memory leaks, N+1 patterns?
+          - Types: Is TypeScript used correctly? Any unsafe type assertions?
+          - Naming: Are variables and functions named clearly?
+          - Comments: Is complex logic explained? Are there outdated comments?
+
+          ## Inline code suggestions
+
+          For specific code improvements, use GitHub'\''s suggestion syntax via `gh api`.
+          This creates suggestions the author can commit with one click.
+
+          Single-line suggestion:
+          ```bash
+          gh api repos/'"${GITHUB_REPO}"'/pulls/'"${PR_NUMBER}"'/reviews \
+            -X POST \
+            -f commit_id="$(gh pr view '"${PR_NUMBER}"' --json headRefOid -q .headRefOid)" \
+            -f event='\''COMMENT'\'' \
+            -f body='\'''\'' \
+            -F comments='\''[
+              {
+                "path": "src/example.ts",
+                "line": 42,
+                "body": "Consider simplifying:\n\n```suggestion\nconst result = items.filter(Boolean);\n```"
+              }
+            ]'\''
+          ```
+
+          Multi-line suggestion (replacing lines 40-45):
+          ```bash
+          gh api repos/'"${GITHUB_REPO}"'/pulls/'"${PR_NUMBER}"'/reviews \
+            -X POST \
+            -f commit_id="$(gh pr view '"${PR_NUMBER}"' --json headRefOid -q .headRefOid)" \
+            -f event='\''COMMENT'\'' \
+            -f body='\'''\'' \
+            -F comments='\''[
+              {
+                "path": "src/example.ts",
+                "start_line": 40,
+                "line": 45,
+                "body": "This can be simplified:\n\n```suggestion\nconst simplified = doThing();\n```"
+              }
+            ]'\''
+          ```
+
+          Use inline suggestions for concrete improvements. Use regular comments for questions or broader feedback.
+
+          ## Post your review
+
+          After your investigation, post a single helpful comment that helps the author and gives maintainers context.
+
+          Start by noting the commit hash you reviewed:
+          ```bash
+          git rev-parse --short HEAD
+          ```
+          Include this at the top of your comment: "Reviewed at commit: <short hash>"
+
+          Then thank them for their contribution. Be conversational, not robotic.
+
+          Include what'\''s relevant:
+          - In-depth explanation of what the PR does - Be comprehensive. A maintainer should be able to read this section and fully understand the author'\''s intent, why they made the changes, how they implemented it, and what files/systems are affected. Don'\''t just summarize - explain.
+          - Related issues/PRs you found that provide useful context (link to them)
+          - Your review findings (issues to address, suggestions, etc.)
+          - Clear next steps for the author
+
+          At the very bottom of your comment, append this exact footer:
+          ```text
+          ---
+          Generated by Cline PR Code Review
+          ```
+
+          Include a '\''For Maintainers'\'' section with:
+          - Anything else useful to help the maintainer resolve this PR
+          - Related issues/PRs with context on why they'\''re relevant
+          - Open issues this PR might fix that weren'\''t linked in the description
+          - Your recommendation: merge as-is, needs changes, needs discussion, close, etc.
+          - SME table - who should review this and why
+
+          For the SME table:
+          | SME | Reason |
+          |-----|--------|
+          | @username | Primary contributor to affected files |
+
+          ## Update labels
+
+          Add appropriate labels based on your analysis:
+          ```bash
+          gh label list --json name,description --limit 100
+          gh pr edit '"${PR_NUMBER}"' --add-label '\''label1,label2'\''
+          ```
+
+          When done, add the reviewed label:
+          ```bash
+          gh pr edit '"${PR_NUMBER}"' --add-label '\''Bot Reviewed'\''
+          ```
+
+          ## Remember
+
+          - This is a one-time automated response - you will NOT see their reply or respond again. Never say things like '\''let me know if you have questions'\'', '\''I can help you with'\'', or '\''feel free to ask'\'' - you won'\''t be there to follow up. Frame any questions for the maintainers who will follow up.
+          - Be helpful and welcoming - Many contributors are new to the project
+          - Be specific - Point to exact lines and suggest fixes, don'\''t give vague feedback
+          - Think deeply - Don'\''t just surface-level review, understand the intent and evaluate the approach
+          - Use inline suggestions - Make it easy for authors to accept changes
+          - You'\''re a first-pass reviewer - A human maintainer will do final approval'

.github/workflows/npm-main.yaml

@@ -1,7 +1,7 @@
 name: Publish NPM Release
 
 on:
-  workflow_call:
+  workflow_dispatch:
     inputs:
       confirm_publish:
         description: 'Type "publish" to confirm you want to publish to NPM'
@@ -10,7 +10,6 @@ on:
 
 permissions:
   contents: write # Required for pushing tags
-  id-token: write # Required for npm trusted publishing (OIDC)
   checks: write # Required by test workflow
   pull-requests: write # Required by test workflow
 
@@ -21,7 +20,7 @@ jobs:
   publish-npm-release:
     needs: test
     name: Publish Cline CLI to NPM
-    if: github.repository == 'cline/cline' && github.ref == 'refs/heads/main' && inputs.confirm_publish == 'publish'
+    if: github.repository == 'cline/cline' && github.ref == 'refs/heads/main' && github.event.inputs.confirm_publish == 'publish'
     runs-on: ubuntu-latest
 
     steps:
@@ -31,10 +30,19 @@ jobs:
       - name: Setup Node.js
         uses: actions/setup-node@v4
         with:
-          node-version: "24.x"
+          node-version: "20.x"
           registry-url: "https://registry.npmjs.org"
 
+      # Cache root dependencies - only reuse if package-lock.json exactly matches
+      - name: Cache root dependencies
+        uses: actions/cache@v4
+        id: root-cache
+        with:
+          path: node_modules
+          key: ${{ runner.os }}-npm-${{ hashFiles('package-lock.json') }}
+
       - name: Install root dependencies and CLI dependencies
+        if: steps.check_commits.outputs.skip != 'true'
         run: npm ci --include=optional # this will also install cli deps because "cli" in included in root package.json workspaces field
 
       - name: Generate Protos
@@ -73,6 +81,8 @@ jobs:
           cat dist-standalone/package.json | grep version
 
       - name: Publish to NPM with latest tag
+        env:
+          NODE_AUTH_TOKEN: ${{ secrets.NPM_RELEASE_TOKEN }}
         run: |
           echo "Publishing version ${{ steps.version.outputs.version }} to NPM with tag 'latest'..."
           cd dist-standalone

.github/workflows/npm-nightly.yaml

@@ -1,17 +1,12 @@
 name: Publish NPM Nightly
 
 on:
-  workflow_call:
-    inputs:
-      force_publish:
-        description: "Force publish even if there are no commits in the last 24 hours"
-        required: false
-        type: boolean
-        default: false
+  schedule:
+    - cron: "0 12 * * *" # 4 AM PST (UTC-8) = 12 UTC
+  workflow_dispatch:
 
 permissions:
   contents: read
-  id-token: write # Required for npm trusted publishing (OIDC)
   checks: write # Required by test workflow
   pull-requests: write # Required by test workflow
 
@@ -32,12 +27,6 @@ jobs:
       - name: Check for recent commits
         id: check_commits
         run: |
-          if [ "${{ inputs.force_publish }}" = "true" ]; then
-            echo "force_publish enabled, proceeding with publish"
-            echo "skip=false" >> $GITHUB_OUTPUT
-            exit 0
-          fi
-
           if [ $(git rev-list --count HEAD --since="24 hours ago") -eq 0 ]; then
             echo "No commits in last 24 hours, skipping publish"
             echo "skip=true" >> $GITHUB_OUTPUT
@@ -50,9 +39,18 @@ jobs:
         if: steps.check_commits.outputs.skip != 'true'
         uses: actions/setup-node@v4
         with:
-          node-version: "24.x"
+          node-version: "20.x"
           registry-url: "https://registry.npmjs.org"
 
+      # Cache root dependencies - only reuse if package-lock.json exactly matches
+      - name: Cache root dependencies
+        if: steps.check_commits.outputs.skip != 'true'
+        uses: actions/cache@v4
+        id: root-cache
+        with:
+          path: node_modules
+          key: ${{ runner.os }}-npm-${{ hashFiles('package-lock.json') }}
+
       - name: Install root dependencies and CLI dependencies
         if: steps.check_commits.outputs.skip != 'true'
         run: npm ci --include=optional # this will also install cli deps because "cli" in included in root package.json workspaces field
@@ -120,6 +118,8 @@ jobs:
 
       - name: Publish to NPM with nightly tag
         if: steps.check_commits.outputs.skip != 'true'
+        env:
+          NODE_AUTH_TOKEN: ${{ secrets.NPM_RELEASE_TOKEN }}
         run: |
           echo "Publishing version ${{ steps.version.outputs.version }} to NPM with tag 'nightly'..."
           cd dist-standalone

.github/workflows/pack-cli.yml

@@ -1,215 +0,0 @@
-# Build and Pack CLI
-#
-# Builds a CLI tarball from any branch/commit and publishes it as a GitHub Release.
-# Requires write access to the repository (maintainers/collaborators only).
-#
-# Security: Split into two jobs to isolate untrusted build code from write tokens.
-# The build job runs arbitrary ref code with zero permissions. The release job
-# only runs trusted GitHub Actions with write scope.
-#
-# Usage (helper script, auto-detects current branch):
-#   ./scripts/build-cli-artifact.sh
-#   ./scripts/build-cli-artifact.sh feature/my-changes
-#   ./scripts/build-cli-artifact.sh feature/my-changes 1234  # comments on PR
-#
-# Usage (gh CLI directly):
-#   gh workflow run pack-cli.yml -f ref=main
-#   gh workflow run pack-cli.yml -f ref=abc123 -f pr_number=1234
-#
-# Install the built CLI (no auth required):
-#   npm install -g https://github.com/cline/cline/releases/download/cli-build-<sha>/cline-<ver>.tgz
-#
-# Find releases:
-#   gh release list --limit 10
-
-name: Build and Pack CLI
-
-permissions:
-  contents: read
-
-on:
-  workflow_dispatch:
-    inputs:
-      ref:
-        description: 'Branch, tag, or commit SHA to build (leave empty for default branch)'
-        required: false
-        type: string
-      pr_number:
-        description: 'PR number to comment on with install instructions (optional)'
-        required: false
-        type: number
-
-jobs:
-  # ── Build job: runs untrusted ref code with ZERO permissions ──
-  build:
-    name: Build CLI
-    runs-on: ubuntu-latest
-    permissions: {}
-    outputs:
-      commit_sha: ${{ steps.commit.outputs.sha }}
-      tarball: ${{ steps.pack.outputs.tarball }}
-    steps:
-      - name: Checkout code
-        uses: actions/checkout@v4
-        with:
-          ref: ${{ inputs.ref || github.ref }}
-          persist-credentials: false
-
-      - name: Get commit SHA
-        id: commit
-        run: |
-          COMMIT_SHA=$(git rev-parse --short HEAD)
-          echo "sha=$COMMIT_SHA" >> $GITHUB_OUTPUT
-          echo "Building from commit: $COMMIT_SHA"
-
-      - name: Setup Node.js
-        uses: actions/setup-node@v4
-        with:
-          node-version: "20.x"
-
-      - name: Install dependencies
-        run: npm ci --include=optional
-
-      - name: Generate Protos
-        run: npm run protos
-
-      - name: Build standalone package
-        run: node scripts/package-npm.mjs
-
-      - name: Create Tarball
-        id: pack
-        run: |
-          cd dist-standalone
-          TARBALL=$(npm pack)
-          echo "tarball=$TARBALL" >> $GITHUB_OUTPUT
-          echo "Created tarball: $TARBALL"
-
-      - name: Upload artifact
-        uses: actions/upload-artifact@v4
-        with:
-          name: cli-tarball
-          path: dist-standalone/*.tgz
-
-  # ── Release job: only trusted Actions code, with write permissions ──
-  release:
-    name: Release CLI
-    needs: build
-    runs-on: ubuntu-latest
-    permissions:
-      contents: write
-      pull-requests: write
-      issues: write
-    steps:
-      - name: Download artifact
-        uses: actions/download-artifact@v4
-        with:
-          name: cli-tarball
-          path: dist-standalone
-
-      - name: Create GitHub Release
-        id: create_release
-        uses: actions/github-script@v7
-        with:
-          script: |
-            const fs = require('fs');
-            const path = require('path');
-            const commit = '${{ needs.build.outputs.commit_sha }}';
-            const tarball = '${{ needs.build.outputs.tarball }}';
-
-            // Delete existing release/tag if re-running for the same commit
-            const tagName = `cli-build-${commit}`;
-            try {
-              const existing = await github.rest.repos.getReleaseByTag({
-                owner: context.repo.owner,
-                repo: context.repo.repo,
-                tag: tagName
-              });
-              await github.rest.repos.deleteRelease({
-                owner: context.repo.owner,
-                repo: context.repo.repo,
-                release_id: existing.data.id
-              });
-              await github.rest.git.deleteRef({
-                owner: context.repo.owner,
-                repo: context.repo.repo,
-                ref: `tags/${tagName}`
-              });
-              core.info(`Deleted existing release for ${tagName}`);
-            } catch (e) {
-              // Release doesn't exist yet, that's fine
-            }
-
-            // Create a release
-            const release = await github.rest.repos.createRelease({
-              owner: context.repo.owner,
-              repo: context.repo.repo,
-              tag_name: tagName,
-              name: `CLI Build (${commit})`,
-              body: `Automated CLI build from commit ${commit}\n\nInstall with:\n\`\`\`bash\nnpm install -g https://github.com/${context.repo.owner}/${context.repo.repo}/releases/download/${tagName}/${tarball}\n\`\`\``,
-              draft: false,
-              prerelease: true
-            });
-
-            // Upload the tarball as a release asset
-            const tarballPath = path.join('dist-standalone', tarball);
-            const tarballData = fs.readFileSync(tarballPath);
-
-            await github.rest.repos.uploadReleaseAsset({
-              owner: context.repo.owner,
-              repo: context.repo.repo,
-              release_id: release.data.id,
-              name: tarball,
-              data: tarballData
-            });
-
-            const downloadUrl = `https://github.com/${context.repo.owner}/${context.repo.repo}/releases/download/${tagName}/${tarball}`;
-            core.setOutput('release_url', release.data.html_url);
-            core.setOutput('download_url', downloadUrl);
-
-      - name: Comment on PR with download instructions
-        if: inputs.pr_number != ''
-        uses: actions/github-script@v7
-        with:
-          script: |
-            const commit = '${{ needs.build.outputs.commit_sha }}';
-            const releaseUrl = '${{ steps.create_release.outputs.release_url }}';
-            const downloadUrl = '${{ steps.create_release.outputs.download_url }}';
-            const prNumber = ${{ inputs.pr_number || 0 }};
-            if (!prNumber) return;
-
-            const comment = `## 📦 CLI Build Ready
-
-            A CLI build has been created for commit \`${commit}\`.
-
-            ### Install Directly from URL (No Authentication Required!)
-
-            \`\`\`bash
-            npm install -g ${downloadUrl}
-            \`\`\`
-
-            ### Alternative: Download and Install
-
-            \`\`\`bash
-            curl -L ${downloadUrl} -o cline.tgz
-            npm install -g ./cline.tgz
-            \`\`\`
-
-            📦 [View Release](${releaseUrl})
-            `;
-
-            await github.rest.issues.createComment({
-              issue_number: prNumber,
-              owner: context.repo.owner,
-              repo: context.repo.repo,
-              body: comment
-            });
-
-      - name: Summary
-        run: |
-          echo "✅ CLI build complete!"
-          echo ""
-          echo "📦 Release: ${{ steps.create_release.outputs.release_url }}"
-          echo "🔗 Download URL: ${{ steps.create_release.outputs.download_url }}"
-          echo ""
-          echo "Install from anywhere (no authentication required):"
-          echo "  npm install -g ${{ steps.create_release.outputs.download_url }}"

.github/workflows/publish-cli-trusted.yaml

@@ -1,53 +0,0 @@
-name: Publish CLI (Trusted)
-
-on:
-  schedule:
-    - cron: "0 12 * * *" # 4 AM PST (UTC-8) = 12 UTC
-  workflow_dispatch:
-    inputs:
-      publish_target:
-        description: "Which publish flow to run"
-        required: true
-        default: "main"
-        type: choice
-        options:
-          - main
-          - nightly
-      confirm_publish:
-        description: 'Required when publish_target=main. Type "publish" to confirm release publish.'
-        required: false
-        type: string
-      force_nightly_publish:
-        description: "Force nightly publish even with no commits in last 24h"
-        required: false
-        type: boolean
-        default: false
-
-permissions:
-  id-token: write # Required for npm trusted publishing (OIDC)
-  contents: write # Required because npm-main creates/pushes git tags
-  checks: write # Required by nested reusable test workflow
-  pull-requests: write # Required by nested reusable test workflow
-
-jobs:
-  publish-main:
-    if: |
-      github.repository == 'cline/cline' && (
-        github.event_name == 'workflow_dispatch' &&
-        github.event.inputs.publish_target == 'main' &&
-        github.event.inputs.confirm_publish == 'publish' &&
-        !endsWith(github.actor, '[bot]')
-      )
-    uses: ./.github/workflows/npm-main.yaml
-    with:
-      confirm_publish: ${{ github.event.inputs.confirm_publish }}
-
-  publish-nightly:
-    if: |
-      github.repository == 'cline/cline' && (
-        github.event_name == 'schedule' ||
-        (github.event_name == 'workflow_dispatch' && github.event.inputs.publish_target == 'nightly')
-      )
-    uses: ./.github/workflows/npm-nightly.yaml
-    with:
-      force_publish: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.force_nightly_publish == 'true' }}

.github/workflows/publish-nightly.yml

@@ -38,6 +38,22 @@ jobs:
               with:
                   node-version: "lts/*"
 
+            # Cache root dependencies - only reuse if package-lock.json exactly matches
+            - name: Cache root dependencies
+              uses: actions/cache@v4
+              id: root-cache
+              with:
+                  path: node_modules
+                  key: ${{ runner.os }}-npm-${{ hashFiles('package-lock.json') }}
+
+            # Cache webview-ui dependencies - only reuse if package-lock.json exactly matches
+            - name: Cache webview-ui dependencies
+              uses: actions/cache@v4
+              id: webview-cache
+              with:
+                  path: webview-ui/node_modules
+                  key: ${{ runner.os }}-npm-webview-${{ hashFiles('webview-ui/package-lock.json') }}
+
             - name: Install root dependencies
               run: npm ci --include=optional
 

.github/workflows/publish.yml

@@ -44,10 +44,28 @@ jobs:
               with:
                   node-version: "lts/*"
 
+            # Cache root dependencies - only reuse if package-lock.json exactly matches
+            - name: Cache root dependencies
+              uses: actions/cache@v4
+              id: root-cache
+              with:
+                  path: node_modules
+                  key: ${{ runner.os }}-npm-${{ hashFiles('package-lock.json') }}
+
+            # Cache webview-ui dependencies - only reuse if package-lock.json exactly matches
+            - name: Cache webview-ui dependencies
+              uses: actions/cache@v4
+              id: webview-cache
+              with:
+                  path: webview-ui/node_modules
+                  key: ${{ runner.os }}-npm-webview-${{ hashFiles('webview-ui/package-lock.json') }}
+
             - name: Install root dependencies
+              if: steps.root-cache.outputs.cache-hit != 'true'
               run: npm install --include=optional
 
             - name: Install webview-ui dependencies
+              if: steps.webview-cache.outputs.cache-hit != 'true'
               run: cd webview-ui && npm install --include=optional
 
             - name: Install Publishing Tools

.gitignore

@@ -48,6 +48,3 @@ test-results
 .secrets
 
 *.tsbuildinfo
-
-# Smoke test results (generated)
-evals/smoke-tests/results/

.gitmodules

@@ -1,3 +0,0 @@
-[submodule "evals/cline-bench"]
-	path = evals/cline-bench
-	url = https://github.com/cline/cline-bench.git

CHANGELOG.md

@@ -1,40 +1,5 @@
 # Changelog
 
-## [3.59.0]
-
-- Added Minimax 2.5 Free Promo
-- Fixed Response chaining for OpenAI's Responses API 
-
-## [3.58.0]
-
-### Added
-- Subagent: replace legacy subagents with the native `use_subagents` tool
-- Bundle `endpoints.json` support so packaged distributions can ship required endpoints out-of-the-box
-- Amazon Bedrock: support parallel tool calling
-- New "double-check completion" experimental feature to verify work before marking tasks complete
-- CLI: new task controls/flags including custom `--thinking` token budget and `--max-consecutive-mistakes` for yolo runs
-- Remote config: new UI/options (including connection/test buttons) and support for syncing deletion of remotely configured MCP servers
-- Vertex / Claude Code: add 1M context model options for Claude Opus 4.6
-- ZAI/GLM: add GLM-5
-
-### Fixed
-- CLI: handle stdin redirection correctly in CI/headless environments
-- CLI: preserve OAuth callback paths during auth redirects
-- VS Code Web: generate auth callback URLs via `vscode.env.asExternalUri` (OAuth callback reliability)
-- Terminal: surface command exit codes in results and improve long-running `execute_command` timeout behavior
-- UI: add loading indicator and fix `api_req_started` rendering
-- Task streaming: prevent duplicate streamed text rows after completion
-- API: preserve selected Vercel model when model metadata is missing
-- Telemetry: route PostHog networking through proxy-aware shared fetch and ensure telemetry flushes on shutdown
-- CI: increase Windows E2E test timeout to reduce flakiness
-
-### Changed
-- Settings/model UX: move "reasoning effort" into model configuration and expose it in settings
-- CLI provider selection: limit provider list to those remotely configured
-- UI: consolidate ViewHeader component/styling across views
-- Tools: add auto-approval support for `attempt_completion` commands
-- Remotely configured MCP server schema now supports custom headers
-
 ## [3.57.1]
 
 ### Fixed

biome.jsonc

@@ -61,8 +61,6 @@
 				"noUselessElse": "info"
 			},
 			"suspicious": {
-				"noSkippedTests": "warn",
-				"noFocusedTests": "error",
 				"noDoubleEquals": "warn",
 				"noImplicitAnyLet": "info",
 				"noThenProperty": "off",

cli/CHANGELOG.md

@@ -1,70 +0,0 @@
-# cline
-
-## [2.2.1]
-
-- Added Minimax 2.5 Free Promo
-- Fixed Response chaining for OpenAI's Responses API 
-
-## [2.2.0]
-
-### Added
-
-- Subagent: replace legacy subagents with the native `use_subagents` tool
-- Bundle `endpoints.json` support so packaged distributions can ship required endpoints out-of-the-box
-- Amazon Bedrock: support parallel tool calling
-- New "double-check completion" experimental feature to verify work before marking tasks complete
-- CLI: new task controls/flags including custom `--thinking` token budget and `--max-consecutive-mistakes` for yolo runs
-- Remote config: new UI/options (including connection/test buttons) and support for syncing deletion of remotely configured MCP servers
-- Vertex / Claude Code: add 1M context model options for Claude Opus 4.6
-- ZAI/GLM: add GLM-5
-
-### Fixed
-
-- CLI: handle stdin redirection correctly in CI/headless environments
-- CLI: preserve OAuth callback paths during auth redirects
-- VS Code Web: generate auth callback URLs via `vscode.env.asExternalUri` (OAuth callback reliability)
-- Terminal: surface command exit codes in results and improve long-running `execute_command` timeout behavior
-- UI: add loading indicator and fix `api_req_started` rendering
-- Task streaming: prevent duplicate streamed text rows after completion
-- API: preserve selected Vercel model when model metadata is missing
-- Telemetry: route PostHog networking through proxy-aware shared fetch and ensure telemetry flushes on shutdown
-- CI: increase Windows E2E test timeout to reduce flakiness
-
-### Changed
-
-- Settings/model UX: move "reasoning effort" into model configuration and expose it in settings
-- CLI provider selection: limit provider list to those remotely configured
-- UI: consolidate ViewHeader component/styling across views
-- Tools: add auto-approval support for `attempt_completion` commands
-- Remotely configured MCP server schema now supports custom headers
-
-## [2.1.0]
-
-### Minor Changes
-
-- 42ce100: Add Generate API Key on Hicap Provider selection
-
-### Patch Changes
-
-- 195294f: Add support for bundled endpoints.json in enterprise distributions. Extensions can now include a pre-configured endpoints.json file that automatically switches Cline to self-hosted mode. Includes packaging scripts for VSIX, NPM, and JetBrains plugins.
-- a1f2601: Replace the LiteLLM model list with a selector
-- 739d75a: Add Claude Code provider support for Claude Opus 4.6 and Sonnet 4.5 1M variants via both full model names and aliases (`opus[1m]`, `sonnet[1m]`), and align the `opus` alias with Opus 4.6.
-- 8440380: Add GitHub Actions workflow to build CLI from any commit for testing
-- b1a8db2: fix(cli): prevent hang when spawned without TTY
-- 7c87017: Add Claude Opus 4.6 model support
-- d116ac5: Supports rendering markdown table in chat view.
-- 6d8fb85: Fix CLI crashing in CI environments and with stdin redirection (e.g., `cline "prompt" < /dev/null`). Now checks both stdin and stdout TTY status before using Ink, and only errors on empty stdin when no prompt is provided.
-- 70a9904: Fix JetBrains sign-in regression by adding fallback for openExternal RPC
-- f440f3a: fix: use vscode.env.openExternal for auth in remote environments
-
-  Fixes OAuth authentication in VS Code Server and remote environments by routing browser URL opening through VS Code's native openExternal API instead of the npm 'open' package.
-
-- 70a9904: fix: use vscode.env.asExternalUri for auth callback URLs only in VS Code Web
-
-  Fixes OAuth callback redirect in VS Code Web (`code serve-web`, Codespaces) by using `vscode.env.asExternalUri()` to resolve the callback URI. This is gated behind a `vscode.env.uiKind === UIKind.Web` check so regular desktop VS Code continues to use the `vscode://` URI directly. The `getCallbackUrl` API now accepts a `path` parameter so the full callback URI (including route) is resolved correctly, and callers pass their path directly instead of appending after.
-
-- 5308ded: Updating script documentation and removing unnecessary continue on error
-- b514f18: Prevent duplicate streamed text rows when a partial text update arrives after the same text was already finalized.
-- 26391c9: Fix Bedrock model id
-- d19a877: Unify ViewHeader Styles Across All Views
-- 5dcaa8c: Add Vertex Claude Opus 4.6 1M model option and global endpoint support, and pass the 1M beta header for Vertex Claude requests.

cli/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "cline",
-	"version": "2.2.1",
+	"version": "2.0.5",
 	"description": "Autonomous coding agent CLI - capable of creating/editing files, running commands, using the browser, and more",
 	"main": "dist/cli.mjs",
 	"bin": {

cli/src/acp/AcpTerminalManager.ts

@@ -12,7 +12,11 @@
 
 import type * as acp from "@agentclientprotocol/sdk"
 import type { TerminalHandle } from "@agentclientprotocol/sdk"
-import { DEFAULT_TERMINAL_OUTPUT_LINE_LIMIT, PROCESS_HOT_TIMEOUT_NORMAL } from "@integrations/terminal/constants"
+import {
+	DEFAULT_SUBAGENT_TERMINAL_OUTPUT_LINE_LIMIT,
+	DEFAULT_TERMINAL_OUTPUT_LINE_LIMIT,
+	PROCESS_HOT_TIMEOUT_NORMAL,
+} from "@integrations/terminal/constants"
 import type {
 	ITerminal,
 	ITerminalManager,
@@ -138,12 +142,12 @@ export interface ManagedTerminal {
  * Wraps ACP terminal operations and emits events compatible with ITerminalProcess.
  */
 class AcpTerminalProcess extends EventEmitter<TerminalProcessEvents> implements ITerminalProcess {
-	isHot = false
-	waitForShellIntegration = false
+	isHot: boolean = false
+	waitForShellIntegration: boolean = false
 
-	private _unretrievedOutput = ""
-	private _continued = false
-	private _completed = false
+	private _unretrievedOutput: string = ""
+	private _continued: boolean = false
+	private _completed: boolean = false
 	private _hotTimeout: NodeJS.Timeout | null = null
 	private _exitWaitTimeout: NodeJS.Timeout | null = null
 	private readonly manager: AcpTerminalManager
@@ -393,7 +397,7 @@ export class AcpTerminalManager implements ITerminalManager {
 	private readonly numericIdToStringId: Map<number, string> = new Map()
 
 	/** Next numeric ID to assign */
-	private nextNumericId = 1
+	private nextNumericId: number = 1
 
 	/** Active processes indexed by numeric terminal ID */
 	private readonly processes: Map<number, AcpTerminalProcess> = new Map()
@@ -402,8 +406,9 @@ export class AcpTerminalManager implements ITerminalManager {
 	private readonly terminalInfos: Map<number, TerminalInfo> = new Map()
 
 	// Configuration options for ITerminalManager
-	private terminalReuseEnabled = true
+	private terminalReuseEnabled: boolean = true
 	private terminalOutputLineLimit: number = DEFAULT_TERMINAL_OUTPUT_LINE_LIMIT
+	private subagentTerminalOutputLineLimit: number = DEFAULT_SUBAGENT_TERMINAL_OUTPUT_LINE_LIMIT
 
 	/**
 	 * Creates a new AcpTerminalManager.
@@ -662,6 +667,14 @@ export class AcpTerminalManager implements ITerminalManager {
 		this.terminalOutputLineLimit = limit
 	}
 
+	/**
+	 * Set the maximum number of output lines for subagent commands.
+	 * @param limit Maximum number of lines
+	 */
+	setSubagentTerminalOutputLineLimit(limit: number): void {
+		this.subagentTerminalOutputLineLimit = limit
+	}
+
 	/**
 	 * Set the default terminal profile.
 	 * @param profile The profile identifier
@@ -674,10 +687,15 @@ export class AcpTerminalManager implements ITerminalManager {
 	 * Process output lines, potentially truncating if over limit.
 	 * @param outputLines Array of output lines
 	 * @param overrideLimit Optional limit override
+	 * @param isSubagentCommand Whether this is a subagent command
 	 * @returns Processed output string
 	 */
-	processOutput(outputLines: string[], overrideLimit?: number): string {
-		const limit = overrideLimit !== undefined ? overrideLimit : this.terminalOutputLineLimit
+	processOutput(outputLines: string[], overrideLimit?: number, isSubagentCommand?: boolean): string {
+		const limit = isSubagentCommand
+			? overrideLimit !== undefined
+				? overrideLimit
+				: this.subagentTerminalOutputLineLimit
+			: this.terminalOutputLineLimit
 
 		if (outputLines.length > limit) {
 			const halfLimit = Math.floor(limit / 2)

cli/src/agent/ClineAgent.ts

@@ -173,7 +173,7 @@ export class ClineAgent implements acp.Agent {
 	async initialize(params: acp.InitializeRequest, connection?: acp.AgentSideConnection): Promise<acp.InitializeResponse> {
 		this.clientCapabilities = params.clientCapabilities
 		this.initializeHostProvider(this.clientCapabilities, connection)
-		await ClineEndpoint.initialize(this.ctx.EXTENSION_DIR)
+		await ClineEndpoint.initialize()
 		await StateManager.initialize(this.ctx.extensionContext)
 
 		return {
@@ -246,8 +246,8 @@ export class ClineAgent implements acp.Agent {
 			},
 			hostBridgeClientProvider,
 			(message: string) => Logger.info(message),
-			async (path: string) => {
-				return AuthHandler.getInstance().getCallbackUrl(path)
+			async () => {
+				return AuthHandler.getInstance().getCallbackUrl()
 			},
 			async () => "", // get binary location not needed in ACP mode
 			this.ctx.EXTENSION_DIR,
@@ -973,7 +973,7 @@ export class ClineAgent implements acp.Agent {
 		// Get the callback URL first to ensure the server is ready
 		let callbackUrl: string
 		try {
-			callbackUrl = await authHandler.getCallbackUrl("/auth")
+			callbackUrl = await authHandler.getCallbackUrl()
 			Logger.debug("[ClineAgent] Callback URL ready:", callbackUrl)
 		} catch (error) {
 			Logger.error("[ClineAgent] Failed to get callback URL:", error)

cli/src/agent/messageTranslator.ts

@@ -312,10 +312,6 @@ function translateSayMessage(
 			// API request finished - no specific update needed
 			break
 
-		case "subagent_usage":
-			// Hidden aggregate metrics event used for task-level accounting.
-			break
-
 		case "task":
 			// Task started - don't echo the user's prompt back to them
 			// The ACP client already knows what they typed

cli/src/components/AuthView.tsx

@@ -5,13 +5,12 @@
 
 import { Box, Text, useApp, useInput } from "ink"
 import Spinner from "ink-spinner"
+// biome-ignore lint/style/useImportType: React is used as a value by JSX (jsx: "react" in tsconfig)
 import React, { useCallback, useEffect, useMemo, useState } from "react"
-import { refreshOcaModels } from "@/core/controller/models/refreshOcaModels"
 import { StateManager } from "@/core/storage/StateManager"
 import { openAiCodexOAuthManager } from "@/integrations/openai-codex/oauth"
 import { AuthService } from "@/services/auth/AuthService"
-import { openAiCodexDefaultModelId, openRouterDefaultModelId } from "@/shared/api"
-import { StringRequest } from "@/shared/proto/cline/common"
+import { liteLlmDefaultModelId, openAiCodexDefaultModelId, openRouterDefaultModelId } from "@/shared/api"
 import { openExternal } from "@/utils/env"
 import { COLORS } from "../constants/colors"
 import { useStdinContext } from "../context/StdinContext"
@@ -32,7 +31,6 @@ import {
 } from "./FeaturedModelPicker"
 import { ImportView } from "./ImportView"
 import { getDefaultModelId, hasModelPicker, ModelPicker } from "./ModelPicker"
-import { OcaEmployeeCheck } from "./OcaEmployeeCheck"
 import { getProviderLabel } from "./ProviderPicker"
 
 type AuthStep =
@@ -45,7 +43,6 @@ type AuthStep =
 	| "success"
 	| "error"
 	| "cline_auth"
-	| "oca_employee_check"
 	| "oca_auth"
 	| "cline_model"
 	| "openai_codex_auth"
@@ -163,6 +160,7 @@ export const AuthView: React.FC<AuthViewProps> = ({ controller, onComplete, onEr
 	const [modelId, setModelId] = useState("")
 	const [baseUrl, setBaseUrl] = useState("")
 	const [errorMessage, setErrorMessage] = useState("")
+	const [authStatus, setAuthStatus] = useState<string>("")
 	const [providerSearch, setProviderSearch] = useState("")
 	const [providerIndex, setProviderIndex] = useState(0)
 	const [clineModelIndex, setClineModelIndex] = useState(0)
@@ -173,14 +171,11 @@ export const AuthView: React.FC<AuthViewProps> = ({ controller, onComplete, onEr
 	// OCA auth hook - enabled when step is oca_auth
 	const handleOcaAuthSuccess = useCallback(async () => {
 		await applyProviderConfig({ providerId: "oca", controller })
-		// Fetch OCA models from the API - this sets actModeOcaModelId/planModeOcaModelId in state
-		await refreshOcaModels(controller, StringRequest.create({ value: "" }))
 		const stateManager = StateManager.get()
 		stateManager.setGlobalState("welcomeViewCompleted", true)
 		await stateManager.flushPendingState()
 		setSelectedProvider("oca")
-		const actModelId = stateManager.getGlobalSettingsKey("actModeOcaModelId") || ""
-		setModelId(actModelId)
+		setModelId(liteLlmDefaultModelId)
 		setStep("success")
 	}, [controller])
 
@@ -322,6 +317,7 @@ export const AuthView: React.FC<AuthViewProps> = ({ controller, onComplete, onEr
 	const startClineAuth = useCallback(async () => {
 		try {
 			setStep("cline_auth")
+			setAuthStatus("Starting authentication...")
 			await AuthService.getInstance(controller).createAuthRequest()
 		} catch (error) {
 			setErrorMessage(error instanceof Error ? error.message : String(error))
@@ -331,6 +327,7 @@ export const AuthView: React.FC<AuthViewProps> = ({ controller, onComplete, onEr
 
 	const startOcaAuth = useCallback(() => {
 		setStep("oca_auth")
+		setAuthStatus("Starting authentication...")
 		initiateOcaAuth()
 	}, [initiateOcaAuth])
 
@@ -361,8 +358,7 @@ export const AuthView: React.FC<AuthViewProps> = ({ controller, onComplete, onEr
 		(value: string) => {
 			setSelectedProvider(value)
 			if (value === "oca") {
-				// Show employee check screen before starting auth
-				setStep("oca_employee_check")
+				startOcaAuth()
 			} else if (value === "openai-codex") {
 				setStep("openai_codex_auth")
 				startOpenAiCodexAuth()
@@ -538,11 +534,8 @@ export const AuthView: React.FC<AuthViewProps> = ({ controller, onComplete, onEr
 				setBaseUrl("")
 				setStep("modelid")
 				break
-			case "oca_employee_check":
-				setStep("provider")
-				break
 			case "oca_auth":
-				setStep("oca_employee_check")
+				setStep("provider")
 				break
 			case "cline_auth":
 				setStep("menu")
@@ -682,9 +675,6 @@ export const AuthView: React.FC<AuthViewProps> = ({ controller, onComplete, onEr
 					</Box>
 				)
 
-			case "oca_employee_check":
-				return <OcaEmployeeCheck isActive={step === "oca_employee_check"} onCancel={goBack} onSignIn={startOcaAuth} />
-
 			case "oca_auth":
 			case "cline_auth":
 				return (
@@ -770,7 +760,6 @@ export const AuthView: React.FC<AuthViewProps> = ({ controller, onComplete, onEr
 	const [menuIndex, setMenuIndex] = useState(0)
 
 	// Steps that allow going back with escape (apikey handled by ApiKeyInput component)
-	// OcaEmployeeCheck handles its own escape key, so oca_employee_check is not in this list
 	const canGoBack = [
 		"provider",
 		"modelid",

cli/src/components/ChatMessage.test.tsx

@@ -1,106 +0,0 @@
-import type { ClineMessage } from "@shared/ExtensionMessage"
-import { render } from "ink-testing-library"
-import React from "react"
-import { describe, expect, it, vi } from "vitest"
-import { ChatMessage } from "./ChatMessage"
-
-vi.mock("../hooks/useTerminalSize", () => ({
-	useTerminalSize: () => ({
-		columns: 120,
-		rows: 40,
-		resizeKey: 0,
-	}),
-}))
-
-describe("ChatMessage subagent rendering", () => {
-	it("renders subagent approval prompts as a tree", () => {
-		const message: ClineMessage = {
-			ts: Date.now(),
-			type: "ask",
-			ask: "use_subagents",
-			text: JSON.stringify({
-				prompts: [
-					"Find codebase stats and size",
-					"Find funny comments and easter eggs",
-					"Find unusual patterns and history",
-				],
-			}),
-		}
-
-		const { lastFrame } = render(React.createElement(ChatMessage, { message, mode: "act" }))
-		const frame = lastFrame() || ""
-
-		expect(frame).toContain("Cline wants to run subagents")
-		expect(frame).toContain("├─   Find codebase stats and size")
-		expect(frame).toContain("├─   Find funny comments and easter eggs")
-		expect(frame).toContain("└─   Find unusual patterns and history")
-	})
-
-	it("renders subagent progress rows with compact token stats and completion checks", () => {
-		const message: ClineMessage = {
-			ts: Date.now(),
-			type: "say",
-			say: "subagent",
-			text: JSON.stringify({
-				status: "running",
-				total: 3,
-				completed: 1,
-				successes: 1,
-				failures: 0,
-				toolCalls: 21,
-				inputTokens: 0,
-				outputTokens: 0,
-				contextWindow: 0,
-				maxContextTokens: 0,
-				maxContextUsagePercentage: 0,
-				items: [
-					{
-						index: 1,
-						prompt: "Find codebase stats and size",
-						status: "completed",
-						toolCalls: 5,
-						inputTokens: 0,
-						outputTokens: 0,
-						totalCost: 0.034,
-						contextTokens: 24400,
-						contextWindow: 200000,
-						contextUsagePercentage: 12.2,
-					},
-					{
-						index: 2,
-						prompt: "Find funny comments and easter eggs",
-						status: "running",
-						toolCalls: 11,
-						inputTokens: 0,
-						outputTokens: 0,
-						totalCost: 0.056,
-						contextTokens: 31600,
-						contextWindow: 200000,
-						contextUsagePercentage: 15.8,
-					},
-					{
-						index: 3,
-						prompt: "Find unusual patterns and history",
-						status: "pending",
-						toolCalls: 5,
-						inputTokens: 0,
-						outputTokens: 0,
-						totalCost: 0,
-						contextTokens: 28900,
-						contextWindow: 200000,
-						contextUsagePercentage: 14.4,
-					},
-				],
-			}),
-		}
-
-		const { lastFrame } = render(React.createElement(ChatMessage, { isStreaming: true, message, mode: "act" }))
-		const frame = lastFrame() || ""
-
-		expect(frame).toContain("Cline is running subagents")
-		expect(frame).toContain("✓ Find codebase stats and size")
-		expect(frame).toContain("5 tool uses · 24.4k tokens · $0.03")
-		expect(frame).toContain("11 tool uses · 31.6k tokens · $0.06")
-		expect(frame).toContain("5 tool uses · 28.9k tokens · $0.00")
-	})
-})

cli/src/components/ChatMessage.tsx

@@ -17,7 +17,6 @@ import { useTerminalSize } from "../hooks/useTerminalSize"
 import { jsonParseSafe } from "../utils/parser"
 import { getToolDescription, isFileEditTool, parseToolFromMessage } from "../utils/tools"
 import { DiffView } from "./DiffView"
-import { SubagentMessage } from "./SubagentMessage"
 
 /**
  * Add "(Tab)" hint after "Act mode" mentions.
@@ -25,7 +24,7 @@ import { SubagentMessage } from "./SubagentMessage"
  * Matches just "Act mode" without requiring "to " prefix because markdown
  * processing may split "toggle to **Act mode**" into separate text chunks.
  */
-function addActModeHint(text: string, keyPrefix: string): React.ReactNode[] {
+function addActModeHint(text: string): React.ReactNode[] {
 	// Match "Act mode" in various capitalizations, but not if already followed by (Tab)
 	const actModeRegex = /\bact\s+mode\b(?!\s*\(tab\))/gi
 	const parts = text.split(actModeRegex)
@@ -42,7 +41,7 @@ function addActModeHint(text: string, keyPrefix: string): React.ReactNode[] {
 		}
 		if (matches[i]) {
 			nodes.push(
-				<React.Fragment key={`${keyPrefix}-act-mode-${i}`}>
+				<React.Fragment key={`act-mode-${i}`}>
 					{matches[i]}
 					<Text color="gray"> (Tab)</Text>
 				</React.Fragment>,
@@ -60,8 +59,6 @@ function addActModeHint(text: string, keyPrefix: string): React.ReactNode[] {
  */
 function renderInlineMarkdown(text: string): React.ReactNode[] {
 	const nodes: React.ReactNode[] = []
-	let hintCallIndex = 0
-	const addHintedText = (value: string) => addActModeHint(value, `hint-${hintCallIndex++}`)
 	// Match **bold**, *italic*, or `code` - order matters (** before *)
 	const regex = /(\*\*[^*]+\*\*|\*[^*]+\*|`[^`]+`)/g
 	let lastIndex = 0
@@ -71,7 +68,7 @@ function renderInlineMarkdown(text: string): React.ReactNode[] {
 		// Add text before match (with Act Mode hint processing)
 		if (match.index > lastIndex) {
 			const beforeText = text.slice(lastIndex, match.index)
-			nodes.push(...addHintedText(beforeText))
+			nodes.push(...addActModeHint(beforeText))
 		}
 
 		const fullMatch = match[0]
@@ -80,7 +77,7 @@ function renderInlineMarkdown(text: string): React.ReactNode[] {
 		if (fullMatch.startsWith("**") && fullMatch.endsWith("**")) {
 			// Bold - also process for Act Mode hints inside bold text
 			const boldContent = fullMatch.slice(2, -2)
-			const hintedContent = addHintedText(boldContent)
+			const hintedContent = addActModeHint(boldContent)
 			nodes.push(
 				<Text bold key={key}>
 					{hintedContent}
@@ -103,10 +100,10 @@ function renderInlineMarkdown(text: string): React.ReactNode[] {
 
 	// Add remaining text (with Act Mode hint processing)
 	if (lastIndex < text.length) {
-		nodes.push(...addHintedText(text.slice(lastIndex)))
+		nodes.push(...addActModeHint(text.slice(lastIndex)))
 	}
 
-	return nodes.length > 0 ? nodes : addHintedText(text)
+	return nodes.length > 0 ? nodes : addActModeHint(text)
 }
 
 /**
@@ -227,7 +224,7 @@ function truncate(text: string, maxLength: number): string {
 /**
  * Format tool result for display
  */
-function formatToolResult(result: string, maxLines = 5): string[] {
+function formatToolResult(result: string, maxLines: number = 5): string[] {
 	const lines = result.split("\n")
 	if (lines.length <= maxLines) {
 		return lines
@@ -449,10 +446,6 @@ export const ChatMessage: React.FC<ChatMessageProps> = ({ message, mode, isStrea
 		)
 	}
 
-	if ((type === "ask" && ask === "use_subagents") || say === "use_subagents" || say === "subagent") {
-		return <SubagentMessage isStreaming={isStreaming} message={message} mode={mode} />
-	}
-
 	// MCP response
 	if (say === "mcp_server_response" && text) {
 		const lines = formatToolResult(text, 8)
@@ -807,8 +800,6 @@ export const ChatMessageList: React.FC<ChatMessageListProps> = ({ messages, maxM
 	const displayMessages = messages.filter((m) => {
 		// Skip api_req_finished, they're just markers
 		if (m.say === "api_req_finished") return false
-		// Skip hidden aggregated usage messages
-		if (m.say === "subagent_usage") return false
 		// Skip empty text messages
 		if (m.say === "text" && !m.text?.trim()) return false
 		// Skip checkpoint messages

cli/src/components/ChatView.tsx

@@ -153,24 +153,6 @@ import { SettingsPanelContent } from "./SettingsPanelContent"
 import { SlashCommandMenu } from "./SlashCommandMenu"
 import { ThinkingIndicator } from "./ThinkingIndicator"
 
-/**
- * Persistent input storage that survives React remounts (e.g., during terminal resize).
- * Keyed by a stable identifier so each task/session maintains its own input state.
- */
-interface PersistedInputState {
-	text: string
-	cursorPos: number
-	pastedTexts: Map<number, string>
-	pasteCounter: number
-}
-
-const inputStateStorage = new Map<string, PersistedInputState>()
-
-function getInputStorageKey(controller: any, taskId?: string): string {
-	// Use taskId if available, otherwise fall back to controller instance
-	return taskId || (controller?.task?.taskId ?? "default")
-}
-
 interface ChatViewProps {
 	controller?: any
 	onExit?: () => void
@@ -369,9 +351,6 @@ export const ChatView: React.FC<ChatViewProps> = ({
 		insertText: insertTextAtCursor,
 	} = useTextInput()
 
-	// Get storage key for persisting input across remounts
-	const storageKey = useMemo(() => getInputStorageKey(ctrl, taskId), [ctrl, taskId])
-
 	// Refs for text input and cursor position (used by useHomeEndKeys and to avoid stale closures in useInput)
 	const textInputRef = useRef(textInput)
 	textInputRef.current = textInput
@@ -388,10 +367,8 @@ export const ChatView: React.FC<ChatViewProps> = ({
 	const [userScrolled, setUserScrolled] = useState(false)
 
 	// Pasted text storage - maps placeholder number to full pasted content
-	const [pastedTexts, setPastedTexts] = useState<Map<number, string>>(() => {
-		return inputStateStorage.get(storageKey)?.pastedTexts ?? new Map()
-	})
-	const pasteCounterRef = useRef<number>(inputStateStorage.get(storageKey)?.pasteCounter ?? 0)
+	const [pastedTexts, setPastedTexts] = useState<Map<number, string>>(new Map())
+	const pasteCounterRef = useRef(0)
 	// Track paste timing to combine chunks that arrive in rapid succession
 	const lastPasteTimeRef = useRef<number>(0)
 	const activePasteNumRef = useRef<number>(0)
@@ -425,29 +402,6 @@ export const ChatView: React.FC<ChatViewProps> = ({
 	// Track when we're exiting to hide UI elements before exit
 	const [isExiting, setIsExiting] = useState(false)
 
-	// Restore input state from storage on mount (after resize remount)
-	useEffect(() => {
-		const stored = inputStateStorage.get(storageKey)
-		if (stored) {
-			setTextInput(stored.text)
-			setCursorPos(stored.cursorPos)
-			setPastedTexts(stored.pastedTexts)
-			pasteCounterRef.current = stored.pasteCounter
-		}
-	}, [storageKey, setTextInput, setCursorPos])
-
-	// Persist input state to storage whenever it changes (survives remount)
-	useEffect(() => {
-		if (textInput || pastedTexts.size > 0) {
-			inputStateStorage.set(storageKey, {
-				text: textInput,
-				cursorPos,
-				pastedTexts: new Map(pastedTexts),
-				pasteCounter: pasteCounterRef.current,
-			})
-		}
-	}, [storageKey, textInput, cursorPos, pastedTexts])
-
 	// Task switch handling: when switching tasks via /history, we clear the terminal and
 	// increment a counter used as the root Box's key. This forces React to remount the tree,
 	// giving us a fresh Static instance. Mirrors how App.tsx handles resize with resizeKey.
@@ -540,14 +494,12 @@ export const ChatView: React.FC<ChatViewProps> = ({
 		clearState() // Force clear React state (bypasses empty messages check)
 		setTextInput("")
 		setCursorPos(0)
-		// Clear persisted state
-		inputStateStorage.delete(storageKey)
 
 		// Post the now-empty state
 		if (ctrl) {
 			ctrl.postStateToWebview()
 		}
-	}, [ctrl, clearState, storageKey])
+	}, [ctrl, clearState])
 
 	const refs = useRef({
 		searchTimeout: null as NodeJS.Timeout | null,
@@ -808,8 +760,6 @@ export const ChatView: React.FC<ChatViewProps> = ({
 			setCursorPos(0)
 			setPastedTexts(new Map()) // Clear stored pastes
 			pasteCounterRef.current = 0
-			// Clear persisted state
-			inputStateStorage.delete(storageKey)
 
 			try {
 				await ctrl.task.handleWebviewAskResponse(responseType, expandedText)
@@ -817,7 +767,7 @@ export const ChatView: React.FC<ChatViewProps> = ({
 				// Controller may be disposed
 			}
 		},
-		[ctrl, pendingAsk, pastedTexts, storageKey],
+		[ctrl, pendingAsk, pastedTexts],
 	)
 
 	// Handle cancel/interrupt
@@ -908,8 +858,6 @@ export const ChatView: React.FC<ChatViewProps> = ({
 			setCursorPos(0)
 			setPastedTexts(new Map()) // Clear stored pastes
 			pasteCounterRef.current = 0
-			// Clear persisted state
-			inputStateStorage.delete(storageKey)
 
 			try {
 				// Convert image paths to data URLs if needed
@@ -937,7 +885,7 @@ export const ChatView: React.FC<ChatViewProps> = ({
 				onError?.()
 			}
 		},
-		[ctrl, onError, pastedTexts, storageKey],
+		[ctrl, onError, pastedTexts],
 	)
 
 	// Auto-submit initial prompt if provided

cli/src/components/ConfigView.tsx

@@ -13,7 +13,6 @@ import {
 import { Box, Text, useApp, useInput } from "ink"
 import React, { useMemo, useState } from "react"
 import { useStdinContext } from "../context/StdinContext"
-import { fuzzyFilter } from "../utils/fuzzy-search"
 import {
 	BooleanSelect,
 	buildConfigEntries,
@@ -22,8 +21,6 @@ import {
 	HookInfo,
 	HookRow,
 	MAX_VISIBLE,
-	ObjectEditorPanel,
-	ObjectEditorState,
 	parseValue,
 	SEPARATOR,
 	SectionHeader,
@@ -108,8 +105,6 @@ export const ConfigView: React.FC<ConfigViewProps> = ({
 	const [isEditing, setIsEditing] = useState(false)
 	const [selectedIndex, setSelectedIndex] = useState(0)
 	const [editValue, setEditValue] = useState("")
-	const [searchQuery, setSearchQuery] = useState("")
-	const [objectEditor, setObjectEditor] = useState<ObjectEditorState | null>(null)
 
 	// Build entries for settings tab
 	const configEntries = useMemo(
@@ -117,13 +112,6 @@ export const ConfigView: React.FC<ConfigViewProps> = ({
 		[globalState, workspaceState],
 	)
 
-	const filteredConfigEntries = useMemo(() => {
-		if (!searchQuery.trim()) {
-			return configEntries
-		}
-		return fuzzyFilter(configEntries, searchQuery, (entry) => `${entry.key} ${String(entry.value ?? "")}`)
-	}, [configEntries, searchQuery])
-
 	// Build entries for rules tab
 	const ruleEntries = useMemo(() => {
 		const entries: ToggleEntry[] = []
@@ -171,7 +159,7 @@ export const ConfigView: React.FC<ConfigViewProps> = ({
 	const currentListLength = useMemo(() => {
 		switch (currentTab) {
 			case "settings":
-				return filteredConfigEntries.length
+				return configEntries.length
 			case "rules":
 				return ruleEntries.length
 			case "workflows":
@@ -183,14 +171,7 @@ export const ConfigView: React.FC<ConfigViewProps> = ({
 			default:
 				return 0
 		}
-	}, [
-		currentTab,
-		filteredConfigEntries.length,
-		ruleEntries.length,
-		workflowEntries.length,
-		hookEntries.length,
-		skillEntries.length,
-	])
+	}, [currentTab, configEntries.length, ruleEntries.length, workflowEntries.length, hookEntries.length, skillEntries.length])
 
 	// Get available tabs
 	const availableTabs = useMemo(() => {
@@ -210,11 +191,10 @@ export const ConfigView: React.FC<ConfigViewProps> = ({
 		setCurrentTab(newTab)
 		setSelectedIndex(0)
 		setIsEditing(false)
-		setObjectEditor(null)
 	}
 
 	// Settings tab handlers
-	const selectedConfigEntry = filteredConfigEntries[selectedIndex]
+	const selectedConfigEntry = configEntries[selectedIndex]
 
 	const handleSettingsSave = (value: string | boolean) => {
 		if (!selectedConfigEntry) {
@@ -230,43 +210,6 @@ export const ConfigView: React.FC<ConfigViewProps> = ({
 		setIsEditing(false)
 	}
 
-	const getObjectAtPath = (root: Record<string, unknown>, path: string[]): Record<string, unknown> => {
-		let current: unknown = root
-		for (const segment of path) {
-			if (!current || typeof current !== "object") {
-				return {}
-			}
-			current = (current as Record<string, unknown>)[segment]
-		}
-		return current && typeof current === "object" ? (current as Record<string, unknown>) : {}
-	}
-
-	const setObjectValueAtPath = (
-		root: Record<string, unknown>,
-		path: string[],
-		key: string,
-		value: unknown,
-	): Record<string, unknown> => {
-		if (path.length === 0) {
-			return { ...root, [key]: value }
-		}
-		const [head, ...rest] = path
-		const child = root[head]
-		const childObj = child && typeof child === "object" ? (child as Record<string, unknown>) : {}
-		return {
-			...root,
-			[head]: setObjectValueAtPath(childObj, rest, key, value),
-		}
-	}
-
-	const persistObjectEditor = (nextObject: Record<string, unknown>, source: "global" | "workspace", key: string) => {
-		if (source === "global" && onUpdateGlobal) {
-			onUpdateGlobal(key as GlobalStateAndSettingsKey, nextObject as never)
-		} else if (source === "workspace" && onUpdateWorkspace) {
-			onUpdateWorkspace(key as LocalStateKey, nextObject as never)
-		}
-	}
-
 	const handleSettingsReset = () => {
 		if (!selectedConfigEntry?.isEditable || selectedConfigEntry.source !== "global") {
 			return
@@ -297,22 +240,15 @@ export const ConfigView: React.FC<ConfigViewProps> = ({
 	// Input handling
 	useInput(
 		(input, key) => {
-			if (objectEditor) {
-				return
-			}
-
-			if (key.escape) {
+			if (input.toLowerCase() === "q" || key.escape) {
 				exit()
 			}
 
-			if (key.leftArrow || key.rightArrow || (input >= "1" && input <= "5")) {
-				const currentTabIndex = availableTabs.findIndex((t) => t.key === currentTab)
-				const targetIdx =
-					input >= "1" && input <= "5"
-						? Number.parseInt(input) - 1
-						: key.leftArrow
-							? (currentTabIndex - 1 + availableTabs.length) % availableTabs.length
-							: (currentTabIndex + 1) % availableTabs.length
+			// Tab navigation with Tab key or number keys
+			if (key.tab || (input >= "1" && input <= "5")) {
+				const targetIdx = key.tab
+					? (availableTabs.findIndex((t) => t.key === currentTab) + 1) % availableTabs.length
+					: parseInt(input) - 1
 				if (targetIdx >= 0 && targetIdx < availableTabs.length) {
 					handleTabChange(availableTabs[targetIdx].key)
 				}
@@ -320,45 +256,21 @@ export const ConfigView: React.FC<ConfigViewProps> = ({
 			}
 
 			// List navigation (arrow keys and vim-style j/k)
-			if (key.upArrow) {
+			if (key.upArrow || input === "k") {
 				setSelectedIndex((i) => (i > 0 ? i - 1 : currentListLength - 1))
-			} else if (key.downArrow) {
+			} else if (key.downArrow || input === "j") {
 				setSelectedIndex((i) => (i < currentListLength - 1 ? i + 1 : 0))
 			}
 
 			// Tab-specific actions
 			if (currentTab === "settings") {
-				if ((key.return || key.tab) && selectedConfigEntry?.isEditable) {
-					if (selectedConfigEntry.type === "boolean") {
-						handleSettingsSave(!selectedConfigEntry.value)
-						return
-					}
-					if (selectedConfigEntry.type === "object") {
-						const value =
-							selectedConfigEntry.value && typeof selectedConfigEntry.value === "object"
-								? (selectedConfigEntry.value as Record<string, unknown>)
-								: {}
-						setObjectEditor({
-							source: selectedConfigEntry.source,
-							key: selectedConfigEntry.key,
-							path: [],
-							value,
-							selectedIndex: 0,
-							isEditingValue: false,
-							editValue: "",
-						})
-						return
-					}
+				if ((key.return || input === "e") && selectedConfigEntry?.isEditable) {
 					setEditValue(selectedConfigEntry.value !== undefined ? String(selectedConfigEntry.value) : "")
 					setIsEditing(true)
-				} else if (key.ctrl && input.toLowerCase() === "r") {
+				} else if (input === "r") {
 					handleSettingsReset()
-				} else if (key.backspace || key.delete) {
-					setSearchQuery((prev) => prev.slice(0, -1))
-				} else if (input && !key.ctrl && !key.meta && !key.escape && !key.upArrow && !key.downArrow) {
-					setSearchQuery((prev) => prev + input)
 				}
-			} else if (key.return || key.tab || input === " ") {
+			} else if (key.return || input === " ") {
 				// Toggle for rules/workflows/hooks/skills
 				handleToggle()
 			}
@@ -426,31 +338,13 @@ export const ConfigView: React.FC<ConfigViewProps> = ({
 		)
 	}
 
-	if (objectEditor && currentTab === "settings") {
-		return (
-			<ObjectEditorPanel
-				getObjectAtPath={getObjectAtPath}
-				onClose={() => setObjectEditor(null)}
-				onPersist={(nextObject) => persistObjectEditor(nextObject, objectEditor.source, objectEditor.key)}
-				setObjectValueAtPath={setObjectValueAtPath}
-				setState={setObjectEditor}
-				state={objectEditor}
-			/>
-		)
-	}
-
 	// Render tab content
 	const renderTabContent = () => {
 		switch (currentTab) {
 			case "settings": {
-				const visibleEntries = filteredConfigEntries.slice(startIndex, startIndex + MAX_VISIBLE)
+				const visibleEntries = configEntries.slice(startIndex, startIndex + MAX_VISIBLE)
 				return (
 					<React.Fragment>
-						<Box>
-							<Text>Search: </Text>
-							<Text color="white">{searchQuery}</Text>
-							<Text inverse> </Text>
-						</Box>
 						<Box>
 							<Text>Data directory: </Text>
 							<Text color="blue" underline>
@@ -613,12 +507,12 @@ export const ConfigView: React.FC<ConfigViewProps> = ({
 
 	// Help text based on current tab
 	const getHelpText = () => {
-		const base = "↑/↓ Navigate • ←/→ tabs • 1-5 tabs • Esc Exit"
+		const base = "↑/↓/j/k Navigate • Tab/1-5 Switch tabs • q/Esc Exit"
 		if (currentTab === "settings") {
-			return `${base} • Type to search • Enter/Tab Edit (booleans toggle) • Backspace clear search • Ctrl+R Reset`
+			return `${base} • Enter/e Edit • r Reset`
 		}
 		const openFolder = onOpenFolder ? " • o Open folder" : ""
-		return `${base} • Enter/Tab/Space Toggle${openFolder}`
+		return `${base} • Enter/Space Toggle${openFolder}`
 	}
 
 	return (

cli/src/components/ConfigViewComponents.tsx

@@ -46,19 +46,16 @@ export interface SkillInfo {
 	enabled: boolean
 }
 
-export interface ObjectEditorState {
-	source: "global" | "workspace"
-	key: string
-	path: string[]
-	value: Record<string, unknown>
-	selectedIndex: number
-	isEditingValue: boolean
-	editValue: string
-}
+export const EXCLUDED_KEYS = new Set([
+	"taskHistory",
+	"primaryRootIndex",
+	"subagentsEnabled",
+	"subagentTerminalOutputLineLimit",
+	"welcomeViewCompleted",
+	"isNewUser",
+])
 
-export const EXCLUDED_KEYS = new Set(["taskHistory", "primaryRootIndex", "welcomeViewCompleted", "isNewUser"])
-
-export const EDITABLE_TYPES: Set<ValueType> = new Set(["string", "number", "boolean", "object"])
+export const EDITABLE_TYPES: Set<ValueType> = new Set(["string", "number", "boolean"])
 export const MAX_VISIBLE = 12
 export const SEPARATOR = "─".repeat(80)
 
@@ -138,7 +135,7 @@ export function parseValue(input: string, type: ValueType): unknown {
 		return input.toLowerCase() === "true" || input === "1"
 	}
 	if (type === "number") {
-		const num = Number.parseFloat(input)
+		const num = parseFloat(input)
 		return Number.isNaN(num) ? 0 : num
 	}
 	if (type === "object") {
@@ -220,7 +217,7 @@ export const TextInput: React.FC<TextInputProps> = ({ label, onChange, onCancel,
 			</Text>
 			<Box>
 				<Text color="white">{value}</Text>
-				<Text color="cyan">|</Text>
+				<Text inverse> </Text>
 			</Box>
 			<Text color="gray">Type: {type} • Enter to save • Esc to cancel</Text>
 		</Box>
@@ -382,169 +379,3 @@ export const SectionHeader: React.FC<{ title: string }> = ({ title }) => (
 		</Text>
 	</Box>
 )
-
-interface ObjectEditorPanelProps {
-	state: ObjectEditorState
-	setState: React.Dispatch<React.SetStateAction<ObjectEditorState | null>>
-	onClose: () => void
-	onPersist: (nextObject: Record<string, unknown>) => void
-	getObjectAtPath: (root: Record<string, unknown>, path: string[]) => Record<string, unknown>
-	setObjectValueAtPath: (root: Record<string, unknown>, path: string[], key: string, value: unknown) => Record<string, unknown>
-}
-
-export const ObjectEditorPanel: React.FC<ObjectEditorPanelProps> = ({
-	state,
-	setState,
-	onClose,
-	onPersist,
-	getObjectAtPath,
-	setObjectValueAtPath,
-}) => {
-	const { isRawModeSupported } = useStdinContext()
-	const currentNode = getObjectAtPath(state.value, state.path)
-	const objectEntries = Object.entries(currentNode).sort(([a], [b]) => a.localeCompare(b))
-	const selectedEntry = objectEntries[state.selectedIndex]
-	const breadcrumb = [state.key, ...state.path].join(" › ")
-
-	useInput(
-		(input, key) => {
-			if (state.isEditingValue) {
-				if (key.escape) {
-					setState((prev) => (prev ? { ...prev, isEditingValue: false, editValue: "" } : prev))
-					return
-				}
-				if (key.return) {
-					if (!selectedEntry) {
-						setState((prev) => (prev ? { ...prev, isEditingValue: false, editValue: "" } : prev))
-						return
-					}
-					const [entryKey, entryValue] = selectedEntry
-					let parsed: unknown = state.editValue
-					if (typeof entryValue === "boolean") {
-						parsed = state.editValue.toLowerCase() === "true" || state.editValue === "1"
-					} else if (typeof entryValue === "number") {
-						const maybeNum = Number(state.editValue)
-						parsed = Number.isNaN(maybeNum) ? 0 : maybeNum
-					}
-					const nextObject = setObjectValueAtPath(state.value, state.path, entryKey, parsed)
-					onPersist(nextObject)
-					setState((prev) => (prev ? { ...prev, value: nextObject, isEditingValue: false, editValue: "" } : prev))
-					return
-				}
-				if (key.backspace || key.delete) {
-					setState((prev) => (prev ? { ...prev, editValue: prev.editValue.slice(0, -1) } : prev))
-					return
-				}
-				if (input && !key.ctrl && !key.meta) {
-					setState((prev) => (prev ? { ...prev, editValue: prev.editValue + input } : prev))
-				}
-				return
-			}
-
-			if (key.escape) {
-				if (state.path.length > 0) {
-					setState((prev) => (prev ? { ...prev, path: prev.path.slice(0, -1), selectedIndex: 0 } : prev))
-				} else {
-					onClose()
-				}
-				return
-			}
-
-			if (key.upArrow || input === "k") {
-				setState((prev) =>
-					prev
-						? {
-								...prev,
-								selectedIndex:
-									objectEntries.length > 0
-										? prev.selectedIndex > 0
-											? prev.selectedIndex - 1
-											: objectEntries.length - 1
-										: 0,
-							}
-						: prev,
-				)
-				return
-			}
-			if (key.downArrow || input === "j") {
-				setState((prev) =>
-					prev
-						? {
-								...prev,
-								selectedIndex:
-									objectEntries.length > 0
-										? prev.selectedIndex < objectEntries.length - 1
-											? prev.selectedIndex + 1
-											: 0
-										: 0,
-							}
-						: prev,
-				)
-				return
-			}
-
-			if (key.return || key.tab) {
-				if (!selectedEntry) {
-					return
-				}
-				const [entryKey, entryValue] = selectedEntry
-				if (typeof entryValue === "boolean") {
-					const nextObject = setObjectValueAtPath(state.value, state.path, entryKey, !entryValue)
-					onPersist(nextObject)
-					setState((prev) => (prev ? { ...prev, value: nextObject } : prev))
-					return
-				}
-				if (entryValue && typeof entryValue === "object" && !Array.isArray(entryValue)) {
-					setState((prev) => (prev ? { ...prev, path: [...prev.path, entryKey], selectedIndex: 0 } : prev))
-					return
-				}
-				setState((prev) =>
-					prev
-						? { ...prev, isEditingValue: true, editValue: entryValue !== undefined ? String(entryValue) : "" }
-						: prev,
-				)
-			}
-		},
-		{ isActive: isRawModeSupported },
-	)
-
-	return (
-		<Box flexDirection="column">
-			<Text bold color="white">
-				⚙️ Edit Nested Object
-			</Text>
-			<Text color="gray">{SEPARATOR}</Text>
-			<Text color="cyan">{breadcrumb}</Text>
-			{state.isEditingValue ? (
-				<Box flexDirection="column" marginTop={1}>
-					<Box>
-						<Text color="white">{state.editValue}</Text>
-						<Text color="cyan">|</Text>
-					</Box>
-					<Text color="gray">Enter to save • Esc to cancel</Text>
-				</Box>
-			) : (
-				<Box flexDirection="column" marginTop={1}>
-					{objectEntries.length === 0 ? (
-						<Text color="gray">No nested keys at this level.</Text>
-					) : (
-						objectEntries.map(([key, value], idx) => {
-							const isSelected = idx === state.selectedIndex
-							const valueText =
-								value && typeof value === "object" && !Array.isArray(value) ? "{...}" : String(value)
-							return (
-								<Text color={isSelected ? "cyan" : undefined} key={key}>
-									{isSelected ? "❯ " : "  "}
-									<Text color="cyan">{key}</Text>
-									<Text color="gray">: </Text>
-									<Text color="white">{valueText}</Text>
-								</Text>
-							)
-						})
-					)}
-					<Text color="gray">↑/↓ Navigate • Enter/Tab Edit or drill in • Esc Back/Close</Text>
-				</Box>
-			)}
-		</Box>
-	)
-}

cli/src/components/HelpPanelContent.tsx

@@ -43,30 +43,6 @@ export const HelpPanelContent: React.FC<HelpPanelContentProps> = ({ onClose }) =
 					</Text>
 				</Box>
 
-				<Box flexDirection="column">
-					<Text bold>Keyboard Shortcuts</Text>
-					<Text>
-						{"  "}
-						<Text color="white">Ctrl+U</Text> - Clear entire input (delete to start)
-					</Text>
-					<Text>
-						{"  "}
-						<Text color="white">Ctrl+K</Text> - Delete from cursor to end
-					</Text>
-					<Text>
-						{"  "}
-						<Text color="white">Ctrl+W</Text> - Delete word backwards
-					</Text>
-					<Text>
-						{"  "}
-						<Text color="white">Ctrl+A / Ctrl+E</Text> - Jump to start / end of input
-					</Text>
-					<Text>
-						{"  "}
-						<Text color="white">Alt/Option+←/→</Text> - Move by word
-					</Text>
-				</Box>
-
 				<Box flexDirection="column">
 					<Text bold>Slash Commands</Text>
 					<Text>

cli/src/components/ModelPicker.tsx

@@ -6,7 +6,6 @@
 import { Box, Text } from "ink"
 import Spinner from "ink-spinner"
 import React, { useEffect, useMemo, useState } from "react"
-import { refreshOcaModels } from "@/core/controller/models/refreshOcaModels"
 import { refreshOpenRouterModels } from "@/core/controller/models/refreshOpenRouterModels"
 import {
 	type ApiProvider,
@@ -65,7 +64,6 @@ import {
 	xaiDefaultModelId,
 	xaiModels,
 } from "@/shared/api"
-import { StringRequest } from "@/shared/proto/cline/common"
 import { filterOpenRouterModelIds } from "@/shared/utils/model-filters"
 import { COLORS } from "../constants/colors"
 import { getOpenRouterDefaultModelId, usesOpenRouterModels } from "../utils/openrouter-models"
@@ -107,7 +105,7 @@ export function hasStaticModels(provider: string): boolean {
 }
 
 export function hasModelPicker(provider: string): boolean {
-	return hasStaticModels(provider) || usesOpenRouterModels(provider) || provider === "oca"
+	return hasStaticModels(provider) || usesOpenRouterModels(provider)
 }
 
 export function getDefaultModelId(provider: string): string {
@@ -134,7 +132,7 @@ export const ModelPicker: React.FC<ModelPickerProps> = ({ provider, controller,
 	const [isLoading, setIsLoading] = useState(false)
 	const [asyncModels, setAsyncModels] = useState<string[]>([])
 
-	// Fetch async models (OpenRouter or OCA) when needed
+	// Fetch OpenRouter models when needed using shared core function
 	useEffect(() => {
 		if (usesOpenRouterModels(provider)) {
 			setIsLoading(true)
@@ -147,23 +145,11 @@ export const ModelPicker: React.FC<ModelPickerProps> = ({ provider, controller,
 				.finally(() => {
 					setIsLoading(false)
 				})
-		} else if (provider === "oca") {
-			setIsLoading(true)
-			refreshOcaModels(controller, StringRequest.create({ value: "" }))
-				.then((result) => {
-					if (result.models) {
-						const modelIds = Object.keys(result.models).sort((a, b) => a.localeCompare(b))
-						setAsyncModels(modelIds)
-					}
-				})
-				.finally(() => {
-					setIsLoading(false)
-				})
 		}
 	}, [provider, controller])
 
 	const modelList = useMemo(() => {
-		if (usesOpenRouterModels(provider) || provider === "oca") {
+		if (usesOpenRouterModels(provider)) {
 			return asyncModels
 		}
 		return getModelList(provider)
@@ -194,7 +180,7 @@ export const ModelPicker: React.FC<ModelPickerProps> = ({ provider, controller,
 	}
 
 	// If async fetch returned no models, render nothing
-	if ((usesOpenRouterModels(provider) || provider === "oca") && modelList.length === 0) {
+	if (usesOpenRouterModels(provider) && modelList.length === 0) {
 		return null
 	}
 

cli/src/components/OcaEmployeeCheck.tsx

@@ -1,88 +0,0 @@
-/**
- * OCA (Oracle Cloud Assist) employee check component.
- * Shows a checkbox for "I'm an Oracle Employee" and a sign-in button.
- * Sets ocaMode in state before triggering the OAuth flow.
- */
-
-import { Box, Text, useInput } from "ink"
-// biome-ignore lint/style/useImportType: React is used as a value by JSX (jsx: "react" in tsconfig)
-import React, { useCallback, useState } from "react"
-import { StateManager } from "@/core/storage/StateManager"
-import { COLORS } from "../constants/colors"
-import { useStdinContext } from "../context/StdinContext"
-
-interface OcaEmployeeCheckProps {
-	/** Whether this component is active and should handle input */
-	isActive: boolean
-	/** Called when user confirms and wants to proceed with sign-in */
-	onSignIn: () => void
-	/** Called when user presses Escape to go back */
-	onCancel: () => void
-}
-
-export const OcaEmployeeCheck: React.FC<OcaEmployeeCheckProps> = ({ isActive, onSignIn, onCancel }) => {
-	const { isRawModeSupported } = useStdinContext()
-	const [isEmployee, setIsEmployee] = useState(true) // Default to checked (internal), matching extension behavior
-	const [selectedIndex, setSelectedIndex] = useState(0) // 0 = checkbox, 1 = sign in button
-
-	const ITEM_COUNT = 2
-
-	const handleSignIn = useCallback(async () => {
-		// Persist ocaMode to state before starting auth
-		const stateManager = StateManager.get()
-		stateManager.setGlobalState("ocaMode", isEmployee ? "internal" : "external")
-		await stateManager.flushPendingState()
-		onSignIn()
-	}, [isEmployee, onSignIn])
-
-	useInput(
-		(_input, key) => {
-			if (key.escape) {
-				onCancel()
-				return
-			}
-			if (key.upArrow) {
-				setSelectedIndex((prev) => (prev > 0 ? prev - 1 : ITEM_COUNT - 1))
-			} else if (key.downArrow) {
-				setSelectedIndex((prev) => (prev < ITEM_COUNT - 1 ? prev + 1 : 0))
-			} else if (key.tab || (key.return && selectedIndex === 0)) {
-				// Toggle checkbox when Tab is pressed or Enter on checkbox item
-				if (selectedIndex === 0) {
-					setIsEmployee((prev) => !prev)
-				}
-			} else if (key.return && selectedIndex === 1) {
-				// Sign in button
-				handleSignIn()
-			}
-		},
-		{ isActive: isRawModeSupported && isActive },
-	)
-
-	return (
-		<Box flexDirection="column">
-			<Text color="white">Oracle Code Assist</Text>
-			<Text> </Text>
-			{/* Checkbox: I'm an Oracle Employee */}
-			<Text>
-				<Text bold color={selectedIndex === 0 ? COLORS.primaryBlue : undefined}>
-					{selectedIndex === 0 ? "❯" : " "}{" "}
-				</Text>
-				<Text color={selectedIndex === 0 || isEmployee ? COLORS.primaryBlue : "gray"}>{isEmployee ? "[✓]" : "[ ]"}</Text>
-				<Text color={selectedIndex === 0 ? COLORS.primaryBlue : "white"}> I'm an Oracle Employee</Text>
-				{selectedIndex === 0 && <Text color="gray"> (Tab to toggle)</Text>}
-			</Text>
-			{/* Sign in button */}
-			<Text>
-				<Text bold color={selectedIndex === 1 ? COLORS.primaryBlue : undefined}>
-					{selectedIndex === 1 ? "❯" : " "}{" "}
-				</Text>
-				<Text color={selectedIndex === 1 ? COLORS.primaryBlue : "white"}>Sign in with Oracle Code Assist</Text>
-				{selectedIndex === 1 && <Text color="gray"> (Enter)</Text>}
-			</Text>
-			<Text> </Text>
-			<Text color="gray">Please ask your IT administrator to set up Oracle Code Assist as a model provider.</Text>
-			<Text> </Text>
-			<Text color="gray">Arrows to navigate, Tab to toggle, Enter to continue, Esc to go back</Text>
-		</Box>
-	)
-}

cli/src/components/SettingsPanelContent.tsx

@@ -7,21 +7,17 @@ import type { AutoApprovalSettings } from "@shared/AutoApprovalSettings"
 import { DEFAULT_AUTO_APPROVAL_SETTINGS } from "@shared/AutoApprovalSettings"
 import type { ApiProvider, ModelInfo } from "@shared/api"
 import { getProviderModelIdKey, isSettingsKey, ProviderToApiKeyMap } from "@shared/storage"
-import { isOpenaiReasoningEffort, OPENAI_REASONING_EFFORT_OPTIONS, type OpenaiReasoningEffort } from "@shared/storage/types"
 import type { TelemetrySetting } from "@shared/TelemetrySetting"
 import { Box, Text, useInput } from "ink"
 import Spinner from "ink-spinner"
 import React, { useCallback, useEffect, useMemo, useState } from "react"
 import { buildApiHandler } from "@/core/api"
 import type { Controller } from "@/core/controller"
-import { refreshOcaModels } from "@/core/controller/models/refreshOcaModels"
 import { StateManager } from "@/core/storage/StateManager"
 import { openAiCodexOAuthManager } from "@/integrations/openai-codex/oauth"
 import { ClineAccountService } from "@/services/account/ClineAccountService"
 import { AuthService, ClineAccountOrganization } from "@/services/auth/AuthService"
-import { StringRequest } from "@/shared/proto/cline/common"
 import { openExternal } from "@/utils/env"
-import { supportsReasoningEffortForModel } from "@/utils/model-utils"
 import { version as CLI_VERSION } from "../../package.json"
 import { COLORS } from "../constants/colors"
 import { useStdinContext } from "../context/StdinContext"
@@ -39,7 +35,6 @@ import {
 } from "./FeaturedModelPicker"
 import { LanguagePicker } from "./LanguagePicker"
 import { hasModelPicker, ModelPicker } from "./ModelPicker"
-import { OcaEmployeeCheck } from "./OcaEmployeeCheck"
 import { OrganizationPicker } from "./OrganizationPicker"
 import { Panel, PanelTab } from "./Panel"
 import { getProviderLabel, ProviderPicker } from "./ProviderPicker"
@@ -56,25 +51,13 @@ type SettingsTab = "api" | "auto-approve" | "features" | "other" | "account"
 interface ListItem {
 	key: string
 	label: string
-	type: "checkbox" | "readonly" | "editable" | "separator" | "header" | "spacer" | "action" | "cycle"
+	type: "checkbox" | "readonly" | "editable" | "separator" | "header" | "spacer" | "action"
 	value: string | boolean
 	description?: string
 	isSubItem?: boolean
 	parentKey?: string
 }
 
-function normalizeReasoningEffort(value: unknown): OpenaiReasoningEffort {
-	if (isOpenaiReasoningEffort(value)) {
-		return value
-	}
-	return "low"
-}
-
-function nextReasoningEffort(current: OpenaiReasoningEffort): OpenaiReasoningEffort {
-	const idx = OPENAI_REASONING_EFFORT_OPTIONS.indexOf(current)
-	return OPENAI_REASONING_EFFORT_OPTIONS[(idx + 1) % OPENAI_REASONING_EFFORT_OPTIONS.length]
-}
-
 const TABS: PanelTab[] = [
 	{ key: "api", label: "API" },
 	{ key: "auto-approve", label: "Auto-approve" },
@@ -85,12 +68,6 @@ const TABS: PanelTab[] = [
 
 // Settings configuration for simple boolean toggles
 const FEATURE_SETTINGS = {
-	subagents: {
-		stateKey: "subagentsEnabled",
-		default: false,
-		label: "Subagents",
-		description: "Let Cline run focused subagents in parallel to explore the codebase for you",
-	},
 	autoCondense: {
 		stateKey: "useAutoCondense",
 		default: false,
@@ -121,12 +98,6 @@ const FEATURE_SETTINGS = {
 		label: "Parallel tool calling",
 		description: "Allow multiple tools in a single response",
 	},
-	doubleCheckCompletion: {
-		stateKey: "doubleCheckCompletionEnabled",
-		default: false,
-		label: "Double-check completion",
-		description: "Reject first completion attempt and require re-verification",
-	},
 } as const
 
 type FeatureKey = keyof typeof FEATURE_SETTINGS
@@ -165,7 +136,6 @@ export const SettingsPanelContent: React.FC<SettingsPanelContentProps> = ({
 	const [isEnteringApiKey, setIsEnteringApiKey] = useState(false)
 	const [isConfiguringBedrock, setIsConfiguringBedrock] = useState(false)
 	const [isWaitingForCodexAuth, setIsWaitingForCodexAuth] = useState(false)
-	const [isShowingOcaEmployeeCheck, setIsShowingOcaEmployeeCheck] = useState(false)
 	const [codexAuthError, setCodexAuthError] = useState<string | null>(null)
 	const [pendingProvider, setPendingProvider] = useState<string | null>(null)
 	const [apiKeyValue, setApiKeyValue] = useState("")
@@ -195,12 +165,6 @@ export const SettingsPanelContent: React.FC<SettingsPanelContentProps> = ({
 	const [planThinkingEnabled, setPlanThinkingEnabled] = useState<boolean>(
 		() => (stateManager.getGlobalSettingsKey("planModeThinkingBudgetTokens") ?? 0) > 0,
 	)
-	const [actReasoningEffort, setActReasoningEffort] = useState<OpenaiReasoningEffort>(() =>
-		normalizeReasoningEffort(stateManager.getGlobalSettingsKey("actModeReasoningEffort")),
-	)
-	const [planReasoningEffort, setPlanReasoningEffort] = useState<OpenaiReasoningEffort>(() =>
-		normalizeReasoningEffort(stateManager.getGlobalSettingsKey("planModeReasoningEffort")),
-	)
 
 	// Auto-approve settings (complex nested object)
 	const [autoApproveSettings, setAutoApproveSettings] = useState<AutoApprovalSettings>(() => {
@@ -239,8 +203,6 @@ export const SettingsPanelContent: React.FC<SettingsPanelContentProps> = ({
 	// OCA auth hook
 	const handleOcaAuthSuccess = useCallback(async () => {
 		await applyProviderConfig({ providerId: "oca", controller })
-		// Fetch OCA models from the API - this sets actModeOcaModelId/planModeOcaModelId in state
-		await refreshOcaModels(controller!, StringRequest.create({ value: "" }))
 		setProvider("oca")
 		refreshModelIds()
 	}, [controller, refreshModelIds])
@@ -433,12 +395,9 @@ export const SettingsPanelContent: React.FC<SettingsPanelContentProps> = ({
 
 	// Build items list based on current tab
 	const items: ListItem[] = useMemo(() => {
-		// Some providers/models expose reasoning effort instead of thinking budget controls.
-		const providerUsesReasoningEffort = provider === "openai-native" || provider === "openai-codex"
-		const showActReasoningEffort = supportsReasoningEffortForModel(actModelId || "")
-		const showPlanReasoningEffort = supportsReasoningEffortForModel(planModelId || "")
-		const showActThinkingOption = !providerUsesReasoningEffort && !showActReasoningEffort
-		const showPlanThinkingOption = !providerUsesReasoningEffort && !showPlanReasoningEffort
+		// OpenAI Native, Codex, and GPT models don't support thinking budget (they use reasoning effort)
+		const isGptModel = actModelId?.toLowerCase().includes("gpt") || planModelId?.toLowerCase().includes("gpt")
+		const showThinkingOption = provider !== "openai-native" && provider !== "openai-codex" && !isGptModel
 
 		switch (currentTab) {
 			case "api":
@@ -462,7 +421,7 @@ export const SettingsPanelContent: React.FC<SettingsPanelContentProps> = ({
 									type: "editable" as const,
 									value: actModelId || "not set",
 								},
-								...(showActThinkingOption
+								...(showThinkingOption
 									? [
 											{
 												key: "actThinkingEnabled",
@@ -472,16 +431,6 @@ export const SettingsPanelContent: React.FC<SettingsPanelContentProps> = ({
 											},
 										]
 									: []),
-								...(showActReasoningEffort
-									? [
-											{
-												key: "actReasoningEffort",
-												label: "Reasoning effort",
-												type: "cycle" as const,
-												value: actReasoningEffort,
-											},
-										]
-									: []),
 								{ key: "planHeader", label: "Plan Mode", type: "header" as const, value: "" },
 								{
 									key: "planModelId",
@@ -489,7 +438,7 @@ export const SettingsPanelContent: React.FC<SettingsPanelContentProps> = ({
 									type: "editable" as const,
 									value: planModelId || "not set",
 								},
-								...(showPlanThinkingOption
+								...(showThinkingOption
 									? [
 											{
 												key: "planThinkingEnabled",
@@ -499,16 +448,6 @@ export const SettingsPanelContent: React.FC<SettingsPanelContentProps> = ({
 											},
 										]
 									: []),
-								...(showPlanReasoningEffort
-									? [
-											{
-												key: "planReasoningEffort",
-												label: "Reasoning effort",
-												type: "cycle" as const,
-												value: planReasoningEffort,
-											},
-										]
-									: []),
 								{ key: "spacer1", label: "", type: "spacer" as const, value: "" },
 							]
 						: [
@@ -518,7 +457,7 @@ export const SettingsPanelContent: React.FC<SettingsPanelContentProps> = ({
 									type: "editable" as const,
 									value: actModelId || "not set",
 								},
-								...(showActThinkingOption
+								...(showThinkingOption
 									? [
 											{
 												key: "actThinkingEnabled",
@@ -528,16 +467,6 @@ export const SettingsPanelContent: React.FC<SettingsPanelContentProps> = ({
 											},
 										]
 									: []),
-								...(showActReasoningEffort
-									? [
-											{
-												key: "actReasoningEffort",
-												label: "Reasoning effort",
-												type: "cycle" as const,
-												value: actReasoningEffort,
-											},
-										]
-									: []),
 							]),
 					{
 						key: "separateModels",
@@ -700,8 +629,6 @@ export const SettingsPanelContent: React.FC<SettingsPanelContentProps> = ({
 		separateModels,
 		actThinkingEnabled,
 		planThinkingEnabled,
-		actReasoningEffort,
-		planReasoningEffort,
 		autoApproveSettings,
 		features,
 		preferredLanguage,
@@ -735,33 +662,6 @@ export const SettingsPanelContent: React.FC<SettingsPanelContentProps> = ({
 		}
 	}, [items.length, selectedIndex])
 
-	const rebuildTaskApi = useCallback(() => {
-		if (!controller?.task) {
-			return
-		}
-		const currentMode = stateManager.getGlobalSettingsKey("mode")
-		const apiConfig = stateManager.getApiConfiguration()
-		controller.task.api = buildApiHandler({ ...apiConfig, ulid: controller.task.ulid }, currentMode)
-	}, [controller, stateManager])
-
-	const setReasoningEffortForMode = useCallback(
-		(mode: "act" | "plan", effort: OpenaiReasoningEffort) => {
-			if (mode === "act") {
-				setActReasoningEffort(effort)
-				stateManager.setGlobalState("actModeReasoningEffort", effort)
-				if (!separateModels) {
-					setPlanReasoningEffort(effort)
-					stateManager.setGlobalState("planModeReasoningEffort", effort)
-				}
-			} else {
-				setPlanReasoningEffort(effort)
-				stateManager.setGlobalState("planModeReasoningEffort", effort)
-			}
-			rebuildTaskApi()
-		},
-		[separateModels, rebuildTaskApi, stateManager],
-	)
-
 	// Handle toggle/edit for selected item
 	const handleAction = useCallback(() => {
 		const item = items[selectedIndex]
@@ -785,15 +685,6 @@ export const SettingsPanelContent: React.FC<SettingsPanelContentProps> = ({
 			return
 		}
 
-		if (item.type === "cycle") {
-			const targetMode = item.key === "actReasoningEffort" ? "act" : item.key === "planReasoningEffort" ? "plan" : undefined
-			if (targetMode) {
-				const currentEffort = targetMode === "act" ? actReasoningEffort : planReasoningEffort
-				setReasoningEffortForMode(targetMode, nextReasoningEffort(currentEffort))
-			}
-			return
-		}
-
 		if (item.type === "editable") {
 			// For provider field, use the provider picker
 			if (item.key === "provider") {
@@ -851,16 +742,7 @@ export const SettingsPanelContent: React.FC<SettingsPanelContentProps> = ({
 					const actModel = stateManager.getGlobalSettingsKey(actKey)
 					if (planKey) stateManager.setGlobalState(planKey, actModel)
 				}
-				const actThinkingBudget = stateManager.getGlobalSettingsKey("actModeThinkingBudgetTokens") ?? 0
-				stateManager.setGlobalState("planModeThinkingBudgetTokens", actThinkingBudget)
-				setPlanThinkingEnabled(actThinkingBudget > 0)
-
-				const actEffort = normalizeReasoningEffort(stateManager.getGlobalSettingsKey("actModeReasoningEffort"))
-				stateManager.setGlobalState("planModeReasoningEffort", actEffort)
-				setPlanReasoningEffort(actEffort)
 			}
-
-			rebuildTaskApi()
 			return
 		}
 
@@ -868,19 +750,23 @@ export const SettingsPanelContent: React.FC<SettingsPanelContentProps> = ({
 		if (item.key === "actThinkingEnabled") {
 			setActThinkingEnabled(newValue)
 			stateManager.setGlobalState("actModeThinkingBudgetTokens", newValue ? 1024 : 0)
-			if (!separateModels) {
-				setPlanThinkingEnabled(newValue)
-				stateManager.setGlobalState("planModeThinkingBudgetTokens", newValue ? 1024 : 0)
-			}
 			// Rebuild API handler to apply thinking budget change
-			rebuildTaskApi()
+			if (controller?.task) {
+				const currentMode = stateManager.getGlobalSettingsKey("mode")
+				const apiConfig = stateManager.getApiConfiguration()
+				controller.task.api = buildApiHandler({ ...apiConfig, ulid: controller.task.ulid }, currentMode)
+			}
 			return
 		}
 		if (item.key === "planThinkingEnabled") {
 			setPlanThinkingEnabled(newValue)
 			stateManager.setGlobalState("planModeThinkingBudgetTokens", newValue ? 1024 : 0)
 			// Rebuild API handler to apply thinking budget change
-			rebuildTaskApi()
+			if (controller?.task) {
+				const currentMode = stateManager.getGlobalSettingsKey("mode")
+				const apiConfig = stateManager.getApiConfiguration()
+				controller.task.api = buildApiHandler({ ...apiConfig, ulid: controller.task.ulid }, currentMode)
+			}
 			return
 		}
 
@@ -937,11 +823,6 @@ export const SettingsPanelContent: React.FC<SettingsPanelContentProps> = ({
 		handleClineLogin,
 		handleClineLogout,
 		accountOrganizations,
-		separateModels,
-		actReasoningEffort,
-		planReasoningEffort,
-		rebuildTaskApi,
-		setReasoningEffortForMode,
 	])
 
 	// Handle model selection from picker
@@ -1084,8 +965,8 @@ export const SettingsPanelContent: React.FC<SettingsPanelContentProps> = ({
 					setProvider("oca")
 					refreshModelIds()
 				} else {
-					// Not logged in - show employee check before auth
-					setIsShowingOcaEmployeeCheck(true)
+					// Not logged in - trigger OAuth
+					startOcaAuth()
 				}
 				return
 			}
@@ -1376,7 +1257,7 @@ export const SettingsPanelContent: React.FC<SettingsPanelContentProps> = ({
 				return
 			}
 		},
-		{ isActive: isRawModeSupported && !isEnteringApiKey && !isConfiguringBedrock && !isShowingOcaEmployeeCheck },
+		{ isActive: isRawModeSupported && !isEnteringApiKey && !isConfiguringBedrock },
 	)
 
 	// Render content
@@ -1552,19 +1433,6 @@ export const SettingsPanelContent: React.FC<SettingsPanelContentProps> = ({
 			)
 		}
 
-		if (isShowingOcaEmployeeCheck) {
-			return (
-				<OcaEmployeeCheck
-					isActive={isShowingOcaEmployeeCheck}
-					onCancel={() => setIsShowingOcaEmployeeCheck(false)}
-					onSignIn={() => {
-						setIsShowingOcaEmployeeCheck(false)
-						startOcaAuth()
-					}}
-				/>
-			)
-		}
-
 		if (isWaitingForOcaAuth) {
 			return (
 				<Box flexDirection="column">
@@ -1701,21 +1569,6 @@ export const SettingsPanelContent: React.FC<SettingsPanelContentProps> = ({
 						)
 					}
 
-					if (item.type === "cycle") {
-						return (
-							<Text key={item.key}>
-								<Text bold color={isSelected ? COLORS.primaryBlue : undefined}>
-									{isSelected ? "❯" : " "}{" "}
-								</Text>
-								<Text color={isSelected ? COLORS.primaryBlue : "white"}>{item.label}: </Text>
-								<Text color={COLORS.primaryBlue}>
-									{typeof item.value === "string" ? item.value : String(item.value)}
-								</Text>
-								{isSelected && <Text color="gray"> (Tab to cycle)</Text>}
-							</Text>
-						)
-					}
-
 					// Readonly or editable field
 					return (
 						<Text key={item.key}>
@@ -1746,7 +1599,6 @@ export const SettingsPanelContent: React.FC<SettingsPanelContentProps> = ({
 		!!codexAuthError ||
 		isPickingOrganization ||
 		isWaitingForClineAuth ||
-		isShowingOcaEmployeeCheck ||
 		isWaitingForOcaAuth ||
 		isEditing
 

cli/src/components/SubagentMessage.tsx

@@ -1,361 +0,0 @@
-import type { ClineAskUseSubagents, ClineMessage, ClineSaySubagentStatus } from "@shared/ExtensionMessage"
-import { Box, Text } from "ink"
-import Spinner from "ink-spinner"
-import React from "react"
-import { COLORS } from "../constants/colors"
-import { useTerminalSize } from "../hooks/useTerminalSize"
-import { jsonParseSafe } from "../utils/parser"
-
-interface SubagentMessageProps {
-	message: ClineMessage
-	isStreaming?: boolean
-	mode?: "act" | "plan"
-}
-
-const TREE_PREFIX_WIDTH = 5
-const MIN_PROMPT_WIDTH = 20
-
-const DotRow: React.FC<{ children: React.ReactNode; color?: string; flashing?: boolean }> = ({
-	children,
-	color,
-	flashing = false,
-}) => (
-	<Box flexDirection="row">
-		<Box width={2}>
-			{flashing ? (
-				<Text color={color}>
-					<Spinner type="toggle8" />
-				</Text>
-			) : (
-				<Text color={color}>⏺</Text>
-			)}
-		</Box>
-		<Box flexGrow={1}>{children}</Box>
-	</Box>
-)
-
-function formatCompactTokens(tokens: number | undefined): string {
-	const value = Number.isFinite(tokens) ? Math.max(0, tokens || 0) : 0
-	return new Intl.NumberFormat("en-US", {
-		notation: "compact",
-		maximumFractionDigits: 1,
-	})
-		.format(value)
-		.toLowerCase()
-}
-
-function formatCompactCost(cost: number | undefined): string {
-	const value = Number.isFinite(cost) ? Math.max(0, cost || 0) : 0
-	const maximumFractionDigits = value >= 0.01 ? 2 : 4
-	return new Intl.NumberFormat("en-US", {
-		style: "currency",
-		currency: "USD",
-		minimumFractionDigits: 2,
-		maximumFractionDigits,
-	}).format(value)
-}
-
-function formatSubagentStatsValues(
-	toolCalls: number | undefined,
-	contextTokens: number | undefined,
-	totalCost: number | undefined,
-	latestToolCall?: string,
-) {
-	const safeToolCalls = Number.isFinite(toolCalls) ? Math.max(0, toolCalls || 0) : 0
-	const toolUses = safeToolCalls === 1 ? "tool use" : "tool uses"
-	const tokensUsed = formatCompactTokens(contextTokens || 0)
-	const formattedCost = formatCompactCost(totalCost || 0)
-	const stats = `${safeToolCalls} ${toolUses} · ${tokensUsed} tokens · ${formattedCost}`
-	const latestTool = latestToolCall?.trim()
-	return latestTool ? `${latestTool} · ${stats}` : stats
-}
-
-function wrapPrompt(text: string, width: number): string[] {
-	if (!text) {
-		return [""]
-	}
-
-	const normalizedWidth = Math.max(1, width)
-	const wrappedLines: string[] = []
-	const paragraphs = text.split("\n")
-
-	for (const paragraph of paragraphs) {
-		const words = paragraph.trim().split(/\s+/).filter(Boolean)
-		if (words.length === 0) {
-			wrappedLines.push("")
-			continue
-		}
-
-		let line = ""
-		for (const word of words) {
-			if (!line) {
-				if (word.length <= normalizedWidth) {
-					line = word
-					continue
-				}
-
-				let remaining = word
-				while (remaining.length > normalizedWidth) {
-					wrappedLines.push(remaining.slice(0, normalizedWidth))
-					remaining = remaining.slice(normalizedWidth)
-				}
-				line = remaining
-				continue
-			}
-
-			if (line.length + 1 + word.length <= normalizedWidth) {
-				line = `${line} ${word}`
-				continue
-			}
-
-			wrappedLines.push(line)
-
-			if (word.length <= normalizedWidth) {
-				line = word
-				continue
-			}
-
-			let remaining = word
-			while (remaining.length > normalizedWidth) {
-				wrappedLines.push(remaining.slice(0, normalizedWidth))
-				remaining = remaining.slice(normalizedWidth)
-			}
-			line = remaining
-		}
-
-		if (line) {
-			wrappedLines.push(line)
-		}
-	}
-
-	return wrappedLines.length > 0 ? wrappedLines : [text]
-}
-
-const TreePromptRow: React.FC<{
-	prefix: React.ReactNode
-	continuationPrefix: string
-	prompt: string
-	promptWidth: number
-	color?: string
-}> = ({ prefix, continuationPrefix, prompt, promptWidth, color }) => {
-	const lines = wrapPrompt(prompt, promptWidth)
-
-	return (
-		<Box flexDirection="column" width="100%">
-			{lines.map((line, index) => (
-				<Box flexDirection="row" key={`${line}-${index}`} width="100%">
-					<Box flexShrink={0} width={TREE_PREFIX_WIDTH}>
-						{index === 0 ? prefix : <Text color="gray">{continuationPrefix}</Text>}
-					</Box>
-					<Box flexGrow={1}>
-						<Text color={color}>{line}</Text>
-					</Box>
-				</Box>
-			))}
-		</Box>
-	)
-}
-
-const TreeStatsRow: React.FC<{ prefix: string; stats: string }> = ({ prefix, stats }) => (
-	<Box flexDirection="row" width="100%">
-		<Box flexShrink={0} width={TREE_PREFIX_WIDTH}>
-			<Text color="gray">{prefix}</Text>
-		</Box>
-		<Box flexGrow={1}>
-			<Text color="gray">⎿ {stats}</Text>
-		</Box>
-	</Box>
-)
-
-export const SubagentMessage: React.FC<SubagentMessageProps> = ({ message, mode, isStreaming }) => {
-	const { type, ask, say, text, partial } = message
-	const toolColor = mode === "plan" ? "yellow" : COLORS.primaryBlue
-	const { columns } = useTerminalSize()
-	const promptWidth = Math.max(MIN_PROMPT_WIDTH, columns - 2 - TREE_PREFIX_WIDTH)
-
-	if ((type === "ask" && ask === "use_subagents") || say === "use_subagents") {
-		const parsed = text
-			? jsonParseSafe<ClineAskUseSubagents>(text, {
-					prompts: [],
-				})
-			: { prompts: [] }
-
-		const prompts = (parsed.prompts || []).map((prompt) => prompt?.trim()).filter(Boolean)
-		if (prompts.length === 0) {
-			return (
-				<Box flexDirection="column" marginBottom={1} width="100%">
-					<DotRow color={toolColor}>
-						<Text color={toolColor}>Cline wants to run subagents:</Text>
-					</DotRow>
-				</Box>
-			)
-		}
-
-		const singular = prompts.length === 1
-		return (
-			<Box flexDirection="column" marginBottom={1} width="100%">
-				<DotRow color={toolColor} flashing={partial === true && isStreaming}>
-					<Text color={toolColor}>{singular ? "Cline wants to run a subagent:" : "Cline wants to run subagents:"}</Text>
-				</DotRow>
-				<Box flexDirection="column" marginLeft={2} width="100%">
-					{prompts.map((prompt, index) => {
-						const isLastPrompt = index === prompts.length - 1
-						const branch = isLastPrompt ? "└─" : "├─"
-						const continuationPrefix = isLastPrompt ? "     " : "│    "
-						const shouldShowPromptStats = partial !== true || !isLastPrompt
-						return (
-							<Box flexDirection="column" key={`${prompt}-${index}`}>
-								<TreePromptRow
-									color={toolColor}
-									continuationPrefix={continuationPrefix}
-									prefix={<Text color={toolColor}>{`${branch} `}</Text>}
-									prompt={prompt}
-									promptWidth={promptWidth}
-								/>
-								{shouldShowPromptStats && (
-									<TreeStatsRow
-										prefix={continuationPrefix}
-										stats={formatSubagentStatsValues(undefined, undefined, undefined)}
-									/>
-								)}
-							</Box>
-						)
-					})}
-				</Box>
-			</Box>
-		)
-	}
-
-	if (say === "subagent" && text) {
-		const parsed = jsonParseSafe<ClineSaySubagentStatus>(text, {
-			status: "running",
-			total: 0,
-			completed: 0,
-			successes: 0,
-			failures: 0,
-			toolCalls: 0,
-			inputTokens: 0,
-			outputTokens: 0,
-			contextWindow: 0,
-			maxContextTokens: 0,
-			maxContextUsagePercentage: 0,
-			items: [],
-		})
-
-		const items = parsed.items || []
-		if (items.length === 0) {
-			return null
-		}
-
-		return (
-			<Box flexDirection="column" marginBottom={1} width="100%">
-				<DotRow color={toolColor} flashing={partial === true && isStreaming}>
-					<Text color={toolColor}>
-						{items.length === 1 ? "Cline is running a subagent:" : "Cline is running subagents:"}
-					</Text>
-				</DotRow>
-				<Box flexDirection="column" marginLeft={2} width="100%">
-					{items.map((entry, index) => {
-						const isLastEntry = index === items.length - 1
-						const branch = isLastEntry ? "└─" : "├─"
-						const continuationPrefix = isLastEntry ? "     " : "│    "
-						const key = `${entry.index}-${index}`
-						const shouldShowStats = true
-
-						if (entry.status === "completed") {
-							return (
-								<Box flexDirection="column" key={key}>
-									<TreePromptRow
-										color="green"
-										continuationPrefix={continuationPrefix}
-										prefix={
-											<Box flexDirection="row">
-												<Text color="gray">{`${branch} `}</Text>
-												<Text color="green">✓</Text>
-											</Box>
-										}
-										prompt={entry.prompt}
-										promptWidth={promptWidth}
-									/>
-									<TreeStatsRow
-										prefix={continuationPrefix}
-										stats={formatSubagentStatsValues(
-											entry.toolCalls,
-											entry.contextTokens,
-											entry.totalCost,
-											entry.latestToolCall,
-										)}
-									/>
-								</Box>
-							)
-						}
-
-						if (entry.status === "failed") {
-							return (
-								<Box flexDirection="column" key={key}>
-									<TreePromptRow
-										color="red"
-										continuationPrefix={continuationPrefix}
-										prefix={
-											<Box flexDirection="row">
-												<Text color="gray">{`${branch} `}</Text>
-												<Text color="red">✗</Text>
-											</Box>
-										}
-										prompt={entry.prompt}
-										promptWidth={promptWidth}
-									/>
-									<TreeStatsRow
-										prefix={continuationPrefix}
-										stats={formatSubagentStatsValues(
-											entry.toolCalls,
-											entry.contextTokens,
-											entry.totalCost,
-											entry.latestToolCall,
-										)}
-									/>
-								</Box>
-							)
-						}
-
-						return (
-							<Box flexDirection="column" key={key}>
-								<TreePromptRow
-									color={toolColor}
-									continuationPrefix={continuationPrefix}
-									prefix={
-										<Box flexDirection="row">
-											<Text color="gray">{branch} </Text>
-											{entry.status === "running" ? (
-												<Text color={toolColor}>
-													<Spinner type="dots" />
-												</Text>
-											) : (
-												<Text color={toolColor}>•</Text>
-											)}
-										</Box>
-									}
-									prompt={entry.prompt}
-									promptWidth={promptWidth}
-								/>
-								{shouldShowStats && (
-									<TreeStatsRow
-										prefix={continuationPrefix}
-										stats={formatSubagentStatsValues(
-											entry.toolCalls,
-											entry.contextTokens,
-											entry.totalCost,
-											entry.latestToolCall,
-										)}
-									/>
-								)}
-							</Box>
-						)
-					})}
-				</Box>
-			</Box>
-		)
-	}
-
-	return null
-}

cli/src/constants/featured-models.test.ts

@@ -1,12 +0,0 @@
-import { describe, expect, it } from "vitest"
-import { getAllFeaturedModels } from "./featured-models"
-
-describe("featured models", () => {
-	it("includes display names for all featured models", () => {
-		const models = getAllFeaturedModels()
-
-		for (const model of models) {
-			expect(model.name).toBeTruthy()
-		}
-	})
-})

cli/src/constants/featured-models.ts

@@ -10,7 +10,7 @@ export interface FeaturedModel {
 	labels: string[]
 }
 
-export const FEATURED_MODELS: { recommended: FeaturedModel[]; free: FeaturedModel[] } = {
+export const FEATURED_MODELS = {
 	recommended: [
 		{
 			id: "anthropic/claude-opus-4.6",
@@ -30,27 +30,33 @@ export const FEATURED_MODELS: { recommended: FeaturedModel[]; free: FeaturedMode
 			description: "1M context window for large codebases",
 			labels: ["TRENDING"],
 		},
-	],
+	] as FeaturedModel[],
 	free: [
 		{
-			id: "minimax/minimax-m2.5",
-			name: "MiniMax M2.5",
-			description: "MiniMax-M2.5 is a lightweight, state-of-the-art LLM optimized for coding and agentic workflows",
+			id: "minimax/minimax-m2.1",
+			name: "MiniMax M2.1",
+			description: "Exceptional Multi-Programming Language Capabilities",
+			labels: ["FREE"],
+		},
+		{
+			id: "moonshotai/kimi-k2.5",
+			name: "Kimi K2.5",
+			description: "State-of-the-art model topping benchmarks",
 			labels: ["FREE"],
 		},
 		{
 			id: "kwaipilot/kat-coder-pro",
 			name: "KAT Coder Pro",
-			description: "KwaiKAT's most advanced agentic coding model in the KAT-Coder series",
+			description: "Advanced agentic coding model",
 			labels: ["FREE"],
 		},
 		{
 			id: "arcee-ai/trinity-large-preview:free",
 			name: "Trinity Large Preview",
-			description: "Arcee AI's advanced large preview model in the Trinity series",
+			description: "US built open source coding model",
 			labels: ["FREE"],
 		},
-	],
+	] as FeaturedModel[],
 }
 
 export function getAllFeaturedModels(): FeaturedModel[] {

cli/src/hooks/useTextInput.ts

@@ -6,7 +6,6 @@
  * - Ctrl+A/E: start/end of line
  * - Ctrl+W: delete word backwards
  * - Ctrl+U: delete to start of line
- * - Ctrl+K: delete to end of line
  *
  * Note: Home/End keys are handled by useHomeEndKeys hook because Ink doesn't
  * expose them in useInput (it sets input='' for these keys).
@@ -153,14 +152,6 @@ export function useTextInput(): UseTextInputReturn {
 		}
 	}, [])
 
-	const deleteToEnd = useCallback(() => {
-		const pos = cursorRef.current
-		if (pos < textRef.current.length) {
-			setTextState((prev) => prev.slice(0, pos))
-			// Cursor stays at same position (now at end of text)
-		}
-	}, [])
-
 	// Cursor movement (internal, used by handlers)
 	const moveToStart = useCallback(() => setCursorPosState(0), [])
 	const moveToEnd = useCallback(() => setCursorPosState(textRef.current.length), [])
@@ -199,9 +190,6 @@ export function useTextInput(): UseTextInputReturn {
 				case "u": // Ctrl+U - delete to start
 					deleteToStart()
 					return true
-				case "k": // Ctrl+K - delete to end
-					deleteToEnd()
-					return true
 				case "w": // Ctrl+W - delete word backwards
 					deleteWordBefore()
 					return true
@@ -209,7 +197,7 @@ export function useTextInput(): UseTextInputReturn {
 					return false
 			}
 		},
-		[moveToStart, moveToEnd, deleteToStart, deleteToEnd, deleteWordBefore],
+		[moveToStart, moveToEnd, deleteToStart, deleteWordBefore],
 	)
 
 	return {

cli/src/index.test.ts

@@ -30,9 +30,7 @@ describe("CLI Commands", () => {
 			.option("-v, --verbose", "Show verbose output")
 			.option("-c, --cwd <path>", "Working directory")
 			.option("--config <path>", "Configuration directory")
-			.option("--thinking [tokens]", "Enable extended thinking")
-			.option("--reasoning-effort <effort>", "Reasoning effort")
-			.option("--max-consecutive-mistakes <count>", "Maximum consecutive mistakes")
+			.option("--thinking", "Enable extended thinking")
 			.action(() => {})
 
 		program
@@ -69,9 +67,7 @@ describe("CLI Commands", () => {
 			.option("-v, --verbose", "Verbose output")
 			.option("-c, --cwd <path>", "Working directory")
 			.option("--config <path>", "Configuration directory")
-			.option("--thinking [tokens]", "Enable extended thinking")
-			.option("--reasoning-effort <effort>", "Reasoning effort")
-			.option("--max-consecutive-mistakes <count>", "Maximum consecutive mistakes")
+			.option("--thinking", "Enable extended thinking")
 			.action(() => {})
 	})
 
@@ -150,27 +146,6 @@ describe("CLI Commands", () => {
 			expect(taskCmd.opts().thinking).toBe(true)
 		})
 
-		it("should parse --thinking with token budget", () => {
-			const taskCmd = program.commands.find((c) => c.name() === "task")!
-			const args = ["test prompt", "--thinking", "8000"]
-			taskCmd.parse(args, { from: "user" })
-			expect(taskCmd.opts().thinking).toBe("8000")
-		})
-
-		it("should parse --reasoning-effort option", () => {
-			const taskCmd = program.commands.find((c) => c.name() === "task")!
-			const args = ["test prompt", "--reasoning-effort", "high"]
-			taskCmd.parse(args, { from: "user" })
-			expect(taskCmd.opts().reasoningEffort).toBe("high")
-		})
-
-		it("should parse --max-consecutive-mistakes option", () => {
-			const taskCmd = program.commands.find((c) => c.name() === "task")!
-			const args = ["test prompt", "--max-consecutive-mistakes", "999"]
-			taskCmd.parse(args, { from: "user" })
-			expect(taskCmd.opts().maxConsecutiveMistakes).toBe("999")
-		})
-
 		it("should parse short flags", () => {
 			const taskCmd = program.commands.find((c) => c.name() === "task")!
 			const args = ["test prompt", "-a", "-v", "-m", "gpt-4"]
@@ -306,21 +281,6 @@ describe("CLI Commands", () => {
 			program.parse(["node", "cli", "--thinking"])
 			expect(program.opts().thinking).toBe(true)
 		})
-
-		it("should parse --thinking with token budget", () => {
-			program.parse(["node", "cli", "--thinking", "4096"])
-			expect(program.opts().thinking).toBe("4096")
-		})
-
-		it("should parse --reasoning-effort option", () => {
-			program.parse(["node", "cli", "--reasoning-effort", "medium"])
-			expect(program.opts().reasoningEffort).toBe("medium")
-		})
-
-		it("should parse --max-consecutive-mistakes option", () => {
-			program.parse(["node", "cli", "--max-consecutive-mistakes", "7"])
-			expect(program.opts().maxConsecutiveMistakes).toBe("7")
-		})
 	})
 
 	describe("command structure", () => {

cli/src/index.ts

@@ -15,14 +15,14 @@ import { HostProvider } from "@/hosts/host-provider"
 import { FileEditProvider } from "@/integrations/editor/FileEditProvider"
 import { openAiCodexOAuthManager } from "@/integrations/openai-codex/oauth"
 import { StandaloneTerminalManager } from "@/integrations/terminal/standalone/StandaloneTerminalManager"
+import { BannerService } from "@/services/banner/BannerService"
 import { ErrorService } from "@/services/error/ErrorService"
+import { initializeDistinctId } from "@/services/logging/distinctId"
 import { telemetryService } from "@/services/telemetry"
-import { PostHogClientProvider } from "@/services/telemetry/providers/posthog/PostHogClientProvider"
 import { HistoryItem } from "@/shared/HistoryItem"
 import { Logger } from "@/shared/services/Logger"
 import { Session } from "@/shared/services/Session"
 import { getProviderModelIdKey, ProviderToApiKeyMap } from "@/shared/storage"
-import { isOpenaiReasoningEffort, OPENAI_REASONING_EFFORT_OPTIONS, type OpenaiReasoningEffort } from "@/shared/storage/types"
 import { version as CLI_VERSION } from "../package.json"
 import { runAcpMode } from "./acp/index.js"
 import { App } from "./components/App"
@@ -32,12 +32,12 @@ import { CliCommentReviewController } from "./controllers/CliCommentReviewContro
 import { CliWebviewProvider } from "./controllers/CliWebviewProvider"
 import { restoreConsole } from "./utils/console"
 import { printInfo, printWarning } from "./utils/display"
-import { selectOutputMode } from "./utils/mode-selection"
 import { parseImagesFromInput, processImagePaths } from "./utils/parser"
 import { CLINE_CLI_DIR, getCliBinaryPath } from "./utils/path"
 import { readStdinIfPiped } from "./utils/piped"
 import { runPlainTextTask } from "./utils/plain-text-task"
 import { applyProviderConfig } from "./utils/provider-config"
+import { selectOutputMode } from "./utils/mode-selection"
 import { getValidCliProviders, isValidCliProvider } from "./utils/providers"
 import { autoUpdateOnStartup, checkForUpdates } from "./utils/update"
 import { initializeCliContext } from "./vscode-context"
@@ -53,92 +53,13 @@ interface TaskOptions {
 	verbose?: boolean
 	cwd?: string
 	config?: string
-	thinking?: boolean | string
-	reasoningEffort?: string
-	maxConsecutiveMistakes?: string
+	thinking?: boolean
 	yolo?: boolean
-	doubleCheckCompletion?: boolean
 	timeout?: string
 	json?: boolean
 	stdinWasPiped?: boolean
 }
 
-let telemetryDisposed = false
-
-async function disposeTelemetryServices(): Promise<void> {
-	if (telemetryDisposed) {
-		return
-	}
-
-	telemetryDisposed = true
-	await Promise.allSettled([telemetryService.dispose(), PostHogClientProvider.getInstance().dispose()])
-}
-
-/**
- * Restore yoloModeToggled to its original value from before this CLI session.
- * This ensures the --yolo flag is session-only and doesn't leak into future runs.
- * Must be called before flushPendingState so the restored value gets persisted.
- */
-function restoreYoloState(): void {
-	if (savedYoloModeToggled !== null) {
-		try {
-			StateManager.get().setGlobalState("yoloModeToggled", savedYoloModeToggled)
-			savedYoloModeToggled = null
-		} catch {
-			// StateManager may not be initialized (e.g., early exit before init)
-		}
-	}
-}
-
-async function disposeCliContext(ctx: CliContext): Promise<void> {
-	restoreYoloState()
-	await ctx.controller.stateManager.flushPendingState()
-	await ctx.controller.dispose()
-	await ErrorService.get().dispose()
-	await disposeTelemetryServices()
-}
-
-function setModeScopedState(currentMode: "act" | "plan", setter: (mode: "act" | "plan") => void): void {
-	const stateManager = StateManager.get()
-	setter(currentMode)
-
-	const separateModels = stateManager.getGlobalSettingsKey("planActSeparateModelsSetting") ?? false
-	if (!separateModels) {
-		const otherMode: "act" | "plan" = currentMode === "act" ? "plan" : "act"
-		setter(otherMode)
-	}
-}
-
-function normalizeReasoningEffort(value?: string): OpenaiReasoningEffort | undefined {
-	if (value === undefined) {
-		return undefined
-	}
-
-	const normalized = value.toLowerCase()
-	if (isOpenaiReasoningEffort(normalized)) {
-		return normalized
-	}
-
-	printWarning(
-		`Invalid --reasoning-effort '${value}'. Using 'medium'. Valid values: ${OPENAI_REASONING_EFFORT_OPTIONS.join(", ")}.`,
-	)
-	return "medium"
-}
-
-function normalizeMaxConsecutiveMistakes(value?: string): number | undefined {
-	if (value === undefined) {
-		return undefined
-	}
-
-	const parsed = Number.parseInt(value, 10)
-	if (Number.isNaN(parsed) || parsed < 1) {
-		printWarning(`Invalid --max-consecutive-mistakes value '${value}'. Expected integer >= 1.`)
-		return undefined
-	}
-
-	return parsed
-}
-
 /**
  * Apply task-related options (mode, model, thinking, yolo) to StateManager.
  * Shared between runTask and resumeTask to avoid duplication.
@@ -165,59 +86,20 @@ function applyTaskOptions(options: TaskOptions): void {
 		telemetryService.captureHostEvent("model_flag", options.model)
 	}
 
-	// Set thinking budget based on --thinking flag (boolean or number)
-	let thinkingBudget = 0
-	if (options.thinking) {
-		if (typeof options.thinking === "string") {
-			const parsed = Number.parseInt(options.thinking, 10)
-			if (Number.isNaN(parsed) || parsed < 0) {
-				printWarning(`Invalid --thinking value '${options.thinking}'. Using default 1024.`)
-				thinkingBudget = 1024
-			} else {
-				thinkingBudget = parsed
-			}
-		} else {
-			thinkingBudget = 1024
-		}
-	}
-	const currentMode = (StateManager.get().getGlobalSettingsKey("mode") || "act") as "act" | "plan"
-	setModeScopedState(currentMode, (mode) => {
-		const thinkingKey = mode === "act" ? "actModeThinkingBudgetTokens" : "planModeThinkingBudgetTokens"
-		StateManager.get().setGlobalState(thinkingKey, thinkingBudget)
-	})
+	// Set thinking budget based on --thinking flag
+	const thinkingBudget = options.thinking ? 1024 : 0
+	const currentMode = StateManager.get().getGlobalSettingsKey("mode") || "act"
+	const thinkingKey = currentMode === "act" ? "actModeThinkingBudgetTokens" : "planModeThinkingBudgetTokens"
+	StateManager.get().setGlobalState(thinkingKey, thinkingBudget)
 	if (options.thinking) {
 		telemetryService.captureHostEvent("thinking_flag", "true")
 	}
 
-	const reasoningEffort = normalizeReasoningEffort(options.reasoningEffort)
-	if (reasoningEffort !== undefined) {
-		setModeScopedState(currentMode, (mode) => {
-			const reasoningKey = mode === "act" ? "actModeReasoningEffort" : "planModeReasoningEffort"
-			StateManager.get().setGlobalState(reasoningKey, reasoningEffort)
-		})
-		telemetryService.captureHostEvent("reasoning_effort_flag", reasoningEffort)
-	}
-
-	const maxConsecutiveMistakes = normalizeMaxConsecutiveMistakes(options.maxConsecutiveMistakes)
-	if (maxConsecutiveMistakes !== undefined) {
-		StateManager.get().setGlobalState("maxConsecutiveMistakes", maxConsecutiveMistakes)
-		telemetryService.captureHostEvent("max_consecutive_mistakes_flag", String(maxConsecutiveMistakes))
-	}
-
-	// Override yolo mode only if --yolo flag is explicitly passed.
-	// The original value is saved in initializeCli and restored on exit.
+	// Set yolo mode based on --yolo flag
 	if (options.yolo) {
-		const state = StateManager.get()
-		savedYoloModeToggled = state.getGlobalSettingsKey("yoloModeToggled") ?? false
-		state.setGlobalState("yoloModeToggled", true)
+		StateManager.get().setGlobalState("yoloModeToggled", true)
 		telemetryService.captureHostEvent("yolo_flag", "true")
 	}
-
-	// Set double-check completion based on flag
-	if (options.doubleCheckCompletion) {
-		StateManager.get().setGlobalState("doubleCheckCompletionEnabled", true)
-		telemetryService.captureHostEvent("double_check_completion_flag", "true")
-	}
 }
 
 /**
@@ -269,7 +151,9 @@ async function runTaskInPlainTextMode(
 	const hasAuth = await isAuthConfigured()
 	if (!hasAuth) {
 		printWarning("Not authenticated. Please run 'cline auth' first to configure your API credentials.")
-		await disposeCliContext(ctx)
+		await ctx.controller.stateManager.flushPendingState()
+		await ctx.controller.dispose()
+		await ErrorService.get().dispose()
 		exit(1)
 	}
 
@@ -284,11 +168,13 @@ async function runTaskInPlainTextMode(
 		imageDataUrls: taskConfig.imageDataUrls,
 		verbose: options.verbose,
 		jsonOutput: options.json,
-		timeoutSeconds: options.timeout ? Number.parseInt(options.timeout, 10) : undefined,
+		timeoutSeconds: options.timeout ? parseInt(options.timeout, 10) : undefined,
 	})
 
 	// Cleanup
-	await disposeCliContext(ctx)
+	await ctx.controller.stateManager.flushPendingState()
+	await ctx.controller.dispose()
+	await ErrorService.get().dispose()
 
 	// Ensure stdout is fully drained before exiting - critical for piping
 	await drainStdout()
@@ -300,7 +186,9 @@ async function runTaskInPlainTextMode(
  */
 function createInkCleanup(ctx: CliContext, onTaskError?: () => boolean): () => Promise<void> {
 	return async () => {
-		await disposeCliContext(ctx)
+		await ctx.controller.stateManager.flushPendingState()
+		await ctx.controller.dispose()
+		await ErrorService.get().dispose()
 		if (onTaskError?.()) {
 			printWarning("Task ended with errors.")
 			exit(1)
@@ -314,9 +202,6 @@ let activeContext: CliContext | null = null
 let isShuttingDown = false
 // Track if we're in plain text mode (no Ink UI) - set by runTask when piped stdin detected
 let isPlainTextMode = false
-// Track the original yoloModeToggled value from before this CLI session so we can restore it on exit.
-// The --yolo flag should only affect the current invocation, not persist across runs.
-let savedYoloModeToggled: boolean | null = null
 
 /**
  * Wait for stdout to fully drain before exiting.
@@ -358,26 +243,15 @@ function setupSignalHandlers() {
 		printWarning(`${signal} received, shutting down...`)
 
 		try {
-			// Restore yolo state before any cleanup - this is idempotent and safe
-			// even if disposeCliContext also calls it (restoreYoloState checks savedYoloModeToggled !== null)
-			restoreYoloState()
-
 			if (activeContext) {
 				const task = activeContext.controller.task
 				if (task) {
 					task.abortTask()
 				}
-				await disposeCliContext(activeContext)
-			} else {
-				// Best-effort flush of restored yolo state when no active context
-				try {
-					await StateManager.get().flushPendingState()
-				} catch {
-					// StateManager may not be initialized yet
-				}
-				await ErrorService.get().dispose()
-				await disposeTelemetryServices()
+				await activeContext.controller.stateManager.flushPendingState()
+				await activeContext.controller.dispose()
 			}
+			await ErrorService.get().dispose()
 		} catch {
 			// Best effort cleanup
 		}
@@ -430,17 +304,8 @@ async function initializeCli(options: InitOptions): Promise<CliContext> {
 		workspaceDir: workspacePath,
 	})
 
-	// Set up output channel and Logger early so ClineEndpoint.initialize logs are captured
-	const outputChannel = window.createOutputChannel("Cline CLI")
-	const logToChannel = (message: string) => outputChannel.appendLine(message)
-
-	// Configure the shared Logging class early to capture all initialization logs
-	Logger.subscribe(logToChannel)
-
-	await ClineEndpoint.initialize(EXTENSION_DIR)
-
-	// Auto-update check (after endpoints initialized, so we can detect bundled configs)
-	autoUpdateOnStartup(CLI_VERSION)
+	await ClineEndpoint.initialize()
+	await initializeDistinctId(extensionContext)
 
 	// Initialize/reset session tracking for this CLI run
 	Session.reset()
@@ -449,9 +314,11 @@ async function initializeCli(options: InitOptions): Promise<CliContext> {
 		AuthHandler.getInstance().setEnabled(true)
 	}
 
+	const outputChannel = window.createOutputChannel("Cline CLI")
 	outputChannel.appendLine(
 		`Cline CLI initialized. Data dir: ${DATA_DIR}, Extension dir: ${EXTENSION_DIR}, Log dir: ${CLINE_CLI_DIR.log}`,
 	)
+	const logToChannel = (message: string) => outputChannel.appendLine(message)
 
 	HostProvider.initialize(
 		() => new CliWebviewProvider(extensionContext as any),
@@ -460,24 +327,28 @@ async function initializeCli(options: InitOptions): Promise<CliContext> {
 		() => new StandaloneTerminalManager(),
 		createCliHostBridgeProvider(workspacePath),
 		logToChannel,
-		async (path: string) => (options.enableAuth ? AuthHandler.getInstance().getCallbackUrl(path) : ""),
+		async () => (options.enableAuth ? AuthHandler.getInstance().getCallbackUrl() : ""),
 		getCliBinaryPath,
 		EXTENSION_DIR,
 		DATA_DIR,
 	)
 
 	await StateManager.initialize(extensionContext as any)
-
 	await ErrorService.initialize()
 
 	// Initialize OpenAI Codex OAuth manager with extension context for secrets storage
 	openAiCodexOAuthManager.initialize(extensionContext)
 
+	// Configure the shared Logging class to use HostProvider's output channel
+	Logger.subscribe((msg: string) => HostProvider.get().logToChannel(msg))
+
 	const webview = HostProvider.get().createWebviewProvider() as CliWebviewProvider
 	const controller = webview.controller
 
-	await telemetryService.captureExtensionActivated()
-	await telemetryService.captureHostEvent("cline_cli", "initialized")
+	BannerService.initialize(webview.controller)
+
+	telemetryService.captureExtensionActivated()
+	telemetryService.captureHostEvent("cline_cli", "initialized")
 
 	const ctx = { extensionContext, dataDir: DATA_DIR, extensionDir: EXTENSION_DIR, workspacePath, controller }
 	activeContext = ctx
@@ -559,7 +430,8 @@ async function runTask(prompt: string, options: TaskOptions & { images?: string[
 				taskError = true
 			},
 			onWelcomeExit: () => {
-				// User pressed Esc; Ink exits and cleanup handles process exit.
+				// User pressed Esc
+				exit(0)
 			},
 		}),
 		createInkCleanup(ctx, () => taskError),
@@ -575,8 +447,8 @@ async function listHistory(options: { config?: string; limit?: number; page?: nu
 	const taskHistory = StateManager.get().getGlobalStateKey("taskHistory") || []
 	// Sort by timestamp (newest first) before pagination
 	const sortedHistory = [...taskHistory].sort((a: any, b: any) => (b.ts || 0) - (a.ts || 0))
-	const limit = typeof options.limit === "string" ? Number.parseInt(options.limit, 10) : options.limit || 10
-	const initialPage = typeof options.page === "string" ? Number.parseInt(options.page, 10) : options.page || 1
+	const limit = typeof options.limit === "string" ? parseInt(options.limit, 10) : options.limit || 10
+	const initialPage = typeof options.page === "string" ? parseInt(options.page, 10) : options.page || 1
 	const totalCount = sortedHistory.length
 	const totalPages = Math.ceil(totalCount / limit)
 
@@ -584,7 +456,9 @@ async function listHistory(options: { config?: string; limit?: number; page?: nu
 
 	if (sortedHistory.length === 0) {
 		printInfo("No task history found.")
-		await disposeCliContext(ctx)
+		await ctx.controller.stateManager.flushPendingState()
+		await ctx.controller.dispose()
+		await ErrorService.get().dispose()
 		exit(0)
 	}
 
@@ -598,7 +472,9 @@ async function listHistory(options: { config?: string; limit?: number; page?: nu
 			isRawModeSupported: checkRawModeSupport(),
 		}),
 		async () => {
-			await disposeCliContext(ctx)
+			await ctx.controller.stateManager.flushPendingState()
+			await ctx.controller.dispose()
+			await ErrorService.get().dispose()
 			exit(0)
 		},
 	)
@@ -627,7 +503,9 @@ async function showConfig(options: { config?: string }) {
 			isRawModeSupported: checkRawModeSupport(),
 		}),
 		async () => {
-			await disposeCliContext(ctx)
+			await ctx.controller.stateManager.flushPendingState()
+			await ctx.controller.dispose()
+			await ErrorService.get().dispose()
 			exit(0)
 		},
 	)
@@ -703,15 +581,17 @@ async function runAuth(options: {
 			baseurl: options.baseurl,
 		})
 
+		await ctx.controller.stateManager.flushPendingState()
+		await ctx.controller.dispose()
+		await ErrorService.get().dispose()
+
 		if (!result.success) {
 			printWarning(result.error || "Quick setup failed")
-			await telemetryService.captureHostEvent("auth", "error")
-			await disposeCliContext(ctx)
+			telemetryService.captureHostEvent("auth", "error")
 			exit(1)
 		}
 
-		await telemetryService.captureHostEvent("auth", "completed")
-		await disposeCliContext(ctx)
+		telemetryService.captureHostEvent("auth", "completed")
 		exit(0)
 	}
 
@@ -725,6 +605,7 @@ async function runAuth(options: {
 			isRawModeSupported: checkRawModeSupport(),
 			onComplete: () => {
 				telemetryService.captureHostEvent("auth", "completed")
+				exit(0)
 			},
 			onError: () => {
 				telemetryService.captureHostEvent("auth", "error")
@@ -732,10 +613,16 @@ async function runAuth(options: {
 			},
 		}),
 		async () => {
-			await disposeCliContext(ctx)
-			exit(authError ? 1 : 0)
+			await ctx.controller.stateManager.flushPendingState()
+			await ctx.controller.dispose()
+			await ErrorService.get().dispose()
+			exit(0)
 		},
 	)
+
+	if (authError) {
+		process.exit(1)
+	}
 }
 
 // Setup CLI commands
@@ -759,11 +646,8 @@ program
 	.option("-v, --verbose", "Show verbose output")
 	.option("-c, --cwd <path>", "Working directory for the task")
 	.option("--config <path>", "Path to Cline configuration directory")
-	.option("--thinking [tokens]", "Enable extended thinking (default: 1024 tokens)")
-	.option("--reasoning-effort <effort>", "Reasoning effort: none|low|medium|high|xhigh")
-	.option("--max-consecutive-mistakes <count>", "Maximum consecutive mistakes before halting in yolo mode")
+	.option("--thinking", "Enable extended thinking (1024 token budget)")
 	.option("--json", "Output messages as JSON instead of styled text")
-	.option("--double-check-completion", "Reject first completion attempt to force re-verification")
 	.option("-T, --taskId <id>", "Resume an existing task by ID")
 	.action((prompt, options) => {
 		if (options.taskId) {
@@ -904,7 +788,9 @@ async function resumeTask(taskId: string, options: TaskOptions & { initialPrompt
 	if (!historyItem) {
 		printWarning(`Task not found: ${taskId}`)
 		printInfo("Use 'cline history' to see available tasks.")
-		await disposeCliContext(ctx)
+		await ctx.controller.stateManager.flushPendingState()
+		await ctx.controller.dispose()
+		await ErrorService.get().dispose()
 		exit(1)
 	}
 
@@ -937,7 +823,7 @@ async function resumeTask(taskId: string, options: TaskOptions & { initialPrompt
 				taskError = true
 			},
 			onWelcomeExit: () => {
-				// User pressed Esc; Ink exits and cleanup handles process exit.
+				exit(0)
 			},
 		}),
 		createInkCleanup(ctx, () => taskError),
@@ -964,14 +850,16 @@ async function showWelcome(options: { verbose?: boolean; cwd?: string; config?:
 			controller: ctx.controller,
 			isRawModeSupported: checkRawModeSupport(),
 			onWelcomeExit: () => {
-				// User pressed Esc; Ink exits and cleanup handles process exit.
+				exit(0)
 			},
 			onError: () => {
 				hadError = true
 			},
 		}),
 		async () => {
-			await disposeCliContext(ctx)
+			await ctx.controller.stateManager.flushPendingState()
+			await ctx.controller.dispose()
+			await ErrorService.get().dispose()
 			exit(hadError ? 1 : 0)
 		},
 	)
@@ -988,11 +876,8 @@ program
 	.option("-v, --verbose", "Show verbose output")
 	.option("-c, --cwd <path>", "Working directory")
 	.option("--config <path>", "Configuration directory")
-	.option("--thinking [tokens]", "Enable extended thinking (default: 1024 tokens)")
-	.option("--reasoning-effort <effort>", "Reasoning effort: none|low|medium|high|xhigh")
-	.option("--max-consecutive-mistakes <count>", "Maximum consecutive mistakes before halting in yolo mode")
+	.option("--thinking", "Enable extended thinking (1024 token budget)")
 	.option("--json", "Output messages as JSON instead of styled text")
-	.option("--double-check-completion", "Reject first completion attempt to force re-verification")
 	.option("--acp", "Run in ACP (Agent Client Protocol) mode for editor integration")
 	.option("-T, --taskId <id>", "Resume an existing task by ID")
 	.action(async (prompt, options) => {
@@ -1062,5 +947,8 @@ program
 		}
 	})
 
+// Background auto-update check (non-blocking)
+autoUpdateOnStartup(CLI_VERSION)
+
 // Parse and run
 program.parse()

cli/src/utils/plain-text-task.test.ts

@@ -1,28 +0,0 @@
-import { afterEach, describe, expect, it, vi } from "vitest"
-import { emitTaskStartedMessage } from "./task-start-output"
-
-describe("emitTaskStartedMessage", () => {
-	afterEach(() => {
-		vi.restoreAllMocks()
-	})
-
-	it("writes structured task_started JSON to stdout in json mode", () => {
-		const stdoutWriteSpy = vi.spyOn(process.stdout, "write").mockImplementation(() => true)
-		const stderrWriteSpy = vi.spyOn(process.stderr, "write").mockImplementation(() => true)
-
-		emitTaskStartedMessage("task-123", true)
-
-		expect(stdoutWriteSpy).toHaveBeenCalledWith('{"type":"task_started","taskId":"task-123"}\n')
-		expect(stderrWriteSpy).not.toHaveBeenCalled()
-	})
-
-	it("writes human-readable task started line to stderr in non-json mode", () => {
-		const stdoutWriteSpy = vi.spyOn(process.stdout, "write").mockImplementation(() => true)
-		const stderrWriteSpy = vi.spyOn(process.stderr, "write").mockImplementation(() => true)
-
-		emitTaskStartedMessage("task-456", false)
-
-		expect(stderrWriteSpy).toHaveBeenCalledWith("Task started: task-456\n")
-		expect(stdoutWriteSpy).not.toHaveBeenCalled()
-	})
-})

cli/src/utils/plain-text-task.ts

@@ -17,7 +17,6 @@ import type { Controller } from "@/core/controller"
 import { getRequestRegistry } from "@/core/controller/grpc-handler"
 import { subscribeToState } from "@/core/controller/state/subscribeToState"
 import { showTaskWithId } from "@/core/controller/task/showTaskWithId"
-import { emitTaskStartedMessage } from "./task-start-output"
 
 export interface PlainTextTaskOptions {
 	controller: Controller
@@ -53,7 +52,6 @@ export async function runPlainTextTask(options: PlainTextTaskOptions): Promise<b
 	})
 
 	let hasError = false
-	let hasEmittedTaskStarted = false
 	// Track which messages have been processed (by timestamp)
 	const processedMessages = new Map<number, string>()
 
@@ -64,20 +62,6 @@ export async function runPlainTextTask(options: PlainTextTaskOptions): Promise<b
 	// results AFTER this time should trigger task completion.
 	const completionCutoffTs = Date.now()
 
-	const emitTaskStarted = () => {
-		if (hasEmittedTaskStarted) {
-			return
-		}
-
-		const taskId = controller.task?.taskId
-		if (!taskId) {
-			return
-		}
-
-		emitTaskStartedMessage(taskId, Boolean(jsonOutput))
-		hasEmittedTaskStarted = true
-	}
-
 	// Helper to process a message and track completion state
 	const processMessage = (message: ClineMessage) => {
 		const ts = message.ts || 0
@@ -135,7 +119,6 @@ export async function runPlainTextTask(options: PlainTextTaskOptions): Promise<b
 		if (options.taskId) {
 			// Load the existing task
 			await showTaskWithId(controller, StringRequest.create({ value: options.taskId }))
-			emitTaskStarted()
 
 			// If a prompt was provided, send it as a message to the resumed task
 			if (prompt && controller.task) {
@@ -148,7 +131,6 @@ export async function runPlainTextTask(options: PlainTextTaskOptions): Promise<b
 		} else if (prompt) {
 			// Start a new task with the prompt
 			await controller.initTask(prompt, imageDataUrls)
-			emitTaskStarted()
 		} else {
 			throw new Error("Either taskId or prompt must be provided")
 		}

cli/src/utils/task-start-output.ts

@@ -1,8 +0,0 @@
-export function emitTaskStartedMessage(taskId: string, jsonOutput: boolean): void {
-	if (jsonOutput) {
-		process.stdout.write(JSON.stringify({ type: "task_started", taskId }) + "\n")
-		return
-	}
-
-	process.stderr.write(`Task started: ${taskId}\n`)
-}

cli/src/utils/update.ts

@@ -1,7 +1,6 @@
 import { spawn } from "node:child_process"
 import { realpathSync } from "node:fs"
 import { exit } from "node:process"
-import { ClineEndpoint } from "@/config"
 import { fetch } from "@/shared/net"
 import { printInfo, printWarning } from "./display"
 
@@ -108,7 +107,7 @@ async function getLatestVersion(currentVersion: string): Promise<string | null>
  * process to install if a newer version is available.
  *
  * Supports npm, pnpm, yarn, and bun global installs.
- * Skipped for npx, local dev, unknown installations, and bundled enterprise packages.
+ * Skipped for npx, local dev, and unknown installations.
  * Can be disabled with CLINE_NO_AUTO_UPDATE=1 environment variable.
  */
 export function autoUpdateOnStartup(currentVersion: string): void {
@@ -122,11 +121,6 @@ export function autoUpdateOnStartup(currentVersion: string): void {
 		return
 	}
 
-	// Skip if using bundled enterprise config (single source of truth)
-	if (ClineEndpoint.isBundledConfig()) {
-		return
-	}
-
 	const { updateCommand } = getInstallationInfo(currentVersion)
 	if (!updateCommand) {
 		return

cli/src/vscode-context.ts

@@ -4,7 +4,6 @@
  */
 
 import { mkdirSync } from "node:fs"
-import { fileURLToPath } from "node:url"
 import os from "os"
 import path from "path"
 import { ExtensionRegistryInfo } from "@/registry"
@@ -12,10 +11,6 @@ import { ClineExtensionContext } from "@/shared/cline"
 import { ClineFileStorage } from "@/shared/storage"
 import { EnvironmentVariableCollection, ExtensionKind, ExtensionMode, readJson, URI } from "./vscode-shim"
 
-// ES module equivalent of __dirname
-const __filename = fileURLToPath(import.meta.url)
-const __dirname = path.dirname(__filename)
-
 const SETTINGS_SUBFOLDER = "data"
 
 /**
@@ -145,8 +140,8 @@ export function initializeCliContext(config: CliContextConfig = {}): CliContextR
 	mkdirSync(DATA_DIR, { recursive: true })
 	mkdirSync(WORKSPACE_STORAGE_DIR, { recursive: true })
 
-	// For CLI, extension dir is the package root (one level up from dist/)
-	const EXTENSION_DIR = path.resolve(__dirname, "..")
+	// For CLI, extension dir is the root of the project (parent of cli)
+	const EXTENSION_DIR = path.resolve(__dirname, "..", "..")
 	const EXTENSION_MODE = process.env.IS_DEV === "true" ? ExtensionMode.Development : ExtensionMode.Production
 
 	const extension: ClineExtensionContext["extension"] = {

cli/vitest.config.ts

@@ -13,7 +13,6 @@ export default defineConfig({
 	},
 	resolve: {
 		alias: {
-			vscode: path.resolve(__dirname, "src/vscode-shim.ts"),
 			// Match tsconfig paths - baseUrl is parent directory
 			"@": path.resolve(__dirname, "../src"),
 			"@api": path.resolve(__dirname, "../src/core/api"),

docs/docs.json

@@ -164,7 +164,6 @@
 							"features/multiroot-workspace",
 							"features/plan-and-act",
 							"features/skills",
-							"features/subagents",
 							{
 								"group": "Slash Commands",
 								"pages": [

docs/enterprise-solutions/bundled-endpoints.mdx

@@ -1,287 +0,0 @@
----
-title: "Bundled Endpoints Configuration"
-description: "Enterprise guide for distributing Cline with pre-configured endpoints"
----
-
-# Bundled Endpoints Configuration
-
-This guide explains how enterprise customers can distribute Cline with pre-configured endpoints bundled directly into the installation packages.
-
-## Overview
-
-Cline supports bundling custom endpoint configurations directly into distribution packages (VSIX, NPM, or JetBrains). This eliminates the need for end users to manually configure endpoints, ensuring consistent configuration across your organization.
-
-### Configuration Priority
-
-When Cline starts, it checks for endpoints configuration in this order:
-
-1. **Bundled endpoints.json** (in extension installation directory) - Highest priority
-2. **User endpoints.json** (`~/.cline/endpoints.json`) - Fallback
-3. **Built-in endpoints** (standard Cline URLs) - Default
-
-When a bundled `endpoints.json` is found, Cline automatically switches to self-hosted mode and uses those endpoints exclusively.
-
-## Prerequisites
-
-- Official Cline release package (VSIX, TGZ, or ZIP)
-- Your `endpoints.json` configuration file
-- `jq` command-line tool (for JSON validation)
-- `unzip`, `zip`, `tar` utilities
-
-## Creating endpoints.json
-
-Create a JSON file with your organization's endpoints:
-
-```json
-{
-  "appBaseUrl": "https://cline.yourcompany.com",
-  "apiBaseUrl": "https://api-cline.yourcompany.com",
-  "mcpBaseUrl": "https://api-cline.yourcompany.com/v1/mcp"
-}
-```
-
-### Required Fields
-
-All three fields are required and must be valid URLs:
-
-- **appBaseUrl**: Web application base URL
-- **apiBaseUrl**: API server base URL  
-- **mcpBaseUrl**: MCP (Model Context Protocol) server URL
-
-### Validation
-
-The packaging scripts automatically validate:
-- Valid JSON syntax
-- All required fields present
-- Non-empty string values
-- Valid URL format (must start with `http://` or `https://`)
-
-## Packaging Scripts
-
-Cline provides three scripts for adding bundled endpoints to packages:
-
-### VSCode Extension (VSIX)
-
-```bash
-./scripts/add-endpoints-to-vsix.sh \
-  cline-3.55.0.vsix \
-  cline-3.55.0-enterprise.vsix \
-  endpoints.json
-```
-
-This script:
-1. Extracts the VSIX package
-2. Adds `endpoints.json` to the `extension/` directory
-3. Repackages as a new VSIX file
-
-### NPM Package (CLI)
-
-```bash
-./scripts/add-endpoints-to-npm.sh \
-  cline-3.55.0.tgz \
-  cline-3.55.0-enterprise.tgz \
-  endpoints.json
-```
-
-This script:
-1. Extracts the NPM tarball
-2. Adds `endpoints.json` to the package root
-3. Repackages as a new tarball
-
-### JetBrains Plugin (ZIP)
-
-```bash
-./scripts/add-endpoints-to-jetbrains.sh \
-  cline-jetbrains-3.55.0.zip \
-  cline-jetbrains-3.55.0-enterprise.zip \
-  endpoints.json
-```
-
-This script:
-1. Extracts the ZIP package
-2. Adds `endpoints.json` to the plugin directory
-3. Repackages as a new ZIP file
-
-## Distribution Workflow
-
-### 1. Download Official Release
-
-Download the official Cline package for your platform:
-
-```bash
-# VSCode - from marketplace or GitHub releases
-curl -LO https://github.com/cline/cline/releases/download/v3.55.0/cline-3.55.0.vsix
-
-# NPM - from npm registry
-npm pack @cline/cline@3.55.0
-
-# JetBrains - from marketplace or GitHub releases
-curl -LO https://github.com/cline/cline/releases/download/v3.55.0/cline-jetbrains-3.55.0.zip
-```
-
-### 2. Create Endpoints Configuration
-
-Create your `endpoints.json` file:
-
-```json
-{
-  "appBaseUrl": "https://cline.internal.company.com",
-  "apiBaseUrl": "https://cline-api.internal.company.com",
-  "mcpBaseUrl": "https://cline-api.internal.company.com/v1/mcp"
-}
-```
-
-### 3. Run Packaging Script
-
-Choose the appropriate script for your platform:
-
-```bash
-# VSCode
-./scripts/add-endpoints-to-vsix.sh \
-  cline-3.55.0.vsix \
-  cline-3.55.0-yourcompany.vsix \
-  endpoints.json
-
-# CLI
-./scripts/add-endpoints-to-npm.sh \
-  cline-3.55.0.tgz \
-  cline-3.55.0-yourcompany.tgz \
-  endpoints.json
-
-# JetBrains
-./scripts/add-endpoints-to-jetbrains.sh \
-  cline-jetbrains-3.55.0.zip \
-  cline-jetbrains-3.55.0-yourcompany.zip \
-  endpoints.json
-```
-
-### 4. Distribute to Users
-
-Distribute the enterprise package to your users through your internal channels:
-
-- **VSCode**: Install via `code --install-extension cline-3.55.0-yourcompany.vsix`
-- **CLI**: Install via `npm install -g cline-3.55.0-yourcompany.tgz`
-- **JetBrains**: Install through IDE plugin manager from disk
-
-## Verification
-
-After installation, verify the configuration is active:
-
-1. Launch Cline
-2. Check the logs for: `"Cline running in self-hosted mode with custom endpoints"`
-3. Confirm that environment switching is disabled (as expected in self-hosted mode)
-
-## User Experience
-
-### What Users See
-
-- Cline automatically uses the bundled endpoints
-- No manual configuration required
-- Environment switching is disabled (prevents accidental misconfiguration)
-- All API calls route to your organization's infrastructure
-
-### User Override
-
-Users **cannot** override bundled endpoints through the UI. The bundled configuration takes absolute precedence. This ensures:
-- Consistent configuration across the organization
-- No accidental connections to external services
-- Simplified deployment and support
-
-If users have a `~/.cline/endpoints.json` file, it will be ignored when bundled configuration is present.
-
-## Troubleshooting
-
-### Invalid Configuration Error
-
-If users see an error about invalid configuration on startup:
-
-```
-ClineConfigurationError: Invalid JSON in bundled endpoints configuration file
-```
-
-**Solution**: The bundled `endpoints.json` is malformed. Repackage with a valid JSON file.
-
-### Missing Required Field Error
-
-```
-ClineConfigurationError: Missing required field "apiBaseUrl" in endpoints configuration file
-```
-
-**Solution**: Ensure all three required fields are present in `endpoints.json`.
-
-### Invalid URL Error
-
-```
-ClineConfigurationError: Field "appBaseUrl" must be a valid URL. Got: "not-a-url"
-```
-
-**Solution**: All URLs must start with `http://` or `https://`.
-
-## Security Considerations
-
-1. **Bundle Validation**: The packaging scripts validate JSON structure and required fields
-2. **Read-Only Configuration**: Users cannot modify bundled endpoints through the UI
-3. **Self-Hosted Mode**: Automatic switch to self-hosted mode prevents external connections
-4. **Audit Trail**: All endpoint access is logged with configuration source
-
-## Updating Endpoints
-
-To update endpoints for existing installations:
-
-1. Create updated `endpoints.json`
-2. Repackage the same Cline version with new endpoints
-3. Distribute updated package
-4. Users reinstall/update the package
-
-The version number remains the same since only configuration changed, not the Cline code.
-
-## Support
-
-For questions or issues with bundled endpoints:
-
-1. Verify your `endpoints.json` is valid JSON with all required fields
-2. Check that URLs are accessible from user networks
-3. Review Cline logs for configuration loading messages
-4. Contact your Cline support representative for assistance
-
-## Example: Complete Workflow
-
-Here's a complete example for VSCode deployment:
-
-```bash
-# 1. Download official release
-curl -LO https://github.com/cline/cline/releases/download/v3.55.0/cline-3.55.0.vsix
-
-# 2. Create endpoints configuration
-cat > endpoints.json << 'EOF'
-{
-  "appBaseUrl": "https://cline.acme.internal",
-  "apiBaseUrl": "https://cline-api.acme.internal",
-  "mcpBaseUrl": "https://cline-api.acme.internal/v1/mcp"
-}
-EOF
-
-# 3. Validate JSON
-jq empty endpoints.json  # Should succeed silently
-
-# 4. Run packaging script
-./scripts/add-endpoints-to-vsix.sh \
-  cline-3.55.0.vsix \
-  cline-3.55.0-acme.vsix \
-  endpoints.json
-
-# 5. Verify output
-unzip -l cline-3.55.0-acme.vsix | grep endpoints.json
-# Should show: extension/endpoints.json
-
-# 6. Test installation (on test machine)
-code --install-extension cline-3.55.0-acme.vsix
-
-# 7. Distribute to organization
-# Upload to internal package repository
-# or distribute via configuration management system
-```
-
-## Changelog
-
-- **v3.55.0**: Initial release of bundled endpoints support

docs/features/skills.mdx

@@ -62,16 +62,15 @@ The description is critical because it's how Cline decides whether to activate a
 Skills can be stored in two locations:
 
 **Global Skills** apply to all your projects:
-- **macOS/Linux:** `~/.agents/skills/` (recommended) or `~/.cline/skills/`
-- **Windows:** `C:\Users\USERNAME\.agents\skills\` (recommended) or `C:\Users\USERNAME\.cline\skills\`
+- **macOS/Linux:** `~/.cline/skills/`
+- **Windows:** `C:\Users\USERNAME\.cline\skills\`
 
 **Project Skills** apply only to the current workspace:
-- `.agents/skills/` (recommended)
-- `.cline/skills/`
+- `.cline/skills/` (recommended)
 - `.clinerules/skills/`
 - `.claude/skills/` (for Claude Code compatibility)
 
-When a global skill and project skill have the same name, the global skill takes precedence. Skills in `.agents/skills` directories take precedence over other locations with the same name, letting you customize skills for your personal workflow while still using project defaults.
+When a global skill and project skill have the same name, the global skill takes precedence. This lets you customize skills for your personal workflow while still using project defaults.
 
 ## Managing Skills
 

docs/features/subagents.mdx

@@ -1,85 +0,0 @@
----
-title: "Subagents"
-sidebarTitle: "Subagents"
-description: "Run parallel research agents to explore your codebase without filling the main agent's context window."
----
-
-Subagents let Cline spawn focused research agents that run in parallel. Each subagent gets its own prompt and context window, explores the codebase independently, and returns a detailed report to the main agent. This keeps the main agent's context clean while gathering broad information fast.
-
-<Tip>
-  Subagents is an experimental feature. Behavior may change in future releases.
-</Tip>
-
-## How It Works
-
-When Cline uses the `use_subagents` tool, it launches independent agents simultaneously. Each one:
-
-- Gets its own prompt describing what to investigate
-- Runs with a separate context window and token budget
-- Can read files, search code, list directories, run read-only commands, and use skills
-- Cannot edit files, use the browser, access MCP servers, or spawn nested subagents
-- Returns a result focused on the most relevant file paths for the main agent to read next
-
-Subagent costs (tokens and API spend) are tracked separately per subagent and rolled into the task's total cost. You can see per-subagent stats (tool calls, tokens, cost) in the chat UI as they run.
-
-## Enabling Subagents
-
-Subagents are disabled by default. To turn them on:
-
-1. Open Cline Settings (click the gear icon in the Cline panel)
-2. Go to **Features**
-3. Under the **Agent** section, toggle **Subagents** on
-
-This setting applies across all editors (VS Code, JetBrains, CLI).
-
-## Using Subagents
-
-Cline does not automatically decide to use subagents. You need to ask for them in your prompt. When the feature is enabled and you mention subagents (or describe a task that benefits from parallel exploration), Cline will use the `use_subagents` tool.
-
-Example prompts:
-
-- "Use subagents to explore how authentication works and where the database models are defined"
-- "Spin up subagents to investigate the API routes, the test setup, and the deployment config"
-- "I'm new to this codebase. Use subagents to map out the main entry points, the routing layer, and the data access patterns"
-
-Each subagent prompt should describe a focused research question. Cline will run them in parallel and synthesize the results.
-You can also run only one subagent when the task is small enough that parallel discovery would be unnecessary overhead.
-
-## Auto-Approve Behavior
-
-Subagents follow the **Read project files** auto-approve permission. If you have "Read project files" enabled in [Auto Approve](/features/auto-approve), subagent launches will be auto-approved.
-
-In [YOLO mode](/features/yolo-mode), subagents are always auto-approved.
-
-If auto-approve is off, Cline will ask for your approval before launching subagents, showing you the prompts it plans to send.
-
-## What Subagents Can Do
-
-Subagents are read-only research agents. Here is what they have access to:
-
-| Tool | Purpose |
-|------|---------|
-| `read_file` | Read file contents |
-| `list_files` | List directory contents |
-| `search_files` | Regex search across files |
-| `list_code_definition_names` | List top-level classes, functions, and methods |
-| `execute_command` | Run read-only commands (`ls`, `grep`, `git log`, `git diff`, etc.) |
-| `use_skill` | Load and activate skills |
-
-Subagents cannot write files, apply patches, use the browser, access MCP servers, or perform web searches. They also cannot spawn their own subagents.
-
-<Note>
-  Commands run by subagents execute in the background and are restricted to read-only operations. Subagents will not run commands that modify files or system state.
-  Subagents also benefit from command pipelines and filters to narrow output quickly before reading files, for example `rg ... | sort | uniq`.
-</Note>
-
-## When to Use Subagents
-
-Subagents work best when you need broad context from multiple areas of a codebase at once:
-
-- **Onboarding to an unfamiliar project**: Ask subagents to map out the architecture, key entry points, and data flow in parallel.
-- **Investigating cross-cutting concerns**: Have separate subagents trace authentication, logging, and error handling simultaneously.
-- **Pre-edit research**: Before making changes, use subagents to gather context from related files so the main agent can make informed edits without burning through its context window.
-- **Large codebases**: When reading many files sequentially would consume too much of the main agent's context, subagents let you explore broadly without that tradeoff.
-
-For small, focused tasks where you already know which files to look at, subagents add unnecessary overhead. Just ask Cline directly.

docs/provider-config/gcp-vertex-ai.mdx

@@ -53,7 +53,7 @@ Vertex AI supports multiple regions. Select a region that meets your latency, co
 -   **asia-southeast1 (Singapore)**
 -   **global (Global)**
 
-The Global endpoint may offer higher availability and reduce resource exhausted errors. Gemini models and supported Claude models can use it, depending on model availability in your project.
+The Global endpoint may offer higher availability and reduce resource exhausted errors. Only Gemini models are supported.
 
 #### 2.2 Enable the Claude 3.5 Sonnet v2 Model
 

evals/.gitignore

@@ -2,28 +2,23 @@ repositories
 temp-files
 results
 
-# Tool precision - results and databases
-benchmarks/tool-precision/replace-in-file/results/
-benchmarks/tool-precision/replace-in-file/*.db
-benchmarks/tool-precision/replace-in-file/*.db-wal
-benchmarks/tool-precision/replace-in-file/*.db-shm
-
-# Tool precision - private test cases (from real sessions)
-# Public/synthetic cases (example-*.json) ARE committed
-benchmarks/tool-precision/replace-in-file/cases/private-*.json
-benchmarks/tool-precision/replace-in-file/cases/session-*.json
-
-# Legacy paths (kept for backwards compatibility)
 diff-edits/cases/
 diff-edits/results/
-diff_editing/test_cases/
-diff_editing/test_outputs/
-diff-edits/cases.zip
 
 # Environment variables
 .env
 
+# backwards compatible
+diff_editing/test_cases/
+diff_editing/test_outputs/
+
+*.db
+*.db-wal
+*.db-shm
+
+.cache
+
 # Python bytecode cache
 *__pycache__/
 
-.cache
+diff-edits/cases.zip

evals/ARCHITECTURE.md

@@ -1,288 +0,0 @@
-# Cline Evals Architecture
-
-## Overview
-
-The evals system provides multi-layered testing for Cline's AI capabilities.
-
-```
-┌─────────────────────────────────────────────────────────────────────────────┐
-│                           TESTING PYRAMID                                   │
-├─────────────────────────────────────────────────────────────────────────────┤
-│                                                                             │
-│                              ┌─────────┐                                    │
-│                             /   E2E    \         Layer 3: Full Agent        │
-│                            /  cline-   \         - Real coding tasks        │
-│                           /   bench     \        - Harbor execution         │
-│                          /_______________\       - Nightly runs             │
-│                                                                             │
-│                        ┌───────────────────┐                                │
-│                       /    Smoke Tests     \     Layer 2: Provider          │
-│                      /   run-smoke-tests    \    - 5 curated scenarios      │
-│                     /    (cline provider)    \   - 3 models via Vercel      │
-│                    /_________________________\   - pass@k metrics           │
-│                                                                             │
-│              ┌─────────────────────────────────┐                            │
-│             /        Contract Tests            \  Layer 1: Unit             │
-│            /   thinking-traces.test.ts          \ - No LLM calls            │
-│           /    tool-parsing.test.ts              \ - Fast, deterministic    │
-│          /______________________________________ \ - API format validation  │
-│                                                                             │
-└─────────────────────────────────────────────────────────────────────────────┘
-```
-
-## Directory Structure
-
-```
-evals/
-├── ARCHITECTURE.md          # This file
-├── README.md                # Quick start guide
-│
-├── analysis/                # Shared metrics & reporting
-│   └── src/
-│       ├── metrics.ts       # pass@k, pass^k, flakiness calculations
-│       └── cli.ts           # Analysis CLI
-│
-├── smoke-tests/             # Layer 2: Provider smoke tests
-│   ├── run-smoke-tests.ts   # Main runner
-│   ├── README.md            # Usage docs
-│   ├── scenarios/           # Test definitions
-│   │   ├── 01-create-file/
-│   │   │   ├── config.json  # Prompt, expected files/content
-│   │   │   ├── template/    # Initial files (if any)
-│   │   │   └── workspace/   # Working dir (cleaned each run)
-│   │   ├── 02-edit-file/
-│   │   ├── 03-read-summarize/
-│   │   ├── 04-multi-file/
-│   │   └── 05-typescript-function/
-│   └── results/             # Generated outputs
-│       ├── latest -> 2026-01-27T.../  # Symlink to most recent
-│       └── 2026-01-27T19-50-54-391Z/
-│           ├── report.json  # Full results
-│           ├── summary.md   # CI-friendly markdown
-│           └── 01-create-file/
-│               └── claude-sonnet/
-│                   ├── trial-1.log           # CLI stdout/stderr
-│                   └── workspace-trial-1/    # Kept for failures only
-│
-├── e2e/                     # Layer 3: Full agent E2E
-│   ├── run-cline-bench.ts   # Harbor runner
-│   └── README.md
-│
-└── cline-bench/             # Git submodule with real coding tasks
-    └── tasks/               # SWE-bench style problems
-```
-
-## Smoke Test Workflow
-
-```
-┌──────────────────────────────────────────────────────────────────────────────┐
-│                         SMOKE TEST EXECUTION FLOW                            │
-└──────────────────────────────────────────────────────────────────────────────┘
-
- npm run eval:smoke
-        │
-        ▼
-┌───────────────────┐
-│ Load scenarios    │  Read config.json from each scenarios/* dir
-│ from disk         │
-└────────┬──────────┘
-         │
-         ▼
-┌───────────────────┐
-│ Create results    │  evals/smoke-tests/results/2026-01-27T.../
-│ directory         │
-└────────┬──────────┘
-         │
-         ▼
-┌───────────────────────────────────────────────────────────────┐
-│                    FOR EACH SCENARIO                          │
-│  ┌─────────────────────────────────────────────────────────┐  │
-│  │                  FOR EACH MODEL                         │  │
-│  │  ┌───────────────────────────────────────────────────┐  │  │
-│  │  │           RUN 3 TRIALS SEQUENTIALLY               │  │  │
-│  │  │                                                   │  │  │
-│  │  │  Trial 1 ──► Trial 2 ──► Trial 3 ──► Results      │  │  │
-│  │  │  (Sequential - Cline instance handles one at a time)  │  │
-│  │  │                                                   │  │  │
-│  │  │  Each trial:                                      │  │  │
-│  │  │  1. Create workspace-trial-N/                     │  │  │
-│  │  │  2. Copy template files (if any)                  │  │  │
-│  │  │  3. Run: cline -y -o "prompt"                     │  │  │
-│  │  │  4. Verify expected files exist                   │  │  │
-│  │  │  5. Verify expected content                       │  │  │
-│  │  │  6. Save trial-N.log                              │  │  │
-│  │  │  7. If failed, copy workspace to results/         │  │  │
-│  │  └───────────────────────────────────────────────────┘  │  │
-│  │                         │                               │  │
-│  │                         ▼                               │  │
-│  │  ┌───────────────────────────────────────────────────┐  │  │
-│  │  │ Calculate metrics: pass@1, pass@3, pass^3, flaky  │  │  │
-│  │  └───────────────────────────────────────────────────┘  │  │
-│  └─────────────────────────────────────────────────────────┘  │
-└───────────────────────────────────────────────────────────────┘
-         │
-         ▼
-┌───────────────────┐
-│ Generate outputs  │
-│ - report.json     │
-│ - summary.md      │
-│ - latest symlink  │
-└───────────────────┘
-```
-
-## Models Tested
-
-```
-┌─────────────────────────────────────────────────────────────────┐
-│                    CLINE PROVIDER ROUTING                       │
-├─────────────────────────────────────────────────────────────────┤
-│                                                                 │
-│   ┌─────────────┐                                               │
-│   │ Smoke Test  │                                               │
-│   │   Runner    │                                               │
-│   └──────┬──────┘                                               │
-│          │                                                      │
-│          │  cline -y -o "prompt" --model <model>                │
-│          │                                                      │
-│          ▼                                                      │
-│   ┌─────────────┐                                               │
-│   │   Cline     │                                               │
-│   │  Provider   │ ◄─── Uses your Cline auth (cline auth)        │
-│   └──────┬──────┘                                               │
-│          │                                                      │
-│          │ Routes to backend                                    │
-│          ▼                                                      │
-│   ┌─────────────────────────────────────────────────────────┐   │
-│   │                   Default Models                        │   │
-│   ├─────────────────────────────────────────────────────────┤   │
-│   │  claude-sonnet-4-20250514                               │   │
-│   │  gpt-4o                                                 │   │
-│   │  gemini-2.5-pro-preview-06-05                           │   │
-│   └─────────────────────────────────────────────────────────┘   │
-│                                                                 │
-└─────────────────────────────────────────────────────────────────┘
-```
-
-## Metrics Explained
-
-```
-┌─────────────────────────────────────────────────────────────────┐
-│                        METRICS                                  │
-├─────────────────────────────────────────────────────────────────┤
-│                                                                 │
-│  pass@k   "What's the probability of getting at least one      │
-│            success if I run k trials?"                          │
-│                                                                 │
-│            Example: 2/3 trials pass → pass@3 ≈ 96%              │
-│            (Very likely to pass if you run 3 times)             │
-│                                                                 │
-├─────────────────────────────────────────────────────────────────┤
-│                                                                 │
-│  pass^k   "What's the probability of ALL k trials succeeding?" │
-│                                                                 │
-│            Example: 2/3 trials pass → pass^3 ≈ 30%              │
-│            (Only 30% chance all 3 would pass)                   │
-│                                                                 │
-├─────────────────────────────────────────────────────────────────┤
-│                                                                 │
-│  Status   PASS  = All trials passed                             │
-│           FLAKY = Some passed, some failed                      │
-│           FAIL  = All trials failed                             │
-│                                                                 │
-└─────────────────────────────────────────────────────────────────┘
-```
-
-## Quick Commands
-
-```bash
-# Run all smoke tests (all models, 3 trials each)
-npm run eval:smoke
-
-# Run single model (use exact model ID for reproducibility)
-npm run eval:smoke -- --model claude-sonnet-4-20250514
-
-# Run single scenario
-npm run eval:smoke -- --scenario 01-create-file
-
-# Quick check (1 trial)
-npm run eval:smoke -- --trials 1
-
-# CI-like run (builds CLI from source, single trial)
-npm run eval:smoke:ci
-
-# View latest results
-cat evals/smoke-tests/results/latest/summary.md
-
-# Debug a failure
-cat evals/smoke-tests/results/latest/<scenario>/<model>/trial-1.log
-ls evals/smoke-tests/results/latest/<scenario>/<model>/workspace-trial-1/
-```
-
-## CI Integration
-
-Smoke tests run automatically on merge to `main` via `.github/workflows/cline-evals-regression.yml`.
-
-**Triggers:**
-- Push to `main` branch (paths: `src/core/**`, `src/shared/**`, `proto/**`)
-- Manual dispatch via `workflow_dispatch`
-
-**What it does:**
-1. Builds the Go CLI from source via `scripts/run-smoke-tests.sh`
-2. Runs all 5 scenarios × 3 models × 1 trial
-3. Uploads results as artifact
-4. Posts summary to GitHub Actions job summary
-
-```bash
-# The CI runs this script which handles proto generation + CLI build:
-bash scripts/run-smoke-tests.sh --trials 1
-```
-
-### Viewing CI Results
-
-1. **Job Summary**: Each run posts results to the Actions tab
-2. **Artifacts**: Full results downloadable as `smoke-test-results-<run_id>`
-
-### Running CI-like Tests Locally
-
-```bash
-# One command - builds CLI from source and runs tests
-npm run eval:smoke:ci
-
-# Or manually:
-npm run protos-go
-cd cli && go build -o cline ./cmd/cline
-export PATH="$(pwd)/cli:$PATH"
-npx tsx evals/smoke-tests/run-smoke-tests.ts --trials 1
-```
-
-### Why Build CLI in CI?
-
-We build the Go CLI from source rather than using a pre-built release because:
-- Tests actual CLI code from the commit (catches CLI regressions)
-- Proto definitions may have changed
-- No dependency on external releases
-
-## Contract Tests (Layer 1)
-
-> **Note**: The old `evals/benchmarks/tool-precision/` tests have been removed. Their functionality is now covered by contract tests in `src/core/**/__tests__/` and the 52 system prompt snapshot tests that run with `npm run test:unit`.
-
-Located in `src/core/api/transform/__tests__/`:
-
-```
-thinking-traces.test.ts
-├── convertToOpenAiMessages preserves reasoning_details
-├── convertToAnthropicMessage preserves thinking blocks
-└── sanitizeGeminiMessages handles provider-specific cleaning
-
-tool-parsing.test.ts
-├── Anthropic tool_use → OpenAI tool_calls conversion
-├── Tool call ID truncation (>40 chars)
-├── OpenAI Responses API ID transformation
-└── Tool result matching
-```
-
-Run with:
-```bash
-npm run test:unit -- --grep "Thinking Trace"  # 9 tests
-npm run test:unit -- --grep "Tool Call"       # 11 tests
-```

evals/README.md

@@ -1,149 +1,341 @@
-# Cline Evaluation Framework
+# Cline Evaluation System
 
-A layered testing system for measuring Cline's performance at different levels.
+This directory contains the evaluation system for benchmarking Cline against various coding evaluation frameworks.
+
+## Overview
+
+The Cline Evaluation System allows you to:
+
+1. Run Cline against standardized coding benchmarks
+2. Collect comprehensive metrics on performance
+3. Generate detailed reports on evaluation results
+4. Compare performance across different models and benchmarks
+
+## Architecture
+
+The evaluation system consists of two main components:
+
+1. **CLI Tool**: Command-line interface in `evals/cli/` for orchestrating evaluations
+2. **Diff Edit Benchmark**: Separate command using the CLI tool that runs a comprehensive diff editing benchmark suite on real world cases, along with a streamlit dashboard displaying the results. For more details, see the Diff Edit Benchmark [README](./diff-edits/README.md). Make sure you add a `evals/diff-edits/cases` folder with all the conversation jsons.
 
 ## Directory Structure
 
 ```
-evals/
-├── smoke-tests/           # Quick provider validation (minutes)
-│   ├── run-smoke-tests.ts
-│   └── scenarios/         # 5 curated test scenarios
-│
-├── e2e/                   # Full E2E with cline-bench (hours)
-│   └── run-cline-bench.ts
-│
-├── cline-bench/           # Real-world tasks (git submodule)
-│   └── tasks/             # 12 production bug fixes
-│
-├── analysis/              # Metrics and reporting framework
-│   ├── src/
-│   │   ├── metrics.ts     # pass@k, pass^k calculations
-│   │   ├── classifier.ts  # Failure pattern matching
-│   │   └── reporters/     # Markdown, JSON output
-│   └── patterns/
-│       └── cline-failures.yaml
-│
-└── baselines/             # Performance baselines for regression detection
+evals/                            # Main directory for evaluation system
+├── cli/                          # CLI tool for orchestrating evaluations
+│   └── src/
+│       ├── index.ts              # CLI entry point
+│       ├── commands/             # CLI commands (setup, run, report)
+│       ├── adapters/             # Benchmark adapters
+│       ├── db/                   # Database management
+│       └── utils/                # Utility functions
+├── diff-edits/                   # Diff editing evaluation suite
+│   ├── cases/                    # Test case JSON files
+│   ├── results/                  # Evaluation results
+│   ├── diff-apply/               # Diff application logic
+│   ├── parsing/                  # Assistant message parsing
+│   └── prompts/                  # System prompts
+├── repositories/                 # Cloned benchmark repositories
+│   └── exercism/                 # Exercism (Aider Polyglot)
+├── results/                      # Evaluation results storage
+│   ├── runs/                     # Individual run results
+│   └── reports/                  # Generated reports
+└── README.md                     # This file
 ```
 
-## Test Layers
+## Getting Started
 
-### Layer 1: Contract Tests (Unit)
+### Prerequisites
 
-Location: `src/core/api/transform/__tests__/`
+- Node.js 16+
+- VSCode with Cline extension installed
+- Git
 
-Tests API transform logic without LLM calls:
-- Thinking trace preservation
-- Tool call parsing (XML, native formats)
-- Provider format conversions
+### Installation
+
+1. Build the CLI tool:
 
 ```bash
-npm run test:unit -- --grep "Thinking\|Tool Call"
+cd evals
+npm install
+npm run build:cli
 ```
 
-### Layer 2: Smoke Tests (Minutes)
+### Usage
 
-Location: `evals/smoke-tests/`
-
-Quick validation across providers with real LLM calls:
-- 5 curated scenarios
-- 3 trials per test for pass@k metrics
-- Runs via cline CLI with `-s` flags
+#### Setting Up Benchmarks
 
 ```bash
-# Set API key (Cline provider)
-export CLINE_API_KEY=sk-...
-
-# Run smoke tests
-npm run eval:smoke
-
-# Run specific scenario
-npm run eval:smoke -- --scenario 01-create-file
-
-# Run with specific model (overrides per-scenario models)
-npm run eval:smoke -- --model anthropic/claude-sonnet-4.5
+cd evals/cli
+node dist/index.js setup
 ```
 
-### Layer 3: E2E Tests (Hours)
-
-Location: `evals/e2e/` + `evals/cline-bench/`
-
-Full agent tests on production-grade tasks:
-- 12 real-world coding problems
-- Docker/Daytona execution via Harbor
-- Nightly CI runs
+This will clone and set up all benchmark repositories. You can specify specific benchmarks:
 
 ```bash
-# Prerequisites: Python 3.13, Harbor, Docker
-npm run eval:e2e
-
-# Specific task
-npm run eval:e2e -- --tasks discord
-
-# Different provider
-npm run eval:e2e -- --provider openai --model gpt-4o
+node dist/index.js setup --benchmarks exercism
 ```
 
+#### Running Evaluations
+
+```bash
+node dist/index.js run --benchmark exercism --count 10
+```
+
+Options:
+- `--benchmark`: Specific benchmark to run (default: exercism)
+- `--count`: Number of tasks to run (default: all available tasks)
+
+**Note:** Model selection is currently configured through the Cline CLI itself, not through evaluation flags.
+
+#### Generating Reports
+
+```bash
+node dist/index.js report
+```
+
+Options:
+- `--format`: Report format (json, markdown) (default: markdown)
+- `--output`: Output path for the report
+
+## Benchmarks
+
+### Exercism
+
+Modified Exercism exercises from the [polyglot-benchmark](https://github.com/Aider-AI/polyglot-benchmark) repository. These are small, focused programming exercises in various languages.
+
+### SWE-Bench (Coming Soon)
+
+Real-world software engineering tasks from the [SWE-bench](https://github.com/SWE-bench/SWE-bench) repository.
+
+### SWELancer (Coming Soon)
+
+Freelance-style programming tasks from the SWELancer benchmark.
+
+### Multi-SWE-Bench (Coming Soon)
+
+Multi-file software engineering tasks from the Multi-SWE-Bench repository.
+
+## Diff Edit Evaluations
+
+The Cline Evaluation System includes a specialized suite for evaluating how well models can make precise edits to files using the `replace_in_file` tool.
+
+### Overview
+
+Diff edit evaluations test a model's ability to:
+
+1. Understand file content and identify specific sections to modify
+2. Generate correct SEARCH/REPLACE blocks for targeted edits
+3. Successfully apply changes without introducing errors
+
+### Directory Structure
+
+```
+diff-edits/
+├── cases/                  # Test case JSON files
+├── results/                # Evaluation results
+├── ClineWrapper.ts         # Wrapper for model interaction
+├── TestRunner.ts           # Main test execution logic
+├── types.ts                # Type definitions
+├── diff-apply/             # Diff application logic
+├── parsing/                # Assistant message parsing
+└── prompts/                # System prompts
+```
+
+### Creating Test Cases
+
+Test cases are defined as JSON files in the `diff-edits/cases/` directory. Each test case should include:
+
+```json
+{
+  "test_id": "example_test_1",
+  "messages": [
+    {
+      "role": "user",
+      "text": "Please fix the bug in this code...",
+      "images": []
+    },
+    {
+      "role": "assistant",
+      "text": "I'll help you fix that bug..."
+    }
+  ],
+  "file_contents": "// Original file content here\nfunction example() {\n  // Code with bug\n}",
+  "file_path": "src/example.js",
+  "system_prompt_details": {
+    "mcp_string": "",
+    "cwd_value": "/path/to/working/directory",
+    "browser_use": false,
+    "width": 900,
+    "height": 600,
+    "os_value": "macOS",
+    "shell_value": "/bin/zsh",
+    "home_value": "/Users/username",
+    "user_custom_instructions": ""
+  },
+  "original_diff_edit_tool_call_message": ""
+}
+```
+
+### Running Diff Edit Evaluations
+
+#### Single Model Evaluation
+
+```bash
+cd evals/cli
+node dist/index.js run-diff-eval --model-ids "anthropic/claude-3-5-sonnet-20241022"
+```
+
+#### Multi-Model Evaluation
+
+Compare multiple models in a single evaluation run:
+
+```bash
+# Compare Claude and Grok models
+node dist/index.js run-diff-eval \
+  --model-ids "anthropic/claude-3-5-sonnet-20241022,x-ai/grok-beta" \
+  --max-cases 10 \
+  --valid-attempts-per-case 3 \
+  --verbose
+
+# Compare multiple Claude variants
+node dist/index.js run-diff-eval \
+  --model-ids "anthropic/claude-3-5-sonnet-20241022,anthropic/claude-3-5-haiku-20241022,anthropic/claude-3-opus-20240229" \
+  --max-cases 5 \
+  --valid-attempts-per-case 2 \
+  --parallel
+```
+
+#### Options
+
+- `--model-ids`: Comma-separated list of model IDs to evaluate (required)
+- `--system-prompt-name`: System prompt to use (default: "basicSystemPrompt")
+- `--valid-attempts-per-case`: Number of attempts per test case per model (default: 1)
+- `--max-cases`: Maximum number of test cases to run (default: all available)
+- `--parsing-function`: Function to parse assistant messages (default: "parseAssistantMessageV2")
+- `--diff-edit-function`: Function to apply diffs (default: "constructNewFileContentV2")
+- `--test-path`: Path to test cases (default: diff-edits/cases)
+- `--thinking-budget`: Tokens allocated for thinking (default: 0)
+- `--parallel`: Run tests in parallel (flag)
+- `--replay`: Use pre-recorded LLM output (flag)
+- `--verbose`: Enable detailed logging (flag)
+
+#### Examples
+
+```bash
+# Quick test with 2 models, 4 cases, 2 attempts each
+node dist/index.js run-diff-eval \
+  --model-ids "anthropic/claude-3-5-sonnet-20241022,x-ai/grok-beta" \
+  --max-cases 4 \
+  --valid-attempts-per-case 2 \
+  --verbose
+
+# Comprehensive evaluation with parallel execution
+node dist/index.js run-diff-eval \
+  --model-ids "anthropic/claude-3-5-sonnet-20241022,anthropic/claude-3-5-haiku-20241022" \
+  --system-prompt-name claude4SystemPrompt \
+  --valid-attempts-per-case 5 \
+  --max-cases 20 \
+  --parallel \
+  --verbose
+```
+
+### Database Storage & Analytics
+
+All evaluation results are automatically stored in a SQLite database (`diff-edits/evals.db`) for advanced analytics and comparison. The database includes:
+
+- **System Prompts**: Versioned system prompt content with hashing for deduplication
+- **Processing Functions**: Versioned parsing and diff-edit function configurations
+- **Files**: Original and edited file content with content-based hashing
+- **Runs**: Evaluation run metadata and configuration
+- **Cases**: Individual test case information with context tokens
+- **Results**: Detailed results with timing, cost, and success metrics
+
+### Interactive Dashboard
+
+Launch the Streamlit dashboard to visualize and analyze evaluation results:
+
+```bash
+cd diff-edits/dashboard
+streamlit run app.py
+```
+
+The dashboard provides:
+
+- **Model Performance Comparison**: Side-by-side comparison of success rates, latency, and costs
+- **Interactive Charts**: Success rate trends, latency vs cost analysis, and performance metrics
+- **Detailed Drill-Down**: Individual result analysis with file content viewing
+- **Run Selection**: Browse and compare different evaluation runs
+- **Real-time Updates**: Automatically refreshes with new evaluation data
+
+#### Dashboard Features
+
+1. **Hero Section**: Overview of current run with key metrics
+2. **Model Cards**: Performance cards with grades and detailed metrics
+3. **Comparison Charts**: Interactive Plotly charts for visual analysis
+4. **Result Explorer**: Detailed view of individual test results including:
+   - Original and edited file content
+   - Raw model output
+   - Parsed tool calls
+   - Timing and cost metrics
+   - Error analysis
+
+#### Quick Start Dashboard
+
+```bash
+# Run a quick evaluation
+node cli/dist/index.js run-diff-eval \
+  --model-ids "anthropic/claude-3-5-sonnet-20241022,x-ai/grok-beta" \
+  --max-cases 4 \
+  --valid-attempts-per-case 2 \
+  --verbose
+
+# Launch dashboard to view results
+cd diff-edits/dashboard && streamlit run app.py
+```
+
+### Legacy Results
+
+For backward compatibility, results are also saved as JSON files in the `diff-edits/results/` directory. The JSON results include:
+- Success/failure status
+- Extracted tool calls
+- Diff edit content
+- Token usage and cost metrics
+
 ## Metrics
 
-The framework calculates:
+The evaluation system collects the following metrics:
 
-| Metric | Formula | Interpretation |
-|--------|---------|----------------|
-| **pass@k** | P(≥1 of k passes) | Solution finding capability |
-| **pass^k** | P(all k pass) | Reliability |
-| **Flakiness** | Entropy of pass rate | Consistency |
+- **Token Usage**: Input and output tokens
+- **Cost**: Estimated cost of API calls
+- **Duration**: Time taken to complete tasks
+- **Tool Usage**: Number of tool calls and failures
+- **Success Rate**: Percentage of tasks completed successfully
+- **Test Success Rate**: Percentage of tests passed
+- **Functional Correctness**: Ratio of tests passed to total tests
 
-With 3 trials:
-- All pass → `pass` (reliable)
-- All fail → `fail` (broken)
-- Mixed → `flaky` (needs investigation)
+## Reports
 
-## CI Integration
+Reports are generated in Markdown or JSON format and include:
 
-- **PR Gate**: Contract tests + smoke tests (fast, ~3min)
-- **Nightly**: E2E tests with cline-bench (not yet implemented, see TODO)
+- Overall summary
+- Benchmark-specific results
+- Model-specific results
+- Tool usage statistics
+- Charts and visualizations
 
-## Quick Start
+## Development
 
-```bash
-# Run all fast tests
-npm run test:unit
-npm run eval:smoke
+### Adding a New Benchmark
 
-# Run E2E (requires setup)
-cd evals/cline-bench
-# Follow README.md for Harbor setup
-npm run eval:e2e
-```
+1. Create a new adapter in `evals/cli/src/adapters/`
+2. Implement the `BenchmarkAdapter` interface
+3. Register the adapter in `evals/cli/src/adapters/index.ts`
 
-## Adding Tests
+### Extending Metrics
 
-### Smoke Test Scenario
+To add new metrics:
 
-1. Create `evals/smoke-tests/scenarios/<name>/config.json`
-2. Add optional `template/` directory with starting files
-3. Run to verify: `npm run eval:smoke -- --scenario <name>`
-
-### Contract Test
-
-1. Add to `src/core/api/transform/__tests__/`
-2. Run: `npm run test:unit -- --grep "YourTest"`
-
-### E2E Task
-
-Contribute to [cline/cline-bench](https://github.com/cline/cline-bench)
-
-## Resources
-
-- [cline-bench tasks](evals/cline-bench/README.md)
-- [Smoke test scenarios](evals/smoke-tests/README.md)
-
-## TODO
-
-- [ ] **Nightly E2E CI**: Add scheduled workflow for cline-bench tests
-  - Requires: Docker runner, Harbor setup, ~1-2 hour timeout
-  - Should run on schedule (e.g., nightly) not per-PR
-  - Separate secrets for E2E environment
-- [ ] **Native tool calling smoke tests**: Add CLI support for `native_tool_call_enabled` setting to test Claude 4 with native tools
+1. Update the database schema in `evals/cli/src/db/schema.ts`
+2. Add collection logic in `evals/cli/src/utils/results.ts`
+3. Update report generation in `evals/cli/src/commands/report.ts`

evals/analysis/package.json

@@ -1,34 +0,0 @@
-{
-	"name": "@cline/analysis",
-	"version": "1.0.0",
-	"description": "Analysis framework for Cline evaluations with failure classification and metrics",
-	"type": "module",
-	"main": "dist/index.js",
-	"types": "dist/index.d.ts",
-	"scripts": {
-		"start": "tsx src/cli.ts",
-		"build": "tsc",
-		"test": "vitest run",
-		"test:watch": "vitest",
-		"test:ui": "vitest --ui"
-	},
-	"keywords": [
-		"cline",
-		"evaluation",
-		"benchmarking",
-		"ai-testing",
-		"metrics"
-	],
-	"dependencies": {
-		"commander": "^12.0.0",
-		"js-yaml": "^4.1.0",
-		"chalk": "^5.0.0"
-	},
-	"devDependencies": {
-		"@types/node": "^20.0.0",
-		"@types/js-yaml": "^4.0.0",
-		"tsx": "^4.0.0",
-		"typescript": "^5.0.0",
-		"vitest": "^1.0.0"
-	}
-}

evals/analysis/patterns/cline-failures.yaml

@@ -1,57 +0,0 @@
-# Cline-specific failure patterns for classification
-# Version 1.0
-
-version: "1.0"
-
-patterns:
-  # Provider-specific bugs (Cline integration issues)
-  - name: "gemini_signature"
-    pattern: "missing.?signature|thoughtSignature"
-    category: "provider_bug"
-    issue: "https://github.com/cline/cline/issues/7974"
-    description: "Gemini 3 Pro requires thoughtSignature for native tool calls"
-
-  - name: "claude_tool_format"
-    pattern: "write_to_file.*missing.*content|content.*parameter.*required"
-    category: "provider_bug"
-    issue: "https://github.com/cline/cline/issues/7998"
-    description: "Claude tool parameter extraction failure"
-
-  # Transient failures (retriable)
-  - name: "rate_limit"
-    pattern: "429|rate.?limit|too.?many.?requests|quota.?exceeded"
-    category: "transient"
-    description: "API rate limiting"
-
-  - name: "network_timeout"
-    pattern: "ECONNREFUSED|ETIMEDOUT|ENOTFOUND|timed.?out"
-    category: "transient"
-    description: "Network connectivity issues"
-
-  - name: "model_overloaded"
-    pattern: "503|service.?unavailable|overloaded"
-    category: "transient"
-    description: "Provider service unavailable"
-
-  # Infrastructure/harness failures
-  - name: "harness_error"
-    pattern: "verifier.*failed|test.*harness.*error|missing.*test.*file"
-    category: "harness"
-    description: "Test harness or verification script failure"
-
-  - name: "environment_failure"
-    pattern: "docker.*failed|container.*exit|OCI.*runtime|pod.*error"
-    category: "environment"
-    description: "Docker/Daytona environment setup failure"
-
-  # Policy/safety failures
-  - name: "safety_refusal"
-    pattern: "content.*policy|safety.*filter|inappropriate.*request"
-    category: "policy"
-    description: "Model refused due to safety/content policy"
-
-  # Auth issues (non-retriable)
-  - name: "auth_error"
-    pattern: "401|unauthorized|invalid.?api.?key"
-    category: "auth"
-    description: "Invalid API credentials"

evals/analysis/src/__tests__/classifier.test.ts

@@ -1,180 +0,0 @@
-import { describe, expect, it } from "vitest"
-import { FailureClassifier } from "../classifier"
-
-describe("FailureClassifier", () => {
-	const classifier = new FailureClassifier()
-
-	describe("Provider Bug Detection", () => {
-		it("detects Gemini signature issue", () => {
-			const logs = `
-				Error: Function call is missing a thought_signature in functionCall parts.
-				This is required for tools to work correctly with Gemini 3 Pro...
-			`
-			const failures = classifier.classify(logs)
-
-			expect(failures.length).toBeGreaterThan(0)
-			expect(failures[0].name).toBe("gemini_signature")
-			expect(failures[0].category).toBe("provider_bug")
-			expect(failures[0].issue_url).toBe("https://github.com/cline/cline/issues/7974")
-		})
-
-		it("detects Claude tool format issue", () => {
-			const logs = `Cline tried to use write_to_file without value for required parameter 'content'`
-			const failures = classifier.classify(logs)
-
-			expect(failures.length).toBeGreaterThan(0)
-			expect(failures[0].name).toBe("claude_tool_format")
-			expect(failures[0].category).toBe("provider_bug")
-			expect(failures[0].issue_url).toBe("https://github.com/cline/cline/issues/7998")
-		})
-	})
-
-	describe("Transient Failure Detection", () => {
-		it("detects rate limiting", () => {
-			const logs = "Error: 429 Too Many Requests - Rate limit exceeded"
-			const failures = classifier.classify(logs)
-
-			expect(failures.length).toBeGreaterThan(0)
-			expect(failures[0].name).toBe("rate_limit")
-			expect(failures[0].category).toBe("transient")
-		})
-
-		it("detects network timeout", () => {
-			const logs = "Error: ETIMEDOUT - Connection timed out"
-			const failures = classifier.classify(logs)
-
-			expect(failures.length).toBeGreaterThan(0)
-			expect(failures[0].name).toBe("network_timeout")
-			expect(failures[0].category).toBe("transient")
-		})
-
-		it("detects service unavailable", () => {
-			const logs = "503 Service Unavailable - Model is currently overloaded"
-			const failures = classifier.classify(logs)
-
-			expect(failures.length).toBeGreaterThan(0)
-			expect(failures[0].name).toBe("model_overloaded")
-			expect(failures[0].category).toBe("transient")
-		})
-	})
-
-	describe("Infrastructure Failure Detection", () => {
-		it("detects harness errors", () => {
-			const logs = "verifier script failed with exit code 1"
-			const failures = classifier.classify(logs)
-
-			expect(failures.length).toBeGreaterThan(0)
-			expect(failures[0].name).toBe("harness_error")
-			expect(failures[0].category).toBe("harness")
-		})
-
-		it("detects environment failures", () => {
-			const logs = "Error: docker container exit code 137"
-			const failures = classifier.classify(logs)
-
-			expect(failures.length).toBeGreaterThan(0)
-			expect(failures[0].name).toBe("environment_failure")
-			expect(failures[0].category).toBe("environment")
-		})
-	})
-
-	describe("Policy and Auth Failures", () => {
-		it("detects safety refusals", () => {
-			const logs = "Request blocked: Content policy violation"
-			const failures = classifier.classify(logs)
-
-			expect(failures.length).toBeGreaterThan(0)
-			expect(failures[0].name).toBe("safety_refusal")
-			expect(failures[0].category).toBe("policy")
-		})
-
-		it("detects auth errors", () => {
-			const logs = "401 Unauthorized: Invalid API key"
-			const failures = classifier.classify(logs)
-
-			expect(failures.length).toBeGreaterThan(0)
-			expect(failures[0].name).toBe("auth_error")
-			expect(failures[0].category).toBe("auth")
-		})
-	})
-
-	describe("Excerpt Extraction", () => {
-		it("extracts context around the matched pattern", () => {
-			const logs = `
-				This is some context before the error.
-				Error: 429 Too Many Requests - Rate limit exceeded
-				This is some context after the error.
-			`
-			const failures = classifier.classify(logs)
-
-			expect(failures[0].excerpt).toContain("Rate limit exceeded")
-			expect(failures[0].excerpt.length).toBeLessThan(500)
-		})
-	})
-
-	describe("Helper Methods", () => {
-		it("hasProviderBug returns true for provider bugs", () => {
-			const logs = "Error: missing thoughtSignature in function call"
-			expect(classifier.hasProviderBug(logs)).toBe(true)
-		})
-
-		it("hasProviderBug returns false for non-provider bugs", () => {
-			const logs = "Error: 429 Too Many Requests"
-			expect(classifier.hasProviderBug(logs)).toBe(false)
-		})
-
-		it("hasTransientFailure returns true for transient errors", () => {
-			const logs = "Error: ETIMEDOUT"
-			expect(classifier.hasTransientFailure(logs)).toBe(true)
-		})
-
-		it("hasTransientFailure returns false for non-transient errors", () => {
-			const logs = "Error: missing thoughtSignature"
-			expect(classifier.hasTransientFailure(logs)).toBe(false)
-		})
-
-		it("getPatternsByCategory returns correct patterns", () => {
-			const providerBugs = classifier.getPatternsByCategory("provider_bug")
-			expect(providerBugs).toContain("gemini_signature")
-			expect(providerBugs).toContain("claude_tool_format")
-
-			const transient = classifier.getPatternsByCategory("transient")
-			expect(transient).toContain("rate_limit")
-			expect(transient).toContain("network_timeout")
-			expect(transient).toContain("model_overloaded")
-		})
-	})
-
-	describe("Multiple Pattern Matching", () => {
-		it("detects multiple failures in same log", () => {
-			const logs = `
-				Error: 429 Too Many Requests
-				Later: Error: ETIMEDOUT
-			`
-			const failures = classifier.classify(logs)
-
-			expect(failures.length).toBe(2)
-			expect(failures.map((f) => f.name)).toContain("rate_limit")
-			expect(failures.map((f) => f.name)).toContain("network_timeout")
-		})
-	})
-
-	describe("Case Insensitivity", () => {
-		it("matches patterns case-insensitively", () => {
-			const logs = "error: RATE LIMIT exceeded"
-			const failures = classifier.classify(logs)
-
-			expect(failures.length).toBeGreaterThan(0)
-			expect(failures[0].name).toBe("rate_limit")
-		})
-	})
-
-	describe("No Match", () => {
-		it("returns empty array when no patterns match", () => {
-			const logs = "Everything completed successfully"
-			const failures = classifier.classify(logs)
-
-			expect(failures).toEqual([])
-		})
-	})
-})

evals/analysis/src/__tests__/metrics.test.ts

@@ -1,249 +0,0 @@
-import { describe, expect, it } from "vitest"
-import { MetricsCalculator } from "../metrics"
-
-describe("MetricsCalculator", () => {
-	const calc = new MetricsCalculator()
-
-	describe("pass@k (solution finding)", () => {
-		it("calculates 100% when at least k trials pass", () => {
-			expect(calc.passAtK([true, true, true], 1)).toBe(1.0)
-			expect(calc.passAtK([true, true, false], 1)).toBe(1.0)
-			expect(calc.passAtK([true, true, true], 3)).toBe(1.0)
-		})
-
-		it("calculates 0% when fewer than k trials pass", () => {
-			expect(calc.passAtK([false, false, false], 1)).toBe(0.0)
-		})
-
-		it("calculates correct probability for mixed results", () => {
-			// With n=3, c=2, k=2: 1 - C(1,2)/C(3,2) = 1 - 0/3 = 1.0
-			expect(calc.passAtK([true, true, false], 2)).toBe(1.0)
-
-			// With n=3, c=1, k=2: 1 - C(2,2)/C(3,2) = 1 - 1/3 = 2/3
-			expect(calc.passAtK([true, false, false], 2)).toBeCloseTo(0.6667, 4)
-		})
-
-		it("throws error when k > n", () => {
-			expect(() => calc.passAtK([true, false], 3)).toThrow()
-		})
-
-		it("handles k=1 correctly (most common case)", () => {
-			expect(calc.passAtK([true, false, false], 1)).toBe(1.0)
-			expect(calc.passAtK([false, false, false], 1)).toBe(0.0)
-		})
-
-		it("handles all-pass scenarios", () => {
-			expect(calc.passAtK([true, true, true, true, true], 3)).toBe(1.0)
-			expect(calc.passAtK([true, true, true, true, true], 5)).toBe(1.0)
-		})
-
-		it("handles all-fail scenarios", () => {
-			expect(calc.passAtK([false, false, false], 1)).toBe(0.0)
-			expect(calc.passAtK([false, false, false], 3)).toBe(0.0)
-		})
-	})
-
-	describe("pass^k (reliability)", () => {
-		it("calculates 100% when all k trials must and do pass", () => {
-			expect(calc.passCaretK([true, true, true], 3)).toBe(1.0)
-			expect(calc.passCaretK([true, true, true, true], 3)).toBeCloseTo(1.0, 4)
-		})
-
-		it("calculates 0% when fewer than k trials pass", () => {
-			expect(calc.passCaretK([true, true, false], 3)).toBe(0.0)
-			expect(calc.passCaretK([true, false, false], 2)).toBe(0.0)
-			expect(calc.passCaretK([false, false, false], 1)).toBe(0.0)
-		})
-
-		it("calculates correct probability for sufficient passes", () => {
-			// With n=4, c=3, k=2: C(3,2)/C(4,2) = 3/6 = 0.5
-			expect(calc.passCaretK([true, true, true, false], 2)).toBeCloseTo(0.5, 4)
-
-			// With n=5, c=3, k=2: C(3,2)/C(5,2) = 3/10 = 0.3
-			expect(calc.passCaretK([true, true, true, false, false], 2)).toBeCloseTo(0.3, 4)
-		})
-
-		it("throws error when k > n", () => {
-			expect(() => calc.passCaretK([true, false], 3)).toThrow()
-		})
-
-		it("diverges from pass@k as trials increase", () => {
-			const trials = [true, true, false, false, false]
-
-			// pass@k increases (eventually finds solution)
-			const passAt1 = calc.passAtK(trials, 1)
-			const passAt3 = calc.passAtK(trials, 3)
-			expect(passAt3).toBeGreaterThanOrEqual(passAt1)
-
-			// pass^k decreases (reliability drops)
-			const passCaret1 = calc.passCaretK(trials, 1)
-			const passCaret3 = calc.passCaretK(trials, 3)
-			expect(passCaret3).toBeLessThanOrEqual(passCaret1)
-		})
-	})
-
-	describe("flakinessScore (variance)", () => {
-		it("returns 0 for all-pass scenarios", () => {
-			expect(calc.flakinessScore([true, true, true])).toBe(0)
-		})
-
-		it("returns 0 for all-fail scenarios", () => {
-			expect(calc.flakinessScore([false, false, false])).toBe(0)
-		})
-
-		it("returns 1 for maximum variance (50% pass rate)", () => {
-			expect(calc.flakinessScore([true, false])).toBe(1)
-			expect(calc.flakinessScore([true, true, false, false])).toBe(1)
-		})
-
-		it("returns values between 0 and 1 for partial variance", () => {
-			const score1 = calc.flakinessScore([true, true, true, false])
-			expect(score1).toBeGreaterThan(0)
-			expect(score1).toBeLessThan(1)
-
-			const score2 = calc.flakinessScore([true, false, false, false])
-			expect(score2).toBeGreaterThan(0)
-			expect(score2).toBeLessThan(1)
-		})
-
-		it("symmetric around 50% pass rate", () => {
-			const score25 = calc.flakinessScore([true, false, false, false])
-			const score75 = calc.flakinessScore([true, true, true, false])
-			expect(score25).toBeCloseTo(score75, 4)
-		})
-
-		it("higher variance for rates closer to 50%", () => {
-			const score25 = calc.flakinessScore([true, false, false, false])
-			const score50 = calc.flakinessScore([true, true, false, false])
-			expect(score50).toBeGreaterThan(score25)
-		})
-	})
-
-	describe("binomial coefficient", () => {
-		it("calculates C(n, 0) = 1", () => {
-			expect(calc["binomial"](5, 0)).toBe(1)
-		})
-
-		it("calculates C(n, n) = 1", () => {
-			expect(calc["binomial"](5, 5)).toBe(1)
-		})
-
-		it("calculates C(n, 1) = n", () => {
-			expect(calc["binomial"](5, 1)).toBe(5)
-		})
-
-		it("calculates C(n, k) correctly", () => {
-			expect(calc["binomial"](5, 2)).toBe(10)
-			expect(calc["binomial"](6, 3)).toBe(20)
-			expect(calc["binomial"](10, 3)).toBe(120)
-		})
-
-		it("returns 0 when k > n", () => {
-			expect(calc["binomial"](3, 5)).toBe(0)
-		})
-
-		it("optimizes by using smaller k", () => {
-			// C(10, 8) = C(10, 2) = 45
-			expect(calc["binomial"](10, 8)).toBe(45)
-			expect(calc["binomial"](10, 2)).toBe(45)
-		})
-	})
-
-	describe("calculateTaskMetrics", () => {
-		it("calculates all metrics for 3 trials", () => {
-			const metrics = calc.calculateTaskMetrics([true, true, false])
-
-			expect(metrics.passAt1).toBe(1.0)
-			expect(metrics.passAt3).toBeGreaterThan(0)
-			expect(metrics.passCaret3).toBe(0.0)
-			expect(metrics.flakinessScore).toBeGreaterThan(0)
-		})
-
-		it("calculates all metrics for perfect pass", () => {
-			const metrics = calc.calculateTaskMetrics([true, true, true])
-
-			expect(metrics.passAt1).toBe(1.0)
-			expect(metrics.passAt3).toBe(1.0)
-			expect(metrics.passCaret3).toBe(1.0)
-			expect(metrics.flakinessScore).toBe(0)
-		})
-
-		it("calculates all metrics for perfect fail", () => {
-			const metrics = calc.calculateTaskMetrics([false, false, false])
-
-			expect(metrics.passAt1).toBe(0.0)
-			expect(metrics.passAt3).toBe(0.0)
-			expect(metrics.passCaret3).toBe(0.0)
-			expect(metrics.flakinessScore).toBe(0)
-		})
-
-		it("throws error for empty trials", () => {
-			expect(() => calc.calculateTaskMetrics([])).toThrow()
-		})
-
-		it("handles fewer than 3 trials gracefully", () => {
-			const metrics = calc.calculateTaskMetrics([true, false])
-
-			expect(metrics.passAt1).toBe(1.0)
-			expect(metrics.passAt3).toBe(0) // Not enough trials
-			expect(metrics.passCaret3).toBe(0)
-			expect(metrics.flakinessScore).toBe(1)
-		})
-	})
-
-	describe("getTaskStatus", () => {
-		it("returns 'pass' when all trials pass", () => {
-			expect(calc.getTaskStatus([true, true, true])).toBe("pass")
-		})
-
-		it("returns 'fail' when all trials fail", () => {
-			expect(calc.getTaskStatus([false, false, false])).toBe("fail")
-		})
-
-		it("returns 'flaky' when some trials pass and some fail", () => {
-			expect(calc.getTaskStatus([true, false, false])).toBe("flaky")
-			expect(calc.getTaskStatus([true, true, false])).toBe("flaky")
-		})
-
-		it("handles single trial", () => {
-			expect(calc.getTaskStatus([true])).toBe("pass")
-			expect(calc.getTaskStatus([false])).toBe("fail")
-		})
-	})
-
-	describe("Real-world scenarios", () => {
-		it("handles typical cline-bench results", () => {
-			// Scenario: Task passed 2/3 times
-			const trials = [true, true, false]
-			const metrics = calc.calculateTaskMetrics(trials)
-
-			expect(metrics.passAt1).toBe(1.0) // Found solution
-			expect(metrics.passAt3).toBeGreaterThan(0.5) // Likely to solve
-			expect(metrics.passCaret3).toBe(0) // Not reliable
-			expect(metrics.flakinessScore).toBeGreaterThan(0) // Has variance
-			expect(calc.getTaskStatus(trials)).toBe("flaky")
-		})
-
-		it("handles consistent success", () => {
-			const trials = [true, true, true]
-			const metrics = calc.calculateTaskMetrics(trials)
-
-			expect(metrics.passAt1).toBe(1.0)
-			expect(metrics.passAt3).toBe(1.0)
-			expect(metrics.passCaret3).toBe(1.0)
-			expect(metrics.flakinessScore).toBe(0)
-			expect(calc.getTaskStatus(trials)).toBe("pass")
-		})
-
-		it("handles consistent failure", () => {
-			const trials = [false, false, false]
-			const metrics = calc.calculateTaskMetrics(trials)
-
-			expect(metrics.passAt1).toBe(0)
-			expect(metrics.passAt3).toBe(0)
-			expect(metrics.passCaret3).toBe(0)
-			expect(metrics.flakinessScore).toBe(0)
-			expect(calc.getTaskStatus(trials)).toBe("fail")
-		})
-	})
-})

evals/analysis/src/classifier.ts

@@ -1,124 +0,0 @@
-/**
- * Failure classification system for Cline evaluations
- *
- * Classifies failures by matching log patterns against known issues:
- * - Provider bugs (Gemini #7974, Claude #7998)
- * - Transient failures (rate limits, timeouts)
- * - Infrastructure issues (harness, environment)
- * - Policy/safety refusals
- * - Auth errors
- */
-
-import * as fs from "fs"
-import * as yaml from "js-yaml"
-import * as path from "path"
-import type { FailureCategory, FailureInfo } from "./schemas"
-
-export interface FailurePattern {
-	name: string
-	pattern: string // Regex pattern as string
-	category: FailureCategory
-	issue?: string // GitHub issue URL
-	description: string
-}
-
-export interface FailurePatternsConfig {
-	version: string
-	patterns: FailurePattern[]
-}
-
-export class FailureClassifier {
-	private patterns: Array<FailurePattern & { regex: RegExp }>
-
-	constructor(patternsPath?: string) {
-		const defaultPath = path.join(__dirname, "../patterns/cline-failures.yaml")
-		const configPath = patternsPath || defaultPath
-
-		const config = this.loadPatternsFromYaml(configPath)
-		this.patterns = config.patterns.map((p) => ({
-			...p,
-			regex: new RegExp(p.pattern, "i"), // Case-insensitive matching
-		}))
-	}
-
-	private loadPatternsFromYaml(filePath: string): FailurePatternsConfig {
-		const content = fs.readFileSync(filePath, "utf-8")
-		const config = yaml.load(content) as FailurePatternsConfig
-
-		if (!config.version || !config.patterns) {
-			throw new Error("Invalid patterns YAML: missing version or patterns")
-		}
-
-		return config
-	}
-
-	/**
-	 * Classify failures in log text
-	 * @param logs Full log text (e.g., cline.txt content)
-	 * @returns Array of matched failure categories with excerpts
-	 */
-	classify(logs: string): FailureInfo[] {
-		const failures: FailureInfo[] = []
-
-		for (const pattern of this.patterns) {
-			const match = pattern.regex.exec(logs)
-			if (match) {
-				failures.push({
-					name: pattern.name,
-					category: pattern.category,
-					excerpt: this.extractExcerpt(logs, match.index, match[0].length),
-					issue_url: pattern.issue,
-				})
-			}
-		}
-
-		return failures
-	}
-
-	/**
-	 * Extract a context snippet around the matched pattern
-	 * @param logs Full log text
-	 * @param matchIndex Index where pattern matched
-	 * @param matchLength Length of the matched text
-	 * @returns Context snippet (up to 200 chars before/after match)
-	 */
-	private extractExcerpt(logs: string, matchIndex: number, matchLength: number): string {
-		const contextSize = 200
-		const start = Math.max(0, matchIndex - contextSize)
-		const end = Math.min(logs.length, matchIndex + matchLength + contextSize)
-
-		let excerpt = logs.slice(start, end)
-
-		// Trim to complete lines for readability
-		excerpt = excerpt.replace(/^\s*\S*\s*/, "") // Remove partial first line
-		excerpt = excerpt.replace(/\s*\S*\s*$/, "") // Remove partial last line
-
-		// Truncate if still too long
-		if (excerpt.length > 400) {
-			excerpt = excerpt.slice(0, 400) + "..."
-		}
-
-		return excerpt.trim()
-	}
-
-	/**
-	 * Check if logs contain any known provider bug patterns
-	 */
-	hasProviderBug(logs: string): boolean {
-		return this.classify(logs).some((f) => f.category === "provider_bug")
-	}
-
-	/**
-	 * Check if logs contain transient failure patterns (retriable)
-	 */
-	hasTransientFailure(logs: string): boolean {
-		return this.classify(logs).some((f) => f.category === "transient")
-	}
-
-	/**
-	 * Get all pattern names for a specific category
-	 */
-	getPatternsByCategory(category: FailureCategory): string[] {
-		return this.patterns.filter((p) => p.category === category).map((p) => p.name)
-	}
-}

evals/analysis/src/cli.ts

@@ -1,240 +0,0 @@
-#!/usr/bin/env node
-
-/**
- * Cline Analysis Framework CLI
- *
- * Commands:
- * - analyze: Parse Harbor job output and generate reports
- * - compare: Compare baseline vs current results for regression detection
- */
-
-import chalk from "chalk"
-import { Command } from "commander"
-import * as fs from "fs"
-import { HarborParser } from "./parsers"
-import { JsonReporter, MarkdownReporter } from "./reporters"
-import type { AnalysisOutputV1, ComparisonResult } from "./schemas"
-
-const program = new Command()
-
-program.name("cline-analysis").description("Analysis framework for Cline evaluations").version("1.0.0")
-
-// Analyze command
-program
-	.command("analyze <job-dir>")
-	.description("Parse Harbor job output and generate analysis report")
-	.option("-f, --format <format>", "Output format: markdown, json, or minimal", "markdown")
-	.option("-o, --output <file>", "Write report to file (default: stdout)")
-	.option("--no-color", "Disable colored output")
-	.action(async (jobDir: string, options: any) => {
-		try {
-			// Validate job directory
-			if (!fs.existsSync(jobDir)) {
-				console.error(chalk.red(`Error: Job directory not found: ${jobDir}`))
-				process.exit(1)
-			}
-
-			console.error(chalk.blue(`Analyzing Harbor job: ${jobDir}`))
-
-			// Parse Harbor output
-			const parser = new HarborParser()
-			const analysis = parser.parseJob(jobDir)
-
-			// Generate report
-			let report: string
-
-			if (options.format === "json") {
-				const jsonReporter = new JsonReporter()
-				report = jsonReporter.generate(analysis, true)
-			} else if (options.format === "minimal") {
-				const jsonReporter = new JsonReporter()
-				report = jsonReporter.generateMinimal(analysis)
-			} else {
-				const markdownReporter = new MarkdownReporter()
-				report = markdownReporter.generate(analysis, options.color)
-			}
-
-			// Output report
-			if (options.output) {
-				fs.writeFileSync(options.output, report)
-				console.error(chalk.green(`✓ Report written to: ${options.output}`))
-
-				// Also write full JSON for future reference
-				if (options.format === "markdown") {
-					const jsonPath = options.output.replace(/\.md$/, ".json")
-					const jsonReporter = new JsonReporter()
-					fs.writeFileSync(jsonPath, jsonReporter.generate(analysis, true))
-					console.error(chalk.gray(`  (Full JSON saved to: ${jsonPath})`))
-				}
-			} else {
-				console.log(report)
-			}
-
-			// Summary on stderr
-			const markdownReporter = new MarkdownReporter()
-			const summary = markdownReporter.generateCompactSummary(analysis)
-			console.error("\n" + chalk.bold("Summary:"))
-			console.error(summary)
-		} catch (error) {
-			console.error(chalk.red("Error during analysis:"))
-			console.error(error)
-			process.exit(1)
-		}
-	})
-
-// Compare command
-program
-	.command("compare <baseline> <current>")
-	.description("Compare baseline and current analysis results")
-	.option("-t, --threshold <number>", "Regression threshold (percentage points)", "10")
-	.option("--no-color", "Disable colored output")
-	.action(async (baselinePath: string, currentPath: string, options: any) => {
-		try {
-			// Load both analysis outputs
-			if (!fs.existsSync(baselinePath)) {
-				console.error(chalk.red(`Error: Baseline file not found: ${baselinePath}`))
-				process.exit(1)
-			}
-
-			if (!fs.existsSync(currentPath)) {
-				console.error(chalk.red(`Error: Current file not found: ${currentPath}`))
-				process.exit(1)
-			}
-
-			const baseline: AnalysisOutputV1 = JSON.parse(fs.readFileSync(baselinePath, "utf-8"))
-			const current: AnalysisOutputV1 = JSON.parse(fs.readFileSync(currentPath, "utf-8"))
-
-			const threshold = parseFloat(options.threshold)
-
-			// Compare results
-			const comparison = compareAnalyses(baseline, current, threshold)
-
-			// Display comparison
-			displayComparison(comparison, options.color)
-
-			// Exit with error if regression detected
-			if (comparison.regression_detected) {
-				console.error(chalk.red("\n✗ Regression detected! See details above."))
-				process.exit(1)
-			} else {
-				console.error(chalk.green("\n✓ No significant regression detected."))
-				process.exit(0)
-			}
-		} catch (error) {
-			console.error(chalk.red("Error during comparison:"))
-			console.error(error)
-			process.exit(1)
-		}
-	})
-
-program.parse()
-
-/**
- * Compare two analysis outputs for regression detection
- */
-function compareAnalyses(baseline: AnalysisOutputV1, current: AnalysisOutputV1, threshold: number): ComparisonResult {
-	const delta = {
-		pass_at_1: (current.summary.pass_at_1 - baseline.summary.pass_at_1) * 100,
-		pass_at_3: (current.summary.pass_at_3 - baseline.summary.pass_at_3) * 100,
-		pass_caret_3: (current.summary.pass_caret_3 - baseline.summary.pass_caret_3) * 100,
-		cost_usd: current.summary.total_cost_usd - baseline.summary.total_cost_usd,
-		duration_sec: current.summary.total_duration_sec - baseline.summary.total_duration_sec,
-	}
-
-	// Detect regression (drop in pass rates exceeding threshold)
-	const regression_detected = delta.pass_at_1 < -threshold || delta.pass_at_3 < -threshold
-
-	// Find tasks that regressed or improved
-	const tasks_regressed: string[] = []
-	const tasks_improved: string[] = []
-
-	const baselineTaskMap = new Map(baseline.tasks.map((t) => [t.task_id, t]))
-
-	for (const currentTask of current.tasks) {
-		const baselineTask = baselineTaskMap.get(currentTask.task_id)
-		if (!baselineTask) {
-			continue
-		}
-
-		const taskDelta = (currentTask.metrics.pass_at_3 - baselineTask.metrics.pass_at_3) * 100
-
-		if (taskDelta < -threshold) {
-			tasks_regressed.push(currentTask.task_name)
-		} else if (taskDelta > threshold) {
-			tasks_improved.push(currentTask.task_name)
-		}
-	}
-
-	return {
-		baseline: baseline.summary,
-		current: current.summary,
-		delta,
-		regression_detected,
-		tasks_regressed,
-		tasks_improved,
-	}
-}
-
-/**
- * Display comparison results with color coding
- */
-function displayComparison(comparison: ComparisonResult, useColor: boolean): void {
-	const separator = "━".repeat(79)
-
-	console.log(useColor ? chalk.bold(separator) : separator)
-	console.log(useColor ? chalk.bold.cyan("Baseline vs Current Comparison") : "Baseline vs Current Comparison")
-	console.log(useColor ? chalk.bold(separator) : separator)
-	console.log("")
-
-	// Pass rate changes
-	console.log(useColor ? chalk.bold("Pass Rate Changes:") : "Pass Rate Changes:")
-	console.log(`  pass@1: ${formatDelta(comparison.delta.pass_at_1, useColor)} percentage points`)
-	console.log(`  pass@3: ${formatDelta(comparison.delta.pass_at_3, useColor)} percentage points`)
-	console.log(`  pass^3: ${formatDelta(comparison.delta.pass_caret_3, useColor)} percentage points`)
-	console.log("")
-
-	// Cost and duration changes
-	console.log(useColor ? chalk.bold("Resource Changes:") : "Resource Changes:")
-	console.log(`  Cost: ${formatDelta(comparison.delta.cost_usd, useColor, true)} USD`)
-	console.log(`  Duration: ${formatDelta(comparison.delta.duration_sec, useColor)} seconds`)
-	console.log("")
-
-	// Tasks regressed
-	if (comparison.tasks_regressed.length > 0) {
-		console.log(useColor ? chalk.bold.red("Tasks Regressed:") : "Tasks Regressed:")
-		for (const task of comparison.tasks_regressed) {
-			console.log(`  • ${task}`)
-		}
-		console.log("")
-	}
-
-	// Tasks improved
-	if (comparison.tasks_improved.length > 0) {
-		console.log(useColor ? chalk.bold.green("Tasks Improved:") : "Tasks Improved:")
-		for (const task of comparison.tasks_improved) {
-			console.log(`  • ${task}`)
-		}
-		console.log("")
-	}
-}
-
-/**
- * Format delta value with color coding
- */
-function formatDelta(value: number, useColor: boolean, invertSign = false): string {
-	const sign = invertSign ? -Math.sign(value) : Math.sign(value)
-	const absValue = Math.abs(value).toFixed(2)
-	const signStr = sign > 0 ? "+" : sign < 0 ? "-" : " "
-
-	if (!useColor) {
-		return `${signStr}${absValue}`
-	}
-
-	if (sign > 0) {
-		return chalk.green(`${signStr}${absValue}`)
-	}
-	if (sign < 0) {
-		return chalk.red(`${signStr}${absValue}`)
-	}
-	return chalk.gray(`${signStr}${absValue}`)
-}

evals/analysis/src/metrics.ts

@@ -1,186 +0,0 @@
-/**
- * Metrics calculation for nondeterministic AI testing
- *
- * Implements:
- * - pass@k: P(at least 1 of k trials passes) - solution finding capability
- * - pass^k: P(all k trials pass) - reliability measure
- * - Flakiness score: Entropy-based variance measurement
- *
- * References:
- * - HumanEval paper: https://arxiv.org/abs/2107.03374
- * - pass@k methodology: https://github.com/openai/human-eval
- */
-
-export class MetricsCalculator {
-	/**
-	 * Calculate pass@k: Probability that at least 1 of k trials succeeds
-	 *
-	 * Formula: 1 - C(n-c, k) / C(n, k)
-	 * where n = total trials, c = number of passes, k = sample size
-	 *
-	 * Interpretation: "Can this model solve the problem?"
-	 *
-	 * @param trials Array of boolean trial results (true = pass, false = fail)
-	 * @param k Number of trials to sample
-	 * @returns Probability [0, 1]
-	 */
-	passAtK(trials: boolean[], k: number): number {
-		const n = trials.length
-		const c = trials.filter(Boolean).length
-
-		if (n < k) {
-			throw new Error(`Cannot calculate pass@${k} with only ${n} trials`)
-		}
-
-		// If we have at least k passes, probability is 100%
-		if (c >= k) {
-			return 1.0
-		}
-
-		// Calculate: 1 - C(n-c, k) / C(n, k)
-		const numerator = this.binomial(n - c, k)
-		const denominator = this.binomial(n, k)
-
-		return 1 - numerator / denominator
-	}
-
-	/**
-	 * Calculate pass^k: Probability that ALL k trials succeed
-	 *
-	 * Formula: C(c, k) / C(n, k)
-	 * where n = total trials, c = number of passes, k = sample size
-	 *
-	 * Interpretation: "Can I rely on this model?" (reliability metric)
-	 *
-	 * @param trials Array of boolean trial results
-	 * @param k Number of trials that must all pass
-	 * @returns Probability [0, 1]
-	 */
-	passCaretK(trials: boolean[], k: number): number {
-		const n = trials.length
-		const c = trials.filter(Boolean).length
-
-		if (n < k) {
-			throw new Error(`Cannot calculate pass^${k} with only ${n} trials`)
-		}
-
-		// If we have fewer than k passes, probability is 0%
-		if (c < k) {
-			return 0.0
-		}
-
-		// Calculate: C(c, k) / C(n, k)
-		const numerator = this.binomial(c, k)
-		const denominator = this.binomial(n, k)
-
-		return numerator / denominator
-	}
-
-	/**
-	 * Calculate flakiness score: Entropy-based measure of variance
-	 *
-	 * Formula: -p*log2(p) - (1-p)*log2(1-p)
-	 * where p = pass rate
-	 *
-	 * Returns:
-	 * - 0.0: No variance (all pass or all fail)
-	 * - 1.0: Maximum variance (50% pass rate)
-	 *
-	 * Interpretation: How unpredictable/inconsistent is this task?
-	 *
-	 * @param trials Array of boolean trial results
-	 * @returns Flakiness score [0, 1]
-	 */
-	flakinessScore(trials: boolean[]): number {
-		const passRate = trials.filter(Boolean).length / trials.length
-
-		// No variance if all pass or all fail
-		if (passRate === 0 || passRate === 1) {
-			return 0
-		}
-
-		// Binary entropy
-		const entropy = -passRate * Math.log2(passRate) - (1 - passRate) * Math.log2(1 - passRate)
-
-		return entropy // Already in [0, 1] range
-	}
-
-	/**
-	 * Binomial coefficient C(n, k) = n! / (k! * (n-k)!)
-	 *
-	 * Uses iterative calculation to avoid factorial overflow
-	 *
-	 * @param n Total items
-	 * @param k Items to choose
-	 * @returns Number of ways to choose k items from n
-	 */
-	private binomial(n: number, k: number): number {
-		if (k > n) {
-			return 0
-		}
-		if (k === 0 || k === n) {
-			return 1
-		}
-
-		// Optimize by using smaller k
-		if (k > n - k) {
-			k = n - k
-		}
-
-		let result = 1
-		for (let i = 1; i <= k; i++) {
-			result *= n - i + 1
-			result /= i
-		}
-
-		return result
-	}
-
-	/**
-	 * Calculate all metrics for a task's trials
-	 *
-	 * @param trials Array of boolean trial results
-	 * @returns Object with pass@1, pass@3, pass^3, and flakiness scores
-	 */
-	calculateTaskMetrics(trials: boolean[]): {
-		passAt1: number
-		passAt3: number
-		passCaret3: number
-		flakinessScore: number
-	} {
-		if (trials.length === 0) {
-			throw new Error("Cannot calculate metrics with no trials")
-		}
-
-		// Calculate pass@k and pass^k for available trials
-		const passAt1 = trials.length >= 1 ? this.passAtK(trials, 1) : 0
-		const passAt3 = trials.length >= 3 ? this.passAtK(trials, 3) : 0
-		const passCaret3 = trials.length >= 3 ? this.passCaretK(trials, 3) : 0
-
-		return {
-			passAt1,
-			passAt3,
-			passCaret3,
-			flakinessScore: this.flakinessScore(trials),
-		}
-	}
-
-	/**
-	 * Determine task status based on trial results
-	 *
-	 * @param trials Array of boolean trial results
-	 * @returns "pass" | "fail" | "flaky"
-	 */
-	getTaskStatus(trials: boolean[]): "pass" | "fail" | "flaky" {
-		const passCount = trials.filter(Boolean).length
-		const totalCount = trials.length
-
-		if (passCount === totalCount) {
-			return "pass"
-		}
-		if (passCount === 0) {
-			return "fail"
-		}
-		return "flaky"
-	}
-}

evals/analysis/src/parsers/harbor.ts

@@ -1,301 +0,0 @@
-/**
- * Parser for Harbor framework job output
- *
- * Parses jobs/ directory structure created by Harbor to extract:
- * - Trial results (pass/fail, duration, cost, tokens)
- * - Task groupings and metrics
- * - Failure classifications
- */
-
-import * as fs from "fs"
-import * as path from "path"
-import { FailureClassifier } from "../classifier"
-import { MetricsCalculator } from "../metrics"
-import type {
-	AnalysisMetadata,
-	AnalysisOutputV1,
-	AnalysisSummary,
-	FailureAnalysis,
-	TaskResultV1,
-	TrialResultV1,
-} from "../schemas"
-
-export interface HarborParserOptions {
-	patternsPath?: string
-}
-
-export class HarborParser {
-	private classifier: FailureClassifier
-	private metrics: MetricsCalculator
-
-	constructor(options: HarborParserOptions = {}) {
-		this.classifier = new FailureClassifier(options.patternsPath)
-		this.metrics = new MetricsCalculator()
-	}
-
-	/**
-	 * Parse a complete Harbor job directory
-	 *
-	 * @param jobDir Path to job directory (e.g., jobs/2025-01-25__10-30-00/)
-	 * @returns Structured analysis output with schema version 1.0
-	 */
-	parseJob(jobDir: string): AnalysisOutputV1 {
-		const configPath = path.join(jobDir, "config.json")
-		const resultPath = path.join(jobDir, "result.json")
-
-		if (!fs.existsSync(configPath) || !fs.existsSync(resultPath)) {
-			throw new Error(`Invalid Harbor job directory: ${jobDir}`)
-		}
-
-		const config = JSON.parse(fs.readFileSync(configPath, "utf-8"))
-		const result = JSON.parse(fs.readFileSync(resultPath, "utf-8"))
-
-		// Find all trial directories
-		const trialDirs = this.findTrialDirectories(jobDir)
-		const trials = trialDirs.map((dir) => this.parseTrialDirectory(dir))
-
-		// Group trials by task ID
-		const taskResults = this.groupTrialsByTask(trials)
-
-		// Calculate aggregate metrics
-		const summary = this.calculateSummary(taskResults)
-
-		// Analyze failures
-		const failures = this.analyzeFailures(taskResults)
-
-		const metadata: AnalysisMetadata = {
-			generated_at: new Date().toISOString(),
-			analysis_version: "1.0.0", // TODO: Get from package.json
-			job_id: path.basename(jobDir),
-			model: config.model,
-			agent: config.agent || "cline-cli",
-			environment: config.environment || "docker",
-		}
-
-		return {
-			schema_version: "1.0",
-			metadata,
-			summary,
-			tasks: taskResults,
-			failures,
-		}
-	}
-
-	/**
-	 * Find all trial directories in a job
-	 */
-	private findTrialDirectories(jobDir: string): string[] {
-		const entries = fs.readdirSync(jobDir, { withFileTypes: true })
-
-		return entries
-			.filter((entry) => entry.isDirectory())
-			.filter((entry) => {
-				// Trial dirs have format: 01k7a12s...disco__fhSEuhr
-				const configExists = fs.existsSync(path.join(jobDir, entry.name, "config.json"))
-				return configExists
-			})
-			.map((entry) => path.join(jobDir, entry.name))
-	}
-
-	/**
-	 * Parse a single trial directory
-	 */
-	private parseTrialDirectory(trialDir: string): ParsedTrial {
-		const configPath = path.join(trialDir, "config.json")
-		const resultPath = path.join(trialDir, "result.json")
-		const rewardPath = path.join(trialDir, "verifier", "reward.txt")
-		const logsPath = path.join(trialDir, "agent", "cline.txt")
-		const testOutputPath = path.join(trialDir, "verifier", "test-stdout.txt")
-
-		const config = JSON.parse(fs.readFileSync(configPath, "utf-8"))
-		const result = JSON.parse(fs.readFileSync(resultPath, "utf-8"))
-		const reward = fs.readFileSync(rewardPath, "utf-8").trim()
-		const logs = fs.existsSync(logsPath) ? fs.readFileSync(logsPath, "utf-8") : ""
-		const testOutput = fs.existsSync(testOutputPath) ? fs.readFileSync(testOutputPath, "utf-8") : ""
-
-		const passed = reward === "1"
-		const failures = passed ? [] : this.classifier.classify(logs)
-
-		return {
-			taskId: config.task_id,
-			trialHash: path.basename(trialDir).split("__")[1] || "",
-			passed,
-			duration: result.duration_sec || 0,
-			cost: result.cost_usd || 0,
-			tokensIn: result.tokens_in,
-			tokensOut: result.tokens_out,
-			logs,
-			testOutput,
-			failures,
-		}
-	}
-
-	/**
-	 * Group trials by task ID and calculate metrics
-	 */
-	private groupTrialsByTask(trials: ParsedTrial[]): TaskResultV1[] {
-		const taskMap = new Map<string, ParsedTrial[]>()
-
-		// Group trials by task ID
-		for (const trial of trials) {
-			const existing = taskMap.get(trial.taskId) || []
-			existing.push(trial)
-			taskMap.set(trial.taskId, existing)
-		}
-
-		// Convert to TaskResultV1 format
-		const taskResults: TaskResultV1[] = []
-
-		for (const [taskId, taskTrials] of taskMap.entries()) {
-			const trialResults: TrialResultV1[] = taskTrials.map((trial, index) => ({
-				trial_index: index,
-				trial_hash: trial.trialHash,
-				passed: trial.passed,
-				duration_sec: trial.duration,
-				cost_usd: trial.cost,
-				tokens_in: trial.tokensIn,
-				tokens_out: trial.tokensOut,
-				failures: trial.failures,
-			}))
-
-			const passResults = taskTrials.map((t) => t.passed)
-			const metrics = this.metrics.calculateTaskMetrics(passResults)
-			const status = this.metrics.getTaskStatus(passResults)
-
-			const totalCost = taskTrials.reduce((sum, t) => sum + t.cost, 0)
-			const avgDuration = taskTrials.reduce((sum, t) => sum + t.duration, 0) / taskTrials.length
-
-			// Extract readable task name from ID
-			const taskName = this.extractTaskName(taskId)
-
-			taskResults.push({
-				task_id: taskId,
-				task_name: taskName,
-				trials: trialResults,
-				metrics,
-				status,
-				total_cost_usd: totalCost,
-				avg_duration_sec: avgDuration,
-			})
-		}
-
-		return taskResults.sort((a, b) => a.task_name.localeCompare(b.task_name))
-	}
-
-	/**
-	 * Extract human-readable task name from task ID
-	 * Example: 01k7a12sd1nk15j08e6x0x7v9e-discord-trivia-approval-keyerror → discord-trivia
-	 */
-	private extractTaskName(taskId: string): string {
-		const parts = taskId.split("-")
-		if (parts.length > 1) {
-			// Remove the ID prefix and get first 2-3 meaningful words
-			const words = parts.slice(1, 4)
-			return words.join("-")
-		}
-		return taskId
-	}
-
-	/**
-	 * Calculate aggregate summary metrics
-	 */
-	private calculateSummary(taskResults: TaskResultV1[]): AnalysisSummary {
-		const totalTasks = taskResults.length
-		const totalTrials = taskResults.reduce((sum, task) => sum + task.trials.length, 0)
-
-		// Calculate overall pass@k metrics
-		const allTrials = taskResults.flatMap((task) => task.trials.map((t) => t.passed))
-		let passAt1 = 0
-		let passAt3 = 0
-		let passCaret3 = 0
-
-		if (allTrials.length >= 1) {
-			passAt1 = this.metrics.passAtK(allTrials, 1)
-		}
-		if (allTrials.length >= 3) {
-			passAt3 = this.metrics.passAtK(allTrials, 3)
-			passCaret3 = this.metrics.passCaretK(allTrials, 3)
-		}
-
-		const totalCost = taskResults.reduce((sum, task) => sum + task.total_cost_usd, 0)
-		const totalDuration = taskResults.reduce((sum, task) => sum + task.avg_duration_sec * task.trials.length, 0)
-
-		const flakyTaskCount = taskResults.filter((task) => task.status === "flaky").length
-
-		return {
-			total_tasks: totalTasks,
-			total_trials: totalTrials,
-			pass_at_1: passAt1,
-			pass_at_3: passAt3,
-			pass_caret_3: passCaret3,
-			total_cost_usd: totalCost,
-			total_duration_sec: totalDuration,
-			flaky_task_count: flakyTaskCount,
-		}
-	}
-
-	/**
-	 * Analyze failure patterns across all tasks
-	 */
-	private analyzeFailures(taskResults: TaskResultV1[]): FailureAnalysis {
-		const categoryCount = new Map<string, number>()
-		const patternCount = new Map<string, { count: number; issue_url?: string; examples: any[] }>()
-
-		for (const task of taskResults) {
-			for (const trial of task.trials) {
-				if (!trial.passed) {
-					for (const failure of trial.failures) {
-						// Count by category
-						categoryCount.set(failure.category, (categoryCount.get(failure.category) || 0) + 1)
-
-						// Count by pattern
-						const existing = patternCount.get(failure.name) || {
-							count: 0,
-							issue_url: failure.issue_url,
-							examples: [],
-						}
-						existing.count++
-
-						// Add example if not too many
-						if (existing.examples.length < 3) {
-							existing.examples.push({
-								task_id: task.task_id,
-								trial_index: trial.trial_index,
-								excerpt: failure.excerpt,
-							})
-						}
-
-						patternCount.set(failure.name, existing)
-					}
-				}
-			}
-		}
-
-		const byCategory: Record<string, number> = {}
-		for (const [category, count] of categoryCount.entries()) {
-			byCategory[category] = count
-		}
-
-		const byPattern = Array.from(patternCount.entries()).map(([name, data]) => ({
-			name,
-			count: data.count,
-			issue_url: data.issue_url,
-			examples: data.examples,
-		}))
-
-		return { by_category: byCategory as any, by_pattern: byPattern }
-	}
-}
-
-interface ParsedTrial {
-	taskId: string
-	trialHash: string
-	passed: boolean
-	duration: number
-	cost: number
-	tokensIn?: number
-	tokensOut?: number
-	logs: string
-	testOutput: string
-	failures: any[]
-}

evals/analysis/src/parsers/index.ts

@@ -1,8 +0,0 @@
-/**
- * Parser exports for Cline Analysis Framework
- *
- * Parsers for different benchmark types:
- * - Harbor: Real-world tasks via cline-bench
- */
-
-export * from "./harbor"

evals/analysis/src/reporters/index.ts

@@ -1,10 +0,0 @@
-/**
- * Reporter exports for Cline Analysis Framework
- *
- * Available reporters:
- * - JsonReporter: Structured JSON output with schema validation
- * - MarkdownReporter: Human-readable terminal reports
- */
-
-export * from "./json"
-export * from "./markdown"

evals/analysis/src/reporters/json.ts

@@ -1,96 +0,0 @@
-/**
- * JSON reporter for Cline analysis results
- *
- * Outputs structured JSON with schema versioning for:
- * - CI integration (baseline diffing, regression detection)
- * - Programmatic analysis
- * - Data archival
- */
-
-import type { AnalysisOutputV1 } from "../schemas"
-
-export class JsonReporter {
-	/**
-	 * Generate JSON report from analysis output
-	 *
-	 * @param output Analysis output (already structured)
-	 * @param pretty Whether to pretty-print the JSON
-	 * @returns JSON string
-	 */
-	generate(output: AnalysisOutputV1, pretty = true): string {
-		return JSON.stringify(output, null, pretty ? 2 : undefined)
-	}
-
-	/**
-	 * Validate that output conforms to AnalysisOutputV1 schema
-	 *
-	 * @param output Analysis output to validate
-	 * @throws Error if schema validation fails
-	 */
-	validate(output: any): asserts output is AnalysisOutputV1 {
-		if (output.schema_version !== "1.0") {
-			throw new Error(`Unsupported schema version: ${output.schema_version}`)
-		}
-
-		// Basic structure validation
-		const required = ["metadata", "summary", "tasks", "failures"]
-		for (const field of required) {
-			if (!(field in output)) {
-				throw new Error(`Missing required field: ${field}`)
-			}
-		}
-
-		// Validate metadata
-		if (!output.metadata.generated_at || !output.metadata.job_id || !output.metadata.model) {
-			throw new Error("Invalid metadata: missing required fields")
-		}
-
-		// Validate summary
-		if (typeof output.summary.total_tasks !== "number") {
-			throw new Error("Invalid summary: total_tasks must be a number")
-		}
-
-		// Validate tasks array
-		if (!Array.isArray(output.tasks)) {
-			throw new Error("Invalid tasks: must be an array")
-		}
-	}
-
-	/**
-	 * Generate a minimal JSON report (without full logs and excerpts)
-	 *
-	 * Useful for CI artifacts where space is limited
-	 *
-	 * @param output Analysis output
-	 * @returns Minified JSON string
-	 */
-	generateMinimal(output: AnalysisOutputV1): string {
-		const minimal = {
-			schema_version: output.schema_version,
-			metadata: {
-				job_id: output.metadata.job_id,
-				model: output.metadata.model,
-				generated_at: output.metadata.generated_at,
-			},
-			summary: output.summary,
-			tasks: output.tasks.map((task) => ({
-				task_id: task.task_id,
-				task_name: task.task_name,
-				status: task.status,
-				metrics: task.metrics,
-				total_cost_usd: task.total_cost_usd,
-				avg_duration_sec: task.avg_duration_sec,
-			})),
-			failures: {
-				by_category: output.failures.by_category,
-				by_pattern: output.failures.by_pattern.map((p) => ({
-					name: p.name,
-					count: p.count,
-					issue_url: p.issue_url,
-				})),
-			},
-		}
-
-		return JSON.stringify(minimal, null, 2)
-	}
-}

evals/analysis/src/reporters/markdown.ts

@@ -1,236 +0,0 @@
-/**
- * Markdown reporter for Cline analysis results
- *
- * Generates human-readable reports with:
- * - Summary metrics (pass@k, cost, duration)
- * - Task-by-task results
- * - Failure analysis with issue links
- * - Terminal-friendly formatting
- */
-
-import chalk from "chalk"
-import type { AnalysisOutputV1, TaskResultV1 } from "../schemas"
-
-export class MarkdownReporter {
-	/**
-	 * Generate markdown report from analysis output
-	 *
-	 * @param output Analysis output
-	 * @param useColor Whether to use terminal colors
-	 * @returns Markdown-formatted report string
-	 */
-	generate(output: AnalysisOutputV1, useColor = true): string {
-		const sections: string[] = []
-
-		sections.push(this.generateHeader(output, useColor))
-		sections.push(this.generateSummary(output, useColor))
-		sections.push(this.generateTaskResults(output, useColor))
-		sections.push(this.generateFailureAnalysis(output, useColor))
-		sections.push(this.generateCostPerformance(output, useColor))
-
-		return sections.join("\n\n")
-	}
-
-	private generateHeader(output: AnalysisOutputV1, useColor: boolean): string {
-		const separator = "━".repeat(79)
-		const title = "Cline Bench Analysis Report"
-
-		const lines = [
-			useColor ? chalk.bold(separator) : separator,
-			useColor ? chalk.bold.cyan(title) : title,
-			useColor ? chalk.bold(separator) : separator,
-			"",
-			`Job: ${output.metadata.job_id}`,
-			`Model: ${output.metadata.model}`,
-			`Tasks: ${output.summary.total_tasks} | Trials per task: ${Math.round(output.summary.total_trials / output.summary.total_tasks)}`,
-		]
-
-		return lines.join("\n")
-	}
-
-	private generateSummary(output: AnalysisOutputV1, useColor: boolean): string {
-		const { summary } = output
-		const separator = "━".repeat(79)
-
-		const passAt1Pct = (summary.pass_at_1 * 100).toFixed(1)
-		const passAt3Pct = (summary.pass_at_3 * 100).toFixed(1)
-		const passCaret3Pct = (summary.pass_caret_3 * 100).toFixed(1)
-
-		const lines = [
-			useColor ? chalk.bold(separator) : separator,
-			useColor ? chalk.bold("Results Summary") : "Results Summary",
-			useColor ? chalk.bold(separator) : separator,
-			"",
-			"Overall Metrics:",
-			`  pass@1: ${passAt1Pct}%  (solution finding)`,
-			`  pass@3: ${passAt3Pct}%  (with 3 attempts)`,
-			`  pass^3: ${passCaret3Pct}%  (reliability - all 3 pass)`,
-			"",
-		]
-
-		if (summary.flaky_task_count > 0) {
-			lines.push(
-				useColor
-					? chalk.yellow(`Flakiness: ${summary.flaky_task_count} tasks showed variance across trials`)
-					: `Flakiness: ${summary.flaky_task_count} tasks showed variance across trials`,
-			)
-		} else {
-			lines.push("Flakiness: No variance detected (all tasks consistent)")
-		}
-
-		return lines.join("\n")
-	}
-
-	private generateTaskResults(output: AnalysisOutputV1, useColor: boolean): string {
-		const separator = "━".repeat(79)
-
-		const lines = [
-			useColor ? chalk.bold(separator) : separator,
-			useColor ? chalk.bold("Task Results") : "Task Results",
-			useColor ? chalk.bold(separator) : separator,
-			"",
-		]
-
-		for (const task of output.tasks) {
-			const statusIcon = this.getStatusIcon(task, useColor)
-			const passAt1Pct = (task.metrics.pass_at_1 * 100).toFixed(0)
-			const passCaret3Pct = (task.metrics.pass_caret_3 * 100).toFixed(0)
-			const trialPattern = this.getTrialPattern(task, useColor)
-			const flakyWarning =
-				task.status === "flaky" && useColor ? chalk.yellow(" ⚠️  FLAKY") : task.status === "flaky" ? " ⚠️  FLAKY" : ""
-
-			const taskLine = `${statusIcon} ${task.task_name.padEnd(30)} | pass@1: ${passAt1Pct.padStart(3)}% | pass^3: ${passCaret3Pct.padStart(3)}% | ${trialPattern}${flakyWarning}`
-			lines.push(taskLine)
-		}
-
-		return lines.join("\n")
-	}
-
-	private getStatusIcon(task: TaskResultV1, useColor: boolean): string {
-		if (task.status === "pass") {
-			return useColor ? chalk.green("✓") : "✓"
-		}
-		if (task.status === "fail") {
-			return useColor ? chalk.red("✗") : "✗"
-		}
-		return useColor ? chalk.yellow("◐") : "◐"
-	}
-
-	private getTrialPattern(task: TaskResultV1, useColor: boolean): string {
-		const pattern = task.trials
-			.map((t) => {
-				if (t.passed) {
-					return useColor ? chalk.green("P") : "P"
-				}
-				return useColor ? chalk.red("F") : "F"
-			})
-			.join("")
-
-		return `[${pattern}]`
-	}
-
-	private generateFailureAnalysis(output: AnalysisOutputV1, useColor: boolean): string {
-		const separator = "━".repeat(79)
-
-		const lines = [
-			useColor ? chalk.bold(separator) : separator,
-			useColor ? chalk.bold("Failure Analysis") : "Failure Analysis",
-			useColor ? chalk.bold(separator) : separator,
-			"",
-		]
-
-		// Known issues (provider bugs)
-		const providerBugs = output.failures.by_pattern.filter((p) => p.issue_url)
-
-		if (providerBugs.length > 0) {
-			lines.push("Known Issues Detected:")
-			for (const bug of providerBugs) {
-				const line = `  • ${bug.name} (${bug.count} occurrence${bug.count > 1 ? "s" : ""}) - ${bug.issue_url}`
-				lines.push(useColor ? chalk.yellow(line) : line)
-
-				// Show first example
-				if (bug.examples.length > 0) {
-					const example = bug.examples[0]
-					lines.push(`    Task: ${example.task_id}, Trial ${example.trial_index}`)
-				}
-			}
-			lines.push("")
-		}
-
-		// Transient failures
-		const transient = Object.entries(output.failures.by_category).filter(([cat]) =>
-			["transient", "harness", "environment"].includes(cat),
-		)
-
-		if (transient.length > 0) {
-			lines.push("Infrastructure/Transient Failures:")
-			for (const [category, count] of transient) {
-				lines.push(`  • ${category}: ${count} occurrence${count > 1 ? "s" : ""}`)
-			}
-			lines.push("")
-		}
-
-		// Task failures (model couldn't solve)
-		const taskFailures = output.tasks.filter((t) => t.status === "fail")
-		if (taskFailures.length > 0) {
-			lines.push("Task Failures (Model couldn't solve):")
-			for (const task of taskFailures) {
-				const allFailed = task.trials.every((t) => !t.passed)
-				if (allFailed) {
-					lines.push(`  • ${task.task_name}: All ${task.trials.length} trials failed verification tests`)
-				}
-			}
-		}
-
-		return lines.join("\n")
-	}
-
-	private generateCostPerformance(output: AnalysisOutputV1, useColor: boolean): string {
-		const separator = "━".repeat(79)
-
-		const avgCostPerTask = output.summary.total_cost_usd / output.summary.total_tasks
-		const avgDurationPerTask = output.summary.total_duration_sec / output.summary.total_tasks
-
-		const formattedDuration = this.formatDuration(output.summary.total_duration_sec)
-		const avgFormattedDuration = this.formatDuration(avgDurationPerTask)
-
-		const lines = [
-			useColor ? chalk.bold(separator) : separator,
-			useColor ? chalk.bold("Cost & Performance") : "Cost & Performance",
-			useColor ? chalk.bold(separator) : separator,
-			"",
-			`Total Cost: $${output.summary.total_cost_usd.toFixed(2)}`,
-			`Avg per task: $${avgCostPerTask.toFixed(2)}`,
-			`Total Duration: ${formattedDuration}`,
-			`Avg per task: ${avgFormattedDuration}`,
-			"",
-			`Full report saved to: ${output.metadata.job_id}/analysis_report.md`,
-		]
-
-		return lines.join("\n")
-	}
-
-	private formatDuration(seconds: number): string {
-		if (seconds < 60) {
-			return `${seconds.toFixed(0)}s`
-		}
-		const minutes = Math.floor(seconds / 60)
-		const secs = Math.floor(seconds % 60)
-		return `${minutes}m ${secs}s`
-	}
-
-	/**
-	 * Generate a compact summary (for CI output)
-	 */
-	generateCompactSummary(output: AnalysisOutputV1): string {
-		const passAt1 = (output.summary.pass_at_1 * 100).toFixed(1)
-		const passAt3 = (output.summary.pass_at_3 * 100).toFixed(1)
-		const cost = output.summary.total_cost_usd.toFixed(2)
-
-		return [
-			`✓ pass@1: ${passAt1}% | pass@3: ${passAt3}%`,
-			`  Cost: $${cost} | Tasks: ${output.summary.total_tasks}`,
-			`  Flaky: ${output.summary.flaky_task_count}`,
-		].join("\n")
-	}
-}

evals/analysis/src/schemas/analysis-output.ts

@@ -1,146 +0,0 @@
-/**
- * Type definitions for Cline Analysis Framework output
- * These schemas define the structured JSON output from our analysis tools
- */
-
-/**
- * Versioned analysis output schema (V1)
- * Breaking changes should increment the version number
- */
-export interface AnalysisOutputV1 {
-	schema_version: "1.0"
-	metadata: AnalysisMetadata
-	summary: AnalysisSummary
-	tasks: TaskResultV1[]
-	failures: FailureAnalysis
-}
-
-export interface AnalysisMetadata {
-	generated_at: string // ISO 8601 timestamp
-	analysis_version: string // Package version (from package.json)
-	job_id: string // Job directory name (e.g., "2025-01-25__10-30-00")
-	model: string // Model used for the run
-	agent: string // Agent name (e.g., "cline-cli")
-	environment: string // "docker" | "daytona"
-}
-
-export interface AnalysisSummary {
-	total_tasks: number
-	total_trials: number
-	pass_at_1: number // Probability at least 1 of k=1 succeeds
-	pass_at_3: number // Probability at least 1 of k=3 succeeds
-	pass_caret_3: number // Probability ALL k=3 succeed (reliability)
-	total_cost_usd: number
-	total_duration_sec: number
-	flaky_task_count: number // Tasks with variance across trials
-}
-
-export interface TaskResultV1 {
-	task_id: string
-	task_name: string
-	trials: TrialResultV1[]
-	metrics: TaskMetrics
-	status: "pass" | "fail" | "flaky"
-	total_cost_usd: number
-	avg_duration_sec: number
-}
-
-export interface TrialResultV1 {
-	trial_index: number // 0-indexed trial number
-	trial_hash: string // Hash from trial directory name
-	passed: boolean
-	duration_sec: number
-	cost_usd: number
-	tokens_in?: number
-	tokens_out?: number
-	failures: FailureInfo[] // Classified failure patterns
-}
-
-export interface TaskMetrics {
-	pass_at_1: number // P(at least 1 of 1 succeeds)
-	pass_at_3: number // P(at least 1 of 3 succeeds)
-	pass_caret_3: number // P(all 3 succeed) - reliability metric
-	flakiness_score: number // Entropy-based variance measure (0-1)
-}
-
-export interface FailureInfo {
-	name: string // Pattern name (e.g., "gemini_signature")
-	category: FailureCategory
-	excerpt: string // Log excerpt showing the failure
-	issue_url?: string // GitHub issue link if applicable
-}
-
-export type FailureCategory =
-	| "provider_bug" // Cline integration bugs (Gemini #7974, Claude #7998)
-	| "transient" // Rate limits, network timeouts, service unavailable
-	| "harness" // Test harness or verification script failure
-	| "environment" // Docker/Daytona setup failure
-	| "policy" // Model safety/content policy refusal
-	| "auth" // Invalid API credentials
-	| "task_failure" // Model couldn't solve the task
-
-export interface FailureAnalysis {
-	by_category: Record<FailureCategory, number>
-	by_pattern: FailurePatternSummary[]
-}
-
-export interface FailurePatternSummary {
-	name: string
-	count: number
-	issue_url?: string
-	examples: FailureExample[]
-}
-
-export interface FailureExample {
-	task_id: string
-	trial_index: number
-	excerpt: string
-}
-
-/**
- * Tool Precision Test Result (for replace_in_file benchmarks)
- */
-export interface ToolPrecisionResult {
-	schema_version: "1.0"
-	benchmark: "tool-precision/replace-in-file"
-	total_cases: number
-	passed: number
-	failed: number
-	pass_rate: number
-	avg_latency_ms: number
-	known_failures: string[]
-	timestamp: string
-}
-
-/**
- * Coding Exercises Result (for small task benchmarks)
- */
-export interface CodingExercisesResult {
-	schema_version: "1.0"
-	benchmark: "coding-exercises"
-	total_exercises: number
-	passed: number
-	failed: number
-	pass_at_1: number
-	avg_duration_sec: number
-	known_failures: string[]
-	timestamp: string
-}
-
-/**
- * Comparison result between two analysis outputs
- */
-export interface ComparisonResult {
-	baseline: AnalysisSummary
-	current: AnalysisSummary
-	delta: {
-		pass_at_1: number // Percentage point change
-		pass_at_3: number
-		pass_caret_3: number
-		cost_usd: number
-		duration_sec: number
-	}
-	regression_detected: boolean
-	tasks_regressed: string[] // Task IDs that got worse
-	tasks_improved: string[] // Task IDs that got better
-}

evals/analysis/src/schemas/harbor-output.ts

@@ -1,111 +0,0 @@
-/**
- * Type definitions for Harbor framework output structure
- * These schemas document Harbor's jobs/ directory format (read-only, we validate against this)
- *
- * Harbor is the execution framework used by cline-bench.
- * See: https://harborframework.com
- */
-
-export interface HarborTrialConfig {
-	task_id: string
-	model: string
-	agent: string
-	environment?: string
-	retries?: number
-}
-
-export interface HarborTrialResult {
-	reward: 0 | 1 // Binary pass/fail from verifier
-	duration_sec: number
-	cost_usd: number
-	tokens_in?: number
-	tokens_out?: number
-	timestamp?: string
-}
-
-export interface HarborTrialFiles {
-	agent: {
-		"cline.txt": string // Full conversation log
-		"setup/stdout.txt": string
-		"setup/stderr.txt": string
-		"setup/return-code.txt": string
-		[key: string]: string // command-N/ directories with stdout, stderr, return-code.txt
-	}
-	verifier: {
-		"reward.txt": "0" | "1"
-		"test-stdout.txt": string
-		"test-stderr.txt": string
-	}
-}
-
-/**
- * Structure of a single trial directory
- * Example: jobs/2025-01-25__10-30-00/01k7a12s...disco__fhSEuhr/
- */
-export interface HarborTrialDirectory {
-	"config.json": HarborTrialConfig
-	"result.json": HarborTrialResult
-	agent: HarborTrialFiles["agent"]
-	verifier: HarborTrialFiles["verifier"]
-}
-
-/**
- * Structure of a job's config.json
- */
-export interface HarborJobConfig {
-	model: string
-	agent: string
-	tasks: string[] // Task IDs
-	trials_per_task: number
-	environment: string // "docker" | "daytona"
-	created_at?: string
-}
-
-/**
- * Structure of a job's result.json (aggregate)
- */
-export interface HarborJobResult {
-	total_tasks: number
-	passed_tasks: number
-	failed_tasks: number
-	total_cost_usd: number
-	total_duration_sec: number
-	started_at?: string
-	completed_at?: string
-}
-
-/**
- * Complete job directory structure
- * Example: jobs/2025-01-25__10-30-00/
- */
-export interface HarborJobDirectory {
-	"config.json": HarborJobConfig
-	"result.json": HarborJobResult
-	trials: Record<string, HarborTrialDirectory> // trial-hash -> trial data
-}
-
-/**
- * Parsed trial result from Harbor output
- */
-export interface ParsedHarborTrial {
-	taskId: string
-	trialHash: string
-	passed: boolean
-	duration: number
-	cost: number
-	tokensIn?: number
-	tokensOut?: number
-	logs: string // Full cline.txt content
-	testOutput: string // Test verification output
-	errors: string[] // Setup or command errors
-}
-
-/**
- * Utility type for extracting trial directory paths
- */
-export interface HarborTrialPath {
-	jobDir: string
-	trialDir: string
-	taskId: string
-	trialHash: string
-}

evals/analysis/src/schemas/index.ts

@@ -1,11 +0,0 @@
-/**
- * Schema exports for Cline Analysis Framework
- *
- * This module provides TypeScript type definitions for:
- * - Harbor framework output (read-only validation)
- * - Analysis framework output (our structured results)
- * - Comparison results (baseline vs current)
- */
-
-export * from "./analysis-output"
-export * from "./harbor-output"

evals/analysis/tsconfig.json

@@ -1,32 +0,0 @@
-{
-	"compilerOptions": {
-		"target": "ES2022",
-		"module": "ESNext",
-		"moduleResolution": "bundler",
-		"lib": [
-			"ES2022"
-		],
-		"outDir": "./dist",
-		"rootDir": "./src",
-		"strict": true,
-		"esModuleInterop": true,
-		"skipLibCheck": true,
-		"forceConsistentCasingInFileNames": true,
-		"resolveJsonModule": true,
-		"declaration": true,
-		"declarationMap": true,
-		"sourceMap": true,
-		"types": [
-			"node",
-			"vitest/globals"
-		]
-	},
-	"include": [
-		"src/**/*"
-	],
-	"exclude": [
-		"node_modules",
-		"dist",
-		"src/**/*.test.ts"
-	]
-}

evals/benchmarks/tool-precision/DEPRECATED.md

@@ -1,17 +0,0 @@
-# DEPRECATED
-
-This framework is kept for backward compatibility but is scheduled for removal.
-
-## Why It's Deprecated
-
-1. **Narrow scope**: Tests diff schema compliance, not general model capability
-2. **Better alternatives**: `cline-bench` provides real-world task evaluation
-3. **Maintenance burden**: Requires manual test case generation from conversations
-
-## If You're Using This
-
-Contact @ara or @robin before this gets removed. We can help migrate your use case to the smoke tests or cline-bench framework.
-
-## Removal Timeline
-
-Target: Q2 2026 (or when cline-bench is fully operational for model comparison)

evals/cli/src/adapters/exercism.ts

@@ -0,0 +1,625 @@
+import chalk from "chalk"
+import execa from "execa"
+import * as fs from "fs"
+import * as path from "path"
+import { BenchmarkAdapter, Task, VerificationResult } from "./types"
+
+const EVALS_DIR = path.resolve(__dirname, "../../../")
+
+/**
+ * Adapter for the modified Exercism benchmark
+ */
+export class ExercismAdapter implements BenchmarkAdapter {
+	name = "exercism"
+
+	/**
+	 * Set up the Exercism benchmark repository
+	 */
+	async setup(): Promise<void> {
+		// Clone repository if needed
+		const exercismDir = path.join(EVALS_DIR, "repositories", "exercism")
+
+		if (!fs.existsSync(exercismDir)) {
+			console.log(`Cloning Exercism repository to ${exercismDir}...`)
+			await execa("git", ["clone", "https://github.com/Aider-AI/polyglot-benchmark.git", exercismDir])
+			console.log("Exercism repository cloned successfully")
+
+			// Unskip all JavaScript and Java tests after cloning
+			this.unskipAllJavaScriptTests(exercismDir)
+			this.unskipAllJavaTests(exercismDir)
+		} else {
+			console.log(`Exercism repository already exists at ${exercismDir}`)
+
+			// Pull latest changes
+			console.log("Pulling latest changes...")
+			await execa("git", ["pull"], { cwd: exercismDir })
+			console.log("Repository updated successfully")
+
+			// Unskip tests again after pulling
+			this.unskipAllJavaScriptTests(exercismDir)
+			this.unskipAllJavaTests(exercismDir)
+		}
+	}
+
+	/**
+	 * List all available tasks in the Exercism benchmark
+	 */
+	async listTasks(): Promise<Task[]> {
+		const tasks: Task[] = []
+		const exercisesDir = path.join(EVALS_DIR, "repositories", "exercism")
+
+		// Ensure the repository exists
+		if (!fs.existsSync(exercisesDir)) {
+			throw new Error(`Exercism repository not found at ${exercisesDir}. Run setup first.`)
+		}
+
+		// Read language directories
+		const languages = fs
+			.readdirSync(exercisesDir)
+			.filter((dir) => fs.statSync(path.join(exercisesDir, dir)).isDirectory())
+			.filter((dir) => !dir.startsWith(".") && !["node_modules", ".git"].includes(dir))
+
+		for (const language of languages) {
+			const languageDir = path.join(exercisesDir, language, "exercises", "practice")
+
+			// Read exercise directories
+			const exercises = fs.readdirSync(languageDir).filter((dir) => fs.statSync(path.join(languageDir, dir)).isDirectory())
+
+			for (const exercise of exercises) {
+				const exerciseDir = path.join(languageDir, exercise)
+
+				// Read instructions
+				let description = ""
+				const instructionsPath = path.join(exerciseDir, ".docs", "instructions.md")
+				if (fs.existsSync(instructionsPath)) {
+					description = fs.readFileSync(instructionsPath, "utf-8")
+				}
+
+				// Determine test commands based on language
+				let testCommands: string[] = []
+				switch (language) {
+					case "cpp":
+						testCommands = ["cmake -DEXERCISM_RUN_ALL_TESTS=1 .", "make"]
+						break
+					case "javascript":
+						testCommands = ["npm install", "npm test -- --testNamePattern=."]
+						break
+					case "python":
+						testCommands = ["python3 -m pytest -o markers=task *_test.py"]
+						break
+					case "go":
+						testCommands = ["GOWORK=off go test -v"]
+						break
+					case "java":
+						testCommands = ["./gradlew test"]
+						break
+					case "rust":
+						testCommands = ["cargo test -- --include-ignored"]
+						break
+					default:
+						testCommands = []
+				}
+
+				tasks.push({
+					id: `exercism-${language}-${exercise}`,
+					name: exercise,
+					description,
+					workspacePath: exerciseDir,
+					setupCommands: [],
+					verificationCommands: testCommands,
+					metadata: {
+						language,
+						type: "exercism",
+					},
+				})
+			}
+		}
+
+		return tasks
+	}
+
+	/**
+	 * Prepare a specific task for execution
+	 * @param taskId The ID of the task to prepare
+	 */
+	async prepareTask(taskId: string): Promise<Task> {
+		const tasks = await this.listTasks()
+		const task = tasks.find((t) => t.id === taskId)
+
+		if (!task) {
+			throw new Error(`Task ${taskId} not found`)
+		}
+
+		// Create temp directory outside workspace for hiding files
+		const tempDir = path.join(EVALS_DIR, "temp-files", task.id)
+		fs.mkdirSync(tempDir, { recursive: true })
+
+		// Read config.json to get solution and test files
+		const configPath = path.join(task.workspacePath, ".meta", "config.json")
+		let config: any = { files: { solution: [], test: [] } }
+
+		if (fs.existsSync(configPath)) {
+			config = JSON.parse(fs.readFileSync(configPath, "utf-8"))
+		}
+
+		// Build enhanced description with instructions
+		let description = ""
+		const instructionsPath = path.join(task.workspacePath, ".docs", "instructions.md")
+		const appendPath = path.join(task.workspacePath, ".docs", "instructions.append.md")
+
+		if (fs.existsSync(instructionsPath)) {
+			description = fs.readFileSync(instructionsPath, "utf-8")
+		}
+
+		if (fs.existsSync(appendPath)) {
+			description += "\n\n" + fs.readFileSync(appendPath, "utf-8")
+		}
+
+		// Add solution files constraint to description
+		const solutionFiles = config.files.solution || []
+		const fileList = solutionFiles.join(", ")
+		description += `\n\nUse the above instructions to modify the supplied files: ${fileList}. Don't change the names of existing functions or classes, as they may be referenced from other code like unit tests, etc. Only use standard libraries, don't suggest installing any packages.`
+		description +=
+			" You should ignore all test or test related files in this directory. The final test file has been removed and will be used to evaluate your work after your implementation is complete. Think deeply about the problem prior to working on the implementation. Consider all edge cases and test your solution prior to finalizing."
+
+		// Move test files to temp directory
+		if (config.files.test) {
+			config.files.test.forEach((testFile: string) => {
+				const src = path.join(task.workspacePath, testFile)
+				if (fs.existsSync(src)) {
+					const dest = path.join(tempDir, testFile)
+					fs.mkdirSync(path.dirname(dest), { recursive: true })
+					fs.renameSync(src, dest)
+				}
+			})
+		}
+
+		// Move all dot directories (except .git) to temp directory
+		const items = fs.readdirSync(task.workspacePath)
+		items.forEach((item) => {
+			if (item.startsWith(".") && item !== ".git") {
+				const src = path.join(task.workspacePath, item)
+				const stat = fs.statSync(src)
+				if (stat.isDirectory()) {
+					const dest = path.join(tempDir, item)
+					fs.renameSync(src, dest)
+				}
+			}
+		})
+
+		return {
+			...task,
+			description,
+			metadata: {
+				...task.metadata,
+				solutionFiles,
+				tempDir,
+				config,
+			},
+		}
+	}
+
+	/**
+	 * Cleanup after task execution (restores hidden files from temp directory)
+	 * @param task The task that was executed
+	 */
+	async cleanupTask(task: Task): Promise<void> {
+		const tempDir = path.join(EVALS_DIR, "temp-files", task.id)
+
+		if (fs.existsSync(tempDir)) {
+			const items = fs.readdirSync(tempDir)
+			items.forEach((item) => {
+				const src = path.join(tempDir, item)
+				const dest = path.join(task.workspacePath, item)
+				// Only move if destination doesn't exist (keeps newer test artifacts like .pytest_cache)
+				if (!fs.existsSync(dest)) {
+					fs.renameSync(src, dest)
+				}
+			})
+
+			// Clean up temp directory
+			fs.rmSync(tempDir, { recursive: true, force: true })
+		}
+	}
+
+	/**
+	 * Verify the result of a task execution by running tests
+	 * @param task The task that was executed
+	 */
+	async verifyResult(task: Task): Promise<VerificationResult> {
+		// Run verification commands
+		let success = true
+		let output = ""
+
+		for (const command of task.verificationCommands) {
+			try {
+				const { stdout, stderr } = await execa(command, {
+					cwd: task.workspacePath,
+					shell: true,
+				})
+				output += stdout + "\n"
+				if (stderr) {
+					output += stderr + "\n"
+				}
+			} catch (error: any) {
+				success = false
+				if (error.stdout) {
+					output += error.stdout + "\n"
+				}
+				if (error.stderr) {
+					output += error.stderr + "\n"
+				}
+			}
+		}
+
+		// Log the raw output
+		// console.log("\n=== TEST OUTPUT START ===")
+		// console.log(output)
+		// console.log("=== TEST OUTPUT END ===\n")
+
+		// Parse test results based on language
+		const language = task.metadata.language
+		let testsPassed = 0
+		let testsFailed = 0
+
+		switch (language) {
+			case "python":
+				const pyPassMatch = output.match(/(\d+) passed/)
+				const pyFailMatch = output.match(/(\d+) failed/)
+				testsPassed = pyPassMatch ? parseInt(pyPassMatch[1]) : 0
+				testsFailed = pyFailMatch ? parseInt(pyFailMatch[1]) : 0
+				break
+
+			case "javascript":
+				const jestMatch = output.match(/Tests:\s+(?:\d+ skipped,\s+)?(\d+) passed(?:,\s+(\d+) failed)?/)
+				if (jestMatch) {
+					testsPassed = parseInt(jestMatch[1])
+					testsFailed = jestMatch[2] ? parseInt(jestMatch[2]) : 0
+				} else {
+					// Fallback to counting test suites
+					testsPassed = (output.match(/PASS/g) || []).length
+					testsFailed = (output.match(/FAIL/g) || []).length
+				}
+				break
+
+			case "go":
+				// This incorrectly counts the parent, but minor and doesn't affect final boolean metric
+				testsPassed = (output.match(/--- PASS:/g) || []).length
+				testsFailed = (output.match(/--- FAIL:/g) || []).length
+				break
+
+			case "rust":
+				// Rust runs multiple test suites (unit, integration, doc tests)
+				// Sum results across all test result lines
+				const resultLines = output.match(/test result:.*?(\d+) passed; (\d+) failed/g)
+				if (resultLines) {
+					testsPassed = 0
+					testsFailed = 0
+					for (const line of resultLines) {
+						const match = line.match(/(\d+) passed; (\d+) failed/)
+						if (match) {
+							testsPassed += parseInt(match[1])
+							testsFailed += parseInt(match[2])
+						}
+					}
+				}
+				break
+
+			case "java":
+				testsPassed = (output.match(/PASSED/g) || []).length
+				testsFailed = (output.match(/FAILED/g) || []).length
+				break
+
+			case "cpp":
+				const cppAllPassedMatch = output.match(/All tests passed \(.*?(\d+) test cases?\)/)
+				const cppTestCasesMatch = output.match(/test cases?: (\d+) \| (\d+) passed/)
+				const cppFailedMatch = output.match(/(\d+) failed/)
+
+				if (cppAllPassedMatch) {
+					// All tests passed - extract total test cases
+					testsPassed = parseInt(cppAllPassedMatch[1])
+					testsFailed = 0
+				} else if (cppTestCasesMatch) {
+					// Mixed results - extract passed count and calculate failed
+					const totalTests = parseInt(cppTestCasesMatch[1])
+					testsPassed = parseInt(cppTestCasesMatch[2])
+					testsFailed = cppFailedMatch ? parseInt(cppFailedMatch[1]) : totalTests - testsPassed
+				}
+				break
+
+			default:
+				// Fallback to generic PASS/FAIL counting
+				testsPassed = (output.match(/PASS/g) || []).length
+				testsFailed = (output.match(/FAIL/g) || []).length
+		}
+
+		const testsTotal = testsPassed + testsFailed
+
+		return {
+			success,
+			rawOutput: output,
+			metrics: {
+				testsPassed,
+				testsFailed,
+				testsTotal,
+				functionalCorrectness: testsTotal > 0 ? testsPassed / testsTotal : 0,
+			},
+		}
+	}
+
+	/**
+	 * Hide test files by moving them to temp directory
+	 * @param task The task to hide test files for
+	 */
+	private hideTestFiles(task: Task): void {
+		const tempDir = task.metadata.tempDir
+		const config = task.metadata.config
+
+		if (config?.files?.test) {
+			config.files.test.forEach((testFile: string) => {
+				const src = path.join(task.workspacePath, testFile)
+				if (fs.existsSync(src)) {
+					const dest = path.join(tempDir, testFile)
+					fs.mkdirSync(path.dirname(dest), { recursive: true })
+					fs.renameSync(src, dest)
+				}
+			})
+		}
+
+		// Hide dot directories again (except .git)
+		const items = fs.readdirSync(task.workspacePath)
+		items.forEach((item) => {
+			if (item.startsWith(".") && item !== ".git") {
+				const src = path.join(task.workspacePath, item)
+				if (fs.existsSync(src)) {
+					const stat = fs.statSync(src)
+					if (stat.isDirectory()) {
+						const dest = path.join(tempDir, item)
+						if (!fs.existsSync(dest)) {
+							fs.renameSync(src, dest)
+						}
+					}
+				}
+			}
+		})
+	}
+
+	/**
+	 * Restore test files by moving them from temp directory
+	 * @param task The task to restore test files for
+	 */
+	private restoreTestFiles(task: Task): void {
+		const tempDir = task.metadata.tempDir
+		const config = task.metadata.config
+
+		if (config?.files?.test) {
+			config.files.test.forEach((testFile: string) => {
+				const src = path.join(tempDir, testFile)
+				if (fs.existsSync(src)) {
+					const dest = path.join(task.workspacePath, testFile)
+					fs.mkdirSync(path.dirname(dest), { recursive: true })
+					fs.renameSync(src, dest)
+				}
+			})
+		}
+
+		// Restore dot directories (except .git)
+		if (fs.existsSync(tempDir)) {
+			const items = fs.readdirSync(tempDir)
+			items.forEach((item) => {
+				if (item.startsWith(".") && item !== ".git") {
+					const src = path.join(tempDir, item)
+					const dest = path.join(task.workspacePath, item)
+					if (fs.existsSync(src) && !fs.existsSync(dest)) {
+						fs.renameSync(src, dest)
+					}
+				}
+			})
+		}
+	}
+
+	/**
+	 * Builds retry message with test errors and fix instructions
+	 * @param testOutput The raw test output showing errors
+	 * @param solutionFiles List of solution files to fix
+	 * @returns Formatted retry message
+	 */
+	private buildRetryMessage(testOutput: string, solutionFiles: string[]): string {
+		const fileList = solutionFiles.join(", ")
+		return `${testOutput}\n\nSee the testing errors above. The tests are correct, don't try and change them. Fix the code in ${fileList} to resolve the errors.`
+	}
+
+	/**
+	 * Unskip all JavaScript tests in the repository by replacing xtest with test
+	 * @param repoPath Path to the exercism repository
+	 */
+	private unskipAllJavaScriptTests(repoPath: string): void {
+		const jsDir = path.join(repoPath, "javascript", "exercises", "practice")
+
+		if (!fs.existsSync(jsDir)) {
+			console.log("JavaScript exercises directory not found, skipping test unskipping")
+			return
+		}
+
+		// Walk through all exercise directories
+		const exercises = fs.readdirSync(jsDir).filter((dir) => {
+			const fullPath = path.join(jsDir, dir)
+			return fs.statSync(fullPath).isDirectory()
+		})
+
+		let filesModified = 0
+		for (const exercise of exercises) {
+			const exerciseDir = path.join(jsDir, exercise)
+
+			// Find all .spec.js files
+			const files = fs.readdirSync(exerciseDir).filter((file) => file.endsWith(".spec.js"))
+
+			for (const file of files) {
+				const filePath = path.join(exerciseDir, file)
+				let content = fs.readFileSync(filePath, "utf-8")
+				const originalContent = content
+
+				// Replace xtest with test to unskip tests
+				content = content.replace(/xtest\(/g, "test(")
+
+				if (content !== originalContent) {
+					fs.writeFileSync(filePath, content)
+					filesModified++
+				}
+			}
+		}
+
+		console.log(`Unskipped tests in ${filesModified} JavaScript test files`)
+	}
+
+	/**
+	 * Unskip all Java tests in the repository by removing @Disabled annotations
+	 * @param repoPath Path to the exercism repository
+	 */
+	private unskipAllJavaTests(repoPath: string): void {
+		const javaDir = path.join(repoPath, "java", "exercises", "practice")
+
+		if (!fs.existsSync(javaDir)) {
+			console.log("Java exercises directory not found, skipping test unskipping")
+			return
+		}
+
+		// Walk through all exercise directories
+		const exercises = fs.readdirSync(javaDir).filter((dir) => {
+			const fullPath = path.join(javaDir, dir)
+			return fs.statSync(fullPath).isDirectory()
+		})
+
+		let filesModified = 0
+		for (const exercise of exercises) {
+			const testDir = path.join(javaDir, exercise, "src", "test", "java")
+
+			if (!fs.existsSync(testDir)) {
+				continue
+			}
+
+			// Find all .java test files
+			const files = fs.readdirSync(testDir).filter((file) => file.endsWith(".java"))
+
+			for (const file of files) {
+				const filePath = path.join(testDir, file)
+				let content = fs.readFileSync(filePath, "utf-8")
+				const originalContent = content
+
+				// Remove @Disabled("Remove to run test") annotations
+				content = content.replace(/@Disabled\("Remove to run test"\)\s*\n/g, "")
+
+				if (content !== originalContent) {
+					fs.writeFileSync(filePath, content)
+					filesModified++
+				}
+			}
+		}
+
+		console.log(`Unskipped tests in ${filesModified} Java test files`)
+	}
+
+	/**
+	 * Runs a Cline task with automatic retry on test failure
+	 * Creates a new Cline instance, runs the task, verifies with tests,
+	 * and retries once if tests fail
+	 * @param task The task to execute
+	 * @returns The final verification result, or null
+	 */
+	async runTask(task: Task): Promise<VerificationResult | null> {
+		const startTime = Date.now()
+		let instanceAddress: string | null = null
+		let attempts = 0
+		let finalVerification: VerificationResult | null = null
+
+		try {
+			// Step 1: Start a new Cline instance in the working directory
+			const instanceResult = await execa("cline", ["instance", "new"], {
+				cwd: task.workspacePath,
+				stdin: "ignore",
+			})
+
+			// Step 2: Parse the instance address from output
+			const addressMatch = instanceResult.stdout.match(/Address:\s*([\d.]+:\d+)/)
+			if (!addressMatch) {
+				throw new Error("Failed to parse instance address from output")
+			}
+			instanceAddress = addressMatch[1]
+
+			// Step 3: Create the initial task on this specific instance
+			await execa("cline", ["task", "new", "--yolo", "--address", instanceAddress, task.description], {
+				cwd: task.workspacePath,
+				stdin: "ignore",
+			})
+
+			// Step 4: Wait for initial implementation to complete
+			console.log(chalk.blue(`Waiting for first attempt to complete...`))
+			await execa("cline", ["task", "view", "--follow-complete", "--address", instanceAddress], {
+				cwd: task.workspacePath,
+				stdin: "ignore",
+			})
+
+			// Step 5: Run first test attempt
+			console.log(chalk.blue(`Running tests (attempt 1)...`))
+			this.restoreTestFiles(task)
+			attempts = 1
+			const firstVerification = await this.verifyResult(task)
+			finalVerification = firstVerification
+
+			// Step 6: Retry if tests failed
+			if (!firstVerification.success) {
+				console.log(chalk.blue(`Tests failed on first attempt. Retrying...`))
+
+				// Hide test files again for retry
+				this.hideTestFiles(task)
+
+				attempts = 2
+				const solutionFiles = task.metadata.solutionFiles || []
+				const retryMessage = this.buildRetryMessage(firstVerification.rawOutput || "", solutionFiles)
+
+				// Send retry task message
+				await execa("cline", ["task", "send", "--yolo", "--address", instanceAddress], {
+					cwd: task.workspacePath,
+					input: retryMessage,
+				})
+
+				// Follow retry until complete
+				await execa("cline", ["task", "view", "--follow-complete", "--address", instanceAddress], {
+					cwd: task.workspacePath,
+					stdin: "ignore",
+				})
+
+				// Run second test attempt (final)
+				console.log(chalk.blue(`Running tests (attempt 2)...`))
+				this.restoreTestFiles(task)
+				const secondVerification = await this.verifyResult(task)
+				finalVerification = secondVerification
+			}
+
+			const duration = Date.now() - startTime
+			console.log(
+				chalk.green(
+					`Task completed in ${(duration / 1000).toFixed(1)}s after ${attempts} attempt${attempts > 1 ? "s" : ""}`,
+				),
+			)
+
+			return finalVerification
+		} catch (error: any) {
+			const duration = Date.now() - startTime
+			console.error(chalk.red(`Task failed after ${(duration / 1000).toFixed(1)}s: ${error.message}`))
+
+			return finalVerification
+		} finally {
+			// Step 7: Always clean up the instance, even if task failed
+			if (instanceAddress) {
+				try {
+					await execa("cline", ["instance", "kill", instanceAddress], {
+						stdin: "ignore",
+					})
+				} catch (cleanupError: any) {
+					console.error(chalk.yellow(`Warning: Failed to kill instance ${instanceAddress}: ${cleanupError.message}`))
+				}
+			}
+		}
+	}
+}

evals/cli/src/adapters/index.ts

@@ -0,0 +1,38 @@
+import { ExercismAdapter } from "./exercism"
+import { BenchmarkAdapter } from "./types"
+
+// Registry of all available adapters
+const adapters: Record<string, BenchmarkAdapter> = {
+	exercism: new ExercismAdapter(),
+}
+
+/**
+ * Get a specific adapter by name
+ * @param name The name of the adapter to get
+ * @returns The requested adapter
+ * @throws Error if the adapter is not found
+ */
+export function getAdapter(name: string): BenchmarkAdapter {
+	const adapter = adapters[name]
+	if (!adapter) {
+		throw new Error(`Adapter for benchmark '${name}' not found`)
+	}
+	return adapter
+}
+
+/**
+ * Get all available adapters
+ * @returns Array of all registered adapters
+ */
+export function getAllAdapters(): BenchmarkAdapter[] {
+	return Object.values(adapters)
+}
+
+/**
+ * Register a new adapter
+ * @param name The name to register the adapter under
+ * @param adapter The adapter to register
+ */
+export function registerAdapter(name: string, adapter: BenchmarkAdapter): void {
+	adapters[name] = adapter
+}

evals/cli/src/adapters/types.ts

@@ -0,0 +1,34 @@
+/**
+ * Represents a task to be executed
+ */
+export interface Task {
+	id: string
+	name: string
+	description: string
+	workspacePath: string
+	setupCommands: string[]
+	verificationCommands: string[]
+	metadata: Record<string, any>
+}
+
+/**
+ * Result of verifying a task execution
+ */
+export interface VerificationResult {
+	success: boolean
+	metrics: Record<string, any>
+	rawOutput?: string
+}
+
+/**
+ * Interface for benchmark adapters
+ */
+export interface BenchmarkAdapter {
+	name: string
+	setup(): Promise<void>
+	listTasks(): Promise<Task[]>
+	prepareTask(taskId: string): Promise<Task>
+	cleanupTask(task: Task): Promise<void>
+	verifyResult(task: Task): Promise<VerificationResult>
+	runTask(task: Task): Promise<VerificationResult | null>
+}

evals/cli/src/commands/report.ts

@@ -0,0 +1,223 @@
+import chalk from "chalk"
+import * as fs from "fs"
+import ora from "ora"
+import * as path from "path"
+import { ResultsDatabase } from "../db"
+import { generateMarkdownReport } from "../utils/markdown"
+
+interface ReportOptions {
+	format?: "json" | "markdown"
+	output?: string
+}
+
+/**
+ * Handler for the report command
+ * @param options Command options
+ */
+export async function reportHandler(options: ReportOptions): Promise<void> {
+	const format = options.format || "markdown"
+	const db = new ResultsDatabase()
+
+	try {
+		const spinner = ora("Generating report...").start()
+
+		// Get all runs
+		const runs = db.getRuns()
+
+		console.log(chalk.blue(`Found ${runs.length} evaluation runs`))
+
+		if (runs.length === 0) {
+			spinner.fail("No evaluation runs found")
+			return
+		}
+
+		// Generate summary report
+		const summary = {
+			runs: runs.length,
+			benchmarks: [...new Set(runs.map((run) => run.benchmark))],
+			tasks: 0,
+			successRate: 0,
+			averageTokens: 0,
+			averageCost: 0,
+			averageDuration: 0,
+			totalToolCalls: 0,
+			totalToolFailures: 0,
+			toolSuccessRate: 0,
+			toolUsage: {} as Record<string, { calls: number; failures: number }>,
+			totalTests: 0,
+			totalTestsPassed: 0,
+			totalTestsFailed: 0,
+			testSuccessRate: 0,
+		}
+
+		let totalTasks = 0
+		let successfulTasks = 0
+		let totalTokens = 0
+		let totalCost = 0
+		let totalDuration = 0
+		let totalToolCalls = 0
+		let totalToolFailures = 0
+		let totalTests = 0
+		let totalTestsPassed = 0
+		let totalTestsFailed = 0
+
+		for (const run of runs) {
+			const tasks = db.getRunTasks(run.id)
+			totalTasks += tasks.length
+
+			for (const task of tasks) {
+				if (task.success) {
+					successfulTasks++
+				}
+
+				const metrics = db.getTaskMetrics(task.id)
+
+				const tokensIn = metrics.find((m) => m.name === "tokensIn")?.value || 0
+				const tokensOut = metrics.find((m) => m.name === "tokensOut")?.value || 0
+				totalTokens += tokensIn + tokensOut
+
+				totalCost += metrics.find((m) => m.name === "cost")?.value || 0
+				totalDuration += metrics.find((m) => m.name === "duration")?.value || 0
+
+				// Collect test metrics
+				const testsPassed = metrics.find((m) => m.name === "testsPassed")?.value || 0
+				const testsFailed = metrics.find((m) => m.name === "testsFailed")?.value || 0
+				const testsTotal = metrics.find((m) => m.name === "testsTotal")?.value || 0
+				totalTestsPassed += testsPassed
+				totalTestsFailed += testsFailed
+				totalTests += testsTotal
+
+				// Collect tool call metrics
+				totalToolCalls += task.total_tool_calls || 0
+				totalToolFailures += task.total_tool_failures || 0
+
+				// Get detailed tool usage
+				const toolCalls = db.getTaskToolCalls(task.id)
+
+				for (const toolCall of toolCalls) {
+					if (!summary.toolUsage[toolCall.tool_name]) {
+						summary.toolUsage[toolCall.tool_name] = {
+							calls: 0,
+							failures: 0,
+						}
+					}
+
+					summary.toolUsage[toolCall.tool_name].calls += toolCall.call_count
+					summary.toolUsage[toolCall.tool_name].failures += toolCall.failure_count
+				}
+			}
+		}
+
+		// Calculate tool success rate
+		summary.totalToolCalls = totalToolCalls
+		summary.totalToolFailures = totalToolFailures
+		summary.toolSuccessRate = totalToolCalls > 0 ? 1 - totalToolFailures / totalToolCalls : 1.0
+
+		// Calculate test metrics
+		summary.totalTests = totalTests
+		summary.totalTestsPassed = totalTestsPassed
+		summary.totalTestsFailed = totalTestsFailed
+		summary.testSuccessRate = totalTests > 0 ? totalTestsPassed / totalTests : 0
+
+		summary.tasks = totalTasks
+		summary.successRate = totalTasks > 0 ? successfulTasks / totalTasks : 0
+		summary.averageTokens = totalTasks > 0 ? totalTokens / totalTasks : 0
+		summary.averageCost = totalTasks > 0 ? totalCost / totalTasks : 0
+		summary.averageDuration = totalTasks > 0 ? totalDuration / totalTasks : 0
+
+		// Generate benchmark-specific reports
+		const benchmarkReports: Record<string, any> = {}
+
+		for (const benchmark of summary.benchmarks) {
+			const benchmarkRuns = runs.filter((run) => run.benchmark === benchmark)
+			const benchmarkSummary = {
+				runs: benchmarkRuns.length,
+				tasks: 0,
+				successRate: 0,
+				averageTokens: 0,
+				averageCost: 0,
+				averageDuration: 0,
+				totalTests: 0,
+				totalTestsPassed: 0,
+				totalTestsFailed: 0,
+				testSuccessRate: 0,
+			}
+
+			let benchmarkTasks = 0
+			let benchmarkSuccessfulTasks = 0
+			let benchmarkTotalTokens = 0
+			let benchmarkTotalCost = 0
+			let benchmarkTotalDuration = 0
+			let benchmarkTotalTests = 0
+			let benchmarkTotalTestsPassed = 0
+			let benchmarkTotalTestsFailed = 0
+
+			for (const run of benchmarkRuns) {
+				const tasks = db.getRunTasks(run.id)
+				benchmarkTasks += tasks.length
+
+				for (const task of tasks) {
+					if (task.success) {
+						benchmarkSuccessfulTasks++
+					}
+
+					const metrics = db.getTaskMetrics(task.id)
+
+					const tokensIn = metrics.find((m) => m.name === "tokensIn")?.value || 0
+					const tokensOut = metrics.find((m) => m.name === "tokensOut")?.value || 0
+					benchmarkTotalTokens += tokensIn + tokensOut
+
+					benchmarkTotalCost += metrics.find((m) => m.name === "cost")?.value || 0
+					benchmarkTotalDuration += metrics.find((m) => m.name === "duration")?.value || 0
+
+					// Collect test metrics
+					const testsPassed = metrics.find((m) => m.name === "testsPassed")?.value || 0
+					const testsFailed = metrics.find((m) => m.name === "testsFailed")?.value || 0
+					const testsTotal = metrics.find((m) => m.name === "testsTotal")?.value || 0
+					benchmarkTotalTestsPassed += testsPassed
+					benchmarkTotalTestsFailed += testsFailed
+					benchmarkTotalTests += testsTotal
+				}
+			}
+
+			benchmarkSummary.tasks = benchmarkTasks
+			benchmarkSummary.successRate = benchmarkTasks > 0 ? benchmarkSuccessfulTasks / benchmarkTasks : 0
+			benchmarkSummary.averageTokens = benchmarkTasks > 0 ? benchmarkTotalTokens / benchmarkTasks : 0
+			benchmarkSummary.averageCost = benchmarkTasks > 0 ? benchmarkTotalCost / benchmarkTasks : 0
+			benchmarkSummary.averageDuration = benchmarkTasks > 0 ? benchmarkTotalDuration / benchmarkTasks : 0
+			benchmarkSummary.totalTests = benchmarkTotalTests
+			benchmarkSummary.totalTestsPassed = benchmarkTotalTestsPassed
+			benchmarkSummary.totalTestsFailed = benchmarkTotalTestsFailed
+			benchmarkSummary.testSuccessRate = benchmarkTotalTests > 0 ? benchmarkTotalTestsPassed / benchmarkTotalTests : 0
+
+			benchmarkReports[benchmark] = benchmarkSummary
+		}
+
+		// Save reports
+		const reportDir = path.join(path.resolve(__dirname, "../../../"), "results", "reports")
+		fs.mkdirSync(reportDir, { recursive: true })
+
+		const timestamp = new Date().toISOString().replace(/:/g, "-")
+
+		if (format === "json") {
+			// Save JSON reports
+			fs.writeFileSync(path.join(reportDir, `summary-${timestamp}.json`), JSON.stringify(summary, null, 2))
+
+			fs.writeFileSync(path.join(reportDir, `benchmarks-${timestamp}.json`), JSON.stringify(benchmarkReports, null, 2))
+
+			spinner.succeed(`JSON reports generated in ${reportDir}`)
+		} else {
+			// Generate markdown report
+			const outputPath = options.output || path.join(reportDir, `report-${timestamp}.md`)
+
+			generateMarkdownReport(summary, benchmarkReports, outputPath)
+
+			spinner.succeed(`Markdown report generated at ${outputPath}`)
+		}
+	} catch (error: any) {
+		console.error(chalk.red(`Error generating report: ${error.message}`))
+		console.error(error.stack)
+	} finally {
+		db.close()
+	}
+}

evals/cli/src/commands/run.ts

@@ -0,0 +1,114 @@
+import chalk from "chalk"
+import ora from "ora"
+import { v4 as uuidv4 } from "uuid"
+import { getAdapter } from "../adapters"
+import { ResultsDatabase } from "../db"
+import { storeTaskResult } from "../utils/results"
+
+interface RunOptions {
+	benchmark?: string
+	count?: number
+}
+
+/**
+ * Handler for the run command
+ * @param options Command options
+ */
+export async function runHandler(options: RunOptions): Promise<void> {
+	// Determine which benchmarks to run
+	const benchmarks = options.benchmark ? [options.benchmark] : ["exercism"] // Default to exercism
+	const count = options.count || Infinity
+
+	console.log(chalk.blue(`Running evaluations for the following benchmarks: ${benchmarks.join(", ")}`))
+
+	// Create a run for each benchmark
+	for (const benchmark of benchmarks) {
+		const runId = uuidv4()
+		const db = new ResultsDatabase()
+
+		console.log(chalk.green(`\nStarting run for benchmark: ${benchmark}`))
+
+		// Create run in database
+		db.createRun(runId, benchmark)
+
+		// Get adapter for this benchmark
+		try {
+			const adapter = getAdapter(benchmark)
+
+			// List tasks
+			const spinner = ora("Listing tasks...").start()
+			const tasks = await adapter.listTasks()
+			spinner.succeed(`Found ${tasks.length} tasks for ${benchmark}`)
+
+			// Limit number of tasks if specified
+			const tasksToRun = tasks.slice(0, count)
+
+			console.log(chalk.blue(`Running ${tasksToRun.length} tasks...`))
+
+			// Run each task
+			for (let i = 0; i < tasksToRun.length; i++) {
+				const task = tasksToRun[i]
+
+				console.log(chalk.cyan(`\nTask ${i + 1}/${tasksToRun.length}: ${task.name}`))
+
+				// Prepare task
+				const prepareSpinner = ora("Preparing task...").start()
+				const preparedTask = await adapter.prepareTask(task.id)
+				prepareSpinner.succeed("Task prepared")
+
+				let cleanedUp = false
+
+				try {
+					// Run task using adapter's execution strategy
+					const finalVerification = await adapter.runTask(preparedTask)
+
+					// Cleanup task
+					const cleanupSpinner = ora("Cleaning up task...").start()
+					await adapter.cleanupTask(preparedTask)
+					cleanedUp = true
+					cleanupSpinner.succeed("Cleanup complete")
+
+					// Use final verification from runTask
+					const verification = finalVerification || (await adapter.verifyResult(preparedTask))
+
+					if (verification.success) {
+						console.log(
+							chalk.green(`Tests passed: ${verification.metrics.testsPassed}/${verification.metrics.testsTotal}`),
+						)
+					} else {
+						console.log(
+							chalk.red(`Tests failed: ${verification.metrics.testsPassed}/${verification.metrics.testsTotal}`),
+						)
+					}
+
+					// Store result
+					const storeSpinner = ora("Storing result...").start()
+					await storeTaskResult(runId, preparedTask, {}, verification)
+					storeSpinner.succeed("Result stored")
+				} catch (error: any) {
+					console.error(chalk.red(`Task failed: ${error.message}`))
+				} finally {
+					// Ensure cleanup always happens
+					if (!cleanedUp) {
+						try {
+							const finalCleanupSpinner = ora("Performing cleanup...").start()
+							await adapter.cleanupTask(preparedTask)
+							finalCleanupSpinner.succeed("Cleanup complete")
+						} catch (cleanupError: any) {
+							console.error(chalk.red(`Cleanup failed: ${cleanupError.message}`))
+						}
+					}
+				}
+			}
+
+			// Mark run as complete
+			db.completeRun(runId)
+
+			console.log(chalk.green(`\nRun complete for benchmark: ${benchmark}`))
+		} catch (error: any) {
+			console.error(chalk.red(`Error running benchmark ${benchmark}: ${error.message}`))
+		}
+	}
+
+	console.log(chalk.green("\nAll evaluations complete"))
+}

evals/cli/src/commands/runDiffEval.ts

@@ -0,0 +1,107 @@
+import execa from "execa"
+import chalk from "chalk"
+import path from "path"
+
+interface RunDiffEvalOptions {
+	modelIds: string
+	systemPromptName: string
+	validAttemptsPerCase: number
+	maxAttemptsPerCase?: number
+	parsingFunction: string
+	diffEditFunction: string
+	thinkingBudget: number
+	provider: string
+	parallel: boolean
+	verbose: boolean
+	testPath: string
+	outputPath: string
+	replay: boolean
+	replayRunId?: string
+	diffApplyFile?: string
+	saveLocally: boolean
+	maxCases?: number
+}
+
+export async function runDiffEvalHandler(options: RunDiffEvalOptions) {
+	console.log(chalk.blue("Starting diff editing evaluation..."))
+
+	// Resolve the path to the TestRunner.ts script relative to the current file
+	const scriptPath = path.resolve(__dirname, "../../../diff-edits/TestRunner.ts")
+
+	// Construct the arguments array for the execa call
+	const args = [
+		"--model-ids",
+		options.modelIds,
+		"--system-prompt-name",
+		options.systemPromptName,
+		"--valid-attempts-per-case",
+		String(options.validAttemptsPerCase),
+		"--parsing-function",
+		options.parsingFunction,
+		"--diff-edit-function",
+		options.diffEditFunction,
+		"--provider",
+		options.provider,
+	]
+
+	// Conditionally add the optional arguments
+	if (options.testPath) {
+		args.push("--test-path", options.testPath)
+	}
+	if (options.outputPath) {
+		args.push("--output-path", options.outputPath)
+	}
+	if (options.thinkingBudget > 0) {
+		args.push("--thinking-budget", String(options.thinkingBudget))
+	}
+
+	if (options.parallel) {
+		args.push("--parallel")
+	}
+
+	if (options.replay) {
+		args.push("--replay")
+	}
+
+	if (options.replayRunId) {
+		args.push("--replay-run-id", options.replayRunId)
+	}
+
+	if (options.diffApplyFile) {
+		args.push("--diff-apply-file", options.diffApplyFile)
+	}
+
+	if (options.verbose) {
+		args.push("--verbose")
+	}
+
+	if (options.maxAttemptsPerCase) {
+		args.push("--max-attempts-per-case", String(options.maxAttemptsPerCase))
+	}
+
+	if (options.maxCases) {
+		args.push("--max-cases", String(options.maxCases))
+	}
+
+	if (options.saveLocally) {
+		args.push("--save-locally")
+	}
+
+	try {
+		console.log(chalk.gray(`Executing: npx tsx ${scriptPath} ${args.join(" ")}`))
+
+		// Execute the script as a child process
+		// We use 'inherit' to stream the stdout/stderr directly to the user's terminal
+		const subprocess = execa("npx", ["tsx", "--tsconfig", path.resolve(__dirname, "../../../tsconfig.json"), scriptPath, ...args], {
+			stdio: "inherit",
+		})
+
+		await subprocess
+
+		console.log(chalk.green("Diff editing evaluation completed successfully."))
+	} catch (error) {
+		console.error(chalk.red("An error occurred during the diff editing evaluation."))
+		// The 'inherit' stdio will have already printed the error details from the script
+		process.exit(1)
+	}
+}

evals/cli/src/commands/setup.ts

@@ -0,0 +1,72 @@
+import * as path from "path"
+import * as fs from "fs"
+import execa from "execa"
+import chalk from "chalk"
+import ora from "ora"
+import { getAllAdapters } from "../adapters/index"
+import { BenchmarkAdapter } from "../adapters/types"
+
+interface SetupOptions {
+	benchmarks: string
+}
+
+/**
+ * Handler for the setup command
+ * @param options Command options
+ */
+export async function setupHandler(options: SetupOptions): Promise<void> {
+	const benchmarks = options.benchmarks.split(",")
+
+	console.log(chalk.blue(`Setting up benchmarks: ${benchmarks.join(", ")}`))
+
+	// Create directories
+	const evalsDir = path.resolve(__dirname, "../../../")
+	const reposDir = path.join(evalsDir, "repositories")
+	const resultsDir = path.join(evalsDir, "results")
+
+	const spinner = ora("Creating directory structure").start()
+
+	try {
+		fs.mkdirSync(reposDir, { recursive: true })
+		fs.mkdirSync(resultsDir, { recursive: true })
+		fs.mkdirSync(path.join(resultsDir, "runs"), { recursive: true })
+		fs.mkdirSync(path.join(resultsDir, "reports"), { recursive: true })
+		spinner.succeed("Directory structure created")
+	} catch (error) {
+		spinner.fail(`Failed to create directory structure: ${(error as Error).message}`)
+		throw error
+	}
+
+	// Set up each benchmark
+	try {
+		const adapters = getAllAdapters().filter((adapter: BenchmarkAdapter) => benchmarks.includes(adapter.name))
+
+		if (adapters.length === 0) {
+			console.warn(chalk.yellow("No valid benchmarks specified. Available benchmarks:"))
+			console.warn(
+				chalk.yellow(
+					getAllAdapters()
+						.map((a: BenchmarkAdapter) => a.name)
+						.join(", "),
+				),
+			)
+			return
+		}
+
+		for (const adapter of adapters) {
+			const setupSpinner = ora(`Setting up ${adapter.name}...`).start()
+			try {
+				await adapter.setup()
+				setupSpinner.succeed(`${adapter.name} setup complete`)
+			} catch (error) {
+				setupSpinner.fail(`Failed to set up ${adapter.name}: ${(error as Error).message}`)
+				throw error
+			}
+		}
+
+		console.log(chalk.green("Setup complete"))
+	} catch (error) {
+		console.error(chalk.red(`Setup failed: ${(error as Error).message}`))
+		throw error
+	}
+}

evals/cli/src/db/index.ts

@@ -0,0 +1,210 @@
+import Database from "better-sqlite3"
+import * as fs from "fs"
+import * as path from "path"
+import { SCHEMA } from "./schema"
+
+const EVALS_DIR = path.resolve(__dirname, "../../../")
+
+/**
+ * Database class for storing evaluation results
+ */
+export class ResultsDatabase {
+	db: Database.Database
+
+	constructor() {
+		// Ensure results directory exists
+		const resultsDir = path.join(EVALS_DIR, "results")
+		fs.mkdirSync(resultsDir, { recursive: true })
+
+		// Create database file
+		const dbPath = path.join(resultsDir, "evals.db")
+		this.db = new Database(dbPath)
+
+		// Initialize schema
+		this.initSchema()
+	}
+
+	/**
+	 * Initialize the database schema
+	 */
+	private initSchema(): void {
+		this.db.exec(SCHEMA)
+	}
+
+	/**
+	 * Create a new evaluation run
+	 * @param id Run ID
+	 * @param benchmark Benchmark name
+	 */
+	createRun(id: string, benchmark: string): void {
+		const stmt = this.db.prepare(`
+      INSERT INTO runs (id, timestamp, benchmark)
+      VALUES (?, ?, ?)
+    `)
+
+		stmt.run(id, Date.now(), benchmark)
+	}
+
+	/**
+	 * Mark a run as completed
+	 * @param id Run ID
+	 */
+	completeRun(id: string): void {
+		const stmt = this.db.prepare(`
+      UPDATE runs SET completed = 1 WHERE id = ?
+    `)
+
+		stmt.run(id)
+	}
+
+	/**
+	 * Create a new task
+	 * @param id Task ID
+	 * @param runId Run ID
+	 * @param taskId Original task ID
+	 */
+	createTask(id: string, runId: string, taskId: string): void {
+		const stmt = this.db.prepare(`
+      INSERT INTO tasks (id, run_id, task_id, timestamp)
+      VALUES (?, ?, ?, ?)
+    `)
+
+		stmt.run(id, runId, taskId, Date.now())
+	}
+
+	/**
+	 * Mark a task as completed
+	 * @param id Task ID
+	 * @param success Whether the task was successful
+	 * @param toolCalls Total tool calls
+	 * @param toolFailures Total tool failures
+	 */
+	completeTask(id: string, success: boolean, toolCalls: number = 0, toolFailures: number = 0): void {
+		const stmt = this.db.prepare(`
+      UPDATE tasks 
+      SET success = ?, total_tool_calls = ?, total_tool_failures = ? 
+      WHERE id = ?
+    `)
+
+		stmt.run(success ? 1 : 0, toolCalls, toolFailures, id)
+	}
+
+	/**
+	 * Add a metric to a task
+	 * @param taskId Task ID
+	 * @param name Metric name
+	 * @param value Metric value
+	 */
+	addMetric(taskId: string, name: string, value: number): void {
+		const stmt = this.db.prepare(`
+      INSERT INTO metrics (task_id, name, value)
+      VALUES (?, ?, ?)
+    `)
+
+		stmt.run(taskId, name, value)
+	}
+
+	/**
+	 * Add a tool call record
+	 * @param taskId Task ID
+	 * @param toolName Tool name
+	 * @param callCount Number of calls
+	 * @param failureCount Number of failures
+	 */
+	addToolCall(taskId: string, toolName: string, callCount: number, failureCount: number): void {
+		const stmt = this.db.prepare(`
+      INSERT INTO tool_calls (task_id, tool_name, call_count, failure_count)
+      VALUES (?, ?, ?, ?)
+    `)
+
+		stmt.run(taskId, toolName, callCount, failureCount)
+	}
+
+	/**
+	 * Add a file record
+	 * @param taskId Task ID
+	 * @param filePath File path
+	 * @param status File status (created, modified, deleted)
+	 */
+	addFile(taskId: string, filePath: string, status: "created" | "modified" | "deleted"): void {
+		const stmt = this.db.prepare(`
+      INSERT INTO files (task_id, path, status)
+      VALUES (?, ?, ?)
+    `)
+
+		stmt.run(taskId, filePath, status)
+	}
+
+	/**
+	 * Get all runs
+	 * @returns Array of runs
+	 */
+	getRuns(): any[] {
+		const stmt = this.db.prepare(`
+      SELECT * FROM runs ORDER BY timestamp DESC
+    `)
+
+		return stmt.all()
+	}
+
+	/**
+	 * Get all tasks for a run
+	 * @param runId Run ID
+	 * @returns Array of tasks
+	 */
+	getRunTasks(runId: string): any[] {
+		const stmt = this.db.prepare(`
+      SELECT * FROM tasks WHERE run_id = ? ORDER BY timestamp ASC
+    `)
+
+		return stmt.all(runId)
+	}
+
+	/**
+	 * Get all metrics for a task
+	 * @param taskId Task ID
+	 * @returns Array of metrics
+	 */
+	getTaskMetrics(taskId: string): any[] {
+		const stmt = this.db.prepare(`
+      SELECT name, value FROM metrics WHERE task_id = ?
+    `)
+
+		return stmt.all(taskId)
+	}
+
+	/**
+	 * Get all tool calls for a task
+	 * @param taskId Task ID
+	 * @returns Array of tool calls
+	 */
+	getTaskToolCalls(taskId: string): any[] {
+		const stmt = this.db.prepare(`
+      SELECT tool_name, call_count, failure_count 
+      FROM tool_calls 
+      WHERE task_id = ?
+    `)
+
+		return stmt.all(taskId)
+	}
+
+	/**
+	 * Get all files for a task
+	 * @param taskId Task ID
+	 * @returns Array of files
+	 */
+	getTaskFiles(taskId: string): any[] {
+		const stmt = this.db.prepare(`
+      SELECT path, status FROM files WHERE task_id = ?
+    `)
+
+		return stmt.all(taskId)
+	}
+
+	/**
+	 * Close the database connection
+	 */
+	close(): void {
+		this.db.close()
+	}
+}

evals/cli/src/db/schema.ts

@@ -0,0 +1,47 @@
+/**
+ * SQL schema for the evaluation database
+ */
+export const SCHEMA = `
+CREATE TABLE IF NOT EXISTS runs (
+  id TEXT PRIMARY KEY,
+  timestamp INTEGER NOT NULL,
+  benchmark TEXT NOT NULL,
+  completed INTEGER NOT NULL DEFAULT 0
+);
+
+CREATE TABLE IF NOT EXISTS tasks (
+  id TEXT PRIMARY KEY,
+  run_id TEXT NOT NULL,
+  task_id TEXT NOT NULL,
+  timestamp INTEGER NOT NULL,
+  success INTEGER NOT NULL DEFAULT 0,
+  total_tool_calls INTEGER DEFAULT 0,
+  total_tool_failures INTEGER DEFAULT 0,
+  FOREIGN KEY (run_id) REFERENCES runs(id)
+);
+
+CREATE TABLE IF NOT EXISTS metrics (
+  id INTEGER PRIMARY KEY AUTOINCREMENT,
+  task_id TEXT NOT NULL,
+  name TEXT NOT NULL,
+  value REAL NOT NULL,
+  FOREIGN KEY (task_id) REFERENCES tasks(id)
+);
+
+CREATE TABLE IF NOT EXISTS tool_calls (
+  id INTEGER PRIMARY KEY AUTOINCREMENT,
+  task_id TEXT NOT NULL,
+  tool_name TEXT NOT NULL,
+  call_count INTEGER NOT NULL,
+  failure_count INTEGER NOT NULL,
+  FOREIGN KEY (task_id) REFERENCES tasks(id)
+);
+
+CREATE TABLE IF NOT EXISTS files (
+  id INTEGER PRIMARY KEY AUTOINCREMENT,
+  task_id TEXT NOT NULL,
+  path TEXT NOT NULL,
+  status TEXT NOT NULL,
+  FOREIGN KEY (task_id) REFERENCES tasks(id)
+);
+`

evals/cli/src/index.ts

@@ -0,0 +1,106 @@
+#!/usr/bin/env node
+import chalk from "chalk"
+import { Command } from "commander"
+import { reportHandler } from "./commands/report"
+import { runHandler } from "./commands/run"
+import { runDiffEvalHandler } from "./commands/runDiffEval"
+import { setupHandler } from "./commands/setup"
+
+// Create the CLI program
+const program = new Command()
+
+// Set up CLI metadata
+program.name("cline-eval").description("CLI tool for orchestrating Cline evaluations across multiple benchmarks").version("0.1.0")
+
+// Setup command
+program
+	.command("setup")
+	.description("Clone and set up benchmark repositories")
+	.option("-b, --benchmarks <benchmarks>", "Comma-separated list of benchmarks to set up", "exercism")
+	.action(async (options) => {
+		try {
+			await setupHandler(options)
+		} catch (error) {
+			console.error(chalk.red(`Error during setup: ${error instanceof Error ? error.message : String(error)}`))
+			process.exit(1)
+		}
+	})
+
+// Run command
+program
+	.command("run")
+	.description("Run evaluations")
+	.option("-b, --benchmark <benchmark>", "Specific benchmark to run")
+	.option("-c, --count <count>", "Number of tasks to run", parseInt)
+	.action(async (options) => {
+		try {
+			await runHandler(options)
+		} catch (error) {
+			console.error(chalk.red(`Error during run: ${error instanceof Error ? error.message : String(error)}`))
+			process.exit(1)
+		}
+	})
+
+// Report command
+program
+	.command("report")
+	.description("Generate reports")
+	.option("-f, --format <format>", "Report format (json, markdown)", "markdown")
+	.option("-o, --output <path>", "Output path for the report")
+	.action(async (options) => {
+		try {
+			await reportHandler(options)
+		} catch (error) {
+			console.error(chalk.red(`Error generating report: ${error instanceof Error ? error.message : String(error)}`))
+			process.exit(1)
+		}
+	})
+
+// Run-diff-eval command
+program
+	.command("run-diff-eval")
+	.description("Run the diff editing evaluation suite")
+	.option("--test-path <path>", "Path to the directory containing test case JSON files")
+	.option("--output-path <path>", "Path to the directory to save the test output JSON files")
+	.option("--model-ids <model_ids>", "Comma-separated list of model IDs to test")
+	.option("--system-prompt-name <name>", "The name of the system prompt to use", "basicSystemPrompt")
+	.option(
+		"-n, --valid-attempts-per-case <number>",
+		"Number of valid attempts per test case per model (will retry until this many valid attempts are collected)",
+		"1",
+	)
+	.option("--max-attempts-per-case <number>", "Maximum total attempts per test case (default: 10x valid attempts)")
+	.option("--max-cases <number>", "Maximum number of test cases to run (limits total cases loaded)")
+	.option("--parsing-function <name>", "The parsing function to use", "parseAssistantMessageV2")
+	.option("--diff-edit-function <name>", "The diff editing function to use", "diff-06-26-25")
+	.option("--thinking-budget <tokens>", "Set the thinking tokens budget", "0")
+	.option("--provider <provider>", "API provider to use (openrouter, openai)", "openrouter")
+	.option("--parallel", "Run tests in parallel", false)
+	.option("--replay", "Run evaluation from a pre-recorded LLM output, skipping the API call", false)
+	.option("--replay-run-id <run_id>", "The ID of the run to replay from the database")
+	.option("--diff-apply-file <filename>", "The name of the diff apply file to use for the replay")
+	.option("--save-locally", "Save results to local JSON files in addition to database", false)
+	.option("-v, --verbose", "Enable verbose logging", false)
+	.action(async (options) => {
+		try {
+			const fullOptions = {
+				...options,
+				validAttemptsPerCase: parseInt(options.validAttemptsPerCase, 10),
+				maxAttemptsPerCase: options.maxAttemptsPerCase ? parseInt(options.maxAttemptsPerCase, 10) : undefined,
+				thinkingBudget: parseInt(options.thinkingBudget, 10),
+				maxCases: options.maxCases ? parseInt(options.maxCases, 10) : undefined,
+			}
+			await runDiffEvalHandler(fullOptions)
+		} catch (error) {
+			console.error(chalk.red(`Error during diff eval run: ${error instanceof Error ? error.message : String(error)}`))
+			process.exit(1)
+		}
+	})
+
+// Parse command line arguments
+program.parse(process.argv)
+
+// If no arguments provided, show help
+if (process.argv.length === 2) {
+	program.help()
+}

evals/cli/src/utils/markdown.ts

@@ -0,0 +1,80 @@
+import * as fs from "fs"
+
+/**
+ * Generate a markdown report from evaluation results
+ * @param summary Overall summary
+ * @param benchmarkReports Benchmark-specific reports
+ * @param outputPath Output file path
+ */
+export function generateMarkdownReport(summary: any, benchmarkReports: Record<string, any>, outputPath: string): void {
+	let markdown = `# Cline Evaluation Report\n\n`
+
+	// Generate summary section
+	markdown += `## Summary\n\n`
+	markdown += `- **Total Runs:** ${summary.runs}\n`
+	markdown += `- **Benchmarks:** ${summary.benchmarks.join(", ")}\n`
+	markdown += `- **Total Tasks:** ${summary.tasks}\n`
+	markdown += `- **Task Success Rate:** ${(summary.successRate * 100).toFixed(2)}%\n`
+	markdown += `- **Total Tests:** ${summary.totalTests}\n`
+	markdown += `- **Tests Passed:** ${summary.totalTestsPassed}\n`
+	markdown += `- **Tests Failed:** ${summary.totalTestsFailed}\n`
+	markdown += `- **Test Success Rate:** ${(summary.testSuccessRate * 100).toFixed(2)}%\n`
+	markdown += `- **Average Tokens:** ${Math.round(summary.averageTokens)}\n`
+	markdown += `- **Average Cost:** $${summary.averageCost.toFixed(4)}\n`
+	markdown += `- **Average Duration:** ${(summary.averageDuration / 1000).toFixed(2)}s\n`
+	markdown += `- **Total Tool Calls:** ${summary.totalToolCalls}\n`
+	markdown += `- **Tool Success Rate:** ${(summary.toolSuccessRate * 100).toFixed(2)}%\n\n`
+
+	// Generate tool usage section
+	markdown += `## Tool Usage\n\n`
+	markdown += `| Tool | Calls | Failures | Success Rate |\n`
+	markdown += `| ---- | ----- | -------- | ------------ |\n`
+
+	for (const [toolName, metrics] of Object.entries(summary.toolUsage)) {
+		const calls = (metrics as any).calls
+		const failures = (metrics as any).failures
+		const successRate = calls > 0 ? (1 - failures / calls) * 100 : 100
+
+		markdown += `| ${toolName} | ${calls} | ${failures} | ${successRate.toFixed(2)}% |\n`
+	}
+
+	// Generate benchmark results section
+	markdown += `\n## Benchmark Results\n\n`
+
+	for (const [benchmark, report] of Object.entries(benchmarkReports)) {
+		markdown += `### ${benchmark}\n\n`
+		markdown += `- **Runs:** ${report.runs}\n`
+		markdown += `- **Tasks:** ${report.tasks}\n`
+		markdown += `- **Task Success Rate:** ${(report.successRate * 100).toFixed(2)}%\n`
+		markdown += `- **Total Tests:** ${report.totalTests}\n`
+		markdown += `- **Tests Passed:** ${report.totalTestsPassed}\n`
+		markdown += `- **Tests Failed:** ${report.totalTestsFailed}\n`
+		markdown += `- **Test Success Rate:** ${(report.testSuccessRate * 100).toFixed(2)}%\n`
+		markdown += `- **Average Tokens:** ${Math.round(report.averageTokens)}\n`
+		markdown += `- **Average Cost:** $${report.averageCost.toFixed(4)}\n`
+		markdown += `- **Average Duration:** ${(report.averageDuration / 1000).toFixed(2)}s\n\n`
+	}
+
+	// Add charts using Mermaid
+	markdown += `## Charts\n\n`
+
+	// Success rate by benchmark chart
+	markdown += `### Success Rate by Benchmark\n\n`
+	markdown += "```mermaid\n"
+	markdown += "graph TD\n"
+	markdown += "  title[Success Rate by Benchmark]\n"
+	markdown += "  style title fill:none,stroke:none\n\n"
+
+	for (const [benchmark, report] of Object.entries(benchmarkReports)) {
+		const successRate = (report.successRate * 100).toFixed(2)
+		markdown += `  ${benchmark}[${benchmark}: ${successRate}%]\n`
+	}
+
+	markdown += "```\n\n"
+
+	// Add timestamp
+	markdown += `\n\n---\n\nReport generated on ${new Date().toISOString()}\n`
+
+	// Write markdown to file
+	fs.writeFileSync(outputPath, markdown)
+}

evals/cli/src/utils/results.ts

@@ -0,0 +1,79 @@
+import { v4 as uuidv4 } from "uuid"
+import { ResultsDatabase } from "../db"
+import { Task } from "../adapters/types"
+
+/**
+ * Store task result in the database
+ * @param runId The run ID
+ * @param task The task that was executed
+ * @param result The result from the test server
+ * @param verification The verification result
+ */
+export async function storeTaskResult(runId: string, task: Task, result: any, verification: any): Promise<void> {
+	const db = new ResultsDatabase()
+	const taskId = uuidv4()
+
+	try {
+		// Extract metrics from the result
+		const { metrics } = result
+		const totalToolCalls = metrics?.totalToolCalls || 0
+		const totalToolFailures = metrics?.totalToolFailures || 0
+
+		// Create task with tool metrics
+		db.createTask(taskId, runId, task.id)
+		db.completeTask(taskId, verification.success, totalToolCalls, totalToolFailures)
+
+		// Store metrics
+		if (metrics) {
+			// Store token metrics
+			if (metrics.tokensIn) db.addMetric(taskId, "tokensIn", metrics.tokensIn)
+			if (metrics.tokensOut) db.addMetric(taskId, "tokensOut", metrics.tokensOut)
+			if (metrics.cost) db.addMetric(taskId, "cost", metrics.cost)
+			if (metrics.duration) db.addMetric(taskId, "duration", metrics.duration)
+
+			// Store tool call metrics
+			if (metrics.toolCalls) {
+				for (const [toolName, callCount] of Object.entries(metrics.toolCalls)) {
+					const failureCount = metrics.toolFailures?.[toolName] || 0
+					db.addToolCall(taskId, toolName, callCount as number, failureCount)
+				}
+			}
+		}
+
+		// Store verification metrics
+		if (verification.metrics) {
+			for (const [key, value] of Object.entries(verification.metrics)) {
+				if (typeof value === "number") {
+					db.addMetric(taskId, key, value)
+				}
+			}
+		}
+
+		// Store file changes
+		if (result.files) {
+			// Store created files
+			if (result.files.created) {
+				for (const file of result.files.created) {
+					db.addFile(taskId, file, "created")
+				}
+			}
+
+			// Store modified files
+			if (result.files.modified) {
+				for (const file of result.files.modified) {
+					db.addFile(taskId, file, "modified")
+				}
+			}
+
+			// Store deleted files
+			if (result.files.deleted) {
+				for (const file of result.files.deleted) {
+					db.addFile(taskId, file, "deleted")
+				}
+			}
+		}
+	} finally {
+		// Close the database connection
+		db.close()
+	}
+}

evals/cli/tsconfig.json

@@ -0,0 +1,17 @@
+{
+	"compilerOptions": {
+		"target": "ES2020",
+		"module": "commonjs",
+		"lib": ["ES2020"],
+		"declaration": true,
+		"outDir": "dist",
+		"rootDir": "src",
+		"strict": true,
+		"esModuleInterop": true,
+		"skipLibCheck": true,
+		"forceConsistentCasingInFileNames": true,
+		"resolveJsonModule": true
+	},
+	"include": ["src/**/*"],
+	"exclude": ["node_modules", "dist"]
+}