images

.changeset/blue-moons-mate.md

@@ -0,0 +1,5 @@
+---
+"claude-dev": minor
+---
+
+Put all the navigation state and message handling and navigation functions in the extension state context instead of the app.tsx

.changeset/famous-shoes-beam.md

@@ -0,0 +1,5 @@
+---
+"claude-dev": minor
+---
+
+Migrate setActiveQuote to protobus

.changeset/five-bikes-impress.md

@@ -0,0 +1,5 @@
+---
+"claude-dev": patch
+---
+
+Fix edge case of changing language in settings

.changeset/funny-adults-tease.md

@@ -0,0 +1,5 @@
+---
+"claude-dev": patch
+---
+
+feat: added gemini flash 05-20

.changeset/good-dingos-try.md

@@ -0,0 +1,5 @@
+---
+"claude-dev": minor
+---
+
+Migrate toggleToolAutoApprove to protobus

.changeset/hot-dodos-rhyme.md

@@ -0,0 +1,5 @@
+---
+"claude-dev": patch
+---
+
+togglePlanActMode protobus migration

.changeset/hungry-worms-draw.md

@@ -0,0 +1,5 @@
+---
+"claude-dev": minor
+---
+
+Migrate toggleClineRule to protobus

.changeset/little-gifts-invite.md

@@ -0,0 +1,5 @@
+---
+"claude-dev": patch
+---
+
+taskCompletionViewChanges protobus migration

.changeset/long-zebras-yawn.md

@@ -0,0 +1,5 @@
+---
+"claude-dev": patch
+---
+
+copyToClipboard protobus migration

.changeset/loud-worms-applaud.md

@@ -0,0 +1,5 @@
+---
+"claude-dev": minor
+---
+
+Migrate updateTerminalConnectionTimeout to protobus

.changeset/nice-cobras-pretend.md

@@ -0,0 +1,5 @@
+---
+"claude-dev": patch
+---
+
+Fixing bug in toggle plan and act

.changeset/old-crews-reply.md

@@ -0,0 +1,5 @@
+---
+"claude-dev": patch
+---
+
+scrollToSetting protobus migration

.changeset/pink-deers-burn.md

@@ -0,0 +1,5 @@
+---
+"claude-dev": minor
+---
+
+migrate accountLogoutClicked to protobus

.changeset/seven-dingos-appear.md

@@ -0,0 +1,5 @@
+---
+"claude-dev": patch
+---
+
+openMention protobus migration

.changeset/silver-seals-care.md

@@ -0,0 +1,5 @@
+---
+"claude-dev": minor
+---
+
+Migrate toggleWindsurfRule to protobus

.changeset/six-candles-check.md

@@ -0,0 +1,5 @@
+---
+"claude-dev": minor
+---
+
+Migrate restartMcpServer to protobus

.changeset/six-carpets-teach.md

@@ -0,0 +1,5 @@
+---
+"claude-dev": minor
+---
+
+Migrate silentlyRefreshMcpMarketplace to protobus

.changeset/six-walls-doubt.md

@@ -1,5 +0,0 @@
----
-"claude-dev": patch
----
-
-telemtrySetting protobus migration

.changeset/sour-ravens-sin.md

@@ -0,0 +1,5 @@
+---
+"claude-dev": minor
+---
+
+Migrate toggleCursorRule to protobus

.changeset/spicy-carpets-begin.md

@@ -1,5 +0,0 @@
----
-"claude-dev": patch
----
-
-clearAllTaskHistory protobus migration

.changeset/spotty-jeans-march.md

@@ -0,0 +1,5 @@
+---
+"claude-dev": patch
+---
+
+browserRelaunchResult protobus migration

.changeset/tame-windows-compete.md

@@ -0,0 +1,5 @@
+---
+"claude-dev": patch
+---
+
+Remind VS code users of clines existence(Open Cline on AutoUpdate+KeyboardShortcuts+Lightbulb icons)

.changeset/wild-gifts-switch.md

@@ -0,0 +1,5 @@
+---
+"claude-dev": minor
+---
+
+Migrate the deleteMcpServer message to protobus

.clinerules/cline-overview.md

@@ -164,7 +164,6 @@ Key providers include:
 - **OpenRouter**: Meta-provider supporting multiple model providers
 - **AWS Bedrock**: Integration with Amazon's AI services
 - **Gemini**: Google's AI models
-- **Cerebras**: High-performance inference with Llama, Qwen, and DeepSeek models
 - **Ollama**: Local model hosting
 - **LM Studio**: Local model hosting
 - **VSCode LM**: VSCode's built-in language models

.clinerules/protobuf-development.md

@@ -1,89 +0,0 @@
-# Cline Protobuf Development Guide
-
-This guide outlines how to add new gRPC endpoints for communication between the webview (frontend) and the extension host (backend).
-
-## Overview
-
-Cline uses [Protobuf](https://protobuf.dev/) to define a strongly-typed API, ensuring efficient and type-safe communication. All definitions are in the `/proto` directory. The compiler and plugins are included as project dependencies, so no manual installation is needed.
-
-## Key Concepts & Best Practices
-
--   **File Structure**: Each feature domain should have its own `.proto` file (e.g., `account.proto`, `task.proto`).
--   **Message Design**:
-    -   For simple, single-value data, use the shared types in `proto/common.proto` (e.g., `StringRequest`, `Empty`, `Int64Request`). This promotes consistency.
-    -   For complex data structures, define custom messages within the feature's `.proto` file (see `task.proto` for examples like `NewTaskRequest`).
--   **Naming Conventions**:
-    -   Services: `PascalCaseService` (e.g., `AccountService`).
-    -   RPCs: `camelCase` (e.g., `accountEmailIdentified`).
-    -   Messages: `PascalCase` (e.g., `StringRequest`).
--   **Streaming**: For server-to-client streaming, use the `stream` keyword on the response type. See `subscribeToAuthCallback` in `account.proto` for an example.
-
----
-
-## 4-Step Development Workflow
-
-Here’s how to add a new RPC, using `scrollToSettings` as an example.
-
-### 1. Define the RPC in a `.proto` File
-
-Add your service method to the appropriate file in the `proto/` directory.
-
-**File: `proto/ui.proto`**
-```proto
-service UiService {
-  // ... other RPCs
-  // Scrolls to a specific settings section in the settings view
-  rpc scrollToSettings(StringRequest) returns (KeyValuePair);
-}
-```
-Here, we use the common `StringRequest` and `KeyValuePair` types.
-
-### 2. Compile Definitions
-
-After editing a `.proto` file, regenerate the TypeScript code. From the project root, run:
-```bash
-npm run protos
-```
-This command compiles all `.proto` files and outputs the generated code to `src/generated/` and `src/shared/`. Do not edit these generated files manually.
-
-### 3. Implement the Backend Handler
-
-Create the RPC implementation in the backend. Handlers are located in `src/core/controller/[service-name]/`.
-
-**File: `src/core/controller/ui/scrollToSettings.ts`**
-```typescript
-import { Controller } from ".."
-import { StringRequest, KeyValuePair } from "../../../shared/proto/common"
-
-/**
- * Executes a scroll to settings action
- * @param controller The controller instance
- * @param request The request containing the ID of the settings section to scroll to
- * @returns KeyValuePair with action and value fields for the UI to process
- */
-export async function scrollToSettings(controller: Controller, request: StringRequest): Promise<KeyValuePair> {
-	return KeyValuePair.create({
-		key: "scrollToSettings",
-		value: request.value || "",
-	})
-}
-```
-
-### 4. Call the RPC from the Webview
-
-Call the new RPC from a React component in `webview-ui/`. The generated client makes this simple.
-
-**File: `webview-ui/src/components/browser/BrowserSettingsMenu.tsx`** (Example)
-```tsx
-import { UiServiceClient } from "../../../services/grpc"
-import { StringRequest } from "../../../../shared/proto/common"
-
-// ... inside a React component
-const handleMenuClick = async () => {
-    try {
-        await UiServiceClient.scrollToSettings(StringRequest.create({ value: "browser" }))
-    } catch (error) {
-        console.error("Error scrolling to browser settings:", error)
-    }
-}
-```

.eslintrc.json

@@ -5,7 +5,7 @@
 		"ecmaVersion": 6,
 		"sourceType": "module"
 	},
-	"plugins": ["@typescript-eslint", "eslint-rules"],
+	"plugins": ["@typescript-eslint"],
 	"rules": {
 		"@typescript-eslint/naming-convention": [
 			"warn",
@@ -19,16 +19,7 @@
 		"eqeqeq": "warn",
 		"no-throw-literal": "warn",
 		"semi": "off",
-		"react-hooks/exhaustive-deps": "off",
-		"eslint-rules/no-protobuf-object-literals": "error",
-		"eslint-rules/no-grpc-client-object-literals": "error",
-		"no-restricted-syntax": [
-			"error",
-			{
-				"selector": "VariableDeclarator[id.type=\"ObjectPattern\"][init.object.name=\"process\"][init.property.name=\"env\"]",
-				"message": "Use process.env.VARIABLE_NAME directly instead of destructuring"
-			}
-		]
+		"react-hooks/exhaustive-deps": "off"
 	},
 	"ignorePatterns": ["out", "dist", "**/*.d.ts"]
 }

.github/ISSUE_TEMPLATE/config.yml

@@ -1,4 +1,4 @@
-blank_issues_enabled: false
+blank_issues_enabled: true
 contact_links:
     - name: ✨ Feature Request
       url: https://github.com/cline/cline/discussions/categories/feature-requests?discussions_q=is%3Aopen+category%3A%22Feature+Requests%22+sort%3Atop
@@ -6,3 +6,6 @@ contact_links:
     - name: 👋 Cline Discord
       url: https://discord.gg/cline
       about: Join our Discord community for discussions and support
+    - name: ❓ Other Questions?
+      url: https://x.com/sdrzn
+      about: Contact the developer on X @sdrzn for other inquiries

.github/ISSUE_TEMPLATE/feature_contribution.yml

@@ -1,116 +0,0 @@
-name: 💡 Feature Proposal & Contribution
-description: Propose a new feature or improvement, and optionally offer to implement feature as a contributor
-labels: ["proposal"]
-body:
-    - type: markdown
-      attributes:
-          value: |
-              **Feature Proposal & Contribution for Cline**
-
-              Thank you for proposing a feature or improvement for Cline! This template helps us understand the problem, evaluate the solution, and coordinate implementation.
-
-              **For detailed proposals:** Please provide comprehensive information to enable fast prioritization and discussion.
-              **For contribution offers:** You can indicate your willingness to implement the feature yourself.
-
-              Before submitting:
-              - Search existing [Issues](https://github.com/cline/cline/issues) and [Discussions](https://github.com/cline/cline/discussions) to avoid duplicates
-              - Read the [Contributing Guide](https://github.com/cline/cline/blob/main/CONTRIBUTING.md) if you plan to contribute
-              - Don't start implementation until the proposal is reviewed and approved
-
-    - type: textarea
-      id: problem-description
-      attributes:
-          label: What problem does this solve?
-          description: |
-              Describe the problem clearly from a user's point of view. Focus on why this matters, who it affects, and when it occurs.
-
-              ✅ Good examples:
-              - "LLM provider returns 400 error when nearing the context window instead of truncating"
-              - "Submit button is invisible in dark mode"
-              - "Users can't easily share their Cline configurations with team members"
-
-              ❌ Avoid vague descriptions:
-              - "Performance is bad"
-              - "UI needs work"
-
-              Your description should include:
-              - Who is affected?
-              - When does it happen?
-              - What's the current vs expected behavior?
-              - What is the impact?
-          placeholder: Be specific about the problem, who it affects, and the impact.
-      validations:
-          required: true
-
-    - type: textarea
-      id: proposed-solution
-      attributes:
-          label: What's the proposed solution?
-          description: |
-              Describe how the problem should be solved. Be specific about UX, system behavior, and any flows that would change.
-
-              ✅ Good examples:
-              - "Add error handling immediately after attempting to create the llm stream and retry after manually truncating"
-              - "Update button styling to ensure contrast in all themes"
-              - "Add export/import functionality in settings with JSON format"
-
-              ❌ Avoid vague solutions:
-              - "Improve performance"
-              - "Fix the bug"
-
-              Your solution should include:
-              - What exactly will change?
-              - How will users interact with it?
-              - What's the expected outcome?
-          placeholder: Describe the proposed changes and how they solve the problem.
-      validations:
-          required: false
-
-    - type: dropdown
-      id: contribution-intent
-      attributes:
-          label: Are you interested in implementing this?
-          description: Let us know if you'd like to contribute to this feature
-          options:
-              - "No, just proposing the idea"
-              - "Yes, I'd like to implement this myself"
-              - "Yes, I'd like to collaborate with others"
-              - "Maybe, depending on complexity and guidance"
-      validations:
-          required: false
-
-    - type: textarea
-      id: implementation-approach
-      attributes:
-          label: Implementation approach (if contributing)
-          description: |
-              **Only fill this out if you selected "Yes" above.**
-
-              How do you plan to implement this? Include:
-              - High-level technical approach
-              - Files/components that would be affected
-              - Any new dependencies required
-              - Potential challenges or considerations you've identified
-
-              This helps us provide better guidance and ensures alignment before you start coding.
-          placeholder: "My implementation approach would be..."
-
-    - type: checkboxes
-      id: checklist
-      attributes:
-          label: Proposal checklist
-          options:
-              - label: I've checked for existing issues or related proposals
-                required: true
-              - label: I understand this needs review before implementation can start
-                required: true
-
-    - type: checkboxes
-      id: contribution-checklist
-      attributes:
-          label: Contribution checklist (if contributing)
-          description: Only check these if you plan to contribute
-          options:
-              - label: I've read the [Contributing Guide](https://github.com/cline/cline/blob/main/CONTRIBUTING.md)
-              - label: I'm willing to make changes based on feedback
-              - label: I understand the code review process and requirements

.github/pull_request_template.md

@@ -1,47 +1,10 @@
-<!--
-Thank you for contributing to Cline!
-
-⚠️ Important: Before submitting this PR, please ensure you have:
-- Opened an issue and discussed your proposed changes with the community / contributors
-- Received approval from a core Cline contributor prior to proceeding with the implementation
-- Link the associated issue in the "Related Issue" section
-
-Limited exceptions:
-Small bug fixes, typo corrections, minor wording improvements, or simple type fixes that don't change functionality may be submitted directly.
-
-Why this requirement?
-We deeply appreciate all community contributions - they are the core reason we're able to operate successfully and keep innovating! We welcome community input and want to make it as easy as possible for people to submit quality work. This process helps our core maintainers review new ideas faster and saves contributor time by ensuring you have the go-ahead before spending time on implementation.
--->
-
-### Related Issue
-
-<!-- Replace XXXX with the issue number that this PR addresses -->
-**Issue:** #XXXX
-
 ### Description
 
-<!-- 
-Help reviewers understand your changes by making this PR readable and well-organized:
-
-- What problem does this PR solve?
-- Why were these changes introduced and what purpose do they serve?
-- For larger changes, provide context about your approach and reasoning
-
-Small PRs may need minimal description, but larger changes benefit from explaining where you're coming from. Much of this context can be in the linked issue above, so feel free to reference it rather than repeating everything here.
--->
+<!-- Describe your changes in detail. What problem does this PR solve? -->
 
 ### Test Procedure
 
-<!-- 
-Please walk us through your testing approach and thought process. This helps reviewers understand that you've thoroughly considered the impact of your changes:
-
-- How did you test this change?
-- What could potentially break and how did you verify it doesn't?
-- What existing functionality might be affected and how did you check it still works?
-- Why are you confident this is ready for merge?
-
-We're not looking for exhaustive documentation - just evidence that you've thought through the implications of your changes and tested accordingly.
--->
+<!-- How did you test this? Are you confident that it will not introduce bugs? If so, why? -->
 
 ### Type of Change
 
@@ -66,15 +29,7 @@ We're not looking for exhaustive documentation - just evidence that you've thoug
 
 ### Screenshots
 
-<!-- 
-Help reviewers quickly understand your changes:
-
-- **UI Changes**: Please include screenshots showing before/after states
-- **Complex Workflows**: Consider uploading a screen recording (video) if your changes involve multiple steps or state transitions
-- **Backend Changes**: Not required, but feel free to include terminal output or other evidence that demonstrates functionality
-
-This helps reviewers see what you've built without having to pull down and test your branch first.
--->
+<!-- For UI changes, add screenshots here -->
 
 ### Additional Notes
 

.github/workflows/publish.yml

@@ -11,10 +11,6 @@ on:
                 options:
                     - pre-release
                     - release
-            tag:
-                description: "Enter existing tag to publish (e.g., v3.1.2)"
-                required: true
-                type: string
 
 permissions:
     contents: write
@@ -34,8 +30,6 @@ jobs:
 
         steps:
             - uses: actions/checkout@v4
-              with:
-                  ref: ${{ github.event.inputs.tag }}
 
             - name: Setup Node.js
               uses: actions/setup-node@v4
@@ -75,20 +69,14 @@ jobs:
                   VERSION=$(node -p "require('./package.json').version")
                   echo "version=$VERSION" >> $GITHUB_OUTPUT
 
-            - name: Validate Tag
-              id: validate_tag
+            - name: Create Git Tag
+              id: create_tag
               run: |
-                  TAG="${{ github.event.inputs.tag }}"
-                  echo "tag=$TAG" >> $GITHUB_OUTPUT
-                  echo "Using existing tag: $TAG"
-
-                  # Verify the tag exists
-                  if ! git rev-parse "$TAG" >/dev/null 2>&1; then
-                    echo "Error: Tag '$TAG' does not exist in the repository"
-                    exit 1
-                  fi
-
-                  echo "Tag '$TAG' validated successfully"
+                  VERSION=v${{ steps.get_version.outputs.version }}
+                  echo "tag=$VERSION" >> $GITHUB_OUTPUT
+                  echo "Tagging with $VERSION"
+                  git tag "$VERSION"
+                  git push origin "$VERSION"
 
             - name: Package and Publish Extension
               env:
@@ -118,7 +106,7 @@ jobs:
             - name: Create GitHub Release
               uses: softprops/action-gh-release@v1
               with:
-                  tag_name: ${{ steps.validate_tag.outputs.tag }}
+                  tag_name: ${{ steps.create_tag.outputs.tag }}
                   files: "*.vsix"
                   # body: ${{ steps.changelog.outputs.content }}
                   generate_release_notes: true

.github/workflows/stale.yml

@@ -1,25 +0,0 @@
-# This workflow will only label and/or close 30 issues at a time in order to avoid exceeding a rate limit.
-# More info: https://docs.github.com/en/actions/use-cases-and-examples/project-management/closing-inactive-issues
-name: Close inactive issues
-on:
-    schedule:
-        - cron: "30 1 * * *"
-
-jobs:
-    close-issues:
-        runs-on: ubuntu-latest
-        permissions:
-            issues: write
-            pull-requests: write
-        steps:
-            - uses: actions/stale@v9
-              with:
-                  days-before-issue-stale: 60
-                  days-before-issue-close: 14
-                  stale-issue-label: "stale"
-                  stale-issue-message: "This issue is stale because it has been open for 60 days with no activity."
-                  close-issue-message: "This issue was closed because it has been inactive for 14 days since being marked as stale."
-                  days-before-pr-stale: -1
-                  days-before-pr-close: -1
-                  exempt-issue-labels: "pinned,security"
-                  repo-token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/test-stale.yml

@@ -1,32 +0,0 @@
-name: Test Stale Issues Workflow
-on:
-    workflow_dispatch:
-        inputs:
-            days-before-stale:
-                description: "Days before an issue becomes stale"
-                required: true
-                default: "1"
-            days-before-close:
-                description: "Days before a stale issue is closed"
-                required: true
-                default: "1"
-
-jobs:
-    test-stale:
-        runs-on: ubuntu-latest
-        permissions:
-            issues: write
-            pull-requests: write
-        steps:
-            - uses: actions/stale@28ca103
-              with:
-                  days-before-issue-stale: ${{ github.event.inputs.days-before-stale }}
-                  days-before-issue-close: ${{ github.event.inputs.days-before-close }}
-                  stale-issue-label: "stale"
-                  stale-issue-message: "This issue is stale because it has been open for ${{ github.event.inputs.days-before-stale }} days with no activity."
-                  close-issue-message: "This issue was closed because it has been inactive for ${{ github.event.inputs.days-before-close }} days since being marked as stale."
-                  days-before-pr-stale: -1
-                  days-before-pr-close: -1
-                  exempt-issue-labels: "pinned,security"
-                  repo-token: ${{ secrets.GITHUB_TOKEN }}
-                  debug-only: true

.github/workflows/test.yml

@@ -68,12 +68,6 @@ jobs:
               if: steps.webview-cache.outputs.cache-hit != 'true'
               run: cd webview-ui && npm ci
 
-            - name: Install local modules on windows
-              if: runner.os == 'Windows' && steps.root-cache.outputs.cache-hit == 'true'
-              run: |
-                  npm install eslint-plugin-eslint-rules 
-                  cd webview-ui/ && npm install eslint-plugin-eslint-rules
-
             - name: Set up NPM on Windows
               if: runner.os == 'Windows'
               run: |
@@ -92,9 +86,8 @@ jobs:
             - name: Build Tests and Extension
               run: npm run pretest
 
-            # Unit Tests disabled due to module system conflicts between backend and webview-ui
-            # - name: Unit Tests
-            #   run: npm run test:unit
+            - name: Unit Tests
+              run: npm run test:unit
 
             # Run extension tests with coverage
             - name: Extension Tests with Coverage

.gitignore

@@ -20,22 +20,3 @@ coverage
 !.github/scripts/coverage/
 
 *evals.env
-
-# Generated files
-src/generated/
-# Core
-src/core/controller/*/methods.ts
-src/core/controller/*/index.ts
-src/core/controller/grpc-service-config.ts
-# Shared
-src/shared/proto/*.ts
-src/shared/proto/host/*.ts
-# Webview
-webview-ui/src/services/grpc-client.ts
-# Host bridge
-src/hosts/vscode/*/methods.ts
-src/hosts/vscode/*/index.ts
-src/hosts/vscode/client/host-grpc-client.ts
-src/hosts/vscode/host-grpc-service-config.ts
-src/standalone/server-setup.ts
-gemini-cli

.husky/pre-commit

@@ -9,9 +9,9 @@ npm run lint || {
 
 # Run Prettier
 echo "Running Prettier..."
-npx lint-staged --verbose || {
-     echo "❌ Prettier failed. Please fix the errors and try committing again."
-     exit 1
- }
+npm run format || {
+    echo "❌ Prettier check failed. Run 'npm run format:fix' to automatically fix formatting issues."
+    exit 1
+}
 
 echo "✅ All checks passed!"

.mocharc.json

@@ -1,6 +1,6 @@
 {
 	"extension": ["ts"],
-	"spec": ["src/**/__tests__/*.ts", "eslint-rules/__tests__/**/*.test.ts"],
+	"spec": "src/**/__tests__/*.ts",
 	"require": ["ts-node/register", "source-map-support/register", "./src/test/requires.ts"],
 	"recursive": true
 }

.prettierignore

@@ -3,6 +3,3 @@ node_modules
 webview-ui/build/
 *.md
 package-lock.json
-src/core/prompts/system.ts
-src/core/prompts/model_prompts/claude4.ts
-evals/

.vscode/launch.json

@@ -42,7 +42,7 @@
 		{
 			"type": "node",
 			"request": "launch",
-			"name": "Run Standalone Service",
+			"name": "Run Standalone Extension",
 			"skipFiles": ["<node_internals>/**"],
 			"sourceMaps": true,
 			"resolveSourceMapLocations": ["${workspaceFolder}/**", "!**/node_modules/**"],
@@ -50,12 +50,9 @@
 			"outFiles": ["${workspaceFolder}/dist-standalone/**/*.js"],
 			"preLaunchTask": "compile-standalone",
 			"env": {
-				// Turns on grpc debug log.
-				//"GRPC_TRACE": "all",
-				//"GRPC_VERBOSITY": "DEBUG",
-				"NODE_PATH": "${workspaceFolder}/dist-standalone/node_modules",
-
-				"HOST_BRIDGE_ADDRESS": "localhost:50052"
+				"GRPC_TRACE": "all",
+				"GRPC_VERBOSITY": "DEBUG",
+				"NODE_PATH": "${workspaceFolder}/dist-standalone/node_modules"
 			},
 			"program": "standalone.js"
 		}

.vscode/tasks.json

@@ -128,25 +128,7 @@
 			"type": "npm",
 			"script": "watch:esbuild",
 			"group": "build",
-			"problemMatcher": {
-				"pattern": [
-					{
-						"regexp": "^✘ \\[ERROR\\] (.*)$",
-						"message": 1
-					},
-					{
-						"regexp": "^\\s+(.*):(\\d+):(\\d+):$",
-						"file": 1,
-						"line": 2,
-						"column": 3
-					}
-				],
-				"background": {
-					"activeOnStart": true,
-					"beginsPattern": "^\\[watch\\] build started$",
-					"endsPattern": "^\\[watch\\] build finished$"
-				}
-			},
+			"problemMatcher": "$esbuild-watch",
 			"isBackground": true,
 			"label": "npm: watch:esbuild",
 			"dependsOn": ["npm: protos"],
@@ -164,25 +146,7 @@
 			"type": "npm",
 			"script": "watch:esbuild:test",
 			"group": "build",
-			"problemMatcher": {
-				"pattern": [
-					{
-						"regexp": "^✘ \\[ERROR\\] (.*)$",
-						"message": 1
-					},
-					{
-						"regexp": "^\\s+(.*):(\\d+):(\\d+):$",
-						"file": 1,
-						"line": 2,
-						"column": 3
-					}
-				],
-				"background": {
-					"activeOnStart": true,
-					"beginsPattern": "^\\[watch\\] build started$",
-					"endsPattern": "^\\[watch\\] build finished$"
-				}
-			},
+			"problemMatcher": "$esbuild-watch",
 			"isBackground": true,
 			"label": "npm: watch:esbuild:test",
 			"dependsOn": ["npm: protos"],

CHANGELOG.md

@@ -1,157 +1,5 @@
 # Changelog
 
-## [3.17.16]
-
--   Fix Claude Code provider error handling for incomplete messages during long-running tasks (Thanks @BarreiroT!)
--   Add taskId as metadata to LiteLLM API requests for better request tracing (Thanks @jorgegarciarey!)
-
-## [3.17.15]
-
--   Fix LiteLLM provider to properly respect selected model IDs when switching between Plan and Act modes (Thanks @sammcj!)
--   Fix chat input being cleared when switching between Plan/Act modes without sending a message (Thanks @BarreiroT!)
--   Fix MCP server name display to avoid showing "undefined" for SSE servers, preventing tool/resource invocation failures (Thanks @ramybenaroya!)
--   Fix AWS Bedrock provider by removing deprecated custom model encoding (Thanks @watany-dev!)
--   Fix timeline tooltips for followup messages and improve color retrieval code (Thanks @char8x!)
--   Improve accessibility by making task header buttons properly announced by screen readers (Thanks @yncat!)
--   Improve accessibility by adding proper state reporting for Plan/Act mode switch for screen readers (Thanks @yncat!)
--   Prevent reading development environment variables from user's environment (Thanks @BarreiroT!)
-
-## [3.17.14]
-
--   Add Claude Code as a new API provider, allowing integration with Anthropic's Claude Code CLI tool and Claude Max Plan (Thanks @BarreiroT!)
--   Add SAP AI Core as a new API provider with support for Claude and GPT models (Thanks @schardosin!)
--   Add configurable default terminal profile setting, allowing users to specify which terminal Cline should use (Thanks @valinha!)
--   Add terminal output size constraint setting to limit how much terminal output is processed
--   Add MCP Rich Display settings to the settings page for persistent configuration (Thanks @Vl4diC0de!)
--   Improve copy button functionality with refactored reusable components (Thanks @shouhanzen!)
--   Improve AWS Bedrock provider by removing deprecated dependency and using standard AWS SDK (Thanks @watany-dev!)
--   Fix list_files tool to properly return files when targeting hidden directories
--   Fix search and replace edge case that could cause file deletion, making the algorithm more lenient for models using different diff formats
--   Fix task restoration issues that could occur when resuming interrupted tasks
--   Fix checkpoint saving to properly track all file changes
--   Improve file context warnings to reduce diff edit errors when resuming restored tasks
--   Clear chat input when switching between Plan/Act modes within a task
--   Exclude .clinerules files from checkpoint tracking
-
-## [3.17.13]
-
--   Add Thinking UX for Gemini models, providing visual feedback during model reasoning
--   Add support for Notifications MCP integration with Cline
--   Add prompt caching indicator for Grok 3 models
--   Sort MCP marketplace by newest listings by default for easier discovery of recent servers
--   Update O3 model family pricing to reflect latest OpenAI rates
--   Remove '-beta' suffix from Grok model identifiers
--   Fix AWS Bedrock provider by removing deprecated Anthropic-Bedrock SDK (Thanks @watany-dev!)
--   Fix menu display issue for terminal timeout settings
--   Improve chat input field styling and behavior
-
-## [3.17.12]
-
--   **Free Grok Model Available!** Access Grok 3 completely free through the Cline provider
--   Add collapsible MCP response panels to keep conversations focused on the main AI responses while still allowing access to detailed MCP output (Thanks @valinha!)
--   Prioritize active files (open tabs) at the top of the file context menu when using @ mentions (Thanks @abeatrix!)
--   Fix context menu to properly default to "File" option instead of incorrectly selecting "Git Commits"
--   Fix diff editing to handle out-of-order SEARCH/REPLACE blocks, improving reliability with models that don't follow strict ordering
--   Fix telemetry warning popup appearing repeatedly for users who have telemetry disabled
-
-## [3.17.11]
-
--   Add support for Gemini 2.5 Pro Preview 06-05 model to Vertex AI and Google Gemini providers
-
-## [3.17.10]
-
--   Add support for Qwen 3 series models with thinking mode options (Thanks @Jonny-china!)
--   Add new AskSage models: Claude 4 Sonnet, Claude 4 Opus, GPT 4.1, Gemini 2.5 Pro (Thanks @swhite24!)
--   Add VSCode walkthrough to help new users get started with Cline
--   Add support for streamable MCP servers
--   Improve Ollama model selection with filterable dropdown instead of radio buttons (Thanks @paulgear!)
--   Add setting to disable aggressive terminal reuse to help users experiencing task lockout issues
--   Fix settings dialog applying changes even when cancel button is clicked
-
-## [3.17.9]
-
--   Aligning Cline to work with Claude 4 model family (Experimental)
--   Add task timeline scrolling feature
--   Add support for uploading CSV and XLSX files for data analysis and processing
--   Add stable Grok-3 models to xAI provider (grok-3, grok-3-fast, grok-3-mini, grok-3-mini-fast) and update default model from grok-3-beta to grok-3 (Thanks @PeterDaveHello!)
--   Add new models to Vertex AI provider
--   Add new model to Nebius AI Studio
--   Remove hard-coded temperature from LM Studio API requests and add support for reasoning_content in LM Studio responses
--   Display delay information when retrying API calls for better user feedback
--   Fix AWS Bedrock credential caching issue where externally updated credentials (e.g., by AWS Identity Manager) were not detected, requiring extension restart (Thanks @DaveFres!)
--   Fix search tool overloading conversation with massive outputs by setting maximum byte limit for responses
--   Fix checkpoints functionality
--   Fix token counting for xAI provider
--   Fix Ollama provider issues
--   Fix window title display for Windows users
--   Improve chat box UI
-
-## [3.17.8]
-
--   Fix bug where terminal would get stuck and output "capture failure"
-
-## [3.17.7]
-
--   Fix diff editing reliability for Claude 4 family models by adding constraints to prevent errors with large replacements
-
-## [3.17.6]
-
--   Add Cerebras as a new API provider with 5 high-performance models including reasoning-capable models (Thanks @kevint-cerebras!)
--   Add support for uploading various file types (XML, JSON, TXT, LOG, MD, DOCX, IPYNB, PDF) alongside images
--   Add improved onboarding experience for new users with guided setup
--   Add prompt cache indicator for Gemini 2.5 Flash models
--   Update SambaNova provider with new model list and documentation links (Thanks @luisfucros!)
--   Fix diff editing support for Claude 4 family of models
--   Improve telemetry and analytics for better user experience insights
-
-## [3.17.5]
-
--   Fix issue with Claude 4 models where after several conversation turns, it would start making invalid diff edits
-
-## [3.17.4]
-
--   Fix thinking budget slider for Claude 4
-
-## [3.17.3]
-
--   Fix diff edit errors with Claude 4 models
-
-## [3.17.2]
-
--   Add support for Claude 4 models (Sonnet 4 and Opus 4) in AWS Bedrock and Vertex AI providers
--   Add support for global workflows, allowing workflows to be shared across workspaces with local workflows taking precedence
--   Fix settings page z-index UI issues that caused display problems
--   Fix AWS Bedrock environment variable handling to properly restore process.env after API calls (Thanks @DaveFres!)
-
-## [3.17.1]
-
--   Add prompt caching for Claude 4 models on Cline and OpenRouter providers
--   Increase max tokens for Claude Opus 4 from 4096 to 8192
-
-## [3.17.0]
-
--   Add support for Anthropic Claude Sonnet 4 and Claude Opus 4 in both Anthropic and Vertex providers
--   Add integration with Nebius AI Studio as a new provider (Thanks @Aktsvigun!)
--   Add custom highlight and hotkey suggestion when the assistant prompts to switch to Act mode
--   Update settings page design, now split into tabs for easier navigation (Thanks Yellow Bat @dlab-anton, and Roo Team!)
--   Fix MCP Server configuration bug
--   Fix model listing for Requesty provider
--   Move all advanced settings to settings page
-
-## [3.16.3]
-
--   Add devstral-small-2505 to the Mistral model list, a new specialized coding model from Mistral AI (Thanks @BarreiroT!)
--   Add documentation links to rules & workflows UI
--   Add support for Streameable HTTP Transport for MCPs (Thanks @alejandropta!)
--   Improve error handling for Mistral SDK API
-
-## [3.16.2]
-
--   Add support for Gemini 2.5 Flash Preview 05-20 model to Vertex AI provider with massive 1M token context window (Thanks @omercelik!)
--   Add keyboard shortcut (Cmd+') to quickly focus Cline from anywhere in VS Code
--   Add lightbulb actions for selected text with options to "Add to Cline", "Explain with Cline", and "Improve with Cline"
--   Automatically focus Cline window after extension updates
-
 ## [3.16.1]
 
 -   Add Enable auto approve toggle switch, allowing users to easily turn auto-approve functionality on or off without losing their action settings

CONTRIBUTING.md

@@ -10,77 +10,16 @@ Bug reports help make Cline better for everyone! Before creating a new issue, pl
      🔐 <b>Important:</b> If you discover a security vulnerability, please use the <a href="https://github.com/cline/cline/security/advisories/new">Github security tool to report it privately</a>.
 </blockquote>
 
-
-## Before Contributing
-
-All contributions must begin with a GitHub Issue, unless the change is for small bug fixes, typo corrections, minor wording improvements, or simple type fixes that don't change functionality.
-
-- **Check existing issues**: Search [GitHub Issues](https://github.com/cline/cline/issues).
-- **Create an issue**: Use appropriate templates:
-    - **Contributions:** Use the "Contribution Request" template to propose what you'd like to work on.
-    - **Bugs:** "Bug Report" template for reporting issues.
-    - **Features:** "Detailed Feature Proposal" template for suggesting new features.
-- **Wait for approval**: A core Cline contributor must approve your contribution request before you start implementation.
-- **Claim issues**: Once approved, the issue will be assigned to you.
-
-**PRs without approved issues may be closed.**
-
-
 ## Deciding What to Work On
 
 Looking for a good first contribution? Check out issues labeled ["good first issue"](https://github.com/cline/cline/labels/good%20first%20issue) or ["help wanted"](https://github.com/cline/cline/labels/help%20wanted). These are specifically curated for new contributors and areas where we'd love some help!
 
 We also welcome contributions to our [documentation](https://github.com/cline/cline/tree/main/docs)! Whether it's fixing typos, improving existing guides, or creating new educational content - we'd love to build a community-driven repository of resources that helps everyone get the most out of Cline. You can start by diving into `/docs` and looking for areas that need improvement.
 
+If you're planning to work on a bigger feature, please create a [feature request](https://github.com/cline/cline/discussions/categories/feature-requests?discussions_q=is%3Aopen+category%3A%22Feature+Requests%22+sort%3Atop) first so we can discuss whether it aligns with Cline's vision.
+
 ## Development Setup
 
-
-### Local Development Instructions
-
-1. Clone the repository _(Requires [git-lfs](https://git-lfs.com/))_:
-    ```bash
-    git clone https://github.com/cline/cline.git
-    ```
-2. Open the project in VSCode:
-    ```bash
-    code cline
-    ```
-3. Install the necessary dependencies for the extension and webview-gui:
-    ```bash
-    npm run install:all
-    ```
-4. Launch by pressing `F5` (or `Run`->`Start Debugging`) to open a new VSCode window with the extension loaded. (You may need to install the [esbuild problem matchers extension](https://marketplace.visualstudio.com/items?itemName=connor4312.esbuild-problem-matchers) if you run into issues building the project.)
-
-
-
-
-### Creating a Pull Request
-
-1. Before creating a PR, generate a changeset entry:
-    ```bash
-    npm run changeset
-    ```
-   This will prompt you for:
-   - Type of change (major, minor, patch)
-     - `major` → breaking changes (1.0.0 → 2.0.0)
-     - `minor` → new features (1.0.0 → 1.1.0)
-     - `patch` → bug fixes (1.0.0 → 1.0.1)
-   - Description of your changes
-
-2. Commit your changes and the generated `.changeset` file
-
-3. Push your branch and create a PR on GitHub. Our CI will:
-   - Run tests and checks
-   - Changesetbot will create a comment showing the version impact
-   - When merged to main, changesetbot will create a Version Packages PR
-   - When the Version Packages PR is merged, a new release will be published
-4. Testing
-    - Run `npm run test` to run tests locally. 
-    - Before submitting PR, run `npm run format:fix` to format your code
-    - Run `npm run test:ci` to run tests locally
-
-### Extension
-
 1. **VS Code Extensions**
 
     - When opening the project, VS Code will prompt you to install recommended extensions
@@ -90,26 +29,23 @@ We also welcome contributions to our [documentation](https://github.com/cline/cl
 2. **Local Development**
     - Run `npm run install:all` to install dependencies
     - Run `npm run test` to run tests locally
-    - Run → Start Debugging or `>Debug: Select and Start Debugging` and wait for a new VS Code instance to open
     - Before submitting PR, run `npm run format:fix` to format your code
 
 3. **Linux-specific Setup**
     VS Code extension tests on Linux require the following system libraries:
 
-    - `dbus`
-    - `libasound2`
-    - `libatk-bridge2.0-0`
     - `libatk1.0-0`
-    - `libdrm2`
-    - `libgbm1`
-    - `libgtk-3-0`
-    - `libnss3`
+    - `libatk-bridge2.0-0`
+    - `libxkbfile1`
     - `libx11-xcb1`
     - `libxcomposite1`
     - `libxdamage1`
     - `libxfixes3`
-    - `libxkbfile1`
     - `libxrandr2`
+    - `libgbm1`
+    - `libdrm2`
+    - `libgtk-3-0`
+    - `dbus`
     - `xvfb`
 
     These libraries provide necessary GUI components and system services for the test environment.
@@ -118,23 +54,13 @@ We also welcome contributions to our [documentation](https://github.com/cline/cl
     ```bash
     sudo apt update
     sudo apt install -y \
-      dbus \
-      libasound2 \
-      libatk-bridge2.0-0 \
-      libatk1.0-0 \
-      libdrm2 \
-      libgbm1 \
-      libgtk-3-0 \
-      libnss3 \
-      libx11-xcb1 \
-      libxcomposite1 \
-      libxdamage1 \
-      libxfixes3 \
-      libxkbfile1 \
-      libxrandr2 \
-      xvfb
+      libatk1.0-0 libatk-bridge2.0-0 libxkbfile1 libx11-xcb1 \
+      libxcomposite1 libxdamage1 libxfixes3 libxrandr2 libgbm1 \
+      libdrm2 libgtk-3-0 dbus xvfb
     ```
 
+    - Run `npm run test:ci` to run tests locally
+
 ## Writing and Submitting Code
 
 Anyone can contribute code to Cline, but we ask that you follow these guidelines to ensure your contributions can be smoothly integrated:

README.md

@@ -51,7 +51,7 @@ Thanks to [Claude 3.7 Sonnet's agentic coding capabilities](https://www.anthrop
 
 ### Use any API and Model
 
-Cline supports API providers like OpenRouter, Anthropic, OpenAI, Google Gemini, AWS Bedrock, Azure, GCP Vertex, and Cerebras. You can also configure any OpenAI compatible API, or use a local model through LM Studio/Ollama. If you're using OpenRouter, the extension fetches their latest model list, allowing you to use the newest models as soon as they're available.
+Cline supports API providers like OpenRouter, Anthropic, OpenAI, Google Gemini, AWS Bedrock, Azure, and GCP Vertex. You can also configure any OpenAI compatible API, or use a local model through LM Studio/Ollama. If you're using OpenRouter, the extension fetches their latest model list, allowing you to use the newest models as soon as they're available.
 
 The extension also keeps track of total tokens and API usage cost for the entire task loop and individual requests, keeping you informed of spend every step of the way.
 
@@ -141,6 +141,50 @@ For example, when working with a local web server, you can use 'Restore Workspac
 
 To contribute to the project, start with our [Contributing Guide](CONTRIBUTING.md) to learn the basics. You can also join our [Discord](https://discord.gg/cline) to chat with other contributors in the `#contributors` channel. If you're looking for full-time work, check out our open positions on our [careers page](https://cline.bot/join-us)!
 
+<details>
+<summary>Local Development Instructions</summary>
+
+1. Clone the repository _(Requires [git-lfs](https://git-lfs.com/))_:
+    ```bash
+    git clone https://github.com/cline/cline.git
+    ```
+2. Open the project in VSCode:
+    ```bash
+    code cline
+    ```
+3. Install the necessary dependencies for the extension and webview-gui:
+    ```bash
+    npm run install:all
+    ```
+4. Launch by pressing `F5` (or `Run`->`Start Debugging`) to open a new VSCode window with the extension loaded. (You may need to install the [esbuild problem matchers extension](https://marketplace.visualstudio.com/items?itemName=connor4312.esbuild-problem-matchers) if you run into issues building the project.)
+
+</details>
+
+<details>
+<summary>Creating a Pull Request</summary>
+
+1. Before creating a PR, generate a changeset entry:
+    ```bash
+    npm run changeset
+    ```
+   This will prompt you for:
+   - Type of change (major, minor, patch)
+     - `major` → breaking changes (1.0.0 → 2.0.0)
+     - `minor` → new features (1.0.0 → 1.1.0)
+     - `patch` → bug fixes (1.0.0 → 1.0.1)
+   - Description of your changes
+
+2. Commit your changes and the generated `.changeset` file
+
+3. Push your branch and create a PR on GitHub. Our CI will:
+   - Run tests and checks
+   - Changesetbot will create a comment showing the version impact
+   - When merged to main, changesetbot will create a Version Packages PR
+   - When the Version Packages PR is merged, a new release will be published
+
+</details>
+
+
 ## License
 
 [Apache 2.0 © 2025 Cline Bot Inc.](./LICENSE)

buf.yaml

@@ -1,22 +0,0 @@
-version: v2
-modules:
-    - path: proto
-      name: cline/cline/lint
-
-lint:
-    use:
-        - STANDARD
-
-    except: # Add exceptions for current patterns that contradict STANDARD settings
-        - RPC_PASCAL_CASE # rpcs are camel case (start with lowercase)
-        - PACKAGE_DIRECTORY_MATCH # the protos in the cline package are not in a dir named cline.
-        - RPC_REQUEST_RESPONSE_UNIQUE # request messages are not unique.
-        - RPC_REQUEST_STANDARD_NAME # request messages dont all end with Request
-        - RPC_RESPONSE_STANDARD_NAME # response messages dont all end with Response
-        - PACKAGE_VERSION_SUFFIX # package name does not contain version.
-        - ENUM_VALUE_PREFIX # enum values dont start with the enum name.
-        - ENUM_ZERO_VALUE_SUFFIX # first value does not have to be UNSPECIFIED.
-
-# breaking:
-#   use:
-#     - WIRE_JSON # Detect changes that break the json wire format (this is the minimum recommended level.)

docs/custom-model-configs/aws-bedrock-with-credentials-authentication.mdx

@@ -25,41 +25,12 @@ description: "Learn how to set up AWS Bedrock with Cline using credentials authe
 
 #### 1.2 Attach the Required Policies
 
-To ensure Cline can interact with AWS Bedrock, your IAM user or role needs specific permissions. While the `AmazonBedrockFullAccess` managed policy provides comprehensive access, for a more restricted and secure setup adhering to the principle of least privilege, the following minimal permissions are sufficient for Cline's core model invocation functionality:
-
--   `bedrock:InvokeModel`
--   `bedrock:InvokeModelWithResponseStream`
-
-You can create a custom IAM policy with these permissions and attach it to your IAM user or role.
-
-**Option 1: Minimal Permissions (Recommended for Production & Least Privilege)**
-
-1.  In the AWS IAM console, create a new policy.
-2.  Use the JSON editor to add the following policy document:
-    ```json
-    {
-    	"Version": "2012-10-17",
-    	"Statement": [
-    		{
-    			"Effect": "Allow",
-    			"Action": ["bedrock:InvokeModel", "bedrock:InvokeModelWithResponseStream"],
-    			"Resource": "*" // For enhanced security, scope this to specific model ARNs if possible.
-    		}
-    	]
-    }
-    ```
-3.  Name the policy (e.g., `ClineBedrockInvokeAccess`) and attach it to your IAM user or role.
-
-**Option 2: Using a Managed Policy (Simpler Initial Setup)**
-
--   Alternatively, you can attach the AWS managed policy **`AmazonBedrockFullAccess`**. This grants broader permissions, including the ability to list models, manage provisioning, and other Bedrock features. This might be simpler for initial setup or if you require these wider capabilities.
-    [View AmazonBedrockFullAccess Policy Details](https://docs.aws.amazon.com/bedrock/latest/userguide/security-iam.html)
-
-**Important Considerations:**
-
--   **Model Listing in Cline:** The minimal permissions (`bedrock:InvokeModel`, `bedrock:InvokeModelWithResponseStream`) are sufficient for Cline to _use_ a model if you specify the model ID directly in Cline's settings. If you rely on Cline to dynamically list available Bedrock models, you might need additional permissions like `bedrock:ListFoundationModels`.
--   **AWS Marketplace Subscriptions:** For third-party models (e.g., Anthropic Claude), ensure you have active AWS Marketplace subscriptions. This is typically managed in the AWS Bedrock console under "Model access" and might require `aws-marketplace:Subscribe` permissions if not already handled.
--   _Enterprise Tip:_ Always apply least-privilege practices. Where possible, scope resource ARNs in your IAM policies to specific models or regions. Utilize [Service Control Policies (SCPs)](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scps.html) for overarching governance in AWS Organizations.
+1. **Attach the Managed Policy:**
+    - Attach the **`AmazonBedrockFullAccess`** managed policy to your user/role.\
+      [View AmazonBedrockFullAccess Policy Details](https://docs.aws.amazon.com/bedrock/latest/userguide/security-iam.html)
+2. **Confirm Additional Permissions:**
+    - Ensure your policy includes permissions for model invocation (e.g., `bedrock:InvokeModel` and `bedrock:InvokeModelWithResponseStream`), model listing, and AWS Marketplace actions (like `aws-marketplace:Subscribe`).
+    - _Enterprise Tip:_ Apply least-privilege practices by scoping resource ARNs and using [Service Control Policies (SCPs)](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scps.html) to restrict access where necessary.
 
 ---
 

docs/docs.json

@@ -143,23 +143,12 @@
 				]
 			},
 			{
-				"group": "Provider Configuration",
+				"group": "Custom Model Configurations",
 				"pages": [
-					"provider-config/anthropic",
-					"provider-config/claude-code",
-					"provider-config/aws-bedrock-with-credentials-authentication",
-					"provider-config/aws-bedrock-with-profile-authentication",
-					"provider-config/gcp-vertex-ai",
-					"provider-config/litellm-and-cline-using-codestral",
-					"provider-config/vscode-language-model-api",
-					"provider-config/xai-grok",
-					"provider-config/mistral-ai",
-					"provider-config/deepseek",
-					"provider-config/ollama",
-					"provider-config/openai",
-					"provider-config/openai-compatible",
-					"provider-config/openrouter",
-					"provider-config/requesty"
+					"custom-model-configs/aws-bedrock-with-credentials-authentication",
+					"custom-model-configs/aws-bedrock-with-profile-authentication",
+					"custom-model-configs/gcp-vertex-ai",
+					"custom-model-configs/litellm-and-cline-using-codestral"
 				]
 			},
 			{

docs/enterprise-solutions/cloud-provider-integration.mdx

@@ -14,9 +14,9 @@ Certain scenarios may warrant using local models, including handling highly sens
 
 #### [IAM Security Best Practices](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html) (For administrators)
 
-#### [AWS Bedrock setup for Legacy IAM (AWS Credentials)](/provider-config/aws-bedrock-with-credentials-authentication)
+#### [AWS Bedrock setup for Legacy IAM (AWS Credentials)](/custom-model-configs/aws-bedrock-with-credentials-authentication.mdx)
 
-#### [AWS Bedrock setup for SSO token (AWS Profile)](/provider-config/aws-bedrock-with-profile-authentication)
+#### [AWS Bedrock setup for SSO token (AWS Profile)](/custom-model-configs/aws-bedrock-with-profile-authentication.mdx)
 
 #### VPC Endpoint Setup
 

docs/features/plan-and-act.mdx

@@ -6,13 +6,10 @@ sidebarTitle: "Plan & Act"
 Plan & Act modes represent Cline's approach to structured AI development, emphasizing thoughtful planning before implementation. This dual-mode system helps developers create more maintainable, accurate code while reducing iteration time.
 
 <Frame>
-	<iframe
-		style={{ width: "100%", aspectRatio: "16/9" }}
-		src="https://www.youtube.com/embed/b7o6URFPp64"
-		title="YouTube video player"
-		frameBorder="0"
-		allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
-		allowFullScreen></iframe>
+	<img
+		src="https://storage.googleapis.com/cline_public_images/docs/assets/planningThenActing%20(1).gif"
+		alt="Use Plan to gather context before using Act to implement the plan"
+	/>
 </Frame>
 
 #### Plan Mode: Think First

docs/gemini-cli-provider.md

@@ -1,93 +0,0 @@
-# Gemini CLI Provider for Cline
-
-This document describes how to use the Gemini CLI as a provider in Cline.
-
-## Overview
-
-The Gemini CLI provider allows you to use Google's Gemini models through the Gemini CLI tool instead of using API keys directly. This can be useful if you:
-
-- Already have the Gemini CLI installed and configured
-- Want to use the CLI's authentication methods (OAuth, etc.)
-- Prefer to manage your Gemini access through the CLI
-
-## Prerequisites
-
-1. **Install the Gemini CLI**: The Gemini CLI must be installed and accessible from your system PATH.
-   
-2. **Set up authentication**: The CLI needs to be authenticated. You can either:
-   - Set the `GEMINI_API_KEY` environment variable
-   - Use the CLI's built-in authentication methods
-
-3. **Build the CLI** (if using from source):
-   ```bash
-   cd gemini-cli/packages/cli
-   npm install
-   npm run build
-   ```
-
-## Configuration
-
-To use the Gemini CLI provider in Cline:
-
-1. Open Cline settings
-2. Select "Gemini CLI" as your API provider
-3. (Optional) Specify the path to the Gemini CLI executable if it's not in your PATH
-4. Select your desired Gemini model
-
-## Supported Models
-
-The Gemini CLI provider supports the same models as the regular Gemini provider:
-
-- gemini-2.5-pro
-- gemini-2.5-flash
-- gemini-2.0-flash-001
-- gemini-1.5-flash-002
-- gemini-1.5-pro-002
-
-## How It Works
-
-1. When you send a message in Cline, it converts your conversation to a format the Gemini CLI understands
-2. Cline spawns the Gemini CLI as a subprocess with your prompt
-3. The CLI's response is streamed back to Cline
-4. The response is displayed in the Cline interface
-
-## Limitations
-
-- **No image support**: The Gemini CLI in non-interactive mode doesn't support images
-- **No tool execution**: The CLI handles its own tool execution, which doesn't integrate with Cline's tools
-- **Token counting**: Token usage is estimated based on text length rather than actual token counts
-- **No caching**: The CLI doesn't support Anthropic-style prompt caching
-
-## Troubleshooting
-
-### CLI not found
-- Ensure the Gemini CLI is installed and in your PATH
-- Or specify the full path to the CLI in Cline settings
-
-### Authentication errors
-- Check that `GEMINI_API_KEY` is set in your environment
-- Or ensure the CLI is authenticated using its built-in methods
-
-### No response
-- Try running the CLI manually to ensure it works:
-  ```bash
-  gemini --prompt "Hello, world!"
-  ```
-
-### Build errors
-If using from source, ensure the CLI is built:
-```bash
-cd gemini-cli/packages/cli
-npm install
-npm run build
-```
-
-## Testing
-
-You can test the integration using the provided test script:
-
-```bash
-node test-gemini-cli.js
-```
-
-This will attempt to run a simple prompt through the CLI and report whether it succeeded.

docs/getting-started/for-new-coders.mdx

@@ -13,13 +13,35 @@ Before you jump into coding, make sure you have these essentials ready:
 
 A popular, free, and powerful code editor.
 
--   [<u>Download VS Code</u>](https://code.visualstudio.com/)
+-   [Download VS Code](https://code.visualstudio.com/)
 
-📺 **Recommended YouTube Tutorial:** [<u>How to Install VS Code</u>](https://www.youtube.com/watch?v=MlIzFUI1QGA)
+📺 **Recommended YouTube Tutorial:** [How to Install VS Code](https://www.youtube.com/watch?v=MlIzFUI1QGA)
 
 > ✅ **Pro Tip:** Install VS Code in your Applications folder (macOS) or Program Files (Windows) for easy access from your dock or start menu.
 
-#### 2. **Organize Your Projects**
+#### 2. **Essential Development Tools**
+
+Basic software required for coding efficiently:
+
+-   Homebrew (macOS)
+-   Node.js
+-   Git
+
+👉 Follow our detailed guide on Installing Essential Development Tools with step-by-step help from Cline.
+
+📺 **Recommended YouTube Tutorials:**
+
+-   **For macOS:**
+    -   [Install Homebrew on Mac](https://www.youtube.com/watch?v=hwGNgVbqasc)
+    -   [Install Git on MacOS 2024](https://www.youtube.com/watch?v=B4qsvQ5IqWk)
+    -   [Install Node.js on Mac (M1 | M2 | M3)](https://www.youtube.com/watch?v=I8H4wolRFBk)
+-   **For Windows:**
+    -   [Install Git on Windows 10/11 (2024)](https://www.youtube.com/watch?v=yjxv1HuRQy0)
+    -   [Install Node.js in Windows 10/11](https://www.youtube.com/watch?v=uCgAuOYpJd0)
+
+> ⚠️ **Note:** If you run into permission issues during installation, try running your terminal or command prompt as an administrator.
+
+#### 3. **Organize Your Projects**
 
 Create a dedicated folder named `Cline` in your Documents folder for all your coding projects:
 
@@ -33,36 +55,14 @@ Inside your `Cline` folder, structure projects clearly:
 
 > 💡 **Tip:** Keeping your projects organized from the start will save you time and confusion later!
 
-#### 3. **Install the Cline VS Code Extension**
+#### 4. **Install the Cline VS Code Extension**
 
 Enhance your coding workflow by installing the Cline extension directly within VS Code:
 
 -   Get Started with Cline Extension Tutorial
 
-📺 **Recommended YouTube Tutorial:** [<u>How To Install Extensions in VS Code</u>](https://www.youtube.com/watch?v=E7trgwZa-mk)
+📺 **Recommended YouTube Tutorial:** [How To Install Extensions in VS Code](https://www.youtube.com/watch?v=E7trgwZa-mk)
 
 > ✅ **Pro Tip:** After installing, reload VS Code to ensure the extension is activated properly.
 
-#### 4. **Essential Development Tools**
-
-Basic software required for coding efficiently:
-
--   Homebrew (macOS)
--   Node.js
--   Git
-
-👉 [<u>Follow our detailed guide on Installing Essential Development Tools with step-by-step help from Cline.</u>](https://docs.cline.bot/getting-started/installing-dev-essentials#installing-dev-essentials)
-
-📺 **Recommended YouTube Tutorials for Manual Installation:**
-
--   **For macOS:**
-    -   [<u>Install Homebrew on Mac</u>](https://www.youtube.com/watch?v=hwGNgVbqasc)
-    -   [<u>Install Git on macOS 2024</u>](https://www.youtube.com/watch?v=B4qsvQ5IqWk)
-    -   [<u>Install Node.js on Mac (M1 | M2 | M3)</u>](https://www.youtube.com/watch?v=I8H4wolRFBk)
--   **For Windows:**
-    -   [<u>Install Git on Windows 10/11 (2024)</u>](https://www.youtube.com/watch?v=yjxv1HuRQy0)
-    -   [<u>Install Node.js in Windows 10/11</u>](https://www.youtube.com/watch?v=uCgAuOYpJd0)
-
-> ⚠️ **Note:** If you run into permission issues during installation, try running your terminal or command prompt as an administrator.
-
 🎉 You're all set! Dive in and start coding smarter and faster with **Cline**.

docs/getting-started/installing-cline.mdx

@@ -60,6 +60,7 @@ Now that you have Cline installed, let's get you set up with your account:
     - DeepSeek Chat (cost-effective alternative)
     - Google Gemini 2.0 Flash
     - And more — all through your Cline account.
+4.  -
 
 ### 💻 Your First Interaction with Cline
 

docs/getting-started/our-favorite-tech-stack.mdx

@@ -69,9 +69,9 @@ Choose your AI assistant based on your needs:
 ### Getting Started
 
 1. Install the development essentials:
-    - Follow our [Development Essentials Installation Guide](https://docs.cline.bot/getting-started/installing-dev-essentials)
+    - Follow our [Development Essentials Installation Guide](https://docs.cline.bot/getting-started/getting-started-new-coders/installing-dev-essentials)
 2. Set up Cline's Memory Bank:
-    - Follow the [Memory Bank setup instructions](https://docs.cline.bot/prompting/cline-memory-bank)
+    - Follow the [Memory Bank setup instructions](https://docs.cline.bot/improving-your-prompting-skills/custom-instructions-library/cline-memory-bank)
     - Create an empty `cline_docs` folder in your project root
     - Create `projectBrief.md` in the `cline_docs` folder (see example below)
     - Tell Cline to "initialize memory bank"
@@ -197,20 +197,20 @@ git push origin main                        # Upload to GitHub
 1.  **Start of day**: Get latest changes
 
     ```bash
-    git pull origin main                     # Download latest code
+    bashCopygit pull origin main                     # Download latest code
     ```
 
 2.  **During development**: Save work regularly
 
     ```bash
-    git add .
+    bashCopygit add .
     git commit -m "Clear message about changes"
     ```
 
 3.  **End of day**: Share your progress
 
     ```bash
-    git push origin main                     # Upload to GitHub
+    bashCopygit push origin main                     # Upload to GitHub
     ```
 
 **Best Practices**

docs/getting-started/understanding-context-management.mdx

@@ -38,7 +38,7 @@ Cline actively builds context in two ways:
     - Guide focus areas
     - Share design thoughts and requirements
 
-💡 **Key Point**: Cline isn't passive - it actively seeks to understand your project. You can either let it explore or guide its focus, especially in [Plan](https://docs.cline.bot/features/plan-and-act) mode.
+💡 **Key Point**: Cline isn't passive - it actively seeks to understand your project. You can either let it explore or guide its focus, especially in [Plan](https://docs.cline.bot/exploring-clines-tools/plan-and-act-modes-a-guide-to-effective-ai-development) mode.
 
 ### Context & Context Windows
 
@@ -93,7 +93,7 @@ Context files help maintain understanding across sessions. They serve as documen
 
 #### Approaches to Context Files
 
-1. **Evergreen Project Context (i.e.** [**Memory Bank**](https://docs.cline.bot/prompting/cline-memory-bank)**)**
+1. **Evergreen Project Context (i.e.** [**Memory Bank**](https://docs.cline.bot/improving-your-prompting-skills/custom-instructions-library/cline-memory-bank)**)**
     - Living documentation that evolves with your project
     - Updated as architecture and patterns emerge
     - Example: The Memory Bank pattern maintains files like `techContext.md` and `systemPatterns.md`
@@ -151,7 +151,7 @@ Context files help maintain understanding across sessions. They serve as documen
     - Use Plan mode for complex discussions
     - Start fresh sessions when needed
 3. **Team Projects**
-    - Share common context files (consider using [.clinerules](https://docs.cline.bot/features/cline-rules) files in project roots)
+    - Share common context files (consider using [.clinerules](https://docs.cline.bot/improving-your-prompting-skills/prompting) files in project roots)
     - Document architectural decisions
     - Maintain consistent patterns
     - Keep documentation current

docs/provider-config/anthropic.mdx

@@ -1,61 +0,0 @@
----
-title: "Anthropic"
-description: "Learn how to configure and use Anthropic Claude models with Cline. Covers API key setup, model selection, and advanced features like prompt caching."
----
-
-**Website:** [https://www.anthropic.com/](https://www.anthropic.com/)
-
-### Getting an API Key
-
-1.  **Sign Up/Sign In:** Go to the [Anthropic Console](https://console.anthropic.com/). Create an account or sign in.
-2.  **Navigate to API Keys:** Go to the [API keys](https://console.anthropic.com/settings/keys) section.
-3.  **Create a Key:** Click "Create Key". Give your key a descriptive name (e.g., "Cline").
-4.  **Copy the Key:** **Important:** Copy the API key _immediately_. You will not be able to see it again. Store it securely.
-
-### Supported Models
-
-Cline supports the following Anthropic Claude models:
-
--   `claude-opus-4-20250514`
--   `claude-opus-4-20250514:thinking` (Extended Thinking variant)
--   `claude-sonnet-4-20250514` (Recommended)
--   `claude-sonnet-4-20250514:thinking` (Extended Thinking variant)
--   `claude-3-7-sonnet-20250219`
--   `claude-3-7-sonnet-20250219:thinking` (Extended Thinking variant)
--   `claude-3-5-sonnet-20241022`
--   `claude-3-5-haiku-20241022`
--   `claude-3-opus-20240229`
--   `claude-3-haiku-20240307`
-
-See [Anthropic's Model Documentation](https://docs.anthropic.com/en/docs/about-claude/models) for more details on each model's capabilities.
-
-### Configuration in Cline
-
-1.  **Open Cline Settings:** Click the settings icon (⚙️) in the Cline panel.
-2.  **Select Provider:** Choose "Anthropic" from the "API Provider" dropdown.
-3.  **Enter API Key:** Paste your Anthropic API key into the "Anthropic API Key" field.
-4.  **Select Model:** Choose your desired Claude model from the "Model" dropdown.
-5.  **(Optional) Custom Base URL:** If you need to use a custom base URL for the Anthropic API, check "Use custom base URL" and enter the URL. Most users won't need to adjust this setting.
-
-### Extended Thinking
-
-Anthropic models offer an "Extended Thinking" feature, designed to give them enhanced reasoning capabilities for complex tasks. This feature allows the model to output its step-by-step thought process before delivering a final answer, providing transparency and enabling more thorough analysis for challenging prompts.
-
-When extended thinking is in Cline, the model generates `thinking` content blocks that detail its internal reasoning. These insights are then incorporated into its final response.
-Cline users can leverage this by checking the `Enable Extended Thinking` box below the model selection menu after selecting a Claude Model from any provider.
-
-**Key Aspects of Extended Thinking:**
-
--   **Supported Models:** This feature is available for select models, including variants of Claude Opus 4, Claude Sonnet 4, and Claude Sonnet 3.7. The specific models listed in the "Supported Models" section above with the `:thinking` suffix are pre-configured in Cline to utilize this.
--   **Summarized Thinking (Claude 4):** For Claude 4 models, the API returns a summary of the full thinking process to balance insight with efficiency and prevent misuse. You are billed for the full thinking tokens, not just the summary.
--   **Streaming:** Extended thinking responses, including the `thinking` blocks, can be streamed.
--   **Tool Use & Prompt Caching:** Extended thinking interacts with tool use (requiring thinking blocks to be passed back) and prompt caching (with specific behaviors around cache invalidation and context).
-
-For comprehensive details on how extended thinking works, including API examples, interaction with tool use, prompt caching, and pricing, please refer to the [official Anthropic documentation on Extended Thinking](https://docs.anthropic.com/en/docs/build-with-claude/extended-thinking).
-
-### Tips and Notes
-
--   **Prompt Caching:** Claude 3 models support [prompt caching](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching), which can significantly reduce costs and latency for repeated prompts.
--   **Context Window:** Claude models have large context windows (200,000 tokens), allowing you to include a significant amount of code and context in your prompts.
--   **Pricing:** Refer to the [Anthropic Pricing](https://www.anthropic.com/pricing) page for the latest pricing information.
--   **Rate Limits:** Anthropic has strict rate limits based on [usage tiers](https://docs.anthropic.com/en/api/rate-limits#requirements-to-advance-tier). If you're repeatedly hitting rate limits, consider contacting Anthropic sales or accessing Claude through a different provider like [OpenRouter](/provider-config/openrouter) or [Requesty](/provider-config/requesty).

docs/provider-config/claude-code.mdx

@@ -1,86 +0,0 @@
----
-title: "Claude Code"
-description: "Use your Claude Max subscription with Cline instead of paying per token. Learn how to set up and configure the Claude Code provider."
----
-
-**Website:** [https://docs.anthropic.com/en/docs/claude-code/setup](https://docs.anthropic.com/en/docs/claude-code/setup)
-
-The Claude Code provider lets you use your existing Claude subscription with Cline. If you have Claude Max, this means you can use Claude in Cline without paying extra API costs.
-
-<Frame>
-	<img
-		src="https://storage.googleapis.com/cline_public_images/docs/assets/claude-code-use-opus.gif"
-		alt="Using the Claude Code provider in Cline with Opus model"
-	/>
-</Frame>
-
-## Setup
-
-First, you'll need to install and authenticate Claude Code on your system:
-
-1. **Install Claude Code**: Follow Anthropic's [official setup guide](https://docs.anthropic.com/en/docs/claude-code/setup) to install and authenticate the Claude CLI.
-
-2. **Configure in Cline**:
-    - Open Cline settings (⚙️ icon)
-    - Select **Claude Code** from the **API Provider** dropdown
-    - Set the path to your Claude CLI executable (usually just `claude` if it's in your PATH)
-
-<Frame>
-	<img
-		src="https://storage.googleapis.com/cline_public_images/docs/assets/claude-code-setup.gif"
-		alt="Setting up the Claude Code provider in Cline"
-	/>
-</Frame>
-
-### Finding your Claude Code path
-
-If you're not sure where Claude Code is installed:
-
--   **macOS / Linux**: Run `which claude` in your terminal
--   **Windows (Command Prompt)**: Run `where claude`
--   **Windows (PowerShell)**: Run `Get-Command claude`
-
-## Supported Models
-
-The Claude Code provider supports these models:
-
--   `claude-sonnet-4-20250514` (Recommended)
--   `claude-opus-4-20250514`
--   `claude-3-7-sonnet-20250219`
--   `claude-3-5-sonnet-20241022`
--   `claude-3-5-haiku-20241022`
-
-## How it works
-
-When you use Claude Code with Cline, here's what happens behind the scenes:
-
-Cline wraps the Claude Code CLI to handle your requests. Each time you send a message, Cline starts a new `claude` process, sends your conversation, and streams the response back. The AI reasoning comes from Claude Code, but all the actual file editing, terminal commands, and other tools are handled by Cline.
-
-The main difference you'll notice is that responses don't stream character-by-character like other providers. Instead, Claude Code processes your full request before sending back the complete response.
-
-## Limitations
-
-There are a few things to keep in mind with Claude Code:
-
--   Images in your messages get converted to text placeholders since Claude Code doesn't support image uploads through the CLI
--   Prompt caching isn't available with this provider
--   Responses don't stream in real-time like other providers
-
-## Troubleshooting
-
-If you run into issues:
-
-**Authentication problems**: Make sure you're logged into Claude Code with your subscription account. Run `claude auth status` to check.
-
-**Path issues**: Double-check that the Claude CLI path in Cline's settings is correct. Try running `claude --version` in your terminal to verify it's working.
-
-**Still having trouble?** We're actively improving this integration. Report issues on our [GitHub](https://github.com/cline/cline/issues) or ask for help in our [Discord](https://discord.gg/cline).
-
-## Usage with subscriptions
-
-If you have a Claude Max subscription, your usage in Cline shows up as $0.00 in the billing interface since you're not paying additional API costs. Your usage still counts against your subscription limits, but you won't see per-token charges.
-
-For more details about using Claude Code with your subscription, check out Anthropic's documentation:
-
--   [Claude Code Setup Guide](https://docs.anthropic.com/en/docs/claude-code/setup)
--   [Using Claude Code with Pro/Max Plans](https://support.anthropic.com/en/articles/11145838-using-claude-code-with-your-pro-or-max-plan)

docs/provider-config/deepseek.mdx

@@ -1,33 +0,0 @@
----
-title: "DeepSeek"
-description: "Learn how to configure and use DeepSeek models like deepseek-chat and deepseek-reasoner with Cline."
----
-
-Cline supports accessing models through the DeepSeek API, including `deepseek-chat` and `deepseek-reasoner`.
-
-**Website:** [https://platform.deepseek.com/](https://platform.deepseek.com/)
-
-### Getting an API Key
-
-1.  **Sign Up/Sign In:** Go to the [DeepSeek Platform](https://platform.deepseek.com/). Create an account or sign in.
-2.  **Navigate to API Keys:** Find your API keys in the [API keys](https://platform.deepseek.com/api_keys) section of the platform.
-3.  **Create a Key:** Click "Create new API key". Give your key a descriptive name (e.g., "Cline").
-4.  **Copy the Key:** **Important:** Copy the API key _immediately_. You will not be able to see it again. Store it securely.
-
-### Supported Models
-
-Cline supports the following DeepSeek models:
-
--   `deepseek-v3-0324` (Recommended for coding tasks)
--   `deepseek-r1` (Recommended for reasoning tasks)
-
-### Configuration in Cline
-
-1.  **Open Cline Settings:** Click the ⚙️ icon in the Cline panel.
-2.  **Select Provider:** Choose "DeepSeek" from the "API Provider" dropdown.
-3.  **Enter API Key:** Paste your DeepSeek API key into the "DeepSeek API Key" field.
-4.  **Select Model:** Choose your desired model from the "Model" dropdown.
-
-### Tips and Notes
-
--   **Pricing:** Refer to the [DeepSeek Pricing](https://api-docs.deepseek.com/quick_start/pricing/) page for details on model costs.

docs/provider-config/mistral-ai.mdx

@@ -1,53 +0,0 @@
----
-title: "Mistral"
-description: "Learn how to configure and use Mistral AI models, including Codestral, with Cline. Covers API key setup and model selection."
----
-
-Cline supports accessing models through the Mistral AI API, including both standard Mistral models and the code-specialized Codestral model.
-
-**Website:** [https://mistral.ai/](https://mistral.ai/)
-
-### Getting an API Key
-
-1.  **Sign Up/Sign In:** Go to the [Mistral Platform](https://console.mistral.ai/). Create an account or sign in. You may need to go through a verification process.
-2.  **Create an API Key:**
-    -   [La Plateforme API Key](https://console.mistral.ai/api-keys/) and/or
-    -   [Codestral API Key](https://console.mistral.ai/codestral)
-
-### Supported Models
-
-Cline supports the following Mistral models:
-
--   pixtral-large-2411
--   ministral-3b-2410
--   ministral-8b-2410
--   mistral-small-latest
--   mistral-medium-latest
--   mistral-small-2501
--   pixtral-12b-2409
--   open-mistral-nemo-2407
--   open-codestral-mamba
--   codestral-2501
--   devstral-small-2505
-
-**Note:** Model availability and specifications may change.
-Refer to the [Mistral AI documentation](https://docs.mistral.ai/api/) and [Mistral Model Overview](https://docs.mistral.ai/getting-started/models/models_overview/) for the most current information.
-
-### Configuration in Cline
-
-1.  **Open Cline Settings:** Click the settings icon (⚙️) in the Cline panel.
-2.  **Select Provider:** Choose "Mistral" from the "API Provider" dropdown.
-3.  **Enter API Key:** Paste your Mistral API key into the "Mistral API Key" field if you're using a standard `mistral` model. If you intend to use `codestral-latest`, see the "Using Codestral" section below.
-4.  **Select Model:** Choose your desired model from the "Model" dropdown.
-
-### Using Codestral
-
-[Codestral](https://docs.mistral.ai/capabilities/code_generation/) is a model specifically designed for code generation and interaction.
-For Codestral, you can use different endpoints (Default: codestral.mistral.ai).
-If using the La Plateforme API Key for Codestral, change the **Codestral Base Url** to: `https://api.mistral.ai`
-
-To use Codestral with Cline:
-
-1.  **Select "Mistral" as the API Provider in Cline Settings.**
-2.  **Select a Codestral Model** (e.g., `codestral-latest`) from the "Model" dropdown.
-3.  **Enter your Codestral API Key** (from `codestral.mistral.ai`) or your La Plateforme API Key (from `api.mistral.ai`) into the appropriate API key field in Cline.

docs/provider-config/ollama.mdx

@@ -1,78 +0,0 @@
----
-title: "Ollama"
----
-
-Cline supports running models locally using Ollama. This approach offers privacy, offline access, and potentially reduced costs. It requires some initial setup and a sufficiently powerful computer. Because of the present state of consumer hardware, it's not recommended to use Ollama with Cline as performance will likely be poor for average hardware configurations.
-
-**Website:** [https://ollama.com/](https://ollama.com/)
-
-### Setting up Ollama
-
-1.  **Download and Install Ollama:**
-    Obtain the Ollama installer for your operating system from the [Ollama website](https://ollama.com/) and follow their installation guide. Ensure Ollama is running. You can typically start it with:
-
-    ```bash
-    ollama serve
-    ```
-
-2.  **Download a Model:**
-    Ollama supports a wide variety of models. A list of available models can be found on the [Ollama model library](https://ollama.com/library). Some models recommended for coding tasks include:
-
-    -   `codellama:7b-code` (a good, smaller starting point)
-    -   `codellama:13b-code` (offers better quality, larger size)
-    -   `codellama:34b-code` (provides even higher quality, very large)
-    -   `qwen2.5-coder:32b`
-    -   `mistralai/Mistral-7B-Instruct-v0.1` (a solid general-purpose model)
-    -   `deepseek-coder:6.7b-base` (effective for coding)
-    -   `llama3:8b-instruct-q5_1` (suitable for general tasks)
-
-    To download a model, open your terminal and execute:
-
-    ```bash
-    ollama pull <model_name>
-    ```
-
-    For instance:
-
-    ```bash
-    ollama pull qwen2.5-coder:32b
-    ```
-
-3.  **Configure the Model's Context Window:**
-    By default, Ollama models often use a context window of 2048 tokens, which can be insufficient for many Cline requests. A minimum of 12,000 tokens is advisable for decent results, with 32,000 tokens being ideal. To adjust this, you'll modify the model's parameters and save it as a new version.
-
-    First, load the model (using `qwen2.5-coder:32b` as an example):
-
-    ```bash
-    ollama run qwen2.5-coder:32b
-    ```
-
-    Once the model is loaded within the Ollama interactive session, set the context size parameter:
-
-    ```
-    /set parameter num_ctx 32768
-    ```
-
-    Then, save this configured model with a new name:
-
-    ```
-    /save your_custom_model_name
-    ```
-
-    (Replace `your_custom_model_name` with a name of your choice.)
-
-4.  **Configure Cline:**
-    -   Open the Cline sidebar (usually indicated by the Cline icon).
-    -   Click the settings gear icon (⚙️).
-    -   Select "ollama" as the API Provider.
-    -   Enter the Model name you saved in the previous step (e.g., `your_custom_model_name`).
-    -   (Optional) Adjust the base URL if Ollama is running on a different machine or port. The default is `http://localhost:11434`.
-    -   (Optional) Configure the Model context size in Cline's Advanced settings. This helps Cline manage its context window effectively with your customized Ollama model.
-
-### Tips and Notes
-
--   **Resource Demands:** Running large language models locally can be demanding on system resources. Ensure your computer meets the requirements for your chosen model.
--   **Model Choice:** Experiment with various models to discover which best fits your specific tasks and preferences.
--   **Offline Capability:** After downloading a model, you can use Cline with that model even without an internet connection.
--   **Token Usage Tracking:** Cline tracks token usage for models accessed via Ollama, allowing you to monitor consumption.
--   **Ollama's Own Documentation:** For more detailed information, consult the official [Ollama documentation](https://ollama.com/docs).

docs/provider-config/openai-compatible.mdx

@@ -1,72 +0,0 @@
----
-title: "OpenAI Compatible"
-description: "Learn how to configure Cline with various AI model providers that offer OpenAI-compatible APIs."
----
-
-Cline supports a wide range of AI model providers that offer APIs compatible with the OpenAI API standard. This allows you to use models from providers _other than_ OpenAI, while still utilizing a familiar API interface. This includes providers such as:
-
--   **Local models** running through tools like Ollama and LM Studio (which are covered in their respective sections).
--   **Cloud providers** like Perplexity, Together AI, Anyscale, and many others.
--   **Any other provider** that offers an OpenAI-compatible API endpoint.
-
-This document focuses on setting up providers _other than_ the official OpenAI API (which has its own [dedicated configuration page](/provider-config/openai)).
-
-### General Configuration
-
-The key to using an OpenAI-compatible provider with Cline is to configure these main settings:
-
-1.  **Base URL:** This is the API endpoint specific to the provider. It will _not_ be `https://api.openai.com/v1` (that URL is for the official OpenAI API).
-2.  **API Key:** This is the secret key you obtain from your chosen provider.
-3.  **Model ID:** This is the specific name or identifier for the model you wish to use.
-
-You'll find these settings in the Cline settings panel (click the ⚙️ icon):
-
--   **API Provider:** Select "OpenAI Compatible".
--   **Base URL:** Enter the base URL provided by your chosen provider. **This is a crucial step.**
--   **API Key:** Enter your API key from the provider.
--   **Model:** Choose or enter the model ID.
--   **Model Configuration:** This section allows you to customize advanced parameters for the model, such as:
-    -   Max Output Tokens
-    -   Context Window size
-    -   Image Support capabilities
-    -   Computer Use (e.g., for models with tool/function calling)
-    -   Input Price (per token/million tokens)
-    -   Output Price (per token/million tokens)
-
-### Supported Models (for OpenAI Native Endpoint)
-
-While the "OpenAI Compatible" provider type allows connecting to various endpoints, if you are connecting directly to the official OpenAI API (or an endpoint that mirrors it exactly), Cline recognizes the following model IDs based on the `openAiNativeModels` definition in its source code:
-
--   `o3-mini`
--   `o3-mini-high`
--   `o3-mini-low`
--   `o1`
--   `o1-preview`
--   `o1-mini`
--   `gpt-4.5-preview`
--   `gpt-4o`
--   `gpt-4o-mini`
-
-**Note:** If you are using a different OpenAI-compatible provider (such as Together AI, Anyscale, etc.), the available model IDs will differ. Always refer to your specific provider's documentation for their supported model names and any unique configuration details.
-
-### v0 (Vercel SDK) in Cline:
-
--   For developers working with v0, their [AI SDK documentation](https://vercel.com/docs/v0/cline) provides valuable insights and examples for integrating various models, many of which are OpenAI-compatible. This can be a helpful resource for understanding how to structure calls and manage configurations when using Cline with services deployed on or integrated with Vercel.
-
--   v0 can be used in Cline with the OpenAI Compatible provider.
-
--   ### Quickstart
-
--   1. With the OpenAI Compatible provider selected, set the Base URL to https://api.v0.dev/v1.
--   2. Paste in your v0 API Key
--   3. Set the Model ID: v0-1.0-md
--   4. Click Verify to confirm the connection.
-
-### Troubleshooting
-
--   **"Invalid API Key":** Double-check that you've entered the API key correctly and that it's for the correct provider.
--   **"Model Not Found":** Ensure you're using a valid model ID for your chosen provider and that it's available at the specified Base URL.
--   **Connection Errors:** Verify the Base URL is correct, that your provider's API is accessible from your machine, and that there are no firewall or network issues.
--   **Unexpected Results:** If you're getting unexpected outputs, try a different model or double-check all configuration parameters.
-
-By using an OpenAI-compatible provider, you can leverage the flexibility of Cline with a wider array of AI models. Remember to always consult your provider's documentation for the most accurate and up-to-date information.

docs/provider-config/openai.mdx

@@ -1,48 +0,0 @@
----
-title: "OpenAI"
-description: "Learn how to configure and use official OpenAI models with Cline."
----
-
-Cline supports accessing models directly through the official OpenAI API.
-
-**Website:** [https://openai.com/](https://openai.com/)
-
-### Getting an API Key
-
-1.  **Sign Up/Sign In:** Visit the [OpenAI Platform](https://platform.openai.com/). You'll need to create an account or sign in if you already have one.
-2.  **Navigate to API Keys:** Once logged in, go to the [API keys section](https://platform.openai.com/api-keys) of your account.
-3.  **Create a Key:** Click on "Create new secret key". It's good practice to give your key a descriptive name (e.g., "Cline API Key").
-4.  **Copy the Key:** **Crucial:** Copy the generated API key immediately. For security reasons, OpenAI will not show it to you again. Store this key in a safe and secure location.
-
-### Supported Models
-
-Cline is compatible with a variety of OpenAI models, including but not limited to:
-
--   'o3'
--   `o3-mini` (medium reasoning effort)
--   'o4-mini'
--   `o3-mini-high` (high reasoning effort)
--   `o3-mini-low` (low reasoning effort)
--   `o1`
--   `o1-preview`
--   `o1-mini`
--   `gpt-4.5-preview`
--   `gpt-4o`
--   `gpt-4o-mini`
--   'gpt-4.1'
--   'gpt-4.1-mini'
-
-For the most current list of available models and their capabilities, please refer to the official [OpenAI Models documentation](https://platform.openai.com/docs/models).
-
-### Configuration in Cline
-
-1.  **Open Cline Settings:** Click the settings gear icon (⚙️) in the Cline panel.
-2.  **Select Provider:** Choose "OpenAI" from the "API Provider" dropdown menu.
-3.  **Enter API Key:** Paste your OpenAI API key into the "OpenAI API Key" field.
-4.  **Select Model:** Choose your desired model from the "Model" dropdown list.
-5.  **(Optional) Base URL:** If you need to use a proxy or a custom base URL for the OpenAI API, you can enter it here. Most users will not need to change this from the default.
-
-### Tips and Notes
-
--   **Pricing:** Be sure to review the [OpenAI Pricing page](https://openai.com/pricing) for detailed information on the costs associated with different models.
--   **Azure OpenAI Service:** If you are looking to use the Azure OpenAI service, please note that specific documentation for Azure OpenAI with Cline may be found separately, or you might need to configure it as an OpenAI-compatible endpoint if such functionality is supported by Cline for custom configurations.

docs/provider-config/openrouter.mdx

@@ -1,40 +0,0 @@
----
-title: "OpenRouter"
-description: "Learn how to use OpenRouter with Cline to access a wide variety of language models through a single API."
----
-
-OpenRouter is an AI platform that provides access to a wide variety of language models from different providers, all through a single API. This can simplify setup and allow you to easily experiment with different models.
-
-**Website:** [https://openrouter.ai/](https://openrouter.ai/)
-
-### Getting an API Key
-
-1.  **Sign Up/Sign In:** Go to the [OpenRouter website](https://openrouter.ai/). Sign in with your Google or GitHub account.
-2.  **Get an API Key:** Go to the [keys page](https://openrouter.ai/keys). You should see an API key listed. If not, create a new key.
-3.  **Copy the Key:** Copy the API key.
-
-### Supported Models
-
-OpenRouter supports a large and growing number of models. Cline automatically fetches the list of available models. Refer to the [OpenRouter Models page](https://openrouter.ai/models) for the complete and up-to-date list.
-
-### Configuration in Cline
-
-1.  **Open Cline Settings:** Click the settings icon (⚙️) in the Cline panel.
-2.  **Select Provider:** Choose "OpenRouter" from the "API Provider" dropdown.
-3.  **Enter API Key:** Paste your OpenRouter API key into the "OpenRouter API Key" field.
-4.  **Select Model:** Choose your desired model from the "Model" dropdown.
-5.  **(Optional) Custom Base URL:** If you need to use a custom base URL for the OpenRouter API, check "Use custom base URL" and enter the URL. Leave this blank for most users.
-
-### Supported Transforms
-
-OpenRouter provides an [optional "middle-out" message transform](https://openrouter.ai/docs/features/message-transforms) to help with prompts that exceed the maximum context size of a model. You can enable it by checking the "Compress prompts and message chains to the context size" box.
-
-### Tips and Notes
-
--   **Model Selection:** OpenRouter offers a wide range of models. Experiment to find the best one for your needs.
--   **Pricing:** OpenRouter charges based on the underlying model's pricing. See the [OpenRouter Models page](https://openrouter.ai/models) for details.
--   **Prompt Caching:**
-    -   OpenRouter passes caching requests to underlying models that support it. Check the [OpenRouter Models page](https://openrouter.ai/models) to see which models offer caching.
-    -   For most models, caching should activate automatically if supported by the model itself (similar to how Requesty works).
-    -   **Exception for Gemini Models via OpenRouter:** Due to potential response delays sometimes observed with Google's caching mechanism when accessed via OpenRouter, a manual activation step is required _specifically for Gemini models_.
-    -   If using a **Gemini model** via OpenRouter, you **must manually check** the "Enable Prompt Caching" box in the provider settings to activate caching for that model. This checkbox serves as a temporary workaround. For non-Gemini models on OpenRouter, this checkbox is not necessary for caching.

docs/provider-config/requesty.mdx

@@ -1,38 +0,0 @@
----
-title: "Requesty"
-description: "Learn how to use Requesty with Cline to access and optimize over 150 large language models."
----
-
-Cline supports accessing models through the [Requesty](https://www.requesty.ai/) AI platform. Requesty provides an easy and optimized API for interacting with 150+ large language models (LLMs).
-
-**Website:** [https://www.requesty.ai/](https://www.requesty.ai/)
-
-### Getting an API Key
-
-1.  **Sign Up/Sign In:** Go to the [Requesty website](https://www.requesty.ai/) and create an account or sign in.
-2.  **Get API Key:** You can get an API key from the [API Management](https://app.requesty.ai/manage-api) section of your Requesty dashboard.
-
-### Supported Models
-
-Requesty provides access to a wide range of models. Cline will automatically fetch the latest list of available models. You can see the full list of available models on the [Model List](https://app.requesty.ai/router/list) page.
-
-### Configuration in Cline
-
-1.  **Open Cline Settings:** Click the settings icon (⚙️) in the Cline panel.
-2.  **Select Provider:** Choose "Requesty" from the "API Provider" dropdown.
-3.  **Enter API Key:** Paste your Requesty API key into the "Requesty API Key" field.
-4.  **Select Model:** Choose your desired model from the "Model" dropdown.
-
-### Tips and Notes
-
--   **Optimizations**: Requesty offers a range of in-flight cost optimizations to lower your costs.
--   **Unified and simplified billing**: Unrestricted access to all providers and models, automatic balance top ups and more via a single [API key](https://app.requesty.ai/manage-api).
--   **Cost tracking**: Track cost per model, coding language, changed file, and more via the [Cost dashboard](https://app.requesty.ai/cost-management) or the [Requesty VS Code extension](https://marketplace.visualstudio.com/items?itemName=Requesty.requesty).
--   **Stats and logs**: See your [coding stats dashboard](https://app.requesty.ai/usage-stats) or go through your [LLM interaction logs](https://app.requesty.ai/logs).
--   **Fallback policies**: Keep your LLM working for you with fallback policies when providers are down.
--   **Prompt Caching:** Some providers support prompt caching. [Search models with caching](https://app.requesty.ai/router/list).
-
-### Relevant resources
-
--   [Requesty Youtube channel](https://www.youtube.com/@requestyAI)
--   [Requesty Discord](https://requesty.ai/discord)

docs/provider-config/vscode-language-model-api.mdx

@@ -1,51 +0,0 @@
----
-title: "VS Code Language Model API"
-description: "Learn how to use Cline with the experimental VS Code Language Model API, enabling access to models from GitHub Copilot and other compatible extensions."
----
-
-Cline offers _experimental_ support for the [VS Code Language Model API](https://code.visualstudio.com/api/extension-guides/language-model). This API enables extensions to grant access to language models directly within the VS Code environment. Consequently, you might be able to leverage models from:
-
--   **GitHub Copilot:** Provided you have an active Copilot subscription and the extension installed.
--   **Other VS Code Extensions:** Any extension that implements the Language Model API.
-
-**Important Note:** This integration is currently in an experimental phase and might not perform as anticipated. Its functionality relies on other extensions correctly implementing the VS Code Language Model API.
-
-### Prerequisites
-
--   **VS Code:** The Language Model API is accessible via VS Code (it is not currently supported by Cursor).
--   **A Language Model Provider Extension:** An extension that furnishes a language model is required. Examples include:
-    -   **GitHub Copilot:** With a Copilot subscription, the GitHub Copilot and GitHub Copilot Chat extensions can serve as model providers.
-    -   **Alternative Extensions:** Explore the VS Code Marketplace for extensions mentioning "Language Model API" or "lm". Other experimental options may be available
-
-### Configuration Steps
-
-1. **Ensure Copilot Account is Active and Extensions are installed:** User logged into either the Copilot or Copilot Chat extension should be able to gain access via Cline.
-2. **Access Cline Settings:** Click the gear icon (⚙️) located in the Cline panel.
-3. **Choose Provider:** Select "VS Code LM API" from the "API Provider" dropdown menu.
-4. **Select Model:** If the Copilot extension(s) are installed and the user is logged into their Copilot account, the "Language Model" dropdown will populate with available models after a short time. The naming convention is `vendor/family`. For instance, if Copilot is active, you might encounter options such as:
-    - `copilot - gpt-3.5-turbo`
-    - `copilot - gpt-4o-mini`
-    - `copilot - gpt-4`
-    - `copilot - gpt-4-turbo`
-    - `copilot - gpt-4o`
-    - `copilot - claude-3.5-sonnet` **NOTE:** this model does not work.
-    - `copilot - gemini-2.0-flash`
-    - `copilot - gpt-4.1`
-
-For best results with the VSCode LM API Provider, we suggest using the OpenAI Models (GPT 3, 4, 4.1, 4o etc.)
-
-### Current Limitations
-
--   **Experimental API Status:** The VS Code Language Model API is still under active development. Anticipate potential changes and instability.
--   **Dependency on Extensions:** This feature is entirely contingent on other extensions making models available. Cline does not directly control the list of accessible models.
--   **Restricted Functionality:** The VS Code Language Model API might not encompass all features available through other API providers (e.g., image input capabilities, streaming responses, detailed usage metrics).
--   **No Direct Cost Management:** Users are subject to the pricing structures and terms of service of the extension providing the model. Cline cannot directly monitor or regulate associated costs.
--   **GitHub Copilot Rate Throttling:** When employing the VS Code LM API with GitHub Copilot, be mindful that GitHub may enforce rate limits on Copilot usage. These limitations are governed by GitHub, not Cline.
-
-### Troubleshooting Tips
-
--   **Models Not Appearing:**
-    -   Confirm that VS Code is installed.
-    -   Verify that a language model provider extension (e.g., GitHub Copilot, GitHub Copilot Chat) is installed and enabled.
-    -   If utilizing Copilot, ensure you have previously sent a Copilot Chat message using the desired model.
--   **Unexpected Operation:** Should you encounter unforeseen behavior, it is likely an issue stemming from the underlying Language Model API or the provider extension. Consider reporting the problem to the developers of the provider extension.

docs/provider-config/xai-grok.mdx

@@ -1,85 +0,0 @@
----
-title: "xAI (Grok)"
-description: "Learn how to configure and use xAI's Grok models with Cline, including API key setup, supported models, and reasoning capabilities."
----
-
-xAI is the company behind Grok, a large language model known for its conversational abilities and large context window. Grok models are designed to provide helpful, informative, and contextually relevant responses.
-
-**Website:** [https://x.ai/](https://x.ai/)
-
-### Getting an API Key
-
-1.  **Sign Up/Sign In:** Go to the [xAI Console](https://console.x.ai/). Create an account or sign in.
-2.  **Navigate to API Keys:** Go to the API keys section in your dashboard.
-3.  **Create a Key:** Click to create a new API key. Give your key a descriptive name (e.g., "Cline").
-4.  **Copy the Key:** **Important:** Copy the API key _immediately_. You will not be able to see it again. Store it securely.
-
-### Supported Models
-
-Cline supports the following xAI Grok models:
-
-#### Grok-3 Models
-
--   `grok-3-beta` (Default) - xAI's Grok-3 beta model with 131K context window
--   `grok-3-fast-beta` - xAI's Grok-3 fast beta model with 131K context window
--   `grok-3-mini-beta` - xAI's Grok-3 mini beta model with 131K context window
--   `grok-3-mini-fast-beta` - xAI's Grok-3 mini fast beta model with 131K context window
-
-#### Grok-2 Models
-
--   `grok-2-latest` - xAI's Grok-2 model - latest version with 131K context window
--   `grok-2` - xAI's Grok-2 model with 131K context window
--   `grok-2-1212` - xAI's Grok-2 model (version 1212) with 131K context window
-
-#### Grok Vision Models
-
--   `grok-2-vision-latest` - xAI's Grok-2 Vision model - latest version with image support and 32K context window
--   `grok-2-vision` - xAI's Grok-2 Vision model with image support and 32K context window
--   `grok-2-vision-1212` - xAI's Grok-2 Vision model (version 1212) with image support and 32K context window
--   `grok-vision-beta` - xAI's Grok Vision Beta model with image support and 8K context window
-
-#### Legacy Models
-
--   `grok-beta` - xAI's Grok Beta model (legacy) with 131K context window
-
-### Configuration in Cline
-
-1.  **Open Cline Settings:** Click the settings icon (⚙️) in the Cline panel.
-2.  **Select Provider:** Choose "xAI" from the "API Provider" dropdown.
-3.  **Enter API Key:** Paste your xAI API key into the "xAI API Key" field.
-4.  **Select Model:** Choose your desired Grok model from the "Model" dropdown.
-
-### Reasoning Capabilities
-
-Grok 3 Mini models feature specialized reasoning capabilities, allowing them to "think before responding" - particularly useful for complex problem-solving tasks.
-
-#### Reasoning-Enabled Models
-
-Reasoning is only supported by:
-
--   `grok-3-mini-beta`
--   `grok-3-mini-fast-beta`
-
-The Grok 3 models `grok-3-beta` and `grok-3-fast-beta` do not support reasoning.
-
-#### Controlling Reasoning Effort
-
-When using reasoning-enabled models, you can control how hard the model thinks with the `reasoning_effort` parameter:
-
--   `low`: Minimal thinking time, using fewer tokens for quick responses
--   `high`: Maximum thinking time, leveraging more tokens for complex problems
-
-Choose `low` for simple queries that should complete quickly, and `high` for harder problems where response latency is less important.
-
-#### Key Features
-
--   **Step-by-Step Problem Solving**: The model thinks through problems methodically before delivering an answer
--   **Math & Quantitative Strength**: Excels at numerical challenges and logic puzzles
--   **Reasoning Trace Access**: The model's thinking process is available via the `reasoning_content` field in the response completion object
-
-### Tips and Notes
-
--   **Context Window:** Most Grok models feature large context windows (up to 131K tokens), allowing you to include substantial amounts of code and context in your prompts.
--   **Vision Capabilities:** Select vision-enabled models (`grok-2-vision-latest`, `grok-2-vision`, etc.) when you need to process or analyze images.
--   **Pricing:** Pricing varies by model, with input costs ranging from $0.3 to $5.0 per million tokens and output costs from $0.5 to $25.0 per million tokens. Refer to the xAI documentation for the most current pricing information.
--   **Performance Tradeoffs:** "Fast" variants typically offer quicker response times but may have higher costs, while "mini" variants are more economical but may have reduced capabilities.

esbuild.js

@@ -125,17 +125,23 @@ const baseConfig = {
 	minify: production,
 	sourcemap: !production,
 	logLevel: "silent",
-	define: production
-		? {
-				"process.env.IS_DEV": JSON.stringify(!production),
-			}
-		: undefined,
+	define: {
+		"process.env.IS_DEV": JSON.stringify(!production),
+	},
 	tsconfig: path.resolve(__dirname, "tsconfig.json"),
 	plugins: [
 		copyWasmFiles,
 		aliasResolverPlugin,
 		/* add to the end of plugins array */
 		esbuildProblemMatcherPlugin,
+		{
+			name: "alias-plugin",
+			setup(build) {
+				build.onResolve({ filter: /^pkce-challenge$/ }, (args) => {
+					return { path: require.resolve("pkce-challenge/dist/index.browser.js") }
+				})
+			},
+		},
 	],
 	format: "cjs",
 	sourcesContent: false,

eslint-rules/__tests__/no-grpc-client-object-literals.test.ts

@@ -1,174 +0,0 @@
-const { RuleTester: GrpcRuleTester } = require("eslint")
-const grpcRule = require("../no-grpc-client-object-literals")
-
-const grpcRuleTester = new GrpcRuleTester({
-	parser: require.resolve("@typescript-eslint/parser"),
-	parserOptions: {
-		ecmaVersion: 2020,
-		sourceType: "module",
-		ecmaFeatures: {
-			jsx: true,
-		},
-	},
-})
-
-grpcRuleTester.run("no-grpc-client-object-literals", grpcRule, {
-	valid: [
-		// Valid case: Using .create() method with gRPC client
-		{
-			code: `
-                import { TogglePlanActModeRequest } from '@shared/proto/state';
-                import { StateServiceClient } from '../services/grpc-client';
-                
-                StateServiceClient.togglePlanActMode(
-                    TogglePlanActModeRequest.create({
-                        chatSettings: {
-                            mode: PlanActMode.PLAN,
-                            preferredLanguage: 'en',
-                        },
-                    })
-                );
-            `,
-		},
-		// Valid case: Using .fromPartial() method with gRPC client
-		{
-			code: `
-                import { TogglePlanActModeRequest, ChatSettings } from '@shared/proto/state';
-                import { StateServiceClient } from '../services/grpc-client';
-                
-                const chatSettings = ChatSettings.fromPartial({
-                    mode: PlanActMode.PLAN,
-                    preferredLanguage: 'en',
-                });
-                
-                StateServiceClient.togglePlanActMode(
-                    TogglePlanActModeRequest.create({
-                        chatSettings: chatSettings,
-                    })
-                );
-            `,
-		},
-		// Valid case: Regular function call with object literal (not a gRPC client)
-		{
-			code: `
-                function processData(data) {
-                    console.log(data);
-                }
-                
-                processData({
-                    id: 123,
-                    name: 'test',
-                });
-            `,
-		},
-		// Valid case: Using proper nested protobuf objects
-		{
-			code: `
-                import { TogglePlanActModeRequest, ChatSettings } from '@shared/proto/state';
-                import { StateServiceClient } from '../services/grpc-client';
-                
-                // Using proper nested protobuf objects
-                const chatSettings = ChatSettings.create({
-                    mode: 0,
-                    preferredLanguage: 'en',
-                });
-                
-                const request = TogglePlanActModeRequest.create({
-                    chatSettings: chatSettings,
-                });
-                
-                StateServiceClient.togglePlanActMode(request);
-            `,
-		},
-		// Valid case: Object literal in second parameter (should not be checked)
-		{
-			code: `
-                import { StateSubscribeRequest } from '@shared/proto/state';
-                import { StateServiceClient } from '../services/grpc-client';
-                
-                const request = StateSubscribeRequest.create({
-                    topics: ['apiConfig', 'tasks']
-                });
-                
-                // Second parameter is an object literal but should not trigger the rule
-                StateServiceClient.subscribe(request, {
-                    metadata: {
-                        userId: 123,
-                        sessionId: "abc-123"
-                    }
-                });
-            `,
-		},
-	],
-	invalid: [
-		// Invalid case: Using object literal directly with gRPC client
-		{
-			code: `
-                import { StateServiceClient } from '../services/grpc-client';
-                
-                StateServiceClient.togglePlanActMode({
-                    chatSettings: {
-                        mode: 0,
-                        preferredLanguage: 'en',
-                    },
-                });
-            `,
-			errors: [{ messageId: "useProtobufMethod" }],
-		},
-		// Invalid case: Using object literal with nested properties
-		{
-			code: `
-                import { ChatSettings } from '@shared/proto/state';
-                import { StateServiceClient } from '../services/grpc-client';
-                
-                const chatSettings = ChatSettings.create({
-                    mode: 0,
-                    preferredLanguage: 'en',
-                });
-                
-                StateServiceClient.togglePlanActMode({
-                    chatSettings: {
-                        mode: 1,
-                        preferredLanguage: 'fr',
-                    },
-                });
-            `,
-			errors: [{ messageId: "useProtobufMethod" }],
-		},
-		// Invalid case: Nested object literal in protobuf create method
-		{
-			code: `
-                import { TogglePlanActModeRequest, ChatSettings } from '@shared/proto/state';
-                import { StateServiceClient } from '../services/grpc-client';
-                
-                // Using nested object literal instead of ChatSettings.create()
-                const request = TogglePlanActModeRequest.create({
-                    chatSettings: {
-                        mode: 0,
-                        preferredLanguage: 'en',
-                    },
-                });
-                
-                StateServiceClient.togglePlanActMode(request);
-            `,
-			errors: [{ messageId: "useProtobufMethod" }],
-		},
-		// Invalid case: Object literal as first parameter to subscribe method
-		{
-			code: `
-                import { StateServiceClient } from '../services/grpc-client';
-                
-                // First parameter is an object literal, which should trigger the rule
-                StateServiceClient.subscribe({
-                    topics: ['apiConfig', 'tasks']
-                }, {
-                    metadata: {
-                        userId: 123,
-                        sessionId: "abc-123"
-                    }
-                });
-            `,
-			errors: [{ messageId: "useProtobufMethod" }],
-		},
-	],
-})

eslint-rules/__tests__/no-protobuf-object-literals.test.ts

@@ -1,214 +0,0 @@
-const { RuleTester } = require("eslint")
-const rule = require("../no-protobuf-object-literals")
-
-const ruleTester = new RuleTester({
-	parser: require.resolve("@typescript-eslint/parser"),
-	parserOptions: {
-		ecmaVersion: 2020,
-		sourceType: "module",
-		ecmaFeatures: {
-			jsx: true,
-		},
-	},
-})
-
-ruleTester.run("no-protobuf-object-literals", rule, {
-	valid: [
-		// Valid case: Using .create() method
-		{
-			code: `
-        import { State } from '@shared/proto/state';
-        
-        const state = State.create({
-          stateJson: '{"apiConfig":{"provider":"anthropic","model":"claude-3-haiku"}}'
-        });
-      `,
-		},
-		// Valid case: Using .fromPartial() method
-		{
-			code: `
-        import { ChatSettings } from '@shared/proto/state';
-        
-        const settings = ChatSettings.fromPartial({
-          mode: 0,
-          preferredLanguage: 'en',
-          openAiReasoningEffort: 'thorough'
-        });
-      `,
-		},
-		// Valid case: Object literal not used with protobuf type
-		{
-			code: `
-        interface MyInterface {
-          id: number;
-          name: string;
-        }
-        
-        const obj: MyInterface = {
-          id: 123,
-          name: 'test'
-        };
-      `,
-		},
-		// Valid case: Using object literal for non-protobuf import
-		{
-			code: `
-        import { SomeType } from '@some/other/package';
-
-        const obj: SomeType = {
-          id: 123,
-          name: 'test'
-        };
-      `,
-		},
-		// Valid case: Regular function call with object literal (should not be flagged)
-		{
-			code: `
-        import { State } from '@shared/proto/state';
-
-        // This should not be flagged because it's a regular function call
-        // not directly tied to a protobuf type
-        process({
-          id: 123,
-          name: 'test',
-          data: { nested: true }
-        });
-      `,
-		},
-	],
-	invalid: [
-		// Invalid case: Using object literal with imported protobuf type
-		{
-			code: `
-        import { State } from '@shared/proto/state';
-
-        const state: State = {
-          stateJson: '{"apiConfig":{"provider":"anthropic","model":"claude-3-haiku"}}'
-        };
-      `,
-			output: `
-        import { State } from '@shared/proto/state';
-
-        const state: State = State.create({
-          stateJson: '{"apiConfig":{"provider":"anthropic","model":"claude-3-haiku"}}'
-        });
-      `,
-			errors: [{ messageId: "useProtobufMethod" }],
-		},
-		// Invalid case: Using object literal with namespaced protobuf type
-		{
-			code: `
-        import * as stateProto from '@shared/proto/state';
-
-        const state: stateProto.State = {
-          stateJson: '{"apiConfig":{"provider":"anthropic","model":"claude-3-haiku"}}'
-        };
-      `,
-			output: `
-        import * as stateProto from '@shared/proto/state';
-
-        const state: stateProto.State = stateProto.State.create({
-          stateJson: '{"apiConfig":{"provider":"anthropic","model":"claude-3-haiku"}}'
-        });
-      `,
-			errors: [{ messageId: "useProtobufMethodGeneric" }],
-		},
-		// Invalid case: Using object literal in a return statement (with protobuf return type)
-		{
-			code: `
-        import { ChatSettings } from '@shared/proto/state';
-
-        function createSettings(): ChatSettings {
-          return {
-            mode: 0,
-            preferredLanguage: 'en',
-            openAiReasoningEffort: 'thorough'
-          };
-        }
-      `,
-			output: `
-        import { ChatSettings } from '@shared/proto/state';
-
-        function createSettings(): ChatSettings {
-          return ChatSettings.create({
-            mode: 0,
-            preferredLanguage: 'en',
-            openAiReasoningEffort: 'thorough'
-          });
-        }
-      `,
-			errors: [{ messageId: "useProtobufMethod" }],
-		},
-		// Invalid case: Using object literal in a function parameter (with protobuf types imported)
-		{
-			code: `
-        import { ChatContent } from '@shared/proto/state';
-
-        function processContent(content: ChatContent) {
-          // process the content
-        }
-
-        processContent({
-          message: 'Hello, this is a test message',
-          images: ['image1.png', 'image2.jpg'],
-          files: ['file1.txt', 'file2.pdf']
-        });
-      `,
-			output: `
-        import { ChatContent } from '@shared/proto/state';
-
-        function processContent(content: ChatContent) {
-          // process the content
-        }
-
-        processContent(ChatContent.create({
-          message: 'Hello, this is a test message',
-          images: ['image1.png', 'image2.jpg'],
-          files: ['file1.txt', 'file2.pdf']
-        }));
-      `,
-			errors: [{ messageId: "useProtobufMethodGeneric" }],
-		},
-		// Invalid case: Using object literal in assignment expression
-		{
-			code: `
-        import { State } from '@shared/proto/state';
-
-        let state: State;
-        state = {
-          stateJson: '{"apiConfig":{"provider":"anthropic","model":"claude-3-haiku"}}'
-        };
-      `,
-			output: `
-        import { State } from '@shared/proto/state';
-
-        let state: State;
-        state = State.create({
-          stateJson: '{"apiConfig":{"provider":"anthropic","model":"claude-3-haiku"}}'
-        });
-      `,
-			errors: [{ messageId: "useProtobufMethod" }],
-		},
-		// Test with custom protobufPackages option
-		{
-			code: `
-        import { CustomProto } from 'custom/proto/package';
-
-        const obj: CustomProto = {
-          field1: 'value',
-          field2: 123
-        };
-      `,
-			output: `
-        import { CustomProto } from 'custom/proto/package';
-
-        const obj: CustomProto = CustomProto.create({
-          field1: 'value',
-          field2: 123
-        });
-      `,
-			options: [{ protobufPackages: ["custom/proto"] }],
-			errors: [{ messageId: "useProtobufMethod" }],
-		},
-	],
-})

eslint-rules/__tests__/no-vscode-postmessage.test.ts

@@ -1,74 +0,0 @@
-const { RuleTester: VscodeRuleTester } = require("eslint")
-const vscodePostmessageRule = require("../no-vscode-postmessage")
-
-const vscodeRuleTester = new VscodeRuleTester({
-	parser: require.resolve("@typescript-eslint/parser"),
-	parserOptions: {
-		ecmaVersion: 2020,
-		sourceType: "module",
-		ecmaFeatures: {
-			jsx: true,
-		},
-	},
-})
-
-vscodeRuleTester.run("no-vscode-postmessage", vscodePostmessageRule, {
-	valid: [
-		// Should allow vscode.postMessage in grpc-client-base.ts
-		{
-			code: `vscode.postMessage({ type: "grpc_request", data: {} })`,
-			filename: "grpc-client-base.ts",
-		},
-		{
-			code: `vscode.postMessage({ type: "grpc_request_cancel" })`,
-			filename: "/path/to/grpc-client-base.ts",
-		},
-		// Should allow other vscode API calls
-		{
-			code: `vscode.window.showInformationMessage("Hello")`,
-			filename: "test.ts",
-		},
-		// Should allow postMessage calls on other objects
-		{
-			code: `window.postMessage({ type: "test" }, "*")`,
-			filename: "test.ts",
-		},
-		// Should allow variables named vscode but not calling postMessage
-		{
-			code: `const vscode = { other: "method" }; vscode.other()`,
-			filename: "test.ts",
-		},
-	],
-	invalid: [
-		// Should ban vscode.postMessage in regular files
-		{
-			code: `vscode.postMessage({ type: "test", data: {} })`,
-			filename: "test.ts",
-			errors: [
-				{
-					messageId: "useGrpcClient",
-				},
-			],
-		},
-		// Should ban vscode.postMessage in components
-		{
-			code: `vscode.postMessage({ type: "apiConfiguration", apiConfiguration })`,
-			filename: "ApiOptions.tsx",
-			errors: [
-				{
-					messageId: "useGrpcClient",
-				},
-			],
-		},
-		// Should ban vscode.postMessage in test files
-		{
-			code: `vscode.postMessage({ type: "newTask", text: message.text })`,
-			filename: "test.test.ts",
-			errors: [
-				{
-					messageId: "useGrpcClient",
-				},
-			],
-		},
-	],
-})

eslint-rules/index.js

@@ -1,22 +0,0 @@
-// eslint-rules/index.js
-const noProtobufObjectLiterals = require("./no-protobuf-object-literals")
-const noGrpcClientObjectLiterals = require("./no-grpc-client-object-literals")
-const noVscodePostmessage = require("./no-vscode-postmessage")
-
-module.exports = {
-	rules: {
-		"no-protobuf-object-literals": noProtobufObjectLiterals,
-		"no-grpc-client-object-literals": noGrpcClientObjectLiterals,
-		"no-vscode-postmessage": noVscodePostmessage,
-	},
-	configs: {
-		recommended: {
-			plugins: ["local"],
-			rules: {
-				"local/no-protobuf-object-literals": "error",
-				"local/no-grpc-client-object-literals": "error",
-				"local/no-vscode-postmessage": "error",
-			},
-		},
-	},
-}

eslint-rules/no-grpc-client-object-literals.js

@@ -1,216 +0,0 @@
-const { ESLintUtils } = require("@typescript-eslint/utils")
-
-const createRule = ESLintUtils.RuleCreator((name) => `https://cline.bot/eslint-rules/${name}`)
-
-module.exports = createRule({
-	name: "no-grpc-client-object-literals",
-	meta: {
-		type: "problem",
-		docs: {
-			description:
-				"Enforce using .create() or .fromPartial() for gRPC service client parameters instead of object literals",
-			recommended: "error",
-		},
-		messages: {
-			useProtobufMethod:
-				"Use the appropriate protobuf .create() or .fromPartial() method instead of " +
-				"object literal for gRPC client parameters.\n" +
-				"Found: {{code}}\n" +
-				"gRPC client methods should always receive properly created protobuf objects.",
-		},
-		schema: [],
-	},
-	defaultOptions: [],
-
-	create(context) {
-		// Check if a name matches the gRPC service client pattern using regex
-		// Must start with an uppercase letter and end with ServiceClient
-		const isGrpcServiceClient = (name) => {
-			return typeof name === "string" && /^[A-Z].*ServiceClient$/.test(name)
-		}
-
-		const safeObjectExpressions = new Map() // Track object expressions in create/fromPartial calls
-
-		return {
-			// Skip object literals inside create() or fromPartial() method calls
-			CallExpression(node) {
-				if (
-					node.callee &&
-					node.callee.type === "MemberExpression" &&
-					(node.callee.property.name === "create" || node.callee.property.name === "fromPartial") &&
-					node.arguments.length > 0 &&
-					node.arguments[0].type === "ObjectExpression"
-				) {
-					// Track this object expression as being used with create/fromPartial
-					safeObjectExpressions.set(node.arguments[0], { isProblematic: false })
-				}
-			},
-
-			// Track create/fromPartial calls that contain nested object literals
-			"CallExpression[callee.type='MemberExpression'][callee.property.name=/^(create|fromPartial)$/]"(node) {
-				if (node.arguments.length > 0 && node.arguments[0].type === "ObjectExpression") {
-					// Track problematic nested object literals
-					const nestedObjectLiterals = new Map() // Map of object expressions to their containing property paths
-
-					// Search for nested object literals
-					const queue = [
-						...node.arguments[0].properties.map((prop) => ({
-							property: prop,
-							path: prop.key && prop.key.name ? prop.key.name : "unknown",
-						})),
-					]
-
-					while (queue.length > 0) {
-						const { property, path } = queue.shift()
-
-						// Skip spread elements
-						if (property.type !== "Property") continue
-
-						// If this is an object literal, mark it as problematic
-						if (property.value.type === "ObjectExpression") {
-							nestedObjectLiterals.set(property.value, path)
-
-							// Add nested properties to queue
-							queue.push(
-								...property.value.properties.map((prop) => ({
-									property: prop,
-									path: `${path}.${prop.key && prop.key.name ? prop.key.name : "unknown"}`,
-								})),
-							)
-						}
-					}
-
-					// For each problematic nested object, track it with its path
-					nestedObjectLiterals.forEach((path, objectExpr) => {
-						safeObjectExpressions.set(objectExpr, {
-							isProblematic: true,
-							path: path,
-							parentNode: node,
-						})
-					})
-				}
-			},
-
-			// Check calls to gRPC service clients
-			"CallExpression[callee.type='MemberExpression']"(node) {
-				// Get the object (left side) of the member expression
-				const callee = node.callee
-				if (callee.object && callee.object.type === "Identifier") {
-					const objectName = callee.object.name
-
-					// Check if this is a call to one of our gRPC service clients
-					if (isGrpcServiceClient(objectName)) {
-						// Only check the first argument of gRPC service client calls
-						if (node.arguments.length > 0) {
-							const arg = node.arguments[0] // Only check the first parameter
-							if (arg.type === "ObjectExpression" && !safeObjectExpressions.has(arg)) {
-								// This is an object literal being passed directly to a gRPC client
-								const sourceCode = context.getSourceCode()
-								const callText = sourceCode.getText(node).trim()
-
-								context.report({
-									node: arg,
-									messageId: "useProtobufMethod",
-									data: {
-										code: callText,
-									},
-								})
-							} else if (arg.type === "ObjectExpression") {
-								// Search for nested object literals that aren't protected
-								const queue = [...arg.properties]
-								while (queue.length > 0) {
-									const property = queue.shift()
-
-									// Skip spread elements
-									if (property.type !== "Property") continue
-
-									// Check value
-									if (
-										property.value.type === "ObjectExpression" &&
-										!safeObjectExpressions.has(property.value)
-									) {
-										// Found a nested object literal
-										const sourceCode = context.getSourceCode()
-										const propertyText = sourceCode.getText(property).trim()
-
-										context.report({
-											node: property.value,
-											messageId: "useProtobufMethod",
-											data: {
-												code: `${objectName}.${callee.property.name}(... ${propertyText} ...)`,
-											},
-										})
-									}
-
-									// Add any nested properties to the queue
-									if (property.value.type === "ObjectExpression") {
-										queue.push(...property.value.properties)
-									}
-								}
-							} else if (arg.type === "Identifier") {
-								// This is a variable - check if it references a problematic protobuf object
-								const varName = arg.name
-								const sourceCode = context.getSourceCode()
-								const scope = sourceCode.getScope(node)
-
-								// Find the variable declaration
-								const variable = scope.variables.find((v) => v.name === varName)
-								if (variable && variable.references && variable.references.length > 0) {
-									// Look for definitions
-									const def = variable.defs.find(
-										(d) => d.node && d.node.type === "VariableDeclarator" && d.node.init,
-									)
-
-									if (
-										def &&
-										def.node.init.type === "CallExpression" &&
-										def.node.init.callee.type === "MemberExpression" &&
-										(def.node.init.callee.property.name === "create" ||
-											def.node.init.callee.property.name === "fromPartial")
-									) {
-										// Flag if we find problematic nested object literals in this create/fromPartial call
-										const callText = sourceCode.getText(node).trim()
-										const initCallText = sourceCode.getText(def.node.init).trim()
-
-										// Check for nested object literals in init node
-										let foundNestedLiteral = false
-										if (
-											def.node.init.arguments.length > 0 &&
-											def.node.init.arguments[0].type === "ObjectExpression"
-										) {
-											// Find any nested object literals
-											const queue = [...def.node.init.arguments[0].properties]
-											while (queue.length > 0 && !foundNestedLiteral) {
-												const property = queue.shift()
-
-												// Skip spread elements
-												if (property.type !== "Property") continue
-
-												if (property.value.type === "ObjectExpression") {
-													foundNestedLiteral = true
-
-													context.report({
-														node,
-														messageId: "useProtobufMethod",
-														data: {
-															code: `${callText} - using request created with nested object literal at: ${property.key.name}`,
-														},
-													})
-												}
-
-												// Add any nested properties to the queue
-												if (property.value.type === "ObjectExpression") {
-													queue.push(...property.value.properties)
-												}
-											}
-										}
-									}
-								}
-							}
-						}
-					}
-				}
-			},
-		}
-	},
-})

eslint-rules/no-protobuf-object-literals.js

@@ -1,556 +0,0 @@
-const { ESLintUtils } = require("@typescript-eslint/utils")
-
-const createRule = ESLintUtils.RuleCreator((name) => `https://cline.bot/eslint-rules/${name}`)
-
-module.exports = createRule({
-	name: "no-protobuf-object-literals",
-	meta: {
-		type: "problem",
-		docs: {
-			description: "Enforce using .create() or .fromPartial() for protobuf objects instead of object literals",
-			recommended: "error",
-		},
-		fixable: "code",
-		messages: {
-			useProtobufMethod:
-				"Use {{typeName}}.create() or {{typeName}}.fromPartial() instead of " +
-				"object literal for protobuf type from @shared/proto\n" +
-				"Found: {{code}}\n  Suggestion: " +
-				"{{typeName}}.create({{objectContent}})",
-			useProtobufMethodGeneric:
-				"Use .create() or .fromPartial() instead of object literal for protobuf " +
-				"type from @shared/proto\n  Found: {{code}}",
-		},
-		schema: [
-			{
-				type: "object",
-				properties: {
-					protobufPackages: {
-						type: "array",
-						items: { type: "string" },
-						default: ["shared/proto/"],
-					},
-				},
-				additionalProperties: false,
-			},
-		],
-	},
-	defaultOptions: [{ protobufPackages: ["shared/proto/"] }],
-
-	create(context, [options]) {
-		const protobufPackages = options.protobufPackages
-		const protobufImports = new Set() // Set of imported protobuf types
-		const protobufNamespaceImports = new Set() // For namespace imports like "import * as proto"
-		const safeObjectExpressions = new Set() // Track object expressions in create/fromPartial calls
-
-		return {
-			// Skip object literals inside create() or fromPartial() method calls
-			CallExpression(node) {
-				if (
-					node.callee &&
-					node.callee.type === "MemberExpression" &&
-					(node.callee.property.name === "create" || node.callee.property.name === "fromPartial") &&
-					node.arguments.length > 0 &&
-					node.arguments[0].type === "ObjectExpression"
-				) {
-					// Track this object expression as being used with create/fromPartial
-					safeObjectExpressions.add(node.arguments[0])
-				}
-			},
-
-			// Track imports from protobuf packages
-			ImportDeclaration(node) {
-				const packageName = node.source.value
-
-				if (matchesProtobufPackage(packageName, protobufPackages)) {
-					// This is a protobuf package.
-					node.specifiers.forEach((spec) => {
-						if (spec.type === "ImportSpecifier") {
-							// import { MyRequest } from '@shared/proto'
-							protobufImports.add(spec.imported.name)
-						} else if (spec.type === "ImportNamespaceSpecifier") {
-							// import * as proto from '@shared/proto'
-							protobufNamespaceImports.add(spec.local.name)
-						}
-					})
-				}
-			},
-
-			// Check variable declarations with type annotations
-			"VariableDeclarator > ObjectExpression"(node) {
-				// Skip if this is inside a create/fromPartial call
-				if (safeObjectExpressions.has(node)) {
-					return
-				}
-
-				// Found object literal in variable declaration
-				const declarator = node.parent
-
-				if (declarator.id && declarator.id.typeAnnotation) {
-					const typeName = getTypeName(declarator.id.typeAnnotation.typeAnnotation)
-					if (typeName) {
-						// Check if it's a direct protobuf import
-						if (protobufImports.has(typeName)) {
-							//console.log('🚨 VIOLATION: Using object literal for protobuf type:', typeName);
-							const sourceCode = context.getSourceCode()
-							const declaratorText = sourceCode.getText(declarator)
-							const objectText = sourceCode.getText(node)
-
-							context.report({
-								node,
-								messageId: "useProtobufMethod",
-								data: {
-									typeName,
-									code: declaratorText,
-									objectContent: objectText,
-								},
-								fix(fixer) {
-									// Replace the object literal with Type.create() call
-									return fixer.replaceText(node, `${typeName}.create(${objectText})`)
-								},
-							})
-							return
-						}
-
-						// Check if it's a namespaced protobuf type (e.g., proto.MyRequest)
-						if (isNamespacedProtobufType(protobufNamespaceImports, typeName)) {
-							//console.log('🚨 VIOLATION: Using object literal for namespaced protobuf type:', typeName);
-							const sourceCode = context.getSourceCode()
-							const declaratorText = sourceCode.getText(declarator)
-							context.report({
-								node,
-								messageId: "useProtobufMethodGeneric",
-								data: { code: declaratorText },
-								fix(fixer) {
-									// For namespaced types, use the full type name to call create()
-									return fixer.replaceText(node, `${typeName}.create(${sourceCode.getText(node)})`)
-								},
-							})
-						}
-					}
-				}
-			},
-
-			// Check assignment expressions
-			"AssignmentExpression > ObjectExpression"(node) {
-				// Skip if this is inside a create/fromPartial call
-				if (safeObjectExpressions.has(node)) {
-					return
-				}
-
-				const assignment = node.parent
-
-				// For assignment to variables without inline type annotation
-				if (assignment.left && assignment.right === node) {
-					let typeName = null
-
-					// Check if there's a typeAnnotation directly on the left
-					if (assignment.left.typeAnnotation) {
-						typeName = getTypeName(assignment.left.typeAnnotation.typeAnnotation)
-					}
-					// Otherwise try to infer from the variable name if it's a simple identifier
-					else if (assignment.left.type === "Identifier") {
-						const varName = assignment.left.name
-						// Check variable declarations in the current scope
-						const sourceCode = context.getSourceCode()
-						const scope = sourceCode.getScope(node)
-						const variable = scope.variables.find((v) => v.name === varName)
-						if (variable && variable.defs.length > 0) {
-							const def = variable.defs[0]
-							if (def.node.id && def.node.id.typeAnnotation) {
-								typeName = getTypeName(def.node.id.typeAnnotation.typeAnnotation)
-							}
-						}
-					}
-
-					if (typeName && protobufImports.has(typeName)) {
-						//console.log('🚨 VIOLATION: Using object literal in assignment for protobuf type:', typeName);
-						const sourceCode = context.getSourceCode()
-						const assignmentText = sourceCode.getText(assignment.left) + " = "
-						const objectText = sourceCode.getText(node)
-
-						context.report({
-							node,
-							messageId: "useProtobufMethod",
-							data: {
-								typeName,
-								code: assignmentText + "{",
-								objectContent: objectText,
-							},
-							fix(fixer) {
-								// Replace the object literal with Type.create() call in assignments
-								return fixer.replaceText(node, `${typeName}.create(${objectText})`)
-							},
-						})
-					}
-				}
-			},
-
-			// Check return statements
-			"ReturnStatement > ObjectExpression"(node) {
-				// Skip if this is inside a create/fromPartial call
-				if (safeObjectExpressions.has(node)) {
-					return
-				}
-
-				// Find the parent function to get its return type
-				const functionNode = findParentFunction(node)
-				if (!functionNode) {
-					return
-				}
-
-				// Try to get the return type using our enhanced helper
-				const sourceCode = context.getSourceCode()
-				let returnTypeName = getFunctionReturnType(functionNode, sourceCode)
-
-				// For async functions with Promise<Type> return type, extract the inner type
-				if (returnTypeName && returnTypeName.startsWith("Promise<") && returnTypeName.endsWith(">")) {
-					returnTypeName = returnTypeName.slice(8, -1)
-				}
-
-				// Check if the return type is a protobuf type
-				if (returnTypeName) {
-					if (protobufImports.has(returnTypeName)) {
-						//console.log('🚨 VIOLATION: Return type is a protobuf type:', returnTypeName);
-						const sourceCode = context.getSourceCode()
-						const returnText = sourceCode.getText(node.parent)
-						context.report({
-							node,
-							messageId: "useProtobufMethod",
-							data: {
-								typeName: returnTypeName,
-								code: returnText,
-								objectContent: sourceCode.getText(node),
-							},
-							fix(fixer) {
-								// Replace the object literal with Type.create() call in return statements
-								return fixer.replaceText(node, `${returnTypeName}.create(${sourceCode.getText(node)})`)
-							},
-						})
-						return
-					}
-
-					// Check if it's a namespaced protobuf type
-					if (isNamespacedProtobufType(protobufNamespaceImports, returnTypeName)) {
-						const sourceCode = context.getSourceCode()
-						const returnText = sourceCode.getText(node.parent)
-						//console.log('🚨 VIOLATION: Return type is a namespaced protobuf type:', returnTypeName);
-						context.report({
-							node,
-							messageId: "useProtobufMethodGeneric",
-							data: { code: returnText },
-							fix(fixer) {
-								// For namespaced types in return statements, we need to extract the full type name
-								const objectCode = sourceCode.getText(node)
-								// Since we may not know the exact type, we'll use the more generic namespaced type
-								return fixer.replaceText(node, `${returnTypeName}.create(${objectCode})`)
-							},
-						})
-						return
-					}
-				}
-
-				// Final fallback - if there are any protobuf imports and the function signature
-				// mentions a return type that matches one of the imported types
-				const functionText = functionNode ? sourceCode.getText(functionNode) : ""
-
-				for (const protoType of protobufImports) {
-					// Use more precise regex to match return type patterns specifically
-					// Rather than just checking if the type name appears anywhere in the signature
-					const returnTypeRegex = new RegExp(
-						// Match arrow function return type
-						`=>\\s*:?\\s*${protoType}\\b|` +
-							// Match function declaration return type
-							`\\)\\s*:?\\s*${protoType}\\b|` +
-							// Match Promise return type
-							`\\)\\s*:?\\s*Promise<\\s*${protoType}\\s*>|` +
-							// Match function type in variable declaration
-							`:\\s*\\(.*\\)\\s*=>\\s*${protoType}\\b`,
-					)
-
-					if (returnTypeRegex.test(functionText)) {
-						const returnText = sourceCode.getText(node.parent)
-						//console.log('🚨 VIOLATION: regex matched protobuf type:', functionText);
-						context.report({
-							node,
-							messageId: "useProtobufMethod",
-							data: {
-								typeName: protoType,
-								code: returnText,
-								objectContent: sourceCode.getText(node),
-							},
-							fix(fixer) {
-								// Replace the object literal with Type.create() call
-								return fixer.replaceText(node, `${protoType}.create(${sourceCode.getText(node)})`)
-							},
-						})
-						return
-					}
-				}
-				// Check for namespace imports too
-				for (const namespace of protobufNamespaceImports) {
-					// Similar to above, but for namespaced types
-					const namespaceReturnTypeRegex = new RegExp(
-						// Match arrow function return type
-						`=>\\s*:?\\s*${namespace}\\.\\w+\\b|` +
-							// Match function declaration return type
-							`\\)\\s*:?\\s*${namespace}\\.\\w+\\b|` +
-							// Match Promise return type
-							`\\)\\s*:?\\s*Promise<\\s*${namespace}\\.\\w+\\s*>|` +
-							// Match function type in variable declaration
-							`:\\s*\\(.*\\)\\s*=>\\s*${namespace}\\.\\w+\\b`,
-					)
-
-					if (namespaceReturnTypeRegex.test(functionText)) {
-						const returnText = sourceCode.getText(node.parent)
-						//console.log('🚨 VIOLATION: regex matched namespaced protobuf type:', functionText, "namespace:", namespace);
-						context.report({
-							node,
-							messageId: "useProtobufMethodGeneric",
-							data: { code: returnText },
-							fix(fixer) {
-								// For namespaced types based on function signature patterns
-								// Extract the namespace and type from the function text using more precise patterns
-								const match = functionText.match(
-									new RegExp(
-										// Match return type patterns more precisely
-										`\\)\\s*:?\\s*(${namespace}\\.[\\w]+)\\b|` + // Function declaration
-											`=>\\s*:?\\s*(${namespace}\\.[\\w]+)\\b|` + // Arrow function
-											`Promise<\\s*(${namespace}\\.[\\w]+)\\s*>`, // Promise wrapped
-									),
-								)
-								if (match) {
-									const fullType = match[1] || match[2]
-									return fixer.replaceText(node, `${fullType}.create(${sourceCode.getText(node)})`)
-								}
-								// Fallback - we can't determine the exact type, but we know it's from the namespace
-								// Use a namespace-based approach
-								return fixer.replaceText(node, `${namespace}.create(${sourceCode.getText(node)})`)
-							},
-						})
-						return
-					}
-				}
-			},
-
-			// Check function call arguments (more selective approach)
-			"CallExpression > ObjectExpression"(node) {
-				// Skip if this is inside a create/fromPartial call
-				if (safeObjectExpressions.has(node)) {
-					return
-				}
-
-				// We need to be more selective to avoid false positives
-				// Only warn if:
-				// 1. The function is called on a protobuf namespace
-				// 2. The call argument has a type annotation that matches a protobuf type
-				// 3. The call is to a function that we know takes a protobuf type
-
-				// Check if it's a call on a protobuf namespace
-				if (
-					node.parent.callee &&
-					node.parent.callee.type === "MemberExpression" &&
-					node.parent.callee.object.type === "Identifier"
-				) {
-					const namespace = node.parent.callee.object.name
-					if (protobufNamespaceImports.has(namespace)) {
-						const sourceCode = context.getSourceCode()
-						const callText = sourceCode.getText(node.parent)
-						//console.log('🚨 VIOLATION: Check function call arguments object literal:', callText);
-						context.report({
-							node,
-							messageId: "useProtobufMethodGeneric",
-							data: { code: callText },
-							fix(fixer) {
-								// For calls on a protobuf namespace
-								const memberExpr = node.parent.callee
-								// Try to determine if this is calling a method that expects a specific type
-								const methodName = memberExpr.property.name
-
-								// If method name looks like 'create' + Type, we can infer the type
-								const possibleTypeName = methodName.replace(/^create/, "")
-
-								// Check if namespace has a type with this name
-								// Since we can't directly check at lint time, we'll use the namespace + inferred type
-								if (possibleTypeName && possibleTypeName !== methodName) {
-									return fixer.replaceText(
-										node,
-										`${namespace}.${possibleTypeName}.create(${sourceCode.getText(node)})`,
-									)
-								}
-
-								// Fallback - use a more generic approach with namespace
-								return fixer.replaceText(node, `${namespace}.create(${sourceCode.getText(node)})`)
-							},
-						})
-						return
-					}
-				}
-
-				// For regular function calls with object literals, check if there are protobuf imports
-				// and if the function might expect a protobuf type
-				if (node.parent.callee) {
-					// This is a more permissive check to catch cases like processContent({ ... })
-					// which might be passing a protobuf type
-					const sourceCode = context.getSourceCode()
-					const scope = sourceCode.getScope(node)
-
-					// Try to find the function definition
-					if (node.parent.callee.type === "Identifier") {
-						const functionName = node.parent.callee.name
-						const variable = scope.variables.find((v) => v.name === functionName)
-
-						// If we found the function and it has parameter type annotations
-						// that match protobuf types, flag it
-						if (variable && variable.defs.length > 0) {
-							const def = variable.defs[0]
-							if (def.node.params && node.parent.arguments.indexOf(node) < def.node.params.length) {
-								const param = def.node.params[node.parent.arguments.indexOf(node)]
-								if (param.typeAnnotation) {
-									const typeName = getTypeName(param.typeAnnotation.typeAnnotation)
-									if (
-										typeName &&
-										(protobufImports.has(typeName) ||
-											isNamespacedProtobufType(protobufNamespaceImports, typeName))
-									) {
-										const callText = sourceCode.getText(node.parent)
-										//console.log('🚨 VIOLATION: Function call arguments object literal:', callText);
-										context.report({
-											node,
-											messageId: "useProtobufMethodGeneric",
-											data: { code: callText },
-											fix(fixer) {
-												// For function calls with protobuf type parameters
-												return fixer.replaceText(node, `${typeName}.create(${sourceCode.getText(node)})`)
-											},
-										})
-										return
-									}
-								}
-							}
-						}
-					}
-				}
-			},
-		}
-	},
-})
-
-// Helper functions
-function getTypeName(typeAnnotation) {
-	if (!typeAnnotation) {
-		return null
-	}
-
-	if (typeAnnotation.type === "TSTypeReference") {
-		if (typeAnnotation.typeName.type === "Identifier") {
-			return typeAnnotation.typeName.name
-		} else if (typeAnnotation.typeName.type === "TSQualifiedName") {
-			// Handle namespaced types like proto.MyRequest
-			return `${typeAnnotation.typeName.left.name}.${typeAnnotation.typeName.right.name}`
-		}
-	}
-	return null
-}
-
-function matchesProtobufPackage(packageName, protobufPackages) {
-	return protobufPackages.some((protobufPackage) => {
-		// Remove leading and trailing @ and / from protobufPackage
-		const cleanedPackage = protobufPackage.replace(/^[@\/]/, "").replace(/[\/]$/, "")
-		const pattern = new RegExp(`(.*[@/]|)${escapeRegex(cleanedPackage)}[/].*`)
-		return pattern.test(packageName)
-	})
-}
-
-// Helper function to escape special regex characters
-function escapeRegex(string) {
-	return string.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")
-}
-
-// Helper to extract function return type more reliably
-function getFunctionReturnType(functionNode, sourceCode) {
-	// 1. Check explicit return type annotation
-	if (functionNode.returnType) {
-		return getTypeName(functionNode.returnType.typeAnnotation)
-	}
-
-	// 2. For variable declarations like const foo: (arg: Type) => ReturnType = ...
-	if (functionNode.parent && functionNode.parent.type === "VariableDeclarator") {
-		const declarator = functionNode.parent
-		if (declarator.id && declarator.id.typeAnnotation) {
-			const typeAnnotation = declarator.id.typeAnnotation.typeAnnotation
-
-			// Handle function type annotations
-			if (typeAnnotation.type === "TSFunctionType" && typeAnnotation.typeAnnotation) {
-				return getTypeName(typeAnnotation.typeAnnotation)
-			}
-
-			// Handle type references to function types
-			if (typeAnnotation.type === "TSTypeReference") {
-				// This might be a type like Promise<ReturnType>
-				if (
-					typeAnnotation.typeName.name === "Promise" &&
-					typeAnnotation.typeParameters &&
-					typeAnnotation.typeParameters.params.length > 0
-				) {
-					return getTypeName(typeAnnotation.typeParameters.params[0])
-				}
-			}
-		}
-	}
-
-	// 3. For class methods, check if it's part of an interface implementation
-	if (
-		functionNode.parent &&
-		functionNode.parent.type === "MethodDefinition" &&
-		functionNode.parent.parent &&
-		functionNode.parent.parent.type === "ClassBody"
-	) {
-		const className = getEnclosingClassName(functionNode)
-		const methodName = functionNode.parent.key.name
-
-		if (className && methodName) {
-			// Look for interface declarations in the scope
-			const scope = sourceCode.getScope(functionNode)
-			// This would require more complex scope analysis which is limited in ESLint
-			// For now, we'll return null and rely on other methods
-		}
-	}
-
-	return null
-}
-
-// Helper to get the class name for a method
-function getEnclosingClassName(node) {
-	let current = node.parent
-	while (current) {
-		if (current.type === "ClassDeclaration" && current.id) {
-			return current.id.name
-		}
-		current = current.parent
-	}
-	return null
-}
-function isNamespacedProtobufType(protobufNamespaceImports, typeName) {
-	if (!typeName.includes(".")) {
-		return false
-	}
-
-	const namespace = typeName.split(".")[0]
-	return protobufNamespaceImports.has(namespace)
-}
-
-function findParentFunction(node) {
-	let current = node.parent
-	while (current) {
-		if (
-			current.type === "FunctionDeclaration" ||
-			current.type === "FunctionExpression" ||
-			current.type === "ArrowFunctionExpression"
-		) {
-			return current
-		}
-		current = current.parent
-	}
-	return null
-}

eslint-rules/no-vscode-postmessage.js

@@ -1,61 +0,0 @@
-const { ESLintUtils } = require("@typescript-eslint/utils")
-const path = require("path")
-
-const createRule = ESLintUtils.RuleCreator((name) => `https://cline.bot/eslint-rules/${name}`)
-
-module.exports = createRule({
-	name: "no-vscode-postmessage",
-	meta: {
-		type: "problem",
-		docs: {
-			description: "Ban vscode.postMessage() calls in favor of gRPC service clients, except in grpc-client-base.ts",
-			recommended: "error",
-		},
-		messages: {
-			useGrpcClient:
-				"Use gRPC service clients instead of vscode.postMessage().\n" +
-				"Example: AccountServiceClient.methodName(RequestType.create({...})) instead of vscode.postMessage({type: '...'}).\n" +
-				"Found: {{code}}",
-		},
-		schema: [],
-	},
-	defaultOptions: [],
-
-	create(context) {
-		// Check if current file is grpc-client-base.ts (exception case)
-		const filename = context.filename
-		const isGrpcClientBase = path.basename(filename) === "grpc-client-base.ts"
-
-		return {
-			// Detect vscode.postMessage calls
-			"CallExpression[callee.type='MemberExpression']"(node) {
-				// Skip if this is grpc-client-base.ts
-				if (isGrpcClientBase) {
-					return
-				}
-
-				const callee = node.callee
-
-				// Check for vscode.postMessage pattern
-				if (
-					callee.object &&
-					callee.object.type === "Identifier" &&
-					callee.object.name === "vscode" &&
-					callee.property &&
-					callee.property.name === "postMessage"
-				) {
-					const sourceCode = context.sourceCode
-					const callText = sourceCode.getText(node).trim()
-
-					context.report({
-						node,
-						messageId: "useGrpcClient",
-						data: {
-							code: callText,
-						},
-					})
-				}
-			},
-		}
-	},
-})

eslint-rules/package.json

@@ -1,31 +0,0 @@
-{
-	"name": "eslint-plugin-eslint-rules",
-	"version": "1.0.0",
-	"description": "Custom ESLint rules for Cline",
-	"main": "index.js",
-	"scripts": {
-		"test": "mocha --no-config --require ts-node/register __tests__/**/*.test.ts"
-	},
-	"keywords": [
-		"eslint",
-		"eslintplugin"
-	],
-	"author": "Cline Bot Inc.",
-	"license": "Apache-2.0",
-	"dependencies": {
-		"@typescript-eslint/utils": "^8.33.0"
-	},
-	"devDependencies": {
-		"@types/eslint": "^8.0.0",
-		"@types/mocha": "^10.0.7",
-		"@types/node": "^20.0.0",
-		"@typescript-eslint/parser": "^7.14.1",
-		"eslint": "^8.57.0",
-		"mocha": "^10.0.0",
-		"ts-node": "^10.9.2",
-		"typescript": "^5.4.5"
-	},
-	"peerDependencies": {
-		"eslint": ">=8.0.0"
-	}
-}

eslint-rules/tsconfig.json

@@ -1,16 +0,0 @@
-{
-	"compilerOptions": {
-		"target": "es2020",
-		"module": "commonjs",
-		"moduleResolution": "node",
-		"esModuleInterop": true,
-		"strict": true,
-		"skipLibCheck": true,
-		"forceConsistentCasingInFileNames": true,
-		"outDir": "dist",
-		"resolveJsonModule": true,
-		"declaration": true
-	},
-	"include": ["**/*.ts", "**/*.js", "**/*.tsx", "__tests__/**/*"],
-	"exclude": ["node_modules", "dist"]
-}

evals/.gitignore

@@ -1,24 +1,3 @@
 repositories
 
-results/evals.db
-
-diff-edits/cases/
-diff-edits/results/
-
-# Environment variables
-.env
-
-# backwards compatible
-diff_editing/test_cases/
-diff_editing/test_outputs/
-
-*.db
-*.db-wal
-*.db-shm
-
-.cache
-
-# Python bytecode cache
-*__pycache__/
-
-diff-edits/cases.zip
+results/evals.db

evals/README.md

@@ -17,7 +17,6 @@ The evaluation system consists of two main components:
 
 1. **Test Server**: Enhanced HTTP server in `src/services/test/TestServer.ts` that provides detailed task results
 2. **CLI Tool**: Command-line interface in `evals/cli/` for orchestrating evaluations
-3. **Diff Edit Benchmark**: Separate command using the CLI tool that runs a comprehensive diff editing benchmark suite on real world cases, along with a streamlit dashboard displaying the results. For more details, see the [Diff Edit Benchmark README](./diff-edits/README.md). Make sure you add a `evals/diff-edits/cases` folder with all the conversation jsons. 
 
 ## Directory Structure
 
@@ -41,12 +40,6 @@ cline-repo/
 │   │   │   └── utils/                # Utility functions
 │   │   ├── package.json
 │   │   └── tsconfig.json
-│   ├── diff-edits/                   # Diff editing evaluation suite
-│   │   ├── cases/                    # Test case JSON files
-│   │   ├── results/                  # Evaluation results
-│   │   ├── diff-apply/               # Diff application logic
-│   │   ├── parsing/                  # Assistant message parsing
-│   │   └── prompts/                  # System prompts
 │   ├── repositories/                 # Cloned benchmark repositories
 │   │   ├── exercism/                 # Modified Exercism (from pashpashpash/evals)
 │   │   ├── swe-bench/                # SWE-Bench repository
@@ -155,192 +148,6 @@ Freelance-style programming tasks from the SWELancer benchmark.
 
 Multi-file software engineering tasks from the Multi-SWE-Bench repository.
 
-## Diff Edit Evaluations
-
-The Cline Evaluation System includes a specialized suite for evaluating how well models can make precise edits to files using the `replace_in_file` tool.
-
-### Overview
-
-Diff edit evaluations test a model's ability to:
-
-1. Understand file content and identify specific sections to modify
-2. Generate correct SEARCH/REPLACE blocks for targeted edits
-3. Successfully apply changes without introducing errors
-
-### Directory Structure
-
-```
-diff-edits/
-├── cases/                  # Test case JSON files
-├── results/                # Evaluation results
-├── ClineWrapper.ts         # Wrapper for model interaction
-├── TestRunner.ts           # Main test execution logic
-├── types.ts                # Type definitions
-├── diff-apply/             # Diff application logic
-├── parsing/                # Assistant message parsing
-└── prompts/                # System prompts
-```
-
-### Creating Test Cases
-
-Test cases are defined as JSON files in the `diff-edits/cases/` directory. Each test case should include:
-
-```json
-{
-  "test_id": "example_test_1",
-  "messages": [
-    {
-      "role": "user",
-      "text": "Please fix the bug in this code...",
-      "images": []
-    },
-    {
-      "role": "assistant",
-      "text": "I'll help you fix that bug..."
-    }
-  ],
-  "file_contents": "// Original file content here\nfunction example() {\n  // Code with bug\n}",
-  "file_path": "src/example.js",
-  "system_prompt_details": {
-    "mcp_string": "",
-    "cwd_value": "/path/to/working/directory",
-    "browser_use": false,
-    "width": 900,
-    "height": 600,
-    "os_value": "macOS",
-    "shell_value": "/bin/zsh",
-    "home_value": "/Users/username",
-    "user_custom_instructions": ""
-  },
-  "original_diff_edit_tool_call_message": ""
-}
-```
-
-### Running Diff Edit Evaluations
-
-#### Single Model Evaluation
-
-```bash
-cd evals/cli
-node dist/index.js run-diff-eval --model-ids "anthropic/claude-3-5-sonnet-20241022"
-```
-
-#### Multi-Model Evaluation
-
-Compare multiple models in a single evaluation run:
-
-```bash
-# Compare Claude and Grok models
-node dist/index.js run-diff-eval \
-  --model-ids "anthropic/claude-3-5-sonnet-20241022,x-ai/grok-beta" \
-  --max-cases 10 \
-  --valid-attempts-per-case 3 \
-  --verbose
-
-# Compare multiple Claude variants
-node dist/index.js run-diff-eval \
-  --model-ids "anthropic/claude-3-5-sonnet-20241022,anthropic/claude-3-5-haiku-20241022,anthropic/claude-3-opus-20240229" \
-  --max-cases 5 \
-  --valid-attempts-per-case 2 \
-  --parallel
-```
-
-#### Options
-
-- `--model-ids`: Comma-separated list of model IDs to evaluate (required)
-- `--system-prompt-name`: System prompt to use (default: "basicSystemPrompt")
-- `--valid-attempts-per-case`: Number of attempts per test case per model (default: 1)
-- `--max-cases`: Maximum number of test cases to run (default: all available)
-- `--parsing-function`: Function to parse assistant messages (default: "parseAssistantMessageV2")
-- `--diff-edit-function`: Function to apply diffs (default: "constructNewFileContentV2")
-- `--test-path`: Path to test cases (default: diff-edits/cases)
-- `--thinking-budget`: Tokens allocated for thinking (default: 0)
-- `--parallel`: Run tests in parallel (flag)
-- `--replay`: Use pre-recorded LLM output (flag)
-- `--verbose`: Enable detailed logging (flag)
-
-#### Examples
-
-```bash
-# Quick test with 2 models, 4 cases, 2 attempts each
-node dist/index.js run-diff-eval \
-  --model-ids "anthropic/claude-3-5-sonnet-20241022,x-ai/grok-beta" \
-  --max-cases 4 \
-  --valid-attempts-per-case 2 \
-  --verbose
-
-# Comprehensive evaluation with parallel execution
-node dist/index.js run-diff-eval \
-  --model-ids "anthropic/claude-3-5-sonnet-20241022,anthropic/claude-3-5-haiku-20241022" \
-  --system-prompt-name claude4SystemPrompt \
-  --valid-attempts-per-case 5 \
-  --max-cases 20 \
-  --parallel \
-  --verbose
-```
-
-### Database Storage & Analytics
-
-All evaluation results are automatically stored in a SQLite database (`diff-edits/evals.db`) for advanced analytics and comparison. The database includes:
-
-- **System Prompts**: Versioned system prompt content with hashing for deduplication
-- **Processing Functions**: Versioned parsing and diff-edit function configurations
-- **Files**: Original and edited file content with content-based hashing
-- **Runs**: Evaluation run metadata and configuration
-- **Cases**: Individual test case information with context tokens
-- **Results**: Detailed results with timing, cost, and success metrics
-
-### Interactive Dashboard
-
-Launch the Streamlit dashboard to visualize and analyze evaluation results:
-
-```bash
-cd diff-edits/dashboard
-streamlit run app.py
-```
-
-The dashboard provides:
-
-- **Model Performance Comparison**: Side-by-side comparison of success rates, latency, and costs
-- **Interactive Charts**: Success rate trends, latency vs cost analysis, and performance metrics
-- **Detailed Drill-Down**: Individual result analysis with file content viewing
-- **Run Selection**: Browse and compare different evaluation runs
-- **Real-time Updates**: Automatically refreshes with new evaluation data
-
-#### Dashboard Features
-
-1. **Hero Section**: Overview of current run with key metrics
-2. **Model Cards**: Performance cards with grades and detailed metrics
-3. **Comparison Charts**: Interactive Plotly charts for visual analysis
-4. **Result Explorer**: Detailed view of individual test results including:
-   - Original and edited file content
-   - Raw model output
-   - Parsed tool calls
-   - Timing and cost metrics
-   - Error analysis
-
-#### Quick Start Dashboard
-
-```bash
-# Run a quick evaluation
-node cli/dist/index.js run-diff-eval \
-  --model-ids "anthropic/claude-3-5-sonnet-20241022,x-ai/grok-beta" \
-  --max-cases 4 \
-  --valid-attempts-per-case 2 \
-  --verbose
-
-# Launch dashboard to view results
-cd diff-edits/dashboard && streamlit run app.py
-```
-
-### Legacy Results
-
-For backward compatibility, results are also saved as JSON files in the `diff-edits/results/` directory. The JSON results include:
-- Success/failure status
-- Extracted tool calls
-- Diff edit content
-- Token usage and cost metrics
-
 ## Metrics
 
 The evaluation system collects the following metrics:

evals/cli/package.json

@@ -0,0 +1,39 @@
+{
+	"name": "cline-evaluation-cli",
+	"version": "0.1.0",
+	"description": "CLI tool for orchestrating Cline evaluations across multiple benchmarks",
+	"main": "dist/index.js",
+	"scripts": {
+		"build": "tsc",
+		"start": "node dist/index.js",
+		"dev": "ts-node src/index.ts",
+		"test": "echo \"Error: no test specified\" && exit 1"
+	},
+	"keywords": [
+		"cline",
+		"evaluation",
+		"benchmark"
+	],
+	"author": "",
+	"license": "MIT",
+	"dependencies": {
+		"better-sqlite3": "^8.0.0",
+		"chalk": "^4.1.2",
+		"commander": "^9.4.1",
+		"execa": "^5.1.1",
+		"node-fetch": "^2.7.0",
+		"ora": "^5.4.1",
+		"sqlite": "^4.1.2",
+		"uuid": "^9.0.0",
+		"yargs": "^17.6.2"
+	},
+	"devDependencies": {
+		"@types/better-sqlite3": "^7.6.3",
+		"@types/node": "^18.11.18",
+		"@types/node-fetch": "^2.6.12",
+		"@types/uuid": "^9.0.0",
+		"@types/yargs": "^17.0.19",
+		"ts-node": "^10.9.1",
+		"typescript": "^4.9.4"
+	}
+}

evals/cli/src/commands/runDiffEval.ts

@@ -1,94 +0,0 @@
-import execa from "execa"
-import chalk from "chalk"
-import path from "path"
-
-interface RunDiffEvalOptions {
-	modelIds: string
-	systemPromptName: string
-	validAttemptsPerCase: number
-	parsingFunction: string
-	diffEditFunction: string
-	thinkingBudget: number
-	parallel: boolean
-	verbose: boolean
-	testPath: string
-	outputPath: string
-	replay: boolean
-	replayRunId?: string
-	diffApplyFile?: string
-	maxCases?: number
-}
-
-export async function runDiffEvalHandler(options: RunDiffEvalOptions) {
-	console.log(chalk.blue("Starting diff editing evaluation..."))
-
-	// Resolve the path to the TestRunner.ts script relative to the current file
-	const scriptPath = path.resolve(__dirname, "../../../diff-edits/TestRunner.ts")
-
-	// Construct the arguments array for the execa call
-	const args = [
-		"--model-ids",
-		options.modelIds,
-		"--system-prompt-name",
-		options.systemPromptName,
-		"--valid-attempts-per-case",
-		String(options.validAttemptsPerCase),
-		"--parsing-function",
-		options.parsingFunction,
-		"--diff-edit-function",
-		options.diffEditFunction,
-	]
-
-	// Conditionally add the optional arguments
-	if (options.testPath) {
-		args.push("--test-path", options.testPath)
-	}
-	if (options.outputPath) {
-		args.push("--output-path", options.outputPath)
-	}
-	if (options.thinkingBudget > 0) {
-		args.push("--thinking-budget", String(options.thinkingBudget))
-	}
-
-	if (options.parallel) {
-		args.push("--parallel")
-	}
-
-	if (options.replay) {
-		args.push("--replay")
-	}
-
-	if (options.replayRunId) {
-		args.push("--replay-run-id", options.replayRunId)
-	}
-
-	if (options.diffApplyFile) {
-		args.push("--diff-apply-file", options.diffApplyFile)
-	}
-
-	if (options.verbose) {
-		args.push("--verbose")
-	}
-
-	if (options.maxCases) {
-		args.push("--max-cases", String(options.maxCases))
-	}
-
-	try {
-		console.log(chalk.gray(`Executing: npx tsx ${scriptPath} ${args.join(" ")}`))
-
-		// Execute the script as a child process
-		// We use 'inherit' to stream the stdout/stderr directly to the user's terminal
-		const subprocess = execa("npx", ["tsx", "--tsconfig", path.resolve(__dirname, "../../../tsconfig.json"), scriptPath, ...args], {
-			stdio: "inherit",
-		})
-
-		await subprocess
-
-		console.log(chalk.green("Diff editing evaluation completed successfully."))
-	} catch (error) {
-		console.error(chalk.red("An error occurred during the diff editing evaluation."))
-		// The 'inherit' stdio will have already printed the error details from the script
-		process.exit(1)
-	}
-}

evals/cli/src/index.ts

@@ -5,7 +5,6 @@ import { setupHandler } from "./commands/setup"
 import { runHandler } from "./commands/run"
 import { reportHandler } from "./commands/report"
 import { evalsEnvHandler } from "./commands/evals-env"
-import { runDiffEvalHandler } from "./commands/runDiffEval"
 
 // Create the CLI program
 const program = new Command()
@@ -78,39 +77,6 @@ program
 		}
 	})
 
-// Run-diff-eval command
-program
-	.command("run-diff-eval")
-	.description("Run the diff editing evaluation suite")
-	.option("--test-path <path>", "Path to the directory containing test case JSON files")
-	.option("--output-path <path>", "Path to the directory to save the test output JSON files")
-	.option("--model-ids <model_ids>", "Comma-separated list of model IDs to test")
-	.option("--system-prompt-name <name>", "The name of the system prompt to use", "basicSystemPrompt")
-	.option("-n, --valid-attempts-per-case <number>", "Number of valid attempts per test case per model (will retry until this many valid attempts are collected)", "1")
-	.option("--max-cases <number>", "Maximum number of test cases to run (limits total cases loaded)")
-	.option("--parsing-function <name>", "The parsing function to use", "parseAssistantMessageV2")
-	.option("--diff-edit-function <name>", "The diff editing function to use", "constructNewFileContentV2")
-	.option("--thinking-budget <tokens>", "Set the thinking tokens budget", "0")
-	.option("--parallel", "Run tests in parallel", false)
-	.option("--replay", "Run evaluation from a pre-recorded LLM output, skipping the API call", false)
-	.option("--replay-run-id <run_id>", "The ID of the run to replay from the database")
-	.option("--diff-apply-file <filename>", "The name of the diff apply file to use for the replay")
-	.option("-v, --verbose", "Enable verbose logging", false)
-	.action(async (options) => {
-		try {
-			const fullOptions = {
-				...options,
-				validAttemptsPerCase: parseInt(options.validAttemptsPerCase, 10),
-				thinkingBudget: parseInt(options.thinkingBudget, 10),
-				maxCases: options.maxCases ? parseInt(options.maxCases, 10) : undefined,
-			}
-			await runDiffEvalHandler(fullOptions)
-		} catch (error) {
-			console.error(chalk.red(`Error during diff eval run: ${error instanceof Error ? error.message : String(error)}`))
-			process.exit(1)
-		}
-	})
-
 // Parse command line arguments
 program.parse(process.argv)
 

evals/diff-edits/ClineWrapper.ts

@@ -1,333 +0,0 @@
-import { OpenRouterHandler } from "../../src/api/providers/openrouter"
-import { ApiHandlerOptions } from "../../src/shared/api"
-import { Anthropic } from "@anthropic-ai/sdk"
-
-import {
-	parseAssistantMessageV1,
-	parseAssistantMessageV2,
-	parseAssistantMessageV3,
-	AssistantMessageContent,
-} from "./parsing/parse-assistant-message-06-06-25" // "../../src/core/assistant-message"
-import { constructNewFileContent as constructNewFileContentV1, constructNewFileContentV2 } from "./diff-apply/diff-06-06-25"
-import { constructNewFileContent as constructNewFileContentV2_1 } from "./diff-apply/diff-06-23-25"
-import { constructNewFileContent as constructNewFileContentV3 } from "../../src/core/assistant-message/diff" // this defaults to the new v1 when called
-
-type ParseAssistantMessageFn = (message: string) => AssistantMessageContent[]
-type ConstructNewFileContentFn = (diff: string, original: string, strict: boolean) => Promise<string>
-
-const parsingFunctions: Record<string, ParseAssistantMessageFn> = {
-	parseAssistantMessageV1: parseAssistantMessageV1,
-	parseAssistantMessageV2: parseAssistantMessageV2,
-	parseAssistantMessageV3: parseAssistantMessageV3,
-}
-
-const diffEditingFunctions: Record<string, ConstructNewFileContentFn> = {
-	"diff-06-06-25": constructNewFileContentV2,
-	"diff-06-23-25": constructNewFileContentV2_1,
-	constructNewFileContentV1: constructNewFileContentV1,
-	constructNewFileContentV2: constructNewFileContentV2,
-	constructNewFileContentV3: constructNewFileContentV3, // position invariant diff
-}
-
-import { TestInput, TestResult, ExtractedToolCall } from "./types"
-export { TestInput, TestResult, ExtractedToolCall }
-
-interface StreamResult {
-	assistantMessage: string
-	reasoningMessage: string
-	usage: {
-		inputTokens: number
-		outputTokens: number
-		cacheWriteTokens: number
-		cacheReadTokens: number
-		totalCost: number
-	}
-	timing?: {
-		timeToFirstTokenMs: number
-		timeToFirstEditMs?: number
-		totalRoundTripMs: number
-	}
-}
-
-/**
- * Process the stream and return full response with timing data
- */
-async function processStream(
-	handler: OpenRouterHandler,
-	systemPrompt: string,
-	messages: Anthropic.Messages.MessageParam[],
-): Promise<StreamResult> {
-	const startTime = Date.now()
-	const stream = handler.createMessage(systemPrompt, messages)
-
-	let assistantMessage = ""
-	let reasoningMessage = ""
-	let inputTokens = 0
-	let outputTokens = 0
-	let cacheWriteTokens = 0
-	let cacheReadTokens = 0
-	let totalCost = 0
-	
-	// Timing tracking
-	let timeToFirstTokenMs: number | null = null
-	let timeToFirstEditMs: number | null = null
-
-	for await (const chunk of stream) {
-		if (!chunk) {
-			continue
-		}
-
-		// Capture time to first token (any chunk type)
-		if (timeToFirstTokenMs === null) {
-			timeToFirstTokenMs = Date.now() - startTime
-		}
-
-		switch (chunk.type) {
-			case "usage":
-				inputTokens += chunk.inputTokens
-				outputTokens += chunk.outputTokens
-				cacheWriteTokens += chunk.cacheWriteTokens ?? 0
-				cacheReadTokens += chunk.cacheReadTokens ?? 0
-				if (chunk.totalCost) {
-					totalCost = chunk.totalCost
-				}
-				break
-			case "reasoning":
-				reasoningMessage += chunk.reasoning
-				break
-			case "text":
-				assistantMessage += chunk.text
-				
-				// Try to detect first tool call by parsing accumulated message
-				if (timeToFirstEditMs === null) {
-					try {
-						const parsed = parseAssistantMessageV2(assistantMessage)
-						const hasToolCall = parsed.some(block => block.type === "tool_use")
-						if (hasToolCall) {
-							timeToFirstEditMs = Date.now() - startTime
-						}
-					} catch {
-						// Parsing failed, continue accumulating
-					}
-				}
-				break
-		}
-	}
-
-	const totalRoundTripMs = Date.now() - startTime
-
-	return {
-		assistantMessage,
-		reasoningMessage,
-		usage: {
-			inputTokens,
-			outputTokens,
-			cacheWriteTokens,
-			cacheReadTokens,
-			totalCost,
-		},
-		timing: {
-			timeToFirstTokenMs: timeToFirstTokenMs || 0,
-			timeToFirstEditMs: timeToFirstEditMs || undefined,
-			totalRoundTripMs,
-		},
-	}
-}
-
-/**
- * Main evaluation function:
- * 1. create and process stream
- * 2. extract any tool calls from the stream
- * 3. if no diff edit, considered a failure (or rerun) - otherwise attempt to apply the diff edit
- */
-export async function runSingleEvaluation(input: TestInput): Promise<TestResult> {
-	try {
-		// Extract parameters
-		const {
-			apiKey,
-			systemPrompt,
-			messages,
-			modelId,
-			originalFile,
-			originalFilePath,
-			parsingFunction,
-			diffEditFunction,
-			thinkingBudgetTokens,
-			originalDiffEditToolCallMessage,
-			diffApplyFile,
-		} = input
-
-		const requiredParams = {
-			systemPrompt,
-			messages,
-			modelId,
-			originalFile,
-			originalFilePath,
-			parsingFunction,
-			diffEditFunction,
-		}
-
-		const missingParams = Object.entries(requiredParams)
-			.filter(([, value]) => !value)
-			.map(([key]) => key)
-
-		if (missingParams.length > 0) {
-			return {
-				success: false,
-				error: "missing_required_parameters",
-				errorString: `Missing required parameters: ${missingParams.join(", ")}`,
-			}
-		}
-
-		const parseAssistantMessage = parsingFunctions[parsingFunction]
-		const constructNewFileContent = diffEditingFunctions[diffApplyFile || diffEditFunction]
-
-		if (!parseAssistantMessage || !constructNewFileContent) {
-			return {
-				success: false,
-				error: "invalid_functions",
-			}
-		}
-
-		const options: ApiHandlerOptions = {
-			openRouterApiKey: apiKey,
-			openRouterModelId: modelId,
-			thinkingBudgetTokens: thinkingBudgetTokens,
-			openRouterModelInfo: {
-				maxTokens: 10_000,
-				contextWindow: 1_000_000,
-				supportsImages: true,
-				supportsPromptCache: true, // may need to turn this on
-				inputPrice: 0,
-				outputPrice: 0,
-			},
-		}
-
-		// Get the output of streaming output of this llm call
-		let streamResult: StreamResult
-		if (originalDiffEditToolCallMessage !== undefined) {
-			// Replay mode: mock the stream result
-			streamResult = {
-				assistantMessage: originalDiffEditToolCallMessage,
-				reasoningMessage: "",
-				usage: { inputTokens: 0, outputTokens: 0, cacheWriteTokens: 0, cacheReadTokens: 0, totalCost: 0 },
-			}
-		} else {
-			// Live mode: existing API call logic
-			try {
-				const openRouterHandler = new OpenRouterHandler(options)
-				streamResult = await processStream(openRouterHandler, systemPrompt, messages)
-			} catch (error: any) {
-				return {
-					success: false,
-					error: "llm_stream_error",
-					errorString: error.message || error.toString(),
-				}
-			}
-		}
-
-		// process the assistant message into its constituent tool calls & text blocks
-		const assistantContentBlocks: AssistantMessageContent[] = parseAssistantMessage(streamResult.assistantMessage)
-
-		const detectedToolCalls: ExtractedToolCall[] = []
-
-		for (const block of assistantContentBlocks) {
-			if (block.type === "tool_use") {
-				detectedToolCalls.push({
-					name: block.name,
-					input: block.params,
-				})
-			}
-		}
-
-		// check if there are any tool calls, if there are none then its a clear error
-		if (detectedToolCalls.length === 0) {
-			return {
-				success: false,
-				streamResult: streamResult,
-				toolCalls: detectedToolCalls,
-				error: "no_tool_calls",
-			}
-		}
-
-		// check that there is exactly one tool call, otherwise an error
-		if (detectedToolCalls.length > 1) {
-			return {
-				success: false,
-				streamResult: streamResult,
-				toolCalls: detectedToolCalls,
-				error: "multi_tool_calls",
-			}
-		}
-
-		// check that the tool call is diff edit tool call
-		if (detectedToolCalls[0].name !== "replace_in_file") {
-			return {
-				success: false,
-				streamResult: streamResult,
-				toolCalls: detectedToolCalls,
-				error: "wrong_tool_call",
-			}
-		}
-
-		const toolCall = detectedToolCalls[0]
-		const diffToolPath = toolCall.input.path
-		const diffToolContent = toolCall.input.diff
-
-		if (!diffToolPath || !diffToolContent) {
-			return {
-				success: false,
-				streamResult: streamResult,
-				toolCalls: detectedToolCalls,
-				error: "tool_call_params_undefined",
-			}
-		}
-
-		// check that we are editing the correct file path
-		console.log(`Expected file path: "${originalFilePath}"`);
-		console.log(`Actual file path used: "${diffToolPath}"`);
-		if (diffToolPath !== originalFilePath) {
-			console.log(`❌ File path mismatch detected!`);
-			// Enhanced logging:
-			if (streamResult?.assistantMessage) {
-				console.log(`   Full model output (assistantMessage):`);
-				console.log(`   -----------------------------------------`);
-				console.log(`   ${streamResult.assistantMessage}`);
-				console.log(`   -----------------------------------------`);
-			}
-			if (toolCall) {
-				console.log(`   Parsed tool call that caused mismatch:`);
-				console.log(`   ${JSON.stringify(toolCall, null, 2)}`);
-				console.log(`   -----------------------------------------`);
-			}
-			return {
-				success: false,
-				streamResult: streamResult,
-				toolCalls: detectedToolCalls,
-				error: "wrong_file_edited",
-			}
-		}
-
-		// checking if the diff edit succeeds, if it failed it will throw an error
-		let diffSuccess = true
-		try {
-			await constructNewFileContent(diffToolContent, originalFile, true)
-		} catch (error: any) {
-			diffSuccess = false
-		}
-
-		return {
-			success: true,
-			streamResult: streamResult,
-			toolCalls: detectedToolCalls,
-			diffEdit: diffToolContent,
-			diffEditSuccess: diffSuccess,
-		}
-	} catch (error: any) {
-		return {
-			success: false,
-			error: "other_error",
-			errorString: error.message || error.toString(),
-		}
-	}
-}

evals/diff-edits/README.md

@@ -1,84 +0,0 @@
-# A Note on Cline's Diff Evaluation Setup
-
-Hey there, this note explains what we're doing with Cline's diff evaluation (evals) system. It's all about checking how well various AI models (which users connect to Cline via their own API keys), prompts, and diffing tools can handle file changes.
-
-## What We're Trying to Figure Out
-
-The main idea here is to figure out which AI models (configured by users) are best at making `replace_in_file` tool calls that work correctly. This helps us understand model capabilities and also speeds up our own experiments with prompts and diffing algorithms to make Cline better over time. We want to know a few key things.
-
-First, can the model create diffs, which are just sets of SEARCH and REPLACE blocks, that apply cleanly to a file? This is what we call `diffEditSuccess`.
-
-Second, how do different LLMs, like Claude or Grok, stack up against each other when they try to make these diff edits? We use a standard set of real-world test cases for this.
-
-Third, do different system prompts, say our `basicSystemPrompt` versus the `claude4SystemPrompt`, change how well a model does at diff editing?
-
-Fourth, we're also looking at different ways to apply the diffs themselves. We have a few algorithms like `constructNewFileContentV1`, `V2`, and `V3`, and we want to see which ones are more robust when fed model-generated diffs.
-
-Fifth, we track how fast the model starts making an edit. The `timeToFirstEditMs` metric gives us a hint about how quickly a user would see changes happening in their editor.
-
-And finally, we keep an eye on how many tokens are used and what it costs for each model and each try. This helps us compare how efficient they are.
-
-Right now, these evals are mostly about whether the diff *applies* correctly. That means, do the SEARCH blocks find a match, and can the REPLACE blocks be put in without an error? We're not yet deeply analyzing if the change is valid code or matches what the user *wanted* semantically. That's a problem for another day, and will require a lot more scaffolding.
-
-## How We Run These Tests
-
-Two prerequisites:
-
-1. Make sure you have an `evals/.env` file with `OPENROUTER_API_KEY=<your-openrouter-key>` 
-
-2. Make sure you add a `evals/diff-edits/cases` folder with all the conversation jsons prior to running this. 
-
-
-Our testing strategy is based on replaying situations from actual user sessions where diff edits were tried.
-
-It starts with our test cases. Each one is a JSON file in `./cases` that has the conversation history that led to a diff edit, the original file content and its path, and the info needed to rebuild the system prompt from that original session.
-
-Then, for every test run, we set up a specific configuration. This includes which LLM we're testing, which system prompt it gets, which function we use to parse the model's raw output, and which function we use to actually apply the diff. Here's the command I've been using:
-
-```bash
-npm run diff-eval -- --model-ids "anthropic/claude-3-5-sonnet-20241022,x-ai/grok-3-beta" --max-cases 4 --valid-attempts-per-case 2 --verbose --parallel
-```
-
-This will build the eval script, run it, and then open the streamlit dashboard to show the results.
-
-The `TestRunner.ts` script is the main coordinator. For each test case and setup, `ClineWrapper.ts` takes over and sends the conversation and system prompt to the LLM. We then watch the model's response as it streams in and parse it to find any tool calls.
-
-We're specifically looking for the model to make a single `replace_in_file` tool call. Multiple edits in one tool call are allowed, and recorded (in case you want to filter results by number of edits in a single tool call and compare success rate for that slice across different models/system prompts/etc). If it does, and it's for the correct file, we grab the diff content it produced. Then, the chosen diff application algorithm tries to apply that diff to the original file. We record whether this worked or not as `diffEditSuccess`.
-
-We record a bunch of data for every attempt into a database. This includes details about the model and prompt, token counts, costs, the raw output from the model, the parsed tool calls, whether it succeeded or failed, any error messages, and timing info. For a detailed explanation of the database schema, see [database.md](./database.md).
-
-A big part of this is how we handle "valid attempts," which I'll explain next.
-
-## Keeping it Fair with "Valid Attempts"
-
-LLMs can be unpredictable. If we replay an old scenario, a new model, or even the same model later, might do something completely different than what happened originally. It might call another tool or ask a question instead of trying a diff edit.
-
-Since we really want to test the *diff editing* part, we need a way to make sure we're comparing fairly. That's why we have this idea of "valid attempts."
-
-An attempt is "valid" for this benchmark if the model actually tries to do what we're interested in. This means two things. One, it must call the `replace_in_file` tool. Two, it must target the *same file path* that was targeted in the original recorded conversation for that test case.
-
-If the model does something else, like calling a different tool or picking the wrong file, we don't count that attempt against its diff editing score. Instead, we consider it an "invalid attempt" for *this specific benchmark* and simply re-run that test case with that model. We keep doing this until we've collected a set number of these "valid attempts."
-
-For example, if we ask for 5 valid attempts per test case, the system will keep re-rolling for that case until the model has tried to edit the correct file using the `replace_in_file` tool 5 times. Only then do we look at how many of those 5 valid attempts actually resulted in a successful diff application (`diffEditSuccess`).
-
-This way, if we're comparing two models and one gets a 10% success rate on its valid diff edit attempts, and another gets 90%, we have a much clearer picture of their actual diff-generating capabilities. It avoids muddying the waters with attempts where the model didn't even try to perform the specific action we're evaluating. This approach helps us isolate and measure the diff-editing skill more directly, despite the non-deterministic nature of these models.
-
-## Replays
-
-You can also use the replay argument to replay a previous benchmark run. This is super useful for iterating on our diffing algorithms without having to re-run expensive and time-consuming LLM calls.
-
-When you run an evaluation, every detail is stored in the database—including the raw, unmodified output from the model. The replay feature takes advantage of this by pulling that raw output and feeding it into a *different* diffing algorithm. This lets you isolate the performance of the diffing logic itself. We can see if a new algorithm is better at applying the exact same set of diffs that a model generated in a previous run.
-
-This process is blazingly fast and free, as it completely bypasses the need to make new API calls. It ensures a true apples-to-apples comparison between diffing strategies, since the model's output—the "ground truth" for the evaluation—remains identical.
-
-Here’s an example of how you would replay a previous run with a new diffing algorithm:
-
-```shell
-cd evals && npm run diff-eval -- --replay-run-id 9902189e-63a8-4210-a4fc-fe59e2eaf2c2 --diff-apply-file diff-06-23-25 --verbose
-```
-
-In this command:
--   `--replay-run-id` specifies the original run we want to use as our ground truth.
--   `--diff-apply-file` tells the script to use the new diffing logic from the `diff-06-23-25.ts` file.
-
-The script will then create a new run in the database that mirrors the original, but with the results of applying the new diffing algorithm. This allows for a direct comparison in the dashboard, helping us quickly see which of our diffing strategies is the most robust.

evals/diff-edits/dashboard/.streamlit/config.toml

@@ -1,8 +0,0 @@
-[theme]
-base="dark"
-
-[browser]
-gatherUsageStats = false
-
-[server]
-headless = true

evals/diff-edits/dashboard/README.md

@@ -1,159 +0,0 @@
-# 🚀 The Sickest Diff Edits Evaluation Dashboard Ever!
-
-A beautiful, modern Streamlit dashboard for visualizing and analyzing diff editing evaluation results with deep drill-down capabilities.
-
-## ✨ Features
-
-### 🎯 **Smart Model Comparison**
-- **Latest Run Focus**: Automatically loads and displays your most recent evaluation run
-- **Beautiful Performance Cards**: Each model gets a stunning card with performance grades (A+ to C)
-- **Best Performer Highlighting**: The top model gets special styling and a trophy 🏆
-- **Interactive Charts**: Success rate comparisons and latency vs cost analysis
-
-### 🔍 **Deep Drill-Down Analysis**
-- **Individual Result Inspection**: Click any model to see detailed results
-- **Side-by-Side File Views**: See original file content with line numbers
-- **Parsed Tool Call Analysis**: View exactly what the model tried to do
-- **Error Analysis**: Detailed error information for failed attempts
-- **Success Metrics**: Line changes, edit counts, and timing breakdowns
-
-### 🎨 **Aesthetic Design**
-- **Modern UI**: Custom CSS with Inter font, gradients, and shadows
-- **Responsive Layout**: Looks great on any screen size
-- **Color-Coded Performance**: Green for excellent, yellow for good, red for poor
-- **Smooth Animations**: Hover effects and transitions
-- **Professional Styling**: Clean, modern design that looks amazing
-
-### 📊 **Comprehensive Metrics**
-- **Success Rates**: Color-coded percentages with performance grades
-- **Timing Analysis**: First token, first edit, and round trip times
-- **Cost Tracking**: Per-result and total cost analysis
-- **Token Metrics**: Context tokens and completion tokens
-- **Edit Statistics**: Number of edits, lines added/deleted
-
-## 🚀 Quick Start
-
-1. **Install dependencies**:
-   ```bash
-   cd diff-edits/dashboard
-   pip install -r requirements.txt
-   ```
-
-2. **Launch the dashboard**:
-   ```bash
-   streamlit run app.py
-   ```
-   
-   Or use the convenient launch script:
-   ```bash
-   ./launch.sh
-   ```
-
-3. **Open your browser** to http://localhost:8501
-
-## 🎯 Dashboard Sections
-
-### **Hero Section**
-- Beautiful gradient header with run information
-- Key metrics overview (models tested, total results, success rate, cost)
-
-### **Model Performance Cards**
-- Each model displayed as a beautiful card
-- Large success rate display with color coding
-- Performance grade badges (A+, A, B+, B, C+, C)
-- Key metrics: latency, cost, results count, first token time
-- "Drill Down" button for detailed analysis
-
-### **Performance Analytics**
-- Interactive bar chart showing success rates
-- Scatter plot of latency vs cost with bubble sizes
-- Hover details and zoom capabilities
-
-### **Detailed Analysis (Drill-Down)**
-- Model-specific success rate, latency, and cost metrics
-- Individual result selector with status icons
-- Tabbed interface for different views:
-
-#### 📄 **File & Edits Tab**
-- **Side-by-side view**: Original file content with line numbers
-- **Edit analysis**: Success/failure status with detailed metrics
-- **Error display**: Clear error information for failed attempts
-- **Success metrics**: Lines added/deleted, number of edits
-- **Parsed tool calls**: JSON view of what the model attempted
-
-#### 🤖 **Raw Output Tab**
-- Complete raw model output in a code viewer
-- Monospace font for easy reading
-
-#### 🔧 **Parsed Tool Call Tab**
-- Pretty-printed JSON of parsed tool calls
-- Diff block visualization for replace_in_file calls
-- Error handling for malformed JSON
-
-#### 📊 **Metrics Tab**
-- Detailed timing metrics (first token, first edit, round trip)
-- Token and cost information
-- Context size and completion tokens
-
-## 🛠 **Technical Features**
-
-### **Smart Data Loading**
-- Automatic latest run detection
-- Efficient SQL queries with proper JOINs
-- Streamlit caching for performance
-- Error handling for missing data
-
-### **Interactive Navigation**
-- Session state management for drill-down views
-- Back button to return to overview
-- Smooth transitions between views
-
-### **Beautiful Styling**
-- Custom CSS with Google Fonts (Inter)
-- Gradient backgrounds and shadows
-- Hover effects and animations
-- Color-coded performance indicators
-- Professional card-based layout
-
-### **Responsive Design**
-- Works on desktop, tablet, and mobile
-- Flexible column layouts
-- Scalable text and metrics
-
-## 🎨 **Design Philosophy**
-
-This dashboard follows modern design principles:
-- **Clarity**: Information is easy to find and understand
-- **Beauty**: Visually appealing with professional styling
-- **Functionality**: Deep drill-down capabilities for detailed analysis
-- **Performance**: Fast loading with efficient data queries
-- **Usability**: Intuitive navigation and clear visual hierarchy
-
-## 📊 **Data Visualization**
-
-- **Plotly Charts**: Interactive, professional-looking visualizations
-- **Color Coding**: Consistent color scheme for performance levels
-- **Performance Badges**: A+ to C grading system
-- **Status Icons**: ✅ for success, ❌ for failure
-- **Metric Cards**: Clean, card-based metric display
-
-## 🔧 **Customization**
-
-The dashboard is highly customizable:
-- **CSS Styling**: Easy to modify colors, fonts, and layouts
-- **Performance Grades**: Adjustable thresholds for A/B/C grades
-- **Metrics Display**: Add or remove metrics as needed
-- **Chart Types**: Easily swap chart types or add new visualizations
-
-## 🚀 **Future Enhancements**
-
-Potential additions:
-- **Historical Trends**: Compare performance across multiple runs
-- **Export Functionality**: Download results as CSV/PDF
-- **Real-time Updates**: Auto-refresh for ongoing evaluations
-- **Custom Filters**: Filter by date range, model type, etc.
-- **Comparison Mode**: Side-by-side model comparisons
-
----
-
-**This is the sickest eval dashboard ever!** 🔥 It combines beautiful design with powerful analysis capabilities, making it easy to understand model performance at a glance while providing deep drill-down capabilities for detailed investigation.

evals/diff-edits/dashboard/app.py

@@ -1,957 +0,0 @@
-import streamlit as st
-import sqlite3
-import pandas as pd
-import plotly.express as px
-import plotly.graph_objects as go
-from plotly.subplots import make_subplots
-import numpy as np
-from datetime import datetime
-import os
-import json
-import difflib
-# import mimetypes # No longer needed here if guess_language_from_filepath handles it
-from utils import get_database_connection, guess_language_from_filepath # Import from utils
-
-# Page config
-st.set_page_config(
-    page_title="Diff Edits Evaluation Dashboard",
-    page_icon="📊",
-    layout="wide",
-    initial_sidebar_state="expanded"
-)
-
-# Custom CSS for beautiful styling
-st.markdown("""
-<style>
-    /* Import Google Fonts */
-    @import url('https://fonts.googleapis.com/css2?family=Azeret+Mono:wght@400;700&display=swap');
-    
-    /* Global Styles */
-    .main {
-        font-family: 'Azeret Mono', monospace;
-    }
-    
-    /* Hero Section */
-    .hero-container {
-        background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
-        padding: 2rem;
-        border-radius: 15px;
-        margin-bottom: 2rem;
-        color: white;
-        text-align: center;
-    }
-    
-    .hero-title {
-        font-size: 3rem;
-        font-weight: 700;
-        margin-bottom: 0.5rem;
-        text-shadow: 2px 2px 4px rgba(0,0,0,0.3);
-    }
-    
-    .hero-subtitle {
-        font-size: 1.2rem;
-        font-weight: 300;
-        opacity: 0.9;
-    }
-    
-    /* Model Performance Cards */
-    .model-card {
-        background: white;
-        border-radius: 15px;
-        padding: 1.5rem;
-        margin: 1rem 0;
-        box-shadow: 0 8px 32px rgba(0,0,0,0.1);
-        border: 1px solid rgba(255,255,255,0.2);
-        transition: transform 0.3s ease, box-shadow 0.3s ease;
-    }
-    
-    .model-card:hover {
-        transform: translateY(-5px);
-        box-shadow: 0 12px 40px rgba(0,0,0,0.15);
-    }
-    
-    .model-card.best-performer {
-        border: 2px solid #00D4AA;
-        background: linear-gradient(135deg, #f0fdf4 0%, #ecfdf5 100%);
-    }
-    
-    .model-name {
-        font-size: 1.5rem;
-        font-weight: 600;
-        margin-bottom: 1rem;
-        color: #1f2937;
-    }
-    
-    .success-rate {
-        font-size: 3rem;
-        font-weight: 700;
-        margin-bottom: 0.5rem;
-    }
-    
-    .success-rate.excellent { color: #10b981; }
-    .success-rate.good { color: #f59e0b; }
-    .success-rate.poor { color: #ef4444; }
-    
-    .metric-row {
-        display: flex;
-        justify-content: space-between;
-        margin: 0.5rem 0;
-        padding: 0.5rem;
-        background: rgba(0,0,0,0.02);
-        border-radius: 8px;
-    }
-    
-    .metric-label {
-        font-weight: 500;
-        color: #6b7280;
-    }
-    
-    .metric-value {
-        font-weight: 600;
-        color: #1f2937;
-    }
-    
-    /* Performance Badge */
-    .performance-badge {
-        display: inline-block;
-        padding: 0.25rem 0.75rem;
-        border-radius: 20px;
-        font-weight: 600;
-        font-size: 0.875rem;
-        margin-left: 1rem;
-    }
-    
-    .badge-a { background: #10b981; color: white; }
-    .badge-b { background: #f59e0b; color: white; }
-    .badge-c { background: #ef4444; color: white; }
-    
-    /* Comparison Charts */
-    .chart-container {
-        background: white;
-        border-radius: 15px;
-        padding: 1.5rem;
-        margin: 1rem 0;
-        box-shadow: 0 4px 20px rgba(0,0,0,0.08);
-    }
-    
-    /* Result Detail Modal */
-    .result-detail {
-        background: white;
-        border-radius: 15px;
-        padding: 2rem;
-        margin: 1rem 0;
-        box-shadow: 0 8px 32px rgba(0,0,0,0.1);
-    }
-    
-    .file-viewer {
-        background: #f8fafc;
-        border: 1px solid #e2e8f0;
-        border-radius: 8px;
-        padding: 1rem;
-        font-family: 'Monaco', 'Menlo', 'Ubuntu Mono', monospace;
-        font-size: 0.875rem;
-        line-height: 1.5;
-        overflow-x: auto;
-    }
-    
-    .diff-added {
-        background-color: #dcfce7;
-        color: #166534;
-    }
-    
-    .diff-removed {
-        background-color: #fef2f2;
-        color: #dc2626;
-    }
-    
-    .error-display {
-        background: #fef2f2;
-        border: 1px solid #fecaca;
-        border-radius: 8px;
-        padding: 1rem;
-        color: #dc2626;
-        font-family: monospace;
-    }
-    
-    /* Sidebar Styling */
-    .sidebar .sidebar-content {
-        background: linear-gradient(180deg, #f8fafc 0%, #f1f5f9 100%);
-    }
-    
-    /* Custom Metrics */
-    .custom-metric {
-        text-align: center;
-        padding: 1rem;
-        background: white;
-        border-radius: 10px;
-        box-shadow: 0 2px 10px rgba(0,0,0,0.05);
-        margin: 0.5rem 0;
-    }
-    
-    .custom-metric-value {
-        font-size: 2rem;
-        font-weight: 700;
-        color: #1f2937;
-    }
-    
-    .custom-metric-label {
-        font-size: 0.875rem;
-        color: #6b7280;
-        font-weight: 500;
-        margin-top: 0.25rem;
-    }
-</style>
-""", unsafe_allow_html=True)
-
-# Enhanced data loading functions
-@st.cache_data
-def load_all_runs():
-    """Load all evaluation runs"""
-    conn = get_database_connection()
-    
-    query = """
-    SELECT run_id, description, created_at, system_prompt_hash
-    FROM runs 
-    ORDER BY created_at DESC
-    """
-    
-    return pd.read_sql_query(query, conn)
-
-@st.cache_data
-def load_run_comparison(run_id):
-    """Load a specific run with model comparison data"""
-    conn = get_database_connection()
-    
-    # Get the run details
-    run_query = f"""
-    SELECT run_id, description, created_at, system_prompt_hash
-    FROM runs 
-    WHERE run_id = '{run_id}'
-    """
-    run_data = pd.read_sql_query(run_query, conn)
-    
-    if run_data.empty:
-        return None, None
-    
-    # Get model performance for this run
-    model_perf_query = f"""
-    SELECT 
-        res.model_id,
-        COUNT(*) as total_results,
-        AVG(CASE WHEN res.succeeded THEN 1.0 ELSE 0.0 END) as success_rate,
-        AVG(res.cost_usd) as avg_cost,
-        SUM(res.cost_usd) as total_cost,
-        AVG(res.time_to_first_token_ms) as avg_first_token_ms,
-        AVG(res.time_to_first_edit_ms) as avg_first_edit_ms,
-        AVG(res.time_round_trip_ms) as avg_round_trip_ms,
-        AVG(res.completion_tokens) as avg_completion_tokens,
-        AVG(res.num_edits) as avg_num_edits,
-        MIN(res.time_round_trip_ms) as min_round_trip_ms,
-        MAX(res.time_round_trip_ms) as max_round_trip_ms
-    FROM results res
-    JOIN cases c ON res.case_id = c.case_id
-    WHERE c.run_id = '{run_id}'
-      AND (res.error_enum NOT IN (1, 6, 7) OR res.error_enum IS NULL)  -- Exclude: no_tool_calls, wrong_tool_call, wrong_file_edited
-    GROUP BY res.model_id
-    ORDER BY success_rate DESC, avg_round_trip_ms ASC
-    """
-    
-    model_performance = pd.read_sql_query(model_perf_query, conn)
-    
-    return run_data.iloc[0], model_performance
-
-@st.cache_data
-def load_latest_run_comparison():
-    """Load the latest run with model comparison data"""
-    conn = get_database_connection()
-    
-    # Get the latest run
-    latest_run_query = """
-    SELECT run_id, description, created_at, system_prompt_hash
-    FROM runs 
-    ORDER BY created_at DESC 
-    LIMIT 1
-    """
-    latest_run = pd.read_sql_query(latest_run_query, conn)
-    
-    if latest_run.empty:
-        return None, None
-    
-    return load_run_comparison(latest_run.iloc[0]['run_id'])
-
-@st.cache_data
-def load_detailed_results(run_id, model_id=None, valid_only=False):
-    """Load detailed results for drill-down analysis"""
-    conn = get_database_connection()
-    
-    where_clause = f"WHERE c.run_id = '{run_id}'"
-    if model_id:
-        where_clause += f" AND res.model_id = '{model_id}'"
-    
-    # Option to filter out invalid attempts
-    if valid_only:
-        where_clause += " AND (res.error_enum NOT IN (1, 6, 7) OR res.error_enum IS NULL)"
-    
-    query = f"""
-    SELECT 
-        res.*,
-        c.task_id,
-        c.description as case_description,
-        c.tokens_in_context,
-        sp.name as system_prompt_name,
-        pf.name as processing_functions_name,
-        orig_f.filepath as original_filepath,
-        orig_f.content as original_file_content,
-        edit_f.filepath as edited_filepath,
-        edit_f.content as edited_file_content
-    FROM results res
-    JOIN cases c ON res.case_id = c.case_id
-    LEFT JOIN system_prompts sp ON c.system_prompt_hash = sp.hash
-    LEFT JOIN processing_functions pf ON res.processing_functions_hash = pf.hash
-    LEFT JOIN files orig_f ON c.file_hash = orig_f.hash
-    LEFT JOIN files edit_f ON res.file_edited_hash = edit_f.hash
-    {where_clause}
-    ORDER BY res.created_at DESC
-    """
-    
-    return pd.read_sql_query(query, conn)
-
-def get_performance_grade(success_rate):
-    """Get performance grade based on success rate"""
-    if success_rate >= 0.9:
-        return "A+", "excellent"
-    elif success_rate >= 0.8:
-        return "A", "excellent"
-    elif success_rate >= 0.7:
-        return "B+", "good"
-    elif success_rate >= 0.6:
-        return "B", "good"
-    elif success_rate >= 0.5:
-        return "C+", "good"
-    else:
-        return "C", "poor"
-
-def render_hero_section(current_run, model_performance):
-    """Render the hero section with key metrics"""
-    run_title = current_run['description'] if current_run['description'] else f"Run {current_run['run_id'][:8]}..."
-    st.markdown(f"""
-    <div class="hero-container">
-        <div class="hero-title">Diff Edit Evaluation Results</div>
-        <div class="hero-subtitle">A comprehensive analysis of model performance on code editing tasks.</div>
-        <div class="hero-subtitle" style="font-size: 0.9rem; margin-top: 10px;">
-            <strong>Current Run:</strong> {run_title} • {current_run['created_at']}
-        </div>
-    </div>
-    """, unsafe_allow_html=True)
-    
-    # Key metrics row
-    col1, col2, col3, col4 = st.columns(4)
-    
-    total_results = model_performance['total_results'].sum()
-    overall_success = model_performance['success_rate'].mean()
-    total_cost = model_performance['total_cost'].sum()
-    avg_latency = model_performance['avg_round_trip_ms'].mean()
-    
-    with col1:
-        st.markdown(f"""
-        <div class="custom-metric">
-            <div class="custom-metric-value">{len(model_performance)}</div>
-            <div class="custom-metric-label">Models Tested</div>
-        </div>
-        """, unsafe_allow_html=True)
-    
-    with col2:
-        st.markdown(f"""
-        <div class="custom-metric">
-            <div class="custom-metric-value">{total_results}</div>
-            <div class="custom-metric-label">Valid Results</div>
-        </div>
-        """, unsafe_allow_html=True)
-    
-    with col3:
-        success_color = "#10b981" if overall_success > 0.8 else "#f59e0b" if overall_success > 0.6 else "#ef4444"
-        st.markdown(f"""
-        <div class="custom-metric">
-            <div class="custom-metric-value" style="color: {success_color}">{overall_success:.1%}</div>
-            <div class="custom-metric-label">Avg Success Rate</div>
-        </div>
-        """, unsafe_allow_html=True)
-    
-    with col4:
-        st.markdown(f"""
-        <div class="custom-metric">
-            <div class="custom-metric-value">${total_cost:.3f}</div>
-            <div class="custom-metric-label">Total Cost</div>
-        </div>
-        """, unsafe_allow_html=True)
-
-def render_model_comparison_cards(model_performance):
-    """Render beautiful model comparison cards"""
-    st.markdown("## Model Leaderboard")
-    
-    # Find best performer
-    best_model = model_performance.iloc[0]['model_id']
-    
-    for idx, model in model_performance.iterrows():
-        is_best = model['model_id'] == best_model
-        grade, grade_class = get_performance_grade(model['success_rate'])
-        
-        # Create a container for each model
-        with st.container():
-            col1, col2 = st.columns([3, 1])
-            
-            with col1:
-                # Use Streamlit's native components instead of raw HTML
-                if is_best:
-                    st.success(f"**{model['model_id']}** - Best Performer")
-                else:
-                    st.info(f"**{model['model_id']}**")
-                
-                # Success rate with color coding
-                success_rate = model['success_rate']
-                if success_rate >= 0.8:
-                    st.success(f"**Success Rate:** {success_rate:.1%} ({grade})")
-                elif success_rate >= 0.6:
-                    st.warning(f"**Success Rate:** {success_rate:.1%} ({grade})")
-                else:
-                    st.error(f"**Success Rate:** {success_rate:.1%} ({grade})")
-                
-                # Metrics in columns
-                metric_col1, metric_col2, metric_col3, metric_col4 = st.columns(4)
-                
-                with metric_col1:
-                    if pd.notna(model['avg_round_trip_ms']):
-                        st.metric("Avg Latency", f"{model['avg_round_trip_ms']:.0f}ms")
-                    else:
-                        st.metric("Avg Latency", "N/A")
-                
-                with metric_col2:
-                    if pd.notna(model['avg_cost']):
-                        st.metric("Avg Cost", f"${model['avg_cost']:.4f}")
-                    else:
-                        st.metric("Avg Cost", "N/A")
-                
-                with metric_col3:
-                    st.metric("Valid Results", f"{model['total_results']}")
-                
-                with metric_col4:
-                    if pd.notna(model['avg_first_token_ms']):
-                        st.metric("First Token", f"{model['avg_first_token_ms']:.0f}ms")
-                    else:
-                        st.metric("First Token", "N/A")
-            
-            with col2:
-                st.write("")  # Add some spacing
-                if st.button(f"Drill Down", key=f"drill_{model['model_id']}", use_container_width=True):
-                    st.session_state.drill_down_model = model['model_id']
-            
-            st.divider()  # Add a divider between models
-
-def render_comparison_charts(model_performance):
-    """Render interactive comparison charts"""
-    st.markdown("## Performance Analysis")
-    
-    col1, col2 = st.columns(2)
-
-    with col1:
-        # Time to First Edit
-        fig_first_edit = px.bar(
-            model_performance,
-            x='model_id',
-            y='avg_first_edit_ms',
-            title="Time to First Edit",
-            labels={'avg_first_edit_ms': 'Time to First Edit (ms)', 'model_id': 'Model'},
-            color='avg_first_edit_ms',
-            color_continuous_scale='bluered',
-            text='avg_first_edit_ms',
-            template='plotly_dark'
-        )
-        fig_first_edit.update_traces(texttemplate='%{text:.0f}ms', textposition='outside')
-        fig_first_edit.update_layout(
-            showlegend=False,
-            plot_bgcolor='rgba(0,0,0,0)',
-            paper_bgcolor='rgba(0,0,0,0)',
-            font=dict(family="Azeret Mono, monospace"),
-            margin=dict(t=50)
-        )
-        st.plotly_chart(fig_first_edit, use_container_width=True)
-
-    with col2:
-        # Latency vs Cost Scatter
-        fig_scatter = px.scatter(
-            model_performance,
-            x='avg_round_trip_ms',
-            y='avg_cost',
-            size='total_results',
-            color='success_rate',
-            hover_name='model_id',
-            title="Latency vs Cost Analysis",
-            labels={
-                'avg_round_trip_ms': 'Avg Round Trip (ms)',
-                'avg_cost': 'Avg Cost ($)',
-                'success_rate': 'Success Rate',
-                'total_results': 'Valid Results'
-            },
-            color_continuous_scale='RdYlGn',
-            template='plotly_dark'
-        )
-        fig_scatter.update_layout(
-            plot_bgcolor='rgba(0,0,0,0)',
-            paper_bgcolor='rgba(0,0,0,0)',
-            font=dict(family="Azeret Mono, monospace")
-        )
-        st.plotly_chart(fig_scatter, use_container_width=True)
-
-def render_detailed_analysis(run_id, model_id):
-    """Render detailed drill-down analysis"""
-    st.markdown(f"## Detailed Analysis: {model_id}")
-    
-    # Load all results (including invalid attempts)
-    detailed_results = load_detailed_results(run_id, model_id)
-    
-    # Also load only valid results for metrics
-    valid_results = load_detailed_results(run_id, model_id, valid_only=True)
-    
-    if detailed_results.empty:
-        st.warning("No detailed results found.")
-        return
-    
-    # Show total vs valid results
-    st.info(f"Showing all {len(detailed_results)} results ({len(valid_results)} valid, {len(detailed_results) - len(valid_results)} invalid)")
-    
-    # Results overview
-    col1, col2, col3 = st.columns(3)
-    
-    with col1:
-        success_count = valid_results['succeeded'].sum()
-        total_count = len(valid_results)
-        st.metric("Success Rate", f"{success_count}/{total_count} ({success_count/total_count:.1%} of valid results)")
-    
-    with col2:
-        avg_latency = detailed_results['time_round_trip_ms'].mean()
-        st.metric("Avg Latency", f"{avg_latency:.0f}ms")
-    
-    with col3:
-        total_cost = detailed_results['cost_usd'].sum()
-        st.metric("Total Cost", f"${total_cost:.4f}")
-    
-    # Interactive results table
-    st.markdown("### 📋 Individual Results")
-    
-    # Add result selector with indicators for valid/invalid attempts
-    result_options = []
-    for idx, row in detailed_results.iterrows():
-        # Check if this is a valid result
-        is_valid = (row['error_enum'] not in [1, 6, 7]) if not pd.isna(row['error_enum']) else True
-        
-        # Create status indicator
-        if is_valid:
-            status = "✅" if row['succeeded'] else "❌"
-        else:
-            status = "⚠️"  # Warning symbol for invalid results
-            
-        # Add validity indicator to the option text
-        validity_text = "" if is_valid else " [INVALID RESULT]"
-        result_options.append(f"{status} {row['task_id']} - {row['time_round_trip_ms']:.0f}ms{validity_text}")
-    
-    selected_result_idx = st.selectbox(
-        "Select a result to analyze:",
-        range(len(result_options)),
-        format_func=lambda x: result_options[x]
-    )
-    
-    if selected_result_idx is not None:
-        render_result_detail(detailed_results.iloc[selected_result_idx])
-
-def render_result_detail(result):
-    """Render detailed view of a single result"""
-    st.markdown("### 🔬 Result Deep Dive")
-    
-    # Check if this is a valid result
-    is_valid = (result['error_enum'] not in [1, 6, 7]) if not pd.isna(result['error_enum']) else True
-    
-    # Show validity warning if needed
-    if not is_valid:
-        st.warning("⚠️ **This is an invalid result** - The model didn't properly call the diff edit tool or edited the wrong file. This result is excluded from success rate calculations.")
-    
-    # Result metadata
-    col1, col2, col3, col4 = st.columns(4)
-    
-    with col1:
-        status_icon = "✅" if result['succeeded'] else "❌"
-        st.markdown(f"**Status:** {status_icon} {'Success' if result['succeeded'] else 'Failed'}")
-    
-    with col2:
-        st.markdown(f"**Task ID:** {result['task_id']}")
-    
-    with col3:
-        st.markdown(f"**Round Trip:** {result['time_round_trip_ms']:.0f}ms")
-    
-    with col4:
-        st.markdown(f"**Cost:** ${result['cost_usd']:.4f}")
-    
-    # Tabbed interface for different views
-    tab1, tab2, tab3, tab4 = st.tabs(["📄 File & Edits", "🤖 Raw Output", "🔧 Parsed Tool Call", "📊 Metrics"])
-    
-    with tab1:
-        render_file_and_edits_view(result)
-    
-    with tab2:
-        render_raw_output_view(result)
-    
-    with tab3:
-        render_parsed_tool_call_view(result)
-    
-    with tab4:
-        render_metrics_view(result)
-
-def render_file_and_edits_view(result):
-    """Render side-by-side file and edits view"""
-    st.markdown("#### 📄 File Content & Edit Analysis")
-    
-    # Check if we have original file content
-    has_original = not pd.isna(result['original_file_content']) and result['original_file_content']
-    has_edited = not pd.isna(result['edited_file_content']) and result['edited_file_content']
-    
-    if not has_original and not has_edited:
-        st.warning("No file content available for this result.")
-        return
-    
-    col1, col2 = st.columns(2)
-    
-    with col1:
-        st.markdown("**Original File:**")
-        if has_original:
-            filepath = result['original_filepath'] if not pd.isna(result['original_filepath']) else 'Unknown file'
-            st.markdown(f"📁 `{filepath}`")
-            
-            # Display full original file content in a scrollable code block
-            with st.expander("View Original File Content", expanded=True):
-                # Prepare content for the copy button (needs JS-specific escaping)
-                raw_content_for_copy = result['original_file_content']
-                # Escape for JavaScript template literal: backticks, backslashes, newlines
-                js_escaped_content = raw_content_for_copy.replace('\\', '\\\\') \
-                                                       .replace('`', '\\`') \
-                                                       .replace('\r\n', '\\n') \
-                                                       .replace('\n', '\\n') \
-                                                       .replace('\r', '\\n')
-
-                unique_suffix = str(result.name if hasattr(result, 'name') else result['task_id']).replace('-', '_').replace('.', '_')
-                button_id = f"copyBtnOriginal_{unique_suffix}"
-                
-                copy_button_html = f"""
-                    <button id="{button_id}" onclick="copyOriginalToClipboard(`{js_escaped_content}`, '{button_id}')" style="margin-bottom: 10px; padding: 5px 10px; border-radius: 5px; border: 1px solid #ccc; cursor: pointer;">Copy Original File</button>
-                    <script>
-                        if (!window.copyOriginalToClipboard) {{
-                            window.copyOriginalToClipboard = async function(text, buttonId) {{
-                                try {{
-                                    await navigator.clipboard.writeText(text);
-                                    const button = document.getElementById(buttonId);
-                                    button.innerText = 'Copied!';
-                                    button.style.backgroundColor = '#d4edda'; // Optional: success feedback
-                                    setTimeout(() => {{ 
-                                        button.innerText = 'Copy Original File'; 
-                                        button.style.backgroundColor = '';
-                                    }}, 2000);
-                                }} catch (err) {{
-                                    console.error('Failed to copy original: ', err);
-                                    const button = document.getElementById(buttonId);
-                                    button.innerText = 'Copy Failed!';
-                                    button.style.backgroundColor = '#f8d7da'; // Optional: error feedback
-                                    setTimeout(() => {{ 
-                                        button.innerText = 'Copy Original File'; 
-                                        button.style.backgroundColor = '';
-                                    }}, 2000);
-                                }}
-                            }}
-                        }}
-                    </script>
-                    """
-                st.components.v1.html(copy_button_html, height=50)
-
-                # Prepare content for st.code (needs actual newlines)
-                content_for_display = result['original_file_content']
-                # Iteratively replace common escaped newline sequences with actual newlines
-                # This handles cases like "\\n" -> "\n" and then "\n" (if it was literally "\n")
-                # Order might matter if there are multiple levels of escaping, but this covers common ones.
-                content_for_display = content_for_display.replace('\\\\r\\\\n', '\r\n').replace('\\\\n', '\n') # Double escaped
-                content_for_display = content_for_display.replace('\\r\\n', '\r\n').replace('\\n', '\n')     # Single escaped
-
-                language = guess_language_from_filepath(filepath)
-                st.code(content_for_display, language=language, line_numbers=False)
-
-        else:
-            st.warning("Original file content not available")
-    
-    with col2:
-        st.markdown("**Edit Analysis:**")
-        
-        if not result['succeeded']:
-            # Show error information
-            st.error("❌ **Edit Failed**")
-            
-            if not pd.isna(result['error_enum']):
-                st.markdown(f"**Error Code:** {result['error_enum']}")
-        else:
-            # Show successful edit information
-            st.success("✅ **Edit Successful**")
-            
-            # Show edit metrics
-            metric_col1, metric_col2, metric_col3 = st.columns(3)
-            
-            with metric_col1:
-                if not pd.isna(result['num_edits']):
-                    st.metric("Edits", int(result['num_edits']))
-            
-            with metric_col2:
-                if not pd.isna(result['num_lines_added']):
-                    st.metric("Added", int(result['num_lines_added']))
-            
-            with metric_col3:
-                if not pd.isna(result['num_lines_deleted']):
-                    st.metric("Deleted", int(result['num_lines_deleted']))
-            
-            # Show edited file if available
-            if has_edited:
-                st.markdown("**Edited File:**")
-                with st.expander("View Edited File Content"):
-                    edited_lines = result['edited_file_content'].split('\n')
-                    for i, line in enumerate(edited_lines[:50], 1):
-                        st.text(f"{i:3d} | {line}")
-                    
-                    if len(edited_lines) > 50:
-                        st.text(f"... ({len(edited_lines) - 50} more lines)")
-        
-        # Show parsed tool call if available
-        if not pd.isna(result['parsed_tool_call_json']):
-            with st.expander("View Parsed Tool Call"):
-                try:
-                    parsed_call = json.loads(result['parsed_tool_call_json'])
-                    st.json(parsed_call)
-                except:
-                    st.text(result['parsed_tool_call_json'])
-
-def render_raw_output_view(result):
-    """Render raw model output"""
-    st.markdown("#### 🤖 Raw Model Output")
-    
-    if pd.isna(result['raw_model_output']) or not result['raw_model_output']:
-        st.warning("No raw output available for this result.")
-        return
-    
-    st.markdown("""
-    <div class="file-viewer">
-    """, unsafe_allow_html=True)
-    
-    st.text(result['raw_model_output'])
-    
-    st.markdown("</div>", unsafe_allow_html=True)
-
-def render_parsed_tool_call_view(result):
-    """Render parsed tool call analysis"""
-    st.markdown("#### 🔧 Parsed Tool Call Analysis")
-    
-    if pd.isna(result['parsed_tool_call_json']) or not result['parsed_tool_call_json']:
-        st.warning("No parsed tool call available for this result.")
-        return
-    
-    try:
-        parsed_call = json.loads(result['parsed_tool_call_json'])
-        
-        # Pretty print the JSON
-        st.json(parsed_call)
-        
-        # If it's a replace_in_file call, show the diff blocks
-        if isinstance(parsed_call, dict) and 'diff' in parsed_call:
-            st.markdown("**Diff Blocks:**")
-            st.code(parsed_call['diff'], language='diff')
-            
-    except json.JSONDecodeError:
-        st.markdown("**Raw Parsed Call (Invalid JSON):**")
-        st.text(result['parsed_tool_call_json'])
-
-def render_metrics_view(result):
-    """Render detailed metrics for the result"""
-    st.markdown("#### 📊 Detailed Metrics")
-    
-    col1, col2 = st.columns(2)
-    
-    with col1:
-        st.markdown("**Timing Metrics:**")
-        if not pd.isna(result['time_to_first_token_ms']):
-            st.metric("Time to First Token", f"{result['time_to_first_token_ms']:.0f}ms")
-        
-        if not pd.isna(result['time_to_first_edit_ms']):
-            st.metric("Time to First Edit", f"{result['time_to_first_edit_ms']:.0f}ms")
-        
-        if not pd.isna(result['time_round_trip_ms']):
-            st.metric("Round Trip Time", f"{result['time_round_trip_ms']:.0f}ms")
-    
-    with col2:
-        st.markdown("**Token & Cost Metrics:**")
-        if not pd.isna(result['completion_tokens']):
-            st.metric("Completion Tokens", int(result['completion_tokens']))
-        
-        if not pd.isna(result['cost_usd']):
-            st.metric("Cost", f"${result['cost_usd']:.4f}")
-        
-        if not pd.isna(result['tokens_in_context']):
-            st.metric("Context Tokens", int(result['tokens_in_context']))
-
-def guess_language_from_filepath(filepath):
-    """Guess the language for syntax highlighting from filepath."""
-    if not filepath or pd.isna(filepath):
-        return None
-    
-    extension_map = {
-        '.py': 'python',
-        '.js': 'javascript',
-        '.ts': 'typescript',
-        '.java': 'java',
-        '.cs': 'csharp',
-        '.cpp': 'cpp',
-        '.c': 'c',
-        '.html': 'html',
-        '.css': 'css',
-        '.json': 'json',
-        '.sql': 'sql',
-        '.md': 'markdown',
-        '.rb': 'ruby',
-        '.php': 'php',
-        '.go': 'go',
-        '.rs': 'rust',
-        '.swift': 'swift',
-        '.kt': 'kotlin',
-        '.sh': 'bash',
-        '.yaml': 'yaml',
-        '.yml': 'yaml',
-        '.xml': 'xml',
-    }
-    
-    _, ext = os.path.splitext(filepath)
-def main():
-    # Add a note about valid attempts
-    st.sidebar.markdown("""
-    ### Note on Metrics
-    Success rates are calculated based on **valid results only**. 
-    
-    Invalid results (where the model didn't call the diff edit tool or edited the wrong file) are excluded from calculations.
-    """)
-    
-    # Initialize session state
-    if 'drill_down_model' not in st.session_state:
-        st.session_state.drill_down_model = None
-    if 'selected_run_id' not in st.session_state:
-        st.session_state.selected_run_id = None
-    
-    # Load all runs for sidebar
-    all_runs = load_all_runs()
-    
-    if all_runs.empty:
-        st.error("No evaluation runs found in the database.")
-        st.stop()
-    
-    # Sidebar for run selection
-    with st.sidebar:
-        st.markdown("## 📊 Evaluation Runs")
-        st.markdown("Select a run to analyze:")
-        
-        # Create run options with nice formatting
-        run_options = []
-        run_ids = []
-        
-        for idx, run in all_runs.iterrows():
-            # Format the run description nicely
-            date_str = run['created_at'][:10]  # Get just the date part
-            time_str = run['created_at'][11:16]  # Get just the time part
-            
-            if run['description']:
-                display_name = f"🚀 {run['description']}"
-            else:
-                display_name = f"📅 Run {run['run_id'][:8]}..."
-            
-            run_options.append(f"{display_name}\n📅 {date_str} {time_str}")
-            run_ids.append(run['run_id'])
-        
-        # Default to latest run if no selection
-        if st.session_state.selected_run_id is None:
-            default_index = 0  # Latest run is first
-            st.session_state.selected_run_id = run_ids[0]
-        else:
-            try:
-                default_index = run_ids.index(st.session_state.selected_run_id)
-            except ValueError:
-                default_index = 0
-                st.session_state.selected_run_id = run_ids[0]
-        
-        selected_run_idx = st.selectbox(
-            "Choose run:",
-            range(len(run_options)),
-            format_func=lambda x: run_options[x],
-            index=default_index,
-            key="run_selector"
-        )
-        
-        # Update selected run if changed
-        if run_ids[selected_run_idx] != st.session_state.selected_run_id:
-            st.session_state.selected_run_id = run_ids[selected_run_idx]
-            st.session_state.drill_down_model = None  # Reset drill down when changing runs
-            st.rerun()
-        
-        # Show run details in sidebar
-        selected_run = all_runs.iloc[selected_run_idx]
-        st.markdown("---")
-        st.markdown("### 📋 Run Details")
-        st.markdown(f"**Run ID:** `{selected_run['run_id'][:12]}...`")
-        st.markdown(f"**Created:** {selected_run['created_at']}")
-        if selected_run['description']:
-            st.markdown(f"**Description:** {selected_run['description']}")
-    
-    # Load data for selected run
-    current_run, model_performance = load_run_comparison(st.session_state.selected_run_id)
-    
-    if current_run is None or model_performance.empty:
-        st.error("No data found for the selected run.")
-        st.stop()
-    
-    # Render main dashboard
-    render_hero_section(current_run, model_performance)
-    
-    # Check if we're in drill-down mode
-    if st.session_state.drill_down_model:
-        col1, col2 = st.columns([1, 4])
-        with col1:
-            if st.button("Back to Overview", use_container_width=True):
-                st.session_state.drill_down_model = None
-                st.rerun()
-        
-        render_detailed_analysis(current_run['run_id'], st.session_state.drill_down_model)
-    else:
-        # Success Rate Comparison
-        fig_success = px.bar(
-            model_performance,
-            x='model_id',
-            y='success_rate',
-            title="Success Rate by Model",
-            labels={'success_rate': 'Success Rate', 'model_id': 'Model'},
-            color='success_rate',
-            color_continuous_scale='RdYlGn',
-            text='success_rate',
-            template='plotly_dark'
-        )
-        fig_success.update_traces(texttemplate='%{text:.1%}', textposition='outside')
-        fig_success.update_layout(
-            showlegend=False,
-            plot_bgcolor='rgba(0,0,0,0)',
-            paper_bgcolor='rgba(0,0,0,0)',
-            font=dict(family="Azeret Mono, monospace"),
-            yaxis_range=[0,1],  # Set y-axis from 0% to 100%
-            margin=dict(t=50)  # Add top margin to prevent clipping
-        )
-        st.plotly_chart(fig_success, use_container_width=True)
-        
-        render_model_comparison_cards(model_performance)
-        render_comparison_charts(model_performance)
-
-if __name__ == "__main__":
-    main()

evals/diff-edits/dashboard/launch.sh

@@ -1,33 +0,0 @@
-#!/bin/bash
-
-# Diff Edits Evaluation Dashboard Launcher
-echo "🚀 Starting Diff Edits Evaluation Dashboard..."
-
-# Check if we're in the right directory
-if [ ! -f "app.py" ]; then
-    echo "❌ Error: app.py not found. Please run this script from the dashboard directory."
-    exit 1
-fi
-
-# Check if database exists
-if [ ! -f "../evals.db" ]; then
-    echo "⚠️  Warning: Database file ../evals.db not found."
-    echo "   Make sure you've run some evaluations first to populate the database."
-    echo "   You can run: node ../cli/dist/index.js run-diff-eval --model-id anthropic/claude-sonnet-4 --max-cases 1"
-    echo ""
-fi
-
-# Check if requirements are installed
-echo "📦 Checking Python dependencies..."
-if ! python -c "import streamlit, plotly, pandas" 2>/dev/null; then
-    echo "📥 Installing required packages..."
-    pip install -r requirements.txt
-fi
-
-echo "🌐 Launching Streamlit dashboard..."
-echo "   Dashboard will open in your browser at http://localhost:8501"
-echo "   Press Ctrl+C to stop the dashboard"
-echo ""
-
-# Launch Streamlit
-streamlit run app.py

evals/diff-edits/dashboard/pages/02_Bad_Cases.py

@@ -1,183 +0,0 @@
-import streamlit as st
-import pandas as pd
-import json
-import os # Need to import os for load_case_raw_data
-from utils import get_database_connection, guess_language_from_filepath # Absolute import
-
-st.set_page_config(
-    page_title="Case Health Inspector",
-    page_icon="🧑‍⚕️",
-    layout="wide"
-)
-
-st.title("Case Health Inspector")
-st.markdown("Identify test cases that are frequently problematic across different models and runs.")
-
-@st.cache_data
-def load_problematic_cases_summary():
-    conn = get_database_connection()
-    query = """
-    WITH case_attempts AS (
-        SELECT
-            c.task_id,
-            c.description AS case_description,
-            f_orig.filepath AS original_filepath, -- Get from files table
-            r.run_id,
-            r.model_id,
-            r.result_id,
-            (CASE WHEN (r.error_enum NOT IN (1, 6, 7) OR r.error_enum IS NULL) THEN 1 ELSE 0 END) AS is_valid_attempt,
-            (CASE WHEN (r.error_enum NOT IN (1, 6, 7) OR r.error_enum IS NULL) THEN r.succeeded ELSE NULL END) AS succeeded_on_valid
-        FROM cases c
-        JOIN results r ON c.case_id = r.case_id
-        LEFT JOIN files f_orig ON c.file_hash = f_orig.hash -- Join to get original filepath
-    ),
-    case_summary AS (
-        SELECT
-            task_id,
-            case_description,
-            original_filepath, -- This is now f_orig.filepath
-            COUNT(DISTINCT run_id) AS num_benchmark_runs,
-            COUNT(result_id) AS total_attempts,
-            SUM(is_valid_attempt) AS total_valid_attempts,
-            SUM(succeeded_on_valid) AS total_successful_valid_attempts
-        FROM case_attempts
-        GROUP BY task_id, case_description, original_filepath -- original_filepath is f_orig.filepath
-    )
-    SELECT
-        task_id,
-        case_description,
-        original_filepath, -- This is f_orig.filepath from case_summary
-        num_benchmark_runs,
-        total_attempts,
-        total_valid_attempts,
-        CAST(total_valid_attempts AS REAL) * 100.0 / total_attempts AS percent_valid_attempts,
-        CASE 
-            WHEN total_valid_attempts > 0 THEN CAST(total_successful_valid_attempts AS REAL) * 100.0 / total_valid_attempts
-            ELSE 0 
-        END AS success_rate_on_valid
-    FROM case_summary
-    ORDER BY percent_valid_attempts ASC, success_rate_on_valid ASC;
-    """
-    df = pd.read_sql_query(query, conn)
-    return df
-
-@st.cache_data
-def load_case_raw_data(task_id):
-    """Loads the original JSON data for a given task_id."""
-    # This assumes test cases are stored in ../cases relative to this script's parent (dashboard)
-    # So, ../../cases from this script's location (pages/02_Bad_Cases.py)
-    # Correct path from this script (pages/02_Bad_Cases.py) to cases/
-    # os.path.dirname(__file__) -> pages
-    # os.path.join(..., '..') -> dashboard
-    # os.path.join(..., '..', '..') -> diff-edits
-    # os.path.join(..., '..', '..', 'cases') -> diff-edits/cases
-    cases_dir = os.path.join(os.path.dirname(__file__), '..', '..', 'cases')
-    
-    # The task_id is usually the filename without .json
-    # However, some task_ids might have suffixes or be different.
-    # We need a robust way to find the file. For now, assume task_id is filename base.
-    # This might need adjustment if task_id format varies significantly from filename.
-    
-    # Try direct match first
-    potential_filename = f"{task_id}.json"
-    filepath = os.path.join(cases_dir, potential_filename)
-
-    if not os.path.exists(filepath):
-        # If direct match fails, list files and try to find one that starts with task_id
-        # This is a simple fallback, might need more robust matching if task_ids are complex
-        try:
-            for f_name in os.listdir(cases_dir):
-                if f_name.startswith(task_id) and f_name.endswith(".json"):
-                    filepath = os.path.join(cases_dir, f_name)
-                    break
-            else: # No break means no file found
-                 return None # File not found
-        except FileNotFoundError:
-            return None # Cases directory itself not found
-
-    if not os.path.exists(filepath): # Check again after potential find
-        return None
-
-    try:
-        with open(filepath, 'r') as f:
-            return json.load(f)
-    except Exception as e:
-        st.error(f"Error loading case file {filepath}: {e}")
-        return None
-
-def render_problematic_cases_page():
-    summary_df = load_problematic_cases_summary()
-
-    if summary_df.empty:
-        st.warning("No case summary data found. Run some evaluations first.")
-        return
-
-    st.markdown("### Cases Overview")
-    st.dataframe(summary_df.style.format({
-        "percent_valid_attempts": "{:.1f}%",
-        "success_rate_on_valid": "{:.1f}%"
-    }), use_container_width=True)
-
-    st.markdown("---")
-    st.markdown("### Case Drill Down")
-    
-    selected_task_id = st.selectbox(
-        "Select a Case ID (task_id) to inspect:",
-        options=[""] + summary_df['task_id'].tolist() # Add a blank option
-    )
-
-    if selected_task_id:
-        case_data = summary_df[summary_df['task_id'] == selected_task_id].iloc[0]
-        st.subheader(f"Details for Case: {case_data['task_id']}")
-        st.markdown(f"**Description:** {case_data['case_description']}")
-        st.markdown(f"**Original Filepath:** `{case_data['original_filepath']}`")
-        
-        raw_json_data = load_case_raw_data(selected_task_id)
-        if raw_json_data:
-            with st.expander("View Raw Case JSON Data", expanded=False):
-                st.json(raw_json_data)
-            
-            if 'file_contents' in raw_json_data and raw_json_data['file_contents']:
-                with st.expander("View Original File Content (from Case JSON)", expanded=True):
-                    # Prepare content for the copy button
-                    raw_content_for_copy = raw_json_data['file_contents']
-                    js_escaped_content = raw_content_for_copy.replace('\\', '\\\\') \
-                                                           .replace('`', '\\`') \
-                                                           .replace('\r\n', '\\n') \
-                                                           .replace('\n', '\\n') \
-                                                           .replace('\r', '\\n')
-                    button_id = f"copyBtnCase_{selected_task_id.replace('-', '_').replace('.', '_')}"
-                    copy_button_html = f"""
-                        <button id="{button_id}" onclick="copyCaseContentToClipboard(`{js_escaped_content}`, '{button_id}')" style="margin-bottom: 10px; padding: 5px 10px; border-radius: 5px; border: 1px solid #ccc; cursor: pointer;">Copy File Content</button>
-                        <script>
-                            if (!window.copyCaseContentToClipboard) {{
-                                window.copyCaseContentToClipboard = async function(text, buttonId) {{
-                                    try {{
-                                        await navigator.clipboard.writeText(text);
-                                        const button = document.getElementById(buttonId);
-                                        button.innerText = 'Copied!';
-                                        setTimeout(() => {{ button.innerText = 'Copy File Content'; }}, 2000);
-                                    }} catch (err) {{ console.error('Failed to copy: ', err); const button = document.getElementById(buttonId); button.innerText = 'Copy Failed!'; setTimeout(() => {{ button.innerText = 'Copy File Content'; }}, 2000); }}
-                                }}
-                            }}
-                        </script>
-                        """
-                    st.components.v1.html(copy_button_html, height=50)
-
-                    # Prepare content for st.code
-                    content_for_display = raw_json_data['file_contents']
-                    content_for_display = content_for_display.replace('\\\\r\\\\n', '\r\n').replace('\\\\n', '\n')
-                    content_for_display = content_for_display.replace('\\r\\n', '\r\n').replace('\\n', '\n')
-                    
-                    language = guess_language_from_filepath(case_data['original_filepath'])
-                    st.code(content_for_display, language=language, line_numbers=False)
-            else:
-                st.warning("Original file content not found in case JSON.")
-        else:
-            st.error(f"Could not load raw JSON data for case: {selected_task_id}")
-        
-        # Placeholder for more detailed stats (per-model performance on this case, error breakdown)
-        st.markdown("*(Further per-model statistics and error breakdowns for this case can be added here.)*")
-
-if __name__ == "__main__":
-    render_problematic_cases_page()

evals/diff-edits/dashboard/requirements.txt

@@ -1,4 +0,0 @@
-streamlit>=1.28.0
-plotly>=5.17.0
-pandas>=2.0.0
-numpy>=1.24.0

evals/diff-edits/dashboard/utils.py

@@ -1,51 +0,0 @@
-import streamlit as st
-import sqlite3
-import pandas as pd
-import os
-
-@st.cache_resource
-def get_database_connection():
-    # Assuming the script is run from the dashboard directory,
-    # evals.db is two levels up from there.
-    # __file__ is utils.py, its dirname is dashboard.
-    # os.path.dirname(__file__) -> dashboard/
-    # os.path.join(..., '..') -> diff-edits/
-    # os.path.join(..., '..', 'evals.db') -> diff-edits/evals.db
-    db_path = os.path.join(os.path.dirname(__file__), '..', 'evals.db')
-    if not os.path.exists(db_path):
-        st.error(f"Database not found. Expected at: {os.path.abspath(db_path)}")
-        st.stop()
-    return sqlite3.connect(db_path, check_same_thread=False)
-
-def guess_language_from_filepath(filepath):
-    """Guess the language for syntax highlighting from filepath."""
-    if not filepath or pd.isna(filepath):
-        return None
-    
-    extension_map = {
-        '.py': 'python',
-        '.js': 'javascript',
-        '.ts': 'typescript',
-        '.java': 'java',
-        '.cs': 'csharp',
-        '.cpp': 'cpp',
-        '.c': 'c',
-        '.html': 'html',
-        '.css': 'css',
-        '.json': 'json',
-        '.sql': 'sql',
-        '.md': 'markdown',
-        '.rb': 'ruby',
-        '.php': 'php',
-        '.go': 'go',
-        '.rs': 'rust',
-        '.swift': 'swift',
-        '.kt': 'kotlin',
-        '.sh': 'bash',
-        '.yaml': 'yaml',
-        '.yml': 'yaml',
-        '.xml': 'xml',
-    }
-    
-    _, ext = os.path.splitext(str(filepath)) # Ensure filepath is string
-    return extension_map.get(ext.lower(), None)

evals/diff-edits/database.md

@@ -1,96 +0,0 @@
-# Diff Edit Evaluation Database Schema
-
-This document provides an overview of the SQLite database schema used for the diff edit evaluation suite. The database is designed to capture every aspect of the evaluation runs in a structured way, allowing for detailed, multi-dimensional analysis and ensuring full reproducibility of our findings.
-
-## Data Model Overview
-
-The database is composed of several interconnected tables that work together to provide a comprehensive picture of each evaluation. The core of the model revolves around `runs`, `cases`, and `results`.
-
-### `runs`
-
-A `run` represents a single, top-level execution of the evaluation script (e.g., one invocation of `npm run diff-eval`). It serves as the main container for a complete benchmark session.
-
--   **Purpose**: To group all the results from a single benchmark execution, allowing for high-level comparison between different runs over time.
--   **Key Columns**:
-    -   `run_id`: A unique identifier for the entire run.
-    -   `description`: A human-readable summary of the run's configuration (e.g., which models were tested, how many cases, etc.).
-    -   `system_prompt_hash`: A foreign key that links this run to the specific system prompt that was used, ensuring we can track performance changes based on prompt modifications.
-
-### `cases`
-
-A `case` represents a single test scenario that is presented to a model. It corresponds to one of the JSON files in the `cases/` directory and links that static definition to a specific benchmark `run`.
-
--   **Purpose**: To track the individual test scenarios within a given run.
--   **Key Columns**:
-    -   `case_id`: A unique identifier for the case *within* a specific run.
-    -   `run_id`: A foreign key linking back to the parent `run`.
-    -   `task_id`: The original, persistent identifier for the test case (typically from the JSON filename).
-    -   `file_hash`: A foreign key linking to the original, un-edited file content for this case.
-
-### `results`
-
-This is the most granular and important table in the database. A `result` represents the outcome of a single attempt by a specific model on a specific case.
-
--   **Purpose**: To store the detailed outcome of every single model attempt, providing the raw data for all quantitative and qualitative analysis.
--   **Key Columns**:
-    -   `result_id`: The primary key for the result.
-    -   `run_id`, `case_id`, `model_id`, `processing_functions_hash`: A set of foreign keys that precisely situate this result within the context of a specific run, case, model, and set of helper functions.
-    -   `succeeded`: A boolean indicating if the generated diff was applied successfully.
-    -   `error_enum`: A numeric code representing the specific type of error if the attempt failed (e.g., `1` for `no_tool_calls`, `7` for `wrong_file_edited`).
-    -   `num_edits`, `num_lines_deleted`, `num_lines_added`: Quantitative metrics about the structure of the generated diff.
-    -   `time_to_first_token_ms`, `time_to_first_edit_ms`, `time_round_trip_ms`: High-precision timing data to measure model latency.
-    -   `cost_usd`, `completion_tokens`: Cost and token usage metrics for efficiency analysis.
-    -   `raw_model_output`, `file_edited_hash`, `parsed_tool_call_json`: The rich, qualitative data. This includes the model's full, raw response and the parsed tool calls, which are invaluable for debugging and understanding the model's reasoning.
-
----
-
-## Supporting Tables
-
-The following tables store versioned, deduplicated content to ensure data integrity and efficiency.
-
-### `system_prompts`
-
--   **Purpose**: Stores the versioned content of the system prompts used in evaluations.
--   **Key Columns**:
-    -   `hash`: A unique hash of the prompt's content, which acts as the primary key. This prevents duplicate storage of the same prompt.
-    -   `name`: A human-readable name for the prompt (e.g., `basicSystemPrompt`, `claude4SystemPrompt`).
-    -   `content`: The full text of the system prompt.
-
-### `processing_functions`
-
--   **Purpose**: Stores the versioned combinations of parsing and diff-editing functions.
--   **Key Columns**:
-    -   `hash`: A unique hash of the function combination name.
-    -   `name`: A human-readable name (e.g., `parseV2-diffV2`).
-    -   `parsing_function`: The name of the function used to parse the model's output.
-    -   `diff_edit_function`: The name of the function used to apply the diff.
-
-### `files`
-
--   **Purpose**: Stores the content of all files involved in the tests, including the original source files and the diffs generated by the models.
--   **Key Columns**:
-    -   `hash`: A content-based hash of the file, ensuring that identical files are only stored once.
-    -   `filepath`: The original path of the file.
-    -   `content`: The full content of the file.
-
-## The Bigger Picture
-
-This relational schema provides a powerful foundation for sophisticated analysis. It moves beyond simple pass/fail metrics and allows us to explore the nuanced interactions between models, prompts, and the code they operate on. With this database, we can answer critical questions like:
-
--   "How does prompt engineering affect not just success rate, but also latency and cost?"
--   "Are certain models more prone to specific types of errors (e.g., hallucinating file paths vs. failing to call a tool)?"
--   "Which of our internal diffing algorithms is the most robust against a wide range of model-generated edits?"
-
-Ultimately, this data model enables us to move from simply *measuring* performance to truly *understanding* it, providing the insights needed to build more capable and reliable AI engineering systems.
-
----
-
-## Viewing the Full Schema
-
-To see the most up-to-date and detailed schema for the database, you can use the `sqlite3` command-line tool. From the `evals/diff-edits` directory, run the following command:
-
-```bash
-sqlite3 evals.db .schema
-```
-
-This will print the complete `CREATE TABLE` statements for all tables in the database, providing a definitive reference for the database structure.

evals/diff-edits/database/client.ts

@@ -1,135 +0,0 @@
-import Database from 'better-sqlite3';
-import * as fs from 'fs';
-import * as path from 'path';
-import * as crypto from 'crypto';
-
-export class DatabaseClient {
-  private static instance: DatabaseClient;
-  private db: Database.Database;
-  private dbPath: string;
-
-  private constructor() {
-    // Get database path from environment or use default
-    this.dbPath = process.env.DIFF_EVALS_DB_PATH || path.join(__dirname, '../evals.db');
-    
-    // Ensure directory exists
-    const dbDir = path.dirname(this.dbPath);
-    if (!fs.existsSync(dbDir)) {
-      fs.mkdirSync(dbDir, { recursive: true });
-    }
-
-    // Initialize database connection
-    this.db = new Database(this.dbPath);
-    
-    // Enable WAL mode for concurrent access
-    this.db.pragma('journal_mode = WAL');
-    
-    // Enable foreign key constraints
-    this.db.pragma('foreign_keys = ON');
-    
-    // Initialize schema if needed
-    this.initializeSchema();
-  }
-
-  static getInstance(): DatabaseClient {
-    if (!DatabaseClient.instance) {
-      DatabaseClient.instance = new DatabaseClient();
-    }
-    return DatabaseClient.instance;
-  }
-
-  private initializeSchema(): void {
-    // Check if tables exist by trying to query one of them
-    try {
-      this.db.prepare('SELECT COUNT(*) FROM system_prompts LIMIT 1').get();
-      // If we get here, tables exist
-      return;
-    } catch (error) {
-      // Tables don't exist, create them
-      console.log('Initializing database schema...');
-      this.createTables();
-    }
-  }
-
-  private createTables(): void {
-    const schemaPath = path.join(__dirname, 'schema.sql');
-    const schema = fs.readFileSync(schemaPath, 'utf8');
-    
-    // Execute the entire schema as one block
-    this.db.transaction(() => {
-      this.db.exec(schema);
-    })();
-
-    console.log('Database schema initialized successfully');
-  }
-
-  getDatabase(): Database.Database {
-    return this.db;
-  }
-
-  getDatabasePath(): string {
-    return this.dbPath;
-  }
-
-  // Utility method to generate SHA-256 hash
-  static generateHash(content: string): string {
-    return crypto.createHash('sha256').update(content).digest('hex');
-  }
-
-  // Utility method to generate UUID-like ID
-  static generateId(): string {
-    return crypto.randomUUID();
-  }
-
-  // Transaction wrapper
-  transaction<T>(fn: () => T): T {
-    return this.db.transaction(fn)();
-  }
-
-  // Close database connection (for cleanup)
-  close(): void {
-    if (this.db) {
-      this.db.close();
-    }
-  }
-
-  // Get database info
-  getInfo(): { path: string; size: number; tables: string[] } {
-    const stats = fs.statSync(this.dbPath);
-    const tables = this.db
-      .prepare("SELECT name FROM sqlite_master WHERE type='table' ORDER BY name")
-      .all()
-      .map((row: any) => row.name);
-
-    return {
-      path: this.dbPath,
-      size: stats.size,
-      tables
-    };
-  }
-
-  // Vacuum database (cleanup and optimize)
-  vacuum(): void {
-    this.db.exec('VACUUM');
-  }
-
-  // Get database statistics
-  getStats(): { [tableName: string]: number } {
-    const tables = ['system_prompts', 'processing_functions', 'files', 'runs', 'cases', 'results'];
-    const stats: { [tableName: string]: number } = {};
-
-    for (const table of tables) {
-      try {
-        const result = this.db.prepare(`SELECT COUNT(*) as count FROM ${table}`).get() as { count: number };
-        stats[table] = result.count;
-      } catch (error) {
-        stats[table] = 0;
-      }
-    }
-
-    return stats;
-  }
-}
-
-// Export singleton instance getter
-export const getDatabase = () => DatabaseClient.getInstance();

evals/diff-edits/database/index.ts

@@ -1,23 +0,0 @@
-// Main database module exports
-export { DatabaseClient, getDatabase } from './client';
-export * from './types';
-export * from './operations';
-export * from './queries';
-
-// Re-export commonly used functions for convenience
-export {
-  upsertSystemPrompt,
-  upsertProcessingFunctions,
-  upsertFile,
-  createBenchmarkRun,
-  createCase,
-  insertResult,
-  getRunStats
-} from './operations';
-
-export {
-  getSuccessRatesByModel,
-  getModelComparisons,
-  getDatabaseSummary,
-  getErrorDistribution
-} from './queries';

evals/diff-edits/database/operations.ts

@@ -1,348 +0,0 @@
-import { DatabaseClient } from './client';
-import {
-  SystemPrompt,
-  ProcessingFunctions,
-  FileRecord,
-  BenchmarkRun,
-  Case,
-  Result,
-  CreateSystemPromptInput,
-  CreateProcessingFunctionsInput,
-  CreateFileInput,
-  CreateBenchmarkRunInput,
-  CreateCaseInput,
-  CreateResultInput
-} from './types';
-
-const db = DatabaseClient.getInstance();
-
-// System Prompts Operations
-export async function upsertSystemPrompt(input: CreateSystemPromptInput): Promise<string> {
-  const hash = DatabaseClient.generateHash(input.content);
-  
-  const stmt = db.getDatabase().prepare(`
-    INSERT OR IGNORE INTO system_prompts (hash, name, content)
-    VALUES (?, ?, ?)
-  `);
-  
-  stmt.run(hash, input.name, input.content);
-  return hash;
-}
-
-export async function getSystemPromptByHash(hash: string): Promise<SystemPrompt | null> {
-  const stmt = db.getDatabase().prepare(`
-    SELECT * FROM system_prompts WHERE hash = ?
-  `);
-  
-  const result = stmt.get(hash) as SystemPrompt | undefined;
-  return result || null;
-}
-
-// Processing Functions Operations
-export async function upsertProcessingFunctions(input: CreateProcessingFunctionsInput): Promise<string> {
-  const hash = DatabaseClient.generateHash(input.parsing_function + input.diff_edit_function);
-  
-  const stmt = db.getDatabase().prepare(`
-    INSERT OR IGNORE INTO processing_functions (hash, name, parsing_function, diff_edit_function)
-    VALUES (?, ?, ?, ?)
-  `);
-  
-  stmt.run(hash, input.name, input.parsing_function, input.diff_edit_function);
-  return hash;
-}
-
-export async function getProcessingFunctionsByHash(hash: string): Promise<ProcessingFunctions | null> {
-  const stmt = db.getDatabase().prepare(`
-    SELECT * FROM processing_functions WHERE hash = ?
-  `);
-  
-  const result = stmt.get(hash) as ProcessingFunctions | undefined;
-  return result || null;
-}
-
-// Files Operations
-export async function upsertFile(input: CreateFileInput): Promise<string> {
-  const hash = DatabaseClient.generateHash(input.content);
-  
-  const stmt = db.getDatabase().prepare(`
-    INSERT OR IGNORE INTO files (hash, filepath, content, tokens)
-    VALUES (?, ?, ?, ?)
-  `);
-  
-  stmt.run(hash, input.filepath, input.content, input.tokens || null);
-  return hash;
-}
-
-export async function getFileByHash(hash: string): Promise<FileRecord | null> {
-  const stmt = db.getDatabase().prepare(`
-    SELECT * FROM files WHERE hash = ?
-  `);
-  
-  const result = stmt.get(hash) as FileRecord | undefined;
-  return result || null;
-}
-
-// Benchmark Runs Operations
-export async function createBenchmarkRun(input: CreateBenchmarkRunInput): Promise<string> {
-  const runId = DatabaseClient.generateId();
-  
-  const stmt = db.getDatabase().prepare(`
-    INSERT INTO runs (run_id, description, system_prompt_hash)
-    VALUES (?, ?, ?)
-  `);
-  
-  stmt.run(runId, input.description || null, input.system_prompt_hash);
-  return runId;
-}
-
-export async function getBenchmarkRun(runId: string): Promise<BenchmarkRun | null> {
-  const stmt = db.getDatabase().prepare(`
-    SELECT * FROM runs WHERE run_id = ?
-  `);
-  
-  const result = stmt.get(runId) as BenchmarkRun | undefined;
-  return result || null;
-}
-
-export async function getAllBenchmarkRuns(): Promise<BenchmarkRun[]> {
-  const stmt = db.getDatabase().prepare(`
-    SELECT * FROM runs ORDER BY created_at DESC
-  `);
-  
-  return stmt.all() as BenchmarkRun[];
-}
-
-// Cases Operations
-export async function createCase(input: CreateCaseInput): Promise<string> {
-  const caseId = DatabaseClient.generateId();
-  
-  const stmt = db.getDatabase().prepare(`
-    INSERT INTO cases (case_id, run_id, description, system_prompt_hash, task_id, tokens_in_context, file_hash)
-    VALUES (?, ?, ?, ?, ?, ?, ?)
-  `);
-  
-  stmt.run(
-    caseId,
-    input.run_id,
-    input.description,
-    input.system_prompt_hash,
-    input.task_id,
-    input.tokens_in_context,
-    input.file_hash || null
-  );
-  
-  return caseId;
-}
-
-export async function getCasesByRun(runId: string): Promise<Case[]> {
-  const stmt = db.getDatabase().prepare(`
-    SELECT * FROM cases WHERE run_id = ? ORDER BY created_at
-  `);
-  
-  return stmt.all(runId) as Case[];
-}
-
-export async function getCaseById(caseId: string): Promise<Case | null> {
-  const stmt = db.getDatabase().prepare(`
-    SELECT * FROM cases WHERE case_id = ?
-  `);
-  
-  const result = stmt.get(caseId) as Case | undefined;
-  return result || null;
-}
-
-// Results Operations
-export async function insertResult(input: CreateResultInput): Promise<string> {
-  const resultId = DatabaseClient.generateId();
-  
-  const stmt = db.getDatabase().prepare(`
-    INSERT INTO results (
-      result_id, run_id, case_id, model_id, processing_functions_hash,
-      succeeded, error_enum, num_edits, num_lines_deleted, num_lines_added,
-      time_to_first_token_ms, time_to_first_edit_ms, time_round_trip_ms,
-      cost_usd, completion_tokens, raw_model_output, file_edited_hash,
-      parsed_tool_call_json
-    ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
-  `);
-  
-  stmt.run(
-    resultId,
-    input.run_id,
-    input.case_id,
-    input.model_id,
-    input.processing_functions_hash,
-    input.succeeded ? 1 : 0, // Convert boolean to integer
-    input.error_enum || null,
-    input.num_edits || null,
-    input.num_lines_deleted || null,
-    input.num_lines_added || null,
-    input.time_to_first_token_ms || null,
-    input.time_to_first_edit_ms || null,
-    input.time_round_trip_ms || null,
-    input.cost_usd || null,
-    input.completion_tokens || null,
-    input.raw_model_output || null,
-    input.file_edited_hash || null,
-    input.parsed_tool_call_json || null
-  );
-  
-  return resultId;
-}
-
-export async function getResultsByRun(runId: string): Promise<Result[]> {
-  const stmt = db.getDatabase().prepare(`
-    SELECT * FROM results WHERE run_id = ? ORDER BY created_at
-  `);
-  
-  return stmt.all(runId) as Result[];
-}
-
-export async function getResultsByCase(caseId: string): Promise<Result[]> {
-  const stmt = db.getDatabase().prepare(`
-    SELECT * FROM results WHERE case_id = ? ORDER BY created_at
-  `);
-  
-  return stmt.all(caseId) as Result[];
-}
-
-export async function getResultById(resultId: string): Promise<Result | null> {
-  const stmt = db.getDatabase().prepare(`
-    SELECT * FROM results WHERE result_id = ?
-  `);
-  
-  const result = stmt.get(resultId) as Result | undefined;
-  return result || null;
-}
-
-// Batch operations for performance
-export async function insertResultsBatch(inputs: CreateResultInput[]): Promise<string[]> {
-  const stmt = db.getDatabase().prepare(`
-    INSERT INTO results (
-      result_id, run_id, case_id, model_id, processing_functions_hash,
-      succeeded, error_enum, num_edits, num_lines_deleted, num_lines_added,
-      time_to_first_token_ms, time_to_first_edit_ms, time_round_trip_ms,
-      cost_usd, completion_tokens, raw_model_output, file_edited_hash,
-      parsed_tool_call_json
-    ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
-  `);
-  
-  return db.transaction(() => {
-    const resultIds: string[] = [];
-    
-    for (const input of inputs) {
-      const resultId = DatabaseClient.generateId();
-      
-      stmt.run(
-        resultId,
-        input.run_id,
-        input.case_id,
-        input.model_id,
-        input.processing_functions_hash,
-        input.succeeded ? 1 : 0, // Convert boolean to integer
-        input.error_enum || null,
-        input.num_edits || null,
-        input.num_lines_deleted || null,
-        input.num_lines_added || null,
-        input.time_to_first_token_ms || null,
-        input.time_to_first_edit_ms || null,
-        input.time_round_trip_ms || null,
-        input.cost_usd || null,
-        input.completion_tokens || null,
-        input.raw_model_output || null,
-        input.file_edited_hash || null,
-        input.parsed_tool_call_json || null
-      );
-      
-      resultIds.push(resultId);
-    }
-    
-    return resultIds;
-  });
-}
-
-export async function createCasesBatch(inputs: CreateCaseInput[]): Promise<string[]> {
-  const stmt = db.getDatabase().prepare(`
-    INSERT INTO cases (case_id, run_id, description, system_prompt_hash, task_id, tokens_in_context)
-    VALUES (?, ?, ?, ?, ?, ?)
-  `);
-  
-  return db.transaction(() => {
-    const caseIds: string[] = [];
-    
-    for (const input of inputs) {
-      const caseId = DatabaseClient.generateId();
-      
-      stmt.run(
-        caseId,
-        input.run_id,
-        input.description,
-        input.system_prompt_hash,
-        input.task_id,
-        input.tokens_in_context
-      );
-      
-      caseIds.push(caseId);
-    }
-    
-    return caseIds;
-  });
-}
-
-// Utility functions
-export async function getRunStats(runId: string): Promise<{
-  total_cases: number;
-  total_results: number;
-  success_rate: number;
-  avg_cost: number;
-  avg_latency: number;
-}> {
-  const stmt = db.getDatabase().prepare(`
-    SELECT 
-      COUNT(DISTINCT c.case_id) as total_cases,
-      COUNT(r.result_id) as total_results,
-      AVG(CASE WHEN r.succeeded THEN 1.0 ELSE 0.0 END) as success_rate,
-      AVG(r.cost_usd) as avg_cost,
-      AVG(r.time_round_trip_ms) as avg_latency
-    FROM cases c
-    LEFT JOIN results r ON c.case_id = r.case_id
-    WHERE c.run_id = ?
-  `);
-  
-  const result = stmt.get(runId) as any;
-  return {
-    total_cases: result.total_cases || 0,
-    total_results: result.total_results || 0,
-    success_rate: result.success_rate || 0,
-    avg_cost: result.avg_cost || 0,
-    avg_latency: result.avg_latency || 0
-  };
-}
-
-// Count valid attempts for a specific case and model
-export async function getValidAttemptCount(caseId: string, modelId: string): Promise<number> {
-  const stmt = db.getDatabase().prepare(`
-    SELECT COUNT(*) as count
-    FROM results 
-    WHERE case_id = ? 
-      AND model_id = ?
-      AND error_enum NOT IN (1, 6, 7)  -- Exclude: no_tool_calls, wrong_tool_call, wrong_file_edited
-  `);
-  
-  const result = stmt.get(caseId, modelId) as { count: number };
-  return result.count;
-}
-
-// Get valid results for a specific case and model (for analysis)
-export async function getValidResults(caseId: string, modelId: string, limit?: number): Promise<Result[]> {
-  const limitClause = limit ? `LIMIT ${limit}` : '';
-  const stmt = db.getDatabase().prepare(`
-    SELECT * FROM results 
-    WHERE case_id = ? 
-      AND model_id = ?
-      AND error_enum NOT IN (1, 6, 7)  -- Only valid attempts
-    ORDER BY created_at
-    ${limitClause}
-  `);
-  
-  return stmt.all(caseId, modelId) as Result[];
-}

evals/diff-edits/database/queries.ts

@@ -1,309 +0,0 @@
-import { DatabaseClient } from './client';
-import {
-  ModelSuccessRate,
-  ModelLatency,
-  CostAnalysis,
-  ErrorDistribution,
-  FailedCase,
-  PerformanceTrend,
-  ModelComparison
-} from './types';
-
-const db = DatabaseClient.getInstance();
-
-// Performance analysis queries
-export async function getSuccessRatesByModel(): Promise<ModelSuccessRate[]> {
-  const stmt = db.getDatabase().prepare(`
-    SELECT 
-      model_id,
-      COUNT(*) as total_runs,
-      SUM(CASE WHEN succeeded THEN 1 ELSE 0 END) as successful_runs,
-      ROUND(AVG(CASE WHEN succeeded THEN 1.0 ELSE 0.0 END) * 100, 2) as success_rate
-    FROM results
-    WHERE error_enum NOT IN (1, 6, 7) OR error_enum IS NULL  -- Exclude: no_tool_calls, wrong_tool_call, wrong_file_edited
-    GROUP BY model_id
-    ORDER BY success_rate DESC, total_runs DESC
-  `);
-  
-  return stmt.all() as ModelSuccessRate[];
-}
-
-export async function getAverageLatencyByModel(): Promise<ModelLatency[]> {
-  const stmt = db.getDatabase().prepare(`
-    SELECT 
-      model_id,
-      ROUND(AVG(time_to_first_token_ms), 2) as avg_time_to_first_token_ms,
-      ROUND(AVG(time_to_first_edit_ms), 2) as avg_time_to_first_edit_ms,
-      ROUND(AVG(time_round_trip_ms), 2) as avg_time_round_trip_ms
-    FROM results
-    WHERE time_to_first_token_ms IS NOT NULL
-    GROUP BY model_id
-    ORDER BY avg_time_round_trip_ms ASC
-  `);
-  
-  return stmt.all() as ModelLatency[];
-}
-
-export async function getCostAnalysisByRun(): Promise<CostAnalysis[]> {
-  const stmt = db.getDatabase().prepare(`
-    SELECT 
-      run_id,
-      model_id,
-      ROUND(SUM(cost_usd), 4) as total_cost_usd,
-      ROUND(AVG(cost_usd), 4) as avg_cost_per_case,
-      SUM(completion_tokens) as total_completion_tokens
-    FROM results
-    WHERE cost_usd IS NOT NULL
-    GROUP BY run_id, model_id
-    ORDER BY total_cost_usd DESC
-  `);
-  
-  return stmt.all() as CostAnalysis[];
-}
-
-// Error analysis queries
-export async function getErrorDistribution(): Promise<ErrorDistribution[]> {
-  const stmt = db.getDatabase().prepare(`
-    SELECT 
-      error_enum,
-      COUNT(*) as count,
-      ROUND(COUNT(*) * 100.0 / (SELECT COUNT(*) FROM results WHERE succeeded = 0), 2) as percentage
-    FROM results
-    WHERE succeeded = 0 AND error_enum IS NOT NULL
-    GROUP BY error_enum
-    ORDER BY count DESC
-  `);
-  
-  return stmt.all() as ErrorDistribution[];
-}
-
-export async function getFailedCasesByError(errorEnum?: number): Promise<FailedCase[]> {
-  let query = `
-    SELECT 
-      r.case_id,
-      r.model_id,
-      r.error_enum,
-      c.description,
-      r.raw_model_output
-    FROM results r
-    JOIN cases c ON r.case_id = c.case_id
-    WHERE r.succeeded = 0
-  `;
-  
-  const params: any[] = [];
-  if (errorEnum !== undefined) {
-    query += ` AND r.error_enum = ?`;
-    params.push(errorEnum);
-  }
-  
-  query += ` ORDER BY r.created_at DESC LIMIT 100`;
-  
-  const stmt = db.getDatabase().prepare(query);
-  return stmt.all(...params) as FailedCase[];
-}
-
-// Trend analysis queries
-export async function getPerformanceTrends(days: number = 30): Promise<PerformanceTrend[]> {
-  const stmt = db.getDatabase().prepare(`
-    SELECT 
-      DATE(r.created_at) as date,
-      r.model_id,
-      ROUND(AVG(CASE WHEN r.succeeded THEN 1.0 ELSE 0.0 END) * 100, 2) as success_rate,
-      ROUND(AVG(r.time_round_trip_ms), 2) as avg_latency_ms,
-      ROUND(AVG(r.cost_usd), 4) as avg_cost_usd
-    FROM results r
-    WHERE r.created_at >= datetime('now', '-' || ? || ' days')
-      AND (r.error_enum NOT IN (1, 6, 7) OR r.error_enum IS NULL)  -- Exclude: no_tool_calls, wrong_tool_call, wrong_file_edited
-    GROUP BY DATE(r.created_at), r.model_id
-    ORDER BY date DESC, model_id
-  `);
-  
-  return stmt.all(days) as PerformanceTrend[];
-}
-
-export async function getModelComparisons(): Promise<ModelComparison[]> {
-  const stmt = db.getDatabase().prepare(`
-    SELECT 
-      model_id,
-      ROUND(AVG(CASE WHEN succeeded THEN 1.0 ELSE 0.0 END) * 100, 2) as success_rate,
-      ROUND(AVG(time_round_trip_ms), 2) as avg_latency_ms,
-      ROUND(AVG(cost_usd), 4) as avg_cost_usd,
-      COUNT(*) as total_runs
-    FROM results
-    WHERE error_enum NOT IN (1, 6, 7) OR error_enum IS NULL  -- Exclude: no_tool_calls, wrong_tool_call, wrong_file_edited
-    GROUP BY model_id
-    HAVING total_runs >= 10
-    ORDER BY success_rate DESC, avg_latency_ms ASC
-  `);
-  
-  return stmt.all() as ModelComparison[];
-}
-
-// Advanced analysis queries
-export async function getTopPerformingCases(limit: number = 10): Promise<Array<{
-  case_id: string;
-  description: string;
-  success_rate: number;
-  avg_latency_ms: number;
-  total_runs: number;
-}>> {
-  const stmt = db.getDatabase().prepare(`
-    SELECT 
-      c.case_id,
-      c.description,
-      ROUND(AVG(CASE WHEN r.succeeded THEN 1.0 ELSE 0.0 END) * 100, 2) as success_rate,
-      ROUND(AVG(r.time_round_trip_ms), 2) as avg_latency_ms,
-      COUNT(r.result_id) as total_runs
-    FROM cases c
-    JOIN results r ON c.case_id = r.case_id
-    WHERE r.error_enum NOT IN (1, 6, 7) OR r.error_enum IS NULL  -- Exclude: no_tool_calls, wrong_tool_call, wrong_file_edited
-    GROUP BY c.case_id, c.description
-    HAVING total_runs >= 5
-    ORDER BY success_rate DESC, avg_latency_ms ASC
-    LIMIT ?
-  `);
-  
-  return stmt.all(limit) as Array<{
-    case_id: string;
-    description: string;
-    success_rate: number;
-    avg_latency_ms: number;
-    total_runs: number;
-  }>;
-}
-
-export async function getWorstPerformingCases(limit: number = 10): Promise<Array<{
-  case_id: string;
-  description: string;
-  success_rate: number;
-  avg_latency_ms: number;
-  total_runs: number;
-}>> {
-  const stmt = db.getDatabase().prepare(`
-    SELECT 
-      c.case_id,
-      c.description,
-      ROUND(AVG(CASE WHEN r.succeeded THEN 1.0 ELSE 0.0 END) * 100, 2) as success_rate,
-      ROUND(AVG(r.time_round_trip_ms), 2) as avg_latency_ms,
-      COUNT(r.result_id) as total_runs
-    FROM cases c
-    JOIN results r ON c.case_id = r.case_id
-    WHERE r.error_enum NOT IN (1, 6, 7) OR r.error_enum IS NULL  -- Exclude: no_tool_calls, wrong_tool_call, wrong_file_edited
-    GROUP BY c.case_id, c.description
-    HAVING total_runs >= 5
-    ORDER BY success_rate ASC, avg_latency_ms DESC
-    LIMIT ?
-  `);
-  
-  return stmt.all(limit) as Array<{
-    case_id: string;
-    description: string;
-    success_rate: number;
-    avg_latency_ms: number;
-    total_runs: number;
-  }>;
-}
-
-export async function getModelPerformanceByTimeOfDay(): Promise<Array<{
-  model_id: string;
-  hour: number;
-  success_rate: number;
-  avg_latency_ms: number;
-  total_runs: number;
-}>> {
-  const stmt = db.getDatabase().prepare(`
-    SELECT 
-      model_id,
-      CAST(strftime('%H', created_at) AS INTEGER) as hour,
-      ROUND(AVG(CASE WHEN succeeded THEN 1.0 ELSE 0.0 END) * 100, 2) as success_rate,
-      ROUND(AVG(time_round_trip_ms), 2) as avg_latency_ms,
-      COUNT(*) as total_runs
-    FROM results
-    GROUP BY model_id, hour
-    HAVING total_runs >= 5
-    ORDER BY model_id, hour
-  `);
-  
-  return stmt.all() as Array<{
-    model_id: string;
-    hour: number;
-    success_rate: number;
-    avg_latency_ms: number;
-    total_runs: number;
-  }>;
-}
-
-export async function getRunComparison(runId1: string, runId2: string): Promise<{
-  run1: { run_id: string; success_rate: number; avg_latency_ms: number; avg_cost_usd: number; total_cases: number };
-  run2: { run_id: string; success_rate: number; avg_latency_ms: number; avg_cost_usd: number; total_cases: number };
-}> {
-  const stmt = db.getDatabase().prepare(`
-    SELECT 
-      run_id,
-      ROUND(AVG(CASE WHEN succeeded THEN 1.0 ELSE 0.0 END) * 100, 2) as success_rate,
-      ROUND(AVG(time_round_trip_ms), 2) as avg_latency_ms,
-      ROUND(AVG(cost_usd), 4) as avg_cost_usd,
-      COUNT(DISTINCT case_id) as total_cases
-    FROM results
-    WHERE run_id IN (?, ?)
-    GROUP BY run_id
-  `);
-  
-  const results = stmt.all(runId1, runId2) as Array<{
-    run_id: string;
-    success_rate: number;
-    avg_latency_ms: number;
-    avg_cost_usd: number;
-    total_cases: number;
-  }>;
-  
-  const run1 = results.find(r => r.run_id === runId1);
-  const run2 = results.find(r => r.run_id === runId2);
-  
-  if (!run1 || !run2) {
-    throw new Error('One or both runs not found');
-  }
-  
-  return { run1, run2 };
-}
-
-// Summary statistics
-export async function getDatabaseSummary(): Promise<{
-  total_runs: number;
-  total_cases: number;
-  total_results: number;
-  valid_results: number;
-  unique_models: number;
-  overall_success_rate: number;
-  date_range: { earliest: string; latest: string };
-}> {
-  const stmt = db.getDatabase().prepare(`
-    SELECT 
-      (SELECT COUNT(*) FROM runs) as total_runs,
-      (SELECT COUNT(*) FROM cases) as total_cases,
-      (SELECT COUNT(*) FROM results) as total_results,
-      (SELECT COUNT(*) FROM results WHERE error_enum NOT IN (1, 6, 7) OR error_enum IS NULL) as valid_results,
-      (SELECT COUNT(DISTINCT model_id) FROM results) as unique_models,
-      (SELECT ROUND(AVG(CASE WHEN succeeded THEN 1.0 ELSE 0.0 END) * 100, 2) 
-       FROM results 
-       WHERE error_enum NOT IN (1, 6, 7) OR error_enum IS NULL) as overall_success_rate,
-      (SELECT MIN(created_at) FROM results) as earliest,
-      (SELECT MAX(created_at) FROM results) as latest
-    FROM results
-    LIMIT 1
-  `);
-  
-  const result = stmt.get() as any;
-  return {
-    total_runs: result.total_runs || 0,
-    total_cases: result.total_cases || 0,
-    total_results: result.total_results || 0,
-    valid_results: result.valid_results || 0,
-    unique_models: result.unique_models || 0,
-    overall_success_rate: result.overall_success_rate || 0,
-    date_range: {
-      earliest: result.earliest || '',
-      latest: result.latest || ''
-    }
-  };
-}

evals/diff-edits/database/schema.sql

@@ -1,78 +0,0 @@
-PRAGMA foreign_keys = ON;
-
-CREATE TABLE system_prompts (
-    hash TEXT PRIMARY KEY,
-    name TEXT NOT NULL,
-    content TEXT NOT NULL,
-    created_at DATETIME DEFAULT CURRENT_TIMESTAMP
-);
-
-CREATE TABLE processing_functions (
-    hash TEXT PRIMARY KEY,
-    name TEXT NOT NULL,
-    parsing_function TEXT NOT NULL,
-    diff_edit_function TEXT NOT NULL,
-    created_at DATETIME DEFAULT CURRENT_TIMESTAMP
-);
-
-CREATE TABLE files (
-    hash TEXT PRIMARY KEY,
-    filepath TEXT NOT NULL,
-    content TEXT NOT NULL,
-    tokens INTEGER,
-    created_at DATETIME DEFAULT CURRENT_TIMESTAMP
-);
-
-CREATE TABLE runs (
-    run_id TEXT PRIMARY KEY,
-    created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
-    description TEXT,
-    system_prompt_hash TEXT NOT NULL,
-    FOREIGN KEY (system_prompt_hash) REFERENCES system_prompts(hash)
-);
-
-CREATE TABLE cases (
-    case_id TEXT PRIMARY KEY,
-    run_id TEXT NOT NULL,
-    created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
-    description TEXT NOT NULL,
-    system_prompt_hash TEXT NOT NULL,
-    task_id TEXT NOT NULL,
-    tokens_in_context INTEGER,
-    file_hash TEXT,
-    FOREIGN KEY (run_id) REFERENCES runs(run_id),
-    FOREIGN KEY (system_prompt_hash) REFERENCES system_prompts(hash),
-    FOREIGN KEY (file_hash) REFERENCES files(hash)
-);
-
-CREATE TABLE results (
-    result_id TEXT PRIMARY KEY,
-    run_id TEXT NOT NULL,
-    case_id TEXT NOT NULL,
-    model_id TEXT NOT NULL,
-    processing_functions_hash TEXT NOT NULL,
-    succeeded BOOLEAN NOT NULL,
-    error_enum INTEGER,
-    num_edits INTEGER,
-    num_lines_deleted INTEGER,
-    num_lines_added INTEGER,
-    time_to_first_token_ms INTEGER,
-    time_to_first_edit_ms INTEGER,
-    time_round_trip_ms INTEGER,
-    cost_usd REAL,
-    completion_tokens INTEGER,
-    raw_model_output TEXT,
-    file_edited_hash TEXT,
-    parsed_tool_call_json TEXT,
-    created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
-    FOREIGN KEY (run_id) REFERENCES runs(run_id),
-    FOREIGN KEY (case_id) REFERENCES cases(case_id),
-    FOREIGN KEY (processing_functions_hash) REFERENCES processing_functions(hash)
-);
-
-CREATE INDEX idx_results_run_model ON results(run_id, model_id);
-CREATE INDEX idx_results_case_model ON results(case_id, model_id);
-CREATE INDEX idx_results_success ON results(succeeded);
-CREATE INDEX idx_cases_run ON cases(run_id);
-CREATE INDEX idx_results_created_at ON results(created_at);
-CREATE INDEX idx_runs_created_at ON runs(created_at);

evals/diff-edits/database/test.ts

@@ -1,53 +0,0 @@
-// Simple test to verify database functionality
-import { getDatabase } from './client';
-import { upsertSystemPrompt, createBenchmarkRun, getDatabaseSummary } from './index';
-
-async function testDatabase() {
-  console.log('Testing database functionality...');
-  
-  try {
-    // Test database connection
-    const db = getDatabase();
-    console.log('✓ Database connection established');
-    console.log('Database path:', db.getDatabasePath());
-    
-    // Test database info
-    const info = db.getInfo();
-    console.log('✓ Database info:', info);
-    
-    // Test database stats
-    const stats = db.getStats();
-    console.log('✓ Database stats:', stats);
-    
-    // Test system prompt creation
-    const systemPromptHash = await upsertSystemPrompt({
-      name: 'test-prompt',
-      content: 'This is a test system prompt for database verification.'
-    });
-    console.log('✓ System prompt created with hash:', systemPromptHash);
-    
-    // Test benchmark run creation
-    const runId = await createBenchmarkRun({
-      description: 'Test run for database verification',
-      system_prompt_hash: systemPromptHash
-    });
-    console.log('✓ Benchmark run created with ID:', runId);
-    
-    // Test database summary
-    const summary = await getDatabaseSummary();
-    console.log('✓ Database summary:', summary);
-    
-    console.log('\n🎉 All database tests passed!');
-    
-  } catch (error) {
-    console.error('❌ Database test failed:', error);
-    process.exit(1);
-  }
-}
-
-// Run test if this file is executed directly
-if (require.main === module) {
-  testDatabase();
-}
-
-export { testDatabase };