changeset

This change allows the `showMessage` host bridge method to pass an array of `items` to the VS Code `window.show*Message` methods. This enables the display of quick pick options in the message box.

.changeset/chatty-turkeys-wait.md

@@ -1,5 +0,0 @@
----
-"claude-dev": patch
----
-
-fix: respect setting litellm models for plan and act

.changeset/curvy-bulldogs-jog.md

@@ -1,5 +0,0 @@
----
-"claude-dev": minor
----
-
-Migrate didBecomeVisible to protobus

.changeset/eight-jokes-punch.md

@@ -1,5 +0,0 @@
----
-"claude-dev": patch
----
-
-Do not clear the input value when changing modes without sending a message

.changeset/hungry-teachers-roll.md

@@ -0,0 +1,5 @@
+---
+"claude-dev": patch
+---
+
+Display user role in organization UI

.changeset/lazy-cherries-search.md

@@ -1,5 +0,0 @@
----
-"claude-dev": patch
----
-
-Prevent reading IS_DEV from the users environment

.changeset/many-zebras-end.md

@@ -0,0 +1,5 @@
+---
+"claude-dev": patch
+---
+
+Fix missing options from window messages.

.changeset/pretty-bikes-tickle.md

@@ -1,5 +0,0 @@
----
-"claude-dev": patch
----
-
-fix(bedrock): remove custom Model encode

.changeset/seven-zebras-grin.md

@@ -1,5 +0,0 @@
----
-"claude-dev": patch
----
-
-Updated Timeline tooltips for `followup` messages and consolidated color retrieval code.

.changeset/thirty-donkeys-whisper.md

@@ -1,5 +0,0 @@
----
-"claude-dev": patch
----
-
-The close task button and delete task button in the task header are now correctly announced by screen readers.

.changeset/twenty-snails-explode.md

@@ -0,0 +1,5 @@
+---
+"claude-dev": patch
+---
+
+Remove state parameter from auth callback link that allows redirect to work when multiple windows are opened

.clinerules/protobuf-development.md

@@ -0,0 +1,89 @@
+# Cline Protobuf Development Guide
+
+This guide outlines how to add new gRPC endpoints for communication between the webview (frontend) and the extension host (backend).
+
+## Overview
+
+Cline uses [Protobuf](https://protobuf.dev/) to define a strongly-typed API, ensuring efficient and type-safe communication. All definitions are in the `/proto` directory. The compiler and plugins are included as project dependencies, so no manual installation is needed.
+
+## Key Concepts & Best Practices
+
+-   **File Structure**: Each feature domain should have its own `.proto` file (e.g., `account.proto`, `task.proto`).
+-   **Message Design**:
+    -   For simple, single-value data, use the shared types in `proto/common.proto` (e.g., `StringRequest`, `Empty`, `Int64Request`). This promotes consistency.
+    -   For complex data structures, define custom messages within the feature's `.proto` file (see `task.proto` for examples like `NewTaskRequest`).
+-   **Naming Conventions**:
+    -   Services: `PascalCaseService` (e.g., `AccountService`).
+    -   RPCs: `camelCase` (e.g., `accountEmailIdentified`).
+    -   Messages: `PascalCase` (e.g., `StringRequest`).
+-   **Streaming**: For server-to-client streaming, use the `stream` keyword on the response type. See `subscribeToAuthCallback` in `account.proto` for an example.
+
+---
+
+## 4-Step Development Workflow
+
+Here’s how to add a new RPC, using `scrollToSettings` as an example.
+
+### 1. Define the RPC in a `.proto` File
+
+Add your service method to the appropriate file in the `proto/` directory.
+
+**File: `proto/ui.proto`**
+```proto
+service UiService {
+  // ... other RPCs
+  // Scrolls to a specific settings section in the settings view
+  rpc scrollToSettings(StringRequest) returns (KeyValuePair);
+}
+```
+Here, we use the common `StringRequest` and `KeyValuePair` types.
+
+### 2. Compile Definitions
+
+After editing a `.proto` file, regenerate the TypeScript code. From the project root, run:
+```bash
+npm run protos
+```
+This command compiles all `.proto` files and outputs the generated code to `src/generated/` and `src/shared/`. Do not edit these generated files manually.
+
+### 3. Implement the Backend Handler
+
+Create the RPC implementation in the backend. Handlers are located in `src/core/controller/[service-name]/`.
+
+**File: `src/core/controller/ui/scrollToSettings.ts`**
+```typescript
+import { Controller } from ".."
+import { StringRequest, KeyValuePair } from "../../../shared/proto/common"
+
+/**
+ * Executes a scroll to settings action
+ * @param controller The controller instance
+ * @param request The request containing the ID of the settings section to scroll to
+ * @returns KeyValuePair with action and value fields for the UI to process
+ */
+export async function scrollToSettings(controller: Controller, request: StringRequest): Promise<KeyValuePair> {
+	return KeyValuePair.create({
+		key: "scrollToSettings",
+		value: request.value || "",
+	})
+}
+```
+
+### 4. Call the RPC from the Webview
+
+Call the new RPC from a React component in `webview-ui/`. The generated client makes this simple.
+
+**File: `webview-ui/src/components/browser/BrowserSettingsMenu.tsx`** (Example)
+```tsx
+import { UiServiceClient } from "../../../services/grpc"
+import { StringRequest } from "../../../../shared/proto/common"
+
+// ... inside a React component
+const handleMenuClick = async () => {
+    try {
+        await UiServiceClient.scrollToSettings(StringRequest.create({ value: "browser" }))
+    } catch (error) {
+        console.error("Error scrolling to browser settings:", error)
+    }
+}
+```

.clinerules/workflows/git-branch-analysis.md

@@ -0,0 +1,75 @@
+# Git Diff Analysis Workflow
+
+## Objective
+Analyze the current branch's changes against main to provide informed insights and context for development decisions.
+
+## Step 1: Gather Git Information
+<important>Do not return any text or conversation other than what is necessary to run these commands</important>
+
+**First, check the expected output size:**
+   ```shell
+   (git branch --show-current && echo "=== STATUS ===" && git status --porcelain | cat && echo "=== COMMIT MESSAGES ===" && git log main..HEAD --oneline | cat && echo "=== CHANGED FILES ===" && git diff main --name-only | cat && echo "=== FULL DIFF ===" && git diff main | cat) | wc -l
+   ```
+
+**If the expected line count is greater than 500 lines, use the file-based approach:**
+   ```shell
+   git branch --show-current > cline-git-analysis.temp && echo "=== STATUS ===" >> cline-git-analysis.temp && git status --porcelain >> cline-git-analysis.temp && echo "=== COMMIT MESSAGES ===" >> cline-git-analysis.temp && git log main..HEAD --oneline >> cline-git-analysis.temp && echo "=== CHANGED FILES ===" >> cline-git-analysis.temp && git diff main --name-only >> cline-git-analysis.temp && echo "=== FULL DIFF ===" >> cline-git-analysis.temp && git diff main >> cline-git-analysis.temp
+   ```
+
+   Then, read the file using the read_file tool. After you have read the file but before you proceed with subsequent steps, delete it:
+   ```shell
+   rm cline-git-analysis.temp
+   ```
+
+**If the expected line count is 500 lines or fewer, use the direct approach:**
+   ```shell
+   git branch --show-current && echo "=== STATUS ===" && git status --porcelain | cat && echo "=== COMMIT MESSAGES ===" && git log main..HEAD --oneline | cat && echo "=== CHANGED FILES ===" && git diff main --name-only | cat && echo "=== FULL DIFF ===" && git diff main | cat
+   ```
+
+<important>If using the direct approach, pipe outputs through `cat` to avoid interactive terminals. If the user's shell is not bash/zsh, adjust the command and chaining
+ syntax accordingly.</important>
+
+## Step 2: Silent, Structured Analysis Phase
+- Analyze all git output without providing commentary or narration
+- Read the full diff to understand the scope and nature of changes
+- Identify patterns, architectural modifications, or potential impacts
+- Use `read_file` to examine any related files providing additional context on the changes you have observed
+
+## Step 3: Context Gathering
+- Analyze related code without providing commentary or narration
+- Read relevant related source files if needed for complete understanding
+- Check dependencies, imports, or cross-references spanning the changes
+- Understand the broader codebase context around modifications
+- This additional context gathering should include related backend code, as well as related ui/frontend code
+- You will typically need to analyze at least several files, potentially many, in order to fully complete this step
+- You should not continue reading additional context if you have exhausted more than 60% of your available context window
+- If you have exhausted less than 40% of your context window, you should continue reviewing additional context
+
+## Step 4: Ready for User Interaction
+**Only after completing the full analysis:**
+- Engage with the user based on comprehensive understanding
+- Provide insights about specific modifications and their impacts
+- If you are certain they exist, note potential breaking changes or compatibility issues
+- Answer questions with informed context from the complete change set and context gathering
+- If the user has not provided a question, or the question is insufficient to provide a quality response, ask brief (one sentence) clarifying questions.
+- Only offer recommendations if they are applicable to the user's request and relevant to the changes that you have observed 
+
+## Key Rules
+- **No prose or conversation during git research phase**
+- **No prose or conversation during context gathering phase**
+- **Complete all analysis before any user interaction**
+- **Use gathered information for all subsequent questions and insights**
+- **Focus on understanding the complete picture before discussing**
+
+## Optional: Additional Analysis Commands
+For deeper investigation when needed:
+
+```shell
+# Detailed commit history with author info
+git log main..HEAD --format="%h %s (%an)" | cat
+
+# Change statistics
+git diff main --stat | cat
+
+# Specific file type changes
+git diff main --name-only | grep -E '\.(ts|js|tsx|jsx|py|md)$' | cat

.eslintrc.json

@@ -22,6 +22,7 @@
 		"react-hooks/exhaustive-deps": "off",
 		"eslint-rules/no-protobuf-object-literals": "error",
 		"eslint-rules/no-grpc-client-object-literals": "error",
+		"eslint-rules/no-direct-vscode-api": "warn",
 		"no-restricted-syntax": [
 			"error",
 			{

.github/ISSUE_TEMPLATE/feature_contribution.yml

@@ -0,0 +1,116 @@
+name: 💡 Feature Proposal & Contribution
+description: Propose a new feature or improvement, and optionally offer to implement feature as a contributor
+labels: ["proposal"]
+body:
+    - type: markdown
+      attributes:
+          value: |
+              **Feature Proposal & Contribution for Cline**
+
+              Thank you for proposing a feature or improvement for Cline! This template helps us understand the problem, evaluate the solution, and coordinate implementation.
+
+              **For detailed proposals:** Please provide comprehensive information to enable fast prioritization and discussion.
+              **For contribution offers:** You can indicate your willingness to implement the feature yourself.
+
+              Before submitting:
+              - Search existing [Issues](https://github.com/cline/cline/issues) and [Discussions](https://github.com/cline/cline/discussions) to avoid duplicates
+              - Read the [Contributing Guide](https://github.com/cline/cline/blob/main/CONTRIBUTING.md) if you plan to contribute
+              - Don't start implementation until the proposal is reviewed and approved
+
+    - type: textarea
+      id: problem-description
+      attributes:
+          label: What problem does this solve?
+          description: |
+              Describe the problem clearly from a user's point of view. Focus on why this matters, who it affects, and when it occurs.
+
+              ✅ Good examples:
+              - "LLM provider returns 400 error when nearing the context window instead of truncating"
+              - "Submit button is invisible in dark mode"
+              - "Users can't easily share their Cline configurations with team members"
+
+              ❌ Avoid vague descriptions:
+              - "Performance is bad"
+              - "UI needs work"
+
+              Your description should include:
+              - Who is affected?
+              - When does it happen?
+              - What's the current vs expected behavior?
+              - What is the impact?
+          placeholder: Be specific about the problem, who it affects, and the impact.
+      validations:
+          required: true
+
+    - type: textarea
+      id: proposed-solution
+      attributes:
+          label: What's the proposed solution?
+          description: |
+              Describe how the problem should be solved. Be specific about UX, system behavior, and any flows that would change.
+
+              ✅ Good examples:
+              - "Add error handling immediately after attempting to create the llm stream and retry after manually truncating"
+              - "Update button styling to ensure contrast in all themes"
+              - "Add export/import functionality in settings with JSON format"
+
+              ❌ Avoid vague solutions:
+              - "Improve performance"
+              - "Fix the bug"
+
+              Your solution should include:
+              - What exactly will change?
+              - How will users interact with it?
+              - What's the expected outcome?
+          placeholder: Describe the proposed changes and how they solve the problem.
+      validations:
+          required: false
+
+    - type: dropdown
+      id: contribution-intent
+      attributes:
+          label: Are you interested in implementing this?
+          description: Let us know if you'd like to contribute to this feature
+          options:
+              - "No, just proposing the idea"
+              - "Yes, I'd like to implement this myself"
+              - "Yes, I'd like to collaborate with others"
+              - "Maybe, depending on complexity and guidance"
+      validations:
+          required: false
+
+    - type: textarea
+      id: implementation-approach
+      attributes:
+          label: Implementation approach (if contributing)
+          description: |
+              **Only fill this out if you selected "Yes" above.**
+
+              How do you plan to implement this? Include:
+              - High-level technical approach
+              - Files/components that would be affected
+              - Any new dependencies required
+              - Potential challenges or considerations you've identified
+
+              This helps us provide better guidance and ensures alignment before you start coding.
+          placeholder: "My implementation approach would be..."
+
+    - type: checkboxes
+      id: checklist
+      attributes:
+          label: Proposal checklist
+          options:
+              - label: I've checked for existing issues or related proposals
+                required: true
+              - label: I understand this needs review before implementation can start
+                required: true
+
+    - type: checkboxes
+      id: contribution-checklist
+      attributes:
+          label: Contribution checklist (if contributing)
+          description: Only check these if you plan to contribute
+          options:
+              - label: I've read the [Contributing Guide](https://github.com/cline/cline/blob/main/CONTRIBUTING.md)
+              - label: I'm willing to make changes based on feedback
+              - label: I understand the code review process and requirements

.github/ISSUE_TEMPLATE/feature_proposal.yml

@@ -1,130 +0,0 @@
-name: 📝 Detailed Feature Proposal
-description: Propose a new feature or improvement
-labels: ["proposal"]
-body:
-    - type: markdown
-      attributes:
-          value: |
-              **Feature Proposal for Cline**
-
-              Thank you for creating a feature proposal for Cline! This template is for clear, actionable proposals that define a specific problem and a high-confidence solution. Please provide enough detail to enable fast prioritization, discussion, and execution.
-
-              Detailed proposals will be prioritized, while vague proposals may be closed or require extensive back and forth communication.
-
-              Before submitting:
-              - Search existing [Issues](https://github.com/cline/cline/issues) and [Discussions](https://github.com/cline/cline/discussions) to avoid duplicates
-              - Don’t start implementation until the proposal is reviewed and approved
-
-    - type: textarea
-      id: problem-description
-      attributes:
-          label: What problem does this solve?
-          description: |
-              Describe the problem clearly from a user's point of view. Focus on why this matters, who it affects, and when it occurs.
-
-              ✅ Solid:
-              - "LLM provider returns 400 error when nearing the context window instead of truncating"
-              - "Submit button is invisible in dark mode"
-
-              ❌ Avoid:
-              - "Performance is bad"
-              - "UI needs work"
-
-              Your description should include:
-              - Who is affected?
-              - When does it happen?
-              - What's the current vs expected behavior?
-              - What is the impact?
-          placeholder: Be specific about the problem, who it affects, and the impact.
-      validations:
-          required: true
-
-    - type: textarea
-      id: proposed-solution
-      attributes:
-          label: What’s the proposed solution?
-          description: |
-              Describe how the problem should be solved. Be specific about UX, system behavior, and any flows that would change.
-
-              ✅ Solid:
-              - "Add error handling immediately after attempting to create the llm stream and retry after manually truncating"
-              - "Update button styling to ensure contrast in all themes"
-
-              ❌ Avoid:
-              - "Improve performance"
-              - "Fix the bug"
-
-              Your solution should include:
-              - What exactly will change?
-              - How will users interact with it?
-              - What’s the expected outcome?
-          placeholder: Describe the proposed changes and how they solve the problem.
-      validations:
-          required: true
-
-    - type: textarea
-      id: acceptance-criteria
-      attributes:
-          label: How will we know it works? (Acceptance Criteria)
-          description: Define clear, testable success criteria.
-          placeholder: Provide specific and testable conditions for success.
-      validations:
-          required: true
-
-    - type: textarea
-      id: estimated-effort
-      attributes:
-          label: Estimated effort and complexity
-          description: |
-              Help us understand scope and risks. Include:
-
-              - Size estimate (XS/S/M/L/XL or hours/days)
-              - Why this size? What’s technically involved?
-              - Any tricky parts, refactors, or risks?
-              - Performance or compatibility concerns?
-              - Any dependencies on systems, teams, or libraries?
-          placeholder: Size, reasoning, risks, and dependencies.
-      validations:
-          required: true
-
-    - type: textarea
-      id: technical-considerations
-      attributes:
-          label: Technical considerations, tradeoffs, and/or risks (optional)
-          description: |
-              Include any technical context that helps us evaluate or implement the proposal more effectively.
-
-              You may include:
-              - Architectural changes or required refactors
-              - Performance implications or system-level impacts
-              - Known limitations, risks, or tricky edge cases
-              - Compatibility concerns or migration steps
-              - Alternative approaches you considered and why they were not chosen
-              - Dependencies on other systems, teams, or libraries
-              - Were other approaches considered? Why is this one preferred?
-          placeholder: Technical considerations, tradeoffs, and/or risks.
-
-    - type: textarea
-      id: additional-context
-      attributes:
-          label: Additional context (optional)
-          description: Diagrams, mockups, logs, links, or anything else that helps explain or justify the proposal.
-          placeholder: Diagrams, mockups, logs, links, or anything else that helps explain or justify the proposal.
-
-    - type: checkboxes
-      id: checklist
-      attributes:
-          label: Proposal checklist
-          options:
-              - label: I’ve checked for existing issues or related proposals
-                required: true
-              - label: I understand this needs review before implementation can start
-                required: true
-
-    - type: checkboxes
-      id: willingness-to-contribute
-      attributes:
-          label: Interested in implementing this?
-          description: Optional
-          options:
-              - label: Yes, I’d like to help implement this feature

.github/workflows/stale.yml

@@ -12,7 +12,7 @@ jobs:
             issues: write
             pull-requests: write
         steps:
-            - uses: actions/stale@28ca103
+            - uses: actions/stale@v9
               with:
                   days-before-issue-stale: 60
                   days-before-issue-close: 14

.gitignore

@@ -7,6 +7,7 @@ tmp
 *.vsix
 
 .DS_Store
+.idea
 
 pnpm-lock.yaml
 
@@ -33,6 +34,4 @@ src/shared/proto/host/*.ts
 # Webview
 webview-ui/src/services/grpc-client.ts
 # Host bridge
-src/hosts/vscode/*/methods.ts
-src/hosts/vscode/*/index.ts
-src/hosts/vscode/host-grpc-service-config.ts
+src/hosts/vscode/client/host-grpc-client.ts

.prettierignore

@@ -5,3 +5,4 @@ webview-ui/build/
 package-lock.json
 src/core/prompts/system.ts
 src/core/prompts/model_prompts/claude4.ts
+evals/

.vscode/launch.json

@@ -23,19 +23,20 @@
 			"request": "launch",
 			"runtimeExecutable": "${execPath}",
 			"args": [
+				"--user-data-dir=${workspaceFolder}/dist/tmp/user",
 				"--profile-temp",
-				"--sync",
-				"off",
+				"--sync=off",
 				"--disable-extensions",
 				"--extensionDevelopmentPath=${workspaceFolder}",
 				"${workspaceFolder}"
 			],
 			"outFiles": ["${workspaceFolder}/dist/**/*.js"],
-			"preLaunchTask": "clean-sandbox",
+			"preLaunchTask": "clean-tmp-user",
 			"internalConsoleOptions": "openOnSessionStart",
 			"postDebugTask": "stop",
 			"env": {
 				"IS_DEV": "true",
+				"TEMP_PROFILE": "true",
 				"DEV_WORKSPACE_FOLDER": "${workspaceFolder}"
 			}
 		},
@@ -50,10 +51,11 @@
 			"outFiles": ["${workspaceFolder}/dist-standalone/**/*.js"],
 			"preLaunchTask": "compile-standalone",
 			"env": {
-				"GRPC_TRACE": "all",
-				"GRPC_VERBOSITY": "DEBUG",
+				// Turns on grpc debug log.
+				//"GRPC_TRACE": "all",
+				//"GRPC_VERBOSITY": "DEBUG",
 				"NODE_PATH": "${workspaceFolder}/dist-standalone/node_modules",
-				"CLINE_DIR": "${userHome}/.cline-standalone",
+
 				"HOST_BRIDGE_ADDRESS": "localhost:50052"
 			},
 			"program": "standalone.js"

.vscode/tasks.json

@@ -233,10 +233,10 @@
 			"type": "shell"
 		},
 		{
-			"label": "clean-sandbox",
+			"label": "clean-tmp-user",
 			"type": "shell",
 			"dependsOn": ["watch"],
-			"command": "rm -rf .vscode-dev"
+			"command": "rm -rf ${workspaceFolder}/dist/tmp/user && mkdir -p ${workspaceFolder}/dist/tmp/user"
 		}
 	],
 	"inputs": [

.vscodeignore

@@ -2,20 +2,24 @@
 .vscode/**
 .vscode-test/**
 out/**
+dist-standalone/**
 node_modules/**
 src/**
+standalone/**
 .gitignore
 .yarnrc
 esbuild.js
-vsc-extension-quickstart.md
 **/tsconfig.json
 **/.eslintrc.json
 **/*.map
 **/*.ts
 **/.vscode-test.*
+eslint-rules/**
+.github/**
+.husky/**
 
 # Custom
-demo.gif
+**/demo.gif
 .nvmrc
 .gitattributes
 .prettierignore
@@ -32,15 +36,12 @@ webview-ui/node_modules/**
 
 # Ignore docs
 docs/**
+old_docs/**
 
 # Fix issue where codicons don't get packaged (https://github.com/microsoft/vscode-extension-samples/issues/692)
 !node_modules/@vscode/codicons/dist/codicon.css
 !node_modules/@vscode/codicons/dist/codicon.ttf
 
-# Include KaTeX CSS and fonts for LaTeX rendering
-!webview-ui/node_modules/katex/dist/katex.min.css
-!webview-ui/node_modules/katex/dist/fonts/**
-
 # Include default themes JSON files used in getTheme
 !src/integrations/theme/default-themes/**
 

CHANGELOG.md

@@ -1,5 +1,125 @@
 # Changelog
 
+## [3.19.4]
+
+-   Add ability to choose Chinese endpoint for Moonshot provider
+
+## [3.19.3]
+
+-   Add Moonshot AI provider
+
+## [3.19.2]
+
+-   Show request ID in error messages returned by Cline Accounts API to help debug user reported issues
+
+## [3.19.1]
+
+-   Fix documentation
+
+## [3.19.0]
+
+-   Add Kimi-K2 as a recommended model in the Cline Provider, and route to Together/Groq for 131k context window and high throughput
+-   Added API Key support for Bedrock integration
+
+## [3.18.14]
+
+-   Fix bug where Cline account users logged in with invalid token would not be shown as logged out in webview presentation layer
+
+## [3.18.13]
+
+-   Fix authentication issue where Cline accounts users would keep getting logged out or seeing 'Unexpected API response' errors
+
+## [3.18.12]
+
+-   Fix flaky organization switching behavior in Cline provider that caused UI inconsistencies and double loading
+-   Fix insufficient credits error display to properly show error messages when account balance is too low
+-   Improve credit balance validation and error handling for Cline provider requests
+
+## [3.18.11]
+
+-   Fix authentication issues with Cline provider by ensuring the client always uses the latest auth token
+
+## [3.18.10]
+
+-   Update recommended fast & cheap model to Grok 4 in OpenRouter model picker
+-   Fix Gemini 2.5 Pro thinking budget slider and add support for Gemini 2.5 Flash Lite Preview model (Thanks @arafatkatze!)
+
+## [3.18.9]
+
+-   Fix streaming reliability issues with Cline provider that could cause connection problems during long conversations
+-   Fix authentication error handling for Cline provider to show clearer error messages when not signed in and prevent recursive failed requests
+-   Remove incorrect pricing display for SAP AI Core provider since it uses non-USD "Capacity Units" that cannot be directly converted (Thanks @ncryptedV1!)
+
+## [3.18.8]
+
+-   Update pricing for Grok 3 model because the promotion ended
+
+## [3.18.7]
+
+-   Remove promotional "free" messaging for Grok 3 model in UI
+
+## [3.18.6]
+
+-   Update request header to include `"ai-client-type": "Cline"` to SAP Api Provider
+-   Add organization accounts
+
+## [3.18.5]
+
+-   Fix Plan/Act mode persistence across sessions and multi-workspace conflicts
+-   Improve provider switching performance by 18x (from 550ms to 30ms) with batched storage operations
+-   Improve SAP AI Core provider model organization and fix exception handling (Thanks @schardosin!)
+
+## [3.18.4]
+
+-   Add support for Gemini 2.5 Pro and Flash to SAP AI Core Provider
+-   Fix logging in with Cline account not getting past welcome screen
+
+## [3.18.3]
+
+-   Improve Cerebras Qwen model performance by removing thinking tokens from model input (Thanks @kevint-cerebras!)
+-   Improve Claude Code provider with better error handling and performance optimizations (Thanks @BarreiroT!)
+
+## [3.18.2]
+
+-   Fix issue where terminal output would not be captured if shell integration fails by falling back to capturing the terminal content.
+-   Add confirmation popup when deleting tasks
+-   Add support for Claude Sonnet 4 and Opus 4 model in SAP AI Core provider (Thanks @lizzzcai!)
+-   Add support for `litellm_session_id` to group requests in a single session (Thanks @jorgegarciarey!)
+-   Add "Thinking Budget" customization for Claude Code (Thanks @BarreiroT!)
+-   Fix issue where the extension would use the user's environment variables for authentication when using Claude Code (Thanks @BarreiroT!)
+
+## [3.18.1]
+
+-   Add support for Claude 4 Sonnet in SAP AI Core provider (Thanks @GTxx!)
+-   Fix ENAMETOOLONG error when using Claude Code provider with long conversation histories (Thanks @BarreiroT!)
+-   Remove Gemini CLI provider because Google asked us to
+-   Fix bug with "Delete All Tasks" functionality
+
+## [3.18.0]
+
+-   Optimized Cline to work with the Claude 4 family of models, resulting in improved performance, reliability, and new capabilities
+-   Added a new Gemini CLI provider that allows you to use your local Gemini CLI authentication to access Gemini models for free (Thanks @google-gemini!)
+-   Optimized Cline to work with the Gemini 2.5 family of models
+-   Updated the default and recommended model to Claude 4 Sonnet for the best performance
+-   Fix race condition in Plan/Act mode switching
+-   Improve robustness of search and replace parsing
+
+## [3.17.16]
+
+-   Fix Claude Code provider error handling for incomplete messages during long-running tasks (Thanks @BarreiroT!)
+-   Add taskId as metadata to LiteLLM API requests for better request tracing (Thanks @jorgegarciarey!)
+
+## [3.17.15]
+
+-   Fix LiteLLM provider to properly respect selected model IDs when switching between Plan and Act modes (Thanks @sammcj!)
+-   Fix chat input being cleared when switching between Plan/Act modes without sending a message (Thanks @BarreiroT!)
+-   Fix MCP server name display to avoid showing "undefined" for SSE servers, preventing tool/resource invocation failures (Thanks @ramybenaroya!)
+-   Fix AWS Bedrock provider by removing deprecated custom model encoding (Thanks @watany-dev!)
+-   Fix timeline tooltips for followup messages and improve color retrieval code (Thanks @char8x!)
+-   Improve accessibility by making task header buttons properly announced by screen readers (Thanks @yncat!)
+-   Improve accessibility by adding proper state reporting for Plan/Act mode switch for screen readers (Thanks @yncat!)
+-   Prevent reading development environment variables from user's environment (Thanks @BarreiroT!)
+
 ## [3.17.14]
 
 -   Add Claude Code as a new API provider, allowing integration with Anthropic's Claude Code CLI tool and Claude Max Plan (Thanks @BarreiroT!)

CONTRIBUTING.md

@@ -10,13 +10,6 @@ Bug reports help make Cline better for everyone! Before creating a new issue, pl
      🔐 <b>Important:</b> If you discover a security vulnerability, please use the <a href="https://github.com/cline/cline/security/advisories/new">Github security tool to report it privately</a>.
 </blockquote>
 
-## Deciding What to Work On
-
-Looking for a good first contribution? Check out issues labeled ["good first issue"](https://github.com/cline/cline/labels/good%20first%20issue) or ["help wanted"](https://github.com/cline/cline/labels/help%20wanted). These are specifically curated for new contributors and areas where we'd love some help!
-
-We also welcome contributions to our [documentation](https://github.com/cline/cline/tree/main/docs)! Whether it's fixing typos, improving existing guides, or creating new educational content - we'd love to build a community-driven repository of resources that helps everyone get the most out of Cline. You can start by diving into `/docs` and looking for areas that need improvement.
-
-If you're planning to work on a bigger feature, please create a [feature request](https://github.com/cline/cline/discussions/categories/feature-requests?discussions_q=is%3Aopen+category%3A%22Feature+Requests%22+sort%3Atop) first so we can discuss whether it aligns with Cline's vision.
 
 ## Before Contributing
 
@@ -24,14 +17,70 @@ All contributions must begin with a GitHub Issue, unless the change is for small
 
 - **Check existing issues**: Search [GitHub Issues](https://github.com/cline/cline/issues).
 - **Create an issue**: Use appropriate templates:
-    - **Bugs:** "Bug Report" template.
-    - **Features:** "Detailed Feature Proposal" template. Approval from a core Cline contributor required before starting.
-- **Claim issues**: Comment your interest.
+    - **Contributions:** Use the "Contribution Request" template to propose what you'd like to work on.
+    - **Bugs:** "Bug Report" template for reporting issues.
+    - **Features:** "Detailed Feature Proposal" template for suggesting new features.
+- **Wait for approval**: A core Cline contributor must approve your contribution request before you start implementation.
+- **Claim issues**: Once approved, the issue will be assigned to you.
 
 **PRs without approved issues may be closed.**
 
+
+## Deciding What to Work On
+
+Looking for a good first contribution? Check out issues labeled ["good first issue"](https://github.com/cline/cline/labels/good%20first%20issue) or ["help wanted"](https://github.com/cline/cline/labels/help%20wanted). These are specifically curated for new contributors and areas where we'd love some help!
+
+We also welcome contributions to our [documentation](https://github.com/cline/cline/tree/main/docs)! Whether it's fixing typos, improving existing guides, or creating new educational content - we'd love to build a community-driven repository of resources that helps everyone get the most out of Cline. You can start by diving into `/docs` and looking for areas that need improvement.
+
 ## Development Setup
 
+
+### Local Development Instructions
+
+1. Clone the repository _(Requires [git-lfs](https://git-lfs.com/))_:
+    ```bash
+    git clone https://github.com/cline/cline.git
+    ```
+2. Open the project in VSCode:
+    ```bash
+    code cline
+    ```
+3. Install the necessary dependencies for the extension and webview-gui:
+    ```bash
+    npm run install:all
+    ```
+4. Launch by pressing `F5` (or `Run`->`Start Debugging`) to open a new VSCode window with the extension loaded. (You may need to install the [esbuild problem matchers extension](https://marketplace.visualstudio.com/items?itemName=connor4312.esbuild-problem-matchers) if you run into issues building the project.)
+
+
+
+
+### Creating a Pull Request
+
+1. Before creating a PR, generate a changeset entry:
+    ```bash
+    npm run changeset
+    ```
+   This will prompt you for:
+   - Type of change (major, minor, patch)
+     - `major` → breaking changes (1.0.0 → 2.0.0)
+     - `minor` → new features (1.0.0 → 1.1.0)
+     - `patch` → bug fixes (1.0.0 → 1.0.1)
+   - Description of your changes
+
+2. Commit your changes and the generated `.changeset` file
+
+3. Push your branch and create a PR on GitHub. Our CI will:
+   - Run tests and checks
+   - Changesetbot will create a comment showing the version impact
+   - When merged to main, changesetbot will create a Version Packages PR
+   - When the Version Packages PR is merged, a new release will be published
+4. Testing
+    - Run `npm run test` to run tests locally. 
+    - Before submitting PR, run `npm run format:fix` to format your code
+    - Run `npm run test:ci` to run tests locally
+
+### Extension
+
 1. **VS Code Extensions**
 
     - When opening the project, VS Code will prompt you to install recommended extensions
@@ -41,6 +90,7 @@ All contributions must begin with a GitHub Issue, unless the change is for small
 2. **Local Development**
     - Run `npm run install:all` to install dependencies
     - Run `npm run test` to run tests locally
+    - Run → Start Debugging or `>Debug: Select and Start Debugging` and wait for a new VS Code instance to open
     - Before submitting PR, run `npm run format:fix` to format your code
 
 3. **Linux-specific Setup**
@@ -85,8 +135,6 @@ All contributions must begin with a GitHub Issue, unless the change is for small
       xvfb
     ```
 
-    - Run `npm run test:ci` to run tests locally
-
 ## Writing and Submitting Code
 
 Anyone can contribute code to Cline, but we ask that you follow these guidelines to ensure your contributions can be smoothly integrated:

README.md

@@ -30,7 +30,7 @@ English | <a href="https://github.com/cline/cline/blob/main/locales/es/README.md
 </table>
 </div>
 
-Meet Cline, an AI assistant that can use your **CLI** a**N**d **E**ditor.
+Meet Cline (pronounced /klaɪn/, like "Klein"), an AI assistant that can use your **CLI** a**N**d **E**ditor.
 
 Thanks to [Claude 3.7 Sonnet's agentic coding capabilities](https://www.anthropic.com/claude/sonnet), Cline can handle complex software development tasks step-by-step. With tools that let him create & edit files, explore large projects, use the browser, and execute terminal commands (after you grant permission), he can assist you in ways that go beyond code completion or tech support. Cline can even use the Model Context Protocol (MCP) to create new tools and extend his own capabilities. While autonomous AI scripts traditionally run in sandboxed environments, this extension provides a human-in-the-loop GUI to approve every file change and terminal command, providing a safe and accessible way to explore the potential of agentic AI.
 
@@ -141,50 +141,6 @@ For example, when working with a local web server, you can use 'Restore Workspac
 
 To contribute to the project, start with our [Contributing Guide](CONTRIBUTING.md) to learn the basics. You can also join our [Discord](https://discord.gg/cline) to chat with other contributors in the `#contributors` channel. If you're looking for full-time work, check out our open positions on our [careers page](https://cline.bot/join-us)!
 
-<details>
-<summary>Local Development Instructions</summary>
-
-1. Clone the repository _(Requires [git-lfs](https://git-lfs.com/))_:
-    ```bash
-    git clone https://github.com/cline/cline.git
-    ```
-2. Open the project in VSCode:
-    ```bash
-    code cline
-    ```
-3. Install the necessary dependencies for the extension and webview-gui:
-    ```bash
-    npm run install:all
-    ```
-4. Launch by pressing `F5` (or `Run`->`Start Debugging`) to open a new VSCode window with the extension loaded. (You may need to install the [esbuild problem matchers extension](https://marketplace.visualstudio.com/items?itemName=connor4312.esbuild-problem-matchers) if you run into issues building the project.)
-
-</details>
-
-<details>
-<summary>Creating a Pull Request</summary>
-
-1. Before creating a PR, generate a changeset entry:
-    ```bash
-    npm run changeset
-    ```
-   This will prompt you for:
-   - Type of change (major, minor, patch)
-     - `major` → breaking changes (1.0.0 → 2.0.0)
-     - `minor` → new features (1.0.0 → 1.1.0)
-     - `patch` → bug fixes (1.0.0 → 1.0.1)
-   - Description of your changes
-
-2. Commit your changes and the generated `.changeset` file
-
-3. Push your branch and create a PR on GitHub. Our CI will:
-   - Run tests and checks
-   - Changesetbot will create a comment showing the version impact
-   - When merged to main, changesetbot will create a Version Packages PR
-   - When the Version Packages PR is merged, a new release will be published
-
-</details>
-
-
 ## License
 
 [Apache 2.0 © 2025 Cline Bot Inc.](./LICENSE)

docs/docs.json

@@ -146,6 +146,8 @@
 				"group": "Provider Configuration",
 				"pages": [
 					"provider-config/anthropic",
+					"provider-config/claude-code",
+					"provider-config/aws-bedrock-with-apikey-authentication",
 					"provider-config/aws-bedrock-with-credentials-authentication",
 					"provider-config/aws-bedrock-with-profile-authentication",
 					"provider-config/gcp-vertex-ai",
@@ -169,6 +171,10 @@
 					"running-models-locally/ollama"
 				]
 			},
+			{
+				"group": "Troubleshooting",
+				"pages": ["troubleshooting/terminal-quick-fixes", "troubleshooting/terminal-integration-guide"]
+			},
 			{
 				"group": "More Info",
 				"pages": ["more-info/telemetry"]

docs/enterprise-solutions/cloud-provider-integration.mdx

@@ -14,6 +14,8 @@ Certain scenarios may warrant using local models, including handling highly sens
 
 #### [IAM Security Best Practices](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html) (For administrators)
 
+#### [AWS Bedrock setup for API Keys](/provider-config/aws-bedrock-with-apikey-authentication)
+
 #### [AWS Bedrock setup for Legacy IAM (AWS Credentials)](/provider-config/aws-bedrock-with-credentials-authentication)
 
 #### [AWS Bedrock setup for SSO token (AWS Profile)](/provider-config/aws-bedrock-with-profile-authentication)

docs/exploring-clines-tools/new-task-tool.mdx

@@ -120,7 +120,7 @@ Example of context window usage over 50% with a 200K context window:
 # Context Window Usage
 
 105,000 / 200,000 tokens (53%)
-Model: anthropic/claude-3.7-sonnet (200K context window)
+Model: anthropic/claude-sonnet-4 (200K context window)
 \`\`\`
 
 **IMPORTANT**: When you see context window usage at or above 50%, you MUST:

docs/features/at-mentions/terminal-mentions.mdx

@@ -58,3 +58,16 @@ When you use the terminal mention in your message, here's what happens behind th
 6. The AI can now "see" the complete terminal output with all formatting preserved
 
 This process happens automatically whenever you use the terminal mention, giving the AI access to your command results, error messages, and other terminal output without you having to copy it manually.
+
+## Troubleshooting Terminal Issues
+
+If you're experiencing issues with terminal mentions or terminal integration in general (such as "Shell Integration Unavailable" or commands not showing output), please refer to our comprehensive [Terminal Integration Troubleshooting Guide](/troubleshooting/terminal-integration-guide).
+
+Common issues include:
+
+-   Terminal mentions not capturing output
+-   "Shell Integration Unavailable" messages in Cline chat
+-   Commands executing but output not visible to Cline
+-   Terminal integration working inconsistently
+
+The troubleshooting guide provides platform-specific solutions and detailed configuration steps to resolve these issues.

docs/features/commands-and-shortcuts/terminal-integration.mdx

@@ -74,8 +74,25 @@ This approach ensures that all terminal output, including colors and formatting,
 
 -   **Select specific output when needed**: By default, the integration captures all terminal content, but you can also select specific lines before right-clicking to focus on just the relevant output.
 
--   **Combine with file mentions**: After sending terminal output to Cline, you can enhance your question by mentioning relevant files using the @ mentions feature.
+-   **Combine terminal outputs with file mentions**: After sending terminal output to Cline, you can enhance your question by mentioning relevant files using the @ mentions feature.
 
--   **Use for build and test output**: Terminal integration is particularly useful for understanding complex build errors or test failures that span multiple lines.
+-   **Contextualize build & test outputs with the terminal**: Terminal integration is particularly useful for understanding complex build errors or test failures that span multiple lines.
 
 Next time you're staring at a cryptic error message in your terminal, try using Cline's terminal integration instead of copying and pasting. You'll get more accurate help because Cline can see the complete terminal context with proper formatting.
+
+## Troubleshooting Terminal Issues
+
+If you're experiencing issues with terminal integration, such as "Shell Integration Unavailable" or commands not showing output, please refer to our comprehensive [Terminal Integration Troubleshooting Guide](/troubleshooting/terminal-integration-guide).
+
+The troubleshooting guide covers:
+
+-   Common terminal integration issues and quick fixes
+-   Platform-specific solutions for Windows, macOS, and Linux
+-   Shell-specific configurations for zsh, bash, PowerShell, and more
+-   Advanced debugging techniques
+-   Terminal settings optimization
+
+<Tip>
+	**Quick Fix**: Most terminal issues can be resolved by switching to bash in the Cline settings and increasing the shell
+	integration timeout to 10 seconds.
+</Tip>

docs/more-info/telemetry.mdx

@@ -6,7 +6,7 @@ title: "Telemetry"
 
 To help make Cline better for everyone, we collect anonymous usage data that helps us understand how developers are using our open-source AI coding agent. This feedback loop is crucial for improving Cline's capabilities and user experience.
 
-We use PostHog, an open-source analytics platform, for data collection and analysis. Our telemetry implementation is fully transparent - you can review the [source code](https://github.com/cline/cline/blob/main/src/services/telemetry/TelemetryService.ts) to see exactly what we track.
+We use PostHog, an open-source analytics platform, for data collection and analysis. Our telemetry implementation is fully transparent - you can review the [source code](https://github.com/cline/cline/blob/main/src/services/posthog/telemetry/TelemetryService.ts) to see exactly what we track.
 
 ### Tracking Policy
 
@@ -22,7 +22,7 @@ We collect basic anonymous usage data including:
 **System Context:** OS type and VS Code environment details\
 **UI Activity:** Navigation patterns and feature usage
 
-For complete transparency, you can inspect our [telemetry implementation](https://github.com/cline/cline/blob/main/src/services/telemetry/TelemetryService.ts) to see the exact events we track.
+For complete transparency, you can inspect our [telemetry implementation](https://github.com/cline/cline/blob/main/src/services/posthog/telemetry/TelemetryService.ts) to see the exact events we track.
 
 ### How to Opt Out
 

docs/provider-config/aws-bedrock-with-apikey-authentication.mdx

@@ -0,0 +1,135 @@
+---
+title: "AWS Bedrock"
+description: "Learn how to set up AWS Bedrock with Cline using credentials authentication. This guide covers AWS environment setup, regional access verification, and secure integration with the Cline VS Code extension."
+---
+
+### Overview
+
+-   **AWS Bedrock:** A fully managed service that offers access to leading generative AI models (e.g., Anthropic Claude, Amazon Nova) through AWS.\
+    [Learn more about AWS Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html).
+-   **Cline:** A VS Code extension that acts as a coding assistant by integrating with AI models—empowering developers to generate code, debug, and analyze data.
+-   **Developer Focus:** This guide is tailored for individual developers that want to enable access to frontier models via AWS Bedrock with a simplified setup using API Keys.
+
+---
+
+### Step 1: Prepare Your AWS Environment
+
+#### 1.1 Individual user setup - Create a Bedrock API Key
+
+For more detailed instructions check the [documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/api-keys.html).
+
+1. **Sign in to the AWS Management Console:**\
+   [AWS Console](https://aws.amazon.com/console/)
+2. **Access Bedrock Console:**
+    - [Bedrock Console](https://console.aws.amazon.com/bedrock)
+    - Create a new Long Lived API Key. This API Key will have by default the `AmazonBedrockLimitedAccess` IAM policy
+      [View AmazonBedrockLimitedAccess Policy Details](https://docs.aws.amazon.com/bedrock/latest/userguide/security-iam.html)
+
+#### 1.2 Create or Modify the Policy
+
+To ensure Cline can interact with AWS Bedrock, your IAM user or role needs specific permissions. While the `AmazonBedrockLimitedAccess` managed policy provides comprehensive access, for a more restricted and secure setup adhering to the principle of least privilege, the following minimal permissions are sufficient for Cline's core model invocation functionality:
+
+-   `bedrock:InvokeModel`
+-   `bedrock:InvokeModelWithResponseStream`
+-   `bedrock:CallWithBearerToken`
+
+You can create a custom IAM policy with these permissions and attach it to your IAM user or role.
+
+1.  In the AWS IAM console, create a new policy.
+2.  Use the JSON editor to add the following policy document:
+    ```json
+    {
+    	"Version": "2012-10-17",
+    	"Statement": [
+    		{
+    			"Effect": "Allow",
+    			"Action": ["bedrock:InvokeModel", "bedrock:InvokeModelWithResponseStream", "bedrock:CallWithBearerToken"],
+    			"Resource": "*" // For enhanced security, scope this to specific model ARNs if possible.
+    		}
+    	]
+    }
+    ```
+3.  Name the policy (e.g., `ClineBedrockInvokeAccess`) and attach it to the IAM user associated with the key you created. The IAM user and the API key have the same prefix.
+
+**Important Considerations:**
+
+-   **Model Listing in Cline:** The minimal permissions (`bedrock:InvokeModel`, `bedrock:InvokeModelWithResponseStream`) are sufficient for Cline to _use_ a model if you specify the model ID directly in Cline's settings. If you rely on Cline to dynamically list available Bedrock models, you might need additional permissions like `bedrock:ListFoundationModels`.
+-   **AWS Marketplace Subscriptions:** For third-party models (e.g., Anthropic Claude), the **`AmazonBedrockLimitedAccess`** policy grants you the necessary permissions to subscribe via the AWS Marketplace. There is no explicit access to be enabled. For Anthropic models you are still required to submit a First Time Use (FTU) form via the Console. If you get the following message in the Cline chat `[ERROR] Failed to process response: Model use case details have not been submitted for this account. Fill out the Anthropic use case details form before using the model.` then open the [Playground in the AWS Bedrock Console](https://console.aws.amazon.com/bedrock/home?#/text-generation-playground), select any Anthropic model and fill in the form (you might need to send a prompt first)
+
+---
+
+### Step 2: Verify Regional and Model Access
+
+#### 2.1 Choose and Confirm a Region
+
+1. **Select a Region:**\
+   AWS Bedrock is available in multiple regions (e.g., US East, Europe, Asia Pacific). Choose the region that meets your latency and compliance needs.\
+   [AWS Global Infrastructure](https://aws.amazon.com/about-aws/global-infrastructure/regions_az/)
+2. **Verify Model Access:**
+    - **Note:** Some models are only accessible via an [Inference Profile](https://docs.aws.amazon.com/bedrock/latest/userguide/inference-profiles-support.html). In such case check the box "Cross Region Inference".
+
+---
+
+### Step 3: Configure the Cline VS Code Extension
+
+#### 3.1 Install and Open Cline
+
+1. **Install VS Code:**\
+   Download from the [VS Code website](https://code.visualstudio.com/).
+2. **Install the Cline Extension:**
+    - Open VS Code.
+    - Go to the Extensions Marketplace (`Ctrl+Shift+X` or `Cmd+Shift+X`).
+    - Search for **Cline** and install it.
+
+#### 3.2 Configure Cline Settings
+
+1. **Open Cline Settings:**
+    - Click on the settings ⚙️ to select your API Provider.
+2. **Select AWS Bedrock as the API Provider:**
+    - From the API Provider dropdown, choose **AWS Bedrock**.
+3. **Enter Your AWS API Key:**
+    - Input your **API Key**
+    - Specify the correct **AWS Region** (e.g., `us-east-1` or your enterprise-approved region).
+4. **Select a Model:**
+    - Choose an on-demand model (e.g., **anthropic.claude-3-5-sonnet-20241022-v2:0**).
+5. **Save and Test:**
+    - Click **Done/Save** to apply your settings.
+    - Test the integration by sending a simple prompt (e.g., "Generate a Python function to check if a number is prime.").
+
+---
+
+### Step 4: Security, Monitoring, and Best Practices
+
+1. **Secure Access:**
+    - Prefer AWS SSO/federated roles over long-lived API Key when possible.
+    - [AWS IAM Best Practices](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html)
+2. **Enhance Network Security:**
+    - Consider setting up [AWS PrivateLink](https://docs.aws.amazon.com/vpc/latest/userguide/endpoint-services-overview.html) to securely connect to Bedrock.
+3. **Monitor and Log Activity:**
+    - Enable AWS CloudTrail to log Bedrock API calls.
+    - Use CloudWatch to monitor metrics like invocation count, latency, and token usage.
+    - Set up alerts for abnormal activity.
+4. **Handle Errors and Manage Costs:**
+    - Implement exponential backoff for throttling errors.
+    - Use AWS Cost Explorer and set billing alerts to track usage.\
+      [AWS Cost Management](https://docs.aws.amazon.com/cost-management/latest/userguide/what-is-aws-cost-management.html)
+5. **Regular Audits and Compliance:**
+    - Periodically review IAM roles and CloudTrail logs.
+    - Follow internal data privacy and governance policies.
+
+---
+
+### Conclusion
+
+By following these steps, your enterprise team can securely integrate AWS Bedrock with the Cline VS Code extension to accelerate development:
+
+1. **Prepare Your AWS Environment:** Create or use a secure IAM role/user, attach the `AmazonBedrockLimitedAccess` policy, and ensure necessary permissions.
+2. **Verify Region and Model Access:** Confirm that your selected region supports your required models.
+3. **Configure Cline in VS Code:** Install and set up Cline with your AWS credentials and choose an appropriate model.
+4. **Implement Security and Monitoring:** Use best practices for IAM, network security, monitoring, and cost management.
+
+For further details, consult the [AWS Bedrock Documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html) and coordinate with your internal cloud team. Happy coding!
+
+---
+
+_This guide will be updated as AWS Bedrock and Cline evolve. Always refer to the latest documentation and internal policies for up-to-date practices._

docs/provider-config/aws-bedrock-with-credentials-authentication.mdx

@@ -5,7 +5,7 @@ description: "Learn how to set up AWS Bedrock with Cline using credentials authe
 
 ### Overview
 
--   **AWS Bedrock:** A fully managed service that offers access to leading generative AI models (e.g., Anthropic Claude, Amazon Titan) through AWS.\
+-   **AWS Bedrock:** A fully managed service that offers access to leading generative AI models (e.g., Anthropic Claude, Amazon Nova) through AWS.\
     [Learn more about AWS Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html).
 -   **Cline:** A VS Code extension that acts as a coding assistant by integrating with AI models—empowering developers to generate code, debug, and analyze data.
 -   **Enterprise Focus:** This guide is tailored for organizations with established AWS environments (using IAM roles, AWS SSO, AWS Organizations, etc.) to ensure secure and compliant usage.
@@ -25,7 +25,7 @@ description: "Learn how to set up AWS Bedrock with Cline using credentials authe
 
 #### 1.2 Attach the Required Policies
 
-To ensure Cline can interact with AWS Bedrock, your IAM user or role needs specific permissions. While the `AmazonBedrockFullAccess` managed policy provides comprehensive access, for a more restricted and secure setup adhering to the principle of least privilege, the following minimal permissions are sufficient for Cline's core model invocation functionality:
+To ensure Cline can interact with AWS Bedrock, your IAM user or role needs specific permissions. While the `AmazonBedrockLimitedAccess` managed policy provides comprehensive access, for a more restricted and secure setup adhering to the principle of least privilege, the following minimal permissions are sufficient for Cline's core model invocation functionality:
 
 -   `bedrock:InvokeModel`
 -   `bedrock:InvokeModelWithResponseStream`
@@ -52,8 +52,8 @@ You can create a custom IAM policy with these permissions and attach it to your
 
 **Option 2: Using a Managed Policy (Simpler Initial Setup)**
 
--   Alternatively, you can attach the AWS managed policy **`AmazonBedrockFullAccess`**. This grants broader permissions, including the ability to list models, manage provisioning, and other Bedrock features. This might be simpler for initial setup or if you require these wider capabilities.
-    [View AmazonBedrockFullAccess Policy Details](https://docs.aws.amazon.com/bedrock/latest/userguide/security-iam.html)
+-   Alternatively, you can attach the AWS managed policy **`AmazonBedrockLimitedAccess`**. This grants broader permissions, including the ability to list models, manage provisioning, and other Bedrock features. This might be simpler for initial setup or if you require these wider capabilities.
+    [View AmazonBedrockLimitedAccess Policy Details](https://docs.aws.amazon.com/bedrock/latest/userguide/security-iam.html)
 
 **Important Considerations:**
 
@@ -71,8 +71,8 @@ You can create a custom IAM policy with these permissions and attach it to your
    AWS Bedrock is available in multiple regions (e.g., US East, Europe, Asia Pacific). Choose the region that meets your latency and compliance needs.\
    [AWS Global Infrastructure](https://aws.amazon.com/about-aws/global-infrastructure/regions_az/)
 2. **Verify Model Access:**
-    - In the AWS Bedrock console, confirm that the models your team requires (e.g., Anthropic Claude, Amazon Titan) are marked as "Access granted."
-    - **Note:** Some advanced models might require an [Inference Profile](https://docs.aws.amazon.com/bedrock/latest/userguide/inference-profiles-prereq.html) if not available on-demand.
+    - In the AWS Bedrock console, confirm that the models your team requires (e.g., Anthropic Claude, Amazon Nova) are marked as "Access granted."
+    - **Note:** Some advanced models might require an [Inference Profile](https://docs.aws.amazon.com/bedrock/latest/userguide/inference-profiles-support.html) if not available on-demand.
 
 #### 2.2 Set Up AWS Marketplace Subscriptions (if needed)
 
@@ -138,7 +138,7 @@ You can create a custom IAM policy with these permissions and attach it to your
 
 By following these steps, your enterprise team can securely integrate AWS Bedrock with the Cline VS Code extension to accelerate development:
 
-1. **Prepare Your AWS Environment:** Create or use a secure IAM role/user, attach the `AmazonBedrockFullAccess` policy, and ensure necessary permissions.
+1. **Prepare Your AWS Environment:** Create or use a secure IAM role/user, attach the `AmazonBedrockLimitedAccess` policy, and ensure necessary permissions.
 2. **Verify Region and Model Access:** Confirm that your selected region supports your required models and subscribe via AWS Marketplace if needed.
 3. **Configure Cline in VS Code:** Install and set up Cline with your AWS credentials and choose an appropriate model.
 4. **Implement Security and Monitoring:** Use best practices for IAM, network security, monitoring, and cost management.

docs/provider-config/claude-code.mdx

@@ -0,0 +1,86 @@
+---
+title: "Claude Code"
+description: "Use your Claude Max or Pro subscription with Cline instead of paying per token. Learn how to set up and configure the Claude Code provider."
+---
+
+**Website:** [https://docs.anthropic.com/en/docs/claude-code/setup](https://docs.anthropic.com/en/docs/claude-code/setup)
+
+The Claude Code provider lets you use your existing Claude subscription with Cline. If you have Claude Max or Pro, this means you can use Claude in Cline without paying extra API costs.
+
+<Frame>
+	<img
+		src="https://storage.googleapis.com/cline_public_images/docs/assets/claude-code-use-opus.gif"
+		alt="Using the Claude Code provider in Cline with Opus model"
+	/>
+</Frame>
+
+## Setup
+
+First, you'll need to install and authenticate Claude Code on your system:
+
+1. **Install Claude Code**: Follow Anthropic's [official setup guide](https://docs.anthropic.com/en/docs/claude-code/setup) to install and authenticate the Claude CLI.
+
+2. **Configure in Cline**:
+    - Open Cline settings (⚙️ icon)
+    - Select **Claude Code** from the **API Provider** dropdown
+    - Set the path to your Claude CLI executable (usually just `claude` if it's in your PATH)
+
+<Frame>
+	<img
+		src="https://storage.googleapis.com/cline_public_images/docs/assets/claude-code-setup.gif"
+		alt="Setting up the Claude Code provider in Cline"
+	/>
+</Frame>
+
+### Finding your Claude Code path
+
+If you're not sure where Claude Code is installed:
+
+-   **macOS / Linux**: Run `which claude` in your terminal
+-   **Windows (Command Prompt)**: Run `where claude`
+-   **Windows (PowerShell)**: Run `Get-Command claude`
+
+## Supported Models
+
+The Claude Code provider supports these models:
+
+-   `claude-sonnet-4-20250514` (Recommended)
+-   `claude-opus-4-20250514`
+-   `claude-3-7-sonnet-20250219`
+-   `claude-3-5-sonnet-20241022`
+-   `claude-3-5-haiku-20241022`
+
+## How it works
+
+When you use Claude Code with Cline, here's what happens behind the scenes:
+
+Cline wraps the Claude Code CLI to handle your requests. Each time you send a message, Cline starts a new `claude` process, sends your conversation, and streams the response back. The AI reasoning comes from Claude Code, but all the actual file editing, terminal commands, and other tools are handled by Cline.
+
+The main difference you'll notice is that responses don't stream character-by-character like other providers. Instead, Claude Code processes your full request before sending back the complete response.
+
+## Limitations
+
+There are a few things to keep in mind with Claude Code:
+
+-   Images in your messages get converted to text placeholders since Claude Code doesn't support image uploads through the CLI
+-   Prompt caching isn't available with this provider
+-   Responses don't stream in real-time like other providers
+
+## Troubleshooting
+
+If you run into issues:
+
+**Authentication problems**: Make sure you're logged into Claude Code with your subscription account. Run `claude auth status` to check.
+
+**Path issues**: Double-check that the Claude CLI path in Cline's settings is correct. Try running `claude --version` in your terminal to verify it's working.
+
+**Still having trouble?** We're actively improving this integration. Report issues on our [GitHub](https://github.com/cline/cline/issues) or ask for help in our [Discord](https://discord.gg/cline).
+
+## Usage with subscriptions
+
+If you have a Claude Max subscription, your usage in Cline shows up as $0.00 in the billing interface since you're not paying additional API costs. Your usage still counts against your subscription limits, but you won't see per-token charges.
+
+For more details about using Claude Code with your subscription, check out Anthropic's documentation:
+
+-   [Claude Code Setup Guide](https://docs.anthropic.com/en/docs/claude-code/setup)
+-   [Using Claude Code with Pro/Max Plans](https://support.anthropic.com/en/articles/11145838-using-claude-code-with-your-pro-or-max-plan)

docs/troubleshooting/terminal-integration-guide.mdx

@@ -0,0 +1,399 @@
+---
+title: "Terminal Integration Troubleshooting Guide"
+sidebarTitle: "Terminal Troubleshooting"
+description: "Complete guide to resolving terminal integration issues in Cline"
+---
+
+This guide helps you resolve terminal integration issues in Cline. Terminal integration is crucial for Cline to execute commands and read their output, enabling it to understand errors, test results, and command responses.
+
+<Tip>
+	If you're experiencing terminal issues, try switching to a simpler shell like `bash` in the Cline settings, under "Terminal Settings"
+
+    This resolves most terminal integration problems.
+
+</Tip>
+
+## Quick Diagnosis Flowchart
+
+Follow this flowchart to quickly identify your issue:
+
+```mermaid
+graph TD
+    A[Terminal Issue] --> B{Can Cline execute commands?}
+    B -->|No| C[Shell Integration Unavailable]
+    B -->|Yes| D{Can Cline see the output?}
+    D -->|No| E[Output Capture Failed]
+    D -->|Yes| F{Is the output corrupted?}
+    F -->|Yes| G[Character Filtering Issue]
+    F -->|No| H{Does the command hang?}
+    H -->|Yes| I[Long-Running Command Issue]
+    H -->|No| J[Check Terminal Settings]
+
+    C --> K[Try Solution 1]
+    E --> L[Try Solution 2]
+    G --> M[Try Solution 3]
+    I --> N[Try Solution 4]
+
+    style A fill:#f9f,stroke:#333,stroke-width:2px
+    style K fill:#9f9,stroke:#333,stroke-width:2px
+    style L fill:#9f9,stroke:#333,stroke-width:2px
+    style M fill:#9f9,stroke:#333,stroke-width:2px
+    style N fill:#9f9,stroke:#333,stroke-width:2px
+```
+
+## Common Issues & Quick Solutions
+
+### 1. Shell Integration Unavailable
+
+**Symptoms:**
+
+-   Message: "Shell Integration Unavailable"
+-   Commands execute but Cline can't read output
+-   Terminal works fine manually but not with Cline
+
+**Quick Solutions:**
+
+#### macOS
+
+-   **Switch to bash**
+
+    1. Go to Cline Settings
+    2. Left-Click the **"Terminal Settings"** tab
+    3. Navigate to **"Default Terminal Profile"** and select **"bash"** from the drop-down menu
+
+-   **Disable Oh-My-Zsh temporarily**:
+
+    1. If using zsh, enter `mv ~/.zshrc ~/.zshrc.backup` into the terminal
+    2. Restart VSCode
+
+-   **Set environment**:
+    1.a For Zsh users, use one of the following Zsh commands to edit your shell profile:
+
+          - `nano ~/.zshrc`
+          - `vim ~/.zshrc`
+          - `code ~/.zshrc`
+
+    1.b For Bash users
+
+          - nano ~/.bash_profile
+
+    2. Add the following to your shell config: `export TERM=xterm-256color`
+    3. Save your configuration
+
+#### Windows
+
+-   **Use PowerShell 7**
+
+    1. Install from Microsoft Store
+    2. Go to Cline Settings
+    3. Left-Click the **"Terminal Settings"** tab
+    4. Navigate to **"Default Terminal Profile"** and select **"PowerShell 7"** from the drop-down menu
+
+-   **Disable Windows ConPTY**
+
+    1. Navigate to your VSCode Settings
+    2. Enter "Integrated: Windows Enable Conpty" into the Settings searchbar
+    3. Uncheck the option
+
+-   **Try Command Prompt**
+    1. Go to Cline Settings
+    2. Left-Click the **"Terminal Settings"** tab
+    3. Navigate to **"Default Terminal Profile"** and select **"Command Prompt"** from the drop-down menu
+
+#### Linux
+
+-   **Use bash**
+
+    1. Go to Cline Settings
+    2. Left-Click the **"Terminal Settings"** tab
+    3. Navigate to **"Default Terminal Profile"** and select **"bash"** from the drop-down menu
+
+-   **Check permissions**
+
+    1. Ensure VSCode has terminal access permissions
+
+-   **Disable custom prompts**
+    1. Comment out prompt customizations in `.bashrc`
+
+### 2. Command Output Not Visible
+
+**Symptoms:**
+
+-   Cline states in chat: "[Command is running but producing no output]"
+-   Commands complete but Cline doesn't see results
+-   Commands work sometimes but not consistently
+
+**Solutions:**
+
+-   **Increase Shell Integration Timeout**
+
+    1. Within Cline, left-click the **Settings** button in the top right-hand corner of the chat window
+    2. Once in the **Settings** window, left-click the **"Terminal Settings"** tab from the left-hand column
+    3. Navigate to "Shell integration timeout (seconds)" and enter **"10"** into the text field
+
+-   **Disable Terminal Reuse**
+
+    1. Within Cline, left-click the **Settings** button in the top right-hand corner of the chat window
+    2. Once in the **Settings** window, left-click the **"Terminal Settings"** tab from the left-hand column
+    3. Look for **"Enable aggressive terminal reuse"**, and **uncheck** this option
+
+-   **Check for interfering extensions**
+    1. Disable other terminal-related VSCode extensions
+
+### 3. Character Filtering Issues
+
+**Symptoms:**
+
+-   Commas missing from output (JSON appears corrupted)
+-   Special characters stripped from terminal output
+-   Syntax errors that don't appear when running manually
+
+**Solution:**
+This is a known bug in output processing. Workarounds:
+
+-   Recommend AI to use file output instead
+    1. Tell Cline in chat or Cline rules, to use `command > output.txt` before reading the file/s
+
+<Tip>
+	This family of issues is only partially solved in the latest Cline versions, so if you still face this, create a GitHub issue
+	if it is a persistent problem.
+</Tip>
+
+### 4. Long-Running Commands & Progress Bars
+
+**Symptoms:**
+
+-   Docker builds never complete in Cline
+-   Progress bars consume thousands of tokens
+-   The Cline button "Proceed while running" doesn't work properly in chat
+
+<Tip>
+	This family of issues has been solved in latest Cline versions but if you still face any issues, then create a GitHub issue
+	for this.
+</Tip>
+
+## Terminal Settings Explained
+
+Access these in Cline by clicking the settings icon, and navigating to the "Terminal Settings" section:
+
+### Default Terminal Profile
+
+-   **What it does**: Selects which shell Cline uses for commands
+-   **When to change**: If experiencing shell integration issues with your default shell
+-   **Recommended**: - macOS: bash (if zsh has issues) - Windows: PowerShell 7 - Linux: bash
+
+### Shell Integration Timeout
+
+-   **What it does**: How long Cline waits for the terminal to be ready
+-   **Default**: 4 seconds
+-   **When to increase**:
+    -   Slow shell startup (heavy .zshrc/.bashrc)
+    -   WSL environments
+    -   SSH connections
+-   **Recommended**: - Start with 10 seconds if having issues
+
+### Enable Aggressive Terminal Reuse
+
+-   **What it does**: Reuses existing terminals even if not in the correct directory
+-   **When to disable**:
+    -   Commands execute in wrong directory
+    -   Virtual environment issues
+    -   Terminal state corruption
+-   **Trade-off**: - Disabling creates more terminals but ensures clean state
+
+### Terminal Output Line Limit
+
+-   **What it does**: Limits how many lines Cline reads from terminal output
+-   **Default**: 500 lines
+-   **When to adjust**:
+    -   Increase for verbose build outputs
+    -   Decrease if hitting token limits
+    -   Set to 100 for commands with progress bars
+
+## Platform-Specific Solutions
+
+### macOS Issues
+
+#### Oh-My-Zsh Conflicts
+
+Oh-My-Zsh often interferes with shell integration. Solutions:
+
+1. Create a minimal `.zshrc` for VSCode:
+    ```bash
+    # ~/.zshrc-vscode
+    export TERM=xterm-256color
+    export PAGER=cat
+    # Minimal PATH and environment setup
+    ```
+2. Configure VSCode to use it:
+    ```json
+    {
+    	"terminal.integrated.env.osx": {
+    		"ZDOTDIR": "~/.zshrc-vscode"
+    	}
+    }
+    ```
+
+#### macOS 15+ Issues
+
+Recent macOS versions have stricter terminal permissions:
+
+1. System Preferences → Privacy & Security → Developer Tools
+2. Add Visual Studio Code
+3. Restart VSCode completely
+
+### Windows Issues
+
+#### PowerShell Execution Policy
+
+If commands fail silently:
+
+```powershell
+Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser
+```
+
+#### WSL Integration
+
+For WSL issues:
+
+1. Use WSL extension for VSCode
+2. Open folder in WSL: `code .` from WSL terminal
+3. Select "WSL Bash" as terminal profile in Cline
+
+#### Path Issues
+
+Windows path problems:
+
+1. Use forward slashes in Cline: `C:/Users/...`
+2. Quote paths with spaces: `"C:/Program Files/..."`
+3. Avoid `~` - use full paths
+
+### Linux/SSH/Container Issues
+
+#### SSH Connections
+
+For remote development:
+
+1. Install Cline on the remote machine, not locally
+2. Use SSH extension's integrated terminal
+3. Increase timeout to 15+ seconds
+
+#### Docker Containers
+
+When developing in containers:
+
+1. Install Cline in the container
+2. Use Dev Containers extension
+3. Ensure shell integration scripts are available
+
+## Shell-Specific Fixes
+
+### Zsh
+
+```bash
+# Add to ~/.zshrc
+export TERM=xterm-256color
+export PAGER=cat
+# Disable fancy prompts for VSCode
+if [[ "$TERM_PROGRAM" == "vscode" ]]; then
+    PS1="%n@%m %1~ %# "
+fi
+```
+
+### Bash
+
+```bash
+# Add to ~/.bashrc
+export TERM=xterm-256color
+export PAGER=cat
+# Simple prompt for VSCode
+if [[ "$TERM_PROGRAM" == "vscode" ]]; then
+    PS1='\u@\h:\w\$ '
+fi
+```
+
+### Fish
+
+```fish
+# Add to ~/.config/fish/config.fish
+set -x TERM xterm-256color
+set -x PAGER cat
+# Disable fancy features in VSCode
+if test "$TERM_PROGRAM" = "vscode"
+    function fish_prompt
+        echo (whoami)'@'(hostname)':'(pwd)'> '
+    end
+end
+```
+
+### PowerShell
+
+```powershell
+# Add to $PROFILE
+$env:PAGER = "cat"
+# Disable progress bars
+$ProgressPreference = 'SilentlyContinue'
+```
+
+## Advanced Troubleshooting
+
+### Debug Mode
+
+Enable terminal debugging to see what's happening:
+
+1. Open VSCode Command Palette (Cmd/Ctrl+Shift+P)
+2. Run: "Developer: Set Log Level..."
+3. Choose "Trace"
+4. Check Output panel → "Cline" for terminal logs
+
+### Manual Shell Integration Test
+
+Test if shell integration works at all:
+
+```bash
+# In VSCode terminal
+echo $TERM_PROGRAM  # Should show "vscode"
+echo $VSCODE_SHELL_INTEGRATION  # Should be "1"
+```
+
+## FAQ
+
+### Why does Cline create so many terminals?
+
+When shell integration fails, Cline can't reuse terminals safely (they might be running long processes). Enable shell integration or adjust the terminal reuse setting.
+
+### Can I use my custom shell (nushell, xonsh, etc.)?
+
+Cline officially supports bash, zsh, fish, and PowerShell. Custom shells may work but aren't guaranteed. Use bash as a fallback.
+
+### Why do some commands work but others don't?
+
+Commands that use interactive features (pagers, progress bars, curses) often fail. Set `PAGER=cat` and use non-interactive flags.
+
+### How do I know if shell integration is working?
+
+Working integration shows command output in Cline's chat. Failed integration shows "Shell Integration Unavailable" or "[Command is running but producing no output]".
+
+## Still Having Issues?
+
+If you've tried everything:
+
+1. **Collect Debug Info**:
+
+    ```bash
+    echo "Shell: $SHELL"
+    echo "Term: $TERM"
+    echo "VSCode: $TERM_PROGRAM"
+    which bash
+    bash --version
+    ```
+
+2. **Report the Issue**:
+    - Use `/reportbug` in Cline github issues
+    - Include your debug info
+    - Mention which solutions you tried
+
+<Tip>
+	Remember: Most terminal issues are resolved by switching to bash and increasing the timeout. Start there before trying complex
+	solutions.
+</Tip>

docs/troubleshooting/terminal-quick-fixes.mdx

@@ -0,0 +1,51 @@
+---
+title: "Terminal Quick Fixes"
+sidebarTitle: "Terminal Quick Fixes"
+description: "Quick solutions for common terminal issues"
+---
+
+**Here is a list of common fixes, starting with the most applicable:**
+
+-   **Switch to bash** (solves most instances)
+
+    1. Within Cline, left-click the **Settings** button in the top right-hand corner of the chat window
+    2. Once in the **Settings** window, left-click the **"Terminal Settings"** tab from the left-hand column
+    3. Navigate to **"Default Terminal Profile"** and select **"bash"** from the drop-down
+
+-   **Increase timeout**
+
+    1. Within Cline, left-click the **Settings** button in the top right-hand corner of the chat window
+    2. Once in the **Settings** window, left-click the **"Terminal Settings"** tab from the left-hand column
+    3. Navigate to "Shell integration timeout (seconds)" and enter **"10"** into the text field
+
+-   **Disable terminal reuse**
+    1. Within Cline, left-click the **Settings** button in the top right-hand corner of the chat window
+    2. Once in the **Settings** window, left-click the **"Terminal Settings"** tab from the left-hand column
+    3. Look for **"Enable aggressive terminal reuse"**, and **uncheck** this option
+
+## Platform-Specific Fixes
+
+### macOS + Oh-My-Zsh
+
+```bash
+# Create minimal config for VSCode
+echo 'export TERM=xterm-256color' > ~/.zshrc-vscode
+echo 'export PAGER=cat' >> ~/.zshrc-vscode
+```
+
+### Windows PowerShell
+
+```powershell
+# Run as Administrator
+Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser
+```
+
+### WSL
+
+-   Open folder from WSL: `code .`
+-   Select **"WSL Bash"** in Cline settings, under **"Terminal Settings"**
+-   Increase **"Shell integration timeout (seconds)"** to **15**
+
+## Full Guide
+
+For detailed troubleshooting, see the [Complete Terminal Troubleshooting Guide](/troubleshooting/terminal-integration-guide).

esbuild.js

@@ -153,8 +153,8 @@ const extensionConfig = {
 // Standalone-specific configuration
 const standaloneConfig = {
 	...baseConfig,
-	entryPoints: ["src/standalone/standalone.ts"],
-	outfile: `${destDir}/standalone.js`,
+	entryPoints: ["src/standalone/cline-core.ts"],
+	outfile: `${destDir}/cline-core.js`,
 	// These gRPC protos need to load files from the module directory at runtime,
 	// so they cannot be bundled.
 	external: ["vscode", "@grpc/reflection", "grpc-health-check"],

eslint-rules/__tests__/no-direct-vscode-api.test.ts

@@ -0,0 +1,123 @@
+const { RuleTester: DirectApiRuleTester } = require("eslint")
+const noDirectVscodeApiRule = require("../no-direct-vscode-api")
+
+const directApiRuleTester = new DirectApiRuleTester({
+	parser: require.resolve("@typescript-eslint/parser"),
+	parserOptions: {
+		ecmaVersion: 2020,
+		sourceType: "module",
+		ecmaFeatures: {
+			jsx: true,
+		},
+	},
+})
+
+directApiRuleTester.run("no-direct-vscode-api", noDirectVscodeApiRule, {
+	valid: [
+		// Should allow vscode.postMessage in grpc-client-base.ts
+		{
+			code: `vscode.postMessage({ type: "grpc_request", data: {} })`,
+			filename: "grpc-client-base.ts",
+		},
+		{
+			code: `vscode.postMessage({ type: "grpc_request_cancel" })`,
+			filename: "/path/to/grpc-client-base.ts",
+		},
+		// Should allow in exception directories
+		{
+			code: `vscode.workspace.workspaceFolders`,
+			filename: "/src/hosts/vscode/host-bridge.ts",
+		},
+		{
+			code: `vscode.workspace.fs.stat(uri)`,
+			filename: "/standalone/runtime-files/helpers.ts",
+		},
+		// Should allow other vscode API calls
+		{
+			code: `vscode.window.showInformationMessage("Hello")`,
+			filename: "test.ts",
+		},
+		// Should allow postMessage calls on other objects
+		{
+			code: `window.postMessage({ type: "test" }, "*")`,
+			filename: "test.ts",
+		},
+		// Should allow variables named vscode but not calling postMessage
+		{
+			code: `const vscode = { other: "method" }; vscode.other()`,
+			filename: "test.ts",
+		},
+	],
+	invalid: [
+		// Should disallow vscode.postMessage in regular files
+		{
+			code: `vscode.postMessage({ type: "test", data: {} })`,
+			filename: "test.ts",
+			errors: [
+				{
+					messageId: "useGrpcClient",
+				},
+			],
+		},
+		// Should disallow vscode.postMessage in components
+		{
+			code: `vscode.postMessage({ type: "apiConfiguration", apiConfiguration })`,
+			filename: "ApiOptions.tsx",
+			errors: [
+				{
+					messageId: "useGrpcClient",
+				},
+			],
+		},
+		// Should disallow vscode.postMessage in test files
+		{
+			code: `vscode.postMessage({ type: "newTask", text: message.text })`,
+			filename: "test.test.ts",
+			errors: [
+				{
+					messageId: "useGrpcClient",
+				},
+			],
+		},
+		// Should disallow property access for disallowed APIs
+		{
+			code: `const folders = vscode.workspace.workspaceFolders;`,
+			filename: "workspace.ts",
+			errors: [
+				{
+					messageId: "useHostBridge",
+				},
+			],
+		},
+		// Should disallow method calls for disallowed APIs
+		{
+			code: `const relativePath = vscode.workspace.asRelativePath(filePath);`,
+			filename: "path-utils.ts",
+			errors: [
+				{
+					messageId: "usePathUtils",
+				},
+			],
+		},
+		// Should disallow nested property access
+		{
+			code: `const stats = await vscode.workspace.fs.stat(uri);`,
+			filename: "file-utils.ts",
+			errors: [
+				{
+					messageId: "useFsUtils",
+				},
+			],
+		},
+		// Should disallow getting a workspace folder
+		{
+			code: `const folder = vscode.workspace.getWorkspaceFolder(uri);`,
+			filename: "path-helper.ts",
+			errors: [
+				{
+					messageId: "usePathUtils",
+				},
+			],
+		},
+	],
+})

eslint-rules/index.js

@@ -1,11 +1,13 @@
 // eslint-rules/index.js
 const noProtobufObjectLiterals = require("./no-protobuf-object-literals")
 const noGrpcClientObjectLiterals = require("./no-grpc-client-object-literals")
+const noDirectVscodeApi = require("./no-direct-vscode-api")
 
 module.exports = {
 	rules: {
 		"no-protobuf-object-literals": noProtobufObjectLiterals,
 		"no-grpc-client-object-literals": noGrpcClientObjectLiterals,
+		"no-direct-vscode-api": noDirectVscodeApi,
 	},
 	configs: {
 		recommended: {
@@ -13,6 +15,7 @@ module.exports = {
 			rules: {
 				"local/no-protobuf-object-literals": "error",
 				"local/no-grpc-client-object-literals": "error",
+				"local/no-direct-vscode-api": "warn",
 			},
 		},
 	},

eslint-rules/no-direct-vscode-api.js

@@ -0,0 +1,164 @@
+const { ESLintUtils } = require("@typescript-eslint/utils")
+const path = require("path")
+
+const createRule = ESLintUtils.RuleCreator((name) => `https://cline.bot/eslint-rules/${name}`)
+
+// Configuration of disallowed VSCode APIs and their recommended alternatives
+const disallowedApis = {
+	"vscode.postMessage": {
+		messageId: "useGrpcClient",
+	},
+	"vscode.workspace.fs.stat": {
+		messageId: "useFsUtils",
+	},
+	"vscode.workspace.workspaceFolders": {
+		messageId: "useHostBridge",
+	},
+	"vscode.workspace.asRelativePath": {
+		messageId: "usePathUtils",
+	},
+	"vscode.workspace.getWorkspaceFolder": {
+		messageId: "usePathUtils",
+	},
+}
+
+module.exports = createRule({
+	name: "no-direct-vscode-api",
+	meta: {
+		type: "problem",
+		docs: {
+			description:
+				"Disallow direct VSCode API usage in favor of Cline's abstraction layers, except in src/hosts/vscode and standalone/runtime-files directories",
+			recommended: "error",
+		},
+		messages: {
+			useGrpcClient:
+				"Use gRPC service clients instead of vscode.postMessage().\n" +
+				"Example: AccountServiceClient.methodName(RequestType.create({...})) instead of vscode.postMessage({type: '...'}).\n" +
+				"Found: {{code}}",
+			useFsUtils:
+				"Use utilities in @/utils/fs instead of vscode.workspace.fs.stat.\n" +
+				"Example: import { isDirectory } from '@/utils/fs' or use the file system methods from the host bridge provider.\n" +
+				"Found: {{code}}",
+			useHostBridge:
+				"Use getHostBridgeProvider().workspaceClient.getWorkspacePaths({}) instead of vscode.workspace.workspaceFolders.\n" +
+				"This provides a consistent abstraction across VSCode and standalone environments.\n" +
+				"Found: {{code}}",
+			usePathUtils:
+				"Use path utilities from @/utils/path instead of direct VSCode workspace path methods.\n" +
+				"This provides consistent path handling across different environments.\n" +
+				"Found: {{code}}",
+		},
+		schema: [],
+	},
+	defaultOptions: [],
+
+	create(context) {
+		// Check if current file is in an exception directory or is grpc-client-base.ts
+		const filename = context.filename
+		const isGrpcClientBase = path.basename(filename) === "grpc-client-base.ts"
+
+		// Skip checking files in src/hosts/vscode or standalone/runtime-files
+		const isExceptionDirectory = filename.includes("/src/hosts/vscode/") || filename.includes("/standalone/runtime-files/")
+
+		// Pattern for checking memberExpressions like vscode.workspace.fs.stat
+		function checkMemberExpression(node) {
+			// Skip if this file is in an exception directory or is grpc-client-base.ts
+			if (isGrpcClientBase || isExceptionDirectory) {
+				return
+			}
+
+			// For handling nested properties like vscode.workspace.fs.stat
+			function getFullPropertyPath(node) {
+				if (node.type !== "MemberExpression") {
+					return node.name || ""
+				}
+
+				const objectPart = getFullPropertyPath(node.object)
+				const propertyPart = node.property.name || ""
+
+				return objectPart ? `${objectPart}.${propertyPart}` : propertyPart
+			}
+
+			// Check if the expression matches one of our disallowed patterns
+			if (node.object && node.object.type === "Identifier" && node.object.name === "vscode") {
+				const fullPath = `vscode.${node.property.name}`
+				checkDisallowedApi(fullPath, node)
+			}
+			// Handle nested expressions like vscode.workspace.fs.stat
+			else if (node.object && node.object.type === "MemberExpression") {
+				const fullPath = getFullPropertyPath(node)
+
+				// Only proceed if it starts with vscode
+				if (fullPath.startsWith("vscode.")) {
+					checkDisallowedApi(fullPath, node)
+				}
+			}
+		}
+
+		// Check if an expression matches a disallowed API and report if it does
+		function checkDisallowedApi(expressionPath, node) {
+			// Check exact matches
+			if (disallowedApis[expressionPath]) {
+				reportViolation(expressionPath, node)
+				return
+			}
+
+			// Check prefix matches (for nested properties)
+			for (const disallowedApi in disallowedApis) {
+				// For direct property access like vscode.workspace.workspaceFolders
+				if (expressionPath === disallowedApi) {
+					reportViolation(disallowedApi, node)
+					return
+				}
+
+				// For method calls like vscode.workspace.asRelativePath(...)
+				if (expressionPath.startsWith(`${disallowedApi}.`) || expressionPath.startsWith(`${disallowedApi}(`)) {
+					reportViolation(disallowedApi, node)
+					return
+				}
+			}
+		}
+
+		// Report a violation with the appropriate message
+		function reportViolation(disallowedApi, node) {
+			const sourceCode = context.sourceCode
+			const config = disallowedApis[disallowedApi]
+
+			// For method calls, get the whole call expression
+			let reportNode = node
+			let parentNode = sourceCode.getAncestors(node).pop()
+			if (parentNode && parentNode.type === "CallExpression" && parentNode.callee === node) {
+				reportNode = parentNode
+			}
+
+			const callText = sourceCode.getText(reportNode).trim()
+
+			context.report({
+				node: reportNode,
+				messageId: config.messageId,
+				data: {
+					code: callText,
+				},
+			})
+		}
+
+		return {
+			// Detect basic member expressions (e.g., vscode.postMessage)
+			MemberExpression(node) {
+				checkMemberExpression(node)
+			},
+
+			// Detect property access through destructuring
+			VariableDeclarator(node) {
+				// Skip if this file is in an exception directory or is grpc-client-base.ts
+				if (isGrpcClientBase || isExceptionDirectory) {
+					return
+				}
+
+				// Destructuring pattern checks removed as developers don't use the API this way
+				// They always use direct imports: import * as vscode from "vscode" and direct access: vscode.thing.foo
+			},
+		}
+	},
+})

evals/.gitignore

@@ -2,5 +2,23 @@ repositories
 
 results/evals.db
 
+diff-edits/cases/
+diff-edits/results/
+
+# Environment variables
+.env
+
+# backwards compatible
 diff_editing/test_cases/
-diff_editing/test_outputs/
+diff_editing/test_outputs/
+
+*.db
+*.db-wal
+*.db-shm
+
+.cache
+
+# Python bytecode cache
+*__pycache__/
+
+diff-edits/cases.zip

evals/README.md

@@ -17,6 +17,7 @@ The evaluation system consists of two main components:
 
 1. **Test Server**: Enhanced HTTP server in `src/services/test/TestServer.ts` that provides detailed task results
 2. **CLI Tool**: Command-line interface in `evals/cli/` for orchestrating evaluations
+3. **Diff Edit Benchmark**: Separate command using the CLI tool that runs a comprehensive diff editing benchmark suite on real world cases, along with a streamlit dashboard displaying the results. For more details, see the [Diff Edit Benchmark README](./diff-edits/README.md). Make sure you add a `evals/diff-edits/cases` folder with all the conversation jsons. 
 
 ## Directory Structure
 
@@ -40,6 +41,12 @@ cline-repo/
 │   │   │   └── utils/                # Utility functions
 │   │   ├── package.json
 │   │   └── tsconfig.json
+│   ├── diff-edits/                   # Diff editing evaluation suite
+│   │   ├── cases/                    # Test case JSON files
+│   │   ├── results/                  # Evaluation results
+│   │   ├── diff-apply/               # Diff application logic
+│   │   ├── parsing/                  # Assistant message parsing
+│   │   └── prompts/                  # System prompts
 │   ├── repositories/                 # Cloned benchmark repositories
 │   │   ├── exercism/                 # Modified Exercism (from pashpashpash/evals)
 │   │   ├── swe-bench/                # SWE-Bench repository
@@ -148,6 +155,192 @@ Freelance-style programming tasks from the SWELancer benchmark.
 
 Multi-file software engineering tasks from the Multi-SWE-Bench repository.
 
+## Diff Edit Evaluations
+
+The Cline Evaluation System includes a specialized suite for evaluating how well models can make precise edits to files using the `replace_in_file` tool.
+
+### Overview
+
+Diff edit evaluations test a model's ability to:
+
+1. Understand file content and identify specific sections to modify
+2. Generate correct SEARCH/REPLACE blocks for targeted edits
+3. Successfully apply changes without introducing errors
+
+### Directory Structure
+
+```
+diff-edits/
+├── cases/                  # Test case JSON files
+├── results/                # Evaluation results
+├── ClineWrapper.ts         # Wrapper for model interaction
+├── TestRunner.ts           # Main test execution logic
+├── types.ts                # Type definitions
+├── diff-apply/             # Diff application logic
+├── parsing/                # Assistant message parsing
+└── prompts/                # System prompts
+```
+
+### Creating Test Cases
+
+Test cases are defined as JSON files in the `diff-edits/cases/` directory. Each test case should include:
+
+```json
+{
+  "test_id": "example_test_1",
+  "messages": [
+    {
+      "role": "user",
+      "text": "Please fix the bug in this code...",
+      "images": []
+    },
+    {
+      "role": "assistant",
+      "text": "I'll help you fix that bug..."
+    }
+  ],
+  "file_contents": "// Original file content here\nfunction example() {\n  // Code with bug\n}",
+  "file_path": "src/example.js",
+  "system_prompt_details": {
+    "mcp_string": "",
+    "cwd_value": "/path/to/working/directory",
+    "browser_use": false,
+    "width": 900,
+    "height": 600,
+    "os_value": "macOS",
+    "shell_value": "/bin/zsh",
+    "home_value": "/Users/username",
+    "user_custom_instructions": ""
+  },
+  "original_diff_edit_tool_call_message": ""
+}
+```
+
+### Running Diff Edit Evaluations
+
+#### Single Model Evaluation
+
+```bash
+cd evals/cli
+node dist/index.js run-diff-eval --model-ids "anthropic/claude-3-5-sonnet-20241022"
+```
+
+#### Multi-Model Evaluation
+
+Compare multiple models in a single evaluation run:
+
+```bash
+# Compare Claude and Grok models
+node dist/index.js run-diff-eval \
+  --model-ids "anthropic/claude-3-5-sonnet-20241022,x-ai/grok-beta" \
+  --max-cases 10 \
+  --valid-attempts-per-case 3 \
+  --verbose
+
+# Compare multiple Claude variants
+node dist/index.js run-diff-eval \
+  --model-ids "anthropic/claude-3-5-sonnet-20241022,anthropic/claude-3-5-haiku-20241022,anthropic/claude-3-opus-20240229" \
+  --max-cases 5 \
+  --valid-attempts-per-case 2 \
+  --parallel
+```
+
+#### Options
+
+- `--model-ids`: Comma-separated list of model IDs to evaluate (required)
+- `--system-prompt-name`: System prompt to use (default: "basicSystemPrompt")
+- `--valid-attempts-per-case`: Number of attempts per test case per model (default: 1)
+- `--max-cases`: Maximum number of test cases to run (default: all available)
+- `--parsing-function`: Function to parse assistant messages (default: "parseAssistantMessageV2")
+- `--diff-edit-function`: Function to apply diffs (default: "constructNewFileContentV2")
+- `--test-path`: Path to test cases (default: diff-edits/cases)
+- `--thinking-budget`: Tokens allocated for thinking (default: 0)
+- `--parallel`: Run tests in parallel (flag)
+- `--replay`: Use pre-recorded LLM output (flag)
+- `--verbose`: Enable detailed logging (flag)
+
+#### Examples
+
+```bash
+# Quick test with 2 models, 4 cases, 2 attempts each
+node dist/index.js run-diff-eval \
+  --model-ids "anthropic/claude-3-5-sonnet-20241022,x-ai/grok-beta" \
+  --max-cases 4 \
+  --valid-attempts-per-case 2 \
+  --verbose
+
+# Comprehensive evaluation with parallel execution
+node dist/index.js run-diff-eval \
+  --model-ids "anthropic/claude-3-5-sonnet-20241022,anthropic/claude-3-5-haiku-20241022" \
+  --system-prompt-name claude4SystemPrompt \
+  --valid-attempts-per-case 5 \
+  --max-cases 20 \
+  --parallel \
+  --verbose
+```
+
+### Database Storage & Analytics
+
+All evaluation results are automatically stored in a SQLite database (`diff-edits/evals.db`) for advanced analytics and comparison. The database includes:
+
+- **System Prompts**: Versioned system prompt content with hashing for deduplication
+- **Processing Functions**: Versioned parsing and diff-edit function configurations
+- **Files**: Original and edited file content with content-based hashing
+- **Runs**: Evaluation run metadata and configuration
+- **Cases**: Individual test case information with context tokens
+- **Results**: Detailed results with timing, cost, and success metrics
+
+### Interactive Dashboard
+
+Launch the Streamlit dashboard to visualize and analyze evaluation results:
+
+```bash
+cd diff-edits/dashboard
+streamlit run app.py
+```
+
+The dashboard provides:
+
+- **Model Performance Comparison**: Side-by-side comparison of success rates, latency, and costs
+- **Interactive Charts**: Success rate trends, latency vs cost analysis, and performance metrics
+- **Detailed Drill-Down**: Individual result analysis with file content viewing
+- **Run Selection**: Browse and compare different evaluation runs
+- **Real-time Updates**: Automatically refreshes with new evaluation data
+
+#### Dashboard Features
+
+1. **Hero Section**: Overview of current run with key metrics
+2. **Model Cards**: Performance cards with grades and detailed metrics
+3. **Comparison Charts**: Interactive Plotly charts for visual analysis
+4. **Result Explorer**: Detailed view of individual test results including:
+   - Original and edited file content
+   - Raw model output
+   - Parsed tool calls
+   - Timing and cost metrics
+   - Error analysis
+
+#### Quick Start Dashboard
+
+```bash
+# Run a quick evaluation
+node cli/dist/index.js run-diff-eval \
+  --model-ids "anthropic/claude-3-5-sonnet-20241022,x-ai/grok-beta" \
+  --max-cases 4 \
+  --valid-attempts-per-case 2 \
+  --verbose
+
+# Launch dashboard to view results
+cd diff-edits/dashboard && streamlit run app.py
+```
+
+### Legacy Results
+
+For backward compatibility, results are also saved as JSON files in the `diff-edits/results/` directory. The JSON results include:
+- Success/failure status
+- Extracted tool calls
+- Diff edit content
+- Token usage and cost metrics
+
 ## Metrics
 
 The evaluation system collects the following metrics:

evals/cli/package.json

@@ -1,39 +0,0 @@
-{
-	"name": "cline-evaluation-cli",
-	"version": "0.1.0",
-	"description": "CLI tool for orchestrating Cline evaluations across multiple benchmarks",
-	"main": "dist/index.js",
-	"scripts": {
-		"build": "tsc",
-		"start": "node dist/index.js",
-		"dev": "ts-node src/index.ts",
-		"test": "echo \"Error: no test specified\" && exit 1"
-	},
-	"keywords": [
-		"cline",
-		"evaluation",
-		"benchmark"
-	],
-	"author": "",
-	"license": "MIT",
-	"dependencies": {
-		"better-sqlite3": "^11.10.0",
-		"chalk": "^4.1.2",
-		"commander": "^9.4.1",
-		"execa": "^5.1.1",
-		"node-fetch": "^2.7.0",
-		"ora": "^5.4.1",
-		"sqlite": "^4.1.2",
-		"uuid": "^9.0.0",
-		"yargs": "^17.6.2"
-	},
-	"devDependencies": {
-		"@types/better-sqlite3": "^7.6.3",
-		"@types/node": "^18.11.18",
-		"@types/node-fetch": "^2.6.12",
-		"@types/uuid": "^9.0.0",
-		"@types/yargs": "^17.0.19",
-		"ts-node": "^10.9.1",
-		"typescript": "^4.9.4"
-	}
-}

evals/cli/src/commands/runDiffEval.ts

@@ -3,9 +3,10 @@ import chalk from "chalk"
 import path from "path"
 
 interface RunDiffEvalOptions {
-	modelId: string
+	modelIds: string
 	systemPromptName: string
-	numberOfRuns: number
+	validAttemptsPerCase: number
+	maxAttemptsPerCase?: number
 	parsingFunction: string
 	diffEditFunction: string
 	thinkingBudget: number
@@ -14,22 +15,26 @@ interface RunDiffEvalOptions {
 	testPath: string
 	outputPath: string
 	replay: boolean
+	replayRunId?: string
+	diffApplyFile?: string
+	saveLocally: boolean
+	maxCases?: number
 }
 
 export async function runDiffEvalHandler(options: RunDiffEvalOptions) {
 	console.log(chalk.blue("Starting diff editing evaluation..."))
 
 	// Resolve the path to the TestRunner.ts script relative to the current file
-	const scriptPath = path.resolve(__dirname, "../../../diff_editing/TestRunner.ts")
+	const scriptPath = path.resolve(__dirname, "../../../diff-edits/TestRunner.ts")
 
 	// Construct the arguments array for the execa call
 	const args = [
-		"--model-id",
-		options.modelId,
+		"--model-ids",
+		options.modelIds,
 		"--system-prompt-name",
 		options.systemPromptName,
-		"--number-of-runs",
-		String(options.numberOfRuns),
+		"--valid-attempts-per-case",
+		String(options.validAttemptsPerCase),
 		"--parsing-function",
 		options.parsingFunction,
 		"--diff-edit-function",
@@ -55,16 +60,36 @@ export async function runDiffEvalHandler(options: RunDiffEvalOptions) {
 		args.push("--replay")
 	}
 
+	if (options.replayRunId) {
+		args.push("--replay-run-id", options.replayRunId)
+	}
+
+	if (options.diffApplyFile) {
+		args.push("--diff-apply-file", options.diffApplyFile)
+	}
+
 	if (options.verbose) {
 		args.push("--verbose")
 	}
 
+	if (options.maxAttemptsPerCase) {
+		args.push("--max-attempts-per-case", String(options.maxAttemptsPerCase))
+	}
+
+	if (options.maxCases) {
+		args.push("--max-cases", String(options.maxCases))
+	}
+
+	if (options.saveLocally) {
+		args.push("--save-locally")
+	}
+
 	try {
 		console.log(chalk.gray(`Executing: npx tsx ${scriptPath} ${args.join(" ")}`))
 
 		// Execute the script as a child process
 		// We use 'inherit' to stream the stdout/stderr directly to the user's terminal
-		const subprocess = execa("npx", ["tsx", scriptPath, ...args], {
+		const subprocess = execa("npx", ["tsx", "--tsconfig", path.resolve(__dirname, "../../../tsconfig.json"), scriptPath, ...args], {
 			stdio: "inherit",
 		})
 

evals/cli/src/index.ts

@@ -84,22 +84,28 @@ program
 	.description("Run the diff editing evaluation suite")
 	.option("--test-path <path>", "Path to the directory containing test case JSON files")
 	.option("--output-path <path>", "Path to the directory to save the test output JSON files")
-	.option("--model-id <model_id>", "The model ID to use for the test")
+	.option("--model-ids <model_ids>", "Comma-separated list of model IDs to test")
 	.option("--system-prompt-name <name>", "The name of the system prompt to use", "basicSystemPrompt")
-	.option("-n, --number-of-runs <number>", "Number of times to run each test case", "1")
+	.option("-n, --valid-attempts-per-case <number>", "Number of valid attempts per test case per model (will retry until this many valid attempts are collected)", "1")
+	.option("--max-attempts-per-case <number>", "Maximum total attempts per test case (default: 10x valid attempts)")
+	.option("--max-cases <number>", "Maximum number of test cases to run (limits total cases loaded)")
 	.option("--parsing-function <name>", "The parsing function to use", "parseAssistantMessageV2")
 	.option("--diff-edit-function <name>", "The diff editing function to use", "constructNewFileContentV2")
 	.option("--thinking-budget <tokens>", "Set the thinking tokens budget", "0")
 	.option("--parallel", "Run tests in parallel", false)
 	.option("--replay", "Run evaluation from a pre-recorded LLM output, skipping the API call", false)
+	.option("--replay-run-id <run_id>", "The ID of the run to replay from the database")
+	.option("--diff-apply-file <filename>", "The name of the diff apply file to use for the replay")
+	.option("--save-locally", "Save results to local JSON files in addition to database", false)
 	.option("-v, --verbose", "Enable verbose logging", false)
 	.action(async (options) => {
 		try {
-			// The logic here simplifies slightly
 			const fullOptions = {
 				...options,
-				numberOfRuns: parseInt(options.numberOfRuns, 10),
+				validAttemptsPerCase: parseInt(options.validAttemptsPerCase, 10),
+				maxAttemptsPerCase: options.maxAttemptsPerCase ? parseInt(options.maxAttemptsPerCase, 10) : undefined,
 				thinkingBudget: parseInt(options.thinkingBudget, 10),
+				maxCases: options.maxCases ? parseInt(options.maxCases, 10) : undefined,
 			}
 			await runDiffEvalHandler(fullOptions)
 		} catch (error) {

evals/diff-edits/ClineWrapper.ts

@@ -8,11 +8,13 @@ import {
 	parseAssistantMessageV3,
 	AssistantMessageContent,
 } from "./parsing/parse-assistant-message-06-06-25" // "../../src/core/assistant-message"
-import { constructNewFileContent as constructNewFileContentV1, constructNewFileContentV2 } from "./diff-apply/diff-06-06-25"
-import { constructNewFileContent as constructNewFileContentV3 } from "../../src/core/assistant-message/diff" // this defaults to the new v1 when called
+import { constructNewFileContent as constructNewFileContent_06_06_25 } from "./diff-apply/diff-06-06-25"
+import { constructNewFileContent as constructNewFileContent_06_23_25 } from "./diff-apply/diff-06-23-25"
+import { constructNewFileContent as constructNewFileContent_06_25_25 } from "./diff-apply/diff-06-25-25"
+import { constructNewFileContent as constructNewFileContent_06_26_25 } from "./diff-apply/diff-06-26-25"
 
 type ParseAssistantMessageFn = (message: string) => AssistantMessageContent[]
-type ConstructNewFileContentFn = (diff: string, original: string, strict: boolean) => Promise<string>
+type ConstructNewFileContentFn = (diff: string, original: string, strict: boolean) => Promise<string | any>
 
 const parsingFunctions: Record<string, ParseAssistantMessageFn> = {
 	parseAssistantMessageV1: parseAssistantMessageV1,
@@ -21,12 +23,14 @@ const parsingFunctions: Record<string, ParseAssistantMessageFn> = {
 }
 
 const diffEditingFunctions: Record<string, ConstructNewFileContentFn> = {
-	constructNewFileContentV1: constructNewFileContentV1,
-	constructNewFileContentV2: constructNewFileContentV2,
-	constructNewFileContentV3: constructNewFileContentV3, // position invariant diff
+	"diff-06-06-25": constructNewFileContent_06_06_25,
+	"diff-06-23-25": constructNewFileContent_06_23_25,
+	"diff-06-25-25": constructNewFileContent_06_25_25,
+	"diff-06-26-25": constructNewFileContent_06_26_25,
 }
 
 import { TestInput, TestResult, ExtractedToolCall } from "./types"
+import { log } from "./helpers"
 export { TestInput, TestResult, ExtractedToolCall }
 
 interface StreamResult {
@@ -39,16 +43,22 @@ interface StreamResult {
 		cacheReadTokens: number
 		totalCost: number
 	}
+	timing?: {
+		timeToFirstTokenMs: number
+		timeToFirstEditMs?: number
+		totalRoundTripMs: number
+	}
 }
 
 /**
- * Process the stream and return full response
+ * Process the stream and return full response with timing data
  */
 async function processStream(
 	handler: OpenRouterHandler,
 	systemPrompt: string,
 	messages: Anthropic.Messages.MessageParam[],
 ): Promise<StreamResult> {
+	const startTime = Date.now()
 	const stream = handler.createMessage(systemPrompt, messages)
 
 	let assistantMessage = ""
@@ -58,12 +68,21 @@ async function processStream(
 	let cacheWriteTokens = 0
 	let cacheReadTokens = 0
 	let totalCost = 0
+	
+	// Timing tracking
+	let timeToFirstTokenMs: number | null = null
+	let timeToFirstEditMs: number | null = null
 
 	for await (const chunk of stream) {
 		if (!chunk) {
 			continue
 		}
 
+		// Capture time to first token (any chunk type)
+		if (timeToFirstTokenMs === null) {
+			timeToFirstTokenMs = Date.now() - startTime
+		}
+
 		switch (chunk.type) {
 			case "usage":
 				inputTokens += chunk.inputTokens
@@ -79,10 +98,25 @@ async function processStream(
 				break
 			case "text":
 				assistantMessage += chunk.text
+				
+				// Try to detect first tool call by parsing accumulated message
+				if (timeToFirstEditMs === null) {
+					try {
+						const parsed = parseAssistantMessageV2(assistantMessage)
+						const hasToolCall = parsed.some(block => block.type === "tool_use")
+						if (hasToolCall) {
+							timeToFirstEditMs = Date.now() - startTime
+						}
+					} catch {
+						// Parsing failed, continue accumulating
+					}
+				}
 				break
 		}
 	}
 
+	const totalRoundTripMs = Date.now() - startTime
+
 	return {
 		assistantMessage,
 		reasoningMessage,
@@ -93,6 +127,11 @@ async function processStream(
 			cacheReadTokens,
 			totalCost,
 		},
+		timing: {
+			timeToFirstTokenMs: timeToFirstTokenMs || 0,
+			timeToFirstEditMs: timeToFirstEditMs || undefined,
+			totalRoundTripMs,
+		},
 	}
 }
 
@@ -116,6 +155,7 @@ export async function runSingleEvaluation(input: TestInput): Promise<TestResult>
 			diffEditFunction,
 			thinkingBudgetTokens,
 			originalDiffEditToolCallMessage,
+			diffApplyFile,
 		} = input
 
 		const requiredParams = {
@@ -141,7 +181,7 @@ export async function runSingleEvaluation(input: TestInput): Promise<TestResult>
 		}
 
 		const parseAssistantMessage = parsingFunctions[parsingFunction]
-		const constructNewFileContent = diffEditingFunctions[diffEditFunction]
+		const constructNewFileContent = diffEditingFunctions[diffApplyFile || diffEditFunction]
 
 		if (!parseAssistantMessage || !constructNewFileContent) {
 			return {
@@ -245,7 +285,22 @@ export async function runSingleEvaluation(input: TestInput): Promise<TestResult>
 		}
 
 		// check that we are editing the correct file path
+		log(input.isVerbose, `Expected file path: "${originalFilePath}"`)
+		log(input.isVerbose, `Actual file path used: "${diffToolPath}"`)
 		if (diffToolPath !== originalFilePath) {
+			log(input.isVerbose, `❌ File path mismatch detected!`)
+			// Enhanced logging:
+			if (streamResult?.assistantMessage) {
+				log(input.isVerbose, `   Full model output (assistantMessage):`)
+				log(input.isVerbose, `   -----------------------------------------`)
+				log(input.isVerbose, `   ${streamResult.assistantMessage}`)
+				log(input.isVerbose, `   -----------------------------------------`)
+			}
+			if (toolCall) {
+				log(input.isVerbose, `   Parsed tool call that caused mismatch:`)
+				log(input.isVerbose, `   ${JSON.stringify(toolCall, null, 2)}`)
+				log(input.isVerbose, `   -----------------------------------------`)
+			}
 			return {
 				success: false,
 				streamResult: streamResult,
@@ -256,10 +311,18 @@ export async function runSingleEvaluation(input: TestInput): Promise<TestResult>
 
 		// checking if the diff edit succeeds, if it failed it will throw an error
 		let diffSuccess = true
+		let replacementData: any = undefined
 		try {
-			await constructNewFileContent(diffToolContent, originalFile, true)
+			const result = await constructNewFileContent(diffToolContent, originalFile, true)
+			
+			// Check if result is an object with replacements (new format)
+			if (typeof result === 'object' && result !== null && 'replacements' in result) {
+				replacementData = result.replacements
+			}
+			// If it's just a string, diffSuccess stays true and replacementData stays undefined
 		} catch (error: any) {
 			diffSuccess = false
+			log(input.isVerbose, `ERROR: ${error}`)
 		}
 
 		return {
@@ -268,6 +331,7 @@ export async function runSingleEvaluation(input: TestInput): Promise<TestResult>
 			toolCalls: detectedToolCalls,
 			diffEdit: diffToolContent,
 			diffEditSuccess: diffSuccess,
+			replacementData: replacementData,
 		}
 	} catch (error: any) {
 		return {

evals/diff-edits/README.md

@@ -0,0 +1,84 @@
+# A Note on Cline's Diff Evaluation Setup
+
+Hey there, this note explains what we're doing with Cline's diff evaluation (evals) system. It's all about checking how well various AI models (which users connect to Cline via their own API keys), prompts, and diffing tools can handle file changes.
+
+## What We're Trying to Figure Out
+
+The main idea here is to figure out which AI models (configured by users) are best at making `replace_in_file` tool calls that work correctly. This helps us understand model capabilities and also speeds up our own experiments with prompts and diffing algorithms to make Cline better over time. We want to know a few key things.
+
+First, can the model create diffs, which are just sets of SEARCH and REPLACE blocks, that apply cleanly to a file? This is what we call `diffEditSuccess`.
+
+Second, how do different LLMs, like Claude or Grok, stack up against each other when they try to make these diff edits? We use a standard set of real-world test cases for this.
+
+Third, do different system prompts, say our `basicSystemPrompt` versus the `claude4SystemPrompt`, change how well a model does at diff editing?
+
+Fourth, we're also looking at different ways to apply the diffs themselves. We have a few algorithms like `constructNewFileContentV1`, `V2`, and `V3`, and we want to see which ones are more robust when fed model-generated diffs.
+
+Fifth, we track how fast the model starts making an edit. The `timeToFirstEditMs` metric gives us a hint about how quickly a user would see changes happening in their editor.
+
+And finally, we keep an eye on how many tokens are used and what it costs for each model and each try. This helps us compare how efficient they are.
+
+Right now, these evals are mostly about whether the diff *applies* correctly. That means, do the SEARCH blocks find a match, and can the REPLACE blocks be put in without an error? We're not yet deeply analyzing if the change is valid code or matches what the user *wanted* semantically. That's a problem for another day, and will require a lot more scaffolding.
+
+## How We Run These Tests
+
+Two prerequisites:
+
+1. Make sure you have an `evals/.env` file with `OPENROUTER_API_KEY=<your-openrouter-key>` 
+
+2. Make sure you add a `evals/diff-edits/cases` folder with all the conversation jsons prior to running this. 
+
+
+Our testing strategy is based on replaying situations from actual user sessions where diff edits were tried.
+
+It starts with our test cases. Each one is a JSON file in `./cases` that has the conversation history that led to a diff edit, the original file content and its path, and the info needed to rebuild the system prompt from that original session.
+
+Then, for every test run, we set up a specific configuration. This includes which LLM we're testing, which system prompt it gets, which function we use to parse the model's raw output, and which function we use to actually apply the diff. Here's the command I've been using:
+
+```bash
+npm run diff-eval -- --model-ids "anthropic/claude-3-5-sonnet,x-ai/grok-3-beta,anthropic/claude-3.7-sonnet,anthropic/claude-sonnet-4,google/gemini-2.5-pro-preview,google/gemini-2.5-flash" --max-cases 5 --valid-attempts-per-case 5 --parallel --diff-edit-function diff-06-26-25 --verbose 
+```
+
+This will build the eval script, run it, and then open the streamlit dashboard to show the results.
+
+The `TestRunner.ts` script is the main coordinator. For each test case and setup, `ClineWrapper.ts` takes over and sends the conversation and system prompt to the LLM. We then watch the model's response as it streams in and parse it to find any tool calls.
+
+We're specifically looking for the model to make a single `replace_in_file` tool call. Multiple edits in one tool call are allowed, and recorded (in case you want to filter results by number of edits in a single tool call and compare success rate for that slice across different models/system prompts/etc). If it does, and it's for the correct file, we grab the diff content it produced. Then, the chosen diff application algorithm tries to apply that diff to the original file. We record whether this worked or not as `diffEditSuccess`.
+
+We record a bunch of data for every attempt into a database. This includes details about the model and prompt, token counts, costs, the raw output from the model, the parsed tool calls, whether it succeeded or failed, any error messages, and timing info. For a detailed explanation of the database schema, see [database.md](./database.md).
+
+A big part of this is how we handle "valid attempts," which I'll explain next.
+
+## Keeping it Fair with "Valid Attempts"
+
+LLMs can be unpredictable. If we replay an old scenario, a new model, or even the same model later, might do something completely different than what happened originally. It might call another tool or ask a question instead of trying a diff edit.
+
+Since we really want to test the *diff editing* part, we need a way to make sure we're comparing fairly. That's why we have this idea of "valid attempts."
+
+An attempt is "valid" for this benchmark if the model actually tries to do what we're interested in. This means two things. One, it must call the `replace_in_file` tool. Two, it must target the *same file path* that was targeted in the original recorded conversation for that test case.
+
+If the model does something else, like calling a different tool or picking the wrong file, we don't count that attempt against its diff editing score. Instead, we consider it an "invalid attempt" for *this specific benchmark* and simply re-run that test case with that model. We keep doing this until we've collected a set number of these "valid attempts."
+
+For example, if we ask for 5 valid attempts per test case, the system will keep re-rolling for that case until the model has tried to edit the correct file using the `replace_in_file` tool 5 times. Only then do we look at how many of those 5 valid attempts actually resulted in a successful diff application (`diffEditSuccess`).
+
+This way, if we're comparing two models and one gets a 10% success rate on its valid diff edit attempts, and another gets 90%, we have a much clearer picture of their actual diff-generating capabilities. It avoids muddying the waters with attempts where the model didn't even try to perform the specific action we're evaluating. This approach helps us isolate and measure the diff-editing skill more directly, despite the non-deterministic nature of these models.
+
+## Replays
+
+You can also use the replay argument to replay a previous benchmark run. This is super useful for iterating on our diffing algorithms without having to re-run expensive and time-consuming LLM calls.
+
+When you run an evaluation, every detail is stored in the database—including the raw, unmodified output from the model. The replay feature takes advantage of this by pulling that raw output and feeding it into a *different* diffing algorithm. This lets you isolate the performance of the diffing logic itself. We can see if a new algorithm is better at applying the exact same set of diffs that a model generated in a previous run.
+
+This process is blazingly fast and free, as it completely bypasses the need to make new API calls. It ensures a true apples-to-apples comparison between diffing strategies, since the model's output—the "ground truth" for the evaluation—remains identical.
+
+Here’s an example of how you would replay a previous run with a new diffing algorithm:
+
+```shell
+cd evals && npm run diff-eval -- --replay-run-id 9902189e-63a8-4210-a4fc-fe59e2eaf2c2 --diff-apply-file diff-06-23-25 --verbose
+```
+
+In this command:
+-   `--replay-run-id` specifies the original run we want to use as our ground truth.
+-   `--diff-apply-file` tells the script to use the new diffing logic from the `diff-06-23-25.ts` file.
+
+The script will then create a new run in the database that mirrors the original, but with the results of applying the new diffing algorithm. This allows for a direct comparison in the dashboard, helping us quickly see which of our diffing strategies is the most robust.

evals/diff-edits/dashboard/.streamlit/config.toml

@@ -0,0 +1,8 @@
+[theme]
+base="dark"
+
+[browser]
+gatherUsageStats = false
+
+[server]
+headless = true

evals/diff-edits/dashboard/README.md

@@ -0,0 +1,159 @@
+# 🚀 The Sickest Diff Edits Evaluation Dashboard Ever!
+
+A beautiful, modern Streamlit dashboard for visualizing and analyzing diff editing evaluation results with deep drill-down capabilities.
+
+## ✨ Features
+
+### 🎯 **Smart Model Comparison**
+- **Latest Run Focus**: Automatically loads and displays your most recent evaluation run
+- **Beautiful Performance Cards**: Each model gets a stunning card with performance grades (A+ to C)
+- **Best Performer Highlighting**: The top model gets special styling and a trophy 🏆
+- **Interactive Charts**: Success rate comparisons and latency vs cost analysis
+
+### 🔍 **Deep Drill-Down Analysis**
+- **Individual Result Inspection**: Click any model to see detailed results
+- **Side-by-Side File Views**: See original file content with line numbers
+- **Parsed Tool Call Analysis**: View exactly what the model tried to do
+- **Error Analysis**: Detailed error information for failed attempts
+- **Success Metrics**: Line changes, edit counts, and timing breakdowns
+
+### 🎨 **Aesthetic Design**
+- **Modern UI**: Custom CSS with Inter font, gradients, and shadows
+- **Responsive Layout**: Looks great on any screen size
+- **Color-Coded Performance**: Green for excellent, yellow for good, red for poor
+- **Smooth Animations**: Hover effects and transitions
+- **Professional Styling**: Clean, modern design that looks amazing
+
+### 📊 **Comprehensive Metrics**
+- **Success Rates**: Color-coded percentages with performance grades
+- **Timing Analysis**: First token, first edit, and round trip times
+- **Cost Tracking**: Per-result and total cost analysis
+- **Token Metrics**: Context tokens and completion tokens
+- **Edit Statistics**: Number of edits, lines added/deleted
+
+## 🚀 Quick Start
+
+1. **Install dependencies**:
+   ```bash
+   cd diff-edits/dashboard
+   pip install -r requirements.txt
+   ```
+
+2. **Launch the dashboard**:
+   ```bash
+   streamlit run app.py
+   ```
+   
+   Or use the convenient launch script:
+   ```bash
+   ./launch.sh
+   ```
+
+3. **Open your browser** to http://localhost:8501
+
+## 🎯 Dashboard Sections
+
+### **Hero Section**
+- Beautiful gradient header with run information
+- Key metrics overview (models tested, total results, success rate, cost)
+
+### **Model Performance Cards**
+- Each model displayed as a beautiful card
+- Large success rate display with color coding
+- Performance grade badges (A+, A, B+, B, C+, C)
+- Key metrics: latency, cost, results count, first token time
+- "Drill Down" button for detailed analysis
+
+### **Performance Analytics**
+- Interactive bar chart showing success rates
+- Scatter plot of latency vs cost with bubble sizes
+- Hover details and zoom capabilities
+
+### **Detailed Analysis (Drill-Down)**
+- Model-specific success rate, latency, and cost metrics
+- Individual result selector with status icons
+- Tabbed interface for different views:
+
+#### 📄 **File & Edits Tab**
+- **Side-by-side view**: Original file content with line numbers
+- **Edit analysis**: Success/failure status with detailed metrics
+- **Error display**: Clear error information for failed attempts
+- **Success metrics**: Lines added/deleted, number of edits
+- **Parsed tool calls**: JSON view of what the model attempted
+
+#### 🤖 **Raw Output Tab**
+- Complete raw model output in a code viewer
+- Monospace font for easy reading
+
+#### 🔧 **Parsed Tool Call Tab**
+- Pretty-printed JSON of parsed tool calls
+- Diff block visualization for replace_in_file calls
+- Error handling for malformed JSON
+
+#### 📊 **Metrics Tab**
+- Detailed timing metrics (first token, first edit, round trip)
+- Token and cost information
+- Context size and completion tokens
+
+## 🛠 **Technical Features**
+
+### **Smart Data Loading**
+- Automatic latest run detection
+- Efficient SQL queries with proper JOINs
+- Streamlit caching for performance
+- Error handling for missing data
+
+### **Interactive Navigation**
+- Session state management for drill-down views
+- Back button to return to overview
+- Smooth transitions between views
+
+### **Beautiful Styling**
+- Custom CSS with Google Fonts (Inter)
+- Gradient backgrounds and shadows
+- Hover effects and animations
+- Color-coded performance indicators
+- Professional card-based layout
+
+### **Responsive Design**
+- Works on desktop, tablet, and mobile
+- Flexible column layouts
+- Scalable text and metrics
+
+## 🎨 **Design Philosophy**
+
+This dashboard follows modern design principles:
+- **Clarity**: Information is easy to find and understand
+- **Beauty**: Visually appealing with professional styling
+- **Functionality**: Deep drill-down capabilities for detailed analysis
+- **Performance**: Fast loading with efficient data queries
+- **Usability**: Intuitive navigation and clear visual hierarchy
+
+## 📊 **Data Visualization**
+
+- **Plotly Charts**: Interactive, professional-looking visualizations
+- **Color Coding**: Consistent color scheme for performance levels
+- **Performance Badges**: A+ to C grading system
+- **Status Icons**: ✅ for success, ❌ for failure
+- **Metric Cards**: Clean, card-based metric display
+
+## 🔧 **Customization**
+
+The dashboard is highly customizable:
+- **CSS Styling**: Easy to modify colors, fonts, and layouts
+- **Performance Grades**: Adjustable thresholds for A/B/C grades
+- **Metrics Display**: Add or remove metrics as needed
+- **Chart Types**: Easily swap chart types or add new visualizations
+
+## 🚀 **Future Enhancements**
+
+Potential additions:
+- **Historical Trends**: Compare performance across multiple runs
+- **Export Functionality**: Download results as CSV/PDF
+- **Real-time Updates**: Auto-refresh for ongoing evaluations
+- **Custom Filters**: Filter by date range, model type, etc.
+- **Comparison Mode**: Side-by-side model comparisons
+
+---
+
+**This is the sickest eval dashboard ever!** 🔥 It combines beautiful design with powerful analysis capabilities, making it easy to understand model performance at a glance while providing deep drill-down capabilities for detailed investigation.

evals/diff-edits/dashboard/launch.sh

@@ -0,0 +1,33 @@
+#!/bin/bash
+
+# Diff Edits Evaluation Dashboard Launcher
+echo "🚀 Starting Diff Edits Evaluation Dashboard..."
+
+# Check if we're in the right directory
+if [ ! -f "app.py" ]; then
+    echo "❌ Error: app.py not found. Please run this script from the dashboard directory."
+    exit 1
+fi
+
+# Check if database exists
+if [ ! -f "../evals.db" ]; then
+    echo "⚠️  Warning: Database file ../evals.db not found."
+    echo "   Make sure you've run some evaluations first to populate the database."
+    echo "   You can run: node ../cli/dist/index.js run-diff-eval --model-id anthropic/claude-sonnet-4 --max-cases 1"
+    echo ""
+fi
+
+# Check if requirements are installed
+echo "📦 Checking Python dependencies..."
+if ! python -c "import streamlit, plotly, pandas" 2>/dev/null; then
+    echo "📥 Installing required packages..."
+    pip install -r requirements.txt
+fi
+
+echo "🌐 Launching Streamlit dashboard..."
+echo "   Dashboard will open in your browser at http://localhost:8501"
+echo "   Press Ctrl+C to stop the dashboard"
+echo ""
+
+# Launch Streamlit
+streamlit run app.py

evals/diff-edits/dashboard/pages/02_Bad_Cases.py

@@ -0,0 +1,183 @@
+import streamlit as st
+import pandas as pd
+import json
+import os # Need to import os for load_case_raw_data
+from utils import get_database_connection, guess_language_from_filepath # Absolute import
+
+st.set_page_config(
+    page_title="Case Health Inspector",
+    page_icon="🧑‍⚕️",
+    layout="wide"
+)
+
+st.title("Case Health Inspector")
+st.markdown("Identify test cases that are frequently problematic across different models and runs.")
+
+@st.cache_data
+def load_problematic_cases_summary():
+    conn = get_database_connection()
+    query = """
+    WITH case_attempts AS (
+        SELECT
+            c.task_id,
+            c.description AS case_description,
+            f_orig.filepath AS original_filepath, -- Get from files table
+            r.run_id,
+            r.model_id,
+            r.result_id,
+            (CASE WHEN (r.error_enum NOT IN (1, 6, 7) OR r.error_enum IS NULL) THEN 1 ELSE 0 END) AS is_valid_attempt,
+            (CASE WHEN (r.error_enum NOT IN (1, 6, 7) OR r.error_enum IS NULL) THEN r.succeeded ELSE NULL END) AS succeeded_on_valid
+        FROM cases c
+        JOIN results r ON c.case_id = r.case_id
+        LEFT JOIN files f_orig ON c.file_hash = f_orig.hash -- Join to get original filepath
+    ),
+    case_summary AS (
+        SELECT
+            task_id,
+            case_description,
+            original_filepath, -- This is now f_orig.filepath
+            COUNT(DISTINCT run_id) AS num_benchmark_runs,
+            COUNT(result_id) AS total_attempts,
+            SUM(is_valid_attempt) AS total_valid_attempts,
+            SUM(succeeded_on_valid) AS total_successful_valid_attempts
+        FROM case_attempts
+        GROUP BY task_id, case_description, original_filepath -- original_filepath is f_orig.filepath
+    )
+    SELECT
+        task_id,
+        case_description,
+        original_filepath, -- This is f_orig.filepath from case_summary
+        num_benchmark_runs,
+        total_attempts,
+        total_valid_attempts,
+        CAST(total_valid_attempts AS REAL) * 100.0 / total_attempts AS percent_valid_attempts,
+        CASE 
+            WHEN total_valid_attempts > 0 THEN CAST(total_successful_valid_attempts AS REAL) * 100.0 / total_valid_attempts
+            ELSE 0 
+        END AS success_rate_on_valid
+    FROM case_summary
+    ORDER BY percent_valid_attempts ASC, success_rate_on_valid ASC;
+    """
+    df = pd.read_sql_query(query, conn)
+    return df
+
+@st.cache_data
+def load_case_raw_data(task_id):
+    """Loads the original JSON data for a given task_id."""
+    # This assumes test cases are stored in ../cases relative to this script's parent (dashboard)
+    # So, ../../cases from this script's location (pages/02_Bad_Cases.py)
+    # Correct path from this script (pages/02_Bad_Cases.py) to cases/
+    # os.path.dirname(__file__) -> pages
+    # os.path.join(..., '..') -> dashboard
+    # os.path.join(..., '..', '..') -> diff-edits
+    # os.path.join(..., '..', '..', 'cases') -> diff-edits/cases
+    cases_dir = os.path.join(os.path.dirname(__file__), '..', '..', 'cases')
+    
+    # The task_id is usually the filename without .json
+    # However, some task_ids might have suffixes or be different.
+    # We need a robust way to find the file. For now, assume task_id is filename base.
+    # This might need adjustment if task_id format varies significantly from filename.
+    
+    # Try direct match first
+    potential_filename = f"{task_id}.json"
+    filepath = os.path.join(cases_dir, potential_filename)
+
+    if not os.path.exists(filepath):
+        # If direct match fails, list files and try to find one that starts with task_id
+        # This is a simple fallback, might need more robust matching if task_ids are complex
+        try:
+            for f_name in os.listdir(cases_dir):
+                if f_name.startswith(task_id) and f_name.endswith(".json"):
+                    filepath = os.path.join(cases_dir, f_name)
+                    break
+            else: # No break means no file found
+                 return None # File not found
+        except FileNotFoundError:
+            return None # Cases directory itself not found
+
+    if not os.path.exists(filepath): # Check again after potential find
+        return None
+
+    try:
+        with open(filepath, 'r') as f:
+            return json.load(f)
+    except Exception as e:
+        st.error(f"Error loading case file {filepath}: {e}")
+        return None
+
+def render_problematic_cases_page():
+    summary_df = load_problematic_cases_summary()
+
+    if summary_df.empty:
+        st.warning("No case summary data found. Run some evaluations first.")
+        return
+
+    st.markdown("### Cases Overview")
+    st.dataframe(summary_df.style.format({
+        "percent_valid_attempts": "{:.1f}%",
+        "success_rate_on_valid": "{:.1f}%"
+    }), use_container_width=True)
+
+    st.markdown("---")
+    st.markdown("### Case Drill Down")
+    
+    selected_task_id = st.selectbox(
+        "Select a Case ID (task_id) to inspect:",
+        options=[""] + summary_df['task_id'].tolist() # Add a blank option
+    )
+
+    if selected_task_id:
+        case_data = summary_df[summary_df['task_id'] == selected_task_id].iloc[0]
+        st.subheader(f"Details for Case: {case_data['task_id']}")
+        st.markdown(f"**Description:** {case_data['case_description']}")
+        st.markdown(f"**Original Filepath:** `{case_data['original_filepath']}`")
+        
+        raw_json_data = load_case_raw_data(selected_task_id)
+        if raw_json_data:
+            with st.expander("View Raw Case JSON Data", expanded=False):
+                st.json(raw_json_data)
+            
+            if 'file_contents' in raw_json_data and raw_json_data['file_contents']:
+                with st.expander("View Original File Content (from Case JSON)", expanded=True):
+                    # Prepare content for the copy button
+                    raw_content_for_copy = raw_json_data['file_contents']
+                    js_escaped_content = raw_content_for_copy.replace('\\', '\\\\') \
+                                                           .replace('`', '\\`') \
+                                                           .replace('\r\n', '\\n') \
+                                                           .replace('\n', '\\n') \
+                                                           .replace('\r', '\\n')
+                    button_id = f"copyBtnCase_{selected_task_id.replace('-', '_').replace('.', '_')}"
+                    copy_button_html = f"""
+                        <button id="{button_id}" onclick="copyCaseContentToClipboard(`{js_escaped_content}`, '{button_id}')" style="margin-bottom: 10px; padding: 5px 10px; border-radius: 5px; border: 1px solid #ccc; cursor: pointer;">Copy File Content</button>
+                        <script>
+                            if (!window.copyCaseContentToClipboard) {{
+                                window.copyCaseContentToClipboard = async function(text, buttonId) {{
+                                    try {{
+                                        await navigator.clipboard.writeText(text);
+                                        const button = document.getElementById(buttonId);
+                                        button.innerText = 'Copied!';
+                                        setTimeout(() => {{ button.innerText = 'Copy File Content'; }}, 2000);
+                                    }} catch (err) {{ console.error('Failed to copy: ', err); const button = document.getElementById(buttonId); button.innerText = 'Copy Failed!'; setTimeout(() => {{ button.innerText = 'Copy File Content'; }}, 2000); }}
+                                }}
+                            }}
+                        </script>
+                        """
+                    st.components.v1.html(copy_button_html, height=50)
+
+                    # Prepare content for st.code
+                    content_for_display = raw_json_data['file_contents']
+                    content_for_display = content_for_display.replace('\\\\r\\\\n', '\r\n').replace('\\\\n', '\n')
+                    content_for_display = content_for_display.replace('\\r\\n', '\r\n').replace('\\n', '\n')
+                    
+                    language = guess_language_from_filepath(case_data['original_filepath'])
+                    st.code(content_for_display, language=language, line_numbers=False)
+            else:
+                st.warning("Original file content not found in case JSON.")
+        else:
+            st.error(f"Could not load raw JSON data for case: {selected_task_id}")
+        
+        # Placeholder for more detailed stats (per-model performance on this case, error breakdown)
+        st.markdown("*(Further per-model statistics and error breakdowns for this case can be added here.)*")
+
+if __name__ == "__main__":
+    render_problematic_cases_page()

evals/diff-edits/dashboard/requirements.txt

@@ -0,0 +1,4 @@
+streamlit>=1.28.0
+plotly>=5.17.0
+pandas>=2.0.0
+numpy>=1.24.0

evals/diff-edits/dashboard/utils.py

@@ -0,0 +1,51 @@
+import streamlit as st
+import sqlite3
+import pandas as pd
+import os
+
+@st.cache_resource
+def get_database_connection():
+    # Assuming the script is run from the dashboard directory,
+    # evals.db is two levels up from there.
+    # __file__ is utils.py, its dirname is dashboard.
+    # os.path.dirname(__file__) -> dashboard/
+    # os.path.join(..., '..') -> diff-edits/
+    # os.path.join(..., '..', 'evals.db') -> diff-edits/evals.db
+    db_path = os.path.join(os.path.dirname(__file__), '..', 'evals.db')
+    if not os.path.exists(db_path):
+        st.error(f"Database not found. Expected at: {os.path.abspath(db_path)}")
+        st.stop()
+    return sqlite3.connect(db_path, check_same_thread=False)
+
+def guess_language_from_filepath(filepath):
+    """Guess the language for syntax highlighting from filepath."""
+    if not filepath or pd.isna(filepath):
+        return None
+    
+    extension_map = {
+        '.py': 'python',
+        '.js': 'javascript',
+        '.ts': 'typescript',
+        '.java': 'java',
+        '.cs': 'csharp',
+        '.cpp': 'cpp',
+        '.c': 'c',
+        '.html': 'html',
+        '.css': 'css',
+        '.json': 'json',
+        '.sql': 'sql',
+        '.md': 'markdown',
+        '.rb': 'ruby',
+        '.php': 'php',
+        '.go': 'go',
+        '.rs': 'rust',
+        '.swift': 'swift',
+        '.kt': 'kotlin',
+        '.sh': 'bash',
+        '.yaml': 'yaml',
+        '.yml': 'yaml',
+        '.xml': 'xml',
+    }
+    
+    _, ext = os.path.splitext(str(filepath)) # Ensure filepath is string
+    return extension_map.get(ext.lower(), None)

evals/diff-edits/database.md

@@ -0,0 +1,96 @@
+# Diff Edit Evaluation Database Schema
+
+This document provides an overview of the SQLite database schema used for the diff edit evaluation suite. The database is designed to capture every aspect of the evaluation runs in a structured way, allowing for detailed, multi-dimensional analysis and ensuring full reproducibility of our findings.
+
+## Data Model Overview
+
+The database is composed of several interconnected tables that work together to provide a comprehensive picture of each evaluation. The core of the model revolves around `runs`, `cases`, and `results`.
+
+### `runs`
+
+A `run` represents a single, top-level execution of the evaluation script (e.g., one invocation of `npm run diff-eval`). It serves as the main container for a complete benchmark session.
+
+-   **Purpose**: To group all the results from a single benchmark execution, allowing for high-level comparison between different runs over time.
+-   **Key Columns**:
+    -   `run_id`: A unique identifier for the entire run.
+    -   `description`: A human-readable summary of the run's configuration (e.g., which models were tested, how many cases, etc.).
+    -   `system_prompt_hash`: A foreign key that links this run to the specific system prompt that was used, ensuring we can track performance changes based on prompt modifications.
+
+### `cases`
+
+A `case` represents a single test scenario that is presented to a model. It corresponds to one of the JSON files in the `cases/` directory and links that static definition to a specific benchmark `run`.
+
+-   **Purpose**: To track the individual test scenarios within a given run.
+-   **Key Columns**:
+    -   `case_id`: A unique identifier for the case *within* a specific run.
+    -   `run_id`: A foreign key linking back to the parent `run`.
+    -   `task_id`: The original, persistent identifier for the test case (typically from the JSON filename).
+    -   `file_hash`: A foreign key linking to the original, un-edited file content for this case.
+
+### `results`
+
+This is the most granular and important table in the database. A `result` represents the outcome of a single attempt by a specific model on a specific case.
+
+-   **Purpose**: To store the detailed outcome of every single model attempt, providing the raw data for all quantitative and qualitative analysis.
+-   **Key Columns**:
+    -   `result_id`: The primary key for the result.
+    -   `run_id`, `case_id`, `model_id`, `processing_functions_hash`: A set of foreign keys that precisely situate this result within the context of a specific run, case, model, and set of helper functions.
+    -   `succeeded`: A boolean indicating if the generated diff was applied successfully.
+    -   `error_enum`: A numeric code representing the specific type of error if the attempt failed (e.g., `1` for `no_tool_calls`, `7` for `wrong_file_edited`).
+    -   `num_edits`, `num_lines_deleted`, `num_lines_added`: Quantitative metrics about the structure of the generated diff.
+    -   `time_to_first_token_ms`, `time_to_first_edit_ms`, `time_round_trip_ms`: High-precision timing data to measure model latency.
+    -   `cost_usd`, `completion_tokens`: Cost and token usage metrics for efficiency analysis.
+    -   `raw_model_output`, `file_edited_hash`, `parsed_tool_call_json`: The rich, qualitative data. This includes the model's full, raw response and the parsed tool calls, which are invaluable for debugging and understanding the model's reasoning.
+
+---
+
+## Supporting Tables
+
+The following tables store versioned, deduplicated content to ensure data integrity and efficiency.
+
+### `system_prompts`
+
+-   **Purpose**: Stores the versioned content of the system prompts used in evaluations.
+-   **Key Columns**:
+    -   `hash`: A unique hash of the prompt's content, which acts as the primary key. This prevents duplicate storage of the same prompt.
+    -   `name`: A human-readable name for the prompt (e.g., `basicSystemPrompt`, `claude4SystemPrompt`).
+    -   `content`: The full text of the system prompt.
+
+### `processing_functions`
+
+-   **Purpose**: Stores the versioned combinations of parsing and diff-editing functions.
+-   **Key Columns**:
+    -   `hash`: A unique hash of the function combination name.
+    -   `name`: A human-readable name (e.g., `parseV2-diffV2`).
+    -   `parsing_function`: The name of the function used to parse the model's output.
+    -   `diff_edit_function`: The name of the function used to apply the diff.
+
+### `files`
+
+-   **Purpose**: Stores the content of all files involved in the tests, including the original source files and the diffs generated by the models.
+-   **Key Columns**:
+    -   `hash`: A content-based hash of the file, ensuring that identical files are only stored once.
+    -   `filepath`: The original path of the file.
+    -   `content`: The full content of the file.
+
+## The Bigger Picture
+
+This relational schema provides a powerful foundation for sophisticated analysis. It moves beyond simple pass/fail metrics and allows us to explore the nuanced interactions between models, prompts, and the code they operate on. With this database, we can answer critical questions like:
+
+-   "How does prompt engineering affect not just success rate, but also latency and cost?"
+-   "Are certain models more prone to specific types of errors (e.g., hallucinating file paths vs. failing to call a tool)?"
+-   "Which of our internal diffing algorithms is the most robust against a wide range of model-generated edits?"
+
+Ultimately, this data model enables us to move from simply *measuring* performance to truly *understanding* it, providing the insights needed to build more capable and reliable AI engineering systems.
+
+---
+
+## Viewing the Full Schema
+
+To see the most up-to-date and detailed schema for the database, you can use the `sqlite3` command-line tool. From the `evals/diff-edits` directory, run the following command:
+
+```bash
+sqlite3 evals.db .schema
+```
+
+This will print the complete `CREATE TABLE` statements for all tables in the database, providing a definitive reference for the database structure.

evals/diff-edits/database/client.ts

@@ -0,0 +1,135 @@
+import Database from 'better-sqlite3';
+import * as fs from 'fs';
+import * as path from 'path';
+import * as crypto from 'crypto';
+
+export class DatabaseClient {
+  private static instance: DatabaseClient;
+  private db: Database.Database;
+  private dbPath: string;
+
+  private constructor() {
+    // Get database path from environment or use default
+    this.dbPath = process.env.DIFF_EVALS_DB_PATH || path.join(__dirname, '../evals.db');
+    
+    // Ensure directory exists
+    const dbDir = path.dirname(this.dbPath);
+    if (!fs.existsSync(dbDir)) {
+      fs.mkdirSync(dbDir, { recursive: true });
+    }
+
+    // Initialize database connection
+    this.db = new Database(this.dbPath);
+    
+    // Enable WAL mode for concurrent access
+    this.db.pragma('journal_mode = WAL');
+    
+    // Enable foreign key constraints
+    this.db.pragma('foreign_keys = ON');
+    
+    // Initialize schema if needed
+    this.initializeSchema();
+  }
+
+  static getInstance(): DatabaseClient {
+    if (!DatabaseClient.instance) {
+      DatabaseClient.instance = new DatabaseClient();
+    }
+    return DatabaseClient.instance;
+  }
+
+  private initializeSchema(): void {
+    // Check if tables exist by trying to query one of them
+    try {
+      this.db.prepare('SELECT COUNT(*) FROM system_prompts LIMIT 1').get();
+      // If we get here, tables exist
+      return;
+    } catch (error) {
+      // Tables don't exist, create them
+      console.log('Initializing database schema...');
+      this.createTables();
+    }
+  }
+
+  private createTables(): void {
+    const schemaPath = path.join(__dirname, 'schema.sql');
+    const schema = fs.readFileSync(schemaPath, 'utf8');
+    
+    // Execute the entire schema as one block
+    this.db.transaction(() => {
+      this.db.exec(schema);
+    })();
+
+    console.log('Database schema initialized successfully');
+  }
+
+  getDatabase(): Database.Database {
+    return this.db;
+  }
+
+  getDatabasePath(): string {
+    return this.dbPath;
+  }
+
+  // Utility method to generate SHA-256 hash
+  static generateHash(content: string): string {
+    return crypto.createHash('sha256').update(content).digest('hex');
+  }
+
+  // Utility method to generate UUID-like ID
+  static generateId(): string {
+    return crypto.randomUUID();
+  }
+
+  // Transaction wrapper
+  transaction<T>(fn: () => T): T {
+    return this.db.transaction(fn)();
+  }
+
+  // Close database connection (for cleanup)
+  close(): void {
+    if (this.db) {
+      this.db.close();
+    }
+  }
+
+  // Get database info
+  getInfo(): { path: string; size: number; tables: string[] } {
+    const stats = fs.statSync(this.dbPath);
+    const tables = this.db
+      .prepare("SELECT name FROM sqlite_master WHERE type='table' ORDER BY name")
+      .all()
+      .map((row: any) => row.name);
+
+    return {
+      path: this.dbPath,
+      size: stats.size,
+      tables
+    };
+  }
+
+  // Vacuum database (cleanup and optimize)
+  vacuum(): void {
+    this.db.exec('VACUUM');
+  }
+
+  // Get database statistics
+  getStats(): { [tableName: string]: number } {
+    const tables = ['system_prompts', 'processing_functions', 'files', 'runs', 'cases', 'results'];
+    const stats: { [tableName: string]: number } = {};
+
+    for (const table of tables) {
+      try {
+        const result = this.db.prepare(`SELECT COUNT(*) as count FROM ${table}`).get() as { count: number };
+        stats[table] = result.count;
+      } catch (error) {
+        stats[table] = 0;
+      }
+    }
+
+    return stats;
+  }
+}
+
+// Export singleton instance getter
+export const getDatabase = () => DatabaseClient.getInstance();

evals/diff-edits/database/index.ts

@@ -0,0 +1,23 @@
+// Main database module exports
+export { DatabaseClient, getDatabase } from './client';
+export * from './types';
+export * from './operations';
+export * from './queries';
+
+// Re-export commonly used functions for convenience
+export {
+  upsertSystemPrompt,
+  upsertProcessingFunctions,
+  upsertFile,
+  createBenchmarkRun,
+  createCase,
+  insertResult,
+  getRunStats
+} from './operations';
+
+export {
+  getSuccessRatesByModel,
+  getModelComparisons,
+  getDatabaseSummary,
+  getErrorDistribution
+} from './queries';

evals/diff-edits/database/operations.ts

@@ -0,0 +1,348 @@
+import { DatabaseClient } from './client';
+import {
+  SystemPrompt,
+  ProcessingFunctions,
+  FileRecord,
+  BenchmarkRun,
+  Case,
+  Result,
+  CreateSystemPromptInput,
+  CreateProcessingFunctionsInput,
+  CreateFileInput,
+  CreateBenchmarkRunInput,
+  CreateCaseInput,
+  CreateResultInput
+} from './types';
+
+const db = DatabaseClient.getInstance();
+
+// System Prompts Operations
+export async function upsertSystemPrompt(input: CreateSystemPromptInput): Promise<string> {
+  const hash = DatabaseClient.generateHash(input.content);
+  
+  const stmt = db.getDatabase().prepare(`
+    INSERT OR IGNORE INTO system_prompts (hash, name, content)
+    VALUES (?, ?, ?)
+  `);
+  
+  stmt.run(hash, input.name, input.content);
+  return hash;
+}
+
+export async function getSystemPromptByHash(hash: string): Promise<SystemPrompt | null> {
+  const stmt = db.getDatabase().prepare(`
+    SELECT * FROM system_prompts WHERE hash = ?
+  `);
+  
+  const result = stmt.get(hash) as SystemPrompt | undefined;
+  return result || null;
+}
+
+// Processing Functions Operations
+export async function upsertProcessingFunctions(input: CreateProcessingFunctionsInput): Promise<string> {
+  const hash = DatabaseClient.generateHash(input.parsing_function + input.diff_edit_function);
+  
+  const stmt = db.getDatabase().prepare(`
+    INSERT OR IGNORE INTO processing_functions (hash, name, parsing_function, diff_edit_function)
+    VALUES (?, ?, ?, ?)
+  `);
+  
+  stmt.run(hash, input.name, input.parsing_function, input.diff_edit_function);
+  return hash;
+}
+
+export async function getProcessingFunctionsByHash(hash: string): Promise<ProcessingFunctions | null> {
+  const stmt = db.getDatabase().prepare(`
+    SELECT * FROM processing_functions WHERE hash = ?
+  `);
+  
+  const result = stmt.get(hash) as ProcessingFunctions | undefined;
+  return result || null;
+}
+
+// Files Operations
+export async function upsertFile(input: CreateFileInput): Promise<string> {
+  const hash = DatabaseClient.generateHash(input.content);
+  
+  const stmt = db.getDatabase().prepare(`
+    INSERT OR IGNORE INTO files (hash, filepath, content, tokens)
+    VALUES (?, ?, ?, ?)
+  `);
+  
+  stmt.run(hash, input.filepath, input.content, input.tokens || null);
+  return hash;
+}
+
+export async function getFileByHash(hash: string): Promise<FileRecord | null> {
+  const stmt = db.getDatabase().prepare(`
+    SELECT * FROM files WHERE hash = ?
+  `);
+  
+  const result = stmt.get(hash) as FileRecord | undefined;
+  return result || null;
+}
+
+// Benchmark Runs Operations
+export async function createBenchmarkRun(input: CreateBenchmarkRunInput): Promise<string> {
+  const runId = DatabaseClient.generateId();
+  
+  const stmt = db.getDatabase().prepare(`
+    INSERT INTO runs (run_id, description, system_prompt_hash)
+    VALUES (?, ?, ?)
+  `);
+  
+  stmt.run(runId, input.description || null, input.system_prompt_hash);
+  return runId;
+}
+
+export async function getBenchmarkRun(runId: string): Promise<BenchmarkRun | null> {
+  const stmt = db.getDatabase().prepare(`
+    SELECT * FROM runs WHERE run_id = ?
+  `);
+  
+  const result = stmt.get(runId) as BenchmarkRun | undefined;
+  return result || null;
+}
+
+export async function getAllBenchmarkRuns(): Promise<BenchmarkRun[]> {
+  const stmt = db.getDatabase().prepare(`
+    SELECT * FROM runs ORDER BY created_at DESC
+  `);
+  
+  return stmt.all() as BenchmarkRun[];
+}
+
+// Cases Operations
+export async function createCase(input: CreateCaseInput): Promise<string> {
+  const caseId = DatabaseClient.generateId();
+  
+  const stmt = db.getDatabase().prepare(`
+    INSERT INTO cases (case_id, run_id, description, system_prompt_hash, task_id, tokens_in_context, file_hash)
+    VALUES (?, ?, ?, ?, ?, ?, ?)
+  `);
+  
+  stmt.run(
+    caseId,
+    input.run_id,
+    input.description,
+    input.system_prompt_hash,
+    input.task_id,
+    input.tokens_in_context,
+    input.file_hash || null
+  );
+  
+  return caseId;
+}
+
+export async function getCasesByRun(runId: string): Promise<Case[]> {
+  const stmt = db.getDatabase().prepare(`
+    SELECT * FROM cases WHERE run_id = ? ORDER BY created_at
+  `);
+  
+  return stmt.all(runId) as Case[];
+}
+
+export async function getCaseById(caseId: string): Promise<Case | null> {
+  const stmt = db.getDatabase().prepare(`
+    SELECT * FROM cases WHERE case_id = ?
+  `);
+  
+  const result = stmt.get(caseId) as Case | undefined;
+  return result || null;
+}
+
+// Results Operations
+export async function insertResult(input: CreateResultInput): Promise<string> {
+  const resultId = DatabaseClient.generateId();
+  
+  const stmt = db.getDatabase().prepare(`
+    INSERT INTO results (
+      result_id, run_id, case_id, model_id, processing_functions_hash,
+      succeeded, error_enum, num_edits, num_lines_deleted, num_lines_added,
+      time_to_first_token_ms, time_to_first_edit_ms, time_round_trip_ms,
+      cost_usd, completion_tokens, raw_model_output, file_edited_hash,
+      parsed_tool_call_json
+    ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+  `);
+  
+  stmt.run(
+    resultId,
+    input.run_id,
+    input.case_id,
+    input.model_id,
+    input.processing_functions_hash,
+    input.succeeded ? 1 : 0, // Convert boolean to integer
+    input.error_enum || null,
+    input.num_edits || null,
+    input.num_lines_deleted || null,
+    input.num_lines_added || null,
+    input.time_to_first_token_ms || null,
+    input.time_to_first_edit_ms || null,
+    input.time_round_trip_ms || null,
+    input.cost_usd || null,
+    input.completion_tokens || null,
+    input.raw_model_output || null,
+    input.file_edited_hash || null,
+    input.parsed_tool_call_json || null
+  );
+  
+  return resultId;
+}
+
+export async function getResultsByRun(runId: string): Promise<Result[]> {
+  const stmt = db.getDatabase().prepare(`
+    SELECT * FROM results WHERE run_id = ? ORDER BY created_at
+  `);
+  
+  return stmt.all(runId) as Result[];
+}
+
+export async function getResultsByCase(caseId: string): Promise<Result[]> {
+  const stmt = db.getDatabase().prepare(`
+    SELECT * FROM results WHERE case_id = ? ORDER BY created_at
+  `);
+  
+  return stmt.all(caseId) as Result[];
+}
+
+export async function getResultById(resultId: string): Promise<Result | null> {
+  const stmt = db.getDatabase().prepare(`
+    SELECT * FROM results WHERE result_id = ?
+  `);
+  
+  const result = stmt.get(resultId) as Result | undefined;
+  return result || null;
+}
+
+// Batch operations for performance
+export async function insertResultsBatch(inputs: CreateResultInput[]): Promise<string[]> {
+  const stmt = db.getDatabase().prepare(`
+    INSERT INTO results (
+      result_id, run_id, case_id, model_id, processing_functions_hash,
+      succeeded, error_enum, num_edits, num_lines_deleted, num_lines_added,
+      time_to_first_token_ms, time_to_first_edit_ms, time_round_trip_ms,
+      cost_usd, completion_tokens, raw_model_output, file_edited_hash,
+      parsed_tool_call_json
+    ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+  `);
+  
+  return db.transaction(() => {
+    const resultIds: string[] = [];
+    
+    for (const input of inputs) {
+      const resultId = DatabaseClient.generateId();
+      
+      stmt.run(
+        resultId,
+        input.run_id,
+        input.case_id,
+        input.model_id,
+        input.processing_functions_hash,
+        input.succeeded ? 1 : 0, // Convert boolean to integer
+        input.error_enum || null,
+        input.num_edits || null,
+        input.num_lines_deleted || null,
+        input.num_lines_added || null,
+        input.time_to_first_token_ms || null,
+        input.time_to_first_edit_ms || null,
+        input.time_round_trip_ms || null,
+        input.cost_usd || null,
+        input.completion_tokens || null,
+        input.raw_model_output || null,
+        input.file_edited_hash || null,
+        input.parsed_tool_call_json || null
+      );
+      
+      resultIds.push(resultId);
+    }
+    
+    return resultIds;
+  });
+}
+
+export async function createCasesBatch(inputs: CreateCaseInput[]): Promise<string[]> {
+  const stmt = db.getDatabase().prepare(`
+    INSERT INTO cases (case_id, run_id, description, system_prompt_hash, task_id, tokens_in_context)
+    VALUES (?, ?, ?, ?, ?, ?)
+  `);
+  
+  return db.transaction(() => {
+    const caseIds: string[] = [];
+    
+    for (const input of inputs) {
+      const caseId = DatabaseClient.generateId();
+      
+      stmt.run(
+        caseId,
+        input.run_id,
+        input.description,
+        input.system_prompt_hash,
+        input.task_id,
+        input.tokens_in_context
+      );
+      
+      caseIds.push(caseId);
+    }
+    
+    return caseIds;
+  });
+}
+
+// Utility functions
+export async function getRunStats(runId: string): Promise<{
+  total_cases: number;
+  total_results: number;
+  success_rate: number;
+  avg_cost: number;
+  avg_latency: number;
+}> {
+  const stmt = db.getDatabase().prepare(`
+    SELECT 
+      COUNT(DISTINCT c.case_id) as total_cases,
+      COUNT(r.result_id) as total_results,
+      AVG(CASE WHEN r.succeeded THEN 1.0 ELSE 0.0 END) as success_rate,
+      AVG(r.cost_usd) as avg_cost,
+      AVG(r.time_round_trip_ms) as avg_latency
+    FROM cases c
+    LEFT JOIN results r ON c.case_id = r.case_id
+    WHERE c.run_id = ?
+  `);
+  
+  const result = stmt.get(runId) as any;
+  return {
+    total_cases: result.total_cases || 0,
+    total_results: result.total_results || 0,
+    success_rate: result.success_rate || 0,
+    avg_cost: result.avg_cost || 0,
+    avg_latency: result.avg_latency || 0
+  };
+}
+
+// Count valid attempts for a specific case and model
+export async function getValidAttemptCount(caseId: string, modelId: string): Promise<number> {
+  const stmt = db.getDatabase().prepare(`
+    SELECT COUNT(*) as count
+    FROM results 
+    WHERE case_id = ? 
+      AND model_id = ?
+      AND error_enum NOT IN (1, 6, 7)  -- Exclude: no_tool_calls, wrong_tool_call, wrong_file_edited
+  `);
+  
+  const result = stmt.get(caseId, modelId) as { count: number };
+  return result.count;
+}
+
+// Get valid results for a specific case and model (for analysis)
+export async function getValidResults(caseId: string, modelId: string, limit?: number): Promise<Result[]> {
+  const limitClause = limit ? `LIMIT ${limit}` : '';
+  const stmt = db.getDatabase().prepare(`
+    SELECT * FROM results 
+    WHERE case_id = ? 
+      AND model_id = ?
+      AND error_enum NOT IN (1, 6, 7)  -- Only valid attempts
+    ORDER BY created_at
+    ${limitClause}
+  `);
+  
+  return stmt.all(caseId, modelId) as Result[];
+}

evals/diff-edits/database/queries.ts

@@ -0,0 +1,309 @@
+import { DatabaseClient } from './client';
+import {
+  ModelSuccessRate,
+  ModelLatency,
+  CostAnalysis,
+  ErrorDistribution,
+  FailedCase,
+  PerformanceTrend,
+  ModelComparison
+} from './types';
+
+const db = DatabaseClient.getInstance();
+
+// Performance analysis queries
+export async function getSuccessRatesByModel(): Promise<ModelSuccessRate[]> {
+  const stmt = db.getDatabase().prepare(`
+    SELECT 
+      model_id,
+      COUNT(*) as total_runs,
+      SUM(CASE WHEN succeeded THEN 1 ELSE 0 END) as successful_runs,
+      ROUND(AVG(CASE WHEN succeeded THEN 1.0 ELSE 0.0 END) * 100, 2) as success_rate
+    FROM results
+    WHERE error_enum NOT IN (1, 6, 7) OR error_enum IS NULL  -- Exclude: no_tool_calls, wrong_tool_call, wrong_file_edited
+    GROUP BY model_id
+    ORDER BY success_rate DESC, total_runs DESC
+  `);
+  
+  return stmt.all() as ModelSuccessRate[];
+}
+
+export async function getAverageLatencyByModel(): Promise<ModelLatency[]> {
+  const stmt = db.getDatabase().prepare(`
+    SELECT 
+      model_id,
+      ROUND(AVG(time_to_first_token_ms), 2) as avg_time_to_first_token_ms,
+      ROUND(AVG(time_to_first_edit_ms), 2) as avg_time_to_first_edit_ms,
+      ROUND(AVG(time_round_trip_ms), 2) as avg_time_round_trip_ms
+    FROM results
+    WHERE time_to_first_token_ms IS NOT NULL
+    GROUP BY model_id
+    ORDER BY avg_time_round_trip_ms ASC
+  `);
+  
+  return stmt.all() as ModelLatency[];
+}
+
+export async function getCostAnalysisByRun(): Promise<CostAnalysis[]> {
+  const stmt = db.getDatabase().prepare(`
+    SELECT 
+      run_id,
+      model_id,
+      ROUND(SUM(cost_usd), 4) as total_cost_usd,
+      ROUND(AVG(cost_usd), 4) as avg_cost_per_case,
+      SUM(completion_tokens) as total_completion_tokens
+    FROM results
+    WHERE cost_usd IS NOT NULL
+    GROUP BY run_id, model_id
+    ORDER BY total_cost_usd DESC
+  `);
+  
+  return stmt.all() as CostAnalysis[];
+}
+
+// Error analysis queries
+export async function getErrorDistribution(): Promise<ErrorDistribution[]> {
+  const stmt = db.getDatabase().prepare(`
+    SELECT 
+      error_enum,
+      COUNT(*) as count,
+      ROUND(COUNT(*) * 100.0 / (SELECT COUNT(*) FROM results WHERE succeeded = 0), 2) as percentage
+    FROM results
+    WHERE succeeded = 0 AND error_enum IS NOT NULL
+    GROUP BY error_enum
+    ORDER BY count DESC
+  `);
+  
+  return stmt.all() as ErrorDistribution[];
+}
+
+export async function getFailedCasesByError(errorEnum?: number): Promise<FailedCase[]> {
+  let query = `
+    SELECT 
+      r.case_id,
+      r.model_id,
+      r.error_enum,
+      c.description,
+      r.raw_model_output
+    FROM results r
+    JOIN cases c ON r.case_id = c.case_id
+    WHERE r.succeeded = 0
+  `;
+  
+  const params: any[] = [];
+  if (errorEnum !== undefined) {
+    query += ` AND r.error_enum = ?`;
+    params.push(errorEnum);
+  }
+  
+  query += ` ORDER BY r.created_at DESC LIMIT 100`;
+  
+  const stmt = db.getDatabase().prepare(query);
+  return stmt.all(...params) as FailedCase[];
+}
+
+// Trend analysis queries
+export async function getPerformanceTrends(days: number = 30): Promise<PerformanceTrend[]> {
+  const stmt = db.getDatabase().prepare(`
+    SELECT 
+      DATE(r.created_at) as date,
+      r.model_id,
+      ROUND(AVG(CASE WHEN r.succeeded THEN 1.0 ELSE 0.0 END) * 100, 2) as success_rate,
+      ROUND(AVG(r.time_round_trip_ms), 2) as avg_latency_ms,
+      ROUND(AVG(r.cost_usd), 4) as avg_cost_usd
+    FROM results r
+    WHERE r.created_at >= datetime('now', '-' || ? || ' days')
+      AND (r.error_enum NOT IN (1, 6, 7) OR r.error_enum IS NULL)  -- Exclude: no_tool_calls, wrong_tool_call, wrong_file_edited
+    GROUP BY DATE(r.created_at), r.model_id
+    ORDER BY date DESC, model_id
+  `);
+  
+  return stmt.all(days) as PerformanceTrend[];
+}
+
+export async function getModelComparisons(): Promise<ModelComparison[]> {
+  const stmt = db.getDatabase().prepare(`
+    SELECT 
+      model_id,
+      ROUND(AVG(CASE WHEN succeeded THEN 1.0 ELSE 0.0 END) * 100, 2) as success_rate,
+      ROUND(AVG(time_round_trip_ms), 2) as avg_latency_ms,
+      ROUND(AVG(cost_usd), 4) as avg_cost_usd,
+      COUNT(*) as total_runs
+    FROM results
+    WHERE error_enum NOT IN (1, 6, 7) OR error_enum IS NULL  -- Exclude: no_tool_calls, wrong_tool_call, wrong_file_edited
+    GROUP BY model_id
+    HAVING total_runs >= 10
+    ORDER BY success_rate DESC, avg_latency_ms ASC
+  `);
+  
+  return stmt.all() as ModelComparison[];
+}
+
+// Advanced analysis queries
+export async function getTopPerformingCases(limit: number = 10): Promise<Array<{
+  case_id: string;
+  description: string;
+  success_rate: number;
+  avg_latency_ms: number;
+  total_runs: number;
+}>> {
+  const stmt = db.getDatabase().prepare(`
+    SELECT 
+      c.case_id,
+      c.description,
+      ROUND(AVG(CASE WHEN r.succeeded THEN 1.0 ELSE 0.0 END) * 100, 2) as success_rate,
+      ROUND(AVG(r.time_round_trip_ms), 2) as avg_latency_ms,
+      COUNT(r.result_id) as total_runs
+    FROM cases c
+    JOIN results r ON c.case_id = r.case_id
+    WHERE r.error_enum NOT IN (1, 6, 7) OR r.error_enum IS NULL  -- Exclude: no_tool_calls, wrong_tool_call, wrong_file_edited
+    GROUP BY c.case_id, c.description
+    HAVING total_runs >= 5
+    ORDER BY success_rate DESC, avg_latency_ms ASC
+    LIMIT ?
+  `);
+  
+  return stmt.all(limit) as Array<{
+    case_id: string;
+    description: string;
+    success_rate: number;
+    avg_latency_ms: number;
+    total_runs: number;
+  }>;
+}
+
+export async function getWorstPerformingCases(limit: number = 10): Promise<Array<{
+  case_id: string;
+  description: string;
+  success_rate: number;
+  avg_latency_ms: number;
+  total_runs: number;
+}>> {
+  const stmt = db.getDatabase().prepare(`
+    SELECT 
+      c.case_id,
+      c.description,
+      ROUND(AVG(CASE WHEN r.succeeded THEN 1.0 ELSE 0.0 END) * 100, 2) as success_rate,
+      ROUND(AVG(r.time_round_trip_ms), 2) as avg_latency_ms,
+      COUNT(r.result_id) as total_runs
+    FROM cases c
+    JOIN results r ON c.case_id = r.case_id
+    WHERE r.error_enum NOT IN (1, 6, 7) OR r.error_enum IS NULL  -- Exclude: no_tool_calls, wrong_tool_call, wrong_file_edited
+    GROUP BY c.case_id, c.description
+    HAVING total_runs >= 5
+    ORDER BY success_rate ASC, avg_latency_ms DESC
+    LIMIT ?
+  `);
+  
+  return stmt.all(limit) as Array<{
+    case_id: string;
+    description: string;
+    success_rate: number;
+    avg_latency_ms: number;
+    total_runs: number;
+  }>;
+}
+
+export async function getModelPerformanceByTimeOfDay(): Promise<Array<{
+  model_id: string;
+  hour: number;
+  success_rate: number;
+  avg_latency_ms: number;
+  total_runs: number;
+}>> {
+  const stmt = db.getDatabase().prepare(`
+    SELECT 
+      model_id,
+      CAST(strftime('%H', created_at) AS INTEGER) as hour,
+      ROUND(AVG(CASE WHEN succeeded THEN 1.0 ELSE 0.0 END) * 100, 2) as success_rate,
+      ROUND(AVG(time_round_trip_ms), 2) as avg_latency_ms,
+      COUNT(*) as total_runs
+    FROM results
+    GROUP BY model_id, hour
+    HAVING total_runs >= 5
+    ORDER BY model_id, hour
+  `);
+  
+  return stmt.all() as Array<{
+    model_id: string;
+    hour: number;
+    success_rate: number;
+    avg_latency_ms: number;
+    total_runs: number;
+  }>;
+}
+
+export async function getRunComparison(runId1: string, runId2: string): Promise<{
+  run1: { run_id: string; success_rate: number; avg_latency_ms: number; avg_cost_usd: number; total_cases: number };
+  run2: { run_id: string; success_rate: number; avg_latency_ms: number; avg_cost_usd: number; total_cases: number };
+}> {
+  const stmt = db.getDatabase().prepare(`
+    SELECT 
+      run_id,
+      ROUND(AVG(CASE WHEN succeeded THEN 1.0 ELSE 0.0 END) * 100, 2) as success_rate,
+      ROUND(AVG(time_round_trip_ms), 2) as avg_latency_ms,
+      ROUND(AVG(cost_usd), 4) as avg_cost_usd,
+      COUNT(DISTINCT case_id) as total_cases
+    FROM results
+    WHERE run_id IN (?, ?)
+    GROUP BY run_id
+  `);
+  
+  const results = stmt.all(runId1, runId2) as Array<{
+    run_id: string;
+    success_rate: number;
+    avg_latency_ms: number;
+    avg_cost_usd: number;
+    total_cases: number;
+  }>;
+  
+  const run1 = results.find(r => r.run_id === runId1);
+  const run2 = results.find(r => r.run_id === runId2);
+  
+  if (!run1 || !run2) {
+    throw new Error('One or both runs not found');
+  }
+  
+  return { run1, run2 };
+}
+
+// Summary statistics
+export async function getDatabaseSummary(): Promise<{
+  total_runs: number;
+  total_cases: number;
+  total_results: number;
+  valid_results: number;
+  unique_models: number;
+  overall_success_rate: number;
+  date_range: { earliest: string; latest: string };
+}> {
+  const stmt = db.getDatabase().prepare(`
+    SELECT 
+      (SELECT COUNT(*) FROM runs) as total_runs,
+      (SELECT COUNT(*) FROM cases) as total_cases,
+      (SELECT COUNT(*) FROM results) as total_results,
+      (SELECT COUNT(*) FROM results WHERE error_enum NOT IN (1, 6, 7) OR error_enum IS NULL) as valid_results,
+      (SELECT COUNT(DISTINCT model_id) FROM results) as unique_models,
+      (SELECT ROUND(AVG(CASE WHEN succeeded THEN 1.0 ELSE 0.0 END) * 100, 2) 
+       FROM results 
+       WHERE error_enum NOT IN (1, 6, 7) OR error_enum IS NULL) as overall_success_rate,
+      (SELECT MIN(created_at) FROM results) as earliest,
+      (SELECT MAX(created_at) FROM results) as latest
+    FROM results
+    LIMIT 1
+  `);
+  
+  const result = stmt.get() as any;
+  return {
+    total_runs: result.total_runs || 0,
+    total_cases: result.total_cases || 0,
+    total_results: result.total_results || 0,
+    valid_results: result.valid_results || 0,
+    unique_models: result.unique_models || 0,
+    overall_success_rate: result.overall_success_rate || 0,
+    date_range: {
+      earliest: result.earliest || '',
+      latest: result.latest || ''
+    }
+  };
+}

evals/diff-edits/database/schema.sql

@@ -0,0 +1,78 @@
+PRAGMA foreign_keys = ON;
+
+CREATE TABLE system_prompts (
+    hash TEXT PRIMARY KEY,
+    name TEXT NOT NULL,
+    content TEXT NOT NULL,
+    created_at DATETIME DEFAULT CURRENT_TIMESTAMP
+);
+
+CREATE TABLE processing_functions (
+    hash TEXT PRIMARY KEY,
+    name TEXT NOT NULL,
+    parsing_function TEXT NOT NULL,
+    diff_edit_function TEXT NOT NULL,
+    created_at DATETIME DEFAULT CURRENT_TIMESTAMP
+);
+
+CREATE TABLE files (
+    hash TEXT PRIMARY KEY,
+    filepath TEXT NOT NULL,
+    content TEXT NOT NULL,
+    tokens INTEGER,
+    created_at DATETIME DEFAULT CURRENT_TIMESTAMP
+);
+
+CREATE TABLE runs (
+    run_id TEXT PRIMARY KEY,
+    created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
+    description TEXT,
+    system_prompt_hash TEXT NOT NULL,
+    FOREIGN KEY (system_prompt_hash) REFERENCES system_prompts(hash)
+);
+
+CREATE TABLE cases (
+    case_id TEXT PRIMARY KEY,
+    run_id TEXT NOT NULL,
+    created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
+    description TEXT NOT NULL,
+    system_prompt_hash TEXT NOT NULL,
+    task_id TEXT NOT NULL,
+    tokens_in_context INTEGER,
+    file_hash TEXT,
+    FOREIGN KEY (run_id) REFERENCES runs(run_id),
+    FOREIGN KEY (system_prompt_hash) REFERENCES system_prompts(hash),
+    FOREIGN KEY (file_hash) REFERENCES files(hash)
+);
+
+CREATE TABLE results (
+    result_id TEXT PRIMARY KEY,
+    run_id TEXT NOT NULL,
+    case_id TEXT NOT NULL,
+    model_id TEXT NOT NULL,
+    processing_functions_hash TEXT NOT NULL,
+    succeeded BOOLEAN NOT NULL,
+    error_enum INTEGER,
+    num_edits INTEGER,
+    num_lines_deleted INTEGER,
+    num_lines_added INTEGER,
+    time_to_first_token_ms INTEGER,
+    time_to_first_edit_ms INTEGER,
+    time_round_trip_ms INTEGER,
+    cost_usd REAL,
+    completion_tokens INTEGER,
+    raw_model_output TEXT,
+    file_edited_hash TEXT,
+    parsed_tool_call_json TEXT,
+    created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
+    FOREIGN KEY (run_id) REFERENCES runs(run_id),
+    FOREIGN KEY (case_id) REFERENCES cases(case_id),
+    FOREIGN KEY (processing_functions_hash) REFERENCES processing_functions(hash)
+);
+
+CREATE INDEX idx_results_run_model ON results(run_id, model_id);
+CREATE INDEX idx_results_case_model ON results(case_id, model_id);
+CREATE INDEX idx_results_success ON results(succeeded);
+CREATE INDEX idx_cases_run ON cases(run_id);
+CREATE INDEX idx_results_created_at ON results(created_at);
+CREATE INDEX idx_runs_created_at ON runs(created_at);

evals/diff-edits/database/test.ts

@@ -0,0 +1,53 @@
+// Simple test to verify database functionality
+import { getDatabase } from './client';
+import { upsertSystemPrompt, createBenchmarkRun, getDatabaseSummary } from './index';
+
+async function testDatabase() {
+  console.log('Testing database functionality...');
+  
+  try {
+    // Test database connection
+    const db = getDatabase();
+    console.log('✓ Database connection established');
+    console.log('Database path:', db.getDatabasePath());
+    
+    // Test database info
+    const info = db.getInfo();
+    console.log('✓ Database info:', info);
+    
+    // Test database stats
+    const stats = db.getStats();
+    console.log('✓ Database stats:', stats);
+    
+    // Test system prompt creation
+    const systemPromptHash = await upsertSystemPrompt({
+      name: 'test-prompt',
+      content: 'This is a test system prompt for database verification.'
+    });
+    console.log('✓ System prompt created with hash:', systemPromptHash);
+    
+    // Test benchmark run creation
+    const runId = await createBenchmarkRun({
+      description: 'Test run for database verification',
+      system_prompt_hash: systemPromptHash
+    });
+    console.log('✓ Benchmark run created with ID:', runId);
+    
+    // Test database summary
+    const summary = await getDatabaseSummary();
+    console.log('✓ Database summary:', summary);
+    
+    console.log('\n🎉 All database tests passed!');
+    
+  } catch (error) {
+    console.error('❌ Database test failed:', error);
+    process.exit(1);
+  }
+}
+
+// Run test if this file is executed directly
+if (require.main === module) {
+  testDatabase();
+}
+
+export { testDatabase };

evals/diff-edits/database/types.ts

@@ -0,0 +1,169 @@
+// Database type definitions for diff-edits evaluation system
+
+export interface SystemPrompt {
+  hash: string;
+  name: string;
+  content: string;
+  created_at: string;
+}
+
+export interface ProcessingFunctions {
+  hash: string;
+  name: string;
+  parsing_function: string;
+  diff_edit_function: string;
+  created_at: string;
+}
+
+export interface FileRecord {
+  hash: string;
+  filepath: string;
+  content: string;
+  tokens?: number;
+  created_at: string;
+}
+
+export interface BenchmarkRun {
+  run_id: string;
+  created_at: string;
+  description?: string;
+  system_prompt_hash: string;
+}
+
+export interface Case {
+	case_id: string
+	run_id: string
+	created_at: string
+	description: string
+	system_prompt_hash: string
+	task_id: string
+	tokens_in_context: number
+	file_hash?: string
+}
+
+export interface Result {
+  result_id: string;
+  run_id: string;
+  case_id: string;
+  model_id: string;
+  processing_functions_hash: string;
+  succeeded: boolean;
+  error_enum?: number;
+  num_edits?: number;
+  num_lines_deleted?: number;
+  num_lines_added?: number;
+  time_to_first_token_ms?: number;
+  time_to_first_edit_ms?: number;
+  time_round_trip_ms?: number;
+  cost_usd?: number;
+  completion_tokens?: number;
+  raw_model_output?: string;
+  file_edited_hash?: string;
+  parsed_tool_call_json?: string;
+  created_at: string;
+}
+
+// Input types for creating records
+export interface CreateSystemPromptInput {
+  name: string;
+  content: string;
+}
+
+export interface CreateProcessingFunctionsInput {
+  name: string;
+  parsing_function: string;
+  diff_edit_function: string;
+}
+
+export interface CreateFileInput {
+  filepath: string;
+  content: string;
+  tokens?: number;
+}
+
+export interface CreateBenchmarkRunInput {
+  description?: string;
+  system_prompt_hash: string;
+}
+
+export interface CreateCaseInput {
+  run_id: string;
+  description: string;
+  system_prompt_hash: string;
+  task_id: string;
+  tokens_in_context: number;
+  file_hash?: string;
+}
+
+export interface CreateResultInput {
+  run_id: string;
+  case_id: string;
+  model_id: string;
+  processing_functions_hash: string;
+  succeeded: boolean;
+  error_enum?: number;
+  num_edits?: number;
+  num_lines_deleted?: number;
+  num_lines_added?: number;
+  time_to_first_token_ms?: number;
+  time_to_first_edit_ms?: number;
+  time_round_trip_ms?: number;
+  cost_usd?: number;
+  completion_tokens?: number;
+  raw_model_output?: string;
+  file_edited_hash?: string;
+  parsed_tool_call_json?: string;
+}
+
+// Analysis result types
+export interface ModelSuccessRate {
+  model_id: string;
+  total_runs: number;
+  successful_runs: number;
+  success_rate: number;
+}
+
+export interface ModelLatency {
+  model_id: string;
+  avg_time_to_first_token_ms: number;
+  avg_time_to_first_edit_ms: number;
+  avg_time_round_trip_ms: number;
+}
+
+export interface CostAnalysis {
+  run_id: string;
+  model_id: string;
+  total_cost_usd: number;
+  avg_cost_per_case: number;
+  total_completion_tokens: number;
+}
+
+export interface ErrorDistribution {
+  error_enum: number;
+  count: number;
+  percentage: number;
+}
+
+export interface FailedCase {
+  case_id: string;
+  model_id: string;
+  error_enum: number;
+  description: string;
+  raw_model_output?: string;
+}
+
+export interface PerformanceTrend {
+  date: string;
+  model_id: string;
+  success_rate: number;
+  avg_latency_ms: number;
+  avg_cost_usd: number;
+}
+
+export interface ModelComparison {
+  model_id: string;
+  success_rate: number;
+  avg_latency_ms: number;
+  avg_cost_usd: number;
+  total_runs: number;
+}

evals/diff-edits/diff-apply/diff-06-23-25.ts

@@ -0,0 +1,827 @@
+const SEARCH_BLOCK_START = "------- SEARCH"
+const SEARCH_BLOCK_END = "======="
+const REPLACE_BLOCK_END = "+++++++ REPLACE"
+
+const SEARCH_BLOCK_CHAR = "-"
+const REPLACE_BLOCK_CHAR = "+"
+const LEGACY_SEARCH_BLOCK_CHAR = "<"
+const LEGACY_REPLACE_BLOCK_CHAR = ">"
+
+// Replace the exact string constants with flexible regex patterns
+const SEARCH_BLOCK_START_REGEX = /^[-]{3,} SEARCH$/
+const SEARCH_BLOCK_END_REGEX = /^[=]{3,}$/
+const REPLACE_BLOCK_END_REGEX = /^[+]{3,} REPLACE$/
+const LEGACY_SEARCH_BLOCK_START_REGEX = /^[<]{3,} SEARCH$/
+const LEGACY_REPLACE_BLOCK_END_REGEX = /^[>]{3,} REPLACE$/
+
+// Helper functions to check if a line matches the flexible patterns
+function isSearchBlockStart(line: string): boolean {
+	return SEARCH_BLOCK_START_REGEX.test(line) || LEGACY_SEARCH_BLOCK_START_REGEX.test(line)
+}
+
+function isSearchBlockEnd(line: string): boolean {
+	return SEARCH_BLOCK_END_REGEX.test(line)
+}
+
+function isReplaceBlockEnd(line: string): boolean {
+	return REPLACE_BLOCK_END_REGEX.test(line) || LEGACY_REPLACE_BLOCK_END_REGEX.test(line)
+}
+
+/**
+ * Attempts a line-trimmed fallback match for the given search content in the original content.
+ * It tries to match `searchContent` lines against a block of lines in `originalContent` starting
+ * from `lastProcessedIndex`. Lines are matched by trimming leading/trailing whitespace and ensuring
+ * they are identical afterwards.
+ *
+ * Returns [matchIndexStart, matchIndexEnd] if found, or false if not found.
+ */
+function lineTrimmedFallbackMatch(originalContent: string, searchContent: string, startIndex: number): [number, number] | false {
+	// Split both contents into lines
+	const originalLines = originalContent.split("\n")
+	const searchLines = searchContent.split("\n")
+
+	// Trim trailing empty line if exists (from the trailing \n in searchContent)
+	if (searchLines[searchLines.length - 1] === "") {
+		searchLines.pop()
+	}
+
+	// Find the line number where startIndex falls
+	let startLineNum = 0
+	let currentIndex = 0
+	while (currentIndex < startIndex && startLineNum < originalLines.length) {
+		currentIndex += originalLines[startLineNum].length + 1 // +1 for \n
+		startLineNum++
+	}
+
+	// For each possible starting position in original content
+	for (let i = startLineNum; i <= originalLines.length - searchLines.length; i++) {
+		let matches = true
+
+		// Try to match all search lines from this position
+		for (let j = 0; j < searchLines.length; j++) {
+			const originalTrimmed = originalLines[i + j].trim()
+			const searchTrimmed = searchLines[j].trim()
+
+			if (originalTrimmed !== searchTrimmed) {
+				matches = false
+				break
+			}
+		}
+
+		// If we found a match, calculate the exact character positions
+		if (matches) {
+			// Find start character index
+			let matchStartIndex = 0
+			for (let k = 0; k < i; k++) {
+				matchStartIndex += originalLines[k].length + 1 // +1 for \n
+			}
+
+			// Find end character index
+			let matchEndIndex = matchStartIndex
+			for (let k = 0; k < searchLines.length; k++) {
+				matchEndIndex += originalLines[i + k].length + 1 // +1 for \n
+			}
+
+			return [matchStartIndex, matchEndIndex]
+		}
+	}
+
+	return false
+}
+
+/**
+ * Attempts to match blocks of code by using the first and last lines as anchors.
+ * This is a third-tier fallback strategy that helps match blocks where we can identify
+ * the correct location by matching the beginning and end, even if the exact content
+ * differs slightly.
+ *
+ * The matching strategy:
+ * 1. Only attempts to match blocks of 3 or more lines to avoid false positives
+ * 2. Extracts from the search content:
+ *    - First line as the "start anchor"
+ *    - Last line as the "end anchor"
+ * 3. For each position in the original content:
+ *    - Checks if the next line matches the start anchor
+ *    - If it does, jumps ahead by the search block size
+ *    - Checks if that line matches the end anchor
+ *    - All comparisons are done after trimming whitespace
+ *
+ * This approach is particularly useful for matching blocks of code where:
+ * - The exact content might have minor differences
+ * - The beginning and end of the block are distinctive enough to serve as anchors
+ * - The overall structure (number of lines) remains the same
+ *
+ * @param originalContent - The full content of the original file
+ * @param searchContent - The content we're trying to find in the original file
+ * @param startIndex - The character index in originalContent where to start searching
+ * @returns A tuple of [startIndex, endIndex] if a match is found, false otherwise
+ */
+function blockAnchorFallbackMatch(originalContent: string, searchContent: string, startIndex: number): [number, number] | false {
+	const originalLines = originalContent.split("\n")
+	const searchLines = searchContent.split("\n")
+
+	// Only use this approach for blocks of 3+ lines
+	if (searchLines.length < 3) {
+		return false
+	}
+
+	// Trim trailing empty line if exists
+	if (searchLines[searchLines.length - 1] === "") {
+		searchLines.pop()
+	}
+
+	const firstLineSearch = searchLines[0].trim()
+	const lastLineSearch = searchLines[searchLines.length - 1].trim()
+	const searchBlockSize = searchLines.length
+
+	// Find the line number where startIndex falls
+	let startLineNum = 0
+	let currentIndex = 0
+	while (currentIndex < startIndex && startLineNum < originalLines.length) {
+		currentIndex += originalLines[startLineNum].length + 1
+		startLineNum++
+	}
+
+	// Look for matching start and end anchors
+	for (let i = startLineNum; i <= originalLines.length - searchBlockSize; i++) {
+		// Check if first line matches
+		if (originalLines[i].trim() !== firstLineSearch) {
+			continue
+		}
+
+		// Check if last line matches at the expected position
+		if (originalLines[i + searchBlockSize - 1].trim() !== lastLineSearch) {
+			continue
+		}
+
+		// Calculate exact character positions
+		let matchStartIndex = 0
+		for (let k = 0; k < i; k++) {
+			matchStartIndex += originalLines[k].length + 1
+		}
+
+		let matchEndIndex = matchStartIndex
+		for (let k = 0; k < searchBlockSize; k++) {
+			matchEndIndex += originalLines[i + k].length + 1
+		}
+
+		return [matchStartIndex, matchEndIndex]
+	}
+
+	return false
+}
+
+/**
+ * This function reconstructs the file content by applying a streamed diff (in a
+ * specialized SEARCH/REPLACE block format) to the original file content. It is designed
+ * to handle both incremental updates and the final resulting file after all chunks have
+ * been processed.
+ *
+ * The diff format is a custom structure that uses three markers to define changes:
+ *
+ *   ------- SEARCH
+ *   [Exact content to find in the original file]
+ *   =======
+ *   [Content to replace with]
+ *   +++++++ REPLACE
+ *
+ * Behavior and Assumptions:
+ * 1. The file is processed chunk-by-chunk. Each chunk of `diffContent` may contain
+ *    partial or complete SEARCH/REPLACE blocks. By calling this function with each
+ *    incremental chunk (with `isFinal` indicating the last chunk), the final reconstructed
+ *    file content is produced.
+ *
+ * 2. Matching Strategy (in order of attempt):
+ *    a. Exact Match: First attempts to find the exact SEARCH block text in the original file
+ *    b. Line-Trimmed Match: Falls back to line-by-line comparison ignoring leading/trailing whitespace
+ *    c. Block Anchor Match: For blocks of 3+ lines, tries to match using first/last lines as anchors
+ *    If all matching strategies fail, an error is thrown.
+ *
+ * 3. Empty SEARCH Section:
+ *    - If SEARCH is empty and the original file is empty, this indicates creating a new file
+ *      (pure insertion).
+ *    - If SEARCH is empty and the original file is not empty, this indicates a complete
+ *      file replacement (the entire original content is considered matched and replaced).
+ *
+ * 4. Applying Changes:
+ *    - Before encountering the "=======" marker, lines are accumulated as search content.
+ *    - After "=======" and before ">>>>>>> REPLACE", lines are accumulated as replacement content.
+ *    - Once the block is complete (">>>>>>> REPLACE"), the matched section in the original
+ *      file is replaced with the accumulated replacement lines, and the position in the original
+ *      file is advanced.
+ *
+ * 5. Incremental Output:
+ *    - As soon as the match location is found and we are in the REPLACE section, each new
+ *      replacement line is appended to the result so that partial updates can be viewed
+ *      incrementally.
+ *
+ * 6. Partial Markers:
+ *    - If the final line of the chunk looks like it might be part of a marker but is not one
+ *      of the known markers, it is removed. This prevents incomplete or partial markers
+ *      from corrupting the output.
+ *
+ * 7. Finalization:
+ *    - Once all chunks have been processed (when `isFinal` is true), any remaining original
+ *      content after the last replaced section is appended to the result.
+ *    - Trailing newlines are not forcibly added. The code tries to output exactly what is specified.
+ *
+ * Errors:
+ * - If the search block cannot be matched using any of the available matching strategies,
+ *   an error is thrown.
+ */
+export async function constructNewFileContent(
+	diffContent: string,
+	originalContent: string,
+	isFinal: boolean,
+	version: "v1" | "v2" = "v1",
+): Promise<string> {
+	const constructor = constructNewFileContentVersionMapping[version]
+	if (!constructor) {
+		throw new Error(`Invalid version '${version}' for file content constructor`)
+	}
+	return constructor(diffContent, originalContent, isFinal)
+}
+
+const constructNewFileContentVersionMapping: Record<
+	string,
+	(diffContent: string, originalContent: string, isFinal: boolean) => Promise<string>
+> = {
+	v1: constructNewFileContentV1,
+	v2: constructNewFileContentV2,
+} as const
+
+async function constructNewFileContentV1(diffContent: string, originalContent: string, isFinal: boolean): Promise<string> {
+	let result = ""
+	let lastProcessedIndex = 0
+
+	let currentSearchContent = ""
+	let currentReplaceContent = ""
+	let inSearch = false
+	let inReplace = false
+
+	let searchMatchIndex = -1
+	let searchEndIndex = -1
+
+	// Track all replacements to handle out-of-order edits
+	let replacements: Array<{ start: number; end: number; content: string }> = []
+	let pendingOutOfOrderReplacement = false
+
+	let lines = diffContent.split("\n")
+
+	// If the last line looks like a partial marker but isn't recognized,
+	// remove it because it might be incomplete.
+	const lastLine = lines[lines.length - 1]
+	if (
+		lines.length > 0 &&
+		(lastLine.startsWith(SEARCH_BLOCK_CHAR) ||
+			lastLine.startsWith(LEGACY_SEARCH_BLOCK_CHAR) ||
+			lastLine.startsWith("=") ||
+			lastLine.startsWith(REPLACE_BLOCK_CHAR) ||
+			lastLine.startsWith(LEGACY_REPLACE_BLOCK_CHAR)) &&
+		!isSearchBlockStart(lastLine) &&
+		!isSearchBlockEnd(lastLine) &&
+		!isReplaceBlockEnd(lastLine)
+	) {
+		lines.pop()
+	}
+
+	for (const line of lines) {
+		if (isSearchBlockStart(line)) {
+			inSearch = true
+			currentSearchContent = ""
+			currentReplaceContent = ""
+			continue
+		}
+
+		if (isSearchBlockEnd(line)) {
+			inSearch = false
+			inReplace = true
+
+			// Remove trailing linebreak for adding the === marker
+			// if (currentSearchContent.endsWith("\r\n")) {
+			// 	currentSearchContent = currentSearchContent.slice(0, -2)
+			// } else if (currentSearchContent.endsWith("\n")) {
+			// 	currentSearchContent = currentSearchContent.slice(0, -1)
+			// }
+
+			if (!currentSearchContent) {
+				// Empty search block
+				if (originalContent.length === 0) {
+					// New file scenario: nothing to match, just start inserting
+					searchMatchIndex = 0
+					searchEndIndex = 0
+				} else {
+					// ERROR: Empty search block with non-empty file indicates malformed SEARCH marker
+					throw new Error(
+						"Empty SEARCH block detected with non-empty file. This usually indicates a malformed SEARCH marker.\n" +
+							"Please ensure your SEARCH marker follows the correct format:\n" +
+							"- Use '------- SEARCH' (7+ dashes + space + SEARCH)\n",
+					)
+				}
+			} else {
+				// Add check for inefficient full-file search
+				// if (currentSearchContent.trim() === originalContent.trim()) {
+				// 	throw new Error(
+				// 		"The SEARCH block contains the entire file content. Please either:\n" +
+				// 			"1. Use an empty SEARCH block to replace the entire file, or\n" +
+				// 			"2. Make focused changes to specific parts of the file that need modification.",
+				// 	)
+				// }
+
+				// Exact search match scenario
+				const exactIndex = originalContent.indexOf(currentSearchContent, lastProcessedIndex)
+				if (exactIndex !== -1) {
+					searchMatchIndex = exactIndex
+					searchEndIndex = exactIndex + currentSearchContent.length
+				} else {
+					// Attempt fallback line-trimmed matching
+					const lineMatch = lineTrimmedFallbackMatch(originalContent, currentSearchContent, lastProcessedIndex)
+					if (lineMatch) {
+						;[searchMatchIndex, searchEndIndex] = lineMatch
+					} else {
+						// Try block anchor fallback for larger blocks
+						const blockMatch = blockAnchorFallbackMatch(originalContent, currentSearchContent, lastProcessedIndex)
+						if (blockMatch) {
+							;[searchMatchIndex, searchEndIndex] = blockMatch
+						} else {
+							// Last resort: search the entire file from the beginning
+							const fullFileIndex = originalContent.indexOf(currentSearchContent, 0)
+							if (fullFileIndex !== -1) {
+								// Found in the file - could be out of order
+								searchMatchIndex = fullFileIndex
+								searchEndIndex = fullFileIndex + currentSearchContent.length
+								if (searchMatchIndex < lastProcessedIndex) {
+									pendingOutOfOrderReplacement = true
+								}
+							} else {
+								throw new Error(
+									`The SEARCH block:\n${currentSearchContent.trimEnd()}\n...does not match anything in the file.`,
+								)
+							}
+						}
+					}
+				}
+			}
+
+			// Check if this is an out-of-order replacement
+			if (searchMatchIndex < lastProcessedIndex) {
+				pendingOutOfOrderReplacement = true
+			}
+
+			// For in-order replacements, output everything up to the match location
+			if (!pendingOutOfOrderReplacement) {
+				result += originalContent.slice(lastProcessedIndex, searchMatchIndex)
+			}
+			continue
+		}
+
+		if (isReplaceBlockEnd(line)) {
+			// Finished one replace block
+
+			// Store this replacement
+			replacements.push({
+				start: searchMatchIndex,
+				end: searchEndIndex,
+				content: currentReplaceContent,
+			})
+
+			// If this was an in-order replacement, advance lastProcessedIndex
+			if (!pendingOutOfOrderReplacement) {
+				lastProcessedIndex = searchEndIndex
+			}
+
+			// Reset for next block
+			inSearch = false
+			inReplace = false
+			currentSearchContent = ""
+			currentReplaceContent = ""
+			searchMatchIndex = -1
+			searchEndIndex = -1
+			pendingOutOfOrderReplacement = false
+			continue
+		}
+
+		// Accumulate content for search or replace
+		// (currentReplaceContent is not being used for anything right now since we directly append to result.)
+		// (We artificially add a linebreak since we split on \n at the beginning. In order to not include a trailing linebreak in the final search/result blocks we need to remove it before using them. This allows for partial line matches to be correctly identified.)
+		// NOTE: search/replace blocks must be arranged in the order they appear in the file due to how we build the content using lastProcessedIndex. We also cannot strip the trailing newline since for non-partial lines it would remove the linebreak from the original content. (If we remove end linebreak from search, then we'd also have to remove it from replace but we can't know if it's a partial line or not since the model may be using the line break to indicate the end of the block rather than as part of the search content.) We require the model to output full lines in order for our fallbacks to work as well.
+		if (inSearch) {
+			currentSearchContent += line + "\n"
+		} else if (inReplace) {
+			currentReplaceContent += line + "\n"
+			// Only output replacement lines immediately for in-order replacements
+			if (searchMatchIndex !== -1 && !pendingOutOfOrderReplacement) {
+				result += line + "\n"
+			}
+		}
+	}
+
+	// If this is the final chunk, we need to apply all replacements and build the final result
+	if (isFinal) {
+		// Handle the case where we're still in replace mode when processing ends
+		// and this is the final chunk - treat it as if we encountered the REPLACE marker
+		if (inReplace && searchMatchIndex !== -1) {
+			// Store this replacement
+			replacements.push({
+				start: searchMatchIndex,
+				end: searchEndIndex,
+				content: currentReplaceContent,
+			})
+
+			// If this was an in-order replacement, advance lastProcessedIndex
+			if (!pendingOutOfOrderReplacement) {
+				lastProcessedIndex = searchEndIndex
+			}
+
+			// Reset state
+			inSearch = false
+			inReplace = false
+			currentSearchContent = ""
+			currentReplaceContent = ""
+			searchMatchIndex = -1
+			searchEndIndex = -1
+			pendingOutOfOrderReplacement = false
+		}
+		// end of handling missing replace marker
+
+		// Sort replacements by start position
+		replacements.sort((a, b) => a.start - b.start)
+
+		// Rebuild the entire result by applying all replacements
+		result = ""
+		let currentPos = 0
+
+		for (const replacement of replacements) {
+			// Add original content up to this replacement
+			result += originalContent.slice(currentPos, replacement.start)
+			// Add the replacement content
+			result += replacement.content
+			// Move position to after the replaced section
+			currentPos = replacement.end
+		}
+
+		// Add any remaining original content
+		result += originalContent.slice(currentPos)
+	}
+
+	return result
+}
+
+enum ProcessingState {
+	Idle = 0,
+	StateSearch = 1 << 0,
+	StateReplace = 1 << 1,
+}
+
+class NewFileContentConstructor {
+	private originalContent: string
+	private isFinal: boolean
+	private state: number
+	private pendingNonStandardLines: string[]
+	private result: string
+	private lastProcessedIndex: number
+	private currentSearchContent: string
+	private currentReplaceContent: string
+	private searchMatchIndex: number
+	private searchEndIndex: number
+
+	constructor(originalContent: string, isFinal: boolean) {
+		this.originalContent = originalContent
+		this.isFinal = isFinal
+		this.pendingNonStandardLines = []
+		this.result = ""
+		this.lastProcessedIndex = 0
+		this.state = ProcessingState.Idle
+		this.currentSearchContent = ""
+		this.currentReplaceContent = ""
+		this.searchMatchIndex = -1
+		this.searchEndIndex = -1
+	}
+
+	private resetForNextBlock() {
+		// Reset for next block
+		this.state = ProcessingState.Idle
+		this.currentSearchContent = ""
+		this.currentReplaceContent = ""
+		this.searchMatchIndex = -1
+		this.searchEndIndex = -1
+	}
+
+	private findLastMatchingLineIndex(regx: RegExp, lineLimit: number) {
+		for (let i = lineLimit; i > 0; ) {
+			i--
+			if (this.pendingNonStandardLines[i].match(regx)) {
+				return i
+			}
+		}
+		return -1
+	}
+
+	private updateProcessingState(newState: ProcessingState) {
+		const isValidTransition =
+			(this.state === ProcessingState.Idle && newState === ProcessingState.StateSearch) ||
+			(this.state === ProcessingState.StateSearch && newState === ProcessingState.StateReplace)
+
+		if (!isValidTransition) {
+			throw new Error(
+				`Invalid state transition.\n` +
+					"Valid transitions are:\n" +
+					"- Idle → StateSearch\n" +
+					"- StateSearch → StateReplace",
+			)
+		}
+
+		this.state |= newState
+	}
+
+	private isStateActive(state: ProcessingState): boolean {
+		return (this.state & state) === state
+	}
+
+	private activateReplaceState() {
+		this.updateProcessingState(ProcessingState.StateReplace)
+	}
+
+	private activateSearchState() {
+		this.updateProcessingState(ProcessingState.StateSearch)
+		this.currentSearchContent = ""
+		this.currentReplaceContent = ""
+	}
+
+	private isSearchingActive(): boolean {
+		return this.isStateActive(ProcessingState.StateSearch)
+	}
+
+	private isReplacingActive(): boolean {
+		return this.isStateActive(ProcessingState.StateReplace)
+	}
+
+	private hasPendingNonStandardLines(pendingNonStandardLineLimit: number): boolean {
+		return this.pendingNonStandardLines.length - pendingNonStandardLineLimit < this.pendingNonStandardLines.length
+	}
+
+	public processLine(line: string) {
+		this.internalProcessLine(line, true, this.pendingNonStandardLines.length)
+	}
+
+	public getResult() {
+		// If this is the final chunk, append any remaining original content
+		if (this.isFinal && this.lastProcessedIndex < this.originalContent.length) {
+			this.result += this.originalContent.slice(this.lastProcessedIndex)
+		}
+		if (this.isFinal && this.state !== ProcessingState.Idle) {
+			throw new Error("File processing incomplete - SEARCH/REPLACE operations still active during finalization")
+		}
+		return this.result
+	}
+
+	private internalProcessLine(
+		line: string,
+		canWritependingNonStandardLines: boolean,
+		pendingNonStandardLineLimit: number,
+	): number {
+		let removeLineCount = 0
+		if (isSearchBlockStart(line)) {
+			removeLineCount = this.trimPendingNonStandardTrailingEmptyLines(pendingNonStandardLineLimit)
+			if (removeLineCount > 0) {
+				pendingNonStandardLineLimit = pendingNonStandardLineLimit - removeLineCount
+			}
+			if (this.hasPendingNonStandardLines(pendingNonStandardLineLimit)) {
+				this.tryFixSearchReplaceBlock(pendingNonStandardLineLimit)
+				canWritependingNonStandardLines && (this.pendingNonStandardLines.length = 0)
+			}
+			this.activateSearchState()
+		} else if (isSearchBlockEnd(line)) {
+			// 校验非标内容
+			if (!this.isSearchingActive()) {
+				this.tryFixSearchBlock(pendingNonStandardLineLimit)
+				canWritependingNonStandardLines && (this.pendingNonStandardLines.length = 0)
+			}
+			this.activateReplaceState()
+			this.beforeReplace()
+		} else if (isReplaceBlockEnd(line)) {
+			if (!this.isReplacingActive()) {
+				this.tryFixReplaceBlock(pendingNonStandardLineLimit)
+				canWritependingNonStandardLines && (this.pendingNonStandardLines.length = 0)
+			}
+			this.lastProcessedIndex = this.searchEndIndex
+			this.resetForNextBlock()
+		} else {
+			// Accumulate content for search or replace
+			// (currentReplaceContent is not being used for anything right now since we directly append to result.)
+			// (We artificially add a linebreak since we split on \n at the beginning. In order to not include a trailing linebreak in the final search/result blocks we need to remove it before using them. This allows for partial line matches to be correctly identified.)
+			// NOTE: search/replace blocks must be arranged in the order they appear in the file due to how we build the content using lastProcessedIndex. We also cannot strip the trailing newline since for non-partial lines it would remove the linebreak from the original content. (If we remove end linebreak from search, then we'd also have to remove it from replace but we can't know if it's a partial line or not since the model may be using the line break to indicate the end of the block rather than as part of the search content.) We require the model to output full lines in order for our fallbacks to work as well.
+			if (this.isReplacingActive()) {
+				this.currentReplaceContent += line + "\n"
+				// Output replacement lines immediately if we know the insertion point
+				if (this.searchMatchIndex !== -1) {
+					this.result += line + "\n"
+				}
+			} else if (this.isSearchingActive()) {
+				this.currentSearchContent += line + "\n"
+			} else {
+				let appendToPendingNonStandardLines = canWritependingNonStandardLines
+				if (appendToPendingNonStandardLines) {
+					// 处理非标内容
+					this.pendingNonStandardLines.push(line)
+				}
+			}
+		}
+		return removeLineCount
+	}
+
+	private beforeReplace() {
+		// Remove trailing linebreak for adding the === marker
+		// if (currentSearchContent.endsWith("\r\n")) {
+		// 	currentSearchContent = currentSearchContent.slice(0, -2)
+		// } else if (currentSearchContent.endsWith("\n")) {
+		// 	currentSearchContent = currentSearchContent.slice(0, -1)
+		// }
+
+		if (!this.currentSearchContent) {
+			// Empty search block
+			if (this.originalContent.length === 0) {
+				// New file scenario: nothing to match, just start inserting
+				this.searchMatchIndex = 0
+				this.searchEndIndex = 0
+			} else {
+				// Complete file replacement scenario: treat the entire file as matched
+				this.searchMatchIndex = 0
+				this.searchEndIndex = this.originalContent.length
+			}
+		} else {
+			// Add check for inefficient full-file search
+			// if (currentSearchContent.trim() === originalContent.trim()) {
+			// 	throw new Error(
+			// 		"The SEARCH block contains the entire file content. Please either:\n" +
+			// 			"1. Use an empty SEARCH block to replace the entire file, or\n" +
+			// 			"2. Make focused changes to specific parts of the file that need modification.",
+			// 	)
+			// }
+			// Exact search match scenario
+			const exactIndex = this.originalContent.indexOf(this.currentSearchContent, this.lastProcessedIndex)
+			if (exactIndex !== -1) {
+				this.searchMatchIndex = exactIndex
+				this.searchEndIndex = exactIndex + this.currentSearchContent.length
+			} else {
+				// Attempt fallback line-trimmed matching
+				const lineMatch = lineTrimmedFallbackMatch(
+					this.originalContent,
+					this.currentSearchContent,
+					this.lastProcessedIndex,
+				)
+				if (lineMatch) {
+					;[this.searchMatchIndex, this.searchEndIndex] = lineMatch
+				} else {
+					// Try block anchor fallback for larger blocks
+					const blockMatch = blockAnchorFallbackMatch(
+						this.originalContent,
+						this.currentSearchContent,
+						this.lastProcessedIndex,
+					)
+					if (blockMatch) {
+						;[this.searchMatchIndex, this.searchEndIndex] = blockMatch
+					} else {
+						throw new Error(
+							`The SEARCH block:\n${this.currentSearchContent.trimEnd()}\n...does not match anything in the file.`,
+						)
+					}
+				}
+			}
+		}
+		if (this.searchMatchIndex < this.lastProcessedIndex) {
+			throw new Error(
+				`The SEARCH block:\n${this.currentSearchContent.trimEnd()}\n...matched an incorrect content in the file.`,
+			)
+		}
+		// Output everything up to the match location
+		this.result += this.originalContent.slice(this.lastProcessedIndex, this.searchMatchIndex)
+	}
+
+	private tryFixSearchBlock(lineLimit: number): number {
+		let removeLineCount = 0
+		if (lineLimit < 0) {
+			lineLimit = this.pendingNonStandardLines.length
+		}
+		if (!lineLimit) {
+			throw new Error("Invalid SEARCH/REPLACE block structure - no lines available to process")
+		}
+		let searchTagRegexp = /^([-]{3,}|[<]{3,}) SEARCH$/
+		const searchTagIndex = this.findLastMatchingLineIndex(searchTagRegexp, lineLimit)
+		if (searchTagIndex !== -1) {
+			let fixLines = this.pendingNonStandardLines.slice(searchTagIndex, lineLimit)
+			fixLines[0] = SEARCH_BLOCK_START
+			for (const line of fixLines) {
+				removeLineCount += this.internalProcessLine(line, false, searchTagIndex)
+			}
+		} else {
+			throw new Error(
+				`Invalid REPLACE marker detected - could not find matching SEARCH block starting from line ${searchTagIndex + 1}`,
+			)
+		}
+		return removeLineCount
+	}
+
+	private tryFixReplaceBlock(lineLimit: number): number {
+		let removeLineCount = 0
+		if (lineLimit < 0) {
+			lineLimit = this.pendingNonStandardLines.length
+		}
+		if (!lineLimit) {
+			throw new Error()
+		}
+		let replaceBeginTagRegexp = /^[=]{3,}$/
+		const replaceBeginTagIndex = this.findLastMatchingLineIndex(replaceBeginTagRegexp, lineLimit)
+		if (replaceBeginTagIndex !== -1) {
+			// // 校验非标内容
+			// if (!this.isSearchingActive()) {
+			// 	removeLineCount += this.tryFixSearchBlock(replaceBeginTagIndex)
+			// }
+			let fixLines = this.pendingNonStandardLines.slice(replaceBeginTagIndex - removeLineCount, lineLimit - removeLineCount)
+			fixLines[0] = SEARCH_BLOCK_END
+			for (const line of fixLines) {
+				removeLineCount += this.internalProcessLine(line, false, replaceBeginTagIndex - removeLineCount)
+			}
+		} else {
+			throw new Error(`Malformed REPLACE block - missing valid separator after line ${replaceBeginTagIndex + 1}`)
+		}
+		return removeLineCount
+	}
+
+	private tryFixSearchReplaceBlock(lineLimit: number): number {
+		let removeLineCount = 0
+		if (lineLimit < 0) {
+			lineLimit = this.pendingNonStandardLines.length
+		}
+		if (!lineLimit) {
+			throw new Error()
+		}
+
+		let replaceEndTagRegexp = /^([+]{3,}|[>]{3,}) REPLACE$/
+		const replaceEndTagIndex = this.findLastMatchingLineIndex(replaceEndTagRegexp, lineLimit)
+		const likeReplaceEndTag = replaceEndTagIndex === lineLimit - 1
+		if (likeReplaceEndTag) {
+			// // 校验非标内容
+			// if (!this.isReplacingActive()) {
+			// 	removeLineCount += this.tryFixReplaceBlock(replaceEndTagIndex)
+			// }
+			let fixLines = this.pendingNonStandardLines.slice(replaceEndTagIndex - removeLineCount, lineLimit - removeLineCount)
+			fixLines[fixLines.length - 1] = REPLACE_BLOCK_END
+			for (const line of fixLines) {
+				removeLineCount += this.internalProcessLine(line, false, replaceEndTagIndex - removeLineCount)
+			}
+		} else {
+			throw new Error("Malformed SEARCH/REPLACE block structure: Missing valid closing REPLACE marker")
+		}
+		return removeLineCount
+	}
+
+	/**
+	 * Removes trailing empty lines from the pendingNonStandardLines array
+	 * @param lineLimit - The index to start checking from (exclusive).
+	 *                    Removes empty lines from lineLimit-1 backwards.
+	 * @returns The number of empty lines removed
+	 */
+	private trimPendingNonStandardTrailingEmptyLines(lineLimit: number): number {
+		let removedCount = 0
+		let i = Math.min(lineLimit, this.pendingNonStandardLines.length) - 1
+
+		while (i >= 0 && this.pendingNonStandardLines[i].trim() === "") {
+			this.pendingNonStandardLines.pop()
+			removedCount++
+			i--
+		}
+
+		return removedCount
+	}
+}
+
+export async function constructNewFileContentV2(diffContent: string, originalContent: string, isFinal: boolean): Promise<string> {
+	let newFileContentConstructor = new NewFileContentConstructor(originalContent, isFinal)
+
+	let lines = diffContent.split("\n")
+
+	// If the last line looks like a partial marker but isn't recognized,
+	// remove it because it might be incomplete.
+	const lastLine = lines[lines.length - 1]
+	if (
+		lines.length > 0 &&
+		(lastLine.startsWith(SEARCH_BLOCK_CHAR) ||
+			lastLine.startsWith(LEGACY_SEARCH_BLOCK_CHAR) ||
+			lastLine.startsWith("=") ||
+			lastLine.startsWith(REPLACE_BLOCK_CHAR) ||
+			lastLine.startsWith(LEGACY_REPLACE_BLOCK_CHAR)) &&
+		lastLine !== SEARCH_BLOCK_START &&
+		lastLine !== SEARCH_BLOCK_END &&
+		lastLine !== REPLACE_BLOCK_END
+	) {
+		lines.pop()
+	}
+
+	for (const line of lines) {
+		newFileContentConstructor.processLine(line)
+	}
+
+	let result = newFileContentConstructor.getResult()
+	return result
+}

evals/diff-edits/diff-apply/diff-06-25-25.ts

@@ -0,0 +1,829 @@
+const SEARCH_BLOCK_START = "------- SEARCH"
+const SEARCH_BLOCK_END = "======="
+const REPLACE_BLOCK_END = "+++++++ REPLACE"
+
+const SEARCH_BLOCK_CHAR = "-"
+const REPLACE_BLOCK_CHAR = "+"
+const LEGACY_SEARCH_BLOCK_CHAR = "<"
+const LEGACY_REPLACE_BLOCK_CHAR = ">"
+
+// Replace the exact string constants with flexible regex patterns
+const SEARCH_BLOCK_START_REGEX = /^[-]{3,} SEARCH>?$/
+const LEGACY_SEARCH_BLOCK_START_REGEX = /^[<]{3,} SEARCH>?$/
+
+const SEARCH_BLOCK_END_REGEX = /^[=]{3,}$/
+
+const REPLACE_BLOCK_END_REGEX = /^[+]{3,} REPLACE>?$/
+const LEGACY_REPLACE_BLOCK_END_REGEX = /^[>]{3,} REPLACE>?$/
+
+// Helper functions to check if a line matches the flexible patterns
+function isSearchBlockStart(line: string): boolean {
+	return SEARCH_BLOCK_START_REGEX.test(line) || LEGACY_SEARCH_BLOCK_START_REGEX.test(line)
+}
+
+function isSearchBlockEnd(line: string): boolean {
+	return SEARCH_BLOCK_END_REGEX.test(line)
+}
+
+function isReplaceBlockEnd(line: string): boolean {
+	return REPLACE_BLOCK_END_REGEX.test(line) || LEGACY_REPLACE_BLOCK_END_REGEX.test(line)
+}
+
+/**
+ * Attempts a line-trimmed fallback match for the given search content in the original content.
+ * It tries to match `searchContent` lines against a block of lines in `originalContent` starting
+ * from `lastProcessedIndex`. Lines are matched by trimming leading/trailing whitespace and ensuring
+ * they are identical afterwards.
+ *
+ * Returns [matchIndexStart, matchIndexEnd] if found, or false if not found.
+ */
+function lineTrimmedFallbackMatch(originalContent: string, searchContent: string, startIndex: number): [number, number] | false {
+	// Split both contents into lines
+	const originalLines = originalContent.split("\n")
+	const searchLines = searchContent.split("\n")
+
+	// Trim trailing empty line if exists (from the trailing \n in searchContent)
+	if (searchLines[searchLines.length - 1] === "") {
+		searchLines.pop()
+	}
+
+	// Find the line number where startIndex falls
+	let startLineNum = 0
+	let currentIndex = 0
+	while (currentIndex < startIndex && startLineNum < originalLines.length) {
+		currentIndex += originalLines[startLineNum].length + 1 // +1 for \n
+		startLineNum++
+	}
+
+	// For each possible starting position in original content
+	for (let i = startLineNum; i <= originalLines.length - searchLines.length; i++) {
+		let matches = true
+
+		// Try to match all search lines from this position
+		for (let j = 0; j < searchLines.length; j++) {
+			const originalTrimmed = originalLines[i + j].trim()
+			const searchTrimmed = searchLines[j].trim()
+
+			if (originalTrimmed !== searchTrimmed) {
+				matches = false
+				break
+			}
+		}
+
+		// If we found a match, calculate the exact character positions
+		if (matches) {
+			// Find start character index
+			let matchStartIndex = 0
+			for (let k = 0; k < i; k++) {
+				matchStartIndex += originalLines[k].length + 1 // +1 for \n
+			}
+
+			// Find end character index
+			let matchEndIndex = matchStartIndex
+			for (let k = 0; k < searchLines.length; k++) {
+				matchEndIndex += originalLines[i + k].length + 1 // +1 for \n
+			}
+
+			return [matchStartIndex, matchEndIndex]
+		}
+	}
+
+	return false
+}
+
+/**
+ * Attempts to match blocks of code by using the first and last lines as anchors.
+ * This is a third-tier fallback strategy that helps match blocks where we can identify
+ * the correct location by matching the beginning and end, even if the exact content
+ * differs slightly.
+ *
+ * The matching strategy:
+ * 1. Only attempts to match blocks of 3 or more lines to avoid false positives
+ * 2. Extracts from the search content:
+ *    - First line as the "start anchor"
+ *    - Last line as the "end anchor"
+ * 3. For each position in the original content:
+ *    - Checks if the next line matches the start anchor
+ *    - If it does, jumps ahead by the search block size
+ *    - Checks if that line matches the end anchor
+ *    - All comparisons are done after trimming whitespace
+ *
+ * This approach is particularly useful for matching blocks of code where:
+ * - The exact content might have minor differences
+ * - The beginning and end of the block are distinctive enough to serve as anchors
+ * - The overall structure (number of lines) remains the same
+ *
+ * @param originalContent - The full content of the original file
+ * @param searchContent - The content we're trying to find in the original file
+ * @param startIndex - The character index in originalContent where to start searching
+ * @returns A tuple of [startIndex, endIndex] if a match is found, false otherwise
+ */
+function blockAnchorFallbackMatch(originalContent: string, searchContent: string, startIndex: number): [number, number] | false {
+	const originalLines = originalContent.split("\n")
+	const searchLines = searchContent.split("\n")
+
+	// Only use this approach for blocks of 3+ lines
+	if (searchLines.length < 3) {
+		return false
+	}
+
+	// Trim trailing empty line if exists
+	if (searchLines[searchLines.length - 1] === "") {
+		searchLines.pop()
+	}
+
+	const firstLineSearch = searchLines[0].trim()
+	const lastLineSearch = searchLines[searchLines.length - 1].trim()
+	const searchBlockSize = searchLines.length
+
+	// Find the line number where startIndex falls
+	let startLineNum = 0
+	let currentIndex = 0
+	while (currentIndex < startIndex && startLineNum < originalLines.length) {
+		currentIndex += originalLines[startLineNum].length + 1
+		startLineNum++
+	}
+
+	// Look for matching start and end anchors
+	for (let i = startLineNum; i <= originalLines.length - searchBlockSize; i++) {
+		// Check if first line matches
+		if (originalLines[i].trim() !== firstLineSearch) {
+			continue
+		}
+
+		// Check if last line matches at the expected position
+		if (originalLines[i + searchBlockSize - 1].trim() !== lastLineSearch) {
+			continue
+		}
+
+		// Calculate exact character positions
+		let matchStartIndex = 0
+		for (let k = 0; k < i; k++) {
+			matchStartIndex += originalLines[k].length + 1
+		}
+
+		let matchEndIndex = matchStartIndex
+		for (let k = 0; k < searchBlockSize; k++) {
+			matchEndIndex += originalLines[i + k].length + 1
+		}
+
+		return [matchStartIndex, matchEndIndex]
+	}
+
+	return false
+}
+
+/**
+ * This function reconstructs the file content by applying a streamed diff (in a
+ * specialized SEARCH/REPLACE block format) to the original file content. It is designed
+ * to handle both incremental updates and the final resulting file after all chunks have
+ * been processed.
+ *
+ * The diff format is a custom structure that uses three markers to define changes:
+ *
+ *   ------- SEARCH
+ *   [Exact content to find in the original file]
+ *   =======
+ *   [Content to replace with]
+ *   +++++++ REPLACE
+ *
+ * Behavior and Assumptions:
+ * 1. The file is processed chunk-by-chunk. Each chunk of `diffContent` may contain
+ *    partial or complete SEARCH/REPLACE blocks. By calling this function with each
+ *    incremental chunk (with `isFinal` indicating the last chunk), the final reconstructed
+ *    file content is produced.
+ *
+ * 2. Matching Strategy (in order of attempt):
+ *    a. Exact Match: First attempts to find the exact SEARCH block text in the original file
+ *    b. Line-Trimmed Match: Falls back to line-by-line comparison ignoring leading/trailing whitespace
+ *    c. Block Anchor Match: For blocks of 3+ lines, tries to match using first/last lines as anchors
+ *    If all matching strategies fail, an error is thrown.
+ *
+ * 3. Empty SEARCH Section:
+ *    - If SEARCH is empty and the original file is empty, this indicates creating a new file
+ *      (pure insertion).
+ *    - If SEARCH is empty and the original file is not empty, this indicates a complete
+ *      file replacement (the entire original content is considered matched and replaced).
+ *
+ * 4. Applying Changes:
+ *    - Before encountering the "=======" marker, lines are accumulated as search content.
+ *    - After "=======" and before ">>>>>>> REPLACE", lines are accumulated as replacement content.
+ *    - Once the block is complete (">>>>>>> REPLACE"), the matched section in the original
+ *      file is replaced with the accumulated replacement lines, and the position in the original
+ *      file is advanced.
+ *
+ * 5. Incremental Output:
+ *    - As soon as the match location is found and we are in the REPLACE section, each new
+ *      replacement line is appended to the result so that partial updates can be viewed
+ *      incrementally.
+ *
+ * 6. Partial Markers:
+ *    - If the final line of the chunk looks like it might be part of a marker but is not one
+ *      of the known markers, it is removed. This prevents incomplete or partial markers
+ *      from corrupting the output.
+ *
+ * 7. Finalization:
+ *    - Once all chunks have been processed (when `isFinal` is true), any remaining original
+ *      content after the last replaced section is appended to the result.
+ *    - Trailing newlines are not forcibly added. The code tries to output exactly what is specified.
+ *
+ * Errors:
+ * - If the search block cannot be matched using any of the available matching strategies,
+ *   an error is thrown.
+ */
+export async function constructNewFileContent(
+	diffContent: string,
+	originalContent: string,
+	isFinal: boolean,
+	version: "v1" | "v2" = "v1",
+): Promise<string> {
+	const constructor = constructNewFileContentVersionMapping[version]
+	if (!constructor) {
+		throw new Error(`Invalid version '${version}' for file content constructor`)
+	}
+	return constructor(diffContent, originalContent, isFinal)
+}
+
+const constructNewFileContentVersionMapping: Record<
+	string,
+	(diffContent: string, originalContent: string, isFinal: boolean) => Promise<string>
+> = {
+	v1: constructNewFileContentV1,
+	v2: constructNewFileContentV2,
+} as const
+
+async function constructNewFileContentV1(diffContent: string, originalContent: string, isFinal: boolean): Promise<string> {
+	let result = ""
+	let lastProcessedIndex = 0
+
+	let currentSearchContent = ""
+	let currentReplaceContent = ""
+	let inSearch = false
+	let inReplace = false
+
+	let searchMatchIndex = -1
+	let searchEndIndex = -1
+
+	// Track all replacements to handle out-of-order edits
+	let replacements: Array<{ start: number; end: number; content: string }> = []
+	let pendingOutOfOrderReplacement = false
+
+	let lines = diffContent.split("\n")
+
+	// If the last line looks like a partial marker but isn't recognized,
+	// remove it because it might be incomplete.
+	const lastLine = lines[lines.length - 1]
+	if (
+		lines.length > 0 &&
+		(lastLine.startsWith(SEARCH_BLOCK_CHAR) ||
+			lastLine.startsWith(LEGACY_SEARCH_BLOCK_CHAR) ||
+			lastLine.startsWith("=") ||
+			lastLine.startsWith(REPLACE_BLOCK_CHAR) ||
+			lastLine.startsWith(LEGACY_REPLACE_BLOCK_CHAR)) &&
+		!isSearchBlockStart(lastLine) &&
+		!isSearchBlockEnd(lastLine) &&
+		!isReplaceBlockEnd(lastLine)
+	) {
+		lines.pop()
+	}
+
+	for (const line of lines) {
+		if (isSearchBlockStart(line)) {
+			inSearch = true
+			currentSearchContent = ""
+			currentReplaceContent = ""
+			continue
+		}
+
+		if (isSearchBlockEnd(line)) {
+			inSearch = false
+			inReplace = true
+
+			// Remove trailing linebreak for adding the === marker
+			// if (currentSearchContent.endsWith("\r\n")) {
+			// 	currentSearchContent = currentSearchContent.slice(0, -2)
+			// } else if (currentSearchContent.endsWith("\n")) {
+			// 	currentSearchContent = currentSearchContent.slice(0, -1)
+			// }
+
+			if (!currentSearchContent) {
+				// Empty search block
+				if (originalContent.length === 0) {
+					// New file scenario: nothing to match, just start inserting
+					searchMatchIndex = 0
+					searchEndIndex = 0
+				} else {
+					// ERROR: Empty search block with non-empty file indicates malformed SEARCH marker
+					throw new Error(
+						"Empty SEARCH block detected with non-empty file. This usually indicates a malformed SEARCH marker.\n" +
+							"Please ensure your SEARCH marker follows the correct format:\n" +
+							"- Use '------- SEARCH' (7+ dashes + space + SEARCH)\n",
+					)
+				}
+			} else {
+				// Add check for inefficient full-file search
+				// if (currentSearchContent.trim() === originalContent.trim()) {
+				// 	throw new Error(
+				// 		"The SEARCH block contains the entire file content. Please either:\n" +
+				// 			"1. Use an empty SEARCH block to replace the entire file, or\n" +
+				// 			"2. Make focused changes to specific parts of the file that need modification.",
+				// 	)
+				// }
+
+				// Exact search match scenario
+				const exactIndex = originalContent.indexOf(currentSearchContent, lastProcessedIndex)
+				if (exactIndex !== -1) {
+					searchMatchIndex = exactIndex
+					searchEndIndex = exactIndex + currentSearchContent.length
+				} else {
+					// Attempt fallback line-trimmed matching
+					const lineMatch = lineTrimmedFallbackMatch(originalContent, currentSearchContent, lastProcessedIndex)
+					if (lineMatch) {
+						;[searchMatchIndex, searchEndIndex] = lineMatch
+					} else {
+						// Try block anchor fallback for larger blocks
+						const blockMatch = blockAnchorFallbackMatch(originalContent, currentSearchContent, lastProcessedIndex)
+						if (blockMatch) {
+							;[searchMatchIndex, searchEndIndex] = blockMatch
+						} else {
+							// Last resort: search the entire file from the beginning
+							const fullFileIndex = originalContent.indexOf(currentSearchContent, 0)
+							if (fullFileIndex !== -1) {
+								// Found in the file - could be out of order
+								searchMatchIndex = fullFileIndex
+								searchEndIndex = fullFileIndex + currentSearchContent.length
+								if (searchMatchIndex < lastProcessedIndex) {
+									pendingOutOfOrderReplacement = true
+								}
+							} else {
+								throw new Error(
+									`The SEARCH block:\n${currentSearchContent.trimEnd()}\n...does not match anything in the file.`,
+								)
+							}
+						}
+					}
+				}
+			}
+
+			// Check if this is an out-of-order replacement
+			if (searchMatchIndex < lastProcessedIndex) {
+				pendingOutOfOrderReplacement = true
+			}
+
+			// For in-order replacements, output everything up to the match location
+			if (!pendingOutOfOrderReplacement) {
+				result += originalContent.slice(lastProcessedIndex, searchMatchIndex)
+			}
+			continue
+		}
+
+		if (isReplaceBlockEnd(line)) {
+			// Finished one replace block
+
+			// Store this replacement
+			replacements.push({
+				start: searchMatchIndex,
+				end: searchEndIndex,
+				content: currentReplaceContent,
+			})
+
+			// If this was an in-order replacement, advance lastProcessedIndex
+			if (!pendingOutOfOrderReplacement) {
+				lastProcessedIndex = searchEndIndex
+			}
+
+			// Reset for next block
+			inSearch = false
+			inReplace = false
+			currentSearchContent = ""
+			currentReplaceContent = ""
+			searchMatchIndex = -1
+			searchEndIndex = -1
+			pendingOutOfOrderReplacement = false
+			continue
+		}
+
+		// Accumulate content for search or replace
+		// (currentReplaceContent is not being used for anything right now since we directly append to result.)
+		// (We artificially add a linebreak since we split on \n at the beginning. In order to not include a trailing linebreak in the final search/result blocks we need to remove it before using them. This allows for partial line matches to be correctly identified.)
+		// NOTE: search/replace blocks must be arranged in the order they appear in the file due to how we build the content using lastProcessedIndex. We also cannot strip the trailing newline since for non-partial lines it would remove the linebreak from the original content. (If we remove end linebreak from search, then we'd also have to remove it from replace but we can't know if it's a partial line or not since the model may be using the line break to indicate the end of the block rather than as part of the search content.) We require the model to output full lines in order for our fallbacks to work as well.
+		if (inSearch) {
+			currentSearchContent += line + "\n"
+		} else if (inReplace) {
+			currentReplaceContent += line + "\n"
+			// Only output replacement lines immediately for in-order replacements
+			if (searchMatchIndex !== -1 && !pendingOutOfOrderReplacement) {
+				result += line + "\n"
+			}
+		}
+	}
+
+	// If this is the final chunk, we need to apply all replacements and build the final result
+	if (isFinal) {
+		// Handle the case where we're still in replace mode when processing ends
+		// and this is the final chunk - treat it as if we encountered the REPLACE marker
+		if (inReplace && searchMatchIndex !== -1) {
+			// Store this replacement
+			replacements.push({
+				start: searchMatchIndex,
+				end: searchEndIndex,
+				content: currentReplaceContent,
+			})
+
+			// If this was an in-order replacement, advance lastProcessedIndex
+			if (!pendingOutOfOrderReplacement) {
+				lastProcessedIndex = searchEndIndex
+			}
+
+			// Reset state
+			inSearch = false
+			inReplace = false
+			currentSearchContent = ""
+			currentReplaceContent = ""
+			searchMatchIndex = -1
+			searchEndIndex = -1
+			pendingOutOfOrderReplacement = false
+		}
+		// end of handling missing replace marker
+
+		// Sort replacements by start position
+		replacements.sort((a, b) => a.start - b.start)
+
+		// Rebuild the entire result by applying all replacements
+		result = ""
+		let currentPos = 0
+
+		for (const replacement of replacements) {
+			// Add original content up to this replacement
+			result += originalContent.slice(currentPos, replacement.start)
+			// Add the replacement content
+			result += replacement.content
+			// Move position to after the replaced section
+			currentPos = replacement.end
+		}
+
+		// Add any remaining original content
+		result += originalContent.slice(currentPos)
+	}
+
+	return result
+}
+
+enum ProcessingState {
+	Idle = 0,
+	StateSearch = 1 << 0,
+	StateReplace = 1 << 1,
+}
+
+class NewFileContentConstructor {
+	private originalContent: string
+	private isFinal: boolean
+	private state: number
+	private pendingNonStandardLines: string[]
+	private result: string
+	private lastProcessedIndex: number
+	private currentSearchContent: string
+	private currentReplaceContent: string
+	private searchMatchIndex: number
+	private searchEndIndex: number
+
+	constructor(originalContent: string, isFinal: boolean) {
+		this.originalContent = originalContent
+		this.isFinal = isFinal
+		this.pendingNonStandardLines = []
+		this.result = ""
+		this.lastProcessedIndex = 0
+		this.state = ProcessingState.Idle
+		this.currentSearchContent = ""
+		this.currentReplaceContent = ""
+		this.searchMatchIndex = -1
+		this.searchEndIndex = -1
+	}
+
+	private resetForNextBlock() {
+		// Reset for next block
+		this.state = ProcessingState.Idle
+		this.currentSearchContent = ""
+		this.currentReplaceContent = ""
+		this.searchMatchIndex = -1
+		this.searchEndIndex = -1
+	}
+
+	private findLastMatchingLineIndex(regx: RegExp, lineLimit: number) {
+		for (let i = lineLimit; i > 0; ) {
+			i--
+			if (this.pendingNonStandardLines[i].match(regx)) {
+				return i
+			}
+		}
+		return -1
+	}
+
+	private updateProcessingState(newState: ProcessingState) {
+		const isValidTransition =
+			(this.state === ProcessingState.Idle && newState === ProcessingState.StateSearch) ||
+			(this.state === ProcessingState.StateSearch && newState === ProcessingState.StateReplace)
+
+		if (!isValidTransition) {
+			throw new Error(
+				`Invalid state transition.\n` +
+					"Valid transitions are:\n" +
+					"- Idle → StateSearch\n" +
+					"- StateSearch → StateReplace",
+			)
+		}
+
+		this.state |= newState
+	}
+
+	private isStateActive(state: ProcessingState): boolean {
+		return (this.state & state) === state
+	}
+
+	private activateReplaceState() {
+		this.updateProcessingState(ProcessingState.StateReplace)
+	}
+
+	private activateSearchState() {
+		this.updateProcessingState(ProcessingState.StateSearch)
+		this.currentSearchContent = ""
+		this.currentReplaceContent = ""
+	}
+
+	private isSearchingActive(): boolean {
+		return this.isStateActive(ProcessingState.StateSearch)
+	}
+
+	private isReplacingActive(): boolean {
+		return this.isStateActive(ProcessingState.StateReplace)
+	}
+
+	private hasPendingNonStandardLines(pendingNonStandardLineLimit: number): boolean {
+		return this.pendingNonStandardLines.length - pendingNonStandardLineLimit < this.pendingNonStandardLines.length
+	}
+
+	public processLine(line: string) {
+		this.internalProcessLine(line, true, this.pendingNonStandardLines.length)
+	}
+
+	public getResult() {
+		// If this is the final chunk, append any remaining original content
+		if (this.isFinal && this.lastProcessedIndex < this.originalContent.length) {
+			this.result += this.originalContent.slice(this.lastProcessedIndex)
+		}
+		if (this.isFinal && this.state !== ProcessingState.Idle) {
+			throw new Error("File processing incomplete - SEARCH/REPLACE operations still active during finalization")
+		}
+		return this.result
+	}
+
+	private internalProcessLine(
+		line: string,
+		canWritependingNonStandardLines: boolean,
+		pendingNonStandardLineLimit: number,
+	): number {
+		let removeLineCount = 0
+		if (isSearchBlockStart(line)) {
+			removeLineCount = this.trimPendingNonStandardTrailingEmptyLines(pendingNonStandardLineLimit)
+			if (removeLineCount > 0) {
+				pendingNonStandardLineLimit = pendingNonStandardLineLimit - removeLineCount
+			}
+			if (this.hasPendingNonStandardLines(pendingNonStandardLineLimit)) {
+				this.tryFixSearchReplaceBlock(pendingNonStandardLineLimit)
+				canWritependingNonStandardLines && (this.pendingNonStandardLines.length = 0)
+			}
+			this.activateSearchState()
+		} else if (isSearchBlockEnd(line)) {
+			// 校验非标内容
+			if (!this.isSearchingActive()) {
+				this.tryFixSearchBlock(pendingNonStandardLineLimit)
+				canWritependingNonStandardLines && (this.pendingNonStandardLines.length = 0)
+			}
+			this.activateReplaceState()
+			this.beforeReplace()
+		} else if (isReplaceBlockEnd(line)) {
+			if (!this.isReplacingActive()) {
+				this.tryFixReplaceBlock(pendingNonStandardLineLimit)
+				canWritependingNonStandardLines && (this.pendingNonStandardLines.length = 0)
+			}
+			this.lastProcessedIndex = this.searchEndIndex
+			this.resetForNextBlock()
+		} else {
+			// Accumulate content for search or replace
+			// (currentReplaceContent is not being used for anything right now since we directly append to result.)
+			// (We artificially add a linebreak since we split on \n at the beginning. In order to not include a trailing linebreak in the final search/result blocks we need to remove it before using them. This allows for partial line matches to be correctly identified.)
+			// NOTE: search/replace blocks must be arranged in the order they appear in the file due to how we build the content using lastProcessedIndex. We also cannot strip the trailing newline since for non-partial lines it would remove the linebreak from the original content. (If we remove end linebreak from search, then we'd also have to remove it from replace but we can't know if it's a partial line or not since the model may be using the line break to indicate the end of the block rather than as part of the search content.) We require the model to output full lines in order for our fallbacks to work as well.
+			if (this.isReplacingActive()) {
+				this.currentReplaceContent += line + "\n"
+				// Output replacement lines immediately if we know the insertion point
+				if (this.searchMatchIndex !== -1) {
+					this.result += line + "\n"
+				}
+			} else if (this.isSearchingActive()) {
+				this.currentSearchContent += line + "\n"
+			} else {
+				let appendToPendingNonStandardLines = canWritependingNonStandardLines
+				if (appendToPendingNonStandardLines) {
+					// 处理非标内容
+					this.pendingNonStandardLines.push(line)
+				}
+			}
+		}
+		return removeLineCount
+	}
+
+	private beforeReplace() {
+		// Remove trailing linebreak for adding the === marker
+		// if (currentSearchContent.endsWith("\r\n")) {
+		// 	currentSearchContent = currentSearchContent.slice(0, -2)
+		// } else if (currentSearchContent.endsWith("\n")) {
+		// 	currentSearchContent = currentSearchContent.slice(0, -1)
+		// }
+
+		if (!this.currentSearchContent) {
+			// Empty search block
+			if (this.originalContent.length === 0) {
+				// New file scenario: nothing to match, just start inserting
+				this.searchMatchIndex = 0
+				this.searchEndIndex = 0
+			} else {
+				// Complete file replacement scenario: treat the entire file as matched
+				this.searchMatchIndex = 0
+				this.searchEndIndex = this.originalContent.length
+			}
+		} else {
+			// Add check for inefficient full-file search
+			// if (currentSearchContent.trim() === originalContent.trim()) {
+			// 	throw new Error(
+			// 		"The SEARCH block contains the entire file content. Please either:\n" +
+			// 			"1. Use an empty SEARCH block to replace the entire file, or\n" +
+			// 			"2. Make focused changes to specific parts of the file that need modification.",
+			// 	)
+			// }
+			// Exact search match scenario
+			const exactIndex = this.originalContent.indexOf(this.currentSearchContent, this.lastProcessedIndex)
+			if (exactIndex !== -1) {
+				this.searchMatchIndex = exactIndex
+				this.searchEndIndex = exactIndex + this.currentSearchContent.length
+			} else {
+				// Attempt fallback line-trimmed matching
+				const lineMatch = lineTrimmedFallbackMatch(
+					this.originalContent,
+					this.currentSearchContent,
+					this.lastProcessedIndex,
+				)
+				if (lineMatch) {
+					;[this.searchMatchIndex, this.searchEndIndex] = lineMatch
+				} else {
+					// Try block anchor fallback for larger blocks
+					const blockMatch = blockAnchorFallbackMatch(
+						this.originalContent,
+						this.currentSearchContent,
+						this.lastProcessedIndex,
+					)
+					if (blockMatch) {
+						;[this.searchMatchIndex, this.searchEndIndex] = blockMatch
+					} else {
+						throw new Error(
+							`The SEARCH block:\n${this.currentSearchContent.trimEnd()}\n...does not match anything in the file.`,
+						)
+					}
+				}
+			}
+		}
+		if (this.searchMatchIndex < this.lastProcessedIndex) {
+			throw new Error(
+				`The SEARCH block:\n${this.currentSearchContent.trimEnd()}\n...matched an incorrect content in the file.`,
+			)
+		}
+		// Output everything up to the match location
+		this.result += this.originalContent.slice(this.lastProcessedIndex, this.searchMatchIndex)
+	}
+
+	private tryFixSearchBlock(lineLimit: number): number {
+		let removeLineCount = 0
+		if (lineLimit < 0) {
+			lineLimit = this.pendingNonStandardLines.length
+		}
+		if (!lineLimit) {
+			throw new Error("Invalid SEARCH/REPLACE block structure - no lines available to process")
+		}
+		let searchTagRegexp = /^([-]{3,}|[<]{3,}) SEARCH$/
+		const searchTagIndex = this.findLastMatchingLineIndex(searchTagRegexp, lineLimit)
+		if (searchTagIndex !== -1) {
+			let fixLines = this.pendingNonStandardLines.slice(searchTagIndex, lineLimit)
+			fixLines[0] = SEARCH_BLOCK_START
+			for (const line of fixLines) {
+				removeLineCount += this.internalProcessLine(line, false, searchTagIndex)
+			}
+		} else {
+			throw new Error(
+				`Invalid REPLACE marker detected - could not find matching SEARCH block starting from line ${searchTagIndex + 1}`,
+			)
+		}
+		return removeLineCount
+	}
+
+	private tryFixReplaceBlock(lineLimit: number): number {
+		let removeLineCount = 0
+		if (lineLimit < 0) {
+			lineLimit = this.pendingNonStandardLines.length
+		}
+		if (!lineLimit) {
+			throw new Error()
+		}
+		let replaceBeginTagRegexp = /^[=]{3,}$/
+		const replaceBeginTagIndex = this.findLastMatchingLineIndex(replaceBeginTagRegexp, lineLimit)
+		if (replaceBeginTagIndex !== -1) {
+			// // 校验非标内容
+			// if (!this.isSearchingActive()) {
+			// 	removeLineCount += this.tryFixSearchBlock(replaceBeginTagIndex)
+			// }
+			let fixLines = this.pendingNonStandardLines.slice(replaceBeginTagIndex - removeLineCount, lineLimit - removeLineCount)
+			fixLines[0] = SEARCH_BLOCK_END
+			for (const line of fixLines) {
+				removeLineCount += this.internalProcessLine(line, false, replaceBeginTagIndex - removeLineCount)
+			}
+		} else {
+			throw new Error(`Malformed REPLACE block - missing valid separator after line ${replaceBeginTagIndex + 1}`)
+		}
+		return removeLineCount
+	}
+
+	private tryFixSearchReplaceBlock(lineLimit: number): number {
+		let removeLineCount = 0
+		if (lineLimit < 0) {
+			lineLimit = this.pendingNonStandardLines.length
+		}
+		if (!lineLimit) {
+			throw new Error()
+		}
+
+		let replaceEndTagRegexp = /^([+]{3,}|[>]{3,}) REPLACE$/
+		const replaceEndTagIndex = this.findLastMatchingLineIndex(replaceEndTagRegexp, lineLimit)
+		const likeReplaceEndTag = replaceEndTagIndex === lineLimit - 1
+		if (likeReplaceEndTag) {
+			// // 校验非标内容
+			// if (!this.isReplacingActive()) {
+			// 	removeLineCount += this.tryFixReplaceBlock(replaceEndTagIndex)
+			// }
+			let fixLines = this.pendingNonStandardLines.slice(replaceEndTagIndex - removeLineCount, lineLimit - removeLineCount)
+			fixLines[fixLines.length - 1] = REPLACE_BLOCK_END
+			for (const line of fixLines) {
+				removeLineCount += this.internalProcessLine(line, false, replaceEndTagIndex - removeLineCount)
+			}
+		} else {
+			throw new Error("Malformed SEARCH/REPLACE block structure: Missing valid closing REPLACE marker")
+		}
+		return removeLineCount
+	}
+
+	/**
+	 * Removes trailing empty lines from the pendingNonStandardLines array
+	 * @param lineLimit - The index to start checking from (exclusive).
+	 *                    Removes empty lines from lineLimit-1 backwards.
+	 * @returns The number of empty lines removed
+	 */
+	private trimPendingNonStandardTrailingEmptyLines(lineLimit: number): number {
+		let removedCount = 0
+		let i = Math.min(lineLimit, this.pendingNonStandardLines.length) - 1
+
+		while (i >= 0 && this.pendingNonStandardLines[i].trim() === "") {
+			this.pendingNonStandardLines.pop()
+			removedCount++
+			i--
+		}
+
+		return removedCount
+	}
+}
+
+export async function constructNewFileContentV2(diffContent: string, originalContent: string, isFinal: boolean): Promise<string> {
+	let newFileContentConstructor = new NewFileContentConstructor(originalContent, isFinal)
+
+	let lines = diffContent.split("\n")
+
+	// If the last line looks like a partial marker but isn't recognized,
+	// remove it because it might be incomplete.
+	const lastLine = lines[lines.length - 1]
+	if (
+		lines.length > 0 &&
+		(lastLine.startsWith(SEARCH_BLOCK_CHAR) ||
+			lastLine.startsWith(LEGACY_SEARCH_BLOCK_CHAR) ||
+			lastLine.startsWith("=") ||
+			lastLine.startsWith(REPLACE_BLOCK_CHAR) ||
+			lastLine.startsWith(LEGACY_REPLACE_BLOCK_CHAR)) &&
+		lastLine !== SEARCH_BLOCK_START &&
+		lastLine !== SEARCH_BLOCK_END &&
+		lastLine !== REPLACE_BLOCK_END
+	) {
+		lines.pop()
+	}
+
+	for (const line of lines) {
+		newFileContentConstructor.processLine(line)
+	}
+
+	let result = newFileContentConstructor.getResult()
+	return result
+}

evals/diff-edits/diff-apply/diff-06-26-25.ts

@@ -0,0 +1,960 @@
+const SEARCH_BLOCK_START = "------- SEARCH"
+const SEARCH_BLOCK_END = "======="
+const REPLACE_BLOCK_END = "+++++++ REPLACE"
+
+const SEARCH_BLOCK_CHAR = "-"
+const REPLACE_BLOCK_CHAR = "+"
+const LEGACY_SEARCH_BLOCK_CHAR = "<"
+const LEGACY_REPLACE_BLOCK_CHAR = ">"
+
+// Replace the exact string constants with flexible regex patterns
+const SEARCH_BLOCK_START_REGEX = /^[-]{3,} SEARCH>?$/
+const LEGACY_SEARCH_BLOCK_START_REGEX = /^[<]{3,} SEARCH>?$/
+
+const SEARCH_BLOCK_END_REGEX = /^[=]{3,}$/
+
+const REPLACE_BLOCK_END_REGEX = /^[+]{3,} REPLACE>?$/
+const LEGACY_REPLACE_BLOCK_END_REGEX = /^[>]{3,} REPLACE>?$/
+
+// Similarity thresholds for block anchor fallback matching
+const SINGLE_CANDIDATE_SIMILARITY_THRESHOLD = 0.0
+const MULTIPLE_CANDIDATES_SIMILARITY_THRESHOLD = 0.0
+
+/**
+ * Levenshtein distance algorithm implementation
+ */
+function levenshtein(a: string, b: string): number {
+	// Handle empty strings
+	if (a === "" || b === "") {
+		return Math.max(a.length, b.length)
+	}
+	const matrix = Array.from({ length: a.length + 1 }, (_, i) =>
+		Array.from({ length: b.length + 1 }, (_, j) => (i === 0 ? j : j === 0 ? i : 0)),
+	)
+
+	for (let i = 1; i <= a.length; i++) {
+		for (let j = 1; j <= b.length; j++) {
+			const cost = a[i - 1] === b[j - 1] ? 0 : 1
+			matrix[i][j] = Math.min(matrix[i - 1][j] + 1, matrix[i][j - 1] + 1, matrix[i - 1][j - 1] + cost)
+		}
+	}
+	return matrix[a.length][b.length]
+}
+
+// Helper functions to check if a line matches the flexible patterns
+function isSearchBlockStart(line: string): boolean {
+	return SEARCH_BLOCK_START_REGEX.test(line) || LEGACY_SEARCH_BLOCK_START_REGEX.test(line)
+}
+
+function isSearchBlockEnd(line: string): boolean {
+	return SEARCH_BLOCK_END_REGEX.test(line)
+}
+
+function isReplaceBlockEnd(line: string): boolean {
+	return REPLACE_BLOCK_END_REGEX.test(line) || LEGACY_REPLACE_BLOCK_END_REGEX.test(line)
+}
+
+/**
+ * Attempts a line-trimmed fallback match for the given search content in the original content.
+ * It tries to match `searchContent` lines against a block of lines in `originalContent` starting
+ * from `lastProcessedIndex`. Lines are matched by trimming leading/trailing whitespace and ensuring
+ * they are identical afterwards.
+ *
+ * Returns [matchIndexStart, matchIndexEnd] if found, or false if not found.
+ */
+function lineTrimmedFallbackMatch(originalContent: string, searchContent: string, startIndex: number): [number, number] | false {
+	// Split both contents into lines
+	const originalLines = originalContent.split("\n")
+	const searchLines = searchContent.split("\n")
+
+	// Trim trailing empty line if exists (from the trailing \n in searchContent)
+	if (searchLines[searchLines.length - 1] === "") {
+		searchLines.pop()
+	}
+
+	// Find the line number where startIndex falls
+	let startLineNum = 0
+	let currentIndex = 0
+	while (currentIndex < startIndex && startLineNum < originalLines.length) {
+		currentIndex += originalLines[startLineNum].length + 1 // +1 for \n
+		startLineNum++
+	}
+
+	// For each possible starting position in original content
+	for (let i = startLineNum; i <= originalLines.length - searchLines.length; i++) {
+		let matches = true
+
+		// Try to match all search lines from this position
+		for (let j = 0; j < searchLines.length; j++) {
+			const originalTrimmed = originalLines[i + j].trim()
+			const searchTrimmed = searchLines[j].trim()
+
+			if (originalTrimmed !== searchTrimmed) {
+				matches = false
+				break
+			}
+		}
+
+		// If we found a match, calculate the exact character positions
+		if (matches) {
+			// Find start character index
+			let matchStartIndex = 0
+			for (let k = 0; k < i; k++) {
+				matchStartIndex += originalLines[k].length + 1 // +1 for \n
+			}
+
+			// Find end character index
+			let matchEndIndex = matchStartIndex
+			for (let k = 0; k < searchLines.length; k++) {
+				matchEndIndex += originalLines[i + k].length + 1 // +1 for \n
+			}
+
+			return [matchStartIndex, matchEndIndex]
+		}
+	}
+
+	return false
+}
+
+/**
+ * Attempts to match blocks of code by using the first and last lines as anchors,
+ * with similarity checking to prevent false positives.
+ * This is a third-tier fallback strategy that helps match blocks where we can identify
+ * the correct location by matching the beginning and end, even if the exact content
+ * differs slightly.
+ *
+ * The matching strategy:
+ * 1. Only attempts to match blocks of 3 or more lines to avoid false positives
+ * 2. Extracts from the search content:
+ *    - First line as the "start anchor"
+ *    - Last line as the "end anchor"
+ * 3. Collects all candidate positions where both anchors match
+ * 4. Uses levenshtein distance to calculate similarity of middle lines
+ * 5. Returns match only if similarity meets threshold requirements
+ *
+ * This approach is particularly useful for matching blocks of code where:
+ * - The exact content might have minor differences
+ * - The beginning and end of the block are distinctive enough to serve as anchors
+ * - The overall structure (number of lines) remains the same
+ * - The middle content is reasonably similar (prevents false positives)
+ *
+ * @param originalContent - The full content of the original file
+ * @param searchContent - The content we're trying to find in the original file
+ * @param startIndex - The character index in originalContent where to start searching
+ * @returns A tuple of [startIndex, endIndex] if a match is found, false otherwise
+ */
+function blockAnchorFallbackMatch(originalContent: string, searchContent: string, startIndex: number): [number, number, number] | false {
+	const originalLines = originalContent.split("\n")
+	const searchLines = searchContent.split("\n")
+
+	// Only use this approach for blocks of 3+ lines
+	if (searchLines.length < 3) {
+		return false
+	}
+
+	// Trim trailing empty line if exists
+	if (searchLines[searchLines.length - 1] === "") {
+		searchLines.pop()
+	}
+
+	const firstLineSearch = searchLines[0].trim()
+	const lastLineSearch = searchLines[searchLines.length - 1].trim()
+	const searchBlockSize = searchLines.length
+
+	// Find the line number where startIndex falls
+	let startLineNum = 0
+	let currentIndex = 0
+	while (currentIndex < startIndex && startLineNum < originalLines.length) {
+		currentIndex += originalLines[startLineNum].length + 1
+		startLineNum++
+	}
+
+	// Collect all candidate positions
+	const candidates: number[] = []
+	for (let i = startLineNum; i <= originalLines.length - searchBlockSize; i++) {
+		if (originalLines[i].trim() === firstLineSearch && originalLines[i + searchBlockSize - 1].trim() === lastLineSearch) {
+			candidates.push(i)
+		}
+	}
+
+	// Return immediately if no candidates
+	if (candidates.length === 0) {
+		return false
+	}
+
+	// Handle single candidate scenario (using relaxed threshold)
+	if (candidates.length === 1) {
+		const i = candidates[0]
+		let similarity = 0
+		let linesToCheck = searchBlockSize - 2
+
+		for (let j = 1; j < searchBlockSize - 1; j++) {
+			const originalLine = originalLines[i + j].trim()
+			const searchLine = searchLines[j].trim()
+			const maxLen = Math.max(originalLine.length, searchLine.length)
+			if (maxLen === 0) {
+				continue
+			}
+			const distance = levenshtein(originalLine, searchLine)
+			similarity += (1 - distance / maxLen) / linesToCheck
+
+			// Exit early when threshold is reached
+			if (similarity >= SINGLE_CANDIDATE_SIMILARITY_THRESHOLD) {
+				break
+			}
+		}
+
+		if (similarity >= SINGLE_CANDIDATE_SIMILARITY_THRESHOLD) {
+			let matchStartIndex = 0
+			for (let k = 0; k < i; k++) {
+				matchStartIndex += originalLines[k].length + 1
+			}
+			let matchEndIndex = matchStartIndex
+			for (let k = 0; k < searchBlockSize; k++) {
+				matchEndIndex += originalLines[i + k].length + 1
+			}
+			return [matchStartIndex, matchEndIndex, similarity]
+		}
+		return false
+	}
+
+	// Calculate similarity for multiple candidates
+	let bestMatchIndex = -1
+	let maxSimilarity = -1
+
+	for (const i of candidates) {
+		let similarity = 0
+		for (let j = 1; j < searchBlockSize - 1; j++) {
+			const originalLine = originalLines[i + j].trim()
+			const searchLine = searchLines[j].trim()
+			const maxLen = Math.max(originalLine.length, searchLine.length)
+			if (maxLen === 0) {
+				continue
+			}
+			const distance = levenshtein(originalLine, searchLine)
+			similarity += 1 - distance / maxLen
+		}
+		similarity /= searchBlockSize - 2 // Average similarity
+
+		if (similarity > maxSimilarity) {
+			maxSimilarity = similarity
+			bestMatchIndex = i
+		}
+	}
+
+	// Threshold judgment
+	if (maxSimilarity >= MULTIPLE_CANDIDATES_SIMILARITY_THRESHOLD) {
+		const i = bestMatchIndex
+		let matchStartIndex = 0
+		for (let k = 0; k < i; k++) {
+			matchStartIndex += originalLines[k].length + 1
+		}
+		let matchEndIndex = matchStartIndex
+		for (let k = 0; k < searchBlockSize; k++) {
+			matchEndIndex += originalLines[i + k].length + 1
+		}
+		return [matchStartIndex, matchEndIndex, maxSimilarity]
+	}
+
+	return false
+}
+
+/**
+ * This function reconstructs the file content by applying a streamed diff (in a
+ * specialized SEARCH/REPLACE block format) to the original file content. It is designed
+ * to handle both incremental updates and the final resulting file after all chunks have
+ * been processed.
+ *
+ * The diff format is a custom structure that uses three markers to define changes:
+ *
+ *   ------- SEARCH
+ *   [Exact content to find in the original file]
+ *   =======
+ *   [Content to replace with]
+ *   +++++++ REPLACE
+ *
+ * Behavior and Assumptions:
+ * 1. The file is processed chunk-by-chunk. Each chunk of `diffContent` may contain
+ *    partial or complete SEARCH/REPLACE blocks. By calling this function with each
+ *    incremental chunk (with `isFinal` indicating the last chunk), the final reconstructed
+ *    file content is produced.
+ *
+ * 2. Matching Strategy (in order of attempt):
+ *    a. Exact Match: First attempts to find the exact SEARCH block text in the original file
+ *    b. Line-Trimmed Match: Falls back to line-by-line comparison ignoring leading/trailing whitespace
+ *    c. Block Anchor Match: For blocks of 3+ lines, tries to match using first/last lines as anchors
+ *    If all matching strategies fail, an error is thrown.
+ *
+ * 3. Empty SEARCH Section:
+ *    - If SEARCH is empty and the original file is empty, this indicates creating a new file
+ *      (pure insertion).
+ *    - If SEARCH is empty and the original file is not empty, this indicates a complete
+ *      file replacement (the entire original content is considered matched and replaced).
+ *
+ * 4. Applying Changes:
+ *    - Before encountering the "=======" marker, lines are accumulated as search content.
+ *    - After "=======" and before ">>>>>>> REPLACE", lines are accumulated as replacement content.
+ *    - Once the block is complete (">>>>>>> REPLACE"), the matched section in the original
+ *      file is replaced with the accumulated replacement lines, and the position in the original
+ *      file is advanced.
+ *
+ * 5. Incremental Output:
+ *    - As soon as the match location is found and we are in the REPLACE section, each new
+ *      replacement line is appended to the result so that partial updates can be viewed
+ *      incrementally.
+ *
+ * 6. Partial Markers:
+ *    - If the final line of the chunk looks like it might be part of a marker but is not one
+ *      of the known markers, it is removed. This prevents incomplete or partial markers
+ *      from corrupting the output.
+ *
+ * 7. Finalization:
+ *    - Once all chunks have been processed (when `isFinal` is true), any remaining original
+ *      content after the last replaced section is appended to the result.
+ *    - Trailing newlines are not forcibly added. The code tries to output exactly what is specified.
+ *
+ * Errors:
+ * - If the search block cannot be matched using any of the available matching strategies,
+ *   an error is thrown.
+ */
+export async function constructNewFileContent(
+	diffContent: string,
+	originalContent: string,
+	isFinal: boolean,
+	version: "v1" | "v2" = "v1",
+): Promise<any> {
+	const constructor = constructNewFileContentVersionMapping[version]
+	if (!constructor) {
+		throw new Error(`Invalid version '${version}' for file content constructor`)
+	}
+	return constructor(diffContent, originalContent, isFinal)
+}
+
+const constructNewFileContentVersionMapping: Record<
+	string,
+	(diffContent: string, originalContent: string, isFinal: boolean) => Promise<any>
+> = {
+	v1: constructNewFileContentV1,
+	v2: constructNewFileContentV2,
+} as const
+
+async function constructNewFileContentV1(diffContent: string, originalContent: string, isFinal: boolean): Promise<{
+	content: string;
+	replacements: Array<{ 
+		start: number; 
+		end: number; 
+		content: string;
+		method: string;
+		similarity: number;
+		searchContent: string;
+		matchedText: string;
+	}>;
+}> {
+	let result = ""
+	let lastProcessedIndex = 0
+
+	let currentSearchContent = ""
+	let currentReplaceContent = ""
+	let inSearch = false
+	let inReplace = false
+
+	let searchMatchIndex = -1
+	let searchEndIndex = -1
+	let matchMethod = ""
+	let similarityScore = -1.0
+
+	// Track all replacements to handle out-of-order edits
+	let replacements: Array<{ 
+		start: number; 
+		end: number; 
+		content: string;
+		method: string;
+		similarity: number;
+		searchContent: string;
+		matchedText: string;
+	}> = []
+	let pendingOutOfOrderReplacement = false
+
+	let lines = diffContent.split("\n")
+
+	// If the last line looks like a partial marker but isn't recognized,
+	// remove it because it might be incomplete.
+	const lastLine = lines[lines.length - 1]
+	if (
+		lines.length > 0 &&
+		(lastLine.startsWith(SEARCH_BLOCK_CHAR) ||
+			lastLine.startsWith(LEGACY_SEARCH_BLOCK_CHAR) ||
+			lastLine.startsWith("=") ||
+			lastLine.startsWith(REPLACE_BLOCK_CHAR) ||
+			lastLine.startsWith(LEGACY_REPLACE_BLOCK_CHAR)) &&
+		!isSearchBlockStart(lastLine) &&
+		!isSearchBlockEnd(lastLine) &&
+		!isReplaceBlockEnd(lastLine)
+	) {
+		lines.pop()
+	}
+
+	for (const line of lines) {
+		if (isSearchBlockStart(line)) {
+			inSearch = true
+			currentSearchContent = ""
+			currentReplaceContent = ""
+			continue
+		}
+
+		if (isSearchBlockEnd(line)) {
+			inSearch = false
+			inReplace = true
+
+			// Remove trailing linebreak for adding the === marker
+			// if (currentSearchContent.endsWith("\r\n")) {
+			// 	currentSearchContent = currentSearchContent.slice(0, -2)
+			// } else if (currentSearchContent.endsWith("\n")) {
+			// 	currentSearchContent = currentSearchContent.slice(0, -1)
+			// }
+
+			if (!currentSearchContent) {
+				// Empty search block
+				if (originalContent.length === 0) {
+					// New file scenario: nothing to match, just start inserting
+					searchMatchIndex = 0
+					searchEndIndex = 0
+					matchMethod = "empty_new_file"
+				} else {
+					// ERROR: Empty search block with non-empty file indicates malformed SEARCH marker
+					throw new Error(
+						"Empty SEARCH block detected with non-empty file. This usually indicates a malformed SEARCH marker.\n" +
+							"Please ensure your SEARCH marker follows the correct format:\n" +
+							"- Use '------- SEARCH' (7+ dashes + space + SEARCH)\n",
+					)
+				}
+			} else {
+				// Add check for inefficient full-file search
+				// if (currentSearchContent.trim() === originalContent.trim()) {
+				// 	throw new Error(
+				// 		"The SEARCH block contains the entire file content. Please either:\n" +
+				// 			"1. Use an empty SEARCH block to replace the entire file, or\n" +
+				// 			"2. Make focused changes to specific parts of the file that need modification.",
+				// 	)
+				// }
+
+				// Exact search match scenario
+				const exactIndex = originalContent.indexOf(currentSearchContent, lastProcessedIndex)
+				if (exactIndex !== -1) {
+					searchMatchIndex = exactIndex
+					searchEndIndex = exactIndex + currentSearchContent.length
+					matchMethod = "exact_match"
+				} else {
+					// Attempt fallback line-trimmed matching
+					const lineMatch = lineTrimmedFallbackMatch(originalContent, currentSearchContent, lastProcessedIndex)
+					if (lineMatch) {
+						;[searchMatchIndex, searchEndIndex] = lineMatch
+						matchMethod = "line_trimmed_fallback"
+					} else {
+						// Try block anchor fallback for larger blocks
+						const blockMatch = blockAnchorFallbackMatch(originalContent, currentSearchContent, lastProcessedIndex)
+						if (blockMatch) {
+							;[searchMatchIndex, searchEndIndex, similarityScore] = blockMatch
+							matchMethod = "block_anchor_fallback"
+						} else {
+							// Last resort: search the entire file from the beginning
+							const fullFileIndex = originalContent.indexOf(currentSearchContent, 0)
+							if (fullFileIndex !== -1) {
+								// Found in the file - could be out of order
+								searchMatchIndex = fullFileIndex
+								searchEndIndex = fullFileIndex + currentSearchContent.length
+								matchMethod = "full_file_search"
+								if (searchMatchIndex < lastProcessedIndex) {
+									pendingOutOfOrderReplacement = true
+								}
+							} else {
+								throw new Error(
+									`The SEARCH block:\n${currentSearchContent.trimEnd()}\n...does not match anything in the file.`,
+								)
+							}
+						}
+					}
+				}
+			}
+
+			// Check if this is an out-of-order replacement
+			if (searchMatchIndex < lastProcessedIndex) {
+				pendingOutOfOrderReplacement = true
+			}
+
+			// For in-order replacements, output everything up to the match location
+			if (!pendingOutOfOrderReplacement) {
+				result += originalContent.slice(lastProcessedIndex, searchMatchIndex)
+			}
+			continue
+		}
+
+		if (isReplaceBlockEnd(line)) {
+			// Finished one replace block
+
+			if (searchMatchIndex === -1) {
+				throw new Error(
+					`The SEARCH block:\n${currentSearchContent.trimEnd()}\n...is malformatted.`,
+				)
+			}
+
+			// Store this replacement
+			replacements.push({
+				start: searchMatchIndex,
+				end: searchEndIndex,
+				content: currentReplaceContent,
+				method: matchMethod,
+				similarity: similarityScore,
+				searchContent: currentSearchContent,
+				matchedText: originalContent.slice(searchMatchIndex, searchEndIndex),
+			})
+
+			// If this was an in-order replacement, advance lastProcessedIndex
+			if (!pendingOutOfOrderReplacement) {
+				lastProcessedIndex = searchEndIndex
+			}
+
+			// Reset for next block
+			inSearch = false
+			inReplace = false
+			currentSearchContent = ""
+			currentReplaceContent = ""
+			searchMatchIndex = -1
+			searchEndIndex = -1
+			similarityScore = -1.0
+			pendingOutOfOrderReplacement = false
+			continue
+		}
+
+		// Accumulate content for search or replace
+		// (currentReplaceContent is not being used for anything right now since we directly append to result.)
+		// (We artificially add a linebreak since we split on \n at the beginning. In order to not include a trailing linebreak in the final search/result blocks we need to remove it before using them. This allows for partial line matches to be correctly identified.)
+		// NOTE: search/replace blocks must be arranged in the order they appear in the file due to how we build the content using lastProcessedIndex. We also cannot strip the trailing newline since for non-partial lines it would remove the linebreak from the original content. (If we remove end linebreak from search, then we'd also have to remove it from replace but we can't know if it's a partial line or not since the model may be using the line break to indicate the end of the block rather than as part of the search content.) We require the model to output full lines in order for our fallbacks to work as well.
+		if (inSearch) {
+			currentSearchContent += line + "\n"
+		} else if (inReplace) {
+			currentReplaceContent += line + "\n"
+			// Only output replacement lines immediately for in-order replacements
+			if (searchMatchIndex !== -1 && !pendingOutOfOrderReplacement) {
+				result += line + "\n"
+			}
+		}
+	}
+
+	// If this is the final chunk, we need to apply all replacements and build the final result
+	if (isFinal) {
+		// Handle the case where we're still in replace mode when processing ends
+		// and this is the final chunk - treat it as if we encountered the REPLACE marker
+		if (inReplace && searchMatchIndex !== -1) {
+			// Store this replacement
+			replacements.push({
+				start: searchMatchIndex,
+				end: searchEndIndex,
+				content: currentReplaceContent,
+				method: matchMethod,
+				similarity: similarityScore,
+				searchContent: currentSearchContent,
+				matchedText: originalContent.slice(searchMatchIndex, searchEndIndex),
+			})
+
+			// If this was an in-order replacement, advance lastProcessedIndex
+			if (!pendingOutOfOrderReplacement) {
+				lastProcessedIndex = searchEndIndex
+			}
+
+			// Reset state
+			inSearch = false
+			inReplace = false
+			currentSearchContent = ""
+			currentReplaceContent = ""
+			searchMatchIndex = -1
+			searchEndIndex = -1
+			pendingOutOfOrderReplacement = false
+		}
+		// end of handling missing replace marker
+
+		// Sort replacements by start position
+		replacements.sort((a, b) => a.start - b.start)
+
+		// Rebuild the entire result by applying all replacements
+		result = ""
+		let currentPos = 0
+
+		for (const replacement of replacements) {
+			// Add original content up to this replacement
+			result += originalContent.slice(currentPos, replacement.start)
+			// Add the replacement content
+			result += replacement.content
+			// Move position to after the replaced section
+			currentPos = replacement.end
+		}
+
+		// Add any remaining original content
+		result += originalContent.slice(currentPos)
+	}
+
+	// For testing - return debug info
+	return {
+		content: result,
+		replacements: replacements
+	}
+}
+
+enum ProcessingState {
+	Idle = 0,
+	StateSearch = 1 << 0,
+	StateReplace = 1 << 1,
+}
+
+class NewFileContentConstructor {
+	private originalContent: string
+	private isFinal: boolean
+	private state: number
+	private pendingNonStandardLines: string[]
+	private result: string
+	private lastProcessedIndex: number
+	private currentSearchContent: string
+	private currentReplaceContent: string
+	private searchMatchIndex: number
+	private searchEndIndex: number
+
+	constructor(originalContent: string, isFinal: boolean) {
+		this.originalContent = originalContent
+		this.isFinal = isFinal
+		this.pendingNonStandardLines = []
+		this.result = ""
+		this.lastProcessedIndex = 0
+		this.state = ProcessingState.Idle
+		this.currentSearchContent = ""
+		this.currentReplaceContent = ""
+		this.searchMatchIndex = -1
+		this.searchEndIndex = -1
+	}
+
+	private resetForNextBlock() {
+		// Reset for next block
+		this.state = ProcessingState.Idle
+		this.currentSearchContent = ""
+		this.currentReplaceContent = ""
+		this.searchMatchIndex = -1
+		this.searchEndIndex = -1
+	}
+
+	private findLastMatchingLineIndex(regx: RegExp, lineLimit: number) {
+		for (let i = lineLimit; i > 0; ) {
+			i--
+			if (this.pendingNonStandardLines[i].match(regx)) {
+				return i
+			}
+		}
+		return -1
+	}
+
+	private updateProcessingState(newState: ProcessingState) {
+		const isValidTransition =
+			(this.state === ProcessingState.Idle && newState === ProcessingState.StateSearch) ||
+			(this.state === ProcessingState.StateSearch && newState === ProcessingState.StateReplace)
+
+		if (!isValidTransition) {
+			throw new Error(
+				`Invalid state transition.\n` +
+					"Valid transitions are:\n" +
+					"- Idle → StateSearch\n" +
+					"- StateSearch → StateReplace",
+			)
+		}
+
+		this.state |= newState
+	}
+
+	private isStateActive(state: ProcessingState): boolean {
+		return (this.state & state) === state
+	}
+
+	private activateReplaceState() {
+		this.updateProcessingState(ProcessingState.StateReplace)
+	}
+
+	private activateSearchState() {
+		this.updateProcessingState(ProcessingState.StateSearch)
+		this.currentSearchContent = ""
+		this.currentReplaceContent = ""
+	}
+
+	private isSearchingActive(): boolean {
+		return this.isStateActive(ProcessingState.StateSearch)
+	}
+
+	private isReplacingActive(): boolean {
+		return this.isStateActive(ProcessingState.StateReplace)
+	}
+
+	private hasPendingNonStandardLines(pendingNonStandardLineLimit: number): boolean {
+		return this.pendingNonStandardLines.length - pendingNonStandardLineLimit < this.pendingNonStandardLines.length
+	}
+
+	public processLine(line: string) {
+		this.internalProcessLine(line, true, this.pendingNonStandardLines.length)
+	}
+
+	public getResult() {
+		// If this is the final chunk, append any remaining original content
+		if (this.isFinal && this.lastProcessedIndex < this.originalContent.length) {
+			this.result += this.originalContent.slice(this.lastProcessedIndex)
+		}
+		if (this.isFinal && this.state !== ProcessingState.Idle) {
+			throw new Error("File processing incomplete - SEARCH/REPLACE operations still active during finalization")
+		}
+		return this.result
+	}
+
+	private internalProcessLine(
+		line: string,
+		canWritependingNonStandardLines: boolean,
+		pendingNonStandardLineLimit: number,
+	): number {
+		let removeLineCount = 0
+		if (isSearchBlockStart(line)) {
+			removeLineCount = this.trimPendingNonStandardTrailingEmptyLines(pendingNonStandardLineLimit)
+			if (removeLineCount > 0) {
+				pendingNonStandardLineLimit = pendingNonStandardLineLimit - removeLineCount
+			}
+			if (this.hasPendingNonStandardLines(pendingNonStandardLineLimit)) {
+				this.tryFixSearchReplaceBlock(pendingNonStandardLineLimit)
+				canWritependingNonStandardLines && (this.pendingNonStandardLines.length = 0)
+			}
+			this.activateSearchState()
+		} else if (isSearchBlockEnd(line)) {
+			// 校验非标内容
+			if (!this.isSearchingActive()) {
+				this.tryFixSearchBlock(pendingNonStandardLineLimit)
+				canWritependingNonStandardLines && (this.pendingNonStandardLines.length = 0)
+			}
+			this.activateReplaceState()
+			this.beforeReplace()
+		} else if (isReplaceBlockEnd(line)) {
+			if (!this.isReplacingActive()) {
+				this.tryFixReplaceBlock(pendingNonStandardLineLimit)
+				canWritependingNonStandardLines && (this.pendingNonStandardLines.length = 0)
+			}
+			this.lastProcessedIndex = this.searchEndIndex
+			this.resetForNextBlock()
+		} else {
+			// Accumulate content for search or replace
+			// (currentReplaceContent is not being used for anything right now since we directly append to result.)
+			// (We artificially add a linebreak since we split on \n at the beginning. In order to not include a trailing linebreak in the final search/result blocks we need to remove it before using them. This allows for partial line matches to be correctly identified.)
+			// NOTE: search/replace blocks must be arranged in the order they appear in the file due to how we build the content using lastProcessedIndex. We also cannot strip the trailing newline since for non-partial lines it would remove the linebreak from the original content. (If we remove end linebreak from search, then we'd also have to remove it from replace but we can't know if it's a partial line or not since the model may be using the line break to indicate the end of the block rather than as part of the search content.) We require the model to output full lines in order for our fallbacks to work as well.
+			if (this.isReplacingActive()) {
+				this.currentReplaceContent += line + "\n"
+				// Output replacement lines immediately if we know the insertion point
+				if (this.searchMatchIndex !== -1) {
+					this.result += line + "\n"
+				}
+			} else if (this.isSearchingActive()) {
+				this.currentSearchContent += line + "\n"
+			} else {
+				let appendToPendingNonStandardLines = canWritependingNonStandardLines
+				if (appendToPendingNonStandardLines) {
+					// 处理非标内容
+					this.pendingNonStandardLines.push(line)
+				}
+			}
+		}
+		return removeLineCount
+	}
+
+	private beforeReplace() {
+		// Remove trailing linebreak for adding the === marker
+		// if (currentSearchContent.endsWith("\r\n")) {
+		// 	currentSearchContent = currentSearchContent.slice(0, -2)
+		// } else if (currentSearchContent.endsWith("\n")) {
+		// 	currentSearchContent = currentSearchContent.slice(0, -1)
+		// }
+
+		if (!this.currentSearchContent) {
+			// Empty search block
+			if (this.originalContent.length === 0) {
+				// New file scenario: nothing to match, just start inserting
+				this.searchMatchIndex = 0
+				this.searchEndIndex = 0
+			} else {
+				// Complete file replacement scenario: treat the entire file as matched
+				this.searchMatchIndex = 0
+				this.searchEndIndex = this.originalContent.length
+			}
+		} else {
+			// Add check for inefficient full-file search
+			// if (currentSearchContent.trim() === originalContent.trim()) {
+			// 	throw new Error(
+			// 		"The SEARCH block contains the entire file content. Please either:\n" +
+			// 			"1. Use an empty SEARCH block to replace the entire file, or\n" +
+			// 			"2. Make focused changes to specific parts of the file that need modification.",
+			// 	)
+			// }
+			// Exact search match scenario
+			const exactIndex = this.originalContent.indexOf(this.currentSearchContent, this.lastProcessedIndex)
+			if (exactIndex !== -1) {
+				this.searchMatchIndex = exactIndex
+				this.searchEndIndex = exactIndex + this.currentSearchContent.length
+			} else {
+				// Attempt fallback line-trimmed matching
+				const lineMatch = lineTrimmedFallbackMatch(
+					this.originalContent,
+					this.currentSearchContent,
+					this.lastProcessedIndex,
+				)
+				if (lineMatch) {
+					;[this.searchMatchIndex, this.searchEndIndex] = lineMatch
+				} else {
+					// Try block anchor fallback for larger blocks
+					const blockMatch = blockAnchorFallbackMatch(
+						this.originalContent,
+						this.currentSearchContent,
+						this.lastProcessedIndex,
+					)
+					if (blockMatch) {
+						;[this.searchMatchIndex, this.searchEndIndex, /* ignore similarity */] = blockMatch
+					} else {
+						throw new Error(
+							`The SEARCH block:\n${this.currentSearchContent.trimEnd()}\n...does not match anything in the file.`,
+						)
+					}
+				}
+			}
+		}
+		if (this.searchMatchIndex < this.lastProcessedIndex) {
+			throw new Error(
+				`The SEARCH block:\n${this.currentSearchContent.trimEnd()}\n...matched an incorrect content in the file.`,
+			)
+		}
+		// Output everything up to the match location
+		this.result += this.originalContent.slice(this.lastProcessedIndex, this.searchMatchIndex)
+	}
+
+	private tryFixSearchBlock(lineLimit: number): number {
+		let removeLineCount = 0
+		if (lineLimit < 0) {
+			lineLimit = this.pendingNonStandardLines.length
+		}
+		if (!lineLimit) {
+			throw new Error("Invalid SEARCH/REPLACE block structure - no lines available to process")
+		}
+		let searchTagRegexp = /^([-]{3,}|[<]{3,}) SEARCH$/
+		const searchTagIndex = this.findLastMatchingLineIndex(searchTagRegexp, lineLimit)
+		if (searchTagIndex !== -1) {
+			let fixLines = this.pendingNonStandardLines.slice(searchTagIndex, lineLimit)
+			fixLines[0] = SEARCH_BLOCK_START
+			for (const line of fixLines) {
+				removeLineCount += this.internalProcessLine(line, false, searchTagIndex)
+			}
+		} else {
+			throw new Error(
+				`Invalid REPLACE marker detected - could not find matching SEARCH block starting from line ${searchTagIndex + 1}`,
+			)
+		}
+		return removeLineCount
+	}
+
+	private tryFixReplaceBlock(lineLimit: number): number {
+		let removeLineCount = 0
+		if (lineLimit < 0) {
+			lineLimit = this.pendingNonStandardLines.length
+		}
+		if (!lineLimit) {
+			throw new Error()
+		}
+		let replaceBeginTagRegexp = /^[=]{3,}$/
+		const replaceBeginTagIndex = this.findLastMatchingLineIndex(replaceBeginTagRegexp, lineLimit)
+		if (replaceBeginTagIndex !== -1) {
+			// // 校验非标内容
+			// if (!this.isSearchingActive()) {
+			// 	removeLineCount += this.tryFixSearchBlock(replaceBeginTagIndex)
+			// }
+			let fixLines = this.pendingNonStandardLines.slice(replaceBeginTagIndex - removeLineCount, lineLimit - removeLineCount)
+			fixLines[0] = SEARCH_BLOCK_END
+			for (const line of fixLines) {
+				removeLineCount += this.internalProcessLine(line, false, replaceBeginTagIndex - removeLineCount)
+			}
+		} else {
+			throw new Error(`Malformed REPLACE block - missing valid separator after line ${replaceBeginTagIndex + 1}`)
+		}
+		return removeLineCount
+	}
+
+	private tryFixSearchReplaceBlock(lineLimit: number): number {
+		let removeLineCount = 0
+		if (lineLimit < 0) {
+			lineLimit = this.pendingNonStandardLines.length
+		}
+		if (!lineLimit) {
+			throw new Error()
+		}
+
+		let replaceEndTagRegexp = /^([+]{3,}|[>]{3,}) REPLACE$/
+		const replaceEndTagIndex = this.findLastMatchingLineIndex(replaceEndTagRegexp, lineLimit)
+		const likeReplaceEndTag = replaceEndTagIndex === lineLimit - 1
+		if (likeReplaceEndTag) {
+			// // 校验非标内容
+			// if (!this.isReplacingActive()) {
+			// 	removeLineCount += this.tryFixReplaceBlock(replaceEndTagIndex)
+			// }
+			let fixLines = this.pendingNonStandardLines.slice(replaceEndTagIndex - removeLineCount, lineLimit - removeLineCount)
+			fixLines[fixLines.length - 1] = REPLACE_BLOCK_END
+			for (const line of fixLines) {
+				removeLineCount += this.internalProcessLine(line, false, replaceEndTagIndex - removeLineCount)
+			}
+		} else {
+			throw new Error("Malformed SEARCH/REPLACE block structure: Missing valid closing REPLACE marker")
+		}
+		return removeLineCount
+	}
+
+	/**
+	 * Removes trailing empty lines from the pendingNonStandardLines array
+	 * @param lineLimit - The index to start checking from (exclusive).
+	 *                    Removes empty lines from lineLimit-1 backwards.
+	 * @returns The number of empty lines removed
+	 */
+	private trimPendingNonStandardTrailingEmptyLines(lineLimit: number): number {
+		let removedCount = 0
+		let i = Math.min(lineLimit, this.pendingNonStandardLines.length) - 1
+
+		while (i >= 0 && this.pendingNonStandardLines[i].trim() === "") {
+			this.pendingNonStandardLines.pop()
+			removedCount++
+			i--
+		}
+
+		return removedCount
+	}
+}
+
+export async function constructNewFileContentV2(diffContent: string, originalContent: string, isFinal: boolean): Promise<string> {
+	let newFileContentConstructor = new NewFileContentConstructor(originalContent, isFinal)
+
+	let lines = diffContent.split("\n")
+
+	// If the last line looks like a partial marker but isn't recognized,
+	// remove it because it might be incomplete.
+	const lastLine = lines[lines.length - 1]
+	if (
+		lines.length > 0 &&
+		(lastLine.startsWith(SEARCH_BLOCK_CHAR) ||
+			lastLine.startsWith(LEGACY_SEARCH_BLOCK_CHAR) ||
+			lastLine.startsWith("=") ||
+			lastLine.startsWith(REPLACE_BLOCK_CHAR) ||
+			lastLine.startsWith(LEGACY_REPLACE_BLOCK_CHAR)) &&
+		lastLine !== SEARCH_BLOCK_START &&
+		lastLine !== SEARCH_BLOCK_END &&
+		lastLine !== REPLACE_BLOCK_END
+	) {
+		lines.pop()
+	}
+
+	for (const line of lines) {
+		newFileContentConstructor.processLine(line)
+	}
+
+	let result = newFileContentConstructor.getResult()
+	return result
+}

evals/diff-edits/helpers.ts

@@ -23,3 +23,9 @@ export const formatResponse = {
 		return formatImagesIntoBlocks(images)
 	},
 }
+
+export function log(isVerbose: boolean, message: string) {
+	if (isVerbose) {
+		console.log(message)
+	}
+}

evals/diff-edits/openRouterModelsHelper.ts

@@ -0,0 +1,98 @@
+import axios from "axios";
+import path from "path";
+import fs from "fs/promises";
+
+// Minimal type for what we need from OpenRouter model info in evals
+export interface EvalOpenRouterModelInfo {
+    id: string;
+    contextWindow: number;
+    inputPrice?: number; // Price per million tokens
+    outputPrice?: number; // Price per million tokens
+    // Add any other fields if they become necessary for evals
+}
+
+function logHelper(isVerbose: boolean, message: string) {
+    if (isVerbose) {
+        console.log(`[OpenRouterModelsHelper] ${message}`);
+    }
+}
+
+/**
+ * Ensures the cache directory exists within evals and returns its path
+ */
+async function ensureEvalCacheDirectoryExists(): Promise<string> {
+    // Cache directory within evals, e.g., evals/.cache/
+    const cacheDir = path.join(__dirname, "..", ".cache");
+    await fs.mkdir(cacheDir, { recursive: true });
+    return cacheDir;
+}
+
+/**
+ * Fetches, parses, and caches OpenRouter model data.
+ * Tries to read from a local cache first.
+ * @param isVerbose Enable verbose logging
+ * @returns A record of model IDs to their info.
+ */
+export async function loadOpenRouterModelData(isVerbose: boolean = false): Promise<Record<string, EvalOpenRouterModelInfo>> {
+    const cacheDir = await ensureEvalCacheDirectoryExists();
+    const cacheFilePath = path.join(cacheDir, "openRouterModels.json");
+    let models: Record<string, EvalOpenRouterModelInfo> = {};
+
+    try {
+        const stats = await fs.stat(cacheFilePath).catch(() => null);
+        // Use cache if less than 24 hours old
+        if (stats && (Date.now() - stats.mtimeMs < 24 * 60 * 60 * 1000)) {
+            logHelper(isVerbose, "Using cached OpenRouter model data.");
+            const fileContents = await fs.readFile(cacheFilePath, "utf8");
+            models = JSON.parse(fileContents);
+            if (Object.keys(models).length > 0) {
+                return models;
+            }
+            logHelper(isVerbose, "Cache was empty or invalid, fetching fresh data.");
+        } else if (stats) {
+            logHelper(isVerbose, "Cached OpenRouter model data is stale, fetching fresh data.");
+        } else {
+            logHelper(isVerbose, "No cached OpenRouter model data found, fetching fresh data.");
+        }
+    } catch (e) {
+        logHelper(isVerbose, `Error accessing cache, fetching fresh data: ${e}`);
+    }
+
+    try {
+        const response = await axios.get("https://openrouter.ai/api/v1/models");
+        if (response.data?.data) {
+            const rawModels = response.data.data;
+            const parsedModels: Record<string, EvalOpenRouterModelInfo> = {};
+            const parsePrice = (price: any) => price ? parseFloat(price) * 1_000_000 : undefined;
+
+            for (const rawModel of rawModels) {
+                parsedModels[rawModel.id] = {
+                    id: rawModel.id,
+                    contextWindow: rawModel.context_length ?? 0,
+                    inputPrice: parsePrice(rawModel.pricing?.prompt),
+                    outputPrice: parsePrice(rawModel.pricing?.completion),
+                };
+            }
+            await fs.writeFile(cacheFilePath, JSON.stringify(parsedModels, null, 2));
+            logHelper(isVerbose, `Fetched and cached ${Object.keys(parsedModels).length} OpenRouter models.`);
+            return parsedModels;
+        } else {
+            logHelper(isVerbose, "Invalid response structure from OpenRouter API.");
+        }
+    } catch (error) {
+        logHelper(isVerbose, `Error fetching OpenRouter models: ${error}. Attempting to use stale cache if available.`);
+        // Attempt to read stale cache as a last resort if fetching failed
+        try {
+            const fileContents = await fs.readFile(cacheFilePath, "utf8");
+            models = JSON.parse(fileContents);
+            if (Object.keys(models).length > 0) {
+                logHelper(isVerbose, "Successfully loaded stale cache after fetch failure.");
+                return models;
+            }
+        } catch (cacheError) {
+            logHelper(isVerbose, `Failed to read stale cache: ${cacheError}. Proceeding without OpenRouter model data.`);
+        }
+    }
+    // Return empty if all attempts fail, so the caller can decide how to handle it
+    return {};
+}

evals/diff-edits/run_and_open_dashboard.sh

@@ -0,0 +1,34 @@
+#!/bin/bash
+
+# Get the directory of this script to make paths robust
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+# The 'evals' directory is the parent of the script's directory
+EVALS_DIR=$(dirname "$SCRIPT_DIR")
+
+# Navigate to the evals directory to ensure npm commands run correctly
+cd "$EVALS_DIR"
+
+# Re-install dependencies and build the CLI
+echo "Ensuring dependencies are up to date and building CLI..."
+npm install && npm run build:cli
+
+# Check if the build was successful before proceeding
+if [ $? -ne 0 ]; then
+    echo "CLI build failed. Aborting evaluation."
+    exit 1
+fi
+
+# Run the evaluation script, passing all arguments from the command line
+echo "Running evaluation..."
+node ./cli/dist/index.js run-diff-eval "$@"
+
+# Check the exit code of the evaluation script
+if [ $? -eq 0 ]; then
+  # If the script succeeded, open the dashboard in the background
+  echo "Evaluation complete. Starting dashboard..."
+  (cd "$SCRIPT_DIR/dashboard" && streamlit run app.py &)
+else
+  # If the script failed, print an error message and exit
+  echo "Evaluation failed. Dashboard will not be started."
+  exit 1
+fi

evals/diff-edits/types.ts

@@ -29,10 +29,12 @@ export interface TestConfig {
 	model_id: string
 	system_prompt_name: string
 	number_of_runs: number
+	max_attempts_per_case: number
 	parsing_function: string
 	diff_edit_function: string
 	thinking_tokens_budget: number
 	replay: boolean
+	diff_apply_file?: string
 }
 
 export interface SystemPromptDetails {
@@ -61,10 +63,26 @@ export type ConstructSystemPromptFn = (
 
 export interface TestResult {
 	success: boolean
-	streamResult?: any
+	streamResult?: {
+		assistantMessage: string
+		reasoningMessage: string
+		usage: {
+			inputTokens: number
+			outputTokens: number
+			cacheWriteTokens: number
+			cacheReadTokens: number
+			totalCost: number
+		}
+		timing?: {
+			timeToFirstTokenMs: number
+			timeToFirstEditMs?: number
+			totalRoundTripMs: number
+		}
+	}
 	diffEdit?: string
 	toolCalls?: ExtractedToolCall[]
 	diffEditSuccess?: boolean
+	replacementData?: any
 	error?: string
 	errorString?: string
 }
@@ -85,4 +103,6 @@ export interface TestInput {
 	diffEditFunction: string
 	thinkingBudgetTokens: number
 	originalDiffEditToolCallMessage?: string
+	diffApplyFile?: string
+	isVerbose: boolean
 }

evals/diff_editing/TestRunner.ts

@@ -1,391 +0,0 @@
-import { runSingleEvaluation, TestInput, TestResult } from "./ClineWrapper"
-import { basicSystemPrompt } from "./prompts/basicSystemPrompt-06-06-25"
-import { claude4SystemPrompt } from "./prompts/claude4SystemPrompt-06-06-25"
-import { formatResponse } from "./helpers"
-import { Anthropic } from "@anthropic-ai/sdk"
-import * as fs from "fs"
-import * as path from "path"
-import { Command } from "commander"
-import { InputMessage, ProcessedTestCase, TestCase, TestConfig, SystemPromptDetails, ConstructSystemPromptFn } from "./types"
-
-function log(isVerbose: boolean, message: string) {
-	if (isVerbose) {
-		console.log(message)
-	}
-}
-
-const systemPromptGeneratorLookup: Record<string, ConstructSystemPromptFn> = {
-	basicSystemPrompt: basicSystemPrompt,
-	claude4SystemPrompt: claude4SystemPrompt,
-}
-
-type TestResultSet = { [test_id: string]: (TestResult & { test_id?: string })[] }
-
-class NodeTestRunner {
-	private apiKey: string | undefined
-
-	constructor(isReplay: boolean) {
-		if (!isReplay) {
-			this.apiKey = process.env.OPENROUTER_API_KEY
-			if (!this.apiKey) {
-				throw new Error("OPENROUTER_API_KEY environment variable not set for a non-replay run.")
-			}
-		}
-	}
-
-	/**
-	 * convert our messages array into a properly formatted Anthropic messages array
-	 */
-	transformMessages(messages: InputMessage[]): Anthropic.Messages.MessageParam[] {
-		return messages.map((msg) => {
-			// Use TextBlockParam here for constructing the input message
-			const content: (Anthropic.TextBlockParam | Anthropic.ImageBlockParam)[] = []
-
-			if (msg.text) {
-				// This object now correctly matches the TextBlockParam type
-				content.push({ type: "text", text: msg.text })
-			}
-
-			if (msg.images && Array.isArray(msg.images)) {
-				const imageBlocks = formatResponse.imageBlocks(msg.images)
-				content.push(...imageBlocks)
-			}
-
-			return {
-				role: msg.role,
-				content: content,
-			}
-		})
-	}
-
-	/**
-	 * Generate the system prompt on the fly
-	 */
-	constructSystemPrompt(systemPromptDetails: SystemPromptDetails, systemPromptName: string) {
-		const systemPromptGenerator = systemPromptGeneratorLookup[systemPromptName]
-
-		const { cwd_value, browser_use, width, height, os_value, shell_value, home_value, mcp_string, user_custom_instructions } =
-			systemPromptDetails
-
-		const systemPrompt = systemPromptGenerator(
-			cwd_value,
-			browser_use,
-			width,
-			height,
-			os_value,
-			shell_value,
-			home_value,
-			mcp_string,
-			user_custom_instructions,
-		)
-
-		return systemPrompt
-	}
-
-	/**
-	 * Loads our test cases from a directory of json files
-	 */
-	loadTestCases(testDirectoryPath: string): TestCase[] {
-		const testCasesArray: TestCase[] = []
-		const dirents = fs.readdirSync(testDirectoryPath, { withFileTypes: true })
-
-		for (const dirent of dirents) {
-			if (dirent.isFile() && dirent.name.endsWith(".json")) {
-				const testFilePath = path.join(testDirectoryPath, dirent.name)
-				const fileContent = fs.readFileSync(testFilePath, "utf8")
-				const testCase: TestCase = JSON.parse(fileContent)
-
-				// Use the filename (without extension) as the test_id if not provided
-				if (!testCase.test_id) {
-					testCase.test_id = path.parse(dirent.name).name
-				}
-				testCasesArray.push(testCase)
-			}
-		}
-
-		return testCasesArray
-	}
-
-	/**
-	 * Saves the test results to the specified output directory.
-	 */
-	saveTestResults(results: TestResultSet, outputPath: string) {
-		// Ensure output directory exists
-		if (!fs.existsSync(outputPath)) {
-			fs.mkdirSync(outputPath, { recursive: true })
-		}
-
-		// Write each test result to its own file
-		for (const testId in results) {
-			const outputFilePath = path.join(outputPath, `${testId}.json`)
-			const testResult = results[testId]
-			fs.writeFileSync(outputFilePath, JSON.stringify(testResult, null, 2))
-		}
-	}
-
-	/**
-	 * Run a single test example
-	 */
-	async runSingleTest(testCase: ProcessedTestCase, testConfig: TestConfig): Promise<TestResult> {
-		if (testConfig.replay && !testCase.original_diff_edit_tool_call_message) {
-			return {
-				success: false,
-				error: "missing_original_diff_edit_tool_call_message",
-				errorString: `Test case ${testCase.test_id} is missing 'original_diff_edit_tool_call_message' for replay.`,
-			}
-		}
-
-		const customSystemPrompt = this.constructSystemPrompt(testCase.system_prompt_details, testConfig.system_prompt_name)
-
-		// messages don't include system prompt and are everything up to the first replace_in_file tool call which results in a diff edit error
-		const input: TestInput = {
-			apiKey: this.apiKey,
-			systemPrompt: customSystemPrompt,
-			messages: testCase.messages,
-			modelId: testConfig.model_id,
-			originalFile: testCase.file_contents,
-			originalFilePath: testCase.file_path,
-			parsingFunction: testConfig.parsing_function,
-			diffEditFunction: testConfig.diff_edit_function,
-			thinkingBudgetTokens: testConfig.thinking_tokens_budget,
-			originalDiffEditToolCallMessage: testConfig.replay ? testCase.original_diff_edit_tool_call_message : undefined,
-		}
-
-		return await runSingleEvaluation(input)
-	}
-
-	/**
-	 * Runs all the text examples synchonously
-	 */
-	async runAllTests(testCases: ProcessedTestCase[], testConfig: TestConfig, isVerbose: boolean): Promise<TestResultSet> {
-		const results: TestResultSet = {}
-
-		for (const testCase of testCases) {
-			results[testCase.test_id] = []
-
-			log(isVerbose, `-Running test: ${testCase.test_id}`)
-			for (let i = 0; i < testConfig.number_of_runs; i++) {
-				const result = await this.runSingleTest(testCase, testConfig)
-				results[testCase.test_id].push(result)
-			}
-		}
-		return results
-	}
-
-	/**
-	 * Runs all of the text examples asynchronously, with concurrency limit
-	 */
-	async runAllTestsParallel(
-		testCases: ProcessedTestCase[],
-		testConfig: TestConfig,
-		isVerbose: boolean,
-		maxConcurrency: number = 20,
-	): Promise<TestResultSet> {
-		const results: TestResultSet = {}
-		testCases.forEach((tc) => {
-			results[tc.test_id] = []
-		})
-
-		// Create a flat list of all individual runs we need to execute
-		const allRuns = testCases.flatMap((testCase) =>
-			Array(testConfig.number_of_runs)
-				.fill(null)
-				.map(() => testCase),
-		)
-
-		for (let i = 0; i < allRuns.length; i += maxConcurrency) {
-			const batch = allRuns.slice(i, i + maxConcurrency)
-
-			const batchPromises = batch.map((testCase) =>
-				this.runSingleTest(testCase, testConfig).then((result) => ({
-					...result,
-					test_id: testCase.test_id,
-				})),
-			)
-
-			const batchResults = await Promise.all(batchPromises)
-
-			// Calculate the total cost for this batch
-			const batchCost = batchResults.reduce((total, result) => {
-				return total + (result.streamResult?.usage?.totalCost || 0)
-			}, 0)
-
-			// Populate the results dictionary
-			for (const result of batchResults) {
-				if (result.test_id) {
-					results[result.test_id].push(result)
-				}
-			}
-
-			const batchNumber = i / maxConcurrency + 1
-			const totalBatches = Math.ceil(allRuns.length / maxConcurrency)
-			log(isVerbose, `-Completed batch ${batchNumber} of ${totalBatches}... (Batch Cost: $${batchCost.toFixed(6)})`)
-		}
-
-		return results
-	}
-
-	/**
-	 * Print output of the tests
-	 */
-	printSummary(results: TestResultSet, isVerbose: boolean) {
-		let totalRuns = 0
-		let totalPasses = 0
-		let totalInputTokens = 0
-		let totalOutputTokens = 0
-		let totalCost = 0
-		let runsWithUsageData = 0
-		let totalDiffEditSuccesses = 0
-		let totalRunsWithToolCalls = 0
-		const testCaseIds = Object.keys(results)
-
-		log(isVerbose, "\n=== TEST SUMMARY ===")
-
-		for (const testId of testCaseIds) {
-			const testResults = results[testId]
-			const passedCount = testResults.filter((r) => r.success && r.diffEditSuccess).length
-			const runCount = testResults.length
-
-			totalRuns += runCount
-			totalPasses += passedCount
-
-			const runsWithToolCalls = testResults.filter((r) => r.success === true).length
-			const diffEditSuccesses = passedCount
-			totalRunsWithToolCalls += runsWithToolCalls
-			totalDiffEditSuccesses += diffEditSuccesses
-
-			// Accumulate token and cost data
-			for (const result of testResults) {
-				if (result.streamResult?.usage) {
-					totalInputTokens += result.streamResult.usage.inputTokens
-					totalOutputTokens += result.streamResult.usage.outputTokens
-					totalCost += result.streamResult.usage.totalCost
-					runsWithUsageData++
-				}
-			}
-
-			log(isVerbose, `\n--- Test Case: ${testId} ---`)
-			log(isVerbose, `  Runs: ${runCount}`)
-			log(isVerbose, `  Passed: ${passedCount}`)
-			log(isVerbose, `  Success Rate: ${runCount > 0 ? ((passedCount / runCount) * 100).toFixed(1) : "N/A"}%`)
-		}
-
-		log(isVerbose, "\n\n=== OVERALL SUMMARY ===")
-		log(isVerbose, `Total Test Cases: ${testCaseIds.length}`)
-		log(isVerbose, `Total Runs Executed: ${totalRuns}`)
-		log(isVerbose, `Overall Passed: ${totalPasses}`)
-		log(isVerbose, `Overall Failed: ${totalRuns - totalPasses}`)
-		log(isVerbose, `Overall Success Rate: ${totalRuns > 0 ? ((totalPasses / totalRuns) * 100).toFixed(1) : "N/A"}%`)
-
-		log(isVerbose, "\n\n=== OVERALL DIFF EDIT SUCCESS RATE ===")
-		if (totalRunsWithToolCalls > 0) {
-			const diffSuccessRate = (totalDiffEditSuccesses / totalRunsWithToolCalls) * 100
-			log(isVerbose, `Total Runs with Successful Tool Calls: ${totalRunsWithToolCalls}`)
-			log(isVerbose, `Total Runs with Successful Diff Edits: ${totalDiffEditSuccesses}`)
-			log(isVerbose, `Diff Edit Success Rate: ${diffSuccessRate.toFixed(1)}%`)
-		} else {
-			log(isVerbose, "No successful tool calls to analyze for diff edit success.")
-		}
-
-		log(isVerbose, "\n\n=== TOKEN & COST ANALYSIS ===")
-		if (runsWithUsageData > 0) {
-			log(isVerbose, `Total Input Tokens: ${totalInputTokens.toLocaleString()}`)
-			log(isVerbose, `Total Output Tokens: ${totalOutputTokens.toLocaleString()}`)
-			log(isVerbose, `Total Cost: $${totalCost.toFixed(6)}`)
-			log(isVerbose, "---")
-			log(
-				isVerbose,
-				`Avg Input Tokens / Run: ${(totalInputTokens / runsWithUsageData).toLocaleString(undefined, {
-					maximumFractionDigits: 0,
-				})}`,
-			)
-			log(
-				isVerbose,
-				`Avg Output Tokens / Run: ${(totalOutputTokens / runsWithUsageData).toLocaleString(undefined, {
-					maximumFractionDigits: 0,
-				})}`,
-			)
-			log(isVerbose, `Avg Cost / Run: $${(totalCost / runsWithUsageData).toFixed(6)}`)
-		} else {
-			log(isVerbose, "No usage data available to analyze.")
-		}
-	}
-}
-
-async function main() {
-	const program = new Command()
-
-	const defaultTestPath = path.join(__dirname, "test_cases")
-	const defaultOutputPath = path.join(__dirname, "test_outputs")
-
-	program
-		.name("TestRunner")
-		.description("Run evaluation tests for diff editing")
-		.version("1.0.0")
-		.option("--test-path <path>", "Path to the directory containing test case JSON files", defaultTestPath)
-		.option("--output-path <path>", "Path to the directory to save the test output JSON files", defaultOutputPath)
-		.option("--model-id <model_id>", "The model ID to use for the test")
-		.option("--system-prompt-name <name>", "The name of the system prompt to use", "basicSystemPrompt")
-		.option("-n, --number-of-runs <number>", "Number of times to run each test case", "1")
-		.option("--parsing-function <name>", "The parsing function to use", "parseAssistantMessageV2")
-		.option("--diff-edit-function <name>", "The diff editing function to use", "constructNewFileContentV2")
-		.option("--thinking-budget <tokens>", "Set the thinking tokens budget", "0")
-		.option("--parallel", "Run tests in parallel", false)
-		.option("--replay", "Run evaluation from a pre-recorded LLM output, skipping the API call", false)
-		.option("-v, --verbose", "Enable verbose logging", false)
-
-	program.parse(process.argv)
-
-	const options = program.opts()
-	const isVerbose = options.verbose
-	const testPath = options.testPath
-	const outputPath = options.outputPath
-
-	const testConfig: TestConfig = {
-		model_id: options.modelId,
-		system_prompt_name: options.systemPromptName,
-		number_of_runs: parseInt(options.numberOfRuns, 10),
-		parsing_function: options.parsingFunction,
-		diff_edit_function: options.diffEditFunction,
-		thinking_tokens_budget: parseInt(options.thinkingBudget, 10),
-		replay: options.replay,
-	}
-
-	try {
-		const startTime = Date.now()
-
-		const runner = new NodeTestRunner(testConfig.replay)
-		const testCases = runner.loadTestCases(testPath)
-
-		const processedTestCases: ProcessedTestCase[] = testCases.map((tc) => ({
-			...tc,
-			messages: runner.transformMessages(tc.messages),
-		}))
-
-		log(isVerbose, `-Loaded ${testCases.length} test cases.`)
-		log(isVerbose, `-Executing ${testConfig.number_of_runs} run(s) per test case.`)
-		if (testConfig.replay) {
-			log(isVerbose, `-Running in REPLAY mode. No API calls will be made.`)
-		}
-		log(isVerbose, "Starting tests...\n")
-
-		const results = options.parallel
-			? await runner.runAllTestsParallel(processedTestCases, testConfig, isVerbose)
-			: await runner.runAllTests(processedTestCases, testConfig, isVerbose)
-
-		runner.printSummary(results, isVerbose)
-
-		const endTime = Date.now()
-		const durationSeconds = ((endTime - startTime) / 1000).toFixed(2)
-		log(isVerbose, `\n-Total execution time: ${durationSeconds} seconds`)
-
-		runner.saveTestResults(results, outputPath)
-	} catch (error) {
-		console.error("\nError running tests:", error)
-		process.exit(1)
-	}
-}
-
-if (require.main === module) {
-	main()
-}

evals/package.json

@@ -0,0 +1,44 @@
+{
+  "name": "cline-evals",
+  "version": "0.1.0",
+  "description": "Evaluation scripts and tools for Cline",
+  "main": "cli/dist/index.js",
+  "scripts": {
+    "build:cli": "cd cli && tsc",
+    "start:cli": "cd cli && node dist/index.js",
+    "dev:cli": "cd cli && ts-node src/index.ts",
+    "diff-eval": "./diff-edits/run_and_open_dashboard.sh",
+    "test": "echo \"Error: no test specified\" && exit 1"
+  },
+  "keywords": [
+    "cline",
+    "evaluation",
+    "benchmark",
+    "diff-edits"
+  ],
+  "author": "",
+  "license": "MIT",
+  "dependencies": {
+    "axios": "^1.8.2",
+    "better-sqlite3": "^11.10.0",
+    "chalk": "^4.1.2",
+    "dotenv": "^16.5.0",
+    "commander": "^9.4.1",
+    "execa": "^5.1.1",
+    "node-fetch": "^2.7.0",
+    "ora": "^5.4.1",
+    "sqlite": "^4.1.2",
+    "tiktoken": "^1.0.21",
+    "uuid": "^9.0.0",
+    "yargs": "^17.6.2"
+  },
+  "devDependencies": {
+    "@types/better-sqlite3": "^7.6.3",
+    "@types/node": "^18.11.18",
+    "@types/node-fetch": "^2.6.12",
+    "@types/uuid": "^9.0.0",
+    "@types/yargs": "^17.0.19",
+    "ts-node": "^10.9.1",
+    "typescript": "^4.9.4"
+  }
+}

evals/tsconfig.json

@@ -0,0 +1,6 @@
+{
+	"extends": "../tsconfig.json",
+	"compilerOptions": {
+		"baseUrl": ".."
+	}
+}

package.json

@@ -2,7 +2,7 @@
 	"name": "claude-dev",
 	"displayName": "Cline",
 	"description": "Autonomous coding agent right in your IDE, capable of creating/editing files, running commands, using the browser, and more with your permission every step of the way.",
-	"version": "3.17.14",
+	"version": "3.19.4",
 	"icon": "assets/icons/icon.png",
 	"engines": {
 		"vscode": "^1.84.0"
@@ -330,15 +330,16 @@
 		"watch:esbuild": "node esbuild.js --watch",
 		"watch:tsc": "tsc --noEmit --watch --project tsconfig.json",
 		"package": "npm run check-types && npm run build:webview && npm run lint && node esbuild.js --production",
-		"protos": "node proto/build-proto.js && node scripts/generate-server-setup.mjs && node scripts/generate-host-bridge-client.mjs",
+		"protos": "node scripts/build-proto.mjs && node scripts/generate-server-setup.mjs && node scripts/generate-host-bridge-client.mjs",
 		"postprotos": "prettier src/shared/proto src/core/controller src/hosts/ webview-ui/src/services src/generated --write --log-level warn",
+		"clean": "rimraf dist dist-standalone webview-ui/build src/generated out/",
 		"compile-tests": "node ./scripts/build-tests.js",
 		"watch-tests": "tsc -p . -w --outDir out",
-		"pretest": "npm run compile-tests && npm run compile && npm run compile-standalone && npm run lint",
-		"check-types": "npm run protos && tsc --noEmit",
+		"check-types": "npm run protos && npx tsc --noEmit && cd webview-ui && npx tsc -b --noEmit",
 		"lint": "eslint src --ext ts && eslint webview-ui/src --ext ts && buf lint && cd webview-ui && npm run lint",
 		"format": "prettier . --check",
 		"format:fix": "prettier . --write",
+		"pretest": "npm run compile && npm run compile-tests && npm run compile-standalone && npm run lint",
 		"test": "npm-run-all test:unit test:integration",
 		"test:ci": "node scripts/test-ci.js",
 		"test:integration": "vscode-test",
@@ -391,11 +392,13 @@
 		"grpc-tools": "^1.13.0",
 		"husky": "^9.1.7",
 		"lint-staged": "^16.1.0",
+		"minimatch": "^3.0.3",
 		"mintlify": "^4.0.515",
 		"npm-run-all": "^4.1.5",
 		"prettier": "^3.3.3",
 		"protoc-gen-ts": "^0.8.7",
 		"proxyquire": "^2.1.3",
+		"rimraf": "^6.0.1",
 		"should": "^13.2.3",
 		"sinon": "^19.0.2",
 		"ts-node": "^10.9.2",
@@ -406,8 +409,8 @@
 	"dependencies": {
 		"@anthropic-ai/sdk": "^0.37.0",
 		"@anthropic-ai/vertex-sdk": "^0.6.4",
-		"@aws-sdk/client-bedrock-runtime": "^3.826.0",
-		"@aws-sdk/credential-providers": "^3.826.0",
+		"@aws-sdk/client-bedrock-runtime": "^3.840.0",
+		"@aws-sdk/credential-providers": "^3.840.0",
 		"@bufbuild/protobuf": "^2.2.5",
 		"@cerebras/cerebras_cloud_sdk": "^1.35.0",
 		"@google-cloud/vertexai": "^1.9.3",
@@ -446,6 +449,7 @@
 		"image-size": "^2.0.2",
 		"isbinaryfile": "^5.0.2",
 		"jschardet": "^3.1.4",
+		"jwt-decode": "^4.0.0",
 		"mammoth": "^1.8.0",
 		"monaco-vscode-textmate-theme-converter": "^0.1.7",
 		"nice-grpc": "^2.1.12",

proto/account.proto

@@ -1,22 +1,129 @@
 syntax = "proto3";
 
 package cline;
+import "common.proto";
 option java_package = "bot.cline.proto";
 option java_multiple_files = true;
 
-import "common.proto";
-
 // Service for account-related operations
 service AccountService {
-    // Handles the user clicking the login link in the UI.
-    // Generates a secure nonce for state validation, stores it in secrets,
-    // and opens the authentication URL in the external browser.
-    rpc accountLoginClicked(EmptyRequest) returns (String);
+  // Handles the user clicking the login link in the UI.
+  // Generates a secure nonce for state validation, stores it in secrets,
+  // and opens the authentication URL in the external browser.
+  rpc accountLoginClicked(EmptyRequest) returns (String);
     
-    // Handles the user clicking the logout button in the UI.
-    // Clears API keys and user state.
-    rpc accountLogoutClicked(EmptyRequest) returns (Empty);
+  // Handles the user clicking the logout button in the UI.
+  // Clears API keys and user state.
+  rpc accountLogoutClicked(EmptyRequest) returns (Empty);
 
-    // Subscribe to auth callback events (when authentication tokens are received)
-    rpc subscribeToAuthCallback(EmptyRequest) returns (stream String);
+  // Subscribe to auth status update events (when authentication state changes)
+  rpc subscribeToAuthStatusUpdate(EmptyRequest) 
+      returns (stream AuthState);
+    
+  // Handles authentication state changes from the Firebase context.
+  // Updates the user info in global state and returns the updated value.
+  rpc authStateChanged(AuthStateChangedRequest) 
+      returns (AuthState);
+    
+  // Fetches all user credits data
+  // (balance, usage transactions, payment transactions)
+  rpc getUserCredits(EmptyRequest) returns (UserCreditsData);
+
+  rpc getOrganizationCredits(GetOrganizationCreditsRequest) returns (OrganizationCreditsData);
+
+  // Fetches all user organizations data
+  // Returns a list of UserOrganization objects
+  rpc getUserOrganizations(EmptyRequest) returns (UserOrganizationsResponse);
+
+  rpc setUserOrganization(UserOrganizationUpdateRequest) returns (Empty);
 }
+
+message AuthStateChangedRequest {
+  Metadata metadata = 1;
+  UserInfo user = 2;
+}
+
+message AuthState {
+  optional UserInfo user = 1;
+}
+
+// User's information
+message UserInfo {
+  string uid = 1;
+  optional string display_name = 2;
+  optional string email = 3;
+  optional string photo_url = 4;
+}
+
+message UserOrganization {
+  bool active = 1;
+  string member_id = 2;
+  string name = 3;
+  string organization_id = 4;
+  repeated string roles = 5; // ["admin", "member", "owner"]
+}
+
+message UserOrganizationsResponse {
+  repeated UserOrganization organizations = 1;
+}
+
+message UserOrganizationUpdateRequest {
+  optional string organization_id = 1;
+}
+
+message UserCreditsData {
+  UserCreditsBalance balance = 1;
+  repeated UsageTransaction usage_transactions = 2;
+  repeated PaymentTransaction payment_transactions = 3;
+}
+
+message GetOrganizationCreditsRequest {
+  string organization_id = 1;
+}
+
+message OrganizationCreditsData {
+  UserCreditsBalance balance = 1;
+  string organization_id = 2;
+  repeated OrganizationUsageTransaction usage_transactions = 3;
+}
+
+message UserCreditsBalance {
+  double current_balance = 1;
+}
+
+message UsageTransaction {
+  string ai_inference_provider_name = 1;
+  string ai_model_name = 2;
+  string ai_model_type_name = 3;
+  int32 completion_tokens = 4;
+  double cost_usd = 5;
+  string created_at = 6;
+  double credits_used = 7;
+  string generation_id = 8;
+  string organization_id = 9;
+  int32 prompt_tokens = 10;
+  int32 total_tokens = 11;
+  string user_id = 12;
+}
+
+message PaymentTransaction {
+  string paid_at = 1;
+  string creator_id = 2;
+  int32 amount_cents = 3;
+  double credits = 4;
+}
+
+message OrganizationUsageTransaction {
+  string ai_inference_provider_name = 1;
+  string ai_model_name = 2;
+  string ai_model_type_name = 3;
+  int32 completion_tokens = 4;
+  double cost_usd = 5;
+  string created_at = 6;
+  double credits_used = 7;
+  string generation_id = 8;
+  string organization_id = 9;
+  int32 prompt_tokens = 10;
+  int32 total_tokens = 11;
+  string user_id = 12;
+}

proto/browser.proto

@@ -1,11 +1,10 @@
 syntax = "proto3";
 
 package cline;
+import "common.proto";
 option java_package = "bot.cline.proto";
 option java_multiple_files = true;
 
-import "common.proto";
-
 service BrowserService {
   rpc getBrowserConnectionInfo(EmptyRequest) returns (BrowserConnectionInfo);
   rpc testBrowserConnection(StringRequest) returns (BrowserConnection);

proto/checkpoints.proto

@@ -1,11 +1,10 @@
 syntax = "proto3";
 
 package cline;
+import "common.proto";
 option java_package = "bot.cline.proto";
 option java_multiple_files = true;
 
-import "common.proto";
-
 service CheckpointsService {
   rpc checkpointDiff(Int64Request) returns (Empty);
   rpc checkpointRestore(CheckpointRestoreRequest) returns (Empty);

proto/file.proto

@@ -1,11 +1,10 @@
 syntax = "proto3";
 
 package cline;
+import "common.proto";
 option java_package = "bot.cline.proto";
 option java_multiple_files = true;
 
-import "common.proto";
-
 // Service for file-related operations
 service FileService {
   // Copies text to clipboard
@@ -29,9 +28,6 @@ service FileService {
   // Search git commits in the workspace
   rpc searchCommits(StringRequest) returns (GitCommits);
 
-  // Select images from the file system and return as data URLs
-  rpc selectImages(EmptyRequest) returns (StringArray);
-
   // Select images and other files from the file system and returns as data URLs & paths respectively
   rpc selectFiles(BooleanRequest) returns (StringArrays);
   

proto/host/diff.proto

@@ -0,0 +1,39 @@
+syntax = "proto3";
+
+package host;
+option java_package = "bot.cline.host.proto";
+option java_multiple_files = true;
+
+import "common.proto";
+
+// Provides methods for diff views.
+service DiffService {
+  // Open the diff view/editor.
+  rpc openDiff(OpenDiffRequest) returns (OpenDiffResponse);
+  rpc replaceText(ReplaceTextRequest) returns (ReplaceTextResponse);
+}
+
+message OpenDiffRequest {
+  optional cline.Metadata metadata = 1;
+  // The absolute path of the document being edited.
+  optional string path = 2;
+  // The new content for the file.
+  optional string content = 3;
+}
+
+message OpenDiffResponse {
+  // A unique identifier for the diff view that was opened.
+  optional string diff_id = 1;
+}
+
+message ReplaceTextRequest {
+  optional cline.Metadata metadata = 1;
+  optional string diff_id = 2;
+  optional string content = 3;
+  optional int32 start_line = 4;
+  optional int32 end_line = 5;
+}
+
+message ReplaceTextResponse {
+  // TBD
+}

proto/host/env.proto

@@ -0,0 +1,19 @@
+syntax = "proto3";
+
+package host;
+option java_package = "bot.cline.host.proto";
+option java_multiple_files = true;
+
+import "common.proto";
+
+// Provides methods for working with the user's environment.
+service EnvService {
+  // Writes text to the system clipboard.
+  rpc clipboardWriteText(cline.StringRequest) returns (cline.Empty);
+
+  // Reads text from the system clipboard.
+  rpc clipboardReadText(cline.EmptyRequest) returns (cline.String);
+
+  // Opens a URL in the user's default browser or application.
+  rpc openExternal(cline.StringRequest) returns (cline.Empty);
+}

proto/host/uri.proto

@@ -1,36 +0,0 @@
-syntax = "proto3";
-
-package host;
-option java_package = "bot.cline.host.proto";
-option java_multiple_files = true;
-
-import "common.proto";
-
-// UriService provides methods for working with URIs in the IDE
-service UriService {
-  // Create a new file URI from a file path
-  rpc file(cline.StringRequest) returns (Uri);
-  
-  // Join a URI with additional path segments
-  rpc joinPath(JoinPathRequest) returns (Uri);
-  
-  // Parse a string URI into a Uri object
-  rpc parse(cline.StringRequest) returns (Uri);
-}
-
-// Uri represents a URI in the IDE
-message Uri {
-  string scheme = 1;
-  string authority = 2;
-  string path = 3;
-  string query = 4;
-  string fragment = 5;
-  string fs_path = 6;
-}
-
-// Request for joining path segments to a URI
-message JoinPathRequest {
-  cline.Metadata metadata = 1;
-  Uri base = 2;
-  repeated string path_segments = 3;
-}

proto/host/window.proto

@@ -0,0 +1,73 @@
+syntax = "proto3";
+
+package host;
+option java_package = "bot.cline.host.proto";
+option java_multiple_files = true;
+
+import "common.proto";
+
+// Provides methods for working with IDE windows and editors.
+service WindowService {
+  // Opens a text document in the editor and returns editor information.
+  rpc showTextDocument(ShowTextDocumentRequest) returns (TextEditorInfo);
+  rpc showOpenDialogue(ShowOpenDialogueRequest) returns (SelectedResources);
+  rpc showMessage(ShowMessageRequest) returns (SelectedResponse);
+}
+
+message ShowTextDocumentRequest {
+  cline.Metadata metadata = 1;
+  string path = 2;
+  optional ShowTextDocumentOptions options = 3;
+}
+
+// See https://code.visualstudio.com/api/references/vscode-api#TextDocumentShowOptions
+message ShowTextDocumentOptions {
+  optional bool preview = 1;
+  optional bool preserve_focus = 2;
+  optional int32 view_column = 3;
+}
+
+message TextEditorInfo {
+  string document_path = 1;
+  optional int32 view_column = 2;
+  bool is_active = 3;
+}
+
+message ShowOpenDialogueRequest {
+  cline.Metadata metadata = 1;
+  optional bool can_select_many = 2;
+  optional string open_label = 3;
+  optional ShowOpenDialogueFilterOption filters = 4;
+}
+
+message ShowOpenDialogueFilterOption {
+  repeated string files = 1;
+}
+
+message SelectedResources {
+  repeated string paths = 1;
+}
+
+enum ShowMessageType {
+  ERROR = 0;
+  INFORMATION = 1;
+  WARNING = 2;
+}
+
+message ShowMessageRequest {
+  cline.Metadata metadata = 1;
+  ShowMessageType type = 2;
+  string message = 3;
+  optional ShowMessageRequestOptions options = 4;
+}
+
+message ShowMessageRequestOptions {
+  repeated string items = 1;
+  optional bool modal = 2;
+  optional string detail = 3;
+
+}
+
+message SelectedResponse {
+  optional string selected_option = 1;
+}

proto/host/workspace.proto

@@ -0,0 +1,24 @@
+syntax = "proto3";
+
+package host;
+option java_package = "bot.cline.host.proto";
+option java_multiple_files = true;
+
+// Provides methods for working with workspaces/projects.
+service WorkspaceService {
+  // Returns a list of the top level directories of the workspace.
+  rpc getWorkspacePaths(GetWorkspacePathsRequest) returns (GetWorkspacePathsResponse);
+}
+
+message GetWorkspacePathsRequest {
+  // The unique ID for the workspace/project.
+  // This is currently optional in vscode. It is required in other environments where cline is running at
+  // the application level, and the user can open multiple projects.
+  optional string id = 1;
+}
+
+message GetWorkspacePathsResponse {
+  // The unique ID for the workspace/project.
+  optional string id = 1;
+  repeated string paths = 2;
+}

proto/mcp.proto

@@ -1,16 +1,15 @@
 syntax = "proto3";
 
 package cline;
+import "common.proto";
 option java_package = "bot.cline.proto";
 option java_multiple_files = true;
 
-import "common.proto";
-
 service McpService {
   rpc toggleMcpServer(ToggleMcpServerRequest) returns (McpServers);
   rpc updateMcpTimeout(UpdateMcpTimeoutRequest) returns (McpServers);
   rpc addRemoteMcpServer(AddRemoteMcpServerRequest) returns (McpServers);
-  rpc downloadMcp(StringRequest) returns (Empty);
+  rpc downloadMcp(StringRequest) returns (McpDownloadResponse);
   rpc restartMcpServer(StringRequest) returns (McpServers);
   rpc deleteMcpServer(StringRequest) returns (McpServers);
   rpc toggleToolAutoApprove(ToggleToolAutoApproveRequest) returns (McpServers);
@@ -119,3 +118,15 @@ message McpMarketplaceItem {
 message McpMarketplaceCatalog {
   repeated McpMarketplaceItem items = 1;
 }
+
+message McpDownloadResponse {
+  string mcp_id = 1;
+  string github_url = 2;
+  string name = 3;
+  string author = 4;
+  string description = 5;
+  string readme_content = 6;
+  string llms_installation_content = 7;
+  bool requires_api_key = 8;
+  optional string error = 9;
+}

proto/models.proto

@@ -1,11 +1,10 @@
 syntax = "proto3";
 
 package cline;
+import "common.proto";
 option java_package = "bot.cline.proto";
 option java_multiple_files = true;
 
-import "common.proto";
-
 // Service for model-related operations
 service ModelsService {
   // Fetches available models from Ollama
@@ -123,6 +122,7 @@ enum ApiProvider {
   CEREBRAS = 23;
   SAPAICORE = 24;
   CLAUDE_CODE = 25;
+  MOONSHOT = 26;
 }
 
 // Model info for OpenAI-compatible models
@@ -165,7 +165,7 @@ message ModelsApiConfiguration {
   // From ApiHandlerOptions (excluding onRetryAttempt function)
   optional string api_model_id = 1;
   optional string api_key = 2;
-  optional string cline_api_key = 3;
+  optional string cline_account_id = 3;
   optional string task_id = 4;
   optional string lite_llm_base_url = 5;
   optional string lite_llm_model_id = 6;
@@ -236,4 +236,8 @@ message ModelsApiConfiguration {
   optional string sap_ai_core_token_url = 71;
   optional string sap_ai_core_base_url = 72;
   optional string claude_code_path = 73;
-}
+  optional string aws_authentication = 74;
+  optional string aws_bedrock_api_key = 75;
+  optional string moonshot_api_key = 76;
+  optional string moonshot_api_line = 77;
+}

proto/package.json

@@ -1,3 +0,0 @@
-{
-	"type": "module"
-}

proto/slash.proto

@@ -1,11 +1,10 @@
 syntax = "proto3";
 
 package cline;
+import "common.proto";
 option java_package = "bot.cline.proto";
 option java_multiple_files = true;
 
-import "common.proto";
-
 // SlashService provides methods for managing slash
 service SlashService {
   // Sends button click message

proto/state.proto

@@ -1,10 +1,9 @@
 syntax = "proto3";
 package cline;
+import "common.proto";
 option java_package = "bot.cline.proto";
 option java_multiple_files = true;
 
-import "common.proto";
-
 service StateService {
   rpc getLatestState(EmptyRequest) returns (State);
   rpc updateTerminalConnectionTimeout(Int64Request) returns (Int64);
@@ -17,6 +16,8 @@ service StateService {
   rpc togglePlanActMode(TogglePlanActModeRequest) returns (Boolean);
   rpc updateAutoApprovalSettings(AutoApprovalSettingsRequest) returns (Empty);
   rpc updateSettings(UpdateSettingsRequest) returns (Empty);
+  rpc updateTelemetrySetting(TelemetrySettingRequest) returns (Empty);
+  rpc setWelcomeViewCompleted(BooleanRequest) returns (Empty);
 }
 
 message State {
@@ -88,6 +89,17 @@ message AutoApprovalSettingsRequest {
   repeated string favorites = 7;
 }
 
+enum TelemetrySettingEnum {
+  UNSET = 0;
+  ENABLED = 1;
+  DISABLED = 2;
+}
+
+message TelemetrySettingRequest {
+  Metadata metadata = 1;
+  TelemetrySettingEnum setting = 2;
+}
+
 // Message for updating settings
 message UpdateSettingsRequest {
   Metadata metadata = 1;
@@ -100,7 +112,7 @@ message UpdateSettingsRequest {
   optional int64 shell_integration_timeout = 8;
   optional bool terminal_reuse_enabled = 9;
   optional bool mcp_responses_collapsed = 10;
-  optional bool mcp_rich_display_enabled = 11;
+  optional string mcp_display_mode = 11;
   optional int64 terminal_output_line_limit = 12;
 }
 
@@ -113,7 +125,7 @@ message ApiConfiguration {
   optional string api_base_url = 4;
   
   // Provider-specific API keys
-  optional string cline_api_key = 5;
+  optional string cline_account_id = 5;
   optional string openrouter_api_key = 6;
   optional string anthropic_base_url = 7;
   optional string openai_api_key = 8;
@@ -220,4 +232,12 @@ message ApiConfiguration {
 
   // Claude Code specific
   optional string claude_code_path = 77;
+
+  // Extension fields for Bedrock Api Keys
+  optional string aws_authentication = 78;
+  optional string aws_bedrock_api_key = 79;
+
+  // Moonshot
+  optional string moonshot_api_key = 80;
+  optional string moonshot_api_line = 81;
 }

proto/task.proto

@@ -1,11 +1,10 @@
 syntax = "proto3";
 
 package cline;
+import "common.proto";
 option java_package = "bot.cline.proto";
 option java_multiple_files = true;
 
-import "common.proto";
-
 service TaskService {
   // Cancels the currently running task
   rpc cancelTask(EmptyRequest) returns (Empty);
@@ -23,8 +22,6 @@ service TaskService {
   rpc exportTaskWithId(StringRequest) returns (Empty);
   // Toggles the favorite status of a task
   rpc toggleTaskFavorite(TaskFavoriteRequest) returns (Empty);
-  // Deletes all non-favorited tasks
-  rpc deleteNonFavoritedTasks(EmptyRequest) returns (DeleteNonFavoritedTasksResults);
   // Gets filtered task history
   rpc getTaskHistory(GetTaskHistoryRequest) returns (TaskHistoryArray);
   // Sends a response to a previous ask operation
@@ -35,6 +32,8 @@ service TaskService {
   rpc taskCompletionViewChanges(Int64Request) returns (Empty);
   // Executes a quick win task with command and title
   rpc executeQuickWin(ExecuteQuickWinRequest) returns (Empty);
+  // Deletes all task history
+  rpc deleteAllTaskHistory(EmptyRequest) returns (DeleteAllTaskHistoryCount);
 }
 
 // Request message for creating a new task
@@ -66,12 +65,6 @@ message TaskResponse {
   int32 cache_reads = 10;
 }
 
-// Results returned when deleting non-favorited tasks
-message DeleteNonFavoritedTasksResults {
-  int32 tasks_preserved = 1;
-  int32 tasks_deleted = 2;
-}
-
 // Request for getting task history with filtering
 message GetTaskHistoryRequest {
   Metadata metadata = 1;
@@ -116,3 +109,8 @@ message ExecuteQuickWinRequest {
   string command = 2;
   string title = 3;
 }
+
+// Results returned when deleting all task history
+message DeleteAllTaskHistoryCount {
+  int32 tasks_deleted = 1;
+}

proto/ui.proto

@@ -1,11 +1,10 @@
 syntax = "proto3";
 
 package cline;
+import "common.proto";
 option java_package = "bot.cline.proto";
 option java_multiple_files = true;
 
-import "common.proto";
-
 // Enum for webview provider types
 enum WebviewProviderType {
   SIDEBAR = 0;
@@ -262,4 +261,10 @@ service UiService {
   
   // Subscribe to webview visibility change events
   rpc subscribeToDidBecomeVisible(EmptyRequest) returns (stream Empty);
+
+  // Returns the HTML for the webview index page. This is only used by external clients, not by the vscode webview.
+  rpc getWebviewHtml(EmptyRequest) returns (String);
+  
+  // Opens a URL in the default browser
+  rpc openUrl(StringRequest) returns (Empty);
 }

proto/web.proto

@@ -1,11 +1,10 @@
 syntax = "proto3";
 
 package cline;
+import "common.proto";
 option java_package = "bot.cline.proto";
 option java_multiple_files = true;
 
-import "common.proto";
-
 service WebService {
   rpc checkIsImageUrl(StringRequest) returns (IsImageUrl);
   rpc fetchOpenGraphData(StringRequest) returns (OpenGraphData);

scripts/build-proto-config.mjs

@@ -0,0 +1,32 @@
+// Configuration file for protocol buffer build scripts
+// Contains service name mappings used by both build-proto.js and build-go-proto.js
+
+// List of gRPC services
+// To add a new service, simply add it to this map and run the build scripts
+// The service handler will be automatically discovered and used by grpc-handler.ts
+export const serviceNameMap = {
+	account: "cline.AccountService",
+	browser: "cline.BrowserService",
+	checkpoints: "cline.CheckpointsService",
+	file: "cline.FileService",
+	mcp: "cline.McpService",
+	state: "cline.StateService",
+	task: "cline.TaskService",
+	web: "cline.WebService",
+	models: "cline.ModelsService",
+	slash: "cline.SlashService",
+	ui: "cline.UiService",
+	// Add new services here - no other code changes needed!
+}
+
+// List of host gRPC services (IDE API bridge)
+// These services are implemented in the IDE extension and called by the standalone Cline Core
+export const hostServiceNameMap = {
+	uri: "host.UriService",
+	watch: "host.WatchService",
+	workspace: "host.WorkspaceService",
+	env: "host.EnvService",
+	window: "host.WindowService",
+	diff: "host.DiffService",
+	// Add new host services here
+}

scripts/build-proto.mjs

@@ -9,20 +9,22 @@ import chalk from "chalk"
 import os from "os"
 
 import { createRequire } from "module"
+import { serviceNameMap, hostServiceNameMap } from "./build-proto-config.mjs"
+
 const require = createRequire(import.meta.url)
 const PROTOC = path.join(require.resolve("grpc-tools"), "../bin/protoc")
 
-const SCRIPT_DIR = path.dirname(fileURLToPath(import.meta.url))
-const ROOT_DIR = path.resolve(SCRIPT_DIR, "..")
+const SCRIPT_NAME = path.relative(process.cwd(), fileURLToPath(import.meta.url))
 
-const TS_OUT_DIR = path.join(ROOT_DIR, "src/shared/proto")
-const GRPC_JS_OUT_DIR = path.join(ROOT_DIR, "src/generated/grpc-js")
-const NICE_JS_OUT_DIR = path.join(ROOT_DIR, "src/generated/nice-grpc")
-const DESCRIPTOR_OUT_DIR = path.join(ROOT_DIR, "dist-standalone/proto")
+const PROTO_DIR = path.resolve("proto")
+const TS_OUT_DIR = path.resolve("src/shared/proto")
+const GRPC_JS_OUT_DIR = path.resolve("src/generated/grpc-js")
+const NICE_JS_OUT_DIR = path.resolve("src/generated/nice-grpc")
+const DESCRIPTOR_OUT_DIR = path.resolve("dist-standalone/proto")
 
 const isWindows = process.platform === "win32"
 const TS_PROTO_PLUGIN = isWindows
-	? path.join(ROOT_DIR, "node_modules", ".bin", "protoc-gen-ts_proto.cmd") // Use the .bin directory path for Windows
+	? path.resolve("node_modules/.bin/protoc-gen-ts_proto.cmd") // Use the .bin directory path for Windows
 	: require.resolve("ts-proto/protoc-gen-ts_proto")
 
 const TS_PROTO_OPTIONS = [
@@ -34,33 +36,8 @@ const TS_PROTO_OPTIONS = [
 	"useDate=false", // Timestamp fields will not be automatically converted to Date.
 ]
 
-// List of gRPC services
-// To add a new service, simply add it to this map and run this script
-// The service handler will be automatically discovered and used by grpc-handler.ts
-const serviceNameMap = {
-	account: "cline.AccountService",
-	browser: "cline.BrowserService",
-	checkpoints: "cline.CheckpointsService",
-	file: "cline.FileService",
-	mcp: "cline.McpService",
-	state: "cline.StateService",
-	task: "cline.TaskService",
-	web: "cline.WebService",
-	models: "cline.ModelsService",
-	slash: "cline.SlashService",
-	ui: "cline.UiService",
-	// Add new services here - no other code changes needed!
-}
-const serviceDirs = Object.keys(serviceNameMap).map((serviceKey) => path.join(ROOT_DIR, "src/core/controller", serviceKey))
-
-// List of host gRPC services (IDE API bridge)
-// These services are implemented in the IDE extension and called by the standalone Cline Core
-const hostServiceNameMap = {
-	uri: "host.UriService",
-	watch: "host.WatchService",
-	// Add new host services here
-}
-const hostServiceDirs = Object.keys(hostServiceNameMap).map((serviceKey) => path.join(ROOT_DIR, "src/hosts/vscode", serviceKey))
+// Service directories derived from imported serviceNameMap
+const serviceDirs = Object.keys(serviceNameMap).map((serviceKey) => path.join("src/core/controller", serviceKey))
 
 async function main() {
 	console.log(chalk.bold.blue("Starting Protocol Buffer code generation..."))
@@ -79,8 +56,8 @@ async function main() {
 	await ensureProtoFilesExist()
 
 	// Process all proto files
-	const protoFiles = await globby("**/*.proto", { cwd: SCRIPT_DIR, realpath: true })
-	console.log(chalk.cyan(`Processing ${protoFiles.length} proto files from`), SCRIPT_DIR)
+	const protoFiles = await globby("**/*.proto", { cwd: PROTO_DIR, realpath: true })
+	console.log(chalk.cyan(`Processing ${protoFiles.length} proto files from`), PROTO_DIR)
 
 	tsProtoc(TS_OUT_DIR, protoFiles, TS_PROTO_OPTIONS)
 	// grpc-js is used to generate service impls for the ProtoBus service.
@@ -91,7 +68,7 @@ async function main() {
 	const descriptorFile = path.join(DESCRIPTOR_OUT_DIR, "descriptor_set.pb")
 	const descriptorProtocCommand = [
 		PROTOC,
-		`--proto_path="${SCRIPT_DIR}"`,
+		`--proto_path="${PROTO_DIR}"`,
 		`--descriptor_set_out="${descriptorFile}"`,
 		"--include_imports",
 		...protoFiles,
@@ -107,11 +84,12 @@ async function main() {
 	log_verbose(chalk.green("Protocol Buffer code generation completed successfully."))
 	log_verbose(chalk.green(`TypeScript files generated in: ${TS_OUT_DIR}`))
 
-	await generateMethodRegistrations()
-	await generateHostMethodRegistrations()
-	await generateServiceConfig()
-	await generateHostServiceConfig()
-	await generateGrpcClientConfig()
+	await generateProtoBusServiceConfig()
+	await generateProtoBusMethodRegistrations()
+	await generateProtoBusGrpcClientConfig()
+
+	await generateHostBridgeServiceConfig()
+	await generateHostBridgeMethodRegistrations()
 
 	console.log(chalk.bold.blue("Finished Protocol Buffer code generation."))
 }
@@ -120,7 +98,7 @@ async function tsProtoc(outDir, protoFiles, protoOptions) {
 	// Build the protoc command with proper path handling for cross-platform
 	const command = [
 		PROTOC,
-		`--proto_path="${SCRIPT_DIR}"`,
+		`--proto_path="${PROTO_DIR}"`,
 		`--plugin=protoc-gen-ts_proto="${TS_PROTO_PLUGIN}"`,
 		`--ts_proto_out="${outDir}"`,
 		`--ts_proto_opt=${protoOptions.join(",")} `,
@@ -140,7 +118,7 @@ async function tsProtoc(outDir, protoFiles, protoOptions) {
  * Generate a gRPC client configuration file for the webview
  * This eliminates the need for manual imports and client creation in grpc-client.ts
  */
-async function generateGrpcClientConfig() {
+async function generateProtoBusGrpcClientConfig() {
 	log_verbose(chalk.cyan("Generating gRPC client configuration..."))
 
 	const serviceImports = []
@@ -165,7 +143,7 @@ async function generateGrpcClientConfig() {
 
 	// Generate the file content
 	const content = `// AUTO-GENERATED FILE - DO NOT MODIFY DIRECTLY
-// Generated by proto/build-proto.js
+// Generated by ${SCRIPT_NAME}
 
 import { createGrpcClient } from "./grpc-client-base"
 ${serviceImports.join("\n")}
@@ -176,7 +154,7 @@ export {
 	${serviceExports.join(",\n\t")}
 }`
 
-	const filePath = path.join(ROOT_DIR, "webview-ui/src/services/grpc-client.ts")
+	const filePath = path.resolve("webview-ui/src/services/grpc-client.ts")
 	await writeFileWithMkdirs(filePath, content)
 	log_verbose(chalk.green(`Generated gRPC client at ${filePath}`))
 }
@@ -239,12 +217,12 @@ async function parseProtoForStreamingMethods(protoFiles, scriptDir) {
 	return streamingMethodsMap
 }
 
-async function generateMethodRegistrations() {
+async function generateProtoBusMethodRegistrations() {
 	log_verbose(chalk.cyan("Generating method registration files..."))
 
 	// Parse proto files for streaming methods
-	const protoFiles = await globby("*.proto", { cwd: SCRIPT_DIR })
-	const streamingMethodsMap = await parseProtoForStreamingMethods(protoFiles, SCRIPT_DIR)
+	const protoFiles = await globby("*.proto", { cwd: PROTO_DIR })
+	const streamingMethodsMap = await parseProtoForStreamingMethods(protoFiles, PROTO_DIR)
 
 	for (const serviceDir of serviceDirs) {
 		const serviceName = path.basename(serviceDir)
@@ -261,7 +239,7 @@ async function generateMethodRegistrations() {
 
 		// Create the methods.ts file with header
 		let methodsContent = `// AUTO-GENERATED FILE - DO NOT MODIFY DIRECTLY
-// Generated by proto/build-proto.js
+// Generated by ${SCRIPT_NAME}
 
 // Import all method implementations
 import { registerMethod } from "./index"\n`
@@ -310,7 +288,7 @@ export function registerAllMethods(): void {
 		// Generate index.ts file
 		const capitalizedServiceName = serviceName.charAt(0).toUpperCase() + serviceName.slice(1)
 		const indexContent = `// AUTO-GENERATED FILE - DO NOT MODIFY DIRECTLY
-// Generated by proto/build-proto.js
+// Generated by ${SCRIPT_NAME}
 
 import { createServiceRegistry, ServiceMethodHandler, StreamingMethodHandler } from "../grpc-service"
 import { StreamingResponseHandler } from "../grpc-handler"
@@ -345,7 +323,7 @@ registerAllMethods()`
  * Generate a service configuration file that maps service names to their handlers
  * This eliminates the need for manual switch/case statements in grpc-handler.ts
  */
-async function generateServiceConfig() {
+async function generateProtoBusServiceConfig() {
 	log_verbose(chalk.cyan("Generating service configuration file..."))
 
 	const serviceImports = []
@@ -365,7 +343,7 @@ async function generateServiceConfig() {
 	}
 
 	const content = `// AUTO-GENERATED FILE - DO NOT MODIFY DIRECTLY
-// Generated by proto/build-proto.js
+// Generated by ${SCRIPT_NAME}
 
 import { Controller } from "./index"
 import { StreamingResponseHandler } from "./grpc-handler"
@@ -385,7 +363,7 @@ export interface ServiceHandlerConfig {
 export const serviceHandlers: Record<string, ServiceHandlerConfig> = {${serviceConfigs.join(",")}
 };`
 
-	const configPath = path.join(ROOT_DIR, "src/core/controller/grpc-service-config.ts")
+	const configPath = path.resolve("src/core/controller/grpc-service-config.ts")
 	await writeFileWithMkdirs(configPath, content)
 	log_verbose(chalk.green(`Generated service configuration at ${configPath}`))
 }
@@ -398,7 +376,7 @@ async function ensureProtoFilesExist() {
 	log_verbose(chalk.cyan("Checking for missing proto files..."))
 
 	// Get existing proto files
-	const existingProtoFiles = await globby("*.proto", { cwd: SCRIPT_DIR })
+	const existingProtoFiles = await globby("*.proto", { cwd: PROTO_DIR })
 	const existingProtoServices = existingProtoFiles.map((file) => path.basename(file, ".proto"))
 
 	// Check each service in serviceNameMap
@@ -435,7 +413,7 @@ service ${serviceClassName} {
 `
 
 			// Write the template proto file
-			const protoFilePath = path.join(SCRIPT_DIR, `${serviceName}.proto`)
+			const protoFilePath = path.join(PROTO_DIR, `${serviceName}.proto`)
 			await fs.writeFile(protoFilePath, protoContent)
 			log_verbose(chalk.green(`Created template proto file at ${protoFilePath}`))
 		}
@@ -445,17 +423,22 @@ service ${serviceClassName} {
 /**
  * Generate method registration files for host services
  */
-async function generateHostMethodRegistrations() {
+async function generateHostBridgeMethodRegistrations() {
 	log_verbose(chalk.cyan("Generating host method registration files..."))
+	// Host service directories derived from imported hostServiceNameMap
+	const hostServiceDirs = Object.keys(hostServiceNameMap).map((serviceKey) =>
+		path.join("src/hosts/vscode/hostbridge", serviceKey),
+	)
 
 	// Parse proto files for streaming methods
-	const hostProtoFiles = await globby("*.proto", { cwd: path.join(SCRIPT_DIR, "host") })
-	const streamingMethodsMap = await parseProtoForStreamingMethods(hostProtoFiles, path.join(SCRIPT_DIR, "host"))
+	const hostProtoFiles = await globby("*.proto", { cwd: path.join(PROTO_DIR, "host") })
+	const streamingMethodsMap = await parseProtoForStreamingMethods(hostProtoFiles, path.join(PROTO_DIR, "host"))
 
 	for (const serviceDir of hostServiceDirs) {
 		const serviceName = path.basename(serviceDir)
 		const fullServiceName = hostServiceNameMap[serviceName]
 		const streamingMethods = streamingMethodsMap.get(fullServiceName) || []
+		const outputDir = path.join("src/generated/hosts/vscode/hostbridge", serviceName)
 
 		log_verbose(chalk.cyan(`Generating method registrations for host ${serviceName}...`))
 
@@ -467,7 +450,7 @@ async function generateHostMethodRegistrations() {
 
 		// Create the methods.ts file with header
 		let methodsContent = `// AUTO-GENERATED FILE - DO NOT MODIFY DIRECTLY
-// Generated by proto/build-proto.js
+// Generated ${SCRIPT_NAME}
 
 // Import all method implementations
 import { registerMethod } from "./index"\n`
@@ -475,7 +458,7 @@ import { registerMethod } from "./index"\n`
 		// Import implementations directly
 		for (const file of implementationFiles) {
 			const baseName = path.basename(file, ".ts")
-			methodsContent += `import { ${baseName} } from "./${baseName}"\n`
+			methodsContent += `import { ${baseName} } from "@hosts/vscode/hostbridge/${serviceName}/${baseName}"\n`
 		}
 
 		// Add streaming methods information
@@ -509,17 +492,17 @@ export function registerAllMethods(): void {
 		methodsContent += `}`
 
 		// Write the methods.ts file
-		const registryFile = path.join(serviceDir, "methods.ts")
+		const registryFile = path.join(outputDir, "methods.ts")
 		await writeFileWithMkdirs(registryFile, methodsContent)
 		log_verbose(chalk.green(`Generated ${registryFile}`))
 
 		// Generate index.ts file
 		const capitalizedServiceName = serviceName.charAt(0).toUpperCase() + serviceName.slice(1)
 		const indexContent = `// AUTO-GENERATED FILE - DO NOT MODIFY DIRECTLY
-// Generated by proto/build-proto.js
+// Generated by ${SCRIPT_NAME}
 
-import { createServiceRegistry, ServiceMethodHandler, StreamingMethodHandler } from "../host-grpc-service"
-import { StreamingResponseHandler } from "../host-grpc-handler"
+import { createServiceRegistry, ServiceMethodHandler, StreamingMethodHandler } from "@hosts/vscode/hostbridge-grpc-service"
+import { StreamingResponseHandler } from "@hosts/vscode/hostbridge-grpc-handler"
 import { registerAllMethods } from "./methods"
 
 // Create ${serviceName} service registry
@@ -539,7 +522,7 @@ export const isStreamingMethod = ${serviceName}Service.isStreamingMethod
 registerAllMethods()`
 
 		// Write the index.ts file
-		const indexFile = path.join(serviceDir, "index.ts")
+		const indexFile = path.join(outputDir, "index.ts")
 		await writeFileWithMkdirs(indexFile, indexContent)
 		log_verbose(chalk.green(`Generated ${indexFile}`))
 	}
@@ -550,7 +533,7 @@ registerAllMethods()`
 /**
  * Generate a service configuration file for host services
  */
-async function generateHostServiceConfig() {
+async function generateHostBridgeServiceConfig() {
 	log_verbose(chalk.cyan("Generating host service configuration file..."))
 
 	const serviceImports = []
@@ -560,7 +543,7 @@ async function generateHostServiceConfig() {
 	for (const [dirName, fullServiceName] of Object.entries(hostServiceNameMap)) {
 		const capitalizedName = dirName.charAt(0).toUpperCase() + dirName.slice(1)
 		serviceImports.push(
-			`import { handle${capitalizedName}ServiceRequest, handle${capitalizedName}ServiceStreamingRequest } from "./${dirName}/index"`,
+			`import { handle${capitalizedName}ServiceRequest, handle${capitalizedName}ServiceStreamingRequest } from "@generated/hosts/vscode/hostbridge/${dirName}/index"`,
 		)
 		serviceConfigs.push(`
   "${fullServiceName}": {
@@ -570,9 +553,9 @@ async function generateHostServiceConfig() {
 	}
 
 	const content = `// AUTO-GENERATED FILE - DO NOT MODIFY DIRECTLY
-// Generated by proto/build-proto.js
+// Generated by ${SCRIPT_NAME}
 
-import { StreamingResponseHandler } from "./host-grpc-handler"
+import { StreamingResponseHandler } from "@/hosts/vscode/hostbridge-grpc-handler"
 ${serviceImports.join("\n")}
 
 /**
@@ -589,7 +572,7 @@ export interface HostServiceHandlerConfig {
 export const hostServiceHandlers: Record<string, HostServiceHandlerConfig> = {${serviceConfigs.join(",")}
 };`
 
-	const filePath = path.join(ROOT_DIR, "src/hosts/vscode/host-grpc-service-config.ts")
+	const filePath = "src/generated/hosts/vscode/hostbridge-grpc-service-config.ts"
 	await writeFileWithMkdirs(filePath, content)
 	log_verbose(chalk.green(`Generated host service configuration at ${filePath}`))
 }
@@ -601,16 +584,33 @@ async function cleanup() {
 	for (const file of existingFiles) {
 		await fs.unlink(path.join(TS_OUT_DIR, file))
 	}
-	await rmdir(path.join(ROOT_DIR, "src/generated"))
+	await rmdir("src/generated")
 
 	// Clean up generated files that were moved.
-	await fs.rm(path.join(ROOT_DIR, "src/standalone/services/host-grpc-client.ts"), { force: true })
-	await rmdir(path.join(ROOT_DIR, "src/standalone/services"))
+	await fs.rm("src/standalone/services/host-grpc-client.ts", { force: true })
+	await rmdir("src/standalone/services")
+	await fs.rm("hosts/vscode", { force: true, recursive: true })
+	await rmdir("hosts")
 
-	await fs.rm(path.join(ROOT_DIR, "hosts/vscode"), { force: true, recursive: true })
-	await rmdir(path.join(ROOT_DIR, "hosts"))
-
-	await fs.rm(path.join(ROOT_DIR, "src/standalone/server-setup.ts"), { force: true })
+	await fs.rm("src/standalone/server-setup.ts", { force: true })
+	await fs.rm("src/hosts/vscode/host-grpc-service-config.ts", { force: true })
+	const oldhostbridgefiles = [
+		"src/hosts/vscode/workspace/methods.ts",
+		"src/hosts/vscode/workspace/index.ts",
+		"src/hosts/vscode/diff/methods.ts",
+		"src/hosts/vscode/diff/index.ts",
+		"src/hosts/vscode/env/methods.ts",
+		"src/hosts/vscode/env/index.ts",
+		"src/hosts/vscode/window/methods.ts",
+		"src/hosts/vscode/window/index.ts",
+		"src/hosts/vscode/watch/methods.ts",
+		"src/hosts/vscode/watch/index.ts",
+		"src/hosts/vscode/uri/methods.ts",
+		"src/hosts/vscode/uri/index.ts",
+	]
+	for (const file of oldhostbridgefiles) {
+		await fs.rm(file, { force: true })
+	}
 }
 
 /**
@@ -635,13 +635,6 @@ async function rmdir(path) {
 	}
 }
 
-function serviceNameWithoutPackage(fullServiceName) {
-	return fullServiceName.replace(/.*\./, "")
-}
-function lowercaseFirstChar(str) {
-	return str.charAt(0).toLowerCase() + str.slice(1)
-}
-
 // Check for Apple Silicon compatibility
 function checkAppleSiliconCompatibility() {
 	// Only run check on macOS

scripts/get-vscode-usages.sh

@@ -5,16 +5,31 @@ DIR=${1:-src/}
 DEST_DIR=dist-standalone
 SDK_DEST=$DEST_DIR/vscode-sdk-uses.txt
 CSS_DEST=$DEST_DIR/vscode-css-uses.txt
+TMP=/tmp/vscode-sdk-uses.txt.tmp
 mkdir -p $DEST_DIR
 
 {
 git grep -h 'vscode\.' $DIR |
 grep -Ev '//.*vscode' | # remove commented out code
+grep -v vscode.commands.executeCommand | # executeCommand is handled separately
+grep -Ev '"vscode' | # remove command strings that get included because they start with vscode
 sed 's|.*vscode\.|vscode.|'| # remove everything before vscode.
 sed 's/[^a-zA-Z0-9_.].*$//' | # remove everything after last identifier
 grep -E '\.[a-z][^.]+$' | # remove types (last part of identifier should be lowercase)
-sort | uniq > $SDK_DEST
+cat > $TMP
 }
+{
+  grep -rh vscode.commands.executeCommand $DIR |
+  perl -ne 'print if /["\x27"]/'  | # Remove occurrences where the command is not on the same line (line doesnt contain quote chars) :(
+  sed -n 's|.*\(vscode.commands.executeCommand[^,]*\).*|\1|p'| # Remove all params after the first one
+  sed 's|\(".*"\).*|\1)|'| # Close the parantheses
+  cat >> $TMP
+}
+
+# Count occurrences
+cat $TMP | sort | uniq -c | sort -n > $SDK_DEST
+rm $TMP
+
 echo Wrote uses of the vscode SDK to $(realpath $SDK_DEST)
 
 {