Compare commits

...
Author SHA1 Message Date
arafatkatze 9038a0330b Embracing the ffmpeg for opus support 2025-07-13 19:56:53 -07:00
arafatkatze 570728b5f8 locally running debug stuff 2025-07-13 17:42:22 -07:00
arafatkatze 944a4626e4 feat: Adding voice mode to Cline
- Add voice recording functionality
- Integrate with Cline Account Service for transcription
- Add language settings for voice input
- Implement duration tooltips and UI improvements
- Add protobuf definitions for voice service
- Fix various UI positioning and animation issues
2025-07-13 05:06:01 -07:00
56 changed files with 1723 additions and 509 deletions
+5
View File
@@ -0,0 +1,5 @@
---
"claude-dev": patch
---
Add voice dictation feature for Cline account users
+5
View File
@@ -0,0 +1,5 @@
---
"claude-dev": patch
---
Default the credits balance to dashes on the account page, making it clear that the balance is not zero
-8
View File
@@ -1,13 +1,5 @@
# Changelog
## [3.18.14]
- Fix bug where Cline account users logged in with invalid token would not be shown as logged out in webview presentation layer
## [3.18.13]
- Fix authentication issue where Cline accounts users would keep getting logged out or seeing 'Unexpected API response' errors
## [3.18.12]
- Fix flaky organization switching behavior in Cline provider that caused UI inconsistencies and double loading
+95
View File
@@ -0,0 +1,95 @@
---
title: Voice Recording
description: Record audio messages and have them transcribed using OpenAI Whisper
---
# Voice Recording
Cline supports voice recording functionality that allows you to record audio messages directly in the chat interface. Your voice is automatically transcribed using OpenAI's Whisper model.
## How It Works
1. **Enable voice recording** in General Settings (enabled by default)
2. **Click the microphone button** in the chat input area
3. **Speak your message** - the button will show a red recording indicator
4. **Click the stop button** when finished
5. **Wait for transcription** - you'll see "[Transcribing...]" in the input field
6. **Review and send** - the transcribed text appears in the input field
## Settings
### Enable/Disable Voice Recording
You can toggle voice recording on or off in the General Settings:
1. Open Cline settings (gear icon)
2. Go to "General Settings"
3. Toggle "Enable Voice Recording" checkbox
4. The microphone button will appear/disappear based on this setting
## Requirements
### OpenAI API Key
Voice transcription requires an OpenAI API key to use the Whisper model. The voice feature will automatically search for any configured OpenAI key in your settings, regardless of which provider you're using for chat.
You can configure an OpenAI key in any of these ways:
- As your main chat provider (OpenAI or OpenAI Native)
- Just having an OpenAI key saved in settings (even if using a different chat provider)
### Audio Recording Tools
Cline uses FFmpeg to capture your voice across all platforms:
- **macOS**: FFmpeg (via Homebrew: `brew install ffmpeg`)
- **Linux**: FFmpeg (via apt: `sudo apt-get install ffmpeg`)
- **Windows**: FFmpeg (via winget: `winget install Gyan.FFmpeg`)
If you don't have FFmpeg installed, Cline will automatically detect this and prompt you to install it with a single click.
## Technical Details
### Independent from Chat Provider
The voice transcription feature is completely independent from your chat provider selection. This means:
- You can use Claude, GPT-4, or any other model for chat
- Voice transcription will always use OpenAI's Whisper model
- As long as you have an OpenAI API key configured somewhere, voice will work
### Audio Format
- Records in WebM format with Opus codec for optimal compression
- Mono channel for optimal speech recognition
- 16kHz sample rate optimized for voice
- 32kbps bitrate for efficient file sizes
### Privacy & Security
- Audio is recorded locally on your machine
- Only the audio file is sent to OpenAI for transcription
- No audio is stored after transcription
- Temporary files are automatically cleaned up
## Troubleshooting
### "No OpenAI API key found"
Make sure you have configured an OpenAI API key in Cline's settings. You don't need to switch to OpenAI as your chat provider - just having the key saved is enough.
### "Failed to start recording"
If you see this error, it means that the audio recording tools are not installed on your system. Cline will prompt you to install them automatically. Follow the on-screen instructions to install the required tools.
### "Transcription failed"
Check that:
- Your OpenAI API key is valid
- You have sufficient OpenAI API credits
- Your internet connection is stable
## API Usage
Voice transcription uses the OpenAI Whisper API, which is billed separately from chat completions. Check OpenAI's pricing page for current rates.
+2 -17
View File
@@ -1,12 +1,12 @@
{
"name": "claude-dev",
"version": "3.18.14",
"version": "3.18.12",
"lockfileVersion": 2,
"requires": true,
"packages": {
"": {
"name": "claude-dev",
"version": "3.18.14",
"version": "3.18.12",
"license": "Apache-2.0",
"dependencies": {
"@anthropic-ai/sdk": "^0.37.0",
@@ -51,7 +51,6 @@
"image-size": "^2.0.2",
"isbinaryfile": "^5.0.2",
"jschardet": "^3.1.4",
"jwt-decode": "^4.0.0",
"mammoth": "^1.8.0",
"monaco-vscode-textmate-theme-converter": "^0.1.7",
"nice-grpc": "^2.1.12",
@@ -15786,15 +15785,6 @@
"safe-buffer": "^5.0.1"
}
},
"node_modules/jwt-decode": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/jwt-decode/-/jwt-decode-4.0.0.tgz",
"integrity": "sha512-+KJGIyHgkGuIq3IEBNftfhW/LfWhXUIY6OmyVWjliu5KH1y0fw7VQ8YndE2O4qZdMSd9SqbnC8GOcZEy0Om7sA==",
"license": "MIT",
"engines": {
"node": ">=18"
}
},
"node_modules/katex": {
"version": "0.16.22",
"resolved": "https://registry.npmjs.org/katex/-/katex-0.16.22.tgz",
@@ -35238,11 +35228,6 @@
"safe-buffer": "^5.0.1"
}
},
"jwt-decode": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/jwt-decode/-/jwt-decode-4.0.0.tgz",
"integrity": "sha512-+KJGIyHgkGuIq3IEBNftfhW/LfWhXUIY6OmyVWjliu5KH1y0fw7VQ8YndE2O4qZdMSd9SqbnC8GOcZEy0Om7sA=="
},
"katex": {
"version": "0.16.22",
"resolved": "https://registry.npmjs.org/katex/-/katex-0.16.22.tgz",
+1 -2
View File
@@ -2,7 +2,7 @@
"name": "claude-dev",
"displayName": "Cline",
"description": "Autonomous coding agent right in your IDE, capable of creating/editing files, running commands, using the browser, and more with your permission every step of the way.",
"version": "3.18.14",
"version": "3.18.12",
"icon": "assets/icons/icon.png",
"engines": {
"vscode": "^1.84.0"
@@ -449,7 +449,6 @@
"image-size": "^2.0.2",
"isbinaryfile": "^5.0.2",
"jschardet": "^3.1.4",
"jwt-decode": "^4.0.0",
"mammoth": "^1.8.0",
"monaco-vscode-textmate-theme-converter": "^0.1.7",
"nice-grpc": "^2.1.12",
+1 -1
View File
@@ -16,6 +16,7 @@ export const serviceNameMap = {
models: "cline.ModelsService",
slash: "cline.SlashService",
ui: "cline.UiService",
voice: "cline.VoiceService",
// Add new services here - no other code changes needed!
}
@@ -27,6 +28,5 @@ export const hostServiceNameMap = {
workspace: "host.WorkspaceService",
env: "host.EnvService",
window: "host.WindowService",
diff: "host.DiffService",
// Add new host services here
}
-25
View File
@@ -1,25 +0,0 @@
syntax = "proto3";
package host;
option java_package = "bot.cline.host.proto";
option java_multiple_files = true;
import "common.proto";
// Provides methods for diff views.
service DiffService {
// Open the diff view/editor.
rpc openDiff(OpenDiffRequest) returns (OpenDiffResponse);
}
message OpenDiffRequest {
optional cline.Metadata metadata = 1;
// The absolute path of the document being edited.
optional string path = 2;
// The new content for the file.
optional string content = 3;
}
message OpenDiffResponse {
// TODO(sfortune) the host needs to return a unique id for the diff editor.
}
+6
View File
@@ -58,6 +58,11 @@ message ChatSettings {
optional string open_ai_reasoning_effort = 3;
}
message DictationSettings {
bool voice_recording_enabled = 1;
string dictation_language = 2;
}
message ChatContent {
optional string message = 1;
repeated string images = 2;
@@ -114,6 +119,7 @@ message UpdateSettingsRequest {
optional bool mcp_responses_collapsed = 10;
optional bool mcp_rich_display_enabled = 11;
optional int64 terminal_output_line_limit = 12;
optional DictationSettings dictation_settings = 13;
}
// Complete API Configuration message
+57
View File
@@ -0,0 +1,57 @@
syntax = "proto3";
import "common.proto";
package cline;
option java_package = "bot.cline.proto";
option java_multiple_files = true;
service VoiceService {
rpc startRecording(StartRecordingRequest) returns (RecordingResult);
rpc stopRecording(StopRecordingRequest) returns (RecordedAudio);
rpc getRecordingStatus(GetRecordingStatusRequest) returns (RecordingStatus);
rpc transcribeAudio(TranscribeAudioRequest) returns (Transcription);
}
// Request messages
message StartRecordingRequest {
Metadata metadata = 1;
// Could add options later
}
message StopRecordingRequest {
Metadata metadata = 1;
}
message GetRecordingStatusRequest {
Metadata metadata = 1;
}
message TranscribeAudioRequest {
Metadata metadata = 1;
string audio_base64 = 2;
string language = 3; // optional language hint
}
// Plain, reusable response types
message RecordingResult {
bool success = 1;
string error = 2;
}
message RecordedAudio {
bool success = 1;
string audio_base64 = 2;
string error = 3;
}
message RecordingStatus {
bool is_recording = 1;
double duration_seconds = 2;
string error = 3;
}
message Transcription {
string text = 1;
string error = 2;
}
+2 -2
View File
@@ -27,7 +27,7 @@ export class ClineHandler implements ApiHandler {
private _authService: AuthService
private client: OpenAI | undefined
// TODO: replace this with a global API Host
private readonly _baseUrl = "https://api.cline.bot"
private readonly _baseUrl = "http://localhost:7777"
// private readonly _baseUrl = "https://core-api.staging.int.cline.bot"
// private readonly _baseUrl = "http://localhost:7777"
lastGenerationId?: string
@@ -182,7 +182,7 @@ export class ClineHandler implements ApiHandler {
}
} catch (error) {
if (error.code === "ERR_BAD_REQUEST" || error.status === 401) {
throw new Error("Unauthorized: Please sign in to Cline before trying again.") // match with webview-ui/src/components/chat/ChatRow.tsx
throw new Error("Unauthorized: Please sign in to Cline before trying again.")
} else if (error.code === "insufficient_credits" || error.status === 402) {
throw new Error(error.error ? JSON.stringify(error.error) : "Insufficient credits or unknown error.")
}
@@ -6,7 +6,7 @@ import * as path from "path"
import { FileContextTracker } from "./FileContextTracker"
import * as diskModule from "@core/storage/disk"
import type { TaskMetadata, FileMetadataEntry } from "./ContextTrackerTypes"
import type { DiffViewProviderCreator, WebviewProviderCreator } from "@/hosts/host-providers"
import type { WebviewProviderCreator } from "@/hosts/host-providers"
import * as hostProviders from "@hosts/host-providers"
import { vscodeHostBridgeClient } from "@/hosts/vscode/client/host-grpc-client"
@@ -53,11 +53,7 @@ describe("FileContextTracker", () => {
mockTaskMetadata = { files_in_context: [], model_usage: [] }
getTaskMetadataStub = sandbox.stub(diskModule, "getTaskMetadata").resolves(mockTaskMetadata)
saveTaskMetadataStub = sandbox.stub(diskModule, "saveTaskMetadata").resolves()
hostProviders.initializeHostProviders(
((_) => {}) as WebviewProviderCreator,
(() => {}) as DiffViewProviderCreator,
vscodeHostBridgeClient,
)
hostProviders.initializeHostProviders(((_) => {}) as WebviewProviderCreator, vscodeHostBridgeClient)
// Create tracker instance
taskId = "test-task-id"
+3 -1
View File
@@ -74,7 +74,7 @@ export class Controller {
)
this.accountService = ClineAccountService.getInstance()
this.authService = AuthService.getInstance(context)
this.authService.restoreRefreshTokenAndRetrieveAuthInfo()
this.authService.restoreAuthToken()
// Clean up legacy checkpoints
cleanupLegacyCheckpoints(this.context.globalStorageUri.fsPath, this.outputChannel).catch((error) => {
@@ -847,6 +847,7 @@ export class Controller {
welcomeViewCompleted,
mcpResponsesCollapsed,
terminalOutputLineLimit,
dictationSettings,
} = await getAllExtensionState(this.context)
// Get current mode using helper function
@@ -905,6 +906,7 @@ export class Controller {
welcomeViewCompleted: welcomeViewCompleted as boolean, // Can be undefined but is set to either true or false by the migration that runs on extension launch in extension.ts
mcpResponsesCollapsed,
terminalOutputLineLimit,
dictationSettings,
}
}
@@ -5,6 +5,7 @@ import { updateApiConfiguration } from "../../storage/state"
import { buildApiHandler } from "../../../api"
import { convertProtoApiConfigurationToApiConfiguration } from "../../../shared/proto-conversions/state/settings-conversion"
import { convertProtoChatSettingsToChatSettings } from "../../../shared/proto-conversions/state/chat-settings-conversion"
import { convertProtoDictationSettingsToDictationSettings } from "../../../shared/proto-conversions/state/dictation-settings-conversion"
import { TelemetrySetting } from "@/shared/TelemetrySetting"
/**
@@ -73,6 +74,12 @@ export async function updateSettings(controller: Controller, request: UpdateSett
}
}
// Update dictation settings
if (request.dictationSettings) {
const dictationSettings = convertProtoDictationSettingsToDictationSettings(request.dictationSettings)
await controller.context.globalState.update("dictationSettings", dictationSettings)
}
// Update terminal timeout setting
if (request.shellIntegrationTimeout !== undefined) {
await controller.context.globalState.update("shellIntegrationTimeout", Number(request.shellIntegrationTimeout))
@@ -0,0 +1,26 @@
import { RecordingStatus } from "@shared/proto/voice"
import { audioRecordingService } from "@/services/dictation/AudioRecordingService"
import { VoiceMethodHandler } from "./index"
/**
* Gets the current recording status
* @returns RecordingStatus with current status
*/
export const getRecordingStatus: VoiceMethodHandler = async (): Promise<RecordingStatus> => {
try {
const status = audioRecordingService.getRecordingStatus()
return RecordingStatus.create({
isRecording: status.isRecording,
durationSeconds: status.durationSeconds,
error: status.error || "",
})
} catch (error) {
console.error("Error getting recording status:", error)
return RecordingStatus.create({
isRecording: false,
durationSeconds: 0,
error: error instanceof Error ? error.message : "Unknown error occurred",
})
}
}
@@ -0,0 +1,41 @@
import { Controller } from ".."
import { StartRecordingRequest, RecordingResult } from "@shared/proto/voice"
import { audioRecordingService } from "@/services/dictation/AudioRecordingService"
import { telemetryService } from "@services/posthog/telemetry/TelemetryService"
import { VoiceMethodHandler } from "./index"
import * as vscode from "vscode"
/**
* Starts audio recording using the Extension Host
* @param controller The controller instance
* @param request StartRecordingRequest
* @returns RecordingResult with success status
*/
export const startRecording: VoiceMethodHandler = async (
controller: Controller,
_request: StartRecordingRequest,
): Promise<RecordingResult> => {
const taskId = controller.task?.taskId
try {
const result = await audioRecordingService.startRecording()
// Capture telemetry for recording start
if (result.success) {
telemetryService.captureVoiceRecordingStarted(taskId, process.platform)
}
return RecordingResult.create({
success: result.success,
error: result.error || "",
})
} catch (error) {
console.error("Error starting recording:", error)
const errorMessage = error instanceof Error ? error.message : "Unknown error occurred"
vscode.window.showErrorMessage(`Voice recording error: ${errorMessage}`)
return RecordingResult.create({
success: false,
error: errorMessage,
})
}
}
@@ -0,0 +1,46 @@
import { Controller } from ".."
import { StopRecordingRequest, RecordedAudio } from "@shared/proto/voice"
import { audioRecordingService } from "@/services/dictation/AudioRecordingService"
import { telemetryService } from "@services/posthog/telemetry/TelemetryService"
import { VoiceMethodHandler } from "./index"
/**
* Stops audio recording and returns the recorded audio
* @param controller The controller instance
* @param request StopRecordingRequest
* @returns RecordedAudio with audio data
*/
export const stopRecording: VoiceMethodHandler = async (
controller: Controller,
_request: StopRecordingRequest,
): Promise<RecordedAudio> => {
const taskId = controller.task?.taskId
const recordingStatus = audioRecordingService.getRecordingStatus()
const recordingDuration = recordingStatus.durationSeconds * 1000 // Convert to milliseconds
try {
const result = await audioRecordingService.stopRecording()
// Capture telemetry for recording stop
telemetryService.captureVoiceRecordingStopped(taskId, recordingDuration, result.success, process.platform)
// Calculate audio size if available
return RecordedAudio.create({
success: result.success,
audioBase64: result.audioBase64 || "",
error: result.error || "",
})
} catch (error) {
console.error("Error stopping recording:", error)
// Capture telemetry for recording failure
telemetryService.captureVoiceRecordingStopped(taskId, recordingDuration, false, process.platform)
return RecordedAudio.create({
success: false,
audioBase64: "",
error: error instanceof Error ? error.message : "Unknown error occurred",
})
}
}
@@ -0,0 +1,84 @@
import { Controller } from ".."
import { TranscribeAudioRequest, Transcription } from "@shared/proto/voice"
import { voiceTranscriptionService } from "@/services/dictation/VoiceTranscriptionService"
import { telemetryService } from "@services/posthog/telemetry/TelemetryService"
import { VoiceMethodHandler } from "./index"
import * as vscode from "vscode"
/**
* Transcribes audio using Cline transcription service
* @param controller The controller instance
* @param request TranscribeAudioRequest containing base64 audio data
* @returns Transcription with transcribed text or error
*/
export const transcribeAudio: VoiceMethodHandler = async (
controller: Controller,
request: TranscribeAudioRequest,
): Promise<Transcription> => {
const taskId = controller.task?.taskId
const startTime = Date.now()
// Calculate audio size from base64
const audioSizeBytes = Math.ceil((request.audioBase64.length * 3) / 4)
// Capture telemetry for transcription start
telemetryService.captureVoiceTranscriptionStarted(taskId, audioSizeBytes, request.language || "en")
try {
// Transcribe the audio
const result = await voiceTranscriptionService.transcribeAudio(request.audioBase64, request.language || undefined)
const durationMs = Date.now() - startTime
// Handle transcription result
if (result.error) {
// Determine error type for telemetry
let errorType = "api_error"
if (result.error.includes("Authentication failed")) {
errorType = "invalid_jwt_token"
} else if (result.error.includes("Insufficient credits")) {
errorType = "insufficient_credits"
} else if (result.error.includes("Invalid audio format")) {
errorType = "invalid_audio_format"
} else if (result.error.includes("Cannot connect")) {
errorType = "connection_error"
} else if (result.error.includes("Network error")) {
errorType = "network_error"
}
// Capture telemetry for transcription error
telemetryService.captureVoiceTranscriptionError(taskId, errorType, result.error, durationMs)
// Show error notification if transcription failed
if (result.error.includes("Authentication failed")) {
vscode.window.showErrorMessage("Authentication failed. ")
} else if (result.error.includes("Insufficient credits")) {
vscode.window.showWarningMessage("Insufficient credits for transcription service.")
} else if (result.error.includes("Cannot connect")) {
vscode.window.showErrorMessage("Cannot connect to transcription service. ")
} else {
vscode.window.showErrorMessage(`Voice transcription failed: ${result.error}`)
}
} else if (result.text) {
// Capture telemetry for successful transcription
telemetryService.captureVoiceTranscriptionCompleted(taskId, result.text.length, durationMs, request.language || "en")
}
// Return the response
return Transcription.create({
text: result.text || "",
error: result.error || "",
})
} catch (error) {
console.error("Error transcribing audio:", error)
const durationMs = Date.now() - startTime
const errorMessage = error instanceof Error ? error.message : "Unknown error occurred"
// Capture telemetry for unexpected error
telemetryService.captureVoiceTranscriptionError(taskId, "unexpected_error", errorMessage, durationMs)
return Transcription.create({
text: "",
error: errorMessage,
})
}
}
+1
View File
@@ -80,6 +80,7 @@ export type GlobalStateKey =
| "claudeCodePath"
// Settings around plan/act and ephemeral model configuration
| "chatSettings"
| "dictationSettings"
| "mode"
// Current active model configuration (per workspace)
| "apiProvider"
+5
View File
@@ -2,12 +2,14 @@ import * as vscode from "vscode"
import { DEFAULT_CHAT_SETTINGS } from "@shared/ChatSettings"
import { DEFAULT_BROWSER_SETTINGS } from "@shared/BrowserSettings"
import { DEFAULT_AUTO_APPROVAL_SETTINGS } from "@shared/AutoApprovalSettings"
import { DEFAULT_DICTATION_SETTINGS } from "@shared/DictationSettings"
import { GlobalStateKey, LocalStateKey, SecretKey } from "./state-keys"
import { ApiConfiguration, ApiProvider, BedrockModelId, ModelInfo } from "@shared/api"
import { HistoryItem } from "@shared/HistoryItem"
import { AutoApprovalSettings } from "@shared/AutoApprovalSettings"
import { BrowserSettings } from "@shared/BrowserSettings"
import { StoredChatSettings } from "@shared/ChatSettings"
import { DictationSettings } from "@shared/DictationSettings"
import { TelemetrySetting } from "@shared/TelemetrySetting"
import { UserInfo } from "@shared/UserInfo"
import { ClineRulesToggles } from "@shared/cline-rules"
@@ -293,6 +295,7 @@ export async function getAllExtensionState(context: vscode.ExtensionContext) {
previousModeAwsBedrockCustomModelBaseId,
previousModeSapAiCoreModelId,
sapAiCoreModelId,
dictationSettings,
] = await Promise.all([
getGlobalState(context, "chatSettings") as Promise<StoredChatSettings | undefined>,
getGlobalState(context, "mode") as Promise<"plan" | "act" | undefined>,
@@ -325,6 +328,7 @@ export async function getAllExtensionState(context: vscode.ExtensionContext) {
getGlobalState(context, "previousModeAwsBedrockCustomModelBaseId") as Promise<BedrockModelId | undefined>,
getGlobalState(context, "previousModeSapAiCoreModelId") as Promise<string | undefined>,
getGlobalState(context, "sapAiCoreModelId") as Promise<string | undefined>,
getGlobalState(context, "dictationSettings") as Promise<DictationSettings | undefined>,
])
const processingStart = performance.now()
@@ -474,6 +478,7 @@ export async function getAllExtensionState(context: vscode.ExtensionContext) {
terminalOutputLineLimit: terminalOutputLineLimit ?? 500,
defaultTerminalProfile: defaultTerminalProfile ?? "default",
globalWorkflowToggles: globalWorkflowToggles || {},
dictationSettings: dictationSettings || DEFAULT_DICTATION_SETTINGS,
}
}
+2 -3
View File
@@ -82,7 +82,6 @@ import { MessageStateHandler } from "./message-state"
import { TaskState } from "./TaskState"
import { ToolExecutor } from "./ToolExecutor"
import { formatErrorWithStatusCode, updateApiReqMsg } from "./utils"
import { createDiffViewProvider, getHostBridgeProvider } from "@/hosts/host-providers"
export const USE_EXPERIMENTAL_CLAUDE4_FEATURES = false
@@ -175,7 +174,7 @@ export class Task {
this.urlContentFetcher = new UrlContentFetcher(context)
this.browserSession = new BrowserSession(context, browserSettings)
this.contextManager = new ContextManager()
this.diffViewProvider = createDiffViewProvider()
this.diffViewProvider = new DiffViewProvider()
this.autoApprovalSettings = autoApprovalSettings
this.browserSettings = browserSettings
this.chatSettings = chatSettings
@@ -1724,7 +1723,7 @@ export class Task {
await this.messageStateHandler.updateClineMessage(lastApiReqStartedIndex, {
text: JSON.stringify({
...currentApiReqInfo, // Spread the modified info (with retryStatus removed)
// cancelReason: "retries_exhausted", // Indicate that automatic retries failed
cancelReason: "retries_exhausted", // Indicate that automatic retries failed
streamingFailedMessage: errorMessage,
} satisfies ClineApiReqInfo),
})
+2 -6
View File
@@ -37,7 +37,6 @@ import { VscodeWebviewProvider } from "./core/webview/VscodeWebviewProvider"
import { ExtensionContext } from "vscode"
import { AuthService } from "./services/auth/AuthService"
import { writeTextToClipboard, readTextFromClipboard } from "@/utils/env"
import { VscodeDiffViewProvider } from "./integrations/editor/VscodeDiffViewProvider"
import { getHostBridgeProvider } from "@hosts/host-providers"
import { ShowMessageRequest, ShowMessageType } from "./shared/proto/host/window"
/*
@@ -716,7 +715,7 @@ export async function activate(context: vscode.ExtensionContext) {
context.subscriptions.push(
context.secrets.onDidChange((event) => {
if (event.key === "clineAccountId") {
AuthService.getInstance(context)?.restoreRefreshTokenAndRetrieveAuthInfo()
AuthService.getInstance(context)?.restoreAuthToken()
}
}),
)
@@ -730,10 +729,7 @@ function maybeSetupHostProviders(context: ExtensionContext) {
const createWebview = function (type: WebviewProviderType) {
return new VscodeWebviewProvider(context, outputChannel, type)
}
const createDiffView = function () {
return new VscodeDiffViewProvider()
}
hostProviders.initializeHostProviders(createWebview, createDiffView, vscodeHostBridgeClient)
hostProviders.initializeHostProviders(createWebview, vscodeHostBridgeClient)
}
}
+11
View File
@@ -0,0 +1,11 @@
// AUTO-GENERATED FILE - DO NOT MODIFY DIRECTLY
// Generated by proto/build-proto.js
import { createChannel, createClientFactory, Client } from "nice-grpc"
import { Channel } from "@grpc/grpc-js"
const channel = createChannel("localhost:50051") // Replace with your server address
export function createGrpcClient<T extends {}>(definition: any): Client<T> {
return createClientFactory().create(definition, channel)
}
-2
View File
@@ -3,7 +3,6 @@ import {
WorkspaceServiceClientInterface,
EnvServiceClientInterface,
WindowServiceClientInterface,
DiffServiceClientInterface,
} from "@generated/hosts/host-bridge-client-types"
/**
@@ -14,7 +13,6 @@ export interface HostBridgeClientProvider {
workspaceClient: WorkspaceServiceClientInterface
envClient: EnvServiceClientInterface
windowClient: WindowServiceClientInterface
diffClient: DiffServiceClientInterface
}
/**
-13
View File
@@ -1,28 +1,22 @@
import { WebviewProvider } from "@core/webview"
import { HostBridgeClientProvider } from "./host-provider-types"
import { WebviewProviderType } from "@/shared/webview/types"
import { DiffViewProvider } from "@/integrations/editor/DiffViewProvider"
/**
* A function that creates WebviewProvider instances
*/
export type WebviewProviderCreator = (providerType: WebviewProviderType) => WebviewProvider
export type DiffViewProviderCreator = () => DiffViewProvider
let _webviewProviderCreator: WebviewProviderCreator | undefined
let _diffViewProviderCreator: DiffViewProviderCreator | undefined
let _hostBridgeProvider: HostBridgeClientProvider | undefined
export var isSetup: boolean = false
export function initializeHostProviders(
webviewProviderCreator: WebviewProviderCreator,
diffViewProviderCreator: DiffViewProviderCreator,
hostBridgeProvider: HostBridgeClientProvider,
) {
_webviewProviderCreator = webviewProviderCreator
_diffViewProviderCreator = diffViewProviderCreator
_hostBridgeProvider = hostBridgeProvider
isSetup = true
}
@@ -34,13 +28,6 @@ export function createWebviewProvider(providerType: WebviewProviderType): Webvie
return _webviewProviderCreator(providerType)
}
export function createDiffViewProvider(): DiffViewProvider {
if (!_diffViewProviderCreator) {
throw Error("Host providers not initialized")
}
return _diffViewProviderCreator()
}
export function getHostBridgeProvider(): HostBridgeClientProvider {
if (!_hostBridgeProvider) {
throw Error("Host providers not initialized")
@@ -7,5 +7,4 @@ export const vscodeHostBridgeClient: HostBridgeClientProvider = {
workspaceClient: createGrpcClient(host.WorkspaceServiceDefinition),
envClient: createGrpcClient(host.EnvServiceDefinition),
windowClient: createGrpcClient(host.WindowServiceDefinition),
diffClient: createGrpcClient(host.DiffServiceDefinition),
}
-5
View File
@@ -1,5 +0,0 @@
import { OpenDiffRequest, OpenDiffResponse } from "@/shared/proto/index.host"
export async function openDiff(_request: OpenDiffRequest): Promise<OpenDiffResponse> {
throw new Error("diffService.openDiff is not supported. Use the VscodeDiffViewProvider.")
}
+94 -27
View File
@@ -10,30 +10,29 @@ import { diagnosticsToProblemsString, getNewDiagnostics } from "../diagnostics"
import { detectEncoding } from "../misc/extract-text"
import * as iconv from "iconv-lite"
import { getHostBridgeProvider } from "@/hosts/host-providers"
import { ShowTextDocumentRequest, ShowTextDocumentOptions } from "@/shared/proto/host/window"
import { ShowTextDocumentRequest, ShowTextDocumentOptions, TextEditorInfo } from "@/shared/proto/host/window"
export const DIFF_VIEW_URI_SCHEME = "cline-diff"
export abstract class DiffViewProvider {
export class DiffViewProvider {
editType?: "create" | "modify"
isEditing = false
originalContent: string | undefined
private createdDirs: string[] = []
protected documentWasOpen = false
protected relPath?: string
protected absolutePath?: string
protected fileEncoding: string = "utf8"
private streamedLines: string[] = []
private documentWasOpen = false
private relPath?: string
private absolutePath?: string
private newContent?: string
protected activeDiffEditor?: vscode.TextEditor
protected fadedOverlayController?: DecorationController
protected activeLineController?: DecorationController
protected preDiagnostics: [vscode.Uri, vscode.Diagnostic[]][] = []
private activeDiffEditor?: vscode.TextEditor
private fadedOverlayController?: DecorationController
private activeLineController?: DecorationController
private streamedLines: string[] = []
private preDiagnostics: [vscode.Uri, vscode.Diagnostic[]][] = []
private fileEncoding: string = "utf8"
constructor() {}
public async open(relPath: string): Promise<void> {
async open(relPath: string): Promise<void> {
this.isEditing = true
this.relPath = relPath
this.absolutePath = path.resolve(await getCwd(), relPath)
@@ -66,20 +65,6 @@ export abstract class DiffViewProvider {
this.streamedLines = []
}
/**
* Opens a diff editor or viewer for the current file.
*
* This abstract method must be implemented by subclasses to create and display
* a diff editor or viewer that shows the difference between the original and
* modified content.
*
* Called automatically by the `open` method after ensuring the file exists and
* creating any necessary directories.
*
* @returns A promise that resolves when the diff editor is open and ready
*/
protected abstract openDiffEditor(): Promise<void>
async update(
accumulatedContent: string,
isFinal: boolean,
@@ -342,6 +327,88 @@ export abstract class DiffViewProvider {
}
}
private async openDiffEditor(): Promise<void> {
if (!this.absolutePath) {
throw new Error("No file path set")
}
// get diagnostics before editing the file, we'll compare to diagnostics after editing to see if cline needs to fix anything
this.preDiagnostics = vscode.languages.getDiagnostics()
// if the file was already open, close it (must happen after showing the diff view since if it's the only tab the column will close)
this.documentWasOpen = false
// close the tab if it's open (it's already been saved)
const tabs = vscode.window.tabGroups.all
.map((tg) => tg.tabs)
.flat()
.filter((tab) => tab.input instanceof vscode.TabInputText && arePathsEqual(tab.input.uri.fsPath, this.absolutePath))
for (const tab of tabs) {
if (!tab.isDirty) {
await vscode.window.tabGroups.close(tab)
}
this.documentWasOpen = true
}
const uri = vscode.Uri.file(this.absolutePath)
// If this diff editor is already open (ie if a previous write file was interrupted) then we should activate that instead of opening a new diff
const diffTab = vscode.window.tabGroups.all
.flatMap((group) => group.tabs)
.find(
(tab) =>
tab.input instanceof vscode.TabInputTextDiff &&
tab.input?.original?.scheme === DIFF_VIEW_URI_SCHEME &&
arePathsEqual(tab.input.modified.fsPath, uri.fsPath),
)
if (diffTab && diffTab.input instanceof vscode.TabInputTextDiff) {
const editorInfo = await getHostBridgeProvider().windowClient.showTextDocument(
ShowTextDocumentRequest.create({
path: diffTab.input.modified.fsPath,
options: ShowTextDocumentOptions.create({
preserveFocus: true,
}),
}),
)
// Find the editor that matches the returned path
const editor = vscode.window.visibleTextEditors.find((e) => e.document.uri.fsPath === editorInfo.documentPath)
if (!editor) {
throw new Error("Failed to find opened text editor")
}
this.activeDiffEditor = editor
} else {
// Open new diff editor
this.activeDiffEditor = await new Promise<vscode.TextEditor>((resolve, reject) => {
const fileName = path.basename(uri.fsPath)
const fileExists = this.editType === "modify"
const disposable = vscode.window.onDidChangeActiveTextEditor((editor) => {
if (editor && arePathsEqual(editor.document.uri.fsPath, uri.fsPath)) {
disposable.dispose()
resolve(editor)
}
})
vscode.commands.executeCommand(
"vscode.diff",
vscode.Uri.parse(`${DIFF_VIEW_URI_SCHEME}:${fileName}`).with({
query: Buffer.from(this.originalContent ?? "").toString("base64"),
}),
uri,
`${fileName}: ${fileExists ? "Original ↔ Cline's Changes" : "New File"} (Editable)`,
{
preserveFocus: true,
},
)
// This may happen on very slow machines ie project idx
setTimeout(() => {
disposable.dispose()
reject(new Error("Failed to open diff editor, please try again..."))
}, 10_000)
})
}
this.fadedOverlayController = new DecorationController("fadedOverlay", this.activeDiffEditor)
this.activeLineController = new DecorationController("activeLine", this.activeDiffEditor)
// Apply faded overlay to all lines initially
this.fadedOverlayController.addLines(0, this.activeDiffEditor.document.lineCount)
}
private scrollEditorToLine(line: number) {
if (this.activeDiffEditor) {
const scrollLine = line + 4
@@ -1,80 +0,0 @@
import { arePathsEqual } from "@/utils/path"
import * as path from "path"
import * as vscode from "vscode"
import { DecorationController } from "./DecorationController"
import { DIFF_VIEW_URI_SCHEME, DiffViewProvider } from "./DiffViewProvider"
export class VscodeDiffViewProvider extends DiffViewProvider {
override async openDiffEditor(): Promise<void> {
if (!this.absolutePath) {
throw new Error("No file path set")
}
// get diagnostics before editing the file, we'll compare to diagnostics after editing to see if cline needs to fix anything
this.preDiagnostics = vscode.languages.getDiagnostics()
// if the file was already open, close it (must happen after showing the diff view since if it's the only tab the column will close)
this.documentWasOpen = false
// close the tab if it's open (it's already been saved)
const tabs = vscode.window.tabGroups.all
.map((tg) => tg.tabs)
.flat()
.filter((tab) => tab.input instanceof vscode.TabInputText && arePathsEqual(tab.input.uri.fsPath, this.absolutePath))
for (const tab of tabs) {
if (!tab.isDirty) {
await vscode.window.tabGroups.close(tab)
}
this.documentWasOpen = true
}
const uri = vscode.Uri.file(this.absolutePath)
// If this diff editor is already open (ie if a previous write file was interrupted) then we should activate that instead of opening a new diff
const diffTab = vscode.window.tabGroups.all
.flatMap((group) => group.tabs)
.find(
(tab) =>
tab.input instanceof vscode.TabInputTextDiff &&
tab.input?.original?.scheme === DIFF_VIEW_URI_SCHEME &&
arePathsEqual(tab.input.modified.fsPath, uri.fsPath),
)
if (diffTab && diffTab.input instanceof vscode.TabInputTextDiff) {
// Use already open diff editor.
this.activeDiffEditor = await vscode.window.showTextDocument(diffTab.input.modified, {
preserveFocus: true,
})
} else {
// Open new diff editor.
this.activeDiffEditor = await new Promise<vscode.TextEditor>((resolve, reject) => {
const fileName = path.basename(uri.fsPath)
const fileExists = this.editType === "modify"
const disposable = vscode.window.onDidChangeActiveTextEditor((editor) => {
if (editor && arePathsEqual(editor.document.uri.fsPath, uri.fsPath)) {
disposable.dispose()
resolve(editor)
}
})
vscode.commands.executeCommand(
"vscode.diff",
vscode.Uri.parse(`${DIFF_VIEW_URI_SCHEME}:${fileName}`).with({
query: Buffer.from(this.originalContent ?? "").toString("base64"),
}),
uri,
`${fileName}: ${fileExists ? "Original ↔ Cline's Changes" : "New File"} (Editable)`,
{
preserveFocus: true,
},
)
// This may happen on very slow machines ie project idx
setTimeout(() => {
disposable.dispose()
reject(new Error("Failed to open diff editor, please try again..."))
}, 10_000)
})
}
this.fadedOverlayController = new DecorationController("fadedOverlay", this.activeDiffEditor)
this.activeLineController = new DecorationController("activeLine", this.activeDiffEditor)
// Apply faded overlay to all lines initially
this.fadedOverlayController.addLines(0, this.activeDiffEditor.document.lineCount)
}
}
+21 -3
View File
@@ -13,7 +13,7 @@ export class ClineAccountService {
private static instance: ClineAccountService
private _authService: AuthService
// TODO: replace this with a global API Host
private readonly _baseUrl = "https://api.cline.bot"
private readonly _baseUrl = "http://localhost:7777"
// private readonly _baseUrl = "https://core-api.staging.int.cline.bot"
// private readonly _baseUrl = "http://localhost:7777"
@@ -274,8 +274,26 @@ export class ClineAccountService {
console.error("Error switching account:", error)
throw error
} finally {
// After user switches account, we will force a refresh of the id token by calling this function that restores the refresh token and retrieves new auth info
await this._authService.restoreRefreshTokenAndRetrieveAuthInfo()
// Request a new authentication token
await this._authService.refreshAuth()
}
}
/**
* Transcribes audio using the Cline transcription service
* @param audioBase64 - Base64 encoded audio data
* @param language - Optional language hint for transcription
* @returns Promise with transcribed text or error
*/
async transcribeAudio(audioBase64: string, language?: string): Promise<{ text: string }> {
const response = await this.authenticatedRequest<{ text: string }>(`/api/v1/chat/transcriptions`, {
method: "POST",
data: {
audioData: audioBase64,
language: language || "en",
},
})
return response
}
}
+69 -71
View File
@@ -1,13 +1,13 @@
import vscode from "vscode"
import crypto from "crypto"
import { EmptyRequest, String } from "../../shared/proto/common"
import { AuthState, UserInfo } from "../../shared/proto/account"
import { AuthState } from "../../shared/proto/account"
import { StreamingResponseHandler, getRequestRegistry } from "@/core/controller/grpc-handler"
import { FirebaseAuthProvider } from "./providers/FirebaseAuthProvider"
import { Controller } from "@/core/controller"
import { storeSecret } from "@/core/storage/state"
const DefaultClineAccountURI = "https://app.cline.bot/auth"
const DefaultClineAccountURI = "http://localhost:3000/auth"
// const DefaultClineAccountURI = "https://staging-app.cline.bot/auth"
// const DefaultClineAccountURI = "http://localhost:3000/auth"
let authProviders: any[] = []
@@ -22,35 +22,14 @@ const availableAuthProviders = {
// Add other providers here as needed
}
export interface ClineAuthInfo {
idToken: string
userInfo: ClineAccountUserInfo
}
export interface ClineAccountUserInfo {
createdAt: string
displayName: string
email: string
id: string
organizations: ClineAccountOrganization[]
}
export interface ClineAccountOrganization {
active: boolean
memberId: string
name: string
organizationId: string
roles: string[]
}
// TODO: Add logic to handle multiple webviews getting auth updates.
export class AuthService {
private static instance: AuthService | null = null
private _config: ServiceConfig
private _authenticated: boolean = false
private _clineAuthInfo: ClineAuthInfo | null = null
private _provider: { provider: FirebaseAuthProvider } | null = null
private _user: any = null
private _provider: any = null
private readonly _authNonce = crypto.randomBytes(32).toString("hex")
private _activeAuthStatusUpdateSubscriptions = new Set<[Controller, StreamingResponseHandler]>()
private _context: vscode.ExtensionContext
@@ -72,14 +51,14 @@ export class AuthService {
const authProvidersConfigs = [
{
name: "firebase",
config: {
apiKey: "AIzaSyC5rx59Xt8UgwdU3PCfzUF7vCwmp9-K2vk",
authDomain: "cline-prod.firebaseapp.com",
projectId: "cline-prod",
storageBucket: "cline-prod.firebasestorage.app",
messagingSenderId: "941048379330",
appId: "1:941048379330:web:45058eedeefc5cdfcc485b",
},
// config: {
// apiKey: "AIzaSyC5rx59Xt8UgwdU3PCfzUF7vCwmp9-K2vk",
// authDomain: "cline-prod.firebaseapp.com",
// projectId: "cline-prod",
// storageBucket: "cline-prod.firebasestorage.app",
// messagingSenderId: "941048379330",
// appId: "1:941048379330:web:45058eedeefc5cdfcc485b",
// },
// Uncomment for staging environment
// config: {
// apiKey: "AIzaSyASSwkwX1kSO8vddjZkE5N19QU9cVQ0CIk",
@@ -90,14 +69,15 @@ export class AuthService {
// appId: "1:853479478430:web:2de0dba1c63c3262d4578f",
// },
// Uncomment for local development environment
// config: {
// apiKey: "AIzaSyASSwkwX1kSO8vddjZkE5N19QU9cVQ0CIk",
// authDomain: "cline-staging.firebaseapp.com",
// projectId: "cline-staging",
// storageBucket: "cline-staging.firebasestorage.app",
// messagingSenderId: "853479478430",
// appId: "1:853479478430:web:2de0dba1c63c3262d4578f",
// },
config: {
apiKey: "AIzaSyD8wtkd1I-EICuAg6xgAQpRdwYTvwxZG2w",
authDomain: "cline-preview.firebaseapp.com",
projectId: "cline-preview",
storageBucket: "cline-preview.firebasestorage.app",
messagingSenderId: "654681443338",
appId: "1:654681443338:web:93bfe710626a573d9123f3",
measurementId: "G-QL8CKK5TNJ",
},
// config: {
// apiKey: "AIzaSyD8wtkd1I-EICuAg6xgAQpRdwYTvwxZG2w",
// authDomain: "cline-preview.firebaseapp.com",
@@ -163,19 +143,13 @@ export class AuthService {
}
async getAuthToken(): Promise<string | null> {
if (!this._clineAuthInfo) {
if (!this._user) {
return null
}
const idToken = this._clineAuthInfo.idToken
const shouldRefreshIdToken = await this._provider?.provider.shouldRefreshIdToken(idToken)
if (shouldRefreshIdToken) {
// Retrieves the stored id token and refreshes it, then updates this._clineAuthInfo
await this.restoreRefreshTokenAndRetrieveAuthInfo()
if (!this._clineAuthInfo) {
return null
}
}
return this._clineAuthInfo.idToken
// TODO: This may need to be dependant on the auth provider
// Return the ID token from the user object
return this._provider.provider.getAuthToken(this._user)
}
private _setProvider(providerName: string): void {
@@ -188,17 +162,9 @@ export class AuthService {
}
getInfo(): AuthState {
// TODO: this logic should be cleaner, but this will determine the authentication state for the webview -- if a user object is returned then the webview assumes authenticated, otherwise it assumes logged out (we previously returned a UserInfo object with empty fields, and this represented a broken logged in state)
let user: any = null
if (this._clineAuthInfo && this._authenticated) {
const userInfo = this._clineAuthInfo.userInfo
user = UserInfo.create({
// TODO: create proto for new user info type
uid: userInfo?.id,
displayName: userInfo?.displayName,
email: userInfo?.email,
photoUrl: undefined,
})
let user = null
if (this._user && this._authenticated) {
user = this._provider.provider.convertUserData(this._user)
}
return AuthState.create({
@@ -235,7 +201,8 @@ export class AuthService {
}
try {
this._clineAuthInfo = null
await this._provider.provider.signOut()
this._user = null
this._authenticated = false
this.sendAuthStatusUpdate()
} catch (error) {
@@ -250,11 +217,12 @@ export class AuthService {
}
try {
this._clineAuthInfo = await this._provider.provider.signIn(this._context, token, provider)
this._user = await this._provider.provider.signIn(this._context, token, provider)
this._authenticated = true
await this.sendAuthStatusUpdate()
// return this._clineAuthInfo
this.setupAutoRefreshAuth()
return this._user
} catch (error) {
console.error("Error signing in with custom token:", error)
throw error
@@ -273,29 +241,59 @@ export class AuthService {
* Restores the authentication token from the extension's storage.
* This is typically called when the extension is activated.
*/
async restoreRefreshTokenAndRetrieveAuthInfo(): Promise<void> {
async restoreAuthToken(): Promise<void> {
if (!this._provider || !this._provider.provider) {
throw new Error("Auth provider is not set")
}
try {
this._clineAuthInfo = await this._provider.provider.retrieveClineAuthInfo(this._context)
if (this._clineAuthInfo) {
this._user = await this._provider.provider.restoreAuthCredential(this._context)
if (this._user) {
this._authenticated = true
await this.sendAuthStatusUpdate()
this.setupAutoRefreshAuth()
// Setup auto-refresh for the auth token
} else {
console.warn("No user found after restoring auth token")
this._authenticated = false
this._clineAuthInfo = null
this._user = null
}
} catch (error) {
console.error("Error restoring auth token:", error)
this._authenticated = false
this._clineAuthInfo = null
this._user = null
return
}
}
/**
* Refreshes the authentication status and sends an update to all subscribers.
*/
async refreshAuth(): Promise<void> {
if (!this._user) {
console.warn("No user is authenticated, skipping auth refresh")
return
}
await this._provider.provider.refreshAuthToken()
this.sendAuthStatusUpdate()
}
private setupAutoRefreshAuth(): void {
// Set timeoutDuration to refresh the auth token 5 minutes before it expires
const timeoutDuration = Math.floor(this._user.stsTokenManager.expirationTime - 5 * 60000 - Date.now()) // Milliseconds until 5 minutes before expiration
setTimeout(() => this._autoRefreshAuth(), timeoutDuration)
}
private async _autoRefreshAuth(): Promise<void> {
if (!this._user) {
console.warn("No user is authenticated, skipping auth refresh")
return
}
await this.refreshAuth()
this.setupAutoRefreshAuth() // Reschedule the next auto-refresh
}
/**
* Subscribe to authStatusUpdate events
* @param controller The controller instance
@@ -1,11 +1,18 @@
import { getSecret, storeSecret } from "@/core/storage/state"
import { ErrorService } from "@/services/error/ErrorService"
import axios from "axios"
import { initializeApp } from "firebase/app"
import { GithubAuthProvider, GoogleAuthProvider, User, getAuth, signInWithCredential } from "firebase/auth"
import {
AuthCredential,
GoogleAuthProvider,
GithubAuthProvider,
OAuthCredential,
User,
UserCredential,
getAuth,
signInWithCredential,
signOut,
} from "firebase/auth"
import { ExtensionContext } from "vscode"
import { ClineAccountUserInfo, ClineAuthInfo } from "../AuthService"
import { jwtDecode } from "jwt-decode"
export class FirebaseAuthProvider {
private _config: any
@@ -22,16 +29,80 @@ export class FirebaseAuthProvider {
this._config = value
}
async shouldRefreshIdToken(existingIdToken: string): Promise<boolean> {
const decodedToken = jwtDecode(existingIdToken)
const exp = decodedToken.exp || 0 // 1752297633
const expirationTime = exp * 1000
const currentTime = Date.now()
const fiveMinutesInMs = 5 * 60 * 1000
if (currentTime > expirationTime - fiveMinutesInMs) {
return true // id token is expired or about to be expired
/**
* Gets the authentication token of the current user.
* @returns {Promise<string | null>} A promise that resolves to the authentication token of the current user, or null if no user is signed in.
*/
async getAuthToken(): Promise<string | null> {
const user = getAuth().currentUser
const idToken = user ? await user.getIdToken() : null
return idToken
}
/**
* Gets the refresh token of the current user.
* @returns {Promise<string | null>} A promise that resolves to the refresh token of the current user, or null if no user is signed in.
*/
async getRefreshToken(): Promise<string | null> {
const user = getAuth().currentUser
const refreshToken = user ? user.refreshToken : null
return refreshToken
}
/**
* Refreshes the authentication token of the current user.
* @returns {Promise<string | null>} A promise that resolves to the refreshed authentication token of the current user, or null if no user is signed in.
*/
async refreshAuthToken(): Promise<string | null> {
const user = getAuth().currentUser
const idToken = user ? await user.getIdToken(true) : null
return idToken
}
/**
* Converts Firebase User object to a generic user object.
* @param user - The Firebase User object.
* @returns {User} A generic user object.
*/
convertUserData(user: User) {
return {
uid: user.uid,
email: user.email,
displayName: user.displayName,
photoUrl: user.photoURL,
}
}
/**
* Signs out the current user from Firebase.
* @returns {Promise<void>} A promise that resolves when the user is signed out.
*/
async signOut(): Promise<void> {
signOut(getAuth(initializeApp(Object.assign({}, this._config))))
.then(() => {
console.log("User signed out successfully.")
})
.catch((error) => {
ErrorService.logMessage("Firebase sign-out error", "error")
ErrorService.logException(error)
throw error
})
}
/**
* Stores the authentication token using a provided token.
* @param token - The authentication token to store.
* @returns {Promise<User>} A promise that resolves with the authenticated user.
* @throws {Error} Throws an error if the storage fails.
*/
private async _storeAuthCredential(context: ExtensionContext, credential: AuthCredential): Promise<void> {
try {
await storeSecret(context, "clineAccountId", JSON.stringify(credential.toJSON()))
} catch (error) {
ErrorService.logMessage("Firebase store token error", "error")
ErrorService.logException(error)
throw error
}
return false
}
/**
@@ -40,68 +111,45 @@ export class FirebaseAuthProvider {
* @returns {Promise<User>} A promise that resolves with the authenticated user.
* @throws {Error} Throws an error if the restoration fails.
*/
async retrieveClineAuthInfo(context: ExtensionContext): Promise<ClineAuthInfo | null> {
const userRefreshToken = await getSecret(context, "clineAccountId")
if (!userRefreshToken) {
async restoreAuthCredential(context: ExtensionContext): Promise<User | null> {
const credentialJSON = await getSecret(context, "clineAccountId")
if (!credentialJSON) {
console.error("No stored authentication credential found.")
return null
}
try {
// Exchange refresh token for new access token using Firebase's secure token endpoint
// https://stackoverflow.com/questions/38233687/how-to-use-the-firebase-refreshtoken-to-reauthenticate/57119131#57119131
const firebaseApiKey = this._config.apiKey
const googleAccessTokenResponse = await axios.post(
`https://securetoken.googleapis.com/v1/token?key=${firebaseApiKey}`,
`grant_type=refresh_token&refresh_token=${encodeURIComponent(userRefreshToken)}`,
{
headers: {
"Content-Type": "application/x-www-form-urlencoded",
},
},
)
// console.log("googleAccessTokenResponse", googleAccessTokenResponse)
// This returns an object with access_token, expires_in (3600), id_token (can be used as bearer token to authenticate requests, we'll use this in the future instead of firebase but need to be aware of how we use firebase sdk for e.g. user info like the profile image), project_id, refresh_token, token_type (always Bearer), and user_id
const idToken = googleAccessTokenResponse.data.id_token
// const idTokenExpirationDate = new Date(Date.now() + googleAccessTokenResponse.data.expires_in * 1000)
// Now retrieve the user info from the backend (this was an easy solution to keep providing user profile details like name and email, but we should move to using the fetchMe() function instead)
// Fetch user info from Cline API
// TODO: consolidate with fetchMe() instead of making the call directly here
const userResponse = await axios.get("https://api.cline.bot/api/v1/users/me", {
headers: {
Authorization: `Bearer ${idToken}`,
},
})
// Store user data
const userInfo: ClineAccountUserInfo = userResponse.data.data
return { idToken, userInfo }
// let userObject = JSON.parse(credentialJSON)
// let user = User.
// userObject = User.constructor._fromJSON(auth, user2);
// const credentialData: AuthCredential = OAuthCredential.fromJSON(credentialJSON) as AuthCredential
// const userCredential = await this._signInWithCredential(context, credentialData)
// return userCredential.user
const credentialData: AuthCredential = OAuthCredential.fromJSON(credentialJSON) as AuthCredential
const userCredential = await this._signInWithCredential(credentialData)
return userCredential.user
} catch (error) {
console.error("Firebase restore token error", error)
ErrorService.logMessage("Firebase restore token error", "error")
ErrorService.logException(error)
throw error
}
}
async _signInWithCredential(credential: AuthCredential): Promise<UserCredential> {
const firebaseConfig = Object.assign({}, this._config)
const app = initializeApp(firebaseConfig)
const auth = getAuth(app)
try {
return await signInWithCredential(auth, credential)
} catch (error) {
ErrorService.logMessage("Firebase sign-in with credential error", "error")
ErrorService.logException(error)
throw error
}
}
/**
* Signs in the user using Firebase authentication with a custom token.
* @returns {Promise<User>} A promise that resolves with the authenticated user.
* @throws {Error} Throws an error if the sign-in fails.
*/
async signIn(context: ExtensionContext, token: string, provider: string): Promise<ClineAuthInfo | null> {
async signIn(context: ExtensionContext, token: string, provider: string): Promise<User> {
try {
let credential
let userCredential
switch (provider) {
case "google":
credential = GoogleAuthProvider.credential(token)
@@ -112,25 +160,9 @@ export class FirebaseAuthProvider {
default:
throw new Error(`Unsupported provider: ${provider}`)
}
// we've received the short-lived tokens from google/github, now we need to sign in to firebase with them
const firebaseConfig = Object.assign({}, this._config)
const app = initializeApp(firebaseConfig)
const auth = getAuth(app)
// this signs the user into firebase sdk internally
const userCredential = (await signInWithCredential(auth, credential)).user
// const userRefreshToken = await userCredential.getIdToken()
// store the long-lived refresh token in secret storage
try {
await storeSecret(context, "clineAccountId", userCredential.refreshToken)
} catch (error) {
ErrorService.logMessage("Firebase store token error", "error")
ErrorService.logException(error)
throw error
}
// userCredential = await this._signInWithCredential(context, credential)
return await this.retrieveClineAuthInfo(context)
this._storeAuthCredential(context, credential)
userCredential = await this._signInWithCredential(credential)
return userCredential.user
} catch (error) {
ErrorService.logMessage("Firebase sign-in error", "error")
ErrorService.logException(error)
@@ -0,0 +1,220 @@
import * as fs from "fs"
import * as path from "path"
import * as os from "os"
import { spawn, ChildProcess } from "child_process"
import { Logger } from "@services/logging/Logger"
import { AUDIO_PROGRAM_CONFIG } from "@/shared/audioProgramConstants"
function isExecutable(filePath: string): boolean {
try {
fs.accessSync(filePath, fs.constants.X_OK)
return true
} catch (e) {
return false
}
}
export class AudioRecordingService {
private recordingProcess: ChildProcess | null = null
private isRecording: boolean = false
private startTime: number = 0
private outputFile: string = ""
constructor() {}
async startRecording(): Promise<{ success: boolean; error?: string }> {
try {
if (this.isRecording) {
return { success: false, error: "Already recording" }
}
// Check if recording software is available
const checkResult = this.checkRecordingDependencies()
if (!checkResult.available) {
return { success: false, error: checkResult.error }
}
// Create temporary file for audio output
const tempDir = os.tmpdir()
this.outputFile = path.join(tempDir, `cline_recording_${Date.now()}.webm`)
Logger.info("Starting audio recording...")
// Get the recording program path
const recordProgram = this.getRecordProgram()
if (!recordProgram) {
return { success: false, error: "Recording program not found" }
}
Logger.info(`Using recording program: ${recordProgram.path}`)
// Set up recording arguments
const args = recordProgram.getArgs(this.outputFile)
// Spawn the recording process
this.recordingProcess = spawn(recordProgram.path, args)
this.isRecording = true
this.startTime = Date.now()
// Handle process errors
this.recordingProcess.on("error", (error) => {
Logger.error(`Recording process error: ${error.message}`)
this.isRecording = false
})
// Handle process exit
this.recordingProcess.on("exit", (code) => {
if (code !== 0 && code !== null) {
Logger.warn(`Recording process exited with code: ${code}`)
}
})
this.recordingProcess.stderr?.on("data", (data) => {
const message = data.toString().trim()
if (message && !message.includes("In:") && !message.includes("Out:")) {
Logger.info(`Recording stderr: ${message}`)
}
})
Logger.info("Audio recording started successfully")
return { success: true }
} catch (error) {
const errorMessage = error instanceof Error ? error.message : String(error)
Logger.error("Failed to start audio recording: " + errorMessage)
return { success: false, error: `Failed to start recording: ${errorMessage}` }
}
}
async stopRecording(): Promise<{ success: boolean; audioBase64?: string; error?: string }> {
try {
if (!this.isRecording || !this.recordingProcess) {
return { success: false, error: "Not currently recording" }
}
Logger.info("Stopping audio recording...")
// Send SIGINT to stop recording gracefully (like Ctrl+C)
this.recordingProcess.kill("SIGINT")
// Wait for the process to finish
await new Promise<void>((resolve) => {
if (this.recordingProcess) {
// Timeout after 5 seconds
const timeoutId = setTimeout(() => {
resolve()
}, 5000)
this.recordingProcess.on("exit", (code) => {
clearTimeout(timeoutId) // Clear the timeout since process exited
resolve()
})
} else {
resolve()
}
})
this.recordingProcess = null
this.isRecording = false
// Wait a moment for file to be fully written
await new Promise((resolve) => setTimeout(resolve, 500))
// Read the audio file and convert to base64
if (!fs.existsSync(this.outputFile)) {
return { success: false, error: "Recording file not found" }
}
const audioBuffer = fs.readFileSync(this.outputFile)
const audioBase64 = audioBuffer.toString("base64")
// Clean up temporary file
try {
fs.unlinkSync(this.outputFile)
} catch (cleanupError) {
Logger.warn(
"Failed to cleanup temporary audio file: " +
(cleanupError instanceof Error ? cleanupError.message : String(cleanupError)),
)
}
Logger.info("Audio recording stopped and converted to base64")
return { success: true, audioBase64 }
} catch (error) {
const errorMessage = error instanceof Error ? error.message : String(error)
Logger.error("Failed to stop audio recording: " + errorMessage)
return { success: false, error: `Failed to stop recording: ${errorMessage}` }
}
}
getRecordingStatus(): { isRecording: boolean; durationSeconds: number; error?: string } {
const durationSeconds = this.isRecording ? (Date.now() - this.startTime) / 1000 : 0
return {
isRecording: this.isRecording,
durationSeconds,
}
}
private checkRecordingDependencies(): { available: boolean; error?: string } {
const program = this.getRecordProgram()
if (!program) {
const platform = os.platform() as keyof typeof AUDIO_PROGRAM_CONFIG
const config = AUDIO_PROGRAM_CONFIG[platform]
const error = config ? config.error : `Audio recording is not supported on platform: ${platform}`
return { available: false, error }
}
return { available: true }
}
private getRecordProgram(): { path: string; getArgs: (outputFile: string) => string[] } | undefined {
const platform = os.platform() as keyof typeof AUDIO_PROGRAM_CONFIG
const config = AUDIO_PROGRAM_CONFIG[platform]
if (!config) {
return undefined
}
// 1. Check if the command is in the system's PATH
const pathDirs = (process.env.PATH || "").split(path.delimiter)
for (const dir of pathDirs) {
const fullPath = path.join(dir, config.command)
if (fs.existsSync(fullPath) && isExecutable(fullPath)) {
return { path: fullPath, getArgs: config.getArgs }
}
}
// 2. Check fallback paths if not in PATH
for (const p of config.fallbackPaths) {
if (fs.existsSync(p) && isExecutable(p)) {
return { path: p, getArgs: config.getArgs }
}
}
return undefined
}
// Cleanup method
cleanup(): void {
if (this.isRecording && this.recordingProcess) {
try {
this.recordingProcess.kill("SIGINT")
this.recordingProcess = null
this.isRecording = false
} catch (error) {
Logger.error("Error during cleanup: " + (error instanceof Error ? error.message : String(error)))
}
}
// Clean up any leftover temp files
if (this.outputFile && fs.existsSync(this.outputFile)) {
try {
fs.unlinkSync(this.outputFile)
} catch (error) {
Logger.warn(
"Failed to cleanup temp file during service cleanup: " +
(error instanceof Error ? error.message : String(error)),
)
}
}
}
}
export const audioRecordingService = new AudioRecordingService()
@@ -0,0 +1,54 @@
import { Logger } from "@services/logging/Logger"
import { ClineAccountService } from "@/services/account/ClineAccountService"
import axios from "axios"
export class VoiceTranscriptionService {
private clineAccountService: ClineAccountService
constructor() {
this.clineAccountService = ClineAccountService.getInstance()
}
async transcribeAudio(audioBase64: string, language?: string): Promise<{ text?: string; error?: string }> {
try {
Logger.info("Transcribing audio with Cline transcription service...")
const result = await this.clineAccountService.transcribeAudio(audioBase64, language)
Logger.info("Transcription successful")
return { text: result.text }
} catch (error) {
Logger.error("Voice transcription error:", error)
// Handle axios errors with proper status code mapping
if (axios.isAxiosError(error)) {
const status = error.response?.status
const message = error.response?.data?.message || error.message
switch (status) {
case 401:
return { error: "Authentication failed. Please reauthenticate your Cline account" }
case 402:
return { error: "Insufficient credits for transcription service." }
case 400:
return { error: "Invalid audio format or request data." }
case 500:
return { error: "Transcription server error. Please try again later." }
default:
return { error: `Transcription failed: ${message}` }
}
}
// Handle network errors
const errorMessage = error instanceof Error ? error.message : String(error)
if (errorMessage.includes("ECONNREFUSED") || errorMessage.includes("Network Error")) {
return { error: "Cannot connect to transcription service." }
}
return { error: `Network error: ${errorMessage}` }
}
}
}
export const voiceTranscriptionService = new VoiceTranscriptionService()
@@ -27,13 +27,14 @@ interface Collection {
* When adding a new category, add it both here and to the initial values in telemetryCategoryEnabled
* Ensure `if (!this.isCategoryEnabled('<category_name>')` is added to the capture method
*/
type TelemetryCategory = "checkpoints" | "browser"
type TelemetryCategory = "checkpoints" | "browser" | "voice"
class TelemetryService {
// Map to control specific telemetry categories (event types)
private telemetryCategoryEnabled: Map<TelemetryCategory, boolean> = new Map([
["checkpoints", false], // Checkpoints telemetry disabled
["browser", true], // Browser telemetry enabled
["voice", true], // Voice telemetry enabled
])
// Stores events when collect=true
@@ -95,6 +96,20 @@ class TelemetryService {
// Tracks when a button is clicked
BUTTON_CLICKED: "ui.button_clicked",
},
// Voice-related events for tracking voice recording and transcription usage
VOICE: {
// Tracks when voice recording is started
RECORDING_STARTED: "voice.recording_started",
// Tracks when voice recording is stopped
RECORDING_STOPPED: "voice.recording_stopped",
// Tracks when voice transcription is started
TRANSCRIPTION_STARTED: "voice.transcription_started",
// Tracks when voice transcription is completed successfully
TRANSCRIPTION_COMPLETED: "voice.transcription_completed",
// Tracks when voice transcription fails
TRANSCRIPTION_ERROR: "voice.transcription_error",
// Tracks when voice feature is enabled or disabled in settings
},
}
/** Singleton instance of the TelemetryService */
@@ -140,16 +155,9 @@ class TelemetryService {
} else {
// Only show warning if user has opted in to Cline telemetry but VS Code telemetry is disabled
if (didUserOptIn) {
void vscode.window
.showWarningMessage(
"Anonymous Cline error and usage reporting is enabled, but VSCode telemetry is disabled. To enable error and usage reporting for this extension, enable VSCode telemetry in settings.",
"Open Settings",
)
.then((selection) => {
if (selection === "Open Settings") {
void vscode.commands.executeCommand("workbench.action.openSettings", "telemetry.telemetryLevel")
}
})
void vscode.window.showWarningMessage(
"Anonymous Cline error and usage reporting is enabled, but VSCode telemetry is disabled. To enable error and usage reporting for this extension, enable VSCode telemetry in settings.",
)
}
this.telemetryEnabled = false
}
@@ -720,6 +728,164 @@ class TelemetryService {
)
}
// Voice events
/**
* Records when voice recording is started
* @param taskId Optional task identifier if recording was started during a task
* @param platform The platform where recording is happening (macOS, Windows, Linux)
* @param collect If true, collect event instead of sending
*/
public captureVoiceRecordingStarted(taskId?: string, platform?: string, collect: boolean = false) {
if (!this.isCategoryEnabled("voice")) {
return
}
this.capture(
{
event: TelemetryService.EVENTS.VOICE.RECORDING_STARTED,
properties: {
taskId,
platform: platform || process.platform,
timestamp: new Date().toISOString(),
},
},
collect,
)
}
/**
* Records when voice recording is stopped
* @param taskId Optional task identifier if recording was stopped during a task
* @param durationMs Duration of the recording in milliseconds
* @param success Whether the recording was successful
* @param platform The platform where recording happened
* @param collect If true, collect event instead of sending
*/
public captureVoiceRecordingStopped(
taskId?: string,
durationMs?: number,
success?: boolean,
platform?: string,
collect?: boolean,
) {
if (!this.isCategoryEnabled("voice")) {
return
}
this.capture(
{
event: TelemetryService.EVENTS.VOICE.RECORDING_STOPPED,
properties: {
taskId,
durationMs,
success,
platform: platform || process.platform,
timestamp: new Date().toISOString(),
},
},
collect || false,
)
}
/**
* Records when voice transcription is started
* @param taskId Optional task identifier if transcription was started during a task
* @param audioSizeBytes Size of the audio data being transcribed
* @param language Language hint provided for transcription
* @param collect If true, collect event instead of sending
*/
public captureVoiceTranscriptionStarted(
taskId?: string,
audioSizeBytes?: number,
language?: string,
collect: boolean = false,
) {
if (!this.isCategoryEnabled("voice")) {
return
}
this.capture(
{
event: TelemetryService.EVENTS.VOICE.TRANSCRIPTION_STARTED,
properties: {
taskId,
audioSizeBytes,
language,
timestamp: new Date().toISOString(),
},
},
collect,
)
}
/**
* Records when voice transcription is completed successfully
* @param taskId Optional task identifier if transcription was completed during a task
* @param transcriptionLength Length of the transcribed text
* @param durationMs Time taken for transcription in milliseconds
* @param language Language used for transcription
* @param collect If true, collect event instead of sending
*/
public captureVoiceTranscriptionCompleted(
taskId?: string,
transcriptionLength?: number,
durationMs?: number,
language?: string,
collect: boolean = false,
) {
if (!this.isCategoryEnabled("voice")) {
return
}
this.capture(
{
event: TelemetryService.EVENTS.VOICE.TRANSCRIPTION_COMPLETED,
properties: {
taskId,
transcriptionLength,
durationMs,
language,
timestamp: new Date().toISOString(),
},
},
collect,
)
}
/**
* Records when voice transcription fails
* @param taskId Optional task identifier if transcription failed during a task
* @param errorType Type of error that occurred (e.g., "no_openai_key", "api_error", "network_error")
* @param errorMessage The error message
* @param durationMs Time taken before failure in milliseconds
* @param collect If true, collect event instead of sending
*/
public captureVoiceTranscriptionError(
taskId?: string,
errorType?: string,
errorMessage?: string,
durationMs?: number,
collect: boolean = false,
) {
if (!this.isCategoryEnabled("voice")) {
return
}
this.capture(
{
event: TelemetryService.EVENTS.VOICE.TRANSCRIPTION_ERROR,
properties: {
taskId,
errorType,
errorMessage,
durationMs,
timestamp: new Date().toISOString(),
},
},
collect,
)
}
/**
* Checks if telemetry is enabled
* @returns Boolean indicating whether telemetry is enabled
+74
View File
@@ -0,0 +1,74 @@
export interface DictationSettings {
voiceRecordingEnabled: boolean
dictationLanguage: string
}
export const DEFAULT_DICTATION_SETTINGS: DictationSettings = {
voiceRecordingEnabled: true,
dictationLanguage: "en",
}
export interface LanguageItem {
name: string
code: string
}
export const SUPPORTED_DICTATION_LANGUAGES: LanguageItem[] = [
{ name: "English", code: "en" },
{ name: "Spanish (Español)", code: "es" },
{ name: "Chinese (中文)", code: "zh" },
{ name: "Japanese (日本語)", code: "ja" },
{ name: "Afrikaans", code: "af" },
{ name: "Arabic (العربية)", code: "ar" },
{ name: "Armenian (Հայերեն)", code: "hy" },
{ name: "Azerbaijani (Azərbaycan)", code: "az" },
{ name: "Belarusian (Беларуская)", code: "be" },
{ name: "Bosnian (Bosanski)", code: "bs" },
{ name: "Bulgarian (Български)", code: "bg" },
{ name: "Catalan (Català)", code: "ca" },
{ name: "Croatian (Hrvatski)", code: "hr" },
{ name: "Czech (Čeština)", code: "cs" },
{ name: "Danish (Dansk)", code: "da" },
{ name: "Dutch (Nederlands)", code: "nl" },
{ name: "Estonian (Eesti)", code: "et" },
{ name: "Finnish (Suomi)", code: "fi" },
{ name: "French (Français)", code: "fr" },
{ name: "Galician (Galego)", code: "gl" },
{ name: "German (Deutsch)", code: "de" },
{ name: "Greek (Ελληνικά)", code: "el" },
{ name: "Hebrew (עברית)", code: "he" },
{ name: "Hindi (हिन्दी)", code: "hi" },
{ name: "Hungarian (Magyar)", code: "hu" },
{ name: "Icelandic (Íslenska)", code: "is" },
{ name: "Indonesian (Bahasa Indonesia)", code: "id" },
{ name: "Italian (Italiano)", code: "it" },
{ name: "Kannada (ಕನ್ನಡ)", code: "kn" },
{ name: "Kazakh (Қазақша)", code: "kk" },
{ name: "Korean (한국어)", code: "ko" },
{ name: "Latvian (Latviešu)", code: "lv" },
{ name: "Lithuanian (Lietuvių)", code: "lt" },
{ name: "Macedonian (Македонски)", code: "mk" },
{ name: "Malay (Bahasa Melayu)", code: "ms" },
{ name: "Marathi (मराठी)", code: "mr" },
{ name: "Maori (Te Reo Māori)", code: "mi" },
{ name: "Nepali (नेपाली)", code: "ne" },
{ name: "Norwegian (Norsk)", code: "no" },
{ name: "Persian (فارسی)", code: "fa" },
{ name: "Polish (Polski)", code: "pl" },
{ name: "Portuguese (Português)", code: "pt" },
{ name: "Romanian (Română)", code: "ro" },
{ name: "Russian (Русский)", code: "ru" },
{ name: "Serbian (Српски)", code: "sr" },
{ name: "Slovak (Slovenčina)", code: "sk" },
{ name: "Slovenian (Slovenščina)", code: "sl" },
{ name: "Swahili (Kiswahili)", code: "sw" },
{ name: "Swedish (Svenska)", code: "sv" },
{ name: "Tagalog", code: "tl" },
{ name: "Tamil (தமிழ்)", code: "ta" },
{ name: "Thai (ไทย)", code: "th" },
{ name: "Turkish (Türkçe)", code: "tr" },
{ name: "Ukrainian (Українська)", code: "uk" },
{ name: "Urdu (اردو)", code: "ur" },
{ name: "Vietnamese (Tiếng Việt)", code: "vi" },
{ name: "Welsh (Cymraeg)", code: "cy" },
]
+2
View File
@@ -3,6 +3,7 @@ import { ApiConfiguration } from "./api"
import { AutoApprovalSettings } from "./AutoApprovalSettings"
import { BrowserSettings } from "./BrowserSettings"
import { ChatSettings } from "./ChatSettings"
import { DictationSettings } from "./DictationSettings"
import { HistoryItem } from "./HistoryItem"
import { TelemetrySetting } from "./TelemetrySetting"
import { ClineRulesToggles } from "./cline-rules"
@@ -31,6 +32,7 @@ export interface ExtensionState {
apiConfiguration?: ApiConfiguration
autoApprovalSettings: AutoApprovalSettings
browserSettings: BrowserSettings
dictationSettings: DictationSettings
remoteBrowserHost?: string
chatSettings: ChatSettings
checkpointTrackerErrorMessage?: string
+68
View File
@@ -0,0 +1,68 @@
export const AUDIO_PROGRAM_CONFIG = {
darwin: {
command: "ffmpeg",
fallbackPaths: ["/usr/local/bin/ffmpeg", "/opt/homebrew/bin/ffmpeg"],
getArgs: (outputFile: string) => [
"-f",
"avfoundation",
"-i",
":default",
"-c:a",
"libopus",
"-b:a",
"32k",
"-application",
"voip",
"-ar",
"16000",
"-ac",
"1",
outputFile,
],
error: "It looks like your system is missing the 'FFmpeg' utility, which is required for voice recording. To install it, please ensure you are in **Act Mode** and then send this message. I will handle the installation for you.\n\n**Installation command:** `brew install ffmpeg`",
},
linux: {
command: "ffmpeg",
fallbackPaths: ["/usr/bin/ffmpeg", "/usr/local/bin/ffmpeg"],
getArgs: (outputFile: string) => [
"-f",
"pulse",
"-i",
"default",
"-c:a",
"libopus",
"-b:a",
"32k",
"-application",
"voip",
"-ar",
"16000",
"-ac",
"1",
outputFile,
],
error: "It looks like your system is missing the 'FFmpeg' utility, which is required for voice recording. To install it, please ensure you are in **Act Mode** and then send this message. I will handle the installation for you.\n\n**Installation command:** `sudo apt-get install ffmpeg`",
},
win32: {
command: "ffmpeg",
fallbackPaths: ["C:\\ffmpeg\\bin\\ffmpeg.exe", "C:\\Program Files\\ffmpeg\\bin\\ffmpeg.exe"],
getArgs: (outputFile: string) => [
"-f",
"dshow",
"-i",
"audio=default",
"-c:a",
"libopus",
"-b:a",
"32k",
"-application",
"voip",
"-ar",
"16000",
"-ac",
"1",
outputFile,
],
error: "It looks like your system is missing the 'FFmpeg' utility, which is required for voice recording. To install it, please ensure you are in **Act Mode** and then send this message. I will handle the installation for you.\n\n**Installation command:** `winget install Gyan.FFmpeg`",
},
}
@@ -0,0 +1,24 @@
import { DictationSettings as ProtoDictationSettings } from "../../proto/state"
import { DictationSettings } from "../../DictationSettings"
/**
* Converts proto DictationSettings to TypeScript DictationSettings
*/
export function convertProtoDictationSettingsToDictationSettings(
protoDictationSettings: ProtoDictationSettings,
): DictationSettings {
return {
voiceRecordingEnabled: protoDictationSettings.voiceRecordingEnabled,
dictationLanguage: protoDictationSettings.dictationLanguage,
}
}
/**
* Converts TypeScript DictationSettings to proto DictationSettings
*/
export function convertDictationSettingsToProtoDictationSettings(dictationSettings: DictationSettings): ProtoDictationSettings {
return ProtoDictationSettings.create({
voiceRecordingEnabled: dictationSettings.voiceRecordingEnabled,
dictationLanguage: dictationSettings.dictationLanguage,
})
}
@@ -1,8 +0,0 @@
import { DiffViewProvider } from "@/integrations/editor/DiffViewProvider"
export class ExternalDiffViewProvider extends DiffViewProvider {
override async openDiffEditor(): Promise<void> {
// The host bridge proto changes are not submitted yet.
//getHostBridgeProvider().diffClient.openDiff(this.absolutePath)
}
}
@@ -4,14 +4,12 @@ import {
WorkspaceServiceClientImpl,
EnvServiceClientImpl,
WindowServiceClientImpl,
DiffServiceClientImpl,
} from "@generated/standalone/host-bridge-clients"
import {
WatchServiceClientInterface,
WorkspaceServiceClientInterface,
EnvServiceClientInterface,
WindowServiceClientInterface,
DiffServiceClientInterface,
} from "@generated/hosts/host-bridge-client-types"
import { HostBridgeClientProvider } from "@/hosts/host-provider-types"
@@ -25,7 +23,6 @@ export class ExternalHostBridgeClientManager implements HostBridgeClientProvider
workspaceClient: WorkspaceServiceClientInterface
envClient: EnvServiceClientInterface
windowClient: WindowServiceClientInterface
diffClient: DiffServiceClientInterface
constructor() {
const address = process.env.HOST_BRIDGE_ADDRESS || "localhost:50052"
@@ -35,7 +32,6 @@ export class ExternalHostBridgeClientManager implements HostBridgeClientProvider
this.workspaceClient = new WorkspaceServiceClientImpl(this.channel)
this.envClient = new EnvServiceClientImpl(this.channel)
this.windowClient = new WindowServiceClientImpl(this.channel)
this.diffClient = new DiffServiceClientImpl(this.channel)
}
public close(): void {
+2 -6
View File
@@ -13,12 +13,11 @@ import { ExternalHostBridgeClientManager } from "./host-bridge-client-manager"
import { ExternalWebviewProvider } from "./ExternalWebviewProvider"
import { WebviewProviderType } from "@/shared/webview/types"
import { v4 as uuidv4 } from "uuid"
import { ExternalDiffViewProvider } from "./ExternalDiffviewProvider"
async function main() {
log("Starting standalone service...")
hostProviders.initializeHostProviders(createWebview, createDiffView, new ExternalHostBridgeClientManager())
hostProviders.initializeHostProviders(createWebview, new ExternalHostBridgeClientManager())
activate(extensionContext)
const controller = new Controller(extensionContext, outputChannel, postMessage, uuidv4())
startProtobusService(controller)
@@ -61,12 +60,9 @@ function getProtobusServiceNames(packageDefinition: { [x: string]: any }): strin
return protobusServiceNames
}
function createWebview() {
const createWebview = () => {
return new ExternalWebviewProvider(extensionContext, outputChannel, WebviewProviderType.SIDEBAR)
}
function createDiffView() {
return new ExternalDiffViewProvider()
}
/**
* Wraps a Promise-based handler function to make it compatible with gRPC's callback-based API.
+19 -10
View File
@@ -9,7 +9,6 @@
"version": "0.3.0",
"dependencies": {
"@floating-ui/react": "^0.27.4",
"@fontsource/azeret-mono": "^5.2.9",
"@heroui/react": "^2.8.0-beta.2",
"@vscode/webview-ui-toolkit": "^1.4.0",
"debounce": "^2.1.1",
@@ -24,6 +23,7 @@
"posthog-js": "^1.224.0",
"pretty-bytes": "^6.1.1",
"react": "^18.3.1",
"react-countup": "^6.5.3",
"react-dom": "^18.3.1",
"react-remark": "^2.1.0",
"react-textarea-autosize": "^8.5.7",
@@ -1261,15 +1261,6 @@
"integrity": "sha512-MDWhGtE+eHw5JW7lq4qhc5yRLS11ERl1c7Z6Xd0a58DozHES6EnNNwUWbMiG4J9Cgj053Bhk8zvlhFYKVhULwg==",
"license": "MIT"
},
"node_modules/@fontsource/azeret-mono": {
"version": "5.2.9",
"resolved": "https://registry.npmjs.org/@fontsource/azeret-mono/-/azeret-mono-5.2.9.tgz",
"integrity": "sha512-1qnbVspQPI38qhSTSidWU4bjG5ynWCfkMwfPxahqxejJO/u4yT1FbPqG73s4fDmQSuDQYoA8jfTpoQiod7+fuA==",
"license": "OFL-1.1",
"funding": {
"url": "https://github.com/sponsors/ayuhito"
}
},
"node_modules/@formatjs/ecma402-abstract": {
"version": "2.3.4",
"resolved": "https://registry.npmjs.org/@formatjs/ecma402-abstract/-/ecma402-abstract-2.3.4.tgz",
@@ -8336,6 +8327,12 @@
"layout-base": "^1.0.0"
}
},
"node_modules/countup.js": {
"version": "2.8.0",
"resolved": "https://registry.npmjs.org/countup.js/-/countup.js-2.8.0.tgz",
"integrity": "sha512-f7xEhX0awl4NOElHulrl4XRfKoNH3rB+qfNSZZyjSZhaAoUk6elvhH+MNxMmlmuUJ2/QNTWPSA7U4mNtIAKljQ==",
"license": "MIT"
},
"node_modules/create-error-class": {
"version": "3.0.2",
"resolved": "https://registry.npmjs.org/create-error-class/-/create-error-class-3.0.2.tgz",
@@ -13714,6 +13711,18 @@
"node": ">=0.10.0"
}
},
"node_modules/react-countup": {
"version": "6.5.3",
"resolved": "https://registry.npmjs.org/react-countup/-/react-countup-6.5.3.tgz",
"integrity": "sha512-udnqVQitxC7QWADSPDOxVWULkLvKUWrDapn5i53HE4DPRVgs+Y5rr4bo25qEl8jSh+0l2cToJgGMx+clxPM3+w==",
"license": "MIT",
"dependencies": {
"countup.js": "^2.8.0"
},
"peerDependencies": {
"react": ">= 16.3.0"
}
},
"node_modules/react-devtools": {
"version": "6.1.2",
"resolved": "https://registry.npmjs.org/react-devtools/-/react-devtools-6.1.2.tgz",
+1 -1
View File
@@ -16,7 +16,6 @@
},
"dependencies": {
"@floating-ui/react": "^0.27.4",
"@fontsource/azeret-mono": "^5.2.9",
"@heroui/react": "^2.8.0-beta.2",
"@vscode/webview-ui-toolkit": "^1.4.0",
"debounce": "^2.1.1",
@@ -31,6 +30,7 @@
"posthog-js": "^1.224.0",
"pretty-bytes": "^6.1.1",
"react": "^18.3.1",
"react-countup": "^6.5.3",
"react-dom": "^18.3.1",
"react-remark": "^2.1.0",
"react-textarea-autosize": "^8.5.7",
@@ -1,8 +1,10 @@
import { VSCodeButton, VSCodeDivider, VSCodeLink, VSCodeDropdown, VSCodeOption } from "@vscode/webview-ui-toolkit/react"
import { memo, useCallback, useEffect, useState, useRef } from "react"
import { memo, useCallback, useEffect, useState } from "react"
import { BadgeCent } from "lucide-react"
import { useClineAuth } from "@/context/ClineAuthContext"
import VSCodeButtonLink from "../common/VSCodeButtonLink"
import ClineLogoWhite from "../../assets/ClineLogoWhite"
import CountUp from "react-countup"
import CreditsHistoryTable from "./CreditsHistoryTable"
import { UsageTransaction, PaymentTransaction } from "@shared/ClineAccount"
import { useExtensionState } from "@/context/ExtensionStateContext"
@@ -11,64 +13,6 @@ import { EmptyRequest } from "@shared/proto/common"
import { UserOrganization, UserOrganizationUpdateRequest } from "@shared/proto/account"
import { formatCreditsBalance } from "@/utils/format"
// Custom hook for animated credit display with styled decimals
const useAnimatedCredits = (targetValue: number, duration: number = 660) => {
const [currentValue, setCurrentValue] = useState(0)
const animationRef = useRef<number>()
const startTimeRef = useRef<number>()
useEffect(() => {
const animate = (timestamp: number) => {
if (!startTimeRef.current) {
startTimeRef.current = timestamp
}
const elapsed = timestamp - startTimeRef.current
const progress = Math.min(elapsed / duration, 1)
// Easing function (ease-out)
const easedProgress = 1 - Math.pow(1 - progress, 3)
const newValue = easedProgress * targetValue
setCurrentValue(newValue)
if (progress < 1) {
animationRef.current = requestAnimationFrame(animate)
}
}
// Reset and start animation
startTimeRef.current = undefined
animationRef.current = requestAnimationFrame(animate)
return () => {
if (animationRef.current) {
cancelAnimationFrame(animationRef.current)
}
}
}, [targetValue, duration])
return currentValue
}
// Custom component to handle styled credit display
const StyledCreditDisplay = ({ balance }: { balance: number }) => {
const animatedValue = useAnimatedCredits(formatCreditsBalance(balance))
const formatted = animatedValue.toFixed(4)
const parts = formatted.split(".")
const wholePart = parts[0]
const decimalPart = parts[1] || "0000"
const firstTwoDecimals = decimalPart.slice(0, 2)
const lastTwoDecimals = decimalPart.slice(2)
return (
<span className="font-azeret-mono font-light tabular-nums">
{wholePart}.{firstTwoDecimals}
<span className="text-[var(--vscode-descriptionForeground)]">{lastTwoDecimals}</span>
</span>
)
}
type VSCodeDropdownChangeEvent = Event & {
target: {
value: string
@@ -202,13 +146,13 @@ export const ClineAccountView = () => {
<div className="flex flex-col pr-3 h-full">
<div className="flex flex-col w-full">
<div className="flex items-center mb-6 flex-wrap gap-y-4">
{/* {user.photoUrl ? (
{user.photoUrl ? (
<img src={user.photoUrl} alt="Profile" className="size-16 rounded-full mr-4" />
) : ( */}
<div className="size-16 rounded-full bg-[var(--vscode-button-background)] flex items-center justify-center text-2xl text-[var(--vscode-button-foreground)] mr-4">
{user.displayName?.[0] || user.email?.[0] || "?"}
</div>
{/* )} */}
) : (
<div className="size-16 rounded-full bg-[var(--vscode-button-background)] flex items-center justify-center text-2xl text-[var(--vscode-button-foreground)] mr-4">
{user.displayName?.[0] || user.email?.[0] || "?"}
</div>
)}
<div className="flex flex-col">
{user.displayName && (
@@ -256,9 +200,7 @@ export const ClineAccountView = () => {
{activeOrganization === null && (
<div className="w-full flex flex-col items-center">
<div className="text-sm text-[var(--vscode-descriptionForeground)] mb-3 font-azeret-mono font-light">
CURRENT BALANCE
</div>
<div className="text-sm text-[var(--vscode-descriptionForeground)] mb-3">CURRENT BALANCE</div>
<div className="text-4xl font-bold text-[var(--vscode-foreground)] mb-6 flex items-center gap-2">
{isLoading ? (
@@ -269,7 +211,8 @@ export const ClineAccountView = () => {
<span>----</span>
) : (
<>
<StyledCreditDisplay balance={balance} />
<BadgeCent className="size-6 text-[var(--vscode-foreground)]" />
<CountUp end={formatCreditsBalance(balance)} duration={0.66} decimals={4} />
</>
)}
<VSCodeButton appearance="icon" className="mt-1" onClick={getUserCredits}>
+1 -20
View File
@@ -1,4 +1,4 @@
import { VSCodeBadge, VSCodeButton, VSCodeProgressRing } from "@vscode/webview-ui-toolkit/react"
import { VSCodeBadge, VSCodeProgressRing } from "@vscode/webview-ui-toolkit/react"
import deepEqual from "fast-deep-equal"
import React, { memo, MouseEvent, useCallback, useEffect, useMemo, useRef, useState } from "react"
import styled from "styled-components"
@@ -36,7 +36,6 @@ import NewTaskPreview from "./NewTaskPreview"
import ReportBugPreview from "./ReportBugPreview"
import UserMessage from "./UserMessage"
import QuoteButton from "./QuoteButton"
import { useClineAuth } from "@/context/ClineAuthContext"
const normalColor = "var(--vscode-foreground)"
const errorColor = "var(--vscode-errorForeground)"
@@ -185,7 +184,6 @@ export const ChatRowContent = memo(
sendMessageFromChatRow,
onSetQuote,
}: ChatRowContentProps) => {
const { handleSignIn, clineUser } = useClineAuth()
const { mcpServers, mcpMarketplaceCatalog, onRelinquishControl, apiConfiguration } = useExtensionState()
const [seeNewChangesDisabled, setSeeNewChangesDisabled] = useState(false)
const [quoteButtonState, setQuoteButtonState] = useState<QuoteButtonState>({
@@ -1008,23 +1006,6 @@ export const ChatRowContent = memo(
.
</>
)}
{apiRequestFailedMessage?.includes(
"Unauthorized: Please sign in to Cline before trying again.", // match with cline.ts (TODO: remove after some time)
) && (
<>
<br />
<br />
{clineUser ? (
<span style={{ color: "var(--vscode-descriptionForeground)" }}>
(Click "Retry" below)
</span>
) : (
<VSCodeButton onClick={handleSignIn} className="w-full mb-4">
Sign in to Cline
</VSCodeButton>
)}
</>
)}
</p>
)
})()}
@@ -45,6 +45,7 @@ import { useClickAway, useEvent, useWindowSize } from "react-use"
import styled from "styled-components"
import ClineRulesToggleModal from "../cline-rules/ClineRulesToggleModal"
import ServersToggleModal from "./ServersToggleModal"
import VoiceRecorder from "./VoiceRecorder"
const getImageDimensions = (dataUrl: string): Promise<{ width: number; height: number }> => {
return new Promise((resolve, reject) => {
@@ -280,6 +281,7 @@ const ChatTextArea = forwardRef<HTMLTextAreaElement, ChatTextAreaProps>(
platform,
localWorkflowToggles,
globalWorkflowToggles,
dictationSettings,
} = useExtensionState()
const [isTextAreaFocused, setIsTextAreaFocused] = useState(false)
const [isDraggingOver, setIsDraggingOver] = useState(false)
@@ -1564,7 +1566,7 @@ const ChatTextArea = forwardRef<HTMLTextAreaElement, ChatTextAreaProps>(
// borderLeft: "9px solid transparent", // NOTE: react-textarea-autosize doesn't calculate correct height when using borderLeft/borderRight so we need to use horizontal padding instead
// Instead of using boxShadow, we use a div with a border to better replicate the behavior when the textarea is focused
// boxShadow: "0px 0px 0px 1px var(--vscode-input-border)",
padding: "9px 28px 9px 9px",
padding: `9px ${dictationSettings?.voiceRecordingEnabled ? "48" : "28"}px 9px 9px`,
cursor: "text",
flex: 1,
zIndex: 1,
@@ -1605,10 +1607,9 @@ const ChatTextArea = forwardRef<HTMLTextAreaElement, ChatTextAreaProps>(
position: "absolute",
right: 23,
display: "flex",
alignItems: "flex-end",
alignItems: "center",
height: textAreaBaseHeight || 31,
bottom: 9.5, // should be 10 but doesn't look good on mac
paddingBottom: "8px",
zIndex: 2,
}}>
<div
@@ -1617,6 +1618,35 @@ const ChatTextArea = forwardRef<HTMLTextAreaElement, ChatTextAreaProps>(
flexDirection: "row",
alignItems: "center",
}}>
{dictationSettings?.voiceRecordingEnabled === true && (
<VoiceRecorder
onTranscription={(text) => {
// Remove any processing text first
const cleanedValue = inputValue.replace(/\s*\[Transcribing\.\.\.\]$/, "")
// Append the transcribed text to the cleaned input
const newValue = cleanedValue + (cleanedValue ? " " : "") + text
setInputValue(newValue)
// Focus the textarea and move cursor to end
setTimeout(() => {
if (textAreaRef.current) {
textAreaRef.current.focus()
const length = newValue.length
textAreaRef.current.setSelectionRange(length, length)
}
}, 0)
}}
onProcessingStateChange={(isProcessing, message) => {
if (isProcessing && message) {
// Show processing message in input
const processingText = inputValue + (inputValue ? " " : "") + `[${message}]`
setInputValue(processingText)
}
// When processing is done, the onTranscription callback will handle the final text
}}
disabled={sendingDisabled}
language={dictationSettings?.dictationLanguage || "en"}
/>
)}
{/* <div
className={`input-icon-button ${shouldDisableImages ? "disabled" : ""} codicon codicon-device-camera`}
onClick={() => {
@@ -0,0 +1,223 @@
import React, { useState, useCallback, useEffect, useRef } from "react"
import { VoiceServiceClient } from "@/services/grpc-client"
import {
StartRecordingRequest,
StopRecordingRequest,
TranscribeAudioRequest,
GetRecordingStatusRequest,
} from "@shared/proto/voice"
import HeroTooltip from "../common/HeroTooltip"
import { formatSeconds } from "@/utils/format"
interface VoiceRecorderProps {
onTranscription: (text: string) => void
onProcessingStateChange?: (isProcessing: boolean, message?: string) => void
disabled?: boolean
language?: string
}
const MAX_DURATION = 5 * 60 // 5 minutes in seconds
const VoiceRecorder: React.FC<VoiceRecorderProps> = ({
onTranscription,
onProcessingStateChange,
disabled = false,
language = "en",
}) => {
const [isRecording, setIsRecording] = useState(false)
const [isProcessing, setIsProcessing] = useState(false)
const [recordingDuration, setRecordingDuration] = useState(0)
const [error, setError] = useState<string | null>(null)
const pollingIntervalRef = useRef<NodeJS.Timeout | null>(null)
const startRecording = useCallback(async () => {
try {
setIsRecording(true)
setError(null) // Clear any previous errors
onProcessingStateChange?.(false) // Clear any previous processing state
setRecordingDuration(0) // Reset recording duration
// Call Extension Host to start recording
const response = await VoiceServiceClient.startRecording(StartRecordingRequest.create({}))
if (!response.success) {
console.error("Failed to start recording:", response.error)
setIsRecording(false)
setError(response.error || "Failed to start recording")
onProcessingStateChange?.(true, response.error || "Failed to start recording")
return
}
console.log("Recording started successfully")
} catch (error) {
console.error("Error starting recording:", error)
setIsRecording(false)
const errorMessage = error instanceof Error ? error.message : "Failed to start recording"
setError(errorMessage)
onProcessingStateChange?.(true, errorMessage)
}
}, [onProcessingStateChange])
const stopRecording = useCallback(async () => {
try {
setIsRecording(false)
setIsProcessing(true)
onProcessingStateChange?.(true, "Processing...")
// Call Extension Host to stop recording and get audio
const response = await VoiceServiceClient.stopRecording(StopRecordingRequest.create({}))
if (!response.success) {
console.error("Failed to stop recording:", response.error)
setIsProcessing(false)
const errorMessage = response.error || "Failed to stop recording"
setError(errorMessage)
onProcessingStateChange?.(true, errorMessage)
return
}
if (!response.audioBase64) {
console.error("No audio data received")
setIsProcessing(false)
const errorMessage = "No audio data received"
setError(errorMessage)
onProcessingStateChange?.(true, errorMessage)
return
}
// Update processing state for transcription
onProcessingStateChange?.(true, "Transcribing...")
// Transcribe the audio using OpenAI Whisper
const transcriptionResponse = await VoiceServiceClient.transcribeAudio(
TranscribeAudioRequest.create({
audioBase64: response.audioBase64,
language: language,
}),
)
if (transcriptionResponse.error) {
console.error("Transcription error:", transcriptionResponse.error)
setError(transcriptionResponse.error)
// Show the error message in the UI
onProcessingStateChange?.(true, transcriptionResponse.error)
// Clear the error after a delay
setTimeout(() => {
setError(null)
onProcessingStateChange?.(false)
}, 5000)
} else if (transcriptionResponse.text) {
setError(null)
onTranscription(transcriptionResponse.text)
onProcessingStateChange?.(false)
}
} catch (error) {
console.error("Error stopping recording:", error)
const errorMessage = error instanceof Error ? error.message : "An error occurred"
setError(errorMessage)
onProcessingStateChange?.(true, errorMessage)
} finally {
setIsProcessing(false)
}
}, [onTranscription, onProcessingStateChange])
// Poll recording status while recording to update duration
useEffect(() => {
const pollRecordingStatus = async () => {
try {
const statusResponse = await VoiceServiceClient.getRecordingStatus(GetRecordingStatusRequest.create({}))
if (statusResponse.isRecording) {
setRecordingDuration(Math.floor(statusResponse.durationSeconds))
// Auto-stop if max duration reached
if (statusResponse.durationSeconds >= MAX_DURATION) {
stopRecording()
}
}
} catch (error) {
console.error("Error polling recording status:", error)
}
}
if (isRecording && !isProcessing) {
// Start polling immediately, then every second
pollRecordingStatus()
pollingIntervalRef.current = setInterval(pollRecordingStatus, 1000)
} else {
// Clear polling when not recording
if (pollingIntervalRef.current) {
clearInterval(pollingIntervalRef.current)
pollingIntervalRef.current = null
}
}
// Cleanup on unmount
return () => {
if (pollingIntervalRef.current) {
clearInterval(pollingIntervalRef.current)
pollingIntervalRef.current = null
}
}
}, [isRecording, isProcessing, stopRecording])
const handleClick = useCallback(() => {
if (disabled || isProcessing) return
if (error) return setError(null)
if (isRecording) {
stopRecording()
} else {
startRecording()
}
}, [isRecording, startRecording, stopRecording, disabled, isProcessing, error])
const getIconClass = () => {
if (isProcessing) return "codicon-loading"
if (isRecording) return "codicon-stop-circle"
if (error) return "codicon-error"
return "codicon-mic"
}
const getIconColor = () => {
if (isRecording) return "var(--vscode-errorForeground)"
if (error) return "var(--vscode-errorForeground)"
return ""
}
const getIconAnimation = () => {
if (isProcessing) return "animate-spin"
if (isRecording) return "animate-pulse"
return ""
}
const getIconAdjustment = () => {
if (isProcessing) return "mt-0"
if (isRecording) return "mt-1"
if (error) return "mt-1"
return "mt-0.5"
}
const getRecTooltipContent = () => {
if (isProcessing) return "Transcribing..."
if (isRecording) return `Stop Recording (${formatSeconds(recordingDuration)}/${formatSeconds(MAX_DURATION)})`
if (error) return `Error: ${error}`
return null
}
return (
<HeroTooltip content={getRecTooltipContent()} placement="top">
<div
className={`input-icon-button mr-1.5 text-base ${getIconAdjustment()} ${getIconAnimation()} ${disabled || isProcessing ? "disabled" : ""}`}
onClick={handleClick}
style={{
color: getIconColor(),
}}>
<span className={`codicon ${getIconClass()}`} />
</div>
</HeroTooltip>
)
}
export default VoiceRecorder
@@ -0,0 +1,16 @@
import styled from "styled-components"
const CollapsibleContent = styled.div<{ isOpen: boolean }>`
overflow: hidden;
transition:
max-height 0.3s ease-in-out,
opacity 0.3s ease-in-out,
margin-top 0.3s ease-in-out,
visibility 0.3s ease-in-out;
max-height: ${({ isOpen }) => (isOpen ? "1000px" : "0")};
opacity: ${({ isOpen }) => (isOpen ? 1 : 0)};
margin-top: ${({ isOpen }) => (isOpen ? "15px" : "0")};
visibility: ${({ isOpen }) => (isOpen ? "visible" : "hidden")};
`
export default CollapsibleContent
@@ -5,6 +5,7 @@ import { BROWSER_VIEWPORT_PRESETS } from "../../../../../src/shared/BrowserSetti
import { useExtensionState } from "../../../context/ExtensionStateContext"
import { BrowserServiceClient } from "../../../services/grpc-client"
import { EmptyRequest, StringRequest } from "@shared/proto/common"
import CollapsibleContent from "../CollapsibleContent"
import { updateBrowserSetting } from "../utils/settingsHandlers"
import { DebouncedTextField } from "../common/DebouncedTextField"
import Section from "../Section"
@@ -43,19 +44,6 @@ const ConnectionStatusIndicator = ({
)
}
const CollapsibleContent = styled.div<{ isOpen: boolean }>`
overflow: hidden;
transition:
max-height 0.3s ease-in-out,
opacity 0.3s ease-in-out,
margin-top 0.3s ease-in-out,
visibility 0.3s ease-in-out;
max-height: ${({ isOpen }) => (isOpen ? "1000px" : "0")}; // Sufficiently large height
opacity: ${({ isOpen }) => (isOpen ? 1 : 0)};
margin-top: ${({ isOpen }) => (isOpen ? "15px" : "0")};
visibility: ${({ isOpen }) => (isOpen ? "visible" : "hidden")};
`
export const BrowserSettingsSection: React.FC<BrowserSettingsSectionProps> = ({ renderSectionHeader }) => {
const { browserSettings } = useExtensionState()
const [isCheckingConnection, setIsCheckingConnection] = useState(false)
@@ -2,8 +2,11 @@ import { VSCodeCheckbox, VSCodeDropdown, VSCodeOption } from "@vscode/webview-ui
import { useExtensionState } from "@/context/ExtensionStateContext"
import { memo } from "react"
import { OpenAIReasoningEffort } from "@shared/ChatSettings"
import { SUPPORTED_DICTATION_LANGUAGES } from "@shared/DictationSettings"
// import CollapsibleContent from "../CollapsibleContent"
import { updateSetting } from "../utils/settingsHandlers"
import { convertChatSettingsToProtoChatSettings } from "@shared/proto-conversions/state/chat-settings-conversion"
import { convertDictationSettingsToProtoDictationSettings } from "@shared/proto-conversions/state/dictation-settings-conversion"
import Section from "../Section"
interface FeatureSettingsSectionProps {
@@ -11,8 +14,14 @@ interface FeatureSettingsSectionProps {
}
const FeatureSettingsSection = ({ renderSectionHeader }: FeatureSettingsSectionProps) => {
const { enableCheckpointsSetting, mcpMarketplaceEnabled, mcpRichDisplayEnabled, mcpResponsesCollapsed, chatSettings } =
useExtensionState()
const {
enableCheckpointsSetting,
mcpMarketplaceEnabled,
mcpRichDisplayEnabled,
mcpResponsesCollapsed,
chatSettings,
dictationSettings,
} = useExtensionState()
const handleReasoningEffortChange = (newValue: OpenAIReasoningEffort) => {
if (!chatSettings) return
@@ -103,9 +112,65 @@ const FeatureSettingsSection = ({ renderSectionHeader }: FeatureSettingsSectionP
<VSCodeOption value="high">High</VSCodeOption>
</VSCodeDropdown>
<p className="text-xs mt-[5px] text-[var(--vscode-descriptionForeground)]">
Reasoning effort for the OpenAI family of models(applies to all OpenAI model providers)
Reasoning effort for the OpenAI family of models (applies to all OpenAI model providers)
</p>
</div>
<div style={{ marginTop: 10 }}>
<div>
<VSCodeCheckbox
checked={dictationSettings?.voiceRecordingEnabled}
onChange={(e: any) => {
const checked = e.target.checked === true
const updatedDictationSettings = {
...dictationSettings,
voiceRecordingEnabled: checked,
}
const protoDictationSettings =
convertDictationSettingsToProtoDictationSettings(updatedDictationSettings)
updateSetting("dictationSettings", protoDictationSettings)
}}>
Enable Dictation
</VSCodeCheckbox>
<p className="text-xs text-[var(--vscode-descriptionForeground)] mt-1">
Enables voice recording with automatic transcription. Requires a Cline Account for the
transcription model.
</p>
</div>
{/* TODO: Fix and use CollapsibleContent, the animation is good but it breaks the dropdown
<CollapsibleContent isOpen={dictationSettings?.voiceRecordingEnabled}> */}
<div className={dictationSettings?.voiceRecordingEnabled ? "mt-4" : "hidden"}>
<label
htmlFor="dictation-language-dropdown"
className="block text-sm font-medium text-[var(--vscode-foreground)] mb-1">
Dictation Language
</label>
<VSCodeDropdown
id="dictation-language-dropdown"
currentValue={dictationSettings?.dictationLanguage || "en"}
onChange={(e: any) => {
const newValue = e.target.value
const updatedDictationSettings = {
...dictationSettings,
dictationLanguage: newValue,
}
const protoDictationSettings =
convertDictationSettingsToProtoDictationSettings(updatedDictationSettings)
updateSetting("dictationSettings", protoDictationSettings)
}}
className="w-full">
{SUPPORTED_DICTATION_LANGUAGES.map((language) => (
<VSCodeOption key={language.code} value={language.code} className="py-0.5">
{language.name}
</VSCodeOption>
))}
</VSCodeDropdown>
<p className="text-xs mt-1 text-[var(--vscode-descriptionForeground)]">
The language you want to speak to the Dictation service in. Separate from preferred UI language.
</p>
</div>
{/* </CollapsibleContent> */}
</div>
</div>
</Section>
</div>
@@ -16,6 +16,7 @@ import { convertProtoMcpServersToMcpServers } from "@shared/proto-conversions/mc
import { DEFAULT_AUTO_APPROVAL_SETTINGS } from "@shared/AutoApprovalSettings"
import { DEFAULT_BROWSER_SETTINGS, BrowserSettings } from "@shared/BrowserSettings"
import { ChatSettings, DEFAULT_CHAT_SETTINGS } from "@shared/ChatSettings"
import { DictationSettings, DEFAULT_DICTATION_SETTINGS } from "@shared/DictationSettings"
import { DEFAULT_PLATFORM, ExtensionMessage, ExtensionState } from "@shared/ExtensionMessage"
import { TelemetrySetting } from "@shared/TelemetrySetting"
import { findLastIndex } from "@shared/array"
@@ -68,6 +69,7 @@ interface ExtensionStateContextType extends ExtensionState {
setTerminalOutputLineLimit: (value: number) => void
setDefaultTerminalProfile: (value: string) => void
setChatSettings: (value: ChatSettings) => void
setDictationSettings: (value: DictationSettings) => void
setMcpServers: (value: McpServer[]) => void
setRequestyModels: (value: Record<string, ModelInfo>) => void
setGlobalClineRulesToggles: (toggles: Record<string, boolean>) => void
@@ -184,6 +186,7 @@ export const ExtensionStateContextProvider: React.FC<{
shouldShowAnnouncement: false,
autoApprovalSettings: DEFAULT_AUTO_APPROVAL_SETTINGS,
browserSettings: DEFAULT_BROWSER_SETTINGS,
dictationSettings: DEFAULT_DICTATION_SETTINGS,
chatSettings: DEFAULT_CHAT_SETTINGS,
platform: DEFAULT_PLATFORM,
telemetrySetting: "unset",
@@ -270,8 +273,13 @@ export const ExtensionStateContextProvider: React.FC<{
const currentVersion = prevState.autoApprovalSettings?.version ?? 1
const shouldUpdateAutoApproval = incomingVersion > currentVersion
// Always preserve chat settings from the current state
// This prevents the backend from overwriting user changes
const chatSettings = prevState.chatSettings || stateData.chatSettings
const newState = {
...stateData,
chatSettings, // Use preserved or incoming chat settings
autoApprovalSettings: shouldUpdateAutoApproval
? stateData.autoApprovalSettings
: prevState.autoApprovalSettings,
@@ -280,9 +288,7 @@ export const ExtensionStateContextProvider: React.FC<{
// Update welcome screen state based on API configuration
setShowWelcome(!newState.welcomeViewCompleted)
setDidHydrateState(true)
console.log("[DEBUG] returning new state in ESC")
return newState
})
} catch (error) {
@@ -816,6 +822,11 @@ export const ExtensionStateContextProvider: React.FC<{
...prevState,
browserSettings: value,
})),
setDictationSettings: (value: DictationSettings) =>
setState((prevState) => ({
...prevState,
dictationSettings: value,
})),
}
return <ExtensionStateContext.Provider value={contextValue}>{children}</ExtensionStateContext.Provider>
-5
View File
@@ -6,11 +6,6 @@
@config "../tailwind.config.js";
/* Import Azeret Mono font from local package */
@import "@fontsource/azeret-mono/300.css";
@import "@fontsource/azeret-mono/400.css";
@import "@fontsource/azeret-mono/700.css";
textarea:focus {
outline: 1.5px solid var(--vscode-focusBorder, #007fd4);
}
+5
View File
@@ -12,6 +12,8 @@ import { WebServiceDefinition } from "@shared/proto/web"
import { ModelsServiceDefinition } from "@shared/proto/models"
import { SlashServiceDefinition } from "@shared/proto/slash"
import { UiServiceDefinition } from "@shared/proto/ui"
import { VoiceServiceDefinition } from "@shared/proto/voice"
const AccountServiceClient = createGrpcClient(AccountServiceDefinition)
const BrowserServiceClient = createGrpcClient(BrowserServiceDefinition)
const CheckpointsServiceClient = createGrpcClient(CheckpointsServiceDefinition)
@@ -23,6 +25,8 @@ const WebServiceClient = createGrpcClient(WebServiceDefinition)
const ModelsServiceClient = createGrpcClient(ModelsServiceDefinition)
const SlashServiceClient = createGrpcClient(SlashServiceDefinition)
const UiServiceClient = createGrpcClient(UiServiceDefinition)
const VoiceServiceClient = createGrpcClient(VoiceServiceDefinition)
export {
AccountServiceClient,
BrowserServiceClient,
@@ -35,5 +39,6 @@ export {
ModelsServiceClient,
SlashServiceClient,
UiServiceClient,
VoiceServiceClient,
}
//# sourceMappingURL=grpc-client.js.map
+13
View File
@@ -62,3 +62,16 @@ export function formatSize(bytes?: number) {
return prettyBytes(bytes)
}
export function formatSeconds(seconds?: number): string {
if (seconds === undefined) {
return "--:--"
}
const mins = Math.floor(seconds / 60)
const secs = Math.floor(seconds % 60)
.toString()
.padStart(2, "0")
return `${mins}:${secs}`
}
+1 -5
View File
@@ -4,11 +4,7 @@ const { heroui } = require("@heroui/react")
module.exports = {
content: ["./src/**/*.{js,ts,jsx,tsx,mdx}", "./node_modules/@heroui/theme/dist/**/*.{js,ts,jsx,tsx}"],
theme: {
extend: {
fontFamily: {
"azeret-mono": ['"Azeret Mono"', "monospace"],
},
},
extend: {},
},
darkMode: "class",
plugins: [