Compare commits

...

5 Commits

Author SHA1 Message Date
Juan Pablo 1136f25bdf feat(discuss-mode): enhance conversational interaction and audio management
- Updated Discuss Mode guidelines to promote a more natural, conversational tone in user interactions.
- Added logic to display a message indicating when Discuss Mode is enabled, emphasizing voice-enabled conversation.
- Implemented UI changes to hide AI-generated text and reasoning in Discuss Mode for a cleaner user experience.
- Enhanced audio playback management with a new mechanism for handling audio elements, ensuring smoother playback and user interaction.
- Adjusted speech speed limits in settings to align with API constraints, improving overall voice synthesis quality.
2025-11-09 17:49:41 -08:00
Juan Pablo 1630f37b3f feat(discuss-mode): enhance voice input functionality with recording and transcription features
- Implemented voice recording capabilities in Discuss Mode, allowing users to record audio directly from the chat interface.
- Integrated ElevenLabs Speech-to-Text API for real-time audio transcription, improving user interaction.
- Added visual indicators for recording status and duration, enhancing user experience.
- Developed custom hooks for managing voice input state and interactions, ensuring smooth functionality.
- Updated UI components to support voice input, including a dedicated voice input button and recording indicator.
2025-11-09 15:23:26 -08:00
Juan Pablo 6ed664942a feat(discuss-mode): enhance voice interaction capabilities with TTS features
- Introduced TTS service with ElevenLabs integration for speech synthesis.
- Added gRPC methods for synthesizing speech, validating API keys, and retrieving available voices.
- Implemented Discuss Mode settings, allowing users to configure voice options and speech speed.
- Developed UI components for Discuss Mode, including toggles and indicators for voice interactions.
- Enhanced audio playback management with dedicated components for message audio and conversation controls.
- Integrated automatic TTS playback for messages in Discuss Mode, improving user experience.
- Added error handling and validation for TTS API key configuration.
2025-11-09 10:24:33 -08:00
Juan Pablo 6dffa3f6ed feat(discuss-mode): implement voice-enabled interactive planning with TTS integration
- Added Discuss Mode Implementation Plan document outlining project goals and architecture.
- Created TTS service with ElevenLabs integration for speech synthesis.
- Implemented gRPC methods for synthesizing speech and retrieving available voices.
- Developed AudioPlayer component for playback of synthesized audio.
- Established base and ElevenLabs TTS provider classes for handling speech requests.
- Included error handling and validation for TTS API key.
2025-11-08 20:11:39 -08:00
Juan Pablo 8b1ed0113c Discussions 2025-11-08 20:11:22 -08:00
50 changed files with 3741 additions and 57 deletions
+556
View File
@@ -0,0 +1,556 @@
# Discuss Mode Implementation Plan
> **Voice-enabled interactive planning conversations for Cline**
> Last Updated: November 8, 2025
> Status: 🚧 In Progress
## 📋 Quick Status
- **Start Date:** November 8, 2025
- **Target Completion:** TBD
- **Current Phase:** Backend Infrastructure Complete (Phases 1-3)
- **Overall Progress:** 35% (15/43 tasks complete)
## 🎯 Project Goals
Enable natural voice conversations with Cline during Plan Mode to create a collaborative planning experience where users can discuss requirements before implementation begins.
### Core Features
- ✅ Voice input (already exists)
- ⬜ Voice output via ElevenLabs TTS
- ⬜ Auto-continue conversation flow
- ⬜ Interactive question-asking behavior
- ⬜ Plan completion detection
- ⬜ Smooth transition to Act Mode
### Scope
- **In Scope:** Voice conversations in Plan Mode only
- **Out of Scope:** Voice during Act Mode (tool execution)
---
## 🏗️ Architecture Overview
### System Components
```
┌─────────────────────────────────────────────────────────────┐
│ Discuss Mode Stack │
├─────────────────────────────────────────────────────────────┤
│ │
│ Frontend (React/TypeScript) │
│ ├── DiscussModeToggle.tsx (UI control) │
│ ├── AudioPlayer.tsx (TTS playback) │
│ ├── VoiceConversationControls.tsx (speaking indicators) │
│ └── VoiceSettingsSection.tsx (settings UI) │
│ │
│ Backend (Node.js/TypeScript) │
│ ├── TextToSpeechService.ts (TTS orchestration) │
│ ├── ElevenLabsProvider.ts (ElevenLabs API) │
│ ├── Task.say() modifications (TTS trigger) │
│ └── System Prompt additions (discuss behavior) │
│ │
│ Infrastructure │
│ ├── proto/tts.proto (gRPC definitions) │
│ ├── Controller handlers (gRPC endpoints) │
│ └── State management (discuss mode state) │
│ │
└─────────────────────────────────────────────────────────────┘
```
### Data Flow
```
User Speech → VoiceRecorder → Transcription → Cline (Plan Mode)
[Processes with
"Discuss Mode" prompt]
Response Text → TextToSpeechService
Audio Buffer → AudioPlayer → Speaker
[On playback complete]
Auto-start VoiceRecorder (if enabled)
```
---
## 📝 Implementation Checklist
### Phase 1: System Prompt & Behavior (Est: 2-3 hours)
- [ ] 1.1 Read existing Plan Mode system prompt
- [ ] 1.2 Design "Discuss Mode" prompt additions
- [ ] 1.3 Add conversational guidelines
- [ ] 1.4 Add proactive questioning instructions
- [ ] 1.5 Add plan completion signal instructions
- [ ] 1.6 Test prompt changes with existing Plan Mode
**Files to Modify:**
- `src/core/prompts/system-prompt.ts`
- `src/core/prompts/system-prompt-legacy/` (if needed)
### Phase 2: TTS Service Backend (Est: 4-6 hours)
- [ ] 2.1 Create `src/services/tts/` directory structure
- [ ] 2.2 Implement `BaseTTSProvider` interface
- [ ] 2.3 Implement `ElevenLabsProvider.ts`
- [ ] API client setup
- [ ] Voice list endpoint
- [ ] Text-to-speech synthesis
- [ ] Error handling
- [ ] 2.4 Implement `TextToSpeechService.ts`
- [ ] Provider factory pattern
- [ ] Audio buffer management
- [ ] Streaming support (optional)
- [ ] 2.5 Add TTS configuration to `src/shared/api.ts`
- [ ] 2.6 Add ElevenLabs API key to secrets storage
- [ ] 2.7 Unit tests for TTS service
**New Files:**
```
src/services/tts/
├── TextToSpeechService.ts
├── providers/
│ ├── BaseTTSProvider.ts
│ ├── ElevenLabsProvider.ts
│ └── OpenAITTSProvider.ts (future)
└── __tests__/
└── TextToSpeechService.test.ts
```
### Phase 3: Protobuf & gRPC (Est: 2-3 hours)
- [ ] 3.1 Create `proto/tts.proto` definition
- [ ] 3.2 Define `TtsService` with methods:
- [ ] `synthesizeSpeech()`
- [ ] `getAvailableVoices()`
- [ ] `updateVoiceSettings()`
- [ ] 3.3 Run proto compilation: `npm run protos`
- [ ] 3.4 Implement gRPC handlers in `src/core/controller/tts/`
- [ ] `synthesizeSpeech.ts`
- [ ] `getAvailableVoices.ts`
- [ ] 3.5 Generate client in `webview-ui/src/services/grpc-client.ts`
**New Files:**
```
proto/tts.proto
src/core/controller/tts/
├── synthesizeSpeech.ts
└── getAvailableVoices.ts
```
### Phase 4: Audio Player Component (Est: 3-4 hours)
- [ ] 4.1 Create `AudioPlayer.tsx` component
- [ ] Audio element management
- [ ] Playback controls
- [ ] Loading states
- [ ] Error handling
- [ ] 4.2 Create audio queue system
- [ ] 4.3 Add speaking animation/indicator
- [ ] 4.4 Integrate into chat message component
- [ ] 4.5 Add auto-play functionality
- [ ] 4.6 Add callback for playback completion
**New Files:**
```
webview-ui/src/components/chat/
├── AudioPlayer.tsx
├── VoiceConversationControls.tsx
└── SpeakingIndicator.tsx
```
### Phase 5: Discuss Mode Integration (Est: 4-5 hours)
- [ ] 5.1 Add `discussModeEnabled` to global state
- [ ] 5.2 Add `voiceModeSettings` to state
- [ ] 5.3 Modify `Task.say()` in `src/core/task/index.ts`:
- [ ] Check if discuss mode is enabled
- [ ] Check if current mode is "plan"
- [ ] Trigger TTS for assistant text
- [ ] Queue audio for playback
- [ ] 5.4 Implement auto-continue logic:
- [ ] Detect audio playback completion
- [ ] Auto-start voice recorder
- [ ] Handle errors gracefully
- [ ] 5.5 Add mode switching guard (auto-disable on Act Mode)
**Files to Modify:**
```
src/core/task/index.ts
src/core/controller/index.ts
src/core/storage/StateManager.ts
```
### Phase 6: UI Components (Est: 3-4 hours)
- [ ] 6.1 Create `DiscussModeToggle.tsx`
- [ ] Toggle button with icon
- [ ] Mode indicator badge
- [ ] Tooltip with description
- [ ] Disable in Act Mode
- [ ] 6.2 Add discuss mode controls to chat header
- [ ] 6.3 Create plan completion UI:
- [ ] "Plan Ready" indicator
- [ ] "Switch to Act Mode" button
- [ ] "Continue Discussing" option
- [ ] 6.4 Add voice conversation status indicators:
- [ ] "🎤 Listening..." when recording
- [ ] "🗣️ Speaking..." when playing audio
- [ ] "💭 Thinking..." when processing
**New Files:**
```
webview-ui/src/components/discuss-mode/
├── DiscussModeToggle.tsx
├── PlanCompletionCard.tsx
└── ConversationStatusIndicator.tsx
```
### Phase 7: Settings Panel (Est: 2-3 hours)
- [ ] 7.1 Extend `VoiceSettingsSection.tsx`
- [ ] 7.2 Add TTS provider selection (ElevenLabs)
- [ ] 7.3 Add ElevenLabs API key input
- [ ] 7.4 Add voice selection dropdown:
- [ ] Fetch voices from ElevenLabs
- [ ] Voice preview button
- [ ] Voice descriptions
- [ ] 7.5 Add speech rate slider
- [ ] 7.6 Add auto-speak toggle
- [ ] 7.7 Add auto-listen toggle
- [ ] 7.8 Save settings to state
**Files to Modify:**
```
webview-ui/src/components/settings/sections/VoiceSettingsSection.tsx
```
### Phase 8: Testing & Polish (Est: 3-4 hours)
- [ ] 8.1 End-to-end testing:
- [ ] Voice input → TTS output flow
- [ ] Auto-continue conversation
- [ ] Plan completion detection
- [ ] Mode switching behavior
- [ ] 8.2 Error handling:
- [ ] API key missing
- [ ] Network failures
- [ ] Audio playback errors
- [ ] Rate limiting (ElevenLabs)
- [ ] 8.3 Edge cases:
- [ ] Empty responses
- [ ] Very long responses (chunking)
- [ ] Interrupted speech
- [ ] Rapid mode switching
- [ ] 8.4 Performance optimization:
- [ ] Audio caching
- [ ] Queue management
- [ ] Memory cleanup
- [ ] 8.5 UX polish:
- [ ] Smooth animations
- [ ] Clear status indicators
- [ ] Helpful error messages
- [ ] Onboarding tooltip
---
## 📁 File Structure
```
cline/
├── proto/
│ └── tts.proto [NEW]
├── src/
│ ├── core/
│ │ ├── controller/
│ │ │ └── tts/ [NEW]
│ │ │ ├── synthesizeSpeech.ts
│ │ │ └── getAvailableVoices.ts
│ │ ├── prompts/
│ │ │ └── system-prompt.ts [MODIFY]
│ │ └── task/
│ │ └── index.ts [MODIFY]
│ │
│ ├── services/
│ │ └── tts/ [NEW]
│ │ ├── TextToSpeechService.ts
│ │ └── providers/
│ │ ├── BaseTTSProvider.ts
│ │ └── ElevenLabsProvider.ts
│ │
│ └── shared/
│ └── api.ts [MODIFY]
└── webview-ui/
└── src/
├── components/
│ ├── chat/
│ │ ├── AudioPlayer.tsx [NEW]
│ │ └── VoiceConversationControls.tsx [NEW]
│ ├── discuss-mode/ [NEW]
│ │ ├── DiscussModeToggle.tsx
│ │ └── PlanCompletionCard.tsx
│ └── settings/
│ └── VoiceSettingsSection.tsx [MODIFY]
└── services/
└── grpc-client.ts [MODIFY]
```
---
## 🔧 Technical Details
### ElevenLabs Integration
**API Endpoints:**
- Text-to-Speech: `POST https://api.elevenlabs.io/v1/text-to-speech/{voice_id}`
- Get Voices: `GET https://api.elevenlabs.io/v1/voices`
**Authentication:**
```typescript
headers: {
'xi-api-key': API_KEY,
'Content-Type': 'application/json'
}
```
**Request Format:**
```typescript
{
text: string,
model_id: "eleven_multilingual_v2",
voice_settings: {
stability: 0.5,
similarity_boost: 0.75
}
}
```
**Response:** Audio file (MP3 format)
### State Management
**New Global State Keys:**
```typescript
interface GlobalState {
discussModeEnabled: boolean
voiceModeSettings: {
ttsProvider: "elevenlabs" | "openai"
elevenLabsApiKey?: string
selectedVoice: string
speechRate: number
autoSpeak: boolean
autoListen: boolean
}
}
```
### Protobuf Schema
```protobuf
syntax = "proto3";
package cline.tts;
service TtsService {
rpc synthesizeSpeech(SynthesizeRequest) returns (SynthesizeResponse);
rpc getAvailableVoices(EmptyRequest) returns (VoicesResponse);
}
message SynthesizeRequest {
string text = 1;
string voice_id = 2;
optional float speech_rate = 3;
}
message SynthesizeResponse {
bytes audio_data = 1;
optional string error = 2;
}
message Voice {
string id = 1;
string name = 2;
string description = 3;
}
message VoicesResponse {
repeated Voice voices = 1;
optional string error = 2;
}
```
---
## 🧪 Testing Strategy
### Unit Tests
- [ ] TTS Service provider selection
- [ ] Audio buffer handling
- [ ] Error handling for API failures
- [ ] State management for discuss mode
### Integration Tests
- [ ] gRPC endpoint communication
- [ ] Full voice input → TTS output flow
- [ ] Mode switching behavior
- [ ] Settings persistence
### Manual Testing Scenarios
1. **Happy Path:**
- Enable Discuss Mode
- Ask initial question via voice
- Cline asks clarifying questions
- Iterative discussion
- Plan completion and approval
- Switch to Act Mode
2. **Error Cases:**
- Missing API key
- Network failure during TTS
- Invalid voice selection
- Audio playback failure
3. **Edge Cases:**
- Mode switching during playback
- Rapid successive voice inputs
- Very long responses (>1000 chars)
- Empty or nonsense responses
---
## 🐛 Known Issues & Future Improvements
### Known Limitations
- ElevenLabs rate limits (adjust queue as needed)
- No offline mode (requires internet for TTS)
- Audio latency depends on network speed
### Future Enhancements
- [ ] Multiple TTS provider support (OpenAI, Azure)
- [ ] Voice cloning integration
- [ ] Conversation history playback
- [ ] Export voice conversations
- [ ] Voice command shortcuts
- [ ] Ambient mode (minimal UI, voice-first)
- [ ] Multi-language support
- [ ] Emotion/tone control for TTS
---
## 📊 Progress Tracking
### Time Estimates vs Actual
| Phase | Estimated | Actual | Status |
|-------|-----------|--------|--------|
| Phase 1: System Prompt | 2-3h | ~1h | ✅ Complete |
| Phase 2: TTS Backend | 4-6h | ~2h | ✅ Complete |
| Phase 3: Protobuf/gRPC | 2-3h | ~1h | ✅ Complete |
| Phase 4: Audio Player | 3-4h | - | ⬜ Not Started |
| Phase 5: Integration | 4-5h | - | ⬜ Not Started |
| Phase 6: UI Components | 3-4h | - | ⬜ Not Started |
| Phase 7: Settings | 2-3h | - | ⬜ Not Started |
| Phase 8: Testing | 3-4h | - | ⬜ Not Started |
| **Total** | **23-32h** | **~4h** | 🟡 35% Complete |
### Sprint Log
_Add daily progress notes here as development proceeds_
**November 8, 2025 - Session 1:**
- ✅ Completed: Initial planning and architecture design
- ✅ Completed: Created comprehensive implementation plan document
- ✅ Completed: **Phase 1 - System Prompt & Behavior**
- Modified `src/core/prompts/system-prompt/components/act_vs_plan_mode.ts`
- Added Discuss Mode conversational guidelines
- Implemented proactive questioning behavior
- Added plan completion signal instructions
- ✅ Completed: **Phase 2 - TTS Service Backend**
- Created complete TTS service architecture in `src/services/tts/`
- Implemented `BaseTTSProvider.ts` abstract base class
- Implemented `ElevenLabsProvider.ts` with full API integration
- Implemented `TextToSpeechService.ts` orchestration layer
- Added voice synthesis, voice selection, and validation
- ✅ Completed: **Phase 3 - Protobuf & gRPC**
- Created `proto/tts.proto` with complete service definitions
- Successfully compiled protobuf definitions
- Implemented gRPC handlers in `src/core/controller/tts/`:
- `synthesizeSpeech.ts` for text-to-speech synthesis
- `getAvailableVoices.ts` for voice listing
- Integrated TTS service into Controller class with getter method
- 📝 Status: Backend infrastructure complete (35% overall progress)
- 🎯 Next: Begin Phase 4 (Audio Player Component implementation)
---
## 📚 References
- [ElevenLabs API Docs](https://elevenlabs.io/docs)
- [Cline Architecture Guide](.clinerules/cline-overview.md)
- [Protobuf Development Guide](.clinerules/protobuf-development.md)
- [Existing Voice Input Implementation](src/services/dictation/)
---
## 👥 Team Notes
_Use this section for brainstorming, decisions, and Q&A during development_
### Design Decisions
- **Why ElevenLabs?** High-quality, natural-sounding voices with proven API reliability
- **Why Plan Mode only?** Clean separation of concerns - discussion vs execution. This creates a natural workflow where users discuss in Plan Mode then execute silently in Act Mode
- **Why auto-continue?** Enables natural conversation flow without manual button clicking between exchanges
### Open Questions
- [ ] Should we support offline TTS? (browser TTS API as fallback?)
- [ ] Max response length before chunking audio?
- [ ] Voice selection defaults for different languages?
- [ ] Should we cache frequently used phrases/responses?
### Example Discuss Mode Session
```
User clicks "Discuss Mode" button in Plan Mode
👤 User: [Speaks] "I need to add authentication to my app"
🤖 Cline: [Speaks] "I'd be happy to help with that! Let me ask a few questions
so I can create the right solution for you. First, what kind of
authentication are you thinking - email/password, OAuth with
providers like Google or GitHub, or something else?"
👤 User: [Speaks] "Email and password for now, but I might add OAuth later"
🤖 Cline: [Speaks] "Perfect. And do you already have a backend API, or should
I create one? Also, what framework is your frontend using?"
👤 User: [Speaks] "I have a Node.js backend with Express, and React on the frontend"
🤖 Cline: [Speaks] "Great! A couple more things - do you need features like
password reset, email verification, or 'remember me' functionality?"
👤 User: [Speaks] "Yes, all of those would be helpful"
🤖 Cline: [Speaks] "Perfect! I have a clear plan now. Here's what I'll do:
I'll create:
1. Backend authentication routes with JWT tokens
2. Password hashing with bcrypt
3. Email service for verification and password reset
4. React components for login, signup, and password reset pages
5. Protected route wrapper for authenticated pages
This will integrate with your existing Express backend and React
frontend. The authentication will be secure and production-ready.
Does this sound like what you need? If so, I'm ready to switch to
Act Mode and start building!"
👤 User: [Speaks] "Yes, that sounds perfect!"
[UI shows: ✅ Plan Complete - Switch to Act Mode?]
[User clicks button → Switches to Act Mode]
[Discuss Mode auto-disables]
[Cline begins implementation silently]
```
---
*This document will be updated throughout implementation. Last updated: November 8, 2025*
+4 -22
View File
@@ -1,12 +1,12 @@
{
"name": "claude-dev",
"version": "3.36.0",
"version": "3.36.1",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "claude-dev",
"version": "3.36.0",
"version": "3.36.1",
"license": "Apache-2.0",
"dependencies": {
"@anthropic-ai/sdk": "^0.37.0",
@@ -151,6 +151,8 @@
},
"node_modules/@anthropic-ai/sdk": {
"version": "0.37.0",
"resolved": "https://registry.npmjs.org/@anthropic-ai/sdk/-/sdk-0.37.0.tgz",
"integrity": "sha512-tHjX2YbkUBwEgg0JZU3EFSSAQPoK4qQR/NFYa8Vtzd5UAyXzZksCw2In69Rml4R/TyHPBfRYaLK35XiOe33pjw==",
"license": "MIT",
"dependencies": {
"@types/node": "^18.11.18",
@@ -177,26 +179,6 @@
"google-auth-library": "^9.4.2"
}
},
"node_modules/@anthropic-ai/vertex-sdk/node_modules/@anthropic-ai/sdk": {
"version": "0.38.0",
"license": "MIT",
"dependencies": {
"@types/node": "^18.11.18",
"@types/node-fetch": "^2.6.4",
"abort-controller": "^3.0.0",
"agentkeepalive": "^4.2.1",
"form-data-encoder": "1.7.2",
"formdata-node": "^4.3.2",
"node-fetch": "^2.6.7"
}
},
"node_modules/@anthropic-ai/vertex-sdk/node_modules/@types/node": {
"version": "18.19.76",
"license": "MIT",
"dependencies": {
"undici-types": "~5.26.4"
}
},
"node_modules/@aws-crypto/crc32": {
"version": "5.2.0",
"resolved": "https://registry.npmjs.org/@aws-crypto/crc32/-/crc32-5.2.0.tgz",
+68
View File
@@ -0,0 +1,68 @@
syntax = "proto3";
package cline;
option go_package = "github.com/cline/grpc-go/cline";
option java_multiple_files = true;
option java_package = "bot.cline.proto";
// TTS Service - Handles text-to-speech synthesis
service TtsService {
// Synthesize speech from text
rpc SynthesizeSpeech(SynthesizeRequest) returns (SynthesizeResponse);
// Get list of available voices for current provider
rpc GetAvailableVoices(GetVoicesRequest) returns (VoicesResponse);
// Validate API key for TTS provider
rpc ValidateApiKey(ValidateApiKeyRequest) returns (ValidateApiKeyResponse);
// Check if API key is configured
rpc CheckApiKeyConfigured(GetVoicesRequest) returns (ValidateApiKeyResponse);
}
// Request to synthesize speech
message SynthesizeRequest {
string text = 1;
string voice_id = 2;
optional float speed = 3;
optional float stability = 4;
optional float similarity_boost = 5;
}
// Response containing synthesized audio
message SynthesizeResponse {
bytes audio_data = 1;
string content_type = 2;
optional string error = 3;
}
// Request to get available voices
message GetVoicesRequest {
// Empty for now, could add filtering options later
}
// Information about a voice
message Voice {
string id = 1;
string name = 2;
optional string description = 3;
optional string preview_url = 4;
}
// Response containing list of voices
message VoicesResponse {
repeated Voice voices = 1;
optional string error = 2;
}
// Request to validate API key
message ValidateApiKeyRequest {
string api_key = 1; // The API key to validate
}
// Response from API key validation
message ValidateApiKeyResponse {
bool is_valid = 1;
optional string error = 2;
}
+14
View File
@@ -264,4 +264,18 @@ service UiService {
// Opens the Cline walkthrough
rpc openWalkthrough(EmptyRequest) returns (Empty);
// Toggle discuss mode on/off
rpc setDiscussModeEnabled(BooleanRequest) returns (BooleanRequest);
// Update discuss mode voice settings
rpc updateDiscussVoiceSettings(DiscussVoiceSettingsRequest) returns (Empty);
}
// Request to update discuss mode voice settings
message DiscussVoiceSettingsRequest {
optional string selected_voice = 1;
optional double speech_speed = 2;
optional bool auto_speak = 3;
optional bool auto_listen = 4;
}
+8 -2
View File
@@ -119,7 +119,8 @@ async function generateVscodeProtobusServers(protobusServices) {
imports.push(`// ${domain} Service`)
servers.push(`const ${serviceName}Handlers: serviceTypes.${serviceName}Handlers = {`)
for (const [rpcName, _rpc] of Object.entries(def.service)) {
imports.push(`import { ${rpcName} } from "@core/controller/${dir}/${rpcName}"`)
const filePath = toFilePath(rpcName)
imports.push(`import { ${rpcName} } from "@core/controller/${dir}/${filePath}"`)
servers.push(` ${rpcName}: ${rpcName},`)
}
servers.push(`} \n`)
@@ -156,7 +157,8 @@ async function generateStandaloneProtobusServiceSetup(protobusServices) {
handlerSetup.push(` // ${domain} Service`)
handlerSetup.push(` server.addService(cline.${name}Service, {`)
for (const [rpcName, rpc] of Object.entries(def.service)) {
imports.push(`import { ${rpcName} } from "@core/controller/${dir}/${rpcName}"`)
const filePath = toFilePath(rpcName)
imports.push(`import { ${rpcName} } from "@core/controller/${dir}/${filePath}"`)
const requestType = "cline." + rpc.requestType.type.name
const responseType = "cline." + rpc.responseType.type.name
if (rpc.requestStream) {
@@ -204,6 +206,10 @@ function getDirName(serviceName) {
const domain = getDomainName(serviceName)
return domain.charAt(0).toLowerCase() + domain.slice(1)
}
function toFilePath(rpcName) {
// Convert PascalCase to camelCase for file paths
return rpcName.charAt(0).toLowerCase() + rpcName.slice(1)
}
// Only run main if this script is executed directly
if (import.meta.url === `file://${process.argv[1]}`) {
@@ -15,6 +15,9 @@ export const cancelRecording = async (controller: Controller): Promise<Recording
let errorMessage = ""
let isSuccess = true
try {
// Clear the silence detection callback
audioRecordingService.setSilenceDetectedCallback(null)
const result = await audioRecordingService.cancelRecording()
isSuccess = !!result?.success
errorMessage = result?.error ?? ""
@@ -117,6 +117,12 @@ export const startRecording = async (controller: Controller): Promise<RecordingR
throw new Error("Please sign in to your Cline Account to use Dictation.")
}
// Set up silence detection callback to auto-stop recording
audioRecordingService.setSilenceDetectedCallback(async () => {
// Automatically stop recording when silence is detected
await audioRecordingService.stopRecording()
})
// Attempt to start recording
const result = await audioRecordingService.startRecording()
@@ -129,6 +135,9 @@ export const startRecording = async (controller: Controller): Promise<RecordingR
})
}
// Clear callback on failure
audioRecordingService.setSilenceDetectedCallback(null)
// Check if the error is due to missing dependencies
const platform = os.platform() as keyof typeof AUDIO_PROGRAM_CONFIG
const config = AUDIO_PROGRAM_CONFIG[platform]
@@ -14,6 +14,9 @@ export const stopRecording = async (controller: Controller): Promise<RecordedAud
const recordingDuration = recordingStatus.durationSeconds * 1000 // Convert to milliseconds
try {
// Clear the silence detection callback
audioRecordingService.setSilenceDetectedCallback(null)
const result = await audioRecordingService.stopRecording()
telemetryService.captureVoiceRecordingStopped(taskId, recordingDuration, result.success, process.platform)
@@ -19,8 +19,36 @@ export const transcribeAudio = async (controller: Controller, request: Transcrib
telemetryService.captureVoiceTranscriptionStarted(taskId, request.language ?? "en")
try {
// Transcribe the audio
const result = await getVoiceTranscriptionService().transcribeAudio(request.audioBase64, request.language ?? "en")
// Try ElevenLabs STT first if API key is available
const apiKey = await controller.context.secrets.get("elevenLabsApiKey")
let result: { text?: string; error?: string } | null = null
if (apiKey) {
try {
const { ElevenLabsProvider } = await import("@/services/tts/providers/ElevenLabsProvider")
const provider = new ElevenLabsProvider(apiKey)
// Convert base64 to Buffer
const audioBuffer = Buffer.from(request.audioBase64, "base64")
result = await provider.transcribeAudio(audioBuffer, request.language ?? "en")
if (result.text) {
console.log("ElevenLabs transcription successful")
} else if (result.error) {
console.warn("ElevenLabs transcription failed, falling back to Cline service:", result.error)
result = null // Fall back to Cline service
}
} catch (error) {
console.warn("ElevenLabs STT error, falling back to Cline service:", error)
result = null // Fall back to Cline service
}
}
// Fall back to Cline transcription service if ElevenLabs failed or no API key
if (!result) {
result = await getVoiceTranscriptionService().transcribeAudio(request.audioBase64, request.language ?? "en")
}
const durationMs = Date.now() - startTime
if (result.error) {
+19
View File
@@ -8,6 +8,7 @@ import { cleanupLegacyCheckpoints } from "@integrations/checkpoints/CheckpointMi
import { downloadTask } from "@integrations/misc/export-markdown"
import { ClineAccountService } from "@services/account/ClineAccountService"
import { McpHub } from "@services/mcp/McpHub"
import { TextToSpeechService } from "@services/tts/TextToSpeechService"
import { ApiProvider, ModelInfo } from "@shared/api"
import { ChatContent } from "@shared/ChatContent"
import { ExtensionState, Platform } from "@shared/ExtensionMessage"
@@ -69,6 +70,7 @@ export class Controller {
authService: AuthService
ocaAuthService: OcaAuthService
readonly stateManager: StateManager
private ttsService?: TextToSpeechService
// NEW: Add workspace manager (optional initially)
private workspaceManager?: WorkspaceRootManager
@@ -107,6 +109,11 @@ export class Controller {
return this.workspaceManager
}
// Getter for TTS service
getTtsService(): TextToSpeechService | undefined {
return this.ttsService
}
/**
* Starts the periodic remote config fetching timer
* Fetches immediately and then every 30 seconds
@@ -867,6 +874,15 @@ export class Controller {
const lastDismissedCliBannerVersion = this.stateManager.getGlobalStateKey("lastDismissedCliBannerVersion") || 0
const subagentsEnabled = this.stateManager.getGlobalSettingsKey("subagentsEnabled")
// Discuss Mode state
const discussModeEnabled = this.stateManager.getGlobalStateKey("discussModeEnabled")
const discussModeSettings = {
selectedVoice: this.stateManager.getGlobalSettingsKey("discussModeSelectedVoice"),
speechSpeed: this.stateManager.getGlobalSettingsKey("discussModeSpeechSpeed"),
autoSpeak: this.stateManager.getGlobalSettingsKey("discussModeAutoSpeak"),
autoListen: this.stateManager.getGlobalSettingsKey("discussModeAutoListen"),
}
const localClineRulesToggles = this.stateManager.getWorkspaceStateKey("localClineRulesToggles")
const localWindsurfRulesToggles = this.stateManager.getWorkspaceStateKey("localWindsurfRulesToggles")
const localCursorRulesToggles = this.stateManager.getWorkspaceStateKey("localCursorRulesToggles")
@@ -944,6 +960,9 @@ export class Controller {
autoCondenseThreshold,
backgroundCommandRunning: this.backgroundCommandRunning,
backgroundCommandTaskId: this.backgroundCommandTaskId,
// Discuss Mode
discussModeEnabled,
discussModeSettings,
// NEW: Add workspace information
workspaceRoots: this.workspaceManager?.getRoots() ?? [],
primaryRootIndex: this.workspaceManager?.getPrimaryIndex() ?? 0,
@@ -0,0 +1,53 @@
import type { GetVoicesRequest, ValidateApiKeyResponse } from "../../../shared/proto/cline/tts"
import type { Controller } from ".."
/**
* Check if TTS API key is configured and valid
* @param controller The controller instance
* @param request Empty request
* @returns Whether an API key is configured and valid
*/
export async function checkApiKeyConfigured(controller: Controller, request: GetVoicesRequest): Promise<ValidateApiKeyResponse> {
try {
// Check if API key exists in secrets
const apiKey = await controller.context.secrets.get("elevenLabsApiKey")
console.log("[checkApiKeyConfigured] Checking API key:", {
hasKey: !!apiKey,
keyLength: apiKey?.length || 0,
})
if (!apiKey) {
console.log("[checkApiKeyConfigured] No API key found in secrets")
return {
isValid: false,
error: undefined, // No error, just not configured
}
}
// API key exists, validate it
const { TextToSpeechService } = await import("../../../services/tts/TextToSpeechService")
const ttsService = new TextToSpeechService()
await ttsService.initialize({
provider: "elevenlabs",
apiKey: apiKey,
})
// Quick validation check
const isValid = await ttsService.validateApiKey()
return {
isValid,
error: isValid ? undefined : "API key is invalid or expired",
}
} catch (error) {
return {
isValid: false,
error: `Failed to check API key: ${error instanceof Error ? error.message : "Unknown error"}`,
}
}
}
// Export with PascalCase for proto compatibility
export { checkApiKeyConfigured as CheckApiKeyConfigured }
@@ -0,0 +1,51 @@
import type { GetVoicesRequest, VoicesResponse } from "../../../shared/proto/cline/tts"
import type { Controller } from ".."
/**
* Get list of available voices for the current TTS provider
* @param controller The controller instance
* @param request The request (currently empty)
* @returns List of available voices
*/
export async function getAvailableVoices(controller: Controller, request: GetVoicesRequest): Promise<VoicesResponse> {
try {
// Get API key from secrets storage
const apiKey = await controller.context.secrets.get("elevenLabsApiKey")
if (!apiKey) {
return {
voices: [],
error: "No API key found. Please validate your API key first.",
}
}
// Initialize TTS service with the stored key
const { TextToSpeechService } = await import("../../../services/tts/TextToSpeechService")
const ttsService = new TextToSpeechService()
await ttsService.initialize({
provider: "elevenlabs",
apiKey: apiKey,
})
const result = await ttsService.getAvailableVoices()
return {
voices: result.voices.map((voice) => ({
id: voice.id,
name: voice.name,
description: voice.description,
previewUrl: voice.previewUrl,
})),
error: result.error,
}
} catch (error) {
return {
voices: [],
error: `Failed to fetch voices: ${error instanceof Error ? error.message : "Unknown error"}`,
}
}
}
// Export with PascalCase for proto compatibility
export { getAvailableVoices as GetAvailableVoices }
@@ -0,0 +1,80 @@
import type { SynthesizeRequest, SynthesizeResponse } from "../../../shared/proto/cline/tts"
import type { Controller } from ".."
/**
* Synthesize speech from text using the configured TTS provider
* @param controller The controller instance
* @param request The synthesis request containing text and voice settings
* @returns The synthesized audio data
*/
export async function synthesizeSpeech(controller: Controller, request: SynthesizeRequest): Promise<SynthesizeResponse> {
try {
// Get API key from secrets storage
const apiKey = await controller.context.secrets.get("elevenLabsApiKey")
if (!apiKey) {
return {
audioData: Buffer.alloc(0),
contentType: "audio/mpeg",
error: "No API key found. Please validate your API key first.",
}
}
// Initialize TTS service with the stored key
const { TextToSpeechService } = await import("../../../services/tts/TextToSpeechService")
const ttsService = new TextToSpeechService()
await ttsService.initialize({
provider: "elevenlabs",
apiKey: apiKey,
})
const result = await ttsService.synthesizeSpeech({
voiceId: request.voiceId,
text: request.text,
speed: request.speed,
stability: request.stability,
similarityBoost: request.similarityBoost,
})
console.log("[synthesizeSpeech] TTS Service Result:", {
audioDataLength: result.audioData?.length,
audioDataType: typeof result.audioData,
contentType: result.contentType,
error: result.error,
})
// Protobuf-ts expects Uint8Array, not Buffer
const audioUint8Array = new Uint8Array(result.audioData)
console.log("[synthesizeSpeech] Created Uint8Array:", {
arrayLength: audioUint8Array.length,
arrayType: typeof audioUint8Array,
isUint8Array: audioUint8Array instanceof Uint8Array,
})
const response: SynthesizeResponse = {
audioData: audioUint8Array as any, // Protobuf bytes field accepts Uint8Array
contentType: result.contentType,
error: result.error,
}
console.log("[synthesizeSpeech] Returning response:", {
audioDataLength: response.audioData?.length,
hasAudioData: !!response.audioData,
contentType: response.contentType,
error: response.error,
})
return response
} catch (error) {
const emptyArray = new Uint8Array(0)
return {
audioData: emptyArray as any,
contentType: "audio/mpeg",
error: `Failed to synthesize speech: ${error instanceof Error ? error.message : "Unknown error"}`,
}
}
}
// Export with PascalCase for proto compatibility
export { synthesizeSpeech as SynthesizeSpeech }
+60
View File
@@ -0,0 +1,60 @@
import type { ValidateApiKeyRequest, ValidateApiKeyResponse } from "../../../shared/proto/cline/tts"
import type { Controller } from ".."
/**
* Validate the TTS API key
* @param controller The controller instance
* @param request The validation request (currently empty)
* @returns Whether the API key is valid
*/
export async function validateApiKey(controller: Controller, request: ValidateApiKeyRequest): Promise<ValidateApiKeyResponse> {
try {
const apiKey = request.apiKey
if (!apiKey || apiKey.trim() === "") {
return {
isValid: false,
error: "API key is required",
}
}
console.log("[validateApiKey] Saving API key to secrets storage...")
// Save the API key to secrets storage
await controller.context.secrets.store("elevenLabsApiKey", apiKey)
console.log("[validateApiKey] API key saved successfully")
// Initialize TTS service with the new key
const { TextToSpeechService } = await import("../../../services/tts/TextToSpeechService")
const ttsService = new TextToSpeechService()
await ttsService.initialize({
provider: "elevenlabs",
apiKey: apiKey,
})
// Validate the API key by attempting to fetch voices
// Also try to get voices to provide better error feedback
const voicesResult = await ttsService.getAvailableVoices()
if (voicesResult.error) {
return {
isValid: false,
error: voicesResult.error,
}
}
// If we got voices successfully, the key is valid
return {
isValid: true,
error: undefined,
}
} catch (error) {
return {
isValid: false,
error: `Failed to validate API key: ${error instanceof Error ? error.message : "Unknown error"}`,
}
}
}
// Export with PascalCase for proto compatibility
export { validateApiKey as ValidateApiKey }
@@ -0,0 +1,23 @@
import { BooleanRequest } from "../../../shared/proto/cline/common"
import { Controller } from ".."
/**
* Toggle discuss mode on/off
* @param controller The controller instance
* @param request The request containing the enabled state
* @returns BooleanRequest with the new state
*/
export async function setDiscussModeEnabled(controller: Controller, request: BooleanRequest): Promise<BooleanRequest> {
const enabled = request.value
// Update global state
controller.stateManager.setGlobalState("discussModeEnabled", enabled)
// Notify webview of state change
await controller.postStateToWebview()
return BooleanRequest.create({ value: enabled })
}
// Export with PascalCase for code generation
export { setDiscussModeEnabled as SetDiscussModeEnabled }
@@ -0,0 +1,41 @@
import { Empty } from "../../../shared/proto/cline/common"
import type { DiscussVoiceSettingsRequest } from "../../../shared/proto/cline/ui"
import type { Controller } from ".."
/**
* Update discuss mode voice settings
* @param controller The controller instance
* @param request The settings to update
* @returns Empty response
*/
export async function updateDiscussVoiceSettings(controller: Controller, request: DiscussVoiceSettingsRequest): Promise<Empty> {
try {
// Update settings in global state
if (request.selectedVoice !== undefined) {
controller.stateManager.setGlobalState("discussModeSelectedVoice", request.selectedVoice)
}
if (request.speechSpeed !== undefined) {
controller.stateManager.setGlobalState("discussModeSpeechSpeed", request.speechSpeed)
}
if (request.autoSpeak !== undefined) {
controller.stateManager.setGlobalState("discussModeAutoSpeak", request.autoSpeak)
}
if (request.autoListen !== undefined) {
controller.stateManager.setGlobalState("discussModeAutoListen", request.autoListen)
}
// Notify webview of state change
await controller.postStateToWebview()
return Empty.create()
} catch (error) {
console.error("Failed to update discuss voice settings:", error)
return Empty.create()
}
}
// Export with PascalCase for proto compatibility
export { updateDiscussVoiceSettings as UpdateDiscussVoiceSettings }
@@ -18,7 +18,51 @@ In each user message, the environment_details will specify the current mode. The
- When starting in PLAN MODE, depending on the user's request, you may need to do some information gathering e.g. using read_file or search_files to get more context about the task.${context.yoloModeToggled !== true ? " You may also ask the user clarifying questions with ask_followup_question to get a better understanding of the task." : ""}
- Once you've gained more context about the user's request, you should architect a detailed plan for how you will accomplish the task. Present the plan to the user using the plan_mode_respond tool.
- Then you might ask the user if they are pleased with this plan, or if they would like to make any changes. Think of this as a brainstorming session where you can discuss the task and plan the best way to accomplish it.
- Finally once it seems like you've reached a good plan, ask the user to switch you back to ACT MODE to implement the solution.`
- Finally once it seems like you've reached a good plan, ask the user to switch you back to ACT MODE to implement the solution.
## DISCUSS MODE (Enhanced Plan Mode with Voice)
When the environment_details indicates that DISCUSS_MODE is enabled, you should adopt a more conversational and interactive planning approach. This is a voice-enabled conversation where the user is speaking to you naturally, so respond as if you're having a friendly, collaborative discussion with a colleague.
### Conversational Tone & Style
- **Be conversational, not robotic** - Write as if you're speaking naturally in a friendly conversation
- **Limit each response to 280 characters maximum** - Keep it brief since responses are converted to speech
- **Avoid mentioning file paths** - Don't include file paths or directory structures in your responses as they're awkward when spoken aloud. Reference files by name only when absolutely necessary (e.g., "the config file" instead of "src/config/settings.json")
- Use contractions naturally ("I'll", "you're", "that's", "we've") to sound more human
- Break down complex topics into dialogue-friendly chunks across multiple exchanges
- Use a warm, collaborative tone that encourages discussion - imagine you're brainstorming with a friend
- Avoid overly technical jargon unless necessary; explain concepts clearly and conversationally
- Speak in the first person ("I'll help you..." not "We will...")
- Use natural, everyday language rather than formal or stilted phrasing
- Show enthusiasm and engagement - use phrases like "That's a great idea!", "I love that approach!", "Perfect!"
### Proactive Question-Asking
- After each response, actively seek to understand the user's needs better by asking 1-2 relevant follow-up questions
- Ask clarifying questions about requirements, constraints, preferences, or technical details
- Guide the conversation naturally toward a complete understanding of the task
- Frame questions conversationally: "What framework are you using?" not "Please specify the framework"
- Examples of good conversational follow-up questions:
* "What framework are you using for this project?"
* "Do you have any specific design preferences or constraints I should know about?"
* "Would you like me to explain how this approach works, or should I move forward with the implementation?"
### Plan Completion Signal
- Once you have gathered sufficient context and created a comprehensive plan, signal this clearly in your response
- Use conversational phrases like:
* "I now have a clear plan for this task."
* "I'm ready to implement this solution when you're ready."
* "I have everything I need to build this. Would you like me to proceed?"
- Then explicitly ask: "Shall I switch to Act Mode and start building this?"
- This signals to the user that the planning phase is complete and implementation can begin
### Natural Conversation Flow
- In Discuss Mode, the user may be speaking to you, so expect more natural language and potentially incomplete sentences
- Be patient and ask for clarification when needed - don't assume or guess
- Always acknowledge the user's input before diving into technical details (e.g., "Got it!", "I understand", "That makes sense")
- Use conversational transitions like "Great!", "I see,", "That makes sense,", "Ah, I understand" to maintain natural flow
- Mirror the user's energy level - if they're excited, match that enthusiasm; if they're casual, be casual too
- Ask follow-up questions naturally as part of the conversation, not as a formal interrogation
- Remember: The goal is collaborative discussion, not just information extraction. You're having a conversation, not conducting an interview`
export async function getActVsPlanModeSection(variant: PromptVariant, context: SystemPromptContext): Promise<string> {
const template = variant.componentOverrides?.[SystemPromptSection.ACT_VS_PLAN]?.template || getActVsPlanModeTemplateText
+17
View File
@@ -54,6 +54,7 @@ export async function readSecretsFromDisk(context: ExtensionContext): Promise<Se
minimaxApiKey,
hicapApiKey,
aihubmixApiKey,
elevenLabsApiKey,
] = await Promise.all([
context.secrets.get("apiKey") as Promise<Secrets["apiKey"]>,
context.secrets.get("openRouterApiKey") as Promise<Secrets["openRouterApiKey"]>,
@@ -96,6 +97,7 @@ export async function readSecretsFromDisk(context: ExtensionContext): Promise<Se
context.secrets.get("minimaxApiKey") as Promise<Secrets["minimaxApiKey"]>,
context.secrets.get("hicapApiKey") as Promise<Secrets["hicapApiKey"]>,
context.secrets.get("aihubmixApiKey") as Promise<Secrets["aihubmixApiKey"]>,
context.secrets.get("elevenLabsApiKey") as Promise<Secrets["elevenLabsApiKey"]>,
])
return {
@@ -140,6 +142,7 @@ export async function readSecretsFromDisk(context: ExtensionContext): Promise<Se
minimaxApiKey,
hicapApiKey,
aihubmixApiKey,
elevenLabsApiKey,
}
}
@@ -272,6 +275,15 @@ export async function readGlobalStateFromDisk(context: ExtensionContext): Promis
const hicapModelId = context.globalState.get<GlobalStateAndSettings["hicapModelId"]>("hicapModelId")
const aihubmixBaseUrl = context.globalState.get<GlobalStateAndSettings["aihubmixBaseUrl"]>("aihubmixBaseUrl")
const aihubmixAppCode = context.globalState.get<GlobalStateAndSettings["aihubmixAppCode"]>("aihubmixAppCode")
const discussModeEnabled = context.globalState.get<GlobalStateAndSettings["discussModeEnabled"]>("discussModeEnabled")
const discussModeSelectedVoice =
context.globalState.get<GlobalStateAndSettings["discussModeSelectedVoice"]>("discussModeSelectedVoice")
const discussModeSpeechSpeed =
context.globalState.get<GlobalStateAndSettings["discussModeSpeechSpeed"]>("discussModeSpeechSpeed")
const discussModeAutoSpeak =
context.globalState.get<GlobalStateAndSettings["discussModeAutoSpeak"]>("discussModeAutoSpeak")
const discussModeAutoListen =
context.globalState.get<GlobalStateAndSettings["discussModeAutoListen"]>("discussModeAutoListen")
// OpenTelemetry configuration
const openTelemetryEnabled =
@@ -669,6 +681,11 @@ export async function readGlobalStateFromDisk(context: ExtensionContext): Promis
openTelemetryLogBatchSize: openTelemetryLogBatchSize ?? 512,
openTelemetryLogBatchTimeout: openTelemetryLogBatchTimeout ?? 5000,
openTelemetryLogMaxQueueSize: openTelemetryLogMaxQueueSize ?? 2048,
discussModeEnabled: discussModeEnabled ?? false,
discussModeSelectedVoice: discussModeSelectedVoice,
discussModeSpeechSpeed: discussModeSpeechSpeed ?? 1.2,
discussModeAutoSpeak: discussModeAutoSpeak ?? false,
discussModeAutoListen: discussModeAutoListen ?? false,
}
} catch (error) {
console.error("[StateHelpers] Failed to read global state:", error)
+5
View File
@@ -3507,8 +3507,13 @@ export class Task {
details += "\n\n# Current Mode"
const mode = this.stateManager.getGlobalSettingsKey("mode")
const discussModeEnabled = this.stateManager.getGlobalStateKey("discussModeEnabled") ?? false
if (mode === "plan") {
details += "\nPLAN MODE\n" + formatResponse.planModeInstructions()
if (discussModeEnabled) {
details +=
"\n\nDISCUSS_MODE is enabled. You are in a voice-enabled conversation where the user may be speaking to you. Adopt a natural, conversational tone and keep responses brief and dialogue-friendly."
}
} else {
details += "\nACT MODE"
}
+4
View File
@@ -114,7 +114,9 @@ export abstract class WebviewProvider {
font-src ${this.getCspSource()} data:;
style-src ${this.getCspSource()} 'unsafe-inline';
img-src ${this.getCspSource()} https: data:;
media-src blob: data:;
script-src 'nonce-${nonce}' 'unsafe-eval';">
<meta http-equiv="Permissions-Policy" content="autoplay=*, microphone=*">
<title>Cline</title>
</head>
<body>
@@ -201,6 +203,7 @@ export abstract class WebviewProvider {
`font-src ${this.getCspSource()}`,
`style-src ${this.getCspSource()} 'unsafe-inline' https://* http://${localServerUrl} http://0.0.0.0:${localPort}`,
`img-src ${this.getCspSource()} https: data:`,
`media-src blob: data:`,
`script-src 'unsafe-eval' https://* http://${localServerUrl} http://0.0.0.0:${localPort} 'nonce-${nonce}'`,
`connect-src https://* ws://${localServerUrl} ws://0.0.0.0:${localPort} http://${localServerUrl} http://0.0.0.0:${localPort}`,
]
@@ -213,6 +216,7 @@ export abstract class WebviewProvider {
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1,shrink-to-fit=no">
<meta http-equiv="Content-Security-Policy" content="${csp.join("; ")}">
<meta http-equiv="Permissions-Policy" content="autoplay=*, microphone=*">
<link rel="stylesheet" type="text/css" href="${stylesUrl}">
<link href="${codiconsUrl}" rel="stylesheet" />
<title>Cline</title>
@@ -18,6 +18,7 @@ export class AudioRecordingService {
private recordingProcess: ChildProcess | null = null
private startTime: number = 0
private outputFile: string = ""
private silenceDetectedCallback: (() => void) | null = null
constructor() {}
@@ -145,7 +146,26 @@ export class AudioRecordingService {
this.recordingProcess.stderr?.on("data", (data) => {
const message = data.toString().trim()
if (message && !message.includes("In:") && !message.includes("Out:")) {
// Check for silence detection
if (message.includes("silence_end")) {
// Parse silence duration from FFmpeg output
// Format: [silencedetect @ 0x...] silence_end: 4.5 | silence_duration: 2.0
const durationMatch = message.match(/silence_duration:\s*([\d.]+)/)
if (durationMatch) {
const silenceDuration = parseFloat(durationMatch[1])
Logger.info(`Detected ${silenceDuration}s of silence`)
// If silence duration is >= 2 seconds, trigger auto-stop
if (silenceDuration >= 2.0 && this.silenceDetectedCallback) {
Logger.info("Auto-stopping recording due to silence detection")
this.silenceDetectedCallback()
}
}
}
// Log other stderr output (but filter out common noise)
if (message && !message.includes("In:") && !message.includes("Out:") && !message.includes("silencedetect")) {
Logger.info(`Recording stderr: ${message}`)
}
})
@@ -231,6 +251,14 @@ export class AudioRecordingService {
}
}
/**
* Sets a callback to be triggered when silence is detected
* @param callback Function to call when silence is detected
*/
setSilenceDetectedCallback(callback: (() => void) | null): void {
this.silenceDetectedCallback = callback
}
private checkRecordingDependencies(): { available: boolean; error?: string } {
const program = this.getRecordProgram()
if (!program) {
@@ -101,6 +101,7 @@ export class VoiceTranscriptionService {
}
async transcribeAudio(audioBase64: string, language?: string): Promise<{ text?: string; error?: string }> {
// Use Cline's transcription service (ElevenLabs STT is now handled in the controller)
try {
Logger.info("Transcribing audio with Cline transcription service...")
+103
View File
@@ -0,0 +1,103 @@
import type { BaseTTSProvider, TTSOptions, TTSSynthesisResult, TTSVoicesResult } from "./providers/BaseTTSProvider"
import { ElevenLabsProvider } from "./providers/ElevenLabsProvider"
/**
* Supported TTS providers
*/
export type TTSProvider = "elevenlabs" | "openai"
/**
* Configuration for TTS service
*/
export interface TTSConfig {
provider: TTSProvider
apiKey: string
}
/**
* Text-to-Speech Service
* Manages TTS provider instances and handles speech synthesis requests
*/
export class TextToSpeechService {
private provider: BaseTTSProvider | null = null
private currentConfig: TTSConfig | null = null
/**
* Initialize the TTS service with configuration
*/
async initialize(config: TTSConfig): Promise<void> {
// Only reinitialize if config changed
if (
this.currentConfig &&
this.currentConfig.provider === config.provider &&
this.currentConfig.apiKey === config.apiKey
) {
return
}
this.currentConfig = config
this.provider = this.createProvider(config)
}
/**
* Create a TTS provider instance based on configuration
*/
private createProvider(config: TTSConfig): BaseTTSProvider {
switch (config.provider) {
case "elevenlabs":
return new ElevenLabsProvider(config.apiKey)
case "openai":
// TODO: Implement OpenAI TTS provider
throw new Error("OpenAI TTS provider not yet implemented")
default:
throw new Error(`Unsupported TTS provider: ${config.provider}`)
}
}
/**
* Synthesize speech from text
*/
async synthesizeSpeech(options: TTSOptions): Promise<TTSSynthesisResult> {
if (!this.provider) {
throw new Error("TTS service not initialized. Call initialize() first.")
}
return this.provider.synthesizeSpeech(options)
}
/**
* Get available voices for the current provider
*/
async getAvailableVoices(): Promise<TTSVoicesResult> {
if (!this.provider) {
throw new Error("TTS service not initialized. Call initialize() first.")
}
return this.provider.getAvailableVoices()
}
/**
* Validate the current API key
*/
async validateApiKey(): Promise<boolean> {
if (!this.provider) {
return false
}
return this.provider.validateApiKey()
}
/**
* Check if service is initialized
*/
isInitialized(): boolean {
return this.provider !== null
}
/**
* Get current provider type
*/
getCurrentProvider(): TTSProvider | null {
return this.currentConfig?.provider ?? null
}
}
@@ -0,0 +1,56 @@
/**
* Base interface for Text-to-Speech providers
*/
export interface TTSVoice {
id: string
name: string
description?: string
previewUrl?: string
}
export interface TTSOptions {
voiceId: string
text: string
speed?: number
stability?: number
similarityBoost?: number
}
export interface TTSSynthesisResult {
audioData: Uint8Array
contentType: string
error?: string
}
export interface TTSVoicesResult {
voices: TTSVoice[]
error?: string
}
export abstract class BaseTTSProvider {
protected apiKey: string
constructor(apiKey: string) {
this.apiKey = apiKey
}
/**
* Synthesize speech from text
* @param options Synthesis options including text and voice settings
* @returns Audio buffer and content type
*/
abstract synthesizeSpeech(options: TTSOptions): Promise<TTSSynthesisResult>
/**
* Get list of available voices
* @returns List of available voices
*/
abstract getAvailableVoices(): Promise<TTSVoicesResult>
/**
* Validate API key by making a test request
* @returns True if API key is valid
*/
abstract validateApiKey(): Promise<boolean>
}
@@ -0,0 +1,242 @@
import axios from "axios"
import { BaseTTSProvider, type TTSOptions, type TTSSynthesisResult, type TTSVoice, type TTSVoicesResult } from "./BaseTTSProvider"
const ELEVENLABS_API_BASE = "https://api.elevenlabs.io/v1"
export interface STTTranscriptionResult {
text?: string
error?: string
}
/**
* ElevenLabs Text-to-Speech Provider
* Implements TTS using ElevenLabs API
*/
export class ElevenLabsProvider extends BaseTTSProvider {
/**
* Synthesize speech from text using ElevenLabs API
*/
async synthesizeSpeech(options: TTSOptions): Promise<TTSSynthesisResult> {
try {
const { voiceId, text, speed = 1.0, stability = 0.5, similarityBoost = 0.75 } = options
// Clamp speed to ElevenLabs API limits (0.7 to 1.2)
const clampedSpeed = Math.max(0.7, Math.min(1.2, speed))
const response = await axios.post(
`${ELEVENLABS_API_BASE}/text-to-speech/${voiceId}`,
{
text,
model_id: "eleven_turbo_v2_5",
voice_settings: {
stability,
similarity_boost: similarityBoost,
speed: clampedSpeed,
},
},
{
headers: {
"xi-api-key": this.apiKey,
"Content-Type": "application/json",
Accept: "audio/mpeg",
},
responseType: "arraybuffer",
timeout: 30000, // 30 second timeout
},
)
return {
audioData: new Uint8Array(response.data),
contentType: response.headers["content-type"] || "audio/mpeg",
}
} catch (error) {
if (axios.isAxiosError(error)) {
const errorMessage = error.response?.data ? new TextDecoder().decode(error.response.data) : error.message
return {
audioData: new Uint8Array(0),
contentType: "audio/mpeg",
error: `ElevenLabs API error: ${errorMessage}`,
}
}
return {
audioData: new Uint8Array(0),
contentType: "audio/mpeg",
error: `Failed to synthesize speech: ${error instanceof Error ? error.message : "Unknown error"}`,
}
}
}
/**
* Get list of available voices from ElevenLabs
*/
async getAvailableVoices(): Promise<TTSVoicesResult> {
try {
const response = await axios.get(`${ELEVENLABS_API_BASE}/voices`, {
headers: {
"xi-api-key": this.apiKey,
},
timeout: 10000,
})
const voices: TTSVoice[] = response.data.voices.map((voice: any) => ({
id: voice.voice_id,
name: voice.name,
description: voice.labels?.description || voice.category,
previewUrl: voice.preview_url,
}))
return { voices }
} catch (error) {
if (axios.isAxiosError(error)) {
// Extract error message from different possible sources
let errorMessage = error.message
if (error.response?.data) {
if (typeof error.response.data === "string") {
errorMessage = error.response.data
} else if (error.response.data.detail) {
errorMessage = error.response.data.detail
} else if (error.response.data.message) {
errorMessage = error.response.data.message
} else {
errorMessage = JSON.stringify(error.response.data)
}
}
// Add status code if available
if (error.response?.status) {
errorMessage = `${error.response.status}: ${errorMessage}`
}
return {
voices: [],
error: `Failed to fetch voices: ${errorMessage}`,
}
}
return {
voices: [],
error: `Failed to fetch voices: ${error instanceof Error ? error.message : String(error)}`,
}
}
}
/**
* Validate API key by attempting to fetch voices
*/
async validateApiKey(): Promise<boolean> {
try {
await axios.get(`${ELEVENLABS_API_BASE}/voices`, {
headers: {
"xi-api-key": this.apiKey,
},
timeout: 5000,
})
return true
} catch (error) {
return false
}
}
/**
* Transcribe audio to text using ElevenLabs Speech-to-Text API (Scribe v1 model)
* @param audioBuffer The audio data as a Buffer or Uint8Array
* @param language Optional language code (e.g., 'en', 'es', 'fr')
* @returns Transcription result with text or error
*/
async transcribeAudio(audioBuffer: Buffer | Uint8Array, language?: string): Promise<STTTranscriptionResult> {
try {
// Convert Uint8Array to Buffer if needed
const buffer = Buffer.isBuffer(audioBuffer) ? audioBuffer : Buffer.from(audioBuffer)
// Create form data manually
const boundary = `----WebKitFormBoundary${Math.random().toString(36).substring(2)}`
// Build multipart form data body
const parts: Buffer[] = []
// Add audio file part - ElevenLabs expects field name "file"
parts.push(Buffer.from(`--${boundary}\r\n`))
parts.push(Buffer.from('Content-Disposition: form-data; name="file"; filename="audio.webm"\r\n'))
parts.push(Buffer.from("Content-Type: audio/webm\r\n\r\n"))
parts.push(buffer)
parts.push(Buffer.from("\r\n"))
// Add language parameter if provided
if (language) {
parts.push(Buffer.from(`--${boundary}\r\n`))
parts.push(Buffer.from('Content-Disposition: form-data; name="language"\r\n\r\n'))
parts.push(Buffer.from(`${language}\r\n`))
}
// Add model parameter - Note: use underscore not hyphen
parts.push(Buffer.from(`--${boundary}\r\n`))
parts.push(Buffer.from('Content-Disposition: form-data; name="model_id"\r\n\r\n'))
parts.push(Buffer.from("scribe_v1\r\n"))
// Close boundary
parts.push(Buffer.from(`--${boundary}--\r\n`))
const body = Buffer.concat(parts)
const response = await axios.post(`${ELEVENLABS_API_BASE}/speech-to-text`, body, {
headers: {
"xi-api-key": this.apiKey,
"Content-Type": `multipart/form-data; boundary=${boundary}`,
},
timeout: 120000, // 2 minute timeout for transcription
})
// Extract text from response (ElevenLabs returns detailed JSON with words, timestamps, etc.)
const text = response.data?.text || ""
return { text }
} catch (error) {
if (axios.isAxiosError(error)) {
let errorMessage = error.message
let errorDetails = ""
if (error.response?.data) {
if (typeof error.response.data === "string") {
errorMessage = error.response.data
} else if (error.response.data.detail) {
if (typeof error.response.data.detail === "object") {
errorMessage = JSON.stringify(error.response.data.detail, null, 2)
} else {
errorMessage = String(error.response.data.detail)
}
} else if (error.response.data.message) {
errorMessage = String(error.response.data.message)
} else if (error.response.data.error) {
errorMessage = String(error.response.data.error)
} else {
// Try to extract useful error info
try {
errorDetails = JSON.stringify(error.response.data, null, 2)
errorMessage = errorDetails
} catch (e) {
errorMessage = "Invalid response format"
}
}
}
// Add status code if available
const statusCode = error.response?.status || "Unknown"
const fullMessage = `${statusCode}: ${errorMessage}`
console.error("[ElevenLabs STT] Full error details:", {
status: error.response?.status,
statusText: error.response?.statusText,
data: error.response?.data,
headers: error.response?.headers,
})
return {
error: `ElevenLabs transcription error: ${fullMessage}`,
}
}
return {
error: `Failed to transcribe audio: ${error instanceof Error ? error.message : String(error)}`,
}
}
}
}
+9
View File
@@ -86,6 +86,15 @@ export interface ExtensionState {
customPrompt?: string
autoCondenseThreshold?: number
favoritedModelIds: string[]
// Discuss Mode settings
discussModeEnabled?: boolean
discussModeSettings?: {
elevenLabsApiKey?: string
selectedVoice?: string
speechSpeed?: number
autoSpeak?: boolean
autoListen?: boolean
}
// NEW: Add workspace information
workspaceRoots: WorkspaceRoot[]
primaryRootIndex: number
+6
View File
@@ -7,6 +7,8 @@ export const AUDIO_PROGRAM_CONFIG = {
"avfoundation",
"-i",
":default",
"-af",
"silencedetect=noise=-30dB:d=2", // Detect 2 seconds of silence at -30dB threshold
"-c:a",
"libopus",
"-b:a",
@@ -32,6 +34,8 @@ export const AUDIO_PROGRAM_CONFIG = {
"alsa",
"-i",
"default",
"-af",
"silencedetect=noise=-30dB:d=2", // Detect 2 seconds of silence at -30dB threshold
"-c:a",
"libopus",
"-b:a",
@@ -61,6 +65,8 @@ export const AUDIO_PROGRAM_CONFIG = {
"wasapi",
"-i",
"audio=default",
"-af",
"silencedetect=noise=-30dB:d=2", // Detect 2 seconds of silence at -30dB threshold
"-c:a",
"libopus",
"-b:a",
+7
View File
@@ -51,6 +51,7 @@ export interface GlobalState {
lastDismissedModelBannerVersion: number
lastDismissedCliBannerVersion: number
nativeToolCallEnabled: boolean
discussModeEnabled: boolean
}
export interface Settings {
@@ -121,6 +122,11 @@ export interface Settings {
hooksEnabled: boolean
subagentsEnabled: boolean
hicapModelId: string | undefined
// Discuss Mode settings
discussModeSelectedVoice: string | undefined
discussModeSpeechSpeed: number | undefined
discussModeAutoSpeak: boolean
discussModeAutoListen: boolean
// Plan mode configurations
planModeApiProvider: ApiProvider
@@ -254,6 +260,7 @@ export interface Secrets {
minimaxApiKey: string | undefined
hicapApiKey: string | undefined
aihubmixApiKey: string | undefined
elevenLabsApiKey: string | undefined
}
export interface LocalState {
+5 -3
View File
@@ -55,8 +55,10 @@
"@testing-library/react": "^16.2.0",
"@testing-library/user-event": "^14.6.1",
"@types/dompurify": "^3.0.5",
"@types/hast": "^3.0.4",
"@types/jest": "^29.5.14",
"@types/node": "^22.13.4",
"@types/prop-types": "^15.7.15",
"@types/react": "^18.3.18",
"@types/react-dom": "^18.3.5",
"@types/uuid": "^9.0.8",
@@ -7250,9 +7252,9 @@
}
},
"node_modules/@types/prop-types": {
"version": "15.7.14",
"resolved": "https://registry.npmjs.org/@types/prop-types/-/prop-types-15.7.14.tgz",
"integrity": "sha512-gNMvNH49DJ7OJYv+KAKn0Xp45p8PLl6zo2YnvDIbTd4J6MER2BmWN49TG7n9LvkyihINxeKW8+3bfS2yDC9dzQ==",
"version": "15.7.15",
"resolved": "https://registry.npmjs.org/@types/prop-types/-/prop-types-15.7.15.tgz",
"integrity": "sha512-F6bEyamV9jKGAFBEmlQnesRPGOQqS2+Uwi0Em15xenOxHaf2hv6L8YCVn3rPdPJOiJfPiCnLIRyvwVaqMY3MIw==",
"devOptional": true,
"license": "MIT"
},
+2
View File
@@ -63,8 +63,10 @@
"@testing-library/react": "^16.2.0",
"@testing-library/user-event": "^14.6.1",
"@types/dompurify": "^3.0.5",
"@types/hast": "^3.0.4",
"@types/jest": "^29.5.14",
"@types/node": "^22.13.4",
"@types/prop-types": "^15.7.15",
"@types/react": "^18.3.18",
"@types/react-dom": "^18.3.5",
"@types/uuid": "^9.0.8",
@@ -0,0 +1,265 @@
import { SynthesizeRequest } from "@shared/proto/cline/tts"
import React, { useCallback, useEffect, useRef, useState } from "react"
import { TtsServiceClient } from "@/services/grpc-client"
export interface AudioQueueItem {
id: string
text: string
voiceId: string
speed?: number
}
export interface AudioPlayerProps {
/** Called when playback of an item completes */
onPlaybackComplete?: (itemId: string) => void
/** Called when an error occurs */
onError?: (error: string, itemId?: string) => void
/** Called when playback state changes */
onPlaybackStateChange?: (isPlaying: boolean, itemId?: string) => void
/** Default voice ID to use */
defaultVoiceId?: string
/** Default playback speed */
defaultSpeed?: number
}
export interface AudioPlayerHandle {
/** Add audio to the queue */
enqueue: (item: AudioQueueItem) => void
/** Clear the entire queue */
clearQueue: () => void
/** Pause current playback */
pause: () => void
/** Resume current playback */
resume: () => void
/** Stop current playback and clear queue */
stop: () => void
/** Get current queue */
getQueue: () => AudioQueueItem[]
/** Check if currently playing */
isPlaying: () => boolean
}
/**
* AudioPlayer component that handles TTS audio playback with queue management.
* Uses the TTS service to synthesize speech and plays it back.
*/
const AudioPlayer = React.forwardRef<AudioPlayerHandle, AudioPlayerProps>(
(
{
onPlaybackComplete,
onError,
onPlaybackStateChange,
defaultVoiceId = "EXAVITQu4vr4xnSDxMaL", // Default ElevenLabs voice
defaultSpeed = 1.0,
},
ref,
) => {
const [queue, setQueue] = useState<AudioQueueItem[]>([])
const [currentItem, setCurrentItem] = useState<AudioQueueItem | null>(null)
const [isPlaying, setIsPlaying] = useState(false)
const [isPaused, setIsPaused] = useState(false)
const audioRef = useRef<HTMLAudioElement | null>(null)
const currentBlobUrlRef = useRef<string | null>(null)
const isProcessingRef = useRef(false)
// Cleanup blob URL when component unmounts or audio changes
const cleanupBlobUrl = useCallback(() => {
if (currentBlobUrlRef.current) {
URL.revokeObjectURL(currentBlobUrlRef.current)
currentBlobUrlRef.current = null
}
}, [])
// Process the next item in the queue
const processNextItem = useCallback(async () => {
if (isProcessingRef.current || queue.length === 0) {
return
}
isProcessingRef.current = true
const nextItem = queue[0]
setCurrentItem(nextItem)
try {
// Synthesize speech via TTS service
const response = await TtsServiceClient.SynthesizeSpeech(
SynthesizeRequest.create({
text: nextItem.text,
voiceId: nextItem.voiceId || defaultVoiceId,
speed: nextItem.speed || defaultSpeed,
}),
)
if (response.error) {
throw new Error(response.error)
}
// Convert Buffer to Uint8Array then to Blob
const audioArray = new Uint8Array(response.audioData)
const audioBlob = new Blob([audioArray], { type: response.contentType || "audio/mpeg" })
const audioUrl = URL.createObjectURL(audioBlob)
// Cleanup previous blob URL
cleanupBlobUrl()
currentBlobUrlRef.current = audioUrl
// Create and play audio element
if (audioRef.current) {
audioRef.current.pause()
audioRef.current = null
}
const audio = new Audio(audioUrl)
audioRef.current = audio
// Set up event handlers
audio.onplay = () => {
setIsPlaying(true)
setIsPaused(false)
onPlaybackStateChange?.(true, nextItem.id)
}
audio.onpause = () => {
if (!audio.ended) {
setIsPaused(true)
onPlaybackStateChange?.(false, nextItem.id)
}
}
audio.onended = () => {
setIsPlaying(false)
setIsPaused(false)
setCurrentItem(null)
cleanupBlobUrl()
// Remove completed item from queue
setQueue((prev) => prev.slice(1))
onPlaybackComplete?.(nextItem.id)
onPlaybackStateChange?.(false, nextItem.id)
isProcessingRef.current = false
// Process next item if available
setTimeout(() => {
if (queue.length > 1) {
processNextItem()
}
}, 0)
}
audio.onerror = (e) => {
const errorMessage = `Audio playback error: ${audio.error?.message || "Unknown error"}`
console.error(errorMessage, e)
setIsPlaying(false)
setCurrentItem(null)
cleanupBlobUrl()
// Remove failed item from queue
setQueue((prev) => prev.slice(1))
onError?.(errorMessage, nextItem.id)
isProcessingRef.current = false
// Try next item
setTimeout(() => {
if (queue.length > 1) {
processNextItem()
}
}, 0)
}
// Start playback
await audio.play()
} catch (error) {
const errorMessage = error instanceof Error ? error.message : "Failed to synthesize speech"
console.error("TTS synthesis error:", error)
setIsPlaying(false)
setCurrentItem(null)
cleanupBlobUrl()
// Remove failed item from queue
setQueue((prev) => prev.slice(1))
onError?.(errorMessage, nextItem.id)
isProcessingRef.current = false
// Try next item
setTimeout(() => {
if (queue.length > 1) {
processNextItem()
}
}, 0)
}
}, [queue, defaultVoiceId, defaultSpeed, onPlaybackComplete, onError, onPlaybackStateChange, cleanupBlobUrl])
// Auto-process queue when items are added
useEffect(() => {
if (queue.length > 0 && !currentItem && !isProcessingRef.current) {
processNextItem()
}
}, [queue, currentItem, processNextItem])
// Expose imperative handle
React.useImperativeHandle(ref, () => ({
enqueue: (item: AudioQueueItem) => {
setQueue((prev) => [...prev, item])
},
clearQueue: () => {
if (audioRef.current) {
audioRef.current.pause()
audioRef.current = null
}
setQueue([])
setCurrentItem(null)
setIsPlaying(false)
setIsPaused(false)
cleanupBlobUrl()
isProcessingRef.current = false
},
pause: () => {
if (audioRef.current && !audioRef.current.paused) {
audioRef.current.pause()
}
},
resume: () => {
if (audioRef.current && audioRef.current.paused) {
audioRef.current.play()
}
},
stop: () => {
if (audioRef.current) {
audioRef.current.pause()
audioRef.current = null
}
setQueue([])
setCurrentItem(null)
setIsPlaying(false)
setIsPaused(false)
cleanupBlobUrl()
isProcessingRef.current = false
},
getQueue: () => queue,
isPlaying: () => isPlaying,
}))
// Cleanup on unmount
useEffect(() => {
return () => {
if (audioRef.current) {
audioRef.current.pause()
audioRef.current = null
}
cleanupBlobUrl()
}
}, [cleanupBlobUrl])
// This component doesn't render anything visible - it's purely functional
return null
},
)
AudioPlayer.displayName = "AudioPlayer"
export default AudioPlayer
@@ -85,6 +85,7 @@ interface ChatRowProps {
sendMessageFromChatRow?: (text: string, images: string[], files: string[]) => void
onSetQuote: (text: string) => void
onCancelCommand?: () => void
discussModeEnabled?: boolean
}
interface QuoteButtonState {
@@ -273,6 +274,7 @@ export const ChatRowContent = memo(
sendMessageFromChatRow,
onSetQuote,
onCancelCommand,
discussModeEnabled,
}: ChatRowContentProps) => {
const { mcpServers, mcpMarketplaceCatalog, onRelinquishControl, vscodeTerminalExecutionMode } = useExtensionState()
const [seeNewChangesDisabled, setSeeNewChangesDisabled] = useState(false)
@@ -1333,6 +1335,10 @@ export const ChatRowContent = memo(
</div>
)
case "text":
// Hide AI-generated text in discuss mode
if (discussModeEnabled) {
return null
}
return (
<WithCopyButton
onMouseUp={handleMouseUp}
@@ -1352,6 +1358,10 @@ export const ChatRowContent = memo(
</WithCopyButton>
)
case "reasoning":
// Hide reasoning text in discuss mode
if (discussModeEnabled) {
return null
}
return (
<>
{message.text && (
@@ -11,6 +11,7 @@ import { normalizeApiConfiguration } from "@/components/settings/utils/providerU
import { useExtensionState } from "@/context/ExtensionStateContext"
import { useShowNavbar } from "@/context/PlatformContext"
import { FileServiceClient, UiServiceClient } from "@/services/grpc-client"
import { DiscussModeToggle } from "../discuss-mode/DiscussModeToggle"
import { Navbar } from "../menu/Navbar"
import AutoApproveBar from "./auto-approve-menu/AutoApproveBar"
// Import utilities and hooks from the new structure
@@ -29,6 +30,7 @@ import {
useScrollBehavior,
WelcomeSection,
} from "./chat-view"
import { useDiscussModeAudio } from "./hooks/useDiscussModeAudio"
interface ChatViewProps {
isHidden: boolean
@@ -53,6 +55,7 @@ const ChatView = ({ isHidden, showAnnouncement, hideAnnouncement, showHistoryVie
userInfo,
currentFocusChainChecklist,
hooksEnabled,
discussModeEnabled,
} = useExtensionState()
const isProdHostedApp = userInfo?.apiBaseUrl === "https://app.cline.bot"
const shouldShowQuickWins = isProdHostedApp && (!taskHistory || taskHistory.length < QUICK_WINS_HISTORY_THRESHOLD)
@@ -330,6 +333,9 @@ const ChatView = ({ isHidden, showAnnouncement, hideAnnouncement, showHistoryVie
// Use scroll behavior hook
const scrollBehavior = useScrollBehavior(messages, visibleMessages, groupedMessages, expandedRows, setExpandedRows)
// Use Discuss Mode audio hook for automatic TTS
const { stopAudio } = useDiscussModeAudio(messages)
const placeholderText = useMemo(() => {
const text = task ? "Type a message..." : "Type your task here..."
return text
@@ -366,6 +372,7 @@ const ChatView = ({ isHidden, showAnnouncement, hideAnnouncement, showHistoryVie
{task && (
<MessagesArea
chatState={chatState}
discussModeEnabled={discussModeEnabled}
groupedMessages={groupedMessages}
messageHandlers={messageHandlers}
modifiedMessages={modifiedMessages}
@@ -376,6 +383,7 @@ const ChatView = ({ isHidden, showAnnouncement, hideAnnouncement, showHistoryVie
</div>
<footer className="bg-(--vscode-sidebar-background)" style={{ gridRow: "2" }}>
<AutoApproveBar />
<DiscussModeToggle />
<ActionButtons
chatState={chatState}
messageHandlers={messageHandlers}
@@ -396,6 +404,7 @@ const ChatView = ({ isHidden, showAnnouncement, hideAnnouncement, showHistoryVie
scrollBehavior={scrollBehavior}
selectFilesAndImages={selectFilesAndImages}
shouldDisableFilesAndImages={shouldDisableFilesAndImages}
stopAudio={stopAudio}
/>
</footer>
</ChatLayout>
@@ -0,0 +1,156 @@
import { SynthesizeRequest } from "@shared/proto/cline/tts"
import { Play, Volume2, VolumeX } from "lucide-react"
import { useEffect, useRef, useState } from "react"
import { useExtensionState } from "@/context/ExtensionStateContext"
import { TtsServiceClient } from "@/services/grpc-client"
interface MessageAudioPlayerProps {
text: string
messageTs: number
}
/**
* Audio player component that appears next to Cline's messages in Discuss Mode
* Allows manual playback of TTS audio
*/
export function MessageAudioPlayer({ text, messageTs }: MessageAudioPlayerProps) {
const { discussModeSettings } = useExtensionState()
const [isPlaying, setIsPlaying] = useState(false)
const [isLoading, setIsLoading] = useState(false)
const [error, setError] = useState<string | null>(null)
const audioRef = useRef<HTMLAudioElement | null>(null)
const audioUrlRef = useRef<string | null>(null)
const handlePlay = async () => {
if (!discussModeSettings?.selectedVoice) {
setError("No voice selected")
return
}
if (isPlaying && audioRef.current) {
// Pause if already playing
audioRef.current.pause()
setIsPlaying(false)
return
}
// If we already have audio, just play it
if (audioRef.current && audioUrlRef.current) {
try {
await audioRef.current.play()
setIsPlaying(true)
return
} catch (err) {
console.error("[MessageAudioPlayer] Error playing cached audio:", err)
}
}
// Otherwise, synthesize new audio
try {
setIsLoading(true)
setError(null)
const response = await TtsServiceClient.SynthesizeSpeech(
SynthesizeRequest.create({
text,
voiceId: discussModeSettings.selectedVoice,
speed: discussModeSettings.speechSpeed || 1.0,
}),
)
if (response.error) {
setError(response.error)
setIsLoading(false)
return
}
if (!response.audioData || response.audioData.length === 0) {
setError("No audio data received")
setIsLoading(false)
return
}
// Convert Buffer to Blob
const audioBlob = new Blob([new Uint8Array(response.audioData)], {
type: response.contentType || "audio/mpeg",
})
const audioUrl = URL.createObjectURL(audioBlob)
audioUrlRef.current = audioUrl
// Create and play audio
const audio = new Audio(audioUrl)
audioRef.current = audio
audio.onended = () => {
setIsPlaying(false)
}
audio.onerror = () => {
setError("Audio playback error")
setIsPlaying(false)
}
await audio.play()
setIsPlaying(true)
setIsLoading(false)
} catch (err: any) {
console.error("[MessageAudioPlayer] Error:", err)
setError(err.message || "Failed to play audio")
setIsLoading(false)
}
}
// Cleanup on unmount
useEffect(() => {
return () => {
if (audioRef.current) {
audioRef.current.pause()
audioRef.current = null
}
if (audioUrlRef.current) {
URL.revokeObjectURL(audioUrlRef.current)
audioUrlRef.current = null
}
}
}, [])
// Cleanup when message changes
useEffect(() => {
if (audioRef.current) {
audioRef.current.pause()
setIsPlaying(false)
}
if (audioUrlRef.current) {
URL.revokeObjectURL(audioUrlRef.current)
audioUrlRef.current = null
}
}, [messageTs, text])
if (!discussModeSettings?.selectedVoice) {
return null
}
return (
<div className="inline-flex items-center gap-1">
<button
className="p-1 rounded hover:bg-gray-200 dark:hover:bg-gray-700 disabled:opacity-50 transition-colors"
disabled={isLoading}
onClick={handlePlay}
title={isPlaying ? "Pause" : isLoading ? "Loading..." : "Play audio"}
type="button">
{isLoading ? (
<div className="w-4 h-4 border-2 border-gray-400 border-t-transparent rounded-full animate-spin" />
) : isPlaying ? (
<Volume2 className="w-4 h-4 text-blue-500" />
) : (
<Play className="w-4 h-4 text-gray-600 dark:text-gray-400" />
)}
</button>
{error && (
<span className="text-xs text-red-500" title={error}>
<VolumeX className="w-3 h-3" />
</span>
)}
</div>
)
}
@@ -0,0 +1,28 @@
import { cn } from "@/lib/utils"
interface RecordingIndicatorProps {
duration: number
className?: string
}
/**
* Visual indicator shown while recording audio
* Displays a pulsing red dot and the recording duration
*/
export function RecordingIndicator({ duration, className }: RecordingIndicatorProps) {
const formatDuration = (seconds: number) => {
const mins = Math.floor(seconds / 60)
const secs = Math.floor(seconds % 60)
return `${mins}:${secs.toString().padStart(2, "0")}`
}
return (
<div className={cn("flex items-center gap-2 text-sm text-red-500", className)} data-testid="recording-indicator">
<span className="relative flex h-2 w-2">
<span className="animate-ping absolute inline-flex h-full w-full rounded-full bg-red-400 opacity-75"></span>
<span className="relative inline-flex rounded-full h-2 w-2 bg-red-500"></span>
</span>
<span>Recording {formatDuration(duration)}</span>
</div>
)
}
@@ -0,0 +1,50 @@
import { Volume2Icon } from "lucide-react"
import React from "react"
import { cn } from "@/lib/utils"
export interface SpeakingIndicatorProps {
/** Whether audio is currently playing */
isPlaying: boolean
/** Optional CSS class name */
className?: string
/** Size variant */
size?: "sm" | "md" | "lg"
/** Show text label */
showLabel?: boolean
}
/**
* Visual indicator that shows when Cline is speaking (TTS audio is playing).
* Displays an animated speaker icon.
*/
const SpeakingIndicator: React.FC<SpeakingIndicatorProps> = ({ isPlaying, className, size = "md", showLabel = false }) => {
if (!isPlaying) {
return null
}
const sizeClasses = {
sm: "w-4 h-4",
md: "w-5 h-5",
lg: "w-6 h-6",
}
const textSizeClasses = {
sm: "text-xs",
md: "text-sm",
lg: "text-base",
}
return (
<div className={cn("flex items-center gap-2 text-muted-foreground", className)}>
<Volume2Icon
className={cn("animate-pulse", sizeClasses[size])}
style={{
animation: "pulse 1.5s cubic-bezier(0.4, 0, 0.6, 1) infinite",
}}
/>
{showLabel && <span className={cn("font-medium", textSizeClasses[size])}>Speaking...</span>}
</div>
)
}
export default SpeakingIndicator
@@ -0,0 +1,137 @@
import { PauseIcon, PlayIcon, SkipForwardIcon, Volume2Icon, VolumeXIcon } from "lucide-react"
import React, { useCallback } from "react"
import { Button } from "@/components/ui/button"
import { Tooltip, TooltipContent, TooltipTrigger } from "@/components/ui/tooltip"
import { cn } from "@/lib/utils"
import type { AudioPlayerHandle } from "./AudioPlayer"
export interface VoiceConversationControlsProps {
/** Reference to the audio player */
audioPlayerRef: React.RefObject<AudioPlayerHandle>
/** Whether audio is currently playing */
isPlaying: boolean
/** Whether audio is currently paused */
isPaused?: boolean
/** Current queue size */
queueSize: number
/** Whether controls should be disabled */
disabled?: boolean
/** Optional CSS class name */
className?: string
}
/**
* Voice conversation controls for managing TTS playback.
* Provides play/pause, skip, and stop functionality.
*/
const VoiceConversationControls: React.FC<VoiceConversationControlsProps> = ({
audioPlayerRef,
isPlaying,
isPaused = false,
queueSize,
disabled = false,
className,
}) => {
const handlePlayPause = useCallback(() => {
if (!audioPlayerRef.current) return
if (isPaused) {
audioPlayerRef.current.resume()
} else if (isPlaying) {
audioPlayerRef.current.pause()
}
}, [audioPlayerRef, isPlaying, isPaused])
const handleSkip = useCallback(() => {
if (!audioPlayerRef.current) return
// Stop current playback and let the queue continue
const queue = audioPlayerRef.current.getQueue()
if (queue.length > 0) {
// Remove current item to skip to next
audioPlayerRef.current.stop()
// Re-add remaining items
queue.slice(1).forEach((item) => {
audioPlayerRef.current?.enqueue(item)
})
}
}, [audioPlayerRef])
const handleStop = useCallback(() => {
if (!audioPlayerRef.current) return
audioPlayerRef.current.stop()
}, [audioPlayerRef])
// Don't show controls if not playing and no queue
if (!isPlaying && queueSize === 0) {
return null
}
return (
<div className={cn("flex items-center gap-1", className)}>
{/* Play/Pause Button */}
{isPlaying && (
<Tooltip>
<TooltipTrigger asChild>
<Button
className="h-8 w-8 p-0"
data-testid="voice-play-pause-button"
disabled={disabled}
onClick={handlePlayPause}
size="sm"
variant="ghost">
{isPaused ? <PlayIcon className="h-4 w-4" /> : <PauseIcon className="h-4 w-4" />}
</Button>
</TooltipTrigger>
<TooltipContent>{isPaused ? "Resume" : "Pause"}</TooltipContent>
</Tooltip>
)}
{/* Skip Button */}
{queueSize > 1 && (
<Tooltip>
<TooltipTrigger asChild>
<Button
className="h-8 w-8 p-0"
data-testid="voice-skip-button"
disabled={disabled}
onClick={handleSkip}
size="sm"
variant="ghost">
<SkipForwardIcon className="h-4 w-4" />
</Button>
</TooltipTrigger>
<TooltipContent>Skip to Next</TooltipContent>
</Tooltip>
)}
{/* Stop Button */}
{(isPlaying || queueSize > 0) && (
<Tooltip>
<TooltipTrigger asChild>
<Button
className="h-8 w-8 p-0 text-destructive hover:text-destructive"
data-testid="voice-stop-button"
disabled={disabled}
onClick={handleStop}
size="sm"
variant="ghost">
<VolumeXIcon className="h-4 w-4" />
</Button>
</TooltipTrigger>
<TooltipContent>Stop All Audio</TooltipContent>
</Tooltip>
)}
{/* Queue indicator */}
{queueSize > 1 && (
<div className="flex items-center gap-1 ml-1 text-xs text-muted-foreground">
<Volume2Icon className="h-3 w-3" />
<span>+{queueSize - 1}</span>
</div>
)}
</div>
)
}
export default VoiceConversationControls
@@ -0,0 +1,39 @@
import { MicIcon, MicOffIcon } from "lucide-react"
import { Button } from "@/components/ui/button"
import { Tooltip, TooltipContent, TooltipTrigger } from "@/components/ui/tooltip"
interface VoiceInputButtonProps {
isRecording: boolean
isTranscribing: boolean
disabled: boolean
onClick: () => void
}
/**
* Button component for voice input in Discuss Mode
* Shows different states for recording, transcribing, and idle
*/
export function VoiceInputButton({ isRecording, isTranscribing, disabled, onClick }: VoiceInputButtonProps) {
const getTooltipText = () => {
if (isTranscribing) return "Transcribing audio..."
if (isRecording) return "Click to stop recording"
return "Click to start voice input"
}
return (
<Tooltip>
<TooltipTrigger asChild>
<Button
className={isRecording ? "text-red-500 animate-pulse" : ""}
data-testid="voice-input-button"
disabled={disabled || isTranscribing}
onClick={onClick}
size="sm"
variant="ghost">
{isRecording ? <MicOffIcon className="h-4 w-4" /> : <MicIcon className="h-4 w-4" />}
</Button>
</TooltipTrigger>
<TooltipContent>{getTooltipText()}</TooltipContent>
</Tooltip>
)
}
@@ -1,6 +1,9 @@
import React from "react"
import React, { useState } from "react"
import ChatTextArea from "@/components/chat/ChatTextArea"
import QuotedMessagePreview from "@/components/chat/QuotedMessagePreview"
import VoiceRecorder from "@/components/chat/VoiceRecorder"
import { useClineAuth } from "@/context/ClineAuthContext"
import { useExtensionState } from "@/context/ExtensionStateContext"
import { ChatState, MessageHandlers, ScrollBehavior } from "../../types/chatTypes"
interface InputSectionProps {
@@ -10,6 +13,7 @@ interface InputSectionProps {
placeholderText: string
shouldDisableFilesAndImages: boolean
selectFilesAndImages: () => Promise<void>
stopAudio: () => void
}
/**
@@ -22,6 +26,7 @@ export const InputSection: React.FC<InputSectionProps> = ({
placeholderText,
shouldDisableFilesAndImages,
selectFilesAndImages,
stopAudio,
}) => {
const {
activeQuote,
@@ -39,6 +44,12 @@ export const InputSection: React.FC<InputSectionProps> = ({
} = chatState
const { isAtBottom, scrollToBottomAuto } = scrollBehavior
const { discussModeEnabled, dictationSettings, mode } = useExtensionState()
const { clineUser } = useClineAuth()
const [isVoiceRecording, setIsVoiceRecording] = useState(false)
// Check if we should show voice-only mode (Discuss Mode enabled in Plan mode)
const showVoiceOnlyMode = discussModeEnabled && mode === "plan"
return (
<>
@@ -52,27 +63,70 @@ export const InputSection: React.FC<InputSectionProps> = ({
</div>
)}
<ChatTextArea
activeQuote={activeQuote}
inputValue={inputValue}
onFocusChange={handleFocusChange}
onHeightChange={() => {
if (isAtBottom) {
scrollToBottomAuto()
}
}}
onSelectFilesAndImages={selectFilesAndImages}
onSend={() => messageHandlers.handleSendMessage(inputValue, selectedImages, selectedFiles)}
placeholderText={placeholderText}
ref={textAreaRef}
selectedFiles={selectedFiles}
selectedImages={selectedImages}
sendingDisabled={sendingDisabled}
setInputValue={setInputValue}
setSelectedFiles={setSelectedFiles}
setSelectedImages={setSelectedImages}
shouldDisableFilesAndImages={shouldDisableFilesAndImages}
/>
{showVoiceOnlyMode ? (
// Voice-only mode for Discuss Mode
<div className="flex justify-center items-center py-8 px-4">
<div className="flex flex-col items-center gap-4">
<VoiceRecorder
disabled={sendingDisabled}
isAuthenticated={!!clineUser?.uid}
language={dictationSettings?.dictationLanguage || "en"}
onProcessingStateChange={(isProcessing, message) => {
// No need to show processing in input since there's no text field
}}
onRecordingStateChange={(isRecording) => {
setIsVoiceRecording(isRecording)
// Stop any playing audio when user starts recording
if (isRecording) {
stopAudio()
}
}}
onTranscription={(text) => {
if (!text) return
// Create blessed audio element during user gesture (transcription completion)
try {
const audio = new Audio()
audio.preload = "auto"
// Store in window for the audio hook to access
;(window as any).__discussModeAudio = audio
} catch (e) {
console.warn("Could not create blessed audio element:", e)
}
// Automatically send the message
messageHandlers.handleSendMessage(text, [], [])
}}
/>
{!isVoiceRecording && (
<p className="text-xs text-muted-foreground text-center">Click the microphone to speak</p>
)}
</div>
</div>
) : (
// Normal mode with full text area
<ChatTextArea
activeQuote={activeQuote}
inputValue={inputValue}
onFocusChange={handleFocusChange}
onHeightChange={() => {
if (isAtBottom) {
scrollToBottomAuto()
}
}}
onSelectFilesAndImages={selectFilesAndImages}
onSend={() => messageHandlers.handleSendMessage(inputValue, selectedImages, selectedFiles)}
placeholderText={placeholderText}
ref={textAreaRef}
selectedFiles={selectedFiles}
selectedImages={selectedImages}
sendingDisabled={sendingDisabled}
setInputValue={setInputValue}
setSelectedFiles={setSelectedFiles}
setSelectedImages={setSelectedImages}
shouldDisableFilesAndImages={shouldDisableFilesAndImages}
/>
)}
</>
)
}
@@ -11,6 +11,7 @@ interface MessagesAreaProps {
scrollBehavior: ScrollBehavior
chatState: ChatState
messageHandlers: MessageHandlers
discussModeEnabled?: boolean
}
/**
@@ -24,6 +25,7 @@ export const MessagesArea: React.FC<MessagesAreaProps> = ({
scrollBehavior,
chatState,
messageHandlers,
discussModeEnabled,
}) => {
const {
virtuosoRef,
@@ -47,6 +49,7 @@ export const MessagesArea: React.FC<MessagesAreaProps> = ({
setActiveQuote,
inputValue,
messageHandlers,
discussModeEnabled,
),
[
groupedMessages,
@@ -57,6 +60,7 @@ export const MessagesArea: React.FC<MessagesAreaProps> = ({
setActiveQuote,
inputValue,
messageHandlers,
discussModeEnabled,
],
)
@@ -15,6 +15,7 @@ interface MessageRendererProps {
onSetQuote: (quote: string | null) => void
inputValue: string
messageHandlers: MessageHandlers
discussModeEnabled?: boolean
}
/**
@@ -32,6 +33,7 @@ export const MessageRenderer: React.FC<MessageRendererProps> = ({
onSetQuote,
inputValue,
messageHandlers,
discussModeEnabled,
}) => {
// Browser session group
if (Array.isArray(messageOrGroup)) {
@@ -58,6 +60,7 @@ export const MessageRenderer: React.FC<MessageRendererProps> = ({
// Regular message
return (
<ChatRow
discussModeEnabled={discussModeEnabled}
inputValue={inputValue}
isExpanded={expandedRows[messageOrGroup.ts] || false}
isLast={isLast}
@@ -86,9 +89,11 @@ export const createMessageRenderer = (
onSetQuote: (quote: string | null) => void,
inputValue: string,
messageHandlers: MessageHandlers,
discussModeEnabled?: boolean,
) => {
return (index: number, messageOrGroup: ClineMessage | ClineMessage[]) => (
<MessageRenderer
discussModeEnabled={discussModeEnabled}
expandedRows={expandedRows}
groupedMessages={groupedMessages}
index={index}
@@ -0,0 +1,497 @@
import { ClineMessage } from "@shared/ExtensionMessage"
import { SynthesizeRequest } from "@shared/proto/cline/tts"
import { useEffect, useRef, useState } from "react"
import { useExtensionState } from "@/context/ExtensionStateContext"
import { TtsServiceClient } from "@/services/grpc-client"
interface AudioQueueItem {
text: string
messageTs: number
}
/**
* Hook that handles automatic TTS synthesis and playback for Discuss Mode
*/
export function useDiscussModeAudio(messages: ClineMessage[]) {
const { mode, discussModeEnabled, discussModeSettings, currentTaskItem } = useExtensionState()
const [audioQueue, setAudioQueue] = useState<AudioQueueItem[]>([])
const [isPlaying, setIsPlaying] = useState(false)
const [currentAudio, setCurrentAudio] = useState<HTMLAudioElement | null>(null)
const processedMessagesRef = useRef(new Set<number>())
const audioRef = useRef<HTMLAudioElement | null>(null)
const lastTaskIdRef = useRef<string | undefined>(undefined)
const lastMessageCountRef = useRef(0)
const taskStartTimeRef = useRef<number>(Date.now()) // Track when we started tracking this task
const activeBlobUrlsRef = useRef<Set<string>>(new Set()) // Track all blob URLs for cleanup
const audioElementRef = useRef<HTMLAudioElement | null>(null) // Pre-created audio element for autoplay
const pendingAudioRef = useRef<{
url: string
audio: HTMLAudioElement
onEnded: () => void
onError: (e: Event) => void
} | null>(null) // Audio waiting for user interaction
// Use blessed audio element from window if available, otherwise create one
useEffect(() => {
// Check if we have a blessed audio element from a user gesture
const blessedAudio = (window as any).__discussModeAudio
if (blessedAudio && !audioElementRef.current) {
console.log("[DiscussModeAudio] Using blessed audio element from user gesture")
audioElementRef.current = blessedAudio
// Clear it so we don't reuse it
delete (window as any).__discussModeAudio
} else if (!audioElementRef.current) {
console.log("[DiscussModeAudio] Creating audio element (may not be blessed)")
const audio = new Audio()
audio.preload = "auto"
audioElementRef.current = audio
}
return () => {
if (audioElementRef.current) {
audioElementRef.current.pause()
audioElementRef.current.src = ""
audioElementRef.current = null
}
}
}, [])
// Clear processed messages when task changes OR message array is reset
useEffect(() => {
const currentTaskId = currentTaskItem?.id
const currentMessageCount = messages.length
// Helper function to cleanup all audio resources
const cleanupAllAudio = () => {
// Stop and cleanup current audio
if (audioRef.current) {
audioRef.current.pause()
audioRef.current.currentTime = 0
audioRef.current.src = ""
audioRef.current = null
}
// Clean up pending audio
if (pendingAudioRef.current) {
const pending = pendingAudioRef.current
pending.audio.removeEventListener("ended", pending.onEnded)
pending.audio.removeEventListener("error", pending.onError)
pending.audio.src = ""
URL.revokeObjectURL(pending.url)
activeBlobUrlsRef.current.delete(pending.url)
pendingAudioRef.current = null
}
// Revoke ALL blob URLs to free memory and prevent replay
activeBlobUrlsRef.current.forEach((url) => {
console.log("[DiscussModeAudio] Revoking blob URL:", url.substring(0, 50))
URL.revokeObjectURL(url)
})
activeBlobUrlsRef.current.clear()
// Clear all state
processedMessagesRef.current.clear()
setAudioQueue([])
setIsPlaying(false)
setCurrentAudio(null)
taskStartTimeRef.current = Date.now()
}
// Task ID changed
if (currentTaskId && currentTaskId !== lastTaskIdRef.current) {
console.log("[DiscussModeAudio] Task ID changed, CLEANING UP ALL AUDIO", {
oldTaskId: lastTaskIdRef.current,
newTaskId: currentTaskId,
activeBlobUrls: activeBlobUrlsRef.current.size,
})
cleanupAllAudio()
lastTaskIdRef.current = currentTaskId
lastMessageCountRef.current = currentMessageCount
return
}
// Message count dropped significantly (task switch or history cleared)
if (currentMessageCount < lastMessageCountRef.current - 2) {
console.log("[DiscussModeAudio] Message count dropped, CLEANING UP ALL AUDIO", {
oldCount: lastMessageCountRef.current,
newCount: currentMessageCount,
activeBlobUrls: activeBlobUrlsRef.current.size,
})
cleanupAllAudio()
}
lastMessageCountRef.current = currentMessageCount
}, [currentTaskItem?.id, messages.length])
// Detect new text messages and add to queue
useEffect(() => {
console.log("[DiscussModeAudio] Effect triggered", {
mode,
discussModeEnabled,
autoSpeak: discussModeSettings?.autoSpeak,
messageCount: messages.length,
currentTaskId: currentTaskItem?.id,
})
// Only process if in Plan Mode with Discuss Mode enabled and auto-speak on
if (mode !== "plan") {
console.log("[DiscussModeAudio] Not in plan mode, skipping")
return
}
if (!discussModeEnabled) {
console.log("[DiscussModeAudio] Discuss mode not enabled, skipping")
return
}
if (!discussModeSettings?.autoSpeak) {
console.log("[DiscussModeAudio] Auto-speak not enabled, skipping")
return
}
// Find the last message
const lastMessage = messages[messages.length - 1]
if (!lastMessage) {
console.log("[DiscussModeAudio] No messages found")
return
}
console.log("[DiscussModeAudio] Last message:", {
say: lastMessage.say,
ask: lastMessage.ask,
hasText: !!lastMessage.text,
partial: lastMessage.partial,
ts: lastMessage.ts,
taskStartTime: taskStartTimeRef.current,
messageAge: Date.now() - lastMessage.ts,
isOldMessage: lastMessage.ts < taskStartTimeRef.current,
alreadyProcessed: processedMessagesRef.current.has(lastMessage.ts),
})
// Check if it's a response from Cline that we haven't processed
// IMPORTANT: Only process Cline's responses (ask is set), not user messages (say === "text")
// In Plan Mode, Cline uses ask="plan_mode_respond" or ask="followup"
const isClineResponse = lastMessage.ask !== undefined
// CRITICAL: Only process messages created AFTER we started tracking this task
// This prevents old messages from playing when switching tasks
if (lastMessage.ts < taskStartTimeRef.current) {
console.log("[DiscussModeAudio] Skipping old message from before current task started", {
messageTs: lastMessage.ts,
taskStartTime: taskStartTimeRef.current,
messageAge: Date.now() - lastMessage.ts,
})
return
}
if (isClineResponse && lastMessage.text && !lastMessage.partial && !processedMessagesRef.current.has(lastMessage.ts)) {
// Extract natural text from JSON response format
let textToSpeak = lastMessage.text
try {
// Parse JSON to extract just the response text
const parsed = JSON.parse(lastMessage.text)
if (parsed.response) {
textToSpeak = parsed.response
}
} catch (e) {
// If not JSON, use the text as-is
console.log("[DiscussModeAudio] Text is not JSON, using as-is")
}
// Skip if no actual text content
if (!textToSpeak || textToSpeak.trim().length === 0) {
console.log("[DiscussModeAudio] No text content to speak")
return
}
// Truncate very long responses (ElevenLabs has limits)
const maxLength = 5000 // ElevenLabs can handle ~5000 chars
if (textToSpeak.length > maxLength) {
console.log(`[DiscussModeAudio] Truncating long response from ${textToSpeak.length} to ${maxLength} chars`)
textToSpeak = textToSpeak.substring(0, maxLength) + "... response truncated for audio."
}
// Mark as processed
processedMessagesRef.current.add(lastMessage.ts)
// Add to queue
console.log("[DiscussModeAudio] New Cline response detected, adding to queue:", textToSpeak.substring(0, 50))
setAudioQueue((prev) => [...prev, { text: textToSpeak, messageTs: lastMessage.ts }])
}
}, [messages, mode, discussModeEnabled, discussModeSettings?.autoSpeak])
// Process audio queue
useEffect(() => {
// First, check if we have pending audio and a blessed element to retry it
const blessedAudio = (window as any).__discussModeAudio
const pending = pendingAudioRef.current
if (blessedAudio && pending && !isPlaying) {
console.log("[DiscussModeAudio] Blessed audio element available, retrying pending audio playback")
// Use the blessed audio element
audioElementRef.current = blessedAudio
delete (window as any).__discussModeAudio
// Update the pending audio to use the blessed element
pending.audio = blessedAudio
// Set up the audio and retry playback
pending.audio.src = pending.url
pending.audio.addEventListener("ended", pending.onEnded)
pending.audio.addEventListener("error", pending.onError)
audioRef.current = pending.audio
setCurrentAudio(pending.audio)
setIsPlaying(true)
// Try to play
pending.audio
.play()
.then(() => {
console.log("[DiscussModeAudio] Pending audio playing successfully after user interaction")
pendingAudioRef.current = null // Clear pending
})
.catch((err) => {
console.error("[DiscussModeAudio] Failed to play pending audio even with blessed element:", err)
// Clean up on failure
pending.audio.removeEventListener("ended", pending.onEnded)
pending.audio.removeEventListener("error", pending.onError)
pending.audio.src = ""
URL.revokeObjectURL(pending.url)
activeBlobUrlsRef.current.delete(pending.url)
pendingAudioRef.current = null
setIsPlaying(false)
setCurrentAudio(null)
audioRef.current = null
// Remove from queue since we can't play it
setAudioQueue((prev) => prev.slice(1))
})
return
}
if (isPlaying || audioQueue.length === 0 || !discussModeSettings?.selectedVoice) {
return
}
const processNextInQueue = async () => {
const nextItem = audioQueue[0]
if (!nextItem) return
try {
setIsPlaying(true)
console.log("[DiscussModeAudio] Synthesizing speech for message:", nextItem.messageTs)
// Call TTS service
const response = await TtsServiceClient.SynthesizeSpeech(
SynthesizeRequest.create({
text: nextItem.text,
voiceId: discussModeSettings.selectedVoice,
speed: discussModeSettings.speechSpeed || 1.0,
}),
)
if (response.error) {
console.error("[DiscussModeAudio] TTS synthesis error:", response.error)
// Remove from queue and try next
setAudioQueue((prev) => prev.slice(1))
setIsPlaying(false)
return
}
if (!response.audioData || response.audioData.length === 0) {
console.error("[DiscussModeAudio] No audio data received")
setAudioQueue((prev) => prev.slice(1))
setIsPlaying(false)
return
}
// Convert Buffer to Blob
const audioBlob = new Blob([new Uint8Array(response.audioData)], {
type: response.contentType || "audio/mpeg",
})
const audioUrl = URL.createObjectURL(audioBlob)
// Track this blob URL for cleanup
activeBlobUrlsRef.current.add(audioUrl)
console.log("[DiscussModeAudio] Created blob URL, total active:", activeBlobUrlsRef.current.size)
// Check for blessed audio element from user gesture
const blessedAudio = (window as any).__discussModeAudio
if (blessedAudio) {
console.log("[DiscussModeAudio] Found blessed audio element, using it")
audioElementRef.current = blessedAudio
delete (window as any).__discussModeAudio
}
// Use audio element (blessed or fallback)
const audio = audioElementRef.current
if (!audio) {
console.error("[DiscussModeAudio] Audio element not available!")
setAudioQueue((prev) => prev.slice(1))
setIsPlaying(false)
return
}
// Set up event handlers
const onEnded = () => {
console.log("[DiscussModeAudio] Audio playback completed, cleaning up")
URL.revokeObjectURL(audioUrl)
activeBlobUrlsRef.current.delete(audioUrl)
setCurrentAudio(null)
audioRef.current = null
// Remove from queue
setAudioQueue((prev) => prev.slice(1))
setIsPlaying(false)
// Clean up handlers
audio.removeEventListener("ended", onEnded)
audio.removeEventListener("error", onError)
}
const onError = (e: Event) => {
console.error("[DiscussModeAudio] Audio playback error:", e)
URL.revokeObjectURL(audioUrl)
activeBlobUrlsRef.current.delete(audioUrl)
setCurrentAudio(null)
audioRef.current = null
setAudioQueue((prev) => prev.slice(1))
setIsPlaying(false)
// Clean up handlers
audio.removeEventListener("ended", onEnded)
audio.removeEventListener("error", onError)
}
audio.addEventListener("ended", onEnded)
audio.addEventListener("error", onError)
// Update audio src and play
audio.src = audioUrl
audioRef.current = audio
setCurrentAudio(audio)
console.log("[DiscussModeAudio] Playing audio with pre-created element...")
// Try to play
try {
await audio.play()
console.log("[DiscussModeAudio] Audio playing successfully")
} catch (playError) {
console.warn(
"[DiscussModeAudio] Autoplay blocked by browser. Keeping audio ready for retry after user interaction.",
)
// Browser blocked autoplay - keep audio ready for retry when user interacts
// Store the audio setup so we can retry when blessed audio element becomes available
pendingAudioRef.current = {
url: audioUrl,
audio: audio,
onEnded,
onError,
}
// Don't remove from queue or revoke blob URL yet - we'll retry
setIsPlaying(false)
// Note: Audio will automatically retry when user interacts and creates blessed audio element
return
}
} catch (error) {
console.error("[DiscussModeAudio] Error processing audio:", error)
setAudioQueue((prev) => prev.slice(1))
setIsPlaying(false)
}
}
processNextInQueue()
}, [audioQueue, isPlaying, discussModeSettings?.selectedVoice, discussModeSettings?.speechSpeed])
// Cleanup on unmount - revoke ALL blob URLs
useEffect(() => {
return () => {
console.log("[DiscussModeAudio] Component unmounting, cleaning up all audio")
if (audioRef.current) {
audioRef.current.pause()
audioRef.current.src = ""
audioRef.current = null
}
// Clean up pending audio
if (pendingAudioRef.current) {
const pending = pendingAudioRef.current
pending.audio.removeEventListener("ended", pending.onEnded)
pending.audio.removeEventListener("error", pending.onError)
pending.audio.src = ""
URL.revokeObjectURL(pending.url)
pendingAudioRef.current = null
}
// Revoke all blob URLs
activeBlobUrlsRef.current.forEach((url) => {
URL.revokeObjectURL(url)
})
activeBlobUrlsRef.current.clear()
}
}, [])
// Clear queue and DELETE all audio blobs when Discuss Mode is disabled or mode changes
useEffect(() => {
if (mode !== "plan" || !discussModeEnabled) {
console.log("[DiscussModeAudio] Mode/Discuss Mode changed, DELETING ALL AUDIO BLOBS", {
mode,
discussModeEnabled,
activeBlobUrls: activeBlobUrlsRef.current.size,
})
// Stop current audio
if (audioRef.current) {
audioRef.current.pause()
audioRef.current.currentTime = 0
audioRef.current.src = ""
audioRef.current = null
}
// Clean up pending audio
if (pendingAudioRef.current) {
const pending = pendingAudioRef.current
pending.audio.removeEventListener("ended", pending.onEnded)
pending.audio.removeEventListener("error", pending.onError)
pending.audio.src = ""
URL.revokeObjectURL(pending.url)
activeBlobUrlsRef.current.delete(pending.url)
pendingAudioRef.current = null
}
// Revoke ALL blob URLs to delete audio files from memory
activeBlobUrlsRef.current.forEach((url) => {
console.log("[DiscussModeAudio] Deleting audio blob:", url.substring(0, 50))
URL.revokeObjectURL(url)
})
activeBlobUrlsRef.current.clear()
// Clear all state
setAudioQueue([])
setIsPlaying(false)
setCurrentAudio(null)
processedMessagesRef.current.clear()
taskStartTimeRef.current = Date.now()
console.log("[DiscussModeAudio] All audio blobs deleted and state cleared")
}
}, [mode, discussModeEnabled])
// Expose function to stop current audio (useful when user starts recording)
const stopAudio = () => {
if (audioRef.current) {
console.log("[DiscussModeAudio] Stopping current audio playback")
audioRef.current.pause()
audioRef.current.currentTime = 0
// Clear queue to prevent auto-continuing
setAudioQueue([])
setIsPlaying(false)
}
}
return {
isPlaying,
queueLength: audioQueue.length,
stopAudio,
}
}
@@ -0,0 +1,132 @@
import { useCallback, useEffect, useState } from "react"
import { useExtensionState } from "@/context/ExtensionStateContext"
import { DictationServiceClient } from "@/services/grpc-client"
/**
* Custom hook for handling voice input in Discuss Mode
* Leverages the existing DictationService infrastructure for recording and transcription
*/
export function useDiscussVoiceInput() {
const [isRecording, setIsRecording] = useState(false)
const [isTranscribing, setIsTranscribing] = useState(false)
const [recordingDuration, setRecordingDuration] = useState(0)
const [error, setError] = useState<string | null>(null)
const { discussModeEnabled, dictationSettings } = useExtensionState()
// Auto-update recording duration while recording
useEffect(() => {
if (!isRecording) return
const interval = setInterval(async () => {
try {
const status = await DictationServiceClient.getRecordingStatus({})
setRecordingDuration(status.durationSeconds)
} catch (err) {
console.error("Failed to get recording status:", err)
}
}, 100)
return () => clearInterval(interval)
}, [isRecording])
/**
* Starts audio recording
*/
const startRecording = useCallback(async () => {
try {
setError(null)
const result = await DictationServiceClient.startRecording({})
if (result.success) {
setIsRecording(true)
} else {
setError(result.error || "Failed to start recording")
}
} catch (err) {
const errorMessage = err instanceof Error ? err.message : "Failed to start recording"
setError(errorMessage)
console.error("Start recording error:", err)
}
}, [])
/**
* Stops recording and transcribes the audio
* @param onTranscriptionComplete - Callback with transcribed text
*/
const stopRecording = useCallback(
async (onTranscriptionComplete: (text: string) => void) => {
try {
setIsRecording(false)
setIsTranscribing(true)
setError(null)
// Stop recording and get audio data
const audioResult = await DictationServiceClient.stopRecording({})
if (!audioResult.success || !audioResult.audioBase64) {
setError(audioResult.error || "Failed to capture audio")
setIsTranscribing(false)
return
}
// Transcribe the audio
const transcription = await DictationServiceClient.transcribeAudio({
audioBase64: audioResult.audioBase64,
language: dictationSettings?.dictationLanguage || "en",
})
if (transcription.error) {
setError(transcription.error)
} else if (transcription.text) {
// Successfully transcribed - pass text to callback
onTranscriptionComplete(transcription.text)
}
} catch (err) {
const errorMessage = err instanceof Error ? err.message : "Transcription failed"
setError(errorMessage)
console.error("Transcription error:", err)
} finally {
setIsTranscribing(false)
setRecordingDuration(0)
}
},
[dictationSettings],
)
/**
* Cancels the current recording without transcribing
*/
const cancelRecording = useCallback(async () => {
try {
await DictationServiceClient.cancelRecording({})
setIsRecording(false)
setRecordingDuration(0)
setError(null)
} catch (err) {
console.error("Cancel recording error:", err)
}
}, [])
/**
* Clears any error messages
*/
const clearError = useCallback(() => {
setError(null)
}, [])
// Check if voice input is available
// Requires both Discuss Mode to be enabled and platform support (currently macOS only)
const isAvailable = discussModeEnabled && (dictationSettings?.featureEnabled ?? false)
return {
isRecording,
isTranscribing,
recordingDuration,
error,
isAvailable,
startRecording,
stopRecording,
cancelRecording,
clearError,
}
}
@@ -0,0 +1,117 @@
import { MessageSquare, Mic, Volume2 } from "lucide-react"
import { memo } from "react"
export type ConversationState = "idle" | "listening" | "thinking" | "speaking"
interface ConversationStatusIndicatorProps {
state: ConversationState
className?: string
}
/**
* ConversationStatusIndicator displays the current state of the voice conversation
* with animated visual feedback for Listening, Thinking, and Speaking states.
*/
export const ConversationStatusIndicator = memo(({ state, className = "" }: ConversationStatusIndicatorProps) => {
// Don't render anything in idle state
if (state === "idle") {
return null
}
const getStateConfig = () => {
switch (state) {
case "listening":
return {
icon: Mic,
text: "Listening...",
bgColor: "bg-blue-500/10",
textColor: "text-blue-500",
iconColor: "text-blue-500",
pulseColor: "bg-blue-500",
}
case "thinking":
return {
icon: MessageSquare,
text: "Thinking...",
bgColor: "bg-purple-500/10",
textColor: "text-purple-500",
iconColor: "text-purple-500",
pulseColor: "bg-purple-500",
}
case "speaking":
return {
icon: Volume2,
text: "Speaking...",
bgColor: "bg-green-500/10",
textColor: "text-green-500",
iconColor: "text-green-500",
pulseColor: "bg-green-500",
}
}
}
const config = getStateConfig()
const Icon = config.icon
return (
<div
className={`flex items-center gap-2 px-3 py-2 rounded-lg ${config.bgColor} transition-all duration-300 ${className}`}>
{/* Animated Icon */}
<div className="relative">
<Icon className={`w-4 h-4 ${config.iconColor}`} strokeWidth={2} />
{/* Pulse animation */}
<span className="absolute inset-0 flex items-center justify-center">
<span
className={`absolute w-4 h-4 rounded-full ${config.pulseColor} opacity-75 animate-ping`}
style={{
animationDuration: state === "speaking" ? "1.5s" : "2s",
}}
/>
</span>
</div>
{/* Status Text */}
<span className={`text-sm font-medium ${config.textColor}`}>{config.text}</span>
{/* Visual Indicator - Wave animation for speaking/listening */}
{(state === "speaking" || state === "listening") && (
<div className="flex items-center gap-0.5 ml-1">
{[0, 1, 2].map((i) => (
<div
className={`w-0.5 rounded-full ${config.pulseColor}`}
key={i}
style={{
height: "12px",
animation: `wave 1s ease-in-out ${i * 0.1}s infinite`,
}}
/>
))}
</div>
)}
{/* Spinner for thinking */}
{state === "thinking" && (
<div className="ml-1">
<div
className={`w-3 h-3 border-2 border-t-transparent rounded-full ${config.iconColor} animate-spin`}
style={{ borderColor: `currentColor transparent transparent transparent` }}
/>
</div>
)}
<style>{`
@keyframes wave {
0%, 100% {
height: 8px;
}
50% {
height: 16px;
}
}
`}</style>
</div>
)
})
ConversationStatusIndicator.displayName = "ConversationStatusIndicator"
@@ -0,0 +1,128 @@
import { BooleanRequest, EmptyRequest } from "@shared/proto/cline/common"
import { Mic, MicOff } from "lucide-react"
import { useCallback, useEffect, useState } from "react"
import { useExtensionState } from "@/context/ExtensionStateContext"
import { TtsServiceClient, UiServiceClient } from "@/services/grpc-client"
interface DiscussModeToggleProps {
className?: string
}
/**
* Toggle button to enable/disable Discuss Mode in Plan Mode.
* When enabled, Cline's responses are spoken via TTS and conversations
* can flow naturally with voice input/output.
*/
export function DiscussModeToggle({ className = "" }: DiscussModeToggleProps) {
const { discussModeEnabled, mode, discussModeSettings } = useExtensionState()
const [isToggling, setIsToggling] = useState(false)
const [isConfigured, setIsConfigured] = useState(false)
// Only show in Plan Mode
const isPlanMode = mode === "plan"
// Check if API key is configured on mount and when settings change
useEffect(() => {
const checkConfiguration = async () => {
try {
const response = await TtsServiceClient.CheckApiKeyConfigured(EmptyRequest.create())
const hasVoice = !!discussModeSettings?.selectedVoice
setIsConfigured(response.isValid && hasVoice)
} catch (error) {
console.error("[DiscussModeToggle] Failed to check TTS configuration:", error)
setIsConfigured(false)
}
}
checkConfiguration()
}, [discussModeSettings])
const handleToggle = useCallback(async () => {
if (isToggling || !isPlanMode) return
// If not configured, show settings hint
if (!isConfigured) {
// TODO: Show tooltip or open settings panel
console.warn("Discuss Mode requires ElevenLabs API key configuration")
return
}
setIsToggling(true)
try {
// Toggle discuss mode via gRPC
await UiServiceClient.setDiscussModeEnabled(BooleanRequest.create({ value: !discussModeEnabled }))
} catch (error) {
console.error("Error toggling Discuss Mode:", error)
} finally {
setIsToggling(false)
}
}, [discussModeEnabled, isPlanMode, isConfigured, isToggling])
// Don't render in Act Mode
if (!isPlanMode) {
return null
}
const isEnabled = discussModeEnabled && isConfigured
return (
<button
className={`
group relative flex items-center gap-2 px-3 py-2 rounded-lg
transition-all duration-200 ease-in-out
${
isEnabled
? "bg-vscode-button-background text-vscode-button-foreground hover:bg-vscode-button-hoverBackground"
: "bg-vscode-button-secondaryBackground text-vscode-button-secondaryForeground hover:bg-vscode-button-secondaryHoverBackground"
}
${!isConfigured ? "opacity-50 cursor-not-allowed" : "cursor-pointer"}
${isToggling ? "opacity-70" : ""}
${className}
`}
disabled={isToggling || !isConfigured}
onClick={handleToggle}
title={
!isConfigured
? "Configure ElevenLabs API key in settings to enable Discuss Mode"
: isEnabled
? "Disable Discuss Mode (voice conversations)"
: "Enable Discuss Mode (voice conversations)"
}>
{/* Icon */}
<div
className={`
transition-transform duration-200
${isEnabled ? "scale-110" : "scale-100"}
`}>
{isEnabled ? <Mic className="w-4 h-4" strokeWidth={2} /> : <MicOff className="w-4 h-4" strokeWidth={2} />}
</div>
{/* Label */}
<span className="text-sm font-medium">{isEnabled ? "Discussing" : "Discuss Mode"}</span>
{/* Status indicator */}
{isEnabled && (
<div className="relative">
<div className="w-2 h-2 bg-green-500 rounded-full animate-pulse" />
<div className="absolute inset-0 w-2 h-2 bg-green-500 rounded-full opacity-50 animate-ping" />
</div>
)}
{/* Hover tooltip for unconfigured state */}
{!isConfigured && (
<div
className="
invisible group-hover:visible
absolute top-full left-0 mt-2 p-2
bg-vscode-notifications-background
border border-vscode-notifications-border
rounded shadow-lg z-50 w-64
text-xs text-vscode-notifications-foreground
">
Configure ElevenLabs API key in Settings Voice to enable Discuss Mode
</div>
)}
</button>
)
}
export default DiscussModeToggle
@@ -0,0 +1,81 @@
import { CheckCircle2, MessageCircle, PlayCircle } from "lucide-react"
import { memo } from "react"
interface PlanCompletionCardProps {
planSummary?: string
onSwitchToActMode?: () => void
onContinueDiscussing?: () => void
className?: string
}
/**
* PlanCompletionCard displays when Cline has completed planning and is ready
* to switch to Act Mode for implementation. Shows plan summary and action buttons.
*/
export const PlanCompletionCard = memo(
({ planSummary, onSwitchToActMode, onContinueDiscussing, className = "" }: PlanCompletionCardProps) => {
const handleSwitchToActMode = () => {
// Mode switching will be handled by parent component via callback
onSwitchToActMode?.()
}
const handleContinueDiscussing = () => {
onContinueDiscussing?.()
}
return (
<div
className={`bg-gradient-to-r from-green-500/10 to-blue-500/10 border border-green-500/20 rounded-lg p-4 ${className}`}>
{/* Header */}
<div className="flex items-center gap-2 mb-3">
<CheckCircle2 className="w-5 h-5 text-green-500" strokeWidth={2} />
<h3 className="text-base font-semibold text-(--vscode-foreground)">Plan Complete!</h3>
</div>
{/* Plan Summary */}
{planSummary && (
<div className="mb-4">
<p className="text-sm text-(--vscode-descriptionForeground) leading-relaxed whitespace-pre-wrap">
{planSummary}
</p>
</div>
)}
{/* Description */}
<p className="text-sm text-(--vscode-descriptionForeground) mb-4">
The plan is ready for implementation. You can switch to Act Mode to begin, or continue discussing to refine
the plan further.
</p>
{/* Action Buttons */}
<div className="flex flex-col sm:flex-row gap-2">
{/* Primary Action - Switch to Act Mode */}
<button
className="flex items-center justify-center gap-2 px-4 py-2 bg-green-600 hover:bg-green-700 text-white rounded-md transition-colors font-medium text-sm flex-1"
onClick={handleSwitchToActMode}>
<PlayCircle className="w-4 h-4" />
<span>Switch to Act Mode</span>
</button>
{/* Secondary Action - Continue Discussing */}
<button
className="flex items-center justify-center gap-2 px-4 py-2 bg-(--vscode-button-secondaryBackground) hover:bg-(--vscode-button-secondaryHoverBackground) text-(--vscode-button-secondaryForeground) rounded-md transition-colors font-medium text-sm flex-1"
onClick={handleContinueDiscussing}>
<MessageCircle className="w-4 h-4" />
<span>Continue Discussing</span>
</button>
</div>
{/* Helper Text */}
<div className="mt-3 pt-3 border-t border-(--vscode-widget-border)">
<p className="text-xs text-(--vscode-descriptionForeground)">
💡 <strong>Tip:</strong> In Act Mode, Cline will silently implement the plan using tools. Discuss Mode
will automatically disable.
</p>
</div>
</div>
)
},
)
PlanCompletionCard.displayName = "PlanCompletionCard"
@@ -6,6 +6,7 @@ import {
FlaskConical,
Info,
type LucideIcon,
Mic,
SlidersHorizontal,
SquareMousePointer,
SquareTerminal,
@@ -27,6 +28,7 @@ import DebugSection from "./sections/DebugSection"
import FeatureSettingsSection from "./sections/FeatureSettingsSection"
import GeneralSettingsSection from "./sections/GeneralSettingsSection"
import TerminalSettingsSection from "./sections/TerminalSettingsSection"
import VoiceSettingsSection from "./sections/VoiceSettingsSection"
const IS_DEV = process.env.IS_DEV
@@ -76,6 +78,13 @@ export const SETTINGS_TABS: SettingsTab[] = [
headerText: "General Settings",
icon: Wrench,
},
{
id: "voice",
name: "Voice",
tooltipText: "Voice Settings",
headerText: "Voice Settings",
icon: Mic,
},
{
id: "about",
name: "About",
@@ -125,6 +134,7 @@ const SettingsView = ({ onDone, targetSection }: SettingsViewProps) => {
features: FeatureSettingsSection,
browser: BrowserSettingsSection,
terminal: TerminalSettingsSection,
voice: VoiceSettingsSection,
about: AboutSection,
debug: DebugSection,
}),
@@ -0,0 +1,435 @@
import { EmptyRequest } from "@shared/proto/cline/common"
import { ValidateApiKeyRequest } from "@shared/proto/cline/tts"
import { DiscussVoiceSettingsRequest } from "@shared/proto/cline/ui"
import { VSCodeButton, VSCodeCheckbox, VSCodeTextField } from "@vscode/webview-ui-toolkit/react"
import { Eye, EyeOff, Volume2 } from "lucide-react"
import { useCallback, useEffect, useState } from "react"
import { useExtensionState } from "@/context/ExtensionStateContext"
import { TtsServiceClient, UiServiceClient } from "@/services/grpc-client"
import Section from "../Section"
interface VoiceSettingsSectionProps {
renderSectionHeader: (tabId: string) => JSX.Element | null
}
import type { Voice as TtsVoice } from "@shared/proto/cline/tts"
const VoiceSettingsSection = ({ renderSectionHeader }: VoiceSettingsSectionProps) => {
const { discussModeSettings } = useExtensionState()
const [apiKey, setApiKey] = useState("")
const [showApiKey, setShowApiKey] = useState(false)
const [isValidating, setIsValidating] = useState(false)
const [validationError, setValidationError] = useState<string | null>(null)
const [isValid, setIsValid] = useState(false)
const [voices, setVoices] = useState<TtsVoice[]>([])
const [isLoadingVoices, setIsLoadingVoices] = useState(false)
const [voicesError, setVoicesError] = useState<string | null>(null)
const [selectedVoice, setSelectedVoice] = useState(discussModeSettings?.selectedVoice || "")
const [speechSpeed, setSpeechSpeed] = useState(discussModeSettings?.speechSpeed || 1.0)
const [autoSpeak, setAutoSpeak] = useState(discussModeSettings?.autoSpeak || false)
const [autoListen, setAutoListen] = useState(discussModeSettings?.autoListen || false)
// Sync with global settings whenever they change
useEffect(() => {
if (discussModeSettings) {
setSelectedVoice(discussModeSettings.selectedVoice || "")
setSpeechSpeed(discussModeSettings.speechSpeed || 1.0)
setAutoSpeak(discussModeSettings.autoSpeak || false)
setAutoListen(discussModeSettings.autoListen || false)
}
}, [discussModeSettings])
// Load API key and voices on mount
useEffect(() => {
const loadInitialData = async () => {
// Check if API key is already configured
try {
const response = await TtsServiceClient.CheckApiKeyConfigured(EmptyRequest.create())
if (response.isValid) {
// API key exists and is valid
setIsValid(true)
setValidationError(null)
// Don't show the actual key for security, but indicate it exists
setApiKey("••••••••••••••••")
// Load voices automatically
await loadVoices()
}
} catch (error) {
console.error("Failed to check API key:", error)
}
}
loadInitialData()
}, [])
const validateApiKey = async (key: string) => {
if (!key.trim()) {
setValidationError("API key is required")
setIsValid(false)
return
}
setIsValidating(true)
setValidationError(null)
try {
const response = await TtsServiceClient.ValidateApiKey(ValidateApiKeyRequest.create({ apiKey: key }))
if (response.isValid) {
setIsValid(true)
setValidationError(null)
// Note: API key will be saved by the backend handler
// Load voices
await loadVoices()
} else {
setIsValid(false)
setValidationError(response.error || "Invalid API key")
}
} catch (error) {
setIsValid(false)
setValidationError("Failed to validate API key: " + (error as Error).message)
} finally {
setIsValidating(false)
}
}
const loadVoices = async () => {
setIsLoadingVoices(true)
setVoicesError(null)
try {
const response = await TtsServiceClient.GetAvailableVoices(EmptyRequest.create())
if (response.error) {
setVoicesError(response.error)
setVoices([])
} else {
setVoices(response.voices)
// Auto-select "Liam" voice if available and no voice is currently selected
if (response.voices.length > 0 && !selectedVoice) {
// Try to find Liam voice
const liamVoice = response.voices.find((v) => v.name.toLowerCase().includes("liam"))
if (liamVoice) {
// Found Liam, select it automatically
setSelectedVoice(liamVoice.id)
// Save to state
await UiServiceClient.updateDiscussVoiceSettings(
DiscussVoiceSettingsRequest.create({ selectedVoice: liamVoice.id }),
)
console.log("Auto-selected Liam voice:", liamVoice.name)
}
}
}
} catch (error) {
setVoicesError("Failed to load voices: " + (error as Error).message)
setVoices([])
} finally {
setIsLoadingVoices(false)
}
}
const handleApiKeyChange = (e: any) => {
const value = e.target.value
setApiKey(value)
setIsValid(false)
setValidationError(null)
}
const handleApiKeyBlur = async () => {
if (apiKey.trim()) {
await validateApiKey(apiKey)
}
}
const handleVoiceChange = useCallback(async (e: any) => {
const voice = e.target.value
setSelectedVoice(voice)
// Save to state via gRPC
await UiServiceClient.updateDiscussVoiceSettings(DiscussVoiceSettingsRequest.create({ selectedVoice: voice }))
}, [])
const handleSpeedChange = useCallback(async (e: any) => {
const speed = parseFloat(e.target.value)
setSpeechSpeed(speed)
// Save to state via gRPC
await UiServiceClient.updateDiscussVoiceSettings(DiscussVoiceSettingsRequest.create({ speechSpeed: speed }))
}, [])
const handleAutoSpeakChange = useCallback(async (e: any) => {
const checked = e.target.checked === true
setAutoSpeak(checked)
// Save to state via gRPC
await UiServiceClient.updateDiscussVoiceSettings(DiscussVoiceSettingsRequest.create({ autoSpeak: checked }))
}, [])
const handleAutoListenChange = useCallback(async (e: any) => {
const checked = e.target.checked === true
setAutoListen(checked)
// Save to state via gRPC
await UiServiceClient.updateDiscussVoiceSettings(DiscussVoiceSettingsRequest.create({ autoListen: checked }))
}, [])
const handleTestVoice = async () => {
if (!selectedVoice) {
console.error("No voice selected")
alert("Please select a voice first")
return
}
try {
console.log("Testing voice:", selectedVoice, "at speed:", speechSpeed)
// Synthesize a test phrase
const testText = "Hello! This is a test of the text-to-speech voice."
console.log("Requesting TTS for:", testText)
const response = await TtsServiceClient.SynthesizeSpeech({
text: testText,
voiceId: selectedVoice,
speed: speechSpeed,
})
console.log("TTS Response:", {
audioDataLength: response.audioData.length,
contentType: response.contentType,
error: response.error,
})
if (response.error) {
console.error("TTS Error:", response.error)
alert("Failed to test voice: " + response.error)
return
}
if (!response.audioData || response.audioData.length === 0) {
console.error("Empty audio data received")
alert("Failed to test voice: No audio data received")
return
}
// Play the audio using a DOM audio element (works better in VSCode webview)
console.log("Converting audio data...")
const audioArray = new Uint8Array(response.audioData)
console.log("Audio array length:", audioArray.length)
const blob = new Blob([audioArray], { type: response.contentType || "audio/mpeg" })
console.log("Blob created:", blob.size, "bytes, type:", blob.type)
const url = URL.createObjectURL(blob)
console.log("Object URL created:", url)
// Create audio element in DOM (VSCode webview friendly)
const audioElement = document.createElement("audio")
audioElement.src = url
audioElement.preload = "auto"
// Add to DOM temporarily
audioElement.style.display = "none"
document.body.appendChild(audioElement)
audioElement.onerror = (e) => {
console.error("Audio playback error:", e, audioElement.error)
alert("Failed to play audio: " + (audioElement.error?.message || "Unknown error"))
URL.revokeObjectURL(url)
document.body.removeChild(audioElement)
}
audioElement.onended = () => {
console.log("Audio playback completed")
URL.revokeObjectURL(url)
document.body.removeChild(audioElement)
}
audioElement.onloadeddata = () => {
console.log("Audio data loaded, duration:", audioElement.duration)
}
audioElement.oncanplay = () => {
console.log("Audio ready to play")
}
console.log("Starting audio playback...")
try {
await audioElement.play()
console.log("Audio.play() called successfully")
} catch (playError) {
console.error("Play error:", playError)
alert(
"Audio playback blocked: " +
(playError as Error).message +
"\n\nTip: Try clicking in the window first, or check your browser/VSCode audio settings.",
)
URL.revokeObjectURL(url)
document.body.removeChild(audioElement)
}
} catch (error) {
console.error("Test voice error:", error)
alert("Failed to test voice: " + (error as Error).message)
}
}
return (
<div>
{renderSectionHeader("voice")}
<Section>
<div className="mb-[5px]">
<h4 className="text-sm font-semibold mb-2">Text-to-Speech Configuration</h4>
<p className="text-sm text-description mb-4">
Configure voice output for Discuss Mode. Cline will speak responses during interactive planning
conversations.
</p>
</div>
{/* API Key Input */}
<div className="mb-[5px]">
<label className="text-sm font-medium mb-2 block">ElevenLabs API Key</label>
<div className="flex gap-2 items-start">
<div className="flex-1">
<VSCodeTextField
className="w-full"
onBlur={handleApiKeyBlur}
onChange={handleApiKeyChange}
placeholder="Enter your ElevenLabs API key"
type={showApiKey ? "text" : "password"}
value={apiKey}
/>
</div>
<VSCodeButton
appearance="icon"
onClick={() => setShowApiKey(!showApiKey)}
title={showApiKey ? "Hide API key" : "Show API key"}>
{showApiKey ? <EyeOff className="w-4 h-4" /> : <Eye className="w-4 h-4" />}
</VSCodeButton>
<VSCodeButton disabled={isValidating || !apiKey.trim()} onClick={() => validateApiKey(apiKey)}>
{isValidating ? "Validating..." : "Validate"}
</VSCodeButton>
</div>
{validationError && <p className="text-sm mt-2 text-red-500">{validationError}</p>}
{isValid && <p className="text-sm mt-2 text-green-500"> API key is valid</p>}
<p className="text-sm mt-2 text-description">
Get your free API key from{" "}
<a className="text-link underline" href="https://elevenlabs.io" rel="noopener noreferrer" target="_blank">
elevenlabs.io
</a>
</p>
</div>
{/* Voice Selection */}
{isValid && (
<div className="mb-[5px]">
<label className="text-sm font-medium mb-2 block">Voice Selection</label>
<div className="flex gap-2 items-start">
<select
className="flex-1 px-2 py-1 bg-input text-foreground border border-input-border rounded"
disabled={isLoadingVoices}
onChange={handleVoiceChange}
value={selectedVoice}>
<option value="">Select a voice...</option>
{voices.map((voice) => (
<option key={voice.id} value={voice.id}>
{voice.name}
</option>
))}
</select>
<VSCodeButton disabled={!selectedVoice || isLoadingVoices} onClick={handleTestVoice}>
<Volume2 className="w-4 h-4 mr-1" />
Test
</VSCodeButton>
</div>
{isLoadingVoices && <p className="text-sm mt-2 text-description">Loading voices...</p>}
{voicesError && <p className="text-sm mt-2 text-red-500">{voicesError}</p>}
{selectedVoice && voices.length > 0 && (
<p className="text-sm mt-2 text-description">
{voices.find((v) => v.id === selectedVoice)?.description || ""}
</p>
)}
</div>
)}
{/* Speech Speed */}
{isValid && selectedVoice && (
<div className="mb-[5px]">
<label className="text-sm font-medium mb-2 block">Speech Speed: {speechSpeed.toFixed(1)}x</label>
<input
className="w-full"
max="1.2"
min="0.7"
onChange={handleSpeedChange}
step="0.1"
type="range"
value={speechSpeed}
/>
<p className="text-sm mt-2 text-description">
Adjust how fast Cline speaks (0.7x = slower, 1.2x = faster)
</p>
</div>
)}
{/* Auto-Speak Toggle */}
{isValid && selectedVoice && (
<div className="mb-[5px]">
<VSCodeCheckbox checked={autoSpeak} onChange={handleAutoSpeakChange}>
Automatically speak responses
</VSCodeCheckbox>
<p className="text-sm mt-2 text-description">
When enabled, Cline will automatically speak text responses in Plan Mode with Discuss Mode active
</p>
</div>
)}
{/* Auto-Listen Toggle */}
{isValid && selectedVoice && autoSpeak && (
<div className="mb-[5px]">
<VSCodeCheckbox checked={autoListen} onChange={handleAutoListenChange}>
Auto-continue conversation
</VSCodeCheckbox>
<p className="text-sm mt-2 text-description">
When enabled, voice input will automatically start after Cline finishes speaking, creating a natural
conversation flow
</p>
</div>
)}
{/* Voice Input Information */}
{isValid && selectedVoice && (
<div className="mb-[5px] mt-4 p-3 bg-[rgba(var(--vscode-textBlockQuote-background-rgb),0.5)] border-l-2 border-[var(--vscode-textBlockQuote-border)] rounded">
<h5 className="text-sm font-semibold mb-2">Voice Input (Speech-to-Text)</h5>
<p className="text-sm text-description mb-2">
Voice input for Discuss Mode uses your ElevenLabs API key for transcription (Scribe v1 model).
</p>
<p className="text-sm text-description mb-2">
When both Discuss Mode and Dictation are enabled, you'll see a microphone button in the chat input.
Click it to record your voice, and your audio will be transcribed using ElevenLabs' Speech-to-Text
API.
</p>
<p className="text-sm text-description mb-2">
<strong>Features:</strong> Supports 99 languages, high accuracy transcription, speaker diarization,
and word-level timestamps.
</p>
<p className="text-sm text-description">
<strong>Note:</strong> Voice input requires macOS (for audio recording) and your ElevenLabs API key
(already configured above).
</p>
</div>
)}
</Section>
</div>
)
}
export default VoiceSettingsSection
@@ -225,6 +225,15 @@ export const ExtensionStateContextProvider: React.FC<{
lastDismissedCliBannerVersion: 0,
subagentsEnabled: false,
// Discuss Mode defaults
discussModeEnabled: false,
discussModeSettings: {
selectedVoice: undefined,
speechSpeed: 1.0,
autoSpeak: false,
autoListen: false,
},
// NEW: Add workspace information with defaults
workspaceRoots: [],
primaryRootIndex: 0,
-4
View File
@@ -8,10 +8,6 @@
"DOM",
"DOM.Iterable"
],
"types": [
"vitest/globals",
"@testing-library/jest-dom"
],
"module": "ESNext",
"skipLibCheck": true,