mirror of
https://github.com/cline/cline.git
synced 2026-09-01 23:19:18 +08:00
Compare commits
5 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 1136f25bdf | |||
| 1630f37b3f | |||
| 6ed664942a | |||
| 6dffa3f6ed | |||
| 8b1ed0113c |
@@ -0,0 +1,556 @@
|
||||
# Discuss Mode Implementation Plan
|
||||
|
||||
> **Voice-enabled interactive planning conversations for Cline**
|
||||
> Last Updated: November 8, 2025
|
||||
> Status: 🚧 In Progress
|
||||
|
||||
## 📋 Quick Status
|
||||
|
||||
- **Start Date:** November 8, 2025
|
||||
- **Target Completion:** TBD
|
||||
- **Current Phase:** Backend Infrastructure Complete (Phases 1-3)
|
||||
- **Overall Progress:** 35% (15/43 tasks complete)
|
||||
|
||||
## 🎯 Project Goals
|
||||
|
||||
Enable natural voice conversations with Cline during Plan Mode to create a collaborative planning experience where users can discuss requirements before implementation begins.
|
||||
|
||||
### Core Features
|
||||
- ✅ Voice input (already exists)
|
||||
- ⬜ Voice output via ElevenLabs TTS
|
||||
- ⬜ Auto-continue conversation flow
|
||||
- ⬜ Interactive question-asking behavior
|
||||
- ⬜ Plan completion detection
|
||||
- ⬜ Smooth transition to Act Mode
|
||||
|
||||
### Scope
|
||||
- **In Scope:** Voice conversations in Plan Mode only
|
||||
- **Out of Scope:** Voice during Act Mode (tool execution)
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ Architecture Overview
|
||||
|
||||
### System Components
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Discuss Mode Stack │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ Frontend (React/TypeScript) │
|
||||
│ ├── DiscussModeToggle.tsx (UI control) │
|
||||
│ ├── AudioPlayer.tsx (TTS playback) │
|
||||
│ ├── VoiceConversationControls.tsx (speaking indicators) │
|
||||
│ └── VoiceSettingsSection.tsx (settings UI) │
|
||||
│ │
|
||||
│ Backend (Node.js/TypeScript) │
|
||||
│ ├── TextToSpeechService.ts (TTS orchestration) │
|
||||
│ ├── ElevenLabsProvider.ts (ElevenLabs API) │
|
||||
│ ├── Task.say() modifications (TTS trigger) │
|
||||
│ └── System Prompt additions (discuss behavior) │
|
||||
│ │
|
||||
│ Infrastructure │
|
||||
│ ├── proto/tts.proto (gRPC definitions) │
|
||||
│ ├── Controller handlers (gRPC endpoints) │
|
||||
│ └── State management (discuss mode state) │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Data Flow
|
||||
|
||||
```
|
||||
User Speech → VoiceRecorder → Transcription → Cline (Plan Mode)
|
||||
↓
|
||||
[Processes with
|
||||
"Discuss Mode" prompt]
|
||||
↓
|
||||
Response Text → TextToSpeechService
|
||||
↓
|
||||
Audio Buffer → AudioPlayer → Speaker
|
||||
↓
|
||||
[On playback complete]
|
||||
↓
|
||||
Auto-start VoiceRecorder (if enabled)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📝 Implementation Checklist
|
||||
|
||||
### Phase 1: System Prompt & Behavior (Est: 2-3 hours)
|
||||
- [ ] 1.1 Read existing Plan Mode system prompt
|
||||
- [ ] 1.2 Design "Discuss Mode" prompt additions
|
||||
- [ ] 1.3 Add conversational guidelines
|
||||
- [ ] 1.4 Add proactive questioning instructions
|
||||
- [ ] 1.5 Add plan completion signal instructions
|
||||
- [ ] 1.6 Test prompt changes with existing Plan Mode
|
||||
|
||||
**Files to Modify:**
|
||||
- `src/core/prompts/system-prompt.ts`
|
||||
- `src/core/prompts/system-prompt-legacy/` (if needed)
|
||||
|
||||
### Phase 2: TTS Service Backend (Est: 4-6 hours)
|
||||
- [ ] 2.1 Create `src/services/tts/` directory structure
|
||||
- [ ] 2.2 Implement `BaseTTSProvider` interface
|
||||
- [ ] 2.3 Implement `ElevenLabsProvider.ts`
|
||||
- [ ] API client setup
|
||||
- [ ] Voice list endpoint
|
||||
- [ ] Text-to-speech synthesis
|
||||
- [ ] Error handling
|
||||
- [ ] 2.4 Implement `TextToSpeechService.ts`
|
||||
- [ ] Provider factory pattern
|
||||
- [ ] Audio buffer management
|
||||
- [ ] Streaming support (optional)
|
||||
- [ ] 2.5 Add TTS configuration to `src/shared/api.ts`
|
||||
- [ ] 2.6 Add ElevenLabs API key to secrets storage
|
||||
- [ ] 2.7 Unit tests for TTS service
|
||||
|
||||
**New Files:**
|
||||
```
|
||||
src/services/tts/
|
||||
├── TextToSpeechService.ts
|
||||
├── providers/
|
||||
│ ├── BaseTTSProvider.ts
|
||||
│ ├── ElevenLabsProvider.ts
|
||||
│ └── OpenAITTSProvider.ts (future)
|
||||
└── __tests__/
|
||||
└── TextToSpeechService.test.ts
|
||||
```
|
||||
|
||||
### Phase 3: Protobuf & gRPC (Est: 2-3 hours)
|
||||
- [ ] 3.1 Create `proto/tts.proto` definition
|
||||
- [ ] 3.2 Define `TtsService` with methods:
|
||||
- [ ] `synthesizeSpeech()`
|
||||
- [ ] `getAvailableVoices()`
|
||||
- [ ] `updateVoiceSettings()`
|
||||
- [ ] 3.3 Run proto compilation: `npm run protos`
|
||||
- [ ] 3.4 Implement gRPC handlers in `src/core/controller/tts/`
|
||||
- [ ] `synthesizeSpeech.ts`
|
||||
- [ ] `getAvailableVoices.ts`
|
||||
- [ ] 3.5 Generate client in `webview-ui/src/services/grpc-client.ts`
|
||||
|
||||
**New Files:**
|
||||
```
|
||||
proto/tts.proto
|
||||
src/core/controller/tts/
|
||||
├── synthesizeSpeech.ts
|
||||
└── getAvailableVoices.ts
|
||||
```
|
||||
|
||||
### Phase 4: Audio Player Component (Est: 3-4 hours)
|
||||
- [ ] 4.1 Create `AudioPlayer.tsx` component
|
||||
- [ ] Audio element management
|
||||
- [ ] Playback controls
|
||||
- [ ] Loading states
|
||||
- [ ] Error handling
|
||||
- [ ] 4.2 Create audio queue system
|
||||
- [ ] 4.3 Add speaking animation/indicator
|
||||
- [ ] 4.4 Integrate into chat message component
|
||||
- [ ] 4.5 Add auto-play functionality
|
||||
- [ ] 4.6 Add callback for playback completion
|
||||
|
||||
**New Files:**
|
||||
```
|
||||
webview-ui/src/components/chat/
|
||||
├── AudioPlayer.tsx
|
||||
├── VoiceConversationControls.tsx
|
||||
└── SpeakingIndicator.tsx
|
||||
```
|
||||
|
||||
### Phase 5: Discuss Mode Integration (Est: 4-5 hours)
|
||||
- [ ] 5.1 Add `discussModeEnabled` to global state
|
||||
- [ ] 5.2 Add `voiceModeSettings` to state
|
||||
- [ ] 5.3 Modify `Task.say()` in `src/core/task/index.ts`:
|
||||
- [ ] Check if discuss mode is enabled
|
||||
- [ ] Check if current mode is "plan"
|
||||
- [ ] Trigger TTS for assistant text
|
||||
- [ ] Queue audio for playback
|
||||
- [ ] 5.4 Implement auto-continue logic:
|
||||
- [ ] Detect audio playback completion
|
||||
- [ ] Auto-start voice recorder
|
||||
- [ ] Handle errors gracefully
|
||||
- [ ] 5.5 Add mode switching guard (auto-disable on Act Mode)
|
||||
|
||||
**Files to Modify:**
|
||||
```
|
||||
src/core/task/index.ts
|
||||
src/core/controller/index.ts
|
||||
src/core/storage/StateManager.ts
|
||||
```
|
||||
|
||||
### Phase 6: UI Components (Est: 3-4 hours)
|
||||
- [ ] 6.1 Create `DiscussModeToggle.tsx`
|
||||
- [ ] Toggle button with icon
|
||||
- [ ] Mode indicator badge
|
||||
- [ ] Tooltip with description
|
||||
- [ ] Disable in Act Mode
|
||||
- [ ] 6.2 Add discuss mode controls to chat header
|
||||
- [ ] 6.3 Create plan completion UI:
|
||||
- [ ] "Plan Ready" indicator
|
||||
- [ ] "Switch to Act Mode" button
|
||||
- [ ] "Continue Discussing" option
|
||||
- [ ] 6.4 Add voice conversation status indicators:
|
||||
- [ ] "🎤 Listening..." when recording
|
||||
- [ ] "🗣️ Speaking..." when playing audio
|
||||
- [ ] "💭 Thinking..." when processing
|
||||
|
||||
**New Files:**
|
||||
```
|
||||
webview-ui/src/components/discuss-mode/
|
||||
├── DiscussModeToggle.tsx
|
||||
├── PlanCompletionCard.tsx
|
||||
└── ConversationStatusIndicator.tsx
|
||||
```
|
||||
|
||||
### Phase 7: Settings Panel (Est: 2-3 hours)
|
||||
- [ ] 7.1 Extend `VoiceSettingsSection.tsx`
|
||||
- [ ] 7.2 Add TTS provider selection (ElevenLabs)
|
||||
- [ ] 7.3 Add ElevenLabs API key input
|
||||
- [ ] 7.4 Add voice selection dropdown:
|
||||
- [ ] Fetch voices from ElevenLabs
|
||||
- [ ] Voice preview button
|
||||
- [ ] Voice descriptions
|
||||
- [ ] 7.5 Add speech rate slider
|
||||
- [ ] 7.6 Add auto-speak toggle
|
||||
- [ ] 7.7 Add auto-listen toggle
|
||||
- [ ] 7.8 Save settings to state
|
||||
|
||||
**Files to Modify:**
|
||||
```
|
||||
webview-ui/src/components/settings/sections/VoiceSettingsSection.tsx
|
||||
```
|
||||
|
||||
### Phase 8: Testing & Polish (Est: 3-4 hours)
|
||||
- [ ] 8.1 End-to-end testing:
|
||||
- [ ] Voice input → TTS output flow
|
||||
- [ ] Auto-continue conversation
|
||||
- [ ] Plan completion detection
|
||||
- [ ] Mode switching behavior
|
||||
- [ ] 8.2 Error handling:
|
||||
- [ ] API key missing
|
||||
- [ ] Network failures
|
||||
- [ ] Audio playback errors
|
||||
- [ ] Rate limiting (ElevenLabs)
|
||||
- [ ] 8.3 Edge cases:
|
||||
- [ ] Empty responses
|
||||
- [ ] Very long responses (chunking)
|
||||
- [ ] Interrupted speech
|
||||
- [ ] Rapid mode switching
|
||||
- [ ] 8.4 Performance optimization:
|
||||
- [ ] Audio caching
|
||||
- [ ] Queue management
|
||||
- [ ] Memory cleanup
|
||||
- [ ] 8.5 UX polish:
|
||||
- [ ] Smooth animations
|
||||
- [ ] Clear status indicators
|
||||
- [ ] Helpful error messages
|
||||
- [ ] Onboarding tooltip
|
||||
|
||||
---
|
||||
|
||||
## 📁 File Structure
|
||||
|
||||
```
|
||||
cline/
|
||||
├── proto/
|
||||
│ └── tts.proto [NEW]
|
||||
│
|
||||
├── src/
|
||||
│ ├── core/
|
||||
│ │ ├── controller/
|
||||
│ │ │ └── tts/ [NEW]
|
||||
│ │ │ ├── synthesizeSpeech.ts
|
||||
│ │ │ └── getAvailableVoices.ts
|
||||
│ │ ├── prompts/
|
||||
│ │ │ └── system-prompt.ts [MODIFY]
|
||||
│ │ └── task/
|
||||
│ │ └── index.ts [MODIFY]
|
||||
│ │
|
||||
│ ├── services/
|
||||
│ │ └── tts/ [NEW]
|
||||
│ │ ├── TextToSpeechService.ts
|
||||
│ │ └── providers/
|
||||
│ │ ├── BaseTTSProvider.ts
|
||||
│ │ └── ElevenLabsProvider.ts
|
||||
│ │
|
||||
│ └── shared/
|
||||
│ └── api.ts [MODIFY]
|
||||
│
|
||||
└── webview-ui/
|
||||
└── src/
|
||||
├── components/
|
||||
│ ├── chat/
|
||||
│ │ ├── AudioPlayer.tsx [NEW]
|
||||
│ │ └── VoiceConversationControls.tsx [NEW]
|
||||
│ ├── discuss-mode/ [NEW]
|
||||
│ │ ├── DiscussModeToggle.tsx
|
||||
│ │ └── PlanCompletionCard.tsx
|
||||
│ └── settings/
|
||||
│ └── VoiceSettingsSection.tsx [MODIFY]
|
||||
│
|
||||
└── services/
|
||||
└── grpc-client.ts [MODIFY]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Technical Details
|
||||
|
||||
### ElevenLabs Integration
|
||||
|
||||
**API Endpoints:**
|
||||
- Text-to-Speech: `POST https://api.elevenlabs.io/v1/text-to-speech/{voice_id}`
|
||||
- Get Voices: `GET https://api.elevenlabs.io/v1/voices`
|
||||
|
||||
**Authentication:**
|
||||
```typescript
|
||||
headers: {
|
||||
'xi-api-key': API_KEY,
|
||||
'Content-Type': 'application/json'
|
||||
}
|
||||
```
|
||||
|
||||
**Request Format:**
|
||||
```typescript
|
||||
{
|
||||
text: string,
|
||||
model_id: "eleven_multilingual_v2",
|
||||
voice_settings: {
|
||||
stability: 0.5,
|
||||
similarity_boost: 0.75
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Response:** Audio file (MP3 format)
|
||||
|
||||
### State Management
|
||||
|
||||
**New Global State Keys:**
|
||||
```typescript
|
||||
interface GlobalState {
|
||||
discussModeEnabled: boolean
|
||||
voiceModeSettings: {
|
||||
ttsProvider: "elevenlabs" | "openai"
|
||||
elevenLabsApiKey?: string
|
||||
selectedVoice: string
|
||||
speechRate: number
|
||||
autoSpeak: boolean
|
||||
autoListen: boolean
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Protobuf Schema
|
||||
|
||||
```protobuf
|
||||
syntax = "proto3";
|
||||
|
||||
package cline.tts;
|
||||
|
||||
service TtsService {
|
||||
rpc synthesizeSpeech(SynthesizeRequest) returns (SynthesizeResponse);
|
||||
rpc getAvailableVoices(EmptyRequest) returns (VoicesResponse);
|
||||
}
|
||||
|
||||
message SynthesizeRequest {
|
||||
string text = 1;
|
||||
string voice_id = 2;
|
||||
optional float speech_rate = 3;
|
||||
}
|
||||
|
||||
message SynthesizeResponse {
|
||||
bytes audio_data = 1;
|
||||
optional string error = 2;
|
||||
}
|
||||
|
||||
message Voice {
|
||||
string id = 1;
|
||||
string name = 2;
|
||||
string description = 3;
|
||||
}
|
||||
|
||||
message VoicesResponse {
|
||||
repeated Voice voices = 1;
|
||||
optional string error = 2;
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
- [ ] TTS Service provider selection
|
||||
- [ ] Audio buffer handling
|
||||
- [ ] Error handling for API failures
|
||||
- [ ] State management for discuss mode
|
||||
|
||||
### Integration Tests
|
||||
- [ ] gRPC endpoint communication
|
||||
- [ ] Full voice input → TTS output flow
|
||||
- [ ] Mode switching behavior
|
||||
- [ ] Settings persistence
|
||||
|
||||
### Manual Testing Scenarios
|
||||
1. **Happy Path:**
|
||||
- Enable Discuss Mode
|
||||
- Ask initial question via voice
|
||||
- Cline asks clarifying questions
|
||||
- Iterative discussion
|
||||
- Plan completion and approval
|
||||
- Switch to Act Mode
|
||||
|
||||
2. **Error Cases:**
|
||||
- Missing API key
|
||||
- Network failure during TTS
|
||||
- Invalid voice selection
|
||||
- Audio playback failure
|
||||
|
||||
3. **Edge Cases:**
|
||||
- Mode switching during playback
|
||||
- Rapid successive voice inputs
|
||||
- Very long responses (>1000 chars)
|
||||
- Empty or nonsense responses
|
||||
|
||||
---
|
||||
|
||||
## 🐛 Known Issues & Future Improvements
|
||||
|
||||
### Known Limitations
|
||||
- ElevenLabs rate limits (adjust queue as needed)
|
||||
- No offline mode (requires internet for TTS)
|
||||
- Audio latency depends on network speed
|
||||
|
||||
### Future Enhancements
|
||||
- [ ] Multiple TTS provider support (OpenAI, Azure)
|
||||
- [ ] Voice cloning integration
|
||||
- [ ] Conversation history playback
|
||||
- [ ] Export voice conversations
|
||||
- [ ] Voice command shortcuts
|
||||
- [ ] Ambient mode (minimal UI, voice-first)
|
||||
- [ ] Multi-language support
|
||||
- [ ] Emotion/tone control for TTS
|
||||
|
||||
---
|
||||
|
||||
## 📊 Progress Tracking
|
||||
|
||||
### Time Estimates vs Actual
|
||||
| Phase | Estimated | Actual | Status |
|
||||
|-------|-----------|--------|--------|
|
||||
| Phase 1: System Prompt | 2-3h | ~1h | ✅ Complete |
|
||||
| Phase 2: TTS Backend | 4-6h | ~2h | ✅ Complete |
|
||||
| Phase 3: Protobuf/gRPC | 2-3h | ~1h | ✅ Complete |
|
||||
| Phase 4: Audio Player | 3-4h | - | ⬜ Not Started |
|
||||
| Phase 5: Integration | 4-5h | - | ⬜ Not Started |
|
||||
| Phase 6: UI Components | 3-4h | - | ⬜ Not Started |
|
||||
| Phase 7: Settings | 2-3h | - | ⬜ Not Started |
|
||||
| Phase 8: Testing | 3-4h | - | ⬜ Not Started |
|
||||
| **Total** | **23-32h** | **~4h** | 🟡 35% Complete |
|
||||
|
||||
### Sprint Log
|
||||
_Add daily progress notes here as development proceeds_
|
||||
|
||||
**November 8, 2025 - Session 1:**
|
||||
- ✅ Completed: Initial planning and architecture design
|
||||
- ✅ Completed: Created comprehensive implementation plan document
|
||||
- ✅ Completed: **Phase 1 - System Prompt & Behavior**
|
||||
- Modified `src/core/prompts/system-prompt/components/act_vs_plan_mode.ts`
|
||||
- Added Discuss Mode conversational guidelines
|
||||
- Implemented proactive questioning behavior
|
||||
- Added plan completion signal instructions
|
||||
- ✅ Completed: **Phase 2 - TTS Service Backend**
|
||||
- Created complete TTS service architecture in `src/services/tts/`
|
||||
- Implemented `BaseTTSProvider.ts` abstract base class
|
||||
- Implemented `ElevenLabsProvider.ts` with full API integration
|
||||
- Implemented `TextToSpeechService.ts` orchestration layer
|
||||
- Added voice synthesis, voice selection, and validation
|
||||
- ✅ Completed: **Phase 3 - Protobuf & gRPC**
|
||||
- Created `proto/tts.proto` with complete service definitions
|
||||
- Successfully compiled protobuf definitions
|
||||
- Implemented gRPC handlers in `src/core/controller/tts/`:
|
||||
- `synthesizeSpeech.ts` for text-to-speech synthesis
|
||||
- `getAvailableVoices.ts` for voice listing
|
||||
- Integrated TTS service into Controller class with getter method
|
||||
- 📝 Status: Backend infrastructure complete (35% overall progress)
|
||||
- 🎯 Next: Begin Phase 4 (Audio Player Component implementation)
|
||||
|
||||
---
|
||||
|
||||
## 📚 References
|
||||
|
||||
- [ElevenLabs API Docs](https://elevenlabs.io/docs)
|
||||
- [Cline Architecture Guide](.clinerules/cline-overview.md)
|
||||
- [Protobuf Development Guide](.clinerules/protobuf-development.md)
|
||||
- [Existing Voice Input Implementation](src/services/dictation/)
|
||||
|
||||
---
|
||||
|
||||
## 👥 Team Notes
|
||||
|
||||
_Use this section for brainstorming, decisions, and Q&A during development_
|
||||
|
||||
### Design Decisions
|
||||
- **Why ElevenLabs?** High-quality, natural-sounding voices with proven API reliability
|
||||
- **Why Plan Mode only?** Clean separation of concerns - discussion vs execution. This creates a natural workflow where users discuss in Plan Mode then execute silently in Act Mode
|
||||
- **Why auto-continue?** Enables natural conversation flow without manual button clicking between exchanges
|
||||
|
||||
### Open Questions
|
||||
- [ ] Should we support offline TTS? (browser TTS API as fallback?)
|
||||
- [ ] Max response length before chunking audio?
|
||||
- [ ] Voice selection defaults for different languages?
|
||||
- [ ] Should we cache frequently used phrases/responses?
|
||||
|
||||
### Example Discuss Mode Session
|
||||
|
||||
```
|
||||
User clicks "Discuss Mode" button in Plan Mode
|
||||
|
||||
👤 User: [Speaks] "I need to add authentication to my app"
|
||||
|
||||
🤖 Cline: [Speaks] "I'd be happy to help with that! Let me ask a few questions
|
||||
so I can create the right solution for you. First, what kind of
|
||||
authentication are you thinking - email/password, OAuth with
|
||||
providers like Google or GitHub, or something else?"
|
||||
|
||||
👤 User: [Speaks] "Email and password for now, but I might add OAuth later"
|
||||
|
||||
🤖 Cline: [Speaks] "Perfect. And do you already have a backend API, or should
|
||||
I create one? Also, what framework is your frontend using?"
|
||||
|
||||
👤 User: [Speaks] "I have a Node.js backend with Express, and React on the frontend"
|
||||
|
||||
🤖 Cline: [Speaks] "Great! A couple more things - do you need features like
|
||||
password reset, email verification, or 'remember me' functionality?"
|
||||
|
||||
👤 User: [Speaks] "Yes, all of those would be helpful"
|
||||
|
||||
🤖 Cline: [Speaks] "Perfect! I have a clear plan now. Here's what I'll do:
|
||||
|
||||
I'll create:
|
||||
1. Backend authentication routes with JWT tokens
|
||||
2. Password hashing with bcrypt
|
||||
3. Email service for verification and password reset
|
||||
4. React components for login, signup, and password reset pages
|
||||
5. Protected route wrapper for authenticated pages
|
||||
|
||||
This will integrate with your existing Express backend and React
|
||||
frontend. The authentication will be secure and production-ready.
|
||||
|
||||
Does this sound like what you need? If so, I'm ready to switch to
|
||||
Act Mode and start building!"
|
||||
|
||||
👤 User: [Speaks] "Yes, that sounds perfect!"
|
||||
|
||||
[UI shows: ✅ Plan Complete - Switch to Act Mode?]
|
||||
[User clicks button → Switches to Act Mode]
|
||||
[Discuss Mode auto-disables]
|
||||
[Cline begins implementation silently]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
*This document will be updated throughout implementation. Last updated: November 8, 2025*
|
||||
Generated
+4
-22
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "claude-dev",
|
||||
"version": "3.36.0",
|
||||
"version": "3.36.1",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "claude-dev",
|
||||
"version": "3.36.0",
|
||||
"version": "3.36.1",
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"@anthropic-ai/sdk": "^0.37.0",
|
||||
@@ -151,6 +151,8 @@
|
||||
},
|
||||
"node_modules/@anthropic-ai/sdk": {
|
||||
"version": "0.37.0",
|
||||
"resolved": "https://registry.npmjs.org/@anthropic-ai/sdk/-/sdk-0.37.0.tgz",
|
||||
"integrity": "sha512-tHjX2YbkUBwEgg0JZU3EFSSAQPoK4qQR/NFYa8Vtzd5UAyXzZksCw2In69Rml4R/TyHPBfRYaLK35XiOe33pjw==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@types/node": "^18.11.18",
|
||||
@@ -177,26 +179,6 @@
|
||||
"google-auth-library": "^9.4.2"
|
||||
}
|
||||
},
|
||||
"node_modules/@anthropic-ai/vertex-sdk/node_modules/@anthropic-ai/sdk": {
|
||||
"version": "0.38.0",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@types/node": "^18.11.18",
|
||||
"@types/node-fetch": "^2.6.4",
|
||||
"abort-controller": "^3.0.0",
|
||||
"agentkeepalive": "^4.2.1",
|
||||
"form-data-encoder": "1.7.2",
|
||||
"formdata-node": "^4.3.2",
|
||||
"node-fetch": "^2.6.7"
|
||||
}
|
||||
},
|
||||
"node_modules/@anthropic-ai/vertex-sdk/node_modules/@types/node": {
|
||||
"version": "18.19.76",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"undici-types": "~5.26.4"
|
||||
}
|
||||
},
|
||||
"node_modules/@aws-crypto/crc32": {
|
||||
"version": "5.2.0",
|
||||
"resolved": "https://registry.npmjs.org/@aws-crypto/crc32/-/crc32-5.2.0.tgz",
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
syntax = "proto3";
|
||||
|
||||
package cline;
|
||||
|
||||
option go_package = "github.com/cline/grpc-go/cline";
|
||||
option java_multiple_files = true;
|
||||
option java_package = "bot.cline.proto";
|
||||
|
||||
// TTS Service - Handles text-to-speech synthesis
|
||||
service TtsService {
|
||||
// Synthesize speech from text
|
||||
rpc SynthesizeSpeech(SynthesizeRequest) returns (SynthesizeResponse);
|
||||
|
||||
// Get list of available voices for current provider
|
||||
rpc GetAvailableVoices(GetVoicesRequest) returns (VoicesResponse);
|
||||
|
||||
// Validate API key for TTS provider
|
||||
rpc ValidateApiKey(ValidateApiKeyRequest) returns (ValidateApiKeyResponse);
|
||||
|
||||
// Check if API key is configured
|
||||
rpc CheckApiKeyConfigured(GetVoicesRequest) returns (ValidateApiKeyResponse);
|
||||
}
|
||||
|
||||
// Request to synthesize speech
|
||||
message SynthesizeRequest {
|
||||
string text = 1;
|
||||
string voice_id = 2;
|
||||
optional float speed = 3;
|
||||
optional float stability = 4;
|
||||
optional float similarity_boost = 5;
|
||||
}
|
||||
|
||||
// Response containing synthesized audio
|
||||
message SynthesizeResponse {
|
||||
bytes audio_data = 1;
|
||||
string content_type = 2;
|
||||
optional string error = 3;
|
||||
}
|
||||
|
||||
// Request to get available voices
|
||||
message GetVoicesRequest {
|
||||
// Empty for now, could add filtering options later
|
||||
}
|
||||
|
||||
// Information about a voice
|
||||
message Voice {
|
||||
string id = 1;
|
||||
string name = 2;
|
||||
optional string description = 3;
|
||||
optional string preview_url = 4;
|
||||
}
|
||||
|
||||
// Response containing list of voices
|
||||
message VoicesResponse {
|
||||
repeated Voice voices = 1;
|
||||
optional string error = 2;
|
||||
}
|
||||
|
||||
// Request to validate API key
|
||||
message ValidateApiKeyRequest {
|
||||
string api_key = 1; // The API key to validate
|
||||
}
|
||||
|
||||
// Response from API key validation
|
||||
message ValidateApiKeyResponse {
|
||||
bool is_valid = 1;
|
||||
optional string error = 2;
|
||||
}
|
||||
@@ -264,4 +264,18 @@ service UiService {
|
||||
|
||||
// Opens the Cline walkthrough
|
||||
rpc openWalkthrough(EmptyRequest) returns (Empty);
|
||||
|
||||
// Toggle discuss mode on/off
|
||||
rpc setDiscussModeEnabled(BooleanRequest) returns (BooleanRequest);
|
||||
|
||||
// Update discuss mode voice settings
|
||||
rpc updateDiscussVoiceSettings(DiscussVoiceSettingsRequest) returns (Empty);
|
||||
}
|
||||
|
||||
// Request to update discuss mode voice settings
|
||||
message DiscussVoiceSettingsRequest {
|
||||
optional string selected_voice = 1;
|
||||
optional double speech_speed = 2;
|
||||
optional bool auto_speak = 3;
|
||||
optional bool auto_listen = 4;
|
||||
}
|
||||
|
||||
@@ -119,7 +119,8 @@ async function generateVscodeProtobusServers(protobusServices) {
|
||||
imports.push(`// ${domain} Service`)
|
||||
servers.push(`const ${serviceName}Handlers: serviceTypes.${serviceName}Handlers = {`)
|
||||
for (const [rpcName, _rpc] of Object.entries(def.service)) {
|
||||
imports.push(`import { ${rpcName} } from "@core/controller/${dir}/${rpcName}"`)
|
||||
const filePath = toFilePath(rpcName)
|
||||
imports.push(`import { ${rpcName} } from "@core/controller/${dir}/${filePath}"`)
|
||||
servers.push(` ${rpcName}: ${rpcName},`)
|
||||
}
|
||||
servers.push(`} \n`)
|
||||
@@ -156,7 +157,8 @@ async function generateStandaloneProtobusServiceSetup(protobusServices) {
|
||||
handlerSetup.push(` // ${domain} Service`)
|
||||
handlerSetup.push(` server.addService(cline.${name}Service, {`)
|
||||
for (const [rpcName, rpc] of Object.entries(def.service)) {
|
||||
imports.push(`import { ${rpcName} } from "@core/controller/${dir}/${rpcName}"`)
|
||||
const filePath = toFilePath(rpcName)
|
||||
imports.push(`import { ${rpcName} } from "@core/controller/${dir}/${filePath}"`)
|
||||
const requestType = "cline." + rpc.requestType.type.name
|
||||
const responseType = "cline." + rpc.responseType.type.name
|
||||
if (rpc.requestStream) {
|
||||
@@ -204,6 +206,10 @@ function getDirName(serviceName) {
|
||||
const domain = getDomainName(serviceName)
|
||||
return domain.charAt(0).toLowerCase() + domain.slice(1)
|
||||
}
|
||||
function toFilePath(rpcName) {
|
||||
// Convert PascalCase to camelCase for file paths
|
||||
return rpcName.charAt(0).toLowerCase() + rpcName.slice(1)
|
||||
}
|
||||
|
||||
// Only run main if this script is executed directly
|
||||
if (import.meta.url === `file://${process.argv[1]}`) {
|
||||
|
||||
@@ -15,6 +15,9 @@ export const cancelRecording = async (controller: Controller): Promise<Recording
|
||||
let errorMessage = ""
|
||||
let isSuccess = true
|
||||
try {
|
||||
// Clear the silence detection callback
|
||||
audioRecordingService.setSilenceDetectedCallback(null)
|
||||
|
||||
const result = await audioRecordingService.cancelRecording()
|
||||
isSuccess = !!result?.success
|
||||
errorMessage = result?.error ?? ""
|
||||
|
||||
@@ -117,6 +117,12 @@ export const startRecording = async (controller: Controller): Promise<RecordingR
|
||||
throw new Error("Please sign in to your Cline Account to use Dictation.")
|
||||
}
|
||||
|
||||
// Set up silence detection callback to auto-stop recording
|
||||
audioRecordingService.setSilenceDetectedCallback(async () => {
|
||||
// Automatically stop recording when silence is detected
|
||||
await audioRecordingService.stopRecording()
|
||||
})
|
||||
|
||||
// Attempt to start recording
|
||||
const result = await audioRecordingService.startRecording()
|
||||
|
||||
@@ -129,6 +135,9 @@ export const startRecording = async (controller: Controller): Promise<RecordingR
|
||||
})
|
||||
}
|
||||
|
||||
// Clear callback on failure
|
||||
audioRecordingService.setSilenceDetectedCallback(null)
|
||||
|
||||
// Check if the error is due to missing dependencies
|
||||
const platform = os.platform() as keyof typeof AUDIO_PROGRAM_CONFIG
|
||||
const config = AUDIO_PROGRAM_CONFIG[platform]
|
||||
|
||||
@@ -14,6 +14,9 @@ export const stopRecording = async (controller: Controller): Promise<RecordedAud
|
||||
const recordingDuration = recordingStatus.durationSeconds * 1000 // Convert to milliseconds
|
||||
|
||||
try {
|
||||
// Clear the silence detection callback
|
||||
audioRecordingService.setSilenceDetectedCallback(null)
|
||||
|
||||
const result = await audioRecordingService.stopRecording()
|
||||
|
||||
telemetryService.captureVoiceRecordingStopped(taskId, recordingDuration, result.success, process.platform)
|
||||
|
||||
@@ -19,8 +19,36 @@ export const transcribeAudio = async (controller: Controller, request: Transcrib
|
||||
telemetryService.captureVoiceTranscriptionStarted(taskId, request.language ?? "en")
|
||||
|
||||
try {
|
||||
// Transcribe the audio
|
||||
const result = await getVoiceTranscriptionService().transcribeAudio(request.audioBase64, request.language ?? "en")
|
||||
// Try ElevenLabs STT first if API key is available
|
||||
const apiKey = await controller.context.secrets.get("elevenLabsApiKey")
|
||||
let result: { text?: string; error?: string } | null = null
|
||||
|
||||
if (apiKey) {
|
||||
try {
|
||||
const { ElevenLabsProvider } = await import("@/services/tts/providers/ElevenLabsProvider")
|
||||
const provider = new ElevenLabsProvider(apiKey)
|
||||
|
||||
// Convert base64 to Buffer
|
||||
const audioBuffer = Buffer.from(request.audioBase64, "base64")
|
||||
result = await provider.transcribeAudio(audioBuffer, request.language ?? "en")
|
||||
|
||||
if (result.text) {
|
||||
console.log("ElevenLabs transcription successful")
|
||||
} else if (result.error) {
|
||||
console.warn("ElevenLabs transcription failed, falling back to Cline service:", result.error)
|
||||
result = null // Fall back to Cline service
|
||||
}
|
||||
} catch (error) {
|
||||
console.warn("ElevenLabs STT error, falling back to Cline service:", error)
|
||||
result = null // Fall back to Cline service
|
||||
}
|
||||
}
|
||||
|
||||
// Fall back to Cline transcription service if ElevenLabs failed or no API key
|
||||
if (!result) {
|
||||
result = await getVoiceTranscriptionService().transcribeAudio(request.audioBase64, request.language ?? "en")
|
||||
}
|
||||
|
||||
const durationMs = Date.now() - startTime
|
||||
|
||||
if (result.error) {
|
||||
|
||||
@@ -8,6 +8,7 @@ import { cleanupLegacyCheckpoints } from "@integrations/checkpoints/CheckpointMi
|
||||
import { downloadTask } from "@integrations/misc/export-markdown"
|
||||
import { ClineAccountService } from "@services/account/ClineAccountService"
|
||||
import { McpHub } from "@services/mcp/McpHub"
|
||||
import { TextToSpeechService } from "@services/tts/TextToSpeechService"
|
||||
import { ApiProvider, ModelInfo } from "@shared/api"
|
||||
import { ChatContent } from "@shared/ChatContent"
|
||||
import { ExtensionState, Platform } from "@shared/ExtensionMessage"
|
||||
@@ -69,6 +70,7 @@ export class Controller {
|
||||
authService: AuthService
|
||||
ocaAuthService: OcaAuthService
|
||||
readonly stateManager: StateManager
|
||||
private ttsService?: TextToSpeechService
|
||||
|
||||
// NEW: Add workspace manager (optional initially)
|
||||
private workspaceManager?: WorkspaceRootManager
|
||||
@@ -107,6 +109,11 @@ export class Controller {
|
||||
return this.workspaceManager
|
||||
}
|
||||
|
||||
// Getter for TTS service
|
||||
getTtsService(): TextToSpeechService | undefined {
|
||||
return this.ttsService
|
||||
}
|
||||
|
||||
/**
|
||||
* Starts the periodic remote config fetching timer
|
||||
* Fetches immediately and then every 30 seconds
|
||||
@@ -867,6 +874,15 @@ export class Controller {
|
||||
const lastDismissedCliBannerVersion = this.stateManager.getGlobalStateKey("lastDismissedCliBannerVersion") || 0
|
||||
const subagentsEnabled = this.stateManager.getGlobalSettingsKey("subagentsEnabled")
|
||||
|
||||
// Discuss Mode state
|
||||
const discussModeEnabled = this.stateManager.getGlobalStateKey("discussModeEnabled")
|
||||
const discussModeSettings = {
|
||||
selectedVoice: this.stateManager.getGlobalSettingsKey("discussModeSelectedVoice"),
|
||||
speechSpeed: this.stateManager.getGlobalSettingsKey("discussModeSpeechSpeed"),
|
||||
autoSpeak: this.stateManager.getGlobalSettingsKey("discussModeAutoSpeak"),
|
||||
autoListen: this.stateManager.getGlobalSettingsKey("discussModeAutoListen"),
|
||||
}
|
||||
|
||||
const localClineRulesToggles = this.stateManager.getWorkspaceStateKey("localClineRulesToggles")
|
||||
const localWindsurfRulesToggles = this.stateManager.getWorkspaceStateKey("localWindsurfRulesToggles")
|
||||
const localCursorRulesToggles = this.stateManager.getWorkspaceStateKey("localCursorRulesToggles")
|
||||
@@ -944,6 +960,9 @@ export class Controller {
|
||||
autoCondenseThreshold,
|
||||
backgroundCommandRunning: this.backgroundCommandRunning,
|
||||
backgroundCommandTaskId: this.backgroundCommandTaskId,
|
||||
// Discuss Mode
|
||||
discussModeEnabled,
|
||||
discussModeSettings,
|
||||
// NEW: Add workspace information
|
||||
workspaceRoots: this.workspaceManager?.getRoots() ?? [],
|
||||
primaryRootIndex: this.workspaceManager?.getPrimaryIndex() ?? 0,
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
import type { GetVoicesRequest, ValidateApiKeyResponse } from "../../../shared/proto/cline/tts"
|
||||
import type { Controller } from ".."
|
||||
|
||||
/**
|
||||
* Check if TTS API key is configured and valid
|
||||
* @param controller The controller instance
|
||||
* @param request Empty request
|
||||
* @returns Whether an API key is configured and valid
|
||||
*/
|
||||
export async function checkApiKeyConfigured(controller: Controller, request: GetVoicesRequest): Promise<ValidateApiKeyResponse> {
|
||||
try {
|
||||
// Check if API key exists in secrets
|
||||
const apiKey = await controller.context.secrets.get("elevenLabsApiKey")
|
||||
|
||||
console.log("[checkApiKeyConfigured] Checking API key:", {
|
||||
hasKey: !!apiKey,
|
||||
keyLength: apiKey?.length || 0,
|
||||
})
|
||||
|
||||
if (!apiKey) {
|
||||
console.log("[checkApiKeyConfigured] No API key found in secrets")
|
||||
return {
|
||||
isValid: false,
|
||||
error: undefined, // No error, just not configured
|
||||
}
|
||||
}
|
||||
|
||||
// API key exists, validate it
|
||||
const { TextToSpeechService } = await import("../../../services/tts/TextToSpeechService")
|
||||
const ttsService = new TextToSpeechService()
|
||||
|
||||
await ttsService.initialize({
|
||||
provider: "elevenlabs",
|
||||
apiKey: apiKey,
|
||||
})
|
||||
|
||||
// Quick validation check
|
||||
const isValid = await ttsService.validateApiKey()
|
||||
|
||||
return {
|
||||
isValid,
|
||||
error: isValid ? undefined : "API key is invalid or expired",
|
||||
}
|
||||
} catch (error) {
|
||||
return {
|
||||
isValid: false,
|
||||
error: `Failed to check API key: ${error instanceof Error ? error.message : "Unknown error"}`,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Export with PascalCase for proto compatibility
|
||||
export { checkApiKeyConfigured as CheckApiKeyConfigured }
|
||||
@@ -0,0 +1,51 @@
|
||||
import type { GetVoicesRequest, VoicesResponse } from "../../../shared/proto/cline/tts"
|
||||
import type { Controller } from ".."
|
||||
|
||||
/**
|
||||
* Get list of available voices for the current TTS provider
|
||||
* @param controller The controller instance
|
||||
* @param request The request (currently empty)
|
||||
* @returns List of available voices
|
||||
*/
|
||||
export async function getAvailableVoices(controller: Controller, request: GetVoicesRequest): Promise<VoicesResponse> {
|
||||
try {
|
||||
// Get API key from secrets storage
|
||||
const apiKey = await controller.context.secrets.get("elevenLabsApiKey")
|
||||
|
||||
if (!apiKey) {
|
||||
return {
|
||||
voices: [],
|
||||
error: "No API key found. Please validate your API key first.",
|
||||
}
|
||||
}
|
||||
|
||||
// Initialize TTS service with the stored key
|
||||
const { TextToSpeechService } = await import("../../../services/tts/TextToSpeechService")
|
||||
const ttsService = new TextToSpeechService()
|
||||
|
||||
await ttsService.initialize({
|
||||
provider: "elevenlabs",
|
||||
apiKey: apiKey,
|
||||
})
|
||||
|
||||
const result = await ttsService.getAvailableVoices()
|
||||
|
||||
return {
|
||||
voices: result.voices.map((voice) => ({
|
||||
id: voice.id,
|
||||
name: voice.name,
|
||||
description: voice.description,
|
||||
previewUrl: voice.previewUrl,
|
||||
})),
|
||||
error: result.error,
|
||||
}
|
||||
} catch (error) {
|
||||
return {
|
||||
voices: [],
|
||||
error: `Failed to fetch voices: ${error instanceof Error ? error.message : "Unknown error"}`,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Export with PascalCase for proto compatibility
|
||||
export { getAvailableVoices as GetAvailableVoices }
|
||||
@@ -0,0 +1,80 @@
|
||||
import type { SynthesizeRequest, SynthesizeResponse } from "../../../shared/proto/cline/tts"
|
||||
import type { Controller } from ".."
|
||||
|
||||
/**
|
||||
* Synthesize speech from text using the configured TTS provider
|
||||
* @param controller The controller instance
|
||||
* @param request The synthesis request containing text and voice settings
|
||||
* @returns The synthesized audio data
|
||||
*/
|
||||
export async function synthesizeSpeech(controller: Controller, request: SynthesizeRequest): Promise<SynthesizeResponse> {
|
||||
try {
|
||||
// Get API key from secrets storage
|
||||
const apiKey = await controller.context.secrets.get("elevenLabsApiKey")
|
||||
|
||||
if (!apiKey) {
|
||||
return {
|
||||
audioData: Buffer.alloc(0),
|
||||
contentType: "audio/mpeg",
|
||||
error: "No API key found. Please validate your API key first.",
|
||||
}
|
||||
}
|
||||
|
||||
// Initialize TTS service with the stored key
|
||||
const { TextToSpeechService } = await import("../../../services/tts/TextToSpeechService")
|
||||
const ttsService = new TextToSpeechService()
|
||||
|
||||
await ttsService.initialize({
|
||||
provider: "elevenlabs",
|
||||
apiKey: apiKey,
|
||||
})
|
||||
|
||||
const result = await ttsService.synthesizeSpeech({
|
||||
voiceId: request.voiceId,
|
||||
text: request.text,
|
||||
speed: request.speed,
|
||||
stability: request.stability,
|
||||
similarityBoost: request.similarityBoost,
|
||||
})
|
||||
|
||||
console.log("[synthesizeSpeech] TTS Service Result:", {
|
||||
audioDataLength: result.audioData?.length,
|
||||
audioDataType: typeof result.audioData,
|
||||
contentType: result.contentType,
|
||||
error: result.error,
|
||||
})
|
||||
|
||||
// Protobuf-ts expects Uint8Array, not Buffer
|
||||
const audioUint8Array = new Uint8Array(result.audioData)
|
||||
console.log("[synthesizeSpeech] Created Uint8Array:", {
|
||||
arrayLength: audioUint8Array.length,
|
||||
arrayType: typeof audioUint8Array,
|
||||
isUint8Array: audioUint8Array instanceof Uint8Array,
|
||||
})
|
||||
|
||||
const response: SynthesizeResponse = {
|
||||
audioData: audioUint8Array as any, // Protobuf bytes field accepts Uint8Array
|
||||
contentType: result.contentType,
|
||||
error: result.error,
|
||||
}
|
||||
|
||||
console.log("[synthesizeSpeech] Returning response:", {
|
||||
audioDataLength: response.audioData?.length,
|
||||
hasAudioData: !!response.audioData,
|
||||
contentType: response.contentType,
|
||||
error: response.error,
|
||||
})
|
||||
|
||||
return response
|
||||
} catch (error) {
|
||||
const emptyArray = new Uint8Array(0)
|
||||
return {
|
||||
audioData: emptyArray as any,
|
||||
contentType: "audio/mpeg",
|
||||
error: `Failed to synthesize speech: ${error instanceof Error ? error.message : "Unknown error"}`,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Export with PascalCase for proto compatibility
|
||||
export { synthesizeSpeech as SynthesizeSpeech }
|
||||
@@ -0,0 +1,60 @@
|
||||
import type { ValidateApiKeyRequest, ValidateApiKeyResponse } from "../../../shared/proto/cline/tts"
|
||||
import type { Controller } from ".."
|
||||
|
||||
/**
|
||||
* Validate the TTS API key
|
||||
* @param controller The controller instance
|
||||
* @param request The validation request (currently empty)
|
||||
* @returns Whether the API key is valid
|
||||
*/
|
||||
export async function validateApiKey(controller: Controller, request: ValidateApiKeyRequest): Promise<ValidateApiKeyResponse> {
|
||||
try {
|
||||
const apiKey = request.apiKey
|
||||
|
||||
if (!apiKey || apiKey.trim() === "") {
|
||||
return {
|
||||
isValid: false,
|
||||
error: "API key is required",
|
||||
}
|
||||
}
|
||||
|
||||
console.log("[validateApiKey] Saving API key to secrets storage...")
|
||||
// Save the API key to secrets storage
|
||||
await controller.context.secrets.store("elevenLabsApiKey", apiKey)
|
||||
console.log("[validateApiKey] API key saved successfully")
|
||||
|
||||
// Initialize TTS service with the new key
|
||||
const { TextToSpeechService } = await import("../../../services/tts/TextToSpeechService")
|
||||
const ttsService = new TextToSpeechService()
|
||||
|
||||
await ttsService.initialize({
|
||||
provider: "elevenlabs",
|
||||
apiKey: apiKey,
|
||||
})
|
||||
|
||||
// Validate the API key by attempting to fetch voices
|
||||
// Also try to get voices to provide better error feedback
|
||||
const voicesResult = await ttsService.getAvailableVoices()
|
||||
|
||||
if (voicesResult.error) {
|
||||
return {
|
||||
isValid: false,
|
||||
error: voicesResult.error,
|
||||
}
|
||||
}
|
||||
|
||||
// If we got voices successfully, the key is valid
|
||||
return {
|
||||
isValid: true,
|
||||
error: undefined,
|
||||
}
|
||||
} catch (error) {
|
||||
return {
|
||||
isValid: false,
|
||||
error: `Failed to validate API key: ${error instanceof Error ? error.message : "Unknown error"}`,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Export with PascalCase for proto compatibility
|
||||
export { validateApiKey as ValidateApiKey }
|
||||
@@ -0,0 +1,23 @@
|
||||
import { BooleanRequest } from "../../../shared/proto/cline/common"
|
||||
import { Controller } from ".."
|
||||
|
||||
/**
|
||||
* Toggle discuss mode on/off
|
||||
* @param controller The controller instance
|
||||
* @param request The request containing the enabled state
|
||||
* @returns BooleanRequest with the new state
|
||||
*/
|
||||
export async function setDiscussModeEnabled(controller: Controller, request: BooleanRequest): Promise<BooleanRequest> {
|
||||
const enabled = request.value
|
||||
|
||||
// Update global state
|
||||
controller.stateManager.setGlobalState("discussModeEnabled", enabled)
|
||||
|
||||
// Notify webview of state change
|
||||
await controller.postStateToWebview()
|
||||
|
||||
return BooleanRequest.create({ value: enabled })
|
||||
}
|
||||
|
||||
// Export with PascalCase for code generation
|
||||
export { setDiscussModeEnabled as SetDiscussModeEnabled }
|
||||
@@ -0,0 +1,41 @@
|
||||
import { Empty } from "../../../shared/proto/cline/common"
|
||||
import type { DiscussVoiceSettingsRequest } from "../../../shared/proto/cline/ui"
|
||||
import type { Controller } from ".."
|
||||
|
||||
/**
|
||||
* Update discuss mode voice settings
|
||||
* @param controller The controller instance
|
||||
* @param request The settings to update
|
||||
* @returns Empty response
|
||||
*/
|
||||
export async function updateDiscussVoiceSettings(controller: Controller, request: DiscussVoiceSettingsRequest): Promise<Empty> {
|
||||
try {
|
||||
// Update settings in global state
|
||||
if (request.selectedVoice !== undefined) {
|
||||
controller.stateManager.setGlobalState("discussModeSelectedVoice", request.selectedVoice)
|
||||
}
|
||||
|
||||
if (request.speechSpeed !== undefined) {
|
||||
controller.stateManager.setGlobalState("discussModeSpeechSpeed", request.speechSpeed)
|
||||
}
|
||||
|
||||
if (request.autoSpeak !== undefined) {
|
||||
controller.stateManager.setGlobalState("discussModeAutoSpeak", request.autoSpeak)
|
||||
}
|
||||
|
||||
if (request.autoListen !== undefined) {
|
||||
controller.stateManager.setGlobalState("discussModeAutoListen", request.autoListen)
|
||||
}
|
||||
|
||||
// Notify webview of state change
|
||||
await controller.postStateToWebview()
|
||||
|
||||
return Empty.create()
|
||||
} catch (error) {
|
||||
console.error("Failed to update discuss voice settings:", error)
|
||||
return Empty.create()
|
||||
}
|
||||
}
|
||||
|
||||
// Export with PascalCase for proto compatibility
|
||||
export { updateDiscussVoiceSettings as UpdateDiscussVoiceSettings }
|
||||
@@ -18,7 +18,51 @@ In each user message, the environment_details will specify the current mode. The
|
||||
- When starting in PLAN MODE, depending on the user's request, you may need to do some information gathering e.g. using read_file or search_files to get more context about the task.${context.yoloModeToggled !== true ? " You may also ask the user clarifying questions with ask_followup_question to get a better understanding of the task." : ""}
|
||||
- Once you've gained more context about the user's request, you should architect a detailed plan for how you will accomplish the task. Present the plan to the user using the plan_mode_respond tool.
|
||||
- Then you might ask the user if they are pleased with this plan, or if they would like to make any changes. Think of this as a brainstorming session where you can discuss the task and plan the best way to accomplish it.
|
||||
- Finally once it seems like you've reached a good plan, ask the user to switch you back to ACT MODE to implement the solution.`
|
||||
- Finally once it seems like you've reached a good plan, ask the user to switch you back to ACT MODE to implement the solution.
|
||||
|
||||
## DISCUSS MODE (Enhanced Plan Mode with Voice)
|
||||
|
||||
When the environment_details indicates that DISCUSS_MODE is enabled, you should adopt a more conversational and interactive planning approach. This is a voice-enabled conversation where the user is speaking to you naturally, so respond as if you're having a friendly, collaborative discussion with a colleague.
|
||||
|
||||
### Conversational Tone & Style
|
||||
- **Be conversational, not robotic** - Write as if you're speaking naturally in a friendly conversation
|
||||
- **Limit each response to 280 characters maximum** - Keep it brief since responses are converted to speech
|
||||
- **Avoid mentioning file paths** - Don't include file paths or directory structures in your responses as they're awkward when spoken aloud. Reference files by name only when absolutely necessary (e.g., "the config file" instead of "src/config/settings.json")
|
||||
- Use contractions naturally ("I'll", "you're", "that's", "we've") to sound more human
|
||||
- Break down complex topics into dialogue-friendly chunks across multiple exchanges
|
||||
- Use a warm, collaborative tone that encourages discussion - imagine you're brainstorming with a friend
|
||||
- Avoid overly technical jargon unless necessary; explain concepts clearly and conversationally
|
||||
- Speak in the first person ("I'll help you..." not "We will...")
|
||||
- Use natural, everyday language rather than formal or stilted phrasing
|
||||
- Show enthusiasm and engagement - use phrases like "That's a great idea!", "I love that approach!", "Perfect!"
|
||||
|
||||
### Proactive Question-Asking
|
||||
- After each response, actively seek to understand the user's needs better by asking 1-2 relevant follow-up questions
|
||||
- Ask clarifying questions about requirements, constraints, preferences, or technical details
|
||||
- Guide the conversation naturally toward a complete understanding of the task
|
||||
- Frame questions conversationally: "What framework are you using?" not "Please specify the framework"
|
||||
- Examples of good conversational follow-up questions:
|
||||
* "What framework are you using for this project?"
|
||||
* "Do you have any specific design preferences or constraints I should know about?"
|
||||
* "Would you like me to explain how this approach works, or should I move forward with the implementation?"
|
||||
|
||||
### Plan Completion Signal
|
||||
- Once you have gathered sufficient context and created a comprehensive plan, signal this clearly in your response
|
||||
- Use conversational phrases like:
|
||||
* "I now have a clear plan for this task."
|
||||
* "I'm ready to implement this solution when you're ready."
|
||||
* "I have everything I need to build this. Would you like me to proceed?"
|
||||
- Then explicitly ask: "Shall I switch to Act Mode and start building this?"
|
||||
- This signals to the user that the planning phase is complete and implementation can begin
|
||||
|
||||
### Natural Conversation Flow
|
||||
- In Discuss Mode, the user may be speaking to you, so expect more natural language and potentially incomplete sentences
|
||||
- Be patient and ask for clarification when needed - don't assume or guess
|
||||
- Always acknowledge the user's input before diving into technical details (e.g., "Got it!", "I understand", "That makes sense")
|
||||
- Use conversational transitions like "Great!", "I see,", "That makes sense,", "Ah, I understand" to maintain natural flow
|
||||
- Mirror the user's energy level - if they're excited, match that enthusiasm; if they're casual, be casual too
|
||||
- Ask follow-up questions naturally as part of the conversation, not as a formal interrogation
|
||||
- Remember: The goal is collaborative discussion, not just information extraction. You're having a conversation, not conducting an interview`
|
||||
|
||||
export async function getActVsPlanModeSection(variant: PromptVariant, context: SystemPromptContext): Promise<string> {
|
||||
const template = variant.componentOverrides?.[SystemPromptSection.ACT_VS_PLAN]?.template || getActVsPlanModeTemplateText
|
||||
|
||||
@@ -54,6 +54,7 @@ export async function readSecretsFromDisk(context: ExtensionContext): Promise<Se
|
||||
minimaxApiKey,
|
||||
hicapApiKey,
|
||||
aihubmixApiKey,
|
||||
elevenLabsApiKey,
|
||||
] = await Promise.all([
|
||||
context.secrets.get("apiKey") as Promise<Secrets["apiKey"]>,
|
||||
context.secrets.get("openRouterApiKey") as Promise<Secrets["openRouterApiKey"]>,
|
||||
@@ -96,6 +97,7 @@ export async function readSecretsFromDisk(context: ExtensionContext): Promise<Se
|
||||
context.secrets.get("minimaxApiKey") as Promise<Secrets["minimaxApiKey"]>,
|
||||
context.secrets.get("hicapApiKey") as Promise<Secrets["hicapApiKey"]>,
|
||||
context.secrets.get("aihubmixApiKey") as Promise<Secrets["aihubmixApiKey"]>,
|
||||
context.secrets.get("elevenLabsApiKey") as Promise<Secrets["elevenLabsApiKey"]>,
|
||||
])
|
||||
|
||||
return {
|
||||
@@ -140,6 +142,7 @@ export async function readSecretsFromDisk(context: ExtensionContext): Promise<Se
|
||||
minimaxApiKey,
|
||||
hicapApiKey,
|
||||
aihubmixApiKey,
|
||||
elevenLabsApiKey,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -272,6 +275,15 @@ export async function readGlobalStateFromDisk(context: ExtensionContext): Promis
|
||||
const hicapModelId = context.globalState.get<GlobalStateAndSettings["hicapModelId"]>("hicapModelId")
|
||||
const aihubmixBaseUrl = context.globalState.get<GlobalStateAndSettings["aihubmixBaseUrl"]>("aihubmixBaseUrl")
|
||||
const aihubmixAppCode = context.globalState.get<GlobalStateAndSettings["aihubmixAppCode"]>("aihubmixAppCode")
|
||||
const discussModeEnabled = context.globalState.get<GlobalStateAndSettings["discussModeEnabled"]>("discussModeEnabled")
|
||||
const discussModeSelectedVoice =
|
||||
context.globalState.get<GlobalStateAndSettings["discussModeSelectedVoice"]>("discussModeSelectedVoice")
|
||||
const discussModeSpeechSpeed =
|
||||
context.globalState.get<GlobalStateAndSettings["discussModeSpeechSpeed"]>("discussModeSpeechSpeed")
|
||||
const discussModeAutoSpeak =
|
||||
context.globalState.get<GlobalStateAndSettings["discussModeAutoSpeak"]>("discussModeAutoSpeak")
|
||||
const discussModeAutoListen =
|
||||
context.globalState.get<GlobalStateAndSettings["discussModeAutoListen"]>("discussModeAutoListen")
|
||||
|
||||
// OpenTelemetry configuration
|
||||
const openTelemetryEnabled =
|
||||
@@ -669,6 +681,11 @@ export async function readGlobalStateFromDisk(context: ExtensionContext): Promis
|
||||
openTelemetryLogBatchSize: openTelemetryLogBatchSize ?? 512,
|
||||
openTelemetryLogBatchTimeout: openTelemetryLogBatchTimeout ?? 5000,
|
||||
openTelemetryLogMaxQueueSize: openTelemetryLogMaxQueueSize ?? 2048,
|
||||
discussModeEnabled: discussModeEnabled ?? false,
|
||||
discussModeSelectedVoice: discussModeSelectedVoice,
|
||||
discussModeSpeechSpeed: discussModeSpeechSpeed ?? 1.2,
|
||||
discussModeAutoSpeak: discussModeAutoSpeak ?? false,
|
||||
discussModeAutoListen: discussModeAutoListen ?? false,
|
||||
}
|
||||
} catch (error) {
|
||||
console.error("[StateHelpers] Failed to read global state:", error)
|
||||
|
||||
@@ -3507,8 +3507,13 @@ export class Task {
|
||||
|
||||
details += "\n\n# Current Mode"
|
||||
const mode = this.stateManager.getGlobalSettingsKey("mode")
|
||||
const discussModeEnabled = this.stateManager.getGlobalStateKey("discussModeEnabled") ?? false
|
||||
if (mode === "plan") {
|
||||
details += "\nPLAN MODE\n" + formatResponse.planModeInstructions()
|
||||
if (discussModeEnabled) {
|
||||
details +=
|
||||
"\n\nDISCUSS_MODE is enabled. You are in a voice-enabled conversation where the user may be speaking to you. Adopt a natural, conversational tone and keep responses brief and dialogue-friendly."
|
||||
}
|
||||
} else {
|
||||
details += "\nACT MODE"
|
||||
}
|
||||
|
||||
@@ -114,7 +114,9 @@ export abstract class WebviewProvider {
|
||||
font-src ${this.getCspSource()} data:;
|
||||
style-src ${this.getCspSource()} 'unsafe-inline';
|
||||
img-src ${this.getCspSource()} https: data:;
|
||||
media-src blob: data:;
|
||||
script-src 'nonce-${nonce}' 'unsafe-eval';">
|
||||
<meta http-equiv="Permissions-Policy" content="autoplay=*, microphone=*">
|
||||
<title>Cline</title>
|
||||
</head>
|
||||
<body>
|
||||
@@ -201,6 +203,7 @@ export abstract class WebviewProvider {
|
||||
`font-src ${this.getCspSource()}`,
|
||||
`style-src ${this.getCspSource()} 'unsafe-inline' https://* http://${localServerUrl} http://0.0.0.0:${localPort}`,
|
||||
`img-src ${this.getCspSource()} https: data:`,
|
||||
`media-src blob: data:`,
|
||||
`script-src 'unsafe-eval' https://* http://${localServerUrl} http://0.0.0.0:${localPort} 'nonce-${nonce}'`,
|
||||
`connect-src https://* ws://${localServerUrl} ws://0.0.0.0:${localPort} http://${localServerUrl} http://0.0.0.0:${localPort}`,
|
||||
]
|
||||
@@ -213,6 +216,7 @@ export abstract class WebviewProvider {
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width,initial-scale=1,shrink-to-fit=no">
|
||||
<meta http-equiv="Content-Security-Policy" content="${csp.join("; ")}">
|
||||
<meta http-equiv="Permissions-Policy" content="autoplay=*, microphone=*">
|
||||
<link rel="stylesheet" type="text/css" href="${stylesUrl}">
|
||||
<link href="${codiconsUrl}" rel="stylesheet" />
|
||||
<title>Cline</title>
|
||||
|
||||
@@ -18,6 +18,7 @@ export class AudioRecordingService {
|
||||
private recordingProcess: ChildProcess | null = null
|
||||
private startTime: number = 0
|
||||
private outputFile: string = ""
|
||||
private silenceDetectedCallback: (() => void) | null = null
|
||||
|
||||
constructor() {}
|
||||
|
||||
@@ -145,7 +146,26 @@ export class AudioRecordingService {
|
||||
|
||||
this.recordingProcess.stderr?.on("data", (data) => {
|
||||
const message = data.toString().trim()
|
||||
if (message && !message.includes("In:") && !message.includes("Out:")) {
|
||||
|
||||
// Check for silence detection
|
||||
if (message.includes("silence_end")) {
|
||||
// Parse silence duration from FFmpeg output
|
||||
// Format: [silencedetect @ 0x...] silence_end: 4.5 | silence_duration: 2.0
|
||||
const durationMatch = message.match(/silence_duration:\s*([\d.]+)/)
|
||||
if (durationMatch) {
|
||||
const silenceDuration = parseFloat(durationMatch[1])
|
||||
Logger.info(`Detected ${silenceDuration}s of silence`)
|
||||
|
||||
// If silence duration is >= 2 seconds, trigger auto-stop
|
||||
if (silenceDuration >= 2.0 && this.silenceDetectedCallback) {
|
||||
Logger.info("Auto-stopping recording due to silence detection")
|
||||
this.silenceDetectedCallback()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Log other stderr output (but filter out common noise)
|
||||
if (message && !message.includes("In:") && !message.includes("Out:") && !message.includes("silencedetect")) {
|
||||
Logger.info(`Recording stderr: ${message}`)
|
||||
}
|
||||
})
|
||||
@@ -231,6 +251,14 @@ export class AudioRecordingService {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Sets a callback to be triggered when silence is detected
|
||||
* @param callback Function to call when silence is detected
|
||||
*/
|
||||
setSilenceDetectedCallback(callback: (() => void) | null): void {
|
||||
this.silenceDetectedCallback = callback
|
||||
}
|
||||
|
||||
private checkRecordingDependencies(): { available: boolean; error?: string } {
|
||||
const program = this.getRecordProgram()
|
||||
if (!program) {
|
||||
|
||||
@@ -101,6 +101,7 @@ export class VoiceTranscriptionService {
|
||||
}
|
||||
|
||||
async transcribeAudio(audioBase64: string, language?: string): Promise<{ text?: string; error?: string }> {
|
||||
// Use Cline's transcription service (ElevenLabs STT is now handled in the controller)
|
||||
try {
|
||||
Logger.info("Transcribing audio with Cline transcription service...")
|
||||
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
import type { BaseTTSProvider, TTSOptions, TTSSynthesisResult, TTSVoicesResult } from "./providers/BaseTTSProvider"
|
||||
import { ElevenLabsProvider } from "./providers/ElevenLabsProvider"
|
||||
|
||||
/**
|
||||
* Supported TTS providers
|
||||
*/
|
||||
export type TTSProvider = "elevenlabs" | "openai"
|
||||
|
||||
/**
|
||||
* Configuration for TTS service
|
||||
*/
|
||||
export interface TTSConfig {
|
||||
provider: TTSProvider
|
||||
apiKey: string
|
||||
}
|
||||
|
||||
/**
|
||||
* Text-to-Speech Service
|
||||
* Manages TTS provider instances and handles speech synthesis requests
|
||||
*/
|
||||
export class TextToSpeechService {
|
||||
private provider: BaseTTSProvider | null = null
|
||||
private currentConfig: TTSConfig | null = null
|
||||
|
||||
/**
|
||||
* Initialize the TTS service with configuration
|
||||
*/
|
||||
async initialize(config: TTSConfig): Promise<void> {
|
||||
// Only reinitialize if config changed
|
||||
if (
|
||||
this.currentConfig &&
|
||||
this.currentConfig.provider === config.provider &&
|
||||
this.currentConfig.apiKey === config.apiKey
|
||||
) {
|
||||
return
|
||||
}
|
||||
|
||||
this.currentConfig = config
|
||||
this.provider = this.createProvider(config)
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a TTS provider instance based on configuration
|
||||
*/
|
||||
private createProvider(config: TTSConfig): BaseTTSProvider {
|
||||
switch (config.provider) {
|
||||
case "elevenlabs":
|
||||
return new ElevenLabsProvider(config.apiKey)
|
||||
case "openai":
|
||||
// TODO: Implement OpenAI TTS provider
|
||||
throw new Error("OpenAI TTS provider not yet implemented")
|
||||
default:
|
||||
throw new Error(`Unsupported TTS provider: ${config.provider}`)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Synthesize speech from text
|
||||
*/
|
||||
async synthesizeSpeech(options: TTSOptions): Promise<TTSSynthesisResult> {
|
||||
if (!this.provider) {
|
||||
throw new Error("TTS service not initialized. Call initialize() first.")
|
||||
}
|
||||
|
||||
return this.provider.synthesizeSpeech(options)
|
||||
}
|
||||
|
||||
/**
|
||||
* Get available voices for the current provider
|
||||
*/
|
||||
async getAvailableVoices(): Promise<TTSVoicesResult> {
|
||||
if (!this.provider) {
|
||||
throw new Error("TTS service not initialized. Call initialize() first.")
|
||||
}
|
||||
|
||||
return this.provider.getAvailableVoices()
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate the current API key
|
||||
*/
|
||||
async validateApiKey(): Promise<boolean> {
|
||||
if (!this.provider) {
|
||||
return false
|
||||
}
|
||||
|
||||
return this.provider.validateApiKey()
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if service is initialized
|
||||
*/
|
||||
isInitialized(): boolean {
|
||||
return this.provider !== null
|
||||
}
|
||||
|
||||
/**
|
||||
* Get current provider type
|
||||
*/
|
||||
getCurrentProvider(): TTSProvider | null {
|
||||
return this.currentConfig?.provider ?? null
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
/**
|
||||
* Base interface for Text-to-Speech providers
|
||||
*/
|
||||
|
||||
export interface TTSVoice {
|
||||
id: string
|
||||
name: string
|
||||
description?: string
|
||||
previewUrl?: string
|
||||
}
|
||||
|
||||
export interface TTSOptions {
|
||||
voiceId: string
|
||||
text: string
|
||||
speed?: number
|
||||
stability?: number
|
||||
similarityBoost?: number
|
||||
}
|
||||
|
||||
export interface TTSSynthesisResult {
|
||||
audioData: Uint8Array
|
||||
contentType: string
|
||||
error?: string
|
||||
}
|
||||
|
||||
export interface TTSVoicesResult {
|
||||
voices: TTSVoice[]
|
||||
error?: string
|
||||
}
|
||||
|
||||
export abstract class BaseTTSProvider {
|
||||
protected apiKey: string
|
||||
|
||||
constructor(apiKey: string) {
|
||||
this.apiKey = apiKey
|
||||
}
|
||||
|
||||
/**
|
||||
* Synthesize speech from text
|
||||
* @param options Synthesis options including text and voice settings
|
||||
* @returns Audio buffer and content type
|
||||
*/
|
||||
abstract synthesizeSpeech(options: TTSOptions): Promise<TTSSynthesisResult>
|
||||
|
||||
/**
|
||||
* Get list of available voices
|
||||
* @returns List of available voices
|
||||
*/
|
||||
abstract getAvailableVoices(): Promise<TTSVoicesResult>
|
||||
|
||||
/**
|
||||
* Validate API key by making a test request
|
||||
* @returns True if API key is valid
|
||||
*/
|
||||
abstract validateApiKey(): Promise<boolean>
|
||||
}
|
||||
@@ -0,0 +1,242 @@
|
||||
import axios from "axios"
|
||||
import { BaseTTSProvider, type TTSOptions, type TTSSynthesisResult, type TTSVoice, type TTSVoicesResult } from "./BaseTTSProvider"
|
||||
|
||||
const ELEVENLABS_API_BASE = "https://api.elevenlabs.io/v1"
|
||||
|
||||
export interface STTTranscriptionResult {
|
||||
text?: string
|
||||
error?: string
|
||||
}
|
||||
|
||||
/**
|
||||
* ElevenLabs Text-to-Speech Provider
|
||||
* Implements TTS using ElevenLabs API
|
||||
*/
|
||||
export class ElevenLabsProvider extends BaseTTSProvider {
|
||||
/**
|
||||
* Synthesize speech from text using ElevenLabs API
|
||||
*/
|
||||
async synthesizeSpeech(options: TTSOptions): Promise<TTSSynthesisResult> {
|
||||
try {
|
||||
const { voiceId, text, speed = 1.0, stability = 0.5, similarityBoost = 0.75 } = options
|
||||
|
||||
// Clamp speed to ElevenLabs API limits (0.7 to 1.2)
|
||||
const clampedSpeed = Math.max(0.7, Math.min(1.2, speed))
|
||||
|
||||
const response = await axios.post(
|
||||
`${ELEVENLABS_API_BASE}/text-to-speech/${voiceId}`,
|
||||
{
|
||||
text,
|
||||
model_id: "eleven_turbo_v2_5",
|
||||
voice_settings: {
|
||||
stability,
|
||||
similarity_boost: similarityBoost,
|
||||
speed: clampedSpeed,
|
||||
},
|
||||
},
|
||||
{
|
||||
headers: {
|
||||
"xi-api-key": this.apiKey,
|
||||
"Content-Type": "application/json",
|
||||
Accept: "audio/mpeg",
|
||||
},
|
||||
responseType: "arraybuffer",
|
||||
timeout: 30000, // 30 second timeout
|
||||
},
|
||||
)
|
||||
|
||||
return {
|
||||
audioData: new Uint8Array(response.data),
|
||||
contentType: response.headers["content-type"] || "audio/mpeg",
|
||||
}
|
||||
} catch (error) {
|
||||
if (axios.isAxiosError(error)) {
|
||||
const errorMessage = error.response?.data ? new TextDecoder().decode(error.response.data) : error.message
|
||||
return {
|
||||
audioData: new Uint8Array(0),
|
||||
contentType: "audio/mpeg",
|
||||
error: `ElevenLabs API error: ${errorMessage}`,
|
||||
}
|
||||
}
|
||||
return {
|
||||
audioData: new Uint8Array(0),
|
||||
contentType: "audio/mpeg",
|
||||
error: `Failed to synthesize speech: ${error instanceof Error ? error.message : "Unknown error"}`,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Get list of available voices from ElevenLabs
|
||||
*/
|
||||
async getAvailableVoices(): Promise<TTSVoicesResult> {
|
||||
try {
|
||||
const response = await axios.get(`${ELEVENLABS_API_BASE}/voices`, {
|
||||
headers: {
|
||||
"xi-api-key": this.apiKey,
|
||||
},
|
||||
timeout: 10000,
|
||||
})
|
||||
|
||||
const voices: TTSVoice[] = response.data.voices.map((voice: any) => ({
|
||||
id: voice.voice_id,
|
||||
name: voice.name,
|
||||
description: voice.labels?.description || voice.category,
|
||||
previewUrl: voice.preview_url,
|
||||
}))
|
||||
|
||||
return { voices }
|
||||
} catch (error) {
|
||||
if (axios.isAxiosError(error)) {
|
||||
// Extract error message from different possible sources
|
||||
let errorMessage = error.message
|
||||
|
||||
if (error.response?.data) {
|
||||
if (typeof error.response.data === "string") {
|
||||
errorMessage = error.response.data
|
||||
} else if (error.response.data.detail) {
|
||||
errorMessage = error.response.data.detail
|
||||
} else if (error.response.data.message) {
|
||||
errorMessage = error.response.data.message
|
||||
} else {
|
||||
errorMessage = JSON.stringify(error.response.data)
|
||||
}
|
||||
}
|
||||
|
||||
// Add status code if available
|
||||
if (error.response?.status) {
|
||||
errorMessage = `${error.response.status}: ${errorMessage}`
|
||||
}
|
||||
|
||||
return {
|
||||
voices: [],
|
||||
error: `Failed to fetch voices: ${errorMessage}`,
|
||||
}
|
||||
}
|
||||
return {
|
||||
voices: [],
|
||||
error: `Failed to fetch voices: ${error instanceof Error ? error.message : String(error)}`,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate API key by attempting to fetch voices
|
||||
*/
|
||||
async validateApiKey(): Promise<boolean> {
|
||||
try {
|
||||
await axios.get(`${ELEVENLABS_API_BASE}/voices`, {
|
||||
headers: {
|
||||
"xi-api-key": this.apiKey,
|
||||
},
|
||||
timeout: 5000,
|
||||
})
|
||||
return true
|
||||
} catch (error) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Transcribe audio to text using ElevenLabs Speech-to-Text API (Scribe v1 model)
|
||||
* @param audioBuffer The audio data as a Buffer or Uint8Array
|
||||
* @param language Optional language code (e.g., 'en', 'es', 'fr')
|
||||
* @returns Transcription result with text or error
|
||||
*/
|
||||
async transcribeAudio(audioBuffer: Buffer | Uint8Array, language?: string): Promise<STTTranscriptionResult> {
|
||||
try {
|
||||
// Convert Uint8Array to Buffer if needed
|
||||
const buffer = Buffer.isBuffer(audioBuffer) ? audioBuffer : Buffer.from(audioBuffer)
|
||||
|
||||
// Create form data manually
|
||||
const boundary = `----WebKitFormBoundary${Math.random().toString(36).substring(2)}`
|
||||
|
||||
// Build multipart form data body
|
||||
const parts: Buffer[] = []
|
||||
|
||||
// Add audio file part - ElevenLabs expects field name "file"
|
||||
parts.push(Buffer.from(`--${boundary}\r\n`))
|
||||
parts.push(Buffer.from('Content-Disposition: form-data; name="file"; filename="audio.webm"\r\n'))
|
||||
parts.push(Buffer.from("Content-Type: audio/webm\r\n\r\n"))
|
||||
parts.push(buffer)
|
||||
parts.push(Buffer.from("\r\n"))
|
||||
|
||||
// Add language parameter if provided
|
||||
if (language) {
|
||||
parts.push(Buffer.from(`--${boundary}\r\n`))
|
||||
parts.push(Buffer.from('Content-Disposition: form-data; name="language"\r\n\r\n'))
|
||||
parts.push(Buffer.from(`${language}\r\n`))
|
||||
}
|
||||
|
||||
// Add model parameter - Note: use underscore not hyphen
|
||||
parts.push(Buffer.from(`--${boundary}\r\n`))
|
||||
parts.push(Buffer.from('Content-Disposition: form-data; name="model_id"\r\n\r\n'))
|
||||
parts.push(Buffer.from("scribe_v1\r\n"))
|
||||
|
||||
// Close boundary
|
||||
parts.push(Buffer.from(`--${boundary}--\r\n`))
|
||||
|
||||
const body = Buffer.concat(parts)
|
||||
|
||||
const response = await axios.post(`${ELEVENLABS_API_BASE}/speech-to-text`, body, {
|
||||
headers: {
|
||||
"xi-api-key": this.apiKey,
|
||||
"Content-Type": `multipart/form-data; boundary=${boundary}`,
|
||||
},
|
||||
timeout: 120000, // 2 minute timeout for transcription
|
||||
})
|
||||
|
||||
// Extract text from response (ElevenLabs returns detailed JSON with words, timestamps, etc.)
|
||||
const text = response.data?.text || ""
|
||||
|
||||
return { text }
|
||||
} catch (error) {
|
||||
if (axios.isAxiosError(error)) {
|
||||
let errorMessage = error.message
|
||||
let errorDetails = ""
|
||||
|
||||
if (error.response?.data) {
|
||||
if (typeof error.response.data === "string") {
|
||||
errorMessage = error.response.data
|
||||
} else if (error.response.data.detail) {
|
||||
if (typeof error.response.data.detail === "object") {
|
||||
errorMessage = JSON.stringify(error.response.data.detail, null, 2)
|
||||
} else {
|
||||
errorMessage = String(error.response.data.detail)
|
||||
}
|
||||
} else if (error.response.data.message) {
|
||||
errorMessage = String(error.response.data.message)
|
||||
} else if (error.response.data.error) {
|
||||
errorMessage = String(error.response.data.error)
|
||||
} else {
|
||||
// Try to extract useful error info
|
||||
try {
|
||||
errorDetails = JSON.stringify(error.response.data, null, 2)
|
||||
errorMessage = errorDetails
|
||||
} catch (e) {
|
||||
errorMessage = "Invalid response format"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Add status code if available
|
||||
const statusCode = error.response?.status || "Unknown"
|
||||
const fullMessage = `${statusCode}: ${errorMessage}`
|
||||
|
||||
console.error("[ElevenLabs STT] Full error details:", {
|
||||
status: error.response?.status,
|
||||
statusText: error.response?.statusText,
|
||||
data: error.response?.data,
|
||||
headers: error.response?.headers,
|
||||
})
|
||||
|
||||
return {
|
||||
error: `ElevenLabs transcription error: ${fullMessage}`,
|
||||
}
|
||||
}
|
||||
return {
|
||||
error: `Failed to transcribe audio: ${error instanceof Error ? error.message : String(error)}`,
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -86,6 +86,15 @@ export interface ExtensionState {
|
||||
customPrompt?: string
|
||||
autoCondenseThreshold?: number
|
||||
favoritedModelIds: string[]
|
||||
// Discuss Mode settings
|
||||
discussModeEnabled?: boolean
|
||||
discussModeSettings?: {
|
||||
elevenLabsApiKey?: string
|
||||
selectedVoice?: string
|
||||
speechSpeed?: number
|
||||
autoSpeak?: boolean
|
||||
autoListen?: boolean
|
||||
}
|
||||
// NEW: Add workspace information
|
||||
workspaceRoots: WorkspaceRoot[]
|
||||
primaryRootIndex: number
|
||||
|
||||
@@ -7,6 +7,8 @@ export const AUDIO_PROGRAM_CONFIG = {
|
||||
"avfoundation",
|
||||
"-i",
|
||||
":default",
|
||||
"-af",
|
||||
"silencedetect=noise=-30dB:d=2", // Detect 2 seconds of silence at -30dB threshold
|
||||
"-c:a",
|
||||
"libopus",
|
||||
"-b:a",
|
||||
@@ -32,6 +34,8 @@ export const AUDIO_PROGRAM_CONFIG = {
|
||||
"alsa",
|
||||
"-i",
|
||||
"default",
|
||||
"-af",
|
||||
"silencedetect=noise=-30dB:d=2", // Detect 2 seconds of silence at -30dB threshold
|
||||
"-c:a",
|
||||
"libopus",
|
||||
"-b:a",
|
||||
@@ -61,6 +65,8 @@ export const AUDIO_PROGRAM_CONFIG = {
|
||||
"wasapi",
|
||||
"-i",
|
||||
"audio=default",
|
||||
"-af",
|
||||
"silencedetect=noise=-30dB:d=2", // Detect 2 seconds of silence at -30dB threshold
|
||||
"-c:a",
|
||||
"libopus",
|
||||
"-b:a",
|
||||
|
||||
@@ -51,6 +51,7 @@ export interface GlobalState {
|
||||
lastDismissedModelBannerVersion: number
|
||||
lastDismissedCliBannerVersion: number
|
||||
nativeToolCallEnabled: boolean
|
||||
discussModeEnabled: boolean
|
||||
}
|
||||
|
||||
export interface Settings {
|
||||
@@ -121,6 +122,11 @@ export interface Settings {
|
||||
hooksEnabled: boolean
|
||||
subagentsEnabled: boolean
|
||||
hicapModelId: string | undefined
|
||||
// Discuss Mode settings
|
||||
discussModeSelectedVoice: string | undefined
|
||||
discussModeSpeechSpeed: number | undefined
|
||||
discussModeAutoSpeak: boolean
|
||||
discussModeAutoListen: boolean
|
||||
|
||||
// Plan mode configurations
|
||||
planModeApiProvider: ApiProvider
|
||||
@@ -254,6 +260,7 @@ export interface Secrets {
|
||||
minimaxApiKey: string | undefined
|
||||
hicapApiKey: string | undefined
|
||||
aihubmixApiKey: string | undefined
|
||||
elevenLabsApiKey: string | undefined
|
||||
}
|
||||
|
||||
export interface LocalState {
|
||||
|
||||
Generated
+5
-3
@@ -55,8 +55,10 @@
|
||||
"@testing-library/react": "^16.2.0",
|
||||
"@testing-library/user-event": "^14.6.1",
|
||||
"@types/dompurify": "^3.0.5",
|
||||
"@types/hast": "^3.0.4",
|
||||
"@types/jest": "^29.5.14",
|
||||
"@types/node": "^22.13.4",
|
||||
"@types/prop-types": "^15.7.15",
|
||||
"@types/react": "^18.3.18",
|
||||
"@types/react-dom": "^18.3.5",
|
||||
"@types/uuid": "^9.0.8",
|
||||
@@ -7250,9 +7252,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@types/prop-types": {
|
||||
"version": "15.7.14",
|
||||
"resolved": "https://registry.npmjs.org/@types/prop-types/-/prop-types-15.7.14.tgz",
|
||||
"integrity": "sha512-gNMvNH49DJ7OJYv+KAKn0Xp45p8PLl6zo2YnvDIbTd4J6MER2BmWN49TG7n9LvkyihINxeKW8+3bfS2yDC9dzQ==",
|
||||
"version": "15.7.15",
|
||||
"resolved": "https://registry.npmjs.org/@types/prop-types/-/prop-types-15.7.15.tgz",
|
||||
"integrity": "sha512-F6bEyamV9jKGAFBEmlQnesRPGOQqS2+Uwi0Em15xenOxHaf2hv6L8YCVn3rPdPJOiJfPiCnLIRyvwVaqMY3MIw==",
|
||||
"devOptional": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
|
||||
@@ -63,8 +63,10 @@
|
||||
"@testing-library/react": "^16.2.0",
|
||||
"@testing-library/user-event": "^14.6.1",
|
||||
"@types/dompurify": "^3.0.5",
|
||||
"@types/hast": "^3.0.4",
|
||||
"@types/jest": "^29.5.14",
|
||||
"@types/node": "^22.13.4",
|
||||
"@types/prop-types": "^15.7.15",
|
||||
"@types/react": "^18.3.18",
|
||||
"@types/react-dom": "^18.3.5",
|
||||
"@types/uuid": "^9.0.8",
|
||||
|
||||
@@ -0,0 +1,265 @@
|
||||
import { SynthesizeRequest } from "@shared/proto/cline/tts"
|
||||
import React, { useCallback, useEffect, useRef, useState } from "react"
|
||||
import { TtsServiceClient } from "@/services/grpc-client"
|
||||
|
||||
export interface AudioQueueItem {
|
||||
id: string
|
||||
text: string
|
||||
voiceId: string
|
||||
speed?: number
|
||||
}
|
||||
|
||||
export interface AudioPlayerProps {
|
||||
/** Called when playback of an item completes */
|
||||
onPlaybackComplete?: (itemId: string) => void
|
||||
/** Called when an error occurs */
|
||||
onError?: (error: string, itemId?: string) => void
|
||||
/** Called when playback state changes */
|
||||
onPlaybackStateChange?: (isPlaying: boolean, itemId?: string) => void
|
||||
/** Default voice ID to use */
|
||||
defaultVoiceId?: string
|
||||
/** Default playback speed */
|
||||
defaultSpeed?: number
|
||||
}
|
||||
|
||||
export interface AudioPlayerHandle {
|
||||
/** Add audio to the queue */
|
||||
enqueue: (item: AudioQueueItem) => void
|
||||
/** Clear the entire queue */
|
||||
clearQueue: () => void
|
||||
/** Pause current playback */
|
||||
pause: () => void
|
||||
/** Resume current playback */
|
||||
resume: () => void
|
||||
/** Stop current playback and clear queue */
|
||||
stop: () => void
|
||||
/** Get current queue */
|
||||
getQueue: () => AudioQueueItem[]
|
||||
/** Check if currently playing */
|
||||
isPlaying: () => boolean
|
||||
}
|
||||
|
||||
/**
|
||||
* AudioPlayer component that handles TTS audio playback with queue management.
|
||||
* Uses the TTS service to synthesize speech and plays it back.
|
||||
*/
|
||||
const AudioPlayer = React.forwardRef<AudioPlayerHandle, AudioPlayerProps>(
|
||||
(
|
||||
{
|
||||
onPlaybackComplete,
|
||||
onError,
|
||||
onPlaybackStateChange,
|
||||
defaultVoiceId = "EXAVITQu4vr4xnSDxMaL", // Default ElevenLabs voice
|
||||
defaultSpeed = 1.0,
|
||||
},
|
||||
ref,
|
||||
) => {
|
||||
const [queue, setQueue] = useState<AudioQueueItem[]>([])
|
||||
const [currentItem, setCurrentItem] = useState<AudioQueueItem | null>(null)
|
||||
const [isPlaying, setIsPlaying] = useState(false)
|
||||
const [isPaused, setIsPaused] = useState(false)
|
||||
|
||||
const audioRef = useRef<HTMLAudioElement | null>(null)
|
||||
const currentBlobUrlRef = useRef<string | null>(null)
|
||||
const isProcessingRef = useRef(false)
|
||||
|
||||
// Cleanup blob URL when component unmounts or audio changes
|
||||
const cleanupBlobUrl = useCallback(() => {
|
||||
if (currentBlobUrlRef.current) {
|
||||
URL.revokeObjectURL(currentBlobUrlRef.current)
|
||||
currentBlobUrlRef.current = null
|
||||
}
|
||||
}, [])
|
||||
|
||||
// Process the next item in the queue
|
||||
const processNextItem = useCallback(async () => {
|
||||
if (isProcessingRef.current || queue.length === 0) {
|
||||
return
|
||||
}
|
||||
|
||||
isProcessingRef.current = true
|
||||
const nextItem = queue[0]
|
||||
setCurrentItem(nextItem)
|
||||
|
||||
try {
|
||||
// Synthesize speech via TTS service
|
||||
const response = await TtsServiceClient.SynthesizeSpeech(
|
||||
SynthesizeRequest.create({
|
||||
text: nextItem.text,
|
||||
voiceId: nextItem.voiceId || defaultVoiceId,
|
||||
speed: nextItem.speed || defaultSpeed,
|
||||
}),
|
||||
)
|
||||
|
||||
if (response.error) {
|
||||
throw new Error(response.error)
|
||||
}
|
||||
|
||||
// Convert Buffer to Uint8Array then to Blob
|
||||
const audioArray = new Uint8Array(response.audioData)
|
||||
const audioBlob = new Blob([audioArray], { type: response.contentType || "audio/mpeg" })
|
||||
const audioUrl = URL.createObjectURL(audioBlob)
|
||||
|
||||
// Cleanup previous blob URL
|
||||
cleanupBlobUrl()
|
||||
currentBlobUrlRef.current = audioUrl
|
||||
|
||||
// Create and play audio element
|
||||
if (audioRef.current) {
|
||||
audioRef.current.pause()
|
||||
audioRef.current = null
|
||||
}
|
||||
|
||||
const audio = new Audio(audioUrl)
|
||||
audioRef.current = audio
|
||||
|
||||
// Set up event handlers
|
||||
audio.onplay = () => {
|
||||
setIsPlaying(true)
|
||||
setIsPaused(false)
|
||||
onPlaybackStateChange?.(true, nextItem.id)
|
||||
}
|
||||
|
||||
audio.onpause = () => {
|
||||
if (!audio.ended) {
|
||||
setIsPaused(true)
|
||||
onPlaybackStateChange?.(false, nextItem.id)
|
||||
}
|
||||
}
|
||||
|
||||
audio.onended = () => {
|
||||
setIsPlaying(false)
|
||||
setIsPaused(false)
|
||||
setCurrentItem(null)
|
||||
cleanupBlobUrl()
|
||||
|
||||
// Remove completed item from queue
|
||||
setQueue((prev) => prev.slice(1))
|
||||
onPlaybackComplete?.(nextItem.id)
|
||||
onPlaybackStateChange?.(false, nextItem.id)
|
||||
|
||||
isProcessingRef.current = false
|
||||
|
||||
// Process next item if available
|
||||
setTimeout(() => {
|
||||
if (queue.length > 1) {
|
||||
processNextItem()
|
||||
}
|
||||
}, 0)
|
||||
}
|
||||
|
||||
audio.onerror = (e) => {
|
||||
const errorMessage = `Audio playback error: ${audio.error?.message || "Unknown error"}`
|
||||
console.error(errorMessage, e)
|
||||
setIsPlaying(false)
|
||||
setCurrentItem(null)
|
||||
cleanupBlobUrl()
|
||||
|
||||
// Remove failed item from queue
|
||||
setQueue((prev) => prev.slice(1))
|
||||
onError?.(errorMessage, nextItem.id)
|
||||
|
||||
isProcessingRef.current = false
|
||||
|
||||
// Try next item
|
||||
setTimeout(() => {
|
||||
if (queue.length > 1) {
|
||||
processNextItem()
|
||||
}
|
||||
}, 0)
|
||||
}
|
||||
|
||||
// Start playback
|
||||
await audio.play()
|
||||
} catch (error) {
|
||||
const errorMessage = error instanceof Error ? error.message : "Failed to synthesize speech"
|
||||
console.error("TTS synthesis error:", error)
|
||||
|
||||
setIsPlaying(false)
|
||||
setCurrentItem(null)
|
||||
cleanupBlobUrl()
|
||||
|
||||
// Remove failed item from queue
|
||||
setQueue((prev) => prev.slice(1))
|
||||
onError?.(errorMessage, nextItem.id)
|
||||
|
||||
isProcessingRef.current = false
|
||||
|
||||
// Try next item
|
||||
setTimeout(() => {
|
||||
if (queue.length > 1) {
|
||||
processNextItem()
|
||||
}
|
||||
}, 0)
|
||||
}
|
||||
}, [queue, defaultVoiceId, defaultSpeed, onPlaybackComplete, onError, onPlaybackStateChange, cleanupBlobUrl])
|
||||
|
||||
// Auto-process queue when items are added
|
||||
useEffect(() => {
|
||||
if (queue.length > 0 && !currentItem && !isProcessingRef.current) {
|
||||
processNextItem()
|
||||
}
|
||||
}, [queue, currentItem, processNextItem])
|
||||
|
||||
// Expose imperative handle
|
||||
React.useImperativeHandle(ref, () => ({
|
||||
enqueue: (item: AudioQueueItem) => {
|
||||
setQueue((prev) => [...prev, item])
|
||||
},
|
||||
clearQueue: () => {
|
||||
if (audioRef.current) {
|
||||
audioRef.current.pause()
|
||||
audioRef.current = null
|
||||
}
|
||||
setQueue([])
|
||||
setCurrentItem(null)
|
||||
setIsPlaying(false)
|
||||
setIsPaused(false)
|
||||
cleanupBlobUrl()
|
||||
isProcessingRef.current = false
|
||||
},
|
||||
pause: () => {
|
||||
if (audioRef.current && !audioRef.current.paused) {
|
||||
audioRef.current.pause()
|
||||
}
|
||||
},
|
||||
resume: () => {
|
||||
if (audioRef.current && audioRef.current.paused) {
|
||||
audioRef.current.play()
|
||||
}
|
||||
},
|
||||
stop: () => {
|
||||
if (audioRef.current) {
|
||||
audioRef.current.pause()
|
||||
audioRef.current = null
|
||||
}
|
||||
setQueue([])
|
||||
setCurrentItem(null)
|
||||
setIsPlaying(false)
|
||||
setIsPaused(false)
|
||||
cleanupBlobUrl()
|
||||
isProcessingRef.current = false
|
||||
},
|
||||
getQueue: () => queue,
|
||||
isPlaying: () => isPlaying,
|
||||
}))
|
||||
|
||||
// Cleanup on unmount
|
||||
useEffect(() => {
|
||||
return () => {
|
||||
if (audioRef.current) {
|
||||
audioRef.current.pause()
|
||||
audioRef.current = null
|
||||
}
|
||||
cleanupBlobUrl()
|
||||
}
|
||||
}, [cleanupBlobUrl])
|
||||
|
||||
// This component doesn't render anything visible - it's purely functional
|
||||
return null
|
||||
},
|
||||
)
|
||||
|
||||
AudioPlayer.displayName = "AudioPlayer"
|
||||
|
||||
export default AudioPlayer
|
||||
@@ -85,6 +85,7 @@ interface ChatRowProps {
|
||||
sendMessageFromChatRow?: (text: string, images: string[], files: string[]) => void
|
||||
onSetQuote: (text: string) => void
|
||||
onCancelCommand?: () => void
|
||||
discussModeEnabled?: boolean
|
||||
}
|
||||
|
||||
interface QuoteButtonState {
|
||||
@@ -273,6 +274,7 @@ export const ChatRowContent = memo(
|
||||
sendMessageFromChatRow,
|
||||
onSetQuote,
|
||||
onCancelCommand,
|
||||
discussModeEnabled,
|
||||
}: ChatRowContentProps) => {
|
||||
const { mcpServers, mcpMarketplaceCatalog, onRelinquishControl, vscodeTerminalExecutionMode } = useExtensionState()
|
||||
const [seeNewChangesDisabled, setSeeNewChangesDisabled] = useState(false)
|
||||
@@ -1333,6 +1335,10 @@ export const ChatRowContent = memo(
|
||||
</div>
|
||||
)
|
||||
case "text":
|
||||
// Hide AI-generated text in discuss mode
|
||||
if (discussModeEnabled) {
|
||||
return null
|
||||
}
|
||||
return (
|
||||
<WithCopyButton
|
||||
onMouseUp={handleMouseUp}
|
||||
@@ -1352,6 +1358,10 @@ export const ChatRowContent = memo(
|
||||
</WithCopyButton>
|
||||
)
|
||||
case "reasoning":
|
||||
// Hide reasoning text in discuss mode
|
||||
if (discussModeEnabled) {
|
||||
return null
|
||||
}
|
||||
return (
|
||||
<>
|
||||
{message.text && (
|
||||
|
||||
@@ -11,6 +11,7 @@ import { normalizeApiConfiguration } from "@/components/settings/utils/providerU
|
||||
import { useExtensionState } from "@/context/ExtensionStateContext"
|
||||
import { useShowNavbar } from "@/context/PlatformContext"
|
||||
import { FileServiceClient, UiServiceClient } from "@/services/grpc-client"
|
||||
import { DiscussModeToggle } from "../discuss-mode/DiscussModeToggle"
|
||||
import { Navbar } from "../menu/Navbar"
|
||||
import AutoApproveBar from "./auto-approve-menu/AutoApproveBar"
|
||||
// Import utilities and hooks from the new structure
|
||||
@@ -29,6 +30,7 @@ import {
|
||||
useScrollBehavior,
|
||||
WelcomeSection,
|
||||
} from "./chat-view"
|
||||
import { useDiscussModeAudio } from "./hooks/useDiscussModeAudio"
|
||||
|
||||
interface ChatViewProps {
|
||||
isHidden: boolean
|
||||
@@ -53,6 +55,7 @@ const ChatView = ({ isHidden, showAnnouncement, hideAnnouncement, showHistoryVie
|
||||
userInfo,
|
||||
currentFocusChainChecklist,
|
||||
hooksEnabled,
|
||||
discussModeEnabled,
|
||||
} = useExtensionState()
|
||||
const isProdHostedApp = userInfo?.apiBaseUrl === "https://app.cline.bot"
|
||||
const shouldShowQuickWins = isProdHostedApp && (!taskHistory || taskHistory.length < QUICK_WINS_HISTORY_THRESHOLD)
|
||||
@@ -330,6 +333,9 @@ const ChatView = ({ isHidden, showAnnouncement, hideAnnouncement, showHistoryVie
|
||||
// Use scroll behavior hook
|
||||
const scrollBehavior = useScrollBehavior(messages, visibleMessages, groupedMessages, expandedRows, setExpandedRows)
|
||||
|
||||
// Use Discuss Mode audio hook for automatic TTS
|
||||
const { stopAudio } = useDiscussModeAudio(messages)
|
||||
|
||||
const placeholderText = useMemo(() => {
|
||||
const text = task ? "Type a message..." : "Type your task here..."
|
||||
return text
|
||||
@@ -366,6 +372,7 @@ const ChatView = ({ isHidden, showAnnouncement, hideAnnouncement, showHistoryVie
|
||||
{task && (
|
||||
<MessagesArea
|
||||
chatState={chatState}
|
||||
discussModeEnabled={discussModeEnabled}
|
||||
groupedMessages={groupedMessages}
|
||||
messageHandlers={messageHandlers}
|
||||
modifiedMessages={modifiedMessages}
|
||||
@@ -376,6 +383,7 @@ const ChatView = ({ isHidden, showAnnouncement, hideAnnouncement, showHistoryVie
|
||||
</div>
|
||||
<footer className="bg-(--vscode-sidebar-background)" style={{ gridRow: "2" }}>
|
||||
<AutoApproveBar />
|
||||
<DiscussModeToggle />
|
||||
<ActionButtons
|
||||
chatState={chatState}
|
||||
messageHandlers={messageHandlers}
|
||||
@@ -396,6 +404,7 @@ const ChatView = ({ isHidden, showAnnouncement, hideAnnouncement, showHistoryVie
|
||||
scrollBehavior={scrollBehavior}
|
||||
selectFilesAndImages={selectFilesAndImages}
|
||||
shouldDisableFilesAndImages={shouldDisableFilesAndImages}
|
||||
stopAudio={stopAudio}
|
||||
/>
|
||||
</footer>
|
||||
</ChatLayout>
|
||||
|
||||
@@ -0,0 +1,156 @@
|
||||
import { SynthesizeRequest } from "@shared/proto/cline/tts"
|
||||
import { Play, Volume2, VolumeX } from "lucide-react"
|
||||
import { useEffect, useRef, useState } from "react"
|
||||
import { useExtensionState } from "@/context/ExtensionStateContext"
|
||||
import { TtsServiceClient } from "@/services/grpc-client"
|
||||
|
||||
interface MessageAudioPlayerProps {
|
||||
text: string
|
||||
messageTs: number
|
||||
}
|
||||
|
||||
/**
|
||||
* Audio player component that appears next to Cline's messages in Discuss Mode
|
||||
* Allows manual playback of TTS audio
|
||||
*/
|
||||
export function MessageAudioPlayer({ text, messageTs }: MessageAudioPlayerProps) {
|
||||
const { discussModeSettings } = useExtensionState()
|
||||
const [isPlaying, setIsPlaying] = useState(false)
|
||||
const [isLoading, setIsLoading] = useState(false)
|
||||
const [error, setError] = useState<string | null>(null)
|
||||
const audioRef = useRef<HTMLAudioElement | null>(null)
|
||||
const audioUrlRef = useRef<string | null>(null)
|
||||
|
||||
const handlePlay = async () => {
|
||||
if (!discussModeSettings?.selectedVoice) {
|
||||
setError("No voice selected")
|
||||
return
|
||||
}
|
||||
|
||||
if (isPlaying && audioRef.current) {
|
||||
// Pause if already playing
|
||||
audioRef.current.pause()
|
||||
setIsPlaying(false)
|
||||
return
|
||||
}
|
||||
|
||||
// If we already have audio, just play it
|
||||
if (audioRef.current && audioUrlRef.current) {
|
||||
try {
|
||||
await audioRef.current.play()
|
||||
setIsPlaying(true)
|
||||
return
|
||||
} catch (err) {
|
||||
console.error("[MessageAudioPlayer] Error playing cached audio:", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Otherwise, synthesize new audio
|
||||
try {
|
||||
setIsLoading(true)
|
||||
setError(null)
|
||||
|
||||
const response = await TtsServiceClient.SynthesizeSpeech(
|
||||
SynthesizeRequest.create({
|
||||
text,
|
||||
voiceId: discussModeSettings.selectedVoice,
|
||||
speed: discussModeSettings.speechSpeed || 1.0,
|
||||
}),
|
||||
)
|
||||
|
||||
if (response.error) {
|
||||
setError(response.error)
|
||||
setIsLoading(false)
|
||||
return
|
||||
}
|
||||
|
||||
if (!response.audioData || response.audioData.length === 0) {
|
||||
setError("No audio data received")
|
||||
setIsLoading(false)
|
||||
return
|
||||
}
|
||||
|
||||
// Convert Buffer to Blob
|
||||
const audioBlob = new Blob([new Uint8Array(response.audioData)], {
|
||||
type: response.contentType || "audio/mpeg",
|
||||
})
|
||||
const audioUrl = URL.createObjectURL(audioBlob)
|
||||
audioUrlRef.current = audioUrl
|
||||
|
||||
// Create and play audio
|
||||
const audio = new Audio(audioUrl)
|
||||
audioRef.current = audio
|
||||
|
||||
audio.onended = () => {
|
||||
setIsPlaying(false)
|
||||
}
|
||||
|
||||
audio.onerror = () => {
|
||||
setError("Audio playback error")
|
||||
setIsPlaying(false)
|
||||
}
|
||||
|
||||
await audio.play()
|
||||
setIsPlaying(true)
|
||||
setIsLoading(false)
|
||||
} catch (err: any) {
|
||||
console.error("[MessageAudioPlayer] Error:", err)
|
||||
setError(err.message || "Failed to play audio")
|
||||
setIsLoading(false)
|
||||
}
|
||||
}
|
||||
|
||||
// Cleanup on unmount
|
||||
useEffect(() => {
|
||||
return () => {
|
||||
if (audioRef.current) {
|
||||
audioRef.current.pause()
|
||||
audioRef.current = null
|
||||
}
|
||||
if (audioUrlRef.current) {
|
||||
URL.revokeObjectURL(audioUrlRef.current)
|
||||
audioUrlRef.current = null
|
||||
}
|
||||
}
|
||||
}, [])
|
||||
|
||||
// Cleanup when message changes
|
||||
useEffect(() => {
|
||||
if (audioRef.current) {
|
||||
audioRef.current.pause()
|
||||
setIsPlaying(false)
|
||||
}
|
||||
if (audioUrlRef.current) {
|
||||
URL.revokeObjectURL(audioUrlRef.current)
|
||||
audioUrlRef.current = null
|
||||
}
|
||||
}, [messageTs, text])
|
||||
|
||||
if (!discussModeSettings?.selectedVoice) {
|
||||
return null
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="inline-flex items-center gap-1">
|
||||
<button
|
||||
className="p-1 rounded hover:bg-gray-200 dark:hover:bg-gray-700 disabled:opacity-50 transition-colors"
|
||||
disabled={isLoading}
|
||||
onClick={handlePlay}
|
||||
title={isPlaying ? "Pause" : isLoading ? "Loading..." : "Play audio"}
|
||||
type="button">
|
||||
{isLoading ? (
|
||||
<div className="w-4 h-4 border-2 border-gray-400 border-t-transparent rounded-full animate-spin" />
|
||||
) : isPlaying ? (
|
||||
<Volume2 className="w-4 h-4 text-blue-500" />
|
||||
) : (
|
||||
<Play className="w-4 h-4 text-gray-600 dark:text-gray-400" />
|
||||
)}
|
||||
</button>
|
||||
{error && (
|
||||
<span className="text-xs text-red-500" title={error}>
|
||||
<VolumeX className="w-3 h-3" />
|
||||
</span>
|
||||
)}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
import { cn } from "@/lib/utils"
|
||||
|
||||
interface RecordingIndicatorProps {
|
||||
duration: number
|
||||
className?: string
|
||||
}
|
||||
|
||||
/**
|
||||
* Visual indicator shown while recording audio
|
||||
* Displays a pulsing red dot and the recording duration
|
||||
*/
|
||||
export function RecordingIndicator({ duration, className }: RecordingIndicatorProps) {
|
||||
const formatDuration = (seconds: number) => {
|
||||
const mins = Math.floor(seconds / 60)
|
||||
const secs = Math.floor(seconds % 60)
|
||||
return `${mins}:${secs.toString().padStart(2, "0")}`
|
||||
}
|
||||
|
||||
return (
|
||||
<div className={cn("flex items-center gap-2 text-sm text-red-500", className)} data-testid="recording-indicator">
|
||||
<span className="relative flex h-2 w-2">
|
||||
<span className="animate-ping absolute inline-flex h-full w-full rounded-full bg-red-400 opacity-75"></span>
|
||||
<span className="relative inline-flex rounded-full h-2 w-2 bg-red-500"></span>
|
||||
</span>
|
||||
<span>Recording {formatDuration(duration)}</span>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,50 @@
|
||||
import { Volume2Icon } from "lucide-react"
|
||||
import React from "react"
|
||||
import { cn } from "@/lib/utils"
|
||||
|
||||
export interface SpeakingIndicatorProps {
|
||||
/** Whether audio is currently playing */
|
||||
isPlaying: boolean
|
||||
/** Optional CSS class name */
|
||||
className?: string
|
||||
/** Size variant */
|
||||
size?: "sm" | "md" | "lg"
|
||||
/** Show text label */
|
||||
showLabel?: boolean
|
||||
}
|
||||
|
||||
/**
|
||||
* Visual indicator that shows when Cline is speaking (TTS audio is playing).
|
||||
* Displays an animated speaker icon.
|
||||
*/
|
||||
const SpeakingIndicator: React.FC<SpeakingIndicatorProps> = ({ isPlaying, className, size = "md", showLabel = false }) => {
|
||||
if (!isPlaying) {
|
||||
return null
|
||||
}
|
||||
|
||||
const sizeClasses = {
|
||||
sm: "w-4 h-4",
|
||||
md: "w-5 h-5",
|
||||
lg: "w-6 h-6",
|
||||
}
|
||||
|
||||
const textSizeClasses = {
|
||||
sm: "text-xs",
|
||||
md: "text-sm",
|
||||
lg: "text-base",
|
||||
}
|
||||
|
||||
return (
|
||||
<div className={cn("flex items-center gap-2 text-muted-foreground", className)}>
|
||||
<Volume2Icon
|
||||
className={cn("animate-pulse", sizeClasses[size])}
|
||||
style={{
|
||||
animation: "pulse 1.5s cubic-bezier(0.4, 0, 0.6, 1) infinite",
|
||||
}}
|
||||
/>
|
||||
{showLabel && <span className={cn("font-medium", textSizeClasses[size])}>Speaking...</span>}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
export default SpeakingIndicator
|
||||
@@ -0,0 +1,137 @@
|
||||
import { PauseIcon, PlayIcon, SkipForwardIcon, Volume2Icon, VolumeXIcon } from "lucide-react"
|
||||
import React, { useCallback } from "react"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import { Tooltip, TooltipContent, TooltipTrigger } from "@/components/ui/tooltip"
|
||||
import { cn } from "@/lib/utils"
|
||||
import type { AudioPlayerHandle } from "./AudioPlayer"
|
||||
|
||||
export interface VoiceConversationControlsProps {
|
||||
/** Reference to the audio player */
|
||||
audioPlayerRef: React.RefObject<AudioPlayerHandle>
|
||||
/** Whether audio is currently playing */
|
||||
isPlaying: boolean
|
||||
/** Whether audio is currently paused */
|
||||
isPaused?: boolean
|
||||
/** Current queue size */
|
||||
queueSize: number
|
||||
/** Whether controls should be disabled */
|
||||
disabled?: boolean
|
||||
/** Optional CSS class name */
|
||||
className?: string
|
||||
}
|
||||
|
||||
/**
|
||||
* Voice conversation controls for managing TTS playback.
|
||||
* Provides play/pause, skip, and stop functionality.
|
||||
*/
|
||||
const VoiceConversationControls: React.FC<VoiceConversationControlsProps> = ({
|
||||
audioPlayerRef,
|
||||
isPlaying,
|
||||
isPaused = false,
|
||||
queueSize,
|
||||
disabled = false,
|
||||
className,
|
||||
}) => {
|
||||
const handlePlayPause = useCallback(() => {
|
||||
if (!audioPlayerRef.current) return
|
||||
|
||||
if (isPaused) {
|
||||
audioPlayerRef.current.resume()
|
||||
} else if (isPlaying) {
|
||||
audioPlayerRef.current.pause()
|
||||
}
|
||||
}, [audioPlayerRef, isPlaying, isPaused])
|
||||
|
||||
const handleSkip = useCallback(() => {
|
||||
if (!audioPlayerRef.current) return
|
||||
|
||||
// Stop current playback and let the queue continue
|
||||
const queue = audioPlayerRef.current.getQueue()
|
||||
if (queue.length > 0) {
|
||||
// Remove current item to skip to next
|
||||
audioPlayerRef.current.stop()
|
||||
// Re-add remaining items
|
||||
queue.slice(1).forEach((item) => {
|
||||
audioPlayerRef.current?.enqueue(item)
|
||||
})
|
||||
}
|
||||
}, [audioPlayerRef])
|
||||
|
||||
const handleStop = useCallback(() => {
|
||||
if (!audioPlayerRef.current) return
|
||||
audioPlayerRef.current.stop()
|
||||
}, [audioPlayerRef])
|
||||
|
||||
// Don't show controls if not playing and no queue
|
||||
if (!isPlaying && queueSize === 0) {
|
||||
return null
|
||||
}
|
||||
|
||||
return (
|
||||
<div className={cn("flex items-center gap-1", className)}>
|
||||
{/* Play/Pause Button */}
|
||||
{isPlaying && (
|
||||
<Tooltip>
|
||||
<TooltipTrigger asChild>
|
||||
<Button
|
||||
className="h-8 w-8 p-0"
|
||||
data-testid="voice-play-pause-button"
|
||||
disabled={disabled}
|
||||
onClick={handlePlayPause}
|
||||
size="sm"
|
||||
variant="ghost">
|
||||
{isPaused ? <PlayIcon className="h-4 w-4" /> : <PauseIcon className="h-4 w-4" />}
|
||||
</Button>
|
||||
</TooltipTrigger>
|
||||
<TooltipContent>{isPaused ? "Resume" : "Pause"}</TooltipContent>
|
||||
</Tooltip>
|
||||
)}
|
||||
|
||||
{/* Skip Button */}
|
||||
{queueSize > 1 && (
|
||||
<Tooltip>
|
||||
<TooltipTrigger asChild>
|
||||
<Button
|
||||
className="h-8 w-8 p-0"
|
||||
data-testid="voice-skip-button"
|
||||
disabled={disabled}
|
||||
onClick={handleSkip}
|
||||
size="sm"
|
||||
variant="ghost">
|
||||
<SkipForwardIcon className="h-4 w-4" />
|
||||
</Button>
|
||||
</TooltipTrigger>
|
||||
<TooltipContent>Skip to Next</TooltipContent>
|
||||
</Tooltip>
|
||||
)}
|
||||
|
||||
{/* Stop Button */}
|
||||
{(isPlaying || queueSize > 0) && (
|
||||
<Tooltip>
|
||||
<TooltipTrigger asChild>
|
||||
<Button
|
||||
className="h-8 w-8 p-0 text-destructive hover:text-destructive"
|
||||
data-testid="voice-stop-button"
|
||||
disabled={disabled}
|
||||
onClick={handleStop}
|
||||
size="sm"
|
||||
variant="ghost">
|
||||
<VolumeXIcon className="h-4 w-4" />
|
||||
</Button>
|
||||
</TooltipTrigger>
|
||||
<TooltipContent>Stop All Audio</TooltipContent>
|
||||
</Tooltip>
|
||||
)}
|
||||
|
||||
{/* Queue indicator */}
|
||||
{queueSize > 1 && (
|
||||
<div className="flex items-center gap-1 ml-1 text-xs text-muted-foreground">
|
||||
<Volume2Icon className="h-3 w-3" />
|
||||
<span>+{queueSize - 1}</span>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
export default VoiceConversationControls
|
||||
@@ -0,0 +1,39 @@
|
||||
import { MicIcon, MicOffIcon } from "lucide-react"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import { Tooltip, TooltipContent, TooltipTrigger } from "@/components/ui/tooltip"
|
||||
|
||||
interface VoiceInputButtonProps {
|
||||
isRecording: boolean
|
||||
isTranscribing: boolean
|
||||
disabled: boolean
|
||||
onClick: () => void
|
||||
}
|
||||
|
||||
/**
|
||||
* Button component for voice input in Discuss Mode
|
||||
* Shows different states for recording, transcribing, and idle
|
||||
*/
|
||||
export function VoiceInputButton({ isRecording, isTranscribing, disabled, onClick }: VoiceInputButtonProps) {
|
||||
const getTooltipText = () => {
|
||||
if (isTranscribing) return "Transcribing audio..."
|
||||
if (isRecording) return "Click to stop recording"
|
||||
return "Click to start voice input"
|
||||
}
|
||||
|
||||
return (
|
||||
<Tooltip>
|
||||
<TooltipTrigger asChild>
|
||||
<Button
|
||||
className={isRecording ? "text-red-500 animate-pulse" : ""}
|
||||
data-testid="voice-input-button"
|
||||
disabled={disabled || isTranscribing}
|
||||
onClick={onClick}
|
||||
size="sm"
|
||||
variant="ghost">
|
||||
{isRecording ? <MicOffIcon className="h-4 w-4" /> : <MicIcon className="h-4 w-4" />}
|
||||
</Button>
|
||||
</TooltipTrigger>
|
||||
<TooltipContent>{getTooltipText()}</TooltipContent>
|
||||
</Tooltip>
|
||||
)
|
||||
}
|
||||
@@ -1,6 +1,9 @@
|
||||
import React from "react"
|
||||
import React, { useState } from "react"
|
||||
import ChatTextArea from "@/components/chat/ChatTextArea"
|
||||
import QuotedMessagePreview from "@/components/chat/QuotedMessagePreview"
|
||||
import VoiceRecorder from "@/components/chat/VoiceRecorder"
|
||||
import { useClineAuth } from "@/context/ClineAuthContext"
|
||||
import { useExtensionState } from "@/context/ExtensionStateContext"
|
||||
import { ChatState, MessageHandlers, ScrollBehavior } from "../../types/chatTypes"
|
||||
|
||||
interface InputSectionProps {
|
||||
@@ -10,6 +13,7 @@ interface InputSectionProps {
|
||||
placeholderText: string
|
||||
shouldDisableFilesAndImages: boolean
|
||||
selectFilesAndImages: () => Promise<void>
|
||||
stopAudio: () => void
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -22,6 +26,7 @@ export const InputSection: React.FC<InputSectionProps> = ({
|
||||
placeholderText,
|
||||
shouldDisableFilesAndImages,
|
||||
selectFilesAndImages,
|
||||
stopAudio,
|
||||
}) => {
|
||||
const {
|
||||
activeQuote,
|
||||
@@ -39,6 +44,12 @@ export const InputSection: React.FC<InputSectionProps> = ({
|
||||
} = chatState
|
||||
|
||||
const { isAtBottom, scrollToBottomAuto } = scrollBehavior
|
||||
const { discussModeEnabled, dictationSettings, mode } = useExtensionState()
|
||||
const { clineUser } = useClineAuth()
|
||||
const [isVoiceRecording, setIsVoiceRecording] = useState(false)
|
||||
|
||||
// Check if we should show voice-only mode (Discuss Mode enabled in Plan mode)
|
||||
const showVoiceOnlyMode = discussModeEnabled && mode === "plan"
|
||||
|
||||
return (
|
||||
<>
|
||||
@@ -52,27 +63,70 @@ export const InputSection: React.FC<InputSectionProps> = ({
|
||||
</div>
|
||||
)}
|
||||
|
||||
<ChatTextArea
|
||||
activeQuote={activeQuote}
|
||||
inputValue={inputValue}
|
||||
onFocusChange={handleFocusChange}
|
||||
onHeightChange={() => {
|
||||
if (isAtBottom) {
|
||||
scrollToBottomAuto()
|
||||
}
|
||||
}}
|
||||
onSelectFilesAndImages={selectFilesAndImages}
|
||||
onSend={() => messageHandlers.handleSendMessage(inputValue, selectedImages, selectedFiles)}
|
||||
placeholderText={placeholderText}
|
||||
ref={textAreaRef}
|
||||
selectedFiles={selectedFiles}
|
||||
selectedImages={selectedImages}
|
||||
sendingDisabled={sendingDisabled}
|
||||
setInputValue={setInputValue}
|
||||
setSelectedFiles={setSelectedFiles}
|
||||
setSelectedImages={setSelectedImages}
|
||||
shouldDisableFilesAndImages={shouldDisableFilesAndImages}
|
||||
/>
|
||||
{showVoiceOnlyMode ? (
|
||||
// Voice-only mode for Discuss Mode
|
||||
<div className="flex justify-center items-center py-8 px-4">
|
||||
<div className="flex flex-col items-center gap-4">
|
||||
<VoiceRecorder
|
||||
disabled={sendingDisabled}
|
||||
isAuthenticated={!!clineUser?.uid}
|
||||
language={dictationSettings?.dictationLanguage || "en"}
|
||||
onProcessingStateChange={(isProcessing, message) => {
|
||||
// No need to show processing in input since there's no text field
|
||||
}}
|
||||
onRecordingStateChange={(isRecording) => {
|
||||
setIsVoiceRecording(isRecording)
|
||||
// Stop any playing audio when user starts recording
|
||||
if (isRecording) {
|
||||
stopAudio()
|
||||
}
|
||||
}}
|
||||
onTranscription={(text) => {
|
||||
if (!text) return
|
||||
|
||||
// Create blessed audio element during user gesture (transcription completion)
|
||||
try {
|
||||
const audio = new Audio()
|
||||
audio.preload = "auto"
|
||||
// Store in window for the audio hook to access
|
||||
;(window as any).__discussModeAudio = audio
|
||||
} catch (e) {
|
||||
console.warn("Could not create blessed audio element:", e)
|
||||
}
|
||||
|
||||
// Automatically send the message
|
||||
messageHandlers.handleSendMessage(text, [], [])
|
||||
}}
|
||||
/>
|
||||
{!isVoiceRecording && (
|
||||
<p className="text-xs text-muted-foreground text-center">Click the microphone to speak</p>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
) : (
|
||||
// Normal mode with full text area
|
||||
<ChatTextArea
|
||||
activeQuote={activeQuote}
|
||||
inputValue={inputValue}
|
||||
onFocusChange={handleFocusChange}
|
||||
onHeightChange={() => {
|
||||
if (isAtBottom) {
|
||||
scrollToBottomAuto()
|
||||
}
|
||||
}}
|
||||
onSelectFilesAndImages={selectFilesAndImages}
|
||||
onSend={() => messageHandlers.handleSendMessage(inputValue, selectedImages, selectedFiles)}
|
||||
placeholderText={placeholderText}
|
||||
ref={textAreaRef}
|
||||
selectedFiles={selectedFiles}
|
||||
selectedImages={selectedImages}
|
||||
sendingDisabled={sendingDisabled}
|
||||
setInputValue={setInputValue}
|
||||
setSelectedFiles={setSelectedFiles}
|
||||
setSelectedImages={setSelectedImages}
|
||||
shouldDisableFilesAndImages={shouldDisableFilesAndImages}
|
||||
/>
|
||||
)}
|
||||
</>
|
||||
)
|
||||
}
|
||||
|
||||
@@ -11,6 +11,7 @@ interface MessagesAreaProps {
|
||||
scrollBehavior: ScrollBehavior
|
||||
chatState: ChatState
|
||||
messageHandlers: MessageHandlers
|
||||
discussModeEnabled?: boolean
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -24,6 +25,7 @@ export const MessagesArea: React.FC<MessagesAreaProps> = ({
|
||||
scrollBehavior,
|
||||
chatState,
|
||||
messageHandlers,
|
||||
discussModeEnabled,
|
||||
}) => {
|
||||
const {
|
||||
virtuosoRef,
|
||||
@@ -47,6 +49,7 @@ export const MessagesArea: React.FC<MessagesAreaProps> = ({
|
||||
setActiveQuote,
|
||||
inputValue,
|
||||
messageHandlers,
|
||||
discussModeEnabled,
|
||||
),
|
||||
[
|
||||
groupedMessages,
|
||||
@@ -57,6 +60,7 @@ export const MessagesArea: React.FC<MessagesAreaProps> = ({
|
||||
setActiveQuote,
|
||||
inputValue,
|
||||
messageHandlers,
|
||||
discussModeEnabled,
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
@@ -15,6 +15,7 @@ interface MessageRendererProps {
|
||||
onSetQuote: (quote: string | null) => void
|
||||
inputValue: string
|
||||
messageHandlers: MessageHandlers
|
||||
discussModeEnabled?: boolean
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -32,6 +33,7 @@ export const MessageRenderer: React.FC<MessageRendererProps> = ({
|
||||
onSetQuote,
|
||||
inputValue,
|
||||
messageHandlers,
|
||||
discussModeEnabled,
|
||||
}) => {
|
||||
// Browser session group
|
||||
if (Array.isArray(messageOrGroup)) {
|
||||
@@ -58,6 +60,7 @@ export const MessageRenderer: React.FC<MessageRendererProps> = ({
|
||||
// Regular message
|
||||
return (
|
||||
<ChatRow
|
||||
discussModeEnabled={discussModeEnabled}
|
||||
inputValue={inputValue}
|
||||
isExpanded={expandedRows[messageOrGroup.ts] || false}
|
||||
isLast={isLast}
|
||||
@@ -86,9 +89,11 @@ export const createMessageRenderer = (
|
||||
onSetQuote: (quote: string | null) => void,
|
||||
inputValue: string,
|
||||
messageHandlers: MessageHandlers,
|
||||
discussModeEnabled?: boolean,
|
||||
) => {
|
||||
return (index: number, messageOrGroup: ClineMessage | ClineMessage[]) => (
|
||||
<MessageRenderer
|
||||
discussModeEnabled={discussModeEnabled}
|
||||
expandedRows={expandedRows}
|
||||
groupedMessages={groupedMessages}
|
||||
index={index}
|
||||
|
||||
@@ -0,0 +1,497 @@
|
||||
import { ClineMessage } from "@shared/ExtensionMessage"
|
||||
import { SynthesizeRequest } from "@shared/proto/cline/tts"
|
||||
import { useEffect, useRef, useState } from "react"
|
||||
import { useExtensionState } from "@/context/ExtensionStateContext"
|
||||
import { TtsServiceClient } from "@/services/grpc-client"
|
||||
|
||||
interface AudioQueueItem {
|
||||
text: string
|
||||
messageTs: number
|
||||
}
|
||||
|
||||
/**
|
||||
* Hook that handles automatic TTS synthesis and playback for Discuss Mode
|
||||
*/
|
||||
export function useDiscussModeAudio(messages: ClineMessage[]) {
|
||||
const { mode, discussModeEnabled, discussModeSettings, currentTaskItem } = useExtensionState()
|
||||
const [audioQueue, setAudioQueue] = useState<AudioQueueItem[]>([])
|
||||
const [isPlaying, setIsPlaying] = useState(false)
|
||||
const [currentAudio, setCurrentAudio] = useState<HTMLAudioElement | null>(null)
|
||||
const processedMessagesRef = useRef(new Set<number>())
|
||||
const audioRef = useRef<HTMLAudioElement | null>(null)
|
||||
const lastTaskIdRef = useRef<string | undefined>(undefined)
|
||||
const lastMessageCountRef = useRef(0)
|
||||
const taskStartTimeRef = useRef<number>(Date.now()) // Track when we started tracking this task
|
||||
const activeBlobUrlsRef = useRef<Set<string>>(new Set()) // Track all blob URLs for cleanup
|
||||
const audioElementRef = useRef<HTMLAudioElement | null>(null) // Pre-created audio element for autoplay
|
||||
const pendingAudioRef = useRef<{
|
||||
url: string
|
||||
audio: HTMLAudioElement
|
||||
onEnded: () => void
|
||||
onError: (e: Event) => void
|
||||
} | null>(null) // Audio waiting for user interaction
|
||||
|
||||
// Use blessed audio element from window if available, otherwise create one
|
||||
useEffect(() => {
|
||||
// Check if we have a blessed audio element from a user gesture
|
||||
const blessedAudio = (window as any).__discussModeAudio
|
||||
if (blessedAudio && !audioElementRef.current) {
|
||||
console.log("[DiscussModeAudio] Using blessed audio element from user gesture")
|
||||
audioElementRef.current = blessedAudio
|
||||
// Clear it so we don't reuse it
|
||||
delete (window as any).__discussModeAudio
|
||||
} else if (!audioElementRef.current) {
|
||||
console.log("[DiscussModeAudio] Creating audio element (may not be blessed)")
|
||||
const audio = new Audio()
|
||||
audio.preload = "auto"
|
||||
audioElementRef.current = audio
|
||||
}
|
||||
return () => {
|
||||
if (audioElementRef.current) {
|
||||
audioElementRef.current.pause()
|
||||
audioElementRef.current.src = ""
|
||||
audioElementRef.current = null
|
||||
}
|
||||
}
|
||||
}, [])
|
||||
|
||||
// Clear processed messages when task changes OR message array is reset
|
||||
useEffect(() => {
|
||||
const currentTaskId = currentTaskItem?.id
|
||||
const currentMessageCount = messages.length
|
||||
|
||||
// Helper function to cleanup all audio resources
|
||||
const cleanupAllAudio = () => {
|
||||
// Stop and cleanup current audio
|
||||
if (audioRef.current) {
|
||||
audioRef.current.pause()
|
||||
audioRef.current.currentTime = 0
|
||||
audioRef.current.src = ""
|
||||
audioRef.current = null
|
||||
}
|
||||
|
||||
// Clean up pending audio
|
||||
if (pendingAudioRef.current) {
|
||||
const pending = pendingAudioRef.current
|
||||
pending.audio.removeEventListener("ended", pending.onEnded)
|
||||
pending.audio.removeEventListener("error", pending.onError)
|
||||
pending.audio.src = ""
|
||||
URL.revokeObjectURL(pending.url)
|
||||
activeBlobUrlsRef.current.delete(pending.url)
|
||||
pendingAudioRef.current = null
|
||||
}
|
||||
|
||||
// Revoke ALL blob URLs to free memory and prevent replay
|
||||
activeBlobUrlsRef.current.forEach((url) => {
|
||||
console.log("[DiscussModeAudio] Revoking blob URL:", url.substring(0, 50))
|
||||
URL.revokeObjectURL(url)
|
||||
})
|
||||
activeBlobUrlsRef.current.clear()
|
||||
|
||||
// Clear all state
|
||||
processedMessagesRef.current.clear()
|
||||
setAudioQueue([])
|
||||
setIsPlaying(false)
|
||||
setCurrentAudio(null)
|
||||
taskStartTimeRef.current = Date.now()
|
||||
}
|
||||
|
||||
// Task ID changed
|
||||
if (currentTaskId && currentTaskId !== lastTaskIdRef.current) {
|
||||
console.log("[DiscussModeAudio] Task ID changed, CLEANING UP ALL AUDIO", {
|
||||
oldTaskId: lastTaskIdRef.current,
|
||||
newTaskId: currentTaskId,
|
||||
activeBlobUrls: activeBlobUrlsRef.current.size,
|
||||
})
|
||||
cleanupAllAudio()
|
||||
lastTaskIdRef.current = currentTaskId
|
||||
lastMessageCountRef.current = currentMessageCount
|
||||
return
|
||||
}
|
||||
|
||||
// Message count dropped significantly (task switch or history cleared)
|
||||
if (currentMessageCount < lastMessageCountRef.current - 2) {
|
||||
console.log("[DiscussModeAudio] Message count dropped, CLEANING UP ALL AUDIO", {
|
||||
oldCount: lastMessageCountRef.current,
|
||||
newCount: currentMessageCount,
|
||||
activeBlobUrls: activeBlobUrlsRef.current.size,
|
||||
})
|
||||
cleanupAllAudio()
|
||||
}
|
||||
|
||||
lastMessageCountRef.current = currentMessageCount
|
||||
}, [currentTaskItem?.id, messages.length])
|
||||
|
||||
// Detect new text messages and add to queue
|
||||
useEffect(() => {
|
||||
console.log("[DiscussModeAudio] Effect triggered", {
|
||||
mode,
|
||||
discussModeEnabled,
|
||||
autoSpeak: discussModeSettings?.autoSpeak,
|
||||
messageCount: messages.length,
|
||||
currentTaskId: currentTaskItem?.id,
|
||||
})
|
||||
|
||||
// Only process if in Plan Mode with Discuss Mode enabled and auto-speak on
|
||||
if (mode !== "plan") {
|
||||
console.log("[DiscussModeAudio] Not in plan mode, skipping")
|
||||
return
|
||||
}
|
||||
|
||||
if (!discussModeEnabled) {
|
||||
console.log("[DiscussModeAudio] Discuss mode not enabled, skipping")
|
||||
return
|
||||
}
|
||||
|
||||
if (!discussModeSettings?.autoSpeak) {
|
||||
console.log("[DiscussModeAudio] Auto-speak not enabled, skipping")
|
||||
return
|
||||
}
|
||||
|
||||
// Find the last message
|
||||
const lastMessage = messages[messages.length - 1]
|
||||
if (!lastMessage) {
|
||||
console.log("[DiscussModeAudio] No messages found")
|
||||
return
|
||||
}
|
||||
|
||||
console.log("[DiscussModeAudio] Last message:", {
|
||||
say: lastMessage.say,
|
||||
ask: lastMessage.ask,
|
||||
hasText: !!lastMessage.text,
|
||||
partial: lastMessage.partial,
|
||||
ts: lastMessage.ts,
|
||||
taskStartTime: taskStartTimeRef.current,
|
||||
messageAge: Date.now() - lastMessage.ts,
|
||||
isOldMessage: lastMessage.ts < taskStartTimeRef.current,
|
||||
alreadyProcessed: processedMessagesRef.current.has(lastMessage.ts),
|
||||
})
|
||||
|
||||
// Check if it's a response from Cline that we haven't processed
|
||||
// IMPORTANT: Only process Cline's responses (ask is set), not user messages (say === "text")
|
||||
// In Plan Mode, Cline uses ask="plan_mode_respond" or ask="followup"
|
||||
const isClineResponse = lastMessage.ask !== undefined
|
||||
|
||||
// CRITICAL: Only process messages created AFTER we started tracking this task
|
||||
// This prevents old messages from playing when switching tasks
|
||||
if (lastMessage.ts < taskStartTimeRef.current) {
|
||||
console.log("[DiscussModeAudio] Skipping old message from before current task started", {
|
||||
messageTs: lastMessage.ts,
|
||||
taskStartTime: taskStartTimeRef.current,
|
||||
messageAge: Date.now() - lastMessage.ts,
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
if (isClineResponse && lastMessage.text && !lastMessage.partial && !processedMessagesRef.current.has(lastMessage.ts)) {
|
||||
// Extract natural text from JSON response format
|
||||
let textToSpeak = lastMessage.text
|
||||
try {
|
||||
// Parse JSON to extract just the response text
|
||||
const parsed = JSON.parse(lastMessage.text)
|
||||
if (parsed.response) {
|
||||
textToSpeak = parsed.response
|
||||
}
|
||||
} catch (e) {
|
||||
// If not JSON, use the text as-is
|
||||
console.log("[DiscussModeAudio] Text is not JSON, using as-is")
|
||||
}
|
||||
|
||||
// Skip if no actual text content
|
||||
if (!textToSpeak || textToSpeak.trim().length === 0) {
|
||||
console.log("[DiscussModeAudio] No text content to speak")
|
||||
return
|
||||
}
|
||||
|
||||
// Truncate very long responses (ElevenLabs has limits)
|
||||
const maxLength = 5000 // ElevenLabs can handle ~5000 chars
|
||||
if (textToSpeak.length > maxLength) {
|
||||
console.log(`[DiscussModeAudio] Truncating long response from ${textToSpeak.length} to ${maxLength} chars`)
|
||||
textToSpeak = textToSpeak.substring(0, maxLength) + "... response truncated for audio."
|
||||
}
|
||||
|
||||
// Mark as processed
|
||||
processedMessagesRef.current.add(lastMessage.ts)
|
||||
|
||||
// Add to queue
|
||||
console.log("[DiscussModeAudio] New Cline response detected, adding to queue:", textToSpeak.substring(0, 50))
|
||||
setAudioQueue((prev) => [...prev, { text: textToSpeak, messageTs: lastMessage.ts }])
|
||||
}
|
||||
}, [messages, mode, discussModeEnabled, discussModeSettings?.autoSpeak])
|
||||
|
||||
// Process audio queue
|
||||
useEffect(() => {
|
||||
// First, check if we have pending audio and a blessed element to retry it
|
||||
const blessedAudio = (window as any).__discussModeAudio
|
||||
const pending = pendingAudioRef.current
|
||||
|
||||
if (blessedAudio && pending && !isPlaying) {
|
||||
console.log("[DiscussModeAudio] Blessed audio element available, retrying pending audio playback")
|
||||
|
||||
// Use the blessed audio element
|
||||
audioElementRef.current = blessedAudio
|
||||
delete (window as any).__discussModeAudio
|
||||
|
||||
// Update the pending audio to use the blessed element
|
||||
pending.audio = blessedAudio
|
||||
|
||||
// Set up the audio and retry playback
|
||||
pending.audio.src = pending.url
|
||||
pending.audio.addEventListener("ended", pending.onEnded)
|
||||
pending.audio.addEventListener("error", pending.onError)
|
||||
|
||||
audioRef.current = pending.audio
|
||||
setCurrentAudio(pending.audio)
|
||||
setIsPlaying(true)
|
||||
|
||||
// Try to play
|
||||
pending.audio
|
||||
.play()
|
||||
.then(() => {
|
||||
console.log("[DiscussModeAudio] Pending audio playing successfully after user interaction")
|
||||
pendingAudioRef.current = null // Clear pending
|
||||
})
|
||||
.catch((err) => {
|
||||
console.error("[DiscussModeAudio] Failed to play pending audio even with blessed element:", err)
|
||||
// Clean up on failure
|
||||
pending.audio.removeEventListener("ended", pending.onEnded)
|
||||
pending.audio.removeEventListener("error", pending.onError)
|
||||
pending.audio.src = ""
|
||||
URL.revokeObjectURL(pending.url)
|
||||
activeBlobUrlsRef.current.delete(pending.url)
|
||||
pendingAudioRef.current = null
|
||||
setIsPlaying(false)
|
||||
setCurrentAudio(null)
|
||||
audioRef.current = null
|
||||
// Remove from queue since we can't play it
|
||||
setAudioQueue((prev) => prev.slice(1))
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
if (isPlaying || audioQueue.length === 0 || !discussModeSettings?.selectedVoice) {
|
||||
return
|
||||
}
|
||||
|
||||
const processNextInQueue = async () => {
|
||||
const nextItem = audioQueue[0]
|
||||
if (!nextItem) return
|
||||
|
||||
try {
|
||||
setIsPlaying(true)
|
||||
console.log("[DiscussModeAudio] Synthesizing speech for message:", nextItem.messageTs)
|
||||
|
||||
// Call TTS service
|
||||
const response = await TtsServiceClient.SynthesizeSpeech(
|
||||
SynthesizeRequest.create({
|
||||
text: nextItem.text,
|
||||
voiceId: discussModeSettings.selectedVoice,
|
||||
speed: discussModeSettings.speechSpeed || 1.0,
|
||||
}),
|
||||
)
|
||||
|
||||
if (response.error) {
|
||||
console.error("[DiscussModeAudio] TTS synthesis error:", response.error)
|
||||
// Remove from queue and try next
|
||||
setAudioQueue((prev) => prev.slice(1))
|
||||
setIsPlaying(false)
|
||||
return
|
||||
}
|
||||
|
||||
if (!response.audioData || response.audioData.length === 0) {
|
||||
console.error("[DiscussModeAudio] No audio data received")
|
||||
setAudioQueue((prev) => prev.slice(1))
|
||||
setIsPlaying(false)
|
||||
return
|
||||
}
|
||||
|
||||
// Convert Buffer to Blob
|
||||
const audioBlob = new Blob([new Uint8Array(response.audioData)], {
|
||||
type: response.contentType || "audio/mpeg",
|
||||
})
|
||||
const audioUrl = URL.createObjectURL(audioBlob)
|
||||
|
||||
// Track this blob URL for cleanup
|
||||
activeBlobUrlsRef.current.add(audioUrl)
|
||||
console.log("[DiscussModeAudio] Created blob URL, total active:", activeBlobUrlsRef.current.size)
|
||||
|
||||
// Check for blessed audio element from user gesture
|
||||
const blessedAudio = (window as any).__discussModeAudio
|
||||
if (blessedAudio) {
|
||||
console.log("[DiscussModeAudio] Found blessed audio element, using it")
|
||||
audioElementRef.current = blessedAudio
|
||||
delete (window as any).__discussModeAudio
|
||||
}
|
||||
|
||||
// Use audio element (blessed or fallback)
|
||||
const audio = audioElementRef.current
|
||||
if (!audio) {
|
||||
console.error("[DiscussModeAudio] Audio element not available!")
|
||||
setAudioQueue((prev) => prev.slice(1))
|
||||
setIsPlaying(false)
|
||||
return
|
||||
}
|
||||
|
||||
// Set up event handlers
|
||||
const onEnded = () => {
|
||||
console.log("[DiscussModeAudio] Audio playback completed, cleaning up")
|
||||
URL.revokeObjectURL(audioUrl)
|
||||
activeBlobUrlsRef.current.delete(audioUrl)
|
||||
setCurrentAudio(null)
|
||||
audioRef.current = null
|
||||
// Remove from queue
|
||||
setAudioQueue((prev) => prev.slice(1))
|
||||
setIsPlaying(false)
|
||||
// Clean up handlers
|
||||
audio.removeEventListener("ended", onEnded)
|
||||
audio.removeEventListener("error", onError)
|
||||
}
|
||||
|
||||
const onError = (e: Event) => {
|
||||
console.error("[DiscussModeAudio] Audio playback error:", e)
|
||||
URL.revokeObjectURL(audioUrl)
|
||||
activeBlobUrlsRef.current.delete(audioUrl)
|
||||
setCurrentAudio(null)
|
||||
audioRef.current = null
|
||||
setAudioQueue((prev) => prev.slice(1))
|
||||
setIsPlaying(false)
|
||||
// Clean up handlers
|
||||
audio.removeEventListener("ended", onEnded)
|
||||
audio.removeEventListener("error", onError)
|
||||
}
|
||||
|
||||
audio.addEventListener("ended", onEnded)
|
||||
audio.addEventListener("error", onError)
|
||||
|
||||
// Update audio src and play
|
||||
audio.src = audioUrl
|
||||
audioRef.current = audio
|
||||
setCurrentAudio(audio)
|
||||
|
||||
console.log("[DiscussModeAudio] Playing audio with pre-created element...")
|
||||
|
||||
// Try to play
|
||||
try {
|
||||
await audio.play()
|
||||
console.log("[DiscussModeAudio] Audio playing successfully")
|
||||
} catch (playError) {
|
||||
console.warn(
|
||||
"[DiscussModeAudio] Autoplay blocked by browser. Keeping audio ready for retry after user interaction.",
|
||||
)
|
||||
|
||||
// Browser blocked autoplay - keep audio ready for retry when user interacts
|
||||
// Store the audio setup so we can retry when blessed audio element becomes available
|
||||
pendingAudioRef.current = {
|
||||
url: audioUrl,
|
||||
audio: audio,
|
||||
onEnded,
|
||||
onError,
|
||||
}
|
||||
|
||||
// Don't remove from queue or revoke blob URL yet - we'll retry
|
||||
setIsPlaying(false)
|
||||
|
||||
// Note: Audio will automatically retry when user interacts and creates blessed audio element
|
||||
return
|
||||
}
|
||||
} catch (error) {
|
||||
console.error("[DiscussModeAudio] Error processing audio:", error)
|
||||
setAudioQueue((prev) => prev.slice(1))
|
||||
setIsPlaying(false)
|
||||
}
|
||||
}
|
||||
|
||||
processNextInQueue()
|
||||
}, [audioQueue, isPlaying, discussModeSettings?.selectedVoice, discussModeSettings?.speechSpeed])
|
||||
|
||||
// Cleanup on unmount - revoke ALL blob URLs
|
||||
useEffect(() => {
|
||||
return () => {
|
||||
console.log("[DiscussModeAudio] Component unmounting, cleaning up all audio")
|
||||
if (audioRef.current) {
|
||||
audioRef.current.pause()
|
||||
audioRef.current.src = ""
|
||||
audioRef.current = null
|
||||
}
|
||||
// Clean up pending audio
|
||||
if (pendingAudioRef.current) {
|
||||
const pending = pendingAudioRef.current
|
||||
pending.audio.removeEventListener("ended", pending.onEnded)
|
||||
pending.audio.removeEventListener("error", pending.onError)
|
||||
pending.audio.src = ""
|
||||
URL.revokeObjectURL(pending.url)
|
||||
pendingAudioRef.current = null
|
||||
}
|
||||
// Revoke all blob URLs
|
||||
activeBlobUrlsRef.current.forEach((url) => {
|
||||
URL.revokeObjectURL(url)
|
||||
})
|
||||
activeBlobUrlsRef.current.clear()
|
||||
}
|
||||
}, [])
|
||||
|
||||
// Clear queue and DELETE all audio blobs when Discuss Mode is disabled or mode changes
|
||||
useEffect(() => {
|
||||
if (mode !== "plan" || !discussModeEnabled) {
|
||||
console.log("[DiscussModeAudio] Mode/Discuss Mode changed, DELETING ALL AUDIO BLOBS", {
|
||||
mode,
|
||||
discussModeEnabled,
|
||||
activeBlobUrls: activeBlobUrlsRef.current.size,
|
||||
})
|
||||
|
||||
// Stop current audio
|
||||
if (audioRef.current) {
|
||||
audioRef.current.pause()
|
||||
audioRef.current.currentTime = 0
|
||||
audioRef.current.src = ""
|
||||
audioRef.current = null
|
||||
}
|
||||
|
||||
// Clean up pending audio
|
||||
if (pendingAudioRef.current) {
|
||||
const pending = pendingAudioRef.current
|
||||
pending.audio.removeEventListener("ended", pending.onEnded)
|
||||
pending.audio.removeEventListener("error", pending.onError)
|
||||
pending.audio.src = ""
|
||||
URL.revokeObjectURL(pending.url)
|
||||
activeBlobUrlsRef.current.delete(pending.url)
|
||||
pendingAudioRef.current = null
|
||||
}
|
||||
|
||||
// Revoke ALL blob URLs to delete audio files from memory
|
||||
activeBlobUrlsRef.current.forEach((url) => {
|
||||
console.log("[DiscussModeAudio] Deleting audio blob:", url.substring(0, 50))
|
||||
URL.revokeObjectURL(url)
|
||||
})
|
||||
activeBlobUrlsRef.current.clear()
|
||||
|
||||
// Clear all state
|
||||
setAudioQueue([])
|
||||
setIsPlaying(false)
|
||||
setCurrentAudio(null)
|
||||
processedMessagesRef.current.clear()
|
||||
taskStartTimeRef.current = Date.now()
|
||||
|
||||
console.log("[DiscussModeAudio] All audio blobs deleted and state cleared")
|
||||
}
|
||||
}, [mode, discussModeEnabled])
|
||||
|
||||
// Expose function to stop current audio (useful when user starts recording)
|
||||
const stopAudio = () => {
|
||||
if (audioRef.current) {
|
||||
console.log("[DiscussModeAudio] Stopping current audio playback")
|
||||
audioRef.current.pause()
|
||||
audioRef.current.currentTime = 0
|
||||
|
||||
// Clear queue to prevent auto-continuing
|
||||
setAudioQueue([])
|
||||
setIsPlaying(false)
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
isPlaying,
|
||||
queueLength: audioQueue.length,
|
||||
stopAudio,
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,132 @@
|
||||
import { useCallback, useEffect, useState } from "react"
|
||||
import { useExtensionState } from "@/context/ExtensionStateContext"
|
||||
import { DictationServiceClient } from "@/services/grpc-client"
|
||||
|
||||
/**
|
||||
* Custom hook for handling voice input in Discuss Mode
|
||||
* Leverages the existing DictationService infrastructure for recording and transcription
|
||||
*/
|
||||
export function useDiscussVoiceInput() {
|
||||
const [isRecording, setIsRecording] = useState(false)
|
||||
const [isTranscribing, setIsTranscribing] = useState(false)
|
||||
const [recordingDuration, setRecordingDuration] = useState(0)
|
||||
const [error, setError] = useState<string | null>(null)
|
||||
const { discussModeEnabled, dictationSettings } = useExtensionState()
|
||||
|
||||
// Auto-update recording duration while recording
|
||||
useEffect(() => {
|
||||
if (!isRecording) return
|
||||
|
||||
const interval = setInterval(async () => {
|
||||
try {
|
||||
const status = await DictationServiceClient.getRecordingStatus({})
|
||||
setRecordingDuration(status.durationSeconds)
|
||||
} catch (err) {
|
||||
console.error("Failed to get recording status:", err)
|
||||
}
|
||||
}, 100)
|
||||
|
||||
return () => clearInterval(interval)
|
||||
}, [isRecording])
|
||||
|
||||
/**
|
||||
* Starts audio recording
|
||||
*/
|
||||
const startRecording = useCallback(async () => {
|
||||
try {
|
||||
setError(null)
|
||||
const result = await DictationServiceClient.startRecording({})
|
||||
|
||||
if (result.success) {
|
||||
setIsRecording(true)
|
||||
} else {
|
||||
setError(result.error || "Failed to start recording")
|
||||
}
|
||||
} catch (err) {
|
||||
const errorMessage = err instanceof Error ? err.message : "Failed to start recording"
|
||||
setError(errorMessage)
|
||||
console.error("Start recording error:", err)
|
||||
}
|
||||
}, [])
|
||||
|
||||
/**
|
||||
* Stops recording and transcribes the audio
|
||||
* @param onTranscriptionComplete - Callback with transcribed text
|
||||
*/
|
||||
const stopRecording = useCallback(
|
||||
async (onTranscriptionComplete: (text: string) => void) => {
|
||||
try {
|
||||
setIsRecording(false)
|
||||
setIsTranscribing(true)
|
||||
setError(null)
|
||||
|
||||
// Stop recording and get audio data
|
||||
const audioResult = await DictationServiceClient.stopRecording({})
|
||||
|
||||
if (!audioResult.success || !audioResult.audioBase64) {
|
||||
setError(audioResult.error || "Failed to capture audio")
|
||||
setIsTranscribing(false)
|
||||
return
|
||||
}
|
||||
|
||||
// Transcribe the audio
|
||||
const transcription = await DictationServiceClient.transcribeAudio({
|
||||
audioBase64: audioResult.audioBase64,
|
||||
language: dictationSettings?.dictationLanguage || "en",
|
||||
})
|
||||
|
||||
if (transcription.error) {
|
||||
setError(transcription.error)
|
||||
} else if (transcription.text) {
|
||||
// Successfully transcribed - pass text to callback
|
||||
onTranscriptionComplete(transcription.text)
|
||||
}
|
||||
} catch (err) {
|
||||
const errorMessage = err instanceof Error ? err.message : "Transcription failed"
|
||||
setError(errorMessage)
|
||||
console.error("Transcription error:", err)
|
||||
} finally {
|
||||
setIsTranscribing(false)
|
||||
setRecordingDuration(0)
|
||||
}
|
||||
},
|
||||
[dictationSettings],
|
||||
)
|
||||
|
||||
/**
|
||||
* Cancels the current recording without transcribing
|
||||
*/
|
||||
const cancelRecording = useCallback(async () => {
|
||||
try {
|
||||
await DictationServiceClient.cancelRecording({})
|
||||
setIsRecording(false)
|
||||
setRecordingDuration(0)
|
||||
setError(null)
|
||||
} catch (err) {
|
||||
console.error("Cancel recording error:", err)
|
||||
}
|
||||
}, [])
|
||||
|
||||
/**
|
||||
* Clears any error messages
|
||||
*/
|
||||
const clearError = useCallback(() => {
|
||||
setError(null)
|
||||
}, [])
|
||||
|
||||
// Check if voice input is available
|
||||
// Requires both Discuss Mode to be enabled and platform support (currently macOS only)
|
||||
const isAvailable = discussModeEnabled && (dictationSettings?.featureEnabled ?? false)
|
||||
|
||||
return {
|
||||
isRecording,
|
||||
isTranscribing,
|
||||
recordingDuration,
|
||||
error,
|
||||
isAvailable,
|
||||
startRecording,
|
||||
stopRecording,
|
||||
cancelRecording,
|
||||
clearError,
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,117 @@
|
||||
import { MessageSquare, Mic, Volume2 } from "lucide-react"
|
||||
import { memo } from "react"
|
||||
|
||||
export type ConversationState = "idle" | "listening" | "thinking" | "speaking"
|
||||
|
||||
interface ConversationStatusIndicatorProps {
|
||||
state: ConversationState
|
||||
className?: string
|
||||
}
|
||||
|
||||
/**
|
||||
* ConversationStatusIndicator displays the current state of the voice conversation
|
||||
* with animated visual feedback for Listening, Thinking, and Speaking states.
|
||||
*/
|
||||
export const ConversationStatusIndicator = memo(({ state, className = "" }: ConversationStatusIndicatorProps) => {
|
||||
// Don't render anything in idle state
|
||||
if (state === "idle") {
|
||||
return null
|
||||
}
|
||||
|
||||
const getStateConfig = () => {
|
||||
switch (state) {
|
||||
case "listening":
|
||||
return {
|
||||
icon: Mic,
|
||||
text: "Listening...",
|
||||
bgColor: "bg-blue-500/10",
|
||||
textColor: "text-blue-500",
|
||||
iconColor: "text-blue-500",
|
||||
pulseColor: "bg-blue-500",
|
||||
}
|
||||
case "thinking":
|
||||
return {
|
||||
icon: MessageSquare,
|
||||
text: "Thinking...",
|
||||
bgColor: "bg-purple-500/10",
|
||||
textColor: "text-purple-500",
|
||||
iconColor: "text-purple-500",
|
||||
pulseColor: "bg-purple-500",
|
||||
}
|
||||
case "speaking":
|
||||
return {
|
||||
icon: Volume2,
|
||||
text: "Speaking...",
|
||||
bgColor: "bg-green-500/10",
|
||||
textColor: "text-green-500",
|
||||
iconColor: "text-green-500",
|
||||
pulseColor: "bg-green-500",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const config = getStateConfig()
|
||||
const Icon = config.icon
|
||||
|
||||
return (
|
||||
<div
|
||||
className={`flex items-center gap-2 px-3 py-2 rounded-lg ${config.bgColor} transition-all duration-300 ${className}`}>
|
||||
{/* Animated Icon */}
|
||||
<div className="relative">
|
||||
<Icon className={`w-4 h-4 ${config.iconColor}`} strokeWidth={2} />
|
||||
|
||||
{/* Pulse animation */}
|
||||
<span className="absolute inset-0 flex items-center justify-center">
|
||||
<span
|
||||
className={`absolute w-4 h-4 rounded-full ${config.pulseColor} opacity-75 animate-ping`}
|
||||
style={{
|
||||
animationDuration: state === "speaking" ? "1.5s" : "2s",
|
||||
}}
|
||||
/>
|
||||
</span>
|
||||
</div>
|
||||
|
||||
{/* Status Text */}
|
||||
<span className={`text-sm font-medium ${config.textColor}`}>{config.text}</span>
|
||||
|
||||
{/* Visual Indicator - Wave animation for speaking/listening */}
|
||||
{(state === "speaking" || state === "listening") && (
|
||||
<div className="flex items-center gap-0.5 ml-1">
|
||||
{[0, 1, 2].map((i) => (
|
||||
<div
|
||||
className={`w-0.5 rounded-full ${config.pulseColor}`}
|
||||
key={i}
|
||||
style={{
|
||||
height: "12px",
|
||||
animation: `wave 1s ease-in-out ${i * 0.1}s infinite`,
|
||||
}}
|
||||
/>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Spinner for thinking */}
|
||||
{state === "thinking" && (
|
||||
<div className="ml-1">
|
||||
<div
|
||||
className={`w-3 h-3 border-2 border-t-transparent rounded-full ${config.iconColor} animate-spin`}
|
||||
style={{ borderColor: `currentColor transparent transparent transparent` }}
|
||||
/>
|
||||
</div>
|
||||
)}
|
||||
|
||||
<style>{`
|
||||
@keyframes wave {
|
||||
0%, 100% {
|
||||
height: 8px;
|
||||
}
|
||||
50% {
|
||||
height: 16px;
|
||||
}
|
||||
}
|
||||
`}</style>
|
||||
</div>
|
||||
)
|
||||
})
|
||||
|
||||
ConversationStatusIndicator.displayName = "ConversationStatusIndicator"
|
||||
@@ -0,0 +1,128 @@
|
||||
import { BooleanRequest, EmptyRequest } from "@shared/proto/cline/common"
|
||||
import { Mic, MicOff } from "lucide-react"
|
||||
import { useCallback, useEffect, useState } from "react"
|
||||
import { useExtensionState } from "@/context/ExtensionStateContext"
|
||||
import { TtsServiceClient, UiServiceClient } from "@/services/grpc-client"
|
||||
|
||||
interface DiscussModeToggleProps {
|
||||
className?: string
|
||||
}
|
||||
|
||||
/**
|
||||
* Toggle button to enable/disable Discuss Mode in Plan Mode.
|
||||
* When enabled, Cline's responses are spoken via TTS and conversations
|
||||
* can flow naturally with voice input/output.
|
||||
*/
|
||||
export function DiscussModeToggle({ className = "" }: DiscussModeToggleProps) {
|
||||
const { discussModeEnabled, mode, discussModeSettings } = useExtensionState()
|
||||
const [isToggling, setIsToggling] = useState(false)
|
||||
const [isConfigured, setIsConfigured] = useState(false)
|
||||
|
||||
// Only show in Plan Mode
|
||||
const isPlanMode = mode === "plan"
|
||||
|
||||
// Check if API key is configured on mount and when settings change
|
||||
useEffect(() => {
|
||||
const checkConfiguration = async () => {
|
||||
try {
|
||||
const response = await TtsServiceClient.CheckApiKeyConfigured(EmptyRequest.create())
|
||||
const hasVoice = !!discussModeSettings?.selectedVoice
|
||||
setIsConfigured(response.isValid && hasVoice)
|
||||
} catch (error) {
|
||||
console.error("[DiscussModeToggle] Failed to check TTS configuration:", error)
|
||||
setIsConfigured(false)
|
||||
}
|
||||
}
|
||||
checkConfiguration()
|
||||
}, [discussModeSettings])
|
||||
|
||||
const handleToggle = useCallback(async () => {
|
||||
if (isToggling || !isPlanMode) return
|
||||
|
||||
// If not configured, show settings hint
|
||||
if (!isConfigured) {
|
||||
// TODO: Show tooltip or open settings panel
|
||||
console.warn("Discuss Mode requires ElevenLabs API key configuration")
|
||||
return
|
||||
}
|
||||
|
||||
setIsToggling(true)
|
||||
try {
|
||||
// Toggle discuss mode via gRPC
|
||||
await UiServiceClient.setDiscussModeEnabled(BooleanRequest.create({ value: !discussModeEnabled }))
|
||||
} catch (error) {
|
||||
console.error("Error toggling Discuss Mode:", error)
|
||||
} finally {
|
||||
setIsToggling(false)
|
||||
}
|
||||
}, [discussModeEnabled, isPlanMode, isConfigured, isToggling])
|
||||
|
||||
// Don't render in Act Mode
|
||||
if (!isPlanMode) {
|
||||
return null
|
||||
}
|
||||
|
||||
const isEnabled = discussModeEnabled && isConfigured
|
||||
|
||||
return (
|
||||
<button
|
||||
className={`
|
||||
group relative flex items-center gap-2 px-3 py-2 rounded-lg
|
||||
transition-all duration-200 ease-in-out
|
||||
${
|
||||
isEnabled
|
||||
? "bg-vscode-button-background text-vscode-button-foreground hover:bg-vscode-button-hoverBackground"
|
||||
: "bg-vscode-button-secondaryBackground text-vscode-button-secondaryForeground hover:bg-vscode-button-secondaryHoverBackground"
|
||||
}
|
||||
${!isConfigured ? "opacity-50 cursor-not-allowed" : "cursor-pointer"}
|
||||
${isToggling ? "opacity-70" : ""}
|
||||
${className}
|
||||
`}
|
||||
disabled={isToggling || !isConfigured}
|
||||
onClick={handleToggle}
|
||||
title={
|
||||
!isConfigured
|
||||
? "Configure ElevenLabs API key in settings to enable Discuss Mode"
|
||||
: isEnabled
|
||||
? "Disable Discuss Mode (voice conversations)"
|
||||
: "Enable Discuss Mode (voice conversations)"
|
||||
}>
|
||||
{/* Icon */}
|
||||
<div
|
||||
className={`
|
||||
transition-transform duration-200
|
||||
${isEnabled ? "scale-110" : "scale-100"}
|
||||
`}>
|
||||
{isEnabled ? <Mic className="w-4 h-4" strokeWidth={2} /> : <MicOff className="w-4 h-4" strokeWidth={2} />}
|
||||
</div>
|
||||
|
||||
{/* Label */}
|
||||
<span className="text-sm font-medium">{isEnabled ? "Discussing" : "Discuss Mode"}</span>
|
||||
|
||||
{/* Status indicator */}
|
||||
{isEnabled && (
|
||||
<div className="relative">
|
||||
<div className="w-2 h-2 bg-green-500 rounded-full animate-pulse" />
|
||||
<div className="absolute inset-0 w-2 h-2 bg-green-500 rounded-full opacity-50 animate-ping" />
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Hover tooltip for unconfigured state */}
|
||||
{!isConfigured && (
|
||||
<div
|
||||
className="
|
||||
invisible group-hover:visible
|
||||
absolute top-full left-0 mt-2 p-2
|
||||
bg-vscode-notifications-background
|
||||
border border-vscode-notifications-border
|
||||
rounded shadow-lg z-50 w-64
|
||||
text-xs text-vscode-notifications-foreground
|
||||
">
|
||||
⚙️ Configure ElevenLabs API key in Settings → Voice to enable Discuss Mode
|
||||
</div>
|
||||
)}
|
||||
</button>
|
||||
)
|
||||
}
|
||||
|
||||
export default DiscussModeToggle
|
||||
@@ -0,0 +1,81 @@
|
||||
import { CheckCircle2, MessageCircle, PlayCircle } from "lucide-react"
|
||||
import { memo } from "react"
|
||||
|
||||
interface PlanCompletionCardProps {
|
||||
planSummary?: string
|
||||
onSwitchToActMode?: () => void
|
||||
onContinueDiscussing?: () => void
|
||||
className?: string
|
||||
}
|
||||
|
||||
/**
|
||||
* PlanCompletionCard displays when Cline has completed planning and is ready
|
||||
* to switch to Act Mode for implementation. Shows plan summary and action buttons.
|
||||
*/
|
||||
export const PlanCompletionCard = memo(
|
||||
({ planSummary, onSwitchToActMode, onContinueDiscussing, className = "" }: PlanCompletionCardProps) => {
|
||||
const handleSwitchToActMode = () => {
|
||||
// Mode switching will be handled by parent component via callback
|
||||
onSwitchToActMode?.()
|
||||
}
|
||||
|
||||
const handleContinueDiscussing = () => {
|
||||
onContinueDiscussing?.()
|
||||
}
|
||||
|
||||
return (
|
||||
<div
|
||||
className={`bg-gradient-to-r from-green-500/10 to-blue-500/10 border border-green-500/20 rounded-lg p-4 ${className}`}>
|
||||
{/* Header */}
|
||||
<div className="flex items-center gap-2 mb-3">
|
||||
<CheckCircle2 className="w-5 h-5 text-green-500" strokeWidth={2} />
|
||||
<h3 className="text-base font-semibold text-(--vscode-foreground)">Plan Complete!</h3>
|
||||
</div>
|
||||
|
||||
{/* Plan Summary */}
|
||||
{planSummary && (
|
||||
<div className="mb-4">
|
||||
<p className="text-sm text-(--vscode-descriptionForeground) leading-relaxed whitespace-pre-wrap">
|
||||
{planSummary}
|
||||
</p>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Description */}
|
||||
<p className="text-sm text-(--vscode-descriptionForeground) mb-4">
|
||||
The plan is ready for implementation. You can switch to Act Mode to begin, or continue discussing to refine
|
||||
the plan further.
|
||||
</p>
|
||||
|
||||
{/* Action Buttons */}
|
||||
<div className="flex flex-col sm:flex-row gap-2">
|
||||
{/* Primary Action - Switch to Act Mode */}
|
||||
<button
|
||||
className="flex items-center justify-center gap-2 px-4 py-2 bg-green-600 hover:bg-green-700 text-white rounded-md transition-colors font-medium text-sm flex-1"
|
||||
onClick={handleSwitchToActMode}>
|
||||
<PlayCircle className="w-4 h-4" />
|
||||
<span>Switch to Act Mode</span>
|
||||
</button>
|
||||
|
||||
{/* Secondary Action - Continue Discussing */}
|
||||
<button
|
||||
className="flex items-center justify-center gap-2 px-4 py-2 bg-(--vscode-button-secondaryBackground) hover:bg-(--vscode-button-secondaryHoverBackground) text-(--vscode-button-secondaryForeground) rounded-md transition-colors font-medium text-sm flex-1"
|
||||
onClick={handleContinueDiscussing}>
|
||||
<MessageCircle className="w-4 h-4" />
|
||||
<span>Continue Discussing</span>
|
||||
</button>
|
||||
</div>
|
||||
|
||||
{/* Helper Text */}
|
||||
<div className="mt-3 pt-3 border-t border-(--vscode-widget-border)">
|
||||
<p className="text-xs text-(--vscode-descriptionForeground)">
|
||||
💡 <strong>Tip:</strong> In Act Mode, Cline will silently implement the plan using tools. Discuss Mode
|
||||
will automatically disable.
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
)
|
||||
},
|
||||
)
|
||||
|
||||
PlanCompletionCard.displayName = "PlanCompletionCard"
|
||||
@@ -6,6 +6,7 @@ import {
|
||||
FlaskConical,
|
||||
Info,
|
||||
type LucideIcon,
|
||||
Mic,
|
||||
SlidersHorizontal,
|
||||
SquareMousePointer,
|
||||
SquareTerminal,
|
||||
@@ -27,6 +28,7 @@ import DebugSection from "./sections/DebugSection"
|
||||
import FeatureSettingsSection from "./sections/FeatureSettingsSection"
|
||||
import GeneralSettingsSection from "./sections/GeneralSettingsSection"
|
||||
import TerminalSettingsSection from "./sections/TerminalSettingsSection"
|
||||
import VoiceSettingsSection from "./sections/VoiceSettingsSection"
|
||||
|
||||
const IS_DEV = process.env.IS_DEV
|
||||
|
||||
@@ -76,6 +78,13 @@ export const SETTINGS_TABS: SettingsTab[] = [
|
||||
headerText: "General Settings",
|
||||
icon: Wrench,
|
||||
},
|
||||
{
|
||||
id: "voice",
|
||||
name: "Voice",
|
||||
tooltipText: "Voice Settings",
|
||||
headerText: "Voice Settings",
|
||||
icon: Mic,
|
||||
},
|
||||
{
|
||||
id: "about",
|
||||
name: "About",
|
||||
@@ -125,6 +134,7 @@ const SettingsView = ({ onDone, targetSection }: SettingsViewProps) => {
|
||||
features: FeatureSettingsSection,
|
||||
browser: BrowserSettingsSection,
|
||||
terminal: TerminalSettingsSection,
|
||||
voice: VoiceSettingsSection,
|
||||
about: AboutSection,
|
||||
debug: DebugSection,
|
||||
}),
|
||||
|
||||
@@ -0,0 +1,435 @@
|
||||
import { EmptyRequest } from "@shared/proto/cline/common"
|
||||
import { ValidateApiKeyRequest } from "@shared/proto/cline/tts"
|
||||
import { DiscussVoiceSettingsRequest } from "@shared/proto/cline/ui"
|
||||
import { VSCodeButton, VSCodeCheckbox, VSCodeTextField } from "@vscode/webview-ui-toolkit/react"
|
||||
import { Eye, EyeOff, Volume2 } from "lucide-react"
|
||||
import { useCallback, useEffect, useState } from "react"
|
||||
import { useExtensionState } from "@/context/ExtensionStateContext"
|
||||
import { TtsServiceClient, UiServiceClient } from "@/services/grpc-client"
|
||||
import Section from "../Section"
|
||||
|
||||
interface VoiceSettingsSectionProps {
|
||||
renderSectionHeader: (tabId: string) => JSX.Element | null
|
||||
}
|
||||
|
||||
import type { Voice as TtsVoice } from "@shared/proto/cline/tts"
|
||||
|
||||
const VoiceSettingsSection = ({ renderSectionHeader }: VoiceSettingsSectionProps) => {
|
||||
const { discussModeSettings } = useExtensionState()
|
||||
|
||||
const [apiKey, setApiKey] = useState("")
|
||||
const [showApiKey, setShowApiKey] = useState(false)
|
||||
const [isValidating, setIsValidating] = useState(false)
|
||||
const [validationError, setValidationError] = useState<string | null>(null)
|
||||
const [isValid, setIsValid] = useState(false)
|
||||
|
||||
const [voices, setVoices] = useState<TtsVoice[]>([])
|
||||
const [isLoadingVoices, setIsLoadingVoices] = useState(false)
|
||||
const [voicesError, setVoicesError] = useState<string | null>(null)
|
||||
|
||||
const [selectedVoice, setSelectedVoice] = useState(discussModeSettings?.selectedVoice || "")
|
||||
const [speechSpeed, setSpeechSpeed] = useState(discussModeSettings?.speechSpeed || 1.0)
|
||||
const [autoSpeak, setAutoSpeak] = useState(discussModeSettings?.autoSpeak || false)
|
||||
const [autoListen, setAutoListen] = useState(discussModeSettings?.autoListen || false)
|
||||
|
||||
// Sync with global settings whenever they change
|
||||
useEffect(() => {
|
||||
if (discussModeSettings) {
|
||||
setSelectedVoice(discussModeSettings.selectedVoice || "")
|
||||
setSpeechSpeed(discussModeSettings.speechSpeed || 1.0)
|
||||
setAutoSpeak(discussModeSettings.autoSpeak || false)
|
||||
setAutoListen(discussModeSettings.autoListen || false)
|
||||
}
|
||||
}, [discussModeSettings])
|
||||
|
||||
// Load API key and voices on mount
|
||||
useEffect(() => {
|
||||
const loadInitialData = async () => {
|
||||
// Check if API key is already configured
|
||||
try {
|
||||
const response = await TtsServiceClient.CheckApiKeyConfigured(EmptyRequest.create())
|
||||
|
||||
if (response.isValid) {
|
||||
// API key exists and is valid
|
||||
setIsValid(true)
|
||||
setValidationError(null)
|
||||
// Don't show the actual key for security, but indicate it exists
|
||||
setApiKey("••••••••••••••••")
|
||||
// Load voices automatically
|
||||
await loadVoices()
|
||||
}
|
||||
} catch (error) {
|
||||
console.error("Failed to check API key:", error)
|
||||
}
|
||||
}
|
||||
loadInitialData()
|
||||
}, [])
|
||||
|
||||
const validateApiKey = async (key: string) => {
|
||||
if (!key.trim()) {
|
||||
setValidationError("API key is required")
|
||||
setIsValid(false)
|
||||
return
|
||||
}
|
||||
|
||||
setIsValidating(true)
|
||||
setValidationError(null)
|
||||
|
||||
try {
|
||||
const response = await TtsServiceClient.ValidateApiKey(ValidateApiKeyRequest.create({ apiKey: key }))
|
||||
|
||||
if (response.isValid) {
|
||||
setIsValid(true)
|
||||
setValidationError(null)
|
||||
// Note: API key will be saved by the backend handler
|
||||
// Load voices
|
||||
await loadVoices()
|
||||
} else {
|
||||
setIsValid(false)
|
||||
setValidationError(response.error || "Invalid API key")
|
||||
}
|
||||
} catch (error) {
|
||||
setIsValid(false)
|
||||
setValidationError("Failed to validate API key: " + (error as Error).message)
|
||||
} finally {
|
||||
setIsValidating(false)
|
||||
}
|
||||
}
|
||||
|
||||
const loadVoices = async () => {
|
||||
setIsLoadingVoices(true)
|
||||
setVoicesError(null)
|
||||
|
||||
try {
|
||||
const response = await TtsServiceClient.GetAvailableVoices(EmptyRequest.create())
|
||||
|
||||
if (response.error) {
|
||||
setVoicesError(response.error)
|
||||
setVoices([])
|
||||
} else {
|
||||
setVoices(response.voices)
|
||||
|
||||
// Auto-select "Liam" voice if available and no voice is currently selected
|
||||
if (response.voices.length > 0 && !selectedVoice) {
|
||||
// Try to find Liam voice
|
||||
const liamVoice = response.voices.find((v) => v.name.toLowerCase().includes("liam"))
|
||||
|
||||
if (liamVoice) {
|
||||
// Found Liam, select it automatically
|
||||
setSelectedVoice(liamVoice.id)
|
||||
// Save to state
|
||||
await UiServiceClient.updateDiscussVoiceSettings(
|
||||
DiscussVoiceSettingsRequest.create({ selectedVoice: liamVoice.id }),
|
||||
)
|
||||
console.log("Auto-selected Liam voice:", liamVoice.name)
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
setVoicesError("Failed to load voices: " + (error as Error).message)
|
||||
setVoices([])
|
||||
} finally {
|
||||
setIsLoadingVoices(false)
|
||||
}
|
||||
}
|
||||
|
||||
const handleApiKeyChange = (e: any) => {
|
||||
const value = e.target.value
|
||||
setApiKey(value)
|
||||
setIsValid(false)
|
||||
setValidationError(null)
|
||||
}
|
||||
|
||||
const handleApiKeyBlur = async () => {
|
||||
if (apiKey.trim()) {
|
||||
await validateApiKey(apiKey)
|
||||
}
|
||||
}
|
||||
|
||||
const handleVoiceChange = useCallback(async (e: any) => {
|
||||
const voice = e.target.value
|
||||
setSelectedVoice(voice)
|
||||
|
||||
// Save to state via gRPC
|
||||
await UiServiceClient.updateDiscussVoiceSettings(DiscussVoiceSettingsRequest.create({ selectedVoice: voice }))
|
||||
}, [])
|
||||
|
||||
const handleSpeedChange = useCallback(async (e: any) => {
|
||||
const speed = parseFloat(e.target.value)
|
||||
setSpeechSpeed(speed)
|
||||
|
||||
// Save to state via gRPC
|
||||
await UiServiceClient.updateDiscussVoiceSettings(DiscussVoiceSettingsRequest.create({ speechSpeed: speed }))
|
||||
}, [])
|
||||
|
||||
const handleAutoSpeakChange = useCallback(async (e: any) => {
|
||||
const checked = e.target.checked === true
|
||||
setAutoSpeak(checked)
|
||||
|
||||
// Save to state via gRPC
|
||||
await UiServiceClient.updateDiscussVoiceSettings(DiscussVoiceSettingsRequest.create({ autoSpeak: checked }))
|
||||
}, [])
|
||||
|
||||
const handleAutoListenChange = useCallback(async (e: any) => {
|
||||
const checked = e.target.checked === true
|
||||
setAutoListen(checked)
|
||||
|
||||
// Save to state via gRPC
|
||||
await UiServiceClient.updateDiscussVoiceSettings(DiscussVoiceSettingsRequest.create({ autoListen: checked }))
|
||||
}, [])
|
||||
|
||||
const handleTestVoice = async () => {
|
||||
if (!selectedVoice) {
|
||||
console.error("No voice selected")
|
||||
alert("Please select a voice first")
|
||||
return
|
||||
}
|
||||
|
||||
try {
|
||||
console.log("Testing voice:", selectedVoice, "at speed:", speechSpeed)
|
||||
|
||||
// Synthesize a test phrase
|
||||
const testText = "Hello! This is a test of the text-to-speech voice."
|
||||
console.log("Requesting TTS for:", testText)
|
||||
|
||||
const response = await TtsServiceClient.SynthesizeSpeech({
|
||||
text: testText,
|
||||
voiceId: selectedVoice,
|
||||
speed: speechSpeed,
|
||||
})
|
||||
|
||||
console.log("TTS Response:", {
|
||||
audioDataLength: response.audioData.length,
|
||||
contentType: response.contentType,
|
||||
error: response.error,
|
||||
})
|
||||
|
||||
if (response.error) {
|
||||
console.error("TTS Error:", response.error)
|
||||
alert("Failed to test voice: " + response.error)
|
||||
return
|
||||
}
|
||||
|
||||
if (!response.audioData || response.audioData.length === 0) {
|
||||
console.error("Empty audio data received")
|
||||
alert("Failed to test voice: No audio data received")
|
||||
return
|
||||
}
|
||||
|
||||
// Play the audio using a DOM audio element (works better in VSCode webview)
|
||||
console.log("Converting audio data...")
|
||||
const audioArray = new Uint8Array(response.audioData)
|
||||
console.log("Audio array length:", audioArray.length)
|
||||
|
||||
const blob = new Blob([audioArray], { type: response.contentType || "audio/mpeg" })
|
||||
console.log("Blob created:", blob.size, "bytes, type:", blob.type)
|
||||
|
||||
const url = URL.createObjectURL(blob)
|
||||
console.log("Object URL created:", url)
|
||||
|
||||
// Create audio element in DOM (VSCode webview friendly)
|
||||
const audioElement = document.createElement("audio")
|
||||
audioElement.src = url
|
||||
audioElement.preload = "auto"
|
||||
|
||||
// Add to DOM temporarily
|
||||
audioElement.style.display = "none"
|
||||
document.body.appendChild(audioElement)
|
||||
|
||||
audioElement.onerror = (e) => {
|
||||
console.error("Audio playback error:", e, audioElement.error)
|
||||
alert("Failed to play audio: " + (audioElement.error?.message || "Unknown error"))
|
||||
URL.revokeObjectURL(url)
|
||||
document.body.removeChild(audioElement)
|
||||
}
|
||||
|
||||
audioElement.onended = () => {
|
||||
console.log("Audio playback completed")
|
||||
URL.revokeObjectURL(url)
|
||||
document.body.removeChild(audioElement)
|
||||
}
|
||||
|
||||
audioElement.onloadeddata = () => {
|
||||
console.log("Audio data loaded, duration:", audioElement.duration)
|
||||
}
|
||||
|
||||
audioElement.oncanplay = () => {
|
||||
console.log("Audio ready to play")
|
||||
}
|
||||
|
||||
console.log("Starting audio playback...")
|
||||
try {
|
||||
await audioElement.play()
|
||||
console.log("Audio.play() called successfully")
|
||||
} catch (playError) {
|
||||
console.error("Play error:", playError)
|
||||
alert(
|
||||
"Audio playback blocked: " +
|
||||
(playError as Error).message +
|
||||
"\n\nTip: Try clicking in the window first, or check your browser/VSCode audio settings.",
|
||||
)
|
||||
URL.revokeObjectURL(url)
|
||||
document.body.removeChild(audioElement)
|
||||
}
|
||||
} catch (error) {
|
||||
console.error("Test voice error:", error)
|
||||
alert("Failed to test voice: " + (error as Error).message)
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<div>
|
||||
{renderSectionHeader("voice")}
|
||||
<Section>
|
||||
<div className="mb-[5px]">
|
||||
<h4 className="text-sm font-semibold mb-2">Text-to-Speech Configuration</h4>
|
||||
<p className="text-sm text-description mb-4">
|
||||
Configure voice output for Discuss Mode. Cline will speak responses during interactive planning
|
||||
conversations.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
{/* API Key Input */}
|
||||
<div className="mb-[5px]">
|
||||
<label className="text-sm font-medium mb-2 block">ElevenLabs API Key</label>
|
||||
<div className="flex gap-2 items-start">
|
||||
<div className="flex-1">
|
||||
<VSCodeTextField
|
||||
className="w-full"
|
||||
onBlur={handleApiKeyBlur}
|
||||
onChange={handleApiKeyChange}
|
||||
placeholder="Enter your ElevenLabs API key"
|
||||
type={showApiKey ? "text" : "password"}
|
||||
value={apiKey}
|
||||
/>
|
||||
</div>
|
||||
<VSCodeButton
|
||||
appearance="icon"
|
||||
onClick={() => setShowApiKey(!showApiKey)}
|
||||
title={showApiKey ? "Hide API key" : "Show API key"}>
|
||||
{showApiKey ? <EyeOff className="w-4 h-4" /> : <Eye className="w-4 h-4" />}
|
||||
</VSCodeButton>
|
||||
<VSCodeButton disabled={isValidating || !apiKey.trim()} onClick={() => validateApiKey(apiKey)}>
|
||||
{isValidating ? "Validating..." : "Validate"}
|
||||
</VSCodeButton>
|
||||
</div>
|
||||
|
||||
{validationError && <p className="text-sm mt-2 text-red-500">{validationError}</p>}
|
||||
|
||||
{isValid && <p className="text-sm mt-2 text-green-500">✓ API key is valid</p>}
|
||||
|
||||
<p className="text-sm mt-2 text-description">
|
||||
Get your free API key from{" "}
|
||||
<a className="text-link underline" href="https://elevenlabs.io" rel="noopener noreferrer" target="_blank">
|
||||
elevenlabs.io
|
||||
</a>
|
||||
</p>
|
||||
</div>
|
||||
|
||||
{/* Voice Selection */}
|
||||
{isValid && (
|
||||
<div className="mb-[5px]">
|
||||
<label className="text-sm font-medium mb-2 block">Voice Selection</label>
|
||||
<div className="flex gap-2 items-start">
|
||||
<select
|
||||
className="flex-1 px-2 py-1 bg-input text-foreground border border-input-border rounded"
|
||||
disabled={isLoadingVoices}
|
||||
onChange={handleVoiceChange}
|
||||
value={selectedVoice}>
|
||||
<option value="">Select a voice...</option>
|
||||
{voices.map((voice) => (
|
||||
<option key={voice.id} value={voice.id}>
|
||||
{voice.name}
|
||||
</option>
|
||||
))}
|
||||
</select>
|
||||
<VSCodeButton disabled={!selectedVoice || isLoadingVoices} onClick={handleTestVoice}>
|
||||
<Volume2 className="w-4 h-4 mr-1" />
|
||||
Test
|
||||
</VSCodeButton>
|
||||
</div>
|
||||
|
||||
{isLoadingVoices && <p className="text-sm mt-2 text-description">Loading voices...</p>}
|
||||
|
||||
{voicesError && <p className="text-sm mt-2 text-red-500">{voicesError}</p>}
|
||||
|
||||
{selectedVoice && voices.length > 0 && (
|
||||
<p className="text-sm mt-2 text-description">
|
||||
{voices.find((v) => v.id === selectedVoice)?.description || ""}
|
||||
</p>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Speech Speed */}
|
||||
{isValid && selectedVoice && (
|
||||
<div className="mb-[5px]">
|
||||
<label className="text-sm font-medium mb-2 block">Speech Speed: {speechSpeed.toFixed(1)}x</label>
|
||||
<input
|
||||
className="w-full"
|
||||
max="1.2"
|
||||
min="0.7"
|
||||
onChange={handleSpeedChange}
|
||||
step="0.1"
|
||||
type="range"
|
||||
value={speechSpeed}
|
||||
/>
|
||||
<p className="text-sm mt-2 text-description">
|
||||
Adjust how fast Cline speaks (0.7x = slower, 1.2x = faster)
|
||||
</p>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Auto-Speak Toggle */}
|
||||
{isValid && selectedVoice && (
|
||||
<div className="mb-[5px]">
|
||||
<VSCodeCheckbox checked={autoSpeak} onChange={handleAutoSpeakChange}>
|
||||
Automatically speak responses
|
||||
</VSCodeCheckbox>
|
||||
<p className="text-sm mt-2 text-description">
|
||||
When enabled, Cline will automatically speak text responses in Plan Mode with Discuss Mode active
|
||||
</p>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Auto-Listen Toggle */}
|
||||
{isValid && selectedVoice && autoSpeak && (
|
||||
<div className="mb-[5px]">
|
||||
<VSCodeCheckbox checked={autoListen} onChange={handleAutoListenChange}>
|
||||
Auto-continue conversation
|
||||
</VSCodeCheckbox>
|
||||
<p className="text-sm mt-2 text-description">
|
||||
When enabled, voice input will automatically start after Cline finishes speaking, creating a natural
|
||||
conversation flow
|
||||
</p>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Voice Input Information */}
|
||||
{isValid && selectedVoice && (
|
||||
<div className="mb-[5px] mt-4 p-3 bg-[rgba(var(--vscode-textBlockQuote-background-rgb),0.5)] border-l-2 border-[var(--vscode-textBlockQuote-border)] rounded">
|
||||
<h5 className="text-sm font-semibold mb-2">Voice Input (Speech-to-Text)</h5>
|
||||
<p className="text-sm text-description mb-2">
|
||||
Voice input for Discuss Mode uses your ElevenLabs API key for transcription (Scribe v1 model).
|
||||
</p>
|
||||
<p className="text-sm text-description mb-2">
|
||||
When both Discuss Mode and Dictation are enabled, you'll see a microphone button in the chat input.
|
||||
Click it to record your voice, and your audio will be transcribed using ElevenLabs' Speech-to-Text
|
||||
API.
|
||||
</p>
|
||||
<p className="text-sm text-description mb-2">
|
||||
<strong>Features:</strong> Supports 99 languages, high accuracy transcription, speaker diarization,
|
||||
and word-level timestamps.
|
||||
</p>
|
||||
<p className="text-sm text-description">
|
||||
<strong>Note:</strong> Voice input requires macOS (for audio recording) and your ElevenLabs API key
|
||||
(already configured above).
|
||||
</p>
|
||||
</div>
|
||||
)}
|
||||
</Section>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
export default VoiceSettingsSection
|
||||
@@ -225,6 +225,15 @@ export const ExtensionStateContextProvider: React.FC<{
|
||||
lastDismissedCliBannerVersion: 0,
|
||||
subagentsEnabled: false,
|
||||
|
||||
// Discuss Mode defaults
|
||||
discussModeEnabled: false,
|
||||
discussModeSettings: {
|
||||
selectedVoice: undefined,
|
||||
speechSpeed: 1.0,
|
||||
autoSpeak: false,
|
||||
autoListen: false,
|
||||
},
|
||||
|
||||
// NEW: Add workspace information with defaults
|
||||
workspaceRoots: [],
|
||||
primaryRootIndex: 0,
|
||||
|
||||
@@ -8,10 +8,6 @@
|
||||
"DOM",
|
||||
"DOM.Iterable"
|
||||
],
|
||||
"types": [
|
||||
"vitest/globals",
|
||||
"@testing-library/jest-dom"
|
||||
],
|
||||
"module": "ESNext",
|
||||
"skipLibCheck": true,
|
||||
|
||||
|
||||
Reference in New Issue
Block a user