mirror of
https://github.com/bytebot-ai/bytebot.git
synced 2026-08-30 17:19:26 +08:00
Documentation updates
This commit is contained in:
Carl Atupem
@@ -0,0 +1,2 @@
|
||||
# Ignore formatting in docs folder
|
||||
/docs/**
|
||||
@@ -2,148 +2,307 @@
|
||||
|
||||
<img src="static/bytebot-logo.png" width="500" alt="Bytebot Logo">
|
||||
|
||||
[🌐 Website](https://bytebot.ai) • [📚 Docs](https://docs.bytebot.ai) • [💬 Discord](https://discord.com/invite/zcb5wA2t4u) • [𝕏 Twitter](https://x.com/bytebot_ai)
|
||||
# Bytebot – Self-Hosted AI Desktop Agent
|
||||
|
||||
## Bytebot – **The Easiest Way to Build Desktop Agents**
|
||||
**Automate any computer task with natural language**
|
||||
|
||||
[🌐 Website](https://bytebot.ai) • [📚 Docs](https://docs.bytebot.ai) • [💬 Discord](https://discord.com/invite/zcb5wA2t4u) • [𝕏 Twitter](https://x.com/bytebot_ai)
|
||||
|
||||
</div>
|
||||
|
||||
## ✨ Why Bytebot?
|
||||
## What is Bytebot?
|
||||
|
||||
Bytebot spins up a containerized Linux desktop with a task-driven agent ready for automation. Chat with it through the web UI or control it programmatically for scraping, CI tasks and remote work.
|
||||
Bytebot is a self-hosted AI desktop agent that transforms how you interact with computers. By combining powerful AI with a containerized Linux desktop, Bytebot can perform complex computer tasks. Think of it as your virtual employee that can actually use a computer – clicking, typing, browsing, and completing workflows just like a human would.
|
||||
|
||||
## Why Self-Host Bytebot?
|
||||
|
||||
- **Complete Privacy**: Your tasks and data never leave your infrastructure
|
||||
- **Full Control**: Customize the desktop environment and installed applications
|
||||
- **No Usage Limits**: Use your own LLM API keys without platform restrictions
|
||||
- **Secure Isolation**: Each desktop runs in its own container, isolated from your host
|
||||
|
||||
## Examples
|
||||
|
||||
|
||||
|
||||
https://github.com/user-attachments/assets/32a76e83-ea3a-4d5e-b34b-3b57f3604948
|
||||
|
||||
|
||||
|
||||
|
||||
https://github.com/user-attachments/assets/5f946df9-9161-4e7e-8262-9eda83ee7d22
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
### Prerequisites
|
||||
|
||||
## 🚀 Features
|
||||
- Docker ≥ 20.10
|
||||
- Docker Compose
|
||||
- Anthropic API key ([get one here](https://console.anthropic.com))
|
||||
|
||||
- 📦 **Containerized Desktop** – XFCE4 on Ubuntu 22.04 in a single Docker image
|
||||
- 🌍 **Access Anywhere** – VNC & browser‑based **noVNC** built‑in
|
||||
- 🛠️ **Unified API** – Script every click & keystroke with a clean REST interface
|
||||
- ⚙️ **Ready‑to‑Go Tools** – Firefox & essentials pre‑installed
|
||||
- 🤖 **Task-Driven Agent** – Manage tasks via REST or Chat UI and watch them run
|
||||
### Start Your Desktop Agent (2 minutes)
|
||||
|
||||
## 🧠 Agent System
|
||||
|
||||
Bytebot's agent stack is orchestrated with `docker-compose`. It starts:
|
||||
|
||||
- `bytebot-desktop` – the Linux desktop and automation daemon
|
||||
- `bytebot-agent` – NestJS service processing tasks with Anthropic's Claude
|
||||
- `bytebot-ui` – Next.js chat interface
|
||||
- `postgres` – stores tasks and conversation history
|
||||
|
||||
Open `http://localhost:9992` to give the agent a task and watch it work.
|
||||
|
||||
## 📖 Documentation
|
||||
|
||||
Dive deeper at [**docs.bytebot.ai**](https://docs.bytebot.ai).
|
||||
|
||||
## ⚡ Quick Start
|
||||
|
||||
### 🛠️ Prerequisites
|
||||
|
||||
- Docker ≥ 20.10
|
||||
|
||||
### 🐳 Run Bytebot
|
||||
|
||||
#### 🤖 Full Agent Stack (fastest way)
|
||||
1. **Clone and configure:**
|
||||
|
||||
```bash
|
||||
git clone https://github.com/bytebot-ai/bytebot.git
|
||||
cd bytebot
|
||||
echo "ANTHROPIC_API_KEY=your_api_key_here" > infrastructure/docker/.env
|
||||
|
||||
docker-compose -f infrastructure/docker/docker-compose.yml \
|
||||
--env-file infrastructure/docker/.env up -d # start desktop, agent & UI
|
||||
```
|
||||
Once running, open `http://localhost:9992` to chat with the agent.
|
||||
|
||||
Stop:
|
||||
2. **Start the agent stack:**
|
||||
|
||||
```bash
|
||||
docker-compose -f infrastructure/docker/docker-compose.yml \
|
||||
--env-file infrastructure/docker/.env down
|
||||
docker-compose -f infrastructure/docker/docker-compose.yml up -d
|
||||
```
|
||||
|
||||
#### Core Container
|
||||
3. **Open the chat interface:**
|
||||
|
||||
```
|
||||
http://localhost:9992
|
||||
```
|
||||
|
||||
That's it! Start chatting with your AI desktop agent. Watch it work in real-time through the embedded desktop viewer.
|
||||
|
||||
### Example Tasks You Can Delegate
|
||||
|
||||
- "Research the top 5 competitors for [product] and create a comparison spreadsheet"
|
||||
- "Fill out this web form with the data from my CSV file"
|
||||
- "Check my email and summarize important messages"
|
||||
- "Download all PDFs from this website and organize them by date"
|
||||
- "Monitor this webpage and alert me when the price drops below $50"
|
||||
|
||||
## 🏗️ Architecture Overview
|
||||
|
||||
Bytebot consists of four main components working together:
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Your Browser │
|
||||
│ http://localhost:9992 │
|
||||
└─────────────────────┬───────────────────────────────────────┘
|
||||
│
|
||||
┌─────────────────────▼───────────────────────────────────────┐
|
||||
│ Bytebot UI (Next.js) │
|
||||
│ • Task interface │
|
||||
│ • Desktop viewer (VNC) │
|
||||
│ • Task management │
|
||||
└─────────────────────┬───────────────────────────────────────┘
|
||||
│ WebSocket
|
||||
┌─────────────────────▼───────────────────────────────────────┐
|
||||
│ Bytebot Agent (NestJS) │
|
||||
│ • LLM integration │
|
||||
│ • Task orchestration │
|
||||
│ • Action planning │
|
||||
└─────────────────────┬───────────────────────────────────────┘
|
||||
│ REST API
|
||||
┌─────────────────────▼───────────────────────────────────────┐
|
||||
│ Bytebot Desktop (Ubuntu + XFCE) │
|
||||
│ • Full Linux desktop │
|
||||
│ • Browser, email, office apps │
|
||||
│ • Automation daemon (bytebotd) │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## 🛠️ Key Features
|
||||
|
||||
### For End Users
|
||||
|
||||
- **Natural Language Control**: Just describe what you want done
|
||||
- **Visual Feedback**: Watch the AI work in real-time
|
||||
- **Task History**: Review and replay previous automations
|
||||
- **Browser-Based**: No software to install on your machine
|
||||
|
||||
### For Developers
|
||||
|
||||
- **REST API**: Integrate desktop automation into your applications
|
||||
- **Extensible**: Add custom tools and applications to the desktop
|
||||
- **Scriptable**: Create complex workflows with the automation API
|
||||
- **Observable**: Full logging and debugging capabilities
|
||||
|
||||
### For IT Teams
|
||||
|
||||
- **Container-Based**: Easy deployment with Docker
|
||||
- **Resource Efficient**: Minimal overhead compared to VMs
|
||||
- **Network Isolated**: Secure by default with customizable access
|
||||
- **Scalable**: Run multiple instances for team use
|
||||
|
||||
## 📊 System Requirements
|
||||
|
||||
### Minimum (Single Agent)
|
||||
|
||||
- 2 CPU cores
|
||||
- 4GB RAM
|
||||
- 10GB storage
|
||||
- Docker & Docker Compose
|
||||
|
||||
### Recommended (Production)
|
||||
|
||||
- 4+ CPU cores
|
||||
- 8GB+ RAM
|
||||
- 20GB+ storage
|
||||
- Linux host OS for best performance
|
||||
|
||||
## 🔧 Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Create `infrastructure/docker/.env`:
|
||||
|
||||
```bash
|
||||
docker-compose -f infrastructure/docker/docker-compose.core.yml pull # pull latest remote image
|
||||
|
||||
docker-compose -f infrastructure/docker/docker-compose.core.yml up -d --no-build # start container
|
||||
# Required
|
||||
ANTHROPIC_API_KEY=sk-ant-...
|
||||
```
|
||||
|
||||
Build locally instead:
|
||||
### Desktop Customization
|
||||
|
||||
Add applications or configurations by extending the Dockerfile:
|
||||
|
||||
```dockerfile
|
||||
# infrastructure/docker/desktop/Dockerfile.custom
|
||||
FROM bytebot/desktop:latest
|
||||
|
||||
# Install additional software
|
||||
RUN apt-get update && apt-get install -y \
|
||||
libreoffice \
|
||||
gimp \
|
||||
your-custom-app
|
||||
|
||||
# Copy custom configs
|
||||
COPY configs/.config /home/bytebot/.config
|
||||
```
|
||||
|
||||
## 🔒 Security Considerations
|
||||
|
||||
- **API Key**: Keep your Anthropic API key secure and never commit it
|
||||
- **Network**: By default, services are only accessible from localhost
|
||||
- **VNC**: Change the default VNC password for production use
|
||||
- **Updates**: Regularly update the container images for security patches
|
||||
|
||||
## 🎯 Common Use Cases
|
||||
|
||||
### Personal Productivity
|
||||
|
||||
- Email management and responses
|
||||
- Calendar scheduling
|
||||
- Document organization
|
||||
- Web research and data collection
|
||||
|
||||
### Business Automation
|
||||
|
||||
- Form filling and data entry
|
||||
- Report generation
|
||||
- Competitive analysis
|
||||
- Customer support tasks
|
||||
|
||||
### Development & Testing
|
||||
|
||||
- UI testing automation
|
||||
- Cross-browser testing
|
||||
- API integration testing
|
||||
- Documentation screenshots
|
||||
|
||||
## 🚦 Managing Your Agent
|
||||
|
||||
### View Logs
|
||||
|
||||
```bash
|
||||
|
||||
docker-compose -f infrastructure/docker/docker-compose.core.yml up -d --build # build image and start container
|
||||
docker-compose -f infrastructure/docker/docker-compose.yml logs -f
|
||||
```
|
||||
|
||||
Stop:
|
||||
### Stop Services
|
||||
|
||||
```bash
|
||||
docker-compose -f infrastructure/docker/docker-compose.core.yml down
|
||||
docker-compose -f infrastructure/docker/docker-compose.yml down
|
||||
```
|
||||
|
||||
More details in the [**Quickstart Guide**](https://docs.bytebot.ai/quickstart).
|
||||
### Update to Latest Version
|
||||
|
||||
### 🔑 Connect
|
||||
```bash
|
||||
docker-compose -f infrastructure/docker/docker-compose.yml pull
|
||||
docker-compose -f infrastructure/docker/docker-compose.yml up -d
|
||||
```
|
||||
|
||||
| Interface | URL / Port | Notes |
|
||||
| ------------- | --------------------------- | ------------------------ |
|
||||
| 💬 Chat UI | `http://localhost:9992` | Agent UI |
|
||||
| 🤖 Agent API | `http://localhost:9991` | REST API |
|
||||
| 🌐 noVNC | `http://localhost:9990/vnc` | open in any browser |
|
||||
### Reset Everything
|
||||
|
||||
```bash
|
||||
docker-compose -f infrastructure/docker/docker-compose.yml down -v
|
||||
```
|
||||
|
||||
## 📚 Advanced Usage
|
||||
|
||||
### Programmatic Control
|
||||
|
||||
## 🤖 Automation API
|
||||
Control Bytebot via REST API:
|
||||
|
||||
Control Bytebot with a single endpoint. Read the [**REST reference**](https://docs.bytebot.ai/rest-api/computer-use). Supported actions:
|
||||
```python
|
||||
import requests
|
||||
|
||||
| 🎮 Action | Description |
|
||||
| ----------------- | -------------------------- |
|
||||
| `move_mouse` | Move cursor to coordinates |
|
||||
| `trace_mouse` | Draw a path |
|
||||
| `click_mouse` | Click (left/right/middle) |
|
||||
| `press_mouse` | Press / release button |
|
||||
| `drag_mouse` | Drag along path |
|
||||
| `scroll` | Scroll direction & amount |
|
||||
| `type_keys` | Type sequence of keys |
|
||||
| `press_keys` | Press / release keys |
|
||||
| `type_text` | Type a string |
|
||||
| `wait` | Wait milliseconds |
|
||||
| `screenshot` | Capture screen |
|
||||
| `cursor_position` | Return cursor position |
|
||||
# Create a task
|
||||
response = requests.post('http://localhost:9991/tasks', json={
|
||||
'description': 'Search for flights from NYC to London next month',
|
||||
})
|
||||
|
||||
_(See docs for parameter details.)_
|
||||
task_id = response.json()['id']
|
||||
|
||||
## 🙌 Contributing
|
||||
# Check task status
|
||||
status = requests.get(f'http://localhost:9991/tasks/{task_id}')
|
||||
print(status.json())
|
||||
```
|
||||
|
||||
1. 🍴 Fork & branch from `main`
|
||||
2. 💡 Commit small, focused changes
|
||||
3. 📩 Open a PR with details
|
||||
4. 🔍 Address review feedback
|
||||
5. 🎉 Merge & celebrate!
|
||||
### Direct Desktop Automation
|
||||
|
||||
## 💬 Support
|
||||
Use the computer control API for precise automation:
|
||||
|
||||
Questions or ideas? Join us on [**Discord**](https://discord.com/invite/zcb5wA2t4u).
|
||||
```javascript
|
||||
// Take screenshot
|
||||
POST http://localhost:9990/computer-use
|
||||
{
|
||||
"action": "screenshot"
|
||||
}
|
||||
|
||||
## 🙏 Acknowledgments
|
||||
// Click at coordinates
|
||||
POST http://localhost:9990/computer-use
|
||||
{
|
||||
"action": "click_mouse",
|
||||
"coordinate": [500, 300]
|
||||
}
|
||||
|
||||
Powered by [**nutjs**](https://github.com/nut-tree/nut.js) and inspired by Anthropic's [**computer‑use demo**](https://github.com/anthropics/anthropic-quickstarts/tree/main/computer-use-demo).
|
||||
// Type text
|
||||
POST http://localhost:9990/computer-use
|
||||
{
|
||||
"action": "type_text",
|
||||
"text": "Hello, Bytebot!"
|
||||
}
|
||||
```
|
||||
|
||||
## 📄 License
|
||||
## 🤝 Contributing
|
||||
|
||||
MIT © 2025 Tantl Labs, Inc.
|
||||
We welcome contributions! Whether it's bug fixes, new features, or documentation improvements:
|
||||
|
||||
1. Fork the repository
|
||||
2. Create your feature branch (`git checkout -b feature/amazing-feature`)
|
||||
3. Commit your changes (`git commit -m 'Add amazing feature'`)
|
||||
4. Push to the branch (`git push origin feature/amazing-feature`)
|
||||
5. Open a Pull Request
|
||||
|
||||
## 💬 Community & Support
|
||||
|
||||
- **Discord**: Join our [community server](https://discord.com/invite/zcb5wA2t4u) for help and discussions
|
||||
- **Documentation**: Comprehensive guides at [docs.bytebot.ai](https://docs.bytebot.ai)
|
||||
- **Issues**: Report bugs on [GitHub](https://github.com/bytebot-ai/bytebot/issues)
|
||||
|
||||
## 🙏 Acknowledgments
|
||||
|
||||
Built with amazing open source projects:
|
||||
|
||||
- [nutjs](https://github.com/nut-tree/nut.js) - Desktop automation framework
|
||||
- [Anthropic Claude](https://www.anthropic.com) - AI reasoning engine
|
||||
- [noVNC](https://novnc.com) - Browser-based VNC client
|
||||
- Inspired by Anthropic's [computer-use demo](https://github.com/anthropics/anthropic-quickstarts)
|
||||
|
||||
## 📄 License
|
||||
|
||||
MIT © 2025 Tantl Labs, Inc.
|
||||
|
||||
---
|
||||
|
||||
<div align="center">
|
||||
<strong>Ready to give your AI its own computer?</strong><br>
|
||||
Start with the Quick Start guide above or dive into the <a href="https://docs.bytebot.ai">full documentation</a>.
|
||||
</div>
|
||||
|
||||
@@ -1,17 +1,17 @@
|
||||
---
|
||||
title: 'Chat UI'
|
||||
description: 'Documentation for the Bytebot Chat UI'
|
||||
title: 'Task UI'
|
||||
description: 'Documentation for the Bytebot Task UI'
|
||||
---
|
||||
|
||||
## Bytebot Chat UI
|
||||
## Bytebot Task UI
|
||||
|
||||
The Bytebot Chat UI provides a web-based interface for interacting with the Bytebot agent system. It combines a chat interface with an embedded noVNC viewer, allowing you to communicate with the agent and watch it perform tasks on the desktop in real-time.
|
||||
The Bytebot Task UI provides a web-based interface for interacting with the Bytebot agent system. It combines a action feed with an embedded noVNC viewer, allowing you to watch it perform task on the desktop in real-time.
|
||||
|
||||
<img src="/static/chat-ui-overview.png" alt="Bytebot Chat UI Overview" className="w-full max-w-4xl" />
|
||||
<img src="/static/chat-ui-overview.png" alt="Bytebot Task Detail" className="w-full max-w-4xl" />
|
||||
|
||||
## Accessing the UI
|
||||
|
||||
When running the full Bytebot agent system, the Chat UI is available at:
|
||||
When running the full Bytebot agent system, the Task UI is available at:
|
||||
|
||||
```
|
||||
http://localhost:9992
|
||||
@@ -30,12 +30,11 @@ The task management panel allows you to:
|
||||
|
||||
<img src="/static/ui-task-management.png" alt="Task Management Panel" className="w-full max-w-4xl" />
|
||||
|
||||
### Chat Interface
|
||||
### Task Interface
|
||||
|
||||
The main chat interface provides:
|
||||
The main task interface provides:
|
||||
|
||||
- Conversation history with the agent
|
||||
- Message input for sending new instructions
|
||||
- Task history with the agent
|
||||
- Support for markdown formatting in messages
|
||||
- Automatic scrolling to new messages
|
||||
|
||||
@@ -45,7 +44,7 @@ The embedded noVNC viewer displays:
|
||||
|
||||
- Real-time view of the desktop environment
|
||||
- Visual feedback of agent actions
|
||||
- Option to expand to full-screen view
|
||||
- Option to expand to take over the desktop
|
||||
- Connection status indicator
|
||||
|
||||
## Features
|
||||
@@ -54,13 +53,12 @@ The embedded noVNC viewer displays:
|
||||
|
||||
To create a new task:
|
||||
|
||||
1. Click the "New Task" button in the task panel
|
||||
2. Enter a description for the task
|
||||
3. Click "Create Task"
|
||||
1. Enter a description for the task
|
||||
2. Click "Start Task" button (or press Enter)
|
||||
|
||||
### Conversation Controls
|
||||
|
||||
The chat interface supports:
|
||||
The task interface supports:
|
||||
|
||||
- Text messages with markdown formatting
|
||||
- Viewing image content in messages
|
||||
@@ -71,13 +69,12 @@ The chat interface supports:
|
||||
|
||||
While primarily for viewing, the desktop panel allows:
|
||||
|
||||
- Expanding to full-screen view
|
||||
- Viewing desktop screenshots taken by the agent
|
||||
- Taking over the desktop
|
||||
- Real-time monitoring of agent actions
|
||||
|
||||
## Message Types
|
||||
|
||||
The chat interface displays different types of messages based on Anthropic's content block structure:
|
||||
The task interface displays different types of messages based on Bytebot's content block structure:
|
||||
|
||||
- **User Messages**: Your instructions and queries
|
||||
- **Assistant Messages**: Responses from the agent, which may include:
|
||||
@@ -92,7 +89,7 @@ The message content structure follows this format:
|
||||
interface Message {
|
||||
id: string;
|
||||
content: MessageContentBlock[];
|
||||
role: MessageRole; // "USER" or "ASSISTANT"
|
||||
role: Role; // "USER" or "ASSISTANT"
|
||||
createdAt?: string;
|
||||
}
|
||||
|
||||
@@ -118,7 +115,7 @@ interface ImageContentBlock extends MessageContentBlock {
|
||||
|
||||
## Technical Details
|
||||
|
||||
The Bytebot Chat UI is built with:
|
||||
The Bytebot Task UI is built with:
|
||||
|
||||
- **Next.js**: React framework for the frontend
|
||||
- **Tailwind CSS**: For styling
|
||||
@@ -142,14 +139,12 @@ If the desktop viewer is not displaying:
|
||||
|
||||
1. Ensure the Bytebot container is running
|
||||
2. Check that the noVNC service is accessible at port 9990
|
||||
3. Try clicking the "Reconnect" button
|
||||
4. Verify that no other VNC client is connected exclusively
|
||||
|
||||
### Message Display Issues
|
||||
|
||||
If messages are not displaying correctly:
|
||||
|
||||
1. Check that the message content is properly formatted
|
||||
2. Ensure the agent service is processing tasks correctly
|
||||
2. Ensure the agent service is processing task correctly
|
||||
3. Check the browser console for any rendering errors
|
||||
4. Try refreshing the browser
|
||||
|
||||
+311
-145
@@ -1,11 +1,11 @@
|
||||
---
|
||||
title: "Agent System"
|
||||
description: "Understanding the Bytebot agent architecture and task management system"
|
||||
description: "The AI brain that powers your self-hosted desktop automation"
|
||||
---
|
||||
|
||||
## Agent System Overview
|
||||
## Overview
|
||||
|
||||
The Bytebot agent system extends the core desktop container with AI-driven automation capabilities. It's designed to execute tasks autonomously using a structured task management system, message-based interactions, and database persistence.
|
||||
The Bytebot Agent System transforms a simple desktop container into an intelligent, autonomous computer user. By combining Claude AI with structured task management, it can understand natural language requests and execute complex workflows just like a human would.
|
||||
|
||||
<img
|
||||
src="/images/agent-architecture.png"
|
||||
@@ -13,171 +13,337 @@ The Bytebot agent system extends the core desktop container with AI-driven autom
|
||||
className="w-full max-w-4xl"
|
||||
/>
|
||||
|
||||
## Core Components
|
||||
## How the AI Agent Works
|
||||
|
||||
### Agent Service
|
||||
### The Brain: Claude AI Integration
|
||||
|
||||
The agent service is the central processing unit of the Bytebot agent system. Built with NestJS, it:
|
||||
At the heart of Bytebot is Claude, Anthropic's advanced AI assistant. The agent:
|
||||
|
||||
- Processes tasks in a structured loop
|
||||
- Integrates with Anthropic's Claude for AI capabilities
|
||||
- Manages the task state and messages
|
||||
- Dispatches computer actions to the bytebotd service
|
||||
- Provides a REST API for task management
|
||||
1. **Understands Context**: Processes your natural language requests with full conversation history
|
||||
2. **Plans Actions**: Breaks down complex tasks into executable computer actions
|
||||
3. **Adapts in Real-time**: Adjusts its approach based on what it sees on screen
|
||||
4. **Learns from Feedback**: Improves task execution through conversation
|
||||
|
||||
### Conversation Flow
|
||||
|
||||
<Steps>
|
||||
<Step title="You Describe a Task">
|
||||
"Research competitors for my SaaS product and create a comparison table"
|
||||
</Step>
|
||||
<Step title="AI Plans the Approach">
|
||||
Claude understands the request and plans: open browser → search → visit sites → extract data → create document
|
||||
</Step>
|
||||
<Step title="Executes Actions">
|
||||
The agent controls the desktop: clicking, typing, taking screenshots, reading content
|
||||
</Step>
|
||||
<Step title="Provides Updates">
|
||||
Real-time status updates and asks for clarification when needed
|
||||
</Step>
|
||||
<Step title="Delivers Results">
|
||||
Completes the task and provides the output (files, screenshots, summaries)
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
## Task Management System
|
||||
|
||||
### Task Lifecycle
|
||||
|
||||
Tasks move through a structured lifecycle:
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[Created] --> B[Queued]
|
||||
B --> C[Running]
|
||||
C --> D[Needs Help]
|
||||
C --> E[Completed]
|
||||
C --> F[Failed]
|
||||
D --> C
|
||||
```
|
||||
|
||||
### Task Properties
|
||||
|
||||
Each task contains:
|
||||
|
||||
- **Description**: What needs to be done
|
||||
- **Priority**: Urgent, High, Medium, or Low
|
||||
- **Status**: Current state in the lifecycle
|
||||
- **Type**: Immediate or Scheduled
|
||||
- **History**: All messages and actions taken
|
||||
|
||||
### Smart Task Processing
|
||||
|
||||
The agent processes tasks intelligently:
|
||||
|
||||
1. **Priority Queue**: Urgent tasks run first
|
||||
2. **Error Recovery**: Automatically retries failed actions
|
||||
3. **Human in the Loop**: Asks for help when stuck
|
||||
4. **Context Preservation**: Maintains conversation history across sessions
|
||||
|
||||
## Real-world Capabilities
|
||||
|
||||
### What the Agent Can Do
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Web Automation" icon="globe">
|
||||
- Browse websites
|
||||
- Fill out forms
|
||||
- Extract data
|
||||
- Download files
|
||||
- Monitor changes
|
||||
</Card>
|
||||
<Card title="Document Work" icon="file">
|
||||
- Create documents
|
||||
- Edit spreadsheets
|
||||
- Generate reports
|
||||
- Organize files
|
||||
- Convert formats
|
||||
</Card>
|
||||
<Card title="Email & Communication" icon="envelope">
|
||||
- Read emails
|
||||
- Draft responses
|
||||
- Manage calendar
|
||||
- Schedule meetings
|
||||
- Send notifications
|
||||
</Card>
|
||||
<Card title="Data Processing" icon="database">
|
||||
- Extract from PDFs
|
||||
- Process CSV files
|
||||
- Create visualizations
|
||||
- Generate summaries
|
||||
- Transform data
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
### Example Use Cases
|
||||
|
||||
#### Research Assistant
|
||||
```
|
||||
User: "Find the top 5 project management tools and compare their pricing"
|
||||
|
||||
Agent Actions:
|
||||
1. Opens browser
|
||||
2. Searches for project management tools
|
||||
3. Visits each tool's website
|
||||
4. Extracts pricing information
|
||||
5. Creates comparison spreadsheet
|
||||
6. Takes screenshots of each pricing page
|
||||
```
|
||||
|
||||
#### Form Automation
|
||||
```
|
||||
User: "Fill out the vendor application form with data from our company profile"
|
||||
|
||||
Agent Actions:
|
||||
1. Opens the form URL
|
||||
2. Reads company profile document
|
||||
3. Maps data to form fields
|
||||
4. Fills out each section
|
||||
5. Uploads required documents
|
||||
6. Submits and saves confirmation
|
||||
```
|
||||
|
||||
#### Email Management
|
||||
```
|
||||
User: "Check my emails and create a summary of action items"
|
||||
|
||||
Agent Actions:
|
||||
1. Opens email client
|
||||
2. Reads unread messages
|
||||
3. Identifies action items
|
||||
4. Creates organized task list
|
||||
5. Drafts response templates
|
||||
6. Flags important messages
|
||||
```
|
||||
|
||||
## Technical Architecture
|
||||
|
||||
### Core Components
|
||||
|
||||
1. **NestJS Agent Service**
|
||||
- Manages task queue with BullMQ
|
||||
- Integrates with Anthropic API
|
||||
- Handles WebSocket connections
|
||||
- Coordinates with desktop API
|
||||
|
||||
2. **Message System**
|
||||
- Structured conversation format
|
||||
- Supports text and images
|
||||
- Maintains full context
|
||||
- Enables rich interactions
|
||||
|
||||
3. **Database Schema**
|
||||
```sql
|
||||
Tasks: id, description, status, priority, timestamps
|
||||
Messages: id, task_id, role, content, timestamps
|
||||
Summaries: id, task_id, content, parent_id
|
||||
```
|
||||
|
||||
4. **Computer Action Bridge**
|
||||
- Translates AI decisions to desktop actions
|
||||
- Handles screenshots and feedback
|
||||
- Manages action timing
|
||||
- Provides error handling
|
||||
|
||||
### API Endpoints
|
||||
|
||||
Key endpoints for programmatic control:
|
||||
|
||||
```typescript
|
||||
// Create a new task
|
||||
POST /tasks
|
||||
{
|
||||
"description": "Your task description",
|
||||
"priority": "HIGH",
|
||||
"type": "IMMEDIATE"
|
||||
}
|
||||
|
||||
// Get task status
|
||||
GET /tasks/:id
|
||||
|
||||
// Send a message
|
||||
POST /tasks/:id/messages
|
||||
{
|
||||
"content": "Additional instructions"
|
||||
}
|
||||
|
||||
// Get task history
|
||||
GET /tasks/:id/messages
|
||||
```
|
||||
|
||||
## Chat UI Features
|
||||
|
||||
The web interface provides:
|
||||
|
||||
### Real-time Interaction
|
||||
- Live chat with the AI agent
|
||||
- Instant status updates
|
||||
- Progress indicators
|
||||
- Error notifications
|
||||
|
||||
### Visual Feedback
|
||||
- Embedded desktop viewer
|
||||
- Screenshot history
|
||||
- Action replay
|
||||
- Task timeline
|
||||
|
||||
### Task Management
|
||||
- Create and prioritize tasks
|
||||
- View active and completed tasks
|
||||
- Export conversation logs
|
||||
- Manage task queues
|
||||
|
||||
Tasks are the primary unit of work in the Bytebot agent system:
|
||||
## Security & Privacy
|
||||
|
||||
```typescript
|
||||
interface Task {
|
||||
id: string;
|
||||
description: string;
|
||||
status: TaskStatus;
|
||||
priority: TaskPriority;
|
||||
type: TaskType;
|
||||
scheduledFor?: Date;
|
||||
createdAt: Date;
|
||||
updatedAt: Date;
|
||||
executedAt?: Date;
|
||||
completedAt?: Date;
|
||||
queuedAt?: Date;
|
||||
}
|
||||
### Data Isolation
|
||||
- All processing happens in your infrastructure
|
||||
- No data sent to external services (except Claude API)
|
||||
- Conversations stored locally
|
||||
- Complete audit trail
|
||||
|
||||
enum TaskType {
|
||||
IMMEDIATE,
|
||||
SCHEDULED,
|
||||
}
|
||||
### Access Control
|
||||
- Configurable authentication
|
||||
- API key management
|
||||
- Network isolation options
|
||||
- Role-based permissions (coming soon)
|
||||
|
||||
enum TaskStatus {
|
||||
PENDING,
|
||||
RUNNING,
|
||||
NEEDS_HELP,
|
||||
NEEDS_REVIEW,
|
||||
COMPLETED,
|
||||
CANCELLED,
|
||||
FAILED,
|
||||
}
|
||||
## Extending the Agent
|
||||
|
||||
enum TaskPriority {
|
||||
LOW,
|
||||
MEDIUM,
|
||||
HIGH,
|
||||
URGENT,
|
||||
}
|
||||
### Custom Tools
|
||||
Add specialized capabilities:
|
||||
```javascript
|
||||
// Register a custom tool
|
||||
agent.registerTool({
|
||||
name: 'database_query',
|
||||
description: 'Query internal database',
|
||||
execute: async (params) => {
|
||||
// Your implementation
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
The task lifecycle involves:
|
||||
|
||||
1. **Creation**: Tasks are created via the API, UI, or by the agent itself with a description
|
||||
2. **Queuing**: Tasks are queued for processing
|
||||
3. **Processing**: The agent processor handles tasks one at a time
|
||||
4. **Completion/Cancellation**: Tasks are marked as complete, cancelled, or other terminal states
|
||||
|
||||
### Message System
|
||||
|
||||
The agent communicates through a structured message system using Anthropic's content block format:
|
||||
|
||||
```typescript
|
||||
interface Message {
|
||||
id: string;
|
||||
content: MessageContentBlock[];
|
||||
role: Role;
|
||||
taskId: string;
|
||||
summaryId?: string;
|
||||
createdAt: Date;
|
||||
updatedAt: Date;
|
||||
}
|
||||
|
||||
enum Role {
|
||||
USER,
|
||||
ASSISTANT,
|
||||
}
|
||||
|
||||
interface MessageContentBlock {
|
||||
type: string;
|
||||
[key: string]: any;
|
||||
}
|
||||
|
||||
interface TextContentBlock extends MessageContentBlock {
|
||||
type: "text";
|
||||
text: string;
|
||||
}
|
||||
|
||||
interface ImageContentBlock extends MessageContentBlock {
|
||||
type: "image";
|
||||
source: {
|
||||
type: "base64";
|
||||
media_type: string;
|
||||
data: string;
|
||||
};
|
||||
}
|
||||
### Workflow Templates
|
||||
Create reusable task templates:
|
||||
```yaml
|
||||
name: "Daily Report"
|
||||
steps:
|
||||
- action: "screenshot"
|
||||
target: "dashboard"
|
||||
- action: "extract_data"
|
||||
format: "table"
|
||||
- action: "create_document"
|
||||
template: "daily_report"
|
||||
```
|
||||
|
||||
Messages are stored with their associated tasks and can be included in summaries for context retention.
|
||||
### Integration Points
|
||||
- Webhook notifications
|
||||
- External API calls
|
||||
- Custom AI prompts
|
||||
- Plugin system (coming soon)
|
||||
|
||||
### Database Structure
|
||||
## Performance Tuning
|
||||
|
||||
The agent system uses PostgreSQL for data persistence with a schema that includes:
|
||||
### Optimization Settings
|
||||
|
||||
- **Tasks**: Storing task metadata and status
|
||||
- **Messages**: Storing conversation history using a JSON structure for content blocks
|
||||
- **Summaries**: Storing context summaries for long-running tasks, with hierarchical relationships
|
||||
```env
|
||||
# Concurrent task limit
|
||||
MAX_CONCURRENT_TASKS=1
|
||||
|
||||
## Agent Processing Loop
|
||||
# Task timeout (ms)
|
||||
TASK_TIMEOUT=300000
|
||||
|
||||
The agent processing loop follows these steps:
|
||||
# Screenshot quality (0-100)
|
||||
SCREENSHOT_QUALITY=80
|
||||
|
||||
1. **Fetch Task**: Retrieve the task and its associated messages
|
||||
2. **Update Status**: Mark the task as in-progress
|
||||
3. **Process Messages**: Send messages to the AI for processing
|
||||
4. **Execute Actions**: Perform computer actions through the bytebotd API
|
||||
5. **Store Results**: Save responses and action results
|
||||
6. **Create Summaries**: Periodically summarize conversation context
|
||||
7. **Complete Task**: Mark task as complete when finished
|
||||
# Message history limit
|
||||
MAX_CONTEXT_MESSAGES=50
|
||||
```
|
||||
|
||||
## Computer Action Integration
|
||||
### Best Practices
|
||||
|
||||
The agent leverages the unified computer action API to perform actions on the desktop:
|
||||
|
||||
1. The AI identifies required actions based on the task and conversation
|
||||
2. The agent service sends computer action requests to the bytebotd daemon
|
||||
3. The bytebotd daemon executes actions on the desktop
|
||||
4. Results (including screenshots) are returned to the agent service
|
||||
5. The agent integrates these results into the conversation
|
||||
|
||||
## Web UI Integration
|
||||
|
||||
The NextJS-based web UI provides:
|
||||
|
||||
- A chat interface for user-agent interaction
|
||||
- Task management controls
|
||||
- Real-time desktop view via embedded noVNC
|
||||
- Task history and status views
|
||||
|
||||
## Security Considerations
|
||||
|
||||
When using the agent system, consider these security aspects:
|
||||
|
||||
- The agent has access to your desktop environment
|
||||
- API keys (like ANTHROPIC_API_KEY) are required and should be secured
|
||||
- Database persistence stores conversation history
|
||||
- Network security for the additional exposed ports (9991, 9992)
|
||||
|
||||
## Customization and Extension
|
||||
|
||||
The agent system can be extended in several ways:
|
||||
|
||||
- Custom tools integration via the NestJS API
|
||||
- UI customization through the NextJS frontend
|
||||
- Additional AI model integration
|
||||
- Workflow automation through the task system
|
||||
1. **Clear Instructions**: Be specific about desired outcomes
|
||||
2. **Break Down Complex Tasks**: Use multiple smaller tasks for better results
|
||||
3. **Provide Context**: Include relevant files or URLs
|
||||
4. **Monitor Progress**: Watch the desktop view for real-time feedback
|
||||
5. **Review Results**: Verify outputs meet requirements
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
Common issues and solutions:
|
||||
<AccordionGroup>
|
||||
<Accordion title="Agent not responding">
|
||||
- Check Anthropic API key is valid
|
||||
- Verify agent service is running
|
||||
- Review logs for errors
|
||||
- Ensure sufficient API credits
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Tasks failing frequently">
|
||||
- Increase task timeout
|
||||
- Check desktop responsiveness
|
||||
- Verify network connectivity
|
||||
- Review action timing
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Slow task execution">
|
||||
- Monitor system resources
|
||||
- Check network latency
|
||||
- Reduce screenshot frequency
|
||||
- Optimize Claude prompts
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
- **Agent not starting**: Check environment variables and database connection
|
||||
- **Task processing errors**: Check the agent logs for error messages
|
||||
- **UI connection issues**: Ensure all services are running and ports are accessible
|
||||
- **Computer action failures**: Verify the bytebotd service is running and accessible
|
||||
## Next Steps
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Quick Start" icon="rocket" href="/quickstart">
|
||||
Get your agent running
|
||||
</Card>
|
||||
<Card title="API Reference" icon="code" href="/api-reference/agent/tasks">
|
||||
Integrate with your apps
|
||||
</Card>
|
||||
<Card title="Use Cases" icon="lightbulb" href="#example-use-cases">
|
||||
See what's possible
|
||||
</Card>
|
||||
<Card title="Best Practices" icon="star" href="#best-practices">
|
||||
Optimize your workflows
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -1,11 +1,11 @@
|
||||
---
|
||||
title: "Architecture"
|
||||
description: "Overview of the Bytebot architecture and components"
|
||||
description: "How Bytebot's desktop agent works under the hood"
|
||||
---
|
||||
|
||||
## Bytebot Architecture
|
||||
## Overview
|
||||
|
||||
Bytebot is designed with a modular architecture that can be run as a standalone desktop container or as a full-featured agent system with a web UI.
|
||||
Bytebot is a self-hosted AI desktop agent built with a modular, containerized architecture. It combines a Linux desktop environment with Claude AI to create an autonomous computer user that can perform tasks through natural language instructions.
|
||||
|
||||
<img
|
||||
src="/images/agent-architecture.png"
|
||||
@@ -13,110 +13,204 @@ Bytebot is designed with a modular architecture that can be run as a standalone
|
||||
className="w-full max-w-4xl"
|
||||
/>
|
||||
|
||||
## Core Components
|
||||
## System Architecture
|
||||
|
||||
### Container Base
|
||||
The system consists of four main components that work together:
|
||||
|
||||
- **Ubuntu 22.04** serves as the base operating system
|
||||
- Provides a stable foundation for the desktop environment and tools
|
||||
### 1. Bytebot Desktop Container
|
||||
The foundation of the system - a containerized Linux desktop that provides:
|
||||
|
||||
### Desktop Environment
|
||||
- **Ubuntu 22.04 LTS** base for stability and compatibility
|
||||
- **XFCE4 Desktop** for a lightweight, responsive UI
|
||||
- **bytebotd Daemon** - The automation service built on nutjs that executes computer actions
|
||||
- **Pre-installed Applications**: Firefox ESR, Thunderbird, text editors, and development tools
|
||||
- **VNC & noVNC** for remote desktop access
|
||||
|
||||
- **XFCE4** desktop environment
|
||||
- Lightweight and customizable
|
||||
- Comes pre-configured with sensible defaults
|
||||
- Includes default user account: `bytebot` with sudo privileges
|
||||
**Key Features:**
|
||||
- Runs completely isolated from your host system
|
||||
- Consistent environment across different platforms
|
||||
- Can be customized with additional software
|
||||
- Accessible via REST API on port 9990
|
||||
|
||||
### Automation Daemon (bytebotd)
|
||||
### 2. AI Agent Service
|
||||
The brain of the system - orchestrates tasks using an LLM:
|
||||
|
||||
- **bytebotd daemon** is the core service that enables automation
|
||||
- Built on top of nutjs for desktop automation
|
||||
- Exposes a REST API for remote control
|
||||
- Provides unified endpoint for all computer actions
|
||||
- Accessible at `localhost:9990`
|
||||
- **NestJS Framework** for robust, scalable backend
|
||||
- **LLM Integration** via OpenAI API for understanding and planning
|
||||
- **WebSocket Support** for real-time updates
|
||||
- **Computer Use API Client** to control the desktop
|
||||
|
||||
### Browser and Tools
|
||||
**Responsibilities:**
|
||||
- Interprets natural language requests
|
||||
- Plans sequences of computer actions
|
||||
- Manages task state and progress
|
||||
- Handles errors and retries
|
||||
|
||||
- **Firefox** pre-installed and configured
|
||||
- Essential utilities for development and testing
|
||||
- Default applications for common tasks
|
||||
### 3. Web Task Interface
|
||||
The user interface for interacting with your AI agent:
|
||||
|
||||
### Remote Access
|
||||
- **Next.js Application** with TypeScript for type safety
|
||||
- **Embedded VNC Viewer** to watch the desktop in action
|
||||
- **Task Management** UI for tracking progress
|
||||
- **WebSocket Connections** for live updates
|
||||
|
||||
- **VNC server** for direct desktop access
|
||||
- **noVNC** for browser-based desktop access
|
||||
**Features:**
|
||||
- Intuitive task interface
|
||||
- Visual feedback of desktop actions
|
||||
- Task history and status
|
||||
- Export conversation logs
|
||||
|
||||
## Agent System Components
|
||||
### 4. PostgreSQL Database
|
||||
Persistent storage for the agent system:
|
||||
|
||||
When running the full Bytebot system using docker-compose, the following additional components are available:
|
||||
- **Tasks Table**: Stores task details, status, and metadata
|
||||
- **Messages Table**: Stores AI conversation history
|
||||
- **Prisma ORM** for type-safe database access
|
||||
|
||||
### Bytebot Agent
|
||||
## Data Flow
|
||||
|
||||
- **Agent service** that manages tasks and AI-driven automation
|
||||
- Built with NestJS for reliable API services
|
||||
- Implements a task processing system with queues via BullMQ
|
||||
- Integrates with Anthropic's Claude for AI capabilities
|
||||
- Accessible at `localhost:9991`
|
||||
### Task Execution Flow
|
||||
|
||||
### Databases
|
||||
<Steps>
|
||||
<Step title="User Input">
|
||||
User describes a task in natural language via the chat UI
|
||||
</Step>
|
||||
<Step title="Task Creation">
|
||||
Agent service creates a task record and adds it to the processing queue
|
||||
</Step>
|
||||
<Step title="AI Planning">
|
||||
Claude AI analyzes the task and generates a plan of computer actions
|
||||
</Step>
|
||||
<Step title="Action Execution">
|
||||
Agent sends computer actions to bytebotd daemon via REST API
|
||||
</Step>
|
||||
<Step title="Desktop Automation">
|
||||
bytebotd executes actions (mouse, keyboard, screenshots) on the desktop
|
||||
</Step>
|
||||
<Step title="Result Processing">
|
||||
Agent receives results, updates task status, and continues or completes
|
||||
</Step>
|
||||
<Step title="User Feedback">
|
||||
Results and status updates are sent back to the user in real-time
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
- **PostgreSQL database** for storing tasks, messages, and agent state
|
||||
- Provides persistence for tasks and conversations
|
||||
### Communication Protocols
|
||||
|
||||
### Chat UI
|
||||
```mermaid
|
||||
graph LR
|
||||
A[Tasks UI] -->|WebSocket| B[Agent Service]
|
||||
A -->|HTTP Proxy| C[Desktop VNC]
|
||||
B -->|REST API| D[Desktop API]
|
||||
B -->|SQL| E[PostgreSQL]
|
||||
B -->|HTTPS| F[Claude AI]
|
||||
D -->|IPC| G[bytebotd]
|
||||
```
|
||||
|
||||
- **NextJS web application** for interacting with the agent
|
||||
- Provides a chat interface for communicating with the AI
|
||||
- Includes an embedded noVNC view for observing desktop actions
|
||||
- Accessible at `localhost:9992`
|
||||
## Security Architecture
|
||||
|
||||
## Task Management
|
||||
### Isolation Layers
|
||||
|
||||
The agent system implements a task-based workflow:
|
||||
1. **Container Isolation**
|
||||
- Each desktop runs in its own Docker container
|
||||
- No access to host filesystem by default
|
||||
- Network isolation with explicit port mapping
|
||||
|
||||
1. **Tasks** are the primary unit of work with properties like status, priority, and description
|
||||
2. **Messages** represent the conversation between user and assistant
|
||||
3. **Summaries** capture the state and progress of tasks
|
||||
2. **Process Isolation**
|
||||
- bytebotd runs as non-root user
|
||||
- Separate processes for different services
|
||||
- Resource limits enforced by Docker
|
||||
|
||||
## Communication Flow
|
||||
3. **Network Security**
|
||||
- Services only accessible from localhost by default
|
||||
- Can be configured with authentication
|
||||
- HTTPS/WSS for external connections
|
||||
|
||||
### Standalone Mode
|
||||
### API Security
|
||||
|
||||
1. **External Application** makes requests to the Bytebot API
|
||||
2. **bytebotd daemon** receives and processes these requests
|
||||
3. **Desktop Automation** is performed using nutjs
|
||||
4. **Results/Screenshots** are returned to the calling application
|
||||
|
||||
### Agent Mode
|
||||
|
||||
1. **User** creates tasks and sends messages via the Chat UI
|
||||
2. **Agent service** processes tasks and messages through a queue system
|
||||
3. **AI Integration** with Claude generates responses and computer actions
|
||||
4. **Computer Use API** executes actions on the Bytebot desktop
|
||||
5. **Results** are returned to the user through the Chat UI
|
||||
|
||||
## Security Considerations
|
||||
- **Desktop API**: No authentication by default (localhost only)
|
||||
- **Agent API**: Can be secured with API keys
|
||||
- **Database**: Password protected, not exposed externally
|
||||
- **VNC Access**: Optional password protection
|
||||
|
||||
<Warning>
|
||||
The default container configuration is intended for development and testing
|
||||
purposes only. It should **not** be used in production environments without
|
||||
security hardening.
|
||||
Default configuration is for development. For production:
|
||||
- Enable authentication on all APIs
|
||||
- Use HTTPS/WSS for all connections
|
||||
- Implement network policies
|
||||
- Rotate credentials regularly
|
||||
</Warning>
|
||||
|
||||
Security aspects to consider before deploying in production:
|
||||
## Deployment Patterns
|
||||
|
||||
1. The container runs with a default user account that has sudo privileges
|
||||
2. Remote access protocols (VNC, noVNC) are not encrypted by default
|
||||
3. The REST API does not implement authentication by default
|
||||
4. Container networking exposes several ports that should be secured
|
||||
5. API keys (like ANTHROPIC_API_KEY) should be properly secured
|
||||
### Single User (Development)
|
||||
```yaml
|
||||
Services: All on one machine
|
||||
Scale: 1 instance each
|
||||
Use Case: Personal automation, development
|
||||
Resources: 4GB RAM, 2 CPU cores
|
||||
```
|
||||
|
||||
## Customization Points
|
||||
### Team Deployment
|
||||
```yaml
|
||||
Services: Separate agent and desktop hosts
|
||||
Scale: Multiple desktop containers
|
||||
Use Case: Shared automation platform
|
||||
Resources: 8GB+ RAM, 4+ CPU cores
|
||||
```
|
||||
|
||||
Bytebot is designed to be customizable for different use cases:
|
||||
### Enterprise Deployment
|
||||
```yaml
|
||||
Services: Kubernetes orchestration
|
||||
Scale: Auto-scaling based on load
|
||||
Use Case: Organization-wide automation
|
||||
Resources: Dedicated cluster
|
||||
```
|
||||
|
||||
- **Docker base image** can be modified for different Linux distributions
|
||||
- **Desktop environment** can be replaced with alternatives (GNOME, KDE, etc.)
|
||||
- **Pre-installed applications** can be customized for specific testing needs
|
||||
- **API endpoints** can be extended for additional functionality
|
||||
- **Agent system** can be extended with custom tools and integrations
|
||||
## Extension Points
|
||||
|
||||
### Custom Tools
|
||||
Add specialized software to the desktop:
|
||||
```dockerfile
|
||||
FROM bytebot/desktop:latest
|
||||
RUN apt-get update && apt-get install -y \
|
||||
your-custom-tools
|
||||
```
|
||||
|
||||
### AI Integrations
|
||||
Extend agent capabilities:
|
||||
- Custom tools for the LLM
|
||||
- Additional AI models
|
||||
- Specialized prompts
|
||||
- Domain-specific knowledge
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Resource Usage
|
||||
- **Desktop Container**: ~1GB RAM idle, 2GB+ active
|
||||
- **Agent Service**: ~256MB RAM
|
||||
- **UI Service**: ~128MB RAM
|
||||
- **Database**: ~256MB RAM
|
||||
|
||||
### Optimization Tips
|
||||
1. Use lightweight desktop environments
|
||||
2. Limit concurrent tasks
|
||||
3. Monitor resource usage
|
||||
4. Scale horizontally for more capacity
|
||||
|
||||
## Next Steps
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Agent System" icon="robot" href="/core-concepts/agent-system">
|
||||
Learn about the AI agent capabilities
|
||||
</Card>
|
||||
<Card title="Desktop Environment" icon="desktop" href="/core-concepts/desktop-environment">
|
||||
Explore the containerized desktop
|
||||
</Card>
|
||||
<Card title="API Reference" icon="code" href="/api-reference/introduction">
|
||||
Integrate with your applications
|
||||
</Card>
|
||||
<Card title="Deployment Guide" icon="rocket" href="/quickstart">
|
||||
Deploy your own instance
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -1,114 +1,278 @@
|
||||
---
|
||||
title: "Desktop Environment"
|
||||
description: "Details about the Bytebot containerized desktop environment"
|
||||
description: "The containerized Linux desktop that serves as your AI agent's workspace"
|
||||
---
|
||||
|
||||
## Containerized Desktop
|
||||
## Overview
|
||||
|
||||
Bytebot's containerized desktop environment provides a lightweight yet fully-functional Linux desktop inside a Docker container. This approach ensures consistency across different host systems and simplifies deployment.
|
||||
The Bytebot Desktop Environment is a fully functional Linux desktop running inside a Docker container. It's the workspace where your AI agent performs tasks - clicking, typing, browsing, and working with applications just like a human would at a physical computer.
|
||||
|
||||
## Components
|
||||
<img
|
||||
src="/images/core-container.png"
|
||||
alt="Bytebot Desktop Environment"
|
||||
className="w-full max-w-4xl"
|
||||
/>
|
||||
|
||||
### XFCE4 Desktop
|
||||
## Why a Containerized Desktop?
|
||||
|
||||
The desktop environment in Bytebot is based on XFCE4, a lightweight and efficient desktop environment for Unix-like operating systems:
|
||||
### Complete Isolation
|
||||
- **No Risk to Host**: All actions happen inside the container
|
||||
- **Sandboxed Environment**: Desktop can't access your host system
|
||||
- **Easy Reset**: Destroy and recreate in seconds
|
||||
- **Multiple Instances**: Run several agents simultaneously
|
||||
|
||||
- **Lightweight**: Requires minimal system resources
|
||||
- **Customizable**: Easily adapted for different use cases
|
||||
- **Fast**: Provides responsive desktop experience
|
||||
- **Compatible**: Works well with automation tools
|
||||
### Consistency Everywhere
|
||||
- **Platform Independent**: Same environment on Mac, Windows, or Linux
|
||||
- **Reproducible**: Identical setup every time
|
||||
- **Version Control**: Pin specific versions for stability
|
||||
- **No Dependencies**: Everything included in the container
|
||||
|
||||
### Built for Automation
|
||||
- **Predictable UI**: Consistent element positioning
|
||||
- **Clean Environment**: No popups or distractions
|
||||
- **Automation-Ready**: Optimized for programmatic control
|
||||
- **Fast Startup**: Desktop ready in seconds
|
||||
|
||||
## Technical Stack
|
||||
|
||||
### Base System
|
||||
- **Ubuntu 22.04 LTS**: Stable, well-supported Linux distribution
|
||||
- **XFCE4 Desktop**: Lightweight, responsive desktop environment
|
||||
- **X11 Display Server**: Standard Linux graphics system
|
||||
- **SystemD**: Modern service management
|
||||
|
||||
- **Ubuntu 22.04** (Jammy Jellyfish) serves as the base operating system
|
||||
- Default user account: `bytebot` with sudo privileges
|
||||
- Pre-configured locale and timezone settings
|
||||
### Pre-installed Software
|
||||
|
||||
### Pre-installed Applications
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Web Browsers" icon="globe">
|
||||
- Firefox ESR (Extended Support Release)
|
||||
- Chrome/Chromium available
|
||||
- Pre-configured for automation
|
||||
- Ad blocker extensions
|
||||
</Card>
|
||||
<Card title="Office Tools" icon="file-lines">
|
||||
- Text editors (nano, vim, gedit)
|
||||
- LibreOffice suite (optional)
|
||||
- PDF viewers
|
||||
- File managers
|
||||
</Card>
|
||||
<Card title="Communication" icon="envelope">
|
||||
- Thunderbird email client
|
||||
- Chat applications
|
||||
- Video conferencing tools
|
||||
- Terminal emulators
|
||||
</Card>
|
||||
<Card title="Development" icon="code">
|
||||
- Git version control
|
||||
- Python 3 environment
|
||||
- Node.js runtime
|
||||
- Common dev tools
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
The Bytebot container comes with essential software pre-installed:
|
||||
### Core Services
|
||||
|
||||
- **Firefox** web browser
|
||||
- **1Password** password manager
|
||||
- **Thunderbird** email client
|
||||
- **Terminal emulator** for command-line access
|
||||
- **Text editor** for viewing and editing files
|
||||
- **File manager** for navigating the filesystem
|
||||
- **Basic system utilities** (calculator, image viewer, etc.)
|
||||
1. **bytebotd Daemon**
|
||||
- Runs on port 9990
|
||||
- Handles all automation requests
|
||||
- Built on nutjs framework
|
||||
- Provides REST API
|
||||
|
||||
## Desktop Configuration
|
||||
2. **VNC Server**
|
||||
- TigerVNC for remote access
|
||||
- Configurable resolution
|
||||
- Multiple connection support
|
||||
|
||||
The desktop environment is configured with automation in mind:
|
||||
3. **noVNC Web Client**
|
||||
- Browser-based desktop access
|
||||
- No client installation needed
|
||||
- WebSocket proxy included
|
||||
|
||||
- **Simplified layout**: Clean desktop with minimal distractions
|
||||
- **Predictable element positioning**: Consistent locations for UI elements
|
||||
- **Auto-login**: Desktop environment starts automatically
|
||||
- **Resolution control**: Configurable display settings
|
||||
4. **Supervisor**
|
||||
- Process management
|
||||
- Service monitoring
|
||||
- Automatic restarts
|
||||
- Log management
|
||||
|
||||
## Remote Access
|
||||
## Desktop Features
|
||||
|
||||
### noVNC
|
||||
### Display Configuration
|
||||
```bash
|
||||
# Default resolution
|
||||
1920x1080 @ 24-bit color
|
||||
|
||||
For browser-based access, noVNC is included:
|
||||
# Configurable via environment
|
||||
DISPLAY_WIDTH=1920
|
||||
DISPLAY_HEIGHT=1080
|
||||
DISPLAY_DEPTH=24
|
||||
```
|
||||
|
||||
- Accessible via web browser at `http://localhost:9990/vnc`
|
||||
- No client software required
|
||||
- Works across different platforms and devices
|
||||
### User Environment
|
||||
- **Username**: `bytebot`
|
||||
- **Home Directory**: `/home/bytebot`
|
||||
- **Sudo Access**: Yes (passwordless)
|
||||
- **Desktop Session**: Auto-login enabled
|
||||
|
||||
## Display Server
|
||||
### File System
|
||||
```
|
||||
/home/bytebot/
|
||||
├── Desktop/ # Desktop shortcuts
|
||||
├── Documents/ # User documents
|
||||
├── Downloads/ # Browser downloads
|
||||
├── .config/ # Application configs
|
||||
└── .local/ # User data
|
||||
```
|
||||
|
||||
Bytebot uses Xvfb (X Virtual Framebuffer) as its display server:
|
||||
## Accessing the Desktop
|
||||
|
||||
- Creates virtual displays in memory without hardware
|
||||
- Suitable for headless environments
|
||||
- Configurable resolution and color depth
|
||||
- Compatible with standard X11 applications
|
||||
### Web Browser (Recommended)
|
||||
Navigate to `http://localhost:9990/vnc` for instant access:
|
||||
- No software installation required
|
||||
- Works on any device with a browser
|
||||
- Supports touch devices
|
||||
- Clipboard sharing
|
||||
|
||||
## Using the Desktop Environment
|
||||
### Direct API Control
|
||||
Most efficient for automation:
|
||||
```bash
|
||||
# Take a screenshot
|
||||
curl -X POST http://localhost:9990/api/computer \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"action": "screenshot"}'
|
||||
|
||||
### Manual Interaction
|
||||
|
||||
You can interact with the desktop environment using your web browser:
|
||||
|
||||
1. Navigate to `http://localhost:9990/vnc`
|
||||
|
||||
### Programmatic Interaction
|
||||
|
||||
The primary purpose of the Bytebot desktop is programmatic control through the Computer Use API:
|
||||
|
||||
- Control mouse and keyboard inputs
|
||||
- Capture screenshots
|
||||
- Interact with desktop applications
|
||||
- Automate workflows
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
The containerized desktop has some performance characteristics to be aware of:
|
||||
|
||||
- **Resource usage**: XFCE4 is lightweight but still requires CPU and memory resources
|
||||
- **Graphics performance**: Limited 3D acceleration compared to native desktop
|
||||
- **Network overhead**: Remote access adds some latency
|
||||
- **Disk I/O**: Container storage may be slower than native filesystem
|
||||
# Move mouse
|
||||
curl -X POST http://localhost:9990/api/computer \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"action": "move_mouse", "coordinate": [500, 300]}'
|
||||
```
|
||||
|
||||
## Customization
|
||||
|
||||
You can customize the desktop environment by:
|
||||
### Adding Software
|
||||
|
||||
1. Modifying the Dockerfile to install additional software
|
||||
2. Adjusting the XFCE4 configuration files for different layouts
|
||||
3. Adding custom startup scripts
|
||||
4. Setting environment variables to control behavior
|
||||
Create a custom Dockerfile:
|
||||
```dockerfile
|
||||
FROM bytebot/desktop:latest
|
||||
|
||||
## Security Notes
|
||||
# Install additional packages
|
||||
RUN apt-get update && apt-get install -y \
|
||||
slack-desktop \
|
||||
zoom \
|
||||
your-custom-app
|
||||
|
||||
# Copy configuration files
|
||||
COPY configs/ /home/bytebot/.config/
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Configure behavior via environment:
|
||||
```yaml
|
||||
# docker-compose.yml
|
||||
environment:
|
||||
- DISPLAY_WIDTH=1920
|
||||
- DISPLAY_HEIGHT=1080
|
||||
```
|
||||
|
||||
## Performance Optimization
|
||||
|
||||
### Resource Allocation
|
||||
```yaml
|
||||
# Recommended settings
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '2'
|
||||
memory: 4G
|
||||
reservations:
|
||||
cpus: '1'
|
||||
memory: 2G
|
||||
```
|
||||
|
||||
## Security Hardening
|
||||
|
||||
<Warning>
|
||||
The default desktop environment provides convenience at the expense of
|
||||
security. For production use, additional hardening is recommended.
|
||||
Default configuration prioritizes ease of use. For production, apply these security measures:
|
||||
</Warning>
|
||||
|
||||
Security considerations:
|
||||
### Essential Security Steps
|
||||
|
||||
- User has sudo privileges by default
|
||||
- VNC password should be changed from default
|
||||
- Desktop autologin removes authentication requirement
|
||||
- NoVNC web access may expose the desktop to unauthorized users
|
||||
1. **Change Default Passwords**
|
||||
```bash
|
||||
# Set user password
|
||||
passwd bytebot
|
||||
```
|
||||
|
||||
2. **Restrict Network Access**
|
||||
```yaml
|
||||
# Only expose to localhost
|
||||
ports:
|
||||
- "127.0.0.1:9990:9990"
|
||||
```
|
||||
|
||||
3. **Remove Sudo Access**
|
||||
```dockerfile
|
||||
# In custom Dockerfile
|
||||
RUN deluser bytebot sudo
|
||||
```
|
||||
|
||||
4. **Enable Authentication**
|
||||
```nginx
|
||||
# Add nginx proxy with auth
|
||||
location / {
|
||||
auth_basic "Restricted";
|
||||
auth_basic_user_file /etc/nginx/.htpasswd;
|
||||
proxy_pass http://bytebot:9990;
|
||||
}
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="Desktop won't start">
|
||||
Check logs:
|
||||
```bash
|
||||
docker logs bytebot-desktop
|
||||
```
|
||||
Common issues:
|
||||
- Insufficient memory
|
||||
- Port conflicts
|
||||
- Display server errors
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Applications crash">
|
||||
Monitor resources:
|
||||
```bash
|
||||
docker stats bytebot-desktop
|
||||
```
|
||||
Solutions:
|
||||
- Increase memory allocation
|
||||
- Check disk space
|
||||
- Update container image
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Regular Updates**: Keep the base image updated for security patches
|
||||
2. **Persistent Storage**: Mount volumes for important data
|
||||
3. **Backup Configurations**: Save customizations outside the container
|
||||
4. **Monitor Resources**: Track CPU/memory usage
|
||||
5. **Clean Temporary Files**: Periodic cleanup for performance
|
||||
|
||||
## Next Steps
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Quick Start" icon="rocket" href="/quickstart">
|
||||
Deploy your first agent
|
||||
</Card>
|
||||
<Card title="API Reference" icon="code" href="/api-reference/computer-use/unified-endpoint">
|
||||
Control the desktop programmatically
|
||||
</Card>
|
||||
<Card title="Agent System" icon="robot" href="/core-concepts/agent-system">
|
||||
Add AI capabilities
|
||||
</Card>
|
||||
<Card title="Security Guide" icon="shield" href="/deployment/security">
|
||||
Harden for production
|
||||
</Card>
|
||||
</CardGroup>
|
||||
+30
-13
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"$schema": "https://mintlify.com/docs.json",
|
||||
"theme": "mint",
|
||||
"name": "Bytebot Documentation",
|
||||
"name": "Bytebot - Self-Hosted AI Desktop Agent",
|
||||
"colors": {
|
||||
"primary": "#000000",
|
||||
"light": "#fbfaf9",
|
||||
@@ -11,32 +11,42 @@
|
||||
"navigation": {
|
||||
"tabs": [
|
||||
{
|
||||
"tab": "Guides",
|
||||
"tab": "Documentation",
|
||||
"groups": [
|
||||
{
|
||||
"group": "Get Started",
|
||||
"group": "Getting Started",
|
||||
"pages": ["introduction", "quickstart"]
|
||||
},
|
||||
{
|
||||
"group": "Core Concepts",
|
||||
"pages": [
|
||||
"core-concepts/architecture",
|
||||
"core-concepts/desktop-environment",
|
||||
"core-concepts/agent-system"
|
||||
"core-concepts/agent-system",
|
||||
"core-concepts/desktop-environment"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"tab": "REST API",
|
||||
"tab": "API Reference",
|
||||
"groups": [
|
||||
{
|
||||
"group": "Overview",
|
||||
"pages": ["rest-api/introduction"]
|
||||
"pages": ["api-reference/introduction"]
|
||||
},
|
||||
{
|
||||
"group": "Endpoints",
|
||||
"pages": ["rest-api/computer-use", "rest-api/examples"]
|
||||
"group": "Agent API",
|
||||
"pages": [
|
||||
"api-reference/agent/tasks",
|
||||
"api-reference/agent/ui"
|
||||
]
|
||||
},
|
||||
{
|
||||
"group": "Computer Control API",
|
||||
"pages": [
|
||||
"api-reference/computer-use/unified-endpoint",
|
||||
"api-reference/computer-use/examples"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -50,7 +60,7 @@
|
||||
},
|
||||
{
|
||||
"anchor": "Discord",
|
||||
"href": "https://discord.gg/6nxuF6cs",
|
||||
"href": "https://discord.gg/zcb5wA2t4u",
|
||||
"icon": "discord"
|
||||
},
|
||||
{
|
||||
@@ -79,14 +89,21 @@
|
||||
],
|
||||
"primary": {
|
||||
"type": "button",
|
||||
"label": "GitHub",
|
||||
"label": "Get Started",
|
||||
"href": "https://github.com/bytebot-ai/bytebot"
|
||||
}
|
||||
},
|
||||
"footer": {
|
||||
"socials": {
|
||||
"github": "https://github.com/bytebot-ai/bytebot",
|
||||
"twitter": "https://twitter.com/bytebotai"
|
||||
"twitter": "https://twitter.com/bytebotai",
|
||||
"discord": "https://discord.gg/zcb5wA2t4u"
|
||||
}
|
||||
},
|
||||
"metadata": {
|
||||
"og:title": "Bytebot - Self-Hosted AI Desktop Agent",
|
||||
"og:description": "Automate any computer task with natural language using your own AI desktop agent",
|
||||
"og:image": "/images/agent-architecture.png",
|
||||
"twitter:card": "summary_large_image"
|
||||
}
|
||||
}
|
||||
}
|
||||
+128
-30
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Introduction
|
||||
description: "A containerized computer use environment with an integrated XFCE4 desktop and automation daemon"
|
||||
description: "Self-hosted AI desktop agent that automates any computer task through natural language"
|
||||
---
|
||||
|
||||
<p align="center">
|
||||
@@ -20,54 +20,152 @@ description: "A containerized computer use environment with an integrated XFCE4
|
||||
|
||||
## What is Bytebot?
|
||||
|
||||
Bytebot provides a complete, self-contained environment for computer use automation. It encapsulates a lightweight XFCE4 desktop environment inside a Docker container with the bytebotd daemon for programmatic control, making it easy to deploy across different platforms.
|
||||
Bytebot is a self-hosted AI desktop agent that transforms how you interact with computers. By combining powerful AI with a containerized Linux desktop, Bytebot performs complex computer tasks. Think of it as your virtual employee that can actually use a computer – clicking, typing, browsing, and completing workflows just like a human would.
|
||||
|
||||
## Key Features
|
||||
## Why Self-Host Bytebot?
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card
|
||||
title="Containerized Desktop"
|
||||
icon="desktop"
|
||||
href="/core-concepts/desktop-environment"
|
||||
>
|
||||
Runs a lightweight XFCE4 desktop on Ubuntu 22.04 with pre-installed tools
|
||||
<Card title="Complete Privacy" icon="shield">
|
||||
Your tasks and data never leave your infrastructure. Everything runs locally
|
||||
on your servers.
|
||||
</Card>
|
||||
<Card
|
||||
title="Computer Use API"
|
||||
icon="code"
|
||||
href="/api-reference/computer-use/unified-endpoint"
|
||||
>
|
||||
Control the desktop environment programmatically through a unified REST API
|
||||
<Card title="Full Control" icon="sliders">
|
||||
Customize the desktop environment, install any applications, and configure
|
||||
to your exact needs.
|
||||
</Card>
|
||||
<Card title="Cross-Platform" icon="globe" href="/quickstart">
|
||||
Works on any system that supports Docker with simple setup
|
||||
<Card title="No Usage Limits" icon="infinity">
|
||||
Use your own LLM API keys without platform restrictions or additional fees.
|
||||
</Card>
|
||||
<Card title="VNC Access" icon="eye" href="/quickstart">
|
||||
View and interact with the desktop through VNC or browser-based noVNC
|
||||
<Card title="Secure Isolation" icon="lock">
|
||||
Each desktop runs in its own container, completely isolated from your host
|
||||
system.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
## Example Tasks You Can Delegate
|
||||
|
||||
- **Data Processing**: "Fill out this web form with the data from my CSV file"
|
||||
- **Email Management**: "Check my email and summarize important messages"
|
||||
- **Web Automation**: "Download all PDFs from this website and organize them by date"
|
||||
- **Testing**: "Test our checkout flow and take screenshots of each step"
|
||||
- **Invoice Processing**: "Download invoices from email, extract data, and update the accounting spreadsheet"
|
||||
- **Data Entry**: "Transfer customer information from PDF forms into our CRM system"
|
||||
- **Report Generation**: "Collect weekly metrics from different dashboards and compile the team report"
|
||||
- **Account Management**: "Update user permissions across our various SaaS tools"
|
||||
- **Document Processing**: "Download attachments from emails, rename them according to our convention, and file them"
|
||||
- **Compliance Tasks**: "Verify all employee records are updated in both HR systems"
|
||||
|
||||
## How It Works
|
||||
|
||||
<Steps>
|
||||
<Step title="Describe Your Task">
|
||||
Simply tell Bytebot what you want done in natural language through the tasks
|
||||
interface
|
||||
</Step>
|
||||
<Step title="AI Plans the Actions">
|
||||
Bytebot understands your request and breaks it down into specific computer
|
||||
actions
|
||||
</Step>
|
||||
<Step title="Executes Actions">
|
||||
Bytebot executes the task on the containerized desktop using the keyboard
|
||||
and mouse
|
||||
</Step>
|
||||
<Step title="Watch or Walk Away">
|
||||
Monitor it working in real-time through the desktop viewer, or let it
|
||||
complete tasks independently.
|
||||
</Step>
|
||||
<Step title="Get Results">
|
||||
Receive the completed task output, screenshots, or confirmation of
|
||||
completion
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
Bytebot is designed as a single, integrated container that provides both a desktop environment and the tools to control it:
|
||||
Bytebot consists of four integrated components working together:
|
||||
|
||||
<img src="/images/core-container.png" alt="Bytebot Core Container" />
|
||||
<img src="/images/agent-architecture.png" alt="Bytebot Agent Architecture" />
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card
|
||||
title="Bytebot Desktop"
|
||||
icon="desktop"
|
||||
href="/core-concepts/desktop-environment"
|
||||
>
|
||||
Ubuntu 22.04 with XFCE4, pre-installed apps, and the automation daemon
|
||||
</Card>
|
||||
<Card title="AI Agent" icon="brain" href="/core-concepts/agent-system">
|
||||
NestJS service that uses Claude AI to plan and execute tasks
|
||||
</Card>
|
||||
<Card
|
||||
title="Task Interface"
|
||||
icon="comments"
|
||||
href="/core-concepts/agent-system#task-ui"
|
||||
>
|
||||
Next.js web app for creating and managing tasks
|
||||
</Card>
|
||||
<Card title="REST API" icon="code" href="/api-reference/introduction">
|
||||
Programmatic access to both task management and direct desktop control
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
## Getting Started
|
||||
|
||||
Get up and running with Bytebot in minutes:
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Quick Start Guide" icon="rocket" href="/quickstart">
|
||||
Set up and run Bytebot on your system
|
||||
<CardGroup cols={3}>
|
||||
<Card title="Quick Start" icon="rocket" href="/quickstart">
|
||||
Get Bytebot running in 2 minutes
|
||||
</Card>
|
||||
<Card title="Architecture" icon="sitemap" href="/core-concepts/architecture">
|
||||
Understand how it all fits together
|
||||
</Card>
|
||||
<Card title="API Reference" icon="book" href="/api-reference/introduction">
|
||||
Learn how to programmatically control the Bytebot environment
|
||||
Integrate with your applications
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
## Use Cases
|
||||
|
||||
### Personal Productivity
|
||||
|
||||
- Automate repetitive tasks
|
||||
- Manage emails and calendars
|
||||
- Organize documents and files
|
||||
- Conduct web research
|
||||
|
||||
### Business Automation
|
||||
|
||||
- Data entry and form filling
|
||||
- Report generation
|
||||
- Competitive analysis
|
||||
- Customer support workflows
|
||||
|
||||
### Development & Testing
|
||||
|
||||
- UI test automation
|
||||
- Cross-browser testing
|
||||
- Documentation generation
|
||||
- API integration testing
|
||||
|
||||
## Community & Support
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card
|
||||
title="Discord Community"
|
||||
icon="discord"
|
||||
href="https://discord.com/invite/zcb5wA2t4u"
|
||||
>
|
||||
Join our community for help, tips, and discussions
|
||||
</Card>
|
||||
<Card
|
||||
title="GitHub"
|
||||
icon="github"
|
||||
href="https://github.com/bytebot-ai/bytebot"
|
||||
>
|
||||
Report issues, contribute, or star the project
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
<Note>
|
||||
The default container configuration is intended for development and testing
|
||||
purposes only. It should **not** be used in production environments without
|
||||
security hardening.
|
||||
**Ready to give your AI its own computer?** Start with our [Quick Start
|
||||
Guide](/quickstart) to have your own AI desktop agent running in minutes.
|
||||
</Note>
|
||||
|
||||
+207
-185
@@ -1,210 +1,232 @@
|
||||
---
|
||||
title: "Quickstart"
|
||||
description: "Get Bytebot up and running in minutes"
|
||||
title: "Quick Start"
|
||||
description: "Get your AI desktop agent running in 2 minutes"
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Before getting started with Bytebot, make sure you have:
|
||||
- Docker ≥ 20.10
|
||||
- Docker Compose
|
||||
- 4GB+ RAM available
|
||||
- Anthropic API key ([get one here](https://console.anthropic.com))
|
||||
|
||||
- Docker installed on your system
|
||||
- Sufficient system resources (recommended: 2+ CPU cores, 4GB+ RAM)
|
||||
- For the agent system: Docker Compose and an Anthropic API key
|
||||
## 🚀 2-Minute Setup
|
||||
|
||||
## Running Bytebot
|
||||
Get your self-hosted AI desktop agent running with just three commands:
|
||||
|
||||
Getting started with Bytebot is simple and straightforward. You can run it as a standalone desktop container or as a full agent system with a chat UI.
|
||||
<Steps>
|
||||
<Step title="Clone and Configure">
|
||||
```bash
|
||||
git clone https://github.com/bytebot-ai/bytebot.git cd bytebot echo "ANTHROPIC_API_KEY=your_api_key_here" > infrastructure/docker/.env
|
||||
```
|
||||
</Step>
|
||||
|
||||
<Step title="Start the Agent Stack">
|
||||
```bash
|
||||
docker-compose -f infrastructure/docker/docker-compose.yml up -d
|
||||
```
|
||||
|
||||
This starts all four services:
|
||||
|
||||
- **Bytebot Desktop**: Containerized Linux environment
|
||||
- **AI Agent**: Claude-powered task processor
|
||||
- **Chat UI**: Web interface for interaction
|
||||
- **Database**: PostgreSQL for persistence
|
||||
|
||||
</Step>
|
||||
|
||||
<Step title="Open the Chat Interface">
|
||||
Navigate to [http://localhost:9992](http://localhost:9992) and start chatting!
|
||||
|
||||
Try asking:
|
||||
|
||||
- "Open Firefox and search for the weather forecast"
|
||||
- "Take a screenshot of the desktop"
|
||||
- "Create a text file with today's date"
|
||||
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
<Note>
|
||||
**First time?** The initial startup may take 2-3 minutes as Docker downloads
|
||||
the images. Subsequent starts will be much faster.
|
||||
</Note>
|
||||
|
||||
## What Just Happened?
|
||||
|
||||
You now have a fully functional AI agent that can:
|
||||
|
||||
- 🖱️ Control mouse and keyboard
|
||||
- 🌐 Browse the web
|
||||
- 📄 Create and edit documents
|
||||
- 📧 Manage emails
|
||||
- 🔄 Automate any desktop task
|
||||
|
||||
Watch it work in real-time through the embedded VNC viewer in the tasks interface!
|
||||
|
||||
## Try These Example Tasks
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Web Research" icon="magnifying-glass">
|
||||
"Find the top 5 news stories about AI today and summarize them"
|
||||
</Card>
|
||||
<Card title="Data Collection" icon="table">
|
||||
"Go to example.com and extract all email addresses into a list"
|
||||
</Card>
|
||||
<Card title="File Management" icon="folder">
|
||||
"Create a folder called 'reports' and organize files by date"
|
||||
</Card>
|
||||
<Card title="Testing" icon="vial">
|
||||
"Test the login flow on our staging site"
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
## Accessing Your Services
|
||||
|
||||
| Service | URL | Purpose |
|
||||
| ---------------- | ------------------------------------------------------------------------ | --------------------------------------------- |
|
||||
| **Tasks UI** | [http://localhost:9992](http://localhost:9992) | Main interface for interacting with the agent |
|
||||
| **Agent API** | [http://localhost:9991/tasks](http://localhost:9991/tasks) | REST API for programmatic task creation |
|
||||
| **Computer API** | [http://localhost:9990/computer-use](http://localhost:9990/computer-use) | Low-level desktop control API |
|
||||
|
||||
## Alternative Deployment Options
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Desktop Container Only">
|
||||
If you just want the containerized desktop without the AI agent:
|
||||
|
||||
```bash
|
||||
#Using pre-built image (recommended)
|
||||
docker-compose -f infrastructure/docker/docker-compose.core.yml pull
|
||||
docker-compose -f infrastructure/docker/docker-compose.core.yml up -d
|
||||
```
|
||||
|
||||
Or build locally:
|
||||
```bash
|
||||
docker-compose -f infrastructure/docker/docker-compose.core.yml up -d --build
|
||||
```
|
||||
|
||||
Access the desktop at
|
||||
[http://localhost:9990/vnc](http://localhost:9990/vnc)
|
||||
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
## Managing Your Agent
|
||||
|
||||
### View Logs
|
||||
|
||||
Monitor what your agent is doing:
|
||||
|
||||
```bash
|
||||
# All services
|
||||
docker-compose -f infrastructure/docker/docker-compose.yml logs -f
|
||||
|
||||
# Just the agent
|
||||
docker-compose -f infrastructure/docker/docker-compose.yml logs -f bytebot-agent
|
||||
```
|
||||
|
||||
### Stop Services
|
||||
|
||||
```bash
|
||||
docker-compose -f infrastructure/docker/docker-compose.yml down
|
||||
```
|
||||
|
||||
### Update to Latest
|
||||
|
||||
```bash
|
||||
docker-compose -f infrastructure/docker/docker-compose.yml pull
|
||||
docker-compose -f infrastructure/docker/docker-compose.yml up -d
|
||||
```
|
||||
|
||||
### Reset Everything
|
||||
|
||||
Remove all data and start fresh:
|
||||
|
||||
```bash
|
||||
docker-compose -f infrastructure/docker/docker-compose.yml down -v
|
||||
```
|
||||
|
||||
## Quick API Examples
|
||||
|
||||
### Create a Task via API
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:9991/tasks \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"description": "Search for flights from NYC to London next month",
|
||||
"type": "browser_task"
|
||||
}'
|
||||
```
|
||||
|
||||
### Direct Desktop Control
|
||||
|
||||
```bash
|
||||
# Take a screenshot
|
||||
curl -X POST http://localhost:9990/api/computer \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"action": "screenshot"}'
|
||||
|
||||
# Type text
|
||||
curl -X POST http://localhost:9990/api/computer \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"action": "type_text", "text": "Hello, Bytebot!"}'
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion icon="cube" title="Running the Standalone Desktop Container">
|
||||
<AccordionGroup>
|
||||
<Accordion icon="play" title="Using the Pre-built Image">
|
||||
The easiest way to get started is to use the pre-built Bytebot image:
|
||||
|
||||
```bash
|
||||
docker-compose -f infrastructure/docker/docker-compose.core.yml pull # pull latest remote image
|
||||
docker-compose -f infrastructure/docker/docker-compose.core.yml up -d --no-build # start container
|
||||
```
|
||||
|
||||
This will start Bytebot with default settings. The container exposes one port:
|
||||
- `9990`: REST API and noVNC web access
|
||||
</Accordion>
|
||||
|
||||
<Accordion icon="cube" title="Building the Docker Image (Alternative)">
|
||||
Alternatively, you can build the Bytebot Docker image locally:
|
||||
|
||||
```bash
|
||||
docker-compose -f infrastructure/docker/docker-compose.core.yml up -d --build # build image and start container
|
||||
```
|
||||
</Accordion>
|
||||
|
||||
<Accordion icon="stop" title="Stopping the Container">
|
||||
To shut down the container:
|
||||
|
||||
```bash
|
||||
docker-compose -f infrastructure/docker/docker-compose.core.yml down
|
||||
```
|
||||
</Accordion>
|
||||
|
||||
<Accordion icon="display" title="Accessing the Desktop">
|
||||
You can access the Bytebot desktop environment in two ways:
|
||||
|
||||
Navigate to `http://localhost:9990/vnc` in your web browser for noVNC access.
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
<Accordion title="Container won't start">
|
||||
Check Docker is running and you have enough resources:
|
||||
```bash
|
||||
docker info
|
||||
docker-compose -f infrastructure/docker/docker-compose.yml logs
|
||||
```
|
||||
</Accordion>
|
||||
|
||||
<Accordion icon="robot" title="Running the Full Agent System (Alpha)">
|
||||
<AccordionGroup>
|
||||
<Accordion icon="key" title="Setting Up Environment Variables">
|
||||
Create a `.env` file with your Anthropic API key:
|
||||
|
||||
```bash
|
||||
echo "ANTHROPIC_API_KEY=your_api_key_here" > infrastructure/docker/.env
|
||||
```
|
||||
|
||||
Replace `your_api_key_here` with your actual Anthropic API key.
|
||||
</Accordion>
|
||||
|
||||
<Accordion icon="play" title="Starting the Services">
|
||||
Run the full Bytebot system using Docker Compose:
|
||||
|
||||
```bash
|
||||
docker-compose -f infrastructure/docker/docker-compose.yml --env-file infrastructure/docker/.env up -d
|
||||
```
|
||||
|
||||
This will start the following services using the pre-built image:
|
||||
- Bytebot desktop container
|
||||
- PostgreSQL database
|
||||
- Bytebot agent service
|
||||
- Bytebot chat UI
|
||||
</Accordion>
|
||||
|
||||
<Accordion icon="stop" title="Stopping the Services">
|
||||
To shut down all services:
|
||||
|
||||
```bash
|
||||
docker-compose -f infrastructure/docker/docker-compose.yml --env-file infrastructure/docker/.env down
|
||||
```
|
||||
</Accordion>
|
||||
|
||||
<Accordion icon="display" title="Accessing the System">
|
||||
Once all services are running, you can access Bytebot through:
|
||||
|
||||
- **Bytebot Desktop (noVNC)**: `http://localhost:9990/vnc`
|
||||
- **Bytebot Agent API**: `http://localhost:9991`
|
||||
- **Bytebot Chat UI**: `http://localhost:9992`
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
<Accordion title="Can't connect to tasks UI">
|
||||
Ensure all services are running:
|
||||
```bash
|
||||
docker-compose -f infrastructure/docker/docker-compose.yml ps
|
||||
```
|
||||
All services should show as "Up".
|
||||
</Accordion>
|
||||
<Accordion title="Agent errors or no response">
|
||||
Check your API key is set correctly:
|
||||
```bash
|
||||
cat infrastructure/docker/.env
|
||||
docker-compose -f infrastructure/docker/docker-compose.yml logs bytebot-agent
|
||||
```
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
## Using the Computer Use API
|
||||
|
||||
The core functionality of Bytebot is programmatic control of the desktop environment through its API.
|
||||
|
||||
<CodeGroup>
|
||||
```bash Terminal
|
||||
# Example: Move the mouse to coordinates (100, 200)
|
||||
curl -X POST http://localhost:9990/computer-use \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"action": "move_mouse", "coordinates": {"x": 100, "y": 200}}'
|
||||
|
||||
# Example: Take a screenshot
|
||||
|
||||
curl -X POST http://localhost:9990/computer-use \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"action": "screenshot"}'
|
||||
|
||||
````
|
||||
|
||||
```python Python
|
||||
import requests
|
||||
|
||||
def control_computer(action, **params):
|
||||
url = "http://localhost:9990/computer-use"
|
||||
data = {"action": action, **params}
|
||||
response = requests.post(url, json=data)
|
||||
return response.json()
|
||||
|
||||
# Move the mouse
|
||||
control_computer("move_mouse", coordinates={"x": 100, "y": 100})
|
||||
|
||||
# Take a screenshot
|
||||
screenshot = control_computer("screenshot")
|
||||
````
|
||||
|
||||
```javascript JavaScript
|
||||
const axios = require("axios");
|
||||
|
||||
async function controlComputer(action, params = {}) {
|
||||
const url = "http://localhost:9990/computer-use";
|
||||
const data = { action, ...params };
|
||||
const response = await axios.post(url, data);
|
||||
return response.data;
|
||||
}
|
||||
|
||||
// Example usage
|
||||
async function runExample() {
|
||||
// Move mouse
|
||||
await controlComputer("move_mouse", { coordinates: { x: 100, y: 100 } });
|
||||
|
||||
// Take screenshot
|
||||
const screenshot = await controlComputer("screenshot");
|
||||
console.log("Screenshot taken:", screenshot);
|
||||
}
|
||||
```
|
||||
|
||||
</CodeGroup>
|
||||
|
||||
<Card
|
||||
title="API Reference"
|
||||
icon="code"
|
||||
href="/api-reference/computer-use/unified-endpoint"
|
||||
>
|
||||
View the complete Computer Use API documentation
|
||||
</Card>
|
||||
|
||||
## Using the Agent Chat UI
|
||||
|
||||
When running the full agent system, you can interact with Bytebot through a chat interface that allows you to:
|
||||
|
||||
1. Create and manage tasks
|
||||
2. Chat with the AI assistant
|
||||
3. View the desktop in real-time as the agent performs actions
|
||||
4. Monitor task progress and results
|
||||
|
||||
<img
|
||||
src="/static/chat-ui-example.png"
|
||||
alt="Bytebot Chat UI"
|
||||
className="w-full max-w-4xl"
|
||||
/>
|
||||
|
||||
## Next Steps
|
||||
|
||||
Now that you have Bytebot up and running, here are some next steps:
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card
|
||||
title="Explore the Architecture"
|
||||
icon="diagram-project"
|
||||
title="Architecture Overview"
|
||||
icon="sitemap"
|
||||
href="/core-concepts/architecture"
|
||||
>
|
||||
Learn more about Bytebot's architecture and components
|
||||
</Card>
|
||||
<Card
|
||||
title="Desktop Environment"
|
||||
icon="desktop"
|
||||
href="/core-concepts/desktop-environment"
|
||||
>
|
||||
Discover details about the containerized desktop environment
|
||||
Understand how all the components work together
|
||||
</Card>
|
||||
<Card title="Agent System" icon="robot" href="/core-concepts/agent-system">
|
||||
Learn about the agent task management system
|
||||
Deep dive into the AI agent capabilities
|
||||
</Card>
|
||||
<Card title="API Reference" icon="code" href="/api-reference/agent/tasks">
|
||||
Explore the Agent API for task management
|
||||
<Card title="API Reference" icon="code" href="/api-reference/introduction">
|
||||
Build integrations with the REST APIs
|
||||
</Card>
|
||||
<Card
|
||||
title="Use Cases"
|
||||
icon="lightbulb"
|
||||
href="/core-concepts/agent-system#use-cases"
|
||||
>
|
||||
Explore what others are building
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
<Note>
|
||||
**Need help?** Join our [Discord
|
||||
community](https://discord.com/invite/zcb5wA2t4u) for support and to share
|
||||
what you're building!
|
||||
</Note>
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
<svg width="256" height="256" viewBox="0 0 256 256" fill="none" xmlns="http://www.w3.org/2000/svg">
|
||||
<rect width="256" height="256" fill="white"/>
|
||||
<path d="M195.022 166.417L130.04 201.594V125.741L195.022 89.2791V166.417ZM204.758 76.3028L130.882 37.6428C129.039 36.678 126.828 36.724 125.025 37.7644L51.3895 80.2784C49.509 81.3648 48.3511 83.3709 48.3511 85.5427V170.707C48.3511 172.878 49.509 174.885 51.3895 175.971L125.145 218.553C127.026 219.638 129.342 219.638 131.224 218.553L204.979 175.971C206.86 174.885 208.018 172.878 208.018 170.707V81.6877C208.018 79.4261 206.762 77.3514 204.758 76.3028Z" fill="black"/>
|
||||
<path d="M158.819 133.386L146.979 140.223V171.295L158.819 164.459V133.386Z" fill="black"/>
|
||||
<path d="M184.811 118.534L172.971 125.371V156.441L184.811 149.606V118.534Z" fill="black"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 805 B |
Reference in New Issue
Block a user