opensource/coder
3
0
Fork 0
mirror of https://github.com/coder/coder.git synced 2026-09-24 15:04:27 +08:00
Code Issues Packages Projects Releases Wiki Activity

docs: add standalone AI Gateway docs (#27592)

Browse Source 
Documents standalone AI Gateway deployment, Gateway key authentication,
monitoring, and the updated embedded vs standalone topology in the AI
Gateway docs.

---------

Co-authored-by: Cian Johnston <cian@coder.com>
This commit is contained in:
Paweł Banaszewski
2026-07-29 19:38:22 +02:00
committed by GitHub
co-authored by Cian Johnston
parent 6c42309ccb
commit 18128b7b52
18 changed files with 773 additions and 136 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/admin/infrastructure/architecture.md
+9 -1
View File
@@ -138,7 +138,15 @@ as OpenAI and Anthropic. Users authenticate through Coder instead of managing se
provider API keys. All prompts, token usage, and tool invocations are recorded
for compliance and cost tracking.
Learn more: [AI Gateway](../../ai-coder/ai-gateway/index.md)
AI Gateway supports 2 deployment topologies:
- **Embedded:** `coderd` runs the AI Gateway data plane in the same process.
- **Standalone:** AI Gateway runs outside `coderd`, as replicas that serve AI traffic and send requests directly to upstream providers.
Standalone replicas hold no durable state. `coderd` is the source of truth and the only component that writes AI Gateway state to the database.
Each replica maintains a control connection to `coderd` for Coder API key validation, provider configuration, and AI session recording, and becomes unready when that connection is unavailable.
Refer to [AI Gateway](../../ai-coder/ai-gateway/index.md) and [standalone deployment](../../ai-coder/ai-gateway/standalone.md) for configuration and operational guidance.
### Agent Firewall
docs/admin/infrastructure/validated-architectures/index.md
+10
View File
@@ -123,6 +123,16 @@ or helper scripts.
Please note that the Registry is a hosted service and isn't available for
offline use.
### AI Gateway
[AI Gateway](../../../ai-coder/ai-gateway/index.md) proxies AI provider traffic
and records each AI session. It runs inside `coderd` by default, and can also
run as a [standalone deployment](../../../ai-coder/ai-gateway/standalone.md)
that scales independently of the control plane. Size replicas from your own AI
request volume and `CODER_AI_GATEWAY_MAX_CONCURRENCY`. For the chart's resource
requests and autoscaling defaults, refer to the
[AI Gateway Helm chart README](https://github.com/coder/coder/blob/main/helm/ai-gateway/README.md).
## Kubernetes Infrastructure
Kubernetes is the recommended, and supported platform for deploying Coder in the
docs/admin/integrations/prometheus.md
+10 -1
View File
@@ -72,6 +72,13 @@ scrape_configs:
apps: "coder"
```
If you run a [standalone AI Gateway](../../ai-coder/ai-gateway/standalone.md),
each replica exports its own metrics on its own listener. Its Helm chart uses the
same `0.0.0.0:2112` default as the `coder` chart, but sets up no scrape
discovery. Refer to
[AI Gateway monitoring](../../ai-coder/ai-gateway/monitoring.md#kubernetes-discovery)
for more details.
To use the Kubernetes Prometheus operator to scrape metrics, you will need to
create a `ServiceMonitor` in your Coder deployment namespace. The following is
an example `ServiceMonitor`.
@@ -102,6 +109,8 @@ You must first enable `coderd_agentstats_*` with the flag
`CODER_PROMETHEUS_COLLECT_AGENT_STATS` before they can be retrieved from the
deployment. They will always be available from the agent.
The `coder_ai_gateway_cost_control_*` metrics are exported only by `coderd`.
<!-- Code generated by 'make docs/admin/integrations/prometheus.md'. DO NOT EDIT -->
| Name | Type | Description | Labels |
@@ -124,7 +133,7 @@ deployment. They will always be available from the agent.
| `coder_ai_gateway_key_pool_exhaustions_total` | counter | The number of times the key pool was exhausted with no usable key (outcome: rate_limited, auth_failed). | `outcome` `provider` |
| `coder_ai_gateway_key_pool_failover_attempts` | histogram | The number of keys attempted before success or exhaustion, per interception for bridged requests and per request for passthrough requests. | `provider` |
| `coder_ai_gateway_key_pool_state` | gauge | The number of keys currently in each state (state: valid, temporary, permanent). | `provider` `state` |
| `coder_ai_gateway_key_pool_state_transitions_total` | counter | The number of API key state transitions during failover (reason: rate_limited, unauthorized, forbidden). | `provider` `reason` |
| `coder_ai_gateway_key_pool_state_transitions_total` | counter | The number of API key state transitions during failover (reason: rate_limited, unauthorized). | `provider` `reason` |
| `coder_ai_gateway_non_injected_tool_selections_total` | counter | The number of times an AI model selected a tool to be invoked by the client. | `model` `name` `provider` |
| `coder_ai_gateway_passthrough_total` | counter | The count of requests which were not intercepted but passed through to the upstream. | `method` `provider` `route` |
| `coder_ai_gateway_prompts_total` | counter | The number of prompts issued by users (initiators). | `client` `initiator_id` `model` `provider` |
docs/admin/users/sessions-tokens.md
+2
View File
@@ -119,6 +119,8 @@ To hard-delete a token, use the `--delete` flag:
coder tokens remove --delete <name|id>
```
Deleting the user that owns a token revokes every token that user holds at the same time.
## API Key Scopes
API key scopes allow you to limit the permissions of a token to specific operations. By default, tokens are created with the `all` scope, granting full access to all actions the user can perform. For improved security, you can create tokens with limited scopes that restrict access to only the operations needed.