mirror of
https://github.com/coder/coder.git
synced 2026-10-31 02:43:21 +08:00
docs: address AI Gateway documentation review feedback (#28309)
Applies the review feedback from #27794. Prose only. --------- Co-authored-by: Nick Vigilante <nickvigilante@users.noreply.github.com>
This commit is contained in:
Paweł Banaszewski
co-authored by
Nick Vigilante
Nick Vigilante
parent
c9a1e2bcee
commit
4ce8e8ed56
@@ -133,17 +133,18 @@ AI-powered development within Coder workspaces.
|
||||
|
||||
### AI Gateway
|
||||
|
||||
AI Gateway is a centralized gateway that sits between coding agents and LLM providers such
|
||||
as OpenAI and Anthropic. Users authenticate through Coder instead of managing separate
|
||||
provider API keys. All prompts, token usage, and tool invocations are recorded
|
||||
for compliance and cost tracking.
|
||||
AI Gateway is a centralized feature that sits between coding agents and LLM providers such as OpenAI and Anthropic.
|
||||
Users authenticate through Coder instead of managing separate provider API keys.
|
||||
All prompts, token usage, and tool invocations are recorded for compliance and cost tracking.
|
||||
|
||||
AI Gateway supports 2 deployment topologies:
|
||||
|
||||
- **Embedded:** `coderd` runs the AI Gateway data plane in the same process.
|
||||
- **Standalone:** AI Gateway runs outside `coderd`, as replicas that serve AI traffic and send requests directly to upstream providers.
|
||||
- **Standalone:** gateway replicas run outside `coderd`, serve AI traffic, and send requests directly to upstream providers.
|
||||
|
||||
Standalone replicas hold no durable state. `coderd` is the source of truth and the only component that writes AI Gateway state to the database.
|
||||
Standalone replicas have no authoritative database state.
|
||||
They keep ephemeral provider snapshots, caches, and metrics in memory, and can optionally write API dumps to local disk.
|
||||
`coderd` is the source of truth and the only component that writes AI Gateway state to the database.
|
||||
Each replica maintains a control connection to `coderd` for Coder API key validation, provider configuration, and AI session recording, and becomes unready when that connection is unavailable.
|
||||
|
||||
Refer to [AI Gateway](../../ai-coder/ai-gateway/index.md) and [standalone deployment](../../ai-coder/ai-gateway/standalone.md) for configuration and operational guidance.
|
||||
|
||||
@@ -125,13 +125,10 @@ offline use.
|
||||
|
||||
### AI Gateway
|
||||
|
||||
[AI Gateway](../../../ai-coder/ai-gateway/index.md) proxies AI provider traffic
|
||||
and records each AI session. It runs inside `coderd` by default, and can also
|
||||
run as a [standalone deployment](../../../ai-coder/ai-gateway/standalone.md)
|
||||
that scales independently of the control plane. Size replicas from your own AI
|
||||
request volume and `CODER_AI_GATEWAY_MAX_CONCURRENCY`. For the chart's resource
|
||||
requests and autoscaling defaults, refer to the
|
||||
[AI Gateway Helm chart README](https://github.com/coder/coder/blob/main/helm/ai-gateway/README.md).
|
||||
[AI Gateway](../../../ai-coder/ai-gateway/index.md) proxies AI provider traffic and records each AI session.
|
||||
It runs inside `coderd` by default and can also run as a [standalone deployment](../../../ai-coder/ai-gateway/standalone.md) that scales independently of the control plane.
|
||||
Size replicas based on your AI request volume and `CODER_AI_GATEWAY_MAX_CONCURRENCY`.
|
||||
For the chart's resource requests and autoscaling defaults, refer to the [AI Gateway Helm chart README](../../../../helm/ai-gateway/README.md).
|
||||
|
||||
## Kubernetes Infrastructure
|
||||
|
||||
|
||||
@@ -46,7 +46,7 @@ coder:
|
||||
readOnly: true
|
||||
```
|
||||
|
||||
[Upgrade Coder](../../install/kubernetes.md#upgrading-coder-via-helm) with these
|
||||
[Upgrade Coder](../../install/kubernetes.md#upgrade-coder-via-helm) with these
|
||||
new values.
|
||||
|
||||
### VM control plane
|
||||
|
||||
@@ -72,12 +72,9 @@ scrape_configs:
|
||||
apps: "coder"
|
||||
```
|
||||
|
||||
If you run a [standalone AI Gateway](../../ai-coder/ai-gateway/standalone.md),
|
||||
each replica exports its own metrics on its own listener. Its Helm chart uses the
|
||||
same `0.0.0.0:2112` default as the `coder` chart, but sets up no scrape
|
||||
discovery. Refer to
|
||||
[AI Gateway monitoring](../../ai-coder/ai-gateway/monitoring.md#kubernetes-discovery)
|
||||
for more details.
|
||||
If you run a [standalone AI gateway](../../ai-coder/ai-gateway/standalone.md), each replica exports its own metrics on its own listener.
|
||||
Its Helm chart uses the same `0.0.0.0:2112` default as the `coder` chart, but sets up no scrape discovery.
|
||||
Refer to [AI gateway monitoring](../../ai-coder/ai-gateway/monitoring.md#kubernetes-discovery) for more details.
|
||||
|
||||
To use the Kubernetes Prometheus operator to scrape metrics, you will need to
|
||||
create a `ServiceMonitor` in your Coder deployment namespace. The following is
|
||||
|
||||
@@ -55,7 +55,7 @@ Intercepted requests are forwarded to the AI Gateway, configured via [`CODER_AI_
|
||||
By default, this is the embedded AI Gateway at `<coderd-access-url>/api/v2/ai-gateway`, and no configuration is needed.
|
||||
|
||||
AI Gateway Proxy remains part of the `coder server` process when you [deploy AI Gateway as a standalone service](../standalone.md).
|
||||
To forward intercepted requests to the standalone Gateway, set:
|
||||
To forward intercepted requests to the standalone gateway, set the following:
|
||||
|
||||
```sh
|
||||
CODER_AI_GATEWAY_PROXY_TARGET=https://ai-gateway.example.com/
|
||||
@@ -73,7 +73,7 @@ For additional configuration options, see the [Coder server configuration](../..
|
||||
> The AI Gateway Proxy should only be accessible within a trusted network and **must not** be directly exposed to the public internet.
|
||||
> Without proper network restrictions, unauthorized users could route traffic through the proxy or intercept credentials.
|
||||
|
||||
### Encrypting client connections
|
||||
### Encrypt client connections
|
||||
|
||||
By default, AI tools send the Coder session token in the proxy credentials over unencrypted HTTP.
|
||||
This only applies to the initial connection between the client and the proxy.
|
||||
@@ -89,7 +89,7 @@ See [Proxy TLS Configuration](#proxy-tls-configuration) for configuration steps.
|
||||
* Internal network only: If the proxy and all clients are on the same trusted network, credentials are not exposed to external attackers.
|
||||
* TLS-terminating load balancer: Place a TLS-terminating load balancer in front of the proxy that terminates TLS and forwards requests over HTTP.
|
||||
|
||||
### Restricting proxy access
|
||||
### Restrict proxy access
|
||||
|
||||
Requests to non-allowlisted domains are tunneled through the proxy, but connections to private and reserved IP ranges are blocked by default.
|
||||
The IP validation and TCP connect happen atomically, preventing DNS rebinding attacks where the resolved address could change between the check and the connection.
|
||||
@@ -151,7 +151,7 @@ CODER_AI_GATEWAY_PROXY_CERT_FILE=/path/to/intermediate-ca.crt
|
||||
CODER_AI_GATEWAY_PROXY_KEY_FILE=/path/to/intermediate-ca.key
|
||||
```
|
||||
|
||||
### Securing the private key
|
||||
### Secure the private key
|
||||
|
||||
> [!WARNING]
|
||||
> The CA private key is used to sign certificates for MITM interception.
|
||||
@@ -162,7 +162,7 @@ Best practices:
|
||||
* Restrict file permissions so only the Coder process can read the key.
|
||||
* Use a secrets manager to store the key where possible.
|
||||
|
||||
### Distributing the certificate
|
||||
### Distribute the certificate
|
||||
|
||||
AI tools need to trust the CA certificate before connecting through the proxy.
|
||||
|
||||
@@ -235,7 +235,7 @@ The certificate must include a SAN matching the proxy's hostname or IP address.
|
||||
|
||||
If clients already trust your organization's root CA, no additional certificate configuration is needed for the TLS connection to the proxy.
|
||||
|
||||
### Trusting the TLS certificate
|
||||
### Trust the TLS certificate
|
||||
|
||||
For **self-signed certificates**, AI tools must be configured to trust the TLS certificate.
|
||||
|
||||
@@ -290,7 +290,7 @@ To use AI Gateway Proxy, AI tools must be configured to:
|
||||
1. Route traffic through the proxy
|
||||
1. Trust the proxy's CA certificate
|
||||
|
||||
### Configuring the proxy
|
||||
### Configure the proxy
|
||||
|
||||
The preferred approach is to configure the proxy directly in the AI tool's settings, as this avoids routing unnecessary traffic through the proxy.
|
||||
Consult the tool's documentation for specific instructions.
|
||||
@@ -311,7 +311,7 @@ Note: if [TLS is not enabled](#proxy-tls-configuration) on the proxy, replace `h
|
||||
|
||||
In order for AI tools that communicate with AI Gateway Proxy to authenticate with Coder via AI Gateway, the Coder session token needs to be passed in the proxy credentials as the password field.
|
||||
|
||||
### Trusting the CA certificate
|
||||
### Trust the CA certificate
|
||||
|
||||
The preferred approach is to configure the CA certificate directly in the AI tool's settings, as this limits the scope of the trusted certificate to that specific application.
|
||||
Consult the tool's documentation for specific instructions.
|
||||
@@ -418,7 +418,7 @@ x509: certificate signed by unknown authority
|
||||
```
|
||||
|
||||
it has not been configured to trust the proxy's
|
||||
MITM CA certificate. See [Trusting the CA certificate](#trusting-the-ca-certificate). If
|
||||
MITM CA certificate. Visit [Trust the CA certificate](#trust-the-ca-certificate). If
|
||||
[TLS is enabled on the listener](#proxy-tls-configuration), the tool must trust that certificate as well.
|
||||
|
||||
### Requests are not being intercepted
|
||||
@@ -465,4 +465,4 @@ WARN blocking connection to private/reserved IP hostname=... port=... resolved
|
||||
```
|
||||
|
||||
To allow specific internal networks, set
|
||||
[`CODER_AI_GATEWAY_PROXY_ALLOWED_PRIVATE_CIDRS`](#restricting-proxy-access).
|
||||
[`CODER_AI_GATEWAY_PROXY_ALLOWED_PRIVATE_CIDRS`](#restrict-proxy-access).
|
||||
|
||||
@@ -7,28 +7,24 @@
|
||||
AI Gateway uses different credentials for different kinds of connections:
|
||||
|
||||
- AI clients use a Coder API token to authenticate with AI Gateway as a user.
|
||||
- Standalone Gateway replicas use AI Gateway keys to connect to the Coder control plane.
|
||||
- Standalone gateway replicas use AI Gateway keys to connect to the Coder control plane.
|
||||
- AI Gateway uses provider credentials configured by an administrator to authenticate to upstream AI providers.
|
||||
- In Bring Your Own Key (BYOK) mode, a user also supplies a personal provider credential or subscription token.
|
||||
- In [Bring Your Own Key (BYOK)](#bring-your-own-key-byok) mode, a user also supplies a personal provider credential or subscription token.
|
||||
|
||||
These credentials are not interchangeable.
|
||||
A Gateway key does not authenticate an AI client, and a Coder API token does not authenticate a standalone replica.
|
||||
A gateway key does not authenticate an AI client, and a Coder API token does not authenticate a standalone replica.
|
||||
|
||||
## Authenticate AI clients
|
||||
|
||||
AI Gateway authenticates clients with the same Coder API token
|
||||
that a user uses for the rest of the Coder API.
|
||||
AI Gateway authenticates clients with the same Coder API token that a user uses for the rest of the Coder API.
|
||||
No separate AI Gateway login is required for client traffic.
|
||||
For token creation, expiration, and revocation, refer to [Sessions and API tokens](../../admin/users/sessions-tokens.md).
|
||||
|
||||
Authenticating with a Coder token avoids distributing centralized provider API keys,
|
||||
such as OpenAI or Anthropic keys, to individual users.
|
||||
AI Gateway handles upstream credentials centrally and
|
||||
forwards each request to the configured provider on the user's behalf.
|
||||
Authenticating with a Coder token avoids distributing centralized provider API keys, such as OpenAI or Anthropic keys, to individual users.
|
||||
AI Gateway handles upstream credentials centrally and forwards each request to the configured provider on the user's behalf.
|
||||
|
||||
The exact environment variable or setting name differs between tools.
|
||||
Refer to the list of [supported clients](./clients/index.md) and
|
||||
your tool's documentation for details.
|
||||
Refer to the list of [supported clients](./clients/index.md) and your tool's documentation for details.
|
||||
|
||||
### Create a Coder API token
|
||||
|
||||
@@ -54,8 +50,7 @@ Use short lifetimes for automation and CI to limit the blast radius if a token l
|
||||
|
||||
### Retrieve your session token
|
||||
|
||||
If you're logged in with the Coder CLI, retrieve your current session token
|
||||
with [`coder login token`](../../reference/cli/login_token.md):
|
||||
If you're logged in with the Coder CLI, retrieve your current session token with [`coder login token`](../../reference/cli/login_token.md):
|
||||
|
||||
```sh
|
||||
export ANTHROPIC_API_KEY=$(coder login token)
|
||||
@@ -75,11 +70,11 @@ export HTTPS_PROXY="https://coder:$(coder login token)@<proxy-host>:8888"
|
||||
The client machine also needs to trust the proxy's CA certificate.
|
||||
For full setup, refer to [AI Gateway Proxy setup](./ai-gateway-proxy/setup.md).
|
||||
|
||||
## Authenticate standalone Gateway replicas
|
||||
## Authenticate standalone gateway replicas
|
||||
|
||||
AI Gateway keys are scoped to the Coder deployment.
|
||||
A [standalone AI Gateway](./standalone.md) uses one of these keys to connect to `coderd`.
|
||||
Only the built-in Owner role can create, list, and delete these keys.
|
||||
A [standalone AI gateway](./standalone.md) uses one of these keys to connect to `coderd`.
|
||||
Only the built-in **owner** role can create, list, and delete these keys.
|
||||
Coder custom roles are organization-scoped and cannot grant the site-level `ai_gateway_key` permissions.
|
||||
|
||||
Create a key with a descriptive name:
|
||||
@@ -89,7 +84,7 @@ coder ai-gateway keys create standalone-production
|
||||
```
|
||||
|
||||
The command displays the plaintext key once.
|
||||
Save it immediately in your secret manager because Coder cannot retrieve it later.
|
||||
Save it immediately in your secret manager, because you cannot retrieve it later.
|
||||
Coder stores only a short prefix of the key for display and a SHA-256 hash for authentication, never the full secret.
|
||||
|
||||
Names must be unique, 64 characters or fewer, and use only lowercase letters, numbers, and hyphens.
|
||||
@@ -102,7 +97,7 @@ Configure the standalone process with either of the following options, but not b
|
||||
- `CODER_AI_GATEWAY_KEY_FILE` or `--key-file` reads the key from a file.
|
||||
|
||||
A user login and `CODER_SESSION_TOKEN` are not used by `coder ai-gateway start`.
|
||||
The same Gateway key can authenticate multiple replicas.
|
||||
The same gateway key can authenticate multiple replicas.
|
||||
Separate keys make it easier to rotate or revoke each deployment independently.
|
||||
|
||||
List keys and the most recent heartbeat for each:
|
||||
@@ -115,14 +110,14 @@ A replica records a heartbeat when its control connection is established, then r
|
||||
The heartbeat reports control-connection liveness rather than client request volume.
|
||||
Coder stores one timestamp per key, so replicas that share a key cannot be distinguished.
|
||||
|
||||
For usage and flags, refer to the generated CLI reference for [creating](../../reference/cli/ai-gateway_keys_create.md), [listing](../../reference/cli/ai-gateway_keys_list.md), and [deleting](../../reference/cli/ai-gateway_keys_delete.md) Gateway keys.
|
||||
For usage and flags, refer to the generated CLI reference for [creating](../../reference/cli/ai-gateway_keys_create.md), [listing](../../reference/cli/ai-gateway_keys_list.md), and [deleting](../../reference/cli/ai-gateway_keys_delete.md) gateway keys.
|
||||
|
||||
### Rotate a Gateway key
|
||||
### Rotate a gateway key
|
||||
|
||||
Rotate a key with a rolling restart.
|
||||
Run more than 1 replica behind a load balancer so client traffic continues during the rollout:
|
||||
|
||||
1. Create a new Gateway key.
|
||||
1. Create a new gateway key.
|
||||
1. Update the Kubernetes Secret, environment variable, or key file used by every replica.
|
||||
1. Restart or roll out the standalone deployment so every replica uses the new key.
|
||||
1. Verify readiness and confirm that the new key has a recent heartbeat.
|
||||
@@ -160,8 +155,7 @@ while AI Gateway continues to provide observability and governance.
|
||||
In BYOK mode, users need two credentials:
|
||||
|
||||
- A Coder API token to authenticate with AI Gateway.
|
||||
- Their own LLM credential, such as a personal API key or subscription token,
|
||||
which AI Gateway forwards to the upstream provider.
|
||||
- Their own LLM credential, such as a personal API key or subscription token, which AI Gateway forwards to the upstream provider.
|
||||
|
||||
BYOK and centralized modes can be used together.
|
||||
When a user provides their own credential, AI Gateway forwards it directly.
|
||||
@@ -173,26 +167,23 @@ while allowing individual users to bring their own key.
|
||||
> When a BYOK credential is present, [key failover](./providers.md#key-failover)
|
||||
> is skipped.
|
||||
|
||||
Coder Agents requests routed through AI Gateway are in-process control plane
|
||||
requests, not external client requests that send their own AI Gateway bearer token.
|
||||
Coder Agents requests routed through AI Gateway are in-process control plane requests, not external client requests that send their own AI Gateway bearer token.
|
||||
Coder Agents use the same global BYOK setting.
|
||||
When BYOK is enabled, users can save personal API keys for any enabled AI provider
|
||||
from the Agents settings page.
|
||||
Refer to [Agents credential selection](../agents/models.md#credential-selection)
|
||||
for the Agents-specific behavior.
|
||||
When BYOK is enabled, users can save personal API keys for any enabled AI provider from the Agents settings page.
|
||||
Refer to [Agents credential selection](../agents/models.md#credential-selection) for the Agents-specific behavior.
|
||||
|
||||
Visit individual [client pages](./clients/index.md) for configuration details.
|
||||
|
||||
### Enable or disable BYOK
|
||||
|
||||
BYOK is enabled by default.
|
||||
Administrators can disable it for the embedded Gateway with `--ai-gateway-allow-byok=false` or `CODER_AI_GATEWAY_ALLOW_BYOK=false`:
|
||||
Administrators can disable it for the embedded gateway with `--ai-gateway-allow-byok=false` or `CODER_AI_GATEWAY_ALLOW_BYOK=false`:
|
||||
|
||||
```sh
|
||||
coder server --ai-gateway-allow-byok=false
|
||||
```
|
||||
|
||||
For a standalone Gateway, set the option on each replica:
|
||||
For a standalone gateway, set the option on each replica:
|
||||
|
||||
```sh
|
||||
CODER_AI_GATEWAY_ALLOW_BYOK=false coder ai-gateway start
|
||||
|
||||
@@ -42,7 +42,7 @@ Copilot CLI is built on Node.js and uses the `NODE_EXTRA_CA_CERTS` environment v
|
||||
export NODE_EXTRA_CA_CERTS="/path/to/coder-ai-gateway-proxy-ca.pem"
|
||||
```
|
||||
|
||||
See [Client Configuration CA certificate trust](../ai-gateway-proxy/setup.md#trusting-the-ca-certificate) for details on how to obtain the certificate file.
|
||||
Refer to [Client Configuration CA certificate trust](../ai-gateway-proxy/setup.md#trust-the-ca-certificate) for details on how to obtain the certificate file.
|
||||
|
||||
When [TLS is enabled](../ai-gateway-proxy/setup.md#proxy-tls-configuration) on the proxy, combine the MITM CA certificate and the TLS certificate into a single file:
|
||||
|
||||
@@ -62,7 +62,7 @@ For installation instructions, see [Installing the GitHub Copilot extension in V
|
||||
### Proxy configuration
|
||||
|
||||
You can configure the proxy using environment variables or VS Code settings.
|
||||
For environment variables, see [AI Gateway Proxy client configuration](../ai-gateway-proxy/setup.md#configuring-the-proxy).
|
||||
For environment variables, visit [AI Gateway Proxy client configuration](../ai-gateway-proxy/setup.md#configure-the-proxy).
|
||||
|
||||
Alternatively, you can configure the proxy directly in VS Code settings:
|
||||
|
||||
@@ -92,11 +92,11 @@ For more details, see [Configuring proxy settings for Copilot](https://docs.gith
|
||||
Add the AI Gateway Proxy CA certificate to your operating system's trust store.
|
||||
By default, VS Code loads system certificates, controlled by the `http.systemCertificates` setting.
|
||||
|
||||
See [Client Configuration CA certificate trust](../ai-gateway-proxy/setup.md#trusting-the-ca-certificate) for details on how to obtain the certificate file.
|
||||
Check out [Client Configuration CA certificate trust](../ai-gateway-proxy/setup.md#trust-the-ca-certificate) for details on how to obtain the certificate file.
|
||||
|
||||
When [TLS is enabled](../ai-gateway-proxy/setup.md#proxy-tls-configuration) on the proxy, add the TLS certificate to the system trust store as well.
|
||||
|
||||
### Using Coder Remote extension
|
||||
### Coder Remote extension
|
||||
|
||||
When connecting to a Coder workspace with the [Coder extension](https://marketplace.visualstudio.com/items?itemName=coder.coder-remote), the Copilot extension runs inside the Coder workspace and not on your local machine.
|
||||
This means proxy and certificate configuration must be done in the Coder workspace environment.
|
||||
@@ -119,7 +119,7 @@ Replace `<proxy-host>` with your AI Gateway Proxy hostname and `<CODER_API_TOKEN
|
||||
|
||||
#### CA certificate trust
|
||||
|
||||
Since the Copilot extension runs inside the Coder workspace, add the [AI Gateway Proxy CA certificate](../ai-gateway-proxy/setup.md#trusting-the-ca-certificate) to the Coder workspace's system trust store.
|
||||
Since the Copilot extension runs inside the Coder workspace, add the [AI Gateway Proxy CA certificate](../ai-gateway-proxy/setup.md#trust-the-ca-certificate) to the Coder workspace's system trust store.
|
||||
See [System trust store](../ai-gateway-proxy/setup.md#system-trust-store) for instructions on how to do this on Linux.
|
||||
|
||||
Restart VS Code for changes to take effect.
|
||||
@@ -161,4 +161,4 @@ Alternatively, you can configure the IDE to accept the certificate:
|
||||
|
||||
For more details, see [Trusted root certificates](https://www.jetbrains.com/help/idea/ssl-certificates.html) in the JetBrains documentation.
|
||||
|
||||
See [Client Configuration CA certificate trust](../ai-gateway-proxy/setup.md#trusting-the-ca-certificate) for details on how to obtain the certificate file.
|
||||
See [Client Configuration CA certificate trust](../ai-gateway-proxy/setup.md#trust-the-ca-certificate) for details on how to obtain the certificate file.
|
||||
|
||||
@@ -26,10 +26,7 @@ The exact configuration method varies by client, some use environment variables,
|
||||
|
||||
Replace `coder.example.com` with your actual Coder deployment URL.
|
||||
|
||||
If you run a [standalone AI Gateway](../standalone.md), point clients at the
|
||||
Gateway endpoint and drop the `/api/v2/ai-gateway` prefix, for example
|
||||
`https://ai-gateway.example.com/openai/v1` or
|
||||
`https://ai-gateway.example.com/anthropic`.
|
||||
If you run a [standalone AI gateway](../standalone.md), point clients at the gateway endpoint and drop the `/api/v2/ai-gateway` path prefix (for example, `https://ai-gateway.example.com/openai/v1` or `https://ai-gateway.example.com/anthropic`).
|
||||
|
||||
## Authentication
|
||||
|
||||
@@ -63,7 +60,7 @@ The table below shows tested AI clients and their compatibility with AI Gateway.
|
||||
|
||||
*Legend: ✅ supported, ⚙️ requires AI Gateway Proxy, ❌ not supported, - not applicable.*
|
||||
|
||||
## Configuring In-Workspace Tools
|
||||
## Configure in-workspace tools
|
||||
|
||||
AI coding tools running inside a Coder workspace, such as IDE extensions, can be configured to use AI Gateway.
|
||||
|
||||
@@ -94,7 +91,8 @@ resource "coder_agent" "dev" {
|
||||
|
||||
You can also configure AI tools running outside of a Coder workspace, such as local IDE extensions or desktop applications, to connect to AI Gateway. Use the same settings as the in-workspace case, configure the [base URL](#base-urls) and authenticate with a Coder API token.
|
||||
|
||||
For base URL setup, the client machine must have network access to the AI Gateway endpoint on your Coder deployment. Clients using [AI Gateway Proxy](../ai-gateway-proxy/index.md) must be able to reach the proxy endpoint and trust its CA certificate.
|
||||
For base URL setup, the client machine must have network access to the gateway endpoint on your Coder deployment.
|
||||
Clients using [AI Gateway Proxy](../ai-gateway-proxy/index.md) must be able to reach the proxy endpoint and trust its CA certificate.
|
||||
|
||||
Users can generate a long-lived API token from the Coder UI or CLI. Follow the instructions at [Sessions and API tokens](../../../admin/users/sessions-tokens.md#generate-a-long-lived-api-token-on-behalf-of-yourself) to create one.
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@ AI Gateway is a smart gateway for AI. It acts as an intermediary between your us
|
||||
and providers like OpenAI and Anthropic. By intercepting all the AI traffic between these clients and
|
||||
the upstream APIs, AI Gateway can record user prompts, token usage, and tool invocations.
|
||||
AI Gateway supports clients running inside or outside Coder workspaces.
|
||||
You can run the Gateway inside the Coder control plane (`coderd`) or as a [standalone service](./standalone.md).
|
||||
You can run the gateway inside the Coder control plane (`coderd`) or as a [standalone service](./standalone.md).
|
||||
|
||||
AI Gateway solves 3 key problems:
|
||||
|
||||
@@ -14,7 +14,7 @@ AI Gateway solves 3 key problems:
|
||||
Users use their Coder session or API tokens to authenticate with `coderd`, and
|
||||
`coderd` securely communicates with the upstream APIs on their behalf.
|
||||
1. **Auditing and attribution**: all interactions with AI services, whether autonomous or human-initiated,
|
||||
will be audited and attributed back to a user.
|
||||
are audited and attributed back to a user.
|
||||
1. **Centralized MCP administration**: define a set of approved MCP servers and tools which your users may
|
||||
use.
|
||||
|
||||
|
||||
@@ -33,7 +33,7 @@ CODER_EXTERNAL_AUTH_0_CLIENT_SECRET=...
|
||||
CODER_EXTERNAL_AUTH_0_MCP_URL=https://api.githubcopilot.com/mcp/
|
||||
```
|
||||
|
||||
Refer to the diagram in [Embedded Gateway](./reference.md#embedded-gateway) for more information.
|
||||
Refer to the diagram in [Embedded gateway](./reference.md#embedded-gateway) for more information.
|
||||
|
||||
You can also control which tools are injected by using an allow and/or a deny regular expression on the tool names:
|
||||
|
||||
@@ -63,7 +63,13 @@ AI Gateway marks automatically injected tools with a prefix `bmcp_` ("bridged MC
|
||||
|
||||
## Tool Injection
|
||||
|
||||
If a model decides to invoke a tool and it has a `bmcp_` prefix and AI Gateway has a connection with the related MCP server, it will invoke the tool. The tool result will be passed back to the upstream AI provider, and this will loop until the model has all of its required data. These inner loops are not relayed back to the client. The client only sees the result of this loop. Refer to [Deployment topologies](./reference.md#deployment-topologies).
|
||||
If a model decides to invoke a tool with a `bmcp_` prefix and AI Gateway has a connection to the related MCP server, AI Gateway invokes the tool.
|
||||
AI Gateway passes the tool result back to the upstream AI provider, and this loop continues until the model has all required data.
|
||||
These inner loops are not relayed back to the client.
|
||||
The client only sees the result of the loop.
|
||||
Injected MCP works with both embedded and standalone gateways.
|
||||
Control connection provides each standalone replica with the required configuration and access tokens.
|
||||
Refer to [Deployment topologies](./reference.md#deployment-topologies) for more information.
|
||||
|
||||
In contrast, tools which are defined by the client (i.e. the [`Bash` tool](https://docs.claude.com/en/docs/claude-code/settings#tools-available-to-claude) defined by _Claude Code_) cannot be invoked by AI Gateway, and the tool call from the model will be relayed to the client, after which it will invoke the tool.
|
||||
|
||||
|
||||
@@ -10,18 +10,18 @@ AI Gateway records the last `user` prompt, token usage, model reasoning, and eve
|
||||
|
||||
|
||||
|
||||
Coder provides an example Grafana dashboard that you can import as a starting point for your metrics.
|
||||
We provide an example Grafana dashboard that you can import as a starting point for your metrics.
|
||||
Refer to the [Grafana dashboard README](../../../examples/monitoring/dashboards/grafana/aibridge/README.md).
|
||||
|
||||
These logs and metrics can be used to determine usage patterns, track costs, and evaluate tooling adoption.
|
||||
|
||||
## Prometheus metrics
|
||||
|
||||
The embedded Gateway and [standalone Gateway](./standalone.md) export the same AI Gateway request metrics.
|
||||
The embedded gateway and [standalone gateway](./standalone.md) export the same AI Gateway request metrics.
|
||||
Each process exports metrics for the traffic that it handles:
|
||||
|
||||
- The Coder control plane (`coderd`) Prometheus listener exports metrics for the embedded Gateway.
|
||||
- Each standalone Gateway replica exports metrics from its own Prometheus listener.
|
||||
- The Coder control plane (`coderd`) Prometheus listener exports metrics for the embedded gateway.
|
||||
- Each standalone gateway replica exports metrics from its own Prometheus listener.
|
||||
|
||||
Refer to [provider configuration](./providers.md) for the provider reload lifecycle these metrics describe.
|
||||
|
||||
@@ -83,10 +83,10 @@ Refer to the [Prometheus reference](../../admin/integrations/prometheus.md) for
|
||||
### Metric name migration
|
||||
|
||||
> [!IMPORTANT]
|
||||
> The embedded Gateway metric prefix changed from `coder_aibridged_*` to `coder_ai_gateway_*`, and the proxy prefix changed from `coder_aibridgeproxyd_*` to `coder_ai_gateway_proxy_*`.
|
||||
> The embedded Gateway and AI Gateway Proxy emit the legacy names with identical values during the v2.35 and v2.36 deprecation window, and the legacy names are planned for removal in v2.37.
|
||||
> The embedded gateway metric prefix changed from `coder_aibridged_*` to `coder_ai_gateway_*`, and the proxy prefix changed from `coder_aibridgeproxyd_*` to `coder_ai_gateway_proxy_*`.
|
||||
> The embedded gateway and AI Gateway Proxy emit the legacy names with identical values during the v2.35 and v2.36 deprecation window, and the legacy names are planned for removal in v2.37.
|
||||
> The cost control metrics were added after the rename and have no legacy alias.
|
||||
> The standalone Gateway emits only the current `coder_ai_gateway_*` names.
|
||||
> The standalone gateway emits only the current `coder_ai_gateway_*` names.
|
||||
> Migrate dashboards and alerts to the new names.
|
||||
> Do not relabel new names back to old names while both are emitted because this creates duplicate legacy series in the same scrape.
|
||||
> After the legacy names are removed, use `metric_relabel_configs` only if you need a temporary compatibility bridge:
|
||||
@@ -123,7 +123,7 @@ Alert when the provider reload loop is firing but failing to refresh the pool fo
|
||||
|
||||
Use the `coder_ai_gateway_proxy_*` metrics when you alert on AI Gateway Proxy.
|
||||
|
||||
## Standalone Gateway monitoring
|
||||
## Standalone gateway monitoring
|
||||
|
||||
### Metrics listener
|
||||
|
||||
@@ -137,7 +137,7 @@ In addition to the common `coder_ai_gateway_*` metrics, the standalone listener
|
||||
|
||||
The [AI Gateway Helm chart](../../../helm/ai-gateway/README.md#metrics) enables metrics and binds the listener to `0.0.0.0:2112` by default.
|
||||
The chart exposes a named `metrics` container port, but it does not include this port in the data-plane Service or create monitoring discovery resources.
|
||||
For Prometheus pod-based discovery, add scrape annotations to each Gateway pod:
|
||||
For Prometheus pod-based discovery, add scrape annotations to each gateway pod:
|
||||
|
||||
```yaml
|
||||
coder:
|
||||
@@ -153,14 +153,15 @@ Refer to the [Helm chart metrics configuration](../../../helm/ai-gateway/README.
|
||||
|
||||
### Health and readiness
|
||||
|
||||
A standalone AI Gateway exposes health endpoints on its data-plane listener:
|
||||
A standalone AI gateway exposes health endpoints on its data-plane listener:
|
||||
|
||||
| Endpoint | Success condition |
|
||||
|------------|-----------------------------------------------------------------------------------------------|
|
||||
| `/healthz` | The HTTP listener is serving. |
|
||||
| `/readyz` | The control connection to `coderd` is active and provider configuration has been initialized. |
|
||||
| Endpoint | Success condition |
|
||||
|------------|----------------------------------------------------------------------------------------------------------|
|
||||
| `/healthz` | The HTTP listener is serving. |
|
||||
| `/readyz` | The control connection to `coderd` is active and the initial provider configuration fetch has completed. |
|
||||
|
||||
`/readyz` returns HTTP 503 until provider configuration is initialized and whenever the control connection to `coderd` is unavailable.
|
||||
`/readyz` returns HTTP 503 until the initial provider configuration fetch completes and whenever the control connection to `coderd` is unavailable.
|
||||
The initial fetch can return an empty provider set.
|
||||
A `200 OK` response from `/healthz` only means the HTTP listener is accepting connections. It returns `200 OK` even when the control connection is down.
|
||||
Both endpoints are unauthenticated, bypass the concurrency, rate limiting, and BYOK middleware, and do not create trace spans.
|
||||
|
||||
@@ -178,7 +179,7 @@ Refer to the [`coder ai-gateway start` logging options](../../reference/cli/ai-g
|
||||
### Structured interception logs
|
||||
|
||||
AI Gateway can emit a structured log for every interception record to an external SIEM or observability platform.
|
||||
The `CODER_AI_GATEWAY_STRUCTURED_LOGGING` setting belongs to `coderd`, standalone Gateway does not consume it.
|
||||
The `CODER_AI_GATEWAY_STRUCTURED_LOGGING` setting belongs to `coderd`, and the standalone gateway does not consume it.
|
||||
Standalone replicas send interception records to `coderd`, which writes the structured logs to the Coder server log output.
|
||||
Refer to [structured logging](./setup.md#structured-logging) for configuration and record types.
|
||||
|
||||
@@ -242,7 +243,7 @@ in the AI Gateway setup guide.
|
||||
## Tracing
|
||||
|
||||
AI Gateway supports tracing through [OpenTelemetry](https://opentelemetry.io/) for request processing, upstream API calls, and MCP server interactions.
|
||||
Embedded Gateway spans are emitted by the `coder server` process.
|
||||
Embedded gateway spans are emitted by the `coder server` process.
|
||||
Standalone spans are emitted independently by every replica with the service name `coder-ai-gateway`.
|
||||
|
||||
### Enable tracing
|
||||
@@ -252,8 +253,8 @@ The exporter always dials without TLS, so an `https://` endpoint is still contac
|
||||
`CODER_TRACE_HONEYCOMB_API_KEY` adds a Honeycomb exporter and works with or without `CODER_TRACE_ENABLE`.
|
||||
Set only the Honeycomb key to export to Honeycomb alone, or set both to export to Honeycomb and an OTLP collector.
|
||||
|
||||
The embedded and standalone Gateways share the same tracing options.
|
||||
Refer to the [`coder server` tracing options](../../reference/cli/server.md#--trace) for the embedded Gateway and the [`coder ai-gateway start` tracing options](../../reference/cli/ai-gateway_start.md#--trace) for standalone replicas.
|
||||
The embedded and standalone gateways support the same tracing configuration options, but each standalone replica must be configured separately.
|
||||
Refer to the [`coder server` tracing options](../../reference/cli/server.md#--trace) for the embedded gateway and the [`coder ai-gateway start` tracing options](../../reference/cli/ai-gateway_start.md#--trace) for standalone replicas.
|
||||
Configure tracing on every standalone process or through `coder.env` in the AI Gateway Helm chart.
|
||||
|
||||
The following minimal configuration enables tracing and exports spans over OTLP/gRPC:
|
||||
@@ -263,7 +264,7 @@ export CODER_TRACE_ENABLE=true
|
||||
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://otel-collector:4317
|
||||
```
|
||||
|
||||
In both deployment modes, each request to the Gateway's LLM API endpoint creates an HTTP request span, including requests that are passed through or rejected instead of intercepted.
|
||||
In both deployment modes, each request to the gateway's LLM API endpoint creates an HTTP request span, including requests that are passed through or rejected instead of intercepted.
|
||||
|
||||
### Traced operations
|
||||
|
||||
|
||||
@@ -38,9 +38,8 @@ After seeding, manage providers through the dashboard or API. A provider
|
||||
that has been edited or removed there is not recreated or overwritten
|
||||
from the environment on the next restart.
|
||||
|
||||
Seeding is a `coderd` operation. A [standalone Gateway](./standalone.md)
|
||||
ignores the deprecated provider variables and fetches provider
|
||||
configuration from `coderd`.
|
||||
Seeding is a `coderd` operation.
|
||||
A [standalone gateway](./standalone.md) ignores the deprecated provider variables and fetches provider configuration from `coderd`.
|
||||
|
||||
## Provider types
|
||||
|
||||
@@ -146,9 +145,9 @@ AI Gateway resolves AWS credentials one of three ways:
|
||||
that role before calling Bedrock, signing requests with the resulting
|
||||
temporary credentials. This works on top of either of the above base
|
||||
identities and supports cross-account Bedrock access. See
|
||||
[Assuming an IAM role](#assuming-an-iam-role).
|
||||
[IAM role assumption](#iam-role-assumption).
|
||||
|
||||
#### Obtaining static Bedrock credentials
|
||||
#### Static Bedrock credentials
|
||||
|
||||
When you cannot use the default credential chain, create a dedicated IAM
|
||||
user and generate a static access key:
|
||||
@@ -175,7 +174,7 @@ user and generate a static access key:
|
||||
[AI Providers API](../../reference/api/aiproviders.md), along with the
|
||||
region (or base URL) and model identifiers.
|
||||
|
||||
#### Assuming an IAM role
|
||||
#### IAM role assumption
|
||||
|
||||
Set the optional **Role ARN** field to have the gateway assume an IAM
|
||||
role before calling Bedrock. The base identity (static credentials or the
|
||||
@@ -303,14 +302,12 @@ attempt and each successful reload, exposed as Prometheus metrics:
|
||||
|
||||
If you run the [external proxy](./ai-gateway-proxy/index.md), it exposes
|
||||
the same pair under the `coder_ai_gateway_proxy_` prefix.
|
||||
Each [standalone Gateway](./standalone.md) replica reloads providers
|
||||
independently.
|
||||
Each [standalone gateway](./standalone.md) replica reloads providers independently and replaces only its own provider snapshot.
|
||||
If a reload fails, that replica retains its previous provider snapshot and continues serving from it.
|
||||
|
||||
A growing gap between the attempt and success timestamps means reloads
|
||||
are firing but failing to apply. Alert on that gap rather than on a
|
||||
single failure, which may resolve on the next change. See
|
||||
[Monitoring](./monitoring.md#prometheus-metrics) for the full metric list
|
||||
and sample alert queries.
|
||||
A growing gap between the attempt and success timestamps means reloads are firing but failing to apply.
|
||||
Alert on that gap rather than on a single failure, which may resolve on the next change.
|
||||
Refer to [Monitoring](./monitoring.md#prometheus-metrics) for the full metric list and sample alert queries.
|
||||
|
||||
## Key failover
|
||||
|
||||
|
||||
@@ -7,26 +7,26 @@
|
||||
## Deployment topologies
|
||||
|
||||
AI Gateway can run inside `coderd` or as a standalone data-plane service.
|
||||
Both topologies run the same Gateway request handling and keep `coderd` as the source of truth for Coder API key validation, provider configuration, and AI session records.
|
||||
They differ in how requests are routed to the Gateway.
|
||||
Both topologies run the same gateway request handling and keep `coderd` as the source of truth for Coder API key validation, provider configuration, and AI session records.
|
||||
They differ in how requests are routed to the gateway.
|
||||
|
||||
### Embedded Gateway
|
||||
### Embedded gateway
|
||||
|
||||
By default, `coder server` runs an in-memory Gateway instance in the `coderd` process.
|
||||
By default, `coder server` runs an in-memory gateway instance in the `coderd` process.
|
||||
AI clients send requests to `<Coder access URL>/api/v2/ai-gateway/<provider-name>/`.
|
||||
The embedded Gateway uses the same control RPC as a standalone deployment, over an in-process transport rather than a network connection.
|
||||
It does not use a Gateway key and does not negotiate an API version.
|
||||
The embedded gateway uses the same control RPC as a standalone deployment, over an in-process transport rather than a network connection.
|
||||
It does not use a gateway key and does not negotiate an AI Gateway API version.
|
||||
|
||||
The following diagram shows the embedded topology:
|
||||
|
||||
|
||||
|
||||
### Standalone Gateway
|
||||
### Standalone gateway
|
||||
|
||||
A [standalone deployment](./standalone.md) runs the AI traffic data plane outside the `coderd` process.
|
||||
Each replica accepts client traffic, sends AI requests directly to upstream providers, and maintains a control connection to `coderd` using a [Gateway key](./standalone.md#create-a-gateway-key).
|
||||
Each replica accepts client traffic, sends AI requests directly to upstream providers, and maintains a control connection to `coderd` by using a [gateway key](./standalone.md#create-a-gateway-key).
|
||||
|
||||
The control connection carries:
|
||||
The control connection carries the following:
|
||||
|
||||
- Coder API key validation, which resolves each request to an active Coder user.
|
||||
- AI budget checks, which reject requests from users over their spend limit.
|
||||
@@ -34,25 +34,25 @@ The control connection carries:
|
||||
- AI session records.
|
||||
- **Deprecated**: the configuration and access tokens used by [injected MCP](./mcp.md).
|
||||
|
||||
Standalone replicas do not own authoritative database state.
|
||||
Standalone replicas have no authoritative database state.
|
||||
They keep ephemeral provider snapshots, request caches, provider key pools, and metrics in memory, and emit their own logs and traces.
|
||||
Each replica writes its own [API dumps](./setup.md#api-dumps) to its own local disk when dumps are enabled.
|
||||
When API dumps are enabled, each replica writes its own [API dumps](./setup.md#api-dumps) to its own local disk.
|
||||
|
||||
`coderd` remains required for standalone operation.
|
||||
A replica becomes unready when its control connection is unavailable, even if its HTTP listener remains healthy.
|
||||
AI Gateway Proxy remains part of `coder server` and can forward its intercepted traffic to either the embedded Gateway or a standalone endpoint.
|
||||
AI Gateway Proxy remains part of `coder server` and can forward its intercepted traffic to either the embedded gateway or a standalone endpoint.
|
||||
|
||||
## Version compatibility
|
||||
|
||||
The control connection between a standalone replica and `coderd` is versioned.
|
||||
The current version is defined in [`coderd/aibridged/proto/version.go`](https://github.com/coder/coder/blob/main/coderd/aibridged/proto/version.go).
|
||||
The control connection between a standalone replica and `coderd` uses a versioned AI Gateway API.
|
||||
The current AI Gateway API version is defined in [`coderd/aibridged/proto/version.go`](../../../coderd/aibridged/proto/version.go).
|
||||
|
||||
`coderd` validates the version that a standalone replica advertises before it accepts the control connection.
|
||||
Compatibility follows these rules:
|
||||
`coderd` validates the AI Gateway API version that a standalone replica advertises before it accepts the control connection.
|
||||
AI Gateway API compatibility follows these rules:
|
||||
|
||||
- The Gateway and `coderd` major versions must match.
|
||||
- The Gateway minor version must be less than or equal to the `coderd` minor version.
|
||||
- `coderd` rejects a standalone Gateway that advertises a newer minor version.
|
||||
- The gateway and `coderd` AI Gateway API major versions must match.
|
||||
- The gateway's AI Gateway API minor version must be less than or equal to the `coderd` instance's AI Gateway API minor version.
|
||||
- `coderd` rejects a standalone gateway that advertises a newer AI Gateway API minor version.
|
||||
|
||||
A rejected replica receives an HTTP 400 response that reports the `client_api_version` and `server_api_version` values.
|
||||
Coder build versions are not the compatibility criterion.
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Setup
|
||||
|
||||
By default, AI Gateway runs inside the Coder control plane (`coderd`) and requires no separate compute.
|
||||
In embedded mode, `coderd` runs the Gateway in memory and brokers traffic to your configured AI providers on behalf of authenticated users.
|
||||
In embedded mode, `coderd` runs the gateway in memory and brokers traffic to your configured AI providers on behalf of authenticated users.
|
||||
|
||||
If AI traffic needs dedicated compute, independent scaling, or a separate network endpoint, you can [deploy AI Gateway as a standalone service](./standalone.md).
|
||||
|
||||
@@ -15,7 +15,7 @@ If AI traffic needs dedicated compute, independent scaling, or a separate networ
|
||||
## Activation
|
||||
|
||||
The AI Gateway feature must be enabled in the Coder deployment configuration before
|
||||
embedded or standalone Gateway instances can serve authenticated traffic.
|
||||
embedded or standalone gateway instances can serve authenticated traffic.
|
||||
|
||||
_AI Gateway is enabled by default as of v2.34._
|
||||
|
||||
@@ -28,7 +28,7 @@ coder server --ai-gateway-enabled=true
|
||||
|
||||
A standalone process does not read `CODER_AI_GATEWAY_ENABLED` from its own environment.
|
||||
However, this setting must remain enabled on `coderd`.
|
||||
It is required for Gateway key management endpoints to work and for standalone replicas to connect to the control plane.
|
||||
It is required for gateway key management endpoints to work and for standalone replicas to connect to the control plane.
|
||||
|
||||
## Configure Providers
|
||||
|
||||
@@ -91,8 +91,7 @@ with `/var/lib/coder/ai-gateway-dumps` configured writes to
|
||||
Sensitive headers are redacted before dumps are written. Leave the value empty
|
||||
to disable dumping.
|
||||
|
||||
Each [standalone Gateway](./standalone.md) replica accepts the same API dump
|
||||
settings and writes dumps to its own local disk.
|
||||
Each [standalone gateway](./standalone.md) replica accepts the same API dump settings and writes optional dumps to its own local disk.
|
||||
|
||||
> [!WARNING]
|
||||
> API dumps are intended for short diagnostic sessions only. Dump files contain
|
||||
@@ -149,7 +148,7 @@ ingestion, set `--log-json` to a file path or `/dev/stderr` so that records are
|
||||
emitted as JSON.
|
||||
|
||||
This setting belongs to `coderd`.
|
||||
A [standalone Gateway](./standalone.md) does not consume it.
|
||||
A [standalone gateway](./standalone.md) does not consume it.
|
||||
|
||||
Filter for AI Gateway records in your logging pipeline by matching on the
|
||||
`"interception log"` message. Each log line includes a `record_type` field that
|
||||
|
||||
@@ -6,12 +6,12 @@
|
||||
|
||||
When AI traffic needs dedicated compute, independent scaling, or a separate network endpoint, you can deploy AI Gateway separately from the Coder control plane (`coderd`).
|
||||
|
||||
A standalone AI Gateway serves client traffic on its own listener and maintains a control connection to `coderd`.
|
||||
A standalone AI gateway serves client traffic on its own listener and maintains a control connection to `coderd`.
|
||||
`coderd` continues to manage authentication, authorization, provider configuration, and AI session records.
|
||||
|
||||
## Before you begin
|
||||
|
||||
Standalone AI Gateway requires:
|
||||
A standalone AI gateway requires the following:
|
||||
|
||||
- Coder v2.36.0 or later.
|
||||
- A [Premium license with AI Governance](../ai-governance.md).
|
||||
@@ -19,15 +19,15 @@ Standalone AI Gateway requires:
|
||||
- The full Coder image or a Coder binary that includes the `coder ai-gateway start` command.
|
||||
|
||||
`coder ai-gateway start` does not read `CODER_AI_GATEWAY_ENABLED` from the standalone process.
|
||||
However, this setting must remain enabled on `coderd` for Gateway key management and standalone control connections.
|
||||
However, this setting must remain enabled on `coderd` for gateway key management and standalone control connections.
|
||||
|
||||
## Create a Gateway key
|
||||
## Create a gateway key
|
||||
|
||||
Each standalone replica uses a Gateway key to authenticate and establish its control connection to `coderd`.
|
||||
Gateway key management requires site-level `ai_gateway_key` permissions, which only the built-in Owner role includes.
|
||||
Each standalone replica uses a gateway key to authenticate and establish its control connection to `coderd`.
|
||||
Gateway key management requires site-level `ai_gateway_key` permissions, which only the built-in **owner** role includes.
|
||||
Coder custom roles are organization-scoped and cannot grant site-level permissions.
|
||||
|
||||
Log in to the Coder CLI as an Owner or another user with these permissions, then create a dedicated key for the standalone deployment:
|
||||
Log in to the Coder CLI as an **owner**, then create a dedicated key for the standalone deployment:
|
||||
|
||||
```sh
|
||||
coder login https://coder.example.com
|
||||
@@ -41,24 +41,24 @@ For independent rotation and revocation, use a separate key for each standalone
|
||||
|
||||
## Start a standalone process
|
||||
|
||||
Set the Coder URL, Gateway key, and listener address, then start the Gateway:
|
||||
Set the Coder URL, gateway key, and listener address, then start the gateway:
|
||||
|
||||
```sh
|
||||
export CODER_URL=https://coder.example.com
|
||||
export CODER_AI_GATEWAY_KEY='<AI Gateway key>'
|
||||
export CODER_AI_GATEWAY_KEY='<gateway_key>'
|
||||
export CODER_AI_GATEWAY_HTTP_ADDRESS=0.0.0.0:4001
|
||||
coder ai-gateway start
|
||||
```
|
||||
|
||||
Use `CODER_AI_GATEWAY_KEY_FILE` instead of `CODER_AI_GATEWAY_KEY` to read the key from a file.
|
||||
The standalone process does not require a user login or `CODER_SESSION_TOKEN` after you provide the Gateway key.
|
||||
The standalone process does not require a user login or `CODER_SESSION_TOKEN` after you provide the gateway key.
|
||||
|
||||
The listener defaults to `127.0.0.1:4001`, which accepts connections only from the local host.
|
||||
Set `CODER_AI_GATEWAY_HTTP_ADDRESS` to a routable address, as shown above, before other hosts or pods can reach the Gateway.
|
||||
Set `CODER_AI_GATEWAY_HTTP_ADDRESS` to a routable address, as shown previously, before other hosts or pods can reach the gateway.
|
||||
|
||||
The standalone Gateway fetches provider configuration from `coderd`.
|
||||
Configure at least one [AI provider](./providers.md) in Coder before sending provider traffic through the Gateway.
|
||||
The standalone Gateway does not use the deprecated [provider seed variables](./providers.md#database-management-of-providers).
|
||||
The standalone gateway fetches provider configuration from `coderd`.
|
||||
Configure at least one [AI provider](./providers.md) in Coder before sending provider traffic through the gateway.
|
||||
The standalone gateway does not use the deprecated [provider seed variables](./providers.md#database-management-of-providers).
|
||||
|
||||
The listener uses HTTP by default.
|
||||
Set both `CODER_AI_GATEWAY_TLS_CERT_FILE` and `CODER_AI_GATEWAY_TLS_KEY_FILE` to terminate TLS in the process.
|
||||
@@ -66,20 +66,20 @@ For all command options, refer to [`coder ai-gateway start`](../../reference/cli
|
||||
|
||||
## Run AI Gateway in Kubernetes
|
||||
|
||||
To run the standalone Gateway as a Kubernetes workload, provide the same environment variables and network access to the Coder URL.
|
||||
To run the standalone gateway as a Kubernetes workload, provide the same environment variables and network access to the Coder URL.
|
||||
You can manage the workload with your own Kubernetes manifests or use the provided Helm chart.
|
||||
The chart configures the Deployment, probes, and Service, plus an optional Ingress or `HTTPRoute`.
|
||||
|
||||
To use the Kubernetes examples, install `kubectl` and configure access to the cluster where AI Gateway will run.
|
||||
Install Helm if you use the chart.
|
||||
|
||||
Create a namespace and store the Gateway key in a Kubernetes Secret:
|
||||
Create a namespace and store the gateway key in a Kubernetes Secret:
|
||||
|
||||
```sh
|
||||
kubectl create namespace coder-ai-gateway
|
||||
kubectl create secret generic coder-ai-gateway-key \
|
||||
--namespace coder-ai-gateway \
|
||||
--from-literal=key='<AI Gateway key>'
|
||||
--from-literal=key='<gateway_key>'
|
||||
```
|
||||
|
||||
### Configure the Helm chart
|
||||
@@ -97,7 +97,7 @@ aigateway:
|
||||
name: coder-ai-gateway-key
|
||||
```
|
||||
|
||||
The Gateway must be able to reach the Coder URL from every replica.
|
||||
The gateway must be able to reach the Coder URL from every replica.
|
||||
For an HTTPS URL signed by a private CA, set `aigateway.coderTLS.caSecret.name` to the Secret holding the CA bundle.
|
||||
If Coder requires client mTLS, also set `aigateway.coderTLS.clientSecret.name`.
|
||||
|
||||
@@ -123,7 +123,7 @@ Chart versions omit the leading `v`, so use `2.36.0` rather than `v2.36.0`.
|
||||
|
||||
When you install the chart directly from a Git checkout, set `coder.image.tag` explicitly and install `./helm/ai-gateway`.
|
||||
Released chart packages select the matching Coder image by default.
|
||||
If you need to run different `coderd` and Gateway versions, refer to [Version compatibility](#version-compatibility).
|
||||
If you need to run different `coderd` and gateway versions, refer to [Version compatibility](#version-compatibility).
|
||||
|
||||
### Validate the Helm deployment
|
||||
|
||||
@@ -138,7 +138,7 @@ kubectl rollout status deployment/coder-ai-gateway \
|
||||
The chart configures the following probes by default:
|
||||
|
||||
- The liveness probe requests `/healthz` and verifies that the HTTP listener is serving.
|
||||
- The readiness probe requests `/readyz` and verifies that the Gateway is connected to `coderd` and has completed its initial provider load.
|
||||
- The readiness probe requests `/readyz` and verifies that the gateway is connected to `coderd` and has completed its initial provider load.
|
||||
- The startup probe is disabled by default.
|
||||
|
||||
`/healthz` can return HTTP 200 while `/readyz` returns HTTP 503.
|
||||
@@ -165,7 +165,7 @@ curl --fail http://127.0.0.1:4001/readyz
|
||||
|
||||
Both endpoints return HTTP 200 with an empty body when the replica is serving and ready.
|
||||
|
||||
List Gateway keys and verify that the key has a recent heartbeat:
|
||||
List gateway keys and verify that the key has a recent heartbeat:
|
||||
|
||||
```sh
|
||||
coder ai-gateway keys list
|
||||
@@ -177,7 +177,7 @@ An active control connection updates the timestamp every 60 seconds.
|
||||
Coder stores one timestamp per key, so a recent heartbeat on a shared key does not confirm that every replica is connected.
|
||||
Check `/readyz` on each replica to verify individual health.
|
||||
|
||||
## Route traffic to the standalone Gateway
|
||||
## Route traffic to the standalone gateway
|
||||
|
||||
The standalone deployment does not move traffic automatically.
|
||||
Configure AI Gateway Proxy and direct AI clients to send requests to the standalone endpoint.
|
||||
@@ -205,28 +205,27 @@ The proxy appends the provider name and request path to the target, so do not in
|
||||
|
||||
### Direct AI clients
|
||||
|
||||
Clients that use `<Coder access URL>/api/v2/ai-gateway/<provider-name>/` continue to reach the embedded Gateway.
|
||||
To route these clients to the standalone Gateway, replace the Coder access URL and embedded route prefix with the standalone endpoint while keeping the provider path.
|
||||
Clients that use `<Coder access URL>/api/v2/ai-gateway/<provider-name>/` continue to reach the embedded gateway.
|
||||
To route these clients to the standalone gateway, replace the Coder access URL and the embedded `/api/v2/ai-gateway` path prefix with the standalone endpoint while keeping the provider path.
|
||||
|
||||
For example, if the Gateway is reachable at `https://ai-gateway.example.com` and the providers are configured, configure Claude Code with:
|
||||
For example, if the gateway is reachable at `https://ai-gateway.example.com` and the providers are configured, configure Claude Code with the following:
|
||||
|
||||
```sh
|
||||
ANTHROPIC_BASE_URL=https://ai-gateway.example.com/anthropic
|
||||
```
|
||||
|
||||
Configure an OpenAI-compatible client with:
|
||||
Configure an OpenAI-compatible client with the following:
|
||||
|
||||
```sh
|
||||
OPENAI_BASE_URL=https://ai-gateway.example.com/openai/v1
|
||||
```
|
||||
|
||||
The standalone listener also accepts the equivalent `/api/v2/ai-gateway/<provider-name>/` paths for compatibility.
|
||||
For full per-client configuration examples, refer to [Client Configuration](./clients/index.md).
|
||||
|
||||
### Expose the standalone AI Gateway
|
||||
### Expose the standalone AI gateway
|
||||
|
||||
If clients cannot access the in-cluster Service, expose the Gateway through an Ingress, an `HTTPRoute`, or a load balancer Service.
|
||||
In the provided Helm chart, set:
|
||||
If clients cannot access the in-cluster Service, expose the gateway through an Ingress, an `HTTPRoute`, or a load balancer Service.
|
||||
In the provided Helm chart, set the following:
|
||||
|
||||
```yaml
|
||||
service:
|
||||
@@ -235,16 +234,16 @@ service:
|
||||
|
||||
Use TLS to protect credentials and AI traffic whenever they cross an untrusted network.
|
||||
Prefer TLS termination at the Ingress or Kubernetes Gateway when your platform already manages certificates there.
|
||||
To terminate TLS in the AI Gateway process, set `aigateway.listenerTLS.name` to an existing TLS Secret.
|
||||
To terminate TLS in the gateway process, set `aigateway.listenerTLS.name` to an existing TLS Secret.
|
||||
|
||||
If `CODER_AI_GATEWAY_PROXY_TARGET` uses HTTPS with a private CA, add that CA to the trust store of the `coderd` pods before changing the proxy target.
|
||||
For the Coder Helm chart, you can mount a CA bundle with `coder.certs.secrets`.
|
||||
This trust configuration is separate from `aigateway.coderTLS.caSecret`, which configures the connection from the standalone Gateway to `coderd`.
|
||||
This trust configuration is separate from `aigateway.coderTLS.caSecret`, which configures the connection from the standalone gateway to `coderd`.
|
||||
|
||||
## Scale replicas
|
||||
|
||||
Run multiple replicas behind a Service or load balancer.
|
||||
In the provided Helm chart, set:
|
||||
In the provided Helm chart, set the following:
|
||||
|
||||
```yaml
|
||||
coder:
|
||||
@@ -261,39 +260,39 @@ Sticky load balancing can improve cache efficiency, but it is not required for c
|
||||
When you enable [API dumps](./setup.md#api-dumps), each replica writes dumps to its own local disk.
|
||||
Use persistent storage if you need those files to survive pod replacement.
|
||||
|
||||
The chart requests 1 CPU and 1 GiB of memory per replica and does not set resource limits.
|
||||
The chart requests 1 CPU and 1 GiB of memory per replica and does not set resource limits.
|
||||
Measure production traffic before changing requests, limits, or `CODER_AI_GATEWAY_MAX_CONCURRENCY`.
|
||||
The chart does not create a Horizontal Pod Autoscaler or PodDisruptionBudget.
|
||||
|
||||
## Migrate from the embedded Gateway
|
||||
## Migrate from the embedded gateway
|
||||
|
||||
Use a gradual cutover so that you can validate the standalone data plane before moving production traffic:
|
||||
|
||||
1. Upgrade `coderd` to the target Coder version.
|
||||
1. Keep `CODER_AI_GATEWAY_ENABLED=true` on `coderd`.
|
||||
1. Create a dedicated Gateway key.
|
||||
1. Create a dedicated gateway key.
|
||||
1. Deploy one standalone replica without changing client or proxy routing.
|
||||
1. Verify `/readyz`, the key heartbeat, metrics, logs, and traces.
|
||||
1. Send a test request directly to the standalone Gateway and confirm that the AI session appears in Coder.
|
||||
1. Send a test request directly to the standalone gateway and confirm that the AI session appears in Coder.
|
||||
1. Set `CODER_AI_GATEWAY_PROXY_TARGET` to the standalone endpoint.
|
||||
1. Update direct client base URLs that should use the standalone endpoint.
|
||||
1. Scale the standalone deployment after the canary path is stable.
|
||||
|
||||
Keep `CODER_AI_GATEWAY_ENABLED=true` on `coderd` after the cutover.
|
||||
The setting is required for standalone control connections and Coder features that use the in-process Gateway.
|
||||
The setting is required for standalone control connections and Coder features that use the in-process gateway.
|
||||
|
||||
## Roll back to the embedded Gateway
|
||||
## Roll back to the embedded gateway
|
||||
|
||||
To return traffic to the embedded Gateway:
|
||||
To return traffic to the embedded gateway:
|
||||
|
||||
1. Remove `CODER_AI_GATEWAY_PROXY_TARGET`, or set it to `<Coder access URL>/api/v2/ai-gateway`.
|
||||
1. Restart or upgrade `coderd` so the proxy target change takes effect.
|
||||
1. Restore direct client base URLs to `<Coder access URL>/api/v2/ai-gateway/<provider-name>/`.
|
||||
1. Verify requests and AI session records through the embedded route.
|
||||
1. Scale down or uninstall the standalone deployment.
|
||||
1. Delete the standalone Gateway key after all replicas have stopped.
|
||||
1. Delete the standalone gateway key after all replicas have stopped.
|
||||
|
||||
Delete the key with:
|
||||
Delete the key with the following command:
|
||||
|
||||
```sh
|
||||
coder ai-gateway keys delete standalone-production
|
||||
@@ -306,20 +305,21 @@ Stop the deployment before deleting its key.
|
||||
|
||||
## Version compatibility
|
||||
|
||||
The standalone Gateway image does not need to match the `coderd` image version exactly.
|
||||
The components can connect when their AI Gateway API versions are compatible, and `coderd` checks compatibility whenever a Gateway replica connects.
|
||||
The standalone gateway image does not need to match the `coderd` image version exactly.
|
||||
The components can connect when their AI Gateway API versions are compatible, and `coderd` checks AI Gateway API compatibility whenever a gateway replica connects.
|
||||
|
||||
A replica may run at the same API version as `coderd` or an earlier minor version of the same major version, but never a newer one.
|
||||
The Gateway treats certain handshake failures from `/api/v2/ai-gateway/serve` as fatal and exits rather than retrying:
|
||||
A replica may use the same AI Gateway API version as `coderd` or an earlier minor version of the same major version, but never a newer one.
|
||||
The gateway treats certain handshake failures from `/api/v2/ai-gateway/serve` as fatal and exits rather than retrying:
|
||||
|
||||
- HTTP 400: incompatible API version.
|
||||
- HTTP 401: invalid Gateway key.
|
||||
- HTTP 401: invalid gateway key.
|
||||
- HTTP 403: missing entitlement.
|
||||
- HTTP 404: endpoint not found. `coderd` may be too old and not expose `/serve` endpoint.
|
||||
- HTTP 404: endpoint not found.
|
||||
`coderd` may be too old and not expose the `/serve` endpoint.
|
||||
|
||||
Sequence changes that move both components:
|
||||
|
||||
- To upgrade, upgrade `coderd` first, then roll out the standalone Gateway release.
|
||||
- To roll back, roll back the standalone Gateway first, then roll back `coderd`.
|
||||
- To upgrade, upgrade `coderd` first, then roll out the standalone gateway release.
|
||||
- To roll back, roll back the standalone gateway first, then roll back `coderd`.
|
||||
|
||||
Running the same Coder release on `coderd` and every standalone replica is the simplest way to stay compatible, but it is not required.
|
||||
|
||||
@@ -185,7 +185,7 @@ If you do not have a domain, you should set `CODER_ACCESS_URL` to this URL in
|
||||
the Helm chart and upgrade Coder (see below). This allows workspaces to connect
|
||||
to the proper Coder URL.
|
||||
|
||||
## Upgrading Coder via Helm
|
||||
## Upgrade Coder via Helm
|
||||
|
||||
To upgrade Coder in the future or change values, you can run the following
|
||||
command:
|
||||
@@ -197,13 +197,11 @@ helm upgrade coder coder-v2/coder \
|
||||
-f values.yaml
|
||||
```
|
||||
|
||||
## Standalone AI Gateway Chart
|
||||
## Standalone AI gateway chart
|
||||
|
||||
Coder also publishes a chart, `oci://ghcr.io/coder/chart/coder-ai-gateway`,
|
||||
that runs [AI Gateway](../ai-coder/ai-gateway/index.md) as its own Deployment
|
||||
alongside the control plane. Use it when you want to scale AI traffic
|
||||
independently of `coderd`. For installation and configuration, visit
|
||||
[Standalone AI Gateway](../ai-coder/ai-gateway/standalone.md).
|
||||
Coder also publishes a chart, `oci://ghcr.io/coder/chart/coder-ai-gateway`, that runs [AI Gateway](../ai-coder/ai-gateway/index.md) as its own Deployment alongside the control plane.
|
||||
Use it when you want to scale AI traffic independently of `coderd`.
|
||||
For installation and configuration, refer to [Deploy AI Gateway as a standalone service](../ai-coder/ai-gateway/standalone.md) and the [AI Gateway Helm chart README](../../helm/ai-gateway/README.md).
|
||||
|
||||
## Coder Observability Chart
|
||||
|
||||
|
||||
@@ -46,7 +46,7 @@ docker-compose pull coder && docker-compose up -d coder
|
||||
### Kubernetes
|
||||
|
||||
See
|
||||
[Upgrading Coder via Helm](../install/kubernetes.md#upgrading-coder-via-helm).
|
||||
[Upgrade Coder via Helm](../install/kubernetes.md#upgrade-coder-via-helm).
|
||||
|
||||
### Coder AMI on AWS
|
||||
|
||||
|
||||
Reference in New Issue
Block a user