mirror of
https://github.com/coder/coder.git
synced 2026-10-16 14:48:56 +08:00
docs(docs/ai-coder/ai-gateway): reformat to one sentence per line (#30701)
Reformats the AI Gateway docs directly under `docs/ai-coder/ai-gateway` to one sentence per line, as the [docs style guide](https://github.com/coder/coder/blob/main/docs/.style/style-guide/formatting.md#one-sentence-per-line) requires. Hard-wrapped paragraphs are unwrapped and lines holding several sentences are split, so a later edit to one sentence shows up as a one-line diff. Apart from the exception below, only whitespace inside paragraphs and list items changes, and the rendered pages are the same. Comparing the changed files against `main` word for word, ignoring whitespace and blockquote `>` markers, finds differences only on the reworded lines. `git diff --word-diff` is easier to read than the line diff. 4 lines that already contained an em or en dash were reworded by hand, because `lint/emdash` treats every reflowed line as newly added. Each uses a parenthetical, a separate sentence, or a colon after a lead-in. Dashes on lines the reflow did not touch are left alone. Applied with an in-progress formatter, limited to pages listed in `docs/manifest.json` and excluding generated reference docs. This is one of a batch of PRs that split the docs by area (10 files changed, 395 insertions(+), 570 deletions(-)). --- 🤖 Built with AI assistance.
This commit is contained in:
Nick Vigilante
@@ -3,12 +3,10 @@ title: Audit AI sessions
|
||||
---
|
||||
|
||||
> [!NOTE]
|
||||
> AI Gateway is part of [AI Governance](../ai-governance.md), which is
|
||||
> included with a Premium license.
|
||||
> AI Gateway is part of [AI Governance](../ai-governance.md), which is included with a Premium license.
|
||||
|
||||
AI Gateway groups intercepted requests into **sessions** and **threads** to show
|
||||
the causal relationships between human prompts and agent actions. This
|
||||
structure gives auditors clear provenance over who initiated what, and why.
|
||||
AI Gateway groups intercepted requests into **sessions** and **threads** to show the causal relationships between human prompts and agent actions.
|
||||
This structure gives auditors clear provenance over who initiated what, and why.
|
||||
|
||||
## Concepts
|
||||
|
||||
@@ -21,28 +19,23 @@ structure gives auditors clear provenance over who initiated what, and why.
|
||||
|
||||
## Human vs. Agent attribution
|
||||
|
||||
AI Gateway distinguishes between human-initiated and agent-initiated requests
|
||||
using the `role` property:
|
||||
AI Gateway distinguishes between human-initiated and agent-initiated requests using the `role` property:
|
||||
|
||||
- A message with `role="user"` indicates a human-initiated action (i.e. prompt).
|
||||
- A message with `role="assistant"` indicates a message generated by a model.
|
||||
- A message with `role="system"` indicates the system prompt for the client.
|
||||
|
||||
The `user` role is currently overloaded by clients like Claude Code and Codex;
|
||||
they inject system instructions
|
||||
within `role="user"` blocks when using agents. AI Gateway applies a heuristic
|
||||
of storing only the **last** prompt from a block of `role="user"` messages.
|
||||
The `user` role is currently overloaded by clients like Claude Code and Codex; they inject system instructions within `role="user"` blocks when using agents.
|
||||
AI Gateway applies a heuristic of storing only the **last** prompt from a block of `role="user"` messages.
|
||||
|
||||
> [!NOTE]
|
||||
> AI Gateway cannot declare with certainty whether a request was human- or
|
||||
> agent-initiated.
|
||||
> AI Gateway cannot declare with certainty whether a request was human- or agent-initiated.
|
||||
|
||||
## LLM reasoning capture
|
||||
|
||||
AI Gateway captures model reasoning and thinking content when available. Both
|
||||
Anthropic (extended thinking) and OpenAI (reasoning summaries) support this
|
||||
feature. Reasoning data gives auditors insight into **why** a tool was called,
|
||||
not just what was called.
|
||||
AI Gateway captures model reasoning and thinking content when available.
|
||||
Both Anthropic (extended thinking) and OpenAI (reasoning summaries) support this feature.
|
||||
Reasoning data gives auditors insight into **why** a tool was called, not just what was called.
|
||||
|
||||
<a id="navigating-the-ui"></a>
|
||||
|
||||
@@ -54,13 +47,10 @@ The sessions page (`http://<deployment-url>/ai-gateway/sessions`) lists sessions
|
||||
By default it shows sessions with activity in the last 24 hours.
|
||||
Use the time range filter in the filter bar to widen or narrow the window, for example to see older sessions.
|
||||
|
||||
Each row shows the last prompt, initiator, provider, client, token usage,
|
||||
network requests, thread count, and timestamp.
|
||||
Each row shows the last prompt, initiator, provider, client, token usage, network requests, thread count, and timestamp.
|
||||
|
||||
The **Network Requests** column reports the total and blocked
|
||||
[Agent Firewall](../agent-firewall/index.md) requests for the session. It shows
|
||||
`No activity` when the session made no requests, and `Disabled` when the session
|
||||
did not pass through Agent Firewall, so no network requests were monitored.
|
||||
The **Network Requests** column reports the total and blocked [Agent Firewall](../agent-firewall/index.md) requests for the session.
|
||||
It shows `No activity` when the session made no requests, and `Disabled` when the session did not pass through Agent Firewall, so no network requests were monitored.
|
||||
|
||||
Select one to view its full details.
|
||||
|
||||
@@ -70,30 +60,21 @@ Select one to view its full details.
|
||||
|
||||
Open a session to see a chronological causal chain of events.
|
||||
|
||||
A search input next to the Back button filters the events by prompt text,
|
||||
tool names, tool inputs, and network call destinations. While a query is
|
||||
active, the timeline shows only matching events and a count below the input
|
||||
reports how many events match. Matches render in bold in prompts, tool names,
|
||||
and network call destinations, and a matching prompt expands to show the
|
||||
match. A "No events match your search in the loaded events." message appears
|
||||
when nothing in the loaded events matches.
|
||||
A search input next to the Back button filters the events by prompt text, tool names, tool inputs, and network call destinations.
|
||||
While a query is active, the timeline shows only matching events and a count below the input reports how many events match.
|
||||
Matches render in bold in prompts, tool names, and network call destinations, and a matching prompt expands to show the match.
|
||||
A "No events match your search in the loaded events." message appears when nothing in the loaded events matches.
|
||||
|
||||
Within a thread, each step shows token usage, tool call details (including
|
||||
arguments and MCP server URLs), duration, and any errors or warnings.
|
||||
Within a thread, each step shows token usage, tool call details (including arguments and MCP server URLs), duration, and any errors or warnings.
|
||||
|
||||
The **Session summary** card beside the timeline reports the session's
|
||||
[Agent Firewall](../agent-firewall/index.md) activity:
|
||||
The **Session summary** card beside the timeline reports the session's [Agent Firewall](../agent-firewall/index.md) activity:
|
||||
|
||||
- **Network requests** is the total number of requests the session made. It
|
||||
shows `Disabled` when the session did not pass through Agent Firewall, so
|
||||
monitoring was not active, and `No activity` when the session made no
|
||||
requests.
|
||||
- **Blocked network requests** is the subset of those requests that the
|
||||
allow-list denied. This row appears only when the session made at least one
|
||||
request, and is highlighted when any were blocked.
|
||||
- **Top domains** names the destination host the session contacted most, and
|
||||
appears whenever at least one domain was recorded. When the session contacted
|
||||
more than one distinct domain, a `+N more` count follows.
|
||||
- **Network requests** is the total number of requests the session made.
|
||||
It shows `Disabled` when the session did not pass through Agent Firewall, so monitoring was not active, and `No activity` when the session made no requests.
|
||||
- **Blocked network requests** is the subset of those requests that the allow-list denied.
|
||||
This row appears only when the session made at least one request, and is highlighted when any were blocked.
|
||||
- **Top domains** names the destination host the session contacted most, and appears whenever at least one domain was recorded.
|
||||
When the session contacted more than one distinct domain, a `+N more` count follows.
|
||||
|
||||
|
||||
|
||||
@@ -103,17 +84,17 @@ The **Session summary** card beside the timeline reports the session's
|
||||
|
||||
When investigating an incident (policy violation, destructive action, etc.):
|
||||
|
||||
1. **Identify the session.** Filter by user, time range, or client to find the
|
||||
relevant session.
|
||||
1. **Locate the thread.** Each thread in a session shows the (likely) human prompt
|
||||
that initiated the chain of actions.
|
||||
1. **Trace the causal chain.** Expand the thread to see every step in the
|
||||
agentic loop — each tool call and its arguments.
|
||||
1. **Review model reasoning.** If extended thinking was enabled, check the
|
||||
model's reasoning at each step to understand why specific tools were called.
|
||||
1. **Assess attribution.** The session identifies the human who
|
||||
initiated the action. Subsequent interceptions represent agent-driven actions
|
||||
that stem from that original prompt.
|
||||
1. **Identify the session.**
|
||||
Filter by user, time range, or client to find the relevant session.
|
||||
1. **Locate the thread.**
|
||||
Each thread in a session shows the (likely) human prompt that initiated the chain of actions.
|
||||
1. **Trace the causal chain.**
|
||||
Expand the thread to see every step in the agentic loop, including each tool call and its arguments.
|
||||
1. **Review model reasoning.**
|
||||
If extended thinking was enabled, check the model's reasoning at each step to understand why specific tools were called.
|
||||
1. **Assess attribution.**
|
||||
The session identifies the human who initiated the action.
|
||||
Subsequent interceptions represent agent-driven actions that stem from that original prompt.
|
||||
|
||||
## What we store
|
||||
|
||||
@@ -126,21 +107,15 @@ AI Gateway captures the following data from each request/response:
|
||||
- In future, we will support storing these results
|
||||
- Model thinking/reasoning
|
||||
|
||||
Model-produced inference text is discarded, as generated text alone
|
||||
cannot affect external systems. The retention philosophy prioritizes:
|
||||
Model-produced inference text is discarded, as generated text alone cannot affect external systems.
|
||||
The retention philosophy prioritizes:
|
||||
|
||||
- **Human prompts** — capture intent and detect policy violations or
|
||||
exfiltration attempts.
|
||||
- **Tool calls** — record how agents interact with external systems,
|
||||
which is critical for understanding how incidents occurred. For
|
||||
example, an agent might delete and recreate a database because it
|
||||
lacks permissions to satisfy a human request to query a table.
|
||||
- **Model reasoning** — preserve thinking content that explains why
|
||||
specific tools were invoked, distinguishing between human instruction
|
||||
and model misunderstanding as the root cause.
|
||||
- **Human prompts** capture intent and detect policy violations or exfiltration attempts.
|
||||
- **Tool calls** record how agents interact with external systems, which is critical for understanding how incidents occurred.
|
||||
For example, an agent might delete and recreate a database because it lacks permissions to satisfy a human request to query a table.
|
||||
- **Model reasoning** preserves thinking content that explains why specific tools were invoked, distinguishing between human instruction and model misunderstanding as the root cause.
|
||||
|
||||
See [data retention](./setup.md#data-retention) to configure how long
|
||||
session data is kept.
|
||||
See [data retention](./setup.md#data-retention) to configure how long session data is kept.
|
||||
|
||||
## Next steps
|
||||
|
||||
|
||||
@@ -3,8 +3,7 @@ title: Authentication
|
||||
---
|
||||
|
||||
> [!NOTE]
|
||||
> AI Gateway is part of [AI Governance](../ai-governance.md), which is
|
||||
> included with a Premium license.
|
||||
> AI Gateway is part of [AI Governance](../ai-governance.md), which is included with a Premium license.
|
||||
|
||||
AI Gateway uses different credentials for different kinds of connections:
|
||||
|
||||
@@ -61,8 +60,7 @@ export ANTHROPIC_BASE_URL="https://coder.example.com/api/v2/ai-gateway/anthropic
|
||||
|
||||
### AI Gateway Proxy authentication
|
||||
|
||||
For tools that don't support a configurable base URL,
|
||||
[AI Gateway Proxy](./ai-gateway-proxy/index.md) intercepts traffic and forwards it to AI Gateway.
|
||||
For tools that don't support a configurable base URL, [AI Gateway Proxy](./ai-gateway-proxy/index.md) intercepts traffic and forwards it to AI Gateway.
|
||||
The Coder token is supplied in the proxy URL:
|
||||
|
||||
```sh
|
||||
@@ -148,9 +146,7 @@ For provider setup and credential failover, refer to [Provider configuration](./
|
||||
## Bring Your Own Key (BYOK)
|
||||
|
||||
In addition to centralized key management, AI Gateway supports Bring Your Own Key (BYOK) mode.
|
||||
Users can provide their own LLM API keys or provider subscriptions,
|
||||
such as Claude Pro or Max and ChatGPT Plus or Pro,
|
||||
while AI Gateway continues to provide observability and governance.
|
||||
Users can provide their own LLM API keys or provider subscriptions, such as Claude Pro or Max and ChatGPT Plus or Pro, while AI Gateway continues to provide observability and governance.
|
||||
|
||||
|
||||
|
||||
@@ -162,16 +158,15 @@ In BYOK mode, users need two credentials:
|
||||
BYOK and centralized modes can be used together.
|
||||
When a user provides their own credential, AI Gateway forwards it directly.
|
||||
When no user credential is present, AI Gateway uses the administrator-configured provider key.
|
||||
This approach offers centralized keys as a default
|
||||
while allowing individual users to bring their own key.
|
||||
This approach offers centralized keys as a default while allowing individual users to bring their own key.
|
||||
|
||||
> [!NOTE]
|
||||
> When a BYOK credential is present, [key failover](./providers.md#key-failover)
|
||||
> is skipped.
|
||||
> When a BYOK credential is present, [key failover](./providers.md#key-failover) is skipped.
|
||||
|
||||
Coder Agents requests routed through AI Gateway are in-process control plane requests, not external client requests that send their own AI Gateway bearer token.
|
||||
Coder Agents use the same global BYOK setting.
|
||||
When BYOK is enabled, users can save personal API keys for any enabled AI provider from the Agents settings page. For AWS Bedrock providers, the personal key is an AWS Bedrock API key that AI Gateway forwards as a bearer token in place of the deployment-managed AWS credentials, for every model family Bedrock serves.
|
||||
When BYOK is enabled, users can save personal API keys for any enabled AI provider from the Agents settings page.
|
||||
For AWS Bedrock providers, the personal key is an AWS Bedrock API key that AI Gateway forwards as a bearer token in place of the deployment-managed AWS credentials, for every model family Bedrock serves.
|
||||
Refer to [Agents credential selection](../agents/models.md#credential-selection) for the Agents-specific behavior.
|
||||
|
||||
Visit individual [client pages](./clients/index.md) for configuration details.
|
||||
|
||||
@@ -3,15 +3,12 @@ title: AI Governance cost control
|
||||
---
|
||||
|
||||
> [!NOTE]
|
||||
> AI Gateway is part of [AI Governance](../ai-governance.md), which is
|
||||
> included with a Premium license.
|
||||
> AI Gateway is part of [AI Governance](../ai-governance.md), which is included with a Premium license.
|
||||
|
||||
AI Governance Cost Control governs AI spend in two complementary ways:
|
||||
|
||||
- **Enforcement** stops a user's requests routed via AI Gateway once their
|
||||
spend reaches their budget.
|
||||
- **Reporting** shows what each user and group has approximately spent in the
|
||||
current budget period.
|
||||
- **Enforcement** stops a user's requests routed via AI Gateway once their spend reaches their budget.
|
||||
- **Reporting** shows what each user and group has approximately spent in the current budget period.
|
||||
|
||||
AI Governance Cost Control requires:
|
||||
|
||||
@@ -20,7 +17,8 @@ AI Governance Cost Control requires:
|
||||
|
||||
> [!NOTE]
|
||||
> AI Governance Cost Control reports approximate spend rather than billed cost.
|
||||
> These figures will not match your provider invoices exactly. For details, visit [How spend is calculated](#how-spend-is-calculated).
|
||||
> These figures will not match your provider invoices exactly.
|
||||
> For details, visit [How spend is calculated](#how-spend-is-calculated).
|
||||
|
||||
These terms appear throughout this page and in the Coder dashboard:
|
||||
|
||||
@@ -35,31 +33,29 @@ These terms appear throughout this page and in the Coder dashboard:
|
||||
|
||||
## Deployment settings
|
||||
|
||||
Two deployment-wide settings govern budget resolution and the reset window. Each
|
||||
accepts a single value today.
|
||||
Two deployment-wide settings govern budget resolution and the reset window.
|
||||
Each accepts a single value today.
|
||||
|
||||
| Setting | Flag | Environment variable | Supported values | Default |
|
||||
|---------------|----------------------|--------------------------|------------------|-----------|
|
||||
| Budget policy | `--ai-budget-policy` | `CODER_AI_BUDGET_POLICY` | `highest` | `highest` |
|
||||
| Budget period | `--ai-budget-period` | `CODER_AI_BUDGET_PERIOD` | `month` | `month` |
|
||||
|
||||
- **Budget policy** determines which budget wins when a user belongs to more
|
||||
than one budgeted group. `highest` selects the largest.
|
||||
- **Budget period** sets the reset window. `month` is the UTC calendar month, so
|
||||
spend resets at 00:00 UTC on the first day of each month.
|
||||
- **Budget policy** determines which budget wins when a user belongs to more than one budgeted group.
|
||||
`highest` selects the largest.
|
||||
- **Budget period** sets the reset window.
|
||||
`month` is the UTC calendar month, so spend resets at 00:00 UTC on the first day of each month.
|
||||
|
||||
These settings are deployment-wide.
|
||||
|
||||
## Budget
|
||||
|
||||
**Every user's spend is recorded against a group.** A user only has a budget
|
||||
when one is set on a group they belong to, or when they are given a user
|
||||
override.
|
||||
**Every user's spend is recorded against a group.**
|
||||
A user only has a budget when one is set on a group they belong to, or when they are given a user override.
|
||||
|
||||
> [!NOTE]
|
||||
> When AI Governance Cost Control is first deployed, users have **unlimited
|
||||
> spend** until an administrator sets budgets. Their spend is recorded against
|
||||
> the `Everyone` group.
|
||||
> When AI Governance Cost Control is first deployed, users have **unlimited spend** until an administrator sets budgets.
|
||||
> Their spend is recorded against the `Everyone` group.
|
||||
|
||||
### Group budget
|
||||
|
||||
@@ -67,9 +63,8 @@ Setting a group budget requires the Owner, User Admin, Organization Admin, or Or
|
||||
Refer to [Groups and roles](../../admin/users/groups-roles.md) for the deployment roles.
|
||||
The [Organization roles](../../admin/users/groups-roles.md#organization-roles) section covers the organization roles.
|
||||
|
||||
A group budget applies to each member individually rather than to the group as a
|
||||
whole. For example, if a group has ten members and a budget of $200 USD, each
|
||||
member can spend $200 USD and the group has a total spend limit of $2,000 USD.
|
||||
A group budget applies to each member individually rather than to the group as a whole.
|
||||
For example, if a group has ten members and a budget of $200 USD, each member can spend $200 USD and the group has a total spend limit of $2,000 USD.
|
||||
|
||||
1. Go to **Groups** and select a group.
|
||||
1. Select **Settings**.
|
||||
@@ -80,30 +75,25 @@ member can spend $200 USD and the group has a total spend limit of $2,000 USD.
|
||||
|
||||
Budget values behave as follows:
|
||||
|
||||
- An empty field means no budget is set. The field displays `no budget`.
|
||||
- `$0 USD` blocks every request routed via AI Gateway from members whose
|
||||
effective group is this group.
|
||||
- An empty field means no budget is set.
|
||||
The field displays `no budget`.
|
||||
- `$0 USD` blocks every request routed via AI Gateway from members whose effective group is this group.
|
||||
- The maximum is `$1,000,000 USD` per member per budget period.
|
||||
|
||||
Note: Members who belong to other groups with budgets are still governed by
|
||||
whichever group the budget policy selects. See
|
||||
[Effective group resolution](#effective-group-resolution).
|
||||
Note: Members who belong to other groups with budgets are still governed by whichever group the budget policy selects.
|
||||
See [Effective group resolution](#effective-group-resolution).
|
||||
|
||||
### User override
|
||||
|
||||
Adding or removing an override requires the Owner or User Admin
|
||||
[role](../../admin/users/groups-roles.md). Other administrator roles can view
|
||||
overrides but cannot change them.
|
||||
Adding or removing an override requires the Owner or User Admin [role](../../admin/users/groups-roles.md).
|
||||
Other administrator roles can view overrides but cannot change them.
|
||||
|
||||
Override a group budget when one user needs a different limit from the rest of
|
||||
their group.
|
||||
Override a group budget when one user needs a different limit from the rest of their group.
|
||||
|
||||
1. Go to **Groups**, select a group, then open the **Members** tab.
|
||||
1. Find the member to apply the override to, open their action menu, and select
|
||||
**Manage AI budget**.
|
||||
1. Find the member to apply the override to, open their action menu, and select **Manage AI budget**.
|
||||
1. Enable **Override group budget**.
|
||||
1. Set **Custom monthly budget**, then choose the group in
|
||||
**Budget assigned to**.
|
||||
1. Set **Custom monthly budget**, then choose the group in **Budget assigned to**.
|
||||
1. Select **Update**.
|
||||
|
||||
|
||||
@@ -112,64 +102,50 @@ Overrides behave as follows:
|
||||
|
||||
- A user can have only one override at a time.
|
||||
- An override supersedes every group budget the user belongs to.
|
||||
- Spend is always attributed to a group, so an override must name the assigned
|
||||
group. This group can be different from the group selected in the **Members**
|
||||
tab. The user must belong to the assigned group.
|
||||
- Spend is always attributed to a group, so an override must name the assigned group.
|
||||
This group can be different from the group selected in the **Members** tab.
|
||||
The user must belong to the assigned group.
|
||||
- The `$0 USD` to `$1,000,000 USD` range applies to overrides as well.
|
||||
- Disabling **Override group budget** removes the override and returns the user
|
||||
to the budget of their [effective group](#effective-group-resolution).
|
||||
- Disabling **Override group budget** removes the override and returns the user to the budget of their [effective group](#effective-group-resolution).
|
||||
|
||||
### Effective group resolution
|
||||
|
||||
A user can belong to several groups that have budgets, and can also hold an
|
||||
override. AI Gateway resolves a single effective group for each request, in this
|
||||
order:
|
||||
A user can belong to several groups that have budgets, and can also hold an override.
|
||||
AI Gateway resolves a single effective group for each request, in this order:
|
||||
|
||||
1. The user's [override](#user-override) takes precedence over every group
|
||||
budget.
|
||||
1. Otherwise, the [budget policy](#deployment-settings) selects one of the
|
||||
groups the user belongs to. The default policy, `highest`, selects the group
|
||||
with the largest budget.
|
||||
1. If none of those groups has a budget, the user has *effectively* unlimited
|
||||
spend and their spend is recorded against the `Everyone` group.
|
||||
1. The user's [override](#user-override) takes precedence over every group budget.
|
||||
1. Otherwise, the [budget policy](#deployment-settings) selects one of the groups the user belongs to.
|
||||
The default policy, `highest`, selects the group with the largest budget.
|
||||
1. If none of those groups has a budget, the user has *effectively* unlimited spend and their spend is recorded against the `Everyone` group.
|
||||
|
||||
> [!NOTE]
|
||||
> Groups with identical budgets are ranked by the organization membership the
|
||||
> user joined first, then by group ID. To see the effective group currently
|
||||
> assigned to a user, see [Monitor spend](#monitor-spend). The effective group is
|
||||
> deployment-wide, so it can be a group in a different organization.
|
||||
> Groups with identical budgets are ranked by the organization membership the user joined first, then by group ID.
|
||||
> To see the effective group currently assigned to a user, see [Monitor spend](#monitor-spend).
|
||||
> The effective group is deployment-wide, so it can be a group in a different organization.
|
||||
|
||||
Recorded spend is immutable. Changing which budget applies to a user affects
|
||||
future requests only: spend that Coder has already recorded stays with the group
|
||||
it was attributed to, so a user's history can span several groups. The effective
|
||||
group is resolved at request time, so changing a group budget can change which
|
||||
group applies to a user on future requests.
|
||||
Recorded spend is immutable.
|
||||
Changing which budget applies to a user affects future requests only: spend that Coder has already recorded stays with the group it was attributed to, so a user's history can span several groups.
|
||||
The effective group is resolved at request time, so changing a group budget can change which group applies to a user on future requests.
|
||||
|
||||
## How enforcement works
|
||||
|
||||
AI Gateway checks each request before forwarding it upstream. The check compares
|
||||
the user's spend in the current budget period with the budget that applies to the
|
||||
request:
|
||||
AI Gateway checks each request before forwarding it upstream.
|
||||
The check compares the user's spend in the current budget period with the budget that applies to the request:
|
||||
|
||||
- Spend below the budget: the request proceeds.
|
||||
- Spend at or above the budget: AI Gateway returns `403 Forbidden` with a
|
||||
message that describes the issue.
|
||||
- Spend at or above the budget: AI Gateway returns `403 Forbidden` with a message that describes the issue.
|
||||
|
||||
Users without a budget have **unlimited spend**, so their requests proceed
|
||||
without enforcement. A blocked user's access resumes when the budget period
|
||||
resets, when an administrator raises the budget, or when an override is added.
|
||||
Users without a budget have **unlimited spend**, so their requests proceed without enforcement.
|
||||
A blocked user's access resumes when the budget period resets, when an administrator raises the budget, or when an override is added.
|
||||
|
||||
> [!NOTE]
|
||||
> Enforcement is approximate. A request's cost is known only after the provider
|
||||
> response reaches AI Gateway, so concurrent in-flight requests can carry a user
|
||||
> slightly past their budget. Subsequent requests are blocked after the recorded
|
||||
> spend reaches the budget.
|
||||
> Enforcement is approximate.
|
||||
> A request's cost is known only after the provider response reaches AI Gateway, so concurrent in-flight requests can carry a user slightly past their budget.
|
||||
> Subsequent requests are blocked after the recorded spend reaches the budget.
|
||||
|
||||
### Notifications
|
||||
|
||||
The first time a user's spend crosses a threshold within a budget period, Coder
|
||||
notifies the user and deployment-wide Owners and User Admins, excluding the
|
||||
affected user:
|
||||
The first time a user's spend crosses a threshold within a budget period, Coder notifies the user and deployment-wide Owners and User Admins, excluding the affected user:
|
||||
|
||||
| Threshold | User notification | Admin notification |
|
||||
|-----------|--------------------------------|------------------------------------------|
|
||||
@@ -178,12 +154,12 @@ affected user:
|
||||
|
||||
- A single expensive request can cross both thresholds at once.
|
||||
- Budgets of `$0 USD` and unpriced usage cross no thresholds.
|
||||
- Notifications are informational. Enforcement does not depend on them.
|
||||
- Notifications are informational.
|
||||
Enforcement does not depend on them.
|
||||
|
||||
Owners also receive **Missing AI Model Prices** weekly, listing models used in the past week that have no price.
|
||||
|
||||
For delivery methods, see
|
||||
[Notifications](../../admin/monitoring/notifications/index.md).
|
||||
For delivery methods, see [Notifications](../../admin/monitoring/notifications/index.md).
|
||||
|
||||
## How spend is calculated
|
||||
|
||||
@@ -201,25 +177,25 @@ Coder uses the first price it finds, in this order:
|
||||
|
||||
A price for the requested model always takes precedence, even when the model the provider reports has a custom price.
|
||||
|
||||
Spend accumulates only from the moment v2.36 is deployed. Upgrading mid-month
|
||||
therefore produces a partial first period.
|
||||
Spend accumulates only from the moment v2.36 is deployed.
|
||||
Upgrading mid-month therefore produces a partial first period.
|
||||
|
||||
To see the prices this deployment uses, list them. The `source` column reports
|
||||
whether each is a default price or a custom price set on this deployment:
|
||||
To see the prices this deployment uses, list them.
|
||||
The `source` column reports whether each is a default price or a custom price set on this deployment:
|
||||
|
||||
```sh
|
||||
coder exp ai-model-prices list
|
||||
```
|
||||
|
||||
A custom price takes precedence over its default, so it is the one listed. To
|
||||
see the default for such a model, filter to it:
|
||||
A custom price takes precedence over its default, so it is the one listed.
|
||||
To see the default for such a model, filter to it:
|
||||
|
||||
```sh
|
||||
coder exp ai-model-prices list --source default
|
||||
```
|
||||
|
||||
Default prices come from the price book that ships with each Coder release. To
|
||||
see the price book for a release:
|
||||
Default prices come from the price book that ships with each Coder release.
|
||||
To see the price book for a release:
|
||||
|
||||
```text
|
||||
https://github.com/coder/coder/blob/release/<VERSION>/coderd/aibridge/prices/data/prices.json
|
||||
@@ -227,20 +203,17 @@ https://github.com/coder/coder/blob/release/<VERSION>/coderd/aibridge/prices/dat
|
||||
|
||||
Replace `<VERSION>` with a Coder minor version, for example `2.36`.
|
||||
|
||||
To use your own price for any of these models, see
|
||||
[Configure model prices](#configure-model-prices).
|
||||
To use your own price for any of these models, see [Configure model prices](#configure-model-prices).
|
||||
|
||||
> [!IMPORTANT]
|
||||
> Spend is an approximation. It can differ from what the provider bills, and
|
||||
> some usage does not count toward it at all:
|
||||
> Spend is an approximation.
|
||||
> It can differ from what the provider bills, and some usage does not count toward it at all:
|
||||
>
|
||||
> - Default prices are list prices. A custom price brings spend closer to the
|
||||
> rates a deployment actually pays, though billing rules that are not a
|
||||
> per-token rate, such as committed-use discounts, cannot be represented.
|
||||
> - A model with no price adds nothing to spend. Its token usage is still
|
||||
> recorded, but it never counts toward a limit, so a user who calls only
|
||||
> unpriced models is effectively unlimited. Setting a price for the model
|
||||
> closes the gap.
|
||||
> - Default prices are list prices.
|
||||
> A custom price brings spend closer to the rates a deployment actually pays, though billing rules that are not a per-token rate, such as committed-use discounts, cannot be represented.
|
||||
> - A model with no price adds nothing to spend.
|
||||
> Its token usage is still recorded, but it never counts toward a limit, so a user who calls only unpriced models is effectively unlimited.
|
||||
> Setting a price for the model closes the gap.
|
||||
|
||||
Monitor `coder_ai_gateway_cost_control_unpriced_token_usage_records_total`, labeled by `provider`, `provider_type`, `model`, and `provider_model`, to detect unpriced usage.
|
||||
`model` is the model the client requested, and `provider_model` is the model the provider reported, or empty when the provider didn't report one.
|
||||
@@ -252,16 +225,12 @@ Coder also notifies Owners weekly about models used in the past week that have n
|
||||
|
||||
### Configure model prices
|
||||
|
||||
Use the experimental `coder exp ai-model-prices` command to set model prices
|
||||
for your deployment. It requires AI Governance, which is included with a
|
||||
Premium license, and the Owner
|
||||
[role](../../admin/users/groups-roles.md) or a custom role granting
|
||||
`ai_model_price:update`. Run `coder exp ai-model-prices --help` for the full
|
||||
reference.
|
||||
Use the experimental `coder exp ai-model-prices` command to set model prices for your deployment.
|
||||
It requires AI Governance, which is included with a Premium license, and the Owner [role](../../admin/users/groups-roles.md) or a custom role granting `ai_model_price:update`.
|
||||
Run `coder exp ai-model-prices --help` for the full reference.
|
||||
|
||||
List the prices this deployment holds, optionally narrowed to one provider or
|
||||
model. The `source` column reports whether a price is a default price
|
||||
(`default`) or one set on this deployment (`custom`):
|
||||
List the prices this deployment holds, optionally narrowed to one provider or model.
|
||||
The `source` column reports whether a price is a default price (`default`) or one set on this deployment (`custom`):
|
||||
|
||||
```sh
|
||||
coder exp ai-model-prices list --provider anthropic
|
||||
@@ -273,16 +242,16 @@ List only the prices you have set:
|
||||
coder exp ai-model-prices list --source custom
|
||||
```
|
||||
|
||||
A listing reports one price per model, so an overridden model's default is not
|
||||
shown. To see every price a model holds, both at once:
|
||||
A listing reports one price per model, so an overridden model's default is not shown.
|
||||
To see every price a model holds, both at once:
|
||||
|
||||
```sh
|
||||
coder exp ai-model-prices list --source all
|
||||
```
|
||||
|
||||
Price a single model. Prices are micro-units per million tokens, so `3000000`
|
||||
is $3.00 per million tokens. Use `null` for a price you do not have, and `0` to
|
||||
declare a model free:
|
||||
Price a single model.
|
||||
Prices are micro-units per million tokens, so `3000000` is $3.00 per million tokens.
|
||||
Use `null` for a price you do not have, and `0` to declare a model free:
|
||||
|
||||
```sh
|
||||
coder exp ai-model-prices update --provider anthropic --model my-model \
|
||||
@@ -290,8 +259,7 @@ coder exp ai-model-prices update --provider anthropic --model my-model \
|
||||
--cache-read-price null --cache-write-price null
|
||||
```
|
||||
|
||||
Price several models at once from a JSON document in the same shape as the
|
||||
price book:
|
||||
Price several models at once from a JSON document in the same shape as the price book:
|
||||
|
||||
```sh
|
||||
coder exp ai-model-prices update prices.json
|
||||
@@ -299,14 +267,12 @@ coder exp ai-model-prices update prices.json
|
||||
|
||||
> [!IMPORTANT]
|
||||
>
|
||||
> - Prices are not retroactive. Usage recorded before you set a price stays
|
||||
> unpriced, so past spend does not change.
|
||||
> - A price you set takes precedence over the default and stays in effect
|
||||
> across upgrades, so it does not pick up later price books.
|
||||
> - A price is keyed by provider type and model, so every configured provider
|
||||
> of that type shares it.
|
||||
> - `openai-compat` providers cannot be priced. They pass through to any
|
||||
> upstream vendor, so a single price would be wrong for most of them.
|
||||
> - Prices are not retroactive.
|
||||
> Usage recorded before you set a price stays unpriced, so past spend does not change.
|
||||
> - A price you set takes precedence over the default and stays in effect across upgrades, so it does not pick up later price books.
|
||||
> - A price is keyed by provider type and model, so every configured provider of that type shares it.
|
||||
> - `openai-compat` providers cannot be priced.
|
||||
> They pass through to any upstream vendor, so a single price would be wrong for most of them.
|
||||
> - This command is experimental and can change without notice.
|
||||
|
||||
## Monitor spend
|
||||
@@ -332,23 +298,14 @@ Visibility follows the viewer's role:
|
||||
If you can also view AI sessions, you can filter it by provider, model, and client.
|
||||
-->
|
||||
|
||||
- The **Groups** page compares each group's spend with the combined limits of
|
||||
the members it covers.
|
||||
- The **Members** tab of a group reports each member's spend, their budget, and
|
||||
its source, labeled `Custom limit` for an override or `Group limit` for a
|
||||
group budget. If the effective group is another group in the same organization,
|
||||
the row shows `Budget managed by another group`. If the effective group is in a
|
||||
different organization, the row shows a dash and explains that the group is not
|
||||
visible there. Because the organization's `Everyone` group includes every
|
||||
member, its **Members** tab is a quick way to look up any user's effective
|
||||
group.
|
||||
- The avatar menu reports the signed-in user's own spend for the budget period
|
||||
as `$<spend> / $<budget> USD`, or `$<spend> / Unlimited USD` when no budget
|
||||
applies.
|
||||
- The **Groups** page compares each group's spend with the combined limits of the members it covers.
|
||||
- The **Members** tab of a group reports each member's spend, their budget, and its source, labeled `Custom limit` for an override or `Group limit` for a group budget.
|
||||
If the effective group is another group in the same organization, the row shows `Budget managed by another group`.
|
||||
If the effective group is in a different organization, the row shows a dash and explains that the group is not visible there.
|
||||
Because the organization's `Everyone` group includes every member, its **Members** tab is a quick way to look up any user's effective group.
|
||||
- The avatar menu reports the signed-in user's own spend for the budget period as `$<spend> / $<budget> USD`, or `$<spend> / Unlimited USD` when no budget applies.
|
||||
|
||||
Administrators can also use the
|
||||
[Get user AI spend](../../reference/api/enterprise.md#get-user-ai-spend) API
|
||||
endpoint to see a user's current effective group.
|
||||
Administrators can also use the [Get user AI spend](../../reference/api/enterprise.md#get-user-ai-spend) API endpoint to see a user's current effective group.
|
||||
|
||||
### Spend export
|
||||
|
||||
@@ -371,39 +328,37 @@ For programmatic per-user reporting, refer to the [List organization AI spend by
|
||||
|
||||
### Prometheus Metrics
|
||||
|
||||
Prometheus metrics report blocked requests, users over budget, unpriced usage,
|
||||
and enforcement latency. For the full metric list, including types and labels,
|
||||
see [Prometheus metrics](../../admin/integrations/prometheus.md).
|
||||
Prometheus metrics report blocked requests, users over budget, unpriced usage, and enforcement latency.
|
||||
For the full metric list, including types and labels, see [Prometheus metrics](../../admin/integrations/prometheus.md).
|
||||
|
||||
## Migrate from Coder Agents Cost Control
|
||||
|
||||
In v2.36, AI Governance Cost Control replaces Coder Agents Cost Control. The
|
||||
legacy Coder Agents Spend page remains available until v2.37.
|
||||
In v2.36, AI Governance Cost Control replaces Coder Agents Cost Control.
|
||||
The legacy Coder Agents Spend page remains available until v2.37.
|
||||
|
||||
> [!WARNING]
|
||||
> Spend limits configured under **Admin settings** > **AI** > **Spend** are no
|
||||
> longer enforced by Coder Agents. To enforce spend, set an AI Governance budget.
|
||||
> Spend limits configured under **Admin settings** > **AI** > **Spend** are no longer enforced by Coder Agents.
|
||||
> To enforce spend, set an AI Governance budget.
|
||||
|
||||
To migrate existing limits:
|
||||
|
||||
1. Record the limits currently set under **Admin settings** > **AI** >
|
||||
**Spend**, including the default limit and any group or user overrides.
|
||||
1. Record the limits currently set under **Admin settings** > **AI** > **Spend**, including the default limit and any group or user overrides.
|
||||
1. Recreate group limits as [group budgets](#group-budget).
|
||||
1. Recreate per-user limits as [user overrides](#user-override).
|
||||
|
||||
Expect the following differences:
|
||||
|
||||
- No deployment-wide default exists. Each group that needs a limit requires its
|
||||
own budget.
|
||||
- The UTC calendar month is the only period. Daily and weekly periods are not
|
||||
currently supported.
|
||||
- Users in several budgeted groups receive the highest budget. Coder Agents Cost
|
||||
Control applied the lowest.
|
||||
- Budgets cover priced AI Gateway traffic. Chat, IDE extensions, and CLI agents
|
||||
draw on the same budget when their provider and model are priced. See
|
||||
[How spend is calculated](#how-spend-is-calculated).
|
||||
- Recorded spend does not carry over. Every user starts the first period at
|
||||
$0 USD.
|
||||
- No deployment-wide default exists.
|
||||
Each group that needs a limit requires its own budget.
|
||||
- The UTC calendar month is the only period.
|
||||
Daily and weekly periods are not currently supported.
|
||||
- Users in several budgeted groups receive the highest budget.
|
||||
Coder Agents Cost Control applied the lowest.
|
||||
- Budgets cover priced AI Gateway traffic.
|
||||
Chat, IDE extensions, and CLI agents draw on the same budget when their provider and model are priced.
|
||||
See [How spend is calculated](#how-spend-is-calculated).
|
||||
- Recorded spend does not carry over.
|
||||
Every user starts the first period at $0 USD.
|
||||
- Coder Agents users who exceed their budget see a usage limit error in chat.
|
||||
The error details include the AI Governance budget limit.
|
||||
|
||||
@@ -412,8 +367,7 @@ Expect the following differences:
|
||||
Cost control reads none of the tables that store prompts, tool call arguments or model thoughts.
|
||||
It depends only on the interception and token usage records, the model prices and the budget tables.
|
||||
|
||||
A deployment that wants budgets and spend reporting without retaining conversation content can therefore set
|
||||
[`--ai-gateway-disable-content-recording`](./setup.md#disable-content-recording) and keep every cost control feature on this page.
|
||||
A deployment that wants budgets and spend reporting without retaining conversation content can therefore set [`--ai-gateway-disable-content-recording`](./setup.md#disable-content-recording) and keep every cost control feature on this page.
|
||||
|
||||
> [!NOTE]
|
||||
> [`--ai-gateway-retention`](./setup.md#data-retention) can't do this.
|
||||
|
||||
@@ -4,31 +4,32 @@ title: AI Gateway
|
||||
|
||||
|
||||
|
||||
AI Gateway is a smart gateway for AI. It acts as an intermediary between your users' coding agents / IDEs
|
||||
and providers like OpenAI and Anthropic. By intercepting all the AI traffic between these clients and
|
||||
the upstream APIs, AI Gateway can record user prompts, token usage, and tool invocations.
|
||||
AI Gateway is a smart gateway for AI.
|
||||
It acts as an intermediary between your users' coding agents / IDEs and providers like OpenAI and Anthropic.
|
||||
By intercepting all the AI traffic between these clients and the upstream APIs, AI Gateway can record user prompts, token usage, and tool invocations.
|
||||
AI Gateway supports clients running inside or outside Coder workspaces.
|
||||
You can run the gateway inside the Coder control plane (`coderd`) or as a [standalone service](./standalone.md).
|
||||
|
||||
AI Gateway solves 3 key problems:
|
||||
|
||||
1. **Centralized authn/z management**: no more issuing & managing API tokens for OpenAI/Anthropic usage.
|
||||
Users use their Coder session or API tokens to authenticate with `coderd`, and
|
||||
`coderd` securely communicates with the upstream APIs on their behalf.
|
||||
1. **Auditing and attribution**: all interactions with AI services, whether autonomous or human-initiated,
|
||||
are audited and attributed back to a user.
|
||||
1. **Centralized MCP administration**: define a set of approved MCP servers and tools which your users may
|
||||
use.
|
||||
1. **Centralized authn/z management**:
|
||||
no more issuing & managing API tokens for OpenAI/Anthropic usage.
|
||||
Users use their Coder session or API tokens to authenticate with `coderd`, and `coderd` securely communicates with the upstream APIs on their behalf.
|
||||
1. **Auditing and attribution**:
|
||||
all interactions with AI services, whether autonomous or human-initiated, are audited and attributed back to a user.
|
||||
1. **Centralized MCP administration**:
|
||||
define a set of approved MCP servers and tools which your users may use.
|
||||
|
||||
> [!NOTE]
|
||||
> AI Gateway is part of [AI Governance](../ai-governance.md), which is
|
||||
> included with a Premium license.
|
||||
> AI Gateway is part of [AI Governance](../ai-governance.md), which is included with a Premium license.
|
||||
>
|
||||
> AI Gateway was previously known as "AI Bridge". Visit [Rebranding Migration](./rebranding-migration.md) for details.
|
||||
> AI Gateway was previously known as "AI Bridge".
|
||||
> Visit [Rebranding Migration](./rebranding-migration.md) for details.
|
||||
|
||||
## When to use AI Gateway
|
||||
|
||||
As LLM adoption grows, administrators need centralized auditing, monitoring, and token management. AI Gateway enables organizations to manage AI tooling access for thousands of engineers from a single control plane.
|
||||
As LLM adoption grows, administrators need centralized auditing, monitoring, and token management.
|
||||
AI Gateway enables organizations to manage AI tooling access for thousands of engineers from a single control plane.
|
||||
|
||||
If you are an administrator or devops leader looking to:
|
||||
|
||||
|
||||
@@ -3,16 +3,14 @@ title: MCP
|
||||
---
|
||||
|
||||
> [!NOTE]
|
||||
> AI Gateway is part of [AI Governance](../ai-governance.md), which is
|
||||
> included with a Premium license.
|
||||
> AI Gateway is part of [AI Governance](../ai-governance.md), which is included with a Premium license.
|
||||
|
||||
<!-- -->
|
||||
|
||||
> [!WARNING]
|
||||
> Injected MCP in AI Gateway is deprecated.
|
||||
> It remains functional and will not be removed until
|
||||
> the new implementation is released. Only critical
|
||||
> security-related patches will be made.
|
||||
> It remains functional and will not be removed until the new implementation is released.
|
||||
> Only critical security-related patches will be made.
|
||||
|
||||
[Model Context Protocol (MCP)](https://modelcontextprotocol.io/docs/getting-started/intro) is a mechanism for connecting AI applications to external systems.
|
||||
|
||||
@@ -21,9 +19,11 @@ AI Gateway can connect to MCP servers and inject tools automatically, enabling y
|
||||
> [!NOTE]
|
||||
> Only MCP servers which support OAuth2 Authorization are supported currently.
|
||||
>
|
||||
> [_Streamable HTTP_](https://modelcontextprotocol.io/specification/2026-07-28/basic/transports) is the only supported transport currently. In future releases we will support the (now deprecated) [_Server-Sent Events_](https://modelcontextprotocol.io/specification/2025-06-18/basic/transports#backwards-compatibility) transport.
|
||||
> [_Streamable HTTP_](https://modelcontextprotocol.io/specification/2026-07-28/basic/transports) is the only supported transport currently.
|
||||
> In future releases we will support the (now deprecated) [_Server-Sent Events_](https://modelcontextprotocol.io/specification/2025-06-18/basic/transports#backwards-compatibility) transport.
|
||||
|
||||
AI Gateway makes use of [External Auth](../../admin/external-auth.md) applications, as they define OAuth2 connections to upstream services. If your External Auth application hosts a remote MCP server, you can configure AI Gateway to connect to it, retrieve its tools and inject them into requests automatically - all while using each individual user's access token.
|
||||
AI Gateway makes use of [External Auth](../../admin/external-auth.md) applications, as they define OAuth2 connections to upstream services.
|
||||
If your External Auth application hosts a remote MCP server, you can configure AI Gateway to connect to it, retrieve its tools and inject them into requests automatically - all while using each individual user's access token.
|
||||
|
||||
For example, GitHub has a [remote MCP server](https://github.com/github/github-mcp-server?tab=readme-ov-file#remote-github-mcp-server) and we can use it as follows.
|
||||
|
||||
@@ -54,14 +54,16 @@ The logic works as follows:
|
||||
|
||||
In the above example, if you prompted your AI model with "list your available github tools by name", it would reply something like:
|
||||
|
||||
> Certainly! Here are the GitHub-related tools that I have available:
|
||||
> Certainly!
|
||||
> Here are the GitHub-related tools that I have available:
|
||||
>
|
||||
> ```text
|
||||
> 1. bmcp_github_update_gist
|
||||
> 2. bmcp_github_list_gists
|
||||
> ```
|
||||
|
||||
AI Gateway marks automatically injected tools with a prefix `bmcp_` ("bridged MCP"). It also namespaces all tool names by the ID of their associated External Auth application (in this case `github`).
|
||||
AI Gateway marks automatically injected tools with a prefix `bmcp_` ("bridged MCP").
|
||||
It also namespaces all tool names by the ID of their associated External Auth application (in this case `github`).
|
||||
|
||||
## Tool Injection
|
||||
|
||||
@@ -79,8 +81,12 @@ If you have [Coder MCP Server](../mcp-server.md) enabled, as well as have `CODER
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
- **Too many tools**: should you receive an error like `Invalid 'tools': array too long. Expected an array with maximum length 128, but got an array with length 132 instead`, you can reduce the number by filtering out tools using the allow/deny patterns documented in this page.
|
||||
- **Too many tools**:
|
||||
should you receive an error like `Invalid 'tools': array too long. Expected an array with maximum length 128, but got an array with length 132 instead`, you can reduce the number by filtering out tools using the allow/deny patterns documented in this page.
|
||||
|
||||
- **Coder MCP tools not being injected**: in order for Coder MCP tools to be injected, the internal MCP server needs to be active. Follow the instructions in the [MCP Server](../mcp-server.md) page to enable it and ensure `CODER_AI_GATEWAY_INJECT_CODER_MCP_TOOLS` is set to `true`.
|
||||
- **Coder MCP tools not being injected**:
|
||||
in order for Coder MCP tools to be injected, the internal MCP server needs to be active.
|
||||
Follow the instructions in the [MCP Server](../mcp-server.md) page to enable it and ensure `CODER_AI_GATEWAY_INJECT_CODER_MCP_TOOLS` is set to `true`.
|
||||
|
||||
- **External Auth tools not being injected**: this is generally due to the requesting user not being authenticated against the [External Auth](../../admin/external-auth.md) app; when this is the case, no attempt is made to connect to the MCP server.
|
||||
- **External Auth tools not being injected**:
|
||||
this is generally due to the requesting user not being authenticated against the [External Auth](../../admin/external-auth.md) app; when this is the case, no attempt is made to connect to the MCP server.
|
||||
|
||||
@@ -3,10 +3,10 @@ title: Monitoring
|
||||
---
|
||||
|
||||
> [!NOTE]
|
||||
> AI Gateway is part of [AI Governance](../ai-governance.md), which is
|
||||
> included with a Premium license.
|
||||
> AI Gateway is part of [AI Governance](../ai-governance.md), which is included with a Premium license.
|
||||
|
||||
AI Gateway records the last `user` prompt, token usage, model reasoning, and every tool invocation for each intercepted request. Each capture is tied to a single "interception" that maps back to the authenticated Coder identity, making it easy to attribute spend and behaviour.
|
||||
AI Gateway records the last `user` prompt, token usage, model reasoning, and every tool invocation for each intercepted request.
|
||||
Each capture is tied to a single "interception" that maps back to the authenticated Coder identity, making it easy to attribute spend and behaviour.
|
||||
|
||||
|
||||
|
||||
@@ -70,12 +70,11 @@ Standalone replicas do not export them.
|
||||
|
||||
AI Gateway Proxy exports metrics from the `coderd` Prometheus listener.
|
||||
|
||||
A `proxy_excluded` status means another enabled provider with the same base URL hostname claimed the proxy route first. The proxy classifier uses first-wins by database name sort order (`ORDER BY name ASC`), so the first provider in that order wins and subsequent providers with the same hostname are excluded from proxy routing. They remain reachable via direct routing at `/api/v2/ai-gateway/{name}/...`.
|
||||
A `proxy_excluded` status means another enabled provider with the same base URL hostname claimed the proxy route first.
|
||||
The proxy classifier uses first-wins by database name sort order (`ORDER BY name ASC`), so the first provider in that order wins and subsequent providers with the same hostname are excluded from proxy routing.
|
||||
They remain reachable via direct routing at `/api/v2/ai-gateway/{name}/...`.
|
||||
|
||||
When alerting on `coder_ai_gateway_proxy_provider_info`, exclude
|
||||
`proxy_excluded` from the alert expression unless you want to be
|
||||
notified of intended duplicate-hostname configurations:
|
||||
`status!~"enabled|proxy_excluded"`.
|
||||
When alerting on `coder_ai_gateway_proxy_provider_info`, exclude `proxy_excluded` from the alert expression unless you want to be notified of intended duplicate-hostname configurations: `status!~"enabled|proxy_excluded"`.
|
||||
|
||||
| Metric | Type | Labels | Purpose |
|
||||
|--------------------------------------------------------------------------|---------|--------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|
|
||||
@@ -171,7 +170,8 @@ A standalone AI gateway exposes health endpoints on its data-plane listener:
|
||||
|
||||
`/readyz` returns HTTP 503 until the initial provider configuration fetch completes and whenever the control connection to `coderd` is unavailable.
|
||||
The initial fetch can return an empty provider set.
|
||||
A `200 OK` response from `/healthz` only means the HTTP listener is accepting connections. It returns `200 OK` even when the control connection is down.
|
||||
A `200 OK` response from `/healthz` only means the HTTP listener is accepting connections.
|
||||
It returns `200 OK` even when the control connection is down.
|
||||
Both endpoints are unauthenticated, bypass the concurrency, rate limiting, and BYOK middleware, and do not create trace spans.
|
||||
|
||||
The standalone Helm chart enables a `/healthz` liveness probe and a `/readyz` readiness probe by default.
|
||||
@@ -208,7 +208,8 @@ curl -X GET "https://coder.example.com/api/v2/ai-gateway/sessions" \
|
||||
|
||||
Available query filters:
|
||||
|
||||
- `client` - Filter by client name. Possible values are listed below.
|
||||
- `client` - Filter by client name.
|
||||
Possible values are listed below.
|
||||
- `initiator` - Filter by user ID or username
|
||||
- `provider` - Filter by AI provider (e.g., `openai`, `anthropic`)
|
||||
- `model` - Filter by model name
|
||||
@@ -219,8 +220,7 @@ Available query filters:
|
||||
<summary>Possible <code>client</code> values</summary>
|
||||
|
||||
> [!NOTE]
|
||||
> Client classification is done on best effort basis using the `User-Agent` header;
|
||||
> not all clients send these headers in an easily-identifiable manner.
|
||||
> Client classification is done on best effort basis using the `User-Agent` header; not all clients send these headers in an easily-identifiable manner.
|
||||
|
||||
- `Claude Code`
|
||||
- `Codex`
|
||||
@@ -246,8 +246,7 @@ Refer to the [API documentation](../../reference/api/aigateway.md) for full deta
|
||||
AI Gateway data is retained for **60 days by default**.
|
||||
Configure the retention period to balance storage costs with your organization's compliance and analysis needs.
|
||||
|
||||
For configuration options and details, refer to [Data Retention](./setup.md#data-retention)
|
||||
in the AI Gateway setup guide.
|
||||
For configuration options and details, refer to [Data Retention](./setup.md#data-retention) in the AI Gateway setup guide.
|
||||
|
||||
## Tracing
|
||||
|
||||
|
||||
@@ -3,17 +3,12 @@ title: Provider configuration
|
||||
---
|
||||
|
||||
> [!NOTE]
|
||||
> AI Gateway is part of [AI Governance](../ai-governance.md), which is
|
||||
> included with a Premium license.
|
||||
> AI Gateway is part of [AI Governance](../ai-governance.md), which is included with a Premium license.
|
||||
|
||||
Providers are deployment-scoped and managed from the dashboard or the
|
||||
[AI Providers API](../../reference/api/aiproviders.md). See
|
||||
[Setup](./setup.md#configure-providers) for the steps to add, edit, and
|
||||
disable a provider.
|
||||
Providers are deployment-scoped and managed from the dashboard or the [AI Providers API](../../reference/api/aiproviders.md).
|
||||
See [Setup](./setup.md#configure-providers) for the steps to add, edit, and disable a provider.
|
||||
|
||||
This page covers the provider types AI Gateway supports, the setup
|
||||
considerations for each, how a provider's lifecycle affects request
|
||||
handling, and how to monitor providers.
|
||||
This page covers the provider types AI Gateway supports, the setup considerations for each, how a provider's lifecycle affects request handling, and how to monitor providers.
|
||||
|
||||
## Database management of providers
|
||||
|
||||
@@ -57,9 +52,8 @@ Both the embedded gateway and a [standalone gateway](./standalone.md) fetch prov
|
||||
|
||||
## Provider types
|
||||
|
||||
AI Gateway speaks two upstream API formats: the **OpenAI** format
|
||||
(Chat Completions and Responses) and the **Anthropic** format
|
||||
(Messages). Every provider type maps to one of these.
|
||||
AI Gateway speaks two upstream API formats: the **OpenAI** format (Chat Completions and Responses) and the **Anthropic** format (Messages).
|
||||
Every provider type maps to one of these.
|
||||
|
||||
| Type | API format | Setup notes |
|
||||
|-----------------|------------|-------------------------------------------------------------------|
|
||||
@@ -73,52 +67,38 @@ AI Gateway speaks two upstream API formats: the **OpenAI** format
|
||||
| `vercel` | OpenAI | OpenAI-compatible endpoint only |
|
||||
| `openai-compat` | OpenAI | Generic OpenAI-compatible endpoint |
|
||||
|
||||
`azure`, `google`, `openrouter`, `vercel`, and `openai-compat` are
|
||||
supported only as OpenAI-compatible endpoints: AI Gateway sends them
|
||||
OpenAI-format requests, so each must expose an OpenAI-compatible API at
|
||||
its base URL. They have no provider-specific integration beyond that.
|
||||
`azure`, `google`, `openrouter`, `vercel`, and `openai-compat` are supported only as OpenAI-compatible endpoints: AI Gateway sends them OpenAI-format requests, so each must expose an OpenAI-compatible API at its base URL.
|
||||
They have no provider-specific integration beyond that.
|
||||
|
||||
### OpenAI
|
||||
|
||||
Set the base URL to the upstream endpoint and provide an API key. The
|
||||
default `https://api.openai.com/v1/` targets the native OpenAI service;
|
||||
point it at any OpenAI-compatible endpoint (for example, a hosted proxy
|
||||
or LiteLLM deployment) when needed.
|
||||
Set the base URL to the upstream endpoint and provide an API key.
|
||||
The default `https://api.openai.com/v1/` targets the native OpenAI service; point it at any OpenAI-compatible endpoint (for example, a hosted proxy or LiteLLM deployment) when needed.
|
||||
|
||||
If you create an [OpenAI key](https://platform.openai.com/api-keys)
|
||||
with minimal privileges, this is the minimum required set:
|
||||
If you create an [OpenAI key](https://platform.openai.com/api-keys) with minimal privileges, this is the minimum required set:
|
||||
|
||||
|
||||
|
||||
### Anthropic
|
||||
|
||||
Set the base URL and provide an API key. The default
|
||||
`https://api.anthropic.com/` targets Anthropic's public API; override it
|
||||
for Anthropic-compatible brokers.
|
||||
Set the base URL and provide an API key.
|
||||
The default `https://api.anthropic.com/` targets Anthropic's public API; override it for Anthropic-compatible brokers.
|
||||
|
||||
Anthropic does not allow [API keys](https://console.anthropic.com/settings/keys)
|
||||
to have restricted permissions at the time of writing (June 2026).
|
||||
Anthropic does not allow [API keys](https://console.anthropic.com/settings/keys) to have restricted permissions at the time of writing (June 2026).
|
||||
|
||||
#### Claude Platform for AWS
|
||||
|
||||
Claude Platform for AWS is Anthropic's own Messages API hosted on AWS. It
|
||||
is an authentication method on the `anthropic` provider type, not a
|
||||
separate provider type: requests and responses are the standard Anthropic
|
||||
format with standard Anthropic model IDs, and only routing and
|
||||
authentication differ. It is distinct from
|
||||
[Amazon Bedrock](#amazon-bedrock), which is a separate provider type.
|
||||
Claude Platform for AWS is Anthropic's own Messages API hosted on AWS.
|
||||
It is an authentication method on the `anthropic` provider type, not a separate provider type: requests and responses are the standard Anthropic format with standard Anthropic model IDs, and only routing and authentication differ.
|
||||
It is distinct from [Amazon Bedrock](#amazon-bedrock), which is a separate provider type.
|
||||
|
||||
Every Claude Platform provider requires:
|
||||
|
||||
- A **region**, for example `us-east-1`. It selects the default endpoint
|
||||
`https://aws-external-anthropic.<region>.api.aws` and, for IAM
|
||||
authentication, the signing scope. Set it explicitly even when you
|
||||
override the endpoint, so requests routed through a proxy are still
|
||||
signed for the correct region.
|
||||
- A **workspace ID**, sent as the `anthropic-workspace-id` header on
|
||||
every request. AI Gateway sets the header from provider configuration
|
||||
and strips any value a client sends, so a client cannot choose which
|
||||
workspace its traffic is attributed to.
|
||||
- A **region**, for example `us-east-1`.
|
||||
It selects the default endpoint `https://aws-external-anthropic.<region>.api.aws` and, for IAM authentication, the signing scope.
|
||||
Set it explicitly even when you override the endpoint, so requests routed through a proxy are still signed for the correct region.
|
||||
- A **workspace ID**, sent as the `anthropic-workspace-id` header on every request.
|
||||
AI Gateway sets the header from provider configuration and strips any value a client sends, so a client cannot choose which workspace its traffic is attributed to.
|
||||
|
||||
AI Gateway chooses authentication from the credentials available for each request:
|
||||
|
||||
@@ -135,36 +115,29 @@ A client sending an ordinary Anthropic key to a Claude Platform provider receive
|
||||
|
||||
### Amazon Bedrock
|
||||
|
||||
Bedrock providers serve Anthropic models hosted on AWS and authenticate
|
||||
with AWS credentials rather than a registered API key. Configure:
|
||||
Bedrock providers serve Anthropic models hosted on AWS and authenticate with AWS credentials rather than a registered API key.
|
||||
Configure:
|
||||
|
||||
- A **protocol**, either `InvokeModel` or `Mantle`. It determines the
|
||||
endpoint format and which of the fields below apply. See
|
||||
[InvokeModel](#invokemodel) and [Mantle](#mantle).
|
||||
- An **endpoint** in the format the protocol requires:
|
||||
`https://bedrock-runtime.<region>.amazonaws.com` for InvokeModel or
|
||||
`https://bedrock-mantle.<region>.api.aws/anthropic` for Mantle. The AWS
|
||||
region is read from the endpoint host.
|
||||
- The **model** and **small fast model** identifiers, for InvokeModel
|
||||
only. Mantle providers do not set them; the client chooses the model on
|
||||
each request and AI Gateway forwards it upstream.
|
||||
- A **protocol**, either `InvokeModel` or `Mantle`.
|
||||
It determines the endpoint format and which of the fields below apply.
|
||||
See [InvokeModel](#invokemodel) and [Mantle](#mantle).
|
||||
- An **endpoint** in the format the protocol requires: `https://bedrock-runtime.<region>.amazonaws.com` for InvokeModel or `https://bedrock-mantle.<region>.api.aws/anthropic` for Mantle.
|
||||
The AWS region is read from the endpoint host.
|
||||
- The **model** and **small fast model** identifiers, for InvokeModel only.
|
||||
Mantle providers do not set them; the client chooses the model on each request and AI Gateway forwards it upstream.
|
||||
|
||||
#### InvokeModel
|
||||
|
||||
The legacy Bedrock runtime API. AI Gateway translates each request into
|
||||
Bedrock's InvokeModel format and sends it to
|
||||
`https://bedrock-runtime.<region>.amazonaws.com`. Still supported, but
|
||||
Mantle is recommended for new deployments.
|
||||
The legacy Bedrock runtime API.
|
||||
AI Gateway translates each request into Bedrock's InvokeModel format and sends it to `https://bedrock-runtime.<region>.amazonaws.com`.
|
||||
Still supported, but Mantle is recommended for new deployments.
|
||||
|
||||
#### Mantle
|
||||
|
||||
The newer Anthropic-compatible Bedrock endpoint, recommended by AWS for new
|
||||
deployments. AI Gateway serves Anthropic models through the native Messages
|
||||
API, forwarding the request body **unchanged** and only applying AWS SigV4
|
||||
signing with the provider's base identity.
|
||||
The newer Anthropic-compatible Bedrock endpoint, recommended by AWS for new deployments.
|
||||
AI Gateway serves Anthropic models through the native Messages API, forwarding the request body **unchanged** and only applying AWS SigV4 signing with the provider's base identity.
|
||||
|
||||
To route Claude Code through a Mantle provider, run it in mantle mode with
|
||||
client-side signing disabled so the gateway signs centrally:
|
||||
To route Claude Code through a Mantle provider, run it in mantle mode with client-side signing disabled so the gateway signs centrally:
|
||||
|
||||
```sh
|
||||
export CLAUDE_CODE_USE_MANTLE=1
|
||||
@@ -185,38 +158,29 @@ curl "${CODER_URL}/api/v2/ai-gateway/${PROVIDER_NAME}/v1/messages" \
|
||||
|
||||
#### AWS credentials
|
||||
|
||||
Do not attach API keys to a Bedrock provider. The provider itself always
|
||||
authenticates with AWS credentials.
|
||||
Do not attach API keys to a Bedrock provider.
|
||||
The provider itself always authenticates with AWS credentials.
|
||||
|
||||
When [BYOK](auth.md#bring-your-own-key-byok) is enabled, a user can save a
|
||||
personal Amazon Bedrock API key under their
|
||||
[Agents API keys](../agents/models.md#user-api-keys-byok). AI Gateway then
|
||||
sends that key as a bearer token for the user's requests instead of signing
|
||||
them with the deployment's AWS credentials. Requests from users without a
|
||||
personal key continue to use the credentials below.
|
||||
When [BYOK](auth.md#bring-your-own-key-byok) is enabled, a user can save a personal Amazon Bedrock API key under their [Agents API keys](../agents/models.md#user-api-keys-byok).
|
||||
AI Gateway then sends that key as a bearer token for the user's requests instead of signing them with the deployment's AWS credentials.
|
||||
Requests from users without a personal key continue to use the credentials below.
|
||||
|
||||
AI Gateway resolves AWS credentials one of three ways:
|
||||
|
||||
- **AWS SDK default credential chain (recommended).** When no explicit
|
||||
credentials are configured, the AWS SDK resolves them automatically
|
||||
from the environment: IAM Roles (instance profiles, IRSA, ECS task
|
||||
roles), shared config files, environment variables, SSO, and more.
|
||||
Attaching an IAM Role to the compute running Coder follows
|
||||
[AWS best practices](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html)
|
||||
for temporary credentials. The role must permit `bedrock:InvokeModel`
|
||||
and `bedrock:InvokeModelWithResponseStream` for the configured models.
|
||||
- **Static credentials.** Provide an access key and secret for an IAM
|
||||
user with the same Bedrock permissions.
|
||||
- **Assumed IAM role.** Set a **Role ARN** to have the gateway assume
|
||||
that role before calling Bedrock, signing requests with the resulting
|
||||
temporary credentials. This works on top of either of the above base
|
||||
identities and supports cross-account Bedrock access. See
|
||||
[IAM role assumption](#iam-role-assumption).
|
||||
- **AWS SDK default credential chain (recommended).**
|
||||
When no explicit credentials are configured, the AWS SDK resolves them automatically from the environment: IAM Roles (instance profiles, IRSA, ECS task roles), shared config files, environment variables, SSO, and more.
|
||||
Attaching an IAM Role to the compute running Coder follows [AWS best practices](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html) for temporary credentials.
|
||||
The role must permit `bedrock:InvokeModel` and `bedrock:InvokeModelWithResponseStream` for the configured models.
|
||||
- **Static credentials.**
|
||||
Provide an access key and secret for an IAM user with the same Bedrock permissions.
|
||||
- **Assumed IAM role.**
|
||||
Set a **Role ARN** to have the gateway assume that role before calling Bedrock, signing requests with the resulting temporary credentials.
|
||||
This works on top of either of the above base identities and supports cross-account Bedrock access.
|
||||
See [IAM role assumption](#iam-role-assumption).
|
||||
|
||||
#### Static Bedrock credentials
|
||||
|
||||
When you cannot use the default credential chain, create a dedicated IAM
|
||||
user and generate a static access key:
|
||||
When you cannot use the default credential chain, create a dedicated IAM user and generate a static access key:
|
||||
|
||||
1. **Choose a region** where you want to use Bedrock.
|
||||
|
||||
@@ -235,49 +199,35 @@ user and generate a static access key:
|
||||
- Select **Create access key**.
|
||||
- Save both the access key ID and secret access key securely.
|
||||
|
||||
4. **Enter the access key ID and secret access key** when you add or edit
|
||||
the Bedrock provider from the dashboard or the
|
||||
[AI Providers API](../../reference/api/aiproviders.md), along with the
|
||||
region (or base URL) and model identifiers.
|
||||
4. **Enter the access key ID and secret access key** when you add or edit the Bedrock provider from the dashboard or the [AI Providers API](../../reference/api/aiproviders.md), along with the region (or base URL) and model identifiers.
|
||||
|
||||
#### IAM role assumption
|
||||
|
||||
Set the optional **Role ARN** field to have the gateway assume an IAM
|
||||
role before calling Bedrock. The base identity (static credentials or the
|
||||
default credential chain) signs an STS `AssumeRole` call, and the
|
||||
temporary credentials it returns sign Bedrock requests. The field is
|
||||
optional: a provider with no Role ARN authenticates with its base
|
||||
identity directly, exactly as described above.
|
||||
Set the optional **Role ARN** field to have the gateway assume an IAM role before calling Bedrock.
|
||||
The base identity (static credentials or the default credential chain) signs an STS `AssumeRole` call, and the temporary credentials it returns sign Bedrock requests.
|
||||
The field is optional: a provider with no Role ARN authenticates with its base identity directly, exactly as described above.
|
||||
|
||||
To use role assumption:
|
||||
|
||||
1. **Create the IAM role** in the target account and grant it
|
||||
`bedrock:InvokeModel` and `bedrock:InvokeModelWithResponseStream` for
|
||||
the configured models. The base identity does not need Bedrock
|
||||
permissions itself; the assumed role does.
|
||||
1. **Create the IAM role** in the target account and grant it `bedrock:InvokeModel` and `bedrock:InvokeModelWithResponseStream` for the configured models.
|
||||
The base identity does not need Bedrock permissions itself; the assumed role does.
|
||||
|
||||
2. **Configure the role's trust policy** to allow the gateway's base
|
||||
identity to assume it.
|
||||
2. **Configure the role's trust policy** to allow the gateway's base identity to assume it.
|
||||
|
||||
3. **Enter the Role ARN** when you add or edit the Bedrock provider. It must be a
|
||||
valid IAM role ARN, for example `arn:aws:iam::123456789012:role/BedrockRole`.
|
||||
3. **Enter the Role ARN** when you add or edit the Bedrock provider.
|
||||
It must be a valid IAM role ARN, for example `arn:aws:iam::123456789012:role/BedrockRole`.
|
||||
|
||||
Each provider assumes a single role. To use several roles, configure one
|
||||
provider per role.
|
||||
Each provider assumes a single role.
|
||||
To use several roles, configure one provider per role.
|
||||
|
||||
#### External ID
|
||||
|
||||
When a Bedrock provider assumes a role, the gateway generates a unique
|
||||
**external ID** for it and sends that value on every `AssumeRole` call.
|
||||
The external ID guards against the
|
||||
[confused deputy problem](https://docs.aws.amazon.com/IAM/latest/UserGuide/confused-deputy.html)
|
||||
on cross-account assumption. The gateway generates and owns the value, as
|
||||
AWS [recommends](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html):
|
||||
you cannot set or change it. It is not a secret, and it is shown on the
|
||||
provider's edit page once a Role ARN is configured.
|
||||
When a Bedrock provider assumes a role, the gateway generates a unique **external ID** for it and sends that value on every `AssumeRole` call.
|
||||
The external ID guards against the [confused deputy problem](https://docs.aws.amazon.com/IAM/latest/UserGuide/confused-deputy.html) on cross-account assumption.
|
||||
The gateway generates and owns the value, as AWS [recommends](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html): you cannot set or change it.
|
||||
It is not a secret, and it is shown on the provider's edit page once a Role ARN is configured.
|
||||
|
||||
To enforce it, add the external ID to the target role's trust policy as an
|
||||
`sts:ExternalId` condition:
|
||||
To enforce it, add the external ID to the target role's trust policy as an `sts:ExternalId` condition:
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -289,32 +239,24 @@ To enforce it, add the external ID to the target role's trust policy as an
|
||||
```
|
||||
|
||||
> [!IMPORTANT]
|
||||
> The gateway sends the external ID whether or not the trust policy checks
|
||||
> it. Until you add the `sts:ExternalId` condition, the value is sent but
|
||||
> not enforced, and the role can still be assumed without it. To rotate the
|
||||
> external ID, recreate the provider.
|
||||
> The gateway sends the external ID whether or not the trust policy checks it.
|
||||
> Until you add the `sts:ExternalId` condition, the value is sent but not enforced, and the role can still be assumed without it.
|
||||
> To rotate the external ID, recreate the provider.
|
||||
|
||||
#### Application inference profiles
|
||||
|
||||
For InvokeModel, the **model** and **small fast model** identifiers can be
|
||||
[application inference profile](https://docs.aws.amazon.com/bedrock/latest/userguide/cost-mgmt-application-inference-profiles.html)
|
||||
ARNs, which attribute Bedrock spend to a team or workload through AWS cost
|
||||
allocation tags.
|
||||
For InvokeModel, the **model** and **small fast model** identifiers can be [application inference profile](https://docs.aws.amazon.com/bedrock/latest/userguide/cost-mgmt-application-inference-profiles.html) ARNs, which attribute Bedrock spend to a team or workload through AWS cost allocation tags.
|
||||
|
||||
AI Gateway passes the profile upstream so AWS records the attribution, while
|
||||
internally resolving and using the underlying model identity, including for
|
||||
usage pricing.
|
||||
AI Gateway passes the profile upstream so AWS records the attribution, while internally resolving and using the underlying model identity, including for usage pricing.
|
||||
|
||||
Resolution requires a `GetInferenceProfile` call, so the AWS identity used by
|
||||
the gateway must have `bedrock:GetInferenceProfile` permission for the
|
||||
profile. Providers configured with plain model identifiers do not need this
|
||||
permission. If resolution fails, the provider is skipped.
|
||||
Resolution requires a `GetInferenceProfile` call, so the AWS identity used by the gateway must have `bedrock:GetInferenceProfile` permission for the profile.
|
||||
Providers configured with plain model identifiers do not need this permission.
|
||||
If resolution fails, the provider is skipped.
|
||||
|
||||
### GitHub Copilot
|
||||
|
||||
GitHub Copilot offers three plans: Individual, Business, and Enterprise,
|
||||
each with its own API endpoint. Add one `copilot` provider per plan your
|
||||
organization uses, setting the base URL accordingly:
|
||||
GitHub Copilot offers three plans: Individual, Business, and Enterprise, each with its own API endpoint.
|
||||
Add one `copilot` provider per plan your organization uses, setting the base URL accordingly:
|
||||
|
||||
| Plan | Base URL |
|
||||
|------------|--------------------------------------------|
|
||||
@@ -322,15 +264,12 @@ organization uses, setting the base URL accordingly:
|
||||
| Business | `https://api.business.githubcopilot.com` |
|
||||
| Enterprise | `https://api.enterprise.githubcopilot.com` |
|
||||
|
||||
Copilot providers authenticate with each user's request-time GitHub
|
||||
OAuth token, so do not attach API keys. For client-side setup (proxy,
|
||||
certificates, IDE configuration), see
|
||||
[GitHub Copilot client configuration](./clients/copilot.md).
|
||||
Copilot providers authenticate with each user's request-time GitHub OAuth token, so do not attach API keys.
|
||||
For client-side setup (proxy, certificates, IDE configuration), see [GitHub Copilot client configuration](./clients/copilot.md).
|
||||
|
||||
### ChatGPT
|
||||
|
||||
ChatGPT subscriptions (Plus, Pro, Business) are supported through a
|
||||
provider of type `openai` with a specific name and base URL:
|
||||
ChatGPT subscriptions (Plus, Pro, Business) are supported through a provider of type `openai` with a specific name and base URL:
|
||||
|
||||
| Field | Value |
|
||||
|----------|-----------------------------------------|
|
||||
@@ -338,26 +277,20 @@ provider of type `openai` with a specific name and base URL:
|
||||
| Name | `chatgpt` |
|
||||
| Base URL | `https://chatgpt.com/backend-api/codex` |
|
||||
|
||||
The name must be exactly `chatgpt`. It determines the route clients use
|
||||
to reach the provider: `/api/v2/ai-gateway/chatgpt/v1`. If no provider
|
||||
with this name exists, requests to that route fail with
|
||||
`404 route not supported`.
|
||||
The name must be exactly `chatgpt`.
|
||||
It determines the route clients use to reach the provider: `/api/v2/ai-gateway/chatgpt/v1`.
|
||||
If no provider with this name exists, requests to that route fail with `404 route not supported`.
|
||||
|
||||
Do not attach API keys. ChatGPT providers authenticate with each user's
|
||||
ChatGPT OAuth token through [BYOK](./auth.md#bring-your-own-key-byok),
|
||||
so BYOK must remain enabled. For client-side setup, see the
|
||||
[Codex CLI ChatGPT subscription configuration](./clients/codex.md#byok-chatgpt-subscription).
|
||||
Do not attach API keys.
|
||||
ChatGPT providers authenticate with each user's ChatGPT OAuth token through [BYOK](./auth.md#bring-your-own-key-byok), so BYOK must remain enabled.
|
||||
For client-side setup, see the [Codex CLI ChatGPT subscription configuration](./clients/codex.md#byok-chatgpt-subscription).
|
||||
|
||||
### OpenAI-compatible providers
|
||||
|
||||
Azure-hosted OpenAI, Google, OpenRouter, Vercel, and any other
|
||||
OpenAI-compatible service are configured with the matching type (or the
|
||||
generic `openai-compat`), the provider's OpenAI-compatible base URL, and
|
||||
an API key.
|
||||
Azure-hosted OpenAI, Google, OpenRouter, Vercel, and any other OpenAI-compatible service are configured with the matching type (or the generic `openai-compat`), the provider's OpenAI-compatible base URL, and an API key.
|
||||
|
||||
> [!NOTE]
|
||||
> See the [Supported APIs](./reference.md#supported-apis) section for
|
||||
> precise endpoint coverage and interception behavior.
|
||||
> See the [Supported APIs](./reference.md#supported-apis) section for precise endpoint coverage and interception behavior.
|
||||
|
||||
## Provider lifecycle
|
||||
|
||||
@@ -371,20 +304,18 @@ Every provider carries an explicit status, reported by the [`provider_info`](./m
|
||||
| `error` | The provider is enabled but cannot be built (missing credentials, bad config) | Requests fail; the error is surfaced in metrics |
|
||||
| `proxy_excluded` | Another enabled provider already claims this hostname; the AI Gateway Proxy routes traffic to the first claimant (by database name sort order, `ORDER BY name ASC`) | Proxy-routed requests go to the first claimant; direct routing (`/api/v2/ai-gateway/{name}/...`) still works for this provider |
|
||||
|
||||
Disabling a provider does not delete it, its credentials, or its
|
||||
historical interception data. Re-enabling restores it to service.
|
||||
Disabling a provider does not delete it, its credentials, or its historical interception data.
|
||||
Re-enabling restores it to service.
|
||||
|
||||
## Monitoring and reloads
|
||||
|
||||
Provider configuration changes take effect automatically, without
|
||||
restarting `coderd`. AI Gateway records the timestamp of each reload
|
||||
attempt and each successful reload, exposed as Prometheus metrics:
|
||||
Provider configuration changes take effect automatically, without restarting `coderd`.
|
||||
AI Gateway records the timestamp of each reload attempt and each successful reload, exposed as Prometheus metrics:
|
||||
|
||||
- `coder_ai_gateway_providers_last_reload_timestamp_seconds`
|
||||
- `coder_ai_gateway_providers_last_reload_success_timestamp_seconds`
|
||||
|
||||
If you run the [external proxy](./ai-gateway-proxy/index.md), it exposes
|
||||
the same pair under the `coder_ai_gateway_proxy_` prefix.
|
||||
If you run the [external proxy](./ai-gateway-proxy/index.md), it exposes the same pair under the `coder_ai_gateway_proxy_` prefix.
|
||||
Each [standalone gateway](./standalone.md) replica reloads providers independently and replaces only its own provider snapshot.
|
||||
If a reload fails, that replica retains its previous provider snapshot and continues serving from it.
|
||||
|
||||
@@ -394,42 +325,33 @@ Refer to [Monitoring](./monitoring.md#prometheus-metrics) for the full metric li
|
||||
|
||||
## Key failover
|
||||
|
||||
You can configure multiple centralized API keys for a single provider instance
|
||||
so that AI Gateway automatically retries with the next key when one fails. This
|
||||
is transparent to end users, and clients see no difference in behavior or need
|
||||
any configuration changes.
|
||||
You can configure multiple centralized API keys for a single provider instance so that AI Gateway automatically retries with the next key when one fails.
|
||||
This is transparent to end users, and clients see no difference in behavior or need any configuration changes.
|
||||
|
||||
Key failover is supported for **OpenAI** and **Anthropic** providers. Amazon
|
||||
Bedrock and GitHub Copilot do not support key failover.
|
||||
Key failover is supported for **OpenAI** and **Anthropic** providers.
|
||||
Amazon Bedrock and GitHub Copilot do not support key failover.
|
||||
|
||||
Multiple keys can be added per provider through the
|
||||
[AI Providers API](../../reference/api/aiproviders.md). Each provider supports
|
||||
a maximum of **5 keys**.
|
||||
Multiple keys can be added per provider through the [AI Providers API](../../reference/api/aiproviders.md).
|
||||
Each provider supports a maximum of **5 keys**.
|
||||
|
||||
### Failover behavior
|
||||
|
||||
Every request starts with the first key in the list. If a key is rate-limited
|
||||
(`429 Too Many Requests`) or fails authentication (`401 Unauthorized`), AI
|
||||
Gateway puts that key on a temporary cooldown and retries the request with the
|
||||
next available key. Keys recover automatically when the cooldown elapses, so
|
||||
failover stays transparent to end users. Any other response, including a
|
||||
`403 Forbidden`, is returned to the caller unchanged.
|
||||
Every request starts with the first key in the list.
|
||||
If a key is rate-limited (`429 Too Many Requests`) or fails authentication (`401 Unauthorized`), AI Gateway puts that key on a temporary cooldown and retries the request with the next available key.
|
||||
Keys recover automatically when the cooldown elapses, so failover stays transparent to end users.
|
||||
Any other response, including a `403 Forbidden`, is returned to the caller unchanged.
|
||||
|
||||
If all keys in the pool are exhausted, AI Gateway returns:
|
||||
|
||||
- `429 Too Many Requests` when at least one key is rate-limited, with a `Retry-After`
|
||||
header set to the shortest cooldown across all keys.
|
||||
- `429 Too Many Requests` when at least one key is rate-limited, with a `Retry-After` header set to the shortest cooldown across all keys.
|
||||
- `502 Bad Gateway` when every key is in an authentication-failure cooldown.
|
||||
The keys still recover automatically once their cooldowns elapse, so no `Retry-After` is sent.
|
||||
The keys still recover automatically once their cooldowns elapse, so no `Retry-After` is sent.
|
||||
|
||||
## Bring Your Own Key
|
||||
|
||||
A provider's configured credentials are the centralized default. When
|
||||
Bring Your Own Key (BYOK) is enabled, a user's own credential takes
|
||||
precedence over the provider's for that user's requests, and AI Gateway
|
||||
falls back to the provider credentials when the user has none. See
|
||||
[Authentication](./auth.md#bring-your-own-key-byok) for the BYOK flow
|
||||
and how to enable or disable it.
|
||||
A provider's configured credentials are the centralized default.
|
||||
When Bring Your Own Key (BYOK) is enabled, a user's own credential takes precedence over the provider's for that user's requests, and AI Gateway falls back to the provider credentials when the user has none.
|
||||
See [Authentication](./auth.md#bring-your-own-key-byok) for the BYOK flow and how to enable or disable it.
|
||||
|
||||
## Failure modes
|
||||
|
||||
|
||||
@@ -2,9 +2,9 @@
|
||||
title: Rebranding migration
|
||||
---
|
||||
|
||||
AI Bridge has been renamed to **AI Gateway**. This is a cosmetic rebrand to make
|
||||
the feature easier to understand. It changes user-visible names, configuration
|
||||
options, the canonical HTTP API path, and the Prometheus metric names.
|
||||
AI Bridge has been renamed to **AI Gateway**.
|
||||
This is a cosmetic rebrand to make the feature easier to understand.
|
||||
It changes user-visible names, configuration options, the canonical HTTP API path, and the Prometheus metric names.
|
||||
|
||||
> [!NOTE]
|
||||
> Deprecated aliases remain available for gateway and proxy controls.
|
||||
@@ -13,9 +13,7 @@ options, the canonical HTTP API path, and the Prometheus metric names.
|
||||
Adopt the `ai_gateway` names for gateway and proxy settings so your configuration matches the current documentation.
|
||||
|
||||
> [!IMPORTANT]
|
||||
> New settings added in every area except the database (configuration options,
|
||||
> environment variables, CLI flags, and API paths) will use only the new
|
||||
> `ai_gateway` name, with no `aibridge` alias.
|
||||
> New settings added in every area except the database (configuration options, environment variables, CLI flags, and API paths) will use only the new `ai_gateway` name, with no `aibridge` alias.
|
||||
|
||||
## At a glance
|
||||
|
||||
@@ -30,20 +28,20 @@ Adopt the `ai_gateway` names for gateway and proxy settings so your configuratio
|
||||
|
||||
## What did not change
|
||||
|
||||
- **No database changes.** Table and column names (for example,
|
||||
`aibridge_interceptions`) are unchanged. No migration runs and no data is
|
||||
rewritten on upgrade.
|
||||
- **Gateway and proxy settings.** Renaming options for logging, retention, or listener addresses does not change their behavior or defaults.
|
||||
- **Internal/library references.** Some internal package names, log fields, and
|
||||
library identifiers still use the `aibridge` name. These are not part of the
|
||||
supported configuration surface and do not affect operators.
|
||||
- **No database changes.**
|
||||
Table and column names (for example, `aibridge_interceptions`) are unchanged.
|
||||
No migration runs and no data is rewritten on upgrade.
|
||||
- **Gateway and proxy settings.**
|
||||
Renaming options for logging, retention, or listener addresses does not change their behavior or defaults.
|
||||
- **Internal/library references.**
|
||||
Some internal package names, log fields, and library identifiers still use the `aibridge` name.
|
||||
These are not part of the supported configuration surface and do not affect operators.
|
||||
|
||||
## Configuration (env vars, flags, YAML)
|
||||
|
||||
The aliases listed below apply only to gateway and proxy controls, not provider setup.
|
||||
|
||||
If both a previous name and a new name are set for the same setting, set only one (prefer
|
||||
the new name).
|
||||
If both a previous name and a new name are set for the same setting, set only one (prefer the new name).
|
||||
|
||||
### Naming rules
|
||||
|
||||
@@ -51,9 +49,8 @@ The rename is a mechanical substitution:
|
||||
|
||||
- Environment variables: `CODER_AIBRIDGE_` becomes `CODER_AI_GATEWAY_`.
|
||||
- CLI flags: `--aibridge-` becomes `--ai-gateway-`.
|
||||
- YAML: only the top-level group key changes
|
||||
(`aibridge:` becomes `ai_gateway:`, `aibridgeproxy:` becomes
|
||||
`ai_gateway_proxy:`). The keys nested under the group are unchanged.
|
||||
- YAML: only the top-level group key changes (`aibridge:` becomes `ai_gateway:`, `aibridgeproxy:` becomes `ai_gateway_proxy:`).
|
||||
The keys nested under the group are unchanged.
|
||||
|
||||
### YAML example
|
||||
|
||||
@@ -115,22 +112,17 @@ AI Gateway Proxy settings:
|
||||
| `CODER_AIBRIDGE_PROXY_DUMP_DIR` | `CODER_AI_GATEWAY_PROXY_DUMP_DIR` | |
|
||||
| `CODER_AIBRIDGE_PROXY_DOMAIN_ALLOWLIST` | `CODER_AI_GATEWAY_PROXY_DOMAIN_ALLOWLIST` | Already deprecated; has no effect |
|
||||
|
||||
CLI flags follow the same mapping with the `--aibridge-*` to `--ai-gateway-*`
|
||||
prefix change.
|
||||
CLI flags follow the same mapping with the `--aibridge-*` to `--ai-gateway-*` prefix change.
|
||||
|
||||
## HTTP API
|
||||
|
||||
The canonical API path is now `/api/v2/ai-gateway` (and
|
||||
`/api/v2/ai-gateway/proxy`). The legacy `/api/v2/aibridge` and
|
||||
`/api/v2/aibridge/proxy` routes are retained for backward compatibility and
|
||||
continue to serve the same handlers.
|
||||
The canonical API path is now `/api/v2/ai-gateway` (and `/api/v2/ai-gateway/proxy`).
|
||||
The legacy `/api/v2/aibridge` and `/api/v2/aibridge/proxy` routes are retained for backward compatibility and continue to serve the same handlers.
|
||||
|
||||
If you have external integrations or agents calling the API directly, update them to the
|
||||
new path at your convenience. No immediate action is required.
|
||||
If you have external integrations or agents calling the API directly, update them to the new path at your convenience.
|
||||
No immediate action is required.
|
||||
|
||||
AI clients (such as Claude Code, Codex, and other tools) that are configured
|
||||
with a base URL pointing at the legacy `/api/v2/aibridge` path continue to work,
|
||||
but should be updated to the new `/api/v2/ai-gateway` base URL.
|
||||
AI clients (such as Claude Code, Codex, and other tools) that are configured with a base URL pointing at the legacy `/api/v2/aibridge` path continue to work, but should be updated to the new `/api/v2/ai-gateway` base URL.
|
||||
|
||||
## Metrics
|
||||
|
||||
@@ -141,25 +133,18 @@ The metric prefixes have been renamed:
|
||||
| `coder_aibridged_*` | `coder_ai_gateway_*` |
|
||||
| `coder_aibridgeproxyd_*` | `coder_ai_gateway_proxy_*` |
|
||||
|
||||
**Both the old and new metric names are emitted simultaneously today.** Every
|
||||
series is exported under both prefixes from the same underlying collector, so
|
||||
existing dashboards, alerts, and recording rules keep working immediately after
|
||||
upgrade with no changes.
|
||||
**Both the old and new metric names are emitted simultaneously today.**
|
||||
Every series is exported under both prefixes from the same underlying collector, so existing dashboards, alerts, and recording rules keep working immediately after upgrade with no changes.
|
||||
|
||||
The old prefixes are retained for backward compatibility with no planned removal
|
||||
date. To keep your observability aligned with the new names:
|
||||
The old prefixes are retained for backward compatibility with no planned removal date.
|
||||
To keep your observability aligned with the new names:
|
||||
|
||||
1. Update Grafana dashboards, Prometheus alerting rules, and recording rules to
|
||||
reference the new `coder_ai_gateway_*` and `coder_ai_gateway_proxy_*` names.
|
||||
2. Verify the new series are present in your monitoring stack (they are emitted
|
||||
as of this release).
|
||||
1. Update Grafana dashboards, Prometheus alerting rules, and recording rules to reference the new `coder_ai_gateway_*` and `coder_ai_gateway_proxy_*` names.
|
||||
2. Verify the new series are present in your monitoring stack (they are emitted as of this release).
|
||||
|
||||
### Optional: dropping the old names
|
||||
|
||||
If you have already migrated to the new names and do not want both prefixes
|
||||
ingested (for example, to avoid doubling cardinality in your time-series
|
||||
database), you can drop the deprecated series at scrape time with Prometheus
|
||||
`metric_relabel_configs`:
|
||||
If you have already migrated to the new names and do not want both prefixes ingested (for example, to avoid doubling cardinality in your time-series database), you can drop the deprecated series at scrape time with Prometheus `metric_relabel_configs`:
|
||||
|
||||
```yaml
|
||||
metric_relabel_configs:
|
||||
@@ -168,7 +153,5 @@ metric_relabel_configs:
|
||||
action: drop
|
||||
```
|
||||
|
||||
This keeps the canonical `coder_ai_gateway_*` and `coder_ai_gateway_proxy_*`
|
||||
series and discards the deprecated `coder_aibridged_*` and
|
||||
`coder_aibridgeproxyd_*` ones before they are stored. Only do this once your
|
||||
dashboards, alerts, and recording rules reference the new names.
|
||||
This keeps the canonical `coder_ai_gateway_*` and `coder_ai_gateway_proxy_*` series and discards the deprecated `coder_aibridged_*` and `coder_aibridgeproxyd_*` ones before they are stored.
|
||||
Only do this once your dashboards, alerts, and recording rules reference the new names.
|
||||
|
||||
@@ -3,8 +3,7 @@ title: Reference
|
||||
---
|
||||
|
||||
> [!NOTE]
|
||||
> AI Gateway is part of [AI Governance](../ai-governance.md), which is
|
||||
> included with a Premium license.
|
||||
> AI Gateway is part of [AI Governance](../ai-governance.md), which is included with a Premium license.
|
||||
|
||||
## Deployment topologies
|
||||
|
||||
@@ -34,7 +33,8 @@ The control connection carries the following:
|
||||
- AI budget checks, which reject requests from users over their spend limit.
|
||||
- Provider configuration, plus a change signal when the provider set changes.
|
||||
- AI session records.
|
||||
- **Deprecated**: the configuration and access tokens used by [injected MCP](./mcp.md).
|
||||
- **Deprecated**:
|
||||
the configuration and access tokens used by [injected MCP](./mcp.md).
|
||||
|
||||
Standalone replicas have no authoritative database state.
|
||||
They keep ephemeral provider snapshots, request caches, provider key pools, and metrics in memory, and emit their own logs and traces.
|
||||
@@ -98,10 +98,14 @@ Nonempty configured names must be unique, ignoring case.
|
||||
|
||||
You can't use any of these reserved names, regardless of casing:
|
||||
|
||||
- **Authentication and session**: `Authorization`, `X-Api-Key`, `Cookie`, `Set-Cookie`.
|
||||
- **Transport and forwarding**: `Host`, `User-Agent`, `Content-Length`, `Content-Type`, `Content-Encoding`, `Accept-Encoding`, `Connection`, `Keep-Alive`, `Te`, `Trailer`, `Transfer-Encoding`, `Upgrade`, `Forwarded`, `X-Forwarded-For`, `X-Forwarded-Host`, `X-Forwarded-Proto`, `X-Forwarded-Port`.
|
||||
- **Proxy**: `Proxy-Authorization`, `Proxy-Authenticate`.
|
||||
- **Coder internal**: `Coder-Session-Token`, `X-Coder-Ai-Governance-Token`, `X-Coder-Ai-Governance-Request-Id`, `X-Coder-Agent-Firewall-Session-Id`, `X-Coder-Agent-Firewall-Sequence-Number`.
|
||||
- **Authentication and session**:
|
||||
`Authorization`, `X-Api-Key`, `Cookie`, `Set-Cookie`.
|
||||
- **Transport and forwarding**:
|
||||
`Host`, `User-Agent`, `Content-Length`, `Content-Type`, `Content-Encoding`, `Accept-Encoding`, `Connection`, `Keep-Alive`, `Te`, `Trailer`, `Transfer-Encoding`, `Upgrade`, `Forwarded`, `X-Forwarded-For`, `X-Forwarded-Host`, `X-Forwarded-Proto`, `X-Forwarded-Port`.
|
||||
- **Proxy**:
|
||||
`Proxy-Authorization`, `Proxy-Authenticate`.
|
||||
- **Coder internal**:
|
||||
`Coder-Session-Token`, `X-Coder-Ai-Governance-Token`, `X-Coder-Ai-Governance-Request-Id`, `X-Coder-Agent-Firewall-Session-Id`, `X-Coder-Agent-Firewall-Sequence-Number`.
|
||||
|
||||
Names beginning with `X-AI-Bridge-Actor` are also reserved, ignoring case, except for the standard name of the attribute you're configuring.
|
||||
For example, `X-User-ID` and `X-Username` are valid custom names.
|
||||
@@ -111,8 +115,10 @@ An invalid configuration prevents `coderd` or a standalone AI Gateway from start
|
||||
|
||||
API support is divided into two categories:
|
||||
|
||||
- **Intercepted**: Requests are intercepted, audited, and augmented.
|
||||
- **Passthrough**: Requests are proxied directly to the upstream provider without auditing or augmentation.
|
||||
- **Intercepted**:
|
||||
Requests are intercepted, audited, and augmented.
|
||||
- **Passthrough**:
|
||||
Requests are proxied directly to the upstream provider without auditing or augmentation.
|
||||
|
||||
Where relevant, both streaming and non-streaming requests are supported.
|
||||
Paths are relative to the provider's base URL, such as `https://ai-gateway.example.com/openai/v1` or `https://ai-gateway.example.com/anthropic`.
|
||||
|
||||
@@ -13,8 +13,7 @@ Refer to [Database management of providers](./providers.md#database-management-o
|
||||
|
||||
## Activation
|
||||
|
||||
The AI Gateway feature must be enabled in the Coder deployment configuration before
|
||||
embedded or standalone gateway instances can serve authenticated traffic.
|
||||
The AI Gateway feature must be enabled in the Coder deployment configuration before embedded or standalone gateway instances can serve authenticated traffic.
|
||||
|
||||
_AI Gateway is enabled by default as of v2.34._
|
||||
|
||||
@@ -33,9 +32,9 @@ It is required for gateway key management endpoints to work and for standalone r
|
||||
|
||||
Configure at least one provider before exposing AI Gateway to end users.
|
||||
|
||||
Providers are deployment-scoped. Add them from the dashboard or the
|
||||
[AI Providers API](../../reference/api/aiproviders.md). Changes take effect
|
||||
without restarting `coderd`.
|
||||
Providers are deployment-scoped.
|
||||
Add them from the dashboard or the [AI Providers API](../../reference/api/aiproviders.md).
|
||||
Changes take effect without restarting `coderd`.
|
||||
|
||||
### Dashboard
|
||||
|
||||
@@ -46,28 +45,24 @@ without restarting `coderd`.
|
||||
1. Enter a unique lowercase name, the upstream endpoint, and the credentials
|
||||
1. Save the provider
|
||||
|
||||
Each provider gets its own AI Gateway route at
|
||||
`/api/v2/ai-gateway/<provider-name>/`.
|
||||
Each provider gets its own AI Gateway route at `/api/v2/ai-gateway/<provider-name>/`.
|
||||
|
||||
> [!NOTE]
|
||||
> Provider names must be unique and use lowercase, hyphen-separated identifiers
|
||||
> such as `anthropic-corp` or `azure-openai`. Once deleted, another provider
|
||||
> may reuse the name.
|
||||
> Provider names must be unique and use lowercase, hyphen-separated identifiers such as `anthropic-corp` or `azure-openai`.
|
||||
> Once deleted, another provider may reuse the name.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Open an existing provider to rotate credentials, update its endpoint, or
|
||||
disable it without restarting `coderd`.
|
||||
Open an existing provider to rotate credentials, update its endpoint, or disable it without restarting `coderd`.
|
||||
|
||||
|
||||
|
||||
## API Dumps
|
||||
|
||||
AI Gateway can dump provider request and response pairs to disk for debugging.
|
||||
Configure the dump directory with `--ai-gateway-dump-dir` or
|
||||
`CODER_AI_GATEWAY_DUMP_DIR`:
|
||||
Configure the dump directory with `--ai-gateway-dump-dir` or `CODER_AI_GATEWAY_DUMP_DIR`:
|
||||
|
||||
```sh
|
||||
coder server --ai-gateway-dump-dir=/var/lib/coder/ai-gateway-dumps
|
||||
@@ -81,22 +76,18 @@ ai_gateway:
|
||||
```
|
||||
|
||||
This top-level setting replaces the previous per-provider `DUMP_DIR` field.
|
||||
For each provider, AI Gateway writes dumps under `<base>/<provider_name>`, where
|
||||
`<base>` is the configured dump directory and `<provider_name>` is the provider
|
||||
instance name used in the route. For example, a provider named `anthropic-corp`
|
||||
with `/var/lib/coder/ai-gateway-dumps` configured writes to
|
||||
`/var/lib/coder/ai-gateway-dumps/anthropic-corp`.
|
||||
For each provider, AI Gateway writes dumps under `<base>/<provider_name>`, where `<base>` is the configured dump directory and `<provider_name>` is the provider instance name used in the route.
|
||||
For example, a provider named `anthropic-corp` with `/var/lib/coder/ai-gateway-dumps` configured writes to `/var/lib/coder/ai-gateway-dumps/anthropic-corp`.
|
||||
|
||||
Sensitive headers are redacted before dumps are written. Leave the value empty
|
||||
to disable dumping.
|
||||
Sensitive headers are redacted before dumps are written.
|
||||
Leave the value empty to disable dumping.
|
||||
|
||||
Each [standalone gateway](./standalone.md) replica accepts the same API dump settings and writes optional dumps to its own local disk.
|
||||
|
||||
> [!WARNING]
|
||||
> API dumps are intended for short diagnostic sessions only. Dump files contain
|
||||
> raw request and response data, which may include proprietary or sensitive
|
||||
> information such as prompts, completions, and tool inputs. Protect the target
|
||||
> directory and disable dumping when diagnostics are complete.
|
||||
> API dumps are intended for short diagnostic sessions only.
|
||||
> Dump files contain raw request and response data, which may include proprietary or sensitive information such as prompts, completions, and tool inputs.
|
||||
> Protect the target directory and disable dumping when diagnostics are complete.
|
||||
|
||||
## Send actor headers
|
||||
|
||||
@@ -126,8 +117,8 @@ For defaults, precedence, and privacy considerations, refer to [Actor header for
|
||||
|
||||
## Data Retention
|
||||
|
||||
AI Gateway records prompts, token usage, tool invocations, and model reasoning for auditing and
|
||||
monitoring purposes. By default, this data is retained for **60 days**.
|
||||
AI Gateway records prompts, token usage, tool invocations, and model reasoning for auditing and monitoring purposes.
|
||||
By default, this data is retained for **60 days**.
|
||||
|
||||
Configure retention using `--ai-gateway-retention` or `CODER_AI_GATEWAY_RETENTION`:
|
||||
|
||||
@@ -144,13 +135,11 @@ ai_gateway:
|
||||
|
||||
Set to `0` to retain data indefinitely.
|
||||
|
||||
For duration formats, how retention works, and best practices, see the
|
||||
[Data Retention](../../admin/setup/data-retention.md) documentation.
|
||||
For duration formats, how retention works, and best practices, see the [Data Retention](../../admin/setup/data-retention.md) documentation.
|
||||
|
||||
## Structured Logging
|
||||
|
||||
AI Gateway can emit structured logs for every interception record, making it
|
||||
straightforward to export data to external SIEM or observability platforms.
|
||||
AI Gateway can emit structured logs for every interception record, making it straightforward to export data to external SIEM or observability platforms.
|
||||
|
||||
Enable with `--ai-gateway-structured-logging` or `CODER_AI_GATEWAY_STRUCTURED_LOGGING`:
|
||||
|
||||
@@ -165,12 +154,8 @@ ai_gateway:
|
||||
structured_logging: true
|
||||
```
|
||||
|
||||
These logs are written to the same output stream as all other `coderd` logs,
|
||||
using the format configured by
|
||||
[`--log-human`](../../reference/cli/server/index.md#--log-human) (default, writes to
|
||||
stderr) or [`--log-json`](../../reference/cli/server/index.md#--log-json). For machine
|
||||
ingestion, set `--log-json` to a file path or `/dev/stderr` so that records are
|
||||
emitted as JSON.
|
||||
These logs are written to the same output stream as all other `coderd` logs, using the format configured by [`--log-human`](../../reference/cli/server/index.md#--log-human) (default, writes to stderr) or [`--log-json`](../../reference/cli/server/index.md#--log-json).
|
||||
For machine ingestion, set `--log-json` to a file path or `/dev/stderr` so that records are emitted as JSON.
|
||||
|
||||
### Choose which process emits the records
|
||||
|
||||
@@ -243,9 +228,8 @@ Disabling content recording is a governance trade-off, not only a storage saving
|
||||
|
||||
Token counts, spend, model and provider attribution, and request volume are all unaffected, as are the [Prometheus metrics](./monitoring.md) for prompt and tool call counts.
|
||||
|
||||
Filter for AI Gateway records in your logging pipeline by matching on the
|
||||
`"interception log"` message. Each log line includes a `record_type` field that
|
||||
indicates the kind of event captured:
|
||||
Filter for AI Gateway records in your logging pipeline by matching on the `"interception log"` message.
|
||||
Each log line includes a `record_type` field that indicates the kind of event captured:
|
||||
|
||||
| `record_type` | Description | Key fields |
|
||||
|----------------------|-----------------------------------------|--------------------------------------------------------------------------------|
|
||||
|
||||
Reference in New Issue
Block a user