From a32f24b3ceddcc918cfb3865ee1389c942ccc234 Mon Sep 17 00:00:00 2001 From: Josh Lambert Date: Sun, 26 Apr 2026 17:52:01 -0400 Subject: [PATCH 1/3] docs: document custom model modalities --- .../code-with-ai/agents/custom-models.md | 21 +++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/packages/kilo-docs/pages/code-with-ai/agents/custom-models.md b/packages/kilo-docs/pages/code-with-ai/agents/custom-models.md index db8dc74c64..20e1180432 100644 --- a/packages/kilo-docs/pages/code-with-ai/agents/custom-models.md +++ b/packages/kilo-docs/pages/code-with-ai/agents/custom-models.md @@ -84,6 +84,7 @@ All fields are optional. When a model ID matches one already in the built-in cat | `reasoning` | `boolean` | Whether the model supports extended thinking | | `temperature` | `boolean` | Whether the model supports the temperature parameter | | `attachment` | `boolean` | Whether the model supports file attachments | +| `modalities` | `object` | Supported input and output types: `{ input, output }` | | `limit` | `object` | Token limits: `{ context, output, input? }` | | `cost` | `object` | Pricing per million tokens: `{ input, output, cache_read?, cache_write? }` | | `options` | `object` | Arbitrary provider-specific model options | @@ -91,6 +92,26 @@ All fields are optional. When a model ID matches one already in the built-in cat | `provider` | `object` | Override `{ npm?, api? }` — the AI SDK package or base API URL for this model | | `variants` | `object` | Named variant configurations (e.g., different reasoning efforts) | +### Modalities (modalities) + +The `modalities` object declares which content types the model can receive and produce. Each array can include `text`, `image`, `audio`, `video`, or `pdf`. + +| Sub-field | Type | Required | Description | +| --------- | ------- | -------- | ------------------------------------------------ | +| `input` | `array` | No | Content types the model accepts from the user | +| `output` | `array` | No | Content types the model can generate in response | + +For a standard text model that can also inspect images, use: + +```jsonc +"modalities": { + "input": ["text", "image"], + "output": ["text"] +} +``` + +If `modalities` is omitted and the model ID matches a models.dev catalog entry for that provider, Kilo uses the catalog's modalities. For completely custom models with no catalog match, Kilo defaults to text input and text output only. Set `attachment: true` alongside image, audio, video, or PDF input modalities when the provider supports sending those files as attachments. + ### Token Limits (limit) The `limit` object controls how Kilo manages the model's context window and output length. These values are specified in **tokens**. From 13df76ac06914efeb789f0a69e9677a8b8ce0c3b Mon Sep 17 00:00:00 2001 From: Josh Lambert Date: Mon, 27 Apr 2026 10:26:56 -0400 Subject: [PATCH 2/3] docs: clarify required modalities fields --- packages/kilo-docs/pages/code-with-ai/agents/custom-models.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/packages/kilo-docs/pages/code-with-ai/agents/custom-models.md b/packages/kilo-docs/pages/code-with-ai/agents/custom-models.md index 20e1180432..1081afe1c5 100644 --- a/packages/kilo-docs/pages/code-with-ai/agents/custom-models.md +++ b/packages/kilo-docs/pages/code-with-ai/agents/custom-models.md @@ -98,8 +98,8 @@ The `modalities` object declares which content types the model can receive and p | Sub-field | Type | Required | Description | | --------- | ------- | -------- | ------------------------------------------------ | -| `input` | `array` | No | Content types the model accepts from the user | -| `output` | `array` | No | Content types the model can generate in response | +| `input` | `array` | Yes | Content types the model accepts from the user | +| `output` | `array` | Yes | Content types the model can generate in response | For a standard text model that can also inspect images, use: From 552706bece31082f326cc0e954e007b6016640f5 Mon Sep 17 00:00:00 2001 From: Josh Lambert Date: Mon, 27 Apr 2026 11:28:45 -0400 Subject: [PATCH 3/3] docs: clarify modalities is optional but requires input/output when specified --- .../pages/code-with-ai/agents/custom-models.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/packages/kilo-docs/pages/code-with-ai/agents/custom-models.md b/packages/kilo-docs/pages/code-with-ai/agents/custom-models.md index 1081afe1c5..834ec18ce1 100644 --- a/packages/kilo-docs/pages/code-with-ai/agents/custom-models.md +++ b/packages/kilo-docs/pages/code-with-ai/agents/custom-models.md @@ -84,7 +84,7 @@ All fields are optional. When a model ID matches one already in the built-in cat | `reasoning` | `boolean` | Whether the model supports extended thinking | | `temperature` | `boolean` | Whether the model supports the temperature parameter | | `attachment` | `boolean` | Whether the model supports file attachments | -| `modalities` | `object` | Supported input and output types: `{ input, output }` | +| `modalities` | `object` | Optional. Supported input and output types: `{ input, output }` | | `limit` | `object` | Token limits: `{ context, output, input? }` | | `cost` | `object` | Pricing per million tokens: `{ input, output, cache_read?, cache_write? }` | | `options` | `object` | Arbitrary provider-specific model options | @@ -94,12 +94,12 @@ All fields are optional. When a model ID matches one already in the built-in cat ### Modalities (modalities) -The `modalities` object declares which content types the model can receive and produce. Each array can include `text`, `image`, `audio`, `video`, or `pdf`. +The `modalities` object declares which content types the model can receive and produce. It is optional — omit it to use defaults from the catalog or fallback to text-only. When `modalities` is provided, both `input` and `output` arrays are required. Each array can include `text`, `image`, `audio`, `video`, or `pdf`. -| Sub-field | Type | Required | Description | -| --------- | ------- | -------- | ------------------------------------------------ | -| `input` | `array` | Yes | Content types the model accepts from the user | -| `output` | `array` | Yes | Content types the model can generate in response | +| Sub-field | Type | Required | Description | +| --------- | ------- | ---------------- | ------------------------------------------------ | +| `input` | `array` | Yes (if present) | Content types the model accepts from the user | +| `output` | `array` | Yes (if present) | Content types the model can generate in response | For a standard text model that can also inspect images, use: