mirror of
https://github.com/Canner/WrenAI.git
synced 2026-08-30 18:00:36 +08:00
docs: add MDL reference docs for Model, Relationship, and View (#1446)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,419 @@
|
||||
# Model
|
||||
|
||||
A **Model** is the core building block of Wren MDL. It maps a physical table (or a SQL expression) to a named semantic entity that AI agents and SQL clients query by name. Models define which columns are exposed and how columns relate to other models.
|
||||
|
||||
## Defining a Model
|
||||
|
||||
Every model requires three things:
|
||||
|
||||
1. A **name** — the identifier used in queries (`SELECT * FROM customers`)
|
||||
2. A **data source** — where the data lives (`table_reference`, `ref_sql`, or `base_object`)
|
||||
3. **Columns** — the fields that are exposed
|
||||
|
||||
### YAML format (wren project)
|
||||
|
||||
```yaml
|
||||
name: customers
|
||||
table_reference:
|
||||
catalog: jaffle_shop
|
||||
schema: main
|
||||
table: customers
|
||||
primary_key: customer_id
|
||||
columns:
|
||||
- name: customer_id
|
||||
type: INTEGER
|
||||
is_primary_key: true
|
||||
- name: first_name
|
||||
type: VARCHAR
|
||||
- name: last_name
|
||||
type: VARCHAR
|
||||
- name: number_of_orders
|
||||
type: BIGINT
|
||||
- name: customer_lifetime_value
|
||||
type: DOUBLE
|
||||
```
|
||||
|
||||
### JSON format (MDL manifest)
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "customers",
|
||||
"tableReference": {
|
||||
"catalog": "jaffle_shop",
|
||||
"schema": "main",
|
||||
"table": "customers"
|
||||
},
|
||||
"primaryKey": "customer_id",
|
||||
"columns": [
|
||||
{ "name": "customer_id", "type": "INTEGER", "isPrimaryKey": true, "isCalculated": false },
|
||||
{ "name": "first_name", "type": "VARCHAR", "isCalculated": false },
|
||||
{ "name": "last_name", "type": "VARCHAR", "isCalculated": false },
|
||||
{ "name": "number_of_orders", "type": "BIGINT", "isCalculated": false },
|
||||
{ "name": "customer_lifetime_value", "type": "DOUBLE", "isCalculated": false }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Model Fields
|
||||
|
||||
| Field | Required | Description |
|
||||
|-------|----------|-------------|
|
||||
| `name` | Yes | Unique identifier used in SQL queries |
|
||||
| `table_reference` | One of three | Points to an existing physical table (`catalog.schema.table`) |
|
||||
| `ref_sql` | One of three | A SQL SELECT statement used as the model's data source |
|
||||
| `base_object` | One of three | References another model or view as the base |
|
||||
| `columns` | Yes | List of columns to expose (see [Column Fields](#column-fields)) |
|
||||
| `primary_key` | No | Column name that uniquely identifies a row; required for relationships |
|
||||
| `properties` | No | Arbitrary key-value metadata (description, tags, etc.) |
|
||||
|
||||
## Data Source: Three Ways to Point at Data
|
||||
|
||||
### 1. `table_reference` — map to a physical table
|
||||
|
||||
Used when the underlying table already exists in the database.
|
||||
|
||||
**jaffle_shop example** — the `orders` model maps directly to `jaffle_shop.main.orders`:
|
||||
|
||||
```yaml
|
||||
name: orders
|
||||
table_reference:
|
||||
catalog: jaffle_shop
|
||||
schema: main
|
||||
table: orders
|
||||
```
|
||||
|
||||
When a query like `SELECT * FROM orders` is executed, Wren rewrites it to the fully-qualified physical table.
|
||||
|
||||
### 2. `ref_sql` — define the model inline with SQL (not yet supported)
|
||||
|
||||
Used when the model is derived — for example, a staging transform or a complex join that doesn't exist as a physical table.
|
||||
|
||||
```yaml
|
||||
name: stg_orders
|
||||
ref_sql: >
|
||||
SELECT
|
||||
id AS order_id,
|
||||
user_id AS customer_id,
|
||||
order_date,
|
||||
status
|
||||
FROM jaffle_shop.main.raw_orders
|
||||
```
|
||||
|
||||
### 3. `base_object` — inherit from another model (not yet supported)
|
||||
|
||||
References an existing model or view by name as the base. Useful for layered modeling (raw → staging → mart).
|
||||
|
||||
```yaml
|
||||
name: active_orders
|
||||
base_object: orders
|
||||
columns:
|
||||
- name: order_id
|
||||
type: INTEGER
|
||||
- name: order_date
|
||||
type: DATE
|
||||
```
|
||||
|
||||
## Column Fields
|
||||
|
||||
| Field | Required | Description |
|
||||
|-------|----------|-------------|
|
||||
| `name` | Yes | Column name used in SQL |
|
||||
| `type` | Yes | SQL data type (`VARCHAR`, `INTEGER`, `DOUBLE`, `DATE`, `TIMESTAMP`, etc.) |
|
||||
| `is_calculated` | No | If `true`, the column value is derived from `expression` at query time |
|
||||
| `expression` | No | SQL expression for calculated columns |
|
||||
| `relationship` | No | Name of a [Relationship](./relationship.md) — makes this column a join handle |
|
||||
| `not_null` | No | Constraint hint; `false` by default |
|
||||
| `is_primary_key` | No | Marks the column as the model's primary key |
|
||||
| `is_hidden` | No | Engine-internal flag; column is excluded from the symbol table and invisible to all clients |
|
||||
| `properties` | No | Arbitrary metadata |
|
||||
|
||||
### Regular columns
|
||||
|
||||
A regular column maps to a field in the underlying table. These are called **source columns** — the engine registers them in the physical schema so DataFusion can read them directly.
|
||||
|
||||
By default, the model column name is used as the physical field name. If the physical column has a different name, use `expression` to declare a simple rename:
|
||||
|
||||
```yaml
|
||||
- name: order_date # model name (exposed to clients)
|
||||
type: DATE
|
||||
is_calculated: false
|
||||
|
||||
- name: customer_id # renamed from the physical column "usr_id"
|
||||
type: INTEGER
|
||||
is_calculated: false
|
||||
expression: usr_id
|
||||
```
|
||||
|
||||
The `expression` on a non-calculated column must be a single column reference — it cannot contain operators or function calls. See [Engine Internals](#engine-internals) for the full resolution rules.
|
||||
|
||||
### Calculated columns
|
||||
|
||||
A calculated column is computed from a SQL expression at query time. Wren inlines the expression into the generated SQL.
|
||||
|
||||
```yaml
|
||||
- name: is_large_order
|
||||
type: BOOLEAN
|
||||
is_calculated: true
|
||||
expression: "amount > 100"
|
||||
```
|
||||
|
||||
Calculated columns can reference other columns in the same model or traverse relationships:
|
||||
|
||||
```yaml
|
||||
- name: customer_name
|
||||
type: VARCHAR
|
||||
is_calculated: true
|
||||
expression: "customers.first_name || ' ' || customers.last_name"
|
||||
relationship: orders_customers
|
||||
```
|
||||
|
||||
### Relationship columns
|
||||
|
||||
A relationship column declares a join path to another model. The `relationship` field names a [Relationship](./relationship.md) defined elsewhere in the MDL.
|
||||
|
||||
```yaml
|
||||
# In the orders model
|
||||
- name: customer
|
||||
type: customers # the related model name
|
||||
relationship: orders_customers
|
||||
```
|
||||
|
||||
This makes `orders.customer.first_name` valid SQL — Wren resolves the join automatically.
|
||||
|
||||
## jaffle_shop Example
|
||||
|
||||
The jaffle_shop dataset has three layers of models that illustrate the full range of modeling patterns:
|
||||
|
||||
```
|
||||
raw_orders ──► stg_orders ──► orders
|
||||
raw_customers ──► stg_customers ──► customers
|
||||
raw_payments ──► stg_payments
|
||||
```
|
||||
|
||||
### Raw layer — `table_reference`
|
||||
|
||||
Raw models point directly at source tables with minimal transformation:
|
||||
|
||||
```yaml
|
||||
name: raw_orders
|
||||
table_reference:
|
||||
catalog: jaffle_shop
|
||||
schema: main
|
||||
table: raw_orders
|
||||
primary_key: id
|
||||
columns:
|
||||
- { name: id, type: INTEGER, is_primary_key: true }
|
||||
- { name: user_id, type: INTEGER }
|
||||
- { name: order_date, type: DATE }
|
||||
- { name: status, type: VARCHAR }
|
||||
```
|
||||
|
||||
### Staging layer — renamed and typed
|
||||
|
||||
Staging models clean column names and enforce types. They use `ref_sql` or point at staging tables:
|
||||
|
||||
```yaml
|
||||
name: stg_orders
|
||||
table_reference:
|
||||
catalog: jaffle_shop
|
||||
schema: main
|
||||
table: stg_orders
|
||||
primary_key: order_id
|
||||
columns:
|
||||
- { name: order_id, type: INTEGER, is_primary_key: true }
|
||||
- { name: customer_id, type: INTEGER }
|
||||
- { name: order_date, type: DATE }
|
||||
- { name: status, type: VARCHAR }
|
||||
```
|
||||
|
||||
### Mart layer — enriched with metrics
|
||||
|
||||
Mart models expose business-ready fields, including pre-aggregated metrics:
|
||||
|
||||
```yaml
|
||||
name: customers
|
||||
table_reference:
|
||||
catalog: jaffle_shop
|
||||
schema: main
|
||||
table: customers
|
||||
primary_key: customer_id
|
||||
columns:
|
||||
- { name: customer_id, type: INTEGER, is_primary_key: true }
|
||||
- { name: first_name, type: VARCHAR }
|
||||
- { name: last_name, type: VARCHAR }
|
||||
- { name: first_order, type: DATE }
|
||||
- { name: most_recent_order, type: DATE }
|
||||
- { name: number_of_orders, type: BIGINT }
|
||||
- { name: customer_lifetime_value, type: DOUBLE }
|
||||
```
|
||||
|
||||
### Cross-model relationships
|
||||
|
||||
The `orders_customers` relationship (defined in `relationships.yml`) links `orders.customer_id → customers.customer_id`. With this in place, you can query across models without writing any JOIN:
|
||||
|
||||
```sql
|
||||
-- Wren resolves the join automatically
|
||||
SELECT
|
||||
order_id,
|
||||
orders.customer.first_name,
|
||||
orders.customer.last_name,
|
||||
amount
|
||||
FROM orders
|
||||
WHERE orders.customer.number_of_orders > 3
|
||||
```
|
||||
|
||||
See [Relationship](./relationship.md) for full details on defining join paths.
|
||||
|
||||
## Using Models in SQL
|
||||
|
||||
Once defined, models are first-class SQL table names:
|
||||
|
||||
```sql
|
||||
SELECT * FROM customers;
|
||||
|
||||
SELECT o.order_id, o.amount, c.first_name
|
||||
FROM orders o
|
||||
JOIN customers c ON o.customer_id = c.customer_id;
|
||||
|
||||
-- Or let Wren handle the join via a relationship column:
|
||||
SELECT order_id, customer.first_name, amount
|
||||
FROM orders;
|
||||
```
|
||||
|
||||
Wren translates these queries to the appropriate dialect SQL for the connected data source before execution.
|
||||
|
||||
## Column-Level Access Control via Selective Exposure
|
||||
|
||||
A model does not have to expose every column in the underlying table. By explicitly listing only the columns a client should see, you create a hard boundary at the semantic layer — columns that are not declared in the model simply do not exist from the client's perspective.
|
||||
|
||||
This is especially valuable in the AI era. When an AI agent (Claude, Cursor, Cline, etc.) connects through Wren MCP, it can only discover and query the columns that are declared in the model. Sensitive fields that are omitted from the model are physically invisible to the agent — no prompt injection or accidental exposure can retrieve them.
|
||||
|
||||
### Example: hiding PII from AI agents
|
||||
|
||||
Suppose the physical `customers` table contains PII columns that should never reach an AI agent:
|
||||
|
||||
| Physical column | Expose to AI? |
|
||||
|-----------------|--------------|
|
||||
| `customer_id` | Yes |
|
||||
| `first_name` | Yes |
|
||||
| `last_name` | Yes |
|
||||
| `email` | **No** |
|
||||
| `phone_number` | **No** |
|
||||
| `date_of_birth` | **No** |
|
||||
| `number_of_orders` | Yes |
|
||||
| `customer_lifetime_value` | Yes |
|
||||
|
||||
Define the model with only the safe columns:
|
||||
|
||||
```yaml
|
||||
name: customers
|
||||
table_reference:
|
||||
catalog: jaffle_shop
|
||||
schema: main
|
||||
table: customers
|
||||
primary_key: customer_id
|
||||
columns:
|
||||
- { name: customer_id, type: INTEGER, is_primary_key: true }
|
||||
- { name: first_name, type: VARCHAR }
|
||||
- { name: last_name, type: VARCHAR }
|
||||
- { name: number_of_orders, type: BIGINT }
|
||||
- { name: customer_lifetime_value, type: DOUBLE }
|
||||
```
|
||||
|
||||
The AI agent sees a `customers` model with five columns. `email`, `phone_number`, and `date_of_birth` do not appear in schema introspection, cannot be referenced in SQL, and are never included in query results — regardless of what the agent asks.
|
||||
|
||||
### Summary
|
||||
|
||||
| Technique | Column reachable via SQL | Visible in schema |
|
||||
|-----------|--------------------------|-------------------|
|
||||
| Declared column | Yes | Yes |
|
||||
| Omitted from model | **No** | **No** |
|
||||
|
||||
Use **omission** to enforce hard boundaries for AI agents.
|
||||
|
||||
## Engine Internals
|
||||
|
||||
### Physical schema registration (`infer_and_register_remote_table`)
|
||||
|
||||
When the engine initialises a model, it builds an Arrow schema that represents the physical table as DataFusion sees it. Only **source columns** — columns that map directly to a field in the underlying table — are registered in this schema. The engine uses `infer_source_column` to decide whether each column qualifies, following these rules in order:
|
||||
|
||||
| Column configuration | Source column? | Physical field name |
|
||||
|----------------------|---------------|---------------------|
|
||||
| `is_calculated: true` | No | — computed at query time from `expression` |
|
||||
| has `relationship` | No | — resolved as a join at query time |
|
||||
| no `expression` | **Yes** | same as `name` |
|
||||
| `expression` is a simple column reference | **Yes** | inferred from the expression (supports rename) |
|
||||
| `expression` is a complex SQL expression | No | — cannot be resolved statically |
|
||||
| `is_hidden: true` | **excluded** | stripped from the symbol table before this step |
|
||||
|
||||
### `is_hidden` — engine-internal columns
|
||||
|
||||
`is_hidden: true` is an engine-internal flag. The engine strips hidden columns from its symbol table during MDL initialisation (`get_visible_columns`), so they never appear in schema introspection, lineage analysis, or access-control checks. They are invisible to every client — AI agents, SQL clients, and the metadata API alike.
|
||||
|
||||
This is used for columns the engine generates internally (e.g. join keys added automatically for relationship resolution) that should not be addressable by user queries.
|
||||
|
||||
### `expression` on a non-calculated column — column rename
|
||||
|
||||
When `is_calculated` is `false` but an `expression` is present, the expression must be a **simple column reference**. The engine uses it to resolve which physical column to read and registers the model column name as an alias.
|
||||
|
||||
```yaml
|
||||
# Physical table has column "usr_id"; expose it as "customer_id" in the model
|
||||
- name: customer_id
|
||||
type: INTEGER
|
||||
is_calculated: false
|
||||
expression: usr_id
|
||||
```
|
||||
|
||||
At query time `SELECT customer_id FROM stg_orders` becomes `SELECT usr_id AS customer_id FROM ...` in the generated SQL.
|
||||
|
||||
If the expression is **compound** (`table.column`), the engine takes the last identifier as the physical column name:
|
||||
|
||||
```yaml
|
||||
- name: customer_id
|
||||
type: INTEGER
|
||||
is_calculated: false
|
||||
expression: raw_orders.user_id # physical name resolved as "user_id"
|
||||
```
|
||||
|
||||
If the expression cannot be reduced to a single identifier (e.g. `amount * 1.1`), the column is not registered as a source column — it must use `is_calculated: true` instead.
|
||||
|
||||
### `is_calculated` + `expression` — computed column
|
||||
|
||||
A calculated column is **never** registered as a source column. The engine inlines the `expression` SQL directly into the generated query at plan time:
|
||||
|
||||
```yaml
|
||||
- name: total_with_tax
|
||||
type: DOUBLE
|
||||
is_calculated: true
|
||||
expression: "amount * 1.1"
|
||||
```
|
||||
|
||||
Generated SQL: `SELECT amount * 1.1 AS total_with_tax FROM orders`
|
||||
|
||||
Calculated columns can also traverse relationship joins:
|
||||
|
||||
```yaml
|
||||
- name: customer_name
|
||||
type: VARCHAR
|
||||
is_calculated: true
|
||||
expression: "customers.first_name || ' ' || customers.last_name"
|
||||
relationship: orders_customers
|
||||
```
|
||||
|
||||
The engine resolves `customers.*` references by expanding the `orders_customers` join automatically.
|
||||
|
||||
### Summary
|
||||
|
||||
```
|
||||
Column definition
|
||||
│
|
||||
├── is_hidden: true → stripped from symbol table; invisible to all clients
|
||||
│
|
||||
├── is_calculated: true → inlined as SQL expression at query time
|
||||
│ └── relationship → join is expanded before inlining
|
||||
│
|
||||
├── no expression → direct physical column (name = physical name)
|
||||
│
|
||||
└── expression (simple) → rename: physical name from expression, model name as alias
|
||||
expression (complex) → must use is_calculated: true
|
||||
```
|
||||
@@ -0,0 +1,172 @@
|
||||
# Relationship
|
||||
|
||||
A **Relationship** defines a join path between two models. Once declared, the engine resolves the join automatically whenever a query traverses a relationship column — no explicit `JOIN` syntax required.
|
||||
|
||||
## Structure
|
||||
|
||||
```yaml
|
||||
# relationships.yml
|
||||
relationships:
|
||||
- name: orders_customers
|
||||
models:
|
||||
- orders
|
||||
- customers
|
||||
join_type: MANY_TO_ONE
|
||||
condition: orders.customer_id = customers.customer_id
|
||||
```
|
||||
|
||||
### JSON format (MDL manifest)
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "orders_customers",
|
||||
"models": ["orders", "customers"],
|
||||
"joinType": "MANY_TO_ONE",
|
||||
"condition": "orders.customer_id = customers.customer_id"
|
||||
}
|
||||
```
|
||||
|
||||
## Fields
|
||||
|
||||
| Field | Required | Description |
|
||||
|-------|----------|-------------|
|
||||
| `name` | Yes | Unique identifier referenced by relationship columns |
|
||||
| `models` | Yes | Exactly two model names: `[from_model, to_model]` |
|
||||
| `join_type` | Yes | Cardinality of the join (see below) |
|
||||
| `condition` | Yes | SQL equality expression linking the two models |
|
||||
|
||||
## Join Types
|
||||
|
||||
| Value | Meaning |
|
||||
|-------|---------|
|
||||
| `ONE_TO_ONE` | Each row in the left model matches at most one row in the right model |
|
||||
| `ONE_TO_MANY` | One row in the left model matches many rows in the right model |
|
||||
| `MANY_TO_ONE` | Many rows in the left model match one row in the right model |
|
||||
| `MANY_TO_MANY` | Many rows on both sides |
|
||||
|
||||
The join type affects how the engine handles aggregation in calculated columns that traverse the relationship. For `TO_ONE` joins (`ONE_TO_ONE`, `MANY_TO_ONE`), the engine uses a simple join. For `TO_MANY` joins, the engine wraps the traversal in an aggregate subquery to avoid row multiplication.
|
||||
|
||||
## The `condition` Field
|
||||
|
||||
The condition is an equality expression using fully-qualified `model.column` references:
|
||||
|
||||
```yaml
|
||||
condition: orders.customer_id = customers.customer_id
|
||||
```
|
||||
|
||||
- Always use `model_name.column_name` on both sides
|
||||
- Only equality conditions are supported
|
||||
- The first model in `models` should appear on the left side of the condition
|
||||
|
||||
## jaffle_shop Example
|
||||
|
||||
The jaffle_shop workspace defines five relationships across its three model layers:
|
||||
|
||||
```yaml
|
||||
relationships:
|
||||
# mart layer
|
||||
- name: orders_customers
|
||||
models: [orders, customers]
|
||||
join_type: MANY_TO_ONE
|
||||
condition: orders.customer_id = customers.customer_id
|
||||
|
||||
# raw layer
|
||||
- name: raw_orders_raw_customers
|
||||
models: [raw_orders, raw_customers]
|
||||
join_type: MANY_TO_ONE
|
||||
condition: raw_orders.user_id = raw_customers.id
|
||||
|
||||
- name: raw_payments_raw_orders
|
||||
models: [raw_payments, raw_orders]
|
||||
join_type: MANY_TO_ONE
|
||||
condition: raw_payments.order_id = raw_orders.id
|
||||
|
||||
# staging layer
|
||||
- name: stg_orders_stg_customers
|
||||
models: [stg_orders, stg_customers]
|
||||
join_type: MANY_TO_ONE
|
||||
condition: stg_orders.customer_id = stg_customers.customer_id
|
||||
|
||||
- name: stg_payments_stg_orders
|
||||
models: [stg_payments, stg_orders]
|
||||
join_type: MANY_TO_ONE
|
||||
condition: stg_payments.order_id = stg_orders.order_id
|
||||
```
|
||||
|
||||
## Using Relationships in Queries
|
||||
|
||||
### Implicit join via relationship column
|
||||
|
||||
Declare a relationship column in a model to expose a join path:
|
||||
|
||||
```yaml
|
||||
# orders model — add a relationship column pointing to customers
|
||||
columns:
|
||||
- name: customer
|
||||
type: customers
|
||||
relationship: orders_customers
|
||||
```
|
||||
|
||||
Then query across models without writing a JOIN:
|
||||
|
||||
```sql
|
||||
-- Wren expands the join automatically
|
||||
SELECT order_id, customer.first_name, customer.last_name, amount
|
||||
FROM orders
|
||||
WHERE customer.number_of_orders > 3
|
||||
ORDER BY amount DESC;
|
||||
```
|
||||
|
||||
The engine resolves `customer.*` by expanding the `orders_customers` join, pushing it only as far as the referenced columns require.
|
||||
|
||||
### Calculated columns that traverse relationships
|
||||
|
||||
Relationship columns can be referenced inside `is_calculated` expressions:
|
||||
|
||||
```yaml
|
||||
- name: customer_name
|
||||
type: VARCHAR
|
||||
is_calculated: true
|
||||
expression: "customer.first_name || ' ' || customer.last_name"
|
||||
relationship: orders_customers
|
||||
```
|
||||
|
||||
For `TO_MANY` relationships, aggregate functions are required to avoid row multiplication:
|
||||
|
||||
```yaml
|
||||
# In the customers model — count orders per customer
|
||||
- name: order_count
|
||||
type: BIGINT
|
||||
is_calculated: true
|
||||
expression: "count(orders.order_id)"
|
||||
relationship: orders_customers
|
||||
```
|
||||
|
||||
The engine detects the aggregate and automatically wraps the join in a subquery.
|
||||
|
||||
## Engine Internals
|
||||
|
||||
### Relationship resolution pipeline
|
||||
|
||||
When the query planner encounters a column reference like `orders.customer.first_name`:
|
||||
|
||||
1. **`ExpandWrenViewRule`** runs first to inline any view definitions
|
||||
2. **`ModelAnalyzeRule`** identifies the `customer` column as a relationship column pointing to `orders_customers`
|
||||
3. **`relation_chain`** resolves the join path, building a `LEFT JOIN customers ON orders.customer_id = customers.customer_id`
|
||||
4. The join is pushed only as far as the referenced columns require — unreferenced relationship columns do not produce joins
|
||||
|
||||
### `TO_MANY` and aggregate subqueries
|
||||
|
||||
The `primary_key` of the base model is required when the relationship is `TO_MANY`. The engine wraps the join in an aggregate subquery keyed on the primary key to prevent row multiplication:
|
||||
|
||||
```sql
|
||||
-- expression: count(orders.order_id) on customers model
|
||||
SELECT
|
||||
customers.customer_id,
|
||||
(SELECT count(orders.order_id)
|
||||
FROM orders
|
||||
WHERE orders.customer_id = customers.customer_id) AS order_count
|
||||
FROM customers
|
||||
```
|
||||
|
||||
If `primary_key` is not declared on the base model, the engine returns an error when a `TO_MANY` calculated column is used.
|
||||
@@ -0,0 +1,144 @@
|
||||
# View
|
||||
|
||||
A **View** is a named SQL query stored in the MDL. It behaves like a virtual table — clients can query it by name, and the engine inlines the `statement` SQL before execution. Unlike a Model, a View does not declare columns explicitly; its schema is inferred from the `statement` at query time.
|
||||
|
||||
## Structure
|
||||
|
||||
```yaml
|
||||
# views.yml
|
||||
views:
|
||||
- name: high_value_orders
|
||||
statement: >
|
||||
SELECT order_id, customer_id, amount
|
||||
FROM orders
|
||||
WHERE amount > 100
|
||||
```
|
||||
|
||||
### JSON format (MDL manifest)
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "high_value_orders",
|
||||
"statement": "SELECT order_id, customer_id, amount FROM orders WHERE amount > 100"
|
||||
}
|
||||
```
|
||||
|
||||
## Fields
|
||||
|
||||
| Field | Required | Description |
|
||||
|-------|----------|-------------|
|
||||
| `name` | Yes | Unique identifier used in SQL queries |
|
||||
| `statement` | Yes | A complete SQL SELECT statement; may reference other models or views |
|
||||
|
||||
## Model vs View
|
||||
|
||||
| | Model | View |
|
||||
|-|-------|------|
|
||||
| Data source | Physical table, `ref_sql`, or `base_object` | SQL `statement` |
|
||||
| Column declarations | Explicit (with types) | Inferred from `statement` |
|
||||
| Relationship columns | Supported | Not supported |
|
||||
| Calculated columns | Supported | Not supported |
|
||||
| Primary key | Supported | Not applicable |
|
||||
| Access control | Column omission, RLAC/CLAC | Column omission via `statement` |
|
||||
|
||||
Use a **Model** when you need typed columns, relationships, or calculated fields. Use a **View** for pre-built queries — dashboards, saved filters, or cross-model aggregations — that you want to expose as a named table.
|
||||
|
||||
## jaffle_shop Example
|
||||
|
||||
The jaffle_shop workspace ships with an empty `views.yml` (`views: []`), but views become useful once you have mart-layer models in place. Here are representative examples:
|
||||
|
||||
### Simple filter view
|
||||
|
||||
```yaml
|
||||
- name: completed_orders
|
||||
statement: >
|
||||
SELECT order_id, customer_id, order_date, amount
|
||||
FROM orders
|
||||
WHERE status = 'completed'
|
||||
```
|
||||
|
||||
```sql
|
||||
SELECT * FROM completed_orders WHERE amount > 50;
|
||||
```
|
||||
|
||||
### Cross-model aggregation view
|
||||
|
||||
```yaml
|
||||
- name: customer_order_summary
|
||||
statement: >
|
||||
SELECT
|
||||
c.customer_id,
|
||||
c.first_name,
|
||||
c.last_name,
|
||||
COUNT(o.order_id) AS total_orders,
|
||||
SUM(o.amount) AS lifetime_value
|
||||
FROM customers c
|
||||
JOIN orders o ON c.customer_id = o.customer_id
|
||||
GROUP BY c.customer_id, c.first_name, c.last_name
|
||||
```
|
||||
|
||||
The `statement` references `customers` and `orders` by their model names. The engine resolves them through the normal model pipeline after expanding the view.
|
||||
|
||||
### View referencing another view
|
||||
|
||||
```yaml
|
||||
- name: vip_customers
|
||||
statement: >
|
||||
SELECT customer_id, first_name, last_name, lifetime_value
|
||||
FROM customer_order_summary
|
||||
WHERE lifetime_value > 500
|
||||
```
|
||||
|
||||
Views can reference other views. The engine expands all view references recursively before resolving model references.
|
||||
|
||||
## Querying a View
|
||||
|
||||
Once defined, a view is a first-class table name:
|
||||
|
||||
```sql
|
||||
SELECT * FROM completed_orders;
|
||||
|
||||
SELECT customer_id, total_orders
|
||||
FROM customer_order_summary
|
||||
ORDER BY total_orders DESC
|
||||
LIMIT 10;
|
||||
```
|
||||
|
||||
The view name can be qualified with catalog and schema:
|
||||
|
||||
```sql
|
||||
SELECT * FROM wren.main.completed_orders;
|
||||
```
|
||||
|
||||
## Engine Internals
|
||||
|
||||
### Session registration
|
||||
|
||||
At session initialisation, each view's `statement` is parsed into a DataFusion `LogicalPlan` and wrapped in a `ViewTable`. The `ViewTable` is registered under the view's fully-qualified name (`catalog.schema.name`) in the DataFusion catalog.
|
||||
|
||||
```
|
||||
view.statement → ctx.state().create_logical_plan()
|
||||
→ ViewTable::new(plan, statement)
|
||||
→ ctx.register_table(catalog.schema.name, view_table)
|
||||
```
|
||||
|
||||
### Query-time expansion: `ExpandWrenViewRule`
|
||||
|
||||
`ExpandWrenViewRule` runs as the **first** analyzer pass — before `ModelAnalyzeRule` and all other rules. It performs a bottom-up walk of the logical plan tree. Whenever it encounters a `TableScan` whose name belongs to the MDL and matches a registered view, it replaces the scan node with the view's `LogicalPlan` wrapped in a subquery alias:
|
||||
|
||||
```
|
||||
TableScan("completed_orders")
|
||||
↓ ExpandWrenViewRule
|
||||
Subquery(
|
||||
Filter(status = 'completed', TableScan("orders")),
|
||||
alias = "completed_orders"
|
||||
)
|
||||
```
|
||||
|
||||
After the view is inlined, the remaining `TableScan("orders")` nodes are processed by `ModelAnalyzeRule` in the next pass, which resolves them to physical tables.
|
||||
|
||||
This ordering ensures that a view's `statement` can freely reference other models or views — all references are resolved in subsequent passes after expansion.
|
||||
|
||||
### Recursive view expansion
|
||||
|
||||
If a view references another view, `ExpandWrenViewRule` handles the recursion automatically. The `transform_up_with_subqueries` traversal processes the tree from leaves to root, so inner views are expanded before outer views reference them.
|
||||
Reference in New Issue
Block a user