docs: add MDL reference docs for Model, Relationship, and View (#1446)

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Jax Liu
2026-03-16 13:38:55 +08:00
committed by GitHub
parent 516be5f67d
commit ec47ed4d95
3 changed files with 735 additions and 0 deletions
+419
View File
@@ -0,0 +1,419 @@
# Model
A **Model** is the core building block of Wren MDL. It maps a physical table (or a SQL expression) to a named semantic entity that AI agents and SQL clients query by name. Models define which columns are exposed and how columns relate to other models.
## Defining a Model
Every model requires three things:
1. A **name** — the identifier used in queries (`SELECT * FROM customers`)
2. A **data source** — where the data lives (`table_reference`, `ref_sql`, or `base_object`)
3. **Columns** — the fields that are exposed
### YAML format (wren project)
```yaml
name: customers
table_reference:
catalog: jaffle_shop
schema: main
table: customers
primary_key: customer_id
columns:
- name: customer_id
type: INTEGER
is_primary_key: true
- name: first_name
type: VARCHAR
- name: last_name
type: VARCHAR
- name: number_of_orders
type: BIGINT
- name: customer_lifetime_value
type: DOUBLE
```
### JSON format (MDL manifest)
```json
{
"name": "customers",
"tableReference": {
"catalog": "jaffle_shop",
"schema": "main",
"table": "customers"
},
"primaryKey": "customer_id",
"columns": [
{ "name": "customer_id", "type": "INTEGER", "isPrimaryKey": true, "isCalculated": false },
{ "name": "first_name", "type": "VARCHAR", "isCalculated": false },
{ "name": "last_name", "type": "VARCHAR", "isCalculated": false },
{ "name": "number_of_orders", "type": "BIGINT", "isCalculated": false },
{ "name": "customer_lifetime_value", "type": "DOUBLE", "isCalculated": false }
]
}
```
## Model Fields
| Field | Required | Description |
|-------|----------|-------------|
| `name` | Yes | Unique identifier used in SQL queries |
| `table_reference` | One of three | Points to an existing physical table (`catalog.schema.table`) |
| `ref_sql` | One of three | A SQL SELECT statement used as the model's data source |
| `base_object` | One of three | References another model or view as the base |
| `columns` | Yes | List of columns to expose (see [Column Fields](#column-fields)) |
| `primary_key` | No | Column name that uniquely identifies a row; required for relationships |
| `properties` | No | Arbitrary key-value metadata (description, tags, etc.) |
## Data Source: Three Ways to Point at Data
### 1. `table_reference` — map to a physical table
Used when the underlying table already exists in the database.
**jaffle_shop example** — the `orders` model maps directly to `jaffle_shop.main.orders`:
```yaml
name: orders
table_reference:
catalog: jaffle_shop
schema: main
table: orders
```
When a query like `SELECT * FROM orders` is executed, Wren rewrites it to the fully-qualified physical table.
### 2. `ref_sql` — define the model inline with SQL (not yet supported)
Used when the model is derived — for example, a staging transform or a complex join that doesn't exist as a physical table.
```yaml
name: stg_orders
ref_sql: >
SELECT
id AS order_id,
user_id AS customer_id,
order_date,
status
FROM jaffle_shop.main.raw_orders
```
### 3. `base_object` — inherit from another model (not yet supported)
References an existing model or view by name as the base. Useful for layered modeling (raw → staging → mart).
```yaml
name: active_orders
base_object: orders
columns:
- name: order_id
type: INTEGER
- name: order_date
type: DATE
```
## Column Fields
| Field | Required | Description |
|-------|----------|-------------|
| `name` | Yes | Column name used in SQL |
| `type` | Yes | SQL data type (`VARCHAR`, `INTEGER`, `DOUBLE`, `DATE`, `TIMESTAMP`, etc.) |
| `is_calculated` | No | If `true`, the column value is derived from `expression` at query time |
| `expression` | No | SQL expression for calculated columns |
| `relationship` | No | Name of a [Relationship](./relationship.md) — makes this column a join handle |
| `not_null` | No | Constraint hint; `false` by default |
| `is_primary_key` | No | Marks the column as the model's primary key |
| `is_hidden` | No | Engine-internal flag; column is excluded from the symbol table and invisible to all clients |
| `properties` | No | Arbitrary metadata |
### Regular columns
A regular column maps to a field in the underlying table. These are called **source columns** — the engine registers them in the physical schema so DataFusion can read them directly.
By default, the model column name is used as the physical field name. If the physical column has a different name, use `expression` to declare a simple rename:
```yaml
- name: order_date # model name (exposed to clients)
type: DATE
is_calculated: false
- name: customer_id # renamed from the physical column "usr_id"
type: INTEGER
is_calculated: false
expression: usr_id
```
The `expression` on a non-calculated column must be a single column reference — it cannot contain operators or function calls. See [Engine Internals](#engine-internals) for the full resolution rules.
### Calculated columns
A calculated column is computed from a SQL expression at query time. Wren inlines the expression into the generated SQL.
```yaml
- name: is_large_order
type: BOOLEAN
is_calculated: true
expression: "amount > 100"
```
Calculated columns can reference other columns in the same model or traverse relationships:
```yaml
- name: customer_name
type: VARCHAR
is_calculated: true
expression: "customers.first_name || ' ' || customers.last_name"
relationship: orders_customers
```
### Relationship columns
A relationship column declares a join path to another model. The `relationship` field names a [Relationship](./relationship.md) defined elsewhere in the MDL.
```yaml
# In the orders model
- name: customer
type: customers # the related model name
relationship: orders_customers
```
This makes `orders.customer.first_name` valid SQL — Wren resolves the join automatically.
## jaffle_shop Example
The jaffle_shop dataset has three layers of models that illustrate the full range of modeling patterns:
```
raw_orders ──► stg_orders ──► orders
raw_customers ──► stg_customers ──► customers
raw_payments ──► stg_payments
```
### Raw layer — `table_reference`
Raw models point directly at source tables with minimal transformation:
```yaml
name: raw_orders
table_reference:
catalog: jaffle_shop
schema: main
table: raw_orders
primary_key: id
columns:
- { name: id, type: INTEGER, is_primary_key: true }
- { name: user_id, type: INTEGER }
- { name: order_date, type: DATE }
- { name: status, type: VARCHAR }
```
### Staging layer — renamed and typed
Staging models clean column names and enforce types. They use `ref_sql` or point at staging tables:
```yaml
name: stg_orders
table_reference:
catalog: jaffle_shop
schema: main
table: stg_orders
primary_key: order_id
columns:
- { name: order_id, type: INTEGER, is_primary_key: true }
- { name: customer_id, type: INTEGER }
- { name: order_date, type: DATE }
- { name: status, type: VARCHAR }
```
### Mart layer — enriched with metrics
Mart models expose business-ready fields, including pre-aggregated metrics:
```yaml
name: customers
table_reference:
catalog: jaffle_shop
schema: main
table: customers
primary_key: customer_id
columns:
- { name: customer_id, type: INTEGER, is_primary_key: true }
- { name: first_name, type: VARCHAR }
- { name: last_name, type: VARCHAR }
- { name: first_order, type: DATE }
- { name: most_recent_order, type: DATE }
- { name: number_of_orders, type: BIGINT }
- { name: customer_lifetime_value, type: DOUBLE }
```
### Cross-model relationships
The `orders_customers` relationship (defined in `relationships.yml`) links `orders.customer_id → customers.customer_id`. With this in place, you can query across models without writing any JOIN:
```sql
-- Wren resolves the join automatically
SELECT
order_id,
orders.customer.first_name,
orders.customer.last_name,
amount
FROM orders
WHERE orders.customer.number_of_orders > 3
```
See [Relationship](./relationship.md) for full details on defining join paths.
## Using Models in SQL
Once defined, models are first-class SQL table names:
```sql
SELECT * FROM customers;
SELECT o.order_id, o.amount, c.first_name
FROM orders o
JOIN customers c ON o.customer_id = c.customer_id;
-- Or let Wren handle the join via a relationship column:
SELECT order_id, customer.first_name, amount
FROM orders;
```
Wren translates these queries to the appropriate dialect SQL for the connected data source before execution.
## Column-Level Access Control via Selective Exposure
A model does not have to expose every column in the underlying table. By explicitly listing only the columns a client should see, you create a hard boundary at the semantic layer — columns that are not declared in the model simply do not exist from the client's perspective.
This is especially valuable in the AI era. When an AI agent (Claude, Cursor, Cline, etc.) connects through Wren MCP, it can only discover and query the columns that are declared in the model. Sensitive fields that are omitted from the model are physically invisible to the agent — no prompt injection or accidental exposure can retrieve them.
### Example: hiding PII from AI agents
Suppose the physical `customers` table contains PII columns that should never reach an AI agent:
| Physical column | Expose to AI? |
|-----------------|--------------|
| `customer_id` | Yes |
| `first_name` | Yes |
| `last_name` | Yes |
| `email` | **No** |
| `phone_number` | **No** |
| `date_of_birth` | **No** |
| `number_of_orders` | Yes |
| `customer_lifetime_value` | Yes |
Define the model with only the safe columns:
```yaml
name: customers
table_reference:
catalog: jaffle_shop
schema: main
table: customers
primary_key: customer_id
columns:
- { name: customer_id, type: INTEGER, is_primary_key: true }
- { name: first_name, type: VARCHAR }
- { name: last_name, type: VARCHAR }
- { name: number_of_orders, type: BIGINT }
- { name: customer_lifetime_value, type: DOUBLE }
```
The AI agent sees a `customers` model with five columns. `email`, `phone_number`, and `date_of_birth` do not appear in schema introspection, cannot be referenced in SQL, and are never included in query results — regardless of what the agent asks.
### Summary
| Technique | Column reachable via SQL | Visible in schema |
|-----------|--------------------------|-------------------|
| Declared column | Yes | Yes |
| Omitted from model | **No** | **No** |
Use **omission** to enforce hard boundaries for AI agents.
## Engine Internals
### Physical schema registration (`infer_and_register_remote_table`)
When the engine initialises a model, it builds an Arrow schema that represents the physical table as DataFusion sees it. Only **source columns** — columns that map directly to a field in the underlying table — are registered in this schema. The engine uses `infer_source_column` to decide whether each column qualifies, following these rules in order:
| Column configuration | Source column? | Physical field name |
|----------------------|---------------|---------------------|
| `is_calculated: true` | No | — computed at query time from `expression` |
| has `relationship` | No | — resolved as a join at query time |
| no `expression` | **Yes** | same as `name` |
| `expression` is a simple column reference | **Yes** | inferred from the expression (supports rename) |
| `expression` is a complex SQL expression | No | — cannot be resolved statically |
| `is_hidden: true` | **excluded** | stripped from the symbol table before this step |
### `is_hidden` — engine-internal columns
`is_hidden: true` is an engine-internal flag. The engine strips hidden columns from its symbol table during MDL initialisation (`get_visible_columns`), so they never appear in schema introspection, lineage analysis, or access-control checks. They are invisible to every client — AI agents, SQL clients, and the metadata API alike.
This is used for columns the engine generates internally (e.g. join keys added automatically for relationship resolution) that should not be addressable by user queries.
### `expression` on a non-calculated column — column rename
When `is_calculated` is `false` but an `expression` is present, the expression must be a **simple column reference**. The engine uses it to resolve which physical column to read and registers the model column name as an alias.
```yaml
# Physical table has column "usr_id"; expose it as "customer_id" in the model
- name: customer_id
type: INTEGER
is_calculated: false
expression: usr_id
```
At query time `SELECT customer_id FROM stg_orders` becomes `SELECT usr_id AS customer_id FROM ...` in the generated SQL.
If the expression is **compound** (`table.column`), the engine takes the last identifier as the physical column name:
```yaml
- name: customer_id
type: INTEGER
is_calculated: false
expression: raw_orders.user_id # physical name resolved as "user_id"
```
If the expression cannot be reduced to a single identifier (e.g. `amount * 1.1`), the column is not registered as a source column — it must use `is_calculated: true` instead.
### `is_calculated` + `expression` — computed column
A calculated column is **never** registered as a source column. The engine inlines the `expression` SQL directly into the generated query at plan time:
```yaml
- name: total_with_tax
type: DOUBLE
is_calculated: true
expression: "amount * 1.1"
```
Generated SQL: `SELECT amount * 1.1 AS total_with_tax FROM orders`
Calculated columns can also traverse relationship joins:
```yaml
- name: customer_name
type: VARCHAR
is_calculated: true
expression: "customers.first_name || ' ' || customers.last_name"
relationship: orders_customers
```
The engine resolves `customers.*` references by expanding the `orders_customers` join automatically.
### Summary
```
Column definition
├── is_hidden: true → stripped from symbol table; invisible to all clients
├── is_calculated: true → inlined as SQL expression at query time
│ └── relationship → join is expanded before inlining
├── no expression → direct physical column (name = physical name)
└── expression (simple) → rename: physical name from expression, model name as alias
expression (complex) → must use is_calculated: true
```
+172
View File
@@ -0,0 +1,172 @@
# Relationship
A **Relationship** defines a join path between two models. Once declared, the engine resolves the join automatically whenever a query traverses a relationship column — no explicit `JOIN` syntax required.
## Structure
```yaml
# relationships.yml
relationships:
- name: orders_customers
models:
- orders
- customers
join_type: MANY_TO_ONE
condition: orders.customer_id = customers.customer_id
```
### JSON format (MDL manifest)
```json
{
"name": "orders_customers",
"models": ["orders", "customers"],
"joinType": "MANY_TO_ONE",
"condition": "orders.customer_id = customers.customer_id"
}
```
## Fields
| Field | Required | Description |
|-------|----------|-------------|
| `name` | Yes | Unique identifier referenced by relationship columns |
| `models` | Yes | Exactly two model names: `[from_model, to_model]` |
| `join_type` | Yes | Cardinality of the join (see below) |
| `condition` | Yes | SQL equality expression linking the two models |
## Join Types
| Value | Meaning |
|-------|---------|
| `ONE_TO_ONE` | Each row in the left model matches at most one row in the right model |
| `ONE_TO_MANY` | One row in the left model matches many rows in the right model |
| `MANY_TO_ONE` | Many rows in the left model match one row in the right model |
| `MANY_TO_MANY` | Many rows on both sides |
The join type affects how the engine handles aggregation in calculated columns that traverse the relationship. For `TO_ONE` joins (`ONE_TO_ONE`, `MANY_TO_ONE`), the engine uses a simple join. For `TO_MANY` joins, the engine wraps the traversal in an aggregate subquery to avoid row multiplication.
## The `condition` Field
The condition is an equality expression using fully-qualified `model.column` references:
```yaml
condition: orders.customer_id = customers.customer_id
```
- Always use `model_name.column_name` on both sides
- Only equality conditions are supported
- The first model in `models` should appear on the left side of the condition
## jaffle_shop Example
The jaffle_shop workspace defines five relationships across its three model layers:
```yaml
relationships:
# mart layer
- name: orders_customers
models: [orders, customers]
join_type: MANY_TO_ONE
condition: orders.customer_id = customers.customer_id
# raw layer
- name: raw_orders_raw_customers
models: [raw_orders, raw_customers]
join_type: MANY_TO_ONE
condition: raw_orders.user_id = raw_customers.id
- name: raw_payments_raw_orders
models: [raw_payments, raw_orders]
join_type: MANY_TO_ONE
condition: raw_payments.order_id = raw_orders.id
# staging layer
- name: stg_orders_stg_customers
models: [stg_orders, stg_customers]
join_type: MANY_TO_ONE
condition: stg_orders.customer_id = stg_customers.customer_id
- name: stg_payments_stg_orders
models: [stg_payments, stg_orders]
join_type: MANY_TO_ONE
condition: stg_payments.order_id = stg_orders.order_id
```
## Using Relationships in Queries
### Implicit join via relationship column
Declare a relationship column in a model to expose a join path:
```yaml
# orders model — add a relationship column pointing to customers
columns:
- name: customer
type: customers
relationship: orders_customers
```
Then query across models without writing a JOIN:
```sql
-- Wren expands the join automatically
SELECT order_id, customer.first_name, customer.last_name, amount
FROM orders
WHERE customer.number_of_orders > 3
ORDER BY amount DESC;
```
The engine resolves `customer.*` by expanding the `orders_customers` join, pushing it only as far as the referenced columns require.
### Calculated columns that traverse relationships
Relationship columns can be referenced inside `is_calculated` expressions:
```yaml
- name: customer_name
type: VARCHAR
is_calculated: true
expression: "customer.first_name || ' ' || customer.last_name"
relationship: orders_customers
```
For `TO_MANY` relationships, aggregate functions are required to avoid row multiplication:
```yaml
# In the customers model — count orders per customer
- name: order_count
type: BIGINT
is_calculated: true
expression: "count(orders.order_id)"
relationship: orders_customers
```
The engine detects the aggregate and automatically wraps the join in a subquery.
## Engine Internals
### Relationship resolution pipeline
When the query planner encounters a column reference like `orders.customer.first_name`:
1. **`ExpandWrenViewRule`** runs first to inline any view definitions
2. **`ModelAnalyzeRule`** identifies the `customer` column as a relationship column pointing to `orders_customers`
3. **`relation_chain`** resolves the join path, building a `LEFT JOIN customers ON orders.customer_id = customers.customer_id`
4. The join is pushed only as far as the referenced columns require — unreferenced relationship columns do not produce joins
### `TO_MANY` and aggregate subqueries
The `primary_key` of the base model is required when the relationship is `TO_MANY`. The engine wraps the join in an aggregate subquery keyed on the primary key to prevent row multiplication:
```sql
-- expression: count(orders.order_id) on customers model
SELECT
customers.customer_id,
(SELECT count(orders.order_id)
FROM orders
WHERE orders.customer_id = customers.customer_id) AS order_count
FROM customers
```
If `primary_key` is not declared on the base model, the engine returns an error when a `TO_MANY` calculated column is used.
+144
View File
@@ -0,0 +1,144 @@
# View
A **View** is a named SQL query stored in the MDL. It behaves like a virtual table — clients can query it by name, and the engine inlines the `statement` SQL before execution. Unlike a Model, a View does not declare columns explicitly; its schema is inferred from the `statement` at query time.
## Structure
```yaml
# views.yml
views:
- name: high_value_orders
statement: >
SELECT order_id, customer_id, amount
FROM orders
WHERE amount > 100
```
### JSON format (MDL manifest)
```json
{
"name": "high_value_orders",
"statement": "SELECT order_id, customer_id, amount FROM orders WHERE amount > 100"
}
```
## Fields
| Field | Required | Description |
|-------|----------|-------------|
| `name` | Yes | Unique identifier used in SQL queries |
| `statement` | Yes | A complete SQL SELECT statement; may reference other models or views |
## Model vs View
| | Model | View |
|-|-------|------|
| Data source | Physical table, `ref_sql`, or `base_object` | SQL `statement` |
| Column declarations | Explicit (with types) | Inferred from `statement` |
| Relationship columns | Supported | Not supported |
| Calculated columns | Supported | Not supported |
| Primary key | Supported | Not applicable |
| Access control | Column omission, RLAC/CLAC | Column omission via `statement` |
Use a **Model** when you need typed columns, relationships, or calculated fields. Use a **View** for pre-built queries — dashboards, saved filters, or cross-model aggregations — that you want to expose as a named table.
## jaffle_shop Example
The jaffle_shop workspace ships with an empty `views.yml` (`views: []`), but views become useful once you have mart-layer models in place. Here are representative examples:
### Simple filter view
```yaml
- name: completed_orders
statement: >
SELECT order_id, customer_id, order_date, amount
FROM orders
WHERE status = 'completed'
```
```sql
SELECT * FROM completed_orders WHERE amount > 50;
```
### Cross-model aggregation view
```yaml
- name: customer_order_summary
statement: >
SELECT
c.customer_id,
c.first_name,
c.last_name,
COUNT(o.order_id) AS total_orders,
SUM(o.amount) AS lifetime_value
FROM customers c
JOIN orders o ON c.customer_id = o.customer_id
GROUP BY c.customer_id, c.first_name, c.last_name
```
The `statement` references `customers` and `orders` by their model names. The engine resolves them through the normal model pipeline after expanding the view.
### View referencing another view
```yaml
- name: vip_customers
statement: >
SELECT customer_id, first_name, last_name, lifetime_value
FROM customer_order_summary
WHERE lifetime_value > 500
```
Views can reference other views. The engine expands all view references recursively before resolving model references.
## Querying a View
Once defined, a view is a first-class table name:
```sql
SELECT * FROM completed_orders;
SELECT customer_id, total_orders
FROM customer_order_summary
ORDER BY total_orders DESC
LIMIT 10;
```
The view name can be qualified with catalog and schema:
```sql
SELECT * FROM wren.main.completed_orders;
```
## Engine Internals
### Session registration
At session initialisation, each view's `statement` is parsed into a DataFusion `LogicalPlan` and wrapped in a `ViewTable`. The `ViewTable` is registered under the view's fully-qualified name (`catalog.schema.name`) in the DataFusion catalog.
```
view.statement → ctx.state().create_logical_plan()
→ ViewTable::new(plan, statement)
→ ctx.register_table(catalog.schema.name, view_table)
```
### Query-time expansion: `ExpandWrenViewRule`
`ExpandWrenViewRule` runs as the **first** analyzer pass — before `ModelAnalyzeRule` and all other rules. It performs a bottom-up walk of the logical plan tree. Whenever it encounters a `TableScan` whose name belongs to the MDL and matches a registered view, it replaces the scan node with the view's `LogicalPlan` wrapped in a subquery alias:
```
TableScan("completed_orders")
↓ ExpandWrenViewRule
Subquery(
Filter(status = 'completed', TableScan("orders")),
alias = "completed_orders"
)
```
After the view is inlined, the remaining `TableScan("orders")` nodes are processed by `ModelAnalyzeRule` in the next pass, which resolves them to physical tables.
This ordering ensures that a view's `statement` can freely reference other models or views — all references are resolved in subsequent passes after expansion.
### Recursive view expansion
If a view references another view, `ExpandWrenViewRule` handles the recursion automatically. The `transform_up_with_subqueries` traversal processes the tree from leaves to root, so inner views are expanded before outer views reference them.