diff --git a/packages/@n8n/expression-runtime/ARCHITECTURE.md b/packages/@n8n/expression-runtime/ARCHITECTURE.md new file mode 100644 index 00000000000..7fb48018067 --- /dev/null +++ b/packages/@n8n/expression-runtime/ARCHITECTURE.md @@ -0,0 +1,427 @@ +# Expression Runtime Architecture + +This package provides a secure, isolated expression evaluation runtime that works across multiple execution environments (isolated-vm, Web Workers, and task runners). + +## Design Goals + +1. **Environment Agnostic**: Single codebase that works in Node.js (isolated-vm), browsers (Web Workers), and task runner processes +2. **Security**: Expressions run in isolated contexts with memory limits and timeouts +3. **Performance**: Lazy data loading, code caching, and efficient data transfer +4. **Observability**: Built-in metrics, traces, and logs +5. **Maintainability**: Clear separation of concerns with well-defined interfaces + +## Three-Layer Architecture + +The architecture is split into three distinct layers: + +``` +┌─────────────────────────────────────────────────────────┐ +│ Host Process │ +│ │ +│ ┌────────────────────────────────────────────────┐ │ +│ │ ExpressionEvaluator (Layer 3) │ │ +│ │ - Public API │ │ +│ │ - Tournament integration │ │ +│ │ - Code caching │ │ +│ │ - Observability │ │ +│ └────────────────┬───────────────────────────────┘ │ +│ │ │ +│ ┌────────────────▼───────────────────────────────┐ │ +│ │ Bridge (Layer 2) │ │ +│ │ - IsolatedVmBridge (Phase 1.1) │ │ +│ │ - WebWorkerBridge (Phase 2+) │ │ +│ │ - Task Runner Integration (TBD) │ │ +│ └────────────────┬───────────────────────────────┘ │ +│ │ IPC/Message Passing │ +└───────────────────┼─────────────────────────────────────┘ + │ +┌───────────────────▼─────────────────────────────────────┐ +│ Isolated Context │ +│ │ +│ ┌────────────────────────────────────────────────┐ │ +│ │ Runtime (Layer 1) │ │ +│ │ - Runs inside isolation │ │ +│ │ - No Node.js dependencies │ │ +│ │ - Lazy loading proxies │ │ +│ │ - Helper functions ($json, $item, etc.) │ │ +│ │ - lodash, Luxon │ │ +│ └────────────────────────────────────────────────┘ │ +│ │ +└─────────────────────────────────────────────────────────┘ +``` + +### Layer 1: Runtime (Isolated Context) + +**Location**: Runs inside the isolated context (isolate, worker, subprocess) + +**Purpose**: Provides the JavaScript execution environment for expressions + +**Key Components**: +- **Lazy Loading Proxies**: Fetch data fields on-demand from host to avoid memory limits +- **Helper Functions**: `$json`, `$item`, `$input`, `$`, etc. +- **Libraries**: lodash, Luxon (bundled) +- **No Node.js APIs**: Pure JavaScript only + +**Bundle**: IIFE format for isolated-vm, ESM for Web Workers + +### Layer 2: Bridge (Host Process) + +**Location**: Runs in the host process + +**Purpose**: Manages communication between host and isolated context + +**Key Components**: +- **RuntimeBridge Interface**: Abstract interface for all bridge implementations +- **IsolatedVmBridge**: Uses isolated-vm API for Node.js backend (Phase 1.1) +- **WebWorkerBridge**: Uses postMessage API for browser (Phase 2+) +- **Task Runner Integration**: TBD - May use IsolatedVmBridge locally or direct evaluation (Phase 2+) + +**Responsibilities**: +- Initialize isolated context +- Transfer code to context +- Handle data requests from runtime (lazy loading) +- Enforce memory limits and timeouts +- Dispose of context when needed + +### Layer 3: Evaluator (Host Process) + +**Location**: Runs in the host process + +**Purpose**: Public API for expression evaluation + +**Key Components**: +- **ExpressionEvaluator**: Main class used by workflow package +- **Tournament Integration**: AST transformation and security validation +- **Code Cache**: Cache transformed code (not evaluation results) +- **Observability**: Emit metrics, traces, and logs + +**Responsibilities**: +- Accept expression strings and workflow data +- Transform expressions with Tournament +- Cache transformed code to avoid re-transformation +- Convert WorkflowData to WorkflowDataProxy for lazy loading +- Use bridge to evaluate in isolated context +- Handle errors gracefully +- Emit observability data + +## Data Flow + +### Expression Evaluation Flow + +```mermaid +sequenceDiagram + participant WF as Workflow + participant Eval as ExpressionEvaluator + participant Bridge as IsolatedVmBridge + participant Runtime as Runtime (Isolated) + + WF->>Eval: evaluate(expr, data) + Eval->>Eval: Transform with Tournament (cached) + Eval->>Bridge: execute(transformedCode, data) + Bridge->>Bridge: registerCallbacks(data) — creates ivm.Reference callbacks + Bridge->>Runtime: resetDataProxies() — initialise $json, $input, etc. as lazy proxies + Bridge->>Runtime: run wrapped code (this === __data) + Runtime->>Runtime: Access $json.field + Runtime->>Bridge: __getValueAtPath(['$json','field']) via ivm.Reference + Bridge->>Bridge: Navigate data object + Bridge-->>Runtime: Metadata or primitive + Runtime-->>Bridge: Expression result + Bridge-->>Eval: Result (copied from isolate) + Eval-->>WF: Result +``` + +### Lazy Data Loading + +Data access from inside the isolate goes through `ivm.Reference` callbacks +registered by the bridge — not through a method on `RuntimeBridge` itself. + +```mermaid +sequenceDiagram + participant Runtime as Runtime (Isolated) + participant Proxy as Lazy Proxy + participant Bridge as IsolatedVmBridge (host) + + Runtime->>Proxy: $json.user.email + Proxy->>Bridge: __getValueAtPath(['$json','user','email']) via ivm.Reference + Bridge->>Bridge: Navigate data object registered via registerCallbacks() + Bridge-->>Proxy: "test@example.com" (primitive copied into isolate) + Proxy-->>Runtime: "test@example.com" +``` + +## Environment-Specific Implementations + +### IsolatedVmBridge (Node.js Backend) + +Uses [isolated-vm](https://github.com/laverdet/isolated-vm) for V8 isolate-based isolation: + +```typescript +class IsolatedVmBridge implements RuntimeBridge { + private isolate: ivm.Isolate; + private context: ivm.Context; + + async initialize(): Promise { + this.isolate = new ivm.Isolate({ + memoryLimit: 128 + }); + this.context = await this.isolate.createContext(); + + // Load runtime code + await this.context.eval(runtimeCode); + } + + async execute(code: string, dataId: string): Promise { + // Implementation... + } +} +``` + +### WebWorkerBridge (Browser Frontend) + +Uses Web Workers for browser-based isolation: + +```typescript +class WebWorkerBridge implements RuntimeBridge { + private worker: Worker; + + async initialize(): Promise { + this.worker = new Worker('/runtime.worker.js'); + // Setup message handlers + } + + async execute(code: string, dataId: string): Promise { + // Implementation... + } +} +``` + +### Task Runner Integration (TBD - Phase 2+) + +Task runners already provide process-level isolation. When code nodes call `evaluateExpression()`, evaluation happens **inside the task runner** (not via IPC to worker). + +**Architecture decision pending - two options**: + +**Option A**: Task runner uses `IsolatedVmBridge` locally +```typescript +// Inside task runner process +const evaluator = new ExpressionEvaluator({ + bridge: new IsolatedVmBridge(config), // Evaluates locally +}); + +// Code node calls evaluateExpression() +const result = await evaluator.evaluate(expression, workflowData); +// ^ All happens inside task runner, no IPC, no lazy loading needed +``` + +**Option B**: Task runner evaluates directly (no extra sandbox) +```typescript +// Task runner already isolated at process level +// No need for isolated-vm sandbox on top +const result = evaluateExpressionDirectly(expression, workflowData); +``` + +**Key point**: Task runner already has all workflow data, so no lazy loading or IPC communication is needed for data access. + +## Package Structure + +``` +packages/@n8n/expression-runtime/ +├── ARCHITECTURE.md # This file +├── README.md +├── package.json +├── tsconfig.json +├── tsconfig.build.json +├── vitest.config.ts +├── esbuild.config.js # Bundles src/runtime/index.ts → dist/bundle/runtime.iife.js +│ +├── src/ +│ ├── index.ts # Public API exports +│ │ +│ ├── types/ # TypeScript interfaces (no implementations) +│ │ ├── index.ts +│ │ ├── bridge.ts # RuntimeBridge, BridgeConfig +│ │ ├── evaluator.ts # IExpressionEvaluator, EvaluatorConfig, error classes +│ │ └── runtime.ts # RuntimeHostInterface, RuntimeGlobals, RuntimeError +│ │ +│ ├── runtime/ # Layer 1: runs inside the V8 isolate +│ │ └── index.ts # Proxy system, resetDataProxies, __sanitize, +│ │ # SafeObject, SafeError, Lodash/Luxon wiring, +│ │ # all extension functions +│ │ +│ ├── bridge/ # Layer 2: host-process isolate management +│ │ └── isolated-vm-bridge.ts # IsolatedVmBridge (ivm.Isolate, callbacks, script cache) +│ │ +│ ├── evaluator/ # Layer 3: public-facing API +│ │ └── expression-evaluator.ts # Tournament integration, expression code cache +│ │ +│ ├── extensions/ # Expression extension functions (bundled into runtime) +│ │ ├── array-extensions.ts +│ │ ├── boolean-extensions.ts +│ │ ├── date-extensions.ts +│ │ ├── number-extensions.ts +│ │ ├── object-extensions.ts +│ │ ├── string-extensions.ts +│ │ ├── extend.ts +│ │ ├── extensions.ts +│ │ ├── expression-extension-error.ts +│ │ └── utils.ts +│ │ +│ └── __tests__/ +│ └── integration.test.ts +│ +└── dist/ + ├── *.js / *.d.ts # Compiled TypeScript (tsc output) + └── bundle/ + └── runtime.iife.js # Self-contained IIFE loaded into isolated-vm +``` + +## Key Design Decisions + +### 1. Why Three Layers? + +**Separation of Concerns**: Each layer has a single responsibility: +- Runtime: Execute expressions in isolation +- Bridge: Handle environment-specific communication +- Evaluator: Provide clean API with observability + +**Environment Agnostic**: The Runtime and Evaluator layers are identical across all environments. Only the Bridge changes. + +### 2. Why Lazy Loading? + +**Memory Efficiency**: Large workflow data (100MB+) cannot fit in isolate memory limits (128MB). Lazy loading fetches only the fields that expressions actually access. + +**Performance**: Transferring only accessed fields is faster than transferring entire objects. + +**Limitation**: Lazy loading requires **synchronous** callbacks from runtime to host. This works for: +- ✅ **isolated-vm**: Uses `ivm.Reference` for true synchronous callbacks +- ✅ **Node.js vm**: Direct synchronous function calls +- ❌ **Web Workers**: postMessage is always async (see Known Limitations below) + +### 3. Why Bundle the Runtime? + +**No Node.js Dependencies**: Runtime must work in environments without Node.js (browser, isolated-vm). Bundling produces a self-contained IIFE/ESM module. + +**Immutability**: Bundled runtime is immutable and can be cached. + +### 4. Why Abstract Bridge? + +**Future-Proofing**: Frontend will use Web Workers. Backend uses isolated-vm. Abstract bridge allows adding new environments without changing other layers. + +**Testing**: NodeVmBridge allows fast testing without native isolated-vm dependency. + +## Known Limitations + +### Lazy Loading with Async Boundaries + +JavaScript Proxy trap handlers are **synchronous**, which creates a fundamental limitation: + +```javascript +const proxy = new Proxy({}, { + get(target, prop) { + // This handler MUST be synchronous + // Cannot use await or return Promise + return someValue; + } +}); +``` + +**Impact by Environment**: + +1. **isolated-vm** ✅ + - Uses `ivm.Reference` for true synchronous callbacks from isolate to host + - Full lazy loading support + +2. **Node.js vm** ✅ + - Direct synchronous function calls + - Full lazy loading support (used for testing) + +3. **Web Workers** ❌ + - `postMessage` is always async + - **Phase 1 Limitation**: No lazy loading, must pre-fetch all data before evaluation + - **Future Enhancement (Phase 2+)**: Explore `SharedArrayBuffer` + `Atomics` for synchronous data access + +### Web Worker Support Roadmap + +**Phase 1** (Initial implementation): +- WebWorkerBridge will pre-fetch all workflow data +- Transfer complete data object to worker before evaluation +- Works for small/medium datasets (< 50MB) +- No lazy loading benefit + +**Phase 2+** (Future enhancement): +- Investigate `SharedArrayBuffer` + `Atomics` for sync access +- Or accept pre-fetching as the Web Worker approach +- Decision based on real-world usage patterns + +### Security Boundaries + +The runtime has **no access** to: +- ❌ Node.js APIs (fs, net, child_process, etc.) +- ❌ Host process memory +- ❌ Other isolates/workers +- ❌ Cookies + +The runtime **can only**: +- ✅ Call `getDataSync()` to fetch workflow data +- ✅ Access lodash and Luxon libraries +- ✅ Execute pure JavaScript code + +## Testing Strategy + +**Runtime Tests** (vitest): +- Use NodeVmBridge for fast, isolated tests +- Test lazy loading, helpers, error handling +- No native dependencies required + +**Bridge Tests** (vitest): +- Test each bridge implementation +- Mock environment-specific APIs +- Test memory limits, timeouts, disposal + +**Evaluator Tests** (vitest): +- Test Tournament integration (transformation and validation) +- Test code caching (transformed code, not results) +- Test WorkflowData to WorkflowDataProxy conversion +- Test observability emission +- Test error handling + +**Integration Tests** (jest in workflow package): +- Test full stack with real isolated-vm +- Test concurrent evaluations +- Test with real workflow data + +## Observability + +All layers emit metrics, traces, and logs: + +**Metrics**: +- `expression.evaluation.count` +- `expression.evaluation.duration_ms` +- `expression.code_cache.hit` (transformed code cache) +- `expression.code_cache.miss` +- `expression.isolate.memory_mb` + +**Traces**: +- `expression.evaluate` span wraps entire evaluation +- `expression.tournament` span for AST transformation +- `expression.isolate.execute` span for isolated execution + +**Logs**: +- Errors at all levels +- Warnings for memory pressure +- Debug logs for development + +See observability package documentation for details. + +## Next Steps + +1. Implement TypeScript interfaces (Phase 0.1) +2. Implement observability infrastructure (Phase 0.2) +3. Create comprehensive benchmarks (Phase 0.3) +4. Implement runtime package (Phase 1.1) +5. Implement isolate pooling (Phase 1.2) + +## References + +- [isolated-vm GitHub](https://github.com/laverdet/isolated-vm) +- [Web Workers MDN](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API) +- [n8n workflow package](../workflow/) diff --git a/packages/@n8n/expression-runtime/README.md b/packages/@n8n/expression-runtime/README.md new file mode 100644 index 00000000000..d19a6de6040 --- /dev/null +++ b/packages/@n8n/expression-runtime/README.md @@ -0,0 +1,305 @@ +# @n8n/expression-runtime + +Secure, isolated expression evaluation runtime for n8n workflows. + +## Status + +**In progress — landing as a series of incremental PRs.** + +Implemented so far: +- ✅ TypeScript interfaces and architecture design +- ✅ Core architecture documentation + +Coming in later PRs: +- 🚧 Runtime bundle: extension functions, deep lazy proxy system (PR 2) +- 🚧 `IsolatedVmBridge`: V8 isolate management via `isolated-vm` (PR 3) +- 🚧 `ExpressionEvaluator`: tournament integration, expression code caching (PR 4) +- 🚧 Integration tests (PR 4) +- 🚧 Workflow integration behind `N8N_EXPRESSION_ENGINE=vm` flag (PR 5) +- 🚧 Web Worker support (Phase 2+) +- 🚧 Performance optimizations (Phase 3) + +## Overview + +This package provides a secure runtime for evaluating expressions in isolated contexts. + +Currently supports: +- **Node.js Backend**: Uses `isolated-vm` for V8 isolate-based isolation with lazy data loading + +Future support (Phase 2+): +- **Browser Frontend**: Will use Web Workers for browser-based isolation +- **Task Runners**: Will use IPC for separate process isolation + +## Features + +- 🔒 **Secure**: Expressions run in isolated V8 contexts with memory limits (128MB) and timeouts (5s) +- 🚀 **Performant**: Lazy data loading via proxies, script compilation caching, and expression code caching +- 📊 **Observable**: Built-in metrics, traces, and logs support (interfaces defined; providers coming later) +- 🌐 **Universal**: Works in Node.js backend (browsers and task runners in Phase 2+) +- 🛡️ **AST Security**: Tournament AST hooks (`ThisSanitizer`, `PrototypeSanitizer`, `DollarSignValidator`) validate expressions before execution + +## Architecture + +The runtime uses a three-layer architecture: + +1. **Runtime** (Layer 1): Runs inside isolated context, provides expression execution environment +2. **Bridge** (Layer 2): Manages communication between host and isolated context +3. **Evaluator** (Layer 3): Public API with Tournament integration and observability + +See [ARCHITECTURE.md](./ARCHITECTURE.md) for detailed design documentation. + +## Installation + +```bash +pnpm add @n8n/expression-runtime +``` + +## Usage + +### Basic Example + +```typescript +import { ExpressionEvaluator, IsolatedVmBridge } from '@n8n/expression-runtime'; + +// Create bridge +const bridge = new IsolatedVmBridge({ + memoryLimit: 128, + timeout: 5000, +}); + +// Create evaluator +const evaluator = new ExpressionEvaluator({ + bridge, +}); + +// Initialize +await evaluator.initialize(); + +// Evaluate expression using {{ }} template syntax +const result = evaluator.evaluate( + '{{ $json.user.email }}', + { + $json: { + user: { email: 'test@example.com' } + } + } +); + +console.log(result); // "test@example.com" + +// Clean up +await evaluator.dispose(); +``` + +### With Security Hooks (Production) + +Pass AST security hooks from `expression-sandboxing.ts` to enable full security validation. This is the pattern used by the workflow package: + +```typescript +import { ExpressionEvaluator, IsolatedVmBridge } from '@n8n/expression-runtime'; +import { + ThisSanitizer, + PrototypeSanitizer, + DollarSignValidator, +} from 'n8n-workflow/expression-sandboxing'; + +const bridge = new IsolatedVmBridge({ timeout: 5000 }); +const evaluator = new ExpressionEvaluator({ + bridge, + hooks: { + before: [ThisSanitizer], + after: [PrototypeSanitizer, DollarSignValidator], + }, +}); + +await evaluator.initialize(); +``` + +When `hooks` is omitted the evaluator still runs tournament transformation (template parsing, `this` binding) but without AST security validation — suitable for development and testing. + +### With Observability (Not Yet Implemented) + +```typescript +import { OpenTelemetryProvider } from '@n8n/expression-runtime/observability'; + +const observability = new OpenTelemetryProvider({ + serviceName: 'n8n-expressions', +}); + +const evaluator = new ExpressionEvaluator({ + bridge, + observability, +}); +``` + +**Note**: Observability providers are not yet implemented. The `ObservabilityProvider` interface exists but no implementations are available yet. + +## API + +### ExpressionEvaluator + +Main class for expression evaluation. + +```typescript +class ExpressionEvaluator { + constructor(config: EvaluatorConfig); + initialize(): Promise; + evaluate(expression: string, data: WorkflowData, options?: EvaluateOptions): unknown; + dispose(): Promise; + isDisposed(): boolean; +} +``` + +### RuntimeBridge + +Abstract interface for bridge implementations. + +```typescript +interface RuntimeBridge { + initialize(): Promise; + execute(code: string, data: Record): unknown; + dispose(): Promise; + isDisposed(): boolean; +} +``` + +### Bridge Implementations + +- **IsolatedVmBridge**: 🚧 For Node.js backend (isolated-vm with V8 isolates) - coming in PR 3 + - Memory isolation with hard 128MB limit + - Timeout enforcement (5s default) + - Deep lazy proxy system for workflow data + - Synchronous callbacks via ivm.Reference + - Security wrappers (SafeObject, SafeError) + - `E()` error handler for tournament-generated try-catch code +- **WebWorkerBridge**: 🚧 For browser frontend (Web Workers) - Phase 2+ +- **Task Runner Integration**: 🚧 TBD - May use IsolatedVmBridge locally or direct evaluation - Phase 2+ + +## Configuration + +```typescript +interface EvaluatorConfig { + bridge: RuntimeBridge; // required + observability?: ObservabilityProvider; // optional - interfaces defined, providers not yet implemented + hooks?: TournamentHooks; // optional - AST security hooks for tournament (PR 4) +} + +interface BridgeConfig { + memoryLimit?: number; // Default: 128 MB (PR 3) + timeout?: number; // Default: 5000 ms (PR 3) + debug?: boolean; // Default: false (PR 3) +} +``` + +## Environment Variables (Not Yet Implemented) + +```bash +# Bridge configuration (not yet implemented) +N8N_EXPRESSION_MEMORY_LIMIT_MB=128 +N8N_EXPRESSION_TIMEOUT_MS=5000 +N8N_EXPRESSION_DEBUG=false + +# Code cache (not yet implemented - caches transformed code, not results) +N8N_EXPRESSION_CODE_CACHE_ENABLED=true +N8N_EXPRESSION_CODE_CACHE_MAX_SIZE=1000 + +# Observability (not yet implemented) +N8N_EXPRESSION_OBSERVABILITY_ENABLED=true +N8N_EXPRESSION_METRICS_ENABLED=true +N8N_EXPRESSION_TRACES_ENABLED=true +N8N_EXPRESSION_TRACE_SAMPLE_RATE=0.01 +``` + +**Note**: Currently, configuration is passed via constructor options. Environment variable support will be added in future phases. + +## Development + +```bash +# Install dependencies +pnpm install + +# Build package +pnpm build + +# Run tests +pnpm test + +# Run tests in watch mode +pnpm test:watch + +# Type check +pnpm typecheck + +# Lint +pnpm lint +``` + +## Testing + +The package uses vitest for fast, isolated testing: + +```typescript +import { ExpressionEvaluator, IsolatedVmBridge } from '@n8n/expression-runtime'; + +describe('ExpressionEvaluator', () => { + it('evaluates simple expression', async () => { + const bridge = new IsolatedVmBridge({ timeout: 5000 }); + const evaluator = new ExpressionEvaluator({ bridge }); + + await evaluator.initialize(); + + const result = evaluator.evaluate('{{ $json.value }}', { $json: { value: 42 } }); + expect(result).toBe(42); + + await evaluator.dispose(); + }); +}); +``` + +Run tests: +```bash +pnpm test # Run all tests +pnpm test integration # Run integration tests only +``` + +## Performance + +The runtime uses several optimizations (implemented in PRs 2–4): + +- **Lazy Loading**: Only fetch data fields that expressions actually access via proxy traps +- **Script Compilation Caching**: Compiled scripts are cached to avoid recompilation +- **Metadata-Driven**: Only structure (keys, lengths) transferred across isolate boundary, not full data +- **Expression Code Caching**: Tournament-transformed code is cached per evaluator instance (same expressions repeat within a workflow, so cache hit rate is high in practice) + +Performance characteristics: +- Arrays: Always lazy-loaded — only length transferred, elements fetched on demand +- Objects: Always lazy-loaded — only keys transferred, values fetched on demand + +## Security + +The runtime enforces strict security at multiple layers (implemented in PRs 2–4): + +- **Memory limits**: Hard 128MB limit via isolated-vm (configurable) +- **Execution timeouts**: 5s default timeout (configurable) +- **Complete isolation**: No access to Node.js APIs (require, fs, process, etc.) +- **Security wrappers**: SafeObject and SafeError prevent dangerous method access +- **Native function blocking**: Prevents access to native code +- **AST transforms**: `ThisSanitizer` rewrites `$json` → `this.$json`; `PrototypeSanitizer` wraps computed property access in `this.__sanitize(key)` to block prototype chain attacks; `DollarSignValidator` enforces correct `$`-variable usage +- **Runtime sanitizer**: `__sanitize()` inside the isolate blocks access to `__proto__`, `constructor`, `prototype`, and other dangerous properties at runtime + +Future security features (Phase 2+): +- 🚧 Additional sandboxing for browser environments + +## Contributing + +See the main n8n repository for contribution guidelines. + +## License + +See [LICENSE.md](../../LICENSE.md) in the n8n repository root. + +## Related + +- [n8n workflow package](../workflow/) +- [isolated-vm](https://github.com/laverdet/isolated-vm) +- [@n8n/tournament](https://github.com/n8n-io/tournament) diff --git a/packages/@n8n/expression-runtime/docs/architecture-diagram.mmd b/packages/@n8n/expression-runtime/docs/architecture-diagram.mmd new file mode 100644 index 00000000000..d5fa16552ac --- /dev/null +++ b/packages/@n8n/expression-runtime/docs/architecture-diagram.mmd @@ -0,0 +1,92 @@ +%% Expression Runtime Architecture +%% Three-layer design for environment-agnostic expression evaluation + +graph TB + subgraph "Host Process" + WF[Workflow Package] + + subgraph "Layer 3: Evaluator" + EVAL[ExpressionEvaluator] + TOUR[Tournament] + CACHE[Code Cache] + OBS[Observability] + end + + subgraph "Layer 2: Bridge" + BRIDGE_IF[RuntimeBridge Interface] + ISOVM[IsolatedVmBridge] + WEBW[WebWorkerBridge] + TASKR[TaskRunnerBridge] + end + + DATASTORE[(Data Store)] + end + + subgraph "Isolated Context (isolate/worker/subprocess)" + subgraph "Layer 1: Runtime" + RUNTIME[Runtime Entry] + PROXY[Lazy Proxy] + HELPERS[Helper Functions] + LODASH[lodash] + LUXON[Luxon] + end + end + + WF -->|evaluate| EVAL + EVAL --> TOUR + EVAL --> CACHE + EVAL --> OBS + EVAL -->|execute| BRIDGE_IF + + BRIDGE_IF -.->|implements| ISOVM + BRIDGE_IF -.->|implements| WEBW + BRIDGE_IF -.->|implements| TASKR + + ISOVM -->|IPC/Reference| RUNTIME + WEBW -->|postMessage| RUNTIME + TASKR -->|IPC| RUNTIME + + RUNTIME --> PROXY + RUNTIME --> HELPERS + RUNTIME --> LODASH + RUNTIME --> LUXON + + PROXY -.->|getData request| ISOVM + ISOVM --> DATASTORE + DATASTORE -.->|value| ISOVM + ISOVM -.->|value| PROXY + + style EVAL fill:#e1f5ff + style BRIDGE_IF fill:#fff4e1 + style RUNTIME fill:#f0ffe1 + + style ISOVM fill:#fff4e1,stroke:#ff9800 + style WEBW fill:#fff4e1,stroke:#9e9e9e,stroke-dasharray: 5 5 + style TASKR fill:#fff4e1,stroke:#9e9e9e,stroke-dasharray: 5 5 + +%% Data Flow Sequence + +sequenceDiagram + participant WF as Workflow + participant Eval as ExpressionEvaluator + participant Bridge as RuntimeBridge + participant Runtime as Runtime (Isolated) + participant Store as Data Store + + WF->>Eval: evaluate(expr, data) + Eval->>Eval: Transform with Tournament + Eval->>Eval: Check cache + Eval->>Store: Store data with ID + Eval->>Bridge: execute(code, dataId) + Bridge->>Runtime: Run code in isolation + + Runtime->>Runtime: Access $json.email + Runtime->>Bridge: getData(dataId, 'email') + Bridge->>Store: Lookup 'email' + Store-->>Bridge: Value + Bridge-->>Runtime: Value + + Runtime-->>Bridge: Expression result + Bridge-->>Eval: Result + Eval->>Eval: Cache result + Eval-->>WF: Result diff --git a/packages/@n8n/expression-runtime/docs/deep-lazy-proxy.md b/packages/@n8n/expression-runtime/docs/deep-lazy-proxy.md new file mode 100644 index 00000000000..7f4a4d7ea82 --- /dev/null +++ b/packages/@n8n/expression-runtime/docs/deep-lazy-proxy.md @@ -0,0 +1,235 @@ +# Deep Lazy Proxy + +## Overview + +The Deep Lazy Proxy is a memory-efficient mechanism for providing workflow data to expression evaluation contexts. Instead of copying entire data structures upfront, it loads data on-demand as properties are accessed. + +## Key Features + +- **On-Demand Loading**: Only fetches data when accessed +- **Metadata-Driven**: Returns object structure (keys, length) without values +- **Caching**: Values are cached after first access to avoid redundant lookups +- **Type Support**: Handles objects, arrays, functions, and primitives correctly +- **Memory Efficient**: Large arrays and objects don't cause memory overhead + +## Architecture + +The deep lazy proxy is implemented entirely within `src/runtime/index.ts`, which is +bundled into `dist/bundle/runtime.iife.js` and injected into the V8 isolate at startup. + +Key functions exposed on `globalThis` inside the isolate: + +- `createDeepLazyProxy(basePath)` — creates recursive object/array proxies +- `resetDataProxies()` — called before each evaluation to reinitialise `$json`, + `$input`, `$node`, etc. as fresh lazy proxies backed by the three host callbacks +- `__sanitize(key)` — runtime property-access guard that blocks `__proto__`, + `constructor`, `prototype`, etc. + +Host-side callbacks registered by `IsolatedVmBridge` as `ivm.Reference` objects +(synchronous cross-isolate calls): + +- `__getValueAtPath(path[])` — returns a primitive, array metadata, or object metadata +- `__getArrayElement(path[], index)` — returns a single array element (or its metadata) +- `__callFunctionAtPath(path[], ...args)` — invokes a host-side function and returns the result + +## Usage + +The proxy system runs **inside the V8 isolate** and is not directly importable from +host code. The host sets up the data context by calling `bridge.execute(code, data)`, +which internally: + +1. Registers three `ivm.Reference` callbacks with the current `data` object +2. Calls `resetDataProxies()` in the isolate to create fresh lazy proxies for + `$json`, `$binary`, `$input`, `$node`, `$parameter`, `$workflow`, `$prevNode` +3. Runs the tournament-transformed expression code with `this === __data` + +From the expression's perspective it just sees normal objects: + +```typescript +// Inside an expression (runs in isolate): +$json.user.email // triggers getValueAtPath(['$json','user','email']) +$json.items[150].id // triggers getArrayElement(['$json','items'], 150) +$items() // triggers callFunctionAtPath(['$items']) +``` + +### Array metadata + +Arrays are **never transferred in full** — only their length is returned. Elements +are loaded individually on demand. Length can be determined from the host object +in O(1), but serialization cost is proportional to the total byte size of all +elements, which cannot be bounded from length alone. + +```typescript +// __getValueAtPath returns: +{ __isArray: true, __length: 1000 } // always metadata only +{ __isObject: true, __keys: ['name','email'] } // object — lazy +42 // primitive +``` + +## How It Works + +### Metadata Pattern + +Instead of transferring entire objects/arrays, the proxy uses metadata: + +**Arrays** (all sizes): +```typescript +{ + __isArray: true, + __length: 1000 // Only length; elements loaded on demand via __getArrayElement +} +``` + +**Objects**: +```typescript +{ + __isObject: true, + __keys: ['name', 'email', 'age'] // Only keys, not values +} +``` + +### Caching + +Once a property is accessed, it's cached in the proxy's target object: + +```typescript +proxy.$json.user.name // First access: fetches via callback +proxy.$json.user.name // Second access: returns cached value +``` + +### Recursive Proxies + +When accessing nested objects or arrays, new proxies are created: + +```typescript +proxy.$json.user // Creates proxy for user object +proxy.$json.items[50] // Creates proxy for object at index 50 +``` + +## Security + +### Function Handling + +- **Custom Functions**: Allowed and passed directly +- **Native Functions**: Blocked for security (e.g., `Object.keys`) + +```typescript +const customFn = (x: number) => x * 2; // Allowed +const nativeFn = Object.keys; // Blocked (returns undefined) +``` + +Detection is done by checking if `fn.toString()` contains `'[native code]'`. + +### Symbol Properties + +Symbol properties return `undefined` to prevent security issues. + +## Performance + +### Memory Efficiency + +- **Arrays**: Always lazy-loaded — only length transferred, elements fetched on demand +- **Objects**: Always lazy-loaded — only keys transferred, values fetched on demand + +### Access Patterns + +Best performance when: +- Accessing few properties from large objects +- Accessing specific array elements (not iterating entire array) +- Accessing the same properties multiple times (caching means only the first access pays) + +Suboptimal performance when: +- Iterating entire arrays (`.map()`, `.filter()`) — each element triggers a separate callback +- Accessing most properties of large objects +- No property reuse (no benefit from caching) + +## Known Limitations + +1. **Array Methods**: Methods like `.map()`, `.filter()` iterate all elements. + Each element triggers a separate `__getArrayElement` callback call, which is slow + for large arrays. + - **Workaround**: Avoid iterating large arrays in expressions; access specific indices instead + +2. **Circular References**: May cause infinite loops in the proxy handler. + - **Current**: No cycle detection; circular structures should be avoided in expression data + +## Testing + +### Unit Tests + +```bash +cd packages/@n8n/expression-runtime +pnpm test proxy +``` + +Test coverage: +- ✅ Basic property access +- ✅ Nested properties +- ✅ Small/large arrays +- ✅ Object proxies +- ✅ Function handling +- ✅ Caching behavior +- ✅ Edge cases (circular refs, symbols, "in" operator) + +### Manual Testing + +Run the example: +```bash +npx tsx src/proxy/__tests__/manual-test.example.ts +``` + +## API Reference (inside the isolate bundle) + +These functions are available on `globalThis` within the V8 isolate after the +runtime bundle (`dist/bundle/runtime.iife.js`) is loaded. + +### `resetDataProxies()` + +Called by the bridge before each expression evaluation. Reads `$json`, `$binary`, +`$input`, `$node`, `$parameter`, `$workflow`, `$prevNode`, `$runIndex`, `$itemIndex`, +and `$items` from `__data` (populated via host callbacks) and exposes them on both +`globalThis` and `__data` so tournament-transformed code can access them via +`this.$json`, `this.$input`, etc. + +### `createDeepLazyProxy(basePath)` + +Creates a recursive Proxy for a given property path. Intercepts property access and +calls back to the host via `__getValueAtPath` to fetch structure metadata, then +creates nested proxies for objects or arrays as needed. + +**Parameter:** +- `basePath: string[]` — path from the root data object to the node this proxy represents + +## Examples + +### Accessing nested data (expression syntax) + +``` +{{ $json.order.customer.name }} // lazy-loads order.customer.name +{{ $json.order.items[1].product }} // lazy-loads array element at index 1 +{{ $json.items[0] }} // fetches only the first element +``` + +### Array iteration is slow for large arrays + +``` +{{ _.sum($json.items) }} +// items has 10 000 elements → length transferred, then 10 000 callback +// calls to fetch each element. Prefer accessing specific indices. +``` + +## Contributing + +When modifying the proxy implementation: + +1. **Run tests**: `pnpm test proxy` +2. **Type check**: `pnpm typecheck` +3. **Build**: `pnpm build` +4. **Add tests** for new features +5. **Update this documentation** + +## Related Files + +- Implementation: `packages/@n8n/expression-runtime/src/runtime/index.ts` — proxy system, `resetDataProxies`, `__sanitize`, `SafeObject`, `SafeError` +- Bridge: `packages/@n8n/expression-runtime/src/bridge/isolated-vm-bridge.ts` — registers `ivm.Reference` callbacks, loads bundle, calls `resetDataProxies` +- Build: `packages/@n8n/expression-runtime/esbuild.config.js` — bundles runtime to `dist/bundle/runtime.iife.js` diff --git a/packages/@n8n/expression-runtime/docs/implementation-phases.md b/packages/@n8n/expression-runtime/docs/implementation-phases.md new file mode 100644 index 00000000000..494740acf58 --- /dev/null +++ b/packages/@n8n/expression-runtime/docs/implementation-phases.md @@ -0,0 +1,155 @@ +# Implementation Phases + +This document maps interfaces to implementation phases to help developers focus on what's needed when. + +## Phase 1.1: Core Runtime Package (MVP) + +**Goal**: Basic expression evaluation working in CLI/backend + +**Interfaces Needed**: +- `RuntimeBridge` - Main bridge interface +- `BridgeConfig` (without `debug` field) +- `RuntimeHostInterface` - Runtime-to-host communication +- `RuntimeGlobals` - Globals injected into runtime +- `WorkflowDataProxy` - Data access helper +- `IExpressionEvaluator` - Public API +- `EvaluatorConfig` (without observability) +- `WorkflowData` - Input data format +- `EvaluateOptions` (basic) + +**Implementations Required**: +- `IsolatedVmBridge` - For CLI/backend +- `ExpressionEvaluator` - Main evaluator class +- Runtime code (runs inside isolate, bundled via esbuild) +- Lazy loading proxies +- Expression code cache (per-evaluator, caches tournament-transformed code) + +**Can Skip**: +- Observability (use `NoOpProvider` stub) +- Debug mode +- Specific error types (use generic `Error`) +- Web Workers +- Task runners + +## Phase 0.2: Observability Infrastructure (PARALLEL) + +**Goal**: Add metrics, traces, and logs + +**Interfaces Needed**: +- `ObservabilityProvider` +- `MetricsAPI` +- `TracesAPI` +- `LogsAPI` +- `Span` + +**Implementations Required**: +- `NoOpProvider` (zero overhead when disabled) +- `OpenTelemetryProvider` +- `PostHogProvider` (optional) +- `CompositeProvider` (use multiple providers) + +**Integration**: +- Add to `EvaluatorConfig.observability` +- Emit metrics/traces from evaluator and bridge +- Smart sampling implementation + +## Phase 1.2: Isolate Pooling + +**Goal**: Handle concurrent evaluations + +**New Interfaces**: None (uses existing `RuntimeBridge`) + +**Implementations Required**: +- `IsolatePool` class +- Pool configuration +- Acquire/release mechanism +- Disposal detection and replacement + +## Phase 1.3: Extension Framework + +**Goal**: 100% test compatibility + +**New Interfaces**: None + +**Implementation**: Extension functions in runtime + +## Phase 1.4: Error Handling + +**Goal**: Graceful error handling with clear messages + +**Interfaces Needed**: +- `ExpressionError` +- `MemoryLimitError` +- `TimeoutError` +- `SecurityViolationError` +- `SyntaxError` + +**Implementation**: Error handling in all code paths + +## Phase 2+: Future Enhancements + +### Web Worker Support +**Interfaces**: Already defined (same `RuntimeBridge`) + +**Implementations**: +- `WebWorkerBridge` +- Runtime bundled as ESM +- Note: No lazy loading initially (pre-fetch data) + +### Chrome DevTools Debugging +**Config**: `BridgeConfig.debug` field + +**Implementation**: +- Inspector protocol integration +- Debug mode in IsolatedVmBridge + +### Task Runner Integration (Architecture TBD) + +Task runners already have process-level isolation. Expression evaluation happens **inside the task runner** (no IPC to worker needed). + +**Option A**: Use `IsolatedVmBridge` locally within task runner +- Adds another sandbox layer for extra security +- Task runner creates local evaluator instance +- No lazy loading needed (task runner has all data) + +**Option B**: Evaluate directly without extra sandbox +- Reuse task runner's existing process isolation +- Simpler, potentially faster +- May be sufficient given process-level isolation + +**Decision pending** - will be made during Phase 2+ implementation. + +--- + +## Quick Start Guides + +### For Frontend Developers (Web Worker Integration) + +**Phase 1**: Skip - Web Workers are Phase 2+ + +**Phase 2**: Focus on: +1. `RuntimeBridge` interface - Your bridge must implement this +2. `BridgeConfig` - Configuration options +3. `WorkflowDataProxy` - How to structure data +4. Ignore: Observability interfaces (optional) + +**Key Difference**: Web Workers can't do lazy loading initially, so you'll need to pre-fetch all data before calling `execute()`. + +### For CLI/Backend Developers + +**Phase 1.1**: Focus on: +1. `IsolatedVmBridge` implementation +2. `ExpressionEvaluator` class +3. Runtime code (runs inside isolate) +4. Code cache implementation + +**Use**: `NoOpProvider` for observability initially + +**Phase 0.2**: Add real observability providers + +### For Testing + +Integration tests use `IsolatedVmBridge` directly (see `src/__tests__/integration.test.ts`). + +All interfaces in `src/types/` are stable enough to write against before the +bridge implementation lands. diff --git a/packages/@n8n/expression-runtime/package.json b/packages/@n8n/expression-runtime/package.json new file mode 100644 index 00000000000..ca77b016648 --- /dev/null +++ b/packages/@n8n/expression-runtime/package.json @@ -0,0 +1,43 @@ +{ + "name": "@n8n/expression-runtime", + "version": "0.1.0", + "description": "Secure, isolated expression evaluation runtime for n8n", + "main": "dist/index.js", + "types": "dist/index.d.ts", + "scripts": { + "build": "tsc -p tsconfig.build.json", + "test": "vitest run", + "test:dev": "vitest --watch --silent false", + "typecheck": "tsc --noEmit" + }, + "keywords": [ + "n8n", + "expression", + "evaluation", + "isolated-vm", + "web-worker", + "security" + ], + "license": "SEE LICENSE IN LICENSE.md", + "dependencies": { + "js-base64": "catalog:", + "jssha": "3.3.1", + "lodash": "catalog:", + "luxon": "catalog:", + "md5": "2.3.0", + "title-case": "3.0.3", + "transliteration": "2.3.5" + }, + "devDependencies": { + "@types/lodash": "catalog:", + "@types/luxon": "3.2.0", + "@types/md5": "^2.3.5", + "typescript": "catalog:", + "vitest": "catalog:" + }, + "files": [ + "dist", + "ARCHITECTURE.md", + "LICENSE.md" + ] +} diff --git a/packages/@n8n/expression-runtime/src/index.ts b/packages/@n8n/expression-runtime/src/index.ts new file mode 100644 index 00000000000..0b6820ce4dd --- /dev/null +++ b/packages/@n8n/expression-runtime/src/index.ts @@ -0,0 +1,24 @@ +// Types — full public API surface +// Implementations (ExpressionEvaluator, IsolatedVmBridge) are added in later PRs. +export type { + IExpressionEvaluator, + EvaluatorConfig, + WorkflowData, + EvaluateOptions, + RuntimeBridge, + BridgeConfig, + ObservabilityProvider, + MetricsAPI, + TracesAPI, + Span, + LogsAPI, + TournamentHooks, +} from './types'; + +export { + ExpressionError, + MemoryLimitError, + TimeoutError, + SecurityViolationError, + SyntaxError, +} from './types'; diff --git a/packages/@n8n/expression-runtime/src/types/bridge.ts b/packages/@n8n/expression-runtime/src/types/bridge.ts new file mode 100644 index 00000000000..3534f362dc0 --- /dev/null +++ b/packages/@n8n/expression-runtime/src/types/bridge.ts @@ -0,0 +1,73 @@ +// ============================================================================ +// Phase 1.1: Bridge Interface (CORE - IMPLEMENT FIRST) +// +// This is the main interface all environments must implement. +// Start here for CLI/backend (IsolatedVmBridge) or frontend (WebWorkerBridge). +// ============================================================================ + +/** + * Abstract interface for runtime bridges. + * + * A bridge manages communication between the host process and the isolated context. + * Different bridge implementations support different isolation mechanisms: + * - IsolatedVmBridge: Uses isolated-vm for Node.js backend (secure isolation with memory limits) + * - WebWorkerBridge: Uses Web Workers for browser frontend (Phase 2+) + * - Task Runner: TBD - May use IsolatedVmBridge locally or direct evaluation (Phase 2+) + */ +export interface RuntimeBridge { + /** + * Initialize the isolated context and load runtime code. + * Must be called before any execute() calls. + */ + initialize(): Promise; + + /** + * Execute JavaScript code in the isolated context. + * + * @param code - Transformed JavaScript code to execute + * @param data - Workflow data proxy from WorkflowDataProxy.getDataProxy() + * @returns Result of the expression evaluation. + * Must be JSON-serializable (no functions, symbols, etc.) + * + * Note: Synchronous for Node.js vm module (Slice 1). + * Will be async for isolated-vm (Slice 2). + */ + execute(code: string, data: Record): unknown; + + /** + * Dispose of the isolated context and free resources. + * After disposal, the bridge cannot be used again. + */ + dispose(): Promise; + + /** + * Check if the bridge has been disposed. + * Disposed bridges cannot execute code. + */ + isDisposed(): boolean; +} + +/** + * Configuration for runtime bridges. + */ +export interface BridgeConfig { + /** + * Memory limit in MB for isolated context. + * Default: 128MB + */ + memoryLimit?: number; + + /** + * Timeout in milliseconds for expression execution. + * Default: 5000ms + */ + timeout?: number; + + /** + * Enable debug mode (inspector protocol). + * Default: false + * + * Phase 2+: Chrome DevTools debugging support + */ + debug?: boolean; +} diff --git a/packages/@n8n/expression-runtime/src/types/evaluator.ts b/packages/@n8n/expression-runtime/src/types/evaluator.ts new file mode 100644 index 00000000000..51b3f119556 --- /dev/null +++ b/packages/@n8n/expression-runtime/src/types/evaluator.ts @@ -0,0 +1,244 @@ +import type { RuntimeBridge } from './bridge'; + +// ============================================================================ +// Phase 1.1: Core Evaluation Interfaces (MVP) +// These are the minimal interfaces needed to evaluate expressions. +// ============================================================================ + +// TournamentHooks is imported from '@n8n/tournament' once that dependency is +// added (PR 4). Defined locally here so the type surface is complete from PR 1. +// See: packages/@n8n/expression-runtime/src/evaluator/expression-evaluator.ts +export interface TournamentHooks { + before?: Array<(ast: unknown) => unknown>; + after?: Array<(ast: unknown) => unknown>; +} + +/** + * Configuration for ExpressionEvaluator. + * + * Note: Slice 1 keeps this minimal. Tournament integration and code caching + * will be added in later slices. + */ +export interface EvaluatorConfig { + /** + * Runtime bridge implementation. + */ + bridge: RuntimeBridge; + + /** + * Observability provider for metrics, traces, and logs. + */ + observability?: ObservabilityProvider; + + /** + * AST security hooks for tournament expression transformation. + * Provided by the caller (e.g., workflow package's expression-sandboxing.ts). + * If omitted, expressions are transformed with no security hooks (dev/testing use). + */ + hooks?: TournamentHooks; +} + +/** + * Expression evaluator - main public API. + * + * This is the primary interface used by the workflow package. + */ +export interface IExpressionEvaluator { + /** + * Initialize the evaluator and bridge. + * Must be called before evaluate(). + */ + initialize(): Promise; + + /** + * Evaluate an expression string against workflow data. + * + * @param expression - Expression string (e.g., "{{ $json.email }}") + * @param data - Workflow data context + * @param options - Evaluation options + * @returns Result of the expression + * + * Note: Synchronous for Slice 1 (Node.js vm module). + * Will be async for Slice 2 (isolated-vm). + */ + evaluate(expression: string, data: WorkflowData, options?: EvaluateOptions): unknown; + + /** + * Dispose of the evaluator and free resources. + */ + dispose(): Promise; + + /** + * Check if the evaluator has been disposed. + */ + isDisposed(): boolean; +} + +/** + * Workflow data proxy from WorkflowDataProxy.getDataProxy(). + * + * For Slice 1: We pass this directly via VM context (simple pass-through). + * Later: Will implement deep lazy proxy for field-level data fetching. + */ +export type WorkflowData = Record; + +/** + * Options for evaluate(). + */ +/** + * Options for evaluate(). + * + * Note: Slice 1 is minimal. Tournament options will be added later. + */ +export interface EvaluateOptions { + /** + * Custom timeout for this evaluation (in milliseconds). + * Overrides the bridge's default timeout. + */ + timeout?: number; +} + +// ============================================================================ +// Phase 0.2 / Phase 1+: Observability Interfaces (OPTIONAL FOR MVP) +// +// These can be stubbed with NoOpProvider initially. +// Full implementation comes in Phase 0.2 (observability infrastructure). +// +// Frontend developers: You can ignore this section for Phase 1. +// CLI/Backend developers: Use NoOpProvider initially, real providers later. +// ============================================================================ + +/** + * Observability provider interface. + * + * Implementations: NoOpProvider, OpenTelemetryProvider, PostHogProvider + */ +export interface ObservabilityProvider { + /** + * Metrics API. + */ + metrics: MetricsAPI; + + /** + * Traces API. + */ + traces: TracesAPI; + + /** + * Logs API. + */ + logs: LogsAPI; +} + +/** + * Metrics API. + */ +export interface MetricsAPI { + /** + * Increment a counter. + */ + counter(name: string, value: number, tags?: Record): void; + + /** + * Set a gauge value. + */ + gauge(name: string, value: number, tags?: Record): void; + + /** + * Record a histogram value. + */ + histogram(name: string, value: number, tags?: Record): void; +} + +/** + * Traces API. + */ +export interface TracesAPI { + /** + * Start a new span. + */ + startSpan(name: string, attributes?: Record): Span; +} + +/** + * Span interface. + */ +export interface Span { + /** + * Set span status. + */ + setStatus(status: 'ok' | 'error'): void; + + /** + * Set span attribute. + */ + setAttribute(key: string, value: unknown): void; + + /** + * Record an exception. + */ + recordException(error: Error): void; + + /** + * End the span. + */ + end(): void; +} + +/** + * Logs API. + */ +export interface LogsAPI { + /** + * Log an error. + */ + error(message: string, error?: Error, context?: Record): void; + + /** + * Log a warning. + */ + warn(message: string, context?: Record): void; + + /** + * Log info. + */ + info(message: string, context?: Record): void; + + /** + * Log debug. + */ + debug(message: string, context?: Record): void; +} + +// ============================================================================ +// Phase 1.4: Error Handling (IMPLEMENT WITH EVALUATOR) +// +// These error types provide structured error information. +// Start with basic Error, add these types in Phase 1.4. +// ============================================================================ + +/** + * Expression evaluation error. + */ +export class ExpressionError extends Error { + constructor( + message: string, + public context: { + expression?: string; + workflowId?: string; + nodeId?: string; + [key: string]: unknown; + }, + ) { + super(message); + this.name = 'ExpressionError'; + } +} + +/** + * Specific error types. + */ +export class MemoryLimitError extends ExpressionError {} +export class TimeoutError extends ExpressionError {} +export class SecurityViolationError extends ExpressionError {} +export class SyntaxError extends ExpressionError {} diff --git a/packages/@n8n/expression-runtime/src/types/index.ts b/packages/@n8n/expression-runtime/src/types/index.ts new file mode 100644 index 00000000000..4778275ab0e --- /dev/null +++ b/packages/@n8n/expression-runtime/src/types/index.ts @@ -0,0 +1,40 @@ +/** + * Expression Runtime Types + * + * This module exports all TypeScript interfaces and types for the expression runtime. + */ + +// Bridge types +export type { RuntimeBridge, BridgeConfig } from './bridge'; + +// Runtime types +export type { + RuntimeHostInterface, + RuntimeGlobals, + RuntimeConfig, + LazyProxyConfig, +} from './runtime'; + +export { RuntimeError } from './runtime'; + +// Evaluator types +export type { + EvaluatorConfig, + IExpressionEvaluator, + WorkflowData, + EvaluateOptions, + ObservabilityProvider, + MetricsAPI, + TracesAPI, + Span, + LogsAPI, + TournamentHooks, +} from './evaluator'; + +export { + ExpressionError, + MemoryLimitError, + TimeoutError, + SecurityViolationError, + SyntaxError, +} from './evaluator'; diff --git a/packages/@n8n/expression-runtime/src/types/runtime.ts b/packages/@n8n/expression-runtime/src/types/runtime.ts new file mode 100644 index 00000000000..07020504e3d --- /dev/null +++ b/packages/@n8n/expression-runtime/src/types/runtime.ts @@ -0,0 +1,138 @@ +// ============================================================================ +// Phase 1.1: Runtime Interfaces (IMPLEMENT WITH BRIDGE) +// +// These interfaces define how the runtime (inside isolation) communicates +// with the host. Implement these when building the runtime code. +// ============================================================================ + +/** + * Runtime interface exposed to isolated context. + * + * This interface defines what the runtime code (running inside the isolated context) + * can call. The bridge implements these functions on the host side. + */ +export interface RuntimeHostInterface { + /** + * Get data from host by path (synchronous). + * Used by lazy loading proxies to fetch data on-demand. + * + * IMPORTANT: This is SYNCHRONOUS because JavaScript Proxy traps cannot be async. + * - IsolatedVmBridge: Uses ivm.Reference for true sync callbacks + * - WebWorkerBridge: Not supported - must pre-fetch all data + * + * @param path - Property path to fetch (e.g., "user.email", "items[0].json") + * @returns Value at the path, or undefined if not found + */ + getDataSync(path: string): unknown; +} + +/** + * Lazy-loading proxy for workflow data. + * + * At runtime, these appear as plain objects but are actually Proxy objects + * that fetch data on-demand from the host using getDataSync(). + */ +type LazyDataProxy = Record; + +/** + * Runtime globals available in isolated context. + * + * These are injected by the bridge when initializing the context. + */ +export interface RuntimeGlobals { + /** + * Host interface for calling back to host process. + */ + __host: RuntimeHostInterface; + + /** + * Workflow data proxy ($json, $item, etc.). + * Set by bridge before each execute() call. + */ + + /** Current item data (lazy-loaded proxy) */ + $json: LazyDataProxy; + + /** Alias for $json (lazy-loaded proxy) */ + $: LazyDataProxy; + + /** Get item by index (lazy-loaded) */ + $item: (index: number, runIndex?: number) => LazyDataProxy; + + /** Access to all items */ + $input: { + all: () => Array<{ json: LazyDataProxy }>; + first: () => { json: LazyDataProxy } | undefined; + last: () => { json: LazyDataProxy } | undefined; + item: { json: LazyDataProxy }; + }; + + /** + * Standard libraries available in runtime. + */ + _: typeof import('lodash'); // lodash + DateTime: typeof import('luxon').DateTime; // Luxon +} + +/** + * Configuration for runtime initialization. + */ +export interface RuntimeConfig { + /** + * Enable debug logging in runtime. + * Default: false + */ + debug?: boolean; + + /** + * Custom timeout for data fetches. + * Default: 1000ms + */ + dataFetchTimeout?: number; +} + +/** + * Runtime error thrown inside isolated context. + * + * These errors are thrown by the runtime code when something goes wrong during + * expression evaluation. The bridge must catch these and translate them to the + * appropriate ExpressionError subclass (see evaluator.ts). + * + * Translation mapping: + * - code: 'MEMORY_LIMIT' → MemoryLimitError + * - code: 'TIMEOUT' → TimeoutError + * - code: 'SECURITY_VIOLATION' → SecurityViolationError + * - code: 'SYNTAX_ERROR' → SyntaxError + * - other → ExpressionError + */ +export class RuntimeError extends Error { + constructor( + message: string, + public code: string, + public details?: Record, + ) { + super(message); + this.name = 'RuntimeError'; + } +} + +/** + * Lazy proxy configuration. + */ +export interface LazyProxyConfig { + /** + * Host interface for fetching data. + */ + host: RuntimeHostInterface; + + /** + * Property path prefix (for nested proxies). + * Example: If this proxy represents $json.user, pathPrefix would be "user" + */ + pathPrefix?: string; + + /** + * Cache for fetched values to avoid repeated host calls. + */ + cache?: Map; +} diff --git a/packages/@n8n/expression-runtime/tsconfig.build.json b/packages/@n8n/expression-runtime/tsconfig.build.json new file mode 100644 index 00000000000..fd7283f34c3 --- /dev/null +++ b/packages/@n8n/expression-runtime/tsconfig.build.json @@ -0,0 +1,10 @@ +{ + "extends": ["./tsconfig.json"], + "compilerOptions": { + "rootDir": "src", + "outDir": "dist", + "tsBuildInfoFile": "dist/build.tsbuildinfo" + }, + "include": ["src/**/*.ts"], + "exclude": ["node_modules", "dist", "src/**/__tests__/**"] +} diff --git a/packages/@n8n/expression-runtime/tsconfig.json b/packages/@n8n/expression-runtime/tsconfig.json new file mode 100644 index 00000000000..ecbc0370e85 --- /dev/null +++ b/packages/@n8n/expression-runtime/tsconfig.json @@ -0,0 +1,10 @@ +{ + "extends": "../typescript-config/modern/tsconfig.json", + "compilerOptions": { + "rootDir": ".", + "noUncheckedIndexedAccess": false, + "types": ["node", "vitest/globals"] + }, + "include": ["src/**/*.ts"], + "exclude": ["node_modules", "dist"] +} diff --git a/packages/@n8n/expression-runtime/vitest.config.ts b/packages/@n8n/expression-runtime/vitest.config.ts new file mode 100644 index 00000000000..cecc0f442d4 --- /dev/null +++ b/packages/@n8n/expression-runtime/vitest.config.ts @@ -0,0 +1,13 @@ +import { defineConfig } from 'vitest/config'; + +export default defineConfig({ + test: { + globals: true, + environment: 'node', + coverage: { + provider: 'v8', + reporter: ['text', 'json', 'html'], + exclude: ['dist/**', 'bundle/**', '**/*.test.ts', '**/*.config.ts'], + }, + }, +}); diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 02fa973402f..efb519f56d3 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -566,7 +566,7 @@ importers: version: 3.0.1 axios: specifier: 1.13.5 - version: 1.13.5(debug@4.4.3) + version: 1.13.5 jest-mock-extended: specifier: ^3.0.4 version: 3.0.4(jest@29.7.0(@types/node@20.19.21)(ts-node@10.9.2(@swc/core@1.15.8(@swc/helpers@0.5.17))(@types/node@20.19.21)(typescript@5.9.2)))(typescript@5.9.2) @@ -1211,6 +1211,46 @@ importers: specifier: 'catalog:' version: 3.1.3(@types/debug@4.1.12)(@types/node@20.19.21)(jiti@2.6.1)(jsdom@23.0.1(bufferutil@4.0.9)(utf-8-validate@5.0.10))(lightningcss@1.30.2)(sass@1.89.2)(terser@5.16.1)(tsx@4.19.3) + packages/@n8n/expression-runtime: + dependencies: + js-base64: + specifier: 'catalog:' + version: 3.7.2(patch_hash=bb02fdf69495c7b0768791b60ab6e1a002053b8decd19a174f5755691e5c9500) + jssha: + specifier: 3.3.1 + version: 3.3.1 + lodash: + specifier: 'catalog:' + version: 4.17.23 + luxon: + specifier: 'catalog:' + version: 3.7.2 + md5: + specifier: 2.3.0 + version: 2.3.0 + title-case: + specifier: 3.0.3 + version: 3.0.3 + transliteration: + specifier: 2.3.5 + version: 2.3.5 + devDependencies: + '@types/lodash': + specifier: 'catalog:' + version: 4.17.17 + '@types/luxon': + specifier: 3.2.0 + version: 3.2.0 + '@types/md5': + specifier: ^2.3.5 + version: 2.3.5 + typescript: + specifier: 5.9.2 + version: 5.9.2 + vitest: + specifier: 'catalog:' + version: 3.1.3(@types/debug@4.1.12)(@types/node@20.19.21)(jiti@2.6.1)(jsdom@23.0.1(bufferutil@4.0.9)(utf-8-validate@5.0.10))(lightningcss@1.30.2)(sass@1.89.2)(terser@5.16.1)(tsx@4.19.3) + packages/@n8n/extension-sdk: dependencies: zod: @@ -14173,9 +14213,6 @@ packages: resolution: {integrity: sha512-MT/xP0CrubFRNLNKvxJ2BYfy53Zkm++5bX9dtuPbqAeQpTVe0MQTFhao8+Cp//EmJp244xt6Drw/GVEGCUj40g==} engines: {node: '>=12', npm: '>=6'} - jssha@3.3.0: - resolution: {integrity: sha512-w9OtT4ALL+fbbwG3gw7erAO0jvS5nfvrukGPMWIAoea359B26ALXGpzy4YJSp9yGnpUvuvOw1nSjSoHDfWSr1w==} - jssha@3.3.1: resolution: {integrity: sha512-VCMZj12FCFMQYcFLPRm/0lOBbLi8uM2BhXPTqw3U4YAfs4AZfiApOoBLoN8cQE60Z50m1MYMTQVCfgF/KaCVhQ==} @@ -22170,7 +22207,7 @@ snapshots: '@currents/commit-info': 1.0.1-beta.0 async-retry: 1.3.3 axios: 1.13.5(debug@4.4.3) - axios-retry: 4.5.0(axios@1.13.5(debug@4.4.3)) + axios-retry: 4.5.0(axios@1.13.5) c12: 1.11.2(magicast@0.3.5) chalk: 4.1.2 commander: 12.1.0 @@ -28155,11 +28192,6 @@ snapshots: axe-core@4.7.2: {} - axios-retry@4.5.0(axios@1.13.5(debug@4.4.3)): - dependencies: - axios: 1.13.5(debug@4.4.3) - is-retry-allowed: 2.2.0 - axios-retry@4.5.0(axios@1.13.5): dependencies: axios: 1.13.5 @@ -31667,7 +31699,7 @@ snapshots: '@types/debug': 4.1.12 '@types/node': 20.19.21 '@types/tough-cookie': 4.0.5 - axios: 1.13.5 + axios: 1.13.5(debug@4.4.3) camelcase: 6.3.0 debug: 4.4.3(supports-color@8.1.1) dotenv: 16.6.1 @@ -31677,7 +31709,7 @@ snapshots: isstream: 0.1.2 jsonwebtoken: 9.0.3 mime-types: 2.1.35 - retry-axios: 2.6.0(axios@1.13.5(debug@4.4.3)) + retry-axios: 2.6.0(axios@1.13.5) tough-cookie: 4.1.4 transitivePeerDependencies: - supports-color @@ -33090,8 +33122,6 @@ snapshots: ms: 2.1.3 semver: 7.7.3 - jssha@3.3.0: {} - jssha@3.3.1: {} jstransformer@1.0.0: @@ -34958,7 +34988,7 @@ snapshots: otpauth@9.1.1: dependencies: - jssha: 3.3.0 + jssha: 3.3.1 otplib@12.0.1: dependencies: @@ -36160,7 +36190,7 @@ snapshots: onetime: 5.1.2 signal-exit: 3.0.7 - retry-axios@2.6.0(axios@1.13.5(debug@4.4.3)): + retry-axios@2.6.0(axios@1.13.5): dependencies: axios: 1.13.5 @@ -37584,7 +37614,7 @@ snapshots: title-case@3.0.3: dependencies: - tslib: 2.6.2 + tslib: 2.8.1 tlds@1.248.0: {}