mirror of
https://github.com/aiclientproxy/proxycast.git
synced 2026-09-24 23:10:56 +08:00
release: v1.9.0
This commit is contained in:
coso
@@ -13,6 +13,7 @@
|
||||
|
||||
- `content/`：对外文档站正文（产品介绍、用户指南、进阶能力）
|
||||
- `aiprompts/`：模块级工程文档（前后端组件、服务、命令、数据层）
|
||||
- `exec-plans/`：执行计划、进度日志、技术债追踪
|
||||
- `tech/`：跨模块技术蓝图与专题工程文档（当前已包含 Harness Engineering 指导文档）
|
||||
- `bussniss/`：商务合作与代理运营方案
|
||||
- `develop/`：开发流程与协作规范
|
||||
@@ -21,6 +22,8 @@
|
||||
- `iteration-notes/`：迭代备忘与下版本建议（暂不进入当前发布范围的问题）
|
||||
- `images/`：文档图片资源
|
||||
- `TECH_SPEC.md`：技术规格文档
|
||||
- `exec-plans/README.md`：执行计划目录说明
|
||||
- `exec-plans/tech-debt-tracker.md`：技术债持续追踪表
|
||||
- `develop/execution-tracker-technical-plan.md`：统一执行追踪（Execution Tracker）专项技术规划
|
||||
- `develop/execution-tracker-deprecation-plan.md`：统一执行追踪旧路径退场计划（P0 收口）
|
||||
- `develop/execution-tracker-p0-acceptance-report.md`：统一执行追踪 P0 验收报告
|
||||
|
||||
@@ -16,6 +16,7 @@
|
||||
|
||||
- `overview.md` - 项目架构总览与模块分层
|
||||
- `governance.md` - 新旧并存治理、迁移收口、禁止回流
|
||||
- `harness-engine-governance.md` - Harness Engine 事实源、evidence pack、replay / analysis / review 治理规范
|
||||
- `quality-workflow.md` - 本地校验、GUI smoke、契约检查、CI 门禁
|
||||
- `command-runtime.md` - `@` / `/` / 轻卡 / viewer / 功能方案包实施手册
|
||||
- `skill-standard.md` - 统一技能标准、skill / adapter / runtime binding 边界
|
||||
@@ -62,6 +63,7 @@
|
||||
- **做网页登录态访问 / 网页导出 / Markdown 落盘场景**：先读 `web-browser-scene-skill.md`
|
||||
- **改 Workspace / GUI 壳 / 主路径**：先读 `workspace.md`、`quality-workflow.md`、`playwright-e2e.md`
|
||||
- **做迁移 / 收口 / 去兼容层**：先读 `governance.md`
|
||||
- **改 handoff / evidence pack / replay / review / HarnessStatusPanel**：先读 `harness-engine-governance.md`，再回看 `governance.md`
|
||||
- **改 Provider / 凭证加载 / Token 刷新**：先读 `providers.md`、`credential-pool.md`
|
||||
- **做跨仓库联动**：先读 `limecore-collaboration-entry.md`
|
||||
|
||||
|
||||
@@ -510,7 +510,7 @@ npm run verify:local
|
||||
|
||||
- **Agent / Codex 主命令**：继续收敛到 `agent_runtime_*`
|
||||
- **子代理运行时主链**：继续收敛到 `agent_runtime_spawn_subagent`；当前 request surface 使用 `name / teamName / runInBackground / mode / isolation / cwd` 等字段，其中 `teamName` 需要与 `name` 搭配并依附现有 Team 上下文，`cwd` 必须是绝对目录，并稳定投影到 child session 的 `working_dir` 与 Team 成员展示；当前 runtime 仍会明确拒绝非空 `mode / isolation`
|
||||
- **Team runtime 工具主链**：当前协作工具面继续收敛到 `Agent / TeamCreate / TeamDelete / SendMessage / ListPeers`；`SubAgentTask` 仅保留兼容入口，不再作为新的多代理主路径
|
||||
- **Team runtime 工具主链**：当前协作工具面继续收敛到 `Agent / TeamCreate / TeamDelete / SendMessage / ListPeers`；不要把已删除的 `SubAgentTask` compat 工具重新接回新的多代理主路径
|
||||
- **用户可见消息工具主链**：继续收敛到 `SendUserMessage`，用于把回复、进度同步、主动提醒和附件送到用户主可见消息面；不要再把这类能力拆到其它平行工具名或旁路协议里
|
||||
- **会话状态回写主链**：继续收敛到 `agent_runtime_update_session`，用于名称、执行策略、session provider/model、`recent_access_mode`、`recent_preferences` 以及 `recent_team_selection` 的轻量持久化回写
|
||||
- **会话权限主链**：`agent_runtime_submit_turn.turn_config.approval_policy / sandbox_policy` 是正式 turn context 权限协议；`getSession` 返回的 `execution_runtime.recent_access_mode` 负责承接会话最近一次 accessMode。当前端已命中同一 steady-state 权限时，不应继续依赖 `harness.access_mode` 作为唯一事实源
|
||||
|
||||
@@ -244,6 +244,17 @@ npm run test:contracts
|
||||
|
||||
只要其中任意一个答案是否定的，就说明治理还没完成。
|
||||
|
||||
### Harness Engine 主链
|
||||
|
||||
遇到 handoff、evidence pack、replay、review、外部诊断交接相关改动时，至少问这几个问题：
|
||||
|
||||
- 运行时事实是不是继续收敛到 `agent_runtime_export_evidence_pack`
|
||||
- replay / analysis / review / GUI 是否只是在消费 evidence pack，而不是重新拼装一套摘要
|
||||
- gap 是否只来自“当前线程真实适用但尚未导出”的信号，而不是历史硬编码模板
|
||||
- request telemetry 是否已经按 `session/thread/turn` 真实关联导出；如果当前线程没有匹配请求，是否导出空摘要而不是继续保留 `unlinked`
|
||||
|
||||
只要其中任意一个答案是否定的，就说明 Harness Engine 还在继续长平行事实源。
|
||||
|
||||
### 记忆与旁路
|
||||
|
||||
遇到记忆系统治理时，至少同时看：
|
||||
@@ -302,6 +313,7 @@ npm run test:contracts
|
||||
## 相关文档
|
||||
|
||||
- `docs/aiprompts/commands.md`
|
||||
- `docs/aiprompts/harness-engine-governance.md`
|
||||
- `docs/aiprompts/quality-workflow.md`
|
||||
- `docs/aiprompts/project-heatmap.md`
|
||||
|
||||
|
||||
@@ -0,0 +1,271 @@
|
||||
# Lime Harness Engine 治理规范
|
||||
|
||||
## 这份文档回答什么
|
||||
|
||||
本文件定义 Lime 的 harness Engine 应该如何长期演进，主要回答：
|
||||
|
||||
- 运行时诊断、evidence pack、replay case、analysis handoff、review template 分别谁负责什么
|
||||
- 什么才算 harness 的唯一事实源，什么只是消费层或展示层
|
||||
- 哪些证据可以导出，哪些只能在“适用时”导出，不能把不存在的信号硬写成全局缺口
|
||||
- 当 Lime 对齐 Claude Code 的治理方式时，哪些原则应该真正落到代码里
|
||||
|
||||
它是 **Harness Engine 的长期治理文档**，不是某次分析导出功能的临时设计说明。
|
||||
|
||||
## 对齐目标
|
||||
|
||||
这份规范参考 Claude Code 的治理方式，但 **不照搬实现细节**。对齐的是以下几条原则：
|
||||
|
||||
1. **单一事实源**
|
||||
2. **先采集，再解释**
|
||||
3. **只有可关联的遥测，才算会话级证据**
|
||||
4. **只有真正发生过的验证，才进入 evidence**
|
||||
5. **消费层不重新拼装另一套运行时真相**
|
||||
|
||||
一句话说，就是：
|
||||
|
||||
**Harness Engine 不是“再做一层报表”，而是把运行时事实稳定导出，供 replay / analysis / review / UI 复用。**
|
||||
|
||||
## 事实源声明
|
||||
|
||||
从现在开始，Harness Engine 相关能力默认收敛到以下主链：
|
||||
|
||||
- 运行时事实导出：`agent_runtime_export_evidence_pack`
|
||||
- 回放样本导出：`agent_runtime_export_replay_case`
|
||||
- 外部诊断交接：`agent_runtime_export_analysis_handoff`
|
||||
- 人工审核模板：`agent_runtime_export_review_decision_template`、`agent_runtime_save_review_decision`
|
||||
|
||||
其中：
|
||||
|
||||
- `evidence pack` 是 **运行时事实源**
|
||||
- `replay / analysis / review / history-record summary` 是 **派生物**
|
||||
- GUI 面板例如 `HarnessStatusPanel` 与 nightly dashboard 是 **展示层**
|
||||
|
||||
允许的数据方向只有：
|
||||
|
||||
`runtime thread/session` -> `evidence pack` -> `replay / analysis / review / summary` -> `trend / cleanup / dashboard` -> `UI`
|
||||
|
||||
禁止反向或旁路：
|
||||
|
||||
- `analysis` 自己再拼一套 observability summary
|
||||
- `replay` 不复用 evidence pack，改为本地猜测缺口
|
||||
- `UI` 的显示状态反过来成为事实源
|
||||
|
||||
## 分类语言
|
||||
|
||||
Harness Engine 相关 surface 继续沿用仓库统一分类：
|
||||
|
||||
- `current`：当前唯一主路径，后续功能只允许继续向这里收敛
|
||||
- `compat`：兼容层，只允许委托和适配
|
||||
- `deprecated`：已知要退出的旧路径，只允许迁移和下线
|
||||
- `dead`：已无入口或确认停用，优先删除
|
||||
|
||||
在 harness 语境里，常见判断如下：
|
||||
|
||||
- `current`
|
||||
- `agent_runtime_export_*`
|
||||
- evidence pack 中的 `runtime.json / timeline.json / artifacts.json`
|
||||
- `compat`
|
||||
- 临时存在但只做委托的旧导出入口
|
||||
- 仍保留的旧 GUI 打开目录/复制命令适配层
|
||||
- `deprecated`
|
||||
- 各消费方各自维护的 observability builder
|
||||
- 无条件导出固定 known gaps 的旧逻辑
|
||||
- `dead`
|
||||
- 已没有入口引用的旧 handoff / review / diagnostics 拼装路径
|
||||
|
||||
## 核心治理原则
|
||||
|
||||
### 1. 先导出事实，再导出判断
|
||||
|
||||
evidence pack 先负责导出：
|
||||
|
||||
- 关联键
|
||||
- timeline
|
||||
- warnings / failed tool / failed command
|
||||
- 最近产物
|
||||
- 已真实发生的 verification 记录
|
||||
|
||||
其中 `observabilitySummary` 应优先承载紧凑的 verification outcome 摘要；更完整的明细仍可放在 `artifacts.json`，但下游不应为了得到“是否通过 / 是否失败 / 是否缺失”再单独重建第二套判断。
|
||||
|
||||
analysis brief、replay input、review template 只能在此基础上做解释，不能自己再回头拼装另一套事实。
|
||||
|
||||
### 2. 信号必须按适用性导出
|
||||
|
||||
Claude Code 风格最重要的一点，不是“信号越多越好”，而是“只导出当前回合真正成立的信号”。
|
||||
|
||||
因此 Lime 的 harness 必须遵守：
|
||||
|
||||
- `requestTelemetry`
|
||||
- RequestLog 必须携带 `session_id / thread_id / turn_id / pending_request_id / queued_turn_id / subagent_session_id`
|
||||
- evidence pack 必须按这些关联键导出会话级 request telemetry 摘要
|
||||
- 如果当前线程没有匹配请求，导出空摘要即可，不能再伪造 `unlinked`
|
||||
- `artifactValidator`
|
||||
- 只有最近产物里存在 ArtifactDocument 或等价 artifact schema 时，才允许出现
|
||||
- `browserVerification`
|
||||
- 只有线程里真实出现浏览器工具或浏览器命令时，才允许出现
|
||||
- `guiSmoke`
|
||||
- 只有线程里真实执行了 `verify:gui-smoke` 时，才允许出现
|
||||
|
||||
不适用的信号不应进入 `known_gaps`，更不应被硬编码成所有线程的默认缺口。
|
||||
|
||||
### 3. gap 代表“已证明缺失”，不是“历史上想接”
|
||||
|
||||
`known_gaps` 只能来自：
|
||||
|
||||
- 已定义的信号覆盖表
|
||||
- 且状态不是 `exported`
|
||||
- 且该信号对当前线程确实适用
|
||||
|
||||
禁止继续保留以下旧做法：
|
||||
|
||||
- 所有线程都写 `artifactValidator/browserVerification/guiSmoke`
|
||||
- 即使没有相关产物或相关命令，也输出空 verification 壳
|
||||
- 用“未来计划支持”冒充“当前已识别缺口”
|
||||
|
||||
补充约束：
|
||||
|
||||
- trend / cleanup report 必须区分 `current` 样本里的 gap 和 `degraded` 样本故意保留的 gap
|
||||
- 只有 `current` 样本出现的 observability gap 才应被当成主线回归风险
|
||||
- `degraded` 样本的作用是保留诊断语义与 known gap 事实，不能反过来把 cleanup 优先级抬高
|
||||
|
||||
### 4. 关联键先于遥测摘要
|
||||
|
||||
Claude Code 的 tracing / event logging 之所以可用，前提是事件先有稳定的 session 级关联。
|
||||
|
||||
因此 Lime 必须把下面这些键视为 harness 级基础设施，而不是锦上添花：
|
||||
|
||||
- `session_id`
|
||||
- `thread_id`
|
||||
- `turn_id`
|
||||
- `pending_request_id`
|
||||
- `queued_turn_id`
|
||||
- `subagent_session_id`
|
||||
|
||||
如果 RequestLog 还没稳定携带这些键，就不能把 request telemetry 当成会话级证据；但一旦键已补齐，后续 analysis / replay / review / UI 必须统一消费同一份 request telemetry 摘要，不能继续并存旧的 `unlinked` 语义。
|
||||
|
||||
### 5. 展示层不能成为新事实源
|
||||
|
||||
以下内容只能消费 evidence pack，不能反向定义事实：
|
||||
|
||||
- `HarnessStatusPanel`
|
||||
- 外部 AI copy prompt
|
||||
- review-summary / analysis-brief 里的文案摘要
|
||||
- GUI 上看到的 warning / gap / status 卡片
|
||||
|
||||
展示层至少要如实透出：
|
||||
|
||||
- `current / degraded` observability gap 角色
|
||||
- `observabilityVerificationOutcomes` 里的紧凑 outcome 焦点
|
||||
- cleanup 主事实源里已经存在的 `blocking_failure / advisory_failure / recovered` 摘要
|
||||
|
||||
对 cleanup / dashboard / nightly report 这条推荐动作链，字段语义也必须固定：
|
||||
|
||||
- failure 焦点只允许走 `focusVerificationFailureOutcomes`
|
||||
- recovered 焦点只允许走 `focusVerificationRecoveredOutcomes`
|
||||
- 不允许继续把两类 outcome 混在一个 recommendation 字段里，再由展示层二次猜测
|
||||
|
||||
一旦 UI 文案和后端 evidence 冲突，以 evidence pack 和运行时导出为准。
|
||||
|
||||
### 6. 诊断链路必须复用下游制品
|
||||
|
||||
标准导出链路应保持为：
|
||||
|
||||
1. `evidence pack` 先落盘
|
||||
2. `replay case` 复用 evidence pack
|
||||
3. `analysis handoff` 复用 handoff bundle + evidence pack + replay case
|
||||
4. `history-record` 复用 summary，并继续派生 `trend / cleanup / dashboard`
|
||||
5. `review decision template` 再复用 analysis handoff
|
||||
|
||||
任何一层如果又回去重新读取 thread 并拼第二套摘要，都视为治理倒退。
|
||||
|
||||
### 7. 旁路系统也必须服从同一事实源
|
||||
|
||||
Harness Engine 不只是导出文件目录。以下旁路也必须收敛：
|
||||
|
||||
- diagnostics 摘要
|
||||
- export copy prompt
|
||||
- review template
|
||||
- GUI 状态卡
|
||||
- 后续 eval / trend / promote 使用的样本事实
|
||||
|
||||
如果主导出已经修了，旁路仍沿旧字段判断，就说明治理还没完成。
|
||||
|
||||
## 典型反模式
|
||||
|
||||
出现以下行为，视为 Harness Engine 治理倒退：
|
||||
|
||||
- 在 `analysis / replay / UI` 各自维护一份 observability 拼装逻辑
|
||||
- 给所有线程硬塞 `artifactValidator`、`browserVerification`、`guiSmoke`
|
||||
- verification 还没发生，就先导出空数组或空对象占位
|
||||
- RequestLog 还无法 join，会话摘要却写成“已有 request telemetry”
|
||||
- 后端证据已经修正，前端或 handoff 文案仍沿旧 gap 模板输出
|
||||
- 为保留旧行为再新增一层 compat 包装，而不是删掉错误事实
|
||||
|
||||
## 变更顺序
|
||||
|
||||
涉及 Harness Engine 改动时，默认按这个顺序做：
|
||||
|
||||
1. 先确认唯一事实源是不是 `agent_runtime_export_*`
|
||||
2. 再确认改动属于 `current / compat / deprecated / dead` 哪一类
|
||||
3. 先修 evidence pack，再修 replay / analysis / review / UI
|
||||
4. 先删错误默认值，再考虑是否需要新增字段
|
||||
5. 最后补定向测试与最小 GUI / 契约验证
|
||||
|
||||
如果说不清这次改动要收敛到哪个导出入口，就不要继续扩。
|
||||
|
||||
## 最低验证要求
|
||||
|
||||
Harness Engine 改动至少应覆盖：
|
||||
|
||||
- 受影响 Rust 服务的定向测试
|
||||
- evidence pack 导出结果检查
|
||||
- replay / analysis 是否复用 evidence pack 的回归检查
|
||||
- 如果改动触及 cleanup report / dashboard 推荐动作契约，默认把 cleanup contract 校验纳入最小门槛：
|
||||
|
||||
```bash
|
||||
npm run harness:cleanup-report:check
|
||||
```
|
||||
|
||||
如果要校验某个指定产物，再显式传入：
|
||||
|
||||
```bash
|
||||
node scripts/check-generated-slop-report.mjs --input "<cleanup-json>"
|
||||
```
|
||||
|
||||
如同时触及命令边界，再补：
|
||||
|
||||
```bash
|
||||
npm run test:contracts
|
||||
```
|
||||
|
||||
如同时触及 GUI 主路径或 `HarnessStatusPanel`，再补：
|
||||
|
||||
```bash
|
||||
npm run verify:gui-smoke
|
||||
```
|
||||
|
||||
人工 spot check 至少看一次：
|
||||
|
||||
- `.lime/harness/sessions/<session_id>/evidence/runtime.json`
|
||||
- `.lime/harness/sessions/<session_id>/evidence/artifacts.json`
|
||||
- `.lime/harness/sessions/<session_id>/replay/input.json`
|
||||
- `.lime/harness/sessions/<session_id>/analysis/analysis-context.json`
|
||||
|
||||
## 当前最值得继续优化的一刀
|
||||
|
||||
如果继续沿这条主线推进，优先级最高的是：
|
||||
|
||||
**把已导出的 request telemetry 继续沉淀到 trend / replay / review 守卫里，确保后续任何新入口都只能复用同一份会话级摘要，而不是再次旁路读取或重建第二套真相。**
|
||||
|
||||
这样才能继续逼近 Claude Code 那种“先 trace，再解释，而且所有解释都复用同一份 trace”的治理状态。
|
||||
|
||||
## 相关文档
|
||||
|
||||
- `docs/aiprompts/governance.md`
|
||||
- `docs/aiprompts/commands.md`
|
||||
- `docs/aiprompts/quality-workflow.md`
|
||||
- `docs/aiprompts/playwright-e2e.md`
|
||||
|
||||
## 一句话总结
|
||||
|
||||
**Harness Engine 的治理目标不是导出更多文件，而是让所有诊断、回放、审核和 GUI 都只复用同一份运行时事实。**
|
||||
@@ -126,6 +126,31 @@ x-api-key: {api_key}
|
||||
anthropic-version: 2023-06-01
|
||||
```
|
||||
|
||||
## Prompt Cache 能力边界
|
||||
|
||||
Lime 当前把 Prompt Cache 能力视为 **Provider 类型能力**，而不是“请求长得像哪家协议”：
|
||||
|
||||
- `anthropic` / `claude` / `claude-oauth`：声明为 `automatic`
|
||||
- `anthropic-compatible`：声明为 `explicit_only`
|
||||
- 其它 Provider：默认 `not_applicable`
|
||||
|
||||
这条事实源当前收敛在：
|
||||
|
||||
- 前端：`src/lib/model/providerPromptCacheSupport.ts`
|
||||
- 后端：Provider 类型与运行时能力判断链
|
||||
|
||||
需要特别注意：
|
||||
|
||||
1. `anthropic-compatible` 只表示接入方兼容 Anthropic wire format，不等于上游已经实现 Anthropic Automatic Prompt Caching
|
||||
2. Lime 不会因为某个自定义渠道“长得像 Anthropic”就默认把它当成官方 Anthropic 自动缓存能力
|
||||
3. 对 `anthropic-compatible` 渠道，Lime 只保留显式 `cache_control` 语义；如果上游没有实现 Automatic Prompt Cache，`cached_input_tokens` 为空不能直接归因到 Lime 没发字段
|
||||
|
||||
排查这类问题时，优先确认三件事：
|
||||
|
||||
1. 当前 Provider 类型是不是 `anthropic-compatible`
|
||||
2. 上游服务是否真的声明支持 Anthropic Automatic Prompt Caching
|
||||
3. 响应 usage 中是否存在 `cache_creation_input_tokens` / `cache_read_input_tokens` / `cached_input_tokens`
|
||||
|
||||
## 凭证管理策略
|
||||
|
||||
### 方案 B: 独立副本策略
|
||||
|
||||
@@ -82,7 +82,7 @@
|
||||
|
||||
如果本轮涉及 `companion_*` 桌宠命令族，还要同步检查本地 companion `WebSocket` 入口、前端 `src/lib/api/companion.ts` 网关、Rust 注册、治理目录册以及浏览器模式 mock 返回形态；浏览器模式下这组命令默认也要保持可 mock，不要让桌宠接入把默认页面渲染链路卡死。
|
||||
|
||||
如果本轮涉及 team runtime 工具面或主线程用户消息工具，还要同步检查 Rust catalog / inventory、runtime 注册、浏览器 fallback mock 与前端 tool display；`Agent / TeamCreate / TeamDelete / SendMessage / ListPeers` 必须保持同一组 current surface，`SendUserMessage` 也必须继续停留在 current 主线程工具面，`SubAgentTask` 只能继续停留在 compat 读取边界。
|
||||
如果本轮涉及 team runtime 工具面或主线程用户消息工具，还要同步检查 Rust catalog / inventory、runtime 注册、浏览器 fallback mock 与前端 tool display；`Agent / TeamCreate / TeamDelete / SendMessage / ListPeers` 必须保持同一组 current surface，`SendUserMessage` 也必须继续停留在 current 主线程工具面，不要把已删除的 `SubAgentTask` compat 工具重新接回 Rust catalog、runtime 注册、mock 或前端 tool display。
|
||||
|
||||
如果本轮涉及 MCP bridge runtime tool surface、inventory 或 ToolSearch，还要同步检查 Rust extension 注入、inventory 快照、浏览器 fallback mock 与 GUI 面板命名；当前唯一命名事实源是 `mcp__<server>__<tool>`，对应 extension surface key 为 `mcp__<server>`，不要让 mock 或 UI 退回裸 `server__tool`。同时，Lime runtime 里的 `ToolSearch` 当前事实源必须是 `ToolSearchBridgeTool`；`aster-rust` 自带 `ToolSearchTool` 只能停留在 compat 存量，不允许再抢占当前 runtime surface。
|
||||
如果本轮还需要对子工作区单独跑 Rust 定向测试，例如 `src-tauri/crates/aster-rust`，必须确认产物仍落在统一的 `src-tauri/target`，不要重新写回子目录自己的 `target/`；否则 `tauri dev` 会把构建产物当成源码变化，反复触发重编译。
|
||||
@@ -183,6 +183,22 @@ npm run test:bridge
|
||||
npm run bridge:health -- --timeout-ms 120000
|
||||
```
|
||||
|
||||
如果本轮改动落在 harness cleanup / dashboard 推荐动作契约，还应补：
|
||||
|
||||
```bash
|
||||
npm run harness:cleanup-report:check
|
||||
```
|
||||
|
||||
这条校验已经进入 `npm run test:contracts` 默认门禁；如果你要点检某个指定产物，再显式执行：
|
||||
|
||||
```bash
|
||||
node scripts/check-generated-slop-report.mjs --input "<cleanup-json>"
|
||||
```
|
||||
|
||||
同时，`scripts/report-generated-slop.mjs`、`scripts/check-generated-slop-report.mjs`、`scripts/harness-eval-history-record.mjs`、`scripts/harness-eval-trend-report.mjs`、`scripts/lib/generated-slop-report-core.mjs`、`scripts/lib/harness-dashboard-core.mjs` 这条 harness cleanup/report 主链，在 `verify:local` 的 smart 模式里默认也按 bridge/contracts 风险处理。
|
||||
本地 `verify:local` 输出里如果看到 `bridge 校验（harness cleanup contract）`，说明命中的就是这条 cleanup/report 契约门禁，而不是普通 DevBridge 变更。
|
||||
CI 里的 `.github/workflows/quality.yml` 结果摘要现在也会透出 `bridge_reasons`，并写入 `GITHUB_STEP_SUMMARY`，用于区分这次是 `harness_cleanup_contract`、`bridge_runtime`，还是 `workflow_full_suite` / `fallback_full_suite` 这类全量触发。
|
||||
|
||||
作用：
|
||||
|
||||
- 检查前端命令调用与 Rust 注册表是否一致
|
||||
|
||||
@@ -28,8 +28,8 @@
|
||||
- `src-tauri/src/services/execution_tracker_service.rs`
|
||||
- `src-tauri/src/services/heartbeat_service/mod.rs`
|
||||
- `src-tauri/src/services/heartbeat_service/delivery.rs`
|
||||
- `src-tauri/src/commands/aster_agent_cmd.rs`
|
||||
- `src-tauri/src/commands/subagent_cmd.rs`
|
||||
- `src-tauri/src/commands/aster_agent_cmd/mod.rs`
|
||||
- `src-tauri/src/commands/aster_agent_cmd/tool_runtime/subagent_tools.rs`
|
||||
- `src-tauri/crates/websocket/src/handlers/rpc_handler.rs`
|
||||
- `src-tauri/crates/core/src/database/schema.rs`
|
||||
|
||||
@@ -256,4 +256,3 @@ Lime 的底座（追踪、权限、沙箱、Provider 能力）已经具备，当
|
||||
1. **远程触发协议与命令面**（入口）
|
||||
2. **任务治理状态模型**（稳定性）
|
||||
3. **任务健康与追踪看板**（运营）
|
||||
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
# `docs/exec-plans`
|
||||
|
||||
本目录存放会影响开发执行的 versioned artifact：执行计划、进度日志、阻塞记录、迁移清单、技术债追踪。
|
||||
|
||||
## 放什么
|
||||
|
||||
- 多轮实现、迁移、治理任务的执行计划
|
||||
- 与计划绑定的阶段进度、阻塞项、决策记录
|
||||
- 需要持续小额偿还的技术债与退出条件
|
||||
|
||||
## 命名约定
|
||||
|
||||
- 专项计划：`<topic>-plan.md`
|
||||
- 进度日志：`<topic>-progress.md`
|
||||
- 常驻追踪：使用固定文件名，例如 `tech-debt-tracker.md`
|
||||
|
||||
## 使用规则
|
||||
|
||||
1. 计划不是一次性文档，推进状态变化时要同步更新
|
||||
2. 会改变实现顺序、范围或回滚策略的决策，必须记录在这里或链接到这里
|
||||
3. 清理类工作如果不能直接回挂路线图，应登记到 `tech-debt-tracker.md`
|
||||
4. 被替代的计划不要悬空，保留跳转说明或归档指针
|
||||
|
||||
## 关联入口
|
||||
|
||||
- 路线图主线：`docs/roadmap/`
|
||||
- 技术债追踪：`docs/exec-plans/tech-debt-tracker.md`
|
||||
- 模块级实施细节：`docs/aiprompts/README.md`
|
||||
@@ -0,0 +1,15 @@
|
||||
# 技术债追踪
|
||||
|
||||
本文件记录需要持续、小额偿还的技术债，避免把问题堆到一次性大清理。
|
||||
|
||||
## 记录规则
|
||||
|
||||
1. 每条技术债都要写清具体代码面、影响和下一小步
|
||||
2. 能回挂路线图主线的，优先回挂路线图；不能回挂的，登记到这里
|
||||
3. 状态至少区分 `待处理`、`进行中`、`已完成`、`放弃`
|
||||
|
||||
## 条目
|
||||
|
||||
| ID | 日期 | 区域 | 差距 / 债务 | 影响 | 下一小步 | 关联文档 | 状态 |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| TBD-000 | 2026-04-12 | 待补 | 初始化占位，后续将零散技术债迁入本表 | 先让规则有稳定落点，避免口头追踪 | 首次登记真实条目时替换本行 | `docs/exec-plans/README.md` | 待处理 |
|
||||
@@ -77,7 +77,6 @@ Lime 实际已经具备这些能力：
|
||||
| current | `src-tauri/crates/mcp/src/manager.rs` | MCP tools runtime registry |
|
||||
| current | `src-tauri/src/commands/aster_agent_cmd.rs` | Aster runtime 注入、`ToolSearch`、inventory 命令 |
|
||||
| current | `src-tauri/src/agent_tools/inventory.rs` | runtime 工具库存快照 |
|
||||
| compat | `SubAgentTask` | 兼容旧子代理工具名；current 协作工具面已收敛到 `Agent / SendMessage / TeamCreate / TeamDelete / ListPeers` |
|
||||
| compat | `workspace_allowed_tool_names(...)` | 当前保留为旧调用入口别名，实际委托默认授权目录 |
|
||||
| dead-candidate | `src-tauri/crates/agent/src/tool_permissions.rs` | 已退出 `lime-agent` 的 `lib.rs` 编译图，仅通过 `src-tauri/crates/agent/tests/legacy_permission_surfaces.rs` 测试夹具加载 |
|
||||
| dead-candidate | `src-tauri/crates/agent/src/shell_security.rs` | 已退出 `lime-agent` 的 `lib.rs` 编译图，仅通过 `src-tauri/crates/agent/tests/legacy_permission_surfaces.rs` 测试夹具加载 |
|
||||
@@ -237,8 +236,8 @@ Lime 实际已经具备这些能力：
|
||||
- **Team runtime current surface**
|
||||
`Agent` / `SendMessage` / `TeamCreate` / `TeamDelete` / `ListPeers`
|
||||
|
||||
- **Compat only**
|
||||
`SubAgentTask`
|
||||
- **Compat tool name**
|
||||
当前不再保留 `SubAgentTask`；team runtime 已收敛到 `Agent / SendMessage / TeamCreate / TeamDelete / ListPeers`
|
||||
|
||||
- **说明**
|
||||
Core surface 现已按 current surface 收敛；精确数量与分类以 `src-tauri/src/agent_tools/catalog.rs` 为准。
|
||||
|
||||
@@ -45,9 +45,9 @@
|
||||
- `TeamDelete`
|
||||
- `ListPeers`
|
||||
|
||||
### Compat only
|
||||
### Compat tool name
|
||||
|
||||
- `SubAgentTask`
|
||||
- 当前不再保留 `SubAgentTask`
|
||||
|
||||
### Core 总数
|
||||
|
||||
@@ -111,7 +111,7 @@ Core surface 当前默认 allow 的工具集已经收敛到 current surface，
|
||||
结论：
|
||||
|
||||
- 默认 allowlist 继续优先保留常驻、小而稳的工具面
|
||||
- `SubAgentTask` 已降为 compat 入口，不再作为 current 默认协作工具面
|
||||
- `SubAgentTask` 已从应用层删除，不再作为 current 默认协作工具面
|
||||
- `read` / `write` / `edit` / `bash` / `WebFetch` 这类需要参数约束或更强执行控制的工具仍不应默认放开
|
||||
|
||||
这符合“常驻工具面小而稳”的原则。
|
||||
@@ -218,7 +218,6 @@ const snapshot = await getAgentRuntimeToolInventory({
|
||||
|
||||
## 5.2 compat
|
||||
|
||||
- `SubAgentTask`
|
||||
- `workspace_allowed_tool_names(...)`
|
||||
|
||||
## 5.3 deprecated
|
||||
|
||||
@@ -134,6 +134,16 @@ npm run harness:eval:trend
|
||||
npm run harness:eval:history:record
|
||||
```
|
||||
|
||||
默认会把完整 artifact 套件写到 `./.lime/harness/reports/`：
|
||||
|
||||
- `harness-eval-summary.json`
|
||||
- `harness-eval-summary.md`
|
||||
- `harness-eval-trend.json`
|
||||
- `harness-eval-trend.md`
|
||||
- `harness-cleanup-report.json`
|
||||
- `harness-cleanup-report.md`
|
||||
- `harness-dashboard.html`
|
||||
|
||||
### 运行 Harness cleanup / slop 报告
|
||||
|
||||
```bash
|
||||
|
||||
@@ -47,6 +47,13 @@
|
||||
"source": "repo_fixture",
|
||||
"caseDir": "docs/test/harness-fixtures/replay/minimal-pending-request",
|
||||
"tags": ["conversation-runtime", "replay", "handoff", "evidence"]
|
||||
},
|
||||
{
|
||||
"id": "fixture-minimal-observability-gap",
|
||||
"title": "最小 observability gap Replay 样本",
|
||||
"source": "repo_fixture",
|
||||
"caseDir": "docs/test/harness-fixtures/replay/minimal-observability-gap",
|
||||
"tags": ["conversation-runtime", "replay", "handoff", "evidence", "observability-gap"]
|
||||
}
|
||||
]
|
||||
},
|
||||
|
||||
+47
-12
@@ -56,8 +56,12 @@ Lime 当前不直接把“真实模型重放平台”一次做完，而是先固
|
||||
|
||||
1. **仓库固定 Replay 样本**
|
||||
- 用于 CI / nightly 的稳定入口
|
||||
- 当前固定一条 fixture：
|
||||
- 当前固定两条 fixture：
|
||||
- `fixture-minimal-pending-request`
|
||||
- `fixture-minimal-observability-gap`
|
||||
- 角色分工必须稳定：
|
||||
- `fixture-minimal-pending-request` 是 `current` 样本，要求 observability fully exported，不能继续挂 `known_gap`
|
||||
- `fixture-minimal-observability-gap` 是 `degraded` 样本，专门承载 `known_gap` 语义，避免把证据缺口混进 current 样本
|
||||
- 目的不是替代真实会话，而是先验证 grader 合同、字段预算和摘要出口不会漂移
|
||||
|
||||
2. **仓库沉淀 Replay 样本**
|
||||
@@ -89,6 +93,7 @@ Lime 当前不直接把“真实模型重放平台”一次做完，而是先固
|
||||
3. 校验 replay case 最小四件套与关键 JSON 字段
|
||||
4. 输出统一 JSON / Markdown 摘要，并聚合 `suite tag / failure mode / review decision status / risk level` 分布
|
||||
5. 继续聚合 `observabilitySignals`，把 `requestTelemetry / artifactValidator / browser/gui smoke` 的证据缺口也纳入 trend 与 cleanup
|
||||
6. 继续聚合 `observabilityVerificationOutcomes`，直接复用 evidence pack 导出的紧凑 outcome，告诉治理层当前更像是 `artifactValidator issues / browser failure / gui smoke failed` 还是已通过
|
||||
|
||||
当前它**不直接执行真实模型重放**，而是先把“样本是否可评估、摘要是否可归档”工程化。
|
||||
|
||||
@@ -167,16 +172,34 @@ node scripts/harness-replay-promote.mjs \
|
||||
|
||||
固定下来。
|
||||
|
||||
当前 nightly 还会恢复并追加 `artifacts/history/*.json` 历史窗口，用于让 trend 不只停留在单次 seed。
|
||||
当前 nightly 会恢复并追加 `artifacts/history/*.json` 历史窗口，用于让 trend 不只停留在单次 seed。
|
||||
|
||||
在当前仓库主链里，nightly 还会基于 trend + doc freshness + governance report 继续生成 `harness-cleanup-report.json/md`，让回归样本、人工审核状态和治理建议进入同一份 nightly artifact。
|
||||
从当前主线开始，nightly 不再手工串 `runner -> trend -> cleanup` 三段命令，而是统一走：
|
||||
|
||||
从 `2026-03-27` 起，这个历史窗口不再依赖 workflow 里的 `cp / ls / xargs` 拼接，而是直接由 `scripts/harness-eval-runner.mjs --record-history-dir` 负责写入和裁剪，保证本地与 nightly 走同一条跨平台主链。
|
||||
```bash
|
||||
node scripts/harness-eval-history-record.mjs \
|
||||
--history-dir "./artifacts/history" \
|
||||
--summary-json "./artifacts/harness-eval-summary.json" \
|
||||
--summary-markdown "./artifacts/harness-eval-summary.md" \
|
||||
--trend-json "./artifacts/harness-eval-trend.json" \
|
||||
--trend-markdown "./artifacts/harness-eval-trend.md" \
|
||||
--cleanup-json "./artifacts/harness-cleanup-report.json" \
|
||||
--cleanup-markdown "./artifacts/harness-cleanup-report.md" \
|
||||
--dashboard-html "./artifacts/harness-dashboard.html"
|
||||
```
|
||||
|
||||
这样本地与 nightly 复用同一条 `summary -> history -> trend -> cleanup -> dashboard` 主线，也让 `current / degraded` observability gap 角色在两侧保持同口径，并把 `browser/gui/artifact` 的 verification outcome 焦点直接带进 nightly HTML 仪表板，继续绑定到同一事实源。
|
||||
|
||||
从 `2026-03-27` 起，这个历史窗口不再依赖 workflow 里的 `cp / ls / xargs` 拼接；当前唯一 current 入口是 `scripts/harness-eval-history-record.mjs`，统一负责写入和裁剪 history window，保证本地与 nightly 走同一条跨平台主链。
|
||||
|
||||
同一天新增的 `scripts/harness-eval-history-record.mjs` 又把这条主链向前推了一步：
|
||||
|
||||
- 本地可以一次命令完成 `summary -> history -> trend -> cleanup`
|
||||
- 本地默认把 history window 写到 `./.lime/harness/history`，并把 trend / cleanup 报表写到 `./.lime/harness/reports`
|
||||
- 本地可以一次命令完成 `summary -> history -> trend -> cleanup -> dashboard`
|
||||
- 本地默认把 history window 写到 `./.lime/harness/history`，并把 `summary / trend / cleanup / dashboard` 全部写到 `./.lime/harness/reports`
|
||||
- 如显式传入 `--dashboard-html`，会覆盖默认 HTML 产物路径，但仍直接复用 in-memory 的 `summary / trend / cleanup` 对象生成 HTML，不再额外读取第二套事实
|
||||
- `history-record` 自身的 JSON / text 结果也应直接带出 cleanup 侧的 verification outcome 焦点，避免调用方为了知道“先修 artifact/browser/gui 哪层”还要再手工下钻 cleanup 报告
|
||||
- `cleanup` 当前还应继续导出稳定的 verification 摘要，例如 `failureCaseCount / recoveredCaseCount`，让 dashboard 与 nightly 值班出口可以直接判断是否需要立刻阻断主线
|
||||
- 对 failure outcome 还应继续区分 `blocking_failure / advisory_failure`；默认像 `guiSmoke:failed`、`browserVerification:failure` 这类直接验证失败应被归到 blocking，而 `artifactValidator:issues_present` 这类更偏证据治理的失败应停留在 advisory
|
||||
- `cleanup report` 在未显式传入 `--trend-input / --trend-history-dir` 时，会优先自动发现 `./.lime/harness/history`，再回退到 `./artifacts/history`
|
||||
- 修复后的趋势积累不再只属于 nightly，开发者本地也能拿到同口径的历史窗口
|
||||
|
||||
@@ -203,13 +226,22 @@ node scripts/harness-eval-runner.mjs \
|
||||
--output-json "./tmp/harness-eval-summary.json" \
|
||||
--output-markdown "./tmp/harness-eval-summary.md"
|
||||
|
||||
# 推荐的本地一体化入口：记录 history，并同时刷新 trend / cleanup
|
||||
# 推荐的本地一体化入口：记录 history，并同时刷新 summary / trend / cleanup / dashboard
|
||||
npm run harness:eval:history:record
|
||||
|
||||
# 如需显式指定本地 history window
|
||||
node scripts/harness-eval-runner.mjs \
|
||||
--record-history-dir "./.lime/harness/history" \
|
||||
--history-retain 30
|
||||
# 默认产物目录：
|
||||
# ./.lime/harness/reports/harness-eval-summary.json
|
||||
# ./.lime/harness/reports/harness-eval-summary.md
|
||||
# ./.lime/harness/reports/harness-eval-trend.json
|
||||
# ./.lime/harness/reports/harness-eval-trend.md
|
||||
# ./.lime/harness/reports/harness-cleanup-report.json
|
||||
# ./.lime/harness/reports/harness-cleanup-report.md
|
||||
# ./.lime/harness/reports/harness-dashboard.html
|
||||
|
||||
# 如需覆盖 dashboard 产物路径
|
||||
node scripts/harness-eval-history-record.mjs \
|
||||
--history-dir "./.lime/harness/history" \
|
||||
--dashboard-html "./tmp/harness-dashboard.html"
|
||||
|
||||
# 从历史 summary 目录生成趋势报告
|
||||
node scripts/harness-eval-trend-report.mjs \
|
||||
@@ -229,7 +261,8 @@ Runner 摘要至少回答下面这些问题：
|
||||
- 哪些 case 属于什么 suite tag / failure mode
|
||||
- 哪些 case 默认需要人工复核
|
||||
- 哪些 case 已经记录人工审核状态与风险等级
|
||||
- 哪些 case 还缺 `observabilitySummary` 或仍处于 `requestTelemetry:unlinked`、`artifactValidator:known_gap` 等证据缺口
|
||||
- 哪些 case 还缺 `observabilitySummary` 或仍处于 `requestTelemetry:known_gap`、`artifactValidator:known_gap` 等证据缺口
|
||||
- 哪些 case 已经导出 `observabilityVerificationOutcomes`，并能直接看出 `artifactValidator:issues_present`、`browserVerification:failure`、`guiSmoke:failed` 等 outcome
|
||||
- 工作区 replay 是否已经开始形成增量样本
|
||||
|
||||
如果摘要回答不了这些问题，就说明 runner 还不算进入 current 主链。
|
||||
@@ -241,6 +274,8 @@ Trend 报告至少还要回答：
|
||||
- 哪些 failure mode / suite tag 在 latest 里增长或退化了
|
||||
- 哪些人工审核状态 / 风险等级在 latest 里新增、减少或发生迁移
|
||||
- 哪些 observability signal gap 在 latest 里增加、减少或仍然停留在 current 样本中
|
||||
- 哪些 verification outcome 在 latest 里新增、减少或迁移，足以直接指出先修 artifact/browser/gui 哪一层
|
||||
- 哪些 observability gap 属于 `current` 样本回归，哪些只是 `degraded` 样本刻意保留的诊断基线
|
||||
- 当前只有 trend seed，还是已经开始形成真正的历史窗口
|
||||
|
||||
## 与其他事实源的关系
|
||||
|
||||
@@ -0,0 +1,125 @@
|
||||
{
|
||||
"replayCaseVersion": "v1",
|
||||
"exportedAt": "2026-03-27T14:10:00Z",
|
||||
"handoffBundle": {
|
||||
"relativeRoot": ".lime/harness/sessions/fixture-session-observability-gap/handoff",
|
||||
"absoluteRoot": "/workspace/lime/.lime/harness/sessions/fixture-session-observability-gap/handoff",
|
||||
"artifacts": [
|
||||
{
|
||||
"kind": "plan",
|
||||
"title": "执行计划",
|
||||
"relativePath": ".lime/harness/sessions/fixture-session-observability-gap/handoff/plan.md"
|
||||
},
|
||||
{
|
||||
"kind": "handoff",
|
||||
"title": "交接摘要",
|
||||
"relativePath": ".lime/harness/sessions/fixture-session-observability-gap/handoff/handoff.md"
|
||||
}
|
||||
]
|
||||
},
|
||||
"evidencePack": {
|
||||
"relativeRoot": ".lime/harness/sessions/fixture-session-observability-gap/evidence",
|
||||
"absoluteRoot": "/workspace/lime/.lime/harness/sessions/fixture-session-observability-gap/evidence",
|
||||
"knownGaps": [
|
||||
"当前环境未找到可读取的 request telemetry 日志目录，Evidence Pack 无法导出会话级请求遥测。",
|
||||
"当前检测到 1 个 ArtifactDocument 产物，但 validator outcome 尚未回挂到当前 evidence pack。"
|
||||
],
|
||||
"artifacts": [
|
||||
{
|
||||
"kind": "summary",
|
||||
"title": "证据摘要",
|
||||
"relativePath": ".lime/harness/sessions/fixture-session-observability-gap/evidence/summary.md"
|
||||
},
|
||||
{
|
||||
"kind": "runtime",
|
||||
"title": "运行时快照",
|
||||
"relativePath": ".lime/harness/sessions/fixture-session-observability-gap/evidence/runtime.json"
|
||||
}
|
||||
]
|
||||
},
|
||||
"observabilitySummary": {
|
||||
"schemaVersion": "v1",
|
||||
"correlation": {
|
||||
"correlationKeys": [
|
||||
"session_id",
|
||||
"thread_id",
|
||||
"turn_id",
|
||||
"pending_request_id",
|
||||
"queued_turn_id",
|
||||
"subagent_session_id"
|
||||
],
|
||||
"sessionId": "fixture-session-observability-gap",
|
||||
"threadId": "fixture-thread-observability-gap",
|
||||
"activeTurnId": "turn-gap-001",
|
||||
"pendingRequestIds": [],
|
||||
"queuedTurnIds": [],
|
||||
"subagentSessionIds": []
|
||||
},
|
||||
"counts": {
|
||||
"turnCount": 1,
|
||||
"itemCount": 2,
|
||||
"pendingRequestCount": 0,
|
||||
"queuedTurnCount": 0,
|
||||
"warningCount": 0,
|
||||
"failedToolCallCount": 0,
|
||||
"failedCommandCount": 0,
|
||||
"subagentCount": 0,
|
||||
"recentArtifactCount": 2
|
||||
},
|
||||
"latest": {
|
||||
"warning": null,
|
||||
"failedTool": null,
|
||||
"failedCommand": null
|
||||
},
|
||||
"requestTelemetry": {
|
||||
"source": "lime_infra.telemetry.request_logs",
|
||||
"searchedRoots": [],
|
||||
"matchedRequestCount": 0,
|
||||
"latestRequestAt": null,
|
||||
"statusCounts": {},
|
||||
"providers": [],
|
||||
"models": [],
|
||||
"requests": []
|
||||
},
|
||||
"signalCoverage": [
|
||||
{
|
||||
"signal": "correlation",
|
||||
"status": "exported",
|
||||
"source": "runtime thread identity",
|
||||
"detail": "当前证据包已导出 session/thread/turn/pending request/subagent 关联键。"
|
||||
},
|
||||
{
|
||||
"signal": "timeline",
|
||||
"status": "exported",
|
||||
"source": "timeline.json",
|
||||
"detail": "当前证据包已导出最近 turn 与 item 时间线。"
|
||||
},
|
||||
{
|
||||
"signal": "warnings",
|
||||
"status": "exported",
|
||||
"source": "thread.diagnostics",
|
||||
"detail": "当前线程没有 diagnostics，但 warning 通道已保留在导出结构中。"
|
||||
},
|
||||
{
|
||||
"signal": "requestTelemetry",
|
||||
"status": "known_gap",
|
||||
"source": "lime_infra.telemetry.request_logs",
|
||||
"detail": "当前环境未找到可读取的 request telemetry 日志目录，Evidence Pack 无法导出会话级请求遥测。"
|
||||
},
|
||||
{
|
||||
"signal": "artifactValidator",
|
||||
"status": "known_gap",
|
||||
"source": "artifact_document_validator",
|
||||
"detail": "当前检测到 1 个 ArtifactDocument 产物，但 validator outcome 尚未回挂到当前 evidence pack。"
|
||||
}
|
||||
],
|
||||
"knownGaps": [
|
||||
"当前环境未找到可读取的 request telemetry 日志目录，Evidence Pack 无法导出会话级请求遥测。",
|
||||
"当前检测到 1 个 ArtifactDocument 产物，但 validator outcome 尚未回挂到当前 evidence pack。"
|
||||
]
|
||||
},
|
||||
"recentArtifacts": [
|
||||
"docs/tech/harness/implementation-blueprint.md",
|
||||
".lime/artifacts/thread-gap/report.artifact.json"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"replayCaseVersion": "v1",
|
||||
"exportedAt": "2026-03-27T14:10:00Z",
|
||||
"sessionId": "fixture-session-observability-gap",
|
||||
"threadId": "fixture-thread-observability-gap",
|
||||
"goalSummary": "确认评估链会把 observability 证据缺口显式识别出来，而不是把证据不完整的会话误判为全量覆盖。",
|
||||
"successCriteria": [
|
||||
"评分结果必须明确指出 requestTelemetry 与 artifactValidator 的 known gap。",
|
||||
"若沿用 handoff bundle 与 evidence pack，结论必须引用至少一条证据来源。",
|
||||
"不得把 evidence 缺口包装成 PASS 证据。"
|
||||
],
|
||||
"blockingChecks": [
|
||||
"确认当前线程虽已完成，但 observability 缺口仍被保留下来。",
|
||||
"确认 requestTelemetry 缺日志目录不会被误判为已导出空摘要。"
|
||||
],
|
||||
"artifactChecks": [
|
||||
"确认 `.lime/harness/sessions/fixture-session-observability-gap/handoff/handoff.md` 仍与目标一致。",
|
||||
"确认 evidence pack 中记录的 known gaps 被正确解读。"
|
||||
],
|
||||
"nonGoals": [
|
||||
"不要要求与原始会话完全相同的工具调用顺序。",
|
||||
"不要把措辞差异当作失败，除非它改变了 observability 风险判断。"
|
||||
],
|
||||
"graderSuggestion": {
|
||||
"preferredMode": "result_artifact_and_observability_gap_resolution",
|
||||
"requiresHumanReview": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
# Replay Case 评分说明
|
||||
|
||||
- 会话：`fixture-session-observability-gap`
|
||||
- 线程：`fixture-thread-observability-gap`
|
||||
- 导出时间：2026-03-27T14:10:00Z
|
||||
- 目标摘要：确认评估链会把 observability 证据缺口显式识别出来，而不是把证据不完整的会话误判为全量覆盖。
|
||||
|
||||
## 建议读取顺序
|
||||
|
||||
1. 先读 `input.json`，理解当前任务、线程状态与 observability 缺口。
|
||||
2. 再读 `expected.json`，确认只评估结果与风险。
|
||||
3. 再读 `evidence-links.json`，跳转到 handoff 与 evidence 入口。
|
||||
4. 如需补证据，优先回看 evidence pack 中的 `knownGaps` 与 `observabilitySummary`。
|
||||
|
||||
## 评分原则
|
||||
|
||||
- 只评结果，不评路径。
|
||||
- 先证据后结论；缺口必须引用 evidence，而不是凭印象补齐。
|
||||
- 如果 evidence 已明确记录 `known_gap`，结论里必须解释它是否阻断判定，不能把缺口包装成 PASS 证据。
|
||||
|
||||
## 最小通过条件
|
||||
|
||||
- 结果必须指出 `requestTelemetry` 与 `artifactValidator` 的缺口状态。
|
||||
- 结果必须引用 handoff 或 evidence 中的至少一条证据。
|
||||
- 不得把 observability 缺口误判为“证据已完整”。
|
||||
|
||||
## 建议输出模板
|
||||
|
||||
```text
|
||||
verdict: pass | fail | needs_review
|
||||
reason:
|
||||
- ...
|
||||
evidence:
|
||||
- ...
|
||||
risks:
|
||||
- ...
|
||||
```
|
||||
@@ -0,0 +1,174 @@
|
||||
{
|
||||
"replayCaseVersion": "v1",
|
||||
"source": "lime.fixture.replay_case",
|
||||
"exportedAt": "2026-03-27T14:10:00Z",
|
||||
"session": {
|
||||
"sessionId": "fixture-session-observability-gap",
|
||||
"threadId": "fixture-thread-observability-gap",
|
||||
"workspaceId": "fixture-workspace",
|
||||
"workspaceRoot": "/workspace/lime",
|
||||
"model": "fixture-model",
|
||||
"executionStrategy": "agent_runtime"
|
||||
},
|
||||
"task": {
|
||||
"goalSummary": "确认评估链会把 observability 证据缺口显式识别出来，而不是把证据不完整的会话误判为全量覆盖。",
|
||||
"latestPlan": "先导出 handoff bundle 与 evidence pack，再由 grader 校验 observability gap。",
|
||||
"latestTurnSummary": "线程已完成，但 request telemetry 与 artifact validator 仍存在 known gap。",
|
||||
"latestTurnPrompt": "请把这次 evidence 不完整的会话导出为 replay 样本。",
|
||||
"latestTurnId": "turn-gap-001",
|
||||
"latestTurnStatus": "completed",
|
||||
"threadStatus": "completed",
|
||||
"primaryBlockingSummary": "当前交付已完成，但 observability 证据仍不完整，评分时必须显式说明。"
|
||||
},
|
||||
"classification": {
|
||||
"sourceKind": "repo_fixture",
|
||||
"suiteTags": [
|
||||
"conversation-runtime",
|
||||
"replay",
|
||||
"observability-gap",
|
||||
"handoff",
|
||||
"evidence"
|
||||
],
|
||||
"failureModes": [
|
||||
"observability_gap"
|
||||
],
|
||||
"primaryBlockingKind": "completed"
|
||||
},
|
||||
"runtimeContext": {
|
||||
"pendingRequests": [],
|
||||
"queuedTurns": [],
|
||||
"todoItems": [
|
||||
{
|
||||
"content": "补 request telemetry 关联",
|
||||
"status": "in_progress"
|
||||
}
|
||||
],
|
||||
"activeSubagents": [],
|
||||
"recentArtifacts": [
|
||||
"docs/tech/harness/implementation-blueprint.md",
|
||||
".lime/artifacts/thread-gap/report.artifact.json"
|
||||
],
|
||||
"recentTimeline": [
|
||||
{
|
||||
"itemId": "timeline-gap-001",
|
||||
"turnId": "turn-gap-001",
|
||||
"payloadKind": "plan",
|
||||
"status": "completed",
|
||||
"summary": "产出 observability gap 导出计划",
|
||||
"updatedAt": "2026-03-27T14:02:00Z"
|
||||
},
|
||||
{
|
||||
"itemId": "timeline-gap-002",
|
||||
"turnId": "turn-gap-001",
|
||||
"payloadKind": "file_artifact",
|
||||
"status": "completed",
|
||||
"summary": "导出 evidence pack 与 handoff",
|
||||
"updatedAt": "2026-03-27T14:07:00Z"
|
||||
}
|
||||
],
|
||||
"lastOutcome": {
|
||||
"thread_id": "fixture-thread-observability-gap",
|
||||
"turn_id": "turn-gap-001",
|
||||
"outcome_type": "success",
|
||||
"summary": "handoff 与 evidence 已导出，但 observability known gap 待后续治理",
|
||||
"primary_cause": "observability_gap",
|
||||
"retryable": true
|
||||
},
|
||||
"incidents": []
|
||||
},
|
||||
"observability": {
|
||||
"schemaVersion": "v1",
|
||||
"correlation": {
|
||||
"correlationKeys": [
|
||||
"session_id",
|
||||
"thread_id",
|
||||
"turn_id",
|
||||
"pending_request_id",
|
||||
"queued_turn_id",
|
||||
"subagent_session_id"
|
||||
],
|
||||
"sessionId": "fixture-session-observability-gap",
|
||||
"threadId": "fixture-thread-observability-gap",
|
||||
"activeTurnId": "turn-gap-001",
|
||||
"pendingRequestIds": [],
|
||||
"queuedTurnIds": [],
|
||||
"subagentSessionIds": []
|
||||
},
|
||||
"counts": {
|
||||
"turnCount": 1,
|
||||
"itemCount": 2,
|
||||
"pendingRequestCount": 0,
|
||||
"queuedTurnCount": 0,
|
||||
"warningCount": 0,
|
||||
"failedToolCallCount": 0,
|
||||
"failedCommandCount": 0,
|
||||
"subagentCount": 0,
|
||||
"recentArtifactCount": 2
|
||||
},
|
||||
"latest": {
|
||||
"warning": null,
|
||||
"failedTool": null,
|
||||
"failedCommand": null
|
||||
},
|
||||
"requestTelemetry": {
|
||||
"source": "lime_infra.telemetry.request_logs",
|
||||
"searchedRoots": [],
|
||||
"matchedRequestCount": 0,
|
||||
"latestRequestAt": null,
|
||||
"statusCounts": {},
|
||||
"providers": [],
|
||||
"models": [],
|
||||
"requests": []
|
||||
},
|
||||
"signalCoverage": [
|
||||
{
|
||||
"signal": "correlation",
|
||||
"status": "exported",
|
||||
"source": "runtime thread identity",
|
||||
"detail": "当前证据包已导出 session/thread/turn/pending request/subagent 关联键。"
|
||||
},
|
||||
{
|
||||
"signal": "timeline",
|
||||
"status": "exported",
|
||||
"source": "timeline.json",
|
||||
"detail": "当前证据包已导出最近 turn 与 item 时间线。"
|
||||
},
|
||||
{
|
||||
"signal": "warnings",
|
||||
"status": "exported",
|
||||
"source": "thread.diagnostics",
|
||||
"detail": "当前线程没有 diagnostics，但 warning 通道已保留在导出结构中。"
|
||||
},
|
||||
{
|
||||
"signal": "requestTelemetry",
|
||||
"status": "known_gap",
|
||||
"source": "lime_infra.telemetry.request_logs",
|
||||
"detail": "当前环境未找到可读取的 request telemetry 日志目录，Evidence Pack 无法导出会话级请求遥测。"
|
||||
},
|
||||
{
|
||||
"signal": "artifactValidator",
|
||||
"status": "known_gap",
|
||||
"source": "artifact_document_validator",
|
||||
"detail": "当前检测到 1 个 ArtifactDocument 产物，但 validator outcome 尚未回挂到当前 evidence pack。"
|
||||
}
|
||||
],
|
||||
"knownGaps": [
|
||||
"当前环境未找到可读取的 request telemetry 日志目录，Evidence Pack 无法导出会话级请求遥测。",
|
||||
"当前检测到 1 个 ArtifactDocument 产物，但 validator outcome 尚未回挂到当前 evidence pack。"
|
||||
]
|
||||
},
|
||||
"linkedArtifacts": {
|
||||
"handoffBundle": {
|
||||
"relativeRoot": ".lime/harness/sessions/fixture-session-observability-gap/handoff",
|
||||
"artifactCount": 4
|
||||
},
|
||||
"evidencePack": {
|
||||
"relativeRoot": ".lime/harness/sessions/fixture-session-observability-gap/evidence",
|
||||
"artifactCount": 4,
|
||||
"knownGaps": [
|
||||
"当前环境未找到可读取的 request telemetry 日志目录，Evidence Pack 无法导出会话级请求遥测。",
|
||||
"当前检测到 1 个 ArtifactDocument 产物，但 validator outcome 尚未回挂到当前 evidence pack。"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -20,11 +20,39 @@
|
||||
"evidencePack": {
|
||||
"relativeRoot": ".lime/harness/sessions/fixture-session-minimal-pending-request/evidence",
|
||||
"absoluteRoot": "/workspace/lime/.lime/harness/sessions/fixture-session-minimal-pending-request/evidence",
|
||||
"knownGaps": [
|
||||
"当前 Lime 虽然已有全局 RequestLog / token / retry / duration 遥测，但 RequestLog 尚未携带 session/thread/turn 关联键，Evidence Pack 只能导出 request telemetry coverage gap，不能给出精确的会话级摘要。",
|
||||
"当前 Evidence Pack 尚未纳入 ArtifactDocument validator outcome 与修复问题摘要。",
|
||||
"当前 Evidence Pack 尚未纳入 GUI smoke / browser 验证结果。"
|
||||
],
|
||||
"knownGaps": [],
|
||||
"verification": {
|
||||
"artifactValidatorIssues": [
|
||||
{
|
||||
"path": ".lime/artifacts/thread-1/report.artifact.json",
|
||||
"issues": [
|
||||
"title 缺失或为空，已使用兜底标题。"
|
||||
],
|
||||
"repaired": true,
|
||||
"fallbackUsed": false
|
||||
}
|
||||
],
|
||||
"browserEvidence": [
|
||||
{
|
||||
"itemId": "browser-tool-fixture-001",
|
||||
"turnId": "turn-fixture-001",
|
||||
"toolName": "browser_snapshot",
|
||||
"status": "completed",
|
||||
"updatedAt": "2026-03-27T11:22:00Z"
|
||||
}
|
||||
],
|
||||
"guiSmoke": {
|
||||
"itemId": "gui-smoke-fixture-001",
|
||||
"turnId": "turn-fixture-001",
|
||||
"status": "completed",
|
||||
"command": "npm run verify:gui-smoke",
|
||||
"cwd": "/workspace/lime",
|
||||
"exitCode": 0,
|
||||
"error": null,
|
||||
"updatedAt": "2026-03-27T11:23:30Z",
|
||||
"outputPreview": "GUI smoke finished successfully"
|
||||
}
|
||||
},
|
||||
"artifacts": [
|
||||
{
|
||||
"kind": "summary",
|
||||
@@ -65,13 +93,25 @@
|
||||
"failedToolCallCount": 0,
|
||||
"failedCommandCount": 0,
|
||||
"subagentCount": 0,
|
||||
"recentArtifactCount": 2
|
||||
"recentArtifactCount": 3
|
||||
},
|
||||
"latest": {
|
||||
"warning": null,
|
||||
"failedTool": null,
|
||||
"failedCommand": null
|
||||
},
|
||||
"requestTelemetry": {
|
||||
"source": "lime_infra.telemetry.request_logs",
|
||||
"searchedRoots": [
|
||||
"/workspace/lime/request_logs"
|
||||
],
|
||||
"matchedRequestCount": 0,
|
||||
"latestRequestAt": null,
|
||||
"statusCounts": {},
|
||||
"providers": [],
|
||||
"models": [],
|
||||
"requests": []
|
||||
},
|
||||
"signalCoverage": [
|
||||
{
|
||||
"signal": "correlation",
|
||||
@@ -93,37 +133,57 @@
|
||||
},
|
||||
{
|
||||
"signal": "requestTelemetry",
|
||||
"status": "unlinked",
|
||||
"source": "lime_infra.telemetry",
|
||||
"detail": "Lime 已有 workspace 级 request telemetry，但当前 RequestLog 还未携带 session/thread/turn 元数据。"
|
||||
"status": "exported",
|
||||
"source": "lime_infra.telemetry.request_logs",
|
||||
"detail": "当前证据包已扫描 request telemetry 日志目录，但当前会话未匹配到 provider request 记录。"
|
||||
},
|
||||
{
|
||||
"signal": "artifactValidator",
|
||||
"status": "known_gap",
|
||||
"status": "exported",
|
||||
"source": "artifact_document_validator",
|
||||
"detail": "Artifact validator outcome 尚未回挂到当前 evidence pack。"
|
||||
"detail": "当前证据包已为 1 个 ArtifactDocument 产物导出 validator outcome。"
|
||||
},
|
||||
{
|
||||
"signal": "browserVerification",
|
||||
"status": "known_gap",
|
||||
"status": "exported",
|
||||
"source": "browser runtime",
|
||||
"detail": "浏览器截图、DOM 快照和交互验证结果尚未接入 evidence pack。"
|
||||
"detail": "当前证据包已导出 1 条浏览器验证线索。"
|
||||
},
|
||||
{
|
||||
"signal": "guiSmoke",
|
||||
"status": "known_gap",
|
||||
"status": "exported",
|
||||
"source": "verify:gui-smoke",
|
||||
"detail": "GUI smoke 结果尚未作为结构化证据回挂当前会话。"
|
||||
"detail": "当前证据包已导出 GUI smoke 运行结果。"
|
||||
}
|
||||
],
|
||||
"knownGaps": [
|
||||
"当前 Lime 虽然已有全局 RequestLog / token / retry / duration 遥测，但 RequestLog 尚未携带 session/thread/turn 关联键，Evidence Pack 只能导出 request telemetry coverage gap，不能给出精确的会话级摘要。",
|
||||
"当前 Evidence Pack 尚未纳入 ArtifactDocument validator outcome 与修复问题摘要。",
|
||||
"当前 Evidence Pack 尚未纳入 GUI smoke / browser 验证结果。"
|
||||
]
|
||||
"verificationSummary": {
|
||||
"artifactValidator": {
|
||||
"applicable": true,
|
||||
"recordCount": 1,
|
||||
"issueCount": 1,
|
||||
"repairedCount": 1,
|
||||
"fallbackUsedCount": 0
|
||||
},
|
||||
"browserVerification": {
|
||||
"recordCount": 1,
|
||||
"successCount": 1,
|
||||
"failureCount": 0,
|
||||
"unknownCount": 0,
|
||||
"latestUpdatedAt": "2026-03-27T11:22:00Z"
|
||||
},
|
||||
"guiSmoke": {
|
||||
"status": "completed",
|
||||
"exitCode": 0,
|
||||
"passed": true,
|
||||
"updatedAt": "2026-03-27T11:23:30Z",
|
||||
"hasOutputPreview": true
|
||||
}
|
||||
},
|
||||
"knownGaps": []
|
||||
},
|
||||
"recentArtifacts": [
|
||||
"docs/tech/harness/implementation-blueprint.md",
|
||||
".lime/artifacts/thread-1/report.artifact.json",
|
||||
".lime/harness/sessions/fixture-session-minimal-pending-request/handoff/handoff.md"
|
||||
]
|
||||
}
|
||||
|
||||
@@ -15,7 +15,7 @@
|
||||
],
|
||||
"artifactChecks": [
|
||||
"确认 `.lime/harness/sessions/fixture-session-minimal-pending-request/handoff/handoff.md` 仍与目标一致。",
|
||||
"确认 evidence pack 中记录的 known gaps 没有被当作 PASS 证据。"
|
||||
"确认 evidence pack 中导出的 verification 证据与 pending request 风险被正确解读。"
|
||||
],
|
||||
"nonGoals": [
|
||||
"不要要求与原始会话完全相同的工具调用顺序。",
|
||||
|
||||
@@ -56,6 +56,7 @@
|
||||
"activeSubagents": [],
|
||||
"recentArtifacts": [
|
||||
"docs/tech/harness/implementation-blueprint.md",
|
||||
".lime/artifacts/thread-1/report.artifact.json",
|
||||
".lime/harness/sessions/fixture-session-minimal-pending-request/handoff/handoff.md"
|
||||
],
|
||||
"recentTimeline": [
|
||||
@@ -113,13 +114,25 @@
|
||||
"failedToolCallCount": 0,
|
||||
"failedCommandCount": 0,
|
||||
"subagentCount": 0,
|
||||
"recentArtifactCount": 2
|
||||
"recentArtifactCount": 3
|
||||
},
|
||||
"latest": {
|
||||
"warning": null,
|
||||
"failedTool": null,
|
||||
"failedCommand": null
|
||||
},
|
||||
"requestTelemetry": {
|
||||
"source": "lime_infra.telemetry.request_logs",
|
||||
"searchedRoots": [
|
||||
"/workspace/lime/request_logs"
|
||||
],
|
||||
"matchedRequestCount": 0,
|
||||
"latestRequestAt": null,
|
||||
"statusCounts": {},
|
||||
"providers": [],
|
||||
"models": [],
|
||||
"requests": []
|
||||
},
|
||||
"signalCoverage": [
|
||||
{
|
||||
"signal": "correlation",
|
||||
@@ -141,34 +154,53 @@
|
||||
},
|
||||
{
|
||||
"signal": "requestTelemetry",
|
||||
"status": "unlinked",
|
||||
"source": "lime_infra.telemetry",
|
||||
"detail": "Lime 已有 workspace 级 request telemetry，但当前 RequestLog 还未携带 session/thread/turn 元数据。"
|
||||
"status": "exported",
|
||||
"source": "lime_infra.telemetry.request_logs",
|
||||
"detail": "当前证据包已扫描 request telemetry 日志目录，但当前会话未匹配到 provider request 记录。"
|
||||
},
|
||||
{
|
||||
"signal": "artifactValidator",
|
||||
"status": "known_gap",
|
||||
"status": "exported",
|
||||
"source": "artifact_document_validator",
|
||||
"detail": "Artifact validator outcome 尚未回挂到当前 evidence pack。"
|
||||
"detail": "当前证据包已为 1 个 ArtifactDocument 产物导出 validator outcome。"
|
||||
},
|
||||
{
|
||||
"signal": "browserVerification",
|
||||
"status": "known_gap",
|
||||
"status": "exported",
|
||||
"source": "browser runtime",
|
||||
"detail": "浏览器截图、DOM 快照和交互验证结果尚未接入 evidence pack。"
|
||||
"detail": "当前证据包已导出 1 条浏览器验证线索。"
|
||||
},
|
||||
{
|
||||
"signal": "guiSmoke",
|
||||
"status": "known_gap",
|
||||
"status": "exported",
|
||||
"source": "verify:gui-smoke",
|
||||
"detail": "GUI smoke 结果尚未作为结构化证据回挂当前会话。"
|
||||
"detail": "当前证据包已导出 GUI smoke 运行结果。"
|
||||
}
|
||||
],
|
||||
"knownGaps": [
|
||||
"当前 Lime 虽然已有全局 RequestLog / token / retry / duration 遥测，但 RequestLog 尚未携带 session/thread/turn 关联键，Evidence Pack 只能导出 request telemetry coverage gap，不能给出精确的会话级摘要。",
|
||||
"当前 Evidence Pack 尚未纳入 ArtifactDocument validator outcome 与修复问题摘要。",
|
||||
"当前 Evidence Pack 尚未纳入 GUI smoke / browser 验证结果。"
|
||||
]
|
||||
"verificationSummary": {
|
||||
"artifactValidator": {
|
||||
"applicable": true,
|
||||
"recordCount": 1,
|
||||
"issueCount": 1,
|
||||
"repairedCount": 1,
|
||||
"fallbackUsedCount": 0
|
||||
},
|
||||
"browserVerification": {
|
||||
"recordCount": 1,
|
||||
"successCount": 1,
|
||||
"failureCount": 0,
|
||||
"unknownCount": 0,
|
||||
"latestUpdatedAt": "2026-03-27T11:22:00Z"
|
||||
},
|
||||
"guiSmoke": {
|
||||
"status": "completed",
|
||||
"exitCode": 0,
|
||||
"passed": true,
|
||||
"updatedAt": "2026-03-27T11:23:30Z",
|
||||
"hasOutputPreview": true
|
||||
}
|
||||
},
|
||||
"knownGaps": []
|
||||
},
|
||||
"linkedArtifacts": {
|
||||
"handoffBundle": {
|
||||
@@ -178,11 +210,39 @@
|
||||
"evidencePack": {
|
||||
"relativeRoot": ".lime/harness/sessions/fixture-session-minimal-pending-request/evidence",
|
||||
"artifactCount": 4,
|
||||
"knownGaps": [
|
||||
"当前 Lime 虽然已有全局 RequestLog / token / retry / duration 遥测，但 RequestLog 尚未携带 session/thread/turn 关联键，Evidence Pack 只能导出 request telemetry coverage gap，不能给出精确的会话级摘要。",
|
||||
"当前 Evidence Pack 尚未纳入 ArtifactDocument validator outcome 与修复问题摘要。",
|
||||
"当前 Evidence Pack 尚未纳入 GUI smoke / browser 验证结果。"
|
||||
]
|
||||
"verification": {
|
||||
"artifactValidatorIssues": [
|
||||
{
|
||||
"path": ".lime/artifacts/thread-1/report.artifact.json",
|
||||
"issues": [
|
||||
"title 缺失或为空，已使用兜底标题。"
|
||||
],
|
||||
"repaired": true,
|
||||
"fallbackUsed": false
|
||||
}
|
||||
],
|
||||
"browserEvidence": [
|
||||
{
|
||||
"itemId": "browser-tool-fixture-001",
|
||||
"turnId": "turn-fixture-001",
|
||||
"toolName": "browser_snapshot",
|
||||
"status": "completed",
|
||||
"updatedAt": "2026-03-27T11:22:00Z"
|
||||
}
|
||||
],
|
||||
"guiSmoke": {
|
||||
"itemId": "gui-smoke-fixture-001",
|
||||
"turnId": "turn-fixture-001",
|
||||
"status": "completed",
|
||||
"command": "npm run verify:gui-smoke",
|
||||
"cwd": "/workspace/lime",
|
||||
"exitCode": 0,
|
||||
"error": null,
|
||||
"updatedAt": "2026-03-27T11:23:30Z",
|
||||
"outputPreview": "GUI smoke finished successfully"
|
||||
}
|
||||
},
|
||||
"knownGaps": []
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user