docs: plan Pi runtime resilience hardening
This commit is contained in:
@@ -0,0 +1,444 @@
|
||||
# Pi Runtime 鲁棒性加固规范与实施计划
|
||||
|
||||
## 0. 文档信息
|
||||
|
||||
| 字段 | 值 |
|
||||
| --- | --- |
|
||||
| 状态 | 已获用户授权,可实施 |
|
||||
| 规划任务 | `20260825-pi-runtime-resilience-plan-4b8e2c71` |
|
||||
| 规划基线 | `274187e3cfe2ae6cb97f2b4bcb422b3fdf06fbf2` |
|
||||
| 原始规范 | `20260822-pi-runtime-spec-b6e2c9a4__pi-runtime-cutover-spec.md` |
|
||||
| 原始票据图 | `20260822-pi-runtime-spec-b6e2c9a4__pi-runtime-cutover-tickets.md` |
|
||||
| 故障诊断 | `20260825-pi-stuck-processing-diagnosis-7c4e91a2` |
|
||||
| 当前实施任务 | `20260825-pi-worker-exit-thinking-5e7c2a91` |
|
||||
| 当前实施基线 | `274187e3cfe2ae6cb97f2b4bcb422b3fdf06fbf2` |
|
||||
| Pi 版本 | `@earendil-works/pi-coding-agent@0.84.2`,保持精确 pin |
|
||||
| 日期 | 2026-08-25 |
|
||||
|
||||
本文是既有 OpenCode → Pi 硬切换规范的事故后补充,不修改原 18 节点 / 31 边 DAG,也不恢复 OpenCode fallback。它把原规范中的 `RUN-004`、`RUN-007`、`RPC-004`、`API-003`、`UX-023`、`UX-026` 和第 18 节故障恢复要求收紧为可执行、可观测、可在最终 Windows 安装包中故障注入的合同。
|
||||
|
||||
本次不承诺第三方 Pi 进程“永不退出”。产品承诺是:Pi worker 可以失败,但 Makelore 应用不随之退出;Conversation 不永久锁死;未确认的请求不自动重发;任何 worker 重建都有明确原因并受 generation 隔离。
|
||||
|
||||
## 1. 事实、推测与未验证边界
|
||||
|
||||
### 1.1 已确认事实
|
||||
|
||||
1. 事故发生后 Electron Main 与 Renderer 仍存活,目标 Pi RPC worker 已不存在。
|
||||
2. 目标 session JSONL 结构有效;最后是 assistant `toolUse(task_state)` 与对应 `toolResult`,没有后续权威 `agent_settled`。
|
||||
3. `PiWorkerPool.handleInvalidation()` 已将 worker 标为 `crashed`、释放 top-level permit 并发出 `worker.crashed`。
|
||||
4. `PiConversationRuntime.onPoolEvent()` 对 `worker.crashed` 只投影 `worker.state`,没有调用已有 `failRun()`,因此 worker 已错误而 run 仍为 `running`。
|
||||
5. `PiConversationRuntime.abort()` 在 worker 已退出时请求失败,当前 catch 会恢复旧的 `running` 状态,造成第二条永久锁死路径。
|
||||
6. Renderer 的“生成中”“当前对话正在处理”和 Send 禁用均由 run 状态派生;Renderer 是无效 Main 状态的下游表现,不是根因。
|
||||
7. 复制 session 后,使用最终安装包 Electron executable、Pi 0.84.2、实际 Makelore extension 与受控 loopback Provider 的 11 类故障矩阵均正常 `agent_settled`;普通 HTTP 500、已测的 partial/malformed SSE、正常 `task_state` 和 copied session 不是此次退出的确定性原因。
|
||||
8. 当前 `PI_RPC_EXITED` 没有携带已有的脱敏 stderr 诊断;Main 日志也不能区分自发退出、RPC 协议失效后强杀和 Main 主动 stop。
|
||||
9. `set_thinking_level=high` 在当前托管模型形状下可能实际得到 effective `off`,但产品仍持久化请求值;该能力合同错误已确认,但没有证据表明它导致了 worker 退出。
|
||||
|
||||
### 1.2 合理推测
|
||||
|
||||
- 当前最严重的产品缺陷不是“发生了一个未知退出”,而是生命周期模块未把退出收敛为 Conversation 终态,并且缺少能定位退出发起方的结构化证据。
|
||||
- 只增加自动重启或固定时长 watchdog 会掩盖状态机缺陷,并可能重复执行已经被 Provider 或 tool 接受的 mutation。
|
||||
- 保持一 Conversation 一 worker 的隔离模型,比改成单一全局 Pi daemon 更能控制故障半径。
|
||||
|
||||
### 1.3 尚未验证
|
||||
|
||||
- 本次 worker 消失究竟是一次真实 HTTPS/Undici/Electron-Node 异常、Main 侧 RPC protocol invalidation,还是主动 stop/rebuild 竞态,现有 artifact 无法判定。
|
||||
- 真实 Provider 的认证、代理、限流、响应差异、真实并发和跨 worker 凭据隔离仍未验证。用户已将其标记为 `Explicitly Waived / Accepted Risk`,`realTurnVerified=false`。
|
||||
- macOS x64/arm64 与 native non-WSL Linux 的发布证据状态不因本规范改变。
|
||||
|
||||
## 2. 目标与非目标
|
||||
|
||||
### 2.1 目标
|
||||
|
||||
1. Pi worker 异常退出不得导致 Electron Main、Renderer 或整个应用退出。
|
||||
2. matching generation 的 worker 一旦不可用,任何仍活动的 run 必须有界收敛到 `status=error`、`terminalReason=failed`、`recoverable=true`。
|
||||
3. Composer 在故障终态恢复编辑/恢复操作,不得继续显示无界“处理中”。
|
||||
4. crash、abort、`agent_settled`、recover 和 generation replacement 必须按 Conversation + generation 串行且终态单调。
|
||||
5. accepted 或 uncertain prompt 永不自动重放;recover 复用同一产品 Conversation 和已有 Pi session binding。
|
||||
6. 每个 Main 主动 stop、protocol invalidation 和 unexpected exit 可明确区分,且诊断不泄露敏感内容。
|
||||
7. thinking level 只持久化 Pi 实际支持并接受的 effective level;切换本身不得隐式重启正常 worker。
|
||||
8. 最终 Windows 安装包具备确定性故障注入证明,而不是只依赖 fake child 单测。
|
||||
|
||||
### 2.2 非目标
|
||||
|
||||
- 不保证操作系统、Electron、Pi、原生依赖或 Provider 永不失败。
|
||||
- 不新增全局 runtime start/stop/restart UI。
|
||||
- 不引入 OpenCode fallback、双 runtime、兼容层、feature flag 或 Pi 上游 fork。
|
||||
- 不添加“长时间无输出就杀 worker”的通用 Provider watchdog。
|
||||
- 不自动重发 prompt、steer、follow-up、fork、compact 或任何 mutation RPC。
|
||||
- 不把 loopback/fault injection 写成真实 Provider Pass。
|
||||
- 不在本票据完成 macOS 或 native non-WSL Linux 发布验收。
|
||||
|
||||
## 3. 架构裁决
|
||||
|
||||
### 3.1 深模块与 seam
|
||||
|
||||
继续由 Main-owned `PiConversationRuntime` 作为产品 Conversation 生命周期的深模块。Renderer、Host route 和 store 只消费 Makelore-owned Snapshot/Patch 与稳定错误;它们不理解 Pi process、RPC、stderr 或 stop reason。
|
||||
|
||||
内部职责保持:
|
||||
|
||||
- `PiWorkerProcess`:spawn、RPC 字节流、exit/close、协议失效、有限诊断和进程树退出。
|
||||
- `PiWorkerPool`:Conversation → worker/generation 归属、permit、LRU、active run、replacement 与 crash event。
|
||||
- `PiConversationRuntime`:产品 run 状态、event projection、terminalization、recover/hydration 与 Host SSE。
|
||||
- `PiSessionRegistry`:Conversation 与 session binding、模型和 thinking metadata 的持久化。
|
||||
- extension/subagent/write-lease 模块:generation/run-owned 资源,失败时可由同一生命周期收敛路径取消。
|
||||
|
||||
不得新增一个只转发这些方法的浅层“supervisor service”。需要的行为应深化现有三个模块,并以现有 `CodingConversationRuntime` interface 作为外部 test seam。
|
||||
|
||||
### 3.2 核心不变量
|
||||
|
||||
| ID | 不变量 |
|
||||
| --- | --- |
|
||||
| `RES-INV-001` | 同一 Conversation 不允许长期存在 `worker=crashed/error` 且 `run=running/aborting/retrying/compacting`。 |
|
||||
| `RES-INV-002` | matching generation 的 crash 只能结束目标 Conversation;其他 Conversation 的 Snapshot、run、queue、worker 与 draft 不变化。 |
|
||||
| `RES-INV-003` | terminal run 单调;较晚到达的 close、abort rejection 或旧 generation event 不得覆盖 completed/aborted/failed。 |
|
||||
| `RES-INV-004` | `clientRequestId` 已 accepted 或状态 uncertain 时,不得因 recover/SSE reconnect/worker replacement 自动再次执行 mutation。 |
|
||||
| `RES-INV-005` | 每个 generation replacement 必须有 Main-owned bounded reason;没有 reason 的 replacement 是测试失败。 |
|
||||
| `RES-INV-006` | running worker 不因 LRU、thinking change、同账号 model change 或普通 diagnostics 被 stop。 |
|
||||
| `RES-INV-007` | worker crash 后 permit、interaction、extension run、child、process lease 与 write lease 最终全部释放。 |
|
||||
| `RES-INV-008` | recover 使用原 Conversation/session binding;不截断、覆盖或静默替换旧 JSONL。 |
|
||||
| `RES-INV-009` | 原始 stderr、Provider body、prompt、tool 内容、Header、Token、账号 ID、完整路径不进入 Renderer 或普通日志。 |
|
||||
|
||||
### 3.3 目标状态流
|
||||
|
||||
```text
|
||||
idle/ready -> preparing/queued -> running -> settling -> idle/completed
|
||||
| |
|
||||
| +-> late close: keep completed
|
||||
+-> worker exit/protocol invalidation
|
||||
-> error/failed/recoverable
|
||||
-> explicit recover
|
||||
-> new generation + same session binding
|
||||
-> hydrate -> idle
|
||||
```
|
||||
|
||||
不存在从 `worker exit` 返回旧 `running` 的边;不存在从 `recover` 自动重发旧 prompt 的边。
|
||||
|
||||
## 4. 故障分类与重建政策
|
||||
|
||||
### 4.1 进程终止分类
|
||||
|
||||
| 分类 | 判断来源 | 产品动作 | 重建政策 |
|
||||
| --- | --- | --- | --- |
|
||||
| Main intentional stop | stop 前已记录 reason | 按原业务完成清理;不伪装 crash | 仅由对应显式操作或安全边界继续 |
|
||||
| RPC protocol invalidation | strict framer/RPC 判定 | 保留 `PI_RPC_PROTOCOL_ERROR`,强制结束目标 generation | 用户 recover;mutation 不重放 |
|
||||
| Unexpected exit | 无 intentional stop、无既有 protocol invalidation 的 exit/close | 保留 `PI_RPC_EXITED` 内部分类并投影稳定 recoverable runtime error | 用户 recover;mutation 不重放 |
|
||||
| App shutdown | `app_shutdown` reason | 拒绝新 prompt,3 秒 graceful 后整树 kill | 下次启动按 registry reopen |
|
||||
| Idle eviction | `idle_eviction` reason,且 run 已 idle | 保留 session binding,不投影 run failure | 下次显式 prepare 可重开 |
|
||||
| Stale resource rebuild | `stale_resource_rebuild` reason | active run 先 settled;旧 generation 失效 | settled 后一次重建 |
|
||||
| Recover/dispose/fork replacement | 对应显式 reason | 取消旧 generation 资源并保持产品语义 | 只执行该操作要求的 replacement |
|
||||
|
||||
Main 主动 stop reason 使用有限内部类型,至少覆盖:
|
||||
|
||||
```text
|
||||
app_shutdown
|
||||
idle_eviction
|
||||
stale_resource_rebuild
|
||||
recover
|
||||
dispose
|
||||
fork_replacement
|
||||
protocol_invalidation
|
||||
test_injection
|
||||
```
|
||||
|
||||
实现可使用与当前调用点等价的更精确名称,但禁止自由文本 reason 和缺省 reason。
|
||||
|
||||
### 4.2 自动动作边界
|
||||
|
||||
- 保留既有只读 RPC 在 generation 不变时最多一次 retry 的合同。
|
||||
- spawn/ready 失败继续进入明确 recoverable error,不添加后台无限拉起。
|
||||
- accepted run 的 worker 退出后不自动新建 generation,也不自动重发请求。
|
||||
- 用户显式 recover 才执行“关闭旧 generation → 清理 → spawn → reopen same binding → hydrate”。
|
||||
- Provider 的 429/5xx 只使用 Pi 自己的有界 retry;Main 不叠加 retry。
|
||||
- 不使用无 token/无事件时长作为 worker 死亡判断。合法的模型推理、工具执行和 compaction 可能很长。
|
||||
|
||||
## 5. 运行时终态与竞态合同
|
||||
|
||||
### 5.1 `worker.crashed`
|
||||
|
||||
`PiConversationRuntime` 必须把 crash handling 放入目标 Conversation 的 projection chain:
|
||||
|
||||
1. 读取事件携带的 Conversation、generation 和原始 `PiProcessError`。
|
||||
2. 若 Snapshot 不存在或 generation 已过期,只做脱敏 lifecycle 记录,不改产品状态。
|
||||
3. 取消该 generation 的 interaction、extension run、child 与其他 run-owned 资源。
|
||||
4. 若 matching run 仍为活动状态,调用统一 `failRun()`,投影 `status=error`、`terminalReason=failed`、`settledAt` 与稳定 recoverable error。
|
||||
5. 投影 worker error state。
|
||||
6. 若 run 已被较早的权威 `agent_settled` 终结,不得改写终态。
|
||||
|
||||
### 5.2 `abort()`
|
||||
|
||||
- live worker 的正常 abort 继续等待权威 `agent_settled`,不得伪造成功。
|
||||
- abort request 失败后必须重新读取最新 run 与 matching worker generation。
|
||||
- run 已 terminal:保留终态并返回/投影对应结果。
|
||||
- worker 已 crashed/absent 且仍是同一活动 run:收敛到与 crash 相同的 failed/recoverable 终态。
|
||||
- worker 仍 live、abort RPC 只是可恢复命令错误:可以恢复原活动态并返回错误,但必须证明 worker 仍归该 generation。
|
||||
- 任何路径都不得把 `error/completed/aborted` 恢复成 `running`。
|
||||
|
||||
### 5.3 事件顺序
|
||||
|
||||
必须覆盖以下交错:
|
||||
|
||||
1. `worker.crashed -> abort rejection`;最终 failed。
|
||||
2. `abort request -> worker.crashed`;最终 failed,不回到 running。
|
||||
3. `agent_settled(completed) -> close`;保持 completed。
|
||||
4. `worker.crashed -> late agent_settled from old generation`;忽略 late event,保持 failed。
|
||||
5. `recover generation N+1 -> generation N late response/event/interaction`;全部丢弃。
|
||||
6. 目标 Conversation crash 时另一个 Conversation running/settled;另一个不受影响。
|
||||
|
||||
## 6. 资源与会话完整性
|
||||
|
||||
### 6.1 必须收敛的资源
|
||||
|
||||
| 资源 | 所有者 | crash 后要求 |
|
||||
| --- | --- | --- |
|
||||
| top-level running permit | Worker pool | 释放一次,计数不得为负 |
|
||||
| queued run | Worker pool | 目标项以原始错误拒绝,不重排其他 FIFO |
|
||||
| process lease | Worker pool/process budget | `finally` 释放 |
|
||||
| interaction | Interaction store | generation/run scoped cancel |
|
||||
| extension run binding | Extension host/UI | clear/end,旧 token 无效 |
|
||||
| subagent dispatch/child permit | Scheduler | parent crash 向下取消并释放 |
|
||||
| child process | Child opener | 有界 stop/kill,零残留 |
|
||||
| project write lease | Write lease | holder/waiter 均可取消,其他项目不受影响 |
|
||||
|
||||
所有清理动作必须可重复;一个 cleanup step 失败不得阻止后续资源释放。不得用吞掉所有错误来伪装成功,首个失败应进入脱敏诊断。
|
||||
|
||||
### 6.2 Session 与请求安全
|
||||
|
||||
- worker exit 保留 session JSONL 和 registry binding。
|
||||
- recover 不创建新的产品 Conversation,不改变稳定 Conversation ID。
|
||||
- hydrate 从 `get_state/get_entries` 重建 Snapshot,并继续使用 generation fence。
|
||||
- 已持久化的 assistant/tool 结果可以水合;尚未确认的 prompt 不重发。
|
||||
- draft、待上传 attachment 和 optimistic user node 按现有 request reconciliation 规则保留。
|
||||
- session unreadable 继续 fail closed;不得自动截断或覆盖文件。
|
||||
|
||||
## 7. Model、thinking 与 revision 合同
|
||||
|
||||
### 7.1 Thinking
|
||||
|
||||
1. 调用前获取或使用已验证的 `get_available_thinking_levels` 能力。
|
||||
2. 不支持的 level 必须拒绝,或由明确产品规则选择并展示有效值;不得请求 `high` 却持久化一个 Pi 实际为 `off` 的假状态。
|
||||
3. `set_thinking_level` 成功后读取 Pi effective state,只有 effective value 可写入 Registry 和 Snapshot。
|
||||
4. 写 Registry 失败时保持旧产品状态并投影稳定 persistence error。
|
||||
5. thinking change 不 replacement worker generation。
|
||||
6. 当 Provider 无法真正关闭 reasoning 时,UI 不得把“关闭思考”表现为可保证抑制 Provider reasoning 的能力。
|
||||
|
||||
### 7.2 Model 与资源 revision
|
||||
|
||||
- 同 Provider account 的 model change 继续使用 `set_model`,不重建 worker。
|
||||
- 跨 Provider account 只重建目标 Conversation,并保持现有 credential isolation。
|
||||
- Provider/resource/credential revision 在 run active 时只标记 stale;settled 后用显式 reason 重建。
|
||||
- revision rebuild、recover、fork replacement 的 generation reason 必须进入 Main-only lifecycle 记录。
|
||||
|
||||
## 8. 可观测性与隐私
|
||||
|
||||
### 8.1 Main-only lifecycle 事件
|
||||
|
||||
至少记录以下结构化事件:
|
||||
|
||||
```text
|
||||
worker.stop_requested
|
||||
worker.protocol_invalidated
|
||||
worker.exit
|
||||
worker.close
|
||||
worker.crashed
|
||||
run.failed
|
||||
worker.replacement_started
|
||||
worker.replacement_ready
|
||||
worker.recovered
|
||||
```
|
||||
|
||||
允许字段:匿名/局部 Conversation 标识、runId 局部标识、generation、stage、stop reason、错误 code、exit code、signal、cold/warm、duration、是否完成各类 cleanup。
|
||||
|
||||
禁止字段:prompt、assistant 内容、thinking 内容、tool 名称/参数/结果、文件内容、完整路径、Provider body、Header、Token、credential、账号 ID、完整 request ID。
|
||||
|
||||
### 8.2 错误保真
|
||||
|
||||
- `PiWorkerProcess` 在 `PI_RPC_EXITED` 中附带已有的 bounded/redacted stderr tail。
|
||||
- protocol invalidation 保留原 `PI_RPC_PROTOCOL_ERROR`,后续 close 不得覆盖错误类别。
|
||||
- intentional stop 不能被 close handler 再标成 crash。
|
||||
- stderr 只进入 Main-owned diagnostic/log,不进入 Renderer DTO。
|
||||
- 若新安装包再次发生问题,证据必须能直接回答“谁发起 stop、哪类 invalidation、哪个 generation、退出码/信号是什么”。
|
||||
|
||||
如结构化证据仍只显示无法解释的原生异常终止,再单独评估 opt-in Windows dump;本票据不先引入 crash-dump 基础设施。
|
||||
|
||||
## 9. 用户体验合同
|
||||
|
||||
- worker crash 后,目标 Conversation 在 Main 收到 exit/invalidated 后不晚于 2 秒进入 recoverable error。
|
||||
- Textarea 保持可编辑;Composer 不再显示无界“当前对话正在处理”。
|
||||
- UI 显示固定中文说明:本地 Agent 已中断,原请求未自动重发。
|
||||
- 提供目标 Conversation 的 recover 操作;recover 失败保持可再次操作的错误态,不进入永久 recovering。
|
||||
- 另一个 Conversation 的生成、输入、队列和错误不变化。
|
||||
- 技术 stderr、exit code、内部路径和 stop reason 不直接展示给普通用户;诊断面板只保留安全结构化状态。
|
||||
|
||||
## 10. 实施票据与阶段
|
||||
|
||||
本规范新增事故后纠正票据 `PI-R001`,不插入或重写原 18 节点 / 31 边 cutover DAG。
|
||||
|
||||
### PI-R001 — Pi worker failure convergence and explainable lifecycle
|
||||
|
||||
| 字段 | 值 |
|
||||
| --- | --- |
|
||||
| 状态 | 已授权;实施中 |
|
||||
| 实施任务 | `20260825-pi-worker-exit-thinking-5e7c2a91` |
|
||||
| 基线 | `274187e3cfe2ae6cb97f2b4bcb422b3fdf06fbf2` |
|
||||
| 依赖 | 已交付的 Pi hard cutover、Windows proxy/session reopen hotfix、故障诊断任务 |
|
||||
| 解锁 | 新 Windows 安装包人工复测;后续 `PI-R002` soak/现场归因 |
|
||||
|
||||
#### 阶段 A:生命周期终态
|
||||
|
||||
- 修改 `electron/coding-runtime/pi/runtime.ts` 的 crash projection 与 abort race。
|
||||
- 必要时调整 `worker-pool.ts` 事件载荷,但不得扩大公共 Renderer interface。
|
||||
- 先添加 red-capable 回归,再修改实现。
|
||||
|
||||
产物:worker crash/abort/settled 的单调状态机;目标 run 不再永久 active。
|
||||
|
||||
#### 阶段 B:退出分类与 stop reason
|
||||
|
||||
- 修改 `worker-process.ts`、`process-errors.ts`、`worker-pool.ts`、`telemetry.ts`。
|
||||
- 显式标注所有 stop 调用点;保留 protocol/exit 原始分类与脱敏 stderr。
|
||||
- 证明 intentional stop 不被 close 误判,protocol failure 不被 exit 覆盖。
|
||||
|
||||
产物:可解释 generation replacement 与下一次现场问题所需的最小证据。
|
||||
|
||||
#### 阶段 C:资源收敛
|
||||
|
||||
- 覆盖 extension、interaction、subagent、process budget 和 write lease 的 crash cleanup。
|
||||
- 复用现有 cleanup seam,不新增全局协调器。
|
||||
- 验证 cleanup 部分失败时仍继续释放其余资源。
|
||||
|
||||
产物:目标故障后所有活动/等待计数归零,其他 Conversation 继续运行。
|
||||
|
||||
#### 阶段 D:thinking 能力合同
|
||||
|
||||
- 修改 `runtime.ts`、`provider-config.ts`、managed catalog/Registry 相关代码。
|
||||
- 只持久化 effective level,unsupported level 不伪装成功。
|
||||
- 验证 thinking change 和同账号 model change 不产生 generation replacement。
|
||||
|
||||
产物:UI、Registry 与 Pi 实际状态一致。
|
||||
|
||||
#### 阶段 E:Host、Renderer 与安装包证明
|
||||
|
||||
- 覆盖 `electron/api/routes/coding-conversations.ts`、`src/stores/coding-conversations.ts`、`CodingChatPanel.tsx`、`CodingComposer.tsx`、`CodingConversationHeader.tsx` 的现有 seam。
|
||||
- 扩展 `release-proof.ts`、fake child、Windows Electron E2E 与 packaged proof。
|
||||
- 生成新的 Windows x64 NSIS,并返回路径、大小、SHA-256 和 clean candidate commit。
|
||||
|
||||
产物:最终安装包中可复现的强制退出、恢复和零残留证明。
|
||||
|
||||
### PI-R002 — Soak and incident classification follow-up
|
||||
|
||||
`PI-R002` 不与 `PI-R001` 并发实施,也不阻断当前修复包交付。它在 `PI-R001` 安装包完成后执行:
|
||||
|
||||
1. 以受控 loopback 连续运行 create/open/prompt/tool/abort/recover,并随机注入目标 worker exit。
|
||||
2. 在 4 parent + 4 child 压力中注入单点失败,确认隔离和计数收敛。
|
||||
3. 观察 normal path 的 generation replacement:每次都必须有已知 reason,未知 reason 为失败。
|
||||
4. 若用户真实环境再次发生退出,读取新的结构化诊断确定 initiating class,再决定是本地修复、上游 Pi issue 还是 Windows/Electron 原生诊断。
|
||||
|
||||
真实 Provider 因用户 waiver 不属于 `PI-R002` hard gate;这意味着产品只能宣称本地 runtime failure containment 已验证,不能宣称所有真实 Provider 行为已验证。
|
||||
|
||||
## 11. 验收矩阵
|
||||
|
||||
| ID | 场景 | 必须结果 |
|
||||
| --- | --- | --- |
|
||||
| `RES-ACC-001` | accepted running + unexpected worker exit | ≤2 秒投影 run error/failed/recoverable;Composer 解锁;应用存活 |
|
||||
| `RES-ACC-002` | worker exit 后点击 abort | 不恢复 running;最终仍为 failed/recoverable |
|
||||
| `RES-ACC-003` | abort 与 exit 交错 | 终态单调;资源只释放一次 |
|
||||
| `RES-ACC-004` | `agent_settled` 后 close | completed/aborted 不被改成 failed |
|
||||
| `RES-ACC-005` | crash 后旧 generation event | 被丢弃,不修改新 Snapshot |
|
||||
| `RES-ACC-006` | Conversation A crash、B running | B 正常 settled,B generation/run/draft 不变 |
|
||||
| `RES-ACC-007` | strict JSONL protocol violation | 保留 protocol error,记录 protocol stop reason,目标 worker 被清理 |
|
||||
| `RES-ACC-008` | deliberate idle eviction/app shutdown/recover | 各自记录正确 reason,不记为 unexpected crash |
|
||||
| `RES-ACC-009` | crash 时存在 interaction/child/write lease | 全部取消/释放;其他 Conversation/项目不受影响 |
|
||||
| `RES-ACC-010` | recover existing Conversation | 复用 session binding,hydrate 成功,不自动重发原 prompt |
|
||||
| `RES-ACC-011` | thinking level unsupported | 拒绝或返回实际 effective level,不持久化假值 |
|
||||
| `RES-ACC-012` | thinking 与同账号 model change | worker generation 不变化 |
|
||||
| `RES-ACC-013` | stderr 含 token/header/content 形状 | Main 诊断有界脱敏;Renderer/日志无敏感值 |
|
||||
| `RES-ACC-014` | final packaged fault injection | 真实 `app.asar` Main + packaged Pi 通过 crash/recover/isolation/cleanup |
|
||||
| `RES-ACC-015` | 应用退出 | 3 秒 graceful/强杀合同生效,最终本任务 Pi/Electron 残留进程为 0 |
|
||||
| `RES-ACC-016` | 正常 packaged flow | 无无原因 generation replacement;既有 cold/warm/acceptance 预算不回退 |
|
||||
|
||||
## 12. 测试与发布门禁
|
||||
|
||||
### 12.1 Focused tests
|
||||
|
||||
- `tests/unit/pi-conversation-runtime.test.ts`
|
||||
- `tests/unit/pi-worker-pool.test.ts`
|
||||
- `tests/unit/pi-worker-process-real.test.ts`
|
||||
- `tests/unit/pi-rpc-foundation.test.ts`
|
||||
- `tests/unit/pi-subagent.test.ts`
|
||||
- `tests/unit/pi-write-lease.test.ts`
|
||||
- `tests/unit/coding-conversations-store.test.tsx`
|
||||
- `tests/unit/coding-chat-panel.test.tsx`
|
||||
- 对应 Host route、release proof 和 managed model tests
|
||||
|
||||
每个行为变更必须先有能在旧实现失败的正确 seam 回归。不得只断言函数“没有抛错”;必须断言最终 Snapshot、generation、资源计数、日志分类和 prompt 未重放。
|
||||
|
||||
### 12.2 必跑命令
|
||||
|
||||
使用 `packageManager` 固定的 pnpm `10.33.4`:
|
||||
|
||||
```text
|
||||
corepack pnpm install --frozen-lockfile
|
||||
corepack pnpm run typecheck
|
||||
corepack pnpm run lint:check
|
||||
corepack pnpm test
|
||||
corepack pnpm run build:vite
|
||||
corepack pnpm run test:electron:windows
|
||||
corepack pnpm run package:win
|
||||
corepack pnpm run verify:publish-runtime
|
||||
corepack pnpm run verify:artifact:win
|
||||
corepack pnpm run verify:artifact:pi
|
||||
corepack pnpm run test:pi-subagent:packaged
|
||||
```
|
||||
|
||||
### 12.3 Packaged fault injection
|
||||
|
||||
最终 clean candidate 的 Windows 安装包必须覆盖:
|
||||
|
||||
1. running 中强制退出真实 packaged Pi worker;Main/Renderer 存活。
|
||||
2. abort-after-exit;不会回到 running。
|
||||
3. settled-before-close;completed 不被覆盖。
|
||||
4. strict framer protocol invalidation;错误分类不被 close 覆盖。
|
||||
5. deliberate stop;reason 与 crash 分类分离。
|
||||
6. 目标 recover;原 binding/hydration 保留,原 uncertain prompt 不重发。
|
||||
7. 两 Conversation 隔离;只杀一个 worker,另一个完成。
|
||||
8. parent/child/write lease/process budget 清理;所有 active/waiting 计数归零。
|
||||
9. thinking/model 变更不造成非预期 generation replacement。
|
||||
10. stderr redaction 与退出后零残留进程。
|
||||
|
||||
报告必须绑定:代码 commit、安装器路径、字节数、SHA-256、Electron/Node/Pi 版本、测试样本、`realTurnVerified=false` 和未完成平台边界。
|
||||
|
||||
## 13. 提交与合并批次
|
||||
|
||||
推荐批次如下,每批都可独立审阅:
|
||||
|
||||
1. `test: reproduce Pi worker exit lifecycle lock`:只加入 crash/abort/ordering red tests。
|
||||
2. `fix: terminalize Pi worker failures`:runtime/pool 最小状态机修复与资源收敛。
|
||||
3. `feat: classify Pi worker lifecycle exits`:stop reason、错误保真、脱敏 diagnostics。
|
||||
4. `fix: persist effective Pi thinking level`:capability/effective state 与 no-restart tests。
|
||||
5. `test: prove packaged Pi failure recovery`:Host/Renderer/E2E/release proof。
|
||||
6. `docs: record Pi resilience qualification`:任务证据、安装器信息和剩余风险。
|
||||
|
||||
合并前必须审阅累计 diff;不得把旧 hotfix worktree、诊断临时脚本、构建输出、安装器、用户数据或敏感日志提交进 Git。
|
||||
|
||||
## 14. 阻断与停止条件
|
||||
|
||||
出现以下任一情况,实施任务不得宣称完成:
|
||||
|
||||
- crash 后仍能得到 worker error + active run 的组合。
|
||||
- abort-after-exit 仍恢复 running/aborting。
|
||||
- accepted prompt 在 recover、SSE reconnect 或 worker replacement 中被再次发送。
|
||||
- cleanup 只在 happy path 归零,fault injection 后有 orphan process/permit/lease。
|
||||
- protocol error 被后续 close 覆盖成普通 exit。
|
||||
- 任一 generation replacement 缺少 reason。
|
||||
- thinking UI/Registry 与 Pi effective state 不一致。
|
||||
- packaged proof 只使用 repo-source composition 或 fake process,没有穿过最终 `app.asar` Main 与 packaged Pi。
|
||||
- 最终 Windows artifact/runtime closure、Electron E2E、build 或文档门禁失败。
|
||||
|
||||
若完成上述本地合同需要修改 Pi 上游包、引入全局 daemon 或改变真实 Provider waiver,实施者必须停止并回报,不得自行扩大架构方向。
|
||||
|
||||
## 15. 风险与发布口径
|
||||
|
||||
- `PI-R001` 通过后可以宣称:Windows 最终安装包已验证 Pi worker 故障隔离、Conversation 终态收敛、显式恢复、无自动重放、退出可观测和零残留。
|
||||
- 不得宣称:Pi 永不崩溃、真实 Provider 永不触发新协议问题、跨平台 release-ready。
|
||||
- 真实 Provider:`Explicitly Waived / Accepted Risk`,`realTurnVerified=false`,不是 Pass。
|
||||
- macOS x64/arm64、native non-WSL Linux:状态不变,不是本票据 Pass。
|
||||
- 新诊断上线后若再现问题,优先使用结构化 exit/stop evidence 定位;不要先增加重启次数、watchdog 或内容日志。
|
||||
@@ -0,0 +1,136 @@
|
||||
# Task: Plan Pi runtime resilience hardening
|
||||
|
||||
## Identity
|
||||
|
||||
- Task ID: 20260825-pi-runtime-resilience-plan-4b8e2c71
|
||||
- Mode: Feature
|
||||
- Branch: codex/20260825-pi-runtime-resilience-plan-4b8e2c71-pi-runtime-resilience-plan
|
||||
- Worktree: D:\Datas\OthersProjects\makelore-pi-runtime-resilience-plan-4b8e2c71
|
||||
- Base commit: 274187e3cfe2ae6cb97f2b4bcb422b3fdf06fbf2
|
||||
- Owner: codex-root
|
||||
- Status: Ready for Integration
|
||||
|
||||
## Scope
|
||||
|
||||
- Produce a task-owned, executable Pi runtime resilience hardening proposal
|
||||
based on the installed Windows worker-loss incident and the existing Pi
|
||||
cutover Spec.
|
||||
- Define lifecycle invariants, crash/abort/settled ordering, explicit restart
|
||||
policy, resource cleanup, session safety, thinking-level correctness,
|
||||
privacy-safe diagnostics, packaged fault injection, release gates, and the
|
||||
executor handoff.
|
||||
- Do not modify product code, the active executor worktree, canonical project
|
||||
memory, or the original 18-node / 31-edge cutover DAG.
|
||||
|
||||
## Intent And Constraints
|
||||
|
||||
- Treat process exit as unavoidable but make application survival, target-only
|
||||
failure convergence, no uncertain prompt replay, and explainable worker
|
||||
replacement mandatory product guarantees.
|
||||
- Keep the Pi hard cutover; do not introduce OpenCode fallback, a second
|
||||
runtime path, a global Pi daemon, generic Provider watchdog, feature flag, or
|
||||
speculative compatibility framework.
|
||||
- Preserve the original Conversation/session binding and generation isolation.
|
||||
Keep real external Provider validation as `Explicitly Waived / Accepted
|
||||
Risk`, `realTurnVerified=false`; do not present loopback evidence as Pass.
|
||||
- Follow repository-pinned pnpm, focused/full tests, `build:vite`, Windows
|
||||
Electron E2E, formal packaging, artifact verification, and final packaged
|
||||
fault injection.
|
||||
- Use no subagents, in accordance with the user's explicit direction and the
|
||||
repository policy.
|
||||
|
||||
## Project Context Loaded
|
||||
|
||||
- Concurrent Task Gate passed in the isolated worktree owned by `codex-root`;
|
||||
task ID, mode, branch, worktree, base commit, and task record match the local
|
||||
registry.
|
||||
- Planning Gate passed after reading the entry and memory indexes, active task,
|
||||
positioning/current-state/decision/architecture/domain/success/evidence/
|
||||
reflection/commitment/stale records, the Pi hard-cutover Spec and tickets,
|
||||
the installed stuck-processing diagnosis, the Windows proxy/session-reopen
|
||||
records, and active peer task scopes.
|
||||
- Base `274187e3cfe2ae6cb97f2b4bcb422b3fdf06fbf2` contains the cumulative Windows
|
||||
Pi proxy and persisted-session reopen fixes used by the installed package
|
||||
diagnosis.
|
||||
- Shared canonical architecture still contains superseded OpenCode-era text.
|
||||
This feature task records a future promotion candidate instead of modifying
|
||||
canonical files outside Integration Gate.
|
||||
- Executor task `20260825-pi-worker-exit-thinking-5e7c2a91` is already isolated
|
||||
on the same base and implements the exact current-fix portion of this plan.
|
||||
This task owns only its proposal/task record, so there is no code-path write
|
||||
conflict or competing architecture direction.
|
||||
- The initiating worker-exit class remains unknown. Eleven installed-runtime
|
||||
copied-session loopback cases settled, so ordinary tested Provider/SSE,
|
||||
session corruption, and `task_state` are not established causes.
|
||||
- Gate result: Passed.
|
||||
|
||||
## Plan
|
||||
|
||||
1. Consolidate confirmed incident evidence and the existing Pi Spec invariants
|
||||
without claiming an initiating cause that current artifacts cannot prove.
|
||||
2. Define the Main-owned lifecycle seam, terminal-state and generation
|
||||
invariants, explicit stop/replacement reasons, no-replay recovery policy,
|
||||
and bounded privacy-safe diagnostics.
|
||||
3. Define resource/session/thinking contracts, exact implementation phases,
|
||||
focused and final packaged fault-injection acceptance, and release blockers.
|
||||
4. Record `PI-R001` as the current corrective implementation task and defer
|
||||
soak/field classification to a dependent `PI-R002`, without changing the
|
||||
original cutover DAG.
|
||||
5. Run document drift/project-doc gates, commit only the owned proposal and
|
||||
task record, then send the fixed document and acceptance contract to the
|
||||
existing implementation thread.
|
||||
|
||||
## Outcome
|
||||
|
||||
- Added task-owned proposal
|
||||
`10-decisions/proposals/20260825-pi-runtime-resilience-plan-4b8e2c71__pi-runtime-resilience-hardening.md`.
|
||||
- The proposal defines the difference between an impossible “never exits”
|
||||
promise and enforceable product guarantees: Electron survival, bounded run
|
||||
terminalization, target isolation, explicit replacement reasons, no mutation
|
||||
replay, same-session recovery, resource convergence, effective thinking
|
||||
state, privacy-safe forensics, and final Windows packaged fault injection.
|
||||
- Assigned current implementation scope to corrective ticket `PI-R001` and the
|
||||
already-active executor task `20260825-pi-worker-exit-thinking-5e7c2a91`.
|
||||
Defined `PI-R002` as a later non-concurrent soak/field-classification task.
|
||||
- Kept the original 18-node / 31-edge cutover DAG unchanged and retained all
|
||||
existing Provider/platform risk labels.
|
||||
|
||||
## Verification
|
||||
|
||||
- Concurrent Task Gate and Planning Gate: passed.
|
||||
- Proposal cross-checked against Pi Spec sections 8, 12.2/12.3, 17, 18, 19,
|
||||
and 20, plus the current implementation/diagnosis records.
|
||||
- Product code and the executor worktree were not modified.
|
||||
- `check_doc_drift.py`: passed; only the task-owned proposal and task record
|
||||
are present.
|
||||
- `check_project_docs.py`: passed.
|
||||
- `git diff --check`: passed.
|
||||
|
||||
## Follow-ups
|
||||
|
||||
- Executor task `20260825-pi-worker-exit-thinking-5e7c2a91` implements and
|
||||
verifies `PI-R001`, produces the Windows NSIS, and returns commit/path/hash.
|
||||
- After `PI-R001` package acceptance, schedule `PI-R002` for repeated
|
||||
loopback/fault-injection soak and use the new lifecycle evidence if a real
|
||||
installed-user recurrence occurs.
|
||||
- Real external Provider validation remains explicitly waived accepted risk;
|
||||
macOS and native non-WSL Linux status is unchanged.
|
||||
|
||||
## Promotion Candidates
|
||||
|
||||
- Target canonical documents: Pi runtime release runbook, accepted Pi
|
||||
architecture/current-state, and future integration status.
|
||||
- Proposal: promote the invariants that worker failure must terminalize the
|
||||
matching run, every generation replacement has a bounded reason, accepted or
|
||||
uncertain mutation is never auto-replayed, and recover reopens the same
|
||||
session binding.
|
||||
- Evidence: installed-package diagnosis
|
||||
`20260825-pi-stuck-processing-diagnosis-7c4e91a2`, this proposal, and the
|
||||
future accepted `PI-R001` implementation/package evidence.
|
||||
- Future impact: all later Pi upgrades and Windows release candidates must keep
|
||||
the packaged failure-injection gate and explainable lifecycle diagnostics.
|
||||
- Semantic conflicts: canonical files still describe the superseded OpenCode
|
||||
runtime; Integration Gate must reconcile them with the completed Pi chain.
|
||||
- Human confirmation: not required for these already-authorized invariants;
|
||||
required if integration changes the existing real-Provider waiver or
|
||||
deferred platform release scope.
|
||||
Reference in New Issue
Block a user