feat: qualify Pi runtime cutover foundation
This commit is contained in:
@@ -0,0 +1,79 @@
|
||||
# Task: Implement PI-000 Pi runtime qualification gate
|
||||
|
||||
## Identity
|
||||
|
||||
- Task ID: 20260822-pi-runtime-qualification-c52e8a14
|
||||
- Mode: Feature
|
||||
- Branch: codex/20260822-pi-runtime-qualification-c52e8a14-pi-runtime-qualification
|
||||
- Worktree: D:\Datas\OthersProjects\makelore-pi-runtime-qualification-c52e8a14
|
||||
- Base commit: fba68e86d93c66d2c50f53d215de76a7c69c97a7
|
||||
- Owner: codex
|
||||
- Status: Completed — Done with explicit waivers
|
||||
|
||||
## Scope
|
||||
|
||||
- Implement the `PI-000 Runtime qualification gate` defined by the accepted Pi runtime cutover specification.
|
||||
- Pin the selected Pi runtime package exactly and keep the frozen lockfile reproducible.
|
||||
- Add a minimal, standalone RPC qualification probe and focused tests for runtime discovery, JSONL framing, session reopen, abort/settled handling, dependency closure, process cleanup, and measurement reporting.
|
||||
- Produce task-scoped evidence for the supported platform/artifact/provider samples that can actually be run from this worktree.
|
||||
- Do not change Chat, Host API routes, Renderer behavior, OpenCode migration paths, or final packaged runtime wiring in this task.
|
||||
|
||||
## Intent And Constraints
|
||||
|
||||
- Candidate runtime: `@earendil-works/pi-coding-agent@0.84.2`, exact version only. Its npm package, CLI entry, Node engine, upstream tag, and commit must agree before it is accepted.
|
||||
- The intended packaged seam is Electron's executable with `ELECTRON_RUN_AS_NODE=1` and Pi's RPC CLI; the gate must disprove this seam if packaging, concurrency, provider compatibility, resource discovery, or latency budgets fail.
|
||||
- Required protocol coverage is the current four-protocol Makelore contract: `openai-completions`, `openai-responses`, `anthropic-messages`, and `openrouter`, including base URL, headers, credential injection, model selection, and image-input capability where applicable.
|
||||
- Windows/Linux and credential-free samples establish the Phase-0 platform evidence. They do not validate real-provider compatibility, concurrency, abort isolation, credential/model isolation, or image-input behavior; the user explicitly waived those remaining `QG-004`/`QG-005` checks on 2026-08-22 and accepted the resulting risk.
|
||||
- The user explicitly waived macOS x64/arm64 from PI-000 on 2026-08-22 and accepted the risk of discovering macOS packaging/native/resource/performance defects later. This is a deferral to PI-150, not a macOS Pass or removal of final cross-platform release validation.
|
||||
- The real Provider waiver is not a Pass. Real authentication, endpoint/proxy/rate-limit/provider variation, base URL/header/environment credential/model/image compatibility, true provider concurrency or serialization, and cross-worker abort/event/session/model/credential isolation remain unverified.
|
||||
- No OpenCode fallback, compatibility layer, RPC/SDK dual path, speculative migration framework, or unrelated refactor is allowed.
|
||||
- If this gate fails, implementation stops at PI-000 and the planner must amend the specification/ADR to the SDK-in-Electron-utility-process alternative.
|
||||
|
||||
## Outcome
|
||||
|
||||
- Implemented the exact Pi dependency pin, frozen lockfile update, standalone qualification probe, temporary production-closure staging, controlled Electron Builder `app.asar` probe, real-provider fixture path, four-protocol loopback contract probe, and focused unit coverage.
|
||||
- Corrected the controlled packaged-closure inspector before macOS execution: the packaged resource directory is now computed once and passed into the artifact process, so a macOS executable reads `Contents/Resources/app.asar` instead of the invalid `Contents/MacOS/resources/app.asar` path.
|
||||
- Windows x64 and WSL2 Ubuntu 24.04 Linux x64 workspace, staged, and controlled packaged samples are a partial pass. Runtime identity, JSONL RPC, session persistence/reopen, deterministic failure settlement, local worker isolation, shell abort isolation, complete artifact dependency/resource resolution, startup budgets, RSS reporting, and clean exit were confirmed.
|
||||
- Windows and Linux packaged loopback samples passed request/SSE/image/header/credential/model contracts for `openai-completions`, `openai-responses`, `anthropic-messages`, and `openrouter`, including two-worker overlap and one-worker abort isolation. This is not real Provider Account evidence.
|
||||
- The planner audited every PI-000 Required work, Acceptance item, and stop condition after the two user decisions. No unwaived Phase-0 item remains incomplete and no confirmed failure exists, so PI-000 is **Done with explicit waivers**, not an all-gates Pass. The new Ready Frontier is `PI-010`, `PI-020`, and `PI-030`.
|
||||
|
||||
## Verification
|
||||
|
||||
- Project-docs preflight and concurrent/planning gates passed in this isolated feature worktree.
|
||||
- `@earendil-works/pi-coding-agent@0.84.2` npm metadata and upstream `v0.84.2` tag both resolve to commit `914cf1472e715297caa30db4b9535d534a9eb718`.
|
||||
- Electron `43.4.0` runs Node `24.18.1` through `ELECTRON_RUN_AS_NODE=1`, satisfying Pi's declared Node `>=22.19.0` engine floor.
|
||||
- `pnpm exec vitest run tests/unit/pi-runtime-probe.test.ts`: 9 tests passed, including macOS packaged-resource path coverage, aggregate-evidence wording, and explicit real-Provider waiver reporting.
|
||||
- `pnpm exec eslint scripts/probe-pi-runtime.mjs scripts/probe-pi-packaged-runtime.mjs scripts/probe-pi-provider-contracts.mjs tests/unit/pi-runtime-probe.test.ts`: passed.
|
||||
- `pnpm run typecheck`: passed.
|
||||
- `pnpm run build:vite`: passed; existing chunk-size and mixed static/dynamic import warnings remain unchanged and are outside PI-000.
|
||||
- `node scripts/probe-pi-runtime.mjs --stage --samples 5 --timeout-ms 10000`: partial pass on Windows x64. Cold ready p95 681 ms, warm ready p95 674 ms, RSS p95 125780 KiB, exit p95 16 ms; 130 applicable shrinkwrap packages present and 9 platform-specific entries skipped.
|
||||
- `node scripts/probe-pi-packaged-runtime.mjs --samples 5 --timeout-ms 10000`: partial pass on a controlled Windows x64 Electron Builder `app.asar` artifact. Cold ready p95 682 ms, warm ready p95 663 ms, RSS p95 141320 KiB, exit p95 18 ms; all 130 applicable packages and six runtime assets resolve from the artifact, with one same-version dependency flattened by Electron Builder.
|
||||
- `node scripts/probe-pi-provider-contracts.mjs ...packaged-runtime-paths...`: passed all four loopback HTTP/SSE contracts through the packaged executable. Provider-shaped overlap was 304/309/290/297 ms; every protocol passed image/header/credential/model/path checks and one-worker abort isolation. Scope was `127.0.0.1` only and no real Provider Account was contacted.
|
||||
- Frozen install in an isolated WSL2 Ubuntu 24.04 x64 snapshot used Node `22.22.2` and the exact repository-pinned `pnpm@10.33.4`; native dependencies installed successfully.
|
||||
- Linux workspace 5+5: cold ready p95 853 ms, warm ready p95 437 ms, RSS p95 170388 KiB, exit p95 8 ms.
|
||||
- Linux staged 5+5: cold ready p95 428 ms, warm ready p95 440 ms, RSS p95 175956 KiB, exit p95 8 ms; all 131 Linux-applicable packages present and 8 platform-specific entries skipped.
|
||||
- Linux controlled packaged 5+5: cold ready p95 545 ms, warm ready p95 569 ms, RSS p95 181928 KiB, exit p95 7 ms; all 131 applicable packages, six runtime assets, and unpacked native assets resolve from the artifact.
|
||||
- Linux packaged loopback: all four contracts passed with 292/290/277/288 ms overlap, correct path/header/credential/model/image serialization, and isolated abort behavior.
|
||||
- Post-fix `node scripts/probe-pi-packaged-runtime.mjs --samples 1 --timeout-ms 10000`: Windows controlled packaged smoke passed through the explicitly supplied resource directory; cold/warm ready were 650/641 ms and no dependency or asset was missing.
|
||||
- Post-Provider-waiver `node scripts/probe-pi-runtime.mjs --samples 1 --timeout-ms 10000`: passed with cold/warm ready 650/588 ms. All four `providerMatrix.realTurnVerified` values remained `false`; real Provider evidence moved out of `missingEvidence` into explicit `waivers`, and macOS remained deferred to PI-150. The per-run `decision` remains `incomplete` because one local sample cannot issue the aggregate PI-000 decision.
|
||||
- Post-probe process queries on Windows and Linux: zero controlled artifact processes and zero Pi RPC-looking processes remained.
|
||||
- Execution-environment inventory found no repository CI definition, macOS runner/SSH host, GitLab runner CLI, or Provider credential environment variable. The internal GitLab remote alone cannot produce the deferred macOS evidence or optional real-Provider diagnostics; neither absence remains a PI-000 blocker after the explicit user decisions.
|
||||
- The evidence gate table was reconciled with the accepted Spec's normative numbering: QG-002 RPC lifecycle, QG-003 dependency closure, QG-004 Provider Accounts, QG-005 real-provider concurrency, QG-006 metrics, and QG-007 stop-and-amend condition.
|
||||
- The planner synchronized the accepted Spec and ticket graph: 134 unique requirements across 19 traceability groups, 18 tickets, 31 edges, and an acyclic topology remain intact; recomputation produced exactly `PI-010`, `PI-020`, and `PI-030` as the new Ready Frontier.
|
||||
- Detailed evidence: `.project-docs/50-evidence/topics/20260822-pi-runtime-qualification-c52e8a14__pi-runtime-qualification.md`.
|
||||
|
||||
## Follow-ups
|
||||
|
||||
- Real Provider fixtures may still be run as optional diagnostics, but they are no longer PI-000 or PI-150 acceptance requirements. Never backfill QG-004/QG-005 as Pass without actual evidence, and do not place secrets in fixtures or task records.
|
||||
- At PI-150, run macOS x64 and macOS arm64 workspace/staged/final packaged samples, packaged loopback, closure/native/resource checks, and metrics before claiming cross-platform release readiness. Use separate frozen installs and artifacts for each architecture.
|
||||
- If Linux qualification must cover a native desktop distribution or release package rather than the accepted controlled `dir` boundary, repeat the Linux probe outside WSL2; the current sample is explicitly Ubuntu 24.04 userspace on a WSL2 kernel.
|
||||
- Any later confirmed runtime, packaging, concurrency, isolation, or performance failure is new evidence and must return to architecture review; this waiver records accepted uncertainty, not permission to misreport a known failure.
|
||||
|
||||
## Promotion Candidates
|
||||
|
||||
- Target: the planner's Pi runtime cutover ticket graph / ready frontier.
|
||||
Proposal: mark PI-000 Done with explicit waivers and advance the Ready Frontier to `PI-010`, `PI-020`, and `PI-030`; keep macOS explicitly deferred to PI-150 and QG-004/QG-005 explicitly waived rather than passed.
|
||||
Evidence: task-scoped PI-000 qualification evidence record.
|
||||
Future impact: downstream implementation proceeds while carrying explicit real-Provider and deferred macOS uncertainty; loopback and packaged evidence remain the proven boundary.
|
||||
Semantic conflicts: none after the planner's user-authorized Spec/ticket amendment.
|
||||
Human confirmation required: already supplied by the user's 2026-08-22 waiver decisions; a later confirmed failure requires a new architecture decision.
|
||||
@@ -0,0 +1,232 @@
|
||||
# PI-000 Pi Runtime Qualification Evidence
|
||||
|
||||
## Scope And Decision State
|
||||
|
||||
- Task: `20260822-pi-runtime-qualification-c52e8a14`
|
||||
- Candidate: `@earendil-works/pi-coding-agent@0.84.2`
|
||||
- Current decision: **PI-000 Done with explicit waivers**. This is not an all-gates Pass.
|
||||
- Ready frontier consequence: `PI-010`, `PI-020`, and `PI-030` are unlocked by the planner's audited 18-node/31-edge DAG.
|
||||
- macOS Phase-0 decision: **Explicitly Waived / Deferred by user on 2026-08-22**. This is not a Pass. macOS x64/arm64 qualification moves to `PI-150` final cross-platform release validation.
|
||||
- Real Provider decision: `QG-004` and `QG-005` are **Explicitly Waived / Accepted Risk by user on 2026-08-22**, not Pass. They no longer block PI-000 or PI-150.
|
||||
- Accepted unverified risk includes real authentication, endpoint/proxy/rate-limit/provider-specific response variation; base URL, header, environment credential, model and image compatibility; real provider concurrency/serialization; and cross-worker abort/event/session/model/credential isolation.
|
||||
- This record summarizes reproducible evidence. Raw generated JSON reports, temporary staged runtimes, and controlled directory artifacts are intentionally not committed.
|
||||
|
||||
## Confirmed Upstream Identity
|
||||
|
||||
- npm package version: `0.84.2`
|
||||
- npm CLI entry: `dist/cli.js`
|
||||
- declared Node engine: `>=22.19.0`
|
||||
- upstream tag: `v0.84.2`
|
||||
- upstream commit and npm `gitHead`: `914cf1472e715297caa30db4b9535d534a9eb718`
|
||||
- package pin in Makelore: exact `0.84.2`, without caret or tilde
|
||||
- project package manager used for the lockfile: the repository-pinned `pnpm@10.33.4`
|
||||
|
||||
The selected Electron `43.4.0` executable exposes Node `24.18.1` when launched with `ELECTRON_RUN_AS_NODE=1`, which satisfies Pi's declared engine floor.
|
||||
|
||||
## Windows x64 Qualification Sample
|
||||
|
||||
Environment:
|
||||
|
||||
- OS: Windows x64, release `10.0.26200`
|
||||
- Runtime seam: Electron executable plus `ELECTRON_RUN_AS_NODE=1`
|
||||
- Artifact: a temporary production-only staging directory installed from Pi's published `npm-shrinkwrap.json`
|
||||
- Startup samples: 5 cold and 5 warm
|
||||
- Cold definition: fresh Pi config, session, and project directories for each process
|
||||
- Warm definition: shared primed Pi config directory with fresh session and project directories for each process
|
||||
|
||||
Measured result:
|
||||
|
||||
| Measurement | Samples | p50 | p95 | max | Budget |
|
||||
|---|---:|---:|---:|---:|---:|
|
||||
| Cold `get_state` ready | 5 | 654 ms | 681 ms | 681 ms | 3000 ms |
|
||||
| Warm `get_state` ready | 5 | 661 ms | 674 ms | 674 ms | 1500 ms |
|
||||
| Worker RSS | 10 | 122780 KiB | 125780 KiB | 125780 KiB | report-only |
|
||||
| RPC process exit | 10 | 14 ms | 16 ms | 16 ms | deterministic |
|
||||
|
||||
All measured processes exited by closing stdin. No SIGTERM, force kill, or retained child was needed in the successful sample.
|
||||
|
||||
The deterministic provider-failure sample used a closed loopback endpoint, disabled automatic retry, and reached `prompt` accepted followed by `agent_settled` with an assistant `error` stop reason in 42 ms. This is below the 10-second deterministic-failure limit and does not claim provider success.
|
||||
|
||||
## Controlled Electron Builder Artifact Sample
|
||||
|
||||
PI-000 also built a Windows x64 Electron Builder `dir` artifact from the same temporary Pi production closure. This is a controlled packaged-boundary probe, not Makelore's final product artifact and not the permanent `PI-030`/`PI-150` bundler wiring. It deliberately does not inherit the product's OpenCode, Python, UV, signing, installer, or other `electron-builder.yml` hooks.
|
||||
|
||||
The packaged boundary was:
|
||||
|
||||
- executable: `MakelorePiProbe.exe`
|
||||
- archive: `resources/app.asar`
|
||||
- native extraction: `resources/app.asar.unpacked`
|
||||
- Pi CLI: `resources/app.asar/dist/cli.js`
|
||||
- process seam: packaged executable plus `ELECTRON_RUN_AS_NODE=1`
|
||||
|
||||
Artifact resolution confirmed Pi `0.84.2`, its CLI, all 130 Windows-applicable production packages, all six runtime WASM/native assets, and native `.node` extraction. The nine non-Windows lock entries were skipped by their declared platform constraints. Electron Builder flattened `@types/retry@0.12.0` from `p-retry/node_modules/@types/retry` to the artifact root `node_modules/@types/retry`; the verifier checks the same version at the resolved artifact location instead of requiring npm's original physical layout.
|
||||
|
||||
Measured result from 5 cold and 5 warm packaged samples:
|
||||
|
||||
| Measurement | Samples | p50 | p95 | max | Budget |
|
||||
|---|---:|---:|---:|---:|---:|
|
||||
| Cold `get_state` ready | 5 | 668 ms | 682 ms | 682 ms | 3000 ms |
|
||||
| Warm `get_state` ready | 5 | 660 ms | 663 ms | 663 ms | 1500 ms |
|
||||
| Worker RSS | 10 | 138404 KiB | 141320 KiB | 141320 KiB | report-only |
|
||||
| RPC process exit | 10 | 16 ms | 18 ms | 18 ms | deterministic |
|
||||
|
||||
The packaged session, deterministic failure settlement, reopen, local shell overlap, one-worker shell abort isolation, and stdin-close exit checks also passed. The deterministic failure settled in 36 ms; the overlapping local shell sample completed in 2098 ms. A post-run process query found zero controlled artifact processes and zero Pi RPC-looking processes.
|
||||
|
||||
## Linux x64 Qualification Sample
|
||||
|
||||
The Linux run used an isolated repository snapshot under WSL2 Ubuntu 24.04 x64. The snapshot installed the frozen lockfile with Node `22.22.2` and the exact repository-pinned `pnpm@10.33.4`; Pi then ran through Electron `43.4.0` and its embedded Node `24.18.1`. This exercises the Linux executable, Linux native dependency selection, asar layout, and process behavior. It is not evidence for a native compositor, AppImage/deb installer, or a non-WSL Linux kernel.
|
||||
|
||||
Workspace 5+5 result:
|
||||
|
||||
| Measurement | Samples | p50 | p95 | max | Budget |
|
||||
|---|---:|---:|---:|---:|---:|
|
||||
| Cold `get_state` ready | 5 | 426 ms | 853 ms | 853 ms | 3000 ms |
|
||||
| Warm `get_state` ready | 5 | 432 ms | 437 ms | 437 ms | 1500 ms |
|
||||
| Worker RSS | 10 | 168228 KiB | 170388 KiB | 170388 KiB | report-only |
|
||||
| RPC process exit | 10 | 7 ms | 8 ms | 8 ms | deterministic |
|
||||
|
||||
Temporary production-closure 5+5 result:
|
||||
|
||||
| Measurement | Samples | p50 | p95 | max | Budget |
|
||||
|---|---:|---:|---:|---:|---:|
|
||||
| Cold `get_state` ready | 5 | 426 ms | 428 ms | 428 ms | 3000 ms |
|
||||
| Warm `get_state` ready | 5 | 423 ms | 440 ms | 440 ms | 1500 ms |
|
||||
| Worker RSS | 10 | 170380 KiB | 175956 KiB | 175956 KiB | report-only |
|
||||
| RPC process exit | 10 | 7 ms | 8 ms | 8 ms | deterministic |
|
||||
|
||||
Controlled Electron Builder `dir`/`app.asar` 5+5 result:
|
||||
|
||||
| Measurement | Samples | p50 | p95 | max | Budget |
|
||||
|---|---:|---:|---:|---:|---:|
|
||||
| Cold `get_state` ready | 5 | 526 ms | 545 ms | 545 ms | 3000 ms |
|
||||
| Warm `get_state` ready | 5 | 545 ms | 569 ms | 569 ms | 1500 ms |
|
||||
| Worker RSS | 10 | 181412 KiB | 181928 KiB | 181928 KiB | report-only |
|
||||
| RPC process exit | 10 | 7 ms | 7 ms | 7 ms | deterministic |
|
||||
|
||||
The Linux shrinkwrap has 131 applicable packages and 8 OS/CPU-skipped entries. The controlled artifact resolved all applicable packages, the Pi CLI, all six tracked runtime assets, and every native asset that must live under `app.asar.unpacked`; no package or asset was missing. Electron Builder applied the same-version `@types/retry@0.12.0` flattening seen on Windows. Session persistence/reopen, deterministic failure settlement, local overlap, shell abort isolation, and stdin-close cleanup passed in all three boundaries. A final process listing contained only WSL init and the listing command itself—no packaged Pi or RPC worker remained.
|
||||
|
||||
## Runtime And Session Behaviors Confirmed On Windows
|
||||
|
||||
- Strict LF-only JSONL framing, including payloads containing U+2028 and U+2029.
|
||||
- `get_state` response and request id correlation.
|
||||
- Persistent session file creation after a settled assistant turn.
|
||||
- Reopen by session file with the same stable session id.
|
||||
- Session history remained readable after reopen; the sample contained six entries.
|
||||
- Direct RPC shell execution returned the expected output.
|
||||
- Idle `abort` was accepted.
|
||||
- Two workers had distinct session ids.
|
||||
- Two direct shell operations overlapped; total elapsed time was 2112 ms for a 4-second left command and 2-second right command.
|
||||
- `abort_bash` cancelled the left worker without cancelling or corrupting the right worker.
|
||||
- Resource discovery was explicitly disabled for the qualification sample with `--no-extensions`, `--no-skills`, `--no-prompt-templates`, `--no-themes`, `--no-context-files`, and `--no-approve`; startup ran with `--offline` and `PI_TELEMETRY=0`.
|
||||
|
||||
The two-worker result proves local process/session/shell isolation only. It is not evidence of two overlapping real-provider turns.
|
||||
|
||||
## Production Closure Evidence
|
||||
|
||||
Pi's published production shrinkwrap contains 139 dependency package entries:
|
||||
|
||||
- 130 entries are applicable and present on Windows x64.
|
||||
- 9 entries are correctly skipped by OS/CPU constraints.
|
||||
- Required image-processing asset is present: `@silvia-odwyer/photon-node/photon_rs_bg.wasm`.
|
||||
- Applicable clipboard native binary is present: `@mariozechner/clipboard-win32-x64-msvc`.
|
||||
- Pi TUI Windows x64/arm64 prebuilds and its packaged Darwin prebuilds are present in the staged closure.
|
||||
|
||||
Published-package caveat: the shrinkwrap is production-only, but the published package still lists ten development dependencies. npm 11 validates those development entries before applying `--omit=dev`, so an unmodified `npm ci --omit=dev` rejects the package as out of sync. The qualification staging copy removes `devDependencies` from only the temporary copied package root and then installs the unchanged published shrinkwrap. This must be considered when `PI-030` defines the permanent staging mechanism.
|
||||
|
||||
## Provider Contract Matrix
|
||||
|
||||
Static Pi contract coverage and a packaged-runtime loopback HTTP/SSE contract sample are confirmed for all current Makelore protocol identifiers:
|
||||
|
||||
| Makelore protocol | Pi configuration | Loopback endpoint | Windows / Linux overlap | Real turn |
|
||||
|---|---|---|---:|---|
|
||||
| `openai-completions` | `api: openai-completions` | `/v1/chat/completions` | 304 / 292 ms | not run — waived/unverified |
|
||||
| `openai-responses` | `api: openai-responses` | `/v1/responses` | 309 / 290 ms | not run — waived/unverified |
|
||||
| `anthropic-messages` | `api: anthropic-messages` | `/v1/messages` | 290 / 277 ms | not run — waived/unverified |
|
||||
| `openrouter` | `api: openai-completions` plus `thinkingFormat/sessionAffinityFormat: openrouter` | `/v1/chat/completions` | 297 / 288 ms | not run — waived/unverified |
|
||||
|
||||
For each protocol, the packaged runtime sent four POST requests through `127.0.0.1`, used the selected model, included the custom header, injected the environment-only credential into the protocol-appropriate header (`Authorization` or `x-api-key`), and carried the PNG image in request history. Two independent workers settled successfully with `stop`, their provider-shaped turns overlapped, aborting the next turn in one worker produced `aborted`, and the unaffected worker still settled with `stop`. All workers exited by closing stdin.
|
||||
|
||||
The Anthropic contract has a meaningful base URL difference: its SDK appends `/v1/messages`, so the configured base URL is the provider root. The OpenAI-style clients receive a base URL ending in `/v1` and append `chat/completions` or `responses`.
|
||||
|
||||
This loopback result proves Pi's request serialization, SSE parsing, image/header/model/credential plumbing, worker overlap, and abort isolation without external cost. It does **not** qualify any real Provider Account, provider authentication, proxy behavior, rate limits, or provider-specific response variation.
|
||||
|
||||
Pi's custom provider contract supports base URL, environment-resolved API key, custom headers, model id, and text/image input declaration. The probe accepts a credential-free fixture file shaped as follows; the secret remains only in the named environment variable:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "qualification-provider",
|
||||
"apiProtocol": "openai-responses",
|
||||
"baseUrl": "https://provider.example/v1",
|
||||
"apiKeyEnv": "QUALIFICATION_PROVIDER_KEY",
|
||||
"headers": { "X-Provider-Header": "literal-or-$ENV_REFERENCE" },
|
||||
"model": {
|
||||
"id": "provider-model-id",
|
||||
"input": ["text", "image"],
|
||||
"contextWindow": 128000,
|
||||
"maxTokens": 8192
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Run a real sample with:
|
||||
|
||||
```text
|
||||
pnpm run probe:pi-runtime -- --provider-fixture <fixture.json> [--image <image-path>]
|
||||
```
|
||||
|
||||
The real-provider path is implemented to require two successful overlapping turns with distinct sessions, then abort a new turn in one worker while the other worker settles successfully. It reports only provider/model ids, environment-variable name, and header names; it does not print credential values.
|
||||
|
||||
No recognized provider credential environment variable was available in the implementation environment, so no real-provider request was sent and no provider cost was incurred.
|
||||
|
||||
Reproduce the local contract sample against an existing controlled artifact with:
|
||||
|
||||
```text
|
||||
pnpm run probe:pi-provider-contracts -- --electron-executable <artifact-executable> --cli-path <artifact-app.asar-cli>
|
||||
```
|
||||
|
||||
## Deferred macOS Execution Handoff For PI-150
|
||||
|
||||
A pre-execution portability review found and fixed one supported-path defect: the artifact-side closure inspector derived `resources` from `dirname(process.execPath)`, which is correct on Windows/Linux but points to `Contents/MacOS/resources` on macOS. The probe now computes the platform resource directory once and passes it into the packaged process, preserving the actual macOS location `Contents/Resources/app.asar`. A focused path test and a real Windows controlled packaged regression pass verify the shared logic. The user explicitly deferred actual macOS execution from PI-000 to PI-150 on 2026-08-22, so no macOS Pass is claimed and the late-discovery risk remains.
|
||||
|
||||
Use a separate checkout and frozen install for each target architecture. Before installing, `node -p "process.platform + ' ' + process.arch + ' ' + process.version"` must report `darwin x64` or `darwin arm64` for the sample being collected. Do not reuse one architecture's `node_modules` or Electron download for the other architecture.
|
||||
|
||||
Run this sequence in each architecture-specific checkout:
|
||||
|
||||
```text
|
||||
pnpm install --frozen-lockfile
|
||||
node scripts/probe-pi-runtime.mjs --samples 5 --timeout-ms 10000 --report release/pi-runtime-macos-workspace.json
|
||||
pnpm run probe:pi-runtime -- --samples 5 --timeout-ms 10000 --report release/pi-runtime-macos-staged.json
|
||||
pnpm run probe:pi-runtime:packaged -- --samples 5 --timeout-ms 10000 --report release/pi-runtime-macos-packaged.json
|
||||
```
|
||||
|
||||
Read `artifact.executable` and `artifact.appAsar` from the packaged JSON report. The Pi CLI path is `<artifact.appAsar>/dist/cli.js`. Run the credential-free packaged four-protocol contract sample before replacing the controlled artifact:
|
||||
|
||||
```text
|
||||
pnpm run probe:pi-provider-contracts -- --electron-executable <artifact.executable> --cli-path <artifact.appAsar>/dist/cli.js --artifact-label controlled-electron-builder-dir-app-asar --report release/pi-provider-contract-macos.json
|
||||
```
|
||||
|
||||
For an optional, non-gating real Provider diagnostic, create one credential-free fixture per protocol being sampled and keep each secret only in the environment variable named by that fixture's `apiKeyEnv`. Reuse the existing packaged artifact through the runtime override so the run exercises the packaged Electron seam without rebuilding it for every protocol:
|
||||
|
||||
```text
|
||||
node scripts/probe-pi-runtime.mjs --samples 5 --timeout-ms 10000 --electron-executable <artifact.executable> --cli-path <artifact.appAsar>/dist/cli.js --artifact-label controlled-electron-builder-dir-app-asar --provider-fixture <protocol-fixture.json> --image <qualification-image> --report release/pi-provider-<protocol>-macos.json
|
||||
```
|
||||
|
||||
The user waived requiring these commands for `openai-completions`, `openai-responses`, `anthropic-messages`, and `openrouter`. If an optional diagnostic is run, preserve only redacted metrics and protocol/provider/model identifiers; do not commit fixtures containing secrets, generated reports, controlled artifacts, caches, or user data. A final process query must show no packaged probe or Pi RPC worker remains.
|
||||
|
||||
## Gate Status
|
||||
|
||||
| Gate | Status | Evidence or blocker |
|
||||
|---|---|---|
|
||||
| QG-001 exact identity and pin | Pass | exact npm/tag/commit/engine/CLI and lockfile pin |
|
||||
| QG-002 packaged Electron Node RPC lifecycle | Pass | Phase-0 scope is Windows x64 + Linux x64 by explicit user decision; both controlled artifacts spawn, become ready, open session, accept prompt, abort, settle, reopen, and exit cleanly. macOS is deferred to PI-150, not passed. |
|
||||
| QG-003 production dependency closure | Pass | Phase-0 controlled Windows/Linux artifacts resolve all 130/131 applicable packages and six runtime assets. Final product artifacts and deferred macOS closure belong to PI-150. |
|
||||
| QG-004 Provider Account matrix | Explicitly Waived / Accepted Risk | four-protocol loopback request/SSE/image contracts pass; real accounts, credentials, endpoints, proxies, authentication, rate-limit and provider-specific behavior remain unverified by user decision on 2026-08-22 |
|
||||
| QG-005 real-provider worker concurrency | Explicitly Waived / Accepted Risk | local shell and loopback protocol turns overlap with isolated abort/model/credential state; real provider overlap/serialization and abort/event/session/model/credential isolation remain unverified by user decision on 2026-08-22 |
|
||||
| QG-006 performance and cleanup metrics | Pass | Phase-0 Windows/Linux cold/warm/RSS/exit sample counts and p50/p95/max are recorded and within budgets. macOS metrics are explicitly deferred to PI-150. |
|
||||
| QG-007 stop-and-amend condition | Pending / not triggered | no confirmed packaging, Provider, or budget failure yet; any mandatory-gate failure stops migration and requires the specified utility-process SDK amendment |
|
||||
|
||||
## Remaining And Optional Evidence
|
||||
|
||||
1. Real Provider fixtures are optional diagnostics after the explicit waiver. Their absence does not block PI-000 or PI-150, but the current record must continue to report every real turn as unverified; a later confirmed failure is new architecture evidence, not something the waiver converts into a Pass.
|
||||
2. At PI-150, run the deferred macOS x64/arm64 workspace, staged, final packaged, loopback, closure/native/resource, and metrics matrix. Missing either architecture blocks a cross-platform release-ready claim even though it no longer blocks PI-000.
|
||||
3. If the accepted Linux definition requires a native desktop distribution or release package, repeat the Linux sample outside WSL2 at PI-150; the current Phase-0 evidence is deliberately labeled as Ubuntu 24.04 userspace on WSL2 with a controlled `dir` artifact.
|
||||
@@ -37,6 +37,9 @@
|
||||
"build": "pnpm run build:vite && node scripts/bundle-opencode.mjs && node scripts/run-electron-builder.mjs",
|
||||
"build:vite": "node --max-old-space-size=6144 ./node_modules/vite/bin/vite.js build",
|
||||
"perf:budget": "node scripts/check-performance-budget.mjs",
|
||||
"probe:pi-runtime": "node scripts/probe-pi-runtime.mjs --stage",
|
||||
"probe:pi-runtime:packaged": "node scripts/probe-pi-packaged-runtime.mjs",
|
||||
"probe:pi-provider-contracts": "node scripts/probe-pi-provider-contracts.mjs",
|
||||
"bundle:opencode": "node scripts/bundle-opencode.mjs",
|
||||
"lint": "eslint . --fix",
|
||||
"lint:check": "eslint .",
|
||||
@@ -84,6 +87,7 @@
|
||||
"dependencies": {
|
||||
"@napi-rs/canvas": "0.1.100",
|
||||
"@monaco-editor/react": "^4.7.0",
|
||||
"@earendil-works/pi-coding-agent": "0.84.2",
|
||||
"@playwright/mcp": "0.0.78",
|
||||
"adm-zip": "0.5.16",
|
||||
"diff": "^9.0.0",
|
||||
|
||||
972
pnpm-lock.yaml
generated
972
pnpm-lock.yaml
generated
File diff suppressed because it is too large
Load Diff
322
scripts/probe-pi-packaged-runtime.mjs
Normal file
322
scripts/probe-pi-packaged-runtime.mjs
Normal file
@@ -0,0 +1,322 @@
|
||||
import { spawn } from 'node:child_process';
|
||||
import { mkdtemp, mkdir, readFile, readdir, rm, stat, writeFile } from 'node:fs/promises';
|
||||
import { arch, platform, tmpdir } from 'node:os';
|
||||
import { dirname, join, relative, resolve, sep } from 'node:path';
|
||||
import { pathToFileURL } from 'node:url';
|
||||
import { Arch, Platform, build } from 'electron-builder';
|
||||
import {
|
||||
parseProbeArgs,
|
||||
resolvePiRuntime,
|
||||
runProbe,
|
||||
stagePiRuntime,
|
||||
validatePiIdentity,
|
||||
validateStagedClosure,
|
||||
} from './probe-pi-runtime.mjs';
|
||||
|
||||
const PRODUCT_NAME = 'MakelorePiProbe';
|
||||
const ARTIFACT_LABEL = 'controlled-electron-builder-dir-app-asar';
|
||||
|
||||
async function pathExists(path) {
|
||||
try {
|
||||
await stat(path);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
function currentTarget() {
|
||||
let platformTarget;
|
||||
if (platform() === 'win32') platformTarget = Platform.WINDOWS;
|
||||
else if (platform() === 'darwin') platformTarget = Platform.MAC;
|
||||
else if (platform() === 'linux') platformTarget = Platform.LINUX;
|
||||
else throw new Error(`Unsupported packaging platform: ${platform()}`);
|
||||
const architecture = Arch[arch()];
|
||||
if (architecture == null) throw new Error(`Unsupported packaging architecture: ${arch()}`);
|
||||
return { platformTarget, architecture };
|
||||
}
|
||||
|
||||
async function findPackagedExecutable(root) {
|
||||
const matches = [];
|
||||
const visit = async (directory) => {
|
||||
for (const entry of await readdir(directory, { withFileTypes: true })) {
|
||||
const path = join(directory, entry.name);
|
||||
if (entry.isDirectory()) await visit(path);
|
||||
else if (
|
||||
(platform() === 'win32' && entry.name === `${PRODUCT_NAME}.exe`)
|
||||
|| (platform() === 'darwin'
|
||||
&& entry.name === PRODUCT_NAME
|
||||
&& directory.endsWith(`${sep}Contents${sep}MacOS`))
|
||||
|| (platform() === 'linux' && entry.name === PRODUCT_NAME)
|
||||
) matches.push(path);
|
||||
}
|
||||
};
|
||||
await visit(root);
|
||||
if (matches.length !== 1) {
|
||||
throw new Error(`Expected one packaged ${PRODUCT_NAME} executable, found ${matches.length}`);
|
||||
}
|
||||
return matches[0];
|
||||
}
|
||||
|
||||
export function packagedResourcesDirectory(executable, currentPlatform = platform()) {
|
||||
return currentPlatform === 'darwin'
|
||||
? resolve(executable, '..', '..', 'Resources')
|
||||
: join(dirname(executable), 'resources');
|
||||
}
|
||||
|
||||
function runCommand(executable, args, options = {}) {
|
||||
return new Promise((resolvePromise, reject) => {
|
||||
const child = spawn(executable, args, {
|
||||
cwd: options.cwd,
|
||||
env: options.env ?? process.env,
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
windowsHide: true,
|
||||
});
|
||||
let stdout = '';
|
||||
let stderr = '';
|
||||
child.stdout.on('data', (chunk) => { stdout += chunk.toString(); });
|
||||
child.stderr.on('data', (chunk) => { stderr += chunk.toString(); });
|
||||
child.once('error', reject);
|
||||
child.once('exit', (code, signal) => {
|
||||
if (code === 0) resolvePromise({ stdout, stderr });
|
||||
else reject(new Error(
|
||||
`${executable} exited with code ${code ?? 'null'} signal ${signal ?? 'none'}: ${stderr || stdout}`,
|
||||
));
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
async function inspectPackagedClosure(executable, resourcesDirectory, assets) {
|
||||
const script = String.raw`
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const resources = process.env.PI_PROBE_RESOURCES_DIRECTORY;
|
||||
if (!resources) throw new Error('PI_PROBE_RESOURCES_DIRECTORY is required');
|
||||
const root = path.join(resources, 'app.asar');
|
||||
const unpackedRoot = path.join(resources, 'app.asar.unpacked');
|
||||
const packageJson = JSON.parse(fs.readFileSync(path.join(root, 'package.json'), 'utf8'));
|
||||
const shrinkwrap = JSON.parse(fs.readFileSync(path.join(root, 'npm-shrinkwrap.json'), 'utf8'));
|
||||
const assets = JSON.parse(process.env.PI_PROBE_ASSETS_JSON);
|
||||
const libc = process.platform === 'linux'
|
||||
? (process.report?.getReport?.().header?.glibcVersionRuntime ? 'glibc' : 'musl')
|
||||
: undefined;
|
||||
const constraintMatches = (constraints, current) => {
|
||||
if (!constraints || constraints.length === 0 || !current) return true;
|
||||
const positive = constraints.filter((value) => !value.startsWith('!'));
|
||||
const negative = constraints.filter((value) => value.startsWith('!')).map((value) => value.slice(1));
|
||||
return !negative.includes(current) && (positive.length === 0 || positive.includes(current));
|
||||
};
|
||||
const supportsCurrentPlatform = (entry) => constraintMatches(entry.os, process.platform)
|
||||
&& constraintMatches(entry.cpu, process.arch)
|
||||
&& constraintMatches(entry.libc, libc);
|
||||
const missingPackages = [];
|
||||
const relocatedPackages = [];
|
||||
let expectedPackages = 0;
|
||||
let platformSkippedPackages = 0;
|
||||
for (const [packagePath, entry] of Object.entries(shrinkwrap.packages || {})) {
|
||||
if (!packagePath || entry.dev) continue;
|
||||
if (!supportsCurrentPlatform(entry)) {
|
||||
platformSkippedPackages += 1;
|
||||
continue;
|
||||
}
|
||||
expectedPackages += 1;
|
||||
const exactPath = path.join(root, ...packagePath.split('/'));
|
||||
if (fs.existsSync(exactPath)) continue;
|
||||
const nestedMarker = '/node_modules/';
|
||||
const packageTail = packagePath.includes(nestedMarker)
|
||||
? packagePath.slice(packagePath.lastIndexOf(nestedMarker) + nestedMarker.length)
|
||||
: packagePath.slice('node_modules/'.length);
|
||||
const nameParts = packageTail.split('/');
|
||||
const packageName = nameParts[0].startsWith('@') ? nameParts.slice(0, 2).join('/') : nameParts[0];
|
||||
const flattenedPath = path.join(root, 'node_modules', ...packageName.split('/'));
|
||||
const flattenedPackageJson = path.join(flattenedPath, 'package.json');
|
||||
if (fs.existsSync(flattenedPackageJson)) {
|
||||
const flattenedVersion = JSON.parse(fs.readFileSync(flattenedPackageJson, 'utf8')).version;
|
||||
if (flattenedVersion === entry.version) {
|
||||
relocatedPackages.push({ from: packagePath, to: 'node_modules/' + packageName, version: entry.version });
|
||||
continue;
|
||||
}
|
||||
}
|
||||
missingPackages.push(packagePath);
|
||||
}
|
||||
const missingAssets = assets.filter((asset) => !fs.existsSync(path.join(root, ...asset.split('/'))));
|
||||
const missingUnpackedNativeAssets = assets
|
||||
.filter((asset) => asset.endsWith('.node'))
|
||||
.filter((asset) => !fs.existsSync(path.join(unpackedRoot, ...asset.split('/'))));
|
||||
process.stdout.write(JSON.stringify({
|
||||
packageName: packageJson.name,
|
||||
packageVersion: packageJson.version,
|
||||
cliExists: fs.existsSync(path.join(root, 'dist', 'cli.js')),
|
||||
lockedPackages: Math.max(0, Object.keys(shrinkwrap.packages || {}).length - 1),
|
||||
expectedPackages,
|
||||
platformSkippedPackages,
|
||||
missingPackages,
|
||||
relocatedPackages,
|
||||
assetCount: assets.length,
|
||||
missingAssets,
|
||||
missingUnpackedNativeAssets,
|
||||
}));
|
||||
`;
|
||||
const { stdout } = await runCommand(executable, ['-e', script], {
|
||||
env: {
|
||||
...process.env,
|
||||
ELECTRON_RUN_AS_NODE: '1',
|
||||
PI_PROBE_RESOURCES_DIRECTORY: resourcesDirectory,
|
||||
PI_PROBE_ASSETS_JSON: JSON.stringify(assets),
|
||||
},
|
||||
});
|
||||
const result = JSON.parse(stdout.trim());
|
||||
if (
|
||||
!result.cliExists
|
||||
|| result.missingPackages.length > 0
|
||||
|| result.missingAssets.length > 0
|
||||
|| result.missingUnpackedNativeAssets.length > 0
|
||||
) {
|
||||
throw new Error(`Packaged Pi closure is incomplete: ${JSON.stringify(result)}`);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
function assertControlledOutput(projectRoot, outputDirectory) {
|
||||
const relativePath = relative(projectRoot, outputDirectory);
|
||||
if (!relativePath || relativePath.startsWith('..') || resolve(projectRoot, relativePath) !== outputDirectory) {
|
||||
throw new Error(`Refusing uncontrolled packaged-probe output: ${outputDirectory}`);
|
||||
}
|
||||
if (relativePath.split(sep).join('/') !== 'release/pi-runtime-probe') {
|
||||
throw new Error(`Unexpected packaged-probe output directory: ${outputDirectory}`);
|
||||
}
|
||||
}
|
||||
|
||||
async function buildControlledArtifact(projectRoot, appDirectory, outputDirectory) {
|
||||
const electronPackage = JSON.parse(await readFile(join(projectRoot, 'node_modules', 'electron', 'package.json'), 'utf8'));
|
||||
const { platformTarget, architecture } = currentTarget();
|
||||
const buildResources = join(appDirectory, 'build-resources');
|
||||
await mkdir(buildResources);
|
||||
const builderArtifacts = await build({
|
||||
// Use the staged Pi root as the project root so this controlled probe does
|
||||
// not inherit Makelore's production hooks or unrelated extraResources.
|
||||
projectDir: appDirectory,
|
||||
targets: platformTarget.createTarget('dir', architecture),
|
||||
publish: 'never',
|
||||
config: {
|
||||
appId: 'app.niancode.desktop.pi-probe',
|
||||
productName: PRODUCT_NAME,
|
||||
electronVersion: electronPackage.version,
|
||||
electronDist: join(projectRoot, 'node_modules', 'electron', 'dist'),
|
||||
directories: {
|
||||
output: outputDirectory,
|
||||
buildResources,
|
||||
},
|
||||
files: [
|
||||
'dist/**/*',
|
||||
'package.json',
|
||||
'npm-shrinkwrap.json',
|
||||
'node_modules/**/*',
|
||||
],
|
||||
asar: true,
|
||||
asarUnpack: ['**/*.node'],
|
||||
npmRebuild: false,
|
||||
win: {
|
||||
executableName: PRODUCT_NAME,
|
||||
signAndEditExecutable: false,
|
||||
verifyUpdateCodeSignature: false,
|
||||
},
|
||||
mac: {
|
||||
identity: null,
|
||||
hardenedRuntime: false,
|
||||
},
|
||||
linux: {
|
||||
executableName: PRODUCT_NAME,
|
||||
},
|
||||
},
|
||||
});
|
||||
return builderArtifacts;
|
||||
}
|
||||
|
||||
export async function runPackagedProbe(options, projectRoot = process.cwd()) {
|
||||
const resolvedProjectRoot = resolve(projectRoot);
|
||||
const outputDirectory = resolve(resolvedProjectRoot, 'release', 'pi-runtime-probe');
|
||||
assertControlledOutput(resolvedProjectRoot, outputDirectory);
|
||||
const scratchRoot = await mkdtemp(join(tmpdir(), 'makelore-pi-packaged-probe-'));
|
||||
const appDirectory = join(scratchRoot, 'app');
|
||||
try {
|
||||
const sourceRuntime = await resolvePiRuntime(resolvedProjectRoot);
|
||||
const identity = await validatePiIdentity(sourceRuntime);
|
||||
await stagePiRuntime(sourceRuntime, appDirectory);
|
||||
const stagedClosure = await validateStagedClosure(appDirectory);
|
||||
await rm(outputDirectory, { recursive: true, force: true, maxRetries: 5, retryDelay: 200 });
|
||||
await mkdir(dirname(outputDirectory), { recursive: true });
|
||||
const builderArtifacts = await buildControlledArtifact(
|
||||
resolvedProjectRoot,
|
||||
appDirectory,
|
||||
outputDirectory,
|
||||
);
|
||||
const executable = await findPackagedExecutable(outputDirectory);
|
||||
const resourcesDirectory = packagedResourcesDirectory(executable);
|
||||
const appAsar = join(resourcesDirectory, 'app.asar');
|
||||
const cliPath = join(appAsar, 'dist', 'cli.js');
|
||||
if (!await pathExists(appAsar)) throw new Error(`Packaged app.asar is missing: ${appAsar}`);
|
||||
const packagedClosure = await inspectPackagedClosure(
|
||||
executable,
|
||||
resourcesDirectory,
|
||||
stagedClosure.assets,
|
||||
);
|
||||
const runtime = await runProbe({
|
||||
...options,
|
||||
reportPath: undefined,
|
||||
stage: false,
|
||||
keepStage: false,
|
||||
electronExecutablePath: executable,
|
||||
cliPath,
|
||||
artifactLabel: ARTIFACT_LABEL,
|
||||
}, resolvedProjectRoot);
|
||||
const report = {
|
||||
schemaVersion: 1,
|
||||
generatedAt: new Date().toISOString(),
|
||||
identity,
|
||||
artifact: {
|
||||
label: ARTIFACT_LABEL,
|
||||
outputDirectory,
|
||||
executable,
|
||||
appAsar,
|
||||
appAsarUnpacked: join(resourcesDirectory, 'app.asar.unpacked'),
|
||||
builderArtifacts,
|
||||
},
|
||||
stagedClosure,
|
||||
packagedClosure,
|
||||
runtime,
|
||||
result: runtime.result,
|
||||
decision: 'incomplete',
|
||||
};
|
||||
if (options.reportPath) {
|
||||
await mkdir(dirname(options.reportPath), { recursive: true });
|
||||
await writeFile(options.reportPath, `${JSON.stringify(report, null, 2)}\n`);
|
||||
}
|
||||
return report;
|
||||
} finally {
|
||||
await rm(scratchRoot, { recursive: true, force: true, maxRetries: 5, retryDelay: 200 });
|
||||
}
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const options = parseProbeArgs(process.argv.slice(2));
|
||||
if (options.help) {
|
||||
process.stdout.write('Usage: node scripts/probe-pi-packaged-runtime.mjs [probe options]\n');
|
||||
return;
|
||||
}
|
||||
if (options.stage || options.keepStage || options.electronExecutablePath) {
|
||||
throw new Error('Packaged probe owns staging and runtime paths; do not pass staging or runtime override flags');
|
||||
}
|
||||
const report = await runPackagedProbe(options);
|
||||
process.stdout.write(`${JSON.stringify(report, null, 2)}\n`);
|
||||
if (report.result === 'fail') process.exitCode = 1;
|
||||
}
|
||||
|
||||
const isMain = process.argv[1]
|
||||
&& pathToFileURL(resolve(process.argv[1])).href === import.meta.url;
|
||||
if (isMain) {
|
||||
main().catch((error) => {
|
||||
process.stderr.write(`${error.stack ?? error.message}\n`);
|
||||
process.exitCode = 1;
|
||||
});
|
||||
}
|
||||
324
scripts/probe-pi-provider-contracts.mjs
Normal file
324
scripts/probe-pi-provider-contracts.mjs
Normal file
@@ -0,0 +1,324 @@
|
||||
import { createServer } from 'node:http';
|
||||
import { mkdtemp, mkdir, rm, writeFile } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { dirname, join, resolve } from 'node:path';
|
||||
import { pathToFileURL } from 'node:url';
|
||||
import {
|
||||
MAKELore_PROVIDER_PROTOCOLS,
|
||||
parseProbeArgs,
|
||||
resolvePiRuntime,
|
||||
runProviderQualification,
|
||||
validatePiIdentity,
|
||||
} from './probe-pi-runtime.mjs';
|
||||
|
||||
const LOCAL_API_KEY_ENV = 'PI_PROBE_LOCAL_CONTRACT_KEY';
|
||||
const LOCAL_API_KEY = 'pi-local-contract-only';
|
||||
const CUSTOM_HEADER = 'x-makelore-pi-probe';
|
||||
const IMAGE_BASE64 = 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAusB9Y9Z4N8AAAAASUVORK5CYII=';
|
||||
|
||||
function readRequestBody(request) {
|
||||
return new Promise((resolvePromise, reject) => {
|
||||
const chunks = [];
|
||||
request.on('data', (chunk) => chunks.push(chunk));
|
||||
request.once('end', () => {
|
||||
try {
|
||||
resolvePromise(JSON.parse(Buffer.concat(chunks).toString('utf8')));
|
||||
} catch (error) {
|
||||
reject(error);
|
||||
}
|
||||
});
|
||||
request.once('error', reject);
|
||||
});
|
||||
}
|
||||
|
||||
function writeSse(response, event) {
|
||||
if (response.destroyed || response.writableEnded) return;
|
||||
response.write(`event: ${event.type}\ndata: ${JSON.stringify(event)}\n\n`);
|
||||
}
|
||||
|
||||
function finishResponse(response, events) {
|
||||
setTimeout(() => {
|
||||
if (response.destroyed || response.writableEnded) return;
|
||||
for (const event of events) writeSse(response, event);
|
||||
response.end();
|
||||
}, 250);
|
||||
}
|
||||
|
||||
function respondOpenAiCompletions(response, body) {
|
||||
const id = `chatcmpl-local-${Date.now()}`;
|
||||
response.writeHead(200, { 'content-type': 'text/event-stream' });
|
||||
response.write(`data: ${JSON.stringify({
|
||||
id,
|
||||
object: 'chat.completion.chunk',
|
||||
created: Math.floor(Date.now() / 1000),
|
||||
model: body.model,
|
||||
choices: [{ index: 0, delta: { role: 'assistant', content: 'PI_PROVIDER_OK' }, finish_reason: null }],
|
||||
})}\n\n`);
|
||||
setTimeout(() => {
|
||||
if (response.destroyed || response.writableEnded) return;
|
||||
response.write(`data: ${JSON.stringify({
|
||||
id,
|
||||
object: 'chat.completion.chunk',
|
||||
created: Math.floor(Date.now() / 1000),
|
||||
model: body.model,
|
||||
choices: [{ index: 0, delta: {}, finish_reason: 'stop' }],
|
||||
usage: { prompt_tokens: 1, completion_tokens: 1, total_tokens: 2 },
|
||||
})}\n\n`);
|
||||
response.end('data: [DONE]\n\n');
|
||||
}, 250);
|
||||
}
|
||||
|
||||
function responseEnvelope(id, model, status, output = []) {
|
||||
return {
|
||||
id,
|
||||
object: 'response',
|
||||
created_at: Math.floor(Date.now() / 1000),
|
||||
status,
|
||||
model,
|
||||
output,
|
||||
error: null,
|
||||
incomplete_details: null,
|
||||
usage: status === 'completed'
|
||||
? {
|
||||
input_tokens: 1,
|
||||
input_tokens_details: { cached_tokens: 0 },
|
||||
output_tokens: 1,
|
||||
output_tokens_details: { reasoning_tokens: 0 },
|
||||
total_tokens: 2,
|
||||
}
|
||||
: null,
|
||||
};
|
||||
}
|
||||
|
||||
function respondOpenAiResponses(response, body) {
|
||||
const id = `resp_local_${Date.now()}`;
|
||||
const item = {
|
||||
id: `msg_local_${Date.now()}`,
|
||||
type: 'message',
|
||||
status: 'completed',
|
||||
role: 'assistant',
|
||||
content: [{ type: 'output_text', text: 'PI_PROVIDER_OK', annotations: [] }],
|
||||
};
|
||||
response.writeHead(200, { 'content-type': 'text/event-stream' });
|
||||
writeSse(response, { type: 'response.created', response: responseEnvelope(id, body.model, 'in_progress') });
|
||||
writeSse(response, { type: 'response.output_item.added', output_index: 0, item: { ...item, status: 'in_progress', content: [] } });
|
||||
writeSse(response, {
|
||||
type: 'response.output_text.delta',
|
||||
item_id: item.id,
|
||||
output_index: 0,
|
||||
content_index: 0,
|
||||
delta: 'PI_PROVIDER_OK',
|
||||
});
|
||||
finishResponse(response, [
|
||||
{ type: 'response.output_item.done', output_index: 0, item },
|
||||
{ type: 'response.completed', response: responseEnvelope(id, body.model, 'completed', [item]) },
|
||||
]);
|
||||
}
|
||||
|
||||
function respondAnthropic(response, body) {
|
||||
const id = `msg_local_${Date.now()}`;
|
||||
response.writeHead(200, { 'content-type': 'text/event-stream' });
|
||||
writeSse(response, {
|
||||
type: 'message_start',
|
||||
message: {
|
||||
id,
|
||||
type: 'message',
|
||||
role: 'assistant',
|
||||
model: body.model,
|
||||
content: [],
|
||||
stop_reason: null,
|
||||
stop_sequence: null,
|
||||
usage: { input_tokens: 1, output_tokens: 0 },
|
||||
},
|
||||
});
|
||||
writeSse(response, { type: 'content_block_start', index: 0, content_block: { type: 'text', text: '' } });
|
||||
writeSse(response, { type: 'content_block_delta', index: 0, delta: { type: 'text_delta', text: 'PI_PROVIDER_OK' } });
|
||||
finishResponse(response, [
|
||||
{ type: 'content_block_stop', index: 0 },
|
||||
{
|
||||
type: 'message_delta',
|
||||
delta: { stop_reason: 'end_turn', stop_sequence: null },
|
||||
usage: { output_tokens: 1 },
|
||||
},
|
||||
{ type: 'message_stop' },
|
||||
]);
|
||||
}
|
||||
|
||||
function protocolFromPath(pathname) {
|
||||
return MAKELore_PROVIDER_PROTOCOLS.find((protocol) => pathname.startsWith(`/${protocol}/`));
|
||||
}
|
||||
|
||||
async function startContractServer(requests) {
|
||||
const server = createServer(async (request, response) => {
|
||||
response.once('error', () => undefined);
|
||||
try {
|
||||
const body = await readRequestBody(request);
|
||||
const protocol = protocolFromPath(request.url ?? '');
|
||||
if (!protocol) {
|
||||
response.writeHead(404).end();
|
||||
return;
|
||||
}
|
||||
requests.push({
|
||||
protocol,
|
||||
method: request.method,
|
||||
path: request.url,
|
||||
headers: request.headers,
|
||||
model: body.model,
|
||||
hasImage: JSON.stringify(body).includes(IMAGE_BASE64),
|
||||
});
|
||||
if (protocol === 'anthropic-messages') respondAnthropic(response, body);
|
||||
else if (protocol === 'openai-responses') respondOpenAiResponses(response, body);
|
||||
else respondOpenAiCompletions(response, body);
|
||||
} catch (error) {
|
||||
if (!response.headersSent) response.writeHead(400, { 'content-type': 'application/json' });
|
||||
if (!response.destroyed && !response.writableEnded) {
|
||||
response.end(JSON.stringify({ error: error instanceof Error ? error.message : String(error) }));
|
||||
}
|
||||
}
|
||||
});
|
||||
await new Promise((resolvePromise, reject) => {
|
||||
server.once('error', reject);
|
||||
server.listen(0, '127.0.0.1', resolvePromise);
|
||||
});
|
||||
const address = server.address();
|
||||
if (!address || typeof address === 'string') throw new Error('Local provider server did not expose a TCP port');
|
||||
return {
|
||||
port: address.port,
|
||||
close: () => new Promise((resolvePromise, reject) => {
|
||||
server.close((error) => (error ? reject(error) : resolvePromise()));
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
function expectedPath(protocol) {
|
||||
const endpoint = protocol === 'openai-responses'
|
||||
? 'responses'
|
||||
: protocol === 'anthropic-messages'
|
||||
? 'messages'
|
||||
: 'chat/completions';
|
||||
return `/${protocol}/v1/${endpoint}`;
|
||||
}
|
||||
|
||||
function validateCapturedRequests(protocol, modelId, requests) {
|
||||
const matching = requests.filter((request) => request.protocol === protocol);
|
||||
const authHeader = protocol === 'anthropic-messages' ? 'x-api-key' : 'authorization';
|
||||
const expectedAuth = protocol === 'anthropic-messages' ? LOCAL_API_KEY : `Bearer ${LOCAL_API_KEY}`;
|
||||
const problems = [];
|
||||
if (matching.length !== 4) problems.push(`expected 4 requests, got ${matching.length}`);
|
||||
if (matching.some((request) => request.method !== 'POST')) problems.push('non-POST request');
|
||||
if (matching.some((request) => request.path !== expectedPath(protocol))) problems.push('unexpected endpoint path');
|
||||
if (matching.some((request) => request.headers[authHeader] !== expectedAuth)) problems.push('credential header mismatch');
|
||||
if (matching.some((request) => request.headers[CUSTOM_HEADER] !== protocol)) problems.push('custom header mismatch');
|
||||
if (matching.some((request) => request.model !== modelId)) problems.push('model mismatch');
|
||||
if (matching.filter((request) => request.hasImage).length < 2) problems.push('image payload missing from initial turns');
|
||||
if (problems.length > 0) throw new Error(`${protocol} local contract failed: ${problems.join('; ')}`);
|
||||
return {
|
||||
requestCount: matching.length,
|
||||
method: 'POST',
|
||||
path: expectedPath(protocol),
|
||||
credentialHeader: authHeader,
|
||||
customHeader: CUSTOM_HEADER,
|
||||
modelId,
|
||||
imageRequests: matching.filter((request) => request.hasImage).length,
|
||||
};
|
||||
}
|
||||
|
||||
export async function runLocalProviderContracts(options, projectRoot = process.cwd()) {
|
||||
const scratchRoot = await mkdtemp(join(tmpdir(), 'makelore-pi-provider-contracts-'));
|
||||
const requests = [];
|
||||
const previousApiKey = process.env[LOCAL_API_KEY_ENV];
|
||||
const server = await startContractServer(requests);
|
||||
process.env[LOCAL_API_KEY_ENV] = LOCAL_API_KEY;
|
||||
try {
|
||||
const sourceRuntime = await resolvePiRuntime(projectRoot);
|
||||
const identity = await validatePiIdentity(sourceRuntime);
|
||||
const runtime = options.electronExecutablePath
|
||||
? {
|
||||
...sourceRuntime,
|
||||
electronExecutable: options.electronExecutablePath,
|
||||
cliPath: options.cliPath,
|
||||
}
|
||||
: sourceRuntime;
|
||||
const imagePath = join(scratchRoot, 'pixel.png');
|
||||
await writeFile(imagePath, Buffer.from(IMAGE_BASE64, 'base64'));
|
||||
const protocols = [];
|
||||
for (const protocol of MAKELore_PROVIDER_PROTOCOLS) {
|
||||
const protocolRoot = join(scratchRoot, protocol);
|
||||
await mkdir(protocolRoot);
|
||||
const fixturePath = join(protocolRoot, 'fixture.json');
|
||||
const modelId = `makelore-local-${protocol}`;
|
||||
const baseUrl = protocol === 'anthropic-messages'
|
||||
? `http://127.0.0.1:${server.port}/${protocol}`
|
||||
: `http://127.0.0.1:${server.port}/${protocol}/v1`;
|
||||
await writeFile(fixturePath, `${JSON.stringify({
|
||||
id: `makelore-local-${protocol}`,
|
||||
apiProtocol: protocol,
|
||||
baseUrl,
|
||||
apiKeyEnv: LOCAL_API_KEY_ENV,
|
||||
headers: { [CUSTOM_HEADER]: protocol },
|
||||
model: { id: modelId, input: ['text', 'image'] },
|
||||
}, null, 2)}\n`);
|
||||
const qualification = await runProviderQualification(
|
||||
runtime,
|
||||
protocolRoot,
|
||||
fixturePath,
|
||||
imagePath,
|
||||
options.timeoutMs,
|
||||
);
|
||||
protocols.push({
|
||||
protocol,
|
||||
qualification,
|
||||
requestContract: validateCapturedRequests(protocol, modelId, requests),
|
||||
});
|
||||
}
|
||||
const report = {
|
||||
schemaVersion: 1,
|
||||
generatedAt: new Date().toISOString(),
|
||||
identity,
|
||||
runtime: {
|
||||
artifact: options.artifactLabel ?? 'workspace-install',
|
||||
executable: runtime.electronExecutable,
|
||||
cliPath: runtime.cliPath,
|
||||
},
|
||||
scope: {
|
||||
network: '127.0.0.1 only',
|
||||
realProvider: false,
|
||||
limitation: 'Validates Pi HTTP/SSE request contracts and worker behavior; does not qualify a real provider account.',
|
||||
},
|
||||
protocols,
|
||||
result: 'pass',
|
||||
};
|
||||
if (options.reportPath) {
|
||||
await mkdir(dirname(options.reportPath), { recursive: true });
|
||||
await writeFile(options.reportPath, `${JSON.stringify(report, null, 2)}\n`);
|
||||
}
|
||||
return report;
|
||||
} finally {
|
||||
if (previousApiKey === undefined) delete process.env[LOCAL_API_KEY_ENV];
|
||||
else process.env[LOCAL_API_KEY_ENV] = previousApiKey;
|
||||
await server.close();
|
||||
await rm(scratchRoot, { recursive: true, force: true, maxRetries: 5, retryDelay: 200 });
|
||||
}
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const options = parseProbeArgs(process.argv.slice(2));
|
||||
if (options.help) {
|
||||
process.stdout.write('Usage: node scripts/probe-pi-provider-contracts.mjs [probe options]\n');
|
||||
return;
|
||||
}
|
||||
if (options.stage || options.keepStage || options.providerFixturePath || options.imagePath) {
|
||||
throw new Error('Local provider contract probe owns its fixtures and does not accept staging/provider fixture flags');
|
||||
}
|
||||
const report = await runLocalProviderContracts(options);
|
||||
process.stdout.write(`${JSON.stringify(report, null, 2)}\n`);
|
||||
}
|
||||
|
||||
const isMain = process.argv[1]
|
||||
&& pathToFileURL(resolve(process.argv[1])).href === import.meta.url;
|
||||
if (isMain) {
|
||||
main().catch((error) => {
|
||||
process.stderr.write(`${error.stack ?? error.message}\n`);
|
||||
process.exitCode = 1;
|
||||
});
|
||||
}
|
||||
1102
scripts/probe-pi-runtime.mjs
Normal file
1102
scripts/probe-pi-runtime.mjs
Normal file
File diff suppressed because it is too large
Load Diff
149
tests/unit/pi-runtime-probe.test.ts
Normal file
149
tests/unit/pi-runtime-probe.test.ts
Normal file
@@ -0,0 +1,149 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { join, sep } from 'node:path';
|
||||
import { packagedResourcesDirectory } from '../../scripts/probe-pi-packaged-runtime.mjs';
|
||||
import {
|
||||
buildEvidenceWaivers,
|
||||
buildMissingEvidence,
|
||||
buildPiProviderConfig,
|
||||
createStrictJsonlParser,
|
||||
lockEntrySupportsCurrentPlatform,
|
||||
MAKELore_PROVIDER_PROTOCOLS,
|
||||
mapMakeloreProtocolToPi,
|
||||
parseProbeArgs,
|
||||
percentile,
|
||||
summarizeMeasurements,
|
||||
} from '../../scripts/probe-pi-runtime.mjs';
|
||||
|
||||
describe('Pi runtime qualification probe', () => {
|
||||
it('locates macOS resources beside Contents/MacOS', () => {
|
||||
const executable = join('probe-root', 'MakelorePiProbe.app', 'Contents', 'MacOS', 'MakelorePiProbe');
|
||||
const resources = packagedResourcesDirectory(executable, 'darwin');
|
||||
|
||||
expect(resources.split(sep).slice(-3)).toEqual([
|
||||
'MakelorePiProbe.app',
|
||||
'Contents',
|
||||
'Resources',
|
||||
]);
|
||||
});
|
||||
|
||||
it('parses strict LF-delimited JSONL without splitting Unicode separators', () => {
|
||||
const records: unknown[] = [];
|
||||
const parser = createStrictJsonlParser((record) => records.push(record));
|
||||
const source = `${JSON.stringify({ text: 'left\u2028middle\u2029right' })}\n${JSON.stringify({ ok: true })}\r\n`;
|
||||
const bytes = Buffer.from(source);
|
||||
|
||||
parser.push(bytes.subarray(0, 9));
|
||||
parser.push(bytes.subarray(9, 25));
|
||||
parser.push(bytes.subarray(25));
|
||||
parser.finish();
|
||||
|
||||
expect(records).toEqual([
|
||||
{ text: 'left\u2028middle\u2029right' },
|
||||
{ ok: true },
|
||||
]);
|
||||
});
|
||||
|
||||
it('uses nearest-rank p50 and p95 measurements', () => {
|
||||
expect(percentile([5, 1, 4, 2, 3], 50)).toBe(3);
|
||||
expect(percentile([5, 1, 4, 2, 3], 95)).toBe(5);
|
||||
expect(summarizeMeasurements([1.2, 2.8, 9.1])).toEqual({
|
||||
samples: 3,
|
||||
p50: 3,
|
||||
p95: 9,
|
||||
max: 9,
|
||||
});
|
||||
});
|
||||
|
||||
it('requires explicit, bounded CLI arguments', () => {
|
||||
expect(parseProbeArgs(['--stage', '--samples', '3', '--timeout-ms', '2000'])).toMatchObject({
|
||||
stage: true,
|
||||
samples: 3,
|
||||
timeoutMs: 2000,
|
||||
});
|
||||
expect(() => parseProbeArgs(['--samples', '0'])).toThrow(/positive integer/);
|
||||
expect(() => parseProbeArgs(['--keep-stage'])).toThrow(/requires --stage/);
|
||||
expect(() => parseProbeArgs(['--electron-executable', 'electron.exe'])).toThrow(/provided together/);
|
||||
expect(() => parseProbeArgs([
|
||||
'--stage',
|
||||
'--electron-executable',
|
||||
'electron.exe',
|
||||
'--cli-path',
|
||||
'app.asar/dist/cli.js',
|
||||
])).toThrow(/cannot be combined/);
|
||||
expect(parseProbeArgs([
|
||||
'--electron-executable',
|
||||
'electron.exe',
|
||||
'--cli-path',
|
||||
'app.asar/dist/cli.js',
|
||||
'--artifact-label',
|
||||
'controlled-app-asar',
|
||||
])).toMatchObject({ artifactLabel: 'controlled-app-asar' });
|
||||
});
|
||||
|
||||
it('covers every current Makelore provider protocol', () => {
|
||||
expect(MAKELore_PROVIDER_PROTOCOLS).toEqual([
|
||||
'openai-completions',
|
||||
'openai-responses',
|
||||
'anthropic-messages',
|
||||
'openrouter',
|
||||
]);
|
||||
expect(mapMakeloreProtocolToPi('openai-responses')).toEqual({
|
||||
api: 'openai-responses',
|
||||
compat: undefined,
|
||||
});
|
||||
expect(mapMakeloreProtocolToPi('openrouter')).toEqual({
|
||||
api: 'openai-completions',
|
||||
compat: {
|
||||
thinkingFormat: 'openrouter',
|
||||
sessionAffinityFormat: 'openrouter',
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it('reports the user-approved macOS deferral separately from Phase-0 aggregation', () => {
|
||||
expect(buildMissingEvidence()).toContain(
|
||||
'Phase-0 artifact aggregation: Windows x64 and Linux x64; macOS x64/arm64 deferred to PI-150 by user decision on 2026-08-22',
|
||||
);
|
||||
expect(buildMissingEvidence()).not.toContain('Linux unpackaged and packaged samples');
|
||||
});
|
||||
|
||||
it('reports real Provider validation as explicitly waived without claiming a real turn', () => {
|
||||
expect(buildMissingEvidence()).not.toEqual(expect.arrayContaining([
|
||||
expect.stringContaining('real-provider'),
|
||||
expect.stringContaining('real provider'),
|
||||
]));
|
||||
expect(buildEvidenceWaivers()).toEqual(expect.arrayContaining([
|
||||
expect.stringContaining('QG-004/QG-005'),
|
||||
expect.stringContaining('explicitly waived'),
|
||||
expect.stringContaining('remain unverified'),
|
||||
]));
|
||||
});
|
||||
|
||||
it('builds a Pi provider model using an environment reference, not a credential value', () => {
|
||||
const config = buildPiProviderConfig({
|
||||
id: 'probe-openrouter',
|
||||
apiProtocol: 'openrouter',
|
||||
baseUrl: 'https://openrouter.ai/api/v1',
|
||||
apiKeyEnv: 'PROBE_OPENROUTER_KEY',
|
||||
headers: { 'X-OpenRouter-Title': 'Makelore qualification' },
|
||||
model: { id: 'anthropic/claude-sonnet-4' },
|
||||
});
|
||||
|
||||
expect(config.providers['probe-openrouter']).toMatchObject({
|
||||
api: 'openai-completions',
|
||||
apiKey: '$PROBE_OPENROUTER_KEY',
|
||||
models: [{
|
||||
id: 'anthropic/claude-sonnet-4',
|
||||
compat: {
|
||||
thinkingFormat: 'openrouter',
|
||||
sessionAffinityFormat: 'openrouter',
|
||||
},
|
||||
}],
|
||||
});
|
||||
});
|
||||
|
||||
it('rejects lock entries for a different operating system', () => {
|
||||
expect(lockEntrySupportsCurrentPlatform({ os: [`!${process.platform}`] })).toBe(false);
|
||||
expect(lockEntrySupportsCurrentPlatform({ os: [process.platform], cpu: [process.arch] })).toBe(true);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user