diff --git a/.project-docs/30-worklog/tasks/20260912-fix-image-delivery-d74b1f08.md b/.project-docs/30-worklog/tasks/20260912-fix-image-delivery-d74b1f08.md new file mode 100644 index 0000000..cbc64f2 --- /dev/null +++ b/.project-docs/30-worklog/tasks/20260912-fix-image-delivery-d74b1f08.md @@ -0,0 +1,56 @@ +# Task: Fix model image delivery and sent attachment visibility + +## Identity + +- Task ID: 20260912-fix-image-delivery-d74b1f08 +- Mode: Feature +- Branch: codex/20260912-fix-image-delivery-d74b1f08-fix-image-delivery +- Worktree: D:\Datas\OthersProjects\.codex-worktrees\makelore\20260912-fix-image-delivery-d74b1f08 +- Base commit: 1d661f7a5695f0ef1364e1fae5874d106f855bd9 +- Owner: codex +- Status: Ready for Integration + +## Scope + +- Diagnose the reported deepseek-flash image refusal and disappearance of sent image attachments. +- Fix Makelore's optimistic user messages and production Pi live/session image projection. Cover binary attachment reuse and renderer reload. +- one-api and Works Square remain read-only in this follow-up; no model-capability catalog or provider control changes were justified by the observations. + +## Intent And Constraints + +- Concurrent Task Gate and Planning Gate passed in this exclusively owned isolated worktree before edits. Read peer task scopes; no conflicting ownership was found. +- No subagents. No external inference, publishing, installation, main-branch changes or live-session mutation. +- The installed app is 1.4.7 with Pi 0.84.2. Preserve runtime version, Host API boundaries, original user history and pending unrelated work. +- Only task-owned project memory is updated. README describes the implemented behavior. + +## Outcome + +- Confirmed the affected persisted user message contains text and a PNG image (base64 length 7,616,308); the configured deepseek-flash model contains text/image. Both global and project image-block settings are absent. +- Used the installed runtime to restore a temporary local copy of the affected session and send to a loopback fixture. Its selected model retained text/image and the affected user message reached the fixture as text plus a complete image_url. Original history stayed untouched, and the temporary copy was removed. Only message types, sizes and model capability fields were reported; no secrets or private message bodies were captured in task artifacts. +- User confirmed that a new empty deepseek-flash conversation can recognize the same image. Old history includes prior omitted-image wording. Historical context influencing the model's denial is a supported inference; the original production HTTP exchange was not captured, so its exact earlier failure is not asserted. Current multimodal support and fresh-session operation are confirmed. +- Fixed the reproducible UI defect: PiConversationRuntime now includes uploaded image references in the optimistic user message, including image-only prompts. Production coding composition now wires projectImage to attachment storage for both live events and persisted sessions. +- Attachment storage uses a content address over MIME and bytes to reuse the original newly uploaded file across repeated Pi projections and application restarts. This digest replaces repeated large-file writes and determines whether writing is skipped; it introduces no separate integrity catalog. Existing randomly identified attachments remain readable. +- No model-name special case, forced vision flag, history deletion or prompt override was added. + +## Verification + +- Initial regression: `pnpm exec vitest run tests/unit/pi-conversation-runtime.test.ts -t 'keeps submitted image references' --maxWorkers=1` failed because the optimistic node contained text only, while the outgoing Pi RPC already contained the image. +- After fixing: composition image storage/projection wiring, event projector, session projector, conversation runtime and attachment routes: 35 tests passed. +- Final focused run: composition images, conversation runtime (text+image and image-only), real Agent Server process, managed capabilities and attachment facade: 17 tests passed. This includes the DeepSeek-compatible real HTTP image assertions for fresh and restored sessions. +- Updated Electron first-chat E2E passed against the final build. It covers binary upload, optimistic preview, replacement with a persisted message, and image display after a complete renderer reload. +- `pnpm run typecheck`, targeted ESLint, `pnpm run build:vite` and `git diff --check` passed. Build retains existing chunk-size, mixed-import and outdated Browserslist notices. +- Additional Main TypeScript check does not pass on this repository baseline. Compiler comparison using the recorded base versions of all three changed production files: 61 diagnostics before, 61 after, zero new diagnostics. These existing errors are outside the image fix. +- Final self-review inspected the source/test/README diff against the recorded base; no temporary diagnostic code, private session copy, dependencies or build artifacts are included. + +## Follow-ups + +- Merge and package this fix before expecting it in the installed 1.4.7 client; this feature task does not install or release a client build. +- For the affected old conversation, a new conversation is the user-confirmed working route. No original messages were removed or rewritten. + +## Promotion Candidates + +- Target: `20-architecture/data-flow.md`. Record the now-connected Pi live/session image projection into Main-owned attachment storage and the reuse of stored bytes during hydration. +- Evidence: production composition regression, projector tests and Electron reload E2E. +- Future impact: prevents future refactors from testing only injected projection callbacks while omitting production composition wiring. +- Semantic conflicts: none identified; follows the accepted Main-owned binary attachment contract. +- Human confirmation required: no product-direction decision; promote through the normal Integration Gate when this fix is merged. diff --git a/README.md b/README.md index c2455b6..52fd5c0 100644 --- a/README.md +++ b/README.md @@ -137,7 +137,7 @@ Pi 正式包必须继续运行 `pnpm run verify:artifact:pi`、`pnpm run smoke:p - 核心聊天页左侧把本地对话嵌套在展开的所属智能体下，并在该智能体子组中提供新建入口；首次选择没有对话的智能体时立即创建本地元数据，同时异步准备对应运行时。即使准备被阻塞或超时，输入框仍可编辑，草稿也不会丢失。 - Conversation 历史按需从 Main-owned Snapshot 读取。Session hydration 沿完整 active branch 投影可见历史；公开 SSE 只交付 Snapshot 与按 Conversation、worker generation 分组的 `patch-batch`。Renderer 整批校验连续 seq 后在一次状态事务中顺序应用，缺口或畸形批次只恢复目标 Conversation，隐藏 Conversation 的流式更新不会提交选中时间线。Main 发出 `lifecycle:sleep` 时页面关闭旧事件流；视图挂载、项目上下文变化、窗口重新可见或获得焦点时，已选 Conversation 会静默重读权威 Snapshot，从而收敛后台已结束的任务，但绝不重放已接受或状态不确定的 mutation。时间线默认挂载最近 120 个节点，接近顶部时按 100 个节点自动追加更早内容并补偿新增高度，保持当前阅读锚点。 - 核心时间线按每条用户消息组织为“用户消息、连续处理过程、结论回复”。Pi 的流式 thinking、工具、压缩、重试、通知和 `subagent.v1` 单个/并行/串行子任务共享一条轻量过程轨道；运行时显示“处理中”和活动指示，完成后按耗时（历史缺少计时时按项目数）收纳为可展开的“已处理”摘要。文字层级按结构化事件类型划分，不按文案关键词猜测：助手在工具调用前后直接对用户说的过程正文与最终结论都使用黑色；thinking、工具输入输出、压缩、子任务和运行状态使用浅灰辅助文字，仍在运行的灰色缩略文字带有低对比度 Shimmer 擦拭高光，结束后停止动画；错误状态保持红色。折叠的 thinking、助手过程正文和工具输出固定展示第一个可见块的首个非空行，横向偏移始终为零；尾部流式更新不会替换未变化的首行预览，展开后仍可查看完整内容。两类流式文字均在原位更新且不做位移动画，助手过程正文也不使用 Shimmer。工具调用用一行摘要持续展示工具名称、最新输入或输出和运行状态，并可按流式批次轻量整行轮动。思考、助手过程正文和工具节点均可展开查看完整详情，工具的累计输出和浏览器附件留在对应的二级展开区而不形成独立气泡。结论回复按安全 Markdown 排版标题、列表、引用、代码和表格；可点击的 HTTP/HTTPS 与本地 HTML 链接统一使用无底色的橙色文字。左键通过 Main-owned 系统入口交给默认浏览器打开；右键可打开系统菜单，网页链接可复制地址，本地链接还可在访达或系统文件夹中显示。同一回复已经给出完整 HTML 路径时，对应的相对查看路径也会解析到该文件。内部 `turn-start`/`turn-end` 只维持事件顺序、不呈现在界面。 -- Composer 使用贴底的复合输入面板，支持文字、登录后的语音转写，以及粘贴或选择 PNG/JPEG/WebP/GIF 图片；每条消息最多 16 张图片、最多并行上传 4 张。语音录制通过现有 Main-owned Works Speech Host API 转成文字，并在项目或账号切换时立即停止；当前对话的模型与思考强度组合入口、语音及发送/中止入口保留在底部控制行，模型入口默认使用透明底色。模型与思考强度通过向上展开的分级菜单切换，只列出当前 Provider 或 Makelore 已验证模型档案实际支持的选项；Qwen3.6 Plus 使用混合思考开关，Qwen3.8 Max 暴露低/中/高三档并把产品“高”映射到服务端 `xhigh`。模型仅报告 `off` 时明确显示“不可调”，不伪造未知模型的推理档位。切换完成后静默刷新当前 Snapshot，只有真正恢复 worker 时才显示本地 Agent 重连提示。重复读取 Snapshot 不发布无变化的 worker 状态；Snapshot 与 SSE 批次竞态时，Renderer 会丢弃已覆盖事件并只应用连续的新尾部，避免形成伪重连循环。上下文接近模型限制或发生溢出时由 Pi 自动整理，Composer 不提供手动整理入口；自动整理的进度和结果仍显示在对话过程轨道中。图片在发送前只保留本地预览，点击发送时才经 Main-owned 有界二进制接口上传一次；Main 在落盘前核对 MIME 与最小图片签名，状态与事件只保存 attachment id，时间线按需读取二进制并创建临时 object URL，不保存重复 base64。智能体报告 `changed-file.v1` 后，输入框上方会展示当前轮次的文件数与增删行统计，并可展开查看 Main-owned 变更快照中的逐文件差异。 +- Composer 使用贴底的复合输入面板，支持文字、登录后的语音转写，以及粘贴或选择 PNG/JPEG/WebP/GIF 图片；每条消息最多 16 张图片、最多并行上传 4 张。语音录制通过现有 Main-owned Works Speech Host API 转成文字，并在项目或账号切换时立即停止；当前对话的模型与思考强度组合入口、语音及发送/中止入口保留在底部控制行，模型入口默认使用透明底色。模型与思考强度通过向上展开的分级菜单切换，只列出当前 Provider 或 Makelore 已验证模型档案实际支持的选项；Qwen3.6 Plus 使用混合思考开关，Qwen3.8 Max 暴露低/中/高三档并把产品“高”映射到服务端 `xhigh`。模型仅报告 `off` 时明确显示“不可调”，不伪造未知模型的推理档位。切换完成后静默刷新当前 Snapshot，只有真正恢复 worker 时才显示本地 Agent 重连提示。重复读取 Snapshot 不发布无变化的 worker 状态；Snapshot 与 SSE 批次竞态时，Renderer 会丢弃已覆盖事件并只应用连续的新尾部，避免形成伪重连循环。上下文接近模型限制或发生溢出时由 Pi 自动整理，Composer 不提供手动整理入口；自动整理的进度和结果仍显示在对话过程轨道中。图片在发送前只保留本地预览，点击发送时才经 Main-owned 有界二进制接口上传一次；Main 在落盘前核对 MIME 与最小图片签名，状态与事件只保存 attachment id，时间线按需读取二进制并创建临时 object URL，不保存重复 base64。发送后立即展示图片附件；Pi 实时消息与历史会话恢复都将图片映射到同一附件存储，重复加载复用已有文件。智能体报告 `changed-file.v1` 后，输入框上方会展示当前轮次的文件数与增删行统计，并可展开查看 Main-owned 变更快照中的逐文件差异。 - Makelore 在应用侧按 Session 独立提交、跟踪和隔离运行状态，不使用“当前对话正在回复”的全局界面锁；同一 Session 的后续消息仍按顺序排队。最终产品中的 Pi 运行时会通过受控 Provider-shaped 回环 smoke 验证同一 Agent Server 内多个 Conversation 逻辑线程的重叠执行、状态隔离、凭证引用隔离、单线程关闭和 Server 崩溃后重启，以及独立子 Agent 进程的隔离与回收；真实外部 Provider 的并发、限流、协议兼容和凭证隔离仍是独立风险，未执行真实 Provider 验证时不得标记为 Pass。 - 首次发送会立即生成稳定的乐观用户消息；HTTP 202 只表示本地 Agent 已接收。后续失败不会删除已接受消息，不确定交付不会自动重发；准备失败可在目标 Conversation 上手动恢复。 - 运行中的 Conversation 可把新消息作为 steer 引导当前回答或 follow-up 排到下一轮，并显示队列位置；队列正常在 `agent_settled` 后释放，用户可中止当前运行。若 Pi 已终结但该握手丢失，Main 会在 30 秒宽限后仅探测目标线程：权威 idle 状态会补齐持久化最终回复并 exactly-once 释放 ownership，矛盾状态则显式失败该目标，且不会重放已接受请求。select/confirm/input/editor 交互在输入区上方回答，select 除预设选项外支持自由输入；交互提交或取消后立即从输入区移除，失效请求会就地提示并允许重试或取消。 diff --git a/electron/api/coding-composition.ts b/electron/api/coding-composition.ts index 7885b60..2fcd489 100644 --- a/electron/api/coding-composition.ts +++ b/electron/api/coding-composition.ts @@ -462,6 +462,7 @@ export function createCodingComposition( ), isAuthenticationError: isCodingProviderAuthenticationError, refreshCredential: refreshCodingProviderCredential, + projectImage: async ({ data, mime }) => await attachments.put(Buffer.from(data, 'base64'), mime), resolveImages: async (refs) => await Promise.all(refs.map(async ({ attachmentId }) => { const record = await attachments.read(attachmentId); return { diff --git a/electron/coding-projects/attachment-store.ts b/electron/coding-projects/attachment-store.ts index 4407ef0..c70c0e2 100644 --- a/electron/coding-projects/attachment-store.ts +++ b/electron/coding-projects/attachment-store.ts @@ -1,5 +1,5 @@ -import { randomUUID } from 'node:crypto'; -import { mkdir, readFile, rename, writeFile } from 'node:fs/promises'; +import { createHash, randomUUID } from 'node:crypto'; +import { mkdir, readFile, rename, stat, writeFile } from 'node:fs/promises'; import path from 'node:path'; const ATTACHMENT_ID_PATTERN = /^[A-Za-z0-9-]{1,64}$/; @@ -22,14 +22,14 @@ export interface CodingAttachmentStoreOptions { } export class CodingAttachmentStore { - private readonly createId: () => string; + private readonly createId: (() => string) | undefined; private readonly maxBytes: number; constructor( private readonly rootDir: string, options: CodingAttachmentStoreOptions = {}, ) { - this.createId = options.createId ?? randomUUID; + this.createId = options.createId; this.maxBytes = options.maxBytes ?? DEFAULT_MAX_ATTACHMENT_BYTES; } @@ -39,11 +39,21 @@ export class CodingAttachmentStore { if (data.byteLength === 0 || data.byteLength > this.maxBytes) { throw new Error('Attachment size is invalid'); } - const attachmentId = this.createId(); + // Live events and session hydration repeat the same large image. Reuse its + // stored bytes, including the original upload, instead of writing a new copy. + const attachmentId = this.createId?.() ?? createHash('sha256') + .update(normalizedMime).update('\0').update(data).digest('hex'); if (!ATTACHMENT_ID_PATTERN.test(attachmentId)) throw new Error('Attachment id is invalid'); await mkdir(this.rootDir, { recursive: true }); const dataPath = path.join(this.rootDir, `${attachmentId}.bin`); const metadataPath = path.join(this.rootDir, `${attachmentId}.json`); + if (!this.createId) { + const existing = await stat(metadataPath).catch((error: NodeJS.ErrnoException) => { + if (error.code === 'ENOENT') return null; + throw error; + }); + if (existing) return { attachmentId, mime: normalizedMime, byteLength: data.byteLength }; + } const temporaryDataPath = `${dataPath}.${randomUUID()}.tmp`; const temporaryMetadataPath = `${metadataPath}.${randomUUID()}.tmp`; const reference: CodingAttachmentRef = { diff --git a/electron/coding-runtime/pi/runtime.ts b/electron/coding-runtime/pi/runtime.ts index 0e0900e..3baa048 100644 --- a/electron/coding-runtime/pi/runtime.ts +++ b/electron/coding-runtime/pi/runtime.ts @@ -104,7 +104,9 @@ export interface PiConversationRuntimeOptions { pool: PiWorkerPool; registry: PiSessionRegistry; resolveModel(model: ProductModelRef): Promise; - resolveImages?(attachments: Array<{ attachmentId: string }>): Promise; + resolveImages?(attachments: Array<{ attachmentId: string }>): Promise>; projectImage?: PiEventProjectorOptions['projectImage']; createId?(kind: RuntimeIdKind): string; now?: () => number; @@ -813,9 +815,17 @@ export class PiConversationRuntime implements CodingConversationRuntime { clientRequestId: input.clientRequestId, role: 'user', status: 'optimistic', - blocks: input.text.length > 0 - ? [{ kind: 'text', id: `${messageId}:content:0`, text: input.text, status: 'complete' }] - : [], + blocks: [ + ...(input.text.length > 0 + ? [{ kind: 'text' as const, id: `${messageId}:content:0`, text: input.text, status: 'complete' as const }] + : []), + ...input.attachments.map(({ attachmentId }, index) => ({ + kind: 'image' as const, + id: `${messageId}:content:${index + (input.text.length > 0 ? 1 : 0)}`, + attachmentId, + mime: images[index].mimeType, + })), + ], }, }); const command: PiRpcCommand = { diff --git a/tests/e2e/pi-coding-first-chat.spec.ts b/tests/e2e/pi-coding-first-chat.spec.ts index 0e7d1a1..7da185d 100644 --- a/tests/e2e/pi-coding-first-chat.spec.ts +++ b/tests/e2e/pi-coding-first-chat.spec.ts @@ -603,6 +603,23 @@ async function installCodingFirstChatHost( await new Promise((resolve) => { state.releaseSnapshot = resolve; }); state.snapshotPending = false; } + const posted = state.captured.find((request) => ( + request.path === `/api/coding/conversations/${conversation.id}/prompt` && request.method === 'POST' + )); + if (!featureComplete && posted?.body) { + const refs = posted.body.attachments as Array<{ attachmentId: string }>; + return respond({ snapshot: { + ...snapshot, + nodes: [{ kind: 'message', id: 'entry:sent-image', sourceEntryId: 'sent-image', + role: 'user', status: 'complete', blocks: [ + { kind: 'text', id: 'entry:sent-image:content:0', text: posted.body.text, status: 'complete' }, + ...refs.map(({ attachmentId }, index) => ({ + kind: 'image', id: `entry:sent-image:image:${index}`, attachmentId, mime: 'image/png', + })), + ] }], + cursor: { workerGeneration: 1, seq: 1 }, + } }); + } const currentSnapshot = state.snapshotSettled ? { ...snapshot, @@ -872,6 +889,20 @@ test('first PI Conversation is editable under 500 ms and submits before runtime { attachmentId: expect.any(String) }, ]); expect(JSON.stringify(prompt?.body)).not.toContain('data:image'); + await releaseSnapshot(electronApp); + const persistedMessage = page.locator('[data-node-id="entry:sent-image"]'); + await expect(persistedMessage).toBeVisible(); + await expect(persistedMessage.getByRole('img', { name: '对话图片附件' })).toBeVisible(); + await expect(persistedMessage.getByRole('button', { name: '从这里创建新对话分支' })).toBeVisible(); + // A full renderer reload discards its optimistic state and blob URLs. + await page.reload(); + page = await getStableWindow(electronApp); + await expect.poll(async () => (await readState(electronApp)).snapshotPending).toBe(true); + await releaseSnapshot(electronApp); + await expect(page.locator('[data-node-id="entry:sent-image"]').getByRole('img', { + name: '对话图片附件', + })).toBeVisible(); + } finally { await releaseSnapshot(electronApp); } diff --git a/tests/unit/coding-composition-images.test.ts b/tests/unit/coding-composition-images.test.ts new file mode 100644 index 0000000..2470b17 --- /dev/null +++ b/tests/unit/coding-composition-images.test.ts @@ -0,0 +1,69 @@ +// @vitest-environment node + +import { mkdtemp, readdir, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; +import { afterEach, describe, expect, it, vi } from 'vitest'; +import type { AgentBrowserModule } from '../../electron/agent-browser'; +import { createCodingComposition } from '../../electron/api/coding-composition'; +import { CodingAttachmentStore } from '../../electron/coding-projects/attachment-store'; +import { createMemoryCodingProjectStorage } from '../../electron/coding-projects/project-store'; +import type { PiConversationRuntimeOptions } from '../../electron/coding-runtime/pi/runtime'; + +const captured = vi.hoisted(() => ({ options: undefined as PiConversationRuntimeOptions | undefined })); +vi.mock('../../electron/coding-runtime/pi/runtime', async (importOriginal) => { + const original = await importOriginal(); + return { ...original, PiConversationRuntime: class extends original.PiConversationRuntime { + constructor(options: PiConversationRuntimeOptions) { + super(options); + captured.options = options; + } + } }; +}); + +const roots: string[] = []; +afterEach(async () => { + captured.options = undefined; + await Promise.all(roots.splice(0).map((root) => rm(root, { recursive: true, force: true }))); +}); + +describe('coding composition images', () => { + it('connects live and persisted Pi images to readable attachments and reuses their bytes after restart', async () => { + const userDataDir = await mkdtemp(path.join(tmpdir(), 'makelore-composition-images-')); + roots.push(userDataDir); + const composition = createCodingComposition({ + storage: createMemoryCodingProjectStorage(), + browser: { close: vi.fn(async () => undefined) } as unknown as AgentBrowserModule, + paths: { + executablePath: process.execPath, + cliPath: path.join(userDataDir, 'unused-cli.js'), + serverPath: path.join(userDataDir, 'unused-server.mjs'), + userDataDir, + bundledSkillsDir: path.resolve('resources/coding-skills'), + }, + }); + try { + const bytes = Buffer.from('image bytes'); + const uploaded = await composition.attachments.put(bytes, 'image/png'); + const images = await captured.options!.resolveImages!([uploaded]); + expect(images).toEqual([{ type: 'image', data: bytes.toString('base64'), mimeType: 'image/png' }]); + expect(captured.options!.projectImage).toBeTypeOf('function'); + for (const source of ['live', 'session'] as const) { + const projected = await captured.options!.projectImage!({ + conversationId: 'conversation-image', source, data: images[0].data, mime: images[0].mimeType, + }); + expect(projected.attachmentId).toBe(uploaded.attachmentId); + expect((await composition.attachments.read(projected.attachmentId)).data).toEqual(bytes); + } + const root = path.join(userDataDir, 'coding-runtime', 'attachments'); + const reopened = new CodingAttachmentStore(root); + expect(await reopened.put(bytes, 'image/png')).toEqual(uploaded); + expect((await readdir(root)).sort()).toEqual([ + `${uploaded.attachmentId}.bin`, `${uploaded.attachmentId}.json`, + ]); + expect((await reopened.put(bytes, 'image/jpeg')).attachmentId).not.toBe(uploaded.attachmentId); + } finally { + await composition.shutdown(); + } + }); +}); diff --git a/tests/unit/pi-agent-server-process-real.test.ts b/tests/unit/pi-agent-server-process-real.test.ts index b0bb251..48f707b 100644 --- a/tests/unit/pi-agent-server-process-real.test.ts +++ b/tests/unit/pi-agent-server-process-real.test.ts @@ -368,6 +368,123 @@ describe('Pi Agent Server real process', () => { } }, 10_000); + it('sends DeepSeek images in fresh and restored real sessions', async () => { + const root = await mkdtemp(path.join(tmpdir(), 'makelore-pi-agent-server-image-delivery-')); + roots.push(root); + const projectPath = path.join(root, 'project'); + const configDir = path.join(root, 'config'); + const sessionDir = path.join(root, 'sessions'); + const extensionDir = path.join(root, 'extensions'); + await Promise.all([ + mkdir(projectPath, { recursive: true }), + mkdir(configDir, { recursive: true }), + mkdir(sessionDir, { recursive: true }), + mkdir(extensionDir, { recursive: true }), + ]); + const provider = await startHeldProvider(); + await Promise.all([ + writeFile(path.join(configDir, 'settings.json'), JSON.stringify({ httpIdleTimeoutMs: 250 })), + writeFile(path.join(configDir, 'models.json'), JSON.stringify({ + providers: { + 'makelore-test': { + baseUrl: provider.baseUrl, + api: 'openai-completions', + apiKey: '$MAKELORE_TEST_KEY', + models: [{ + id: 'deepseek-flash', + name: 'Test model', + reasoning: true, + compat: { supportsDeveloperRole: false, supportsReasoningEffort: false, + thinkingFormat: 'deepseek', requiresReasoningContentOnAssistantMessages: true }, + input: ['text', 'image'], + contextWindow: 32_000, + maxTokens: 4_096, + cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }, + }], + }, + }, + })), + ]); + const promptPath = path.join(root, 'system.md'); + const languagePromptPath = path.join(root, 'language.md'); + await Promise.all([ + writeFile(promptPath, 'Describe the supplied image.'), + writeFile(languagePromptPath, MAKELORE_DEFAULT_LANGUAGE_PROMPT), + ]); + const extensionHost = new PiManagedExtensionHost(); + const registration = await extensionHost.registerWorker({ + conversationId: 'image-delivery', + generation: 1, + projectId: 'project-a', + projectPath, + extensionsDir: extensionDir, + }); + const runtimeRoot = path.resolve('node_modules/@earendil-works/pi-coding-agent'); + const server = new PiAgentServerProcess({ + executablePath: process.execPath, + serverPath: path.resolve('resources/pi-agent-server.mjs'), + runtimeRoot, + configDir, + }); + const workerOptions: PiWorkerProcessOptions = { + executablePath: process.execPath, + cliPath: path.join(runtimeRoot, 'dist', 'cli.js'), + cwd: projectPath, + configDir, + sessionDir, + conversationId: 'image-delivery', + workerGeneration: 1, + tools: ['bash'], + additionalArgs: [ + '--provider', 'makelore-test', + '--model', 'deepseek-flash', + '--thinking', 'off', + '--system-prompt', promptPath, + '--append-system-prompt', languagePromptPath, + '--extension', registration.extensionPath, + '--session-id', 'session-image-delivery', + ], + env: { MAKELORE_TEST_KEY: 'secret-image-delivery', ...registration.env }, + sensitiveValues: ['secret-image-delivery', ...registration.sensitiveValues], + }; + let worker = server.createWorker(workerOptions); + try { + for (let pass = 0; pass < 2; pass += 1) { + await worker.start(); + await extensionHost.bindRun('image-delivery', 1, 'run-image-delivery'); + const settled = new Promise((resolve) => { + const unsubscribe = worker.subscribe((event) => { + if (event.type !== 'agent_settled') return; + unsubscribe(); + resolve(); + }); + }); + const image = { type: 'image', data: 'aW1hZ2U=', mimeType: 'image/png' }; + await worker.request(pass === 0 + ? { type: 'prompt', message: 'Describe this image', images: [image] } + : { type: 'prompt', message: 'Describe the earlier image again' }); + await expect.poll(() => provider.requests.length).toBe(pass + 1); + const request = JSON.parse(provider.requests[pass].body); + expect(request.messages).toContainEqual({ + role: 'user', content: [ + { type: 'text', text: 'Describe this image' }, + { type: 'image_url', image_url: { url: 'data:image/png;base64,aW1hZ2U=' } }, + ], + }); + provider.release(); + await settled; + await worker.stop('test_injection'); + if (pass === 0) worker = server.createWorker(workerOptions); + } + } finally { + await worker.stop('test_injection').catch(() => undefined); + await server.stop().catch(() => undefined); + await registration.dispose().catch(() => undefined); + await extensionHost.close().catch(() => undefined); + await provider.close().catch(() => undefined); + } + }, 10_000); + it('boots from the packaged sibling resource layout', async () => { const root = await mkdtemp(path.join(tmpdir(), 'makelore-pi-agent-server-layout-')); roots.push(root); diff --git a/tests/unit/pi-conversation-runtime.test.ts b/tests/unit/pi-conversation-runtime.test.ts index ec5d375..4341dc2 100644 --- a/tests/unit/pi-conversation-runtime.test.ts +++ b/tests/unit/pi-conversation-runtime.test.ts @@ -171,6 +171,49 @@ afterEach(async () => { }); describe('Pi Conversation runtime', () => { + + it.each(['What is in this picture?', ''])('keeps submitted image references visible before Pi emits its user message (%s)', async (text) => { + const projectPath = await mkdtemp(path.join(tmpdir(), 'makelore-image-runtime-')); + roots.push(projectPath); + const projectStore = createCodingProjectStore(createMemoryCodingProjectStorage(), { + createId: () => 'project-image', now: () => NOW, + }); + await createLocalCodingProject({ projectPath, now: NOW }, projectStore); + const model = { accountId: 'account-image', modelId: 'deepseek-flash', thinkingLevel: 'off' as const }; + await createCodingProjectAgent(projectPath, { + id: 'agent-image', avatarId: 'avatar-01', roleName: 'Reader', name: 'Reader', + model, modelResolution: 'resolved', + responsibility: { mission: 'Read images', owns: [], boundaries: [], collaborators: [], principles: [] }, + }, { now: NOW }); + const conversation = await createCodingConversationStore(projectPath).create({ + agentId: 'agent-image', title: 'Image', model, modelResolution: 'resolved', + }); + let worker: RuntimeFakeWorker; + const runtime = new PiConversationRuntime({ + pool: new PiWorkerPool({ openWorker: async ({ generation }) => { + worker = new RuntimeFakeWorker('image-worker', generation); + return { worker, session: { piSessionId: 'image-session', sessionKey: 'image-key' } }; + } }), + registry: new PiSessionRegistry({ projectStore }), + resolveModel: async (candidate) => ({ ...candidate, runtimeProviderId: 'image-provider', input: ['text', 'image'] }), + resolveImages: async () => [{ type: 'image', data: 'aW1hZ2U=', mimeType: 'image/png' }], + }); + try { + await runtime.prepare({ conversationId: conversation.id, projectId: 'project-image', + agentId: 'agent-image', title: 'Image', model: { model, modelResolution: 'resolved' } }); + await runtime.prompt({ conversationId: conversation.id, clientRequestId: 'image-request', + mode: 'prompt', text, attachments: [{ attachmentId: 'image-attachment' }] }); + expect(worker!.requests.find(command => command.type === 'prompt')).toMatchObject({ + images: [{ type: 'image', data: 'aW1hZ2U=', mimeType: 'image/png' }], + }); + const snapshot = await runtime.getSnapshot(conversation.id); + expect(snapshot.nodes).toContainEqual(expect.objectContaining({ + role: 'user', blocks: expect.arrayContaining([expect.objectContaining({ + kind: 'image', attachmentId: 'image-attachment', mime: 'image/png', + })]), + })); + } finally { await runtime.shutdown(); } + }); it('persists native managed choices and rejects removed effort or unsupported images before prompt', async () => { const projectPath = await mkdtemp(path.join(tmpdir(), 'makelore-managed-runtime-')); roots.push(projectPath);