diff --git a/.project-docs/30-worklog/tasks/20260921-game-audio-client-b74d9f31.md b/.project-docs/30-worklog/tasks/20260921-game-audio-client-b74d9f31.md new file mode 100644 index 0000000..49fe178 --- /dev/null +++ b/.project-docs/30-worklog/tasks/20260921-game-audio-client-b74d9f31.md @@ -0,0 +1,51 @@ +# Task: Implement official game audio client + +## Identity + +- Task ID: 20260921-game-audio-client-b74d9f31 +- Mode: Feature +- Branch: codex/20260921-game-audio-client-b74d9f31-game-audio-client +- Worktree: D:\Datas\OthersProjects\.codex-worktrees\makelore\20260921-game-audio-client-b74d9f31 +- Base commit: d222d175000008bad5f344577679256d22a636c2 +- Owner: codex +- Status: Ready for Integration + +## Scope + +- GA-03 offline implementation of ML-GAME-AUDIO-001: typed Main audio client, durable delivery, adapter, bundled resources/project-wide catalog, composition and focused tests/README. Preserve image generation, Data Service, local packages and selected-model Web Search. No root workspace, server, deployment or paid-provider writes. + +## Intent And Constraints + +- Concurrent Task Gate PASS: bundled start exited 0; status matches this task, feature mode, owner codex, exact branch/worktree/base. Root retains three foreign untracked records. Peer records were read at registry-derived paths; planning peers are old read-only reviews/undefined records and separate WeChat diagnosis. Undefined scopes remain unknown context, not ownership authority; no concrete overlapping audio implementation decision found. Completed image delivery is a read-only prerequisite. +- Project Context Loaded: entry/read-before-planning, planning-gate, memory-index, own record, positioning (placeholder; AGENTS/README establish actual product), current-state integrated snapshot, decision index/system overview, module map/data flow, domain rules/glossary/success criteria, relevant evidence/reflection/commitment/stale indexes and ADR-008. Approved server-side design at f6eabf253a4b4cd27f7f31fdf8e93afe21d7fcdd is the audio authority. +- Planning Gate Passed: one optional bundled project-wide plugin; current Plugin sheet stays in Project Configuration (older /plugins design route is superseded by actual integrated routing). Main owns Works auth, exact admission, durable identity, polling and all-output delivery. Child remains empty; no partner gate, picker, new store authority, Provider credentials, remote playback URLs, second save confirmation, or local generation rerun. +- Plan: client public HTTP contract TDD; durable pre-submit receipt/query-only recovery and sequential materialization with shared lease; adapter/manifest/static operations; composition/startup resume; focused image/Plugin regressions, typecheck/build and relevant Electron proof. No subagents. +- External limits: no authorized paid Meowa call or production activation; real response/media samples and PG proof remain separate. Routine client compatibility floor will be 2.0.1, paired with server migration 0093; no package publication implied. + +## Outcome + +- GA-03 offline implementation complete. Main-only typed client uses fixed server routes, closed receipts, bounded streamed bodies, frozen account epoch and one same-body 401 refresh. Nullable hosted amounts normalize to the generic capability receipt's omitted optional field. +- Durable intent is saved before POST; lost response uses by-operation only. Saved outputs and missing indices persist in userData. Startup/account reconnect resumes original jobs; transient download/status errors schedule query/delivery-only recovery. Definite pre-submit rejections remain not_started, while unknown post-submit failures cannot claim no charge. Shared local-project lease covers only writes/tracker, never generation/network waiting. Every output uses wx semantics in the frozen original project and no second confirmation. +- New adapter + one official optional bundled game-audio plugin expose music full, explicit music preview, sound single/pack/variants, cancel. Native price/release/project/request authority stays in Main. Four static tool-operation mappings, project-wide parent availability and empty child scope; no new picker/store/partner gate or provider secrets. +- Result card displays saved relative paths/MIME/bytes and loads local audio only after a click, with native controls/no autoplay. Reuses bounded project media reader through a receipt/account-owned Host route; no Provider URL or audio payload enters model messages. Composition shares leases/tracker and resumes receipts. Registry manifest duplicate checks now use plugin/capability/operation identity (global tool names remain unique), permitting two distinct audio capabilities to both name operation generate. +- Additional owned paths beyond the initial outline: read-only saved-audio Host route and composition service port, export of existing media reader, Chat progress/preview component and test, relevant effective-resolver/manifest/Registry/artifact assertions, one synthetic-audio Electron case reusing the existing first-chat fixture, README and package version 2.0.1. No plugin navigation/store authority changes. + +## Verification + +- TDD RED→GREEN at transport, delivery, adapter/Registry seams; tests cover literal confirmation, fixed mode/pricing/identity, streamed bounds, account change, lost response, automatic recovery without repeat POST, partial restart, owner-only local preview, no fake not_started after submission and nullable receipt normalization. +- Final focused set before the normalization refinement: 7 files / 45 passed. Final refined transport/adapter/delivery slice: 3 files / 13 passed. Typecheck PASS. Full lint PASS (0 errors, 8 existing CloudAgents/CloudCosts/Home/Makelore warnings); final scoped ESLint includes the added Electron test, PASS. +- Full initial unit run: 2,133 passed / 2 skipped / 1 failed (unchanged pi-managed-worker-opener test expects runtime-v6, unchanged runtime is v7). Subsequent broad run excluding only that stale test file and isolated pressure: 244 files / 2,136 passed / 2 skipped; pressure 1/1 PASS. Later additions tested in focused runs; no false full-clean claim. +- Vite Renderer/Main/Preload/utility build PASS. Artifact verifier now requires game-audio ID and hosted route; artifact contract tests PASS. No Windows installer or signed app.asar was produced, so packaged GA-04 acceptance remains open. +- Actual built Electron synthetic-audio test PASS 1/1 (2.5s): saved result→expand process/tool→explicit preview→native audio duration 0.25s and paused/no autoplay; screenshot visually inspected. It mocks the existing Host seam with a synthetic PCM WAV, not Meowa generation or live billing. First run only needed opening the existing collapsed process group in the test. +- Existing plugin-marketplace/project-plugins Electron diagnostics 4/4 FAIL on stale assertions (removed introductory copy, old local/bundled buttons, old scope=project URL). Those test files and Plugins/ProjectConfiguration product modules have no task diff; current base uses 可选/已安装 tabs and scope=all. These failures are recorded, not hidden or repaired as part of audio. + +## Follow-ups + +- GA-00/GA-04 HOLD: real Meowa final result shape, MIME/size/count/expiry/charge and PostgreSQL concurrency not verified. No new fee/rate invented and no live generation, publication, install, push, deploy or merge. +- A real accepted generation using the paired server and packaged client still requires separately bounded paid-canary authorization, plus actual price activation by Operations. The playable synthetic Electron fixture is not a substitute. +- Unrelated base runtime-v6 and old plugin-page E2E assertions need their own maintenance task; no source reset/stash/clean or root worktree writes performed. + +## Promotion Candidates + +- Target: current-state/module-map/domain rules at integration. Add project-wide Game Audio >=2.0.1, single confirmation→internal polling→all-file auto-save, separate job/delivery/billing phases and receipt-owned local preview. Evidence: public seams, actual built Electron test and paired server record. No conflict with image delivery, Data Service or selected-model Web Search; shared IDs and protocol remain unchanged. +- Target: evidence/commitments. Preserve real-provider/PG/packaged activation holds and exact baseline diagnostic failures. Human confirmation required for paid canary/activation; no canonical promotion in this feature worktree. diff --git a/README.md b/README.md index 98a9cc1..d84305a 100644 --- a/README.md +++ b/README.md @@ -123,6 +123,8 @@ Pi 正式包必须继续运行 `pnpm run verify:artifact:pi`、`pnpm run smoke:p ### 插件与 Skill +- 游戏音频生成（`makelore.game-audio`，客户端 2.0.1+）是独立的官方项目级 bundled 插件，获取并启用到项目后供 parent 使用，不需伙伴分配。包含完整音乐、明确请求的音乐预览、单条/成组/变体音效和取消；一次费用与项目写入确认后，Main 内部查询并将全部音频保存到原项目 `assets/generated/game-audio//`。结果卡点击后才加载本地试听，不自动播放。userData 收据支持原账号启动恢复、网络恢复和只补缺失文件，不重复提交生成；音乐完整/预览各按次计价，音效按每条每 0.5 秒计价。当前是离线合同实现：真实 Meowa 音频字段/媒体格式/收费验收与生产价格启用尚未完成，不应直接开放收费流量。 + - Makelore 不再打包独立的 `resources/coding-skills/` 或默认勾选本地 Skill。官方 Skill 作为 Marketplace 插件 Release 从云端目录下发，代码所有的特殊能力只存在于 `resources/coding-plugins/` 的插件包内；插件清单、已验证 Package Store Release 或明确安装的 Device Package 是 Skill 内容的唯一来源。 - Marketplace 官方插件 Skill 从已签名 Package Store Release 或受控内置插件定义动态物化；parent worker 的冻结 effective snapshot 必须满足对应账号、设备、项目、Policy，以及该插件确实要求时的智能体分配条件，child worker 不继承 hosted tool。所选模型的 Web Search 与 Conversation 安装的设备包使用各自独立的 Main-owned 合同，不属于这条 Marketplace 生命周期。 - `makelore.data-service`、`makelore.game-resource` 与 `makelore.project-scaffold` 是代码所有的官方项目级例外：满足各自既有账号获取或随应用提供条件并在项目启用后，其完整 Skill/tool 集合自动进入该项目的每个 parent Agent，不要求或展示伙伴分配；禁用项目插件后未来 worker 不再加载，既有 assignment 数据可原样保留但不参与生效判断。游戏资源生成只需一次计费与项目写入确认，Main 会在后台完成状态查询并把全部输出自动保存到发起操作时的原项目；本地交付续作不会重新生成或扣费。其他 Marketplace 插件继续按自身 assignment 规则计算有效资源。 diff --git a/electron/api/coding-composition.ts b/electron/api/coding-composition.ts index b66e32b..83d1f81 100644 --- a/electron/api/coding-composition.ts +++ b/electron/api/coding-composition.ts @@ -48,6 +48,10 @@ import { import { createDataServicePluginAdapter } from '../coding-plugins/adapters/data-service'; import { createGameResourcePluginAdapter } from '../coding-plugins/adapters/game-resource'; import { GameResourceClient } from '../services/game-resource-client'; +import { GameAudioClient } from '../services/game-audio-client'; +import { GameAudioDeliveryCoordinator, GameAudioReceiptStore } from '../services/game-audio-delivery'; +import { GameAudioPluginAdapter } from '../coding-plugins/adapters/game-audio'; +import { MarketplaceHostedAdmissionResolver } from '../coding-plugins/hosted-admission'; import { GameResourceDeliveryCoordinator, GameResourceDeliveryReceiptStore, @@ -339,6 +343,27 @@ export function createCodingComposition( bundledReleases: CODE_OWNED_OPTIONAL_BUNDLED_RELEASES, }); const policyClient = options.policyClient ?? new PluginPolicyClient(); + const gameAudioClient = new GameAudioClient(); + const gameAudioDelivery = new GameAudioDeliveryCoordinator({ + client: gameAudioClient, + receipts: new GameAudioReceiptStore(path.join(options.paths.userDataDir, 'coding-runtime', 'game-audio', 'receipts.json')), + leases: projectWriteLeases, + recordTouchedPaths: (conversationId, runId, paths) => productTools.recordTouchedPaths(conversationId, runId, paths), + ...(options.acquireBackgroundLease ? { acquireBackgroundLease: options.acquireBackgroundLease } : {}), + }); + const gameAudioAdapter = new GameAudioPluginAdapter({ + client: gameAudioClient, + delivery: gameAudioDelivery, + admission: new MarketplaceHostedAdmissionResolver({ + marketplace: marketplaceClient, packageStore, + makeloreVersion: options.clientVersion ?? '2.0.1', + bundledReleases: CODE_OWNED_OPTIONAL_BUNDLED_RELEASES, + }), + getPricingVersion: () => { + const state = policyClient.getState(); + return state.status === 'current' ? state.catalog?.pricing_version?.version_id ?? null : null; + }, + }); const knownPluginIds = new Set(pluginDefinitions.map(({ id }) => id)); // Existing user Releases are discovered from the device index at startup; // unlike the bundled catalog they cannot be enumerated synchronously. Keep @@ -373,7 +398,7 @@ export function createCodingComposition( const capabilityRegistry = createCodingCapabilityRegistry({ policyClient, projectPlugins, - adapters: [dataServiceAdapter, gameResourceAdapter], + adapters: [dataServiceAdapter, gameResourceAdapter, gameAudioAdapter], definitions: pluginDefinitions, effectiveResolver, getDurableProjectId: async (projectPath, localProjectId) => { @@ -389,7 +414,7 @@ export function createCodingComposition( projects, projectPlugins, policyClient, - adapters: [dataServiceAdapter, gameResourceAdapter], + adapters: [dataServiceAdapter, gameResourceAdapter, gameAudioAdapter], definitions: pluginDefinitions, effectiveResolver, getDefinitions: async () => { @@ -505,9 +530,11 @@ export function createCodingComposition( }, }); void gameResourceDelivery.resumePending().catch(() => undefined); + void gameAudioDelivery.resumePending().catch(() => undefined); const unsubscribeMarketplaceSession = subscribeWorksSquareSession(() => { void invalidateManagedResources(); void gameResourceDelivery.resumePending().catch(() => undefined); + void gameAudioDelivery.resumePending().catch(() => undefined); }); const host = createCodingProductHost({ projects, @@ -526,6 +553,7 @@ export function createCodingComposition( }) : () => undefined; return { + gameAudio: gameAudioDelivery, attachments, dataService, devicePackages: devicePackageManager, @@ -557,6 +585,7 @@ export function createCodingComposition( async shutdown() { conversations.dispose(); gameResourceDelivery.dispose(); + gameAudioDelivery.dispose(); previewDataSession?.dispose(); if (typeof options.browser.configurePreviewDataSession === 'function') { options.browser.configurePreviewDataSession(undefined); diff --git a/electron/api/coding-product-services.ts b/electron/api/coding-product-services.ts index a0ef6e9..098bf88 100644 --- a/electron/api/coding-product-services.ts +++ b/electron/api/coding-product-services.ts @@ -48,6 +48,7 @@ import type { import type { MarketplaceLibrarySnapshot } from '../coding-plugins/account-plugin-cache'; import type { EffectivePluginResolver } from '../coding-plugins/effective-resolver'; import type { DevicePackageManager } from '../coding-packages/device-package-manager'; +import type { GameAudioDeliveryCoordinator } from '../services/game-audio-delivery'; export interface ActiveCodingProject { id: string; @@ -65,6 +66,7 @@ export interface CodingProductHost { } export interface CodingProductComposition { + gameAudio?: Pick; attachments: CodingAttachmentStore; dataService: DataServiceOperations; devicePackages: DevicePackageManager; diff --git a/electron/api/routes/coding-plugins.ts b/electron/api/routes/coding-plugins.ts index 41127f8..ccbc266 100644 --- a/electron/api/routes/coding-plugins.ts +++ b/electron/api/routes/coding-plugins.ts @@ -4,6 +4,7 @@ import { CodingProjectServiceError } from '../../coding-projects/project-service import type { HostApiContext } from '../context'; import { parseJsonBody, sendJson } from '../route-utils'; import { CodingProjectPluginServiceError } from '../coding-product-services'; +import { GameAudioClientError } from '../../services/game-audio-client'; const PLUGIN_ROUTE = /^\/api\/coding\/plugins\/([^/]+)$/u; const PLUGIN_ID_PATTERN = /^[a-z][a-z0-9.-]{0,127}$/u; @@ -96,6 +97,20 @@ export async function handleCodingPluginRoutes( url: URL, ctx: HostApiContext, ): Promise { + const savedAudio = /^\/api\/coding\/game-audio\/([0-9a-f-]{36})\/outputs\/([0-9])$/u.exec(url.pathname); + if (savedAudio && req.method === 'GET') { + try { + const delivery = ctx.codingProducts?.gameAudio; + if (!delivery) throw new CodingPluginRouteError(503, 'game_audio_unavailable', 'Audio delivery is unavailable'); + if (url.search) requestError('Saved audio lookup does not accept query parameters'); + sendJson(res, 200, await delivery.readSavedOutput(savedAudio[1], Number(savedAudio[2]))); + } catch (error) { + if (error instanceof GameAudioClientError) { + sendJson(res, error.status, { code: error.code, error: error.message }); + } else sendError(res, error); + } + return true; + } if (url.pathname !== '/api/coding/plugins' && !PLUGIN_ROUTE.test(url.pathname)) return false; const plugins = ctx.codingProducts?.plugins; if (!plugins) { diff --git a/electron/coding-plugins/adapters/game-audio.ts b/electron/coding-plugins/adapters/game-audio.ts new file mode 100644 index 0000000..70c07b6 --- /dev/null +++ b/electron/coding-plugins/adapters/game-audio.ts @@ -0,0 +1,119 @@ +import { readFile, stat } from 'node:fs/promises'; +import path from 'node:path'; +import type { CodingPluginToolDefinition } from '../../../shared/coding-plugins'; +import type { MarketplaceHostedAdmissionResolver } from '../hosted-admission'; +import { MarketplaceHostedAdmissionError } from '../hosted-admission'; +import type { AdapterInvocationResult, CodingPluginAdapter, TrustedCodingCapabilityContext } from '../registry'; +import { audioRecord, GameAudioClientError, type AudioCommand, type GameAudioClient } from '../../services/game-audio-client'; +import type { AudioDeliveryResult, GameAudioDeliveryCoordinator } from '../../services/game-audio-delivery'; +import { getWorksSquareAccountBinding } from '../../services/works-square-session'; + +interface Options { + client: Pick; + delivery: Pick; + admission: Pick; + getPricingVersion: () => string | null; + getAccountKey?: () => string | null; +} +function invalid(): never { throw new GameAudioClientError('plugin_input_invalid', 422, 'Audio generation input is invalid'); } +function result(value: AudioDeliveryResult, projectId: string): AdapterInvocationResult { + const { billing, executionId, progressPercent, ...remaining } = value; + const data = { ...remaining, projectId, + ...(executionId === null ? {} : { executionId }), + ...(progressPercent === null ? {} : { progressPercent }) }; + if (value.providerStatus === 'failed') return { + success: false, status: 503, code: value.errorCode ?? 'game_audio_generation_failed', + error: 'Audio generation failed; inspect the billing receipt before a new attempt', + retryable: false, payload_schema: 'game-audio.v1', data: null, billing, + }; + return { success: true, status: value.deliveryStatus === 'saved' ? 200 : 202, + code: null, error: null, + retryable: false, payload_schema: 'game-audio.v1', data, billing }; +} +async function references(project: string, value: unknown) { + if (value === undefined) return []; + if (!Array.isArray(value) || value.length > 4) invalid(); + const items: { name: string; mime_type: string; data_base64: string }[] = []; + let total = 0; + for (const candidate of value) { + if (typeof candidate !== 'string' || !candidate || candidate.length > 1024) invalid(); + const absolute = path.resolve(project, candidate); + const relative = path.relative(path.resolve(project), absolute); + if (!relative || relative === '..' || relative.startsWith('..' + path.sep) || path.isAbsolute(relative)) invalid(); + const mime = ({ '.png': 'image/png', '.jpg': 'image/jpeg', '.jpeg': 'image/jpeg', '.webp': 'image/webp' } as Record)[path.extname(absolute).toLowerCase()]; + const metadata = await stat(absolute); + if (!mime || !metadata.isFile() || metadata.size <= 0 || metadata.size > 8 * 1024 * 1024) invalid(); + const content = await readFile(absolute); + total += content.byteLength; + if (!content.length || content.length > 8 * 1024 * 1024 || total > 16 * 1024 * 1024) invalid(); + items.push({ name: path.basename(absolute).slice(0, 160), mime_type: mime, data_base64: content.toString('base64') }); + } + return items; +} +export class GameAudioPluginAdapter implements CodingPluginAdapter { + readonly pluginId = 'makelore.game-audio'; + constructor(private readonly options: Options) {} + async inspect() { return { status: 'ready' as const }; } + async invoke(context: TrustedCodingCapabilityContext, tool: CodingPluginToolDefinition, + input: unknown, onProgress?: (value: AdapterInvocationResult) => void): Promise { + let submitted = false; + try { + if (!audioRecord(input)) invalid(); + const account = this.options.getAccountKey ? this.options.getAccountKey() : getWorksSquareAccountBinding()?.accountKey; + if (!account) throw new GameAudioClientError('authentication_required', 401, 'Sign-in is required'); + if (tool.name === 'game_audio_cancel') { + if (typeof input.executionId !== 'string' || Object.keys(input).some((key) => key !== 'executionId')) invalid(); + const generation = await this.options.client.cancel(input.executionId, account); + return { success: true, status: 200, code: null, error: null, retryable: false, + payload_schema: 'game-audio.v1', data: { executionId: generation.execution_id, + providerStatus: generation.provider_status, outputCount: generation.outputs.length, + generationBilling: Object.fromEntries(Object.entries(generation.billing).filter(([, value]) => value !== null)) } }; + } + if (input.confirmed !== true) throw new GameAudioClientError('confirmation_required', 400, 'Confirm Token Point generation and automatic project save first'); + const sound = tool.name === 'game_sound_generate'; + if (!sound && tool.name !== 'game_music_generate' && tool.name !== 'game_music_preview') invalid(); + const allowed = sound ? ['prompt', 'confirmed', 'mode', 'duration', 'count', 'loop'] : ['prompt', 'confirmed', 'referencePaths']; + if (Object.keys(input).some((key) => !allowed.includes(key)) || typeof input.prompt !== 'string' + || !input.prompt.trim() || input.prompt.trim().length > 4000) invalid(); + const pricing = this.options.getPricingVersion(); + if (!pricing) throw new GameAudioClientError('plugin_billing_unavailable', 503, 'Current audio pricing is unavailable'); + let specification: Pick, 'kind' | 'mode' | 'duration' | 'count' | 'loop'> + | Pick, 'kind' | 'mode' | 'reference_files'>; + if (sound) { + const mode = input.mode ?? 'single'; + const duration = input.duration ?? 2; + const count = input.count ?? (mode === 'single' ? 1 : 4); + const loop = input.loop ?? false; + if (!['single', 'pack', 'variants'].includes(String(mode)) || typeof mode !== 'string' + || typeof duration !== 'number' || !(duration === 0.5 || Number.isInteger(duration) && duration >= 1 && duration <= 10) + || typeof count !== 'number' || !Number.isInteger(count) || count < 1 || count > 10 + || (mode === 'single' && count !== 1) || typeof loop !== 'boolean') invalid(); + specification = { kind: 'sound', mode: mode as 'single' | 'pack' | 'variants', duration, count, loop }; + } else { + specification = { kind: 'music', mode: tool.name === 'game_music_preview' ? 'demo' : 'full', + reference_files: await references(context.projectPath, input.referencePaths) }; + } + const admission = await this.options.admission.resolve({ pluginId: this.pluginId, + workerSnapshot: { requestId: context.requestId, pluginReleaseId: context.pluginReleaseId } }); + submitted = true; + const delivered = await this.options.delivery.generateAndMaterialize({ + accountKey: account, conversationId: context.conversationId, runId: context.runId, + localProjectId: context.localProjectId, durableProjectId: context.durableProjectId, + projectPath: context.projectPath, logicalOperationId: context.requestId, + }, { ...specification, release_id: admission.releaseId, release_admission_id: admission.releaseAdmissionId, + project_id: context.durableProjectId, logical_operation_id: context.requestId, + pricing_version_id: pricing, prompt: input.prompt.trim(), confirmed: true }, + (progress) => onProgress?.(result(progress, context.localProjectId))); + return result(delivered, context.localProjectId); + } catch (error) { + const known = error instanceof GameAudioClientError || error instanceof MarketplaceHostedAdmissionError; + return { success: false, status: known ? error.status : 503, + code: known ? error.code : 'plugin_backend_unavailable', + error: known ? error.message : submitted + ? 'Audio receipt is unavailable; recover the original operation instead of generating again' + : 'Audio service is unavailable; no new generation was started', + retryable: false, payload_schema: 'game-audio.v1', data: null, + billing: { mode: 'platform_metered', status: submitted ? 'receipt_unavailable' : 'not_started' } }; + } + } +} diff --git a/electron/coding-plugins/manifest.ts b/electron/coding-plugins/manifest.ts index 1ace1ba..1bfce8a 100644 --- a/electron/coding-plugins/manifest.ts +++ b/electron/coding-plugins/manifest.ts @@ -12,6 +12,8 @@ import { CODE_OWNED_PLUGIN_PERMISSION_IDS, GAME_RESOURCE_BUNDLED_RELEASE_ID, GAME_RESOURCE_PLUGIN_ID, + GAME_AUDIO_PLUGIN_ID, + GAME_AUDIO_BUNDLED_RELEASE_ID, PROJECT_SCAFFOLD_BUNDLED_RELEASE_ID, PROJECT_SCAFFOLD_PLUGIN_ID, DATA_SERVICE_CAPABILITY_IDS, @@ -38,9 +40,17 @@ export const BUNDLED_CODING_PLUGIN_ROOTS = Object.freeze([ 'data-service', 'game-resource', 'project-scaffold', + 'game-audio', ] as const); const BUNDLED_CODING_PLUGIN_METADATA = Object.freeze({ + 'game-audio': Object.freeze({ + pluginId: GAME_AUDIO_PLUGIN_ID, + runtimeKind: 'platform_hosted' as const, + acquisitionMode: 'user_acquired' as const, + releaseId: GAME_AUDIO_BUNDLED_RELEASE_ID, + bundledV2: true, + }), 'data-service': Object.freeze({ pluginId: DATA_SERVICE_PLUGIN_ID, runtimeKind: 'bundled_typed' as const, @@ -1182,9 +1192,10 @@ function validateBundledDefinitions(definitions: readonly CodingPluginDefinition } for (const tool of definition.tools) { if (toolIds.has(tool.name)) throw new Error(`Duplicate bundled tool identifier: ${tool.name}`); - if (operationIds.has(tool.operation)) throw new Error(`Duplicate bundled operation identifier: ${tool.operation}`); + const operationKey = `${definition.id}:${tool.capabilityId}:${tool.operation}`; + if (operationIds.has(operationKey)) throw new Error(`Duplicate bundled operation identifier: ${operationKey}`); toolIds.add(tool.name); - operationIds.add(tool.operation); + operationIds.add(operationKey); capabilityIds.add(tool.capabilityId); } } diff --git a/electron/coding-projects/game-asset-browser.ts b/electron/coding-projects/game-asset-browser.ts index af643c6..ce94b04 100644 --- a/electron/coding-projects/game-asset-browser.ts +++ b/electron/coding-projects/game-asset-browser.ts @@ -145,7 +145,7 @@ async function resolveInsideProject(projectPath: string, candidatePath: string): } } -async function readMediaDataUrl(projectPath: string, candidatePath: string, maxBytes: number): Promise { +export async function readMediaDataUrl(projectPath: string, candidatePath: string, maxBytes: number): Promise { const resolved = await resolveInsideProject(projectPath, candidatePath); if (!resolved) return null; const fileStat = await stat(resolved); diff --git a/electron/services/game-audio-client.ts b/electron/services/game-audio-client.ts new file mode 100644 index 0000000..75878c4 --- /dev/null +++ b/electron/services/game-audio-client.ts @@ -0,0 +1,233 @@ +import type { CapabilityBillingReceiptV1 } from '../../shared/data-service'; +import { WORKS_SQUARE_CONFIG } from '../api/works-config'; +import { proxyAwareFetch } from '../utils/proxy-fetch'; +import { getValidWorksSquareAccessToken, getWorksSquareAccountBinding } from './works-square-session'; + +const ROOT = '/api/plugins/v1/hosted/game-audio/generations'; +const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/u; +const DECIMAL = /^(?:0|[1-9]\d*)\.\d{2}$/u; +const AUDIO_EXTENSIONS: Readonly> = { + 'audio/mpeg': 'mp3', 'audio/wav': 'wav', 'audio/x-wav': 'wav', + 'audio/ogg': 'ogg', 'audio/flac': 'flac', 'audio/mp4': 'm4a', +}; +export type AudioBilling = Extract; +export interface AudioGeneration { + schema_version: 1; + plugin_id: 'makelore.game-audio'; + execution_id: string; + release_id: string; + project_id: string; + logical_operation_id: string; + kind: 'music' | 'sound'; + provider_status: 'accepted' | 'running' | 'succeeded' | 'failed' | 'cancelled' | 'submission_unknown'; + output_ready: boolean; + outputs: { index: number; name: string }[]; + progress_percent: number | null; + poll_interval_seconds: number; + error_code: string | null; + billing: AudioBilling; +} +interface AudioIdentity { + release_id: string; + release_admission_id: string; + project_id: string; + logical_operation_id: string; + pricing_version_id: string; + prompt: string; + confirmed: true; +} +export type AudioCommand = AudioIdentity & ({ + kind: 'music'; mode: 'full' | 'demo'; + reference_files?: { name: string; mime_type: string; data_base64: string }[]; +} | { + kind: 'sound'; mode: 'single' | 'pack' | 'variants'; + duration: number; count: number; loop: boolean; +}); +export class GameAudioClientError extends Error { + constructor(readonly code: string, readonly status: number, message: string) { + super(message); + this.name = 'GameAudioClientError'; + } +} +export function audioRecord(value: unknown): value is Record { + return value !== null && typeof value === 'object' && !Array.isArray(value); +} +function invalid(): never { + throw new GameAudioClientError('plugin_backend_invalid', 502, 'Audio response is invalid'); +} +function bounded(value: unknown, max: number): value is string { + return typeof value === 'string' && value.length > 0 && value.length <= max; +} +function integer(value: unknown, min: number, max: number): value is number { + return typeof value === 'number' && Number.isSafeInteger(value) && value >= min && value <= max; +} +export function parseAudioGeneration(value: unknown): AudioGeneration { + if (!audioRecord(value) || value.schema_version !== 1 || value.plugin_id !== 'makelore.game-audio' + || !bounded(value.execution_id, 36) || !UUID.test(value.execution_id) + || !bounded(value.release_id, 36) || !UUID.test(value.release_id) + || !bounded(value.project_id, 36) || !UUID.test(value.project_id) + || !bounded(value.logical_operation_id, 128) + || !['music', 'sound'].includes(String(value.kind)) + || !['accepted', 'running', 'succeeded', 'failed', 'cancelled', 'submission_unknown'].includes(String(value.provider_status)) + || typeof value.output_ready !== 'boolean' + || !Array.isArray(value.outputs) || value.outputs.length > 10 + || !value.outputs.every((item, index) => audioRecord(item) + && Object.keys(item).every((key) => ['index', 'name'].includes(key)) + && item.index === index && bounded(item.name, 160)) + || value.output_ready !== (value.outputs.length > 0) + || (value.output_ready && value.provider_status !== 'succeeded') + || !integer(value.poll_interval_seconds, 1, 300) + || (value.progress_percent !== null && !integer(value.progress_percent, 0, 100)) + || (value.error_code !== null && !bounded(value.error_code, 100))) invalid(); + const bill = value.billing; + if (!audioRecord(bill) || bill.mode !== 'platform_metered' + || !['reserved', 'dispatched', 'settled', 'released', 'expired', 'pending_review', 'refunded'].includes(String(bill.status)) + || typeof bill.reserved_points !== 'string' || !DECIMAL.test(bill.reserved_points) + || (bill.actual_points != null && (typeof bill.actual_points !== 'string' || !DECIMAL.test(bill.actual_points))) + || !integer(bill.usage_amount, 0, 200) + || bill.unit !== (value.kind === 'sound' ? 'half_second' : 'generation') + || (['settled', 'refunded'].includes(String(bill.status)) && bill.actual_points == null) + || (['accepted', 'running'].includes(String(value.provider_status)) && !['reserved', 'dispatched'].includes(String(bill.status))) + || (value.provider_status === 'succeeded' && !['settled', 'pending_review', 'refunded'].includes(String(bill.status))) + || (value.provider_status === 'submission_unknown' && bill.status !== 'pending_review') + || (['failed', 'cancelled'].includes(String(value.provider_status)) && !['released', 'expired', 'refunded', 'pending_review'].includes(String(bill.status)))) invalid(); + const fields = ['schema_version', 'plugin_id', 'execution_id', 'release_id', 'project_id', + 'logical_operation_id', 'kind', 'provider_status', 'output_ready', 'outputs', + 'progress_percent', 'poll_interval_seconds', 'error_code', 'billing']; + if (Object.keys(value).some((key) => !fields.includes(key)) + || Object.keys(bill).some((key) => !['mode', 'status', 'reserved_points', 'actual_points', 'usage_amount', 'unit'].includes(key))) invalid(); + // Hosted JSON uses null; shared capability receipts use an absent optional amount. + return { ...value, billing: Object.fromEntries(Object.entries(bill).filter(([, field]) => field !== null)) } as unknown as AudioGeneration; +} +async function bytes(response: Response, limit: number): Promise { + if (Number(response.headers.get('content-length')) > limit) { + await response.body?.cancel(); + invalid(); + } + const reader = response.body?.getReader(); + if (!reader) invalid(); + const chunks: Uint8Array[] = []; + let total = 0; + try { + while (true) { + const next = await reader.read(); + if (next.done) break; + total += next.value.byteLength; + if (total > limit) invalid(); + chunks.push(next.value); + } + } finally { + await reader.cancel().catch(() => undefined); + reader.releaseLock(); + } + const result = new Uint8Array(total); + let offset = 0; + for (const chunk of chunks) { result.set(chunk, offset); offset += chunk.byteLength; } + return result; +} +export interface GameAudioClientOptions { + fetchImpl?: typeof fetch; + getAccessToken?: typeof getValidWorksSquareAccessToken; + getAccountBinding?: typeof getWorksSquareAccountBinding; + apiBaseUrl?: string; +} +export class GameAudioClient { + constructor(private readonly options: GameAudioClientOptions = {}) {} + + async generate(command: AudioCommand, account: string): Promise { + const result = await this.json(ROOT, account, 'POST', command); + if (result.project_id !== command.project_id || result.release_id !== command.release_id + || result.logical_operation_id !== command.logical_operation_id || result.kind !== command.kind) invalid(); + return result; + } + async get(id: string, account: string): Promise { + if (!UUID.test(id)) invalid(); + const result = await this.json(ROOT + '/' + id, account); + if (result.execution_id !== id) invalid(); + return result; + } + async byOperation(id: string, account: string): Promise { + if (!bounded(id, 128)) invalid(); + const result = await this.json(ROOT + '/by-operation/' + encodeURIComponent(id), account); + if (result.logical_operation_id !== id) invalid(); + return result; + } + async cancel(id: string, account: string): Promise { + if (!UUID.test(id)) invalid(); + const result = await this.json(ROOT + '/' + id + '/cancel', account, 'POST'); + if (result.execution_id !== id) invalid(); + return result; + } + async download(id: string, index: number, account: string) { + if (!UUID.test(id) || !integer(index, 0, 9)) invalid(); + return await this.request(ROOT + '/' + id + '/content?output_index=' + index, account, 'GET', undefined, async (response) => { + const mime = response.headers.get('content-type')?.split(';', 1)[0]?.trim().toLowerCase() ?? ''; + const extension = AUDIO_EXTENSIONS[mime]; + if (!extension) { + await response.body?.cancel().catch(() => undefined); + throw new GameAudioClientError('game_audio_content_invalid', 502, 'Unsupported audio media'); + } + const content = await bytes(response, 32 * 1024 * 1024); + if (content.length === 0) invalid(); + return { bytes: content, extension, mimeType: mime }; + }); + } + private async json(route: string, account: string, method = 'GET', command?: AudioCommand) { + return await this.request(route, account, method, command, async (response) => { + const content = await bytes(response, 1024 * 1024); + let value: unknown; + try { value = JSON.parse(new TextDecoder().decode(content)); } catch { invalid(); } + return parseAudioGeneration(value); + }); + } + private async request( + route: string, account: string, method: string, command: AudioCommand | undefined, + consume: (response: Response) => Promise, + ): Promise { + const binding = this.options.getAccountBinding ?? getWorksSquareAccountBinding; + const original = binding(); + const assertAccount = () => { + const current = binding(); + if (!original || original.accountKey !== account || current?.accountKey !== account || current.epoch !== original.epoch) { + throw new GameAudioClientError('plugin_account_changed', 409, 'Original signed-in account is required'); + } + }; + assertAccount(); + const fetchImpl = this.options.fetchImpl ?? proxyAwareFetch; + const getToken = this.options.getAccessToken ?? getValidWorksSquareAccessToken; + const body = command ? JSON.stringify(command) : undefined; + if (body && Buffer.byteLength(body) > 24 * 1024 * 1024) invalid(); + const controller = new AbortController(); + const timeout = setTimeout(() => controller.abort(), method === 'POST' ? 90_000 : 40_000); + try { + for (let attempt = 0; attempt < 2; attempt++) { + const token = await getToken({ fetchImpl, ...(attempt ? { forceRefresh: true } : {}) }); + assertAccount(); + if (!token) throw new GameAudioClientError('authentication_required', 401, 'Sign-in is required'); + const response = await fetchImpl((this.options.apiBaseUrl ?? WORKS_SQUARE_CONFIG.apiBaseUrl).replace(/\/$/u, '') + route, { + method, redirect: 'manual', signal: controller.signal, + headers: { Accept: 'application/json', Authorization: 'Bearer ' + token, + ...(body ? { 'Content-Type': 'application/json' } : {}) }, + ...(body ? { body } : {}), + }); + assertAccount(); + if (response.status === 401 && attempt === 0) { await response.body?.cancel(); continue; } + if (!response.ok) { + let code = response.status === 429 ? 'plugin_rate_limited' : 'plugin_backend_unavailable'; + try { + const error: unknown = JSON.parse(new TextDecoder().decode(await bytes(response, 1024 * 1024))); + if (response.status !== 429 && audioRecord(error) && audioRecord(error.detail) + && bounded(error.detail.error_code, 100)) code = error.detail.error_code; + } catch { /* HTTP status still owns the failure even if the error body is invalid. */ } + throw new GameAudioClientError(code, response.status, 'Audio request failed (' + response.status + ')'); + } + const value = await consume(response); + assertAccount(); + return value; + } + throw new GameAudioClientError('authentication_required', 401, 'Sign-in is required'); + } finally { + clearTimeout(timeout); + } + } +} diff --git a/electron/services/game-audio-delivery.ts b/electron/services/game-audio-delivery.ts new file mode 100644 index 0000000..6116074 --- /dev/null +++ b/electron/services/game-audio-delivery.ts @@ -0,0 +1,268 @@ +import { mkdir, readFile, stat, writeFile } from 'node:fs/promises'; +import path from 'node:path'; +import { setTimeout as delay } from 'node:timers/promises'; +import { atomicWriteJson, readJsonFile } from '../coding-projects/atomic-json'; +import type { PiProjectWriteLeaseCoordinator } from '../coding-runtime/pi/write-lease'; +import { audioRecord, GameAudioClientError, parseAudioGeneration, + type AudioCommand, type AudioGeneration, type GameAudioClient, type AudioBilling } from './game-audio-client'; +import { getWorksSquareAccountBinding } from './works-square-session'; +import { readMediaDataUrl } from '../coding-projects/game-asset-browser'; + +export interface AudioDeliveryContext { + accountKey: string; + conversationId: string; + runId: string; + localProjectId: string; + durableProjectId: string; + projectPath: string; + logicalOperationId: string; +} +export interface AudioDeliveredFile { index: number; path: string; bytes: number; mimeType: string } +export interface AudioDeliveryResult { + executionId: string | null; + logicalOperationId: string; + providerStatus: AudioGeneration['provider_status']; + deliveryStatus: 'waiting' | 'saving' | 'saved' | 'delivery_failed' | 'needs_reconciliation'; + outputCount: number; + files: AudioDeliveredFile[]; + billing: AudioBilling; + phase: 'submitted' | 'generating' | 'saving' | 'saved' | 'paused'; + progressPercent: number | null; + errorCode?: string; +} +interface Receipt { + context: AudioDeliveryContext; + generation: AudioGeneration | null; + deliveryStatus: AudioDeliveryResult['deliveryStatus']; + files: AudioDeliveredFile[]; + errorCode?: string; + submissionRejected?: boolean; +} +function sameIdentity(left: AudioDeliveryContext, right: AudioDeliveryContext) { + return left.accountKey === right.accountKey && left.logicalOperationId === right.logicalOperationId; +} +function validReceipt(value: unknown): value is Receipt { + if (!audioRecord(value) || !audioRecord(value.context) + || !['accountKey', 'conversationId', 'runId', 'localProjectId', 'durableProjectId', 'projectPath', 'logicalOperationId'] + .every((key) => typeof value.context[key] === 'string' && value.context[key].length > 0) + || !path.isAbsolute(String(value.context.projectPath)) + || !['waiting', 'saving', 'saved', 'delivery_failed', 'needs_reconciliation'].includes(String(value.deliveryStatus)) + || !Array.isArray(value.files) || value.files.length > 10) return false; + try { + if (value.generation !== null) parseAudioGeneration(value.generation); + } catch { return false; } + return value.files.every((file) => audioRecord(file) && Number.isInteger(file.index) + && typeof file.bytes === 'number' && file.bytes > 0 && typeof file.path === 'string' + && /^assets\/generated\/game-audio\/[0-9a-f-]{36}\/(?:music|sound)-\d+\.(?:mp3|wav|ogg|flac|m4a)$/u.test(file.path)); +} +/** One atomic userData file; no project file serves as the job database. */ +export class GameAudioReceiptStore { + private writes: Promise = Promise.resolve(); + constructor(private readonly filePath: string) {} + async list(): Promise { + let value: unknown; + try { value = await readJsonFile(this.filePath); } catch (error) { + if (audioRecord(error) && error.code === 'ENOENT') return []; + throw error; + } + if (!audioRecord(value) || value.schemaVersion !== 1 || !Array.isArray(value.receipts) + || !value.receipts.every(validReceipt)) throw new Error('Audio receipt file is invalid'); + return value.receipts; + } + async put(receipt: Receipt): Promise { + const operation = this.writes.then(async () => { + const rows = (await this.list()).filter((row) => !sameIdentity(row.context, receipt.context)); + rows.push(receipt); + await atomicWriteJson(this.filePath, { schemaVersion: 1, receipts: rows }); + }); + this.writes = operation.catch(() => undefined); + await operation; + } +} +interface Options { + client: Pick; + receipts: GameAudioReceiptStore; + leases: PiProjectWriteLeaseCoordinator; + getAccountKey?: () => string | null; + sleep?: (ms: number, signal?: AbortSignal) => Promise; + recordTouchedPaths?: (conversationId: string, runId: string, paths: readonly string[]) => Promise; + acquireBackgroundLease?: (lease: { id: string; kind: 'coding-run' }) => () => void; +} +type Progress = (value: AudioDeliveryResult) => void; +export class GameAudioDeliveryCoordinator { + private readonly flights = new Map>(); + private readonly shutdown = new AbortController(); + private recoveryTimer: ReturnType | undefined; + constructor(private readonly options: Options) {} + dispose(): void { + this.shutdown.abort(); + clearTimeout(this.recoveryTimer); + } + private scheduleRecovery(): void { + if (this.shutdown.signal.aborted || this.recoveryTimer) return; + this.recoveryTimer = setTimeout(() => { + this.recoveryTimer = undefined; + void this.resumePending().catch(() => this.scheduleRecovery()); + }, 30_000); + this.recoveryTimer.unref?.(); + } + async readSavedOutput(executionId: string, index: number): Promise<{ path: string; dataUrl: string }> { + const account = this.account(); + const receipt = (await this.options.receipts.list()).find((row) => + row.context.accountKey === account && row.generation?.execution_id === executionId); + const file = receipt?.files.find((item) => item.index === index); + if (!receipt || !file) throw new GameAudioClientError('game_audio_output_not_found', 404, 'Saved audio is unavailable'); + this.assertAccount(receipt.context); + const dataUrl = await readMediaDataUrl(receipt.context.projectPath, file.path, 32 * 1024 * 1024); + this.assertAccount(receipt.context); + if (!dataUrl?.startsWith('data:audio/')) throw new GameAudioClientError('game_audio_content_invalid', 422, 'Saved audio cannot be read'); + return { path: file.path, dataUrl }; + } + async generateAndMaterialize(context: AudioDeliveryContext, command: AudioCommand, progress?: Progress) { + return await this.singleFlight(context, async () => { + const old = (await this.options.receipts.list()).find((row) => sameIdentity(row.context, context)); + if (old) return await this.advance(old, progress); + this.assertAccount(context); + const receipt: Receipt = { context: { ...context }, generation: null, deliveryStatus: 'waiting', files: [] }; + // The durable identity must exist BEFORE the first possibly charged request. + await this.options.receipts.put(receipt); + try { + receipt.generation = await this.options.client.generate(command, context.accountKey); + this.checkGeneration(receipt); + await this.options.receipts.put(receipt); + } catch (error) { + if (error instanceof GameAudioClientError && (error.status === 401 || error.status === 422 + || error.code === 'plugin_pricing_changed')) { + receipt.deliveryStatus = 'needs_reconciliation'; + receipt.submissionRejected = true; + receipt.errorCode = error.code; + await this.options.receipts.put(receipt); + return this.result(receipt, 'paused'); + } + // Response loss is not permission to submit again. + } + return await this.advance(receipt, progress); + }); + } + async resumePending(): Promise { + const account = this.account(); + if (!account || this.shutdown.signal.aborted) return; + const rows = await this.options.receipts.list(); + await Promise.allSettled(rows.filter((row) => row.context.accountKey === account + && !row.submissionRejected + && row.deliveryStatus !== 'saved' + && !['failed', 'cancelled'].includes(row.generation?.provider_status ?? '')) + .map((row) => this.singleFlight(row.context, () => this.advance(row)))); + } + private async singleFlight(context: AudioDeliveryContext, run: () => Promise) { + const key = JSON.stringify([context.accountKey, context.logicalOperationId]); + const old = this.flights.get(key); + if (old) return await old; + const release = this.options.acquireBackgroundLease?.({ id: 'audio:' + key, kind: 'coding-run' }); + const flight = run().finally(() => { this.flights.delete(key); release?.(); }); + this.flights.set(key, flight); + return await flight; + } + private account(): string | null { + return this.options.getAccountKey ? this.options.getAccountKey() : getWorksSquareAccountBinding()?.accountKey ?? null; + } + private assertAccount(context: AudioDeliveryContext): void { + if (this.shutdown.signal.aborted || this.account() !== context.accountKey) { + throw new GameAudioClientError('plugin_account_changed', 409, 'Audio recovery waits for the original account'); + } + } + private checkGeneration(receipt: Receipt): void { + const job = receipt.generation; + if (job && (job.project_id !== receipt.context.durableProjectId + || job.logical_operation_id !== receipt.context.logicalOperationId)) { + throw new GameAudioClientError('plugin_backend_invalid', 502, 'Audio execution identity changed'); + } + } + private async sleep(ms: number): Promise { + if (this.options.sleep) await this.options.sleep(ms, this.shutdown.signal); + else await delay(ms, undefined, { signal: this.shutdown.signal }); + } + private async advance(receipt: Receipt, progress?: Progress): Promise { + if (receipt.submissionRejected) return this.result(receipt, 'paused'); + if (receipt.deliveryStatus === 'saved') return this.result(receipt, 'saved'); + const report = (phase: AudioDeliveryResult['phase']) => { + try { progress?.(this.result(receipt, phase)); } catch { /* Observational only. */ } + }; + try { + this.assertAccount(receipt.context); + let errors = 0; + while (true) { + this.assertAccount(receipt.context); + const job = receipt.generation; + if (job && !['accepted', 'running'].includes(job.provider_status)) break; + report(job ? 'generating' : 'submitted'); + try { + receipt.generation = job + ? await this.options.client.get(job.execution_id, receipt.context.accountKey) + : await this.options.client.byOperation(receipt.context.logicalOperationId, receipt.context.accountKey); + this.checkGeneration(receipt); + await this.options.receipts.put(receipt); + errors = 0; + } catch (error) { + errors++; + if (errors >= 3 || (error instanceof GameAudioClientError && error.code === 'plugin_account_changed')) throw error; + await this.sleep(1000 * errors); + continue; + } + if (['accepted', 'running'].includes(receipt.generation.provider_status)) { + await this.sleep(receipt.generation.poll_interval_seconds * 1000); + } + } + const job = receipt.generation; + if (!job?.output_ready) return this.result(receipt, 'paused'); + this.assertAccount(receipt.context); + const project = await stat(receipt.context.projectPath); + if (!project.isDirectory()) throw new Error('Original project is unavailable'); + receipt.deliveryStatus = 'saving'; + report('saving'); + for (const output of job.outputs) { + if (receipt.files.some((file) => file.index === output.index)) continue; + this.assertAccount(receipt.context); + const content = await this.options.client.download(job.execution_id, output.index, receipt.context.accountKey); + const relative = 'assets/generated/game-audio/' + job.execution_id + '/' + job.kind + '-' + (output.index + 1) + '.' + content.extension; + const target = path.join(receipt.context.projectPath, relative); + const lease = await this.options.leases.acquire(receipt.context.localProjectId, 'audio:' + job.execution_id, this.shutdown.signal); + try { + this.assertAccount(receipt.context); + await mkdir(path.dirname(target), { recursive: true }); + try { await writeFile(target, content.bytes, { flag: 'wx' }); } catch (error) { + if (!audioRecord(error) || error.code !== 'EEXIST') throw error; + const existing = await readFile(target); + if (!existing.equals(Buffer.from(content.bytes))) throw new Error('Audio target already contains a different file', { cause: error }); + } + receipt.files.push({ index: output.index, path: relative, bytes: content.bytes.byteLength, mimeType: content.mimeType }); + await this.options.receipts.put(receipt); + try { + await this.options.recordTouchedPaths?.(receipt.context.conversationId, receipt.context.runId, [relative]); + } catch { /* Restored runs may no longer have an in-memory change tracker. Saved paths remain durable. */ } + } finally { lease.release(); } + } + receipt.deliveryStatus = 'saved'; + delete receipt.errorCode; + await this.options.receipts.put(receipt); + report('saved'); + return this.result(receipt, 'saved'); + } catch (error) { + receipt.deliveryStatus = receipt.generation?.output_ready ? 'delivery_failed' : 'needs_reconciliation'; + receipt.errorCode = error instanceof GameAudioClientError ? error.code : 'game_audio_delivery_paused'; + await this.options.receipts.put(receipt); + report('paused'); + if (!(error instanceof GameAudioClientError && error.code === 'plugin_account_changed')) this.scheduleRecovery(); + return this.result(receipt, 'paused'); + } + } + private result(receipt: Receipt, phase: AudioDeliveryResult['phase']): AudioDeliveryResult { + const job = receipt.generation; + return { executionId: job?.execution_id ?? null, logicalOperationId: receipt.context.logicalOperationId, + providerStatus: job?.provider_status ?? (receipt.submissionRejected ? 'failed' : 'submission_unknown'), deliveryStatus: receipt.deliveryStatus, + outputCount: job?.outputs.length ?? 0, files: [...receipt.files], phase, + progressPercent: job?.progress_percent ?? null, + billing: job?.billing ?? { mode: 'platform_metered', status: receipt.submissionRejected ? 'not_started' : 'receipt_unavailable' }, + ...(receipt.errorCode || job?.error_code ? { errorCode: receipt.errorCode ?? job?.error_code ?? undefined } : {}) }; + } +} diff --git a/package.json b/package.json index 5d3e9f1..673da61 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "niancode", - "version": "2.0.0", + "version": "2.0.1", "pnpm": { "onlyBuiltDependencies": [ "@discordjs/opus", diff --git a/resources/coding-plugins/game-audio/com.makelore/capability.json b/resources/coding-plugins/game-audio/com.makelore/capability.json new file mode 100644 index 0000000..5b83367 --- /dev/null +++ b/resources/coding-plugins/game-audio/com.makelore/capability.json @@ -0,0 +1,545 @@ +{ + "schemaVersion": 2, + "pluginId": "makelore.game-audio", + "contractVersion": 1, + "scope": "project", + "runtime": { + "kind": "platform_hosted", + "protocol": "makelore-hosted.v1" + }, + "skills": [ + { + "id": "game-audio", + "entry": "../skills/game-audio/SKILL.md", + "grants": [ + "game-audio.music", + "game-audio.sound", + "game-audio.control" + ] + } + ], + "tools": [ + { + "name": "game_music_generate", + "label": "生成游戏音乐", + "description": "Generate audio after Token Point confirmation; internally wait and save all outputs to the original project.", + "capabilityId": "game-audio.music", + "operation": "generate", + "roles": [ + "parent" + ], + "mutation": "write", + "projectWriteLease": false, + "permissions": [ + "hosted.game-audio.game-music-generate" + ], + "executionMode": "job", + "inputSchema": { + "type": "object", + "additionalProperties": false, + "required": [ + "prompt", + "confirmed" + ], + "properties": { + "prompt": { + "type": "string", + "minLength": 1, + "maxLength": 4000 + }, + "confirmed": { + "type": "boolean" + }, + "referencePaths": { + "type": "array", + "maxItems": 4, + "items": { + "type": "string", + "minLength": 1, + "maxLength": 1024 + } + } + } + }, + "outputSchema": { + "type": "object", + "additionalProperties": false, + "required": [ + "logicalOperationId", + "providerStatus", + "deliveryStatus", + "outputCount", + "files", + "phase", + "projectId" + ], + "properties": { + "executionId": { + "type": "string", + "maxLength": 36 + }, + "logicalOperationId": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "providerStatus": { + "type": "string", + "minLength": 1, + "maxLength": 32 + }, + "deliveryStatus": { + "type": "string", + "minLength": 1, + "maxLength": 32 + }, + "outputCount": { + "type": "integer", + "minimum": 0, + "maximum": 10 + }, + "files": { + "type": "array", + "maxItems": 10, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "index", + "path", + "bytes", + "mimeType" + ], + "properties": { + "index": { + "type": "integer", + "minimum": 0, + "maximum": 9 + }, + "path": { + "type": "string", + "minLength": 1, + "maxLength": 1024 + }, + "bytes": { + "type": "integer", + "minimum": 1, + "maximum": 33554432 + }, + "mimeType": { + "type": "string", + "minLength": 1, + "maxLength": 40 + } + } + } + }, + "phase": { + "type": "string", + "minLength": 1, + "maxLength": 32 + }, + "progressPercent": { + "type": "integer", + "minimum": 0, + "maximum": 100 + }, + "errorCode": { + "type": "string", + "minLength": 1, + "maxLength": 100 + }, + "projectId": { + "type": "string", + "minLength": 1, + "maxLength": 128 + } + } + } + }, + { + "name": "game_music_preview", + "label": "生成音乐短预览", + "description": "Generate audio after Token Point confirmation; internally wait and save all outputs to the original project.", + "capabilityId": "game-audio.music", + "operation": "preview", + "roles": [ + "parent" + ], + "mutation": "write", + "projectWriteLease": false, + "permissions": [ + "hosted.game-audio.game-music-preview" + ], + "executionMode": "job", + "inputSchema": { + "type": "object", + "additionalProperties": false, + "required": [ + "prompt", + "confirmed" + ], + "properties": { + "prompt": { + "type": "string", + "minLength": 1, + "maxLength": 4000 + }, + "confirmed": { + "type": "boolean" + }, + "referencePaths": { + "type": "array", + "maxItems": 4, + "items": { + "type": "string", + "minLength": 1, + "maxLength": 1024 + } + } + } + }, + "outputSchema": { + "type": "object", + "additionalProperties": false, + "required": [ + "logicalOperationId", + "providerStatus", + "deliveryStatus", + "outputCount", + "files", + "phase", + "projectId" + ], + "properties": { + "executionId": { + "type": "string", + "maxLength": 36 + }, + "logicalOperationId": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "providerStatus": { + "type": "string", + "minLength": 1, + "maxLength": 32 + }, + "deliveryStatus": { + "type": "string", + "minLength": 1, + "maxLength": 32 + }, + "outputCount": { + "type": "integer", + "minimum": 0, + "maximum": 10 + }, + "files": { + "type": "array", + "maxItems": 10, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "index", + "path", + "bytes", + "mimeType" + ], + "properties": { + "index": { + "type": "integer", + "minimum": 0, + "maximum": 9 + }, + "path": { + "type": "string", + "minLength": 1, + "maxLength": 1024 + }, + "bytes": { + "type": "integer", + "minimum": 1, + "maximum": 33554432 + }, + "mimeType": { + "type": "string", + "minLength": 1, + "maxLength": 40 + } + } + } + }, + "phase": { + "type": "string", + "minLength": 1, + "maxLength": 32 + }, + "progressPercent": { + "type": "integer", + "minimum": 0, + "maximum": 100 + }, + "errorCode": { + "type": "string", + "minLength": 1, + "maxLength": 100 + }, + "projectId": { + "type": "string", + "minLength": 1, + "maxLength": 128 + } + } + } + }, + { + "name": "game_sound_generate", + "label": "生成游戏音效", + "description": "Generate audio after Token Point confirmation; internally wait and save all outputs to the original project.", + "capabilityId": "game-audio.sound", + "operation": "generate", + "roles": [ + "parent" + ], + "mutation": "write", + "projectWriteLease": false, + "permissions": [ + "hosted.game-audio.game-sound-generate" + ], + "executionMode": "job", + "inputSchema": { + "type": "object", + "additionalProperties": false, + "required": [ + "prompt", + "confirmed" + ], + "properties": { + "prompt": { + "type": "string", + "minLength": 1, + "maxLength": 4000 + }, + "confirmed": { + "type": "boolean" + }, + "mode": { + "type": "string", + "minLength": 1, + "maxLength": 16 + }, + "duration": { + "type": "number", + "minimum": 0.5, + "maximum": 10 + }, + "count": { + "type": "integer", + "minimum": 1, + "maximum": 10 + }, + "loop": { + "type": "boolean" + } + } + }, + "outputSchema": { + "type": "object", + "additionalProperties": false, + "required": [ + "logicalOperationId", + "providerStatus", + "deliveryStatus", + "outputCount", + "files", + "phase", + "projectId" + ], + "properties": { + "executionId": { + "type": "string", + "maxLength": 36 + }, + "logicalOperationId": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "providerStatus": { + "type": "string", + "minLength": 1, + "maxLength": 32 + }, + "deliveryStatus": { + "type": "string", + "minLength": 1, + "maxLength": 32 + }, + "outputCount": { + "type": "integer", + "minimum": 0, + "maximum": 10 + }, + "files": { + "type": "array", + "maxItems": 10, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "index", + "path", + "bytes", + "mimeType" + ], + "properties": { + "index": { + "type": "integer", + "minimum": 0, + "maximum": 9 + }, + "path": { + "type": "string", + "minLength": 1, + "maxLength": 1024 + }, + "bytes": { + "type": "integer", + "minimum": 1, + "maximum": 33554432 + }, + "mimeType": { + "type": "string", + "minLength": 1, + "maxLength": 40 + } + } + } + }, + "phase": { + "type": "string", + "minLength": 1, + "maxLength": 32 + }, + "progressPercent": { + "type": "integer", + "minimum": 0, + "maximum": 100 + }, + "errorCode": { + "type": "string", + "minLength": 1, + "maxLength": 100 + }, + "projectId": { + "type": "string", + "minLength": 1, + "maxLength": 128 + } + } + } + }, + { + "name": "game_audio_cancel", + "label": "取消游戏音频任务", + "description": "Cancel only an existing audio execution when the user requests it.", + "capabilityId": "game-audio.control", + "operation": "cancel", + "roles": [ + "parent" + ], + "mutation": "write", + "projectWriteLease": false, + "permissions": [ + "hosted.game-audio.game-audio-cancel" + ], + "executionMode": "synchronous", + "inputSchema": { + "type": "object", + "additionalProperties": false, + "required": [ + "executionId" + ], + "properties": { + "executionId": { + "type": "string", + "minLength": 1, + "maxLength": 36 + } + } + }, + "outputSchema": { + "type": "object", + "additionalProperties": false, + "required": [ + "executionId", + "providerStatus", + "outputCount", + "generationBilling" + ], + "properties": { + "executionId": { + "type": "string", + "minLength": 1, + "maxLength": 36 + }, + "providerStatus": { + "type": "string", + "minLength": 1, + "maxLength": 32 + }, + "outputCount": { + "type": "integer", + "minimum": 0, + "maximum": 10 + }, + "generationBilling": { + "type": "object", + "additionalProperties": false, + "required": [ + "mode", + "status", + "reserved_points", + "usage_amount", + "unit" + ], + "properties": { + "mode": { + "type": "string", + "minLength": 1, + "maxLength": 32 + }, + "status": { + "type": "string", + "minLength": 1, + "maxLength": 32 + }, + "reserved_points": { + "type": "string", + "minLength": 1, + "maxLength": 64 + }, + "actual_points": { + "type": "string", + "maxLength": 64 + }, + "usage_amount": { + "type": "integer", + "minimum": 0, + "maximum": 200 + }, + "unit": { + "type": "string", + "minLength": 1, + "maxLength": 32 + } + } + } + } + } + } + ] +} diff --git a/resources/coding-plugins/game-audio/plugin.json b/resources/coding-plugins/game-audio/plugin.json new file mode 100644 index 0000000..bddc7e3 --- /dev/null +++ b/resources/coding-plugins/game-audio/plugin.json @@ -0,0 +1,14 @@ +{ + "$schema": "https://agent-plugins.org/schemas/1.0.0/plugin.schema.json", + "name": "makelore.game-audio", + "version": "1.0.0", + "description": "Generate game music, short previews and sound effects, automatically saved to the project.", + "author": { + "name": "MakeLore" + }, + "extensions": { + "com.makelore": { + "capabilityManifest": "./com.makelore/capability.json" + } + } +} diff --git a/resources/coding-plugins/game-audio/skills/game-audio/SKILL.md b/resources/coding-plugins/game-audio/skills/game-audio/SKILL.md new file mode 100644 index 0000000..5fd62ef --- /dev/null +++ b/resources/coding-plugins/game-audio/skills/game-audio/SKILL.md @@ -0,0 +1,26 @@ +--- +name: game-audio +description: 为当前 MakeLore 项目生成背景音乐、音乐短预览或游戏音效时使用；提交前确认 Token Point 费用与自动保存。 +--- + +# 游戏音频生成 + +1. 确定用途与规格。完整音乐用 `game_music_generate`；只有用户明确要短预览才用 + `game_music_preview`。两者独立计费，预览不是免费步骤，也不自动升级生成完整版。 + 音乐最多使用 4 张用户选定的项目内参考图；不要猜 duration、bpm 或 model。 +2. 音效用 `game_sound_generate`：single=1 条，pack=一组，variants=同音效变体。 + 每条时长支持 0.5 秒或 1–10 的整数秒；pack/variants 默认最多 4 条，上限 10； + loop 默认 false。先让用户知道时长、最多条数与循环设置。 +3. 提交前说明 Token Point 规则：音乐按任务、音效按每半秒×实际成功条数； + 预留按最多条数，未返回部分不自动补生成。引用当前可信价格；没有可靠数值就不报总价。 + 本次确认同时授权向托管 Meowa 发送提示词/参考图，以及保存全部输出到发起项目。 +4. 用户确认后以 confirmed=true 调用对应生成工具一次。Main 内部等待任务、更新进度卡、 + 顺序下载并保存到 assets/generated/game-audio//。无需轮询或再次询问保存。 +5. 只有 deliveryStatus=saved 才报告完成，并链接实际 files 中的项目相对路径供用户点击试听； + 不自动播放。区分 providerStatus、deliveryStatus 和 billing；pending_review 不等于失败或退款。 +6. delivery_failed/needs_reconciliation/submission_unknown 时报告原任务身份与真实状态， + 由本机收据恢复原任务。不要为了修复下载或状态不明再次调用生成工具或更换模板重新收费。 + 只有用户明确要求取消时调用 game_audio_cancel；取消返回 succeeded 时仍以实际完成为准。 + +插件随客户端提供，项目启用后所有父 Agent 可用。只用上述固定工具；Provider 凭据、 +URL、账户、价格版本和项目身份由 Main/服务端管理，不从对话构造。 diff --git a/scripts/lib/pi-product-artifact.mjs b/scripts/lib/pi-product-artifact.mjs index 432a052..94c2ffe 100644 --- a/scripts/lib/pi-product-artifact.mjs +++ b/scripts/lib/pi-product-artifact.mjs @@ -44,6 +44,8 @@ const MARKETPLACE_ARTIFACT_MARKERS = Object.freeze({ hostedRuntime: Object.freeze([ 'makelore.game-resource', '/api/plugins/v1/hosted/game-resource/generations', + 'makelore.game-audio', + '/api/plugins/v1/hosted/game-audio/generations', ]), modelTools: Object.freeze([ 'makelore-model-tool.v1', diff --git a/shared/coding-plugins.ts b/shared/coding-plugins.ts index d7c89ea..6f6e33c 100644 --- a/shared/coding-plugins.ts +++ b/shared/coding-plugins.ts @@ -103,6 +103,8 @@ export const DATA_SERVICE_PREVIEW_RUNTIME_SURFACE = 'data-service-v1' as const; export const GAME_RESOURCE_PLUGIN_ID = 'makelore.game-resource' as const; export const GAME_RESOURCE_BUNDLED_RELEASE_ID = '00000000-0000-4000-8000-000000000105' as const; +export const GAME_AUDIO_PLUGIN_ID = 'makelore.game-audio' as const; +export const GAME_AUDIO_BUNDLED_RELEASE_ID = '00000000-0000-4000-8000-000000000505' as const; export const PROJECT_SCAFFOLD_PLUGIN_ID = 'makelore.project-scaffold' as const; export const PROJECT_SCAFFOLD_BUNDLED_RELEASE_ID = '00000000-0000-4000-8000-000000000303' as const; @@ -113,11 +115,17 @@ export const PROJECT_SCAFFOLD_BUNDLED_RELEASE_ID = '00000000-0000-4000-8000-0000 */ export function isProjectWideCodingPluginId(pluginId: string): boolean { return pluginId === DATA_SERVICE_PLUGIN_ID + || pluginId === GAME_AUDIO_PLUGIN_ID || pluginId === GAME_RESOURCE_PLUGIN_ID || pluginId === PROJECT_SCAFFOLD_PLUGIN_ID; } export const CODE_OWNED_OPTIONAL_BUNDLED_RELEASES = Object.freeze({ + [GAME_AUDIO_PLUGIN_ID]: Object.freeze({ + releaseId: GAME_AUDIO_BUNDLED_RELEASE_ID, + version: '1.0.0', + channel: 'stable' as const, + }), [GAME_RESOURCE_PLUGIN_ID]: Object.freeze({ releaseId: GAME_RESOURCE_BUNDLED_RELEASE_ID, version: '1.0.0', diff --git a/src/pages/Chat/CodingConversationTimeline.tsx b/src/pages/Chat/CodingConversationTimeline.tsx index 366c767..83d0746 100644 --- a/src/pages/Chat/CodingConversationTimeline.tsx +++ b/src/pages/Chat/CodingConversationTimeline.tsx @@ -64,6 +64,7 @@ import type { } from '../../../shared/coding-conversation-contracts'; import { CodingAttachmentPreview } from './CodingAttachmentPreview'; import { CodingWelcomeHero } from './CodingWelcomeHero'; +import { GameAudioFiles } from './GameAudioFiles'; const EMPTY_NODES: ConversationNode[] = []; const IDLE_RUN: ConversationRunState = { status: 'idle' }; @@ -105,26 +106,29 @@ function capabilityBillingLabel(billing: CapabilityToolDetails['billing']): stri } function gameResourceProgressLabel(details: CapabilityToolDetails): string | null { - if (details.plugin_id !== 'makelore.game-resource' || details.operation !== 'generate' + if (!['makelore.game-resource', 'makelore.game-audio'].includes(details.plugin_id) + || !['generate', 'preview'].includes(details.operation) || !details.data || typeof details.data !== 'object' || Array.isArray(details.data)) return null; const data = details.data as Record; + const label = details.plugin_id === 'makelore.game-audio' ? '游戏音频' : '游戏资源'; const phase = typeof data.phase === 'string' ? data.phase : null; const deliveryStatus = typeof data.deliveryStatus === 'string' ? data.deliveryStatus : null; const providerStatus = typeof data.providerStatus === 'string' ? data.providerStatus : null; const outputCount = Number.isSafeInteger(data.outputCount) ? data.outputCount as number : 0; if (phase === 'saved' || deliveryStatus === 'saved') { - return `游戏资源 · 已保存 ${outputCount} 个文件`; + return `${label} · 已保存 ${outputCount} 个文件`; } - if (phase === 'saving' || deliveryStatus === 'saving') return '游戏资源 · 正在保存到项目'; - if (deliveryStatus === 'delivery_failed') return '游戏资源 · 自动保存失败，等待重试'; - if (phase === 'submitted') return '游戏资源 · 已提交'; + if (phase === 'saving' || deliveryStatus === 'saving') return label + ' · 正在保存到项目'; + if (deliveryStatus === 'delivery_failed') return label + ' · 自动保存失败，等待恢复'; + if (deliveryStatus === 'needs_reconciliation') return label + ' · 等待原任务状态恢复，请勿重复生成'; + if (phase === 'submitted') return label + ' · 已提交'; if (phase === 'generating' || ['reserved', 'accepted', 'running'].includes(providerStatus ?? '')) { - return '游戏资源 · 正在生成'; + return label + ' · 正在生成'; } - if (providerStatus === 'pending_review') return '游戏资源 · 账单待审核'; - if (providerStatus === 'submission_unknown') return '游戏资源 · 提交状态待确认'; - if (providerStatus === 'cancelled') return '游戏资源 · 已取消'; - if (providerStatus === 'failed') return '游戏资源 · 生成失败'; + if (providerStatus === 'pending_review') return label + ' · 账单待审核'; + if (providerStatus === 'submission_unknown') return label + ' · 提交状态待确认'; + if (providerStatus === 'cancelled') return label + ' · 已取消'; + if (providerStatus === 'failed') return label + ' · 生成失败'; return null; } @@ -652,6 +656,7 @@ const ToolDetails = memo(function ToolDetails({ details }: { details: KnownToolD {details.success ? '成功' : details.error ?? '请求失败'} · HTTP {details.status}

{capabilityBillingLabel(details.billing)}

+ {details.plugin_id === 'makelore.game-audio' && } ); } diff --git a/src/pages/Chat/GameAudioFiles.tsx b/src/pages/Chat/GameAudioFiles.tsx new file mode 100644 index 0000000..a545199 --- /dev/null +++ b/src/pages/Chat/GameAudioFiles.tsx @@ -0,0 +1,53 @@ +import { useEffect, useRef, useState } from 'react'; +import { Button } from '@/components/ui/button'; +import { hostApiFetch } from '@/lib/host-api'; + +function record(value: unknown): value is Record { + return value !== null && typeof value === 'object' && !Array.isArray(value); +} +const AUDIO_PATH = /^assets\/generated\/game-audio\/[0-9a-f-]{36}\/(?:music|sound)-\d+\.(?:mp3|wav|ogg|flac|m4a)$/u; +function AudioFile({ executionId, index, relative }: { executionId: string; index: number; relative: string }) { + const [source, setSource] = useState(null); + const [loading, setLoading] = useState(false); + const [error, setError] = useState(false); + const mounted = useRef(true); + useEffect(() => { + mounted.current = true; + return () => { + mounted.current = false; + }; + }, []); + const name = relative.split('/').at(-1) ?? relative; + const preview = async () => { + setLoading(true); + setError(false); + try { + const file = await hostApiFetch<{ path: string; dataUrl: string }>( + '/api/coding/game-audio/' + executionId + '/outputs/' + index, + ); + if (file.path !== relative || !file.dataUrl.startsWith('data:audio/')) throw new Error('Audio preview unavailable'); + if (!mounted.current) return; + setSource(file.dataUrl); + } catch { if (mounted.current) setError(true); } + finally { if (mounted.current) setLoading(false); } + }; + return
  • +

    {relative}

    + {source ?
    • ; +} +export function GameAudioFiles({ data }: { data: unknown }) { + if (!record(data) || !Array.isArray(data.files) || typeof data.executionId !== 'string' + || !/^[0-9a-f-]{36}$/u.test(data.executionId)) return null; + const files = data.files.flatMap((file) => record(file) && typeof file.path === 'string' + && AUDIO_PATH.test(file.path) && typeof file.index === 'number' && Number.isInteger(file.index) + && file.index >= 0 && file.index < 10 ? [{ path: file.path, index: file.index }] : []); + if (!files.length) return null; + return
      + {files.map((file) => )} +
    ; +} diff --git a/tests/e2e/pi-coding-first-chat.spec.ts b/tests/e2e/pi-coding-first-chat.spec.ts index ba90d44..1298b39 100644 --- a/tests/e2e/pi-coding-first-chat.spec.ts +++ b/tests/e2e/pi-coding-first-chat.spec.ts @@ -82,9 +82,10 @@ async function installCodingFirstChatHost( hostConnection: HostConnection, featureComplete = false, managedCapabilities = false, + audioPreview?: { executionId: string; path: string; dataUrl: string }, ): Promise { await electronApp.evaluate(async (_, payload) => { - const { connection, featureComplete, managedCapabilities } = payload; + const { connection, featureComplete, managedCapabilities, audioPreview } = payload; const { ipcMain } = process.mainModule!.require('electron') as typeof import('electron'); type MainState = { captured: CapturedRequest[]; @@ -456,6 +457,9 @@ async function installCodingFirstChatHost( ) => { const path = request.path ?? ''; const method = request.method ?? 'GET'; + if (audioPreview && path === '/api/coding/game-audio/' + audioPreview.executionId + '/outputs/0') { + return respond({ path: audioPreview.path, dataUrl: audioPreview.dataUrl }); + } const body = typeof request.body === 'string' && request.body ? JSON.parse(request.body) as Record : undefined; @@ -737,9 +741,71 @@ async function installCodingFirstChatHost( if (path === '/api/coding/runtime/diagnostics') return respond({ runtime: { revision: { provider: 1, resources: 1 }, workers: [{ conversationId: conversation.id, generation: 1, state: 'running', stage: 'running' }] } }); return respond({ success: false, error: `Unhandled E2E route: ${method} ${path}` }, 404); }); - }, { connection: hostConnection, featureComplete, managedCapabilities }); + }, { connection: hostConnection, featureComplete, managedCapabilities, audioPreview }); } +test('saved Game Audio offers click-only playable local preview in Electron', async ({ launchElectronApp }, testInfo) => { + const app = await launchElectronApp({ skipSetup: true }); + let page = await getStableWindow(app); + const connection = await page.evaluate(async () => ({ + token: await window.electron.ipcRenderer.invoke('hostapi:token') as string, + baseUrl: await window.electron.ipcRenderer.invoke('hostapi:base-url') as string, + })); + const executionId = '00000000-0000-4000-8000-000000000601'; + const relative = 'assets/generated/game-audio/' + executionId + '/sound-1.wav'; + // Synthetic quarter-second PCM sample, never a claimed Meowa output. + const wav = Buffer.alloc(4044); + wav.write('RIFF', 0); wav.writeUInt32LE(4036, 4); wav.write('WAVEfmt ', 8); + wav.writeUInt32LE(16, 16); wav.writeUInt16LE(1, 20); wav.writeUInt16LE(1, 22); + wav.writeUInt32LE(8000, 24); wav.writeUInt32LE(16000, 28); + wav.writeUInt16LE(2, 32); wav.writeUInt16LE(16, 34); + wav.write('data', 36); wav.writeUInt32LE(4000, 40); + await installCodingFirstChatHost(app, connection, true, false, { + executionId, path: relative, dataUrl: 'data:audio/wav;base64,' + wav.toString('base64'), + }); + await settleSnapshot(app); + await disableCodingEventSource(page); + await page.reload(); + page = await getStableWindow(app); + await page.getByTestId('ai-module-option-programming').click(); + await page.evaluate(() => { window.location.hash = '/chat'; }); + await expect(page.getByTestId('coding-conversation-header')).toBeVisible(); + const snapshot = await page.evaluate(async () => { + const response = await window.electron.ipcRenderer.invoke('hostapi:fetch', { + path: '/api/coding/conversations/conversation-pi-first-chat/snapshot', method: 'GET', + }) as { data: { json: { snapshot: Record } } }; + return response.data.json.snapshot; + }); + const details = { + schema: 'makelore-capability.v1', plugin_id: 'makelore.game-audio', plugin_version: '1.0.0', + capability_id: 'game-audio.sound', operation: 'generate', request_id: 'pi:r:t', + success: true, status: 200, code: null, error: null, retryable: false, payload_schema: 'game-audio.v1', + billing: { mode: 'platform_metered', status: 'settled', reserved_points: '0.04', + actual_points: '0.04', usage_amount: 4, unit: 'half_second' }, + data: { executionId, logicalOperationId: 'pi:r:t', projectId: 'project-e2e', + providerStatus: 'succeeded', deliveryStatus: 'saved', phase: 'saved', outputCount: 1, + files: [{ index: 0, path: relative, bytes: wav.length, mimeType: 'audio/wav' }] }, + }; + await emitCodingEvent(page, 'snapshot', { + type: 'snapshot', conversationId: 'conversation-pi-first-chat', workerGeneration: 1, seq: 10, + snapshot: { ...snapshot, cursor: { workerGeneration: 1, seq: 10 }, nodes: [{ + kind: 'tool', id: 'audio-preview', toolCallId: 'audio-preview', toolName: 'game_sound_generate', + title: '生成游戏音效', status: 'complete', inputText: '', output: [], details, + }] }, + }); + await expect(page.getByTestId('tool-progress-preview')).toContainText('游戏音频 · 已保存'); + await page.getByTestId('coding-process-group').locator('summary').first().click(); + await page.locator('[data-node-id="audio-preview"] > summary').click(); + await expect(page.locator('audio')).toHaveCount(0); + await page.getByRole('button', { name: '试听 sound-1.wav' }).click(); + const player = page.getByLabel('试听 sound-1.wav', { exact: true }); + await expect(player).toHaveAttribute('controls', ''); + expect(await player.getAttribute('autoplay')).toBeNull(); + await expect.poll(() => player.evaluate((element) => (element as HTMLAudioElement).duration)).toBe(0.25); + expect(await player.evaluate((element) => (element as HTMLAudioElement).paused)).toBe(true); + await page.screenshot({ path: testInfo.outputPath('audio-preview.png') }); +}); + async function readState(electronApp: ElectronApplication): Promise<{ captured: CapturedRequest[]; snapshotPending: boolean; diff --git a/tests/unit/coding-conversation-timeline.test.tsx b/tests/unit/coding-conversation-timeline.test.tsx index c37e723..2462150 100644 --- a/tests/unit/coding-conversation-timeline.test.tsx +++ b/tests/unit/coding-conversation-timeline.test.tsx @@ -12,15 +12,19 @@ vi.mock('@/lib/coding-attachments', () => ({ describe('CodingConversationTimeline', () => { afterEach(() => vi.unstubAllGlobals()); - it('shows one Game Resource progress card from generation through automatic project save', async () => { + it.each([ + ['makelore.game-resource', '游戏资源'], + ['makelore.game-audio', '游戏音频'], + ])('shows one %s progress card through automatic project save', async (pluginId, label) => { const { codingConversationStore } = await import('@/stores/coding-conversations'); const { CodingConversationTimeline } = await import( '@/pages/Chat/CodingConversationTimeline' ); - const base = createProductSnapshot('conversation-game-resource-progress', 1); + const conversationId = 'conversation-progress-' + pluginId; + const base = createProductSnapshot(conversationId, 1); const details = (phase: 'saving' | 'saved') => ({ schema: 'makelore-capability.v1' as const, - plugin_id: 'makelore.game-resource', + plugin_id: pluginId, plugin_version: '1.0.0', capability_id: 'game-resource.generate', operation: 'generate', @@ -82,28 +86,28 @@ describe('CodingConversationTimeline', () => { }; codingConversationStore.getState().applySnapshotEvent({ type: 'snapshot', - conversationId: 'conversation-game-resource-progress', + conversationId, workerGeneration: 1, seq: base.cursor.seq, snapshot: snapshot(base.cursor.seq, 'saving'), }); - render(); + render(); expect(screen.getByTestId('tool-progress-preview')).toHaveTextContent( - '游戏资源 · 正在保存到项目', + label + ' · 正在保存到项目', ); const nextSeq = base.cursor.seq + 1; act(() => codingConversationStore.getState().applySnapshotEvent({ type: 'snapshot', - conversationId: 'conversation-game-resource-progress', + conversationId, workerGeneration: 1, seq: nextSeq, snapshot: snapshot(nextSeq, 'saved'), })); await waitFor(() => expect(screen.getByTestId('tool-progress-preview')) - .toHaveTextContent('游戏资源 · 已保存 2 个文件')); + .toHaveTextContent(label + ' · 已保存 2 个文件')); }); it('resolves attachment refs into temporary object URLs without base64 state', async () => { diff --git a/tests/unit/coding-plugin-effective-resolver.test.ts b/tests/unit/coding-plugin-effective-resolver.test.ts index fe7bd24..19b7343 100644 --- a/tests/unit/coding-plugin-effective-resolver.test.ts +++ b/tests/unit/coding-plugin-effective-resolver.test.ts @@ -9,6 +9,8 @@ import { DATA_SERVICE_PLUGIN_DEFINITION, GAME_RESOURCE_BUNDLED_RELEASE_ID, GAME_RESOURCE_PLUGIN_ID, + GAME_AUDIO_PLUGIN_ID, + GAME_AUDIO_BUNDLED_RELEASE_ID, PROJECT_SCAFFOLD_BUNDLED_RELEASE_ID, PROJECT_SCAFFOLD_PLUGIN_ID, type CodingPluginDefinition, @@ -458,11 +460,14 @@ describe('effective plugin resolver', () => { expect(refresh).not.toHaveBeenCalled(); }); - it('materializes an acquired code-owned bundled hosted Plugin without Package Store bytes or Agent assignment', async () => { + it.each([ + [GAME_RESOURCE_PLUGIN_ID, GAME_RESOURCE_BUNDLED_RELEASE_ID], + [GAME_AUDIO_PLUGIN_ID, GAME_AUDIO_BUNDLED_RELEASE_ID], + ])('materializes acquired bundled %s without Package Store bytes or Agent assignment', async (pluginId, releaseId) => { const bundled: CodingPluginDefinition = { ...serverDefinition, - id: GAME_RESOURCE_PLUGIN_ID, - releaseId: GAME_RESOURCE_BUNDLED_RELEASE_ID, + id: pluginId, + releaseId, provenance: { source: 'bundled', packageRoot: 'game-resource' }, skills: [{ id: 'game-resource', diff --git a/tests/unit/coding-plugin-manifest.test.ts b/tests/unit/coding-plugin-manifest.test.ts index 654ba0f..4b05010 100644 --- a/tests/unit/coding-plugin-manifest.test.ts +++ b/tests/unit/coding-plugin-manifest.test.ts @@ -38,13 +38,15 @@ describe('bundled coding plugin manifests', () => { 'data-service', 'game-resource', 'project-scaffold', + 'game-audio', ]); expect(resolveBundledCodingPluginRootPaths(path.resolve('resources/coding-plugins'))).toEqual([ PACKAGE_ROOT, GAME_RESOURCE_ROOT, PROJECT_SCAFFOLD_ROOT, + path.resolve('resources/coding-plugins/game-audio'), ]); - expect(definitions).toHaveLength(3); + expect(definitions).toHaveLength(4); expect(definitions[0]).toMatchObject({ id: 'makelore.data-service', adapterId: 'data-service', @@ -53,7 +55,7 @@ describe('bundled coding plugin manifests', () => { }); expect(definitions[0]?.tools.map(({ name }) => name)).toEqual(DATA_SERVICE_TOOL_NAMES); expect(definitions[0]).toEqual(DATA_SERVICE_PLUGIN_DEFINITION); - expect(definitions.slice(1)).toMatchObject([ + expect(definitions.slice(1, 3)).toMatchObject([ { id: 'makelore.game-resource', version: '1.0.0', runtimeKind: 'platform_hosted', acquisitionMode: 'user_acquired', releaseId: '00000000-0000-4000-8000-000000000105', @@ -160,6 +162,7 @@ describe('bundled coding plugin manifests', () => { path.resolve('tmp/data-service'), path.resolve('tmp/game-resource'), path.resolve('tmp/project-scaffold'), + path.resolve('tmp/game-audio'), ]); }); diff --git a/tests/unit/coding-plugin-routes.test.ts b/tests/unit/coding-plugin-routes.test.ts index 96ccd55..fb03d43 100644 --- a/tests/unit/coding-plugin-routes.test.ts +++ b/tests/unit/coding-plugin-routes.test.ts @@ -46,6 +46,16 @@ async function invoke(ctx: HostApiContext, method: string, target: string, body? } describe('coding plugin Host routes', () => { + it('previews only a receipt output, without accepting project paths or remote URLs', async () => { + const readSavedOutput = vi.fn().mockResolvedValue({ path: 'assets/generated/game-audio/file.mp3', dataUrl: 'data:audio/mpeg;base64,SUQz' }); + const ctx = { codingProducts: { gameAudio: { readSavedOutput } } } as unknown as HostApiContext; + const id = '00000000-0000-4000-8000-000000000601'; + const route = '/api/coding/game-audio/' + id + '/outputs/0'; + expect(await invoke(ctx, 'GET', route)).toMatchObject({ handled: true, status: 200 }); + expect(readSavedOutput).toHaveBeenCalledWith(id, 0); + expect(await invoke(ctx, 'GET', route + '?projectPath=/forged')).toMatchObject({ status: 400 }); + expect(readSavedOutput).toHaveBeenCalledTimes(1); + }); it('uses only the local project handle for GET and preserves the bounded projection', async () => { const projection = { schemaVersion: 1 as const, diff --git a/tests/unit/game-audio-client.test.ts b/tests/unit/game-audio-client.test.ts new file mode 100644 index 0000000..104c1d4 --- /dev/null +++ b/tests/unit/game-audio-client.test.ts @@ -0,0 +1,85 @@ +import { describe, expect, it, vi } from 'vitest'; +import { GameAudioClient } from '../../electron/services/game-audio-client'; + +const execution = '00000000-0000-4000-8000-000000000601'; +const release = '00000000-0000-4000-8000-000000000505'; +const project = '00000000-0000-4000-8000-000000000602'; +const receipt = () => ({ + schema_version: 1, plugin_id: 'makelore.game-audio', execution_id: execution, + release_id: release, project_id: project, logical_operation_id: 'audio-request', + kind: 'sound', provider_status: 'succeeded', output_ready: true, + outputs: [{ index: 0, name: 'sound-1' }], progress_percent: 100, + poll_interval_seconds: 2, error_code: null, + billing: { mode: 'platform_metered', status: 'settled', reserved_points: '0.16', + actual_points: '0.04', usage_amount: 4, unit: 'half_second' }, +}); +const command = { + release_id: release, release_admission_id: project, project_id: project, + logical_operation_id: 'audio-request', pricing_version_id: project, + kind: 'sound' as const, mode: 'single' as const, duration: 2, count: 1, + loop: false, prompt: 'A coin chime', confirmed: true as const, +}; +function setup(fetchImpl: typeof fetch) { + return new GameAudioClient({ fetchImpl, apiBaseUrl: 'https://square.test', + getAccessToken: async () => 'session-token', + getAccountBinding: () => ({ accountKey: 'owner', epoch: 1 }) }); +} +describe('Game Audio typed Main transport', () => { + it('normalizes nullable unsettled amounts into the shared optional receipt field', async () => { + const wire = { ...receipt(), provider_status: 'running', output_ready: false, outputs: [], + billing: { ...receipt().billing, status: 'dispatched', actual_points: null } }; + const job = await setup(vi.fn().mockResolvedValue(Response.json(wire))).get(execution, 'owner'); + expect(job.billing).not.toHaveProperty('actual_points'); + }); + it('uses fixed routes and retries a 401 only once with exactly the same command', async () => { + const fetchImpl = vi.fn() + .mockResolvedValueOnce(new Response('', { status: 401 })) + .mockResolvedValueOnce(Response.json(receipt())); + const client = setup(fetchImpl); + expect(await client.generate(command, 'owner')).toEqual(receipt()); + expect(fetchImpl).toHaveBeenCalledTimes(2); + expect(fetchImpl.mock.calls[0]?.[0]).toBe('https://square.test/api/plugins/v1/hosted/game-audio/generations'); + expect(fetchImpl.mock.calls[0]?.[1]?.body).toBe(JSON.stringify(command)); + expect(fetchImpl.mock.calls[1]?.[1]?.body).toBe(fetchImpl.mock.calls[0]?.[1]?.body); + }); + it('rejects contradictory output/billing and response identity', async () => { + for (const value of [ + { ...receipt(), output_ready: false }, + { ...receipt(), execution_id: '../escape' }, + { ...receipt(), project_id: execution }, + { ...receipt(), billing: { ...receipt().billing, actual_points: 'bad' } }, + { ...receipt(), outputs: [{ index: 1, name: 'sound-2' }] }, + ]) { + await expect(setup(vi.fn().mockResolvedValue(Response.json(value))) + .generate(command, 'owner')).rejects.toMatchObject({ code: 'plugin_backend_invalid' }); + } + }); + it('checks account binding before network and again after a response', async () => { + const fetchImpl = vi.fn().mockResolvedValue(Response.json(receipt())); + await expect(setup(fetchImpl).get(execution, 'other')).rejects.toMatchObject({ code: 'plugin_account_changed' }); + expect(fetchImpl).not.toHaveBeenCalled(); + let epoch = 1; + const client = new GameAudioClient({ + getAccessToken: async () => 'token', getAccountBinding: () => ({ accountKey: 'owner', epoch }), + fetchImpl: async () => { epoch++; return Response.json(receipt()); }, + }); + await expect(client.get(execution, 'owner')).rejects.toMatchObject({ code: 'plugin_account_changed' }); + }); + it('bounds chunked JSON and downloads only supported audio MIME', async () => { + let cancelled = false; + const body = new ReadableStream({ + pull(controller) { controller.enqueue(new Uint8Array(600_000)); }, + cancel() { cancelled = true; }, + }); + await expect(setup(vi.fn().mockResolvedValue(new Response(body))) + .get(execution, 'owner')).rejects.toMatchObject({ code: 'plugin_backend_invalid' }); + expect(cancelled).toBe(true); + await expect(setup(vi.fn().mockResolvedValue(new Response('html', { + headers: { 'Content-Type': 'text/html' }, + }))).download(execution, 0, 'owner')).rejects.toMatchObject({ code: 'game_audio_content_invalid' }); + const audio = await setup(vi.fn().mockResolvedValue(new Response('ID3test', { + headers: { 'Content-Type': 'audio/mpeg' }, + }))).download(execution, 0, 'owner'); + expect(audio.extension).toBe('mp3'); + }); +}); diff --git a/tests/unit/game-audio-delivery.test.ts b/tests/unit/game-audio-delivery.test.ts new file mode 100644 index 0000000..338370e --- /dev/null +++ b/tests/unit/game-audio-delivery.test.ts @@ -0,0 +1,127 @@ +import { mkdtemp, readFile, rm } from 'node:fs/promises'; +import os from 'node:os'; +import path from 'node:path'; +import { afterEach, describe, expect, it, vi } from 'vitest'; +import { PiProjectWriteLeaseCoordinator } from '../../electron/coding-runtime/pi/write-lease'; +import { GameAudioDeliveryCoordinator, GameAudioReceiptStore } from '../../electron/services/game-audio-delivery'; +import { GameAudioClientError, type AudioCommand, type AudioGeneration } from '../../electron/services/game-audio-client'; + +const directories: string[] = []; +afterEach(async () => { await Promise.all(directories.splice(0).map((dir) => rm(dir, { recursive: true, force: true }))); }); +const execution = '00000000-0000-4000-8000-000000000601'; +const project = '00000000-0000-4000-8000-000000000602'; +const release = '00000000-0000-4000-8000-000000000505'; +const command: AudioCommand = { release_id: release, release_admission_id: project, + project_id: project, logical_operation_id: 'request', pricing_version_id: project, prompt: 'coin', + confirmed: true, kind: 'sound', mode: 'pack', count: 2, duration: 2, loop: false }; +function generation(active = false): AudioGeneration { + return { schema_version: 1, plugin_id: 'makelore.game-audio', execution_id: execution, release_id: release, + project_id: project, logical_operation_id: 'request', kind: 'sound', provider_status: active ? 'running' : 'succeeded', + output_ready: !active, outputs: active ? [] : [{ index: 0, name: 'sound-1' }, { index: 1, name: 'sound-2' }], + progress_percent: active ? 40 : 100, poll_interval_seconds: 2, error_code: null, + billing: { mode: 'platform_metered', status: active ? 'dispatched' : 'settled', + reserved_points: '0.08', actual_points: active ? null : '0.08', usage_amount: 8, unit: 'half_second' } }; +} +async function setup() { + const root = await mkdtemp(path.join(os.tmpdir(), 'makelore-audio-')); + directories.push(root); + const receipts = new GameAudioReceiptStore(path.join(root, 'user-data', 'receipts.json')); + const context = { accountKey: 'owner', conversationId: 'conversation', runId: 'run', + localProjectId: 'local-project', durableProjectId: project, projectPath: root, logicalOperationId: 'request' }; + const leases = new PiProjectWriteLeaseCoordinator(); + const client = { + generate: vi.fn(async () => generation(true)), get: vi.fn(async () => generation()), + byOperation: vi.fn(async () => generation()), + download: vi.fn(async () => ({ bytes: new TextEncoder().encode('ID3audio'), extension: 'mp3', mimeType: 'audio/mpeg' })), + }; + const recordTouchedPaths = vi.fn(async () => undefined); + const options = { client, receipts, leases, recordTouchedPaths, getAccountKey: () => 'owner', + sleep: vi.fn(async () => undefined) }; + return { root, context, receipts, leases, client, options }; +} +describe('Game Audio durable automatic delivery', () => { + it('persists intent before POST and saves all outputs under the frozen project without another confirmation', async () => { + const s = await setup(); + s.client.generate.mockImplementation(async () => { + expect((await s.receipts.list())[0]?.context).toEqual(s.context); + expect(s.leases.activeCount).toBe(0); + return generation(true); + }); + s.client.download.mockImplementation(async () => { + expect(s.leases.activeCount).toBe(0); + return { bytes: new TextEncoder().encode('ID3audio'), extension: 'mp3', mimeType: 'audio/mpeg' }; + }); + const delivery = new GameAudioDeliveryCoordinator(s.options); + const progress = vi.fn(); + const result = await delivery.generateAndMaterialize(s.context, command, progress); + expect(result.deliveryStatus).toBe('saved'); + expect(result.files).toHaveLength(2); + for (const file of result.files) expect(await readFile(path.join(s.root, file.path), 'utf8')).toBe('ID3audio'); + expect(result.files[1]?.path).toBe('assets/generated/game-audio/' + execution + '/sound-2.mp3'); + expect(s.options.recordTouchedPaths).toHaveBeenCalled(); + expect(progress.mock.calls.some(([value]) => value.phase === 'generating')).toBe(true); + await delivery.generateAndMaterialize(s.context, command); + expect(s.client.generate).toHaveBeenCalledTimes(1); + expect(s.client.download).toHaveBeenCalledTimes(2); + expect(await delivery.readSavedOutput(execution, 1)).toEqual({ + path: result.files[1]?.path, dataUrl: 'data:audio/mpeg;base64,' + Buffer.from('ID3audio').toString('base64'), + }); + await expect(delivery.readSavedOutput(execution, 9)).rejects.toMatchObject({ status: 404 }); + const other = new GameAudioDeliveryCoordinator({ ...s.options, getAccountKey: () => 'other' }); + await expect(other.readSavedOutput(execution, 0)).rejects.toMatchObject({ status: 404 }); + other.dispose(); + delivery.dispose(); + }); + it('recovers a lost POST response with by-operation, never re-submits', async () => { + const s = await setup(); + s.client.generate.mockRejectedValue(new Error('response lost')); + const delivery = new GameAudioDeliveryCoordinator(s.options); + expect((await delivery.generateAndMaterialize(s.context, command)).deliveryStatus).toBe('saved'); + expect(s.client.byOperation).toHaveBeenCalledWith('request', 'owner'); + expect(s.client.generate).toHaveBeenCalledTimes(1); + delivery.dispose(); + }); + it('reports a definite pre-dispatch rejection as not started and never resumes it', async () => { + const s = await setup(); + s.client.generate.mockRejectedValue(new GameAudioClientError('plugin_pricing_changed', 409, 'Price changed')); + const delivery = new GameAudioDeliveryCoordinator(s.options); + expect(await delivery.generateAndMaterialize(s.context, command)) + .toMatchObject({ providerStatus: 'failed', billing: { status: 'not_started' } }); + await delivery.resumePending(); + expect(s.client.byOperation).not.toHaveBeenCalled(); + delivery.dispose(); + }); + it('automatically resumes a transient download failure without resubmitting', async () => { + const s = await setup(); + s.client.download.mockRejectedValueOnce(new Error('offline')); + vi.useFakeTimers(); + const delivery = new GameAudioDeliveryCoordinator(s.options); + try { + expect((await delivery.generateAndMaterialize(s.context, command)).deliveryStatus).toBe('delivery_failed'); + await vi.advanceTimersByTimeAsync(30_000); + vi.useRealTimers(); + await vi.waitFor(async () => expect((await s.receipts.list())[0]?.deliveryStatus).toBe('saved')); + expect(s.client.generate).toHaveBeenCalledTimes(1); + } finally { delivery.dispose(); vi.useRealTimers(); } + }); + it('resumes only the missing output after restart and pauses for another account', async () => { + const s = await setup(); + s.client.download.mockResolvedValueOnce({ bytes: new TextEncoder().encode('ID3one'), extension: 'mp3', mimeType: 'audio/mpeg' }) + .mockRejectedValue(new Error('download offline')); + const first = new GameAudioDeliveryCoordinator(s.options); + expect((await first.generateAndMaterialize(s.context, command)).deliveryStatus).toBe('delivery_failed'); + first.dispose(); + s.client.download.mockClear().mockResolvedValue({ bytes: new TextEncoder().encode('ID3two'), extension: 'mp3', mimeType: 'audio/mpeg' }); + const other = new GameAudioDeliveryCoordinator({ ...s.options, getAccountKey: () => 'other' }); + await other.resumePending(); + expect(s.client.download).not.toHaveBeenCalled(); + other.dispose(); + const restored = new GameAudioDeliveryCoordinator(s.options); + await restored.resumePending(); + expect(s.client.generate).toHaveBeenCalledTimes(1); + expect(s.client.download).toHaveBeenCalledTimes(1); + expect(s.client.download).toHaveBeenCalledWith(execution, 1, 'owner'); + expect((await s.receipts.list())[0]?.deliveryStatus).toBe('saved'); + restored.dispose(); + }); +}); diff --git a/tests/unit/game-audio-files.test.tsx b/tests/unit/game-audio-files.test.tsx new file mode 100644 index 0000000..6d6974b --- /dev/null +++ b/tests/unit/game-audio-files.test.tsx @@ -0,0 +1,19 @@ +import { fireEvent, render, screen } from '@testing-library/react'; +import { afterEach, expect, it, vi } from 'vitest'; +import { GameAudioFiles } from '../../src/pages/Chat/GameAudioFiles'; +const mocks = vi.hoisted(() => ({ read: vi.fn() })); +vi.mock('@/lib/host-api', () => ({ hostApiFetch: mocks.read })); +afterEach(() => vi.unstubAllGlobals()); +it('loads only saved local audio on explicit preview; never auto-plays or asks to save again', async () => { + const executionId = '00000000-0000-4000-8000-000000000601'; + const relative = 'assets/generated/game-audio/00000000-0000-4000-8000-000000000601/sound-1.mp3'; + mocks.read.mockResolvedValue({ path: relative, dataUrl: 'data:audio/mpeg;base64,SUQz' }); + const { unmount } = render(); + expect(mocks.read).not.toHaveBeenCalled(); + fireEvent.click(screen.getByRole('button', { name: '试听 sound-1.mp3' })); + const player = await screen.findByLabelText('试听 sound-1.mp3', { selector: 'audio' }); + expect(player).toHaveAttribute('controls'); + expect(player).not.toHaveAttribute('autoplay'); + expect(mocks.read).toHaveBeenCalledWith('/api/coding/game-audio/' + executionId + '/outputs/0'); + unmount(); +}); diff --git a/tests/unit/game-audio-plugin.test.ts b/tests/unit/game-audio-plugin.test.ts new file mode 100644 index 0000000..2be06d9 --- /dev/null +++ b/tests/unit/game-audio-plugin.test.ts @@ -0,0 +1,99 @@ +// @vitest-environment node +import path from 'node:path'; +import { describe, expect, it, vi } from 'vitest'; +import { loadBundledCodingPluginDefinitionsSync } from '../../electron/coding-plugins/manifest'; +import { isProjectWideCodingPluginId } from '../../shared/coding-plugins'; +import { GameAudioPluginAdapter } from '../../electron/coding-plugins/adapters/game-audio'; +import { CodingCapabilityRegistryImpl } from '../../electron/coding-plugins/registry'; +import type { PluginPolicyClientState } from '../../electron/services/plugin-policy-client'; +import type { AudioDeliveryResult } from '../../electron/services/game-audio-delivery'; + +const context = { conversationId: 'conversation', runId: 'run', resourceId: 'tool', + requestId: 'trusted-id', localProjectId: 'local', projectPath: '/project', + durableProjectId: 'durable', workerRole: 'parent' as const, effectiveSkillIds: ['game-audio'], + pluginReleaseId: '00000000-0000-4000-8000-000000000505' }; +function definition() { + return loadBundledCodingPluginDefinitionsSync(path.resolve('resources/coding-plugins')) + .find((item) => item.id === 'makelore.game-audio')!; +} +describe('official Game Audio plugin', () => { + it('passes normalized receipts through the registry with progress and saved files', async () => { + const plugin = definition(); + const delivered: AudioDeliveryResult = { + executionId: '00000000-0000-4000-8000-000000000601', logicalOperationId: 'request', + providerStatus: 'succeeded', deliveryStatus: 'saved', phase: 'saved', progressPercent: 100, + outputCount: 1, files: [{ index: 0, path: 'assets/generated/game-audio/00000000-0000-4000-8000-000000000601/music-1.mp3', bytes: 10, mimeType: 'audio/mpeg' }], + billing: { mode: 'platform_metered', status: 'settled', reserved_points: '1.00', actual_points: '1.00', usage_amount: 1, unit: 'generation' }, + }; + const adapter = new GameAudioPluginAdapter({ + delivery: { generateAndMaterialize: async (_context, _command, progress) => { + progress?.({ ...delivered, providerStatus: 'running', deliveryStatus: 'waiting', phase: 'generating', files: [], + billing: { mode: 'platform_metered', status: 'dispatched', reserved_points: '1.00', usage_amount: 1, unit: 'generation' } }); + return delivered; + } }, + client: { cancel: vi.fn() }, + admission: { resolve: vi.fn(async () => ({ releaseId: context.pluginReleaseId, releaseAdmissionId: 'admission' })) }, + getPricingVersion: () => 'price', getAccountKey: () => 'owner', + }); + const policy: PluginPolicyClientState = { status: 'current', revision: 1, lastVerifiedAt: 1, + catalog: { schema_version: 1, catalog_version: '1', pricing_version: 'price', + plugins: [{ plugin_id: plugin.id, supported_contract_versions: [1], status: 'active', + capabilities: plugin.operations.map((op) => ({ capability_id: op.capabilityId, operations: [{ + operation: op.operation, billing: { mode: 'platform_metered', entitlement_scope: 'plugin_usage', notice: 'Metered', + unit_name: 'generation', unit_size: 1, rate_points: '1.00', minimum_charge_points: '0.00', rounding_mode: 'ceil' }, + }] })) }] } }; + const registry = new CodingCapabilityRegistryImpl({ definitions: [plugin], adapters: [adapter], + getEnabledPluginIds: async () => [plugin.id], policyClient: { getState: () => policy, refresh: vi.fn() } }); + const onUpdate = vi.fn(); + const output = await registry.invoke({ toolName: 'game_music_generate', workerRole: 'parent', + effectiveSkillIds: ['game-audio'], context: { conversationId: 'c', runId: 'r', resourceId: 't', + projectId: 'p', projectPath: '/project', skillIds: ['game-audio'] }, + value: { prompt: 'forest', confirmed: true }, onUpdate }); + expect(output.details).toMatchObject({ success: true, data: { files: delivered.files } }); + expect(onUpdate).toHaveBeenCalledWith(expect.objectContaining({ + details: expect.objectContaining({ success: true, billing: { ...delivered.billing, status: 'dispatched', actual_points: undefined } }), + })); + }); + it('ships one project-wide Skill and exactly four statically priced parent tools', () => { + const plugin = definition(); + expect(plugin).toBeDefined(); + expect(isProjectWideCodingPluginId(plugin.id)).toBe(true); + expect(plugin.tools.map((tool) => [tool.name, tool.capabilityId, tool.operation])).toEqual([ + ['game_music_generate', 'game-audio.music', 'generate'], + ['game_music_preview', 'game-audio.music', 'preview'], + ['game_sound_generate', 'game-audio.sound', 'generate'], + ['game_audio_cancel', 'game-audio.control', 'cancel'], + ]); + expect(plugin.tools.every((tool) => tool.roles.length === 1 && tool.roles[0] === 'parent' && !tool.projectWriteLease)).toBe(true); + }); + it('requires literal confirmation; Main freezes identity, mode and trusted price before auto-delivery', async () => { + const generateAndMaterialize = vi.fn(async (): Promise => ({ + executionId: null, logicalOperationId: 'trusted-id', providerStatus: 'running', + deliveryStatus: 'waiting', files: [], outputCount: 0, phase: 'generating', progressPercent: 40, + billing: { mode: 'platform_metered', status: 'receipt_unavailable' }, + })); + const resolve = vi.fn(async () => ({ releaseId: context.pluginReleaseId, releaseAdmissionId: 'admission' })); + const adapter = new GameAudioPluginAdapter({ + delivery: { generateAndMaterialize }, + client: { cancel: vi.fn() }, admission: { resolve }, + getPricingVersion: () => 'trusted-price', getAccountKey: () => 'owner', + }); + const tool = definition().tools[1]!; + expect(await adapter.invoke(context, tool, { prompt: 'ambient forest', confirmed: 1 })) + .toMatchObject({ success: false, code: 'confirmation_required', billing: { status: 'not_started' } }); + expect(generateAndMaterialize).not.toHaveBeenCalled(); + expect(await adapter.invoke(context, tool, { prompt: 'ambient forest', confirmed: true })) + .toMatchObject({ success: true, status: 202 }); + expect(generateAndMaterialize).toHaveBeenCalledWith(expect.objectContaining({ + accountKey: 'owner', logicalOperationId: 'trusted-id', projectPath: '/project', + }), expect.objectContaining({ + kind: 'music', mode: 'demo', pricing_version_id: 'trusted-price', project_id: 'durable', + release_id: context.pluginReleaseId, release_admission_id: 'admission', logical_operation_id: 'trusted-id', + }), expect.any(Function)); + expect(await adapter.invoke(context, tool, { prompt: 'x', confirmed: true, mode: 'full' })) + .toMatchObject({ success: false, code: 'plugin_input_invalid' }); + generateAndMaterialize.mockRejectedValueOnce(new Error('receipt disk failed after POST')); + expect(await adapter.invoke(context, tool, { prompt: 'forest', confirmed: true })) + .toMatchObject({ success: false, billing: { status: 'receipt_unavailable' } }); + }); +}); diff --git a/tests/unit/pi-product-artifact.test.ts b/tests/unit/pi-product-artifact.test.ts index 8fc287e..ee65371 100644 --- a/tests/unit/pi-product-artifact.test.ts +++ b/tests/unit/pi-product-artifact.test.ts @@ -30,6 +30,7 @@ const MARKETPLACE_ARTIFACT_TEXT = [ 'makelore-plugin-release.v1', 'skill_only', 'platform_hosted', 'plugin_signature_invalid', 'signing key is not trusted', 'makelore.game-resource', '/api/plugins/v1/hosted/game-resource/generations', + 'makelore.game-audio', '/api/plugins/v1/hosted/game-audio/generations', 'makelore-model-tool.v1', 'model.web-search', 'forced_search', 'makelore-device-package.v1', '/api/coding/device-packages', 'device-parent-workers', '/api/coding/plugin-marketplace', @@ -399,6 +400,7 @@ describe('final Pi product artifact verification', () => { await writeFile(path.join(source, 'dist', 'assets', 'plugin-marketplace.js'), [ 'makelore-plugin-release.v1 skill_only platform_hosted plugin_signature_invalid signing key is not trusted', 'makelore.game-resource /api/plugins/v1/hosted/game-resource/generations', + 'makelore.game-audio /api/plugins/v1/hosted/game-audio/generations', 'makelore-model-tool.v1 model.web-search forced_search', 'makelore-device-package.v1 /api/coding/device-packages device-parent-workers', '/api/coding/plugin-marketplace plugin-marketplace\\/install\\/ plugin-marketplace\\/update\\/', @@ -419,6 +421,7 @@ describe('final Pi product artifact verification', () => { await writeFile(path.join(staleSource, 'dist-electron', 'stale-marketplace.js'), [ 'makelore-plugin-release.v1 skill_only platform_hosted plugin_signature_invalid signing key is not trusted', 'makelore.game-resource /api/plugins/v1/hosted/game-resource/generations', + 'makelore.game-audio /api/plugins/v1/hosted/game-audio/generations', '/api/coding/plugin-marketplace plugin-marketplace\\/install\\/ plugin-marketplace\\/update\\/', 'effectiveSkillIds pluginReleaseIds', '/api/coding/plugin-marketplace/catalog /api/coding/plugin-marketplace/library 为你的智能体添加插件，拓展更多能力。 acquire enable_project /project-config/plugins',