# Task Trace Terminal Verification

## Scope

Replace the decorative daily-processing status module with a diagnostic view
that exposes the persisted task flow from upload through downstream outbox
state. After the first structured UI pass, the user explicitly required the
plainest possible presentation: one black code console with raw log lines.

## Implemented Evidence Path

- `GET /api/jobs/{job_id}/trace` validates the job identifier and asks the
  repository for a privacy-minimized diagnostic projection.
- New tasks are submitted once through
  `POST /api/open/agent-sessions/{session_id}/messages/stream?include_trace=true`.
  The first `run.started` binds the ARR attempt; the same generator is then
  consumed on a background thread until the SuperAgent run terminal event and
  SSE `end`.
- Raw SSE and assistant text never reach disk. `arr_processing.agent_trace`
  persists only allowlisted run/task/step/end projections in a private,
  append-only local JSONL store keyed by the job-id digest.
- `arr_web.job_trace` reconstructs ordered events from the processing run,
  attempts, artifact deliveries or direct MCP submissions, Finance daily
  versions and downstream outbox events, then merges the sanitized SuperAgent
  events by timestamp.
- Equal PostgreSQL transaction timestamps use an explicit business-sequence
  tie-breaker so validation/Finance commit precede attempt/job success and
  downstream queue events.
- The desktop page renders the response in a single black monospace `<pre>`.
  Each event is one line containing timestamp, level, stage, code, title,
  message and allowlisted details JSON. The controls remain limited to
  historical task selection, four-second active-task polling, manual refresh
  and one-click copying of the complete current `<pre>` plaintext.

## Privacy Boundary

The SQL and projection expose only diagnostic columns. `message.delta` and
`message.final` are always dropped. Trace summaries redact credential
assignments, Bearer values, URLs, absolute paths, emails, long numbers and XML
fragments before persistence, and the read path validates the field/details
allowlist again. The response and UI do not contain grants, credentials,
private object paths, source bytes, raw payload/envelope/receipt JSON or guest
PII. Outbox rows are described only as queued/published notifications, not as
proof of monthly-report completion.

The SuperAgent endpoint accepts only POST and does not replay historical runs.
It was also verified to create a different `run_id` when the exact same message
and idempotency key are sent twice. ARR therefore never POSTs a second stream to
backfill logs. An uncertain failure before `run.started` is terminalized as
`PROCESSING_REMOTE_SUBMISSION_AMBIGUOUS` rather than automatically retried.

## Runtime Checks

An isolated read-only Web instance on `127.0.0.1:8766` used the configured
`booking_test` database while the standing 8765 process was left untouched.

- A current queued/running task rendered upload, attempt and current-state
  events and continued four-second polling.
- Successful fixture `mvp-v1-fixture-20260727` rendered 11 events in the
  required upload → dispatch → result → validation → Finance → job → outbox
  order.
- Failed task `arrjob-fdc2c1a0695c41b3b372a52033b2f567` rendered six events
  and the terminal line
  `FAILURE stage=writeback code=PROCESSING_RESULT_MISSING`.
- Computed UI evidence was `tag=PRE`, black `rgb(8, 11, 16)`, monospace font,
  and zero old filter/timeline/evidence elements.
- At a 375×812 viewport, document width stayed 375 pixels; long raw lines
  scroll only inside the console. Browser console error/warning count was zero.
- Live contract probes confirmed `405 Allow: POST` for GET/HEAD/OPTIONS, the
  outer `trace` SSE event, inner `run.started/task.updated/step.updated/`
  `message.delta/message.final/run.completed`, and terminal `end`.
- A live isolated background-handoff probe returned the run handle and then
  persisted 45 safe events (28 task updates and 14 step updates) through
  completion/end with no capture failure.
- The controlled local LaunchAgent was restarted after implementation. Port
  8765 reports database/processing/writeback/monthly readiness, serves
  `arr-job-trace-2`, and correctly returns `agent_trace_events=0` for a job that
  predates trace capture.
- On 2026-07-30, the live 8765 page showed the copy control disabled before a
  trace loaded and enabled for a 228-line terminal containing the final `END`
  line. A real click completed the page Clipboard API promise and displayed the
  existing success toast; browser console warning/error count remained zero.
  The in-app Browser isolates page, automation and macOS clipboard channels, so
  no external pasteback-equality claim is made. Source and regression checks
  verify that the exact `#process-log.textContent` is passed to the Clipboard
  API, with selected-textarea fallback and explicit failure feedback.

## Automated Verification

- `node --check arr_web/static/app.js`: pass.
- 38 post-hardening targeted tests: pass.
- Full repository suite: 275 tests passed with 2 expected artifact-tool skips.

## Result

The status module now reports the real SuperAgent execution path plus the
authoritative persisted ARR processing chain in the user-requested plain
terminal form. It locates failures by stage and stable error code without
treating Agent trace as proof of Finance commit or widening the privacy surface.
Jobs submitted before this capture path was enabled retain only their persisted
ARR facts because historical Agent trace cannot be replayed.

The 2026-07-30 v3 run also established an important limitation: trace capture can
finish normally while the authoritative ARR job remains `running`. Four-second
refresh continued and the trace ended at `AGENT_TRACE_STREAM_ENDED`, but no MCP
callback arrived and the production lifecycle had no terminal reconciler. The UI
was faithfully rendering stale backend state; trace completion alone must not be
promoted to business success, but it must trigger or feed a separate terminal
no-submission reconciliation path.