ARR validated daily ingestion

This package is the ARR-owned boundary between a SuperAgent delivery and PostgreSQL. It does not run the Agent or generate monthly reports. Production object materialization is supplied by the separate arr_storage.ManagedObjectStore boundary.

It supports two result-delivery modes:

artifact_callback: the existing signed callback with Agent-produced OSS artifacts;
direct_mcp: one complete structured payload submitted through the future ARR MCP gateway, independently replayed from the original private XML before Finance activation.

Authority boundary

private object store
  -> DeliveryEnvelope 1.0
  -> DeliveryValidator (hash/size/MIME + v3 contract + independent replay)
  -> PostgresIngestionRepository (one atomic transaction)
  -> current retained facts + privacy-minimized outbox

Only ARR calls this package and owns ARR_DATABASE_URL. SuperAgent/Agent receives opaque job/file handles and never receives database credentials.

Direct MCP service boundary

The framework-neutral direct service is already implemented; the MCP protocol layer must remain a thin adapter over it:

from arr_web.direct_ingestion_runtime import compose_oss_direct_ingestion

runtime = compose_oss_direct_ingestion(project_root=project_root)
try:
    receipt = runtime.service.submit({
        "submission_grant": short_lived_attempt_grant,
        "job_id": job_id,
        "attempt_no": attempt_no,
        "payload": frozen_structured_result,
    })
finally:
    runtime.close()

ARR computes the canonical payload SHA-256 and byte size. The caller must not send a duplicate hash field. The payload is limited to 3 MiB, bound to one job_id/attempt grant, and accepts only successful, activation-eligible structured-result v3 data. The service then:

atomically consumes the hashed short-lived grant and stores a validating ledger row;
materializes the registered source.xml from private object storage;
reruns the allowlisted deterministic processor and compares every business fact;
writes the daily version, all outcome rows, channel metrics, current pointer and outbox atomically;
purges the guest-bearing temporary payload at every terminal state;
returns the prior receipt for an exact retry and rejects changed parameters as a conflict.

The public request and receipt contracts are database/contracts/arr-submit-processing-result-v1.schema.json and database/contracts/arr-submit-processing-result-receipt-v1.schema.json. Migration 010_mcp_result_ingestion.sql must be applied before this mode is enabled.

Runtime requirements

Python 3.9+
openpyxl==3.1.5 through requirements-data-processing.txt
psycopg[binary]==3.2.13 through requirements-arr-ingestion.txt
applied PostgreSQL migration 008_arr_mvp_v1_rebuild.sql
the allowlisted arr-opera-daily-ingest Skill and its independent validator

Install the root requirements.txt; do not place a real DSN in source, CLI arguments, logs, prompts, envelopes, or test fixtures.

Required lifecycle

ARR stores the uploaded XML as an immutable private object and constructs its ArtifactRef.
ARR registers JobRegistration before dispatch. The SHA-256 idempotency key identifies one job attempt.
SuperAgent returns artifacts only. ARR builds the strict DeliveryEnvelope and serializes it as UTF-8 JSON.
DeliveryValidator.validate() materializes objects into private temporary files, recomputes identity, checks processor/rule allowlist and v3 payload invariants, then invokes the independent Skill validator.
IngestionService.ingest() calls the PostgreSQL repository only after validation succeeds.
A successful transaction stores all source outcomes, exact Group Code lookup status and ordered channel metrics; it activates the new version and switches current_daily_versions at the end.
A deterministic processing failure stores a rejected daily version and all available source outcomes for audit, but never creates a current pointer or changes the last active business facts.

Minimal application wiring:

from pathlib import Path

from arr_ingestion.postgres import DatabaseConfig, PostgresIngestionRepository
from arr_ingestion.service import IngestionService
from arr_ingestion.validation import DeliveryValidator, ProcessorPolicy
from arr_storage import FilesystemObjectBackend, ManagedObjectStore

policy = ProcessorPolicy(
    processor_version="3.0.0",
    rule_set_sha256="<allowlisted-64-hex-rule-hash>",
    skill_root=Path("/private/runtime/arr-opera-daily-ingest"),
)
repository = PostgresIngestionRepository(DatabaseConfig.from_environment())
object_store = ManagedObjectStore(
    FilesystemObjectBackend(Path("/private/object-store"))
)
service = IngestionService(
    DeliveryValidator(object_store, policy),
    repository,
)

outcome = service.ingest(raw_delivery_envelope)

FilesystemObjectBackend is for controlled local tests only. arr_storage.CloudObjectBackend exposes the same guarded path to a provider-specific OSS/S3 client port. Only immutable committed objects can be materialized, and the ARR validator still recomputes the delivery-declared identity before any database write.

Transaction and retry semantics

PostgreSQL target is hard-gated to booking_test in the current test deployment.
Each write uses a new SERIALIZABLE transaction with local lock/statement timeouts.
Same callback bytes are idempotent; same callback ID with different bytes is rejected.
Same source/date/processor/rule reuses the immutable version; a different source on the same date creates the next version and supersedes the old current only after all inserts pass.
SQLSTATE 40001, 40P01, 55P03 and concurrent 23505 are retried by reopening the entire transaction, at most four attempts. Business validation errors are never retried.
Outbox payloads contain only opaque IDs, dates, counts, version numbers, disposition and safe failure codes.

Verified test-database state

Migration 008 and a privacy-safe 2026-07-27 vertical slice are deployed to the controlled remote test database <ARR_DB_HOST>:5432/booking_test. Successful and rejected delivery paths, all-outcome storage, booking lookup, total-price enforcement, current activation and idempotent replay have been tested. The 2026-07-28 full migration from the LAN source also passed independent structure, content-hash and application read-path checks. Exact hashes and counts are recorded in database/APPLIED_MIGRATIONS.md and database/REMOTE_MIGRATION_20260728.md.

The active controlled configuration is /path/to/private/booking-test-db.env. The old LAN configuration /path/to/private/booking-test-lan-db.env is retained only for temporary rollback checks. Production remains blocked until ARR has dedicated least-privilege roles, a concrete private OSS provider/bucket configuration in secret management, and the real SuperAgent file/callback contract.

6.8 KiB Raw Blame History