# ARR validated daily ingestion

This package is the ARR-owned boundary between a SuperAgent delivery and PostgreSQL. It does not run the Agent or generate monthly reports. Production object materialization is supplied by the separate `arr_storage.ManagedObjectStore` boundary.

It supports two result-delivery modes:

- `artifact_callback`: the existing signed callback with Agent-produced OSS artifacts;
- `direct_mcp`: one complete structured payload submitted through the future ARR MCP gateway, independently replayed from the original private XML before Finance activation.

## Authority boundary

```text
private object store
  -> DeliveryEnvelope 1.0
  -> DeliveryValidator (hash/size/MIME + v3 contract + independent replay)
  -> PostgresIngestionRepository (one atomic transaction)
  -> current retained facts + privacy-minimized outbox
```

Only ARR calls this package and owns `ARR_DATABASE_URL`. SuperAgent/Agent receives opaque job/file handles and never receives database credentials.

## Direct MCP service boundary

The framework-neutral direct service is already implemented; the MCP protocol layer must remain a thin adapter over it:

```python
from arr_web.direct_ingestion_runtime import compose_oss_direct_ingestion

runtime = compose_oss_direct_ingestion(project_root=project_root)
try:
    receipt = runtime.service.submit({
        "submission_grant": short_lived_attempt_grant,
        "job_id": job_id,
        "attempt_no": attempt_no,
        "payload": frozen_structured_result,
    })
finally:
    runtime.close()
```

ARR computes the canonical payload SHA-256 and byte size. The caller must not send a duplicate hash field. The payload is limited to 3 MiB, bound to one `job_id`/attempt grant, and accepts only successful, activation-eligible structured-result v3 data. The service then:

1. atomically consumes the hashed short-lived grant and stores a validating ledger row;
2. materializes the registered `source.xml` from private object storage;
3. reruns the allowlisted deterministic processor and compares every business fact;
4. writes the daily version, all outcome rows, channel metrics, current pointer and outbox atomically;
5. purges the guest-bearing temporary payload at every terminal state;
6. returns the prior receipt for an exact retry and rejects changed parameters as a conflict.

The public request and receipt contracts are
`database/contracts/arr-submit-processing-result-v1.schema.json` and
`database/contracts/arr-submit-processing-result-receipt-v1.schema.json`.
Migration `010_mcp_result_ingestion.sql` must be applied before this mode is enabled.

## Runtime requirements

- Python 3.9+
- `openpyxl==3.1.5` through `requirements-data-processing.txt`
- `psycopg[binary]==3.2.13` through `requirements-arr-ingestion.txt`
- applied PostgreSQL migration `008_arr_mvp_v1_rebuild.sql`
- the allowlisted `arr-opera-daily-ingest` Skill and its independent validator

Install the root `requirements.txt`; do not place a real DSN in source, CLI arguments, logs, prompts, envelopes, or test fixtures.

## Required lifecycle

1. ARR stores the uploaded XML as an immutable private object and constructs its `ArtifactRef`.
2. ARR registers `JobRegistration` before dispatch. The SHA-256 idempotency key identifies one job attempt.
3. SuperAgent returns artifacts only. ARR builds the strict `DeliveryEnvelope` and serializes it as UTF-8 JSON.
4. `DeliveryValidator.validate()` materializes objects into private temporary files, recomputes identity, checks processor/rule allowlist and v3 payload invariants, then invokes the independent Skill validator.
5. `IngestionService.ingest()` calls the PostgreSQL repository only after validation succeeds.
6. A successful transaction stores all source outcomes, exact Group Code lookup status and ordered channel metrics; it activates the new version and switches `current_daily_versions` at the end.
7. A deterministic processing failure stores a rejected daily version and all available source outcomes for audit, but never creates a current pointer or changes the last active business facts.

Minimal application wiring:

```python
from pathlib import Path

from arr_ingestion.postgres import DatabaseConfig, PostgresIngestionRepository
from arr_ingestion.service import IngestionService
from arr_ingestion.validation import DeliveryValidator, ProcessorPolicy
from arr_storage import FilesystemObjectBackend, ManagedObjectStore

policy = ProcessorPolicy(
    processor_version="3.0.0",
    rule_set_sha256="<allowlisted-64-hex-rule-hash>",
    skill_root=Path("/private/runtime/arr-opera-daily-ingest"),
)
repository = PostgresIngestionRepository(DatabaseConfig.from_environment())
object_store = ManagedObjectStore(
    FilesystemObjectBackend(Path("/private/object-store"))
)
service = IngestionService(
    DeliveryValidator(object_store, policy),
    repository,
)

outcome = service.ingest(raw_delivery_envelope)
```

`FilesystemObjectBackend` is for controlled local tests only. `arr_storage.CloudObjectBackend` exposes the same guarded path to a provider-specific OSS/S3 client port. Only immutable `committed` objects can be materialized, and the ARR validator still recomputes the delivery-declared identity before any database write.

## Transaction and retry semantics

- PostgreSQL target is hard-gated to `booking_test` in the current test deployment.
- Each write uses a new SERIALIZABLE transaction with local lock/statement timeouts.
- Same callback bytes are idempotent; same callback ID with different bytes is rejected.
- Same source/date/processor/rule reuses the immutable version; a different source on the same date creates the next version and supersedes the old current only after all inserts pass.
- SQLSTATE 40001, 40P01, 55P03 and concurrent 23505 are retried by reopening the entire transaction, at most four attempts. Business validation errors are never retried.
- Outbox payloads contain only opaque IDs, dates, counts, version numbers, disposition and safe failure codes.

## Verified test-database state

Migration 008 and a privacy-safe 2026-07-27 vertical slice are deployed to the controlled remote test database `<ARR_DB_HOST>:5432/booking_test`. Successful and rejected delivery paths, all-outcome storage, booking lookup, total-price enforcement, current activation and idempotent replay have been tested. The 2026-07-28 full migration from the LAN source also passed independent structure, content-hash and application read-path checks. Exact hashes and counts are recorded in `database/APPLIED_MIGRATIONS.md` and `database/REMOTE_MIGRATION_20260728.md`.

The active controlled configuration is `/path/to/private/booking-test-db.env`. The old LAN configuration `/path/to/private/booking-test-lan-db.env` is retained only for temporary rollback checks. Production remains blocked until ARR has dedicated least-privilege roles, a concrete private OSS provider/bucket configuration in secret management, and the real SuperAgent file/callback contract.