118 lines
6.8 KiB
Markdown
118 lines
6.8 KiB
Markdown
# ARR validated daily ingestion
|
|
|
|
This package is the ARR-owned boundary between a SuperAgent delivery and PostgreSQL. It does not run the Agent or generate monthly reports. Production object materialization is supplied by the separate `arr_storage.ManagedObjectStore` boundary.
|
|
|
|
It supports two result-delivery modes:
|
|
|
|
- `artifact_callback`: the existing signed callback with Agent-produced OSS artifacts;
|
|
- `direct_mcp`: one complete structured payload submitted through the future ARR MCP gateway, independently replayed from the original private XML before Finance activation.
|
|
|
|
## Authority boundary
|
|
|
|
```text
|
|
private object store
|
|
-> DeliveryEnvelope 1.0
|
|
-> DeliveryValidator (hash/size/MIME + v3 contract + independent replay)
|
|
-> PostgresIngestionRepository (one atomic transaction)
|
|
-> current retained facts + privacy-minimized outbox
|
|
```
|
|
|
|
Only ARR calls this package and owns `ARR_DATABASE_URL`. SuperAgent/Agent receives opaque job/file handles and never receives database credentials.
|
|
|
|
## Direct MCP service boundary
|
|
|
|
The framework-neutral direct service is already implemented; the MCP protocol layer must remain a thin adapter over it:
|
|
|
|
```python
|
|
from arr_web.direct_ingestion_runtime import compose_oss_direct_ingestion
|
|
|
|
runtime = compose_oss_direct_ingestion(project_root=project_root)
|
|
try:
|
|
receipt = runtime.service.submit({
|
|
"submission_grant": short_lived_attempt_grant,
|
|
"job_id": job_id,
|
|
"attempt_no": attempt_no,
|
|
"payload": frozen_structured_result,
|
|
})
|
|
finally:
|
|
runtime.close()
|
|
```
|
|
|
|
ARR computes the canonical payload SHA-256 and byte size. The caller must not send a duplicate hash field. The payload is limited to 3 MiB, bound to one `job_id`/attempt grant, and accepts only successful, activation-eligible structured-result v3 data. The service then:
|
|
|
|
1. atomically consumes the hashed short-lived grant and stores a validating ledger row;
|
|
2. materializes the registered `source.xml` from private object storage;
|
|
3. reruns the allowlisted deterministic processor and compares every business fact;
|
|
4. writes the daily version, all outcome rows, channel metrics, current pointer and outbox atomically;
|
|
5. purges the guest-bearing temporary payload at every terminal state;
|
|
6. returns the prior receipt for an exact retry and rejects changed parameters as a conflict.
|
|
|
|
The public request and receipt contracts are
|
|
`database/contracts/arr-submit-processing-result-v1.schema.json` and
|
|
`database/contracts/arr-submit-processing-result-receipt-v1.schema.json`.
|
|
Migration `010_mcp_result_ingestion.sql` must be applied before this mode is enabled.
|
|
|
|
## Runtime requirements
|
|
|
|
- Python 3.9+
|
|
- `openpyxl==3.1.5` through `requirements-data-processing.txt`
|
|
- `psycopg[binary]==3.2.13` through `requirements-arr-ingestion.txt`
|
|
- applied PostgreSQL migration `008_arr_mvp_v1_rebuild.sql`
|
|
- the allowlisted `arr-opera-daily-ingest` Skill and its independent validator
|
|
|
|
Install the root `requirements.txt`; do not place a real DSN in source, CLI arguments, logs, prompts, envelopes, or test fixtures.
|
|
|
|
## Required lifecycle
|
|
|
|
1. ARR stores the uploaded XML as an immutable private object and constructs its `ArtifactRef`.
|
|
2. ARR registers `JobRegistration` before dispatch. The SHA-256 idempotency key identifies one job attempt.
|
|
3. SuperAgent returns artifacts only. ARR builds the strict `DeliveryEnvelope` and serializes it as UTF-8 JSON.
|
|
4. `DeliveryValidator.validate()` materializes objects into private temporary files, recomputes identity, checks processor/rule allowlist and v3 payload invariants, then invokes the independent Skill validator.
|
|
5. `IngestionService.ingest()` calls the PostgreSQL repository only after validation succeeds.
|
|
6. A successful transaction stores all source outcomes, exact Group Code lookup status and ordered channel metrics; it activates the new version and switches `current_daily_versions` at the end.
|
|
7. A deterministic processing failure stores a rejected daily version and all available source outcomes for audit, but never creates a current pointer or changes the last active business facts.
|
|
|
|
Minimal application wiring:
|
|
|
|
```python
|
|
from pathlib import Path
|
|
|
|
from arr_ingestion.postgres import DatabaseConfig, PostgresIngestionRepository
|
|
from arr_ingestion.service import IngestionService
|
|
from arr_ingestion.validation import DeliveryValidator, ProcessorPolicy
|
|
from arr_storage import FilesystemObjectBackend, ManagedObjectStore
|
|
|
|
policy = ProcessorPolicy(
|
|
processor_version="3.0.0",
|
|
rule_set_sha256="<allowlisted-64-hex-rule-hash>",
|
|
skill_root=Path("/private/runtime/arr-opera-daily-ingest"),
|
|
)
|
|
repository = PostgresIngestionRepository(DatabaseConfig.from_environment())
|
|
object_store = ManagedObjectStore(
|
|
FilesystemObjectBackend(Path("/private/object-store"))
|
|
)
|
|
service = IngestionService(
|
|
DeliveryValidator(object_store, policy),
|
|
repository,
|
|
)
|
|
|
|
outcome = service.ingest(raw_delivery_envelope)
|
|
```
|
|
|
|
`FilesystemObjectBackend` is for controlled local tests only. `arr_storage.CloudObjectBackend` exposes the same guarded path to a provider-specific OSS/S3 client port. Only immutable `committed` objects can be materialized, and the ARR validator still recomputes the delivery-declared identity before any database write.
|
|
|
|
## Transaction and retry semantics
|
|
|
|
- PostgreSQL target is hard-gated to `booking_test` in the current test deployment.
|
|
- Each write uses a new SERIALIZABLE transaction with local lock/statement timeouts.
|
|
- Same callback bytes are idempotent; same callback ID with different bytes is rejected.
|
|
- Same source/date/processor/rule reuses the immutable version; a different source on the same date creates the next version and supersedes the old current only after all inserts pass.
|
|
- SQLSTATE 40001, 40P01, 55P03 and concurrent 23505 are retried by reopening the entire transaction, at most four attempts. Business validation errors are never retried.
|
|
- Outbox payloads contain only opaque IDs, dates, counts, version numbers, disposition and safe failure codes.
|
|
|
|
## Verified test-database state
|
|
|
|
Migration 008 and a privacy-safe 2026-07-27 vertical slice are deployed to the controlled remote test database `<ARR_DB_HOST>:5432/booking_test`. Successful and rejected delivery paths, all-outcome storage, booking lookup, total-price enforcement, current activation and idempotent replay have been tested. The 2026-07-28 full migration from the LAN source also passed independent structure, content-hash and application read-path checks. Exact hashes and counts are recorded in `database/APPLIED_MIGRATIONS.md` and `database/REMOTE_MIGRATION_20260728.md`.
|
|
|
|
The active controlled configuration is `/path/to/private/booking-test-db.env`. The old LAN configuration `/path/to/private/booking-test-lan-db.env` is retained only for temporary rollback checks. Production remains blocked until ARR has dedicated least-privilege roles, a concrete private OSS provider/bucket configuration in secret management, and the real SuperAgent file/callback contract.
|