shiyuyun/wyndham-ARR
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: sync latest ARR implementation

Browse Source
This commit is contained in:
Wyndham ARR
2026-07-31 15:11:42 +08:00
parent d6f8a747fa
commit bf7939dd1a
185 changed files with 17527 additions and 2260 deletions
Download Patch File Download Diff File Expand all files Collapse all files

118
arr_ingestion/README.md
View File

@@ -1,117 +1,21 @@
# ARR validated daily ingestion
This package is the ARR-owned boundary between a SuperAgent delivery and PostgreSQL. It does not run the Agent or generate monthly reports. Production object materialization is supplied by the separate `arr_storage.ManagedObjectStore` boundary.
It supports two result-delivery modes:
- `artifact_callback`: the existing signed callback with Agent-produced OSS artifacts;
- `direct_mcp`: one complete structured payload submitted through the future ARR MCP gateway, independently replayed from the original private XML before Finance activation.
## Authority boundary
This package is the authoritative validate-before-write boundary for ARR2.
```text
private object store
committed source/output objects
-> DeliveryEnvelope 1.0
-> DeliveryValidator (hash/size/MIME + v3 contract + independent replay)
-> DeliveryValidator (identity + strict v3 reconciliation + validate_daily.py)
-> PostgresIngestionRepository (one atomic transaction)
-> current retained facts + privacy-minimized outbox
-> current Finance facts + privacy-minimized outbox
```
Only ARR calls this package and owns `ARR_DATABASE_URL`. SuperAgent/Agent receives opaque job/file handles and never receives database credentials.
`JobRegistration` records source/processor/rule/attempt identity before processing. Controlled processor failures use a
complete failed delivery and retain auditable outcomes without changing the current Finance version. Timeouts,
process-start errors, malformed artifacts or pre-commit validation errors use explicit lifecycle failure state.
## Direct MCP service boundary
Only recognized transient PostgreSQL SQLSTATEs reopen the full transaction, at most four attempts. Business, Schema and
identity errors are never retried. Exact delivery replay is idempotent; changed bytes under the same ID conflict.
The framework-neutral direct service is already implemented; the MCP protocol layer must remain a thin adapter over it:
```python
from arr_web.direct_ingestion_runtime import compose_oss_direct_ingestion
runtime = compose_oss_direct_ingestion(project_root=project_root)
try:
receipt = runtime.service.submit({
"submission_grant": short_lived_attempt_grant,
"job_id": job_id,
"attempt_no": attempt_no,
"payload": frozen_structured_result,
})
finally:
runtime.close()
```
ARR computes the canonical payload SHA-256 and byte size. The caller must not send a duplicate hash field. The payload is limited to 3 MiB, bound to one `job_id`/attempt grant, and accepts only successful, activation-eligible structured-result v3 data. The service then:
1. atomically consumes the hashed short-lived grant and stores a validating ledger row;
2. materializes the registered `source.xml` from private object storage;
3. reruns the allowlisted deterministic processor and compares every business fact;
4. writes the daily version, all outcome rows, channel metrics, current pointer and outbox atomically;
5. purges the guest-bearing temporary payload at every terminal state;
6. returns the prior receipt for an exact retry and rejects changed parameters as a conflict.
The public request and receipt contracts are
`database/contracts/arr-submit-processing-result-v1.schema.json` and
`database/contracts/arr-submit-processing-result-receipt-v1.schema.json`.
Migration `010_mcp_result_ingestion.sql` must be applied before this mode is enabled.
## Runtime requirements
- Python 3.9+
- `openpyxl==3.1.5` through `requirements-data-processing.txt`
- `psycopg[binary]==3.2.13` through `requirements-arr-ingestion.txt`
- applied PostgreSQL migration `008_arr_mvp_v1_rebuild.sql`
- the allowlisted `arr-opera-daily-ingest` Skill and its independent validator
Install the root `requirements.txt`; do not place a real DSN in source, CLI arguments, logs, prompts, envelopes, or test fixtures.
## Required lifecycle
1. ARR stores the uploaded XML as an immutable private object and constructs its `ArtifactRef`.
2. ARR registers `JobRegistration` before dispatch. The SHA-256 idempotency key identifies one job attempt.
3. SuperAgent returns artifacts only. ARR builds the strict `DeliveryEnvelope` and serializes it as UTF-8 JSON.
4. `DeliveryValidator.validate()` materializes objects into private temporary files, recomputes identity, checks processor/rule allowlist and v3 payload invariants, then invokes the independent Skill validator.
5. `IngestionService.ingest()` calls the PostgreSQL repository only after validation succeeds.
6. A successful transaction stores all source outcomes, exact Group Code lookup status and ordered channel metrics; it activates the new version and switches `current_daily_versions` at the end.
7. A deterministic processing failure stores a rejected daily version and all available source outcomes for audit, but never creates a current pointer or changes the last active business facts.
Minimal application wiring:
```python
from pathlib import Path
from arr_ingestion.postgres import DatabaseConfig, PostgresIngestionRepository
from arr_ingestion.service import IngestionService
from arr_ingestion.validation import DeliveryValidator, ProcessorPolicy
from arr_storage import FilesystemObjectBackend, ManagedObjectStore
policy = ProcessorPolicy(
processor_version="3.0.0",
rule_set_sha256="<allowlisted-64-hex-rule-hash>",
skill_root=Path("/private/runtime/arr-opera-daily-ingest"),
)
repository = PostgresIngestionRepository(DatabaseConfig.from_environment())
object_store = ManagedObjectStore(
FilesystemObjectBackend(Path("/private/object-store"))
)
service = IngestionService(
DeliveryValidator(object_store, policy),
repository,
)
outcome = service.ingest(raw_delivery_envelope)
```
`FilesystemObjectBackend` is for controlled local tests only. `arr_storage.CloudObjectBackend` exposes the same guarded path to a provider-specific OSS/S3 client port. Only immutable `committed` objects can be materialized, and the ARR validator still recomputes the delivery-declared identity before any database write.
## Transaction and retry semantics
- PostgreSQL target is hard-gated to `booking_test` in the current test deployment.
- Each write uses a new SERIALIZABLE transaction with local lock/statement timeouts.
- Same callback bytes are idempotent; same callback ID with different bytes is rejected.
- Same source/date/processor/rule reuses the immutable version; a different source on the same date creates the next version and supersedes the old current only after all inserts pass.
- SQLSTATE 40001, 40P01, 55P03 and concurrent 23505 are retried by reopening the entire transaction, at most four attempts. Business validation errors are never retried.
- Outbox payloads contain only opaque IDs, dates, counts, version numbers, disposition and safe failure codes.
## Verified test-database state
Migration 008 and a privacy-safe 2026-07-27 vertical slice are deployed to the controlled remote test database `<ARR_DB_HOST>:5432/booking_test`. Successful and rejected delivery paths, all-outcome storage, booking lookup, total-price enforcement, current activation and idempotent replay have been tested. The 2026-07-28 full migration from the LAN source also passed independent structure, content-hash and application read-path checks. Exact hashes and counts are recorded in `database/APPLIED_MIGRATIONS.md` and `database/REMOTE_MIGRATION_20260728.md`.
The active controlled configuration is `/path/to/private/booking-test-db.env`. The old LAN configuration `/path/to/private/booking-test-lan-db.env` is retained only for temporary rollback checks. Production remains blocked until ARR has dedicated least-privilege roles, a concrete private OSS provider/bucket configuration in secret management, and the real SuperAgent file/callback contract.
Migrations 009/010 and direct-submission modules remain historical compatibility surfaces, but ARR2 uses the generic
artifact-delivery tables with their existing `artifact_callback` default and does not expose an MCP gateway.

2
arr_ingestion/contracts.py
View File

@@ -44,6 +44,7 @@ ARTIFACT_ROLES = {
}
ROLE_CONTRACTS = {
"source_xml": ("opera_xml", ".xml", "application/xml"),
"booking_source": ("booking_excel", ".xlsx", XLSX_MIME),
"daily_report": ("daily_xlsx", ".xlsx", XLSX_MIME),
"result_json": ("result_json", ".json", "application/json"),
"structured_result_json": (
@@ -55,6 +56,7 @@ ROLE_CONTRACTS = {
}
ARTIFACT_SIZE_LIMITS = {
"source_xml": 100 * 1024 * 1024,
"booking_source": 25 * 1024 * 1024,
"daily_report": 100 * 1024 * 1024,
"result_json": 5 * 1024 * 1024,
"structured_result_json": 50 * 1024 * 1024,

161
arr_ingestion/postgres.py
View File

@@ -216,19 +216,34 @@ class PostgresIngestionRepository(IngestionRepository):
def _ensure_artifact(cursor: Any, reference: ArtifactRef) -> int:
cursor.execute(
"""
SELECT id, byte_size, mime_type
SELECT
id,
artifact_kind,
original_filename,
sha256,
byte_size,
mime_type
FROM ingestion.artifacts
WHERE artifact_kind = %s
AND sha256 = %s
WHERE storage_provider = %s
AND bucket_alias = %s
AND object_key = %s
AND object_version_id IS NULL
FOR SHARE
""",
(reference.file_kind, reference.sha256),
(
ARTIFACT_STORAGE_PROVIDER,
ARTIFACT_BUCKET_ALIAS,
reference.object_key,
),
)
row = cursor.fetchone()
if row:
if (
int(row[1]) != reference.byte_size
or (row[2] or "") != reference.mime_type
str(row[1]) != reference.file_kind
or str(row[2]) != reference.original_filename
or str(row[3]) != reference.sha256
or int(row[4]) != reference.byte_size
or (row[5] or "") != reference.mime_type
):
raise IngestionError(
"ARTIFACT_CONFLICT",
@@ -278,6 +293,127 @@ class PostgresIngestionRepository(IngestionRepository):
"processing job could not be stored",
)
def mark_running(self, job_id: str, attempt_no: int) -> None:
self._run_transaction(
lambda cursor: self._mark_running(cursor, job_id, attempt_no),
"processing job could not be started",
)
def _mark_running(self, cursor: Any, job_id: str, attempt_no: int) -> None:
run_id, run_status, attempt_id, attempt_status = self._lock_attempt(
cursor, job_id, attempt_no
)
if run_status in {"accepted", "rejected", "failed", "cancelled"}:
raise IngestionError("JOB_TERMINAL", "processing job is already terminal")
if attempt_status in {"succeeded", "failed", "cancelled"}:
raise IngestionError("JOB_TERMINAL", "processing attempt is already terminal")
cursor.execute(
"""
UPDATE ingestion.processing_attempts
SET attempt_status = 'running',
started_at = COALESCE(started_at, now())
WHERE id = %s
""",
(attempt_id,),
)
cursor.execute(
"""
UPDATE ingestion.processing_runs
SET run_status = 'running',
failure_code = NULL,
failure_message = NULL,
updated_at = now()
WHERE id = %s
""",
(run_id,),
)
def record_failure(self, job_id: str, attempt_no: int, failure_code: str) -> None:
if not isinstance(failure_code, str) or re.fullmatch(
r"[A-Z][A-Z0-9_]{0,63}", failure_code
) is None:
failure_code = "PROCESSING_FAILED"
self._run_transaction(
lambda cursor: self._record_runtime_failure(
cursor, job_id, attempt_no, failure_code
),
"processing failure could not be stored",
)
def _record_runtime_failure(
self,
cursor: Any,
job_id: str,
attempt_no: int,
failure_code: str,
) -> None:
run_id, run_status, attempt_id, _attempt_status = self._lock_attempt(
cursor, job_id, attempt_no
)
if run_status == "failed":
return
if run_status in {"accepted", "rejected", "cancelled"}:
raise IngestionError("JOB_TERMINAL", "processing job is already terminal")
cursor.execute(
"""
UPDATE ingestion.processing_attempts
SET attempt_status = 'failed',
failure_code = %s,
failure_message = 'programmatic processing failed',
finished_at = now()
WHERE id = %s
""",
(failure_code, attempt_id),
)
cursor.execute(
"""
UPDATE ingestion.processing_runs
SET run_status = 'failed',
failure_code = %s,
failure_message = 'programmatic processing failed',
updated_at = now(),
finished_at = now()
WHERE id = %s
""",
(failure_code, run_id),
)
self._insert_outbox(
cursor,
f"processing-run:{run_id}:failed",
"processing_run",
run_id,
"arr.processing_failed",
{
"job_id": job_id,
"business_date": None,
"daily_version_id": None,
"failure_code": failure_code,
},
)
@staticmethod
def _lock_attempt(
cursor: Any,
job_id: str,
attempt_no: int,
) -> Tuple[int, str, int, str]:
cursor.execute(
"""
SELECT run.id, run.run_status, attempt.id, attempt.attempt_status
FROM ingestion.processing_runs AS run
JOIN ingestion.processing_attempts AS attempt
ON attempt.processing_run_id = run.id
AND attempt.attempt_no = %s
WHERE run.run_key = %s
FOR UPDATE OF run, attempt
""",
(attempt_no, job_id),
)
row = cursor.fetchone()
if not row:
raise IngestionError("JOB_NOT_FOUND", "processing job was not registered")
return int(row[0]), str(row[1]), int(row[2]), str(row[3])
def _register_job(self, cursor: Any, registration: JobRegistration) -> None:
source_artifact_id = self._ensure_artifact(cursor, registration.source)
cursor.execute(
@@ -287,6 +423,7 @@ class PostgresIngestionRepository(IngestionRepository):
source_artifact_id,
requested_processor_version,
requested_rule_set_sha256,
uploaded_filename,
run_status
FROM ingestion.processing_runs
WHERE run_key = %s
@@ -309,9 +446,10 @@ class PostgresIngestionRepository(IngestionRepository):
source_artifact_id,
run_status,
requested_processor_version,
requested_rule_set_sha256
requested_rule_set_sha256,
uploaded_filename
)
VALUES (%s, 'opera_daily', %s, 'queued', %s, %s)
VALUES (%s, 'opera_daily', %s, 'queued', %s, %s, %s)
RETURNING id
""",
(
@@ -319,6 +457,7 @@ class PostgresIngestionRepository(IngestionRepository):
source_artifact_id,
registration.processor_version,
registration.rule_set_sha256,
registration.uploaded_filename,
),
)
processing_run_id = int(cursor.fetchone()[0])
@@ -327,6 +466,7 @@ class PostgresIngestionRepository(IngestionRepository):
int(existing[1]) != source_artifact_id
or existing[2] != registration.processor_version
or str(existing[3]) != registration.rule_set_sha256
or existing[4] != registration.uploaded_filename
):
raise IngestionError(
"JOB_CONFLICT",
@@ -350,7 +490,7 @@ class PostgresIngestionRepository(IngestionRepository):
"processing attempt identity conflicts",
)
return
if str(existing[4]) in {
if str(existing[5]) in {
"accepted",
"rejected",
"failed",
@@ -1121,6 +1261,9 @@ class PostgresIngestionRepository(IngestionRepository):
source.group_code_key,
count(DISTINCT source.id)
FROM booking.source_rows AS source
JOIN booking.current_source_batch AS active_source
ON active_source.source_batch_id = source.source_batch_id
AND active_source.singleton
JOIN booking.source_batches AS batch
ON batch.id = source.source_batch_id
AND batch.batch_status = 'accepted'

30
arr_ingestion/repository.py
View File

@@ -20,6 +20,7 @@ class JobRegistration:
rule_set_sha256: str
attempt_no: int
idempotency_key: str
uploaded_filename: Optional[str] = None
@dataclass(frozen=True)
@@ -35,6 +36,12 @@ class IngestionRepository(Protocol):
def register_job(self, registration: JobRegistration) -> None:
...
def mark_running(self, job_id: str, attempt_no: int) -> None:
...
def record_failure(self, job_id: str, attempt_no: int, failure_code: str) -> None:
...
def commit_delivery(self, delivery: VerifiedDelivery) -> IngestionOutcome:
...
@@ -81,6 +88,29 @@ class InMemoryIngestionRepository:
return
self._jobs[registration.job_id] = _MemoryJob(registration=registration)
def mark_running(self, job_id: str, attempt_no: int) -> None:
with self._lock:
job = self._jobs.get(job_id)
if job is None:
raise IngestionError("JOB_NOT_FOUND", "processing job was not registered")
if attempt_no != job.registration.attempt_no:
raise IngestionError("JOB_NOT_FOUND", "processing attempt was not registered")
if job.status in {"succeeded", "failed"}:
raise IngestionError("JOB_TERMINAL", "processing job is already terminal")
job.status = "running"
def record_failure(self, job_id: str, attempt_no: int, failure_code: str) -> None:
with self._lock:
job = self._jobs.get(job_id)
if job is None:
raise IngestionError("JOB_NOT_FOUND", "processing job was not registered")
if attempt_no != job.registration.attempt_no:
raise IngestionError("JOB_NOT_FOUND", "processing attempt was not registered")
if job.status == "succeeded":
raise IngestionError("JOB_TERMINAL", "processing job is already terminal")
job.status = "failed"
job.failure_code = failure_code
def commit_delivery(self, delivery: VerifiedDelivery) -> IngestionOutcome:
envelope = delivery.envelope
with self._lock:

2
arr_ingestion/validation.py
View File

@@ -169,6 +169,8 @@ class ProcessorPolicy:
raise ValueError("processor policy identity is invalid")
resolved = self.skill_root.resolve()
object.__setattr__(self, "skill_root", resolved)
if not (resolved / "scripts" / "process_daily.py").is_file():
raise ValueError("processor executable is unavailable")
if not (resolved / "scripts" / "validate_daily.py").is_file():
raise ValueError("processor validator is unavailable")
if not (resolved / "references" / "价格对照.xlsx").is_file():