shiyuyun/wyndham-ARR
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add daily manual price review workflow

Browse Source
This commit is contained in:
Wyndham ARR
2026-08-06 22:40:18 +08:00
parent aae3d8e1db
commit ca3e8e18fa
77 changed files with 7750 additions and 602 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
.project-docs/20-architecture/data-flow.md
View File

@@ -18,15 +18,17 @@
|---|---|---|---|
| 1 | Browser XML upload | Uploaded-basename task provenance + private canonical source object + queued DB job | Filename/content/size and immutable object identity |
| 2 | Committed source object | Isolated processor input | Materialization rechecks stored bytes |
| 3 | Fixed `process_daily.py` | Daily/result/structured or failure artifacts | Timeout, confined paths, exit/JSON agreement |
| 3 | Fixed `process_daily.py` | Final daily/result/structured, pure-price review JSON, or failure artifacts | Timeout, confined paths, exit/JSON agreement |
| 4 | Validated processor/report artifacts | Private committed OSS objects | Role, MIME, size and SHA-256 |
| 5 | Canonical `DeliveryEnvelope` | `DeliveryValidator` | Strict Schema, reconciliation, source/artifact hashes and independent validation |
| 6 | Verified delivery | PostgreSQL Finance facts | Serializable atomic commit/version activation |
| 7 | Accepted/failed run | Task trace and outbox | Persisted state is authoritative |
| 8 | `arr.daily_version_committed` | Dedicated monthly worker | Lease, `SKIP LOCKED`, bounded retry/dead-letter |
| 9 | Retained Finance `ARRIVAL` facts | Monthly snapshot request | Affected month from event version; “更新至” is max included `ARRIVAL` |
| 10 | Monthly snapshot | XLSX/result + `reporting` metadata | Formula reopen-validation, current-pin recheck, atomic activation |
| 11 | Registered active report | Web list/download + outbox acknowledgement | Real report ID; visible monthly tab polls automatically; path/size/SHA-256 recheck before download |
| 5 | Canonical `DeliveryEnvelope` | `DeliveryValidator` | Strict v4 Schema, reconciliation, source/artifact hashes and independent final/review replay |
| 6 | Pure `PRICE_UNMATCHED` delivery | PostgreSQL review case/items/events | No daily XLSX, Finance version, failure event or monthly event; task remains active as `awaiting_review` |
| 7 | Frozen complete review manifest + registered original source | Same processor and independent validator | UI/API accepts only a non-negative integer for a missing key; storage and the manifest normalize it to exact `.00`, and the manifest binds job/case/source/date/processor/rules/full key set |
| 8 | Final verified delivery | PostgreSQL Finance facts | Serializable atomic version activation, review completion and one commit outbox event |
| 9 | Accepted/failed run | Task trace and outbox | Persisted state is authoritative; retryable final infrastructure failure returns to `generation_failed` with manifest retained |
| 10 | `arr.daily_version_committed` | Dedicated monthly worker | Lease, `SKIP LOCKED`, bounded retry/dead-letter |
| 11 | Retained Finance `ARRIVAL` facts | Monthly snapshot request | Affected month from event version; “更新至” is max included `ARRIVAL` |
| 12 | Monthly snapshot | XLSX/result + `reporting` metadata | Formula reopen-validation, current-pin recheck, atomic activation |
| 13 | Registered active report | Web list/download + outbox acknowledgement | Real report ID; visible monthly tab polls automatically; path/size/SHA-256 recheck before download |
## State Ownership
@@ -36,6 +38,7 @@
- `processing_runs.uploaded_filename` owns the user-facing browser basename; the source artifact remains canonically
named `source.xml` for processing and validation.
- Temporary processor/validator paths are deleted after each request.
- `daily_review_cases`/items/events retain only normalized price keys, aggregate impact, actor/revision and before/after price; they deliberately contain no guest names, comments or raw traces. Frozen manifests and items are immutable.
- New monthly/company XLSX and `result.json` bytes live in immutable private OSS objects; the database stores provider,
bucket alias, object key and identity. The controlled local reader remains available for historical local records,
while `/app/outputs` is only staging/cache and local `.web-jobs` state.
@@ -65,4 +68,4 @@ equals its persisted `as_of_date`. Every data-row `TOTAL PRICE` cell is `=R[row]
## Last Updated
2026-07-31
2026-08-06

14
.project-docs/20-architecture/module-map.md
View File

@@ -3,28 +3,30 @@
| Path | Responsibility | Notes |
|---|---|---|
| `arr_web/auth.py` | Login credential verification and bounded attempt ledger | Environment-owned single-operator identity; dependency-free constant-time comparison; no secret logging |
| `arr_web/programmatic.py` | Upload-to-terminal coordinator | Registers the validated browser basename separately from canonical `source.xml`; no remote/model transport |
| `arr_web/programmatic.py` | XML upload and manual-price finalization coordinator | Registers the validated browser basename separately from canonical `source.xml`; pure price misses create no Finance version, while a frozen manifest replays the original source for final commit |
| `arr_web/booking_uploads.py`, `booking_ingestion/excel.py` | Authenticated Booking source upload and bounded raw XLSX parsing | Parser 2.0 reads Tour Code plus exact `โรงแรม`, latest-row cancellations and extracted room items with review state; coordinator/backend route contracts target drafts |
| `booking_ingestion/excel_postgres.py` | Earlier direct Booking full-source import/activation path | Its one-item immediate-accept implementation does not represent parser 2.0 review and must not be treated as the completed review repository |
| `booking_ingestion/excel_review.py`, `booking_ingestion/excel_review_postgres.py` | Editable extraction drafts and atomic reviewed-source activation | Single-operator repository uses advisory locking and a zero-pending gate; real PostgreSQL transaction/rollback acceptance passes; actor/reason/revision history is not implemented |
| `arr_web/processing_runtime.py` | PostgreSQL + OSS + processor composition | Active production processing composition |
| `arr_processing/local.py` | Bounded subprocess and output manifest extraction | Never exposes stdout/stderr |
| `arr_processing/policy.py` | Frozen processor/rule identity | Neutral, reusable loader |
| `arr_ingestion/validation.py` | Strict artifact/result validation | Runs `validate_daily.py` on success |
| `arr_ingestion/postgres.py` | Atomic Finance commit and lifecycle state | Four retries only for transient SQLSTATEs |
| `arr_processing/policy.py` | Frozen processor/rule identity | Active v4 loader plus isolated legacy direct-MCP v3 compatibility projection |
| `arr_ingestion/validation.py` | Strict artifact/result validation | Dispatches v3/v4; independently replays review and frozen-manifest final results |
| `arr_ingestion/postgres.py` | Job/review lifecycle and atomic Finance commit | `recorded_review` has no Finance/outbox write; final success is the only activation boundary; four retries only for transient SQLSTATEs |
| `arr_storage/aliyun_oss_v2.py` | Encrypted/unversioned OSS adapter | Writes all objects private |
| `arr_web/downloads.py` | OSS daily/report + controlled local legacy download routing | Rechecks provider metadata, size and SHA-256 |
| `arr_web/job_trace.py` | Programmatic persisted-fact trace | No external trace store |
| `arr_web/app.py`, `arr_web/repository.py`, `arr_web/company_jobs.py` | Authenticated portal routes plus public H5 aggregate routes, paged history reads and read-only history-month discovery | Default-deny login gate protects the desktop/API/download surface; purpose-built `/api/public/h5/*` exposes only sanitized aggregate metrics; daily/monthly counts and rows share a repeatable-read snapshot; `/api/history-months` merges daily/monthly database counts with company job-state counts; company totals/slices share one lock |
| `arr_web/server.py` | Standard-library HTTP transport | Dispatches GET/POST/PATCH/DELETE with one bounded body reader; real socket tests cover review update/delete and missing/oversized lengths |
| `arr_web/static/login.html`, `login.css`, `login.js` | Responsive ARR login gateway | Labeled form, password visibility, generic inline failures, safe desktop/H5 return target and reduced-motion support |
| `arr_web/static/app.js`, `arr_web/static/h5.js` | Authenticated desktop and anonymous-capable H5 client state, rendering and polling | Desktop daily/monthly/company histories own independent viewing-month state and default to the latest non-empty month; company generation month remains separate. H5 reads only public aggregate endpoints and may retain optional logout for an authenticated session. Desktop also includes 50-row Booking draft review/edit, the draft's validated uploaded filename below the review title, individual/all-visible selection, count-aware in-page delete confirmation and activation; company generation keeps the fixed five-company context beside the page title, a four-card upload/period setup row, cumulative CO display labels, short centered period actions and an in-page generation confirmation dialog; session-expiry redirect and CSRF logout remain shared; monthly versions auto-refresh the selected viewing month every four seconds |
| `arr_web/static/app.js`, `arr_web/static/i18n.js`, `arr_web/static/h5.js` | Authenticated desktop and anonymous-capable H5 client state, rendering, localization and polling | Desktop daily/monthly/company histories own independent viewing-month state and default to the latest non-empty month; needs-review status opens/focuses the paged privacy-safe price panel, finalization projects a generating upload state, and review controls/errors follow the shared Chinese/English/Thai catalog. H5 reads only public aggregate endpoints; session-expiry redirect and CSRF logout remain shared |
| `monthly_reports/worker.py` | Dedicated outbox consumer | Lease/reclaim, retry/dead-letter, success acknowledgement after activation |
| `monthly_reports/repository.py` | Monthly snapshot and publication repository | Derives scope from `ARRIVAL`; persists metadata/lineage/artifact identities |
| `monthly_reports/publishing.py` | Python/openpyxl monthly builder and atomic/OSS publisher | Reopens sheets, headers, row counts, semantic hash and exact row-relative `TOTAL PRICE` formulas |
| `monthly_reports/`, `company_reports/`, `channel_analytics/` | Downstream reports/BI | Consume accepted Finance facts |
| `database/012_monthly_report_publication.sql` | Additive metadata-only publication schema | Applied after immutable 008–011 baseline |
| `database/016_monthly_report_oss_artifacts.sql` | Monthly publication provider compatibility | Allows new OSS/S3 identities while retaining legacy local records |
| `database/017_daily_price_review.sql` | Daily missing-price review lifecycle | Applied to controlled `booking_test` on 2026-08-06 after exact 016 semantic verification and corrected up/down probe; adds protected case/item/event audit, `awaiting_review`, manual lineage and guarded rollback |
| `database/018_daily_review_manual_override_artifact.sql` | Frozen manual-manifest artifact compatibility | Applied as a forward-only correction after rollback probing; permits only `manual_override_json` in the existing artifact-kind constraint and refuses destructive rollback once such an immutable artifact exists |
| `database/014_booking_current_source_batch.sql` | Booking full-workbook current-source pointer and view scoping | Formally applied on 2026-07-31; batch 1 remains selected |
| `database/015_booking_excel_review_drafts.sql` | Item-level Booking extraction draft state | Formally applied and empty; basic latest-state review only, with no actor/reason/revision history and no DB-enforced zero-pending activation |
| `compose.yaml`, `deploy/` | Web-login + Caddy-HTTPS template and worker deployment boundary | Requires Web credentials for the desktop/operational surface; public H5 aggregate routes and `/healthz` remain anonymously reachable; no MCP port/domain/service |
@@ -35,4 +37,4 @@ by the active ARR2.0 entrypoint.
## Last Updated
2026-08-02
2026-08-06

14
.project-docs/20-architecture/system-overview.md
View File

@@ -5,7 +5,9 @@
ARR2.0 accepts Opera XML plus a separate Booking Tour Code/`โรงแรม` workbook and owns both deterministic processing
lifecycles. It writes immutable private
objects, runs the frozen processor locally, independently validates the complete artifacts, and atomically commits
accepted facts to PostgreSQL. Downstream analytics and report programs read committed facts.
accepted facts to PostgreSQL. A pure `PRICE_UNMATCHED` result instead enters an authenticated, auditable manual-price
review state and cannot create Finance facts until the original XML is replayed with a frozen canonical manifest.
Downstream analytics and report programs read committed facts.
Successful daily commits also create a durable outbox event. A separate worker derives the affected month and watermark
from committed `ARRIVAL` facts, publishes a validated workbook, and records metadata-only report state in PostgreSQL.
@@ -14,11 +16,11 @@ from committed `ARRIVAL` facts, publishes a validated workbook, and records meta
| Component | Responsibility |
|---|---|
| `arr_web.auth` / authenticated `SessionLedger` | Environment-owned operator credentials, bounded login attempts, random server-side sessions, CSRF and logout revocation |
| `arr_web.programmatic` | Terminal upload orchestration and safe public error mapping |
| `arr_web.programmatic` | XML upload plus synchronous frozen-review final replay orchestration and safe public error mapping |
| `arr_processing.local` / `arr_processing.policy` | Isolated frozen processor execution and approved version/rule identity |
| `arr_storage` | Immutable staged/committed OSS workflow; all object ACLs private |
| `arr_ingestion.validation` | Strict delivery/result/structured validation plus independent daily validation |
| `arr_ingestion.postgres` | Job lifecycle, artifact registration, atomic Finance version commit and outbox |
| `arr_ingestion.validation` | Strict v3/v4 delivery/result/structured validation plus independent daily/review-manifest replay |
| `arr_ingestion.postgres` | Job/review lifecycle, artifact registration, atomic final Finance version commit and outbox |
| `booking_ingestion.excel` | Bounded Tour Code/`โรงแรม` extraction, latest-row/cancellation semantics and room-item normalization |
| `booking_ingestion.excel_review_postgres` | Durable editable draft state and zero-pending atomic current-source activation |
| `arr_web.booking_uploads` | Private Booking workbook upload, draft review orchestration and privacy-minimized source summaries |
@@ -37,6 +39,8 @@ from committed `ARRIVAL` facts, publishes a validated workbook, and records meta
- Operator credentials come only from runtime `ARR_WEB_USERNAME` / `ARR_WEB_PASSWORD`; the application fails closed when either is absent. Authenticated mutations also require the per-session CSRF token.
- Processor exit code and JSON status must agree; output paths are confined to an isolated temporary directory.
- Validation completes before the database transaction; any transaction failure leaves no partial active version.
- `review_required` is valid only for a non-empty error set entirely composed of `PRICE_UNMATCHED`. It records neither a Finance version nor an `arr.processing_failed`/monthly event; a final manual value is limited to the verified missing key and is entered as a non-negative integer (including `0`) before exact `.00` storage/manifest normalization.
- All review mutations use the authenticated session plus CSRF protection and optimistic revision checks. Finalization is idempotent; infrastructure failures retain the frozen manifest for retry, while deterministic replay failures close the case and job as failed.
- Booking uploads create review drafts first. Only an explicit activation with no retained pending items can atomically
create immutable Booking facts and switch `booking.current_source_batch`; an open draft blocks company-report creation.
- Only recognized transient PostgreSQL concurrency errors receive bounded transaction retries.
@@ -53,4 +57,4 @@ from committed `ARRIVAL` facts, publishes a validated workbook, and records meta
## Last Updated
2026-08-03
2026-08-06