feat: move report artifacts to OSS storage

This commit is contained in:
Wyndham ARR
2026-08-04 12:37:26 +08:00
parent 727643f1f2
commit a2b86cdf10
45 changed files with 1288 additions and 1417 deletions

View File

@@ -19,7 +19,7 @@
| 1 | Browser XML upload | Uploaded-basename task provenance + private canonical source object + queued DB job | Filename/content/size and immutable object identity |
| 2 | Committed source object | Isolated processor input | Materialization rechecks stored bytes |
| 3 | Fixed `process_daily.py` | Daily/result/structured or failure artifacts | Timeout, confined paths, exit/JSON agreement |
| 4 | Local artifacts | Private committed OSS objects | Role, MIME, size and SHA-256 |
| 4 | Validated processor/report artifacts | Private committed OSS objects | Role, MIME, size and SHA-256 |
| 5 | Canonical `DeliveryEnvelope` | `DeliveryValidator` | Strict Schema, reconciliation, source/artifact hashes and independent validation |
| 6 | Verified delivery | PostgreSQL Finance facts | Serializable atomic commit/version activation |
| 7 | Accepted/failed run | Task trace and outbox | Persisted state is authoritative |
@@ -36,7 +36,9 @@
- `processing_runs.uploaded_filename` owns the user-facing browser basename; the source artifact remains canonically
named `source.xml` for processing and validation.
- Temporary processor/validator paths are deleted after each request.
- Monthly XLSX/result bytes live in immutable local archive paths on the current shared output volume; generated report artifacts are derived outputs, not an alternative business fact source.
- New monthly/company XLSX and `result.json` bytes live in immutable private OSS objects; the database stores provider,
bucket alias, object key and identity. The controlled local reader remains available for historical local records,
while `/app/outputs` is only staging/cache and local `.web-jobs` state.
## Booking Source And Company-Report Flow

View File

@@ -13,7 +13,7 @@
| `arr_ingestion/validation.py` | Strict artifact/result validation | Runs `validate_daily.py` on success |
| `arr_ingestion/postgres.py` | Atomic Finance commit and lifecycle state | Four retries only for transient SQLSTATEs |
| `arr_storage/aliyun_oss_v2.py` | Encrypted/unversioned OSS adapter | Writes all objects private |
| `arr_web/downloads.py` | OSS daily + controlled local report download routing | Rechecks metadata, size and SHA-256 |
| `arr_web/downloads.py` | OSS daily/report + controlled local legacy download routing | Rechecks provider metadata, size and SHA-256 |
| `arr_web/job_trace.py` | Programmatic persisted-fact trace | No external trace store |
| `arr_web/app.py`, `arr_web/repository.py`, `arr_web/company_jobs.py` | Authenticated portal routes plus public H5 aggregate routes, paged history reads and read-only history-month discovery | Default-deny login gate protects the desktop/API/download surface; purpose-built `/api/public/h5/*` exposes only sanitized aggregate metrics; daily/monthly counts and rows share a repeatable-read snapshot; `/api/history-months` merges daily/monthly database counts with company job-state counts; company totals/slices share one lock |
| `arr_web/server.py` | Standard-library HTTP transport | Dispatches GET/POST/PATCH/DELETE with one bounded body reader; real socket tests cover review update/delete and missing/oversized lengths |
@@ -21,9 +21,10 @@
| `arr_web/static/app.js`, `arr_web/static/h5.js` | Authenticated desktop and anonymous-capable H5 client state, rendering and polling | Desktop daily/monthly/company histories own independent viewing-month state and default to the latest non-empty month; company generation month remains separate. H5 reads only public aggregate endpoints and may retain optional logout for an authenticated session. Desktop also includes 50-row Booking draft review/edit, the draft's validated uploaded filename below the review title, individual/all-visible selection, count-aware in-page delete confirmation and activation; company generation keeps the fixed five-company context beside the page title, a four-card upload/period setup row, cumulative CO display labels, short centered period actions and an in-page generation confirmation dialog; session-expiry redirect and CSRF logout remain shared; monthly versions auto-refresh the selected viewing month every four seconds |
| `monthly_reports/worker.py` | Dedicated outbox consumer | Lease/reclaim, retry/dead-letter, success acknowledgement after activation |
| `monthly_reports/repository.py` | Monthly snapshot and publication repository | Derives scope from `ARRIVAL`; persists metadata/lineage/artifact identities |
| `monthly_reports/xlsx/build_workbook.mjs` | Monthly XLSX builder and reopen validator | Exact row-relative `TOTAL PRICE` formulas only in column S |
| `monthly_reports/publishing.py` | Python/openpyxl monthly builder and atomic/OSS publisher | Reopens sheets, headers, row counts, semantic hash and exact row-relative `TOTAL PRICE` formulas |
| `monthly_reports/`, `company_reports/`, `channel_analytics/` | Downstream reports/BI | Consume accepted Finance facts |
| `database/012_monthly_report_publication.sql` | Additive metadata-only publication schema | Applied after immutable 008011 baseline |
| `database/016_monthly_report_oss_artifacts.sql` | Monthly publication provider compatibility | Allows new OSS/S3 identities while retaining legacy local records |
| `database/014_booking_current_source_batch.sql` | Booking full-workbook current-source pointer and view scoping | Formally applied on 2026-07-31; batch 1 remains selected |
| `database/015_booking_excel_review_drafts.sql` | Item-level Booking extraction draft state | Formally applied and empty; basic latest-state review only, with no actor/reason/revision history and no DB-enforced zero-pending activation |
| `compose.yaml`, `deploy/` | Web-login + Caddy-HTTPS template and worker deployment boundary | Requires Web credentials for the desktop/operational surface; public H5 aggregate routes and `/healthz` remain anonymously reachable; no MCP port/domain/service |

View File

@@ -41,7 +41,8 @@ from committed `ARRIVAL` facts, publishes a validated workbook, and records meta
create immutable Booking facts and switch `booking.current_source_batch`; an open draft blocks company-report creation.
- Only recognized transient PostgreSQL concurrency errors receive bounded transaction retries.
- PostgreSQL state, not HTTP/console output, is authoritative for success.
- Daily downloads are read from OSS and rechecked; monthly downloads resolve only registered active/superseded local artifacts and recheck path, size and SHA-256.
- Daily and new report downloads are read from OSS and rechecked; historical local report artifacts remain readable through
the controlled project-root fallback with the same size and SHA-256 checks.
- Fresh uploads are new jobs/versions; exact delivery replay is idempotent.
- Monthly success is acknowledged only after both artifacts are registered and the publication is active; replay of the same snapshot returns the existing report.