shiyuyun/wyndham-ARR
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: move report artifacts to OSS storage

Browse Source
This commit is contained in:
Wyndham ARR
2026-08-04 12:37:26 +08:00
parent 727643f1f2
commit a2b86cdf10
45 changed files with 1288 additions and 1417 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
.project-docs/20-architecture/data-flow.md
View File

@@ -19,7 +19,7 @@
| 1 | Browser XML upload | Uploaded-basename task provenance + private canonical source object + queued DB job | Filename/content/size and immutable object identity |
| 2 | Committed source object | Isolated processor input | Materialization rechecks stored bytes |
| 3 | Fixed `process_daily.py` | Daily/result/structured or failure artifacts | Timeout, confined paths, exit/JSON agreement |
| 4 | Local artifacts | Private committed OSS objects | Role, MIME, size and SHA-256 |
| 4 | Validated processor/report artifacts | Private committed OSS objects | Role, MIME, size and SHA-256 |
| 5 | Canonical `DeliveryEnvelope` | `DeliveryValidator` | Strict Schema, reconciliation, source/artifact hashes and independent validation |
| 6 | Verified delivery | PostgreSQL Finance facts | Serializable atomic commit/version activation |
| 7 | Accepted/failed run | Task trace and outbox | Persisted state is authoritative |
@@ -36,7 +36,9 @@
- `processing_runs.uploaded_filename` owns the user-facing browser basename; the source artifact remains canonically
named `source.xml` for processing and validation.
- Temporary processor/validator paths are deleted after each request.
- Monthly XLSX/result bytes live in immutable local archive paths on the current shared output volume; generated report artifacts are derived outputs, not an alternative business fact source.
- New monthly/company XLSX and `result.json` bytes live in immutable private OSS objects; the database stores provider,
bucket alias, object key and identity. The controlled local reader remains available for historical local records,
while `/app/outputs` is only staging/cache and local `.web-jobs` state.
## Booking Source And Company-Report Flow

5
.project-docs/20-architecture/module-map.md
View File

@@ -13,7 +13,7 @@
| `arr_ingestion/validation.py` | Strict artifact/result validation | Runs `validate_daily.py` on success |
| `arr_ingestion/postgres.py` | Atomic Finance commit and lifecycle state | Four retries only for transient SQLSTATEs |
| `arr_storage/aliyun_oss_v2.py` | Encrypted/unversioned OSS adapter | Writes all objects private |
| `arr_web/downloads.py` | OSS daily + controlled local report download routing | Rechecks metadata, size and SHA-256 |
| `arr_web/downloads.py` | OSS daily/report + controlled local legacy download routing | Rechecks provider metadata, size and SHA-256 |
| `arr_web/job_trace.py` | Programmatic persisted-fact trace | No external trace store |
| `arr_web/app.py`, `arr_web/repository.py`, `arr_web/company_jobs.py` | Authenticated portal routes plus public H5 aggregate routes, paged history reads and read-only history-month discovery | Default-deny login gate protects the desktop/API/download surface; purpose-built `/api/public/h5/*` exposes only sanitized aggregate metrics; daily/monthly counts and rows share a repeatable-read snapshot; `/api/history-months` merges daily/monthly database counts with company job-state counts; company totals/slices share one lock |
| `arr_web/server.py` | Standard-library HTTP transport | Dispatches GET/POST/PATCH/DELETE with one bounded body reader; real socket tests cover review update/delete and missing/oversized lengths |
@@ -21,9 +21,10 @@
| `arr_web/static/app.js`, `arr_web/static/h5.js` | Authenticated desktop and anonymous-capable H5 client state, rendering and polling | Desktop daily/monthly/company histories own independent viewing-month state and default to the latest non-empty month; company generation month remains separate. H5 reads only public aggregate endpoints and may retain optional logout for an authenticated session. Desktop also includes 50-row Booking draft review/edit, the draft's validated uploaded filename below the review title, individual/all-visible selection, count-aware in-page delete confirmation and activation; company generation keeps the fixed five-company context beside the page title, a four-card upload/period setup row, cumulative CO display labels, short centered period actions and an in-page generation confirmation dialog; session-expiry redirect and CSRF logout remain shared; monthly versions auto-refresh the selected viewing month every four seconds |
| `monthly_reports/worker.py` | Dedicated outbox consumer | Lease/reclaim, retry/dead-letter, success acknowledgement after activation |
| `monthly_reports/repository.py` | Monthly snapshot and publication repository | Derives scope from `ARRIVAL`; persists metadata/lineage/artifact identities |
| `monthly_reports/xlsx/build_workbook.mjs` | Monthly XLSX builder and reopen validator | Exact row-relative `TOTAL PRICE` formulas only in column S |
| `monthly_reports/publishing.py` | Python/openpyxl monthly builder and atomic/OSS publisher | Reopens sheets, headers, row counts, semantic hash and exact row-relative `TOTAL PRICE` formulas |
| `monthly_reports/`, `company_reports/`, `channel_analytics/` | Downstream reports/BI | Consume accepted Finance facts |
| `database/012_monthly_report_publication.sql` | Additive metadata-only publication schema | Applied after immutable 008–011 baseline |
| `database/016_monthly_report_oss_artifacts.sql` | Monthly publication provider compatibility | Allows new OSS/S3 identities while retaining legacy local records |
| `database/014_booking_current_source_batch.sql` | Booking full-workbook current-source pointer and view scoping | Formally applied on 2026-07-31; batch 1 remains selected |
| `database/015_booking_excel_review_drafts.sql` | Item-level Booking extraction draft state | Formally applied and empty; basic latest-state review only, with no actor/reason/revision history and no DB-enforced zero-pending activation |
| `compose.yaml`, `deploy/` | Web-login + Caddy-HTTPS template and worker deployment boundary | Requires Web credentials for the desktop/operational surface; public H5 aggregate routes and `/healthz` remain anonymously reachable; no MCP port/domain/service |

3
.project-docs/20-architecture/system-overview.md
View File

@@ -41,7 +41,8 @@ from committed `ARRIVAL` facts, publishes a validated workbook, and records meta
create immutable Booking facts and switch `booking.current_source_batch`; an open draft blocks company-report creation.
- Only recognized transient PostgreSQL concurrency errors receive bounded transaction retries.
- PostgreSQL state, not HTTP/console output, is authoritative for success.
- Daily downloads are read from OSS and rechecked; monthly downloads resolve only registered active/superseded local artifacts and recheck path, size and SHA-256.
- Daily and new report downloads are read from OSS and rechecked; historical local report artifacts remain readable through
the controlled project-root fallback with the same size and SHA-256 checks.
- Fresh uploads are new jobs/versions; exact delivery replay is idempotent.
- Monthly success is acknowledged only after both artifacts are registered and the publication is active; replay of the same snapshot returns the existing report.

28
.project-docs/30-worklog/current-state.md
View File

@@ -13,10 +13,20 @@ blocked. A not-yet-ended period keeps its fixed C/O cutoff and must be rerun aft
workbook needs those facts. Duplicate publication of the same semantic snapshot is idempotent: the first validated
archive/result pair remains authoritative even if a retry rebuilds different XLSX bytes.
Company-channel XLSX generation is now Web-container deployable without the private `@oai/artifact-tool` npm package:
`company_reports.publishing.ArtifactToolBuilder` builds and reopens workbooks with Python/openpyxl, validates sheet
names/headers/row counts/no-formulas, and treats blank Excel cells as business-empty strings. Node/artifact-tool remains
only a monthly-worker packaging concern.
Monthly and company-channel XLSX generation is now Web/container deployable with Python/openpyxl only. The monthly
builder preserves sheet/header/row/formula/semantic validation, including `=R[row]*C[row]*G[row]`; the company builder
preserves its no-formula contract. Both publishers upload new workbook and `result.json` objects through the existing
immutable OSS adapter, while download routing retains controlled-local compatibility for historical records and keeps
`.web-jobs` queue state local.
## Completed On 2026-08-04
- Repaired deployment availability for monthly and company report artifacts. Removed the monthly Node/private-package
runtime path and the stale Node builders/package manifests, added Python/openpyxl monthly validation, and added
shared OSS publication/read routing with hash/size/MIME rechecks. Migration 016 allows OSS monthly artifact
identities while preserving historical local rows. Local fake-object-store, builder, publisher, download-router and
migration tests pass; the development machine has no Docker, so CentOS image/Compose acceptance is explicitly handed
to the operator. No live report or business data was changed.
## Completed On 2026-08-03
@@ -450,14 +460,12 @@ only a monthly-worker packaging concern.
detached and are not reboot-persistent.
- The latest operator-selected Web credentials were rotated on 2026-07-31, but the password still matches the username.
Rotate it again to a distinct high-entropy value in Keychain, followed by one controlled Web restart.
- The workstation runs Web and worker as separate processes. The checked-in Compose image now supports company-channel
XLSX generation through Python/openpyxl in Web, but intentionally does not claim to run the monthly worker because it
does not package the workstation-only monthly artifact-tool module; a production image/process manager must supply
Node/artifact-tool and the shared output volume before enabling that monthly service.
- A formally controlled no-PII server acceptance run remains appropriate after that deployment packaging is complete.
- The workstation runs Web and worker as separate processes. The checked-in Compose image supports both report paths
through Python/openpyxl; the worker still needs the existing database/OSS secrets and may use `/app/outputs` only for
staging and local `.web-jobs` state. A formally controlled no-PII CentOS Docker acceptance run remains appropriate.
- ARR2.0 now has Git metadata; `main` tracks `origin/main`. Runtime credential values remain outside Git and project
files.
## Last Updated
2026-08-03
2026-08-04

1
.project-docs/30-worklog/task-history.md
View File

@@ -4,6 +4,7 @@
| Date | Task | Outcome | Docs Updated |
|---|---|---|---|
| 2026-08-04 | Remove private monthly XLSX runtime and make report artifacts deployment-safe | Replaced monthly and company report builders' production path with Python/openpyxl; preserved the exact monthly `TOTAL PRICE` formula and workbook semantic checks; published new monthly/company XLSX and `result.json` artifacts through the existing OSS adapter while retaining legacy local reads; added migration 016 for OSS monthly artifacts, removed Node builders/flags, and added OSS/local download routing tests. Targeted report/deployment tests pass; the full local suite has 307 passes, 3 skips and 8 environment-only errors (`httpx`/Aliyun test setup). Docker and live OSS were intentionally not run in this development environment | Current state, architecture, deployment runbook, migration ledger, evidence/index, stale item |
| 2026-08-03 | Make company-report XLSX generation deploy without private npm | Replaced the company-report builder's Node/private `@oai/artifact-tool` runtime dependency with Python/openpyxl workbook generation and self-validation, removed the private package dependency, enabled `--enable-company-reports` in Compose Web, and documented that only the monthly worker still needs Node/artifact-tool packaging. Focused company-report and deployment-entrypoint tests pass 19/19; no migration or live deployment was performed | Current state/history, architecture, deployment evidence/index, stale item, runbooks |
| 2026-08-03 | Execute controlled fix and live-accept 2026-08 `01-10` company report | With explicit confirmation, stopped only the stale PID 11176 and started new listener PID 54127 through the existing Keychain-backed launcher. One authorized job `05cc547d…` succeeded 5/5 with row counts `54/18/7/1/15`, captured active Booking batch-7 source metadata, reused the existing version/artifact identities, and passed five HTTP download/hash checks. Logout and temporary-file cleanup completed; no Booking/Finance source or fact mutation occurred | Current state, runtime evidence/index, stale item, commitments, scoped planning record |
| 2026-08-03 | Diagnose the latest 2026-08 `01-10` official-Excel save failure | Read-only evidence confirms newest job `087dceca...` built all five companies (`54/18/7/1/15`) and failed only at publish. Port 8766 is still PID 11176 from 2026-08-02, older than the semantic-reuse publisher fix; the earlier successful five-file archive/result/current set remains hash-consistent and intact. The launcher preflight is ready and six publisher tests pass. Repair is an exact controlled listener replacement followed by one authorized rerun; no restart, report write or business-data mutation was performed | Current state, runtime evidence/index, stale item, scoped planning record |

2
.project-docs/50-evidence/evidence-index.md
View File

@@ -4,6 +4,8 @@ Use this index for searchable, traceable evidence records.
| Date | Topic | Status | Source | Detail |
|---|---|---|---|---|
| 2026-08-04 | Report artifact deployability repair | Implemented locally; CentOS/Docker acceptance pending operator execution | [Evidence topic](topics/2026-08-04-report-artifact-deployability.md) | Monthly XLSX now uses Python/openpyxl with exact row-relative formulas and semantic validation. New monthly/company workbook and result artifacts use the existing immutable OSS adapter; download routing supports OSS plus legacy local records, and migration 016 permits OSS monthly identities. Focused report/storage/Web/migration tests pass; Docker/real OSS were intentionally not run locally. |
| 2026-08-04 | Deployed monthly-report download diagnosis | Superseded for implementation; optional authenticated remote capture pending | [Evidence topic](topics/2026-08-04-deployed-monthly-download-diagnosis.md) | The pre-repair Node/local-output diagnosis remains useful as historical cause evidence. The implementation now uses Python/openpyxl plus OSS-backed report identities and legacy local fallback; remote acceptance still belongs on the CentOS deployment. |
| 2026-08-03 | Company-report openpyxl builder for deployment | Implemented; focused deployment checks pass | [Evidence topic](topics/2026-08-03-company-report-openpyxl-builder.md) | CentOS Docker build failed because public npm cannot install private `@oai/artifact-tool`. Company-report XLSX generation now runs through Python/openpyxl in Web, while monthly worker packaging remains the only Node/artifact-tool concern. Focused company-report/deployment tests pass 19/19. |
| 2026-08-03 | Booking source and company-report retry semantics | Source contract confirmed; reuse visibility gap identified; live acceptance pending restart | [Evidence topic](topics/2026-08-03-company-report-retry-semantics.md) | Byte-identical XLSX uploads reuse the activated source; same rows with different file bytes are new drafts. Same report snapshots reuse complete publication pairs, changed Finance/Booking pins create new versions, and an activated subset changes Booking detail coverage while retaining Finance rows. The current API/UI do not expose whether a retry reused an existing artifact. |
| 2026-08-03 | Company-report retry after reported 8766 restart | Resolved by controlled listener replacement and live 5/5 rerun | [Evidence topic](topics/2026-08-03-company-source-runtime-diagnosis.md) | PID 11176 was replaced by PID 54127. Job `05cc547d…` succeeded all five August `01-10` companies, reused the prior version/artifact identities, and all five authenticated downloads returned hash-matching 200 responses. No Booking/Finance source or fact changed. |

35
.project-docs/50-evidence/topics/2026-08-04-deployed-monthly-download-diagnosis.md Normal file
View File

@@ -0,0 +1,35 @@
# Deployed monthly-report download diagnosis
## Metadata
- Date: 2026-08-04
- Status: Superseded for implementation by `2026-08-04-report-artifact-deployability`; authenticated remote capture remains optional operational evidence
- Scope: `http://8.138.234.141:8765`, monthly desktop list/download, comparison with Daily and company-channel downloads
- Confidence: Mixed — remote reachability/version facts are Fact; exact download failure classification is Inference until an authenticated click is captured
- Source: read-only `curl` probes, local source inspection at `727643f`, focused tests, deployment documentation
- Last verified: 2026-08-04
- Stale trigger: deployed image or migration 016 not yet applied; the source-level worker/download diagnosis below is now superseded by the OSS/openpyxl repair
## Facts
- The remote root redirects unauthenticated users to `/login`; protected desktop APIs and downloads return JSON `401 AUTH_REQUIRED`.
- Remote `/healthz` returns `ready`. Public H5 HTML/CSS/JS/i18n assets match the local `727643f` checkout byte-for-byte.
- Public `/api/public/h5/months` shows current Finance projections for 2026-08 and 2026-07, with the latest August projection updated through 2026-08-03. This proves the Web/database analytics read path is alive, but it does not prove that a local monthly XLSX archive is readable.
- The desktop monthly download route resolves a registered `monthly_xlsx` artifact from `reporting.monthly_runs`, then reads and re-hashes the file under Web's controlled project root. Daily artifacts are read from OSS; company artifacts are generated/read by Web from its local output root.
- The `727643f` change removed `@oai/artifact-tool` only from the company-report builder. Monthly XLSX generation still imports the private module through `monthly_reports/xlsx/build_workbook.mjs` and requires Node plus a separately supplied artifact-tool module.
- The checked-in Compose service mounts `/app/outputs` for Web but does not start a monthly worker. Deployment instructions require a separately managed worker with the same database and shared `/app/outputs` volume.
- Focused Web/repository/monthly worker/service/publisher tests pass 40/40 locally.
The source-level diagnosis above described the pre-repair Node/local-output architecture. The implementation now uses
Python/openpyxl and OSS-backed report identities with legacy local fallback; use the new deployability evidence topic
for the acceptance contract.
## Inference
The symptom pattern — Daily and company downloads work while a monthly workbook download fails — points first to deployment/runtime publication rather than a generic Web download-route defect. The monthly path uniquely depends on a locally registered archive being produced by the independent Node/artifact-tool worker and being visible at the same `/app/outputs` path inside Web. A missing worker, wrong worker output root, missing shared volume, or worker/Web running on different hosts can leave monthly metadata/analytics visible while the workbook download returns an artifact-read or integrity error.
The exact branch is not yet confirmed because the deployed desktop page requires a user login session. No credentials were read or submitted, and no server restart, report rerun, database write, or artifact repair was performed.
## Next verification
After an operator signs in through the in-app browser, capture one monthly row's `report_id`, the browser download response/status, and the corresponding error code. Then inspect only the deployed worker/Web runtime state: worker process/log, `reporting.monthly_runs` artifact identity, Web-visible `/app/outputs` path, and registered SHA-256/size. Do not rerun the report until the mismatch is identified.

45
.project-docs/50-evidence/topics/2026-08-04-report-artifact-deployability.md Normal file
View File

@@ -0,0 +1,45 @@
# Report artifact deployability repair
## Metadata
- Date: 2026-08-04
- Status: Implemented locally; CentOS/Docker deployment acceptance pending operator execution
- Scope: monthly openpyxl builder, monthly/company OSS publication, OSS/local download routing, migration 016
- Confidence: High for code/tests; Docker image and live OSS acceptance intentionally not run on this workstation
- Source: local source inspection, filesystem-backed object-store tests, Python unit/integration tests
## Implementation facts
- `monthly_reports/publishing.py` now builds monthly workbooks with Python/openpyxl, reopens each sheet, checks
sheet names, headers, row counts, exact `TOTAL PRICE` formulas, formula count and semantic SHA-256. Each data row
uses `=R[row]*C[row]*G[row]`.
- The monthly and company publishers use the existing `ManagedObjectStore` seam when composed for deployment. New
workbook and `result.json` objects are committed under report-specific immutable roles and registered with
`storage_provider='oss'`, bucket alias `arr-private`, object key, SHA-256, byte size and MIME type.
- `arr_web/downloads.py` routes `oss`/`s3` descriptors through the managed-object reader and keeps old local
descriptors on the controlled project-root reader. Both routes recheck stored/actual identity.
- `.web-jobs` remains local queue state. It is not uploaded to OSS.
- `database/016_monthly_report_oss_artifacts.sql` updates the 012 publication trigger to accept OSS/S3 or historical
local monthly artifacts. The down migration refuses to restore local-only validation while published non-local
artifacts exist.
- The stale Node builders, package manifests and old optional Node test were removed. Production entrypoints and
deployment docs no longer expose Node/npm/module-path flags.
- The Docker root dependency chain now installs `openpyxl==3.1.5` through `requirements-monthly-reports.txt`; no
separate npm or Codex-runtime bootstrap is needed.
## Verification
- Targeted report/web/storage/migration suite: 41 tests passed; builder/company integration slice: 15 tests passed
(56/56 together).
- Full local discovery: 315 tests executed; 307 passed, 3 skipped, and 8 environment errors were limited to the
pre-existing missing `httpx` Agent modules and unavailable Aliyun SDK/client construction. No failure was caused by
the report/OSS changes.
- Docker build/Compose and real Aliyun OSS were not run because the development environment has no Docker and the user
explicitly requested fake/in-memory storage tests instead.
## Operator acceptance still required
Apply migration 016 on the isolated `booking_test` database, rebuild the clean image, start Web and the independent
worker with the existing OSS/DB secrets, submit one controlled XML, wait for monthly publication, and verify that
monthly/company downloads still work after the local report cache is removed. The exact commands are in `deploy/README.md`
and the final task handoff.

2
.project-docs/90-maintenance/stale-items.md
View File

@@ -16,7 +16,7 @@
| 2026-07-30 | `booking_test` current Finance projection | The earlier 417-row snapshot included accepted run `mvp-v1-fixture-20260727` from `synthetic.xml`/`local_fixture`. A 2026-07-31 14:17 read-only company-report recheck found 986 current supported-company facts, so the old 416/417 remediation target is no longer a complete description of current Finance state | Re-audit current daily-version pins and source provenance before any fixture retirement; require explicit authorization for version changes, then verify the clean target and watermark |
| 2026-07-30 | Channel BI refresh lifecycle | Resolved 2026-08-03. Desktop and public H5 now check selected-month metadata every five seconds while visible and reload full analytics only after `updated_at` changes; hidden views pause and transient failures preserve the last good snapshot | Refresh the browser once to load the new static assets, then observe the next authorized publication as a live acceptance check |
| 2026-07-30 | Channel BI KPI label | The card labeled `公司数` renders worksheet-level `channel_count`; LianTai GROUP/FIT are two channels, so six does not mean six companies | Decide whether to relabel it `渠道/子表数` or implement an explicit five-company aggregation |
| 2026-07-30 | Production monthly-worker packaging | Local Web/worker separation is live, and company-report XLSX generation now runs in Web through Python/openpyxl. The checked-in Compose image still does not contain the workstation-only Node/artifact-tool runtime required by the monthly workbook builder | Package an approved monthly builder runtime and shared output volume before adding/enabling the managed production monthly-worker service |
| 2026-08-04 | Production monthly-worker packaging | Resolved in code: monthly/company builders are Python/openpyxl-only, new report artifacts use private OSS, and `.web-jobs` remains local. Docker/CentOS acceptance and migration 016 application are still operator-side deployment actions | Apply 016, rebuild the clean image, start Web plus the independent worker with existing OSS secrets, then verify OSS-backed downloads after removing the report cache |
## Superseded For ARR2.0