feat: move report artifacts to OSS storage

This commit is contained in:
Wyndham ARR
2026-08-04 12:37:26 +08:00
parent 727643f1f2
commit a2b86cdf10
45 changed files with 1288 additions and 1417 deletions

View File

@@ -4,6 +4,8 @@ Use this index for searchable, traceable evidence records.
| Date | Topic | Status | Source | Detail |
|---|---|---|---|---|
| 2026-08-04 | Report artifact deployability repair | Implemented locally; CentOS/Docker acceptance pending operator execution | [Evidence topic](topics/2026-08-04-report-artifact-deployability.md) | Monthly XLSX now uses Python/openpyxl with exact row-relative formulas and semantic validation. New monthly/company workbook and result artifacts use the existing immutable OSS adapter; download routing supports OSS plus legacy local records, and migration 016 permits OSS monthly identities. Focused report/storage/Web/migration tests pass; Docker/real OSS were intentionally not run locally. |
| 2026-08-04 | Deployed monthly-report download diagnosis | Superseded for implementation; optional authenticated remote capture pending | [Evidence topic](topics/2026-08-04-deployed-monthly-download-diagnosis.md) | The pre-repair Node/local-output diagnosis remains useful as historical cause evidence. The implementation now uses Python/openpyxl plus OSS-backed report identities and legacy local fallback; remote acceptance still belongs on the CentOS deployment. |
| 2026-08-03 | Company-report openpyxl builder for deployment | Implemented; focused deployment checks pass | [Evidence topic](topics/2026-08-03-company-report-openpyxl-builder.md) | CentOS Docker build failed because public npm cannot install private `@oai/artifact-tool`. Company-report XLSX generation now runs through Python/openpyxl in Web, while monthly worker packaging remains the only Node/artifact-tool concern. Focused company-report/deployment tests pass 19/19. |
| 2026-08-03 | Booking source and company-report retry semantics | Source contract confirmed; reuse visibility gap identified; live acceptance pending restart | [Evidence topic](topics/2026-08-03-company-report-retry-semantics.md) | Byte-identical XLSX uploads reuse the activated source; same rows with different file bytes are new drafts. Same report snapshots reuse complete publication pairs, changed Finance/Booking pins create new versions, and an activated subset changes Booking detail coverage while retaining Finance rows. The current API/UI do not expose whether a retry reused an existing artifact. |
| 2026-08-03 | Company-report retry after reported 8766 restart | Resolved by controlled listener replacement and live 5/5 rerun | [Evidence topic](topics/2026-08-03-company-source-runtime-diagnosis.md) | PID 11176 was replaced by PID 54127. Job `05cc547d…` succeeded all five August `01-10` companies, reused the prior version/artifact identities, and all five authenticated downloads returned hash-matching 200 responses. No Booking/Finance source or fact changed. |

View File

@@ -0,0 +1,35 @@
# Deployed monthly-report download diagnosis
## Metadata
- Date: 2026-08-04
- Status: Superseded for implementation by `2026-08-04-report-artifact-deployability`; authenticated remote capture remains optional operational evidence
- Scope: `http://8.138.234.141:8765`, monthly desktop list/download, comparison with Daily and company-channel downloads
- Confidence: Mixed — remote reachability/version facts are Fact; exact download failure classification is Inference until an authenticated click is captured
- Source: read-only `curl` probes, local source inspection at `727643f`, focused tests, deployment documentation
- Last verified: 2026-08-04
- Stale trigger: deployed image or migration 016 not yet applied; the source-level worker/download diagnosis below is now superseded by the OSS/openpyxl repair
## Facts
- The remote root redirects unauthenticated users to `/login`; protected desktop APIs and downloads return JSON `401 AUTH_REQUIRED`.
- Remote `/healthz` returns `ready`. Public H5 HTML/CSS/JS/i18n assets match the local `727643f` checkout byte-for-byte.
- Public `/api/public/h5/months` shows current Finance projections for 2026-08 and 2026-07, with the latest August projection updated through 2026-08-03. This proves the Web/database analytics read path is alive, but it does not prove that a local monthly XLSX archive is readable.
- The desktop monthly download route resolves a registered `monthly_xlsx` artifact from `reporting.monthly_runs`, then reads and re-hashes the file under Web's controlled project root. Daily artifacts are read from OSS; company artifacts are generated/read by Web from its local output root.
- The `727643f` change removed `@oai/artifact-tool` only from the company-report builder. Monthly XLSX generation still imports the private module through `monthly_reports/xlsx/build_workbook.mjs` and requires Node plus a separately supplied artifact-tool module.
- The checked-in Compose service mounts `/app/outputs` for Web but does not start a monthly worker. Deployment instructions require a separately managed worker with the same database and shared `/app/outputs` volume.
- Focused Web/repository/monthly worker/service/publisher tests pass 40/40 locally.
The source-level diagnosis above described the pre-repair Node/local-output architecture. The implementation now uses
Python/openpyxl and OSS-backed report identities with legacy local fallback; use the new deployability evidence topic
for the acceptance contract.
## Inference
The symptom pattern — Daily and company downloads work while a monthly workbook download fails — points first to deployment/runtime publication rather than a generic Web download-route defect. The monthly path uniquely depends on a locally registered archive being produced by the independent Node/artifact-tool worker and being visible at the same `/app/outputs` path inside Web. A missing worker, wrong worker output root, missing shared volume, or worker/Web running on different hosts can leave monthly metadata/analytics visible while the workbook download returns an artifact-read or integrity error.
The exact branch is not yet confirmed because the deployed desktop page requires a user login session. No credentials were read or submitted, and no server restart, report rerun, database write, or artifact repair was performed.
## Next verification
After an operator signs in through the in-app browser, capture one monthly row's `report_id`, the browser download response/status, and the corresponding error code. Then inspect only the deployed worker/Web runtime state: worker process/log, `reporting.monthly_runs` artifact identity, Web-visible `/app/outputs` path, and registered SHA-256/size. Do not rerun the report until the mismatch is identified.

View File

@@ -0,0 +1,45 @@
# Report artifact deployability repair
## Metadata
- Date: 2026-08-04
- Status: Implemented locally; CentOS/Docker deployment acceptance pending operator execution
- Scope: monthly openpyxl builder, monthly/company OSS publication, OSS/local download routing, migration 016
- Confidence: High for code/tests; Docker image and live OSS acceptance intentionally not run on this workstation
- Source: local source inspection, filesystem-backed object-store tests, Python unit/integration tests
## Implementation facts
- `monthly_reports/publishing.py` now builds monthly workbooks with Python/openpyxl, reopens each sheet, checks
sheet names, headers, row counts, exact `TOTAL PRICE` formulas, formula count and semantic SHA-256. Each data row
uses `=R[row]*C[row]*G[row]`.
- The monthly and company publishers use the existing `ManagedObjectStore` seam when composed for deployment. New
workbook and `result.json` objects are committed under report-specific immutable roles and registered with
`storage_provider='oss'`, bucket alias `arr-private`, object key, SHA-256, byte size and MIME type.
- `arr_web/downloads.py` routes `oss`/`s3` descriptors through the managed-object reader and keeps old local
descriptors on the controlled project-root reader. Both routes recheck stored/actual identity.
- `.web-jobs` remains local queue state. It is not uploaded to OSS.
- `database/016_monthly_report_oss_artifacts.sql` updates the 012 publication trigger to accept OSS/S3 or historical
local monthly artifacts. The down migration refuses to restore local-only validation while published non-local
artifacts exist.
- The stale Node builders, package manifests and old optional Node test were removed. Production entrypoints and
deployment docs no longer expose Node/npm/module-path flags.
- The Docker root dependency chain now installs `openpyxl==3.1.5` through `requirements-monthly-reports.txt`; no
separate npm or Codex-runtime bootstrap is needed.
## Verification
- Targeted report/web/storage/migration suite: 41 tests passed; builder/company integration slice: 15 tests passed
(56/56 together).
- Full local discovery: 315 tests executed; 307 passed, 3 skipped, and 8 environment errors were limited to the
pre-existing missing `httpx` Agent modules and unavailable Aliyun SDK/client construction. No failure was caused by
the report/OSS changes.
- Docker build/Compose and real Aliyun OSS were not run because the development environment has no Docker and the user
explicitly requested fake/in-memory storage tests instead.
## Operator acceptance still required
Apply migration 016 on the isolated `booking_test` database, rebuild the clean image, start Web and the independent
worker with the existing OSS/DB secrets, submit one controlled XML, wait for monthly publication, and verify that
monthly/company downloads still work after the local report cache is removed. The exact commands are in `deploy/README.md`
and the final task handoff.