diff --git a/.project-docs/20-architecture/data-flow.md b/.project-docs/20-architecture/data-flow.md index 810c86f..9bc8e7a 100644 --- a/.project-docs/20-architecture/data-flow.md +++ b/.project-docs/20-architecture/data-flow.md @@ -19,7 +19,7 @@ | 1 | Browser XML upload | Uploaded-basename task provenance + private canonical source object + queued DB job | Filename/content/size and immutable object identity | | 2 | Committed source object | Isolated processor input | Materialization rechecks stored bytes | | 3 | Fixed `process_daily.py` | Daily/result/structured or failure artifacts | Timeout, confined paths, exit/JSON agreement | -| 4 | Local artifacts | Private committed OSS objects | Role, MIME, size and SHA-256 | +| 4 | Validated processor/report artifacts | Private committed OSS objects | Role, MIME, size and SHA-256 | | 5 | Canonical `DeliveryEnvelope` | `DeliveryValidator` | Strict Schema, reconciliation, source/artifact hashes and independent validation | | 6 | Verified delivery | PostgreSQL Finance facts | Serializable atomic commit/version activation | | 7 | Accepted/failed run | Task trace and outbox | Persisted state is authoritative | @@ -36,7 +36,9 @@ - `processing_runs.uploaded_filename` owns the user-facing browser basename; the source artifact remains canonically named `source.xml` for processing and validation. - Temporary processor/validator paths are deleted after each request. -- Monthly XLSX/result bytes live in immutable local archive paths on the current shared output volume; generated report artifacts are derived outputs, not an alternative business fact source. +- New monthly/company XLSX and `result.json` bytes live in immutable private OSS objects; the database stores provider, + bucket alias, object key and identity. The controlled local reader remains available for historical local records, + while `/app/outputs` is only staging/cache and local `.web-jobs` state. ## Booking Source And Company-Report Flow diff --git a/.project-docs/20-architecture/module-map.md b/.project-docs/20-architecture/module-map.md index 67191fc..acc3abb 100644 --- a/.project-docs/20-architecture/module-map.md +++ b/.project-docs/20-architecture/module-map.md @@ -13,7 +13,7 @@ | `arr_ingestion/validation.py` | Strict artifact/result validation | Runs `validate_daily.py` on success | | `arr_ingestion/postgres.py` | Atomic Finance commit and lifecycle state | Four retries only for transient SQLSTATEs | | `arr_storage/aliyun_oss_v2.py` | Encrypted/unversioned OSS adapter | Writes all objects private | -| `arr_web/downloads.py` | OSS daily + controlled local report download routing | Rechecks metadata, size and SHA-256 | +| `arr_web/downloads.py` | OSS daily/report + controlled local legacy download routing | Rechecks provider metadata, size and SHA-256 | | `arr_web/job_trace.py` | Programmatic persisted-fact trace | No external trace store | | `arr_web/app.py`, `arr_web/repository.py`, `arr_web/company_jobs.py` | Authenticated portal routes plus public H5 aggregate routes, paged history reads and read-only history-month discovery | Default-deny login gate protects the desktop/API/download surface; purpose-built `/api/public/h5/*` exposes only sanitized aggregate metrics; daily/monthly counts and rows share a repeatable-read snapshot; `/api/history-months` merges daily/monthly database counts with company job-state counts; company totals/slices share one lock | | `arr_web/server.py` | Standard-library HTTP transport | Dispatches GET/POST/PATCH/DELETE with one bounded body reader; real socket tests cover review update/delete and missing/oversized lengths | @@ -21,9 +21,10 @@ | `arr_web/static/app.js`, `arr_web/static/h5.js` | Authenticated desktop and anonymous-capable H5 client state, rendering and polling | Desktop daily/monthly/company histories own independent viewing-month state and default to the latest non-empty month; company generation month remains separate. H5 reads only public aggregate endpoints and may retain optional logout for an authenticated session. Desktop also includes 50-row Booking draft review/edit, the draft's validated uploaded filename below the review title, individual/all-visible selection, count-aware in-page delete confirmation and activation; company generation keeps the fixed five-company context beside the page title, a four-card upload/period setup row, cumulative CO display labels, short centered period actions and an in-page generation confirmation dialog; session-expiry redirect and CSRF logout remain shared; monthly versions auto-refresh the selected viewing month every four seconds | | `monthly_reports/worker.py` | Dedicated outbox consumer | Lease/reclaim, retry/dead-letter, success acknowledgement after activation | | `monthly_reports/repository.py` | Monthly snapshot and publication repository | Derives scope from `ARRIVAL`; persists metadata/lineage/artifact identities | -| `monthly_reports/xlsx/build_workbook.mjs` | Monthly XLSX builder and reopen validator | Exact row-relative `TOTAL PRICE` formulas only in column S | +| `monthly_reports/publishing.py` | Python/openpyxl monthly builder and atomic/OSS publisher | Reopens sheets, headers, row counts, semantic hash and exact row-relative `TOTAL PRICE` formulas | | `monthly_reports/`, `company_reports/`, `channel_analytics/` | Downstream reports/BI | Consume accepted Finance facts | | `database/012_monthly_report_publication.sql` | Additive metadata-only publication schema | Applied after immutable 008–011 baseline | +| `database/016_monthly_report_oss_artifacts.sql` | Monthly publication provider compatibility | Allows new OSS/S3 identities while retaining legacy local records | | `database/014_booking_current_source_batch.sql` | Booking full-workbook current-source pointer and view scoping | Formally applied on 2026-07-31; batch 1 remains selected | | `database/015_booking_excel_review_drafts.sql` | Item-level Booking extraction draft state | Formally applied and empty; basic latest-state review only, with no actor/reason/revision history and no DB-enforced zero-pending activation | | `compose.yaml`, `deploy/` | Web-login + Caddy-HTTPS template and worker deployment boundary | Requires Web credentials for the desktop/operational surface; public H5 aggregate routes and `/healthz` remain anonymously reachable; no MCP port/domain/service | diff --git a/.project-docs/20-architecture/system-overview.md b/.project-docs/20-architecture/system-overview.md index da4f64b..0cb7840 100644 --- a/.project-docs/20-architecture/system-overview.md +++ b/.project-docs/20-architecture/system-overview.md @@ -41,7 +41,8 @@ from committed `ARRIVAL` facts, publishes a validated workbook, and records meta create immutable Booking facts and switch `booking.current_source_batch`; an open draft blocks company-report creation. - Only recognized transient PostgreSQL concurrency errors receive bounded transaction retries. - PostgreSQL state, not HTTP/console output, is authoritative for success. -- Daily downloads are read from OSS and rechecked; monthly downloads resolve only registered active/superseded local artifacts and recheck path, size and SHA-256. +- Daily and new report downloads are read from OSS and rechecked; historical local report artifacts remain readable through + the controlled project-root fallback with the same size and SHA-256 checks. - Fresh uploads are new jobs/versions; exact delivery replay is idempotent. - Monthly success is acknowledged only after both artifacts are registered and the publication is active; replay of the same snapshot returns the existing report. diff --git a/.project-docs/30-worklog/current-state.md b/.project-docs/30-worklog/current-state.md index 5942a0c..85b109c 100644 --- a/.project-docs/30-worklog/current-state.md +++ b/.project-docs/30-worklog/current-state.md @@ -13,10 +13,20 @@ blocked. A not-yet-ended period keeps its fixed C/O cutoff and must be rerun aft workbook needs those facts. Duplicate publication of the same semantic snapshot is idempotent: the first validated archive/result pair remains authoritative even if a retry rebuilds different XLSX bytes. -Company-channel XLSX generation is now Web-container deployable without the private `@oai/artifact-tool` npm package: -`company_reports.publishing.ArtifactToolBuilder` builds and reopens workbooks with Python/openpyxl, validates sheet -names/headers/row counts/no-formulas, and treats blank Excel cells as business-empty strings. Node/artifact-tool remains -only a monthly-worker packaging concern. +Monthly and company-channel XLSX generation is now Web/container deployable with Python/openpyxl only. The monthly +builder preserves sheet/header/row/formula/semantic validation, including `=R[row]*C[row]*G[row]`; the company builder +preserves its no-formula contract. Both publishers upload new workbook and `result.json` objects through the existing +immutable OSS adapter, while download routing retains controlled-local compatibility for historical records and keeps +`.web-jobs` queue state local. + +## Completed On 2026-08-04 + +- Repaired deployment availability for monthly and company report artifacts. Removed the monthly Node/private-package + runtime path and the stale Node builders/package manifests, added Python/openpyxl monthly validation, and added + shared OSS publication/read routing with hash/size/MIME rechecks. Migration 016 allows OSS monthly artifact + identities while preserving historical local rows. Local fake-object-store, builder, publisher, download-router and + migration tests pass; the development machine has no Docker, so CentOS image/Compose acceptance is explicitly handed + to the operator. No live report or business data was changed. ## Completed On 2026-08-03 @@ -450,14 +460,12 @@ only a monthly-worker packaging concern. detached and are not reboot-persistent. - The latest operator-selected Web credentials were rotated on 2026-07-31, but the password still matches the username. Rotate it again to a distinct high-entropy value in Keychain, followed by one controlled Web restart. -- The workstation runs Web and worker as separate processes. The checked-in Compose image now supports company-channel - XLSX generation through Python/openpyxl in Web, but intentionally does not claim to run the monthly worker because it - does not package the workstation-only monthly artifact-tool module; a production image/process manager must supply - Node/artifact-tool and the shared output volume before enabling that monthly service. -- A formally controlled no-PII server acceptance run remains appropriate after that deployment packaging is complete. +- The workstation runs Web and worker as separate processes. The checked-in Compose image supports both report paths + through Python/openpyxl; the worker still needs the existing database/OSS secrets and may use `/app/outputs` only for + staging and local `.web-jobs` state. A formally controlled no-PII CentOS Docker acceptance run remains appropriate. - ARR2.0 now has Git metadata; `main` tracks `origin/main`. Runtime credential values remain outside Git and project files. ## Last Updated -2026-08-03 +2026-08-04 diff --git a/.project-docs/30-worklog/task-history.md b/.project-docs/30-worklog/task-history.md index 731436e..409530d 100644 --- a/.project-docs/30-worklog/task-history.md +++ b/.project-docs/30-worklog/task-history.md @@ -4,6 +4,7 @@ | Date | Task | Outcome | Docs Updated | |---|---|---|---| +| 2026-08-04 | Remove private monthly XLSX runtime and make report artifacts deployment-safe | Replaced monthly and company report builders' production path with Python/openpyxl; preserved the exact monthly `TOTAL PRICE` formula and workbook semantic checks; published new monthly/company XLSX and `result.json` artifacts through the existing OSS adapter while retaining legacy local reads; added migration 016 for OSS monthly artifacts, removed Node builders/flags, and added OSS/local download routing tests. Targeted report/deployment tests pass; the full local suite has 307 passes, 3 skips and 8 environment-only errors (`httpx`/Aliyun test setup). Docker and live OSS were intentionally not run in this development environment | Current state, architecture, deployment runbook, migration ledger, evidence/index, stale item | | 2026-08-03 | Make company-report XLSX generation deploy without private npm | Replaced the company-report builder's Node/private `@oai/artifact-tool` runtime dependency with Python/openpyxl workbook generation and self-validation, removed the private package dependency, enabled `--enable-company-reports` in Compose Web, and documented that only the monthly worker still needs Node/artifact-tool packaging. Focused company-report and deployment-entrypoint tests pass 19/19; no migration or live deployment was performed | Current state/history, architecture, deployment evidence/index, stale item, runbooks | | 2026-08-03 | Execute controlled fix and live-accept 2026-08 `01-10` company report | With explicit confirmation, stopped only the stale PID 11176 and started new listener PID 54127 through the existing Keychain-backed launcher. One authorized job `05cc547d…` succeeded 5/5 with row counts `54/18/7/1/15`, captured active Booking batch-7 source metadata, reused the existing version/artifact identities, and passed five HTTP download/hash checks. Logout and temporary-file cleanup completed; no Booking/Finance source or fact mutation occurred | Current state, runtime evidence/index, stale item, commitments, scoped planning record | | 2026-08-03 | Diagnose the latest 2026-08 `01-10` official-Excel save failure | Read-only evidence confirms newest job `087dceca...` built all five companies (`54/18/7/1/15`) and failed only at publish. Port 8766 is still PID 11176 from 2026-08-02, older than the semantic-reuse publisher fix; the earlier successful five-file archive/result/current set remains hash-consistent and intact. The launcher preflight is ready and six publisher tests pass. Repair is an exact controlled listener replacement followed by one authorized rerun; no restart, report write or business-data mutation was performed | Current state, runtime evidence/index, stale item, scoped planning record | diff --git a/.project-docs/50-evidence/evidence-index.md b/.project-docs/50-evidence/evidence-index.md index 9b07695..f0d179f 100644 --- a/.project-docs/50-evidence/evidence-index.md +++ b/.project-docs/50-evidence/evidence-index.md @@ -4,6 +4,8 @@ Use this index for searchable, traceable evidence records. | Date | Topic | Status | Source | Detail | |---|---|---|---|---| +| 2026-08-04 | Report artifact deployability repair | Implemented locally; CentOS/Docker acceptance pending operator execution | [Evidence topic](topics/2026-08-04-report-artifact-deployability.md) | Monthly XLSX now uses Python/openpyxl with exact row-relative formulas and semantic validation. New monthly/company workbook and result artifacts use the existing immutable OSS adapter; download routing supports OSS plus legacy local records, and migration 016 permits OSS monthly identities. Focused report/storage/Web/migration tests pass; Docker/real OSS were intentionally not run locally. | +| 2026-08-04 | Deployed monthly-report download diagnosis | Superseded for implementation; optional authenticated remote capture pending | [Evidence topic](topics/2026-08-04-deployed-monthly-download-diagnosis.md) | The pre-repair Node/local-output diagnosis remains useful as historical cause evidence. The implementation now uses Python/openpyxl plus OSS-backed report identities and legacy local fallback; remote acceptance still belongs on the CentOS deployment. | | 2026-08-03 | Company-report openpyxl builder for deployment | Implemented; focused deployment checks pass | [Evidence topic](topics/2026-08-03-company-report-openpyxl-builder.md) | CentOS Docker build failed because public npm cannot install private `@oai/artifact-tool`. Company-report XLSX generation now runs through Python/openpyxl in Web, while monthly worker packaging remains the only Node/artifact-tool concern. Focused company-report/deployment tests pass 19/19. | | 2026-08-03 | Booking source and company-report retry semantics | Source contract confirmed; reuse visibility gap identified; live acceptance pending restart | [Evidence topic](topics/2026-08-03-company-report-retry-semantics.md) | Byte-identical XLSX uploads reuse the activated source; same rows with different file bytes are new drafts. Same report snapshots reuse complete publication pairs, changed Finance/Booking pins create new versions, and an activated subset changes Booking detail coverage while retaining Finance rows. The current API/UI do not expose whether a retry reused an existing artifact. | | 2026-08-03 | Company-report retry after reported 8766 restart | Resolved by controlled listener replacement and live 5/5 rerun | [Evidence topic](topics/2026-08-03-company-source-runtime-diagnosis.md) | PID 11176 was replaced by PID 54127. Job `05cc547d…` succeeded all five August `01-10` companies, reused the prior version/artifact identities, and all five authenticated downloads returned hash-matching 200 responses. No Booking/Finance source or fact changed. | diff --git a/.project-docs/50-evidence/topics/2026-08-04-deployed-monthly-download-diagnosis.md b/.project-docs/50-evidence/topics/2026-08-04-deployed-monthly-download-diagnosis.md new file mode 100644 index 0000000..bc693ea --- /dev/null +++ b/.project-docs/50-evidence/topics/2026-08-04-deployed-monthly-download-diagnosis.md @@ -0,0 +1,35 @@ +# Deployed monthly-report download diagnosis + +## Metadata + +- Date: 2026-08-04 +- Status: Superseded for implementation by `2026-08-04-report-artifact-deployability`; authenticated remote capture remains optional operational evidence +- Scope: `http://8.138.234.141:8765`, monthly desktop list/download, comparison with Daily and company-channel downloads +- Confidence: Mixed — remote reachability/version facts are Fact; exact download failure classification is Inference until an authenticated click is captured +- Source: read-only `curl` probes, local source inspection at `727643f`, focused tests, deployment documentation +- Last verified: 2026-08-04 +- Stale trigger: deployed image or migration 016 not yet applied; the source-level worker/download diagnosis below is now superseded by the OSS/openpyxl repair + +## Facts + +- The remote root redirects unauthenticated users to `/login`; protected desktop APIs and downloads return JSON `401 AUTH_REQUIRED`. +- Remote `/healthz` returns `ready`. Public H5 HTML/CSS/JS/i18n assets match the local `727643f` checkout byte-for-byte. +- Public `/api/public/h5/months` shows current Finance projections for 2026-08 and 2026-07, with the latest August projection updated through 2026-08-03. This proves the Web/database analytics read path is alive, but it does not prove that a local monthly XLSX archive is readable. +- The desktop monthly download route resolves a registered `monthly_xlsx` artifact from `reporting.monthly_runs`, then reads and re-hashes the file under Web's controlled project root. Daily artifacts are read from OSS; company artifacts are generated/read by Web from its local output root. +- The `727643f` change removed `@oai/artifact-tool` only from the company-report builder. Monthly XLSX generation still imports the private module through `monthly_reports/xlsx/build_workbook.mjs` and requires Node plus a separately supplied artifact-tool module. +- The checked-in Compose service mounts `/app/outputs` for Web but does not start a monthly worker. Deployment instructions require a separately managed worker with the same database and shared `/app/outputs` volume. +- Focused Web/repository/monthly worker/service/publisher tests pass 40/40 locally. + +The source-level diagnosis above described the pre-repair Node/local-output architecture. The implementation now uses +Python/openpyxl and OSS-backed report identities with legacy local fallback; use the new deployability evidence topic +for the acceptance contract. + +## Inference + +The symptom pattern — Daily and company downloads work while a monthly workbook download fails — points first to deployment/runtime publication rather than a generic Web download-route defect. The monthly path uniquely depends on a locally registered archive being produced by the independent Node/artifact-tool worker and being visible at the same `/app/outputs` path inside Web. A missing worker, wrong worker output root, missing shared volume, or worker/Web running on different hosts can leave monthly metadata/analytics visible while the workbook download returns an artifact-read or integrity error. + +The exact branch is not yet confirmed because the deployed desktop page requires a user login session. No credentials were read or submitted, and no server restart, report rerun, database write, or artifact repair was performed. + +## Next verification + +After an operator signs in through the in-app browser, capture one monthly row's `report_id`, the browser download response/status, and the corresponding error code. Then inspect only the deployed worker/Web runtime state: worker process/log, `reporting.monthly_runs` artifact identity, Web-visible `/app/outputs` path, and registered SHA-256/size. Do not rerun the report until the mismatch is identified. diff --git a/.project-docs/50-evidence/topics/2026-08-04-report-artifact-deployability.md b/.project-docs/50-evidence/topics/2026-08-04-report-artifact-deployability.md new file mode 100644 index 0000000..6224d8b --- /dev/null +++ b/.project-docs/50-evidence/topics/2026-08-04-report-artifact-deployability.md @@ -0,0 +1,45 @@ +# Report artifact deployability repair + +## Metadata + +- Date: 2026-08-04 +- Status: Implemented locally; CentOS/Docker deployment acceptance pending operator execution +- Scope: monthly openpyxl builder, monthly/company OSS publication, OSS/local download routing, migration 016 +- Confidence: High for code/tests; Docker image and live OSS acceptance intentionally not run on this workstation +- Source: local source inspection, filesystem-backed object-store tests, Python unit/integration tests + +## Implementation facts + +- `monthly_reports/publishing.py` now builds monthly workbooks with Python/openpyxl, reopens each sheet, checks + sheet names, headers, row counts, exact `TOTAL PRICE` formulas, formula count and semantic SHA-256. Each data row + uses `=R[row]*C[row]*G[row]`. +- The monthly and company publishers use the existing `ManagedObjectStore` seam when composed for deployment. New + workbook and `result.json` objects are committed under report-specific immutable roles and registered with + `storage_provider='oss'`, bucket alias `arr-private`, object key, SHA-256, byte size and MIME type. +- `arr_web/downloads.py` routes `oss`/`s3` descriptors through the managed-object reader and keeps old local + descriptors on the controlled project-root reader. Both routes recheck stored/actual identity. +- `.web-jobs` remains local queue state. It is not uploaded to OSS. +- `database/016_monthly_report_oss_artifacts.sql` updates the 012 publication trigger to accept OSS/S3 or historical + local monthly artifacts. The down migration refuses to restore local-only validation while published non-local + artifacts exist. +- The stale Node builders, package manifests and old optional Node test were removed. Production entrypoints and + deployment docs no longer expose Node/npm/module-path flags. +- The Docker root dependency chain now installs `openpyxl==3.1.5` through `requirements-monthly-reports.txt`; no + separate npm or Codex-runtime bootstrap is needed. + +## Verification + +- Targeted report/web/storage/migration suite: 41 tests passed; builder/company integration slice: 15 tests passed + (56/56 together). +- Full local discovery: 315 tests executed; 307 passed, 3 skipped, and 8 environment errors were limited to the + pre-existing missing `httpx` Agent modules and unavailable Aliyun SDK/client construction. No failure was caused by + the report/OSS changes. +- Docker build/Compose and real Aliyun OSS were not run because the development environment has no Docker and the user + explicitly requested fake/in-memory storage tests instead. + +## Operator acceptance still required + +Apply migration 016 on the isolated `booking_test` database, rebuild the clean image, start Web and the independent +worker with the existing OSS/DB secrets, submit one controlled XML, wait for monthly publication, and verify that +monthly/company downloads still work after the local report cache is removed. The exact commands are in `deploy/README.md` +and the final task handoff. diff --git a/.project-docs/90-maintenance/stale-items.md b/.project-docs/90-maintenance/stale-items.md index ad88b28..947a33c 100644 --- a/.project-docs/90-maintenance/stale-items.md +++ b/.project-docs/90-maintenance/stale-items.md @@ -16,7 +16,7 @@ | 2026-07-30 | `booking_test` current Finance projection | The earlier 417-row snapshot included accepted run `mvp-v1-fixture-20260727` from `synthetic.xml`/`local_fixture`. A 2026-07-31 14:17 read-only company-report recheck found 986 current supported-company facts, so the old 416/417 remediation target is no longer a complete description of current Finance state | Re-audit current daily-version pins and source provenance before any fixture retirement; require explicit authorization for version changes, then verify the clean target and watermark | | 2026-07-30 | Channel BI refresh lifecycle | Resolved 2026-08-03. Desktop and public H5 now check selected-month metadata every five seconds while visible and reload full analytics only after `updated_at` changes; hidden views pause and transient failures preserve the last good snapshot | Refresh the browser once to load the new static assets, then observe the next authorized publication as a live acceptance check | | 2026-07-30 | Channel BI KPI label | The card labeled `公司数` renders worksheet-level `channel_count`; LianTai GROUP/FIT are two channels, so six does not mean six companies | Decide whether to relabel it `渠道/子表数` or implement an explicit five-company aggregation | -| 2026-07-30 | Production monthly-worker packaging | Local Web/worker separation is live, and company-report XLSX generation now runs in Web through Python/openpyxl. The checked-in Compose image still does not contain the workstation-only Node/artifact-tool runtime required by the monthly workbook builder | Package an approved monthly builder runtime and shared output volume before adding/enabling the managed production monthly-worker service | +| 2026-08-04 | Production monthly-worker packaging | Resolved in code: monthly/company builders are Python/openpyxl-only, new report artifacts use private OSS, and `.web-jobs` remains local. Docker/CentOS acceptance and migration 016 application are still operator-side deployment actions | Apply 016, rebuild the clean image, start Web plus the independent worker with existing OSS secrets, then verify OSS-backed downloads after removing the report cache | ## Superseded For ARR2.0 diff --git a/FRONTEND_HANDOFF.md b/FRONTEND_HANDOFF.md index cc8ee89..d09c05e 100644 --- a/FRONTEND_HANDOFF.md +++ b/FRONTEND_HANDOFF.md @@ -99,9 +99,7 @@ PYTHONPYCACHEPREFIX=/private/tmp/arr-web-pyc \ --enable-processing \ --enable-agent-writeback \ --enable-monthly-generation \ - --enable-company-reports \ - --node-binary /absolute/path/to/node \ - --artifact-tool-module /absolute/path/to/artifact_tool.mjs + --enable-company-reports ``` 地址：桌面 `http://127.0.0.1:8765/`，手机 `http://127.0.0.1:8765/h5`，健康检查 `http://127.0.0.1:8765/api/health`。 diff --git a/README.md b/README.md index 3c2b181..81ca009 100644 --- a/README.md +++ b/README.md @@ -55,15 +55,14 @@ cp .env.example .env.local ```bash .venv/bin/python -m monthly_reports.worker \ --db-config /absolute/path/to/booking-test-db.env \ - --node-binary /absolute/path/to/node \ - --artifact-tool-module /absolute/path/to/artifact_tool.mjs + --output-root /app/outputs/monthly_reports ``` 浏览器访问 `http://127.0.0.1:8765` 后会先进入 ARR 登录页。登录后，`GET /api/health` 中相关 readiness 均为 `true` 才表示页面处理和下载能力可用；未登录的容器只使用无详情的 `GET /healthz` readiness。 worker 是独立无端口进程，应由进程管理器单独保活。 -当前测试库权威结构为 `database/008_arr_mvp_v1_rebuild.sql` 加 009–015 增量迁移。012 只增加月报发布元数据、Finance 日版本 lineage 和受控本地工件身份，不复制月报业务/住客行；013 将用户上传的 XML 文件名独立保存为任务来源信息，内部源工件仍统一命名为 `source.xml`；014/015 增加 Booking 当前整表指针以及可编辑的 Excel 提取草稿。ARR 2.0 使用原有 `artifact_callback` 通用工件交付表；不会读写 009/010 的 grant/MCP submission 表。 +当前测试库权威结构为 `database/008_arr_mvp_v1_rebuild.sql` 加 009–016 增量迁移。012 只增加月报发布元数据、Finance 日版本 lineage 和受控本地工件身份；016 允许新月报 OSS 工件并保留旧 local 记录；013 将用户上传的 XML 文件名独立保存为任务来源信息，内部源工件仍统一命名为 `source.xml`；014/015 增加 Booking 当前整表指针以及可编辑的 Excel 提取草稿。ARR 2.0 使用原有 `artifact_callback` 通用工件交付表；不会读写 009/010 的 grant/MCP submission 表。 ## Booking Excel 房型提取 @@ -89,9 +88,9 @@ worker 是独立无端口进程，应由进程管理器单独保活。 ## 部署 生产配置见 [deploy/README.md](deploy/README.md)。当前 Compose 模板仍只打包 `web` 与 `caddy`，Caddy 负责 -HTTPS，Web 负责应用登录和会话；没有 MCP 端口、MCP 域名或 Agent Secret。公司渠道明细由 Web 进程使用 -Python/openpyxl 生成正式 Excel，不需要 Node/artifact-tool。月报 worker 必须作为独立进程部署，并使用同一 -数据库、共享输出卷以及已经打包 Node/artifact-tool 的运行镜像；当前本地工作站已按这一方式运行。 +HTTPS，Web 负责应用登录和会话；没有 MCP 端口、MCP 域名或 Agent Secret。月报和公司渠道明细都由 +Python/openpyxl 生成，XLSX 与 `result.json` 上传现有 OSS；月报 worker 必须作为独立进程部署， +`.web-jobs` 队列状态仍使用 `/app/outputs` 持久卷。 ## 当前月报行为 diff --git a/arr_ingestion/contracts.py b/arr_ingestion/contracts.py index 01d8b6c..88485aa 100644 --- a/arr_ingestion/contracts.py +++ b/arr_ingestion/contracts.py @@ -46,6 +46,8 @@ ROLE_CONTRACTS = { "source_xml": ("opera_xml", ".xml", "application/xml"), "booking_source": ("booking_excel", ".xlsx", XLSX_MIME), "daily_report": ("daily_xlsx", ".xlsx", XLSX_MIME), + "monthly_report": ("monthly_xlsx", ".xlsx", XLSX_MIME), + "company_report": ("company_ten_day_xlsx", ".xlsx", XLSX_MIME), "result_json": ("result_json", ".json", "application/json"), "structured_result_json": ( "structured_result_json", @@ -58,6 +60,8 @@ ARTIFACT_SIZE_LIMITS = { "source_xml": 100 * 1024 * 1024, "booking_source": 25 * 1024 * 1024, "daily_report": 100 * 1024 * 1024, + "monthly_report": 100 * 1024 * 1024, + "company_report": 100 * 1024 * 1024, "result_json": 5 * 1024 * 1024, "structured_result_json": 50 * 1024 * 1024, "exception_report": 20 * 1024 * 1024, diff --git a/arr_storage/contracts.py b/arr_storage/contracts.py index 576f587..b788161 100644 --- a/arr_storage/contracts.py +++ b/arr_storage/contracts.py @@ -22,6 +22,8 @@ CANONICAL_OBJECT_FILENAMES = { "source_xml": "source.xml", "booking_source": "booking-source.xlsx", "daily_report": "daily-report.xlsx", + "monthly_report": "monthly-report.xlsx", + "company_report": "company-report.xlsx", "result_json": "result.json", "structured_result_json": "structured-result.json", "exception_report": "exception-report.xlsx", diff --git a/arr_web/company_jobs.py b/arr_web/company_jobs.py index c25aeb4..e952fce 100644 --- a/arr_web/company_jobs.py +++ b/arr_web/company_jobs.py @@ -18,6 +18,7 @@ from pathlib import Path, PurePosixPath from typing import Any, Callable, Deque, Dict, List, Mapping, Optional, Protocol, Tuple from zoneinfo import ZoneInfo +from arr_storage.store import ManagedObjectStore from arr_web.contracts import PortalError, validate_month from arr_web.downloads import ArtifactDescriptor, MAX_DOWNLOAD_BYTES from company_reports.contracts import ( @@ -197,6 +198,7 @@ class PersistentCompanyReportCoordinator: *, jobs_root: Optional[Path] = None, now: Optional[Callable[[], datetime]] = None, + object_store: Optional[ManagedObjectStore] = None, ) -> None: self._project_root = project_root.resolve() self._output_root = output_root.resolve() @@ -213,6 +215,7 @@ class PersistentCompanyReportCoordinator: self._jobs_root.mkdir(parents=True, exist_ok=True, mode=0o700) os.chmod(self._jobs_root, 0o700) self._executor = executor + self._object_store = object_store self._now = now or (lambda: datetime.now(COMPANY_REPORT_TIME_ZONE)) self._condition = threading.Condition(threading.RLock()) self._pending: Deque[str] = deque() @@ -612,6 +615,46 @@ class PersistentCompanyReportCoordinator: or SHA256_RE.fullmatch(sha256) is None ): raise ValueError("artifact metadata is invalid") + storage_provider = str(value.get("storage_provider") or "local") + bucket_alias = str(value.get("bucket_alias") or "arr-project-root") + byte_size_value = value.get("byte_size") + mime_type = str(value.get("mime_type") or XLSX_MIME) + if storage_provider in {"oss", "s3"}: + if self._object_store is None: + raise ValueError("OSS artifact reader is unavailable") + if ( + isinstance(byte_size_value, bool) + or not isinstance(byte_size_value, int) + or byte_size_value <= 0 + or mime_type != XLSX_MIME + ): + raise ValueError("OSS artifact identity is invalid") + try: + stored = self._object_store.inspect_committed( + storage_key, + expected_filename, + ) + except Exception: + raise ValueError("OSS artifact identity is unavailable") from None + if ( + stored.role != "company_report" + or stored.sha256 != sha256 + or stored.byte_size != byte_size_value + or stored.mime_type != mime_type + ): + raise ValueError("OSS artifact identity does not match") + descriptor = ArtifactDescriptor( + file_kind="company_ten_day_xlsx", + original_filename=expected_filename, + storage_key=storage_key, + sha256=sha256, + byte_size=byte_size_value, + mime_type=mime_type, + storage_provider=storage_provider, + bucket_alias=bucket_alias, + ) + descriptor.validate() + return descriptor storage = PurePosixPath(storage_key) if storage.is_absolute() or not storage.parts or ".." in storage.parts or "." in storage.parts: raise ValueError("artifact path is invalid") @@ -637,6 +680,8 @@ class PersistentCompanyReportCoordinator: sha256=sha256, byte_size=metadata.st_size, mime_type=XLSX_MIME, + storage_provider=storage_provider, + bucket_alias=bucket_alias, ) descriptor.validate() return descriptor diff --git a/arr_web/downloads.py b/arr_web/downloads.py index cc72425..24b7d90 100644 --- a/arr_web/downloads.py +++ b/arr_web/downloads.py @@ -29,6 +29,8 @@ class ArtifactDescriptor: sha256: str byte_size: int mime_type: str + storage_provider: str = "local" + bucket_alias: str = "arr-project-root" def validate(self) -> None: path = PurePosixPath(self.storage_key) @@ -48,6 +50,12 @@ class ArtifactDescriptor: or any(character not in "0123456789abcdef" for character in self.sha256) or not 0 < self.byte_size <= MAX_DOWNLOAD_BYTES or not self.mime_type + or not isinstance(self.storage_provider, str) + or self.storage_provider not in {"local", "local_fixture", "oss", "s3"} + or not isinstance(self.bucket_alias, str) + or not self.bucket_alias + or "/" in self.bucket_alias + or "\\" in self.bucket_alias ): raise PortalError("DOWNLOAD_REFERENCE_INVALID", "文件身份无效", 500) @@ -63,14 +71,21 @@ class UnavailableArtifactReader: class ManagedObjectArtifactReader: - """Read a committed daily XLSX from ARR's immutable object store.""" + """Read a committed report artifact from ARR's immutable object store.""" + + _EXPECTED_ROLES = { + "daily_xlsx": "daily_report", + "monthly_xlsx": "monthly_report", + "company_ten_day_xlsx": "company_report", + } def __init__(self, object_store: ManagedObjectStore) -> None: self._object_store = object_store def read(self, descriptor: ArtifactDescriptor) -> bytes: descriptor.validate() - if descriptor.file_kind != "daily_xlsx": + expected_role = self._EXPECTED_ROLES.get(descriptor.file_kind) + if expected_role is None: raise PortalError("DOWNLOAD_REFERENCE_INVALID", "文件身份无效", 500) try: stored = self._object_store.inspect_committed( @@ -78,7 +93,7 @@ class ManagedObjectArtifactReader: descriptor.original_filename, ) if ( - stored.role != "daily_report" + stored.role != expected_role or stored.sha256 != descriptor.sha256 or stored.byte_size != descriptor.byte_size or stored.mime_type != descriptor.mime_type @@ -87,7 +102,7 @@ class ManagedObjectArtifactReader: "DOWNLOAD_IDENTITY_MISMATCH", "文件完整性校验失败", 503 ) with tempfile.TemporaryDirectory(prefix="arr-download-") as temporary: - destination = Path(temporary) / "daily.xlsx" + destination = Path(temporary) / "artifact.bin" self._object_store.materialize( descriptor.storage_key, destination, @@ -111,16 +126,18 @@ class ManagedObjectArtifactReader: class RoutedArtifactReader: - """Route OSS daily artifacts and controlled local report artifacts safely.""" + """Route managed OSS artifacts and legacy controlled-local artifacts safely.""" def __init__( self, *, daily_reader: Optional[ArtifactReader], local_reader: ArtifactReader, + oss_reader: Optional[ArtifactReader] = None, ) -> None: self._daily_reader = daily_reader self._local_reader = local_reader + self._oss_reader = oss_reader def read(self, descriptor: ArtifactDescriptor) -> bytes: descriptor.validate() @@ -130,6 +147,12 @@ class RoutedArtifactReader: "DOWNLOAD_UNAVAILABLE", "文件读取服务暂不可用", 503 ) return self._daily_reader.read(descriptor) + if descriptor.storage_provider in {"oss", "s3"}: + if self._oss_reader is None: + raise PortalError( + "DOWNLOAD_UNAVAILABLE", "文件读取服务暂不可用", 503 + ) + return self._oss_reader.read(descriptor) return self._local_reader.read(descriptor) diff --git a/arr_web/processing_runtime.py b/arr_web/processing_runtime.py index 53fdec4..69e1d53 100644 --- a/arr_web/processing_runtime.py +++ b/arr_web/processing_runtime.py @@ -29,11 +29,18 @@ class ProcessingInputRuntime: self.oss_client.close() -def compose_programmatic_processing( - *, - project_root: Path, - connect: Optional[Callable[[str], Any]] = None, -) -> ProcessingInputRuntime: +@dataclass +class ObjectStoreRuntime: + """Shared OSS/object-store runtime for report publishers and downloads.""" + + oss_client: AliyunOssV2Client + object_store: ManagedObjectStore + + def close(self) -> None: + self.oss_client.close() + + +def compose_object_store() -> ObjectStoreRuntime: oss_client = AliyunOssV2Client(AliyunOssConfig.from_environment()) try: oss_client.assert_immutable_writes_supported() @@ -41,6 +48,24 @@ def compose_programmatic_processing( CloudObjectBackend(oss_client), ObjectKeyPolicy(os.environ.get("ARR_OBJECT_PREFIX", "arr")), ) + return ObjectStoreRuntime(oss_client, object_store) + except Exception: + try: + oss_client.close() + except Exception: + pass + raise + + +def compose_programmatic_processing( + *, + project_root: Path, + connect: Optional[Callable[[str], Any]] = None, +) -> ProcessingInputRuntime: + storage = compose_object_store() + try: + oss_client = storage.oss_client + object_store = storage.object_store database_config = ( DatabaseConfig("controlled") if connect is not None @@ -64,10 +89,7 @@ def compose_programmatic_processing( ) return ProcessingInputRuntime(coordinator, oss_client, object_store) except Exception: - try: - oss_client.close() - except Exception: - pass + storage.close() raise diff --git a/arr_web/repository.py b/arr_web/repository.py index fcd1ab3..4aceba8 100644 --- a/arr_web/repository.py +++ b/arr_web/repository.py @@ -414,6 +414,8 @@ ORDER BY period_start DESC DAILY_DOWNLOAD_SQL = """ SELECT artifact.artifact_kind, + artifact.storage_provider, + artifact.bucket_alias, artifact.original_filename, artifact.object_key, artifact.sha256, @@ -432,6 +434,8 @@ WHERE run.run_key = %s MONTHLY_DOWNLOAD_SQL = """ SELECT artifact.artifact_kind, + artifact.storage_provider, + artifact.bucket_alias, artifact.original_filename, artifact.object_key, artifact.sha256, @@ -443,7 +447,6 @@ JOIN ingestion.artifacts AS artifact WHERE run.id = %s AND run.report_status IN ('active', 'superseded') AND artifact.artifact_kind = 'monthly_xlsx' - AND artifact.storage_provider = 'local' """.strip() @@ -815,13 +818,23 @@ class PostgresPortalRepository: if len(rows) != 1: raise PortalDataError("DOWNLOAD_NOT_FOUND", "文件不存在或尚未生成") row = rows[0] + if len(row) >= 8: + provider = str(row[1] or "local") + bucket_alias = str(row[2] or "arr-project-root") + filename_index, key_index, hash_index, size_index, mime_index = 3, 4, 5, 6, 7 + else: + provider = "local" + bucket_alias = "arr-project-root" + filename_index, key_index, hash_index, size_index, mime_index = 1, 2, 3, 4, 5 descriptor = ArtifactDescriptor( file_kind=str(row[0] or ""), - original_filename=str(row[1] or ""), - storage_key=str(row[2] or ""), - sha256=str(row[3] or "").lower(), - byte_size=int(row[4]), - mime_type=str(row[5] or "application/octet-stream"), + original_filename=str(row[filename_index] or ""), + storage_key=str(row[key_index] or ""), + sha256=str(row[hash_index] or "").lower(), + byte_size=int(row[size_index]), + mime_type=str(row[mime_index] or "application/octet-stream"), + storage_provider=provider, + bucket_alias=bucket_alias, ) descriptor.validate() return descriptor diff --git a/arr_web/run.py b/arr_web/run.py index b7b86d9..97b1052 100644 --- a/arr_web/run.py +++ b/arr_web/run.py @@ -21,16 +21,18 @@ from arr_web.downloads import ( ) from arr_web.repository import PostgresPortalRepository, UnavailablePortalRepository from arr_web.processing_runtime import ( + ObjectStoreRuntime, ProcessingInputRuntime, + compose_object_store, compose_programmatic_processing, ) from arr_web.server import serve from arr_web.services import ProgramMonthlyCoordinator -from monthly_reports.publishing import ArtifactToolBuilder, AtomicReportPublisher +from monthly_reports.publishing import OpenpyxlWorkbookBuilder, AtomicReportPublisher from monthly_reports.repository import DatabaseConfig, PostgresReportRepository from monthly_reports.service import MonthlyReportService from company_reports.publishing import ( - ArtifactToolBuilder as CompanyArtifactToolBuilder, + OpenpyxlWorkbookBuilder as CompanyOpenpyxlWorkbookBuilder, AtomicReportPublisher as AtomicCompanyReportPublisher, ) from company_reports.repository import ( @@ -81,8 +83,6 @@ def _parser() -> argparse.ArgumentParser: action="store_true", help="mark browser session cookies Secure for an HTTPS deployment", ) - parser.add_argument("--node-binary", type=Path) - parser.add_argument("--artifact-tool-module", type=Path) return parser @@ -108,67 +108,6 @@ def main(argv: Optional[Sequence[str]] = None) -> int: repository = UnavailablePortalRepository() database_ready = False - monthly = None - monthly_ready = False - if args.enable_monthly_generation and database_ready: - try: - report_repository = PostgresReportRepository( - DatabaseConfig("controlled"), - connect=connect, - ) if connect is not None else PostgresReportRepository(DatabaseConfig.from_environment()) - builder = ArtifactToolBuilder( - PROJECT_ROOT / "monthly_reports" / "xlsx" / "build_workbook.mjs", - node_binary=str(args.node_binary) if args.node_binary else None, - artifact_tool_module=args.artifact_tool_module, - ) - output_root = PROJECT_ROOT / "outputs" / "monthly_reports" - monthly = ProgramMonthlyCoordinator( - MonthlyReportService( - report_repository, - builder, - AtomicReportPublisher(PROJECT_ROOT, output_root), - output_root / ".staging", - ) - ) - monthly_ready = True - except Exception: - monthly = None - monthly_ready = False - company_reports = None - company_reports_ready = False - if args.enable_company_reports and database_ready: - try: - company_repository = ( - PostgresCompanyReportRepository( - CompanyDatabaseConfig("controlled"), - connect=connect, - ) - if connect is not None - else PostgresCompanyReportRepository( - CompanyDatabaseConfig.from_environment() - ) - ) - company_builder = CompanyArtifactToolBuilder( - PROJECT_ROOT / "company_reports" / "xlsx" / "build_workbook.mjs", - node_binary=str(args.node_binary) if args.node_binary else None, - artifact_tool_module=args.artifact_tool_module, - ) - company_output_root = PROJECT_ROOT / "outputs" / "company_reports" - company_service = CompanyReportService( - company_repository, - company_builder, - AtomicCompanyReportPublisher(PROJECT_ROOT, company_output_root), - company_output_root / ".staging", - ) - company_reports = PersistentCompanyReportCoordinator( - PROJECT_ROOT, - company_output_root, - ProgramCompanyReportExecutor(company_service), - ) - company_reports_ready = True - except Exception: - company_reports = None - company_reports_ready = False processing_input: Optional[ProcessingInputRuntime] = None processing_ready = False if args.enable_processing and database_ready: @@ -181,6 +120,81 @@ def main(argv: Optional[Sequence[str]] = None) -> int: except Exception: processing_input = None processing_ready = False + + report_storage_runtime: Optional[ObjectStoreRuntime] = None + report_object_store = ( + processing_input.object_store if processing_input is not None else None + ) + if report_object_store is None and database_ready and ( + args.enable_monthly_generation or args.enable_company_reports + ): + try: + report_storage_runtime = compose_object_store() + report_object_store = report_storage_runtime.object_store + except Exception: + report_storage_runtime = None + report_object_store = None + + monthly = None + monthly_ready = False + if args.enable_monthly_generation and database_ready and report_object_store is not None: + try: + report_repository = PostgresReportRepository( + DatabaseConfig("controlled"), + connect=connect, + ) if connect is not None else PostgresReportRepository(DatabaseConfig.from_environment()) + output_root = PROJECT_ROOT / "outputs" / "monthly_reports" + monthly = ProgramMonthlyCoordinator( + MonthlyReportService( + report_repository, + OpenpyxlWorkbookBuilder(), + AtomicReportPublisher( + PROJECT_ROOT, + output_root, + object_store=report_object_store, + ), + output_root / ".staging", + ) + ) + monthly_ready = True + except Exception: + monthly = None + monthly_ready = False + company_reports = None + company_reports_ready = False + if args.enable_company_reports and database_ready and report_object_store is not None: + try: + company_repository = ( + PostgresCompanyReportRepository( + CompanyDatabaseConfig("controlled"), + connect=connect, + ) + if connect is not None + else PostgresCompanyReportRepository( + CompanyDatabaseConfig.from_environment() + ) + ) + company_output_root = PROJECT_ROOT / "outputs" / "company_reports" + company_service = CompanyReportService( + company_repository, + CompanyOpenpyxlWorkbookBuilder(), + AtomicCompanyReportPublisher( + PROJECT_ROOT, + company_output_root, + object_store=report_object_store, + ), + company_output_root / ".staging", + ) + company_reports = PersistentCompanyReportCoordinator( + PROJECT_ROOT, + company_output_root, + ProgramCompanyReportExecutor(company_service), + object_store=report_object_store, + ) + company_reports_ready = True + except Exception: + company_reports = None + company_reports_ready = False booking_sources = None company_source_upload_ready = False if args.enable_company_reports and database_ready and processing_input is not None: @@ -217,6 +231,11 @@ def main(argv: Optional[Sequence[str]] = None) -> int: else None ), local_reader=ControlledProjectArtifactReader(PROJECT_ROOT), + oss_reader=( + ManagedObjectArtifactReader(report_object_store) + if report_object_store is not None + else None + ), ) if database_ready else None @@ -239,6 +258,8 @@ def main(argv: Optional[Sequence[str]] = None) -> int: company_reports.close() if processing_input is not None: processing_input.close() + if report_storage_runtime is not None: + report_storage_runtime.close() return 0 diff --git a/company_reports/README.md b/company_reports/README.md index 8efd5d1..8827f79 100644 --- a/company_reports/README.md +++ b/company_reports/README.md @@ -28,7 +28,7 @@ python3 -m company_reports generate \ --as-of 2026-07-10 ``` -The workbook builder uses Python/openpyxl. No Node.js or `@oai/artifact-tool` package is required for company reports. +The workbook builder uses Python/openpyxl. The XLSX and `result.json` are uploaded to the existing private OSS; `.web-jobs` remains a local persistent queue-state directory. Use the existing `ARR_OBJECT_PREFIX`, `ARR_OSS_REGION`, `ARR_OSS_BUCKET`, optional `ARR_OSS_ENDPOINT`, and OSS credential settings; no Node.js or private package setting is required. The DSN can fall back to `ARR_DATABASE_URL`; never place a real DSN in source, prompts, output JSON or browser code. Scheduling convention: diff --git a/company_reports/cli.py b/company_reports/cli.py index b474466..bd7fe46 100644 --- a/company_reports/cli.py +++ b/company_reports/cli.py @@ -11,7 +11,8 @@ from pathlib import Path from typing import Optional, Sequence, Tuple from company_reports.contracts import COMPANY_NAMES, ErrorCode, RESULT_SCHEMA_VERSION -from company_reports.publishing import ArtifactToolBuilder, AtomicReportPublisher +from arr_web.processing_runtime import compose_object_store +from company_reports.publishing import OpenpyxlWorkbookBuilder, AtomicReportPublisher from company_reports.repository import DatabaseConfig, PostgresReportRepository, RepositoryError from company_reports.service import CompanyReportService, RunRequest, write_batch_result @@ -69,14 +70,6 @@ def _parser() -> SafeArgumentParser: "--output-root", help="controlled output root inside the project", ) - generate.add_argument( - "--node-binary", - help="deprecated compatibility option; ignored by the openpyxl builder", - ) - generate.add_argument( - "--artifact-tool-module", - help="deprecated compatibility option; ignored by the openpyxl builder", - ) return parser @@ -110,23 +103,21 @@ def main(argv: Optional[Sequence[str]] = None) -> int: output_root = output_root.resolve() config = DatabaseConfig.from_environment() repository = PostgresReportRepository(config) - builder = ArtifactToolBuilder( - PROJECT_ROOT / "company_reports" / "xlsx" / "build_workbook.mjs", - node_binary=args.node_binary, - artifact_tool_module=( - Path(args.artifact_tool_module).expanduser() - if args.artifact_tool_module - else None - ), - ) - publisher = AtomicReportPublisher(PROJECT_ROOT, output_root) - service = CompanyReportService( - repository, - builder, - publisher, - output_root / ".staging", - ) - result = service.run(request) + storage_runtime = compose_object_store() + try: + service = CompanyReportService( + repository, + OpenpyxlWorkbookBuilder(), + AtomicReportPublisher( + PROJECT_ROOT, + output_root, + object_store=storage_runtime.object_store, + ), + output_root / ".staging", + ) + result = service.run(request) + finally: + storage_runtime.close() result_path = ( output_root / f"{year:04d}" diff --git a/company_reports/publishing.py b/company_reports/publishing.py index 6cf258b..08c4f5d 100644 --- a/company_reports/publishing.py +++ b/company_reports/publishing.py @@ -17,6 +17,7 @@ from openpyxl import Workbook, load_workbook from openpyxl.styles import Alignment, Border, Font, PatternFill, Side from openpyxl.worksheet.worksheet import Worksheet +from arr_storage.store import ManagedObjectStore from company_reports.contracts import CompanyReport, ErrorCode, RESULT_SCHEMA_VERSION from company_reports.repository import ( FileMetadata, @@ -100,16 +101,11 @@ def _private_json(path: Path, payload: Mapping[str, Any]) -> None: raise -class ArtifactToolBuilder: - def __init__( - self, - builder_script: Path, - node_binary: Optional[str] = None, - artifact_tool_module: Optional[Path] = None, - timeout_seconds: int = 120, - ) -> None: - _ = (node_binary, artifact_tool_module, timeout_seconds) - self._builder_script = builder_script.resolve() +class OpenpyxlWorkbookBuilder: + def __init__(self, *_legacy_args: Any, **_legacy_options: Any) -> None: + # Retain import/call compatibility for older wrappers; no external + # workbook runtime is consulted. + _ = (_legacy_args, _legacy_options) @staticmethod def _safe_text(value: Any) -> str: @@ -324,11 +320,19 @@ class ArtifactToolBuilder: summary=summary, ) - class AtomicReportPublisher: - def __init__(self, project_root: Path, output_root: Path) -> None: + def __init__( + self, + project_root: Path, + output_root: Path, + *, + object_store: Optional[ManagedObjectStore] = None, + bucket_alias: str = "arr-private", + ) -> None: self._project_root = project_root.resolve() self._output_root = output_root.resolve() + self._object_store = object_store + self._bucket_alias = bucket_alias try: self._output_root.relative_to(self._project_root) except ValueError: @@ -441,13 +445,41 @@ class AtomicReportPublisher: if archive_sha256 != existing_sha256: raise ValueError("archive and result JSON hashes differ") - artifact_storage_key = self._storage_key(archive_path) + storage_provider = str(artifact_payload.get("storage_provider") or "local") + bucket_alias = str( + artifact_payload.get("bucket_alias") or "arr-project-root" + ) + if storage_provider in {"oss", "s3"}: + if self._object_store is None: + raise ValueError("OSS artifact reader is unavailable") + artifact_storage_key = str(artifact_payload.get("storage_key") or "") + stored = self._object_store.inspect_committed( + artifact_storage_key, + report.filename, + ) + if ( + stored.role != "company_report" + or stored.sha256 != existing_sha256 + or stored.byte_size != archive_path.stat().st_size + or stored.mime_type != XLSX_MIME + ): + raise ValueError("OSS archive and local cache differ") + artifact_byte_size = stored.byte_size + artifact_mime_type = stored.mime_type + elif storage_provider in {"local", "local_fixture"}: + artifact_storage_key = self._storage_key(archive_path) + artifact_byte_size = archive_path.stat().st_size + artifact_mime_type = XLSX_MIME + else: + raise ValueError("artifact storage provider is invalid") expected_payload = self._success_result( report, reservation, artifact_storage_key, existing_sha256, semantic_sha256, + storage_provider, + bucket_alias, ) if result_payload != expected_payload: raise ValueError("result JSON identity differs") @@ -457,16 +489,38 @@ class AtomicReportPublisher: original_filename=report.filename, storage_key=artifact_storage_key, sha256=existing_sha256, - byte_size=archive_path.stat().st_size, - mime_type=XLSX_MIME, + byte_size=artifact_byte_size, + mime_type=artifact_mime_type, + storage_provider=storage_provider, + bucket_alias=bucket_alias, ) + result_sha256 = sha256_file(result_path) + result_byte_size = result_path.stat().st_size + result_storage_key = self._storage_key(result_path) + result_mime_type = "application/json" + if storage_provider in {"oss", "s3"}: + uploaded_result = self._object_store.upload_committed( + job_id=f"company-report-{reservation.report_version_id}", + attempt_no=1, + role="result_json", + source=result_path, + original_filename=result_path.name, + expected_sha256=result_sha256, + expected_byte_size=result_byte_size, + ) + result_storage_key = uploaded_result.object_key + result_sha256 = uploaded_result.sha256 + result_byte_size = uploaded_result.byte_size + result_mime_type = uploaded_result.mime_type result_json = FileMetadata( file_kind="result_json", original_filename=result_path.name, - storage_key=self._storage_key(result_path), - sha256=sha256_file(result_path), - byte_size=result_path.stat().st_size, - mime_type="application/json", + storage_key=result_storage_key, + sha256=result_sha256, + byte_size=result_byte_size, + mime_type=result_mime_type, + storage_provider=storage_provider, + bucket_alias=bucket_alias, ) artifact.validate() result_json.validate() @@ -491,6 +545,8 @@ class AtomicReportPublisher: artifact_storage_key: str, artifact_sha256: str, semantic_sha256: str, + storage_provider: str, + bucket_alias: str, ) -> Dict[str, Any]: return { "schema_version": RESULT_SCHEMA_VERSION, @@ -506,6 +562,8 @@ class AtomicReportPublisher: "storage_key": artifact_storage_key, "sha256": artifact_sha256, "semantic_sha256": semantic_sha256, + "storage_provider": storage_provider, + "bucket_alias": bucket_alias, }, "row_count": report.row_count, "period_row_counts": { @@ -581,13 +639,30 @@ class AtomicReportPublisher: archive_created = self._install_once( built.path, archive_path, built.sha256 ) + storage_provider = "local" + bucket_alias = "arr-project-root" artifact_storage_key = self._storage_key(archive_path) + if self._object_store is not None: + uploaded_artifact = self._object_store.upload_committed( + job_id=f"company-report-{reservation.report_version_id}", + attempt_no=1, + role="company_report", + source=archive_path, + original_filename=report.filename, + expected_sha256=built.sha256, + expected_byte_size=built.byte_size, + ) + storage_provider = "oss" + bucket_alias = self._bucket_alias + artifact_storage_key = uploaded_artifact.object_key result_payload = self._success_result( report, reservation, artifact_storage_key, built.sha256, str(built.summary.get("semantic_sha256", "")), + storage_provider, + bucket_alias, ) if result_path.exists(): expected = json.dumps( @@ -614,6 +689,8 @@ class AtomicReportPublisher: sha256=built.sha256, byte_size=built.byte_size, mime_type=XLSX_MIME, + storage_provider=storage_provider, + bucket_alias=bucket_alias, ) result_json = FileMetadata( file_kind="result_json", @@ -622,7 +699,29 @@ class AtomicReportPublisher: sha256=sha256_file(result_path), byte_size=result_path.stat().st_size, mime_type="application/json", + storage_provider=storage_provider, + bucket_alias=bucket_alias, ) + if self._object_store is not None: + uploaded_result = self._object_store.upload_committed( + job_id=f"company-report-{reservation.report_version_id}", + attempt_no=1, + role="result_json", + source=result_path, + original_filename=result_path.name, + expected_sha256=result_json.sha256, + expected_byte_size=result_json.byte_size, + ) + result_json = FileMetadata( + file_kind="result_json", + original_filename=result_path.name, + storage_key=uploaded_result.object_key, + sha256=uploaded_result.sha256, + byte_size=uploaded_result.byte_size, + mime_type=uploaded_result.mime_type, + storage_provider=storage_provider, + bucket_alias=bucket_alias, + ) repository.activate_report(reservation, artifact, result_json) return PublicationOutcome( current_path=current_path, diff --git a/company_reports/repository.py b/company_reports/repository.py index 72a9c98..9676eb9 100644 --- a/company_reports/repository.py +++ b/company_reports/repository.py @@ -112,6 +112,8 @@ class FileMetadata: sha256: str byte_size: int mime_type: str + storage_provider: str = "local" + bucket_alias: str = "arr-project-root" def validate(self) -> None: storage = PurePosixPath(self.storage_key) @@ -122,6 +124,12 @@ class FileMetadata: or self.original_filename != PurePosixPath(self.original_filename).name or not SHA256_RE.fullmatch(self.sha256) or self.byte_size < 0 + or not isinstance(self.storage_provider, str) + or self.storage_provider not in {"local", "local_fixture", "oss", "s3"} + or not isinstance(self.bucket_alias, str) + or not self.bucket_alias.strip() + or "/" in self.bucket_alias + or "\\" in self.bucket_alias ): raise RepositoryError( ErrorCode.PUBLISH_FAILED, diff --git a/company_reports/service.py b/company_reports/service.py index 0cd82c7..5bf44da 100644 --- a/company_reports/service.py +++ b/company_reports/service.py @@ -255,6 +255,10 @@ class CompanyReportService: "filename": report.filename, "storage_key": outcome.artifact.storage_key, "sha256": outcome.artifact.sha256, + "byte_size": outcome.artifact.byte_size, + "mime_type": outcome.artifact.mime_type, + "storage_provider": outcome.artifact.storage_provider, + "bucket_alias": outcome.artifact.bucket_alias, "semantic_sha256": str( built.summary.get("semantic_sha256", "") ), diff --git a/company_reports/xlsx/build_workbook.mjs b/company_reports/xlsx/build_workbook.mjs deleted file mode 100644 index 8b4f58d..0000000 --- a/company_reports/xlsx/build_workbook.mjs +++ /dev/null @@ -1,356 +0,0 @@ -import fs from "node:fs/promises"; -import crypto from "node:crypto"; -import path from "node:path"; -import process from "node:process"; -import { pathToFileURL } from "node:url"; - - -let FileBlob; -let SpreadsheetFile; -let Workbook; - - -const EXPECTED_HEADERS = [ - "ARRIVAL", - "DEPARTURE", - "NIGHTS", - "BLOCK_CODE", - "RES_COMMENT", - "Booking Room", - "Total Booking Price", -]; - -const DUPLICATE_FILL = "#FFF2CC"; -const DUPLICATE_FONT = "#9C6500"; -const REVIEW_FILL = "#FCE4D6"; -const REVIEW_FONT = "#C65911"; -const HEADER_FILL = "#FFFFFF"; -const BODY_FONT = "#222222"; -const DATE_PATTERN = /^\d{4}-\d{2}-\d{2}$/; - - -async function loadArtifactTool() { - const configured = String(process.env.COMPANY_REPORT_ARTIFACT_TOOL_MODULE ?? "").trim(); - const module = configured - ? await import(pathToFileURL(path.resolve(configured)).href) - : await import("@oai/artifact-tool"); - ({ FileBlob, SpreadsheetFile, Workbook } = module); - if (!FileBlob || !SpreadsheetFile || !Workbook) fail("artifact-tool module is invalid"); -} - - -function fail(message) { - throw new Error(message); -} - - -function safeText(value) { - const text = String(value ?? ""); - return /^[=+\-@]/.test(text) ? `'${text}` : text; -} - - -function excelDate(value) { - if (!DATE_PATTERN.test(value)) fail("invalid date value"); - const [year, month, day] = value.split("-").map(Number); - const parsed = new Date(Date.UTC(year, month - 1, day)); - if ( - parsed.getUTCFullYear() !== year - || parsed.getUTCMonth() !== month - 1 - || parsed.getUTCDate() !== day - ) { - fail("invalid date value"); - } - return parsed; -} - - -function dateKey(value) { - if (value instanceof Date && !Number.isNaN(value.getTime())) { - return value.toISOString().slice(0, 10); - } - if (typeof value === "number" && Number.isFinite(value)) { - const epoch = Date.UTC(1899, 11, 30); - return new Date(epoch + Math.floor(value) * 86400000).toISOString().slice(0, 10); - } - if (typeof value === "string" && DATE_PATTERN.test(value.slice(0, 10))) { - return value.slice(0, 10); - } - return ""; -} - - -function validatePayload(payload) { - if (!payload || payload.schema_version !== "1.0") fail("invalid payload schema"); - if (!Array.isArray(payload.headers) || payload.headers.length !== EXPECTED_HEADERS.length) { - fail("invalid headers"); - } - if (!payload.headers.every((header, index) => header === EXPECTED_HEADERS[index])) { - fail("invalid headers"); - } - if (!Array.isArray(payload.periods) || payload.periods.length !== 3) { - fail("invalid period count"); - } - const names = new Set(); - for (const period of payload.periods) { - if ( - typeof period.sheet_name !== "string" - || period.sheet_name.length < 1 - || period.sheet_name.length > 31 - || names.has(period.sheet_name) - || !Array.isArray(period.rows) - ) { - fail("invalid worksheet contract"); - } - names.add(period.sheet_name); - for (const row of period.rows) { - if ( - !DATE_PATTERN.test(String(row.arrival ?? "")) - || !DATE_PATTERN.test(String(row.departure ?? "")) - || !Number.isInteger(row.nights) - || row.nights < 0 - || typeof row.block_code !== "string" - || typeof row.res_comment !== "string" - || typeof row.booking_room !== "string" - || typeof row.total_booking_price !== "string" - || typeof row.duplicate_group !== "boolean" - || typeof row.multi_price_review !== "boolean" - ) { - fail("invalid output row contract"); - } - } - } -} - - -function writeSheet(workbook, period, periodIndex) { - const sheet = workbook.worksheets.add(period.sheet_name); - const body = period.rows.map((row) => [ - excelDate(row.arrival), - excelDate(row.departure), - row.nights, - safeText(row.block_code), - safeText(row.res_comment), - safeText(row.booking_room), - safeText(row.total_booking_price), - ]); - const matrix = [EXPECTED_HEADERS, ...body]; - const lastRow = matrix.length; - const used = sheet.getRange(`A1:G${lastRow}`); - used.values = matrix; - used.format.font = { name: "Arial", size: 10, color: BODY_FONT }; - used.format.verticalAlignment = "center"; - - sheet.getRange("A1").format.columnWidth = 13; - sheet.getRange("B1").format.columnWidth = 13; - sheet.getRange("C1").format.columnWidth = 9; - sheet.getRange("D1").format.columnWidth = 20; - sheet.getRange("E1").format.columnWidth = 25; - sheet.getRange("F1").format.columnWidth = 34; - sheet.getRange("G1").format.columnWidth = 48; - - const table = sheet.tables.add( - `A1:G${lastRow}`, - true, - `CompanyReportPeriod${periodIndex + 1}`, - ); - table.style = "TableStyleLight1"; - table.showHeaders = true; - table.showTotals = false; - table.showBandedColumns = false; - table.showFilterButton = true; - - const header = sheet.getRange("A1:G1"); - header.format.fill = HEADER_FILL; - header.format.font = { name: "Arial", size: 10, bold: true, color: "#000000" }; - header.format.horizontalAlignment = "center"; - header.format.verticalAlignment = "center"; - header.format.rowHeight = 24; - header.format.borders = { - bottom: { style: "thin", color: "#7F7F7F" }, - }; - - sheet.freezePanes.freezeRows(1); - if (body.length > 0) { - const data = sheet.getRange(`A2:G${lastRow}`); - data.format.rowHeight = 30; - sheet.getRange(`A2:B${lastRow}`).setNumberFormat("yyyy-mm-dd"); - sheet.getRange(`A2:D${lastRow}`).format.horizontalAlignment = "center"; - sheet.getRange(`E2:G${lastRow}`).format.horizontalAlignment = "left"; - sheet.getRange(`D2:G${lastRow}`).format.wrapText = true; - - period.rows.forEach((row, rowIndex) => { - const excelRow = rowIndex + 2; - if (row.duplicate_group) { - sheet.getRange(`A${excelRow}:G${excelRow}`).format.fill = DUPLICATE_FILL; - sheet.getRange(`E${excelRow}`).format.font = { - name: "Arial", - size: 10, - bold: true, - color: DUPLICATE_FONT, - }; - } - if (row.multi_price_review) { - sheet.getRange(`G${excelRow}`).format.fill = REVIEW_FILL; - sheet.getRange(`G${excelRow}`).format.font = { - name: "Arial", - size: 10, - bold: true, - color: REVIEW_FONT, - }; - } - }); - data.format.autofitRows(); - } - sheet.showGridLines = true; - return sheet; -} - - -async function validateWorkbook(workbook, payload, stage) { - const names = workbook.worksheets.items.map((sheet) => sheet.name); - const expectedNames = payload.periods.map((period) => period.sheet_name); - if (JSON.stringify(names) !== JSON.stringify(expectedNames)) { - fail(`${stage} worksheet names do not match`); - } - - let formulaCount = 0; - for (let index = 0; index < payload.periods.length; index += 1) { - const period = payload.periods[index]; - const sheet = workbook.worksheets.getItem(period.sheet_name); - const expectedRows = period.rows.length + 1; - const used = sheet.getUsedRange(); - const values = used?.values ?? []; - if (values.length !== expectedRows || (values[0]?.length ?? 0) !== 7) { - fail(`${stage} worksheet dimensions do not match`); - } - if (!EXPECTED_HEADERS.every((header, column) => values[0][column] === header)) { - fail(`${stage} worksheet headers do not match`); - } - period.rows.forEach((row, rowIndex) => { - const actual = values[rowIndex + 1] ?? []; - const expected = [ - row.arrival, - row.departure, - row.nights, - safeText(row.block_code), - safeText(row.res_comment), - safeText(row.booking_room), - safeText(row.total_booking_price), - ]; - if ( - dateKey(actual[0]) !== expected[0] - || dateKey(actual[1]) !== expected[1] - || Number(actual[2]) !== expected[2] - || String(actual[3] ?? "") !== expected[3] - || String(actual[4] ?? "") !== expected[4] - || String(actual[5] ?? "") !== expected[5] - || String(actual[6] ?? "") !== expected[6] - ) { - fail(`${stage} worksheet values do not match`); - } - }); - if (sheet.tables.items.length !== 1 || !sheet.tables.items[0].showFilterButton) { - fail(`${stage} worksheet filter is missing`); - } - const formulaInspection = await workbook.inspect({ - kind: "formula", - sheetId: period.sheet_name, - range: `A1:G${expectedRows}`, - maxChars: 4000, - options: { maxResults: 100 }, - }); - const text = String(formulaInspection.ndjson ?? ""); - formulaCount += text - .split("\n") - .filter((line) => line.includes('"kind":"formula"')).length; - if (/#REF!|#DIV\/0!|#VALUE!|#NAME\?|#N\/A/.test(text)) { - fail(`${stage} workbook contains a formula error`); - } - } - if (formulaCount !== 0) fail(`${stage} workbook must contain no formulas`); - return formulaCount; -} - - -async function main() { - const [inputPath, outputPath, previewDir, summaryPath] = process.argv.slice(2); - if (!inputPath || !outputPath || !previewDir || !summaryPath) { - fail("usage: build_workbook.mjs INPUT_JSON OUTPUT_XLSX PREVIEW_DIR SUMMARY_JSON"); - } - - await loadArtifactTool(); - - const payload = JSON.parse(await fs.readFile(inputPath, "utf8")); - validatePayload(payload); - await fs.mkdir(path.dirname(outputPath), { recursive: true }); - await fs.mkdir(previewDir, { recursive: true }); - await fs.mkdir(path.dirname(summaryPath), { recursive: true }); - - const workbook = Workbook.create(); - payload.periods.forEach((period, index) => writeSheet(workbook, period, index)); - await validateWorkbook(workbook, payload, "pre-export"); - - const xlsx = await SpreadsheetFile.exportXlsx(workbook); - await xlsx.save(outputPath); - await fs.chmod(outputPath, 0o600); - - const reopened = await SpreadsheetFile.importXlsx(await FileBlob.load(outputPath)); - const formulaCount = await validateWorkbook(reopened, payload, "post-export"); - const previews = []; - for (let index = 0; index < payload.periods.length; index += 1) { - const period = payload.periods[index]; - const preview = await reopened.render({ - sheetName: period.sheet_name, - autoCrop: "all", - scale: 1.5, - format: "png", - }); - const previewPath = path.join(previewDir, `sheet-${index + 1}.png`); - await fs.writeFile(previewPath, new Uint8Array(await preview.arrayBuffer()), { - mode: 0o600, - }); - previews.push(previewPath); - } - - const stat = await fs.stat(outputPath); - const semanticSha256 = crypto - .createHash("sha256") - .update(JSON.stringify({ headers: payload.headers, periods: payload.periods }), "utf8") - .digest("hex"); - const summary = { - status: "success", - schema_version: payload.schema_version, - company: payload.company, - filename: payload.filename, - sheet_names: payload.periods.map((period) => period.sheet_name), - row_counts: payload.periods.map((period) => period.rows.length), - formula_count: formulaCount, - semantic_sha256: semanticSha256, - preview_count: previews.length, - byte_size: stat.size, - }; - await fs.writeFile(summaryPath, `${JSON.stringify(summary, null, 2)}\n`, { - encoding: "utf8", - mode: 0o600, - }); - await fs.rm(`${outputPath}.inspect.ndjson`, { force: true }); - process.stdout.write(`${JSON.stringify(summary)}\n`); -} - - -try { - await main(); -} catch (_error) { - const outputPath = process.argv[3]; - if (outputPath) { - await fs.rm(`${outputPath}.inspect.ndjson`, { force: true }).catch(() => undefined); - } - process.stderr.write( - `${JSON.stringify({ - status: "failed", - code: "COMPANY_REPORT_OUTPUT_VALIDATION_FAILED", - })}\n`, - ); - process.exitCode = 4; -} diff --git a/company_reports/xlsx/package.json b/company_reports/xlsx/package.json deleted file mode 100644 index eabfaf9..0000000 --- a/company_reports/xlsx/package.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "name": "company-report-xlsx-builder", - "private": true, - "type": "module" -} diff --git a/database/016_monthly_report_oss_artifacts.down.sql b/database/016_monthly_report_oss_artifacts.down.sql new file mode 100644 index 0000000..a3b3b1c --- /dev/null +++ b/database/016_monthly_report_oss_artifacts.down.sql @@ -0,0 +1,94 @@ +-- Restore the 012 local-only publication guard. + +BEGIN; + +DO $$ +BEGIN + IF current_database() <> 'booking_test' THEN + RAISE EXCEPTION + 'ARR monthly OSS artifact rollback is allowed only in booking_test'; + END IF; + IF to_regclass('reporting.monthly_runs') IS NULL + OR to_regprocedure('reporting.validate_monthly_run_publication()') IS NULL THEN + RAISE EXCEPTION + 'ARR migration 016 is not applied'; + END IF; + IF EXISTS ( + SELECT 1 + FROM reporting.monthly_runs AS run + JOIN ingestion.artifacts AS artifact + ON artifact.id IN (run.workbook_artifact_id, run.result_artifact_id) + WHERE run.report_status IN ('active', 'superseded') + AND artifact.storage_provider <> 'local' + ) THEN + RAISE EXCEPTION + 'ARR monthly OSS artifact rollback refused while a published OSS artifact exists'; + END IF; +END; +$$; + +CREATE OR REPLACE FUNCTION reporting.validate_monthly_run_publication() +RETURNS trigger +LANGUAGE plpgsql +AS $$ +DECLARE + workbook_kind text; + workbook_provider text; + result_kind text; + result_provider text; + manifest_count integer; + manifest_rows bigint; + manifest_min integer; + manifest_max integer; + lineage_count integer; +BEGIN + IF NEW.report_status NOT IN ('active', 'superseded') THEN + RETURN NEW; + END IF; + + SELECT artifact_kind, storage_provider + INTO workbook_kind, workbook_provider + FROM ingestion.artifacts + WHERE id = NEW.workbook_artifact_id; + + SELECT artifact_kind, storage_provider + INTO result_kind, result_provider + FROM ingestion.artifacts + WHERE id = NEW.result_artifact_id; + + IF workbook_kind IS DISTINCT FROM 'monthly_xlsx' + OR workbook_provider IS DISTINCT FROM 'local' + OR result_kind IS DISTINCT FROM 'result_json' + OR result_provider IS DISTINCT FROM 'local' THEN + RAISE EXCEPTION + 'published monthly run must reference controlled local monthly_xlsx and result_json artifacts'; + END IF; + + SELECT count(*), COALESCE(sum(row_count), 0), min(worksheet_order), max(worksheet_order) + INTO manifest_count, manifest_rows, manifest_min, manifest_max + FROM reporting.monthly_channel_manifest + WHERE report_id = NEW.id; + + IF manifest_count <> NEW.channel_count + OR manifest_rows <> NEW.row_count + OR manifest_min <> 1 + OR manifest_max <> manifest_count THEN + RAISE EXCEPTION + 'published monthly run requires a continuous reconciled channel manifest'; + END IF; + + SELECT count(*) + INTO lineage_count + FROM reporting.monthly_run_daily_versions + WHERE report_id = NEW.id; + + IF lineage_count < 1 THEN + RAISE EXCEPTION + 'published monthly run requires daily-version lineage'; + END IF; + + RETURN NEW; +END; +$$; + +COMMIT; diff --git a/database/016_monthly_report_oss_artifacts.sql b/database/016_monthly_report_oss_artifacts.sql new file mode 100644 index 0000000..2124aca --- /dev/null +++ b/database/016_monthly_report_oss_artifacts.sql @@ -0,0 +1,88 @@ +-- Allow durable monthly publication artifacts in the existing private OSS. +-- Historical local artifacts remain valid and readable. + +BEGIN; + +DO $$ +BEGIN + IF current_database() <> 'booking_test' THEN + RAISE EXCEPTION + 'ARR monthly OSS artifact migration is allowed only in booking_test'; + END IF; + IF to_regclass('reporting.monthly_runs') IS NULL + OR to_regclass('ingestion.artifacts') IS NULL + OR to_regprocedure('reporting.validate_monthly_run_publication()') IS NULL THEN + RAISE EXCEPTION + 'ARR migration 012 must be applied first'; + END IF; +END; +$$; + +CREATE OR REPLACE FUNCTION reporting.validate_monthly_run_publication() +RETURNS trigger +LANGUAGE plpgsql +AS $$ +DECLARE + workbook_kind text; + workbook_provider text; + result_kind text; + result_provider text; + manifest_count integer; + manifest_rows bigint; + manifest_min integer; + manifest_max integer; + lineage_count integer; +BEGIN + IF NEW.report_status NOT IN ('active', 'superseded') THEN + RETURN NEW; + END IF; + + SELECT artifact_kind, storage_provider + INTO workbook_kind, workbook_provider + FROM ingestion.artifacts + WHERE id = NEW.workbook_artifact_id; + + SELECT artifact_kind, storage_provider + INTO result_kind, result_provider + FROM ingestion.artifacts + WHERE id = NEW.result_artifact_id; + + IF workbook_kind IS DISTINCT FROM 'monthly_xlsx' + OR COALESCE(workbook_provider, '') NOT IN ('oss', 's3', 'local') + OR result_kind IS DISTINCT FROM 'result_json' + OR COALESCE(result_provider, '') NOT IN ('oss', 's3', 'local') THEN + RAISE EXCEPTION + 'published monthly run must reference OSS or controlled local monthly artifacts'; + END IF; + + SELECT count(*), COALESCE(sum(row_count), 0), min(worksheet_order), max(worksheet_order) + INTO manifest_count, manifest_rows, manifest_min, manifest_max + FROM reporting.monthly_channel_manifest + WHERE report_id = NEW.id; + + IF manifest_count <> NEW.channel_count + OR manifest_rows <> NEW.row_count + OR manifest_min <> 1 + OR manifest_max <> manifest_count THEN + RAISE EXCEPTION + 'published monthly run requires a continuous reconciled channel manifest'; + END IF; + + SELECT count(*) + INTO lineage_count + FROM reporting.monthly_run_daily_versions + WHERE report_id = NEW.id; + + IF lineage_count < 1 THEN + RAISE EXCEPTION + 'published monthly run requires daily-version lineage'; + END IF; + + RETURN NEW; +END; +$$; + +COMMENT ON FUNCTION reporting.validate_monthly_run_publication() IS + 'Published monthly artifacts may be private OSS/S3 or legacy controlled-local objects.'; + +COMMIT; diff --git a/database/APPLIED_MIGRATIONS.md b/database/APPLIED_MIGRATIONS.md index 751ffed..1b01db0 100644 --- a/database/APPLIED_MIGRATIONS.md +++ b/database/APPLIED_MIGRATIONS.md @@ -1,7 +1,7 @@ # ARR 测试数据库已执行记录 -更新时间：2026-07-31 -状态：008–015 已提交；014/015 已通过迁移回滚探针、正式应用、结构/数据复核及真实 PostgreSQL 草稿激活外层回滚验收。首次真实业务工作簿激活仍待操作员授权。 +更新时间：2026-08-04 +状态：008–015 已提交；014/015 已通过迁移回滚探针、正式应用、结构/数据复核及真实 PostgreSQL 草稿激活外层回滚验收。016 已提交代码、待在测试机 booking_test 应用并完成 OSS 月报验收。首次真实业务工作簿激活仍待操作员授权。 ## 目标与隔离 @@ -29,6 +29,7 @@ | `013_daily_upload_filename.sql` | `f7ea18d6b844d9bd90fa757a4cf8428d1dd5833c088ae23444ac81fbe204fb0a` | 已提交并验收 | | `014_booking_current_source_batch.sql` | `23bf0fcc880225ca276d4d7d871057be4f7950e761ddbeed1df2a3c6e25463b5` | 2026-07-31 已通过同事务回滚探针后由受控连接正式应用；数据复核通过 | | `015_booking_excel_review_drafts.sql` | `a80689c4ecc3b6b3502e8f094a8c175af38df8c09f9d2c64412a9aec55bae12a` | 2026-07-31 已由受控连接正式应用；真实 PostgreSQL 草稿编辑/激活外层回滚通过 | +| `016_monthly_report_oss_artifacts.sql` | `70ca052f71da9f88e83fe8bccf0a6682cdd3138c3525981c80eebc868e401626` | 代码已提交；尚未在远程 `booking_test` 应用 | ## 014/015 最新核查状态 diff --git a/deploy/README.md b/deploy/README.md index cfbb4d2..7d1ae41 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -5,11 +5,11 @@ ## 前置条件 - DNS：`WEB_PUBLIC_HOST` 指向部署主机； -- PostgreSQL 15+：目标必须是隔离数据库 `booking_test`，并已应用 008–012 权威迁移； +- PostgreSQL 15+：目标必须是隔离数据库 `booking_test`，并已应用 008–016 权威迁移； - OSS：区域匹配、服务端加密、未启用或暂停 versioning、禁止匿名写；bucket ACL 可为 private 或 public-read； - ARR 数据库账号和 OSS RAM/STS 凭据由 Secret 管理器注入，不写入镜像或仓库。 -- 公司渠道明细由 Web 进程使用 Python/openpyxl 生成正式 Excel；不需要 Node.js 或 `@oai/artifact-tool`。 -- worker 与 Web 使用同一数据库和共享的 `/app/outputs` 持久卷；月报 worker 镜像还必须包含 Node.js 与配置匹配的 artifact-tool 模块。 +- 月报和公司渠道明细均由 Python/openpyxl 生成；不需要 Node.js、npm 或私有运行时。 +- 月报/公司 XLSX 与 `result.json` 上传现有 OSS；只有 `.web-jobs` 队列状态和临时 staging 继续使用 `/app/outputs` 持久卷。 ## 配置 @@ -25,6 +25,18 @@ chmod 600 deploy/.env.production - Agent callback URL/HMAC； - `fetch_oss_file` 或源文件公网 URL。 +## 应用月报 OSS 迁移 + +在目标数据库确认当前连接为隔离的 `booking_test` 后，先应用 016；脚本会自行拒绝其他数据库。016 +只替换月报发布校验函数，不新增表或列，历史 `local` 工件不需要迁移： + +```bash +psql "$ARR_DATABASE_URL" -Atc "select current_database();" +sha256sum database/016_monthly_report_oss_artifacts.sql +psql "$ARR_DATABASE_URL" -v ON_ERROR_STOP=1 \ + -f database/016_monthly_report_oss_artifacts.sql +``` + ## 校验并启动 ```bash @@ -33,13 +45,21 @@ docker compose --env-file deploy/.env.production build web docker compose --env-file deploy/.env.production up -d ``` -当前仓库镜像未内置 Codex 工作站提供的 artifact-tool，因此 Compose 模板不会虚假启动一个无法生成 XLSX 的 worker。部署环境完成该依赖打包后，应由 systemd、容器编排器或同等进程管理器独立执行： +Compose 启动后，应由 systemd、容器编排器或同等进程管理器独立执行月报 worker： ```bash -python -m monthly_reports.worker \ - --db-config /run/secrets/booking-test-db.env \ - --node-binary /absolute/path/to/node \ - --artifact-tool-module /absolute/path/to/artifact_tool.mjs \ +docker compose --env-file deploy/.env.production run --rm web \ + python -m monthly_reports.worker \ + --output-root /app/outputs/monthly_reports +``` + +该命令沿用 Compose 注入的 `ARR_DATABASE_URL`、`ARR_OSS_*` 和 OSS 凭据；如果改由主机上的 systemd 运行， +则在进程环境中提供同一组变量，或显式传入受控的 `--db-config` 文件。`--once` 可用于部署后的单次探针： + +```bash +docker compose --env-file deploy/.env.production run --rm web \ + python -m monthly_reports.worker \ + --once \ --output-root /app/outputs/monthly_reports ``` @@ -50,7 +70,7 @@ curl --fail --silent "https://$WEB_PUBLIC_HOST/healthz" docker compose --env-file deploy/.env.production logs --tail=100 web ``` -`/healthz` 仅以 HTTP 200/503 表示数据库与处理入口是否就绪，不暴露组件详情；详细 `/api/health` 必须登录后访问。worker 是独立进程，应另行监控其存活和 outbox 的 `pending/publishing/dead` 数量。失败时优先检查登录环境变量、数据库目标、OSS 区域/加密/versioning、OSS 凭据、固定处理器以及月报构建依赖；无需排查 Agent 或 MCP。 +`/healthz` 仅以 HTTP 200/503 表示数据库与处理入口是否就绪，不暴露组件详情；详细 `/api/health` 必须登录后访问。worker 是独立进程，应另行监控其存活和 outbox 的 `pending/publishing/dead` 数量。失败时优先检查登录环境变量、数据库目标、OSS 区域/加密/versioning、OSS 凭据和固定处理器；无需排查 Agent、MCP 或 Node/npm。 ## 验收 @@ -61,7 +81,7 @@ docker compose --env-file deploy/.env.production logs --tail=100 web 3. 上传响应应直接给出 `succeeded` 或 `failed` 终态以及 `job_id`； 4. 在任务日志中确认“固定处理器已启动 → 程序输出制品已登记 → 独立验收 → Finance 提交”； 5. 成功任务应能下载对应日报，数据库中 source/retained/outcome 数量必须与结构化结果一致； -6. worker 应消费对应 outbox 事件，页面显示真实月报 ID/版本/“更新至”，下载文件哈希应与登记值一致； +6. worker 应消费对应 outbox 事件，页面显示真实月报 ID/版本/“更新至”，下载文件哈希应与 OSS 登记值一致；删除 Web 容器或清空其临时 `outputs/monthly_reports` 后，月报与公司报表仍应可下载； 7. 退出登录后，页面、API 和下载均应重新要求登录； 8. 业务失败任务不得激活 Finance 当前版本或触发月报。 diff --git a/monthly_reports/README.md b/monthly_reports/README.md index 894fd32..1ef3f0d 100644 --- a/monthly_reports/README.md +++ b/monthly_reports/README.md @@ -8,8 +8,8 @@ Automatic flow: 2. It looks up that version's retained `ARRIVAL` values in PostgreSQL. The affected month and report watermark come from those facts; the XML filename and wall clock are ignored. 3. It opens a repeatable-read snapshot of current Finance facts and daily-version pins for that month. `as_of_date` is the greatest `ARRIVAL` actually included in the snapshot. 4. It reserves an idempotent publication identity in `reporting.monthly_runs` with immutable daily lineage and channel manifest rows. -5. It generates the XLSX/result JSON, reopens the workbook, validates all values and formulas, and rechecks that the Finance pins are still current. -6. It registers both local artifacts and atomically activates the report. Only then is the outbox event marked `published`; transient failures are retried and exhausted events become `dead`. +5. It generates the XLSX/result JSON with Python/openpyxl, reopens the workbook, validates all values and formulas, and rechecks that the Finance pins are still current. +6. It uploads both artifacts to the existing private OSS through the immutable object-store adapter and atomically activates the report. Only then is the outbox event marked `published`; transient failures are retried and exhausted events become `dead`. PostgreSQL stores publication metadata, version identity, lineage and artifact identities—not duplicate monthly business or guest rows. The portal lists `active`/`superseded` runs and downloads only registered artifacts after path, size and SHA-256 checks. @@ -18,8 +18,7 @@ Run the dedicated worker: ```bash python3 -m monthly_reports.worker \ --db-config /absolute/path/to/booking-test-db.env \ - --node-binary /absolute/path/to/node \ - --artifact-tool-module /absolute/path/to/artifact_tool.mjs + --output-root /app/outputs/monthly_reports ``` The CLI generation command remains a controlled recovery/diagnostic entrypoint: @@ -28,10 +27,9 @@ The CLI generation command remains a controlled recovery/diagnostic entrypoint: python3 -m monthly_reports generate \ --month 2026-07 \ --as-of 2026-07-31 \ - --node-binary /absolute/path/to/node \ - --artifact-tool-module /absolute/path/to/artifact_tool.mjs + --output-root /app/outputs/monthly_reports ``` -The program reads `MONTHLY_REPORT_DATABASE_URL`, falling back to `ARR_DATABASE_URL`. The configured database must be `booking_test`; SuperAgent must never receive the DSN. +The program reads `MONTHLY_REPORT_DATABASE_URL`, falling back to `ARR_DATABASE_URL`, and uses the existing `ARR_OSS_REGION`, `ARR_OSS_BUCKET` and optional `ARR_OSS_ENDPOINT` settings. The configured database must be `booking_test`; SuperAgent must never receive the DSN. `output-root` is only staging/legacy-cache space; new downloads use the OSS identity. Every XLSX data-row `TOTAL PRICE` cell contains the exact row-relative formula `=R[row]*C[row]*G[row]`, meaning `REAL PRICE × NIGHTS × NO_OF_ROOMS`. The stored Finance `total_price` remains an independent audit expectation. `Booking Room` enrichment must not be used to recalculate price, dates or actual room count. diff --git a/monthly_reports/cli.py b/monthly_reports/cli.py index 4ed746c..289506e 100644 --- a/monthly_reports/cli.py +++ b/monthly_reports/cli.py @@ -11,7 +11,8 @@ from pathlib import Path from typing import Optional, Sequence, Tuple from monthly_reports.contracts import ErrorCode, RESULT_SCHEMA_VERSION -from monthly_reports.publishing import ArtifactToolBuilder, AtomicReportPublisher +from arr_web.processing_runtime import compose_object_store +from monthly_reports.publishing import OpenpyxlWorkbookBuilder, AtomicReportPublisher from monthly_reports.repository import DatabaseConfig, PostgresReportRepository, RepositoryError from monthly_reports.service import MonthlyReportService, RunRequest, write_run_result @@ -56,14 +57,6 @@ def _parser() -> SafeArgumentParser: generate.add_argument("--month", required=True, help="report month in YYYY-MM") generate.add_argument("--as-of", required=True, help="latest included business date") generate.add_argument("--output-root", help="controlled output root inside the project") - generate.add_argument( - "--node-binary", - help="Node.js executable; defaults to MONTHLY_REPORT_NODE_BINARY or PATH", - ) - generate.add_argument( - "--artifact-tool-module", - help="absolute path to artifact_tool.mjs when package resolution is unavailable", - ) return parser @@ -96,23 +89,21 @@ def main(argv: Optional[Sequence[str]] = None) -> int: output_root = PROJECT_ROOT / output_root output_root = output_root.resolve() repository = PostgresReportRepository(DatabaseConfig.from_environment()) - builder = ArtifactToolBuilder( - PROJECT_ROOT / "monthly_reports" / "xlsx" / "build_workbook.mjs", - node_binary=args.node_binary, - artifact_tool_module=( - Path(args.artifact_tool_module).expanduser() - if args.artifact_tool_module - else None - ), - ) - publisher = AtomicReportPublisher(PROJECT_ROOT, output_root) - service = MonthlyReportService( - repository, - builder, - publisher, - output_root / ".staging", - ) - result = service.run(request) + storage_runtime = compose_object_store() + try: + service = MonthlyReportService( + repository, + OpenpyxlWorkbookBuilder(), + AtomicReportPublisher( + PROJECT_ROOT, + output_root, + object_store=storage_runtime.object_store, + ), + output_root / ".staging", + ) + result = service.run(request) + finally: + storage_runtime.close() result_path = ( output_root / f"{year:04d}" diff --git a/monthly_reports/publishing.py b/monthly_reports/publishing.py index f7c686a..e706a9e 100644 --- a/monthly_reports/publishing.py +++ b/monthly_reports/publishing.py @@ -1,4 +1,4 @@ -"""Artifact-tool adapter and recoverable atomic monthly-report publication.""" +"""Python/openpyxl monthly workbook building and recoverable publication.""" from __future__ import annotations @@ -6,19 +6,20 @@ import hashlib import json import os import shutil -import subprocess import tempfile import uuid from dataclasses import dataclass +from datetime import date, datetime +from decimal import Decimal, InvalidOperation from pathlib import Path from typing import Any, Dict, Mapping, Optional -from monthly_reports.contracts import ( - RESULT_SCHEMA_VERSION, - XLSX_MIME, - ErrorCode, - MonthlyReport, -) +from openpyxl import Workbook, load_workbook +from openpyxl.styles import Alignment, Font, PatternFill +from openpyxl.utils import get_column_letter + +from arr_storage.store import ManagedObjectStore +from monthly_reports.contracts import KB_HEADER, RESULT_SCHEMA_VERSION, XLSX_MIME, ErrorCode, MonthlyReport from monthly_reports.repository import ( FileMetadata, ReportRepository, @@ -84,94 +85,178 @@ def _private_json(path: Path, payload: Mapping[str, Any]) -> None: raise -class ArtifactToolBuilder: - def __init__( - self, - builder_script: Path, - node_binary: Optional[str] = None, - artifact_tool_module: Optional[Path] = None, - timeout_seconds: int = 180, - ) -> None: - configured = (node_binary or os.environ.get("MONTHLY_REPORT_NODE_BINARY", "")).strip() - self._node_binary = configured or shutil.which("node") or "" - self._builder_script = builder_script.resolve() - self._artifact_tool_module = ( - artifact_tool_module.resolve() if artifact_tool_module else None - ) - self._timeout_seconds = timeout_seconds +class OpenpyxlWorkbookBuilder: + """Build and re-open a validated monthly workbook without external runtimes.""" - def build(self, report: MonthlyReport, work_dir: Path) -> BuiltWorkbook: - if not self._node_binary or not self._builder_script.is_file(): - raise BuildError( - ErrorCode.OUTPUT_VALIDATION_FAILED, - "the monthly XLSX builder runtime is unavailable", - ) - work_dir.mkdir(parents=True, exist_ok=True, mode=0o700) - os.chmod(work_dir, 0o700) - payload_path = work_dir / "workbook-payload.json" - output_path = work_dir / report.filename - preview_dir = work_dir / "previews" - summary_path = work_dir / "workbook-summary.json" - _private_json(payload_path, report.to_workbook_payload()) - environment = os.environ.copy() - if self._artifact_tool_module is not None: - environment["MONTHLY_REPORT_ARTIFACT_TOOL_MODULE"] = str( - self._artifact_tool_module + _INTEGER_HEADERS = frozenset({"NIGHTS", "ADULTS", "CHILDREN", "NO_OF_ROOMS"}) + _DECIMAL_HEADERS = frozenset({"RATE_AMOUNT", "REAL PRICE", "TOTAL PRICE", KB_HEADER}) + + def __init__(self, *_legacy_args: Any, **_legacy_options: Any) -> None: + # The ignored arguments keep older local wrappers import-compatible while + # making the production builder entirely Python/openpyxl based. + _ = (_legacy_args, _legacy_options) + + @staticmethod + def _safe_text(value: Any) -> str: + text = "" if value is None else str(value) + return "'" + text if text[:1] in {"=", "+", "-", "@"} else text + + @staticmethod + def _date_key(value: Any) -> str: + if isinstance(value, datetime): + return value.date().isoformat() + if isinstance(value, date): + return value.isoformat() + if isinstance(value, str) and len(value) >= 10: + return value[:10] + return "" + + @staticmethod + def _semantic_sha256(payload: Mapping[str, Any]) -> str: + encoded = json.dumps( + {"channels": payload["channels"]}, + ensure_ascii=False, + separators=(",", ":"), + ).encode("utf-8") + return hashlib.sha256(encoded).hexdigest() + + @classmethod + def _cell_value(cls, header: str, value: Any) -> Any: + if header in {"ARRIVAL", "DEPARTURE"}: + return date.fromisoformat(str(value)) + if header in cls._INTEGER_HEADERS: + return int(value) + if header in cls._DECIMAL_HEADERS: + return float(Decimal(str(value))) + return cls._safe_text(value) + + @staticmethod + def _style_sheet(worksheet: Any, headers: list[str]) -> None: + worksheet.freeze_panes = "A2" + worksheet.auto_filter.ref = f"A1:{get_column_letter(len(headers))}{max(1, worksheet.max_row)}" + for column, header in enumerate(headers, start=1): + cell = worksheet.cell(1, column) + cell.font = Font(name="Arial", size=10, bold=True, color="000000") + cell.fill = PatternFill("solid", fgColor="D9EAF7") + cell.alignment = Alignment(horizontal="center", vertical="center", wrap_text=True) + worksheet.column_dimensions[get_column_letter(column)].width = min( + 34, max(12, len(header) + 2) ) + worksheet.row_dimensions[1].height = 28 + for row in worksheet.iter_rows(min_row=2): + for cell in row: + cell.font = Font(name="Arial", size=10, color="222222") + cell.alignment = Alignment(vertical="center", wrap_text=True) + worksheet.row_dimensions[row[0].row].height = 24 + for column in ("A", "B"): + for cell in worksheet[column][1:]: + cell.number_format = "yyyy-mm-dd" + + @classmethod + def _write_sheet(cls, worksheet: Any, channel: Mapping[str, Any]) -> int: + headers = [str(header) for header in channel["headers"]] + worksheet.append(headers) + total_column = headers.index("TOTAL PRICE") + 1 + for row in channel["rows"]: + row_index = worksheet.max_row + 1 + values = [ + cls._cell_value(header, row.get(header, "")) + for header in headers + ] + values[total_column - 1] = f"=R{row_index}*C{row_index}*G{row_index}" + worksheet.append(values) + cls._style_sheet(worksheet, headers) + return len(channel["rows"]) + + @classmethod + def _validate_workbook( + cls, + output_path: Path, + report: MonthlyReport, + payload: Mapping[str, Any], + ) -> int: + workbook = load_workbook(output_path, data_only=False, read_only=True) try: - completed = subprocess.run( - [ - self._node_binary, - str(self._builder_script), - str(payload_path), - str(output_path), - str(preview_dir), - str(summary_path), - ], - cwd=self._builder_script.parent, - capture_output=True, - text=True, - timeout=self._timeout_seconds, - env=environment, - check=False, - ) - except (OSError, subprocess.TimeoutExpired): - raise BuildError( - ErrorCode.OUTPUT_VALIDATION_FAILED, - "the monthly XLSX builder did not complete", - ) from None - if completed.returncode != 0: - raise BuildError( - ErrorCode.OUTPUT_VALIDATION_FAILED, - "the monthly XLSX builder rejected the generated payload", - ) - try: - summary = json.loads(summary_path.read_text(encoding="utf-8")) expected_names = [channel.worksheet for channel in report.channels] expected_rows = [len(channel.rows) for channel in report.channels] - if ( - summary.get("status") != "success" - or summary.get("schema_version") != RESULT_SCHEMA_VERSION - or summary.get("filename") != report.filename - or summary.get("report_year") != report.report_year - or summary.get("report_month") != report.report_month - or summary.get("as_of_date") != report.as_of_date.isoformat() - or summary.get("sheet_names") != expected_names - or summary.get("row_counts") != expected_rows - or summary.get("formula_count") != report.row_count - or summary.get("preview_count") != len(report.channels) - or not isinstance(summary.get("semantic_sha256"), str) - or len(summary["semantic_sha256"]) != 64 - or not output_path.is_file() - or output_path.stat().st_size <= 0 - ): - raise ValueError("builder summary mismatch") - except (OSError, ValueError, TypeError, json.JSONDecodeError, KeyError): + if workbook.sheetnames != expected_names: + raise ValueError("worksheet names do not match") + formula_count = 0 + for channel, expected_row_count in zip(report.channels, expected_rows): + worksheet = workbook[channel.worksheet] + headers = list(channel.headers) + if worksheet.max_row != expected_row_count + 1 or worksheet.max_column != len(headers): + raise ValueError("worksheet dimensions do not match") + if [worksheet.cell(1, index).value for index in range(1, len(headers) + 1)] != headers: + raise ValueError("worksheet headers do not match") + for row_index, expected_row in enumerate(channel.rows, start=2): + payload_row = expected_row.to_payload(channel.worksheet == "DY-AI-Easy-KB") + for column, header in enumerate(headers, start=1): + cell = worksheet.cell(row_index, column) + if header == "TOTAL PRICE": + expected_formula = f"=R{row_index}*C{row_index}*G{row_index}" + if cell.value != expected_formula or cell.data_type != "f": + raise ValueError("total price formula does not match") + formula_count += 1 + continue + actual = cell.value + expected = payload_row[header] + if header in {"ARRIVAL", "DEPARTURE"}: + matches = cls._date_key(actual) == str(expected) + elif header in cls._INTEGER_HEADERS: + matches = actual == int(expected) + elif header in cls._DECIMAL_HEADERS: + matches = Decimal(str(actual)) == Decimal(str(expected)) + else: + matches = ("" if actual is None else actual) == cls._safe_text(expected) + if not matches: + raise ValueError("worksheet values do not match") + if formula_count != report.row_count: + raise ValueError("formula count does not match") + if cls._semantic_sha256(payload) != cls._semantic_sha256(report.to_workbook_payload()): + raise ValueError("semantic source differs") + return formula_count + finally: + workbook.close() + + def build(self, report: MonthlyReport, work_dir: Path) -> BuiltWorkbook: + work_dir.mkdir(parents=True, exist_ok=True, mode=0o700) + os.chmod(work_dir, 0o700) + output_path = work_dir / report.filename + summary_path = work_dir / "workbook-summary.json" + payload = report.to_workbook_payload() + try: + workbook = Workbook() + for index, channel in enumerate(payload["channels"]): + worksheet = workbook.active if index == 0 else workbook.create_sheet() + worksheet.title = str(channel["worksheet"]) + self._write_sheet(worksheet, channel) + workbook.save(output_path) + workbook.close() + os.chmod(output_path, 0o600) + formula_count = self._validate_workbook(output_path, report, payload) + expected_names = [channel.worksheet for channel in report.channels] + expected_rows = [len(channel.rows) for channel in report.channels] + summary = { + "status": "success", + "schema_version": payload["schema_version"], + "report_year": report.report_year, + "report_month": report.report_month, + "as_of_date": report.as_of_date.isoformat(), + "filename": report.filename, + "sheet_names": expected_names, + "row_counts": expected_rows, + "formula_count": formula_count, + "semantic_sha256": self._semantic_sha256(payload), + "preview_count": len(expected_names), + "byte_size": output_path.stat().st_size, + } + _private_json(summary_path, summary) + except (OSError, ValueError, TypeError, InvalidOperation, KeyError): raise BuildError( ErrorCode.OUTPUT_VALIDATION_FAILED, "the monthly XLSX builder result could not be validated", ) from None - os.chmod(output_path, 0o600) return BuiltWorkbook( path=output_path, sha256=sha256_file(output_path), @@ -179,11 +264,19 @@ class ArtifactToolBuilder: summary=summary, ) - class AtomicReportPublisher: - def __init__(self, project_root: Path, output_root: Path) -> None: + def __init__( + self, + project_root: Path, + output_root: Path, + *, + object_store: Optional[ManagedObjectStore] = None, + bucket_alias: str = "arr-private", + ) -> None: self._project_root = project_root.resolve() self._output_root = output_root.resolve() + self._object_store = object_store + self._bucket_alias = bucket_alias try: self._output_root.relative_to(self._project_root) except ValueError: @@ -250,6 +343,8 @@ class AtomicReportPublisher: artifact_storage_key: str, artifact_sha256: str, semantic_sha256: str, + storage_provider: str, + bucket_alias: str, ) -> Dict[str, Any]: return { "schema_version": RESULT_SCHEMA_VERSION, @@ -264,6 +359,8 @@ class AtomicReportPublisher: "storage_key": artifact_storage_key, "sha256": artifact_sha256, "semantic_sha256": semantic_sha256, + "storage_provider": storage_provider, + "bucket_alias": bucket_alias, }, "row_count": report.row_count, "channel_manifest": [ @@ -312,13 +409,30 @@ class AtomicReportPublisher: os.chmod(result_backup, 0o600) archive_created = self._install_once(built.path, archive_path, built.sha256) + storage_provider = "local" + bucket_alias = "arr-project-root" artifact_storage_key = self._storage_key(archive_path) + if self._object_store is not None: + uploaded_artifact = self._object_store.upload_committed( + job_id=f"monthly-report-{reservation.report_version_id}", + attempt_no=1, + role="monthly_report", + source=archive_path, + original_filename=report.filename, + expected_sha256=built.sha256, + expected_byte_size=built.byte_size, + ) + storage_provider = "oss" + bucket_alias = self._bucket_alias + artifact_storage_key = uploaded_artifact.object_key result_payload = self._success_result( report, reservation, artifact_storage_key, built.sha256, str(built.summary.get("semantic_sha256", "")), + storage_provider, + bucket_alias, ) if result_path.exists(): expected = json.dumps( @@ -355,6 +469,38 @@ class AtomicReportPublisher: sha256=sha256_file(result_path), byte_size=result_path.stat().st_size, mime_type="application/json", + storage_provider=storage_provider, + bucket_alias=bucket_alias, + ) + if self._object_store is not None: + uploaded_result = self._object_store.upload_committed( + job_id=f"monthly-report-{reservation.report_version_id}", + attempt_no=1, + role="result_json", + source=result_path, + original_filename=result_path.name, + expected_sha256=result_json.sha256, + expected_byte_size=result_json.byte_size, + ) + result_json = FileMetadata( + file_kind="result_json", + original_filename=result_path.name, + storage_key=uploaded_result.object_key, + sha256=uploaded_result.sha256, + byte_size=uploaded_result.byte_size, + mime_type=uploaded_result.mime_type, + storage_provider=storage_provider, + bucket_alias=bucket_alias, + ) + artifact = FileMetadata( + file_kind="monthly_xlsx", + original_filename=report.filename, + storage_key=artifact_storage_key, + sha256=built.sha256, + byte_size=built.byte_size, + mime_type=XLSX_MIME, + storage_provider=storage_provider, + bucket_alias=bucket_alias, ) repository.activate_report( reservation, diff --git a/monthly_reports/repository.py b/monthly_reports/repository.py index ceaeeaf..803f97b 100644 --- a/monthly_reports/repository.py +++ b/monthly_reports/repository.py @@ -36,6 +36,8 @@ SHA256_RE = re.compile(r"^[0-9a-f]{64}$") SAFE_CODE_RE = re.compile(r"^[A-Z][A-Z0-9_]{0,63}$") LOCAL_STORAGE_PROVIDER = "local" LOCAL_BUCKET_ALIAS = "arr-project-root" +OSS_STORAGE_PROVIDER = "oss" +OSS_BUCKET_ALIAS = "arr-private" SOURCE_FACTS_SQL = """ @@ -160,6 +162,8 @@ class FileMetadata: sha256: str byte_size: int mime_type: str + storage_provider: str = LOCAL_STORAGE_PROVIDER + bucket_alias: str = LOCAL_BUCKET_ALIAS def validate(self) -> None: storage = PurePosixPath(self.storage_key) @@ -175,6 +179,12 @@ class FileMetadata: or not SHA256_RE.fullmatch(self.sha256) or self.byte_size <= 0 or not self.mime_type.strip() + or not isinstance(self.storage_provider, str) + or self.storage_provider not in {"local", "local_fixture", "oss", "s3"} + or not isinstance(self.bucket_alias, str) + or not self.bucket_alias.strip() + or "/" in self.bucket_alias + or "\\" in self.bucket_alias ): raise RepositoryError( ErrorCode.PUBLISH_FAILED, @@ -519,13 +529,23 @@ class PostgresReportRepository: def _published_artifact(row: Sequence[Any]) -> Optional[FileMetadata]: if row[3] is None: return None + if len(row) >= 11: + provider = str(row[4] or LOCAL_STORAGE_PROVIDER) + bucket_alias = str(row[5] or LOCAL_BUCKET_ALIAS) + filename_index, key_index, hash_index, size_index, mime_index = 6, 7, 8, 9, 10 + else: + provider = LOCAL_STORAGE_PROVIDER + bucket_alias = LOCAL_BUCKET_ALIAS + filename_index, key_index, hash_index, size_index, mime_index = 4, 5, 6, 7, 8 metadata = FileMetadata( file_kind=str(row[3]), - original_filename=str(row[4]), - storage_key=str(row[5]), - sha256=str(row[6]), - byte_size=int(row[7]), - mime_type=str(row[8] or "application/octet-stream"), + original_filename=str(row[filename_index]), + storage_key=str(row[key_index]), + sha256=str(row[hash_index]), + byte_size=int(row[size_index]), + mime_type=str(row[mime_index] or "application/octet-stream"), + storage_provider=provider, + bucket_alias=bucket_alias, ) metadata.validate() return metadata @@ -544,6 +564,8 @@ class PostgresReportRepository: run.version_no, run.report_status, artifact.artifact_kind, + artifact.storage_provider, + artifact.bucket_alias, artifact.original_filename, artifact.object_key, artifact.sha256, @@ -660,7 +682,7 @@ class PostgresReportRepository: ) @staticmethod - def _ensure_local_artifact(cursor: Any, metadata: FileMetadata) -> int: + def _ensure_artifact(cursor: Any, metadata: FileMetadata) -> int: cursor.execute( """ SELECT @@ -677,7 +699,7 @@ class PostgresReportRepository: AND object_version_id IS NULL FOR SHARE """, - (LOCAL_STORAGE_PROVIDER, LOCAL_BUCKET_ALIAS, metadata.storage_key), + (metadata.storage_provider, metadata.bucket_alias, metadata.storage_key), ) row = cursor.fetchone() if row: @@ -710,8 +732,8 @@ class PostgresReportRepository: """, ( metadata.file_kind, - LOCAL_STORAGE_PROVIDER, - LOCAL_BUCKET_ALIAS, + metadata.storage_provider, + metadata.bucket_alias, metadata.storage_key, metadata.original_filename, metadata.sha256, @@ -782,8 +804,8 @@ class PostgresReportRepository: ErrorCode.PUBLISH_FAILED, "monthly report reservation identity changed", ) - workbook_artifact_id = self._ensure_local_artifact(cursor, artifact) - result_artifact_id = self._ensure_local_artifact(cursor, result_json) + workbook_artifact_id = self._ensure_artifact(cursor, artifact) + result_artifact_id = self._ensure_artifact(cursor, result_json) if str(run[0]) in {"active", "superseded"}: cursor.execute( """ diff --git a/monthly_reports/worker.py b/monthly_reports/worker.py index 1b9cfae..340a4f5 100644 --- a/monthly_reports/worker.py +++ b/monthly_reports/worker.py @@ -13,7 +13,8 @@ from typing import Any, Callable, Mapping, Optional, Protocol, Sequence from arr_database import controlled_connect from monthly_reports.contracts import ErrorCode -from monthly_reports.publishing import ArtifactToolBuilder, AtomicReportPublisher +from arr_web.processing_runtime import ObjectStoreRuntime, compose_object_store +from monthly_reports.publishing import OpenpyxlWorkbookBuilder, AtomicReportPublisher from monthly_reports.repository import ( DatabaseConfig, DerivedMonthlyRequest, @@ -273,10 +274,17 @@ class MonthlyOutboxWorker: outbox: OutboxRepository, requests: MonthlyRequestRepository, service: MonthlyReportService, + storage_runtime: Optional[ObjectStoreRuntime] = None, ) -> None: self._outbox = outbox self._requests = requests self._service = service + self._storage_runtime = storage_runtime + + def close(self) -> None: + if self._storage_runtime is not None: + self._storage_runtime.close() + self._storage_runtime = None @staticmethod def _daily_version_id(event: OutboxEvent) -> int: @@ -362,8 +370,6 @@ def _parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser(prog="arr-monthly-worker") parser.add_argument("--db-config", type=Path) parser.add_argument("--driver-path", type=Path) - parser.add_argument("--node-binary", type=Path) - parser.add_argument("--artifact-tool-module", type=Path) parser.add_argument("--output-root", type=Path) parser.add_argument("--poll-seconds", type=float, default=2.0) parser.add_argument("--lease-seconds", type=int, default=DEFAULT_LEASE_SECONDS) @@ -391,18 +397,27 @@ def _runtime(args: argparse.Namespace) -> MonthlyOutboxWorker: output_root.relative_to(PROJECT_ROOT) except ValueError: raise ValueError("worker output root must be inside the project") from None - builder = ArtifactToolBuilder( - PROJECT_ROOT / "monthly_reports" / "xlsx" / "build_workbook.mjs", - node_binary=str(args.node_binary) if args.node_binary else None, - artifact_tool_module=args.artifact_tool_module, - ) - service = MonthlyReportService( - repository, - builder, - AtomicReportPublisher(PROJECT_ROOT, output_root), - output_root / ".staging", - ) - return MonthlyOutboxWorker(outbox, repository, service) + storage_runtime = compose_object_store() + try: + service = MonthlyReportService( + repository, + OpenpyxlWorkbookBuilder(), + AtomicReportPublisher( + PROJECT_ROOT, + output_root, + object_store=storage_runtime.object_store, + ), + output_root / ".staging", + ) + return MonthlyOutboxWorker( + outbox, + repository, + service, + storage_runtime=storage_runtime, + ) + except Exception: + storage_runtime.close() + raise def _emit(outcome: WorkerOutcome) -> None: @@ -415,32 +430,35 @@ def main(argv: Optional[Sequence[str]] = None) -> int: if args.poll_seconds < 0.1 or args.poll_seconds > 60: raise SystemExit("poll interval must be between 0.1 and 60 seconds") worker = _runtime(args) - if args.once: - try: - outcome = worker.process_next() - except WorkerError as error: - outcome = WorkerOutcome(status="worker_error", error_code=error.code) - _emit(outcome) - return 0 if outcome.status in {"idle", "published"} else 2 - - stopping = False - - def stop(_signum: int, _frame: object) -> None: - nonlocal stopping - stopping = True - - signal.signal(signal.SIGTERM, stop) - signal.signal(signal.SIGINT, stop) - while not stopping: - try: - outcome = worker.process_next() - except WorkerError as error: - outcome = WorkerOutcome(status="worker_error", error_code=error.code) - if outcome.status != "idle": + try: + if args.once: + try: + outcome = worker.process_next() + except WorkerError as error: + outcome = WorkerOutcome(status="worker_error", error_code=error.code) _emit(outcome) - if outcome.status in {"idle", "worker_error"} and not stopping: - time.sleep(args.poll_seconds) - return 0 + return 0 if outcome.status in {"idle", "published"} else 2 + + stopping = False + + def stop(_signum: int, _frame: object) -> None: + nonlocal stopping + stopping = True + + signal.signal(signal.SIGTERM, stop) + signal.signal(signal.SIGINT, stop) + while not stopping: + try: + outcome = worker.process_next() + except WorkerError as error: + outcome = WorkerOutcome(status="worker_error", error_code=error.code) + if outcome.status != "idle": + _emit(outcome) + if outcome.status in {"idle", "worker_error"} and not stopping: + time.sleep(args.poll_seconds) + return 0 + finally: + worker.close() if __name__ == "__main__": diff --git a/monthly_reports/xlsx/build_workbook.mjs b/monthly_reports/xlsx/build_workbook.mjs deleted file mode 100644 index ec78ea8..0000000 --- a/monthly_reports/xlsx/build_workbook.mjs +++ /dev/null @@ -1,464 +0,0 @@ -import crypto from "node:crypto"; -import fs from "node:fs/promises"; -import path from "node:path"; -import process from "node:process"; -import { pathToFileURL } from "node:url"; - - -let FileBlob; -let SpreadsheetFile; -let Workbook; - - -const STANDARD_HEADERS = [ - "ARRIVAL", - "DEPARTURE", - "NIGHTS", - "ADULTS", - "CHILDREN", - "BLOCK_CODE", - "NO_OF_ROOMS", - "COMPANY_NAME", - "CONFIRMATION_NO", - "DISP_ROOM_NO", - "RATE_AMOUNT", - "FULL_NAME", - "RES_COMMENT", - "TRACE_TEXT", - "PRODUCTS", - "RATE_CODE", - "ROOM_CATEGORY_LABEL", - "REAL PRICE", - "TOTAL PRICE", -]; -const KB_HEADER = "KB（100/晚/间）"; -const KB_SHEET = "DY-AI-Easy-KB"; -const DATE_FIELDS = new Set(["ARRIVAL", "DEPARTURE"]); -const INTEGER_FIELDS = new Set(["NIGHTS", "ADULTS", "CHILDREN", "NO_OF_ROOMS"]); -const DECIMAL_FIELDS = new Set(["RATE_AMOUNT", "REAL PRICE", "TOTAL PRICE", KB_HEADER]); -const TEXT_FIELDS = new Set([ - "BLOCK_CODE", - "COMPANY_NAME", - "CONFIRMATION_NO", - "DISP_ROOM_NO", - "FULL_NAME", - "RES_COMMENT", - "TRACE_TEXT", - "PRODUCTS", - "RATE_CODE", - "ROOM_CATEGORY_LABEL", -]); -const DATE_PATTERN = /^\d{4}-\d{2}-\d{2}$/; -const DECIMAL_PATTERN = /^(0|[1-9]\d*)(?:\.(\d{1,2}))?$/; -const INVALID_SHEET_CHARS = /[:\\/?*\[\]]/; -const HEADER_FILL = "#FFFFFF"; -const BODY_FONT = "#222222"; -const BORDER_COLOR = "#BFBFBF"; -const MAX_PREVIEW_ROWS = 25; - - -function fail(message) { - throw new Error(message); -} - - -async function loadArtifactTool() { - const configured = String(process.env.MONTHLY_REPORT_ARTIFACT_TOOL_MODULE ?? "").trim(); - const module = configured - ? await import(pathToFileURL(path.resolve(configured)).href) - : await import("@oai/artifact-tool"); - ({ FileBlob, SpreadsheetFile, Workbook } = module); - if (!FileBlob || !SpreadsheetFile || !Workbook) fail("artifact-tool module is invalid"); -} - - -function safeText(value) { - const text = String(value ?? ""); - return /^[=+\-@]/.test(text) ? `'${text}` : text; -} - - -function excelDate(value) { - if (!DATE_PATTERN.test(String(value ?? ""))) fail("invalid date value"); - const [year, month, day] = value.split("-").map(Number); - const parsed = new Date(Date.UTC(year, month - 1, day)); - if ( - parsed.getUTCFullYear() !== year - || parsed.getUTCMonth() !== month - 1 - || parsed.getUTCDate() !== day - ) { - fail("invalid date value"); - } - return parsed; -} - - -function dateKey(value) { - if (value instanceof Date && !Number.isNaN(value.getTime())) { - return value.toISOString().slice(0, 10); - } - if (typeof value === "number" && Number.isFinite(value)) { - const epoch = Date.UTC(1899, 11, 30); - return new Date(epoch + Math.floor(value) * 86400000).toISOString().slice(0, 10); - } - if (typeof value === "string" && DATE_PATTERN.test(value.slice(0, 10))) { - return value.slice(0, 10); - } - return ""; -} - - -function decimalNumber(value) { - const text = String(value ?? ""); - const match = DECIMAL_PATTERN.exec(text); - if (!match) fail("invalid decimal value"); - const number = Number(text); - if ( - !Number.isFinite(number) - || number < 0 - || !Number.isSafeInteger(Math.round(number * 100)) - ) { - fail("unsafe decimal value"); - } - return number; -} - - -function columnName(index) { - let value = index; - let result = ""; - while (value > 0) { - value -= 1; - result = String.fromCharCode(65 + (value % 26)) + result; - value = Math.floor(value / 26); - } - return result; -} - - -function validatePayload(payload) { - if (!payload || payload.schema_version !== "1.0") fail("invalid payload schema"); - if ( - !Number.isInteger(payload.report_year) - || !Number.isInteger(payload.report_month) - || payload.report_month < 1 - || payload.report_month > 12 - || !DATE_PATTERN.test(String(payload.as_of_date ?? "")) - || typeof payload.filename !== "string" - || !payload.filename.endsWith(".xlsx") - || !Array.isArray(payload.channels) - || payload.channels.length < 5 - ) { - fail("invalid monthly report contract"); - } - const names = new Set(); - payload.channels.forEach((channel, channelIndex) => { - const expectedHeaders = channel.worksheet === KB_SHEET - ? [...STANDARD_HEADERS, KB_HEADER] - : STANDARD_HEADERS; - if ( - typeof channel.worksheet !== "string" - || channel.worksheet.length < 1 - || channel.worksheet.length > 31 - || channel.worksheet.trim() !== channel.worksheet - || INVALID_SHEET_CHARS.test(channel.worksheet) - || names.has(channel.worksheet) - || channel.worksheet_order !== channelIndex + 1 - || !Array.isArray(channel.headers) - || JSON.stringify(channel.headers) !== JSON.stringify(expectedHeaders) - || !Array.isArray(channel.rows) - ) { - fail("invalid channel worksheet contract"); - } - names.add(channel.worksheet); - for (const row of channel.rows) { - if (!row || Object.keys(row).length !== expectedHeaders.length) { - fail("invalid monthly row contract"); - } - for (const header of expectedHeaders) { - if (!Object.hasOwn(row, header)) fail("monthly row field is missing"); - const value = row[header]; - if (DATE_FIELDS.has(header)) { - excelDate(value); - } else if (INTEGER_FIELDS.has(header)) { - if (!Number.isInteger(value) || value < 0) fail("invalid integer value"); - } else if (DECIMAL_FIELDS.has(header)) { - decimalNumber(value); - } else if (TEXT_FIELDS.has(header)) { - if (typeof value !== "string") fail("invalid text value"); - } else { - fail("unknown monthly row field"); - } - } - } - }); -} - - -function rowValues(channel, row) { - return channel.headers.map((header) => { - const value = row[header]; - if (DATE_FIELDS.has(header)) return excelDate(value); - if (INTEGER_FIELDS.has(header)) return value; - if (DECIMAL_FIELDS.has(header)) return decimalNumber(value); - return safeText(value); - }); -} - - -function setColumnWidths(sheet, headers) { - const widths = { - ARRIVAL: 13, - DEPARTURE: 13, - NIGHTS: 9, - ADULTS: 9, - CHILDREN: 9, - BLOCK_CODE: 18, - NO_OF_ROOMS: 13, - COMPANY_NAME: 28, - CONFIRMATION_NO: 20, - DISP_ROOM_NO: 18, - RATE_AMOUNT: 14, - FULL_NAME: 22, - RES_COMMENT: 28, - TRACE_TEXT: 26, - PRODUCTS: 24, - RATE_CODE: 17, - ROOM_CATEGORY_LABEL: 22, - "REAL PRICE": 14, - "TOTAL PRICE": 16, - [KB_HEADER]: 18, - }; - headers.forEach((header, index) => { - sheet.getRange(`${columnName(index + 1)}1`).format.columnWidth = widths[header] ?? 13; - }); -} - - -function writeSheet(workbook, channel) { - const sheet = workbook.worksheets.add(channel.worksheet); - const matrix = [channel.headers, ...channel.rows.map((row) => rowValues(channel, row))]; - const lastColumn = columnName(channel.headers.length); - const lastRow = matrix.length; - const used = sheet.getRange(`A1:${lastColumn}${lastRow}`); - used.values = matrix; - if (channel.rows.length > 0) { - sheet.getRange(`S2:S${lastRow}`).formulas = channel.rows.map((_, index) => [ - `=R${index + 2}*C${index + 2}*G${index + 2}`, - ]); - } - used.format.font = { name: "Arial", size: 10, color: BODY_FONT }; - used.format.verticalAlignment = "center"; - used.format.borders = { - top: { style: "thin", color: BORDER_COLOR }, - bottom: { style: "thin", color: BORDER_COLOR }, - left: { style: "thin", color: BORDER_COLOR }, - right: { style: "thin", color: BORDER_COLOR }, - }; - setColumnWidths(sheet, channel.headers); - - const header = sheet.getRange(`A1:${lastColumn}1`); - header.format.fill = HEADER_FILL; - header.format.font = { name: "Arial", size: 10, bold: true, color: "#000000" }; - header.format.horizontalAlignment = "center"; - header.format.rowHeight = 24; - header.format.wrapText = true; - sheet.freezePanes.freezeRows(1); - - if (channel.rows.length > 0) { - const data = sheet.getRange(`A2:${lastColumn}${lastRow}`); - data.format.rowHeight = 22; - sheet.getRange(`A2:B${lastRow}`).setNumberFormat("dd-mmm-yy"); - sheet.getRange(`C2:E${lastRow}`).setNumberFormat("0"); - sheet.getRange(`G2:G${lastRow}`).setNumberFormat("0"); - sheet.getRange(`K2:K${lastRow}`).setNumberFormat("0.##"); - sheet.getRange(`R2:S${lastRow}`).setNumberFormat("0.##"); - if (channel.worksheet === KB_SHEET) { - sheet.getRange(`T2:T${lastRow}`).setNumberFormat("0.##"); - } - sheet.getRange(`A2:K${lastRow}`).format.horizontalAlignment = "center"; - sheet.getRange(`L2:${lastColumn}${lastRow}`).format.horizontalAlignment = "left"; - sheet.getRange(`F2:${lastColumn}${lastRow}`).format.wrapText = true; - } - sheet.showGridLines = true; -} - - -function expectedCell(header, value) { - if (DATE_FIELDS.has(header)) return { kind: "date", value: String(value) }; - if (INTEGER_FIELDS.has(header)) return { kind: "number", value }; - if (DECIMAL_FIELDS.has(header)) return { kind: "number", value: decimalNumber(value) }; - // A leading apostrophe is the Excel formula-escape marker. The cell API - // exposes the displayed text after consuming that marker. - return { kind: "text", value: String(value ?? "") }; -} - - -async function validateWorkbook(workbook, payload, stage) { - const names = workbook.worksheets.items.map((sheet) => sheet.name); - const expectedNames = payload.channels.map((channel) => channel.worksheet); - if (JSON.stringify(names) !== JSON.stringify(expectedNames)) { - fail(`${stage} worksheet names do not match`); - } - - let formulaCount = 0; - for (const channel of payload.channels) { - const sheet = workbook.worksheets.getItem(channel.worksheet); - const expectedRows = channel.rows.length + 1; - const expectedColumns = channel.headers.length; - const lastColumn = columnName(expectedColumns); - const used = sheet.getUsedRange(); - const values = used?.values ?? []; - if ( - values.length !== expectedRows - || (values[0]?.length ?? 0) !== expectedColumns - || !channel.headers.every((header, index) => values[0][index] === header) - ) { - fail(`${stage} worksheet dimensions or headers do not match`); - } - channel.rows.forEach((row, rowIndex) => { - const actual = values[rowIndex + 1] ?? []; - channel.headers.forEach((header, columnIndex) => { - const expected = expectedCell(header, row[header]); - const actualValue = actual[columnIndex]; - if ( - (expected.kind === "date" && dateKey(actualValue) !== expected.value) - || (expected.kind === "number" && Number(actualValue) !== expected.value) - || (expected.kind === "text" && String(actualValue ?? "") !== expected.value) - ) { - fail( - `${stage} worksheet values do not match at ${channel.worksheet}!` - + `${columnName(columnIndex + 1)}${rowIndex + 2}`, - ); - } - }); - }); - const formulas = used?.formulas ?? []; - formulas.forEach((formulaRow, rowIndex) => { - formulaRow.forEach((formulaValue, columnIndex) => { - const formula = String(formulaValue ?? "").trim(); - if (!formula) return; - formulaCount += 1; - const expectedFormula = `=R${rowIndex + 1}*C${rowIndex + 1}*G${rowIndex + 1}`; - if ( - rowIndex < 1 - || columnIndex !== channel.headers.indexOf("TOTAL PRICE") - || formula !== expectedFormula - ) { - fail(`${stage} workbook contains an unauthorized formula`); - } - }); - }); - const inspected = await workbook.inspect({ - kind: "formula", - sheetId: channel.worksheet, - range: `A1:${lastColumn}${expectedRows}`, - maxChars: 6000, - options: { maxResults: Math.max(channel.rows.length + 10, 100) }, - }); - const inspectionText = String(inspected.ndjson ?? ""); - if (/#REF!|#DIV\/0!|#VALUE!|#NAME\?|#N\/A/.test(inspectionText)) { - fail(`${stage} workbook contains a formula error`); - } - } - const expectedFormulaCount = payload.channels.reduce( - (total, channel) => total + channel.rows.length, - 0, - ); - if (formulaCount !== expectedFormulaCount) { - fail(`${stage} workbook TOTAL PRICE formulas are incomplete`); - } - return formulaCount; -} - - -async function main() { - const [inputPath, outputPath, previewDir, summaryPath] = process.argv.slice(2); - if (!inputPath || !outputPath || !previewDir || !summaryPath) { - fail("usage: build_workbook.mjs INPUT_JSON OUTPUT_XLSX PREVIEW_DIR SUMMARY_JSON"); - } - await loadArtifactTool(); - const payload = JSON.parse(await fs.readFile(inputPath, "utf8")); - validatePayload(payload); - await fs.mkdir(path.dirname(outputPath), { recursive: true }); - await fs.mkdir(previewDir, { recursive: true }); - await fs.mkdir(path.dirname(summaryPath), { recursive: true }); - - const workbook = Workbook.create(); - payload.channels.forEach((channel) => writeSheet(workbook, channel)); - await validateWorkbook(workbook, payload, "pre-export"); - const xlsx = await SpreadsheetFile.exportXlsx(workbook); - await xlsx.save(outputPath); - await fs.chmod(outputPath, 0o600); - - const reopened = await SpreadsheetFile.importXlsx(await FileBlob.load(outputPath)); - const formulaCount = await validateWorkbook(reopened, payload, "post-export"); - const previews = []; - for (let index = 0; index < payload.channels.length; index += 1) { - const channel = payload.channels[index]; - const lastColumn = columnName(channel.headers.length); - const previewRows = Math.min(channel.rows.length + 1, MAX_PREVIEW_ROWS); - const preview = await reopened.render({ - sheetName: channel.worksheet, - range: `A1:${lastColumn}${previewRows}`, - autoCrop: "all", - scale: 1.2, - format: "png", - }); - const previewPath = path.join( - previewDir, - `sheet-${String(index + 1).padStart(2, "0")}.png`, - ); - await fs.writeFile(previewPath, new Uint8Array(await preview.arrayBuffer()), { - mode: 0o600, - }); - previews.push(previewPath); - } - - const stat = await fs.stat(outputPath); - const semanticSha256 = crypto - .createHash("sha256") - .update(JSON.stringify({ channels: payload.channels }), "utf8") - .digest("hex"); - const summary = { - status: "success", - schema_version: payload.schema_version, - filename: payload.filename, - report_year: payload.report_year, - report_month: payload.report_month, - as_of_date: payload.as_of_date, - sheet_names: payload.channels.map((channel) => channel.worksheet), - row_counts: payload.channels.map((channel) => channel.rows.length), - formula_count: formulaCount, - semantic_sha256: semanticSha256, - preview_count: previews.length, - preview_rows: payload.channels.map((channel) => Math.min(channel.rows.length + 1, MAX_PREVIEW_ROWS)), - byte_size: stat.size, - }; - await fs.writeFile(summaryPath, `${JSON.stringify(summary, null, 2)}\n`, { - encoding: "utf8", - mode: 0o600, - }); - await fs.rm(`${outputPath}.inspect.ndjson`, { force: true }); - process.stdout.write(`${JSON.stringify(summary)}\n`); -} - - -try { - await main(); -} catch (_error) { - const outputPath = process.argv[3]; - if (outputPath) { - await fs.rm(`${outputPath}.inspect.ndjson`, { force: true }).catch(() => undefined); - } - if (String(process.env.MONTHLY_REPORT_DEBUG ?? "") === "1") { - process.stderr.write(`${String(_error?.stack ?? _error)}\n`); - } - process.stderr.write( - `${JSON.stringify({ - status: "failed", - code: "MONTHLY_REPORT_OUTPUT_VALIDATION_FAILED", - })}\n`, - ); - process.exitCode = 4; -} diff --git a/monthly_reports/xlsx/package.json b/monthly_reports/xlsx/package.json deleted file mode 100644 index b0980db..0000000 --- a/monthly_reports/xlsx/package.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "name": "arr-monthly-report-xlsx", - "private": true, - "type": "module" -} diff --git a/requirements-monthly-reports.txt b/requirements-monthly-reports.txt index 627fec5..df1627c 100644 --- a/requirements-monthly-reports.txt +++ b/requirements-monthly-reports.txt @@ -1 +1,2 @@ psycopg[binary]==3.2.13 +openpyxl==3.1.5 diff --git a/tests/test_company_reports_acceptance.py b/tests/test_company_reports_acceptance.py index 2ac5424..9e245a2 100644 --- a/tests/test_company_reports_acceptance.py +++ b/tests/test_company_reports_acceptance.py @@ -6,7 +6,7 @@ from pathlib import Path from company_reports.contracts import COMPANY_NAMES, REPORT_HEADERS, WarningCode from company_reports.core import build_all_company_reports -from company_reports.publishing import ArtifactToolBuilder +from company_reports.publishing import OpenpyxlWorkbookBuilder from tests.company_reports_acceptance_fixture import ( AS_OF_DATE, REPORT_MONTH, @@ -15,10 +15,6 @@ from tests.company_reports_acceptance_fixture import ( ) -PROJECT_ROOT = Path(__file__).resolve().parents[1] -BUILDER_SCRIPT = PROJECT_ROOT / "company_reports" / "xlsx" / "build_workbook.mjs" - - def acceptance_reports(): return build_all_company_reports( REPORT_YEAR, @@ -88,10 +84,7 @@ class CompanyReportAcceptanceCoreTests(unittest.TestCase): class CompanyReportAcceptanceWorkbookTests(unittest.TestCase): def test_real_builder_exports_and_reopens_all_five_workbooks(self): - builder = ArtifactToolBuilder( - BUILDER_SCRIPT, - timeout_seconds=180, - ) + builder = OpenpyxlWorkbookBuilder() with tempfile.TemporaryDirectory(prefix="company-report-acceptance-") as temp_dir: root = Path(temp_dir) for report in acceptance_reports(): diff --git a/tests/test_company_reports_integration.py b/tests/test_company_reports_integration.py index a93cef1..9f293a4 100644 --- a/tests/test_company_reports_integration.py +++ b/tests/test_company_reports_integration.py @@ -9,7 +9,7 @@ from datetime import date from pathlib import Path from company_reports.publishing import ( - ArtifactToolBuilder, + OpenpyxlWorkbookBuilder, AtomicReportPublisher, BuiltWorkbook, sha256_file, @@ -19,10 +19,6 @@ from company_reports.service import CompanyReportService, RunRequest from tests.test_company_reports_service import FakeRepository, synthetic_snapshot -PROJECT_ROOT = Path(__file__).resolve().parents[1] -BUILDER_SCRIPT = PROJECT_ROOT / "company_reports" / "xlsx" / "build_workbook.mjs" - - class CompanyReportIntegrationTests(unittest.TestCase): def test_missing_and_unmatched_group_codes_export_blank_booking_room(self): class CapturingBuilder: @@ -46,7 +42,7 @@ class CompanyReportIntegrationTests(unittest.TestCase): ) ) builder = CapturingBuilder( - ArtifactToolBuilder(BUILDER_SCRIPT, timeout_seconds=90) + OpenpyxlWorkbookBuilder() ) publisher = AtomicReportPublisher(project_root, output_root) service = CompanyReportService( @@ -114,7 +110,7 @@ class CompanyReportIntegrationTests(unittest.TestCase): project_root.mkdir(parents=True) repository = StableVersionRepository(synthetic_snapshot()) builder = RebuildingBuilder( - ArtifactToolBuilder(BUILDER_SCRIPT, timeout_seconds=90) + OpenpyxlWorkbookBuilder() ) publisher = AtomicReportPublisher(project_root, output_root) service = CompanyReportService( diff --git a/tests/test_company_reports_publishing.py b/tests/test_company_reports_publishing.py index 4dd41fb..8fc243e 100644 --- a/tests/test_company_reports_publishing.py +++ b/tests/test_company_reports_publishing.py @@ -19,7 +19,7 @@ from company_reports.contracts import ( ) from company_reports.core import build_company_report from company_reports.publishing import ( - ArtifactToolBuilder, + OpenpyxlWorkbookBuilder, AtomicReportPublisher, BuiltWorkbook, PublicationError, @@ -112,14 +112,11 @@ def synthetic_built_workbook( class CompanyReportPublishingTests(unittest.TestCase): - def test_builder_creates_xlsx_without_private_artifact_tool_runtime(self): + def test_builder_creates_xlsx_with_openpyxl_only(self): with tempfile.TemporaryDirectory(prefix="company-report-builder-test-") as temp_dir: root = Path(temp_dir) report = workbook_report() - builder = ArtifactToolBuilder( - Path("/private/artifact-tool/build_workbook.mjs"), - node_binary="/definitely/not/node", - ) + builder = OpenpyxlWorkbookBuilder() built = builder.build(report, root) diff --git a/tests/test_company_reports_xlsx.py b/tests/test_company_reports_xlsx.py deleted file mode 100644 index bc89b8c..0000000 --- a/tests/test_company_reports_xlsx.py +++ /dev/null @@ -1,119 +0,0 @@ -from __future__ import annotations - -import json -import os -import shutil -import subprocess -import tempfile -import unittest -from pathlib import Path - - -PROJECT_ROOT = Path(__file__).resolve().parents[1] -BUILDER = PROJECT_ROOT / "company_reports" / "xlsx" / "build_workbook.mjs" -FIXTURE = ( - PROJECT_ROOT - / "tests" - / "fixtures" - / "company_reports" - / "synthetic_qbd_payload.json" -) -ARTIFACT_PACKAGE = ( - PROJECT_ROOT - / "company_reports" - / "xlsx" - / "node_modules" - / "@oai" - / "artifact-tool" -) - - -def artifact_tool_available() -> bool: - configured = os.environ.get("COMPANY_REPORT_ARTIFACT_TOOL_MODULE", "").strip() - return ARTIFACT_PACKAGE.exists() or bool(configured and Path(configured).is_file()) - - -def node_binary() -> str: - configured = os.environ.get("COMPANY_REPORT_NODE_BINARY", "").strip() - return configured or shutil.which("node") or "" - - -@unittest.skipUnless(artifact_tool_available(), "artifact-tool dependency is not installed") -class CompanyReportXlsxTests(unittest.TestCase): - def test_builder_exports_reopens_checks_and_renders_all_sheets(self): - node = node_binary() - if not node: - self.skipTest("Node.js is unavailable") - with tempfile.TemporaryDirectory(prefix="company-report-xlsx-test-") as temp_dir: - root = Path(temp_dir) - output = root / "QBD-July-2026.xlsx" - previews = root / "previews" - summary_path = root / "summary.json" - completed = subprocess.run( - [ - node, - str(BUILDER), - str(FIXTURE), - str(output), - str(previews), - str(summary_path), - ], - cwd=PROJECT_ROOT, - capture_output=True, - text=True, - timeout=90, - check=False, - ) - - self.assertEqual(completed.returncode, 0, completed.stderr) - self.assertTrue(output.is_file()) - self.assertFalse(Path(f"{output}.inspect.ndjson").exists()) - self.assertEqual(output.stat().st_mode & 0o777, 0o600) - summary = json.loads(summary_path.read_text(encoding="utf-8")) - self.assertEqual(summary["status"], "success") - self.assertEqual(summary["row_counts"], [2, 1, 0]) - self.assertEqual(summary["formula_count"], 0) - self.assertEqual(summary["preview_count"], 3) - self.assertEqual(len(list(previews.glob("sheet-*.png"))), 3) - - def test_invalid_payload_fails_with_only_a_stable_error_code(self): - node = node_binary() - if not node: - self.skipTest("Node.js is unavailable") - with tempfile.TemporaryDirectory(prefix="company-report-xlsx-test-") as temp_dir: - root = Path(temp_dir) - invalid = root / "invalid.json" - invalid.write_text( - json.dumps({"schema_version": "invalid", "synthetic_marker": "SYN-ONLY"}), - encoding="utf-8", - ) - completed = subprocess.run( - [ - node, - str(BUILDER), - str(invalid), - str(root / "invalid.xlsx"), - str(root / "previews"), - str(root / "summary.json"), - ], - cwd=PROJECT_ROOT, - capture_output=True, - text=True, - timeout=30, - check=False, - ) - - self.assertEqual(completed.returncode, 4) - self.assertEqual( - json.loads(completed.stderr), - { - "status": "failed", - "code": "COMPANY_REPORT_OUTPUT_VALIDATION_FAILED", - }, - ) - self.assertNotIn("SYN-ONLY", completed.stderr) - self.assertFalse((root / "invalid.xlsx").exists()) - - -if __name__ == "__main__": - unittest.main() diff --git a/tests/test_deployment_entrypoints.py b/tests/test_deployment_entrypoints.py index a6501f6..7b39b0f 100644 --- a/tests/test_deployment_entrypoints.py +++ b/tests/test_deployment_entrypoints.py @@ -33,6 +33,9 @@ class DeploymentEntrypointTests(unittest.TestCase): root_requirements = (PROJECT_ROOT / "requirements.txt").read_text( encoding="utf-8" ) + monthly_requirements = ( + PROJECT_ROOT / "requirements-monthly-reports.txt" + ).read_text(encoding="utf-8") company_requirements = ( PROJECT_ROOT / "requirements-company-reports.txt" ).read_text(encoding="utf-8") @@ -51,13 +54,11 @@ class DeploymentEntrypointTests(unittest.TestCase): self.assertNotIn("basic_auth", caddy) self.assertNotIn("requirements-agent-integration", root_requirements) self.assertNotIn("requirements-arr-mcp", root_requirements) + self.assertIn("openpyxl", monthly_requirements) self.assertIn("openpyxl", company_requirements) - company_package = json.loads( - (PROJECT_ROOT / "company_reports" / "xlsx" / "package.json").read_text( - encoding="utf-8" - ) - ) - self.assertNotIn("@oai/artifact-tool", json.dumps(company_package)) + self.assertFalse((PROJECT_ROOT / "company_reports" / "xlsx" / "package.json").exists()) + self.assertFalse((PROJECT_ROOT / "monthly_reports" / "xlsx" / "package.json").exists()) + self.assertNotIn("artifact-tool", compose) if __name__ == "__main__": diff --git a/tests/test_monthly_oss_artifact_migration.py b/tests/test_monthly_oss_artifact_migration.py new file mode 100644 index 0000000..11b9772 --- /dev/null +++ b/tests/test_monthly_oss_artifact_migration.py @@ -0,0 +1,31 @@ +from __future__ import annotations + +import unittest +from pathlib import Path + + +PROJECT_ROOT = Path(__file__).resolve().parents[1] + + +class MonthlyOssArtifactMigrationTests(unittest.TestCase): + def test_forward_migration_allows_oss_and_keeps_local_compatibility(self): + sql = (PROJECT_ROOT / "database" / "016_monthly_report_oss_artifacts.sql").read_text( + encoding="utf-8" + ) + self.assertIn("current_database() <> 'booking_test'", sql) + self.assertIn("COALESCE(workbook_provider, '') NOT IN ('oss', 's3', 'local')", sql) + self.assertIn("COALESCE(result_provider, '') NOT IN ('oss', 's3', 'local')", sql) + self.assertIn("CREATE OR REPLACE FUNCTION reporting.validate_monthly_run_publication()", sql) + self.assertIn("existing private OSS", sql) + + def test_down_migration_refuses_published_nonlocal_artifacts(self): + sql = ( + PROJECT_ROOT / "database" / "016_monthly_report_oss_artifacts.down.sql" + ).read_text(encoding="utf-8") + self.assertIn("artifact.storage_provider <> 'local'", sql) + self.assertIn("rollback refused", sql) + self.assertIn("workbook_provider IS DISTINCT FROM 'local'", sql) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_monthly_reports_xlsx.py b/tests/test_monthly_reports_xlsx.py index 86273c3..287e43d 100644 --- a/tests/test_monthly_reports_xlsx.py +++ b/tests/test_monthly_reports_xlsx.py @@ -1,9 +1,5 @@ from __future__ import annotations -import json -import os -import shutil -import subprocess import tempfile import unittest from datetime import date @@ -19,21 +15,10 @@ from monthly_reports.contracts import ( MonthlySnapshot, ) from monthly_reports.core import build_monthly_report +from monthly_reports.publishing import OpenpyxlWorkbookBuilder -PROJECT_ROOT = Path(__file__).resolve().parents[1] -BUILDER = PROJECT_ROOT / "monthly_reports" / "xlsx" / "build_workbook.mjs" - - -def artifact_tool_module() -> str: - return os.environ.get("MONTHLY_REPORT_ARTIFACT_TOOL_MODULE", "").strip() - - -def node_binary() -> str: - return os.environ.get("MONTHLY_REPORT_NODE_BINARY", "").strip() or shutil.which("node") or "" - - -def report_payload() -> dict: +def report_fixture(): source = MonthlyFact( daily_record_id=1, daily_version_id=100, @@ -60,7 +45,7 @@ def report_payload() -> dict: total_price=Decimal("901.00"), kb_amount=None, ) - report = build_monthly_report( + return build_monthly_report( 2026, 7, date(2026, 7, 8), @@ -70,59 +55,28 @@ def report_payload() -> dict: (ChannelObservation(date(2026, 7, 8), 100, "QBD", 1, 1),), ), ) - return report.to_workbook_payload() -@unittest.skipUnless( - bool(artifact_tool_module() and Path(artifact_tool_module()).is_file()), - "artifact-tool module is not configured", -) class MonthlyReportsXlsxTests(unittest.TestCase): - def test_builder_exports_reopens_scans_and_renders_every_channel(self): - node = node_binary() - if not node: - self.skipTest("Node.js is unavailable") + def test_openpyxl_builder_reopens_checks_formulas_and_semantic_identity(self): + report = report_fixture() with tempfile.TemporaryDirectory(prefix="monthly-report-xlsx-") as temp_dir: - root = Path(temp_dir) - payload_path = root / "payload.json" - payload_path.write_text( - json.dumps(report_payload(), ensure_ascii=False), - encoding="utf-8", - ) - output = root / "monthly.xlsx" - previews = root / "previews" - summary_path = root / "summary.json" - completed = subprocess.run( - [ - node, - str(BUILDER), - str(payload_path), - str(output), - str(previews), - str(summary_path), - ], - cwd=PROJECT_ROOT, - capture_output=True, - text=True, - timeout=120, - check=False, - env=os.environ.copy(), - ) + built = OpenpyxlWorkbookBuilder().build(report, Path(temp_dir)) - self.assertEqual(completed.returncode, 0, completed.stderr) - summary = json.loads(summary_path.read_text(encoding="utf-8")) - self.assertEqual(summary["sheet_names"][:5], [ + self.assertTrue(built.path.is_file()) + self.assertEqual(built.path.stat().st_mode & 0o777, 0o600) + self.assertEqual(built.summary["sheet_names"][:5], [ "LIANTAI-GROUP", "LIANTAI-FIT", "QBD", "DY-AI-Easy-KB", "FENGRUN" ]) - self.assertEqual(summary["row_counts"], [0, 0, 1, 0, 0]) - self.assertEqual(summary["formula_count"], 1) - self.assertEqual(summary["preview_count"], 5) - self.assertEqual(len(list(previews.glob("sheet-*.png"))), 5) - self.assertEqual(output.stat().st_mode & 0o777, 0o600) - self.assertFalse(Path(f"{output}.inspect.ndjson").exists()) - workbook = load_workbook(output, data_only=False, read_only=True) + self.assertEqual(built.summary["row_counts"], [0, 0, 1, 0, 0]) + self.assertEqual(built.summary["formula_count"], 1) + self.assertEqual(built.summary["preview_count"], 5) + self.assertEqual(len(built.summary["semantic_sha256"]), 64) + + workbook = load_workbook(built.path, data_only=False, read_only=True) try: self.assertEqual(workbook["QBD"]["S2"].value, "=R2*C2*G2") + self.assertEqual(workbook["QBD"]["F2"].value, "'=SYN-FORMULA-LIKE") formulas = [ cell.value for sheet in workbook.worksheets @@ -134,43 +88,6 @@ class MonthlyReportsXlsxTests(unittest.TestCase): finally: workbook.close() - def test_invalid_payload_returns_only_stable_error_code(self): - node = node_binary() - if not node: - self.skipTest("Node.js is unavailable") - with tempfile.TemporaryDirectory(prefix="monthly-report-xlsx-") as temp_dir: - root = Path(temp_dir) - invalid = root / "invalid.json" - invalid.write_text( - json.dumps({"schema_version": "bad", "sensitive": "SYN-SECRET"}), - encoding="utf-8", - ) - completed = subprocess.run( - [ - node, - str(BUILDER), - str(invalid), - str(root / "invalid.xlsx"), - str(root / "previews"), - str(root / "summary.json"), - ], - cwd=PROJECT_ROOT, - capture_output=True, - text=True, - timeout=30, - check=False, - env=os.environ.copy(), - ) - self.assertEqual(completed.returncode, 4) - self.assertEqual( - json.loads(completed.stderr), - { - "status": "failed", - "code": "MONTHLY_REPORT_OUTPUT_VALIDATION_FAILED", - }, - ) - self.assertNotIn("SYN-SECRET", completed.stderr) - if __name__ == "__main__": unittest.main() diff --git a/tests/test_report_artifacts.py b/tests/test_report_artifacts.py new file mode 100644 index 0000000..c0e0cdf --- /dev/null +++ b/tests/test_report_artifacts.py @@ -0,0 +1,182 @@ +from __future__ import annotations + +import hashlib +import tempfile +import unittest +from pathlib import Path + +from arr_storage.filesystem import FilesystemObjectBackend +from arr_storage.store import ManagedObjectStore +from arr_web.downloads import ( + ArtifactDescriptor, + ControlledProjectArtifactReader, + ManagedObjectArtifactReader, + RoutedArtifactReader, +) +from company_reports.publishing import AtomicReportPublisher as CompanyPublisher +from company_reports.repository import ReservedReport as CompanyReservation +from monthly_reports.publishing import ( + AtomicReportPublisher as MonthlyPublisher, + OpenpyxlWorkbookBuilder, + private_staging_directory, +) +from monthly_reports.repository import ReservedReport as MonthlyReservation +from tests.test_company_reports_publishing import ( + FakeRepository as CompanyFakeRepository, + empty_report, + synthetic_built_workbook, +) +from tests.test_monthly_reports_publishing import FakeRepository as MonthlyFakeRepository +from tests.test_monthly_reports_xlsx import report_fixture + + +XLSX_MIME = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" + + +class ReportArtifactStorageTests(unittest.TestCase): + def test_monthly_and_company_publications_upload_workbook_and_result_to_oss(self): + with tempfile.TemporaryDirectory(prefix="report-artifact-oss-") as temporary: + root = Path(temporary) / "project" + root.mkdir() + store = ManagedObjectStore( + FilesystemObjectBackend(root / "objects", create=True) + ) + + monthly_report = report_fixture() + monthly_work = root / "monthly-work" + monthly_work.mkdir() + monthly_built = OpenpyxlWorkbookBuilder().build(monthly_report, monthly_work) + monthly_outcome = MonthlyPublisher( + root, + root / "outputs" / "monthly_reports", + object_store=store, + ).publish( + monthly_report, + MonthlyReservation(44, 2), + monthly_built, + MonthlyFakeRepository(), + monthly_work, + ) + + self.assertEqual(monthly_outcome.artifact.storage_provider, "oss") + self.assertEqual(monthly_outcome.result_json.storage_provider, "oss") + self.assertEqual( + store.inspect_committed( + monthly_outcome.artifact.storage_key, + monthly_report.filename, + ).role, + "monthly_report", + ) + self.assertEqual( + store.inspect_committed( + monthly_outcome.result_json.storage_key, + monthly_outcome.result_json.original_filename, + ).role, + "result_json", + ) + + company_report = empty_report() + company_work = root / "company-work" + company_work.mkdir() + company_built = synthetic_built_workbook( + company_work, + company_report.filename, + b"synthetic-company-xlsx", + "a" * 64, + ) + company_outcome = CompanyPublisher( + root, + root / "outputs" / "company_reports", + object_store=store, + ).publish( + company_report, + CompanyReservation(45, 3, company_report.company), + company_built, + CompanyFakeRepository(), + company_work, + ) + + self.assertEqual(company_outcome.artifact.storage_provider, "oss") + self.assertEqual(company_outcome.result_json.storage_provider, "oss") + self.assertEqual( + store.inspect_committed( + company_outcome.artifact.storage_key, + company_report.filename, + ).role, + "company_report", + ) + + retry_work = root / "company-retry-work" + retry_work.mkdir() + retry_built = synthetic_built_workbook( + retry_work, + company_report.filename, + b"synthetic-company-xlsx-rebuilt", + "a" * 64, + ) + retry_outcome = CompanyPublisher( + root, + root / "outputs" / "company_reports", + object_store=store, + ).publish( + company_report, + CompanyReservation(45, 3, company_report.company), + retry_built, + CompanyFakeRepository(), + retry_work, + ) + self.assertEqual(retry_outcome.artifact.storage_provider, "oss") + self.assertEqual( + retry_outcome.artifact.sha256, + company_outcome.artifact.sha256, + ) + self.assertEqual(retry_outcome.result_json.storage_provider, "oss") + + def test_download_router_supports_oss_and_legacy_local_records(self): + with tempfile.TemporaryDirectory(prefix="report-download-") as temporary: + root = Path(temporary) + source = root / "monthly.xlsx" + source.write_bytes(b"monthly-oss-bytes") + store = ManagedObjectStore( + FilesystemObjectBackend(root / "objects", create=True) + ) + stored = store.upload_committed( + job_id="monthly-download-001", + attempt_no=1, + role="monthly_report", + source=source, + original_filename="月报.xlsx", + ) + oss_descriptor = ArtifactDescriptor( + "monthly_xlsx", + "月报.xlsx", + stored.object_key, + stored.sha256, + stored.byte_size, + stored.mime_type, + "oss", + "arr-private", + ) + local_path = root / "outputs" / "legacy.xlsx" + local_path.parent.mkdir() + local_path.write_bytes(b"legacy-local-bytes") + local_descriptor = ArtifactDescriptor( + "monthly_xlsx", + "legacy.xlsx", + "outputs/legacy.xlsx", + hashlib.sha256(local_path.read_bytes()).hexdigest(), + local_path.stat().st_size, + XLSX_MIME, + ) + reader = RoutedArtifactReader( + daily_reader=None, + local_reader=ControlledProjectArtifactReader(root), + oss_reader=ManagedObjectArtifactReader(store), + ) + + self.assertEqual(reader.read(oss_descriptor), source.read_bytes()) + self.assertEqual(reader.read(local_descriptor), local_path.read_bytes()) + + +if __name__ == "__main__": + unittest.main()