diff --git a/.project-docs/30-worklog/tasks/20261009-daily-lifecycle-7bc2.md b/.project-docs/30-worklog/tasks/20261009-daily-lifecycle-7bc2.md new file mode 100644 index 0000000..2681dab --- /dev/null +++ b/.project-docs/30-worklog/tasks/20261009-daily-lifecycle-7bc2.md @@ -0,0 +1,76 @@ +# Task: 日报按日期覆盖、删除及月报同步 + +## Identity + +- Task ID: 20261009-daily-lifecycle-7bc2 +- Mode: Feature +- Branch: codex/20261009-daily-lifecycle-7bc2-daily-lifecycle +- Worktree: /Users/chillishark/.codex/worktrees/arr-daily-lifecycle +- Base commit: 976a4fa7b2bc445e327f412531c6ccc5e7ddc99a +- Owner: codex +- Status: Ready for Integration + +## Scope + +- User-approved daily report lifecycle: one current report per business date, successful new XML/OHIP versions replace old only after acceptance, manual deletion with monthly refresh, grouped date/history UI and consistent selected-date overview. + +## Intent And Constraints + +- Preserve shared business rules, immutable original/audit facts and live user decisions. Deletion retires the report while retaining source/objects and audit, with no product undo button; deleting a current day must not resurrect an older version. A fresh upload/request may generate a new version. New pending/failed input must not remove the current accepted day. No new Oracle reads or fabricated prices. Use synthetic isolated database acceptance, then safe local activation with backup and no active tasks. Prior release push authorization remains valid. + +## Outcome + +- Implemented business-date grouping and exact selected-date overview, with current accepted report separate from a new pending/failed version. Shared XML/OHIP rules and processor4.4.0 identity unchanged; a successful new version alone replaces the current day. +- Added authenticated/CSRF-protected exact-job/date retirement. Serializability, day/month locks, immutable retirement receipts, current-version guard and outbox atomically remove only the requested effective version; old/pending history retirement preserves current facts. No source/object/Finance fact deletion and no automatic old-version restoration. +- Migration022 adds append-only operator receipts and withdrawn monthly publications, partial live-snapshot uniqueness and immutable withdrawn lineage; grants are confined to known application role. Startup checks022 structure/protection, including missing index/disabled audit trigger rejection. Native XML/API simulation and OHIP owned instances now request schema022 explicitly. +- Monthly worker rebuilds from remaining current Finance days, refuses stale cutoff/pins, consumes committed/scope-changed events, withdraws an empty month without publishing an empty workbook, and blocks pre-deletion builds from becoming current. Withdrawn publications disappear from downloads, lists and analytics; audit and artifacts remain intact. +- Frontend offers one date row, explicit current/history/new statuses, date selection, paged history, deletion impact confirmation, safe duplicate submission and stale-response guards; preserves other-date pending work and unsaved prices. Deleted old successful versions are clearly history-only when no current day remains. Chinese/English/Thai wording and empty read-only preview updated. +- Updated product/deployment docs with022 order, role grants, worker coupling, backed-up upgrade and rollback boundaries. Canonical project memory left unchanged in feature mode. +- Activated loopback8875 after confirming no active acquisition/processing, graceful launchd stop and cold backup. Private launcher/config/plist now point to this owned worktree, schema008-022. Existing login and production hotel57106 routing retained. No actual daily report retired and no Oracle acquisition submitted. + +## Verification + +- Backend: initial33 retirement/ingestion checks passed; after adding022 startup gate, final17 retirement/PostgreSQL cases plus15 runtime composition cases passed (32 total). Covers old/new identity, pending price cancellation, busy generation, duplicate/concurrent retirement, source/fact retention, transaction rollback, day/month races and022 gate failures. +- Monthly:31 repository/service/worker unit checks and7 real isolated PostgreSQL lifecycle cases passed. Includes delete last/middle/latest date, empty whole month, ABA snapshot republication, paused stale builders and immutable withdrawn lineage. +- Portal:7 real isolated PostgreSQL date/history/download/analytics/read-only role cases passed. Root50 API/legacy/schema/runtime checks passed; later19 API/schema/empty-preview checks passed after preview compatibility was added. HTTP checks cover auth, CSRF, strict date/body and session actor binding. +- Local entrypoints:19 native XML/API simulation checks passed with real owned PostgreSQL, explicitly verifying new grouped API on the new schema and restart/download/monthly behavior. +- Frontend:105 JavaScript tests and14 UI/static/privacy tests passed after final history-only status correction. Syntax and whitespace checks passed. Independent review found and resolved old-success label ambiguity and missing local/schema022 startup gate; no remaining blocking issue reported. +- Independent browser port8891 with synthetic data: two effective days plus one pending replacement. Deleted pending09/16 and verified old effective day stayed; deleted effective09/16 and verified list removed the day, selected09/16 overview stayed empty instead of displaying09/17, and rebuilt September month dropped from2 data rows to1. Real hotel data/services were isolated from this fixture. +- Actual local upgrade: baseline and after-upgrade hashes/counts of all26 existing business tables match, ignoring only newly added NULL monthly.withdrawn_at.3 current daily versions,4,896 original source files, queue statuses, legacy jobs/months/monthly responses and existing pending tasks match.022 retirement table remains0 rows. Actual grouped September API returns unique09/17 and09/16 rows; pending09/15 and09/13 survive. +- Actual browser: selected09/13 shows09/13 throughout acquisition/date overview, no borrowed rooms/duration; existing09/16 shows150 rooms, one main row and2 history records with delete controls. No business input saved, deleted or refetched. Screenshots saved privately. +- Backup: `/Users/chillishark/Library/Application Support/ARR2.0/daily-lifecycle-backup-20261009-025721` (cold owned database, config/source/objects/queues; existing runtime excluded). Private evidence/scripts: `/Users/chillishark/Library/Application Support/ARR2.0/daily-lifecycle-validation-20261009`; captures before/after snapshots, activation, synthetic monthly deletion and live date/history screenshots. No credentials/data committed. +- Project documentation and task-aware drift gates passed; canonical snapshot migration remains pending for a separate Integration Gate. Final JavaScript syntax and whitespace checks passed. Approved target is arr0918 `https://git.nianxx.cn/shiyuyun/ARR-2.0-0918.git`, main; pre-publication remote HEAD equals base976a4fa. Publish non-force and verify remote HEAD; store exact post-push receipt privately with activation evidence. + +## Follow-ups + +- Target-host deployment still requires backed-up022 migration, role grants and coordinated new Web/processing/monthly worker. Local acceptance does not claim target-host deployment or full real-report financial business reconciliation. +- Retain this managed worktree:8875 now runs its code. Do not retire/archive it until runtime has moved to another reviewed checkout. Backup must be restored with matching schema/code/worker if an operator later needs full rollback; no rollback was needed. + +## Promotion Candidates + +- Targets: canonical domain business-rules/success-criteria and architecture data-flow. Proposal: daily report identity is business date with one current accepted version; replacement only on success, logical retirement with immutable facts, current-day deletion excludes that day from monthly facts, historical deletion preserves current. Evidence: explicit user approval2026-10-09, isolated PostgreSQL concurrency/withdrawal tests and browser lifecycle proof. Future impact: daily/monthly API, deployments and audit expectations. Conflict: extends existing immutable processing/audit design without changing rate/price/PM/cancellation rules; canonical September snapshot is stale. Human confirmation: product behavior already approved; apply only through a separate serialized Integration Gate. +- Targets: canonical current-state/evidence index. Proposal: local8875 runs this checkout with processor4.4.0/schema022, existing actual data preserved, feature acceptance synthetic and actual portal read-only. Evidence: private activation/baseline/after snapshots and screenshots. Future impact: accurate maintenance/rollback/worktree lifecycle. Conflict: prior production-review checkout is no longer the live code path; primary dirty checkout remains untouched. Human confirmation: no additional decision required to promote verified environment facts, but Integration Gate remains mandatory. + +## Concurrent Task Gate + +- Passed: unique feature task20261009-daily-lifecycle-7bc2, codex owner, branchcodex/20261009-daily-lifecycle-7bc2-daily-lifecycle, managed worktree/Users/chillishark/.codex/worktrees/arr-daily-lifecycle, base976a4fa7b2bc445e327f412531c6ccc5e7ddc99a. start exited0 and status matched. Existing production-review task is Ready for Integration and runs the current service; its task record was inspected read-only. Its code is included in this base, no concurrent write overlap. Dirty primary checkout remains untouched. + +## Project Context Loaded + +Read: read-before-planning/planning-gate, own task record, memory-index, positioning, current-state, decision-index, ADR001/004/006/007, system-overview/module-map/data-flow, business-rules/success-criteria, evidence/reflection/commitments/stale indexes and prior production-review outcome. + +- Project goal: deterministic shared XML/OHIP daily processing, accepted Finance facts and automatically updated monthly reports. +- Current focus: business-date report identity/lifecycle, recoverable deletion and date-consistent presentation. +- Constraints: immutable source/artifacts/audit, atomic activation and outbox, frozen manual-price replay, no invented values or new acquisition, private runtime outside Git. +- Decisions: successful commits activate one Finance version per date; dedicated monthly worker with current-pin validation and derived scope; user now explicitly approves report retirement/monthly removal and date-grouped display. +- Evidence/conflicts: canonical memory last integrated September18 is stale regarding October9 production hotel57106/processor4.4.0/migration021 and multi-agent consent; recent task evidence and direct user authorization govern these current facts. No canonical edits in feature mode. +- Modules: arr_web repository/app/downloads/programmatic/UI, arr_ingestion Finance SQL/protected roles, monthly_reports snapshot/publication/outbox and local runtime migrations. +- Unknowns: exact existing activation, monthly empty-date/month behavior and race boundaries require code inspection; no unresolved product choice blocks the authorized behavior. +- Gate: Passed. + +## Plan + +1. Inspect existing date-version activation and monthly publication; establish a recoverable auditable retirement contract. +2. Implement protected atomic day/report retirement and monthly refresh, reject unsafe stale/concurrent operations and preserve current version until replacement succeeds. +3. Group daily list by date with accessible history/delete controls; bind overview to selected day/current published version versus pending replacement. +4. Verify real isolated PostgreSQL daily/monthly lifecycle, race/idempotent cases, HTTP/auth and UI; safely activate local service without deleting actual reports, capture UI proof and push the reviewed code. diff --git a/arr_ingestion/postgres.py b/arr_ingestion/postgres.py index 7eda27f..c31df2e 100644 --- a/arr_ingestion/postgres.py +++ b/arr_ingestion/postgres.py @@ -8,7 +8,7 @@ import os import re import time from dataclasses import dataclass -from datetime import date +from datetime import date, timedelta from decimal import Decimal, InvalidOperation from typing import Any, Callable, Dict, Mapping, Optional, Sequence, Tuple @@ -215,6 +215,119 @@ class PostgresIngestionRepository(IngestionRepository): "database connection failed", ) from None + def retire_daily_job( + self, job_id: str, business_date: date, actor_username: str + ) -> Dict[str, Any]: + """Retire one exact run, retaining all source, facts and audit lineage.""" + if not isinstance(job_id, str) or re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9._:-]{0,127}", job_id) is None: + raise IngestionError("DAILY_JOB_NOT_FOUND", "日报记录不存在") + if type(business_date) is not date: + raise IngestionError("DAILY_DATE_CONFLICT", "报表日期无效,请刷新后重试") + actor = self._review_actor(actor_username) + return self._run_transaction( + lambda cursor: self._retire_daily_job(cursor, job_id, business_date, actor), + "daily report could not be retired", + ) + + @staticmethod + def _retirement_receipt(job_id: str, business_date: date, current_removed: bool) -> Dict[str, Any]: + return { + "job_id": job_id, + "business_date": business_date.isoformat(), + "is_retired": True, + "current_removed": current_removed, + "monthly_refresh_required": current_removed, + } + + def _retire_daily_job(self, cursor: Any, job_id: str, business_date: date, actor: str) -> Dict[str, Any]: + cursor.execute( + """SELECT id, business_date, run_status + FROM ingestion.processing_runs + WHERE run_key = %s AND pipeline_type = 'opera_daily' + FOR UPDATE""", (job_id,), + ) + run = cursor.fetchone() + if run is None: + raise IngestionError("DAILY_JOB_NOT_FOUND", "日报记录不存在") + run_id, stored_date, run_status = int(run[0]), run[1], str(run[2]) + if stored_date != business_date: + raise IngestionError("DAILY_DATE_CONFLICT", "报表日期已变化,请刷新后重试") + cursor.execute( + """SELECT current_removed FROM ingestion.daily_run_retirements + WHERE processing_run_id = %s""", (run_id,), + ) + retired = cursor.fetchone() + if retired is not None: + return self._retirement_receipt(job_id, business_date, bool(retired[0])) + if run_status not in {"accepted", "rejected", "failed", "cancelled", "awaiting_review"}: + raise IngestionError("DAILY_JOB_BUSY", "任务正在处理,请完成后再删除") + cursor.execute( + "SELECT pg_advisory_xact_lock(hashtextextended(%s, 0))", + (f"finance-daily:{business_date.isoformat()}",), + ) + cursor.execute( + """SELECT id, version_status FROM finance.daily_versions + WHERE processing_run_id = %s FOR UPDATE""", (run_id,), + ) + versions = cursor.fetchall() + if len(versions) > 1: + raise IngestionError("DATABASE_STATE_INVALID", "日报版本关联无效") + version_id = int(versions[0][0]) if versions else None + cursor.execute( + """SELECT daily_version_id FROM finance.current_daily_versions + WHERE business_date = %s FOR UPDATE""", (business_date,), + ) + current = cursor.fetchone() + current_removed = version_id is not None and current is not None and int(current[0]) == version_id + if run_status == "awaiting_review": + cursor.execute( + """SELECT case_key, revision, case_status FROM ingestion.daily_review_cases + WHERE processing_run_id = %s FOR UPDATE""", (run_id,), + ) + case = cursor.fetchone() + if case is None or str(case[2]) not in {"open", "generation_failed"}: + raise IngestionError("DAILY_JOB_BUSY", "任务正在生成日报,请完成后再删除") + self._cancel_price_review(cursor, job_id, str(case[0]), int(case[1]), actor) + if current_removed: + period_start = business_date.replace(day=1) + period_end = (period_start.replace(day=28) + timedelta(days=4)).replace(day=1) - timedelta(days=1) + cursor.execute( + "SELECT pg_advisory_xact_lock(hashtextextended(%s, 0))", + (f"monthly_channel:{period_start.isoformat()}:{period_end.isoformat()}:*",), + ) + cursor.execute( + """DELETE FROM finance.current_daily_versions + WHERE business_date = %s AND daily_version_id = %s""", (business_date, version_id), + ) + if cursor.rowcount != 1: + raise IngestionError("DAILY_DATE_CONFLICT", "该日有效报表已变化,请刷新后重试") + cursor.execute( + """UPDATE finance.daily_versions SET version_status = 'superseded', superseded_at = now() + WHERE id = %s AND version_status = 'active'""", (version_id,), + ) + if cursor.rowcount != 1: + raise IngestionError("DATABASE_STATE_INVALID", "该日有效报表状态无效") + # Withdraw downloadable snapshots immediately, before asynchronous rebuilding. + cursor.execute( + """UPDATE reporting.monthly_runs + SET report_status = 'withdrawn', withdrawn_at = now(), updated_at = now() + WHERE period_start = %s AND report_status IN ('reserved', 'active', 'superseded', 'failed')""", + (period_start,), + ) + self._insert_outbox( + cursor, f"daily-version:{version_id}:retired", "daily_version", version_id, + "arr.daily_scope_changed", + {"daily_version_id": version_id, "business_date": business_date.isoformat(), + "period_start": period_start.isoformat(), "action": "delete"}, + ) + cursor.execute( + """INSERT INTO ingestion.daily_run_retirements + (processing_run_id, business_date, daily_version_id, current_removed, actor_username) + VALUES (%s, %s, %s, %s, %s)""", + (run_id, business_date, version_id, current_removed, actor), + ) + return self._retirement_receipt(job_id, business_date, current_removed) + def _run_transaction( self, operation: Callable[[Any], Any], @@ -401,6 +514,44 @@ class PostgresIngestionRepository(IngestionRepository): "ARR processing requires migration 021") self._run_transaction(check, "ARR PM schema could not be checked") + def assert_daily_lifecycle_schema(self) -> None: + """Do not expose date history/deletion with an incomplete 022 upgrade.""" + def check(cursor): + cursor.execute("""SELECT + to_regclass('ingestion.daily_run_retirements') IS NOT NULL, + (SELECT count(*) = 7 FROM information_schema.columns + WHERE table_schema = 'ingestion' AND table_name = 'daily_run_retirements' + AND column_name IN ('id','processing_run_id','business_date','daily_version_id', + 'current_removed','actor_username','retired_at')), + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema = 'reporting' AND table_name = 'monthly_runs' + AND column_name = 'withdrawn_at' AND data_type = 'timestamp with time zone'), + EXISTS (SELECT 1 FROM pg_constraint + WHERE conrelid = 'reporting.monthly_runs'::regclass + AND conname = 'monthly_runs_status_shape' + AND pg_get_constraintdef(oid) LIKE '%withdrawn_at%'), + EXISTS (SELECT 1 FROM pg_index AS index_row + JOIN pg_class AS index_name ON index_name.oid = index_row.indexrelid + WHERE index_row.indrelid = 'reporting.monthly_runs'::regclass + AND index_name.relname = 'monthly_runs_live_snapshot_unique' + AND index_row.indisunique AND index_row.indisvalid + AND pg_get_indexdef(index_row.indexrelid) LIKE '%period_start, source_snapshot_sha256%' + AND pg_get_expr(index_row.indpred,index_row.indrelid) LIKE '%report_status <>%withdrawn%'), + EXISTS (SELECT 1 FROM pg_trigger + WHERE tgrelid = to_regclass('ingestion.daily_run_retirements') + AND tgname = 'daily_run_retirements_immutable' AND tgenabled = 'O'), + EXISTS (SELECT 1 FROM pg_proc + WHERE oid = to_regprocedure('finance.validate_current_daily_version()') + AND prosrc LIKE '%daily_run_retirements%'), + EXISTS (SELECT 1 FROM pg_proc + WHERE oid = to_regprocedure('reporting.protect_published_monthly_child()') + AND prosrc LIKE '%withdrawn%')""") + row = cursor.fetchone() + if not row or len(row) != 8 or any(value is not True for value in row): + raise IngestionError("DATABASE_MIGRATION_MISSING", + "ARR daily report lifecycle requires migration 022") + self._run_transaction(check, "ARR daily lifecycle schema could not be checked") + def register_job(self, registration: JobRegistration) -> None: if ( registration.source.role not in {"source_xml", "source_data"} @@ -1280,6 +1431,17 @@ class PostgresIngestionRepository(IngestionRepository): "processing job identity conflicts", ) processing_run_id = int(existing[0]) + # Pre-022 ingestion remains compatible; once retirement is enabled, + # an old run identity can never be reused to undo a user's deletion. + cursor.execute("SELECT to_regclass('ingestion.daily_run_retirements')") + retirement_table = cursor.fetchone() + if retirement_table is not None and retirement_table[0] is not None: + cursor.execute( + "SELECT 1 FROM ingestion.daily_run_retirements WHERE processing_run_id = %s", + (processing_run_id,), + ) + if cursor.fetchone() is not None: + raise IngestionError("JOB_RETIRED", "该报表记录已删除,请重新提交新任务") cursor.execute( """ SELECT idempotency_key @@ -2275,6 +2437,14 @@ class PostgresIngestionRepository(IngestionRepository): business_date: date, daily_version_id: int, ) -> None: + # Publication and retirement share this month lock. A successful daily + # replacement cannot slip between a monthly source check and activation. + period_start = business_date.replace(day=1) + period_end = (period_start.replace(day=28) + timedelta(days=4)).replace(day=1) - timedelta(days=1) + cursor.execute( + "SELECT pg_advisory_xact_lock(hashtextextended(%s, 0))", + (f"monthly_channel:{period_start.isoformat()}:{period_end.isoformat()}:*",), + ) cursor.execute( """ SELECT daily_version_id diff --git a/arr_web/DIRECT_DATA_ENTRY.md b/arr_web/DIRECT_DATA_ENTRY.md index 0419a9a..d17e109 100644 --- a/arr_web/DIRECT_DATA_ENTRY.md +++ b/arr_web/DIRECT_DATA_ENTRY.md @@ -29,6 +29,21 @@ 完整备注、套餐明细和查询出处保留在原始数据中。Trace 不获取,旧报表对应列留空。 这是一套明确的新入口显示规则,不声明与 Oracle 原生报表的文字格式完全一致。 +## 日报覆盖、删除与日期概览(2026-10-09) + +日报列表按业务日期展示,一天只展示一行。接口取数和XML上传遵循相同的覆盖规则:新版本成功生成并入库后,才替换该日有效日报并更新月报;新版本尚待核对或失败时,原有效日报仍可下载并继续计入月报。 + +点击列表中的日期,可查看该日概览;上方日期、房数和耗时始终对应当前查看日期。待处理新版本会单独提示,尚无有效日报时显示空值,不借用其他日期的成功结果。 + +“历史记录”保留同一天的旧版本和未完成版本。删除前页面会说明影响: + +- 删除当前有效日报:该日从月报中移除,系统自动生成剩余日期的月报;整月已无有效日报时,撤下该月报。旧日报不会自动恢复。 +- 删除历史版本或待价格核对版本:保留当前有效日报,月报不变。运行中的版本须等任务停止后再删除。 + +删除会保留原始数据及操作记录,已删除记录不能继续下载或处理;如需重新纳入,可重新上传或提交新的取数请求。此功能不提供撤销删除按钮。 + +页面使用`GET /api/daily-reports`按日期分页、`GET /api/daily-reports/{date}`读取所选日期、`GET /api/daily-reports/{date}/history`读取历史;删除使用`DELETE /api/jobs/{job_id}`,正文为`{"business_date":"YYYY-MM-DD"}`,沿用登录及CSRF保护,并记录当前操作者。原`GET /api/jobs`仍供任务记录查询使用。 + ## 字段完善的保存与继续处理 `DataFieldReviews`只接受`collection_complete=true`的完整采集,以当前冻结处理器的费率白名单和字段验证识别阻塞项。 diff --git a/arr_web/README.md b/arr_web/README.md index 65ce35e..c8b3a7c 100644 --- a/arr_web/README.md +++ b/arr_web/README.md @@ -1,5 +1,7 @@ # ARR 2.0 Web +2026-10-09:日报按日期展示一个有效版本,支持查看历史、删除及月报同步;概览跟随所选日期。发布需要数据库022及配套月报worker,详见[交付说明](../deploy/OHIP_RELEASE_HANDOVER.md#日报版本管理升级2026-10-09)。 + 2026-09-18:[直接数据入口](DIRECT_DATA_ENTRY.md)已接通日期按钮、日报、复核和月报;独立本机模拟8875,未查询实际Oracle。 2026-09-17:[本机接口模拟全流程及按钮联调](LOCAL_API_SIMULATION.md)已在独立8874验证;[原文件重放](LOCAL_XML_REPLAY.md)8873保留。两个入口均明确未连接Oracle,不改变默认运行配置。 @@ -47,6 +49,8 @@ Web 层提供固定页面和审计友好的 API。生产 XML 写路径为: - `GET /healthz`:仅供容器读取的无详情 readiness 探针 - `POST /api/jobs` - `GET /api/jobs` +- `GET /api/daily-reports`、`GET /api/daily-reports/{date}`、`GET /api/daily-reports/{date}/history`:按日期查询有效日报与历史 +- `DELETE /api/jobs/{job_id}`:删除指定日期的日报版本,当前有效版本删除后自动更新月报;需要登录和CSRF - `GET /api/jobs/{job_id}/trace` - `GET /api/download/daily` - `GET /api/monthly-runs?month=YYYY-MM`:读取已持久化的月报版本 diff --git a/arr_web/app.py b/arr_web/app.py index b4e3964..9cf8ec4 100644 --- a/arr_web/app.py +++ b/arr_web/app.py @@ -39,6 +39,7 @@ from arr_web.repository import ( UnavailablePortalRepository, ) from arr_web.downloads import ArtifactReader, UnavailableArtifactReader +from arr_web.daily_reports import UnavailableDailyReports from arr_web.arr_downloads import ( ARRDownloadCoordinator, UnavailableARRDownloads, default_report_date, validate_report_date, validate_request_id, @@ -254,6 +255,7 @@ class PortalApplication: login_attempts: Optional[LoginAttemptLedger] = None, secure_cookies: bool = False, arr_downloads: Optional[ARRDownloadCoordinator] = None, + daily_reports=None, ) -> None: self._repository = repository or UnavailablePortalRepository() self._upload = upload or UnavailableUploadCoordinator() @@ -273,6 +275,7 @@ class PortalApplication: self._login_attempts = login_attempts or LoginAttemptLedger() self._secure_cookies = secure_cookies self._arr_downloads = arr_downloads or UnavailableARRDownloads() + self._daily_reports = daily_reports or UnavailableDailyReports() def handle( self, @@ -425,7 +428,35 @@ class PortalApplication: offset=offset, ), ) + if method == "GET" and route.path == "/api/daily-reports": + query = parse_qs(route.query, keep_blank_values=True) + month = validate_month(self._one(query, "month", self._current_month())) + limit = self._integer(query, "limit", 50, 1, 200) + offset = self._integer(query, "offset", 0, 0, 10_000_000) + reports, total = self._repository.list_daily_reports(month, limit, offset) + return Response.json(200, _paged_success(reports, total=total, limit=limit, offset=offset)) + if method == "GET" and route.path.startswith("/api/daily-reports/"): + parts = route.path[len("/api/daily-reports/"):].split("/") + day = validate_report_date(parts[0]) + if len(parts) == 1: + return Response.json(200, success(self._repository.get_daily_report(day))) + if len(parts) == 2 and parts[1] == "history": + query = parse_qs(route.query, keep_blank_values=True) + limit = self._integer(query, "limit", 50, 1, 200) + offset = self._integer(query, "offset", 0, 0, 10_000_000) + jobs, total = self._repository.list_daily_report_history(day, limit, offset) + return Response.json(200, _paged_success(jobs, total=total, limit=limit, offset=offset)) job_parts = route.path.strip("/").split("/") + if method == "DELETE" and len(job_parts) == 3 and job_parts[:2] == ["api", "jobs"]: + self._require_csrf(normalized_headers) + if len(body) > 4096: + raise PortalError("REQUEST_TOO_LARGE", "请求内容过大", 413) + payload = _strict_json(body) + if set(payload) != {"business_date"}: + raise PortalError("DAILY_DELETE_REQUEST_INVALID", "请确认要删除的日报日期") + day = validate_report_date(payload["business_date"]) + job_id = validate_job_id(job_parts[2]) + return Response.json(200, success(self._daily_reports.retire(job_id, day, current_session[0]))) if ( method == "GET" and len(job_parts) == 4 @@ -887,11 +918,13 @@ class PortalApplication: ) def _arr_download_snapshot(self, task: Optional[dict]) -> Optional[dict]: - if task and task.get("status") == "needs_review" and task.get("job_id"): + if task and task.get("status") in {"needs_review", "succeeded"} and task.get("job_id"): try: status = self._repository.get_job_trace(task["job_id"])["job"]["status"] except (PortalDataError, KeyError): return task + if status == "deleted": + return {**task, "status": "deleted", "is_retired": True, "can_retry": False} if status in {"accepted", "succeeded", "failed", "rejected", "cancelled"}: return {**task, "status": "succeeded" if status in {"accepted", "succeeded"} else "failed", "can_retry": False} return task diff --git a/arr_web/daily_reports.py b/arr_web/daily_reports.py new file mode 100644 index 0000000..555fb40 --- /dev/null +++ b/arr_web/daily_reports.py @@ -0,0 +1,27 @@ +"""Authenticated daily-report retirement, separate from read-only portal access.""" +from dataclasses import dataclass +from datetime import date + +from arr_ingestion.contracts import IngestionError +from arr_web.contracts import PortalError + + +class UnavailableDailyReports: + def retire(self, job_id, business_date, actor): + raise PortalError("DAILY_RETIREMENT_UNAVAILABLE", "日报删除服务暂不可用", 503) + + +@dataclass +class ProgramDailyReports: + repository: object + + def retire(self, job_id, business_date, actor): + try: + result = self.repository.retire_daily_job(job_id, date.fromisoformat(business_date), actor) + except IngestionError as error: + status = 404 if error.code == "DAILY_JOB_NOT_FOUND" else ( + 409 if error.code in {"DAILY_DATE_CONFLICT", "DAILY_JOB_BUSY", "REVIEW_STATE_INVALID", + "JOB_RETIRED", "DAILY_RETIREMENT_CONFLICT"} else 503) + raise PortalError(error.code, "日报记录已变化,请刷新后重试" if status == 409 else + "日报删除未完成,请稍后重试", status) from None + return {**result, "status": "deleted", "removed_from_month": result["current_removed"]} diff --git a/arr_web/local_ohip.py b/arr_web/local_ohip.py index a475993..c778def 100644 --- a/arr_web/local_ohip.py +++ b/arr_web/local_ohip.py @@ -224,7 +224,7 @@ def open_portal(root, port): load_key(credential_file) access = AccessState(root) with job_lock(root): - database = ReplayDatabase(root, schema_version=21) + database = ReplayDatabase(root, schema_version=22) try: database.start() def factory(*, root, snapshot, **dependencies): diff --git a/arr_web/local_replay.py b/arr_web/local_replay.py index ca90fef..ff3b657 100644 --- a/arr_web/local_replay.py +++ b/arr_web/local_replay.py @@ -22,6 +22,7 @@ from arr_web.app import PortalApplication, RuntimeHealth from arr_web.arr_downloads import PersistentARRDownloads from arr_web.auth import LoginCredentials from arr_web.downloads import ManagedObjectArtifactReader +from arr_web.daily_reports import ProgramDailyReports from arr_web.local_replay_database import ReplayDatabase from arr_web.local_xml_replay import (PROJECT, LocalReplayPortal, NativeXMLReplayExecutor, NativeXMLSnapshot, SourceDateDownloads, document) @@ -81,6 +82,7 @@ class ReplayRuntime: monthly_repository, service) self.portal_repository = PostgresPortalRepository("local-replay", connect=database.connect) app = PortalApplication(repository=self.portal_repository, price_reviews=review, + daily_reports=ProgramDailyReports(self.repository), upload=review if enable_upload else None, artifact_reader=ManagedObjectArtifactReader(self.store), credentials=LoginCredentials(**document(root / "login.json")), @@ -132,7 +134,7 @@ def open_instance(root: Path, port: int, *, executor_factory=None, portal_type=L "processor_version": policy.processor_version, "rule_set_sha256": policy.rule_set_sha256} if fingerprint(document(root / "instance.json")) != fingerprint(expected): raise ValueError("replay_instance_identity_or_policy_changed") - database = ReplayDatabase(root) + database = ReplayDatabase(root, schema_version=22) try: database.start() runtime = ReplayRuntime(root, database, snapshot, policy, port, diff --git a/arr_web/local_replay_database.py b/arr_web/local_replay_database.py index 68e262e..c122fa1 100644 --- a/arr_web/local_replay_database.py +++ b/arr_web/local_replay_database.py @@ -15,7 +15,7 @@ from integrations.ohip.capture_job import atomic_json, private_directory class ReplayDatabase: def __init__(self, root: Path, *, schema_version: int = 21): - if schema_version not in (18, 19, 20, 21): + if schema_version not in (18, 19, 20, 21, 22): raise ValueError("unsupported_local_schema") self.schema_version = schema_version self.root = root diff --git a/arr_web/local_xml_replay.py b/arr_web/local_xml_replay.py index 2987d3a..d49906d 100644 --- a/arr_web/local_xml_replay.py +++ b/arr_web/local_xml_replay.py @@ -178,7 +178,8 @@ class LocalReplayPortal: allowed = (self.allow_xml_upload and path == "/api/jobs") or path in {"/api/login", "/api/logout", "/api/arr-downloads"} or bool( re.fullmatch(r"/api/arr-downloads/[a-f0-9]{32}/retry", path)) or bool( re.fullmatch(r"/api/arr-downloads/[a-f0-9]{32}/data-review/(?:items/[^/]+|finalize)", path)) or bool( - re.fullmatch(r"/api/jobs/[^/]+/review/(?:items/[0-9]+|finalize|cancel)", path)) + re.fullmatch(r"/api/jobs/[^/]+/review/(?:items/[0-9]+|finalize|cancel)", path)) or bool( + method == "DELETE" and re.fullmatch(r"/api/jobs/[^/]+", path)) if method != "GET" and not allowed: return self._error("REPLAY_ROUTE_DISABLED", 405) cookie = SimpleCookie() diff --git a/arr_web/preview.py b/arr_web/preview.py index a8e87b4..5df758d 100644 --- a/arr_web/preview.py +++ b/arr_web/preview.py @@ -14,6 +14,15 @@ from arr_web.server import serve class EmptyPreviewRepository(UnavailablePortalRepository): + def list_daily_reports(self, month_key, limit=50, offset=0): + return [], 0 + + def get_daily_report(self, business_date): + return {"business_date": business_date, "current_job": None, "latest_job": None, "history_count": 0} + + def list_daily_report_history(self, business_date, limit=50, offset=0): + return [], 0 + def list_jobs(self, month_key, limit=50, offset=0): return [], 0 diff --git a/arr_web/processing_runtime.py b/arr_web/processing_runtime.py index b8f4928..d095592 100644 --- a/arr_web/processing_runtime.py +++ b/arr_web/processing_runtime.py @@ -81,6 +81,7 @@ def compose_programmatic_processing( ) repository.assert_cancelled_scope_schema() repository.assert_pm_scope_schema() + repository.assert_daily_lifecycle_schema() policy = load_processor_policy(project_root) ingestion = IngestionService(DeliveryValidator(object_store, policy), repository) processor = LocalDailyProcessor(policy) diff --git a/arr_web/repository.py b/arr_web/repository.py index d7a441e..85897f9 100644 --- a/arr_web/repository.py +++ b/arr_web/repository.py @@ -27,6 +27,15 @@ class PortalDataError(RuntimeError): class PortalRepository(Protocol): + def list_daily_reports(self, month_key: str, limit: int = 50, offset: int = 0): + ... + + def get_daily_report(self, business_date: str): + ... + + def list_daily_report_history(self, business_date: str, limit: int = 50, offset: int = 0): + ... + def list_jobs( self, month_key: str, @@ -78,6 +87,15 @@ class UnavailablePortalRepository: def _raise() -> None: raise PortalDataError("DATABASE_UNAVAILABLE", "数据库读取服务暂不可用") + def list_daily_reports(self, month_key, limit=50, offset=0): + self._raise() + + def get_daily_report(self, business_date): + self._raise() + + def list_daily_report_history(self, business_date, limit=50, offset=0): + self._raise() + def list_jobs( self, month_key: str, @@ -225,6 +243,61 @@ WHERE run.pipeline_type = 'opera_daily' ) """.strip() +# Date-level views deliberately leave the original task-log API intact. +DAILY_REPORT_JOBS_SQL = """ +SELECT run.run_key, + CASE run.run_status WHEN 'accepted' THEN 'succeeded' + WHEN 'awaiting_review' THEN 'needs_review' WHEN 'rejected' THEN 'failed' + ELSE run.run_status END, + run.business_date, run.failure_code, run.created_at, run.updated_at, + run.finished_at, run.uploaded_filename, source.byte_size, + version.retained_rows, version.version_no, daily.sha256, + coalesce(rooms.total, 0), review.case_key, review.case_status, + review.revision, coalesce(review.completed, 0), coalesce(review.total, 0), + version.id, current_version.daily_version_id IS NOT NULL, + retirement.id IS NOT NULL +FROM ingestion.processing_runs AS run +JOIN ingestion.artifacts AS source ON source.id = run.source_artifact_id +LEFT JOIN finance.daily_versions AS version ON version.processing_run_id = run.id +LEFT JOIN finance.current_daily_versions AS current_version + ON current_version.daily_version_id = version.id +LEFT JOIN ingestion.daily_run_retirements AS retirement ON retirement.processing_run_id = run.id +LEFT JOIN ingestion.artifacts AS daily ON daily.id = version.daily_report_artifact_id +LEFT JOIN LATERAL ( + SELECT sum(record.no_of_rooms) AS total FROM finance.daily_records AS record + WHERE record.daily_version_id = version.id AND record.outcome = 'retained' +) AS rooms ON true +LEFT JOIN LATERAL ( + SELECT case_row.case_key, case_row.case_status, case_row.revision, + count(item.id) AS total, + count(item.id) FILTER (WHERE item.real_price IS NOT NULL) AS completed + FROM ingestion.daily_review_cases AS case_row + LEFT JOIN ingestion.daily_review_items AS item ON item.review_case_id = case_row.id + WHERE case_row.processing_run_id = run.id + GROUP BY case_row.case_key, case_row.case_status, case_row.revision +) AS review ON true +WHERE run.pipeline_type = 'opera_daily' AND run.business_date = ANY(%s) +ORDER BY run.created_at DESC, run.id DESC +""".strip() + +DAILY_REPORT_DATES_SQL = """ +SELECT DISTINCT run.business_date +FROM ingestion.processing_runs AS run +LEFT JOIN ingestion.daily_run_retirements AS retirement ON retirement.processing_run_id = run.id +WHERE run.pipeline_type = 'opera_daily' AND run.business_date >= %s AND run.business_date < %s + AND retirement.id IS NULL +ORDER BY run.business_date DESC +LIMIT %s OFFSET %s +""".strip() + +DAILY_REPORT_COUNT_SQL = """ +SELECT count(DISTINCT run.business_date) +FROM ingestion.processing_runs AS run +LEFT JOIN ingestion.daily_run_retirements AS retirement ON retirement.processing_run_id = run.id +WHERE run.pipeline_type = 'opera_daily' AND run.business_date >= %s AND run.business_date < %s + AND retirement.id IS NULL +""".strip() + JOB_TRACE_RUN_SQL = """ SELECT @@ -439,7 +512,7 @@ SELECT FROM reporting.monthly_runs AS run LEFT JOIN ingestion.artifacts AS artifact ON artifact.id = run.workbook_artifact_id -WHERE run.period_start = %s +WHERE run.period_start = %s AND run.report_status <> 'withdrawn' ORDER BY run.version_no DESC, run.id DESC LIMIT %s OFFSET %s """.strip() @@ -448,7 +521,7 @@ LIMIT %s OFFSET %s MONTHLY_RUNS_COUNT_SQL = """ SELECT count(*) FROM reporting.monthly_runs AS run -WHERE run.period_start = %s +WHERE run.period_start = %s AND run.report_status <> 'withdrawn' """.strip() @@ -471,6 +544,7 @@ WITH daily AS ( monthly AS ( SELECT run.period_start, count(*) AS record_count FROM reporting.monthly_runs AS run + WHERE run.report_status <> 'withdrawn' GROUP BY run.period_start ) SELECT @@ -594,8 +668,17 @@ class PostgresPortalRepository: (start, end, start, end, limit, offset), ) rows = list(cursor.fetchall()) - return [ - { + return [self._job_from_row(row) for row in rows], total + except PortalDataError: + raise + except Exception: + raise PortalDataError("DATABASE_QUERY_FAILED", "日报记录查询失败") from None + finally: + connection.close() + + @staticmethod + def _job_from_row(row): + return { "job_id": str(row[0]), "status": str(row[1]), "arrival_date": _iso(row[2]), @@ -615,15 +698,97 @@ class PostgresPortalRepository: "review_completed_items": int(row[16] or 0), "review_total_items": int(row[17] or 0), } - for row in rows - ], total + + @classmethod + def _report_job(cls, row): + job = cls._job_from_row(row) + retired = bool(row[20]) + job.update(daily_version_id=int(row[18]) if row[18] is not None else None, + is_current=bool(row[19]) and not retired, is_retired=retired, + can_delete=not retired and job["status"] in {"succeeded", "failed", "cancelled", "needs_review"} + and job["review_status"] not in {"generating", "generation_processing"}) + if retired: + job["status"] = "deleted" + job["daily_report_sha256"] = None + return job + + @staticmethod + def _date_group(day, jobs): + active = [job for job in jobs if not job["is_retired"]] + return {"business_date": day.isoformat(), + "current_job": next((job for job in active if job["is_current"]), None), + "latest_job": active[0] if active else None, + "history_count": max(0, len(jobs) - 1)} + + def list_daily_reports(self, month_key, limit=50, offset=0): + start, end = self._month_bounds(month_key) + connection = self._open() + try: + with connection.transaction(): + with connection.cursor() as cursor: + self._begin(cursor) + cursor.execute(DAILY_REPORT_COUNT_SQL, (start, end)) + total = int(cursor.fetchone()[0]) + cursor.execute(DAILY_REPORT_DATES_SQL, (start, end, limit, offset)) + days = [row[0] for row in cursor.fetchall()] + if not days: + return [], total + cursor.execute(DAILY_REPORT_JOBS_SQL, (days,)) + jobs = [self._report_job(row) for row in cursor.fetchall()] + return [self._date_group(day, [job for job in jobs if job["arrival_date"] == day.isoformat()]) + for day in days], total except PortalDataError: raise except Exception: - raise PortalDataError("DATABASE_QUERY_FAILED", "日报记录查询失败") from None + raise PortalDataError("DATABASE_QUERY_FAILED", "日报日期查询失败") from None finally: connection.close() + def get_daily_report(self, business_date): + day = date.fromisoformat(business_date) + connection = self._open() + try: + with connection.transaction(): + with connection.cursor() as cursor: + self._begin(cursor) + cursor.execute(DAILY_REPORT_JOBS_SQL, ([day],)) + jobs = [self._report_job(row) for row in cursor.fetchall()] + return self._date_group(day, jobs) + except PortalDataError: + raise + except Exception: + raise PortalDataError("DATABASE_QUERY_FAILED", "日报日期查询失败") from None + finally: + connection.close() + + def list_daily_report_history(self, business_date, limit=50, offset=0): + day = date.fromisoformat(business_date) + connection = self._open() + try: + with connection.transaction(): + with connection.cursor() as cursor: + self._begin(cursor) + cursor.execute(DAILY_REPORT_JOBS_SQL, ([day],)) + jobs = [self._report_job(row) for row in cursor.fetchall()] + return jobs[offset:offset + limit], len(jobs) + except PortalDataError: + raise + except Exception: + raise PortalDataError("DATABASE_QUERY_FAILED", "日报历史查询失败") from None + finally: + connection.close() + + @staticmethod + def _is_retired(cursor, job_id): + # Old, explicitly supported read-only deployments may predate migration022. + cursor.execute("SELECT to_regclass('ingestion.daily_run_retirements')") + if cursor.fetchone()[0] is None: + return False + cursor.execute("""SELECT 1 FROM ingestion.daily_run_retirements AS retirement + JOIN ingestion.processing_runs AS run ON run.id = retirement.processing_run_id + WHERE run.run_key = %s""", (job_id,)) + return cursor.fetchone() is not None + def get_job_trace(self, job_id: str) -> Dict[str, Any]: connection = self._open() try: @@ -664,6 +829,7 @@ class PostgresPortalRepository: ), "result_schema_version": str(row[19]) if row[19] else None, } + retired = self._is_retired(cursor, job_id) run_id = int(row[0]) cursor.execute(JOB_TRACE_ATTEMPTS_SQL, (run_id,)) @@ -815,7 +981,7 @@ class PostgresPortalRepository: } for item in cursor.fetchall() ] - return build_job_trace( + trace = build_job_trace( run, attempts=attempts, deliveries=deliveries, @@ -825,6 +991,9 @@ class PostgresPortalRepository: review_case=review_case, review_events=review_events, ) + if retired: + trace["job"].update(status="deleted", is_retired=True, active=False) + return trace except PortalDataError: raise except Exception: @@ -940,6 +1109,8 @@ class PostgresPortalRepository: with connection.transaction(): with connection.cursor() as cursor: self._begin(cursor) + if statement == DAILY_DOWNLOAD_SQL and self._is_retired(cursor, str(value)): + raise PortalDataError("DOWNLOAD_NOT_FOUND", "该日报已删除") cursor.execute(statement, (value,)) rows = list(cursor.fetchall()) if len(rows) != 1: diff --git a/arr_web/run.py b/arr_web/run.py index 8b4afd7..afcbfe5 100644 --- a/arr_web/run.py +++ b/arr_web/run.py @@ -32,6 +32,7 @@ from arr_web.processing_runtime import ( ) from arr_web.server import serve from arr_web.services import ProgramMonthlyCoordinator +from arr_web.daily_reports import ProgramDailyReports from monthly_reports.publishing import OpenpyxlWorkbookBuilder, AtomicReportPublisher from monthly_reports.repository import DatabaseConfig, PostgresReportRepository from monthly_reports.service import MonthlyReportService @@ -257,6 +258,7 @@ def _run(args, login_credentials, resources: ExitStack, *, arr_source: CapturedA resources.callback(arr_downloads.close, wait=True) application = PortalApplication( repository=repository, + daily_reports=ProgramDailyReports(processing_input.repository) if processing_input is not None else None, arr_downloads=arr_downloads, upload=( processing_input.coordinator if processing_input is not None else None diff --git a/arr_web/static/app.js b/arr_web/static/app.js index 951d6e3..d41e3d0 100644 --- a/arr_web/static/app.js +++ b/arr_web/static/app.js @@ -54,6 +54,18 @@ biPollLoading: false, toastTimer: null, jobs: [], + dailyReports: [], + dailyOverviewDate: "", + dailyOverview: null, + dailyOverviewTimer: null, + dailyOverviewRequest: 0, + dailyHistoryDate: "", + dailyHistoryItems: [], + dailyHistoryTotal: 0, + dailyHistoryOffset: 0, + dailyHistoryLoading: false, + dailyDeleteRequest: null, + dailyDeleting: false, selectedJobId: "", jobTrace: null, jobTraceLoading: false, @@ -119,6 +131,7 @@ rejected: ["status.validation_failed", "failed"], failed: ["status.failed", "failed"], cancelled: ["status.cancelled", "failed"], + deleted: ["daily.deleted", ""], }; const reportStatus = { reserved: ["status.generating", "running"], @@ -522,16 +535,18 @@ feedback.className = `arr-download-feedback${arrDownloadActive() ? " is-running" : status === "succeeded" ? " is-success" : ["needs_review", "needs_data_review"].includes(status) ? " is-review" : ["failed", "interrupted", "reconnecting", "service_reconnecting", "date_unavailable"].includes(status) || rejected ? " is-error" : ""}`; const taskError = !task?.job_id && status === "failed" && task.error_code === "ARR_SOURCE_FETCH_FAILED" ? "source_failed" : !task?.job_id && status === "interrupted" && task.error_code === "ARR_DOWNLOAD_INTERRUPTED" ? "source_interrupted" : status; - const text = rejected + const viewingAnotherDate = task && task.report_date !== dateInput.value && !arrDownloadActive() && !reconciling; + const text = viewingAnotherDate ? I18N.t("arr_download.ready") : rejected ? I18N.t("arr_download.submission_rejected", { reason: task.submission_error }) : I18N.t(`arr_download.${taskError}`); - const taskDate = task?.report_date || state.arrDownloadIntent?.report_date; + const taskDate = viewingAnotherDate ? dateInput.value : task?.report_date || state.arrDownloadIntent?.report_date; $("#arr-download-status").textContent = taskDate ? `${taskDate} · ${text}` : text; - $("#arr-download-review").hidden = task?.status !== "needs_review" || !task.job_id; - $("#arr-download-data-review").hidden = !arrDownloadNeedsDataReview(); + $("#arr-download-review").hidden = viewingAnotherDate || task?.status !== "needs_review" || !task.job_id; + $("#arr-download-data-review").hidden = !selectedARRDateNeedsDataReview(); $("#arr-download-data-review").disabled = locked; renderARRAcquisitionProgress(task); renderARRPendingReviews(); + syncDailyOverviewDate(dateInput.value); } function clearARRDownloadPoll() { @@ -560,7 +575,12 @@ state.arrDownloadTask = task; rememberARRPendingReview(task, { allowAdd: allowPendingAdd }); state.arrDownloadDisconnected = false; - rememberARRIntent({ request_id: task.request_id, report_date: task.report_date }); + rememberARRIntent(task.status === "deleted" ? null : { request_id: task.request_id, report_date: task.report_date }); + if (task.status === "deleted" && state.dailyPriceReviewJobId === task.job_id) { + state.dailyPriceReview = null; + state.dailyPriceReviewJobId = ""; + renderDailyPriceReview(null); + } if (!state.arrDownloadDateEdited) $("#arr-download-date").value = task.report_date; renderARRDownload(); if (arrDownloadNeedsDataReview()) { @@ -576,7 +596,7 @@ state.jobsOffset = 0; renderHistoryMonthControl("jobs"); await loadJobs(false, false); - await selectJob(task.job_id); + await selectJob(task.job_id, { syncDate: !state.arrDownloadDateEdited }); state.arrDownloadSynced = signature; if (task.status === "succeeded" && $("#panel-monthly").classList.contains("is-active")) await loadMonthly(false, true); } @@ -627,6 +647,7 @@ } const storageKey = `arr:last-download:v2:${encodeURIComponent(config.context_id)}:${encodeURIComponent(state.arrDownloadUsername)}`; if (state.arrDownloadStorageKey !== storageKey) { + const contextChanged = state.arrDownloadContextId && state.arrDownloadContextId !== config.context_id; state.arrDownloadStorageKey = storageKey; state.arrDownloadContextId = config.context_id; state.arrDownloadRestored = false; @@ -636,6 +657,19 @@ state.arrDownloadDisconnected = false; state.arrDownloadSynced = ""; state.arrDownloadPendingReviews = []; + state.dailyOverview = null; + state.dailyOverviewDate = ""; + state.dailyOverviewRequest += 1; + window.clearTimeout(state.dailyOverviewTimer); + if (contextChanged) { + state.dailyReports = []; + state.jobs = []; + state.dailyHistoryItems = []; + state.dailyHistoryTotal = 0; + state.dailyHistoryDate = ""; + renderJobs(); + renderDailyHistory(); + } state.manualReviewMode = ""; state.dailyPriceReview = null; state.dailyPriceReviewJobId = ""; @@ -1285,40 +1319,280 @@ updateCompanyReportControls(); } - function renderJobs(jobs) { - const body = $("#jobs-body"); - if (!jobs.length) { - body.innerHTML = `${historyEmptyMarkup("jobs", "history.daily_empty", "暂无 Daily Report 记录")}`; - $("#metric-arrival").textContent = "—"; - $("#metric-duration").textContent = "—"; - $("#metric-rooms").textContent = "—"; + function dailyJobSource(job) { + return job?.filename === "ARR.json" ? I18N.t("arr_download.source_label") : job?.filename || "—"; + } + + function dailyJobStatus(job) { + if (!job) return `${escapeHtml(I18N.t("daily.no_current"))}`; + const completed = formatInteger(job.review_completed_items || 0); + const total = formatInteger(job.review_total_items || 0); + return job.status === "needs_review" && job.review_case_id + ? `` + : chip(job.status, jobStatus); + } + + function dailyJobDownload(job) { + return state.health.download_ready && job?.status === "succeeded" && job.daily_report_sha256 && !job.is_retired + ? `${escapeHtml(I18N.t("common.download"))}` + : '—'; + } + + function dailyOverviewRow() { + return (state.dailyOverview?.business_date === state.dailyOverviewDate ? state.dailyOverview : null) + || state.dailyReports.find((day) => day.business_date === state.dailyOverviewDate); + } + + function renderDailyOverview() { + const date = state.dailyOverviewDate; + const day = dailyOverviewRow(); + const task = state.arrDownloadTask?.report_date === date ? state.arrDownloadTask : null; + const latest = day?.latest_job; + const current = day?.current_job; + const taskPending = task && ["queued", "downloading", "processing", "needs_data_review", "needs_review", "interrupted"].includes(task.status); + const replacement = latest && latest.job_id !== current?.job_id && !latest.is_retired; + const jobPending = replacement && !["succeeded", "accepted", "cancelled", "rejected", "failed"].includes(latest.status); + const failed = !taskPending && !jobPending && (task?.status === "failed" || (replacement && ["failed", "rejected"].includes(latest.status))); + const pending = Boolean(taskPending || jobPending); + const status = taskPending ? task.status : jobPending ? latest.status : ""; + const review = ["needs_data_review", "needs_review"].includes(status); + let hint = I18N.t("daily.no_current"); + if (pending) hint = I18N.t(review ? "daily.overview_review" : ["queued", "downloading"].includes(status) ? "daily.overview_fetching" : "daily.overview_processing"); + else if (current) hint = I18N.t("daily.current"); + else if (failed) hint = I18N.t("status.failed"); + $("#metric-arrival").textContent = validARRDate(date) ? formatDate(date) : "—"; + $("#metric-arrival-note").textContent = hint; + $("#metric-current-version").hidden = !current || (!pending && !failed); + $("#metric-current-version").textContent = I18N.t("daily.current_preserved"); + const showResult = Boolean(current && !pending); + $("#metric-duration").textContent = showResult ? duration(current) : "—"; + $("#metric-rooms").textContent = showResult ? formatInteger(current.no_of_rooms) : "—"; + $("#metric-duration-note").textContent = I18N.t(showResult ? "metric.duration_from_submit" : "daily.after_generation"); + $("#metric-rooms-note").textContent = I18N.t(showResult ? "metric.rooms" : "daily.after_generation"); + } + + function syncDailyOverviewDate(date) { + if (!validARRDate(date)) { + state.dailyOverviewDate = ""; + state.dailyOverview = null; + state.dailyOverviewRequest += 1; + window.clearTimeout(state.dailyOverviewTimer); + renderDailyOverview(); return; } - body.innerHTML = jobs.map((job) => { - const sourceLabel = job.filename === "ARR.json" - ? (I18N?.t("arr_download.source_label") || "接口获取") - : (job.filename || "—"); - const completedItems = formatInteger(job.review_completed_items || 0); - const totalItems = formatInteger(job.review_total_items || 0); - const statusMarkup = job.status === "needs_review" && job.review_case_id - ? `` - : chip(job.status, jobStatus); - const canDownload = state.health.download_ready && job.status === "succeeded" && job.daily_report_sha256; - return ` - - ${escapeHtml(sourceLabel)} - ${statusMarkup}${job.failure_code ? `${escapeHtml(job.failure_code)}` : ""} - ${escapeHtml(job.arrival_date ? formatDate(job.arrival_date) : "—")} - ${formatInteger(job.no_of_rooms)} - ${duration(job)} - ${formatDate(job.created_at, true)} - ${canDownload ? `${escapeHtml(I18N?.t("common.download") || "下载")}` : '—'} + if (state.dailyOverviewDate !== date) { + state.dailyOverviewDate = date; + state.dailyOverview = null; + state.dailyOverviewRequest += 1; + window.clearTimeout(state.dailyOverviewTimer); + if (!state.dailyReports.some((day) => day.business_date === date)) { + state.dailyOverviewTimer = window.setTimeout(() => loadDailyOverview(date), 250); + } + renderJobs(); + return; + } + renderDailyOverview(); + } + + async function loadDailyOverview(date = state.dailyOverviewDate) { + if (!validARRDate(date)) return; + const sequence = ++state.dailyOverviewRequest; + const context = state.arrDownloadContextId; + try { + const day = await api(`/api/daily-reports/${encodeURIComponent(date)}`); + if (state.dailyOverviewDate !== date || state.dailyOverviewRequest !== sequence || state.arrDownloadContextId !== context) return; + if (day?.business_date === date) state.dailyOverview = day; + renderDailyOverview(); + } catch (_) { /* Do not substitute a different date's result when this day cannot be read. */ } + } + + async function selectDailyReportDate(date) { + if (!validARRDate(date) || state.dailyDeleting || state.arrDownloadBusy || dailyPriceReviewHasUnsavedChanges()) { + if (dailyPriceReviewHasUnsavedChanges()) showToast(I18N.t("daily_review.save_before_switch"), true); + return; + } + $("#arr-download-date").value = date; + state.arrDownloadDateEdited = true; + renderARRDownload(); + if (!dailyOverviewRow()) await loadDailyOverview(date); + } + + function renderJobs() { + const body = $("#jobs-body"); + if (!state.dailyReports.length) { + body.innerHTML = `${historyEmptyMarkup("jobs", "history.daily_empty", "暂无 Daily Report 记录")}`; + renderDailyOverview(); + return; + } + body.innerHTML = state.dailyReports.map((day) => { + const current = day.current_job; + const latest = day.latest_job; + const replacement = latest && latest.job_id !== current?.job_id && !latest.is_retired ? latest : null; + const primary = current || replacement; + const historicalOnly = !current && latest && ["succeeded", "accepted"].includes(latest.status); + const date = day.business_date; + const deleteButton = primary?.can_delete + ? `` : ""; + const pendingMarkup = current && replacement + ? `
${escapeHtml(I18N.t("daily.replacement"))}${dailyJobStatus(replacement)}
` : ""; + return ` + ${escapeHtml(dailyJobSource(primary))}${current || historicalOnly ? `${escapeHtml(I18N.t(current ? "daily.current" : "daily.history_only"))}` : ""} + ${historicalOnly ? `${escapeHtml(I18N.t("daily.no_current"))}` : dailyJobStatus(primary)}${pendingMarkup} + + ${current ? formatInteger(current.no_of_rooms) : "—"} + ${current ? duration(current) : "—"} + ${primary?.created_at ? formatDate(primary.created_at, true) : "—"} +
${dailyJobDownload(current)}${deleteButton}
`; }).join(""); - const latest = jobs.find((job) => job.status === "succeeded"); - $("#metric-arrival").textContent = latest?.arrival_date ? formatDate(latest.arrival_date) : "—"; - $("#metric-duration").textContent = latest ? duration(latest) : "—"; - $("#metric-rooms").textContent = latest ? formatInteger(latest.no_of_rooms) : "—"; + renderDailyOverview(); + } + + function renderDailyHistory() { + $("#daily-history-description").textContent = I18N.t("daily.history_date", { date: state.dailyHistoryDate }); + const items = state.dailyHistoryItems; + $("#daily-history-body").innerHTML = items.length ? items.map((job) => { + const label = I18N.t(job.is_retired ? "daily.deleted" : job.is_current ? "daily.current" : "daily.historical"); + const remove = job.can_delete && !job.is_retired + ? `` : ""; + return `${escapeHtml(dailyJobSource(job))}${escapeHtml(label)}${job.is_retired ? `${escapeHtml(label)}` : dailyJobStatus(job)}${job.status === "succeeded" ? formatInteger(job.no_of_rooms) : "—"}${formatDate(job.created_at, true)}
${dailyJobDownload(job)}${remove}
`; + }).join("") : `${escapeHtml(I18N.t(state.dailyHistoryLoading ? "daily.loading" : "daily.history_empty"))}`; + const total = state.dailyHistoryTotal; + const offset = state.dailyHistoryOffset; + $("#daily-history-page-summary").textContent = I18N.t("pagination.summary", { total: formatInteger(total), start: formatInteger(total ? offset + 1 : 0), end: formatInteger(Math.min(total, offset + HISTORY_PAGE_SIZE)) }); + $("#daily-history-prev").disabled = state.dailyHistoryLoading || state.dailyDeleting || offset === 0; + $("#daily-history-next").disabled = state.dailyHistoryLoading || state.dailyDeleting || offset + HISTORY_PAGE_SIZE >= total; + $("#daily-history-close").disabled = state.dailyDeleting; + } + + async function loadDailyHistory(date = state.dailyHistoryDate) { + if (!validARRDate(date) || state.dailyHistoryLoading) return; + state.dailyHistoryDate = date; + state.dailyHistoryLoading = true; + renderDailyHistory(); + const context = state.arrDownloadContextId; + const offset = state.dailyHistoryOffset; + try { + const page = readPage(await api(`/api/daily-reports/${encodeURIComponent(date)}/history?limit=${HISTORY_PAGE_SIZE}&offset=${offset}`, { returnEnvelope: true })); + if (state.dailyHistoryDate !== date || state.arrDownloadContextId !== context) return; + state.dailyHistoryItems = page.items; + state.dailyHistoryTotal = page.total; + $("#daily-history-error").hidden = true; + } catch (error) { + if (state.dailyHistoryDate !== date || state.arrDownloadContextId !== context) return; + $("#daily-history-error").textContent = error.message; + $("#daily-history-error").hidden = false; + } finally { + state.dailyHistoryLoading = false; + renderDailyHistory(); + } + } + + async function openDailyHistory(date, trigger) { + if (!validARRDate(date) || state.dailyDeleting || state.dailyHistoryLoading) return; + state.dailyHistoryDate = date; + state.dailyHistoryItems = []; + state.dailyHistoryOffset = 0; + state.dailyHistoryTotal = 0; + $("#daily-history-error").hidden = true; + const dialog = $("#daily-history-dialog"); + dialog._returnFocus = trigger; + renderDailyHistory(); + if (!dialog.open) { + if (typeof dialog.showModal === "function") dialog.showModal(); + else dialog.setAttribute("open", ""); + } + $("#daily-history-close").focus(); + await loadDailyHistory(date); + } + + function closeDailyHistory() { + if (state.dailyDeleting) return; + const dialog = $("#daily-history-dialog"); + if (typeof dialog.close === "function") dialog.close(); + else dialog.removeAttribute("open"); + if (dialog._returnFocus?.isConnected) dialog._returnFocus.focus(); + } + + function openDailyDelete(button) { + if (dailyPriceReviewHasUnsavedChanges()) { + showToast(I18N.t("daily_review.save_before_switch"), true); + return; + } + if (state.dailyDeleting || state.dailyPriceReviewMutating || state.arrDataReviewMutating) return; + const jobId = button.dataset.dailyDelete; + const date = button.dataset.dailyDate; + const current = button.dataset.dailyCurrent === "true"; + if (!jobId || !validARRDate(date)) return; + state.dailyDeleteRequest = { jobId, date, current, trigger: button }; + $("#daily-delete-description").textContent = I18N.t(current ? "daily.delete_current_description" : "daily.delete_history_description", { date }); + $("#daily-delete-note").textContent = I18N.t("daily.delete_note"); + $("#daily-delete-error").hidden = true; + const dialog = $("#daily-delete-dialog"); + if (typeof dialog.showModal === "function") dialog.showModal(); + else dialog.setAttribute("open", ""); + $("#daily-delete-cancel").focus(); + } + + function closeDailyDelete(force = false) { + if (state.dailyDeleting && !force) return; + const request = state.dailyDeleteRequest; + state.dailyDeleteRequest = null; + const dialog = $("#daily-delete-dialog"); + if (typeof dialog.close === "function") dialog.close(); + else dialog.removeAttribute("open"); + if (request?.trigger?.isConnected) request.trigger.focus(); + } + + async function confirmDailyDelete() { + const request = state.dailyDeleteRequest; + if (!request || state.dailyDeleting) return; + state.dailyDeleting = true; + $("#daily-delete-dialog").setAttribute("aria-busy", "true"); + $("#daily-delete-cancel").disabled = true; + $("#daily-delete-submit").disabled = true; + $("#daily-delete-error").hidden = true; + renderJobs(); + renderDailyHistory(); + let deleted = false; + try { + const result = await api(`/api/jobs/${encodeURIComponent(request.jobId)}`, { + method: "DELETE", headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ business_date: request.date }), + }); + deleted = true; + // Once deletion succeeds, never present a later refresh failure as a failed deletion. + if (state.dailyPriceReviewJobId === request.jobId) { + state.dailyPriceReview = null; + state.dailyPriceReviewJobId = ""; + renderDailyPriceReview(null); + } + state.arrDownloadPendingReviews = state.arrDownloadPendingReviews.filter((task) => task.job_id !== request.jobId); + if (state.arrDownloadTask?.job_id === request.jobId) { + await acceptARRDownloadTask({ ...state.arrDownloadTask, status: "deleted", can_retry: false, is_retired: true }, { sync: false }); + } + closeDailyDelete(true); + showToast(I18N.t(result.current_removed ? "daily.deleted_current" : "daily.deleted_history")); + await Promise.allSettled([loadJobs(false, false), loadDailyOverview(state.dailyOverviewDate), loadHistoryMonths()]); + if ($("#daily-history-dialog").open) await loadDailyHistory(request.date); + if ($("#panel-monthly").classList.contains("is-active")) await loadMonthly(false, true); + if ($("#panel-bi").classList.contains("is-active")) await loadMonthsAndAnalytics(); + } catch (error) { + $("#daily-delete-error").textContent = error.message; + $("#daily-delete-error").hidden = false; + } finally { + state.dailyDeleting = false; + $("#daily-delete-dialog").setAttribute("aria-busy", "false"); + $("#daily-delete-cancel").disabled = false; + $("#daily-delete-submit").disabled = false; + renderJobs(); + renderDailyHistory(); + if (deleted) { + const focusTarget = $("#daily-history-dialog").open ? $("#daily-history-close") + : $(`[data-daily-date-select="${request.date}"]`) || $("#refresh-jobs"); + focusTarget.focus(); + } + } } function reviewPrice(value) { @@ -1525,6 +1799,11 @@ } setDailyPriceReviewMessage(); await loadDailyPriceReview(jobId, { reset: jobId !== state.dailyPriceReviewJobId, scroll }); + if (state.dailyPriceReviewJobId === jobId && validARRDate(state.dailyPriceReview?.business_date)) { + $("#arr-download-date").value = state.dailyPriceReview.business_date; + state.arrDownloadDateEdited = true; + renderARRDownload(); + } } async function saveDailyPriceReviewItem(button) { @@ -1924,7 +2203,7 @@ } } - async function selectJob(jobId) { + async function selectJob(jobId, { syncDate = true } = {}) { if (!jobId) return; clearTracePoll(); const changed = state.selectedJobId !== jobId; @@ -1932,30 +2211,47 @@ if (changed) state.jobTrace = null; renderJobs(state.jobs); await loadJobTrace(true, !changed); + const date = state.jobs.find((job) => job.job_id === jobId)?.arrival_date + || (state.jobTrace?.job?.job_id === jobId ? state.jobTrace.job.business_date : ""); + if (syncDate && state.selectedJobId === jobId && validARRDate(date)) { + $("#arr-download-date").value = date; + renderARRDownload(); + } } async function loadJobs(showErrors = false, loadTrace = true) { if (state.jobsLoading) return; state.jobsLoading = true; + const requestedMonth = historyMonth("jobs"); + const context = state.arrDownloadContextId; + let requestedOffset = state.jobsOffset; + const isCurrent = () => historyMonth("jobs") === requestedMonth + && state.arrDownloadContextId === context && state.jobsOffset === requestedOffset; renderPagination("jobs", state.jobsTotal, state.jobsOffset, true); try { - const month = encodeURIComponent(historyMonth("jobs")); + const month = encodeURIComponent(requestedMonth); let page = readPage(await api( - `/api/jobs?month=${month}&limit=${HISTORY_PAGE_SIZE}&offset=${state.jobsOffset}`, + `/api/daily-reports?month=${month}&limit=${HISTORY_PAGE_SIZE}&offset=${requestedOffset}`, { returnEnvelope: true }, )); + if (!isCurrent()) return; if (!page.items.length && page.total > 0 && state.jobsOffset >= page.total) { state.jobsOffset = pageOffset(page.total, state.jobsOffset, -1); + requestedOffset = state.jobsOffset; page = readPage(await api( - `/api/jobs?month=${month}&limit=${HISTORY_PAGE_SIZE}&offset=${state.jobsOffset}`, + `/api/daily-reports?month=${month}&limit=${HISTORY_PAGE_SIZE}&offset=${requestedOffset}`, { returnEnvelope: true }, )); + if (!isCurrent()) return; } - state.jobs = page.items; + state.dailyReports = page.items; + if (page.items.some((day) => day.business_date === state.dailyOverviewDate)) state.dailyOverview = null; + state.jobs = [...new Map(page.items.flatMap((day) => [day.current_job, day.latest_job]) + .filter(Boolean).map((job) => [job.job_id, job])).values()]; state.jobsTotal = page.total; renderJobs(state.jobs); if (!state.selectedJobId && state.jobs.length) { - await selectJob(state.jobs[0].job_id); + await selectJob(state.jobs[0].job_id, { syncDate: false }); } else if (loadTrace && state.selectedJobId) { await loadJobTrace(showErrors, Boolean(state.jobTrace)); } else if (!state.jobs.length && !state.selectedJobId) { @@ -1966,11 +2262,13 @@ setTraceLiveState(I18N?.t("task.live_unselected") || "未选择任务"); } } catch (error) { + if (!isCurrent()) return; $("#jobs-body").innerHTML = `${escapeHtml(I18N?.t("daily.unavailable") || "日报记录暂时无法读取")}`; if (showErrors) showToast(error.message, true); } finally { state.jobsLoading = false; renderPagination("jobs", state.jobsTotal, state.jobsOffset); + if (!isCurrent()) await loadJobs(false, false); } } @@ -3341,6 +3639,11 @@ renderARRDataReview(); renderARRCalendar(); renderJobs(state.jobs); + renderDailyHistory(); + if (state.dailyDeleteRequest) { + $("#daily-delete-description").textContent = I18N.t(state.dailyDeleteRequest.current ? "daily.delete_current_description" : "daily.delete_history_description", { date: state.dailyDeleteRequest.date }); + $("#daily-delete-note").textContent = I18N.t("daily.delete_note"); + } renderPagination("jobs", state.jobsTotal, state.jobsOffset, state.jobsLoading); renderMonthly(state.monthlyRuns); renderPagination("monthly", state.monthlyTotal, state.monthlyOffset, state.monthlyLoading); @@ -3553,12 +3856,56 @@ changeHistoryPage("companyReportsTotal", "companyReportsOffset", 1, () => loadCompanyReportHistory(false)) ); $("#jobs-body").addEventListener("click", (event) => { + const dateTrigger = event.target.closest("[data-daily-date-select]"); + if (dateTrigger) { selectDailyReportDate(dateTrigger.dataset.dailyDateSelect); return; } + const historyTrigger = event.target.closest("[data-daily-history]"); + if (historyTrigger) { openDailyHistory(historyTrigger.dataset.dailyHistory, historyTrigger); return; } + const deleteTrigger = event.target.closest("[data-daily-delete]"); + if (deleteTrigger) { openDailyDelete(deleteTrigger); return; } const reviewTrigger = event.target.closest("[data-open-price-review]"); if (reviewTrigger) { event.preventDefault(); openDailyPriceReview(reviewTrigger.dataset.openPriceReview, true); } }); + $("#daily-history-body").addEventListener("click", async (event) => { + const deleteTrigger = event.target.closest("[data-daily-delete]"); + if (deleteTrigger) { openDailyDelete(deleteTrigger); return; } + const reviewTrigger = event.target.closest("[data-open-price-review]"); + if (reviewTrigger) { + closeDailyHistory(); + await selectDailyReportDate(state.dailyHistoryDate); + await openDailyPriceReview(reviewTrigger.dataset.openPriceReview, true); + return; + } + const logTrigger = event.target.closest("[data-daily-job-log]"); + if (logTrigger) { + closeDailyHistory(); + await selectJob(logTrigger.dataset.dailyJobLog); + openTaskLog(); + } + }); + $("#daily-history-close").addEventListener("click", closeDailyHistory); + $("#daily-history-dialog").addEventListener("cancel", (event) => { + event.preventDefault(); + closeDailyHistory(); + }); + $("#daily-history-prev").addEventListener("click", async () => { + if (state.dailyHistoryLoading || state.dailyDeleting) return; + state.dailyHistoryOffset = Math.max(0, state.dailyHistoryOffset - HISTORY_PAGE_SIZE); + await loadDailyHistory(); + }); + $("#daily-history-next").addEventListener("click", async () => { + if (state.dailyHistoryLoading || state.dailyDeleting) return; + state.dailyHistoryOffset += HISTORY_PAGE_SIZE; + await loadDailyHistory(); + }); + $("#daily-delete-cancel").addEventListener("click", () => closeDailyDelete()); + $("#daily-delete-submit").addEventListener("click", confirmDailyDelete); + $("#daily-delete-dialog").addEventListener("cancel", (event) => { + event.preventDefault(); + closeDailyDelete(); + }); $("#daily-price-review-body").addEventListener("click", (event) => { const save = event.target.closest("[data-daily-price-review-save]"); if (save) saveDailyPriceReviewItem(save); diff --git a/arr_web/static/i18n.js b/arr_web/static/i18n.js index 54a44e1..c698bf5 100644 --- a/arr_web/static/i18n.js +++ b/arr_web/static/i18n.js @@ -250,10 +250,41 @@ "upload.stage_processor": ["正在运行固定处理器", "Running the fixed processor", "กำลังเรียกใช้ตัวประมวลผล"], "upload.stage_validate": ["正在独立验收结果", "Independently validating the result", "กำลังตรวจสอบผลอย่างอิสระ"], "upload.stage_commit": ["正在提交数据库", "Committing to the database", "กำลังบันทึกลงฐานข้อมูล"], - "metric.arrival_date": ["最近成功处理日期", "Latest successful processing date", "วันที่ประมวลผลสำเร็จล่าสุด"], + "metric.arrival_date": ["当前查看日期", "Selected report date", "วันที่รายงานที่เลือก"], "metric.duration": ["处理耗时", "Processing time", "เวลาประมวลผล"], "metric.duration_from_submit": ["从提交至完成", "From submission to completion", "ตั้งแต่ส่งจนเสร็จ"], "metric.rooms": ["保留记录的售出房数", "Sold rooms in retained records", "จำนวนห้องที่ขายในรายการที่เก็บไว้"], + "daily.current": ["当前有效日报", "Current daily report", "รายงานรายวันที่ใช้อยู่"], + "daily.no_current": ["无当前有效日报", "No current daily report", "ไม่มีรายงานรายวันที่ใช้อยู่"], + "daily.history_only": ["仅保留历史版本", "Historical versions only", "มีเฉพาะรุ่นในประวัติ"], + "daily.current_preserved": ["当前有效日报仍保留", "The current daily report remains valid", "รายงานรายวันปัจจุบันยังคงใช้ได้"], + "daily.overview_review": ["待人工核对", "Awaiting manual review", "รอการตรวจสอบด้วยตนเอง"], + "daily.overview_fetching": ["正在获取该日数据", "Fetching data for this date", "กำลังดึงข้อมูลของวันที่นี้"], + "daily.overview_processing": ["正在生成该日日报", "Generating this daily report", "กำลังสร้างรายงานรายวันของวันที่นี้"], + "daily.after_generation": ["待新日报生成后显示", "Shown after the new report is generated", "แสดงเมื่อสร้างรายงานใหม่แล้ว"], + "daily.replacement": ["新版本", "New version", "รุ่นใหม่"], + "daily.history_versions": ["历史记录({count})", "History ({count})", "ประวัติ ({count})"], + "daily.historical": ["历史版本", "Historical version", "รุ่นในประวัติ"], + "daily.deleted": ["已删除", "Deleted", "ลบแล้ว"], + "daily.history_title": ["日报历史记录", "Daily report history", "ประวัติรายงานรายวัน"], + "daily.history_date": ["{date} · 全部处理记录", "{date} · All processing records", "{date} · ประวัติการประมวลผลทั้งหมด"], + "daily.history_empty": ["暂无历史记录", "No history records", "ยังไม่มีประวัติ"], + "daily.history_note": ["每天仅展示当前有效日报;新版本处理成功后替换,月报同步更新。", "One current report per date. A successfully processed version replaces it and updates the monthly report.", "แสดงรายงานที่ใช้อยู่หนึ่งฉบับต่อวัน รุ่นใหม่จะแทนที่เมื่อประมวลผลสำเร็จและอัปเดตรายงานประจำเดือน"], + "daily.history_dialog_note": ["旧版本仅供追溯,不计入月报。已删除记录保留操作痕迹,不会自动恢复。", "Old versions are kept for reference and excluded from the monthly report. Deleted records retain their audit history and are not restored automatically.", "เก็บรุ่นเก่าไว้ตรวจสอบย้อนหลังและไม่นับในรายงานประจำเดือน รายการที่ลบยังมีประวัติการดำเนินการและจะไม่กู้คืนอัตโนมัติ"], + "daily.history_region": ["日报历史版本", "Daily report versions", "รุ่นของรายงานรายวัน"], + "daily.history_pagination": ["日报历史分页", "Daily report history pages", "การแบ่งหน้าประวัติรายงานรายวัน"], + "daily.view_log": ["查看日志", "View log", "ดูบันทึก"], + "daily.delete_title": ["删除日报", "Delete daily report", "ลบรายงานรายวัน"], + "daily.delete_current_description": ["删除 {date} 当前有效日报?该日数据将从月报中移除。", "Delete the current report for {date}? This date will be removed from the monthly report.", "ลบรายงานที่ใช้อยู่ของวันที่ {date} หรือไม่ ข้อมูลวันนี้จะถูกลบออกจากรายงานประจำเดือน"], + "daily.delete_history_description": ["删除 {date} 的这条处理记录?当前有效日报和月报不变。", "Delete this processing record for {date}? The current daily and monthly reports will not change.", "ลบประวัติการประมวลผลนี้ของวันที่ {date} หรือไม่ รายงานรายวันปัจจุบันและรายงานประจำเดือนจะไม่เปลี่ยน"], + "daily.delete_note": ["原始数据和操作记录保留;删除后不会自动恢复旧版本。", "Original data and audit history are retained. Deleting a report will not reactivate an older version.", "เก็บข้อมูลต้นฉบับและประวัติการดำเนินการไว้ การลบจะไม่เปิดใช้รุ่นเก่าโดยอัตโนมัติ"], + "daily.confirm_delete": ["确认删除", "Confirm delete", "ยืนยันการลบ"], + "daily.deleted_current": ["该日日报已移除,月报正在同步更新", "Daily report removed; the monthly report is updating", "ลบรายงานรายวันแล้ว กำลังอัปเดตรายงานประจำเดือน"], + "daily.deleted_history": ["处理记录已删除,当前日报和月报保持不变", "Record deleted; the current daily and monthly reports are unchanged", "ลบประวัติแล้ว รายงานรายวันปัจจุบันและรายงานประจำเดือนไม่เปลี่ยน"], + "arr_download.deleted": ["报表已删除,可重新获取该日数据", "Report deleted; you may fetch this date again", "ลบรายงานแล้ว สามารถดึงข้อมูลวันนี้ใหม่ได้"], + "error.daily_delete_invalid": ["删除请求无效,请刷新日报列表后重试", "Invalid delete request. Refresh the daily list and try again.", "คำขอลบไม่ถูกต้อง โปรดรีเฟรชรายการรายวันแล้วลองใหม่"], + "error.daily_delete_busy": ["这份报表正在处理,请完成后再删除", "This report is being processed. Try deleting it after processing finishes.", "รายงานนี้กำลังประมวลผล โปรดลองลบเมื่อประมวลผลเสร็จ"], + "error.daily_delete_unavailable": ["日报删除服务暂不可用,请稍后重试", "Report deletion is temporarily unavailable. Try again later.", "บริการลบรายงานไม่พร้อมใช้งานชั่วคราว โปรดลองภายหลัง"], "daily.history": ["Daily Report", "Daily Report", "Daily Report"], "daily.refresh": ["刷新", "Refresh", "รีเฟรช"], "daily.download": ["日报", "Daily report", "รายงานรายวัน"], @@ -678,6 +709,13 @@ }; const ERROR_KEYS = { + DAILY_DELETE_REQUEST_INVALID: "error.daily_delete_invalid", + DAILY_DATE_CONFLICT: "error.daily_delete_invalid", + DAILY_JOB_BUSY: "error.daily_delete_busy", + DAILY_RETIREMENT_UNAVAILABLE: "error.daily_delete_unavailable", + DAILY_RETIREMENT_CONFLICT: "error.daily_delete_invalid", + DAILY_JOB_NOT_FOUND: "error.not_found", + JOB_RETIRED: "arr_download.deleted", ARR_DOWNLOAD_UNAVAILABLE: "arr_download.unavailable", ARR_DOWNLOAD_DATE_INVALID: "arr_download.date_invalid", ARR_DOWNLOAD_REQUEST_INVALID: "arr_download.request_invalid", diff --git a/arr_web/static/index.html b/arr_web/static/index.html index 4b23709..23ef146 100644 --- a/arr_web/static/index.html +++ b/arr_web/static/index.html @@ -124,17 +124,18 @@
ARRIVAL DATE — - 最近成功处理日期 + 当前查看日期 +
处理耗时 — - 从提交至完成 + 从提交至完成
NO. OF ROOM — - 保留记录的售出房数 + 保留记录的售出房数
@@ -214,6 +215,7 @@

Daily Report

+

每天仅展示当前有效日报;新版本处理成功后替换,月报同步更新。

@@ -228,7 +230,7 @@
- +
文件状态ARRIVAL DATENO. OF ROOM处理耗时提交时间日报
文件状态ARRIVAL DATENO. OF ROOM处理耗时提交时间操作
正在读取本月记录…
@@ -510,6 +512,29 @@ + +
+

日报历史记录

+ +
+

旧版本仅供追溯,不计入月报。已删除记录保留操作痕迹,不会自动恢复。

+ +
+
文件状态NO. OF ROOM提交时间操作
+
+ +
+ + +
+

删除日报

+

+

原始数据和操作记录保留;删除后不会自动恢复旧版本。

+ +
+
+
+

显示当前选中的日报处理任务从上传到下游通知的处理日志。

diff --git a/arr_web/static/styles.css b/arr_web/static/styles.css index eb37165..525e690 100644 --- a/arr_web/static/styles.css +++ b/arr_web/static/styles.css @@ -390,6 +390,37 @@ button { color: inherit; } .text-button:hover { color: var(--blue); border-color: #a9c1f5; } .download-link { color: var(--blue); font-size: 12px; font-weight: 700; text-decoration: none; } .download-link:hover { text-decoration: underline; text-underline-offset: 3px; } +.daily-history-note { max-width: 58ch; margin: 7px 0 0; color: var(--subtle); font-size: 12px; line-height: 1.6; } +.daily-version-label { display: block; margin-top: 5px; color: var(--subtle); font-size: 11px; } +.daily-row-actions { display: flex; flex-wrap: wrap; align-items: center; gap: 10px 14px; } +.daily-date-button, .daily-history-button, .daily-delete-button { padding: 5px 0; border: 0; border-radius: 4px; background: transparent; color: var(--blue); font: inherit; font-size: 12px; cursor: pointer; } +.daily-date-button { font-size: 13px; font-variant-numeric: tabular-nums; } +.daily-date-button[aria-pressed="true"] { font-weight: 750; text-decoration: underline; text-underline-offset: 4px; } +.daily-date-button:hover, .daily-history-button:hover, .daily-delete-button:hover { text-decoration: underline; text-underline-offset: 3px; } +.daily-delete-button { color: #b42318; } +.daily-row-actions button:disabled { opacity: .45; cursor: wait; } +.daily-date-button:focus-visible, .daily-history-button:focus-visible, .daily-delete-button:focus-visible { outline: 3px solid rgba(37, 99, 235, .2); outline-offset: 3px; } +.daily-replacement { display: flex; align-items: flex-start; flex-wrap: wrap; gap: 5px 9px; margin-top: 9px; padding-top: 8px; border-top: 1px solid var(--line); } +.daily-replacement > span { padding-top: 4px; color: var(--subtle); font-size: 11px; } +#metric-current-version { color: #9a4b0a; line-height: 1.5; } +.daily-history-dialog { width: min(1100px, calc(100% - 32px)); max-width: none; max-height: calc(100vh - 48px); padding: 0; overflow: auto; border: 1px solid var(--line); border-radius: 16px; color: var(--ink); background: var(--surface); box-shadow: 0 28px 80px rgba(16, 24, 40, .3); } +.daily-history-dialog::backdrop, .daily-delete-dialog::backdrop { background: rgba(15, 23, 42, .48); } +.daily-history-dialog-heading { display: flex; justify-content: space-between; align-items: flex-start; gap: 20px; padding: 24px 26px 12px; } +.daily-history-dialog h2, .daily-delete-dialog h2 { margin: 0; font-size: 20px; } +.daily-history-dialog-heading p { margin: 8px 0 0; color: var(--blue); font-size: 13px; } +.daily-history-dialog-heading > button, .daily-delete-actions button { min-height: 40px; padding: 9px 16px; border: 1px solid var(--line); border-radius: 8px; background: white; color: var(--ink); font: inherit; font-size: 13px; cursor: pointer; } +.daily-history-dialog-note { margin: 0; padding: 0 26px 20px; color: var(--subtle); font-size: 12px; line-height: 1.6; } +.daily-history-dialog table { min-width: 650px; } +.daily-delete-dialog { width: min(460px, calc(100% - 32px)); max-width: none; padding: 0; border: 1px solid var(--line); border-radius: 16px; color: var(--ink); background: var(--surface); box-shadow: 0 28px 80px rgba(16, 24, 40, .3); } +.daily-delete-card { padding: 26px; } +.daily-delete-card > p { margin: 16px 0 0; font-size: 14px; line-height: 1.7; } +#daily-delete-note { color: var(--subtle); font-size: 12px; } +.daily-delete-error { margin: 0; padding: 12px 26px; color: #b42318; font-size: 13px; line-height: 1.6; } +.daily-delete-card .daily-delete-error { padding: 0; } +.daily-delete-actions { display: flex; justify-content: flex-end; gap: 10px; margin-top: 24px; } +#daily-delete-submit { border-color: #b42318; color: white; background: #b42318; } +.daily-delete-actions button:disabled { opacity: .5; cursor: wait; } +.daily-delete-actions button:focus-visible, .daily-history-dialog-heading > button:focus-visible { outline: 3px solid rgba(37, 99, 235, .2); outline-offset: 3px; } .table-scroll { overflow: auto; border-top: 1px solid var(--line); } .pagination-bar { min-height: 58px; padding: 11px 18px; display: flex; align-items: center; justify-content: space-between; gap: 18px; border-top: 1px solid var(--line); background: #fbfcfe; } .pagination-summary, .pagination-page { color: var(--muted); font-size: 11px; font-weight: 650; font-variant-numeric: tabular-nums; } diff --git a/database/022_daily_report_retirement.down.sql b/database/022_daily_report_retirement.down.sql new file mode 100644 index 0000000..499d1bd --- /dev/null +++ b/database/022_daily_report_retirement.down.sql @@ -0,0 +1,111 @@ +-- Operator-only rollback; refuses to discard any retirement or withdrawn publication. +BEGIN; +DO $$ +BEGIN + IF current_database() <> 'booking_test' THEN + RAISE EXCEPTION 'ARR daily lifecycle rollback is allowed only in booking_test'; + END IF; + IF EXISTS (SELECT 1 FROM ingestion.daily_run_retirements) + OR EXISTS (SELECT 1 FROM reporting.monthly_runs WHERE report_status = 'withdrawn' OR withdrawn_at IS NOT NULL) THEN + RAISE EXCEPTION 'refusing rollback: daily retirement or withdrawn monthly audit exists'; + END IF; +END $$; +DROP TRIGGER daily_run_retirements_immutable ON ingestion.daily_run_retirements; +DROP FUNCTION ingestion.protect_daily_run_retirement(); +DROP TABLE ingestion.daily_run_retirements; +DROP INDEX reporting.monthly_runs_live_snapshot_unique; +ALTER TABLE reporting.monthly_runs + DROP CONSTRAINT monthly_runs_report_status_check, + DROP CONSTRAINT monthly_runs_status_shape, + DROP COLUMN withdrawn_at; +ALTER TABLE reporting.monthly_runs + ADD CONSTRAINT monthly_runs_report_status_check CHECK (report_status IN ('reserved', 'active', 'superseded', 'failed')), + ADD CONSTRAINT monthly_runs_snapshot_unique UNIQUE (period_start, source_snapshot_sha256), + ADD CONSTRAINT monthly_runs_status_shape CHECK ( + ( + report_status = 'reserved' + AND workbook_artifact_id IS NULL + AND result_artifact_id IS NULL + AND semantic_sha256 IS NULL + AND failure_code IS NULL + AND failure_message IS NULL + AND published_at IS NULL + AND superseded_at IS NULL + AND failed_at IS NULL + ) + OR ( + report_status = 'active' + AND workbook_artifact_id IS NOT NULL + AND result_artifact_id IS NOT NULL + AND semantic_sha256 IS NOT NULL + AND failure_code IS NULL + AND failure_message IS NULL + AND published_at IS NOT NULL + AND superseded_at IS NULL + AND failed_at IS NULL + ) + OR ( + report_status = 'superseded' + AND workbook_artifact_id IS NOT NULL + AND result_artifact_id IS NOT NULL + AND semantic_sha256 IS NOT NULL + AND failure_code IS NULL + AND failure_message IS NULL + AND published_at IS NOT NULL + AND superseded_at IS NOT NULL + AND failed_at IS NULL + ) + OR ( + report_status = 'failed' + AND workbook_artifact_id IS NULL + AND result_artifact_id IS NULL + AND semantic_sha256 IS NULL + AND failure_code IS NOT NULL + AND btrim(failure_code) <> '' + AND published_at IS NULL + AND superseded_at IS NULL + AND failed_at IS NOT NULL + ) + ); +CREATE OR REPLACE FUNCTION finance.validate_current_daily_version() +RETURNS trigger +LANGUAGE plpgsql +AS $$ +BEGIN + IF NOT EXISTS ( + SELECT 1 + FROM finance.daily_versions AS version + WHERE version.id = NEW.daily_version_id + AND version.business_date = NEW.business_date + AND version.version_status = 'active' + ) THEN + RAISE EXCEPTION + 'current business date must reference an active daily version'; + END IF; + RETURN NEW; +END; +$$; + +CREATE OR REPLACE FUNCTION reporting.protect_published_monthly_child() +RETURNS trigger +LANGUAGE plpgsql +AS $$ +DECLARE + parent_id bigint; +BEGIN + parent_id := COALESCE(OLD.report_id, NEW.report_id); + IF EXISTS ( + SELECT 1 + FROM reporting.monthly_runs AS run + WHERE run.id = parent_id + AND run.report_status IN ('active', 'superseded') + ) THEN + RAISE EXCEPTION + 'published monthly report lineage and manifest are immutable'; + END IF; + RETURN COALESCE(NEW, OLD); +END; +$$; + + +COMMIT; diff --git a/database/022_daily_report_retirement.sql b/database/022_daily_report_retirement.sql new file mode 100644 index 0000000..acc8816 --- /dev/null +++ b/database/022_daily_report_retirement.sql @@ -0,0 +1,170 @@ +-- Recoverable daily retirement and withdrawal of stale monthly publications. +-- PostgreSQL 15+. Migrations 008 through 021 remain immutable. +BEGIN; +DO $$ +BEGIN + IF current_database() <> 'booking_test' THEN + RAISE EXCEPTION 'ARR daily lifecycle migration is allowed only in booking_test'; + END IF; + IF NOT EXISTS ( + SELECT 1 FROM information_schema.columns + WHERE table_schema = 'finance' AND table_name = 'daily_versions' + AND column_name = 'excluded_pm_rows' + ) OR to_regprocedure('reporting.protect_published_monthly_child()') IS NULL THEN + RAISE EXCEPTION 'ARR migrations 008 through 021 must be applied before 022'; + END IF; + IF to_regclass('ingestion.daily_run_retirements') IS NOT NULL THEN + RAISE EXCEPTION 'ARR daily lifecycle migration 022 is already applied'; + END IF; +END $$; + +CREATE TABLE ingestion.daily_run_retirements ( + id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + processing_run_id bigint NOT NULL UNIQUE REFERENCES ingestion.processing_runs(id), + business_date date NOT NULL, + daily_version_id bigint REFERENCES finance.daily_versions(id), + current_removed boolean NOT NULL, + actor_username text NOT NULL CHECK (btrim(actor_username) <> '' AND char_length(actor_username) <= 255), + retired_at timestamptz NOT NULL DEFAULT now(), + CONSTRAINT daily_run_retirements_current_shape CHECK (NOT current_removed OR daily_version_id IS NOT NULL) +); +COMMENT ON TABLE ingestion.daily_run_retirements IS + 'Append-only retirement receipts. Sources, processing runs, original Finance facts and prior publications are retained; no older daily version is automatically restored.'; + +CREATE FUNCTION ingestion.protect_daily_run_retirement() +RETURNS trigger LANGUAGE plpgsql AS $$ +BEGIN + IF TG_OP <> 'INSERT' THEN + RAISE EXCEPTION 'daily retirement audit is immutable'; + END IF; + IF NOT EXISTS ( + SELECT 1 FROM ingestion.processing_runs AS run + WHERE run.id = NEW.processing_run_id AND run.pipeline_type = 'opera_daily' + AND run.business_date = NEW.business_date + AND run.run_status IN ('accepted', 'rejected', 'failed', 'cancelled') + ) OR (NEW.daily_version_id IS NOT NULL AND NOT EXISTS ( + SELECT 1 FROM finance.daily_versions AS version + WHERE version.id = NEW.daily_version_id AND version.processing_run_id = NEW.processing_run_id + AND (version.business_date = NEW.business_date OR version.version_status = 'rejected') + )) THEN + RAISE EXCEPTION 'daily retirement must reference its terminal run and exact daily version'; + END IF; + IF EXISTS ( + SELECT 1 FROM finance.current_daily_versions AS current_version + WHERE current_version.daily_version_id = NEW.daily_version_id + ) THEN + RAISE EXCEPTION 'a retired daily version cannot remain the current daily source'; + END IF; + RETURN NEW; +END $$; +CREATE TRIGGER daily_run_retirements_immutable +BEFORE INSERT OR UPDATE OR DELETE ON ingestion.daily_run_retirements +FOR EACH ROW EXECUTE FUNCTION ingestion.protect_daily_run_retirement(); + +CREATE OR REPLACE FUNCTION finance.validate_current_daily_version() +RETURNS trigger LANGUAGE plpgsql AS $$ +BEGIN + IF NOT EXISTS ( + SELECT 1 FROM finance.daily_versions AS version + WHERE version.id = NEW.daily_version_id + AND version.business_date = NEW.business_date + AND version.version_status = 'active' + AND NOT EXISTS ( + SELECT 1 FROM ingestion.daily_run_retirements AS retired + WHERE retired.processing_run_id = version.processing_run_id + ) + ) THEN + RAISE EXCEPTION 'current business date must reference an active non-retired daily version'; + END IF; + RETURN NEW; +END $$; + +ALTER TABLE reporting.monthly_runs + ADD COLUMN withdrawn_at timestamptz, + DROP CONSTRAINT monthly_runs_report_status_check, + DROP CONSTRAINT monthly_runs_status_shape, + DROP CONSTRAINT monthly_runs_snapshot_unique; +ALTER TABLE reporting.monthly_runs + ADD CONSTRAINT monthly_runs_report_status_check CHECK (report_status IN ( + 'reserved', 'active', 'superseded', 'failed', 'withdrawn' + )), + ADD CONSTRAINT monthly_runs_status_shape CHECK ( + (withdrawn_at IS NULL AND ( + ( + report_status = 'reserved' + AND workbook_artifact_id IS NULL + AND result_artifact_id IS NULL + AND semantic_sha256 IS NULL + AND failure_code IS NULL + AND failure_message IS NULL + AND published_at IS NULL + AND superseded_at IS NULL + AND failed_at IS NULL + ) + OR ( + report_status = 'active' + AND workbook_artifact_id IS NOT NULL + AND result_artifact_id IS NOT NULL + AND semantic_sha256 IS NOT NULL + AND failure_code IS NULL + AND failure_message IS NULL + AND published_at IS NOT NULL + AND superseded_at IS NULL + AND failed_at IS NULL + ) + OR ( + report_status = 'superseded' + AND workbook_artifact_id IS NOT NULL + AND result_artifact_id IS NOT NULL + AND semantic_sha256 IS NOT NULL + AND failure_code IS NULL + AND failure_message IS NULL + AND published_at IS NOT NULL + AND superseded_at IS NOT NULL + AND failed_at IS NULL + ) + OR ( + report_status = 'failed' + AND workbook_artifact_id IS NULL + AND result_artifact_id IS NULL + AND semantic_sha256 IS NULL + AND failure_code IS NOT NULL + AND btrim(failure_code) <> '' + AND published_at IS NULL + AND superseded_at IS NULL + AND failed_at IS NOT NULL + ) + )) OR (report_status = 'withdrawn' AND withdrawn_at IS NOT NULL) + ); +CREATE UNIQUE INDEX monthly_runs_live_snapshot_unique + ON reporting.monthly_runs (period_start, source_snapshot_sha256) + WHERE report_status <> 'withdrawn'; + +CREATE OR REPLACE FUNCTION reporting.protect_published_monthly_child() +RETURNS trigger LANGUAGE plpgsql AS $$ +DECLARE parent_id bigint; +BEGIN + parent_id := COALESCE(OLD.report_id, NEW.report_id); + IF EXISTS ( + SELECT 1 FROM reporting.monthly_runs AS run + WHERE run.id = parent_id AND run.report_status IN ('active', 'superseded', 'withdrawn') + ) THEN + RAISE EXCEPTION 'published monthly report lineage and manifest are immutable'; + END IF; + RETURN COALESCE(NEW, OLD); +END $$; +COMMENT ON COLUMN reporting.monthly_runs.withdrawn_at IS + 'Publication removed from downloads because a current daily source was retired. Historical artifacts and lineage remain intact; reserved builds cannot later publish.'; + +-- Existing consuming application role; no source, artifact or Finance-fact +-- DELETE privileges are added. Differently named roles require equivalent +-- explicit grants in the release deployment, never PUBLIC or web read roles. +DO $$ +BEGIN + IF to_regrole('arr_app') IS NOT NULL THEN + EXECUTE 'GRANT SELECT, INSERT ON ingestion.daily_run_retirements TO arr_app'; + EXECUTE 'GRANT USAGE, SELECT ON SEQUENCE ingestion.daily_run_retirements_id_seq TO arr_app'; + EXECUTE 'GRANT DELETE ON finance.current_daily_versions TO arr_app'; + END IF; +END $$; +COMMIT; diff --git a/deploy/OHIP_RELEASE_HANDOVER.md b/deploy/OHIP_RELEASE_HANDOVER.md index e2be383..7d7750b 100644 --- a/deploy/OHIP_RELEASE_HANDOVER.md +++ b/deploy/OHIP_RELEASE_HANDOVER.md @@ -1,10 +1,10 @@ # ARR 自动取数交付与生产启用说明 -更新日期:2026-10-08。状态:**代码收尾完成,本机已连接生产酒店57106;目标主机部署和完整报表业务验收仍需执行。** +更新日期:2026-10-09。状态:**本机已连接生产酒店57106,并完成022及日报版本管理升级;目标主机部署和完整报表业务验收仍需执行。** 本机10月7日任务已保存生成成功结果,38间房;该状态不代表已逐项核对实际日报文件及月报内容。 早期沙箱/模拟检查是历史验证,不替代生产业务验收。生产连接、修复和本机验证记录见[本次任务](../.project-docs/30-worklog/tasks/20261008-production-review-9e7b.md)。 -当前处理器为4.4.0。目标数据库需按顺序具备019、020和021,Web与处理器必须一起发布。 +当前处理器为4.4.0。目标数据库需按顺序具备019、020、021和022,Web、处理服务与月报worker必须一起发布。 用户已确认两条输入路径都排除已取消预订和PM房型;完整原始数据仍保留,不要求被排除记录补房号。 已有未处理任务接续必须备份并显式记录新旧规则身份,不能复用旧身份直接运行新规则;已发布结果不自动重新处理。 @@ -29,7 +29,7 @@ | 酒店与平台 | 当前本机已连接生产酒店57106。目标部署仍须核对平台实际路由和授权;本地hotel-id仅校验返回酒店,不能把沙箱接口切成生产接口。 | | 应用身份 | 当前客户端固定平台`https://ohip.nianxx.cn`及应用`caller_zloxalzsQ9pYsztt`。如生产改用其他平台或新应用,需适配并验证固定身份配置;替换一份不同身份的密钥文件会被拒绝。 | | 读取权限 | 目标应用须覆盖预订、档案、团队、房间历史及套餐8个操作;现有配置需求包含reservations.read、profiles.read、blocks.read、configuration.read。保留平台已授其他权限,不盲目覆盖权限集。 | -| 数据库 | 在已具备018的目标库上按顺序应用019、020、021。本机已升级至021;既有迁移及运行配置要求隔离库名`booking_test`。若目标使用其他库名,先适配和验证数据库边界,不能直接删除保护或假设已有兼容。 | +| 数据库 | 在已具备018的目标库上按顺序应用019、020、021、022。022支持日报删除及月报撤下;既有迁移及运行配置要求隔离库名`booking_test`。若目标使用其他库名,先适配和验证数据库边界,不能直接删除保护或假设已有兼容。 | | 自动取数入口 | 标准启动器支持三个OHIP参数,现有基础Compose尚未传入,部署时须明确加入。仅按基础Compose启动会保留原处理入口而不启用自动取数。 | | 凭据和持久数据 | 挂载正式凭据目录,并为任务/原始数据配置仓库外的持久目录。目录0700、凭据文件0600,属主为实际运行用户;容器须按容器UID准备属主。凭据只读挂载,任务目录可写,不能放临时目录或镜像内。 | | 月报与报表存储 | 复用正确配置的数据库和OSS,独立月报worker须由进程管理器持续运行。网页可访问、Web健康检查成功不等于月报worker已启动。 | @@ -52,9 +52,22 @@ 月报另按部署说明运行`python -m monthly_reports.worker --output-root /app/outputs/monthly_reports`并配置持续监管。 `--enable-monthly-generation`是另一个手动生成入口,不替代自动月报worker。 -数据库升级文件为`database/019_ohip_data_source.sql`、`database/020_daily_cancelled_exclusion.sql`、`database/021_daily_pm_exclusion.sql`,各有对应`.down.sql`。 +数据库升级文件为`database/019_ohip_data_source.sql`、`database/020_daily_cancelled_exclusion.sql`、`database/021_daily_pm_exclusion.sql`、`database/022_daily_report_retirement.sql`,各有对应`.down.sql`。 先检查现有版本、备份和副本验证,再在正式启用窗口按顺序升级;已有相应来源/排除事实时回退会被保护机制拒绝。 +## 日报版本管理升级(2026-10-09) + +同一天只展示一个有效日报;新输入待核对或失败时保留原日报,成功入库后才覆盖。删除当前有效日报会原子移除当日有效指针、保留原始事实和审计、撤下受影响月报,并通过`arr.daily_scope_changed`事件重建该月剩余有效日期的月报。整月为空时撤下月报,不发布空工作簿。删除旧版本或待价格核对版本不会改变当前日报及月报;旧版本不会自动恢复。 + +启用前停止新任务并等现有处理停止,备份数据库、对象存储及私有任务目录;应用022后同时更新Web/处理服务/月报worker,再开放删除操作。旧worker不认识新事件,不能单独发布Web。处理服务启动时检查022的表、索引及审计保护;不满足时拒绝启用。旧本机实例的schema标记须在受控备份及实际升级后更新,不能只改标记绕过升级。 + +022给已有`arr_app`角色精确增加审计表SELECT/INSERT、对应sequence权限以及`finance.current_daily_versions`的DELETE,不授予原始数据、对象或Finance事实的删除权限。自定义应用角色须获得相同权限;自定义网页只读角色须另授予`SELECT ON ingestion.daily_run_retirements`(沿用已有schema USAGE),不能给它写入或DELETE权限。所有角色名按目标环境实际配置,不向PUBLIC授权。 + +迁移不自动删除、合并或重新处理已有报表。同日历史仍可追溯,界面按当前有效指针分组展示。已有删除审计或withdrawn月报时,022回滚会拒绝执行,须先评估恢复方案;恢复整个备份也必须连同匹配的旧代码和worker一起恢复。 + +本次功能仅使用独立合成数据验证删除和月报同步;本机升级保留实际报表及待办,不为验收删除用户日报或重新读取Oracle数据。 +本机升级后逐表核对26张既有业务表、3个有效日报、4,896份原始捕获文件及待办均保持一致(月报新增的空withdrawn_at列不参与原列比较)。022删除审计为0;浏览器已确认9月16日按日期合并及9月13日概览一致。备份、验收及保留运行工作树的记录见[本次版本管理任务](../.project-docs/30-worklog/tasks/20261009-daily-lifecycle-7bc2.md)。 + ## 有真实数据后的首日验收 以下是目标部署后的验收步骤,本次推送不发起查询: @@ -74,4 +87,4 @@ - 最新功能验证:取数进度118项后端、84项JavaScript及13项静态UI检查通过,实际本机9月15日显示85/85笔且人工核对仍保留。取消/PM、字段/价格核对、日报/月报和迁移的既有验证及边界见本次任务记录。 - 当前处理器`.zip`及`.skill`已随4.4.0更新,两包SHA-256均为`aac23951ca72cc6878b617cddaaa5dffd9dc6fe2dee1c9036db61c880118a134`;压缩包内工作簿与原版一致。 - 本机没有Docker命令,本次未构建或启动生产镜像;正式主机部署验证待执行。 -- `.zip/.skill`是处理器包,完整Web交付还包含项目源码、数据库019至021及运行配置,不能只上传处理器包来启用自动下载。 +- `.zip/.skill`是处理器包,完整Web交付还包含项目源码、数据库019至022及运行配置,不能只上传处理器包来启用自动下载。 diff --git a/deploy/README.md b/deploy/README.md index c7ab41e..9fe8a7c 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -1,15 +1,15 @@ # ARR 2.0 单机 Compose 部署 -2026-09-18新增按日期OHIP取数:本文基础Compose尚未启用自动取数,也未包含019的启用步骤。 +2026-10-09:本文基础Compose尚未启用自动取数;当前Web/处理服务还要求数据库升级至022。 使用新增入口必须同时完成[OHIP交付与生产启用说明](OHIP_RELEASE_HANDOVER.md),核对正式平台身份、 -018→019升级、三个启动参数、凭据/任务目录挂载和独立月报worker。 +018→022升级、三个启动参数、凭据/任务目录挂载和独立月报worker。下文016–018步骤保留作为前置迁移说明。 本 Compose 模板部署两个 HTTP 服务:`web` 负责应用登录、页面、程序化 XML 处理、验证和入库,`caddy` 负责 HTTPS。没有 Agent 回调服务或 MCP 公网入口。自动月报由另一个无端口的 `monthly_reports.worker` 进程执行,不得放进 Web 请求进程。 ## 前置条件 - DNS:`WEB_PUBLIC_HOST` 指向部署主机; -- PostgreSQL 15+:目标必须是隔离数据库 `booking_test`,发布窗口内按 016 → 017 → 018 应用; +- PostgreSQL 15+:目标必须是隔离数据库 `booking_test`,发布窗口内按顺序应用至022(016–018见下文,019–022见交付说明); - OSS:区域匹配、服务端加密、未启用或暂停 versioning、禁止匿名写;bucket ACL 可为 private 或 public-read; - ARR 数据库账号和 OSS RAM/STS 凭据由 Secret 管理器注入,不写入镜像或仓库。 - 月报和公司渠道明细均由 Python/openpyxl 生成;不需要 Node.js、npm 或私有运行时。 diff --git a/monthly_reports/repository.py b/monthly_reports/repository.py index 803f97b..e6d8aca 100644 --- a/monthly_reports/repository.py +++ b/monthly_reports/repository.py @@ -215,7 +215,9 @@ class ReservedReport: class DerivedMonthlyRequest: report_year: int report_month: int - as_of_date: date + # A known month with no remaining active facts is a withdrawal, not an + # invalid source. No artificial cutoff or empty workbook is manufactured. + as_of_date: Optional[date] class ReportRepository(Protocol): @@ -452,11 +454,9 @@ class PostgresReportRepository: ) latest = cursor.fetchone() as_of_date = latest[0] if latest else None - if as_of_date is None: - return None return DerivedMonthlyRequest( - report_year=as_of_date.year, - report_month=as_of_date.month, + report_year=first_arrival.year, + report_month=first_arrival.month, as_of_date=as_of_date, ) @@ -491,12 +491,23 @@ class PostgresReportRepository: @classmethod def _validate_pins_current(cls, cursor: Any, report: MonthlyReport) -> None: - period_start, _period_end = month_bounds(report.report_year, report.report_month) + period_start, period_end = month_bounds(report.report_year, report.report_month) if cls._current_pins(cursor, period_start, report.as_of_date) != report.daily_versions: raise RepositoryError( ErrorCode.SOURCE_SNAPSHOT_STALE, "daily source versions changed while the monthly report was being generated", ) + cls._validate_month_watermark(cursor, period_start, period_end, report.as_of_date) + + @staticmethod + def _validate_month_watermark(cursor: Any, period_start: date, period_end: date, as_of_date: date) -> None: + cursor.execute(MONTH_MAX_ARRIVAL_SQL, (period_start, period_end + timedelta(days=1))) + latest = cursor.fetchone() + if latest is None or latest[0] != as_of_date: + raise RepositoryError( + ErrorCode.SOURCE_SNAPSHOT_STALE, + "the monthly report cutoff no longer matches the current Finance facts", + ) @staticmethod def _source_snapshot_sha256(report: MonthlyReport) -> str: @@ -576,6 +587,7 @@ class PostgresReportRepository: ON artifact.id = run.workbook_artifact_id WHERE run.period_start = %s AND run.source_snapshot_sha256 = %s + AND run.report_status <> 'withdrawn' FOR UPDATE OF run """, (period_start, snapshot_sha256), @@ -779,6 +791,7 @@ class PostgresReportRepository: ErrorCode.SOURCE_SNAPSHOT_STALE, "daily source versions changed before publication", ) + self._validate_month_watermark(cursor, period_start, period_end, reservation.as_of_date) cursor.execute( """ SELECT report_status, source_snapshot_sha256 diff --git a/monthly_reports/worker.py b/monthly_reports/worker.py index 340a4f5..dd8b5e6 100644 --- a/monthly_reports/worker.py +++ b/monthly_reports/worker.py @@ -8,11 +8,13 @@ import signal import sys import time from dataclasses import dataclass +from datetime import date, timedelta from pathlib import Path from typing import Any, Callable, Mapping, Optional, Protocol, Sequence from arr_database import controlled_connect from monthly_reports.contracts import ErrorCode +from monthly_reports.core import month_bounds from arr_web.processing_runtime import ObjectStoreRuntime, compose_object_store from monthly_reports.publishing import OpenpyxlWorkbookBuilder, AtomicReportPublisher from monthly_reports.repository import ( @@ -26,6 +28,7 @@ from monthly_reports.service import MonthlyReportService, RunRequest, RunResult PROJECT_ROOT = Path(__file__).resolve().parents[1] EVENT_TYPE = "arr.daily_version_committed" +SCOPE_CHANGED_EVENT_TYPE = "arr.daily_scope_changed" DEFAULT_LEASE_SECONDS = 300 DEFAULT_MAX_ATTEMPTS = 8 @@ -34,8 +37,8 @@ CLAIM_EVENT_SQL = """ WITH candidate AS ( SELECT event.id FROM ingestion.outbox_events AS event - WHERE event.event_type = %s - AND event.aggregate_type = 'processing_run' + WHERE ((event.event_type = %s AND event.aggregate_type = 'processing_run') + OR (event.event_type = %s AND event.aggregate_type = 'daily_version')) AND event.available_at <= now() AND event.publish_status IN ('pending', 'publishing') ORDER BY event.created_at, event.id @@ -53,7 +56,8 @@ RETURNING event.id, event.event_key, event.payload, - event.publish_attempts + event.publish_attempts, + event.event_type """.strip() @@ -80,6 +84,23 @@ WHERE event.id = %s """.strip() +WITHDRAW_EVENT_SQL = """ +UPDATE ingestion.outbox_events AS event +SET publish_status = 'published', published_at = now(), available_at = now(), last_error_code = NULL +WHERE event.id = %s + AND event.publish_status = 'publishing' + AND NOT EXISTS ( + SELECT 1 FROM finance.v_active_daily_facts AS fact + WHERE fact.arrival >= %s AND fact.arrival < %s + ) + AND NOT EXISTS ( + SELECT 1 FROM reporting.monthly_runs AS run + WHERE run.period_start = %s + AND run.report_status IN ('reserved', 'active', 'superseded') + ) +""".strip() + + FAIL_EVENT_SQL = """ UPDATE ingestion.outbox_events SET publish_status = %s, @@ -107,6 +128,7 @@ class OutboxEvent: event_key: str payload: Mapping[str, Any] publish_attempts: int + event_type: str = EVENT_TYPE @dataclass(frozen=True) @@ -134,6 +156,9 @@ class OutboxRepository(Protocol): def mark_published(self, event_id: int, report_id: int) -> None: ... + def mark_withdrawn(self, event_id: int, period_start: date) -> None: + ... + def mark_failed(self, event: OutboxEvent, code: str, *, retryable: bool) -> str: ... @@ -199,7 +224,7 @@ class PostgresOutboxRepository: with connection.transaction(): with connection.cursor() as cursor: self._begin(cursor) - cursor.execute(CLAIM_EVENT_SQL, (EVENT_TYPE, self._lease_seconds)) + cursor.execute(CLAIM_EVENT_SQL, (EVENT_TYPE, SCOPE_CHANGED_EVENT_TYPE, self._lease_seconds)) row = cursor.fetchone() if not row: return None @@ -213,6 +238,7 @@ class PostgresOutboxRepository: event_key=str(row[1]), payload=payload, publish_attempts=int(row[3]), + event_type=str(row[4]), ) except WorkerError: raise @@ -267,6 +293,34 @@ class PostgresOutboxRepository: finally: connection.close() + def mark_withdrawn(self, event_id: int, period_start: date) -> None: + """Acknowledge a verified empty month without inventing an artifact. + + Retirement withdraws publications in the same Finance transaction. The + matching month lock coordinates the acknowledgement with publication + activation and retirement of the month. + """ + if type(period_start) is not date or period_start.day != 1: + raise WorkerError("MONTHLY_WORKER_EVENT_INVALID", retryable=False) + _start, period_end = month_bounds(period_start.year, period_start.month) + connection = self._open() + try: + with connection.transaction(): + with connection.cursor() as cursor: + self._begin(cursor) + PostgresReportRepository._lock_scope(cursor, period_start, period_end) + cursor.execute(WITHDRAW_EVENT_SQL, ( + event_id, period_start, period_end + timedelta(days=1), period_start, + )) + if cursor.rowcount != 1: + raise WorkerError("MONTHLY_WORKER_WITHDRAW_ACK_FAILED", retryable=True) + except WorkerError: + raise + except Exception: + raise WorkerError("MONTHLY_WORKER_WITHDRAW_ACK_FAILED", retryable=True) from None + finally: + connection.close() + class MonthlyOutboxWorker: def __init__( @@ -317,6 +371,16 @@ class MonthlyOutboxWorker: derived = self._requests.derive_monthly_request(daily_version_id) if derived is None: raise WorkerError("MONTHLY_WORKER_ARRIVAL_SCOPE_EMPTY", retryable=False) + period_start = date(derived.report_year, derived.report_month, 1) + if event.event_type == SCOPE_CHANGED_EVENT_TYPE: + if (event.payload.get("action") != "delete" + or event.payload.get("period_start") != period_start.isoformat()): + raise WorkerError("MONTHLY_WORKER_EVENT_INVALID", retryable=False) + elif event.event_type != EVENT_TYPE: + raise WorkerError("MONTHLY_WORKER_EVENT_INVALID", retryable=False) + if derived.as_of_date is None: + self._outbox.mark_withdrawn(event.event_id, period_start) + return WorkerOutcome(status="withdrawn", event_id=event.event_id) result = self._service.run( RunRequest( derived.report_year, diff --git a/tests/javascript/arr_data_review.cjs b/tests/javascript/arr_data_review.cjs index 639d31a..ed6f78b 100644 --- a/tests/javascript/arr_data_review.cjs +++ b/tests/javascript/arr_data_review.cjs @@ -24,7 +24,7 @@ function pendingAPI(url,currentPrice=priceTask) { if(url.includes('/review?')) return response(200,priceReview()); if(url.endsWith('/trace')) return response(200,{job:{job_id:priceTask.job_id, business_date:priceTask.report_date,status:currentPrice.status,active:currentPrice.status==='needs_review'},logs:[]}); - if(url.startsWith('/api/jobs?')) return response(200,[]); + if(url.startsWith('/api/daily-reports?')) return response(200,[]); throw new Error('unexpected request '+url); } @@ -34,7 +34,7 @@ test('an empty validated field review polls the same task and follows processing const h=harness(url=>{ if(url.endsWith('/data-review')) return response(200,review([])); if(url===`/api/arr-downloads/${requestId}`) return response(200,{...task,status:nextStatus,job_id:nextStatus==='processing'?null:jobId}); - if(url.startsWith('/api/jobs?')) return response(200,[]); + if(url.startsWith('/api/daily-reports?')) return response(200,[]); if(url.endsWith('/trace')) return response(200,{job:{job_id:jobId,business_date:task.report_date,status:nextStatus,active:true},logs:[]}); if(url.includes('/review?')) return response(200,{...priceReview(),business_date:task.report_date}); throw new Error('unexpected request '+url); diff --git a/tests/javascript/daily_price_review.cjs b/tests/javascript/daily_price_review.cjs index c139322..fb24daa 100644 --- a/tests/javascript/daily_price_review.cjs +++ b/tests/javascript/daily_price_review.cjs @@ -43,7 +43,7 @@ test('explicit zero is saved as zero and survives reloading without becoming emp return response(200,{revision:2}); } if(url.startsWith('/api/jobs/fixture-job/review?')) return response(200,current); - if(url.startsWith('/api/jobs?')) return response(200,[]); + if(url.startsWith('/api/daily-reports?')) return response(200,[]); throw new Error('unexpected request '+url); }); h.state.dailyPriceReview=fixture(); diff --git a/tests/javascript/daily_report_lifecycle.cjs b/tests/javascript/daily_report_lifecycle.cjs new file mode 100644 index 0000000..4762427 --- /dev/null +++ b/tests/javascript/daily_report_lifecycle.cjs @@ -0,0 +1,290 @@ +// Date identity, replacement and recoverable deletion against a synthetic API only. +const test = require('node:test'); +const assert = require('node:assert/strict'); +const fs = require('node:fs'); +const path = require('node:path'); +const vm = require('node:vm'); +const {harness,response} = require('./helpers/arr_ui_harness.cjs'); +const date='2026-09-16'; +const job=(id,extra={})=>({job_id:id,arrival_date:date,status:'succeeded',filename:`${id}.XML`, + no_of_rooms:150,created_at:'2026-10-08T01:00:00Z',finished_at:'2026-10-08T01:00:52Z', + daily_report_sha256:'f'.repeat(64),is_current:true,is_retired:false,can_delete:true,...extra}); +const day=(current=job('current'),latest=current,extra={})=>({business_date:date,current_job:current,latest_job:latest,history_count:1,...extra}); +const page=(items,total=items.length)=>({status:200,ok:true,json:async()=>({ok:true,data:items,pagination:{total}})}); +const button=(id='current',current=true)=>({dataset:{dailyDelete:id,dailyDate:date,dailyCurrent:String(current)},isConnected:true,focus(){}}); +const plain=value=>JSON.parse(JSON.stringify(value)); +function setup(respond=()=>{throw new Error('Unexpected request');}) { + const h=harness(respond); + h.state.health.download_ready=true; + h.state.dailyOverviewDate=date; + h.state.selectedJobId='current'; + h.element('#arr-download-date').value=date; + return h; +} + +test('overview follows the selected incomplete date and never borrows another successful date',()=>{ + const h=setup(); + h.state.dailyReports=[day(job('september17',{arrival_date:'2026-09-17'}),undefined,{business_date:'2026-09-17'})]; + h.state.arrDownloadTask={request_id:'a'.repeat(32),report_date:date,status:'needs_data_review'}; + h.renderDailyOverview(); + assert.equal(h.element('#metric-arrival').textContent,date); + assert.equal(h.element('#metric-arrival-note').textContent,'daily.overview_review'); + assert.equal(h.element('#metric-rooms').textContent,'—'); + assert.equal(h.element('#metric-duration').textContent,'—'); + assert.equal(h.element('#metric-current-version').hidden,true); + assert.equal(h.calls.length,0); +}); + +test('a same-date pending replacement preserves the accepted day and hides ungenerated metrics',()=>{ + const h=setup(); + const pending=job('new',{status:'needs_review',is_current:false,no_of_rooms:0,daily_report_sha256:null,review_case_id:'case',review_total_items:1}); + h.state.dailyReports=[day(job('old'),pending)]; + h.renderJobs(); + const html=h.element('#jobs-body').innerHTML; + assert.equal((html.match(//g)||[]).length,1); + assert.match(html,/daily.current/); + assert.match(html,/daily.replacement/); + assert.match(html,/data-open-price-review="new"/); + assert.match(html,/job_id=old/); + assert.doesNotMatch(html,/job_id=new/); + assert.equal(h.element('#metric-arrival').textContent,date); + assert.equal(h.element('#metric-rooms').textContent,'—'); + assert.equal(h.element('#metric-duration').textContent,'—'); + assert.equal(h.element('#metric-current-version').hidden,false); + assert.doesNotMatch(html,/]+(?:tabindex|onclick|role="button")/); +}); + +test('a successful replacement displays only the accepted new report for the same day',()=>{ + const h=setup(); + h.state.dailyReports=[day(job('new',{no_of_rooms:43}))]; + h.renderJobs(); + assert.equal(h.element('#metric-rooms').textContent,'43'); + assert.equal(h.element('#metric-duration').textContent,'52s'); + assert.equal(h.element('#metric-current-version').hidden,true); + assert.match(h.element('#jobs-body').innerHTML,/job_id=new/); + assert.doesNotMatch(h.element('#jobs-body').innerHTML,/job_id=old/); +}); + +test('a failed replacement never removes the valid old report',()=>{ + const h=setup(); + h.state.dailyReports=[day(job('old'),job('new',{status:'failed',is_current:false,daily_report_sha256:null}))]; + h.renderJobs(); + assert.equal(h.element('#metric-rooms').textContent,'150'); + assert.equal(h.element('#metric-current-version').hidden,false); + assert.match(h.element('#jobs-body').innerHTML,/job_id=old/); + assert.match(h.element('#jobs-body').innerHTML,/status.failed/); +}); + +test('deleting the current report leaves older successful versions explicitly historical instead of completed current reports',()=>{ + const h=setup(); + const historical=job('superseded',{is_current:false}); + h.state.dailyReports=[day(null,historical,{history_count:2})]; + h.renderJobs(); + const html=h.element('#jobs-body').innerHTML; + assert.match(html,/daily.no_current/); + assert.match(html,/daily.history_only/); + assert.doesNotMatch(html,/status\.completed|daily\.current|download-link/); + assert.match(html,/data-daily-history="2026-09-16"/); + assert.match(html,/data-daily-delete="superseded"[^>]+data-daily-current="false"/); + assert.equal(h.element('#metric-arrival-note').textContent,'daily.no_current'); + assert.equal(h.element('#metric-rooms').textContent,'—'); + assert.equal(h.element('#metric-duration').textContent,'—'); +}); + +test('an out-of-page date reads its own overview and ignores stale and cross-context replies',async()=>{ + let resolve; + const h=setup(()=>new Promise(r=>{resolve=r;})); + const pending=h.loadDailyOverview(date); + h.state.dailyOverviewDate='2026-10-07'; + resolve(response(200,day())); + await pending; + assert.equal(h.state.dailyOverview,null); + h.state.dailyOverviewDate=date; + const other=h.loadDailyOverview(date); + h.state.arrDownloadContextId='another-hotel'; + resolve(response(200,day())); + await other; + assert.equal(h.state.dailyOverview,null); + const current=h.loadDailyOverview(date); + resolve(response(200,day())); + await current; + assert.equal(h.element('#metric-arrival').textContent,date); + assert.equal(h.element('#metric-rooms').textContent,'150'); + assert.equal(h.calls.every(c=>c.method==='GET'),true); +}); + +test('date pagination uses grouped API results and preserves job objects for traces',async()=>{ + const h=setup(url=>{ + assert.equal(url,'/api/daily-reports?month=2026-09&limit=50&offset=0'); + return page([day(job('old'),job('new',{status:'needs_review',is_current:false}))],31); + }); + h.state.jobsMonth='2026-09'; + await h.loadJobs(false,false); + assert.equal(h.state.jobsTotal,31); + assert.equal(h.state.dailyReports.length,1); + assert.deepEqual(plain(h.state.jobs.map(j=>j.job_id)),['old','new']); + assert.equal(h.calls.length,1); +}); + +test('history distinguishes old, current and deleted records without exposing missing filenames',async()=>{ + const h=setup(url=>{ + assert.equal(url,`/api/daily-reports/${date}/history?limit=50&offset=0`); + return page([job('current'),job('old',{is_current:false,filename:'.XML'}), + job('deleted',{is_current:false,is_retired:true,status:'deleted',can_delete:false,filename:null})]); + }); + await h.openDailyHistory(date,button()); + const html=h.element('#daily-history-body').innerHTML; + assert.match(html,/daily.current/); + assert.match(html,/daily.historical/); + assert.match(html,/daily.deleted/); + assert.match(html,/<old>.XML/); + assert.match(html,/data-daily-delete="old"/); + assert.doesNotMatch(html,/data-daily-delete="deleted"|job_id=deleted/); + assert.doesNotMatch(html,/deleted\.XML|null|undefined/); +}); + +function deletionSetup({removed=true,fail=false,wait=false}={}) { + let release; + const next=removed?day(null,null,{history_count:2}):day(job('new')); + const h=setup((url,options)=>{ + if(options.method==='DELETE') { + assert.equal(url,'/api/jobs/current'); + assert.deepEqual(JSON.parse(options.body),{business_date:date}); + if(fail) return response(409,null,'DAILY_JOB_BUSY'); + const value=response(200,{job_id:'current',business_date:date,status:'deleted',current_removed:removed}); + return wait?new Promise(resolve=>{release=()=>resolve(value);}):value; + } + if(url.startsWith('/api/daily-reports?')) return page([next]); + if(url===`/api/daily-reports/${date}`) return response(200,next); + if(url==='/api/history-months') return response(200,[]); + throw new Error('Unexpected request '+url); + }); + h.state.dailyReports=[day()]; + h.state.jobsMonth='2026-09'; + return {h,release:()=>release()}; +} + +test('deleting an exact current job clears only its pending review and refreshes the day without resurrecting history',async()=>{ + const {h}=deletionSetup(); + const task={request_id:'a'.repeat(32),report_date:date,status:'needs_review',job_id:'current',can_retry:false}; + h.state.arrDownloadTask=task; + h.state.arrDownloadPendingReviews=[task,{request_id:'b'.repeat(32),report_date:'2026-10-07',status:'needs_data_review'}]; + h.rememberARRIntent({request_id:task.request_id,report_date:date}); + h.state.dailyPriceReviewJobId='current'; + h.state.dailyPriceReview={case_status:'open',items:[]}; + h.openDailyDelete(button()); + assert.equal(h.element('#daily-delete-description').textContent,'daily.delete_current_description'); + await h.confirmDailyDelete(); + assert.equal(h.state.arrDownloadTask.status,'deleted'); + assert.equal(h.state.arrDownloadIntent,null); + assert.equal(h.storage.has('test'),false); + assert.equal(h.state.dailyPriceReviewJobId,''); + assert.equal(h.state.arrDownloadPendingReviews.length,1); + assert.equal(h.state.arrDownloadPendingReviews[0].report_date,'2026-10-07'); + assert.equal(h.state.dailyReports[0].current_job,null); + assert.equal(h.element('#metric-rooms').textContent,'—'); + assert.equal(h.element('#toast').textContent,'daily.deleted_current'); + const deletes=h.calls.filter(c=>c.method==='DELETE'); + assert.equal(deletes.length,1); + assert.equal(deletes[0].csrf,'fixture'); + assert.equal(h.calls.every(c=>['GET','DELETE'].includes(c.method)),true); +}); + +test('when a newer report wins during deletion, the old target alone is deleted and the actual outcome is shown',async()=>{ + const {h}=deletionSetup({removed:false}); + h.openDailyDelete(button()); + await h.confirmDailyDelete(); + assert.equal(h.state.dailyReports[0].current_job.job_id,'new'); + assert.equal(h.element('#toast').textContent,'daily.deleted_history'); + assert.match(h.element('#jobs-body').innerHTML,/job_id=new/); +}); + +test('failed deletion keeps the current report, same review and confirmation available for retry',async()=>{ + const {h}=deletionSetup({fail:true}); + h.openDailyDelete(button()); + await h.confirmDailyDelete(); + assert.equal(h.state.dailyDeleteRequest.jobId,'current'); + assert.equal(h.state.dailyDeleting,false); + assert.equal(h.state.dailyReports[0].current_job.job_id,'current'); + assert.equal(h.element('#daily-delete-error').hidden,false); + assert.equal(h.element('#daily-delete-error').textContent,'DAILY_JOB_BUSY'); + assert.equal(h.calls.length,1); +}); + +test('double confirmation while deletion is pending cannot send another mutation',async()=>{ + const {h,release}=deletionSetup({wait:true}); + h.openDailyDelete(button()); + const first=h.confirmDailyDelete(); + await h.confirmDailyDelete(); + h.closeDailyDelete(); + assert.equal(h.state.dailyDeleteRequest.jobId,'current'); + assert.equal(h.calls.filter(c=>c.method==='DELETE').length,1); + release(); + await first; + assert.equal(h.state.dailyDeleteRequest,null); +}); + +test('unsaved price edits prevent date changes and deletion instead of discarding human work',async()=>{ + const h=setup(); + h.state.dailyPriceReview={case_status:'open',items:[{item_id:1,real_price:null}]}; + h.element('#daily-price-review-body').querySelectorAll=()=>[{value:'0',closest:()=>({dataset:{dailyPriceReviewItemId:'1'}})}]; + h.openDailyDelete(button()); + assert.equal(h.state.dailyDeleteRequest,null); + await h.selectDailyReportDate('2026-10-07'); + assert.equal(h.element('#arr-download-date').value,date); + assert.equal(h.element('#toast').textContent,'daily_review.save_before_switch'); + assert.equal(h.calls.length,0); +}); + +test('deleted OHIP intent becomes terminal and permits a fresh explicit same-date request',async()=>{ + const h=setup((url,options)=>{ + assert.equal(url,'/api/arr-downloads'); + assert.equal(options.method,'POST'); + return response(202,{...JSON.parse(options.body),status:'queued',can_retry:false}); + }); + const old={request_id:'a'.repeat(32),report_date:date,status:'deleted',job_id:'current',can_retry:false,is_retired:true}; + await h.acceptARRDownloadTask(old,{sync:false}); + assert.equal(h.state.arrDownloadIntent,null); + assert.equal(h.state.arrDownloadPendingReviews.length,0); + assert.equal(h.element('#arr-download-review').hidden,true); + await h.submit(); + assert.equal(h.calls.length,1); + assert.notEqual(JSON.parse(h.calls[0].body).request_id,old.request_id); + assert.equal(JSON.parse(h.calls[0].body).report_date,date); +}); + +test('all new lifecycle labels and warnings are available in Chinese, English and Thai',()=>{ + const context=vm.createContext({window:{},document:{readyState:'loading',addEventListener(){}}}); + vm.runInContext(fs.readFileSync(path.resolve(__dirname,'../../arr_web/static/i18n.js'),'utf8'),context); + const catalog=context.window.ARRI18n.CATALOG; + for(const key of ['daily.current','daily.no_current','daily.history_only','daily.current_preserved','daily.overview_review', + 'daily.overview_fetching','daily.overview_processing','daily.after_generation','daily.replacement','daily.history_versions', + 'daily.history_title','daily.history_dialog_note','daily.history_region','daily.history_pagination','daily.delete_title', + 'daily.delete_current_description','daily.delete_history_description','daily.delete_note','daily.deleted_current', + 'daily.deleted_history','arr_download.deleted','error.daily_delete_busy']) { + assert.equal(catalog[key]?.length,3,key); + assert.equal(catalog[key].every(Boolean),true,key); + assert.match(catalog[key][2],/[\u0e00-\u0e7f]/,key); + } +}); + +for(const scope of ['month','context']) { + test(`a late daily list response after changing ${scope} is ignored and the current selection is loaded`,async()=>{ + let resolveOld; + const nextDate='2026-10-07'; + const h=setup((url,options,calls)=>calls.length===1 ? new Promise(resolve=>{resolveOld=resolve;}) + : page([day(job('fresh',{arrival_date:nextDate}),undefined,{business_date:nextDate})])); + h.state.jobsMonth='2026-09'; + const renders=[]; + Object.defineProperty(h.element('#jobs-body'),'innerHTML',{get(){return renders.at(-1)||'';},set(value){renders.push(value);}}); + const loading=h.loadJobs(false,false); + if(scope==='month') h.state.jobsMonth='2026-10'; + else h.state.arrDownloadContextId='another-hotel'; + resolveOld(page([day(job('stale'))])); + await loading; + assert.equal(h.calls.length,2); + assert.equal(h.state.dailyReports[0].current_job.job_id,'fresh'); + assert.equal(renders.some(html=>html.includes('stale.XML')),false); + if(scope==='month') assert.match(h.calls[1].url,/month=2026-10/); + }); +} diff --git a/tests/javascript/helpers/arr_ui_harness.cjs b/tests/javascript/helpers/arr_ui_harness.cjs index 852fe81..38aa3c0 100644 --- a/tests/javascript/helpers/arr_ui_harness.cjs +++ b/tests/javascript/helpers/arr_ui_harness.cjs @@ -26,13 +26,13 @@ function harness(respond) { }; const context=vm.createContext({Headers, console, Date, Intl, Uint8Array, document:{hidden:false,querySelector:element}, - window:{ARRI18n:{t:(key,values)=>key==='manual_review.context'?`${values.date} · ${values.type}`:key,text:value=>value,errorMessage:code=>code,formatInteger:value=>String(value),formatMonth:value=>value},crypto:webcrypto, + window:{ARRI18n:{t:(key,values)=>key==='manual_review.context'?`${values.date} · ${values.type}`:key,text:value=>value,errorMessage:code=>code,formatInteger:value=>String(value),formatMonth:value=>value,formatDate:value=>value,formatDurationSeconds:value=>`${value}s`},crypto:webcrypto, setTimeout(callback,delay){const timer={id:timers.length+1,callback,delay,cancelled:false};timers.push(timer);return timer.id;}, clearTimeout(id){const timer=timers.find(entry=>entry.id===id);if(timer) timer.cancelled=true;},location:{replace(){}}}, localStorage:{setItem:(k,v)=>storage.set(k,v),getItem:k=>{storageReads.push(k);return storage.get(k);},removeItem:k=>storage.delete(k)}, fetch:async (url, options) => {calls.push({url,method:options.method||'GET',body:options.body,csrf:options.headers.get('X-ARR-CSRF')});return respond(url,options,calls);}, }); - const exports='state, api, submitARRDownload, rememberARRIntent, acceptARRDownloadTask, handleARRDownloadDateChange, initARRDownload, loadARRDownloadTask, loadARRDataReview, renderARRDataReview, arrDataReviewCanFinalize, saveARRDataReviewItem, finalizeARRDataReview, trackARRDataReviewDraft, parseARRDataReviewValue, selectARRPendingReview, strictReviewPrice, reviewPriceInput, renderDailyPriceReview, saveDailyPriceReviewItem, loadDailyPriceReview, openDailyPriceReview, dailyPriceReviewCanFinalize, dailyPriceReviewHasUnsavedChanges, finalizeDailyPriceReview, confirmDailyPriceReviewCancel'; + const exports='state, api, submitARRDownload, rememberARRIntent, acceptARRDownloadTask, handleARRDownloadDateChange, initARRDownload, loadARRDownloadTask, loadARRDataReview, renderARRDataReview, arrDataReviewCanFinalize, saveARRDataReviewItem, finalizeARRDataReview, trackARRDataReviewDraft, parseARRDataReviewValue, selectARRPendingReview, strictReviewPrice, reviewPriceInput, renderDailyPriceReview, saveDailyPriceReviewItem, loadDailyPriceReview, openDailyPriceReview, dailyPriceReviewCanFinalize, dailyPriceReviewHasUnsavedChanges, finalizeDailyPriceReview, confirmDailyPriceReviewCancel, renderJobs, renderDailyOverview, syncDailyOverviewDate, loadDailyOverview, selectDailyReportDate, loadJobs, loadDailyHistory, openDailyHistory, closeDailyHistory, openDailyDelete, closeDailyDelete, confirmDailyDelete'; vm.runInContext(source.replace(boot,` globalThis.subject = {${exports}};\n})();`),context); const subject=context.subject; Object.assign(subject.state,{arrDownloadReady:true,arrDownloadLoaded:true,arrDownloadStorageKey:'test',arrDownloadUsername:'operator',csrf:'fixture'}); diff --git a/tests/local_postgres.py b/tests/local_postgres.py index 5d66697..04048cb 100644 --- a/tests/local_postgres.py +++ b/tests/local_postgres.py @@ -77,7 +77,7 @@ class TemporaryPostgres: raise def reset_database(self, *, schema_version=21): - if schema_version not in (19, 20, 21): + if schema_version not in (19, 20, 21, 22): raise ValueError("unsupported fixture schema version") # Only this owned cluster has passed data_directory/socket checks above. with self.connect(database="postgres", autocommit=True) as connection: diff --git a/tests/test_arr_daily_report_api.py b/tests/test_arr_daily_report_api.py new file mode 100644 index 0000000..bcbd5f0 --- /dev/null +++ b/tests/test_arr_daily_report_api.py @@ -0,0 +1,120 @@ +"""HTTP identity, authorization and no-side-effect boundaries for daily retirement.""" +import json +from datetime import date +from unittest import TestCase +from unittest.mock import Mock + +from arr_ingestion.contracts import IngestionError +from arr_web.app import PortalApplication +from arr_web.daily_reports import ProgramDailyReports +from arr_web.local_xml_replay import LocalReplayPortal +from tests.test_arr_web import TEST_CREDENTIALS, decoded, login + + +class DailyReportApiTests(TestCase): + def setUp(self): + self.repository = Mock() + self.retirement = Mock() + self.app = PortalApplication(repository=self.repository, daily_reports=self.retirement, + credentials=TEST_CREDENTIALS) + _, self.headers = login(self.app) + self.day = "2026-09-16" + self.group = {"business_date": self.day, "current_job": None, + "latest_job": None, "history_count": 0} + self.repository.list_daily_reports.return_value = ([self.group], 1) + self.repository.get_daily_report.return_value = self.group + self.repository.list_daily_report_history.return_value = ([], 0) + + def delete(self, payload=None, headers=None): + return self.app.handle("DELETE", "/api/jobs/arrjob-synthetic", + self.headers if headers is None else headers, + json.dumps(payload or {"business_date": self.day}).encode()) + + def test_grouped_pagination_counts_dates_and_preserves_single_day_context(self): + response = self.app.handle("GET", "/api/daily-reports?month=2026-09&limit=2&offset=4", self.headers) + self.assertEqual(response.status, 200) + self.assertEqual(decoded(response)["data"], [self.group]) + self.repository.list_daily_reports.assert_called_once_with("2026-09", 2, 4) + result = self.app.handle("GET", "/api/daily-reports/2026-09-16", self.headers) + self.assertEqual(decoded(result)["data"], self.group) + history = self.app.handle("GET", "/api/daily-reports/2026-09-16/history?limit=5&offset=10", self.headers) + self.assertEqual(history.status, 200) + self.repository.list_daily_report_history.assert_called_once_with(self.day, 5, 10) + + def test_all_new_reads_and_delete_require_login(self): + for path in ("/api/daily-reports", "/api/daily-reports/2026-09-16", + "/api/daily-reports/2026-09-16/history"): + self.assertEqual(self.app.handle("GET", path, {}).status, 401) + self.assertEqual(self.delete(headers={}).status, 401) + self.retirement.retire.assert_not_called() + + def test_delete_requires_csrf_and_exact_date_body(self): + headers = {key: value for key, value in self.headers.items() if key.lower() != "x-arr-csrf"} + self.assertEqual(self.delete(headers=headers).status, 403) + for payload in ({"business_date": self.day, "all": True}, {"job_id": "other"}, + {"business_date": "2026-09-31"}, {"business_date": None}): + self.assertGreaterEqual(self.delete(payload).status, 400) + self.retirement.retire.assert_not_called() + + def test_delete_binds_route_job_date_and_authenticated_actor(self): + self.retirement.retire.return_value = {"job_id": "arrjob-synthetic", "business_date": self.day, + "status": "deleted", "removed_from_month": True} + result = self.delete() + self.assertEqual(result.status, 200) + self.retirement.retire.assert_called_once_with("arrjob-synthetic", self.day, TEST_CREDENTIALS.username) + self.assertTrue(decoded(result)["data"]["removed_from_month"]) + + def test_unconfigured_mutation_is_unavailable_without_affecting_reads(self): + app = PortalApplication(repository=self.repository, credentials=TEST_CREDENTIALS) + _, headers = login(app) + result = app.handle("DELETE", "/api/jobs/arrjob-synthetic", headers, + json.dumps({"business_date": self.day}).encode()) + self.assertEqual(result.status, 503) + self.assertEqual(decoded(result)["error"]["code"], "DAILY_RETIREMENT_UNAVAILABLE") + + def test_get_requests_never_retire_and_invalid_reads_are_rejected(self): + for path in ("/api/daily-reports?month=wrong", "/api/daily-reports/2026-09-31", + "/api/daily-reports/2026-09-16/history?limit=0"): + self.assertGreaterEqual(self.app.handle("GET", path, self.headers).status, 400) + self.retirement.retire.assert_not_called() + + def test_deleted_ohip_job_converges_without_requery_or_retry(self): + self.repository.get_job_trace.return_value = {"job": {"status": "deleted"}} + for status in ("needs_review", "succeeded"): + task = {"request_id": "a" * 32, "report_date": self.day, + "job_id": "arrjob-synthetic", "status": status, "can_retry": True} + result = self.app._arr_download_snapshot(task) + self.assertEqual(result["status"], "deleted") + self.assertFalse(result["can_retry"]) + self.retirement.retire.assert_not_called() + + def test_local_wrapper_allows_only_exact_authenticated_delete_route(self): + portal = LocalReplayPortal(self.app, Mock(), 8875) + headers = {**self.headers, "host": "127.0.0.1:8875", "origin": "http://127.0.0.1:8875"} + headers["Cookie"] = headers["Cookie"].replace("arr_session=", portal.cookie_name + "=") + self.retirement.retire.return_value = {"status": "deleted"} + result = portal.handle("DELETE", "/api/jobs/arrjob-synthetic", headers, + json.dumps({"business_date": self.day}).encode()) + self.assertEqual(result.status, 200) + self.assertEqual(portal.handle("DELETE", "/api/jobs", headers).status, 405) + + +class DailyRetirementCoordinatorTests(TestCase): + def test_receipt_reports_actual_current_removal_after_concurrent_replacement(self): + repository = Mock() + repository.retire_daily_job.return_value = {"current_removed": False, "is_retired": True} + result = ProgramDailyReports(repository).retire("job-synthetic", "2026-09-16", "operator") + repository.retire_daily_job.assert_called_once_with("job-synthetic", date(2026, 9, 16), "operator") + self.assertEqual(result["status"], "deleted") + self.assertFalse(result["removed_from_month"]) + + def test_busy_or_changed_job_is_conflict_missing_job_404_database_503(self): + for code, status in (("DAILY_JOB_BUSY", 409), ("DAILY_DATE_CONFLICT", 409), + ("DAILY_JOB_NOT_FOUND", 404), ("DATABASE_WRITE_FAILED", 503)): + with self.subTest(code=code): + repository = Mock() + repository.retire_daily_job.side_effect = IngestionError(code, "private detail") + with self.assertRaises(Exception) as caught: + ProgramDailyReports(repository).retire("job-synthetic", "2026-09-16", "operator") + self.assertEqual(caught.exception.status, status) + self.assertNotIn("private detail", str(caught.exception)) diff --git a/tests/test_arr_daily_report_reads_postgres.py b/tests/test_arr_daily_report_reads_postgres.py new file mode 100644 index 0000000..70ad3d6 --- /dev/null +++ b/tests/test_arr_daily_report_reads_postgres.py @@ -0,0 +1,227 @@ +"""Date-grouped portal reads against owned disposable PostgreSQL only.""" +from datetime import date, timedelta +import os +from pathlib import Path +import tempfile +import unittest + +from arr_ingestion.postgres import DatabaseConfig as IngestionConfig, PostgresIngestionRepository +from arr_ingestion.service import IngestionService +from arr_ingestion.validation import DeliveryValidator +from arr_processing.local import LocalDailyProcessor +from arr_processing.policy import load_processor_policy +from arr_storage.filesystem import FilesystemObjectBackend +from arr_storage.store import ManagedObjectStore +from arr_web.programmatic import ProgrammaticUploadCoordinator +from arr_web.repository import PortalDataError, PostgresPortalRepository +from monthly_reports.publishing import AtomicReportPublisher, OpenpyxlWorkbookBuilder +from monthly_reports.repository import DatabaseConfig, PostgresReportRepository +from monthly_reports.service import MonthlyReportService +from monthly_reports.worker import MonthlyOutboxWorker, PostgresOutboxRepository +from tests.local_postgres import TemporaryPostgres +from tests.test_arr_opera_daily_ingest import reservation, xml_document + + +PROJECT = Path(__file__).resolve().parents[1] +DAY = date(2026, 7, 2) + + +@unittest.skipUnless(os.environ.get("ARR_TEST_LOCAL_POSTGRES") == "1", "owned disposable PostgreSQL opt-in required") +class DailyReportReadPostgresTests(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.database = TemporaryPostgres().__enter__() + cls.addClassCleanup(cls.database.__exit__, None, None, None) + cls.policy = load_processor_policy(PROJECT) + + def setUp(self): + self.database.reset_database(schema_version=22) + temporary = tempfile.TemporaryDirectory(prefix="arr-daily-read-fixture-") + self.addCleanup(temporary.cleanup) + self.root = Path(temporary.name) + self.store = ManagedObjectStore(FilesystemObjectBackend(self.root / "objects", create=True)) + self.ingestion = PostgresIngestionRepository(IngestionConfig("owned-fixture"), connect=self.database.connect) + service = IngestionService(DeliveryValidator(self.store, self.policy), self.ingestion) + self.uploads = ProgrammaticUploadCoordinator(self.store, self.ingestion, service, LocalDailyProcessor(self.policy), + self.policy.processor_version, self.policy.rule_set_sha256) + self.portal = PostgresPortalRepository("owned-fixture", connect=self.database.connect) + config = DatabaseConfig("owned-fixture") + reports = PostgresReportRepository(config, connect=self.database.connect) + output = self.root / "outputs/monthly" + monthly = MonthlyReportService(reports, OpenpyxlWorkbookBuilder(), + AtomicReportPublisher(self.root, output, object_store=self.store), output / ".staging") + self.worker = MonthlyOutboxWorker(PostgresOutboxRepository(config, connect=self.database.connect), reports, monthly) + + def sql(self, query, parameters=None): + with self.database.connect() as connection: + cursor = connection.execute(query, parameters) + return cursor.fetchall() if cursor.description else [] + + def upload(self, label, day=DAY, **changes): + xml = xml_document(reservation(1, confirmation=f"SYN-{label}", room=f"SYN-ROOM-{label}", **changes)) + xml = (xml.replace("2026-07-27", day.isoformat()) + .replace("2026-07-28", (day + timedelta(days=1)).isoformat()) + .replace("20260727", day.strftime("%Y%m%d")) + .replace("27-07-26", day.strftime("%d-%m-%y"))) + return self.uploads.submit(f"synthetic-{label}.XML", xml.encode()) + + def accepted(self, label, day=DAY): + result = self.upload(label, day) + self.assertEqual(result["status"], "succeeded", result) + return result + + def retire(self, uploaded): + return self.ingestion.retire_daily_job(uploaded["job_id"], date.fromisoformat(uploaded["business_date"]), "synthetic-operator") + + def drain(self): + for _attempt in range(20): + result = self.worker.process_next() + if result.status == "idle": + return + self.assertIn(result.status, {"published", "withdrawn"}, result) + self.fail("synthetic monthly worker did not settle") + + def assert_download_missing(self, callback): + with self.assertRaises(PortalDataError) as raised: + callback() + self.assertEqual(raised.exception.code, "DOWNLOAD_NOT_FOUND") + + def test_one_date_group_keeps_current_success_separate_from_latest_pending_then_failed_replacement(self): + accepted = self.accepted("current") + pending = self.upload("pending", company="SYNTHETIC UNPRICED COMPANY") + self.assertEqual(pending["status"], "needs_review") + groups, total = self.portal.list_daily_reports("2026-07") + self.assertEqual((total, len(groups)), (1, 1)) + group = groups[0] + self.assertEqual(group["business_date"], DAY.isoformat()) + self.assertEqual(group["current_job"]["job_id"], accepted["job_id"]) + self.assertTrue(group["current_job"]["is_current"]) + self.assertEqual(group["latest_job"]["job_id"], pending["job_id"]) + self.assertEqual(group["latest_job"]["status"], "needs_review") + self.assertFalse(group["latest_job"]["is_current"]) + self.assertIsNone(group["latest_job"]["daily_report_sha256"]) + self.assertEqual(group["history_count"], 1) + failed = self.upload("failed", departure="2026-07-01") + self.assertEqual(failed["status"], "failed") + selected = self.portal.get_daily_report(DAY.isoformat()) + self.assertEqual(selected["current_job"]["job_id"], accepted["job_id"]) + self.assertEqual((selected["latest_job"]["job_id"], selected["latest_job"]["status"]), (failed["job_id"], "failed")) + self.assertEqual(selected["history_count"], 2) + self.assertEqual(self.portal.resolve_daily_download(accepted["job_id"]).sha256, + selected["current_job"]["daily_report_sha256"]) + self.assert_download_missing(lambda: self.portal.resolve_daily_download(pending["job_id"])) + + def test_deleting_an_old_success_preserves_current_and_monthly_but_blocks_old_download(self): + old = self.accepted("old") + current = self.accepted("current") + self.drain() + monthly_before = self.portal.list_monthly_runs("2026-07") + result = self.retire(old) + self.assertFalse(result["current_removed"]) + self.assertFalse(result["monthly_refresh_required"]) + selected = self.portal.get_daily_report(DAY.isoformat()) + self.assertEqual(selected["current_job"]["job_id"], current["job_id"]) + self.assertEqual(selected["latest_job"]["job_id"], current["job_id"]) + self.assertEqual(self.portal.list_monthly_runs("2026-07"), monthly_before) + history, total = self.portal.list_daily_report_history(DAY.isoformat()) + self.assertEqual(total, 2) + deleted = next(job for job in history if job["job_id"] == old["job_id"]) + self.assertEqual(deleted["status"], "deleted") + self.assertTrue(deleted["is_retired"]) + self.assertFalse(deleted["is_current"]) + self.assertFalse(deleted["can_delete"]) + self.assertIsNone(deleted["daily_report_sha256"]) + self.assert_download_missing(lambda: self.portal.resolve_daily_download(old["job_id"])) + self.assertIsNotNone(self.portal.resolve_daily_download(current["job_id"])) + + def test_deleting_current_does_not_turn_old_history_into_an_effective_report(self): + old = self.accepted("old") + current = self.accepted("current") + self.retire(current) + selected = self.portal.get_daily_report(DAY.isoformat()) + self.assertIsNone(selected["current_job"]) + self.assertFalse(selected["latest_job"]["is_current"]) + self.assertEqual(selected["latest_job"]["job_id"], old["job_id"]) + history, total = self.portal.list_daily_report_history(DAY.isoformat()) + self.assertEqual(total, 2) + self.assertFalse(any(job["is_current"] for job in history)) + deleted = next(job for job in history if job["job_id"] == current["job_id"]) + self.assertTrue(deleted["is_retired"]) + self.assertFalse(deleted["can_delete"]) + self.assertIsNone(deleted["daily_report_sha256"]) + self.assertEqual(self.sql("SELECT count(*) FROM finance.current_daily_versions"), [(0,)]) + self.assertEqual(self.portal.list_months(), []) + self.assert_download_missing(lambda: self.portal.resolve_daily_download(current["job_id"])) + + def test_empty_month_hides_withdrawn_publications_counts_old_download_and_bi(self): + uploaded = self.accepted("only") + self.drain() + monthly, total = self.portal.list_monthly_runs("2026-07") + self.assertEqual(total, 1) + report_id = monthly[0]["report_id"] + self.assertIsNotNone(self.portal.resolve_monthly_download(report_id)) + self.assertEqual(len(self.portal.list_months()), 1) + self.retire(uploaded) + # Visibility changes with retirement; the asynchronous ack is not needed + # before old artifacts and analytics disappear from the portal. + self.assertEqual(self.portal.list_monthly_runs("2026-07"), ([], 0)) + history_months = self.portal.list_history_month_counts() + self.assertEqual(history_months[0]["monthly_count"], 0) + self.assertEqual(self.portal.list_months(), []) + self.assert_download_missing(lambda: self.portal.resolve_monthly_download(report_id)) + with self.assertRaises(PortalDataError) as raised: + self.portal.read_dashboard("2026-07") + self.assertEqual(raised.exception.code, "ANALYTICS_MONTH_NOT_FOUND") + self.assertEqual(self.portal.list_daily_reports("2026-07"), ([], 0)) + selected = self.portal.get_daily_report(DAY.isoformat()) + self.assertIsNone(selected["current_job"]) + self.assertIsNone(selected["latest_job"]) + history, total = self.portal.list_daily_report_history(DAY.isoformat()) + self.assertEqual((total, history[0]["status"]), (1, "deleted")) + self.drain() + self.assertEqual(self.portal.list_monthly_runs("2026-07"), ([], 0)) + + def test_empty_selected_day_and_history_return_date_context_without_a_fabricated_report(self): + selected = self.portal.get_daily_report("2026-07-19") + self.assertEqual(selected, {"business_date": "2026-07-19", "current_job": None, "latest_job": None, "history_count": 0}) + self.assertEqual(self.portal.list_daily_report_history("2026-07-19"), ([], 0)) + self.assertEqual(self.portal.list_daily_reports("2026-07"), ([], 0)) + + def test_selected_date_is_independent_of_latest_upload_and_date_pagination_counts_days(self): + first = self.accepted("first", date(2026, 7, 2)) + second = self.accepted("second", date(2026, 7, 3)) + self.accepted("first-replaced", date(2026, 7, 2)) + groups, total = self.portal.list_daily_reports("2026-07", limit=1) + self.assertEqual(total, 2) + self.assertEqual(groups[0]["business_date"], "2026-07-03") + self.assertEqual(groups[0]["current_job"]["job_id"], second["job_id"]) + selected = self.portal.get_daily_report("2026-07-03") + self.assertEqual(selected["current_job"]["job_id"], second["job_id"]) + history, count = self.portal.list_daily_report_history("2026-07-02", limit=1, offset=1) + self.assertEqual(count, 2) + self.assertEqual(history[0]["job_id"], first["job_id"]) + + def test_date_views_work_under_a_select_only_role_that_cannot_retire_reports(self): + uploaded = self.accepted("read-role") + self.retire(uploaded) + self.sql("CREATE ROLE arr_read_lifecycle_fixture NOLOGIN") + self.sql("GRANT USAGE ON SCHEMA ingestion, finance, reporting TO arr_read_lifecycle_fixture") + self.sql("GRANT SELECT ON ALL TABLES IN SCHEMA ingestion, finance, reporting TO arr_read_lifecycle_fixture") + def read_connection(_dsn): + connection = self.database.connect() + connection.execute("SET ROLE arr_read_lifecycle_fixture") + connection.commit() + return connection + portal = PostgresPortalRepository("owned-select-only", connect=read_connection) + self.assertEqual(portal.list_daily_reports("2026-07"), ([], 0)) + history, total = portal.list_daily_report_history(DAY.isoformat()) + self.assertEqual((total, history[0]["status"]), (1, "deleted")) + self.assertIsNone(portal.get_daily_report(DAY.isoformat())["current_job"]) + import psycopg + with read_connection(None) as connection: + with self.assertRaises(psycopg.errors.InsufficientPrivilege): + connection.execute("DELETE FROM finance.current_daily_versions") + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_arr_download_runtime.py b/tests/test_arr_download_runtime.py index 80ae040..52b8357 100644 --- a/tests/test_arr_download_runtime.py +++ b/tests/test_arr_download_runtime.py @@ -42,6 +42,7 @@ class RuntimeTests(unittest.TestCase): self.repository = InMemoryIngestionRepository() self.repository.assert_cancelled_scope_schema = Mock() self.repository.assert_pm_scope_schema = Mock() + self.repository.assert_daily_lifecycle_schema = Mock() self.processor = CountingProcessor(self.policy) self.client = Mock() with patch.object(processing_runtime, 'compose_object_store', return_value= @@ -61,6 +62,23 @@ class RuntimeTests(unittest.TestCase): def test_processing_composition_checks_finance_cancellation_schema(self): self.repository.assert_cancelled_scope_schema.assert_called_once_with() self.repository.assert_pm_scope_schema.assert_called_once_with() + self.repository.assert_daily_lifecycle_schema.assert_called_once_with() + + def test_missing_daily_lifecycle_migration_closes_storage_before_enabling_processing(self): + from arr_ingestion.contracts import IngestionError + self.repository.assert_daily_lifecycle_schema.side_effect = IngestionError( + "DATABASE_MIGRATION_MISSING", "ARR daily report lifecycle requires migration 022") + client = Mock() + with patch.object(processing_runtime, 'compose_object_store', return_value= + processing_runtime.ObjectStoreRuntime(client, self.store)), \ + patch.object(processing_runtime, 'PostgresIngestionRepository', return_value=self.repository), \ + patch.object(processing_runtime, 'LocalDailyProcessor') as processor, \ + self.assertRaises(IngestionError) as error: + processing_runtime.compose_programmatic_processing( + project_root=Path(__file__).resolve().parents[1], connect=Mock()) + self.assertEqual(error.exception.code, "DATABASE_MIGRATION_MISSING") + processor.assert_not_called() + client.close.assert_called_once_with() def test_missing_pm_migration_closes_storage_and_disables_composition(self): from arr_ingestion.contracts import IngestionError diff --git a/tests/test_arr_local_api_simulation.py b/tests/test_arr_local_api_simulation.py index 8bb40e2..15f4dba 100644 --- a/tests/test_arr_local_api_simulation.py +++ b/tests/test_arr_local_api_simulation.py @@ -236,6 +236,9 @@ class LocalAPISQLTests(unittest.TestCase): self.assertEqual(response.status, 202) result = native_tests.ReplaySQLTests.wait(runtime.queue) self.assertEqual(result["status"], "succeeded") + grouped = runtime.app.handle("GET", "/api/daily-reports?month=2026-07", headers) + self.assertEqual(grouped.status, 200) + self.assertEqual(json.loads(grouped.body)["data"][0]["current_job"]["job_id"], result["job_id"]) daily = runtime.app.handle("GET", "/api/download/daily?job_id=" + result["job_id"], headers) self.assertEqual(daily.status, 200) self.assertIn("LOCAL-API-SIMULATION-", daily.headers["Content-Disposition"]) diff --git a/tests/test_arr_local_xml_replay.py b/tests/test_arr_local_xml_replay.py index 4b6f2e1..dde8c7e 100644 --- a/tests/test_arr_local_xml_replay.py +++ b/tests/test_arr_local_xml_replay.py @@ -193,6 +193,9 @@ class ReplaySQLTests(unittest.TestCase): self.assertEqual(response.status, 202) result = self.wait(runtime.queue) self.assertEqual(result["status"], "succeeded") + grouped = runtime.app.handle("GET", "/api/daily-reports?month=2026-07", headers) + self.assertEqual(grouped.status, 200) + self.assertEqual(json.loads(grouped.body)["data"][0]["current_job"]["job_id"], result["job_id"]) daily = runtime.app.handle("GET", "/api/download/daily?job_id=" + result["job_id"], headers) self.assertEqual(daily.status, 200) self.assertIn("LOCAL-REPLAY-", daily.headers["Content-Disposition"]) diff --git a/tests/test_arr_web.py b/tests/test_arr_web.py index 930e476..d79b4e4 100644 --- a/tests/test_arr_web.py +++ b/tests/test_arr_web.py @@ -295,7 +295,7 @@ class PortalApplicationTests(unittest.TestCase): def test_daily_history_hides_internal_filename_when_provenance_is_missing(self) -> None: script = (STATIC_ROOT / "app.js").read_text(encoding="utf-8") - self.assertIn('job.filename || "—"', script) + self.assertIn('job?.filename || "—"', script) def session_headers(self) -> Dict[str, str]: return dict(self.auth_headers) diff --git a/tests/test_arr_web_daily_price_review_ui.py b/tests/test_arr_web_daily_price_review_ui.py index 8a8560f..d0ba1ab 100644 --- a/tests/test_arr_web_daily_price_review_ui.py +++ b/tests/test_arr_web_daily_price_review_ui.py @@ -19,11 +19,11 @@ class DailyPriceReviewUiContractTests(unittest.TestCase): cls.i18n = (STATIC_ROOT / "i18n.js").read_text(encoding="utf-8") def test_needs_review_status_is_the_review_action(self) -> None: - render_start = self.script.index("function renderJobs(jobs)") - render_end = self.script.index("function reviewPrice(value)", render_start) + render_start = self.script.index("function dailyJobStatus(job)") + render_end = self.script.index("function dailyJobDownload(job)", render_start) renderer = self.script[render_start:render_end] self.assertIn('needs_review: ["status.needs_review", "review"]', self.script) - self.assertIn('const statusMarkup = job.status === "needs_review"', renderer) + self.assertIn('return job.status === "needs_review" && job.review_case_id', renderer) self.assertIn('data-open-price-review=', renderer) self.assertIn('${chip(job.status, jobStatus)}', renderer) self.assertNotIn("const reviewProgress", renderer) diff --git a/tests/test_arr_web_preview.py b/tests/test_arr_web_preview.py index ba5ac86..3d6a092 100644 --- a/tests/test_arr_web_preview.py +++ b/tests/test_arr_web_preview.py @@ -36,6 +36,9 @@ class PreviewTests(unittest.TestCase): self.assertFalse(health[key]) self.assertFalse(get("/api/arr-downloads")["ready"]) self.assertEqual(get("/api/jobs?month=2026-09"), []) + self.assertEqual(get("/api/daily-reports?month=2026-09"), []) + self.assertIsNone(get("/api/daily-reports/2026-09-16")["current_job"]) + self.assertEqual(get("/api/daily-reports/2026-09-16/history"), []) self.assertEqual(get("/api/months"), []) def test_all_mutations_blocked_before_dispatch(self): diff --git a/tests/test_daily_report_retirement.py b/tests/test_daily_report_retirement.py new file mode 100644 index 0000000..2a1838e --- /dev/null +++ b/tests/test_daily_report_retirement.py @@ -0,0 +1,344 @@ +"""Recoverable daily retirement: real isolated PostgreSQL, synthetic rows only.""" +from concurrent.futures import ThreadPoolExecutor +from datetime import date +import hashlib +import json +import os +from pathlib import Path +import threading +import time +import unittest + +from arr_ingestion.contracts import ArtifactRef, IngestionError +from arr_ingestion.postgres import DatabaseConfig, PostgresIngestionRepository +from arr_ingestion.repository import JobRegistration +from tests.local_postgres import TemporaryPostgres + +PROJECT = Path(__file__).resolve().parents[1] +DAY = date(2026, 9, 16) +ACTOR = "fixture-user" + + +class RetirementMigrationContractTests(unittest.TestCase): + def test_additive_contract_and_no_fact_or_artifact_deletion_grants(self): + up = (PROJECT / "database/022_daily_report_retirement.sql").read_text() + down = (PROJECT / "database/022_daily_report_retirement.down.sql").read_text() + for text in ("current_database() <> 'booking_test'", "daily_run_retirements_immutable", + "monthly_runs_live_snapshot_unique", "WHERE report_status <> 'withdrawn'", + "'active', 'superseded', 'withdrawn'", "active non-retired daily version"): + self.assertIn(text, up) + self.assertIn("GRANT SELECT, INSERT ON ingestion.daily_run_retirements", up) + self.assertIn("GRANT DELETE ON finance.current_daily_versions", up) + self.assertNotIn("GRANT DELETE ON finance.daily_records", up) + self.assertNotIn("GRANT DELETE ON ingestion.artifacts", up) + self.assertIn("refusing rollback: daily retirement or withdrawn monthly audit exists", down) + + +@unittest.skipUnless(os.environ.get("ARR_TEST_LOCAL_POSTGRES") == "1", "owned PostgreSQL opt-in required") +class DailyRetirementPostgresTests(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.database = TemporaryPostgres().__enter__() + cls.addClassCleanup(cls.database.__exit__, None, None, None) + + def setUp(self): + self.database.reset_database(schema_version=22) + self.repository = PostgresIngestionRepository(DatabaseConfig("owned-fixture"), connect=self.database.connect) + + def sql(self, query, parameters=None): + with self.database.connect() as connection: + cursor = connection.execute(query, parameters) + return cursor.fetchall() if cursor.description else [] + + def seed(self, job, *, status="accepted", day=DAY, version=None, active=True): + sha = hashlib.sha256(job.encode()).hexdigest() + with self.database.connect() as connection: + source = connection.execute("""INSERT INTO ingestion.artifacts + (artifact_kind, storage_provider, bucket_alias, object_key, original_filename, sha256, byte_size, mime_type) + VALUES ('opera_xml','oss','arr-private',%s,'source.xml',%s,0,'application/xml') RETURNING id""", + (job + "/source.xml", sha)).fetchone()[0] + result = connection.execute("""INSERT INTO ingestion.artifacts + (artifact_kind, storage_provider, bucket_alias, object_key, original_filename, sha256, byte_size) + VALUES ('result_json','local_fixture','fixture',%s,'result.json',%s,0) RETURNING id""", + (job + "/result.json", sha)).fetchone()[0] + run = connection.execute("""INSERT INTO ingestion.processing_runs + (run_key,pipeline_type,source_artifact_id,result_artifact_id,run_status,business_date, + requested_processor_version,requested_rule_set_sha256,delivered_processor_version, + delivered_rule_set_sha256,result_schema_version,delivery_sha256,validated_at,finished_at,failure_code) + VALUES (%s,'opera_daily',%s,%s,%s,%s,'4.4.0',%s,'4.4.0',%s,'4.0',%s,now(), + CASE WHEN %s IN ('accepted','failed','rejected','cancelled') THEN now() END, + CASE WHEN %s IN ('failed','rejected') THEN 'FIXTURE_FAILED' END) RETURNING id""", + (job,source,result,status,day,sha,sha,sha,status,status)).fetchone()[0] + version_id = None + if version is not None: + outputs = [] + for kind in ("daily_xlsx", "structured_result_json"): + outputs.append(connection.execute("""INSERT INTO ingestion.artifacts + (artifact_kind,storage_provider,bucket_alias,object_key,original_filename,sha256,byte_size) + VALUES (%s,'local_fixture','fixture',%s,%s,%s,0) RETURNING id""", + (kind,job + "/" + kind,kind,sha)).fetchone()[0]) + version_id = connection.execute("""INSERT INTO finance.daily_versions + (business_date,version_no,processing_run_id,source_artifact_id,daily_report_artifact_id, + result_json_artifact_id,structured_result_artifact_id,version_status,processor_version, + rule_set_sha256,result_schema_version,result_sha256,source_rows,retained_rows, + excluded_rate_code_rows,duplicate_rows,validation_failed_rows,price_unmatched_rows,validated_at) + VALUES (%s,%s,%s,%s,%s,%s,%s,'validated','4.4.0',%s,'4.0',%s,1,1,0,0,0,0,now()) RETURNING id""", + (day,version,run,source,outputs[0],result,outputs[1],sha,sha)).fetchone()[0] + connection.execute("""INSERT INTO finance.daily_records + (daily_version_id,source_sequence,source_location,outcome,decision_codes,block_code, + adults,children,company_name,company_key,confirmation_no,disp_room_no,effective_rate_amount, + full_name,no_of_rooms,rate_code,arrival,departure,nights,real_price,total_price,channel_key, + pricing_method,booking_source_match_status) + VALUES (%s,1,'synthetic:1','retained',ARRAY['PRICE_REFERENCE_MATCHED'],'', + 2,0,'FIXTURE','FIXTURE',%s,%s,10,'SYNTHETIC',2,'WHO2',%s,%s,2,10,40,'FIT', + 'price_reference_exact','missing_group_code')""", + (version_id,job,job,day,day.replace(day=day.day+2))) + connection.execute("""INSERT INTO finance.daily_channel_metrics + (daily_version_id,channel_key,channel_order,row_count) VALUES (%s,'FIT',1,1)""",(version_id,)) + if active: + with connection.cursor() as cursor: + self.repository._activate_daily_version(cursor,day,version_id) + if status == "awaiting_review": + delivery = connection.execute("""INSERT INTO ingestion.processing_deliveries + (delivery_key,processing_run_id,envelope_sha256,envelope_json,delivery_status,result_status, + processor_version,rule_set_sha256,result_schema_version,business_date) + VALUES (%s,%s,%s,'{}','recorded_review','review_required','4.4.0',%s,'4.0',%s) RETURNING id""", + (job + "-delivery",run,sha,sha,day)).fetchone()[0] + case = connection.execute("""INSERT INTO ingestion.daily_review_cases + (case_key,processing_run_id,initial_delivery_id,business_date,source_sha256, + processor_version,rule_set_sha256,review_version,case_status) + VALUES (%s,%s,%s,%s,%s,'4.4.0',%s,'1.0','open') RETURNING id""", + ("dailyreview-" + sha[:32],run,delivery,day,sha,sha)).fetchone()[0] + connection.execute("""INSERT INTO ingestion.daily_review_items + (review_case_id,company_key,rate_code,effective_rate_amount,affected_records,affected_rooms,affected_room_nights) + VALUES (%s,'FIXTURE','WHO2',1000,1,1,2)""",(case,)) + return run,version_id,source + + def month(self, version_id, *, snapshot="f" * 64): + with self.database.connect() as connection: + report = connection.execute("""INSERT INTO reporting.monthly_runs + (period_start,as_of_date,version_no,source_snapshot_sha256,report_status,processor_version, + rule_set_sha256,result_schema_version,row_count,channel_count) + VALUES ('2026-09-01','2026-09-16',1,%s,'reserved','fixture',%s,'1.0',0,5) RETURNING id""", + (snapshot,"a" * 64)).fetchone()[0] + connection.execute("""INSERT INTO reporting.monthly_run_daily_versions + (report_id,business_date,daily_version_id) VALUES (%s,%s,%s)""",(report,DAY,version_id)) + return report + + def test_current_delete_removes_pointer_withdraws_month_and_keeps_original_data(self): + run,version,_ = self.seed("current",version=1) + report = self.month(version) + before = self.sql("SELECT id,object_key,sha256 FROM ingestion.artifacts ORDER BY id") + original_facts = self.sql("SELECT * FROM finance.daily_records ORDER BY id") + self.assertEqual(self.sql("SELECT count(*) FROM finance.v_active_daily_facts"),[(1,)]) + receipt = self.repository.retire_daily_job("current",DAY,ACTOR) + self.assertEqual(receipt,{"job_id":"current","business_date":DAY.isoformat(),"is_retired":True, + "current_removed":True,"monthly_refresh_required":True}) + self.assertEqual(self.sql("SELECT * FROM finance.current_daily_versions"),[]) + self.assertEqual(self.sql("SELECT run_status FROM ingestion.processing_runs WHERE id=%s",(run,)),[("accepted",)]) + self.assertEqual(self.sql("SELECT version_status FROM finance.daily_versions WHERE id=%s",(version,)),[("superseded",)]) + self.assertEqual(before,self.sql("SELECT id,object_key,sha256 FROM ingestion.artifacts ORDER BY id")) + self.assertEqual(original_facts,self.sql("SELECT * FROM finance.daily_records ORDER BY id")) + self.assertEqual(self.sql("SELECT count(*) FROM finance.v_active_daily_facts"),[(0,)]) + self.assertEqual(self.sql("SELECT report_status, withdrawn_at IS NOT NULL FROM reporting.monthly_runs WHERE id=%s",(report,)),[("withdrawn",True)]) + event = self.sql("SELECT aggregate_type,event_type,payload FROM ingestion.outbox_events")[0] + self.assertEqual(event[:2],("daily_version","arr.daily_scope_changed")) + self.assertEqual(event[2]["period_start"],"2026-09-01") + self.assertEqual(self.repository.retire_daily_job("current",DAY,"another-user"),receipt) + self.assertEqual(self.sql("SELECT actor_username FROM ingestion.daily_run_retirements"),[(ACTOR,)]) + self.assertEqual(self.sql("SELECT count(*) FROM ingestion.outbox_events"),[(1,)]) + + def test_old_history_delete_keeps_current_and_month(self): + _,old,_ = self.seed("old",version=1) + _,new,_ = self.seed("new",version=2) + report = self.month(new) + receipt = self.repository.retire_daily_job("old",DAY,ACTOR) + self.assertFalse(receipt["current_removed"]) + self.assertEqual(self.sql("SELECT daily_version_id FROM finance.current_daily_versions"),[(new,)]) + self.assertEqual(self.sql("SELECT report_status FROM reporting.monthly_runs WHERE id=%s",(report,)),[("reserved",)]) + self.assertEqual(self.sql("SELECT count(*) FROM ingestion.outbox_events"),[(0,)]) + self.assertEqual(self.sql("SELECT count(*) FROM finance.daily_versions WHERE id=%s",(old,)),[(1,)]) + + def test_pending_price_review_deletes_only_that_task_and_audits_cancellation(self): + _,current,_ = self.seed("current",version=1) + run,_,_ = self.seed("pending",status="awaiting_review") + report = self.month(current) + self.assertFalse(self.repository.retire_daily_job("pending",DAY,ACTOR)["monthly_refresh_required"]) + self.assertEqual(self.sql("SELECT run_status FROM ingestion.processing_runs WHERE id=%s",(run,)),[("cancelled",)]) + self.assertEqual(self.sql("SELECT case_status,revision FROM ingestion.daily_review_cases"),[("cancelled",1)]) + self.assertEqual(self.sql("SELECT event_type,actor_username FROM ingestion.daily_review_events"),[("PRICE_REVIEW_CANCELLED",ACTOR)]) + self.assertEqual(self.sql("SELECT daily_version_id FROM finance.current_daily_versions"),[(current,)]) + self.assertEqual(self.sql("SELECT report_status FROM reporting.monthly_runs WHERE id=%s",(report,)),[("reserved",)]) + self.assertEqual(self.sql("SELECT count(*) FROM ingestion.daily_review_items"),[(1,)]) + + def test_running_and_generating_tasks_are_blocked_without_changes(self): + for status in ("received","queued","running","validating"): + self.seed(status,status=status) + with self.subTest(status=status), self.assertRaises(IngestionError) as raised: + self.repository.retire_daily_job(status,DAY,ACTOR) + self.assertEqual(raised.exception.code,"DAILY_JOB_BUSY") + self.seed("generating",status="awaiting_review") + self.sql("UPDATE ingestion.daily_review_cases SET case_status='processing'") + with self.assertRaises(IngestionError) as raised: + self.repository.retire_daily_job("generating",DAY,ACTOR) + self.assertEqual(raised.exception.code,"DAILY_JOB_BUSY") + self.assertEqual(self.sql("SELECT count(*) FROM ingestion.daily_run_retirements"),[(0,)]) + + def test_stale_date_unknown_job_and_invalid_identity_fail_closed(self): + self.seed("current",version=1) + for job,day,code in (("current",date(2026,9,17),"DAILY_DATE_CONFLICT"), + ("missing",DAY,"DAILY_JOB_NOT_FOUND"), + ("bad/identity",DAY,"DAILY_JOB_NOT_FOUND"), + ("current",DAY.isoformat(),"DAILY_DATE_CONFLICT")): + with self.subTest(job=job,day=day),self.assertRaises(IngestionError) as raised: + self.repository.retire_daily_job(job,day,ACTOR) + self.assertEqual(raised.exception.code,code) + self.assertEqual(self.sql("SELECT count(*) FROM ingestion.daily_run_retirements"),[(0,)]) + + def test_failed_and_cancelled_terminal_records_without_versions_can_be_deleted(self): + for status in ("failed","rejected","cancelled"): + self.seed(status,status=status) + self.assertFalse(self.repository.retire_daily_job(status,DAY,ACTOR)["current_removed"]) + self.assertEqual(self.sql("SELECT count(*) FROM ingestion.daily_run_retirements"),[(3,)]) + self.assertEqual(self.sql("SELECT count(*) FROM ingestion.outbox_events"),[(0,)]) + + def test_concurrent_repeated_delete_has_one_audit_and_outbox(self): + self.seed("current",version=1) + with ThreadPoolExecutor(max_workers=2) as pool: + results = list(pool.map(lambda _: self.repository.retire_daily_job("current",DAY,ACTOR),range(2))) + self.assertEqual(results[0],results[1]) + self.assertEqual(self.sql("SELECT count(*) FROM ingestion.daily_run_retirements"),[(1,)]) + self.assertEqual(self.sql("SELECT count(*) FROM ingestion.outbox_events"),[(1,)]) + + def test_replacement_racing_old_delete_keeps_new_current_without_resurrection(self): + self.seed("old",version=1) + _,new,_ = self.seed("new",version=2,active=False) + barrier = threading.Barrier(2) + def activate(): + barrier.wait() + def operation(cursor): + cursor.execute("SELECT pg_advisory_xact_lock(hashtextextended(%s,0))",("finance-daily:"+DAY.isoformat(),)) + self.repository._activate_daily_version(cursor,DAY,new) + self.repository._run_transaction(operation,"fixture activation failed") + def retire(): + barrier.wait() + return self.repository.retire_daily_job("old",DAY,ACTOR) + with ThreadPoolExecutor(max_workers=2) as pool: + a,b = pool.submit(activate),pool.submit(retire) + a.result(); b.result() + self.assertEqual(self.sql("SELECT daily_version_id FROM finance.current_daily_versions"),[(new,)]) + self.assertEqual(self.sql("SELECT version_status FROM finance.daily_versions WHERE id=%s",(new,)),[("active",)]) + + def test_outbox_failure_rolls_back_current_removal_month_withdrawal_and_audit(self): + _,version,_ = self.seed("current",version=1) + self.month(version) + self.sql("""CREATE FUNCTION ingestion.fail_retirement_event() RETURNS trigger LANGUAGE plpgsql AS $$ + BEGIN RAISE EXCEPTION 'synthetic event failure'; END $$; + CREATE TRIGGER fixture_reject_retirement_event BEFORE INSERT ON ingestion.outbox_events + FOR EACH ROW EXECUTE FUNCTION ingestion.fail_retirement_event()""") + with self.assertRaises(IngestionError): + self.repository.retire_daily_job("current",DAY,ACTOR) + self.assertEqual(self.sql("SELECT daily_version_id FROM finance.current_daily_versions"),[(version,)]) + self.assertEqual(self.sql("SELECT version_status FROM finance.daily_versions"),[("active",)]) + self.assertEqual(self.sql("SELECT report_status FROM reporting.monthly_runs"),[("reserved",)]) + self.assertEqual(self.sql("SELECT count(*) FROM ingestion.daily_run_retirements"),[(0,)]) + + def test_daily_activation_waits_for_monthly_publication_scope_lock(self): + self.seed("old",version=1) + _,new,_ = self.seed("new",version=2,active=False) + held = self.database.connect() + self.addCleanup(held.close) + held.execute("SELECT pg_advisory_xact_lock(hashtextextended(%s,0))", + ("monthly_channel:2026-09-01:2026-09-30:*",)) + def activate(): + self.repository._run_transaction( + lambda cursor: self.repository._activate_daily_version(cursor,DAY,new), + "fixture activation failed") + with ThreadPoolExecutor(max_workers=1) as pool: + future = pool.submit(activate) + try: + deadline = time.monotonic()+5 + while self.sql("SELECT count(*) FROM pg_locks WHERE locktype='advisory' AND NOT granted")[0][0] == 0: + self.assertLess(time.monotonic(),deadline,"activation did not wait for month lock") + time.sleep(.01) + self.assertFalse(future.done()) + finally: + held.commit() + future.result(timeout=10) + self.assertEqual(self.sql("SELECT daily_version_id FROM finance.current_daily_versions"),[(new,)]) + + def test_retirement_audit_and_withdrawn_lineage_are_immutable_and_old_version_cannot_be_current(self): + import psycopg + _,version,_ = self.seed("current",version=1) + report = self.month(version) + self.repository.retire_daily_job("current",DAY,ACTOR) + for query in ("UPDATE ingestion.daily_run_retirements SET actor_username='changed'", + "DELETE FROM ingestion.daily_run_retirements", + "DELETE FROM reporting.monthly_run_daily_versions WHERE report_id="+str(report)): + with self.subTest(query=query),self.assertRaises(psycopg.Error): + self.sql(query) + # Even an accidental lifecycle update cannot repoint a retired version. + self.sql("UPDATE finance.daily_versions SET version_status='active' WHERE id=%s",(version,)) + with self.assertRaises(psycopg.Error): + self.sql("INSERT INTO finance.current_daily_versions (business_date,daily_version_id) VALUES (%s,%s)",(DAY,version)) + + def test_same_retired_job_registration_replay_is_rejected_but_new_upload_identity_is_allowed(self): + _,_,source = self.seed("current",version=1) + sha = hashlib.sha256(b"current").hexdigest() + self.repository.retire_daily_job("current",DAY,ACTOR) + source_ref = ArtifactRef("source_xml","opera_xml","current/source.xml","source.xml",sha,0,"application/xml") + registration = JobRegistration("current",source_ref,"4.4.0",sha,1,"b"*64,None) + with self.assertRaises(IngestionError) as raised: + self.repository.register_job(registration) + self.assertEqual(raised.exception.code,"JOB_RETIRED") + self.repository.register_job(JobRegistration("new-upload",source_ref,"4.4.0",sha,1,"c"*64,None)) + self.assertEqual(self.sql("SELECT run_status FROM ingestion.processing_runs WHERE run_key='new-upload'"),[("queued",)]) + + def test_empty_rollback_reapply_and_rollback_refusal_after_audit(self): + import psycopg + with self.database.connect(autocommit=True) as connection: + connection.execute((PROJECT/"database/022_daily_report_retirement.down.sql").read_text(),prepare=False) + connection.execute((PROJECT/"database/022_daily_report_retirement.sql").read_text(),prepare=False) + self.seed("current",version=1) + self.repository.retire_daily_job("current",DAY,ACTOR) + with self.database.connect(autocommit=True) as connection: + with self.assertRaises(psycopg.Error): + connection.execute((PROJECT/"database/022_daily_report_retirement.down.sql").read_text(),prepare=False) + connection.execute("ROLLBACK") + self.assertEqual(self.sql("SELECT count(*) FROM ingestion.daily_run_retirements"),[(1,)]) + + def test_upgrade_preserves_existing_history_and_grants_only_pointer_delete(self): + self.database.reset_database(schema_version=21) + run,version,_ = self.seed("existing",version=1) + before = self.sql("SELECT id,business_date,version_status FROM finance.daily_versions") + with self.database.connect(autocommit=True) as connection: + connection.execute("DO $$ BEGIN IF to_regrole('arr_app') IS NULL THEN CREATE ROLE arr_app; END IF; END $$") + connection.execute((PROJECT/"database/022_daily_report_retirement.sql").read_text(),prepare=False) + self.assertEqual(self.sql("SELECT id,business_date,version_status FROM finance.daily_versions"),before) + self.assertEqual(self.sql("SELECT daily_version_id FROM finance.current_daily_versions"),[(version,)]) + self.assertEqual(self.sql("""SELECT has_table_privilege('arr_app','ingestion.daily_run_retirements','SELECT'), + has_table_privilege('arr_app','ingestion.daily_run_retirements','INSERT'), + has_table_privilege('arr_app','ingestion.daily_run_retirements','DELETE'), + has_table_privilege('arr_app','finance.current_daily_versions','DELETE'), + has_table_privilege('arr_app','finance.daily_records','DELETE'), + has_table_privilege('arr_app','ingestion.artifacts','DELETE')"""),[(True,True,False,True,False,False)]) + + def test_daily_lifecycle_gate_rejects_021_then_accepts_complete_022_and_rejects_missing_index(self): + self.database.reset_database(schema_version=21) + with self.assertRaises(IngestionError) as missing: + self.repository.assert_daily_lifecycle_schema() + self.assertEqual(missing.exception.code,"DATABASE_MIGRATION_MISSING") + with self.database.connect(autocommit=True) as connection: + connection.execute((PROJECT/"database/022_daily_report_retirement.sql").read_text(),prepare=False) + self.repository.assert_daily_lifecycle_schema() + self.sql("DROP INDEX reporting.monthly_runs_live_snapshot_unique") + with self.assertRaises(IngestionError) as missing: + self.repository.assert_daily_lifecycle_schema() + self.assertEqual(missing.exception.code,"DATABASE_MIGRATION_MISSING") + + def test_daily_lifecycle_gate_rejects_disabled_retirement_audit_guard(self): + self.repository.assert_daily_lifecycle_schema() + self.sql("ALTER TABLE ingestion.daily_run_retirements DISABLE TRIGGER daily_run_retirements_immutable") + with self.assertRaises(IngestionError) as missing: + self.repository.assert_daily_lifecycle_schema() + self.assertEqual(missing.exception.code,"DATABASE_MIGRATION_MISSING") diff --git a/tests/test_monthly_reports_lifecycle_postgres.py b/tests/test_monthly_reports_lifecycle_postgres.py new file mode 100644 index 0000000..a7a9188 --- /dev/null +++ b/tests/test_monthly_reports_lifecycle_postgres.py @@ -0,0 +1,217 @@ +"""Owned disposable PostgreSQL acceptance for monthly retirement and races. + +No existing database, live source, production configuration or hotel data is used. +""" +from concurrent.futures import ThreadPoolExecutor +from datetime import date, timedelta +import os +from pathlib import Path +import tempfile +import threading +import unittest + +from arr_ingestion.postgres import DatabaseConfig as IngestionConfig, PostgresIngestionRepository +from arr_ingestion.service import IngestionService +from arr_ingestion.validation import DeliveryValidator +from arr_processing.local import LocalDailyProcessor +from arr_processing.policy import load_processor_policy +from arr_storage.filesystem import FilesystemObjectBackend +from arr_storage.store import ManagedObjectStore +from arr_web.programmatic import ProgrammaticUploadCoordinator +from monthly_reports.contracts import ErrorCode +from monthly_reports.core import build_monthly_report +from monthly_reports.publishing import AtomicReportPublisher, OpenpyxlWorkbookBuilder +from monthly_reports.repository import DatabaseConfig, PostgresReportRepository +from monthly_reports.service import MonthlyReportService, RunRequest +from monthly_reports.worker import MonthlyOutboxWorker, PostgresOutboxRepository, WorkerError +from tests.local_postgres import TemporaryPostgres +from tests.test_arr_opera_daily_ingest import reservation, xml_document + + +PROJECT = Path(__file__).resolve().parents[1] + + +class PausedBuilder: + def __init__(self): + self.entered, self.release = threading.Event(), threading.Event() + self.delegate = OpenpyxlWorkbookBuilder() + + def build(self, report, work_dir): + self.entered.set() + if not self.release.wait(15): + raise RuntimeError("synthetic monthly builder wait expired") + return self.delegate.build(report, work_dir) + + +@unittest.skipUnless(os.environ.get("ARR_TEST_LOCAL_POSTGRES") == "1", "owned disposable PostgreSQL opt-in required") +class MonthlyLifecyclePostgresTests(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.database = TemporaryPostgres().__enter__() + cls.addClassCleanup(cls.database.__exit__, None, None, None) + cls.policy = load_processor_policy(PROJECT) + + def setUp(self): + self.database.reset_database(schema_version=22) + temporary = tempfile.TemporaryDirectory(prefix="arr-monthly-lifecycle-") + self.addCleanup(temporary.cleanup) + self.root = Path(temporary.name) + self.store = ManagedObjectStore(FilesystemObjectBackend(self.root / "objects", create=True)) + self.ingestion = PostgresIngestionRepository(IngestionConfig("owned-fixture"), connect=self.database.connect) + service = IngestionService(DeliveryValidator(self.store, self.policy), self.ingestion) + self.uploads = ProgrammaticUploadCoordinator(self.store, self.ingestion, service, LocalDailyProcessor(self.policy), + self.policy.processor_version, self.policy.rule_set_sha256) + config = DatabaseConfig("owned-fixture") + self.reports = PostgresReportRepository(config, connect=self.database.connect) + self.outbox = PostgresOutboxRepository(config, connect=self.database.connect) + self.service = self.monthly_service() + self.worker = MonthlyOutboxWorker(self.outbox, self.reports, self.service) + + def sql(self, query, parameters=None): + with self.database.connect() as connection: + cursor = connection.execute(query, parameters) + return cursor.fetchall() if cursor.description else [] + + def monthly_service(self, builder=None): + output = self.root / "outputs/monthly" + return MonthlyReportService(self.reports, builder or OpenpyxlWorkbookBuilder(), + AtomicReportPublisher(self.root, output, object_store=self.store), output / ".staging") + + def accepted(self, day): + xml = xml_document(reservation(1, confirmation=f"SYN-{day.isoformat()}", room=f"SYN-ROOM-{day.day}")) + xml = (xml.replace("2026-07-27", day.isoformat()) + .replace("2026-07-28", (day + timedelta(days=1)).isoformat()) + .replace("20260727", day.strftime("%Y%m%d")) + .replace("27-07-26", day.strftime("%d-%m-%y"))) + result = self.uploads.submit(f"synthetic-{day.isoformat()}.XML", xml.encode()) + self.assertEqual(result["status"], "succeeded", result) + return result + + def retire(self, uploaded): + return self.ingestion.retire_daily_job(uploaded["job_id"], date.fromisoformat(uploaded["business_date"]), "synthetic-operator") + + def drain(self): + outcomes = [] + for _attempt in range(20): + outcome = self.worker.process_next() + if outcome.status == "idle": + return outcomes + self.assertIn(outcome.status, {"published", "withdrawn"}, outcome) + outcomes.append(outcome) + self.fail("synthetic monthly outbox did not settle") + + def active(self): + return self.sql("SELECT id, as_of_date, version_no, row_count FROM reporting.monthly_runs WHERE report_status='active'") + + def test_last_day_deletion_withdraws_all_old_publications_and_acknowledges_without_empty_report(self): + uploaded = self.accepted(date(2026, 7, 2)) + self.drain() + published = self.active()[0] + count_before = self.sql("SELECT count(*) FROM reporting.monthly_runs")[0][0] + self.assertTrue(self.retire(uploaded)["current_removed"]) + self.assertEqual(self.active(), []) + outcomes = self.drain() + self.assertEqual([outcome.status for outcome in outcomes], ["withdrawn"]) + self.assertEqual(self.sql("SELECT count(*) FROM reporting.monthly_runs")[0][0], count_before) + self.assertEqual(self.sql("SELECT count(*) FROM finance.v_active_daily_facts"), [(0,)]) + self.assertEqual(self.sql("SELECT report_status, withdrawn_at IS NOT NULL FROM reporting.monthly_runs WHERE id=%s", (published[0],)), + [("withdrawn", True)]) + self.assertEqual(self.sql("SELECT publish_status FROM ingestion.outbox_events WHERE event_type='arr.daily_scope_changed'"), [("published",)]) + self.assertEqual(self.sql("SELECT count(*) FROM finance.daily_records WHERE outcome='retained'"), [(1,)]) + + def test_deleting_middle_day_rebuilds_only_remaining_facts_at_same_watermark(self): + first = self.accepted(date(2026, 7, 2)) + middle = self.accepted(date(2026, 7, 3)) + last = self.accepted(date(2026, 7, 4)) + self.drain() + self.assertEqual(self.active()[0][1:], (date(2026, 7, 4), 1, 3)) + self.retire(middle) + self.assertEqual(self.active(), []) + self.drain() + active = self.active()[0] + self.assertEqual(active[1], date(2026, 7, 4)) + self.assertEqual(active[3], 2) + self.assertEqual(self.sql("SELECT daily_version_id FROM reporting.monthly_run_daily_versions WHERE report_id=%s ORDER BY business_date", (active[0],)), + [(first["daily_version_id"],), (last["daily_version_id"],)]) + + def test_deleting_every_day_withdraws_the_month_and_preserves_published_lineage(self): + first = self.accepted(date(2026, 7, 2)) + self.drain() + second = self.accepted(date(2026, 7, 3)) + self.drain() + published_ids = [row[0] for row in self.sql("SELECT id FROM reporting.monthly_runs ORDER BY id")] + self.retire(first) + self.retire(second) + self.assertEqual([outcome.status for outcome in self.drain()], ["withdrawn", "withdrawn"]) + self.assertEqual(self.active(), []) + self.assertEqual(self.sql("SELECT count(*) FROM finance.current_daily_versions"), [(0,)]) + self.assertEqual(self.sql("SELECT count(*) FROM finance.v_active_daily_facts"), [(0,)]) + self.assertEqual(self.sql("SELECT count(*) FROM reporting.monthly_runs WHERE report_status='withdrawn'"), [(2,)]) + import psycopg + with self.assertRaisesRegex(psycopg.errors.RaiseException, "lineage and manifest are immutable"): + self.sql("DELETE FROM reporting.monthly_run_daily_versions WHERE report_id=%s", (published_ids[0],)) + self.assertEqual(self.sql("SELECT count(*) FROM reporting.monthly_run_daily_versions WHERE report_id=%s", (published_ids[0],)), [(1,)]) + + def test_deleting_latest_day_creates_new_A_snapshot_without_resurrecting_earlier_A_publication(self): + first = self.accepted(date(2026, 7, 2)) + self.drain() + original = self.active()[0] + second = self.accepted(date(2026, 7, 3)) + self.drain() + self.retire(second) + self.drain() + remaining = self.active()[0] + self.assertEqual(remaining[1:], (date(2026, 7, 2), 3, 1)) + self.assertNotEqual(remaining[0], original[0]) + self.assertEqual(self.sql("SELECT source_snapshot_sha256 FROM reporting.monthly_runs WHERE id IN (%s,%s) ORDER BY id", (original[0], remaining[0]))[0], + self.sql("SELECT source_snapshot_sha256 FROM reporting.monthly_runs WHERE id=%s", (remaining[0],))[0]) + self.assertEqual(self.sql("SELECT daily_version_id FROM finance.current_daily_versions"), [(first["daily_version_id"],)]) + self.assertEqual(self.sql("SELECT report_status FROM reporting.monthly_runs WHERE id=%s", (original[0],)), [("withdrawn",)]) + + def test_reserved_worker_cannot_publish_a_day_deleted_while_it_was_building(self): + uploaded = self.accepted(date(2026, 7, 2)) + paused = PausedBuilder() + service = self.monthly_service(paused) + with ThreadPoolExecutor(max_workers=1) as pool: + result = pool.submit(service.run, RunRequest(2026, 7, date(2026, 7, 2))) + try: + self.assertTrue(paused.entered.wait(10)) + self.retire(uploaded) + finally: + paused.release.set() + final = result.result(timeout=20) + self.assertEqual((final.status, final.error_code), ("failed", ErrorCode.SOURCE_SNAPSHOT_STALE)) + self.assertEqual(self.active(), []) + self.assertEqual(self.sql("SELECT report_status FROM reporting.monthly_runs"), [("withdrawn",)]) + self.assertTrue(all(outcome.status == "withdrawn" for outcome in self.drain())) + + def test_old_cutoff_worker_cannot_replace_a_later_successful_publication(self): + self.accepted(date(2026, 7, 2)) + paused = PausedBuilder() + service = self.monthly_service(paused) + with ThreadPoolExecutor(max_workers=1) as pool: + result = pool.submit(service.run, RunRequest(2026, 7, date(2026, 7, 2))) + try: + self.assertTrue(paused.entered.wait(10)) + self.accepted(date(2026, 7, 3)) + self.drain() + later_id = self.active()[0][0] + finally: + paused.release.set() + final = result.result(timeout=20) + self.assertEqual((final.status, final.error_code), ("failed", ErrorCode.SOURCE_SNAPSHOT_STALE)) + self.assertEqual(self.active()[0][0], later_id) + self.assertEqual(self.active()[0][1], date(2026, 7, 3)) + + def test_empty_ack_refuses_a_month_that_still_has_active_facts(self): + self.accepted(date(2026, 7, 2)) + event = self.outbox.claim_next() + self.assertIsNotNone(event) + with self.assertRaises(WorkerError) as raised: + self.outbox.mark_withdrawn(event.event_id, date(2026, 7, 1)) + self.assertEqual(raised.exception.code, "MONTHLY_WORKER_WITHDRAW_ACK_FAILED") + self.assertEqual(self.sql("SELECT publish_status FROM ingestion.outbox_events WHERE id=%s", (event.event_id,)), [("publishing",)]) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_monthly_reports_repository.py b/tests/test_monthly_reports_repository.py index 42a53fc..7e9f4c2 100644 --- a/tests/test_monthly_reports_repository.py +++ b/tests/test_monthly_reports_repository.py @@ -230,6 +230,8 @@ class MonthlyReportsRepositoryTests(unittest.TestCase): return [("booking_test",)], 1 if "FROM finance.current_daily_versions" in sql: return [(date(2026, 7, 2), 102)], 1 + if "SELECT max(fact.arrival)" in sql: + return [(date(2026, 7, 2),)], 1 if "FROM reporting.monthly_runs AS run" in sql: return [], 0 if "SELECT COALESCE(max(version_no), 0) + 1" in sql: @@ -321,6 +323,8 @@ class MonthlyReportsRepositoryTests(unittest.TestCase): return [("booking_test",)], 1 if "FROM finance.current_daily_versions" in sql: return [(date(2026, 7, 2), 102)], 1 + if "SELECT max(fact.arrival)" in sql: + return [(date(2026, 7, 2),)], 1 if "SELECT report_status, source_snapshot_sha256" in sql: return [("reserved", "c" * 64)], 1 if "FROM ingestion.artifacts" in sql: @@ -373,6 +377,64 @@ class MonthlyReportsRepositoryTests(unittest.TestCase): self.assertNotIn("finance.report_versions", executed) self.assertNotIn("booking.file_objects", executed) + def test_deleted_last_day_preserves_month_identity_with_no_fabricated_cutoff(self): + def handler(sql, _params): + if sql == "SELECT current_database()": + return [("booking_test",)], 1 + if "SELECT min(record.arrival), max(record.arrival)" in sql: + return [(date(2026, 7, 21), date(2026, 7, 21))], 1 + if "SELECT max(fact.arrival)" in sql: + return [(None,)], 1 + return [], 0 + connection = FakeConnection(handler) + repository = PostgresReportRepository(DatabaseConfig("synthetic"), connect=lambda _dsn: connection) + derived = repository.derive_monthly_request(4) + self.assertIsNotNone(derived) + self.assertEqual((derived.report_year, derived.report_month, derived.as_of_date), (2026, 7, None)) + + def test_earlier_cutoff_cannot_publish_after_later_day_was_added_outside_its_pins(self): + from tests.test_monthly_reports_xlsx import report_fixture + report = report_fixture() + for latest in (date(2026, 7, 9), None): + with self.subTest(latest=latest): + def handler(sql, _params): + if sql == "SELECT current_database()": + return [("booking_test",)], 1 + if "FROM finance.current_daily_versions" in sql: + return [(date(2026, 7, 8), 100)], 1 + if "SELECT max(fact.arrival)" in sql: + return [(latest,)], 1 + return [], 0 + connection = FakeConnection(handler) + repository = PostgresReportRepository(DatabaseConfig("synthetic"), connect=lambda _dsn: connection) + with self.assertRaises(RepositoryError) as raised: + repository.reserve_report(report) + self.assertEqual(raised.exception.code, ErrorCode.SOURCE_SNAPSHOT_STALE) + self.assertFalse(any("INSERT INTO reporting.monthly_runs" in sql + for sql, _params in connection.cursor_instance.calls)) + + def test_reservation_lookup_excludes_withdrawn_snapshot_instead_of_reusing_its_artifact(self): + from tests.test_monthly_reports_xlsx import report_fixture + report = report_fixture() + def handler(sql, _params): + if sql == "SELECT current_database()": + return [("booking_test",)], 1 + if "FROM finance.current_daily_versions" in sql: + return [(date(2026, 7, 8), 100)], 1 + if "SELECT max(fact.arrival)" in sql: + return [(date(2026, 7, 8),)], 1 + if "FROM reporting.monthly_runs AS run" in sql: + self.assertIn("run.report_status <> 'withdrawn'", sql) + return [], 0 + if "SELECT COALESCE(max(version_no), 0) + 1" in sql: + return [(3,)], 1 + if "INSERT INTO reporting.monthly_runs" in sql: + return [(45,)], 1 + return [], 1 + connection = FakeConnection(handler) + reservation = PostgresReportRepository(DatabaseConfig("synthetic"), connect=lambda _dsn: connection).reserve_report(report) + self.assertEqual((reservation.report_version_id, reservation.version_no, reservation.already_published), (45, 3, False)) + if __name__ == "__main__": unittest.main() diff --git a/tests/test_monthly_reports_worker.py b/tests/test_monthly_reports_worker.py index ba82af9..e30dd30 100644 --- a/tests/test_monthly_reports_worker.py +++ b/tests/test_monthly_reports_worker.py @@ -10,6 +10,8 @@ from monthly_reports.worker import ( CLAIM_EVENT_SQL, FAIL_EVENT_SQL, PUBLISH_EVENT_SQL, + WITHDRAW_EVENT_SQL, + SCOPE_CHANGED_EVENT_TYPE, MonthlyOutboxWorker, OutboxEvent, ) @@ -20,6 +22,7 @@ class FakeOutbox: self.event = event self.published: list[tuple[int, int]] = [] self.failed: list[tuple[OutboxEvent, str, bool]] = [] + self.withdrawn: list[tuple[int, date]] = [] def claim_next(self): value, self.event = self.event, None @@ -28,6 +31,9 @@ class FakeOutbox: def mark_published(self, event_id: int, report_id: int) -> None: self.published.append((event_id, report_id)) + def mark_withdrawn(self, event_id: int, period_start: date) -> None: + self.withdrawn.append((event_id, period_start)) + def mark_failed(self, event: OutboxEvent, code: str, *, retryable: bool) -> str: self.failed.append((event, code, retryable)) return "pending" if retryable else "dead" @@ -119,10 +125,62 @@ class MonthlyReportsWorkerTests(unittest.TestCase): self.assertTrue(outbox.failed[0][2]) self.assertEqual(outbox.published, []) + def test_last_day_deletion_acknowledges_withdrawal_without_an_empty_workbook(self): + deletion = OutboxEvent(9, "retired-day:4", { + "daily_version_id": 4, "period_start": "2026-07-01", "action": "delete", + }, 1, SCOPE_CHANGED_EVENT_TYPE) + outbox = FakeOutbox(deletion) + service = FakeService(lambda request: self.fail("an empty month must not produce a workbook")) + result = MonthlyOutboxWorker(outbox, FakeRequests(DerivedMonthlyRequest(2026, 7, None)), service).process_next() + self.assertEqual(result.status, "withdrawn") + self.assertIsNone(result.report_id) + self.assertEqual(outbox.withdrawn, [(9, date(2026, 7, 1))]) + self.assertEqual(outbox.failed, []) + self.assertEqual(service.requests, []) + + def test_old_commit_event_for_now_deleted_month_also_acknowledges_withdrawal(self): + outbox = FakeOutbox(event()) + service = FakeService(lambda request: self.fail("retired source must not be republished")) + result = MonthlyOutboxWorker(outbox, FakeRequests(DerivedMonthlyRequest(2026, 7, None)), service).process_next() + self.assertEqual(result.status, "withdrawn") + self.assertEqual(outbox.withdrawn, [(9, date(2026, 7, 1))]) + + def test_mid_month_deletion_rebuilds_at_remaining_fact_watermark(self): + outbox = FakeOutbox(OutboxEvent(9, "retired-day:4", { + "daily_version_id": 4, "period_start": "2026-07-01", "action": "delete", + }, 1, SCOPE_CHANGED_EVENT_TYPE)) + service = FakeService(lambda request: RunResult(request=request, status="success", report_version_id=45, + artifact={"sha256": "b" * 64})) + result = MonthlyOutboxWorker(outbox, FakeRequests(DerivedMonthlyRequest(2026, 7, date(2026, 7, 2))), service).process_next() + self.assertEqual(result.status, "published") + self.assertEqual(service.requests[0].as_of_date, date(2026, 7, 2)) + self.assertEqual(outbox.withdrawn, []) + + def test_deletion_cannot_withdraw_a_month_other_than_its_original_fact_scope(self): + outbox = FakeOutbox(OutboxEvent(9, "retired-day:4", { + "daily_version_id": 4, "period_start": "2026-08-01", "action": "delete", + }, 1, SCOPE_CHANGED_EVENT_TYPE)) + result = MonthlyOutboxWorker(outbox, FakeRequests(DerivedMonthlyRequest(2026, 7, None)), FakeService(None)).process_next() + self.assertEqual((result.status, result.error_code), ("dead", "MONTHLY_WORKER_EVENT_INVALID")) + self.assertEqual(outbox.withdrawn, []) + + def test_empty_ack_failure_retries_instead_of_claiming_withdrawal(self): + from monthly_reports.worker import WorkerError + outbox = FakeOutbox(event()) + def fail_ack(*_args): + raise WorkerError("MONTHLY_WORKER_WITHDRAW_ACK_FAILED", retryable=True) + outbox.mark_withdrawn = fail_ack + result = MonthlyOutboxWorker(outbox, FakeRequests(DerivedMonthlyRequest(2026, 7, None)), FakeService(None)).process_next() + self.assertEqual(result.status, "pending") + self.assertEqual(result.error_code, "MONTHLY_WORKER_WITHDRAW_ACK_FAILED") + def test_claim_and_ack_sql_are_leased_locked_and_downloadable(self): self.assertIn("FOR UPDATE SKIP LOCKED", CLAIM_EVENT_SQL) self.assertIn("publish_status IN ('pending', 'publishing')", CLAIM_EVENT_SQL) self.assertIn("available_at <= now()", CLAIM_EVENT_SQL) + self.assertIn("aggregate_type = 'daily_version'", CLAIM_EVENT_SQL) + self.assertIn("finance.v_active_daily_facts", WITHDRAW_EVENT_SQL) + self.assertIn("'reserved', 'active', 'superseded'", WITHDRAW_EVENT_SQL) self.assertIn("reporting.monthly_runs", PUBLISH_EVENT_SQL) self.assertIn("monthly_xlsx", PUBLISH_EVENT_SQL) self.assertIn("result_json", PUBLISH_EVENT_SQL)