fix: refresh public dashboard and BI telemetry
This commit is contained in:
Wyndham ARR
@@ -0,0 +1,32 @@
|
||||
# Channel BI freshness after August publication
|
||||
|
||||
## Question
|
||||
|
||||
The operator reported that the monthly report had updated but Channel BI still showed `8.1`.
|
||||
|
||||
## Read-only evidence
|
||||
|
||||
- `reporting.monthly_runs` has active August V02 with `as_of_date=2026-08-02`, 268 rows and 6 channels. August V01
|
||||
at `2026-08-01` is superseded.
|
||||
- A fresh `PostgresAnalyticsRepository.read_dashboard("2026-08")` repeatable-read query returns
|
||||
`max_arrival_date=2026-08-02`, `min_arrival_date=2026-08-01`, `updated_at=2026-08-03T14:21:29+08:00` and 268
|
||||
retained rows. The source projection and database publication are therefore current.
|
||||
- `arr_web/static/app.js` calls `loadAnalytics()` during initial boot, on the first BI tab entry only when
|
||||
`state.analytics` is empty, and when the month selector changes. The existing four-second timer belongs to the
|
||||
monthly-history list and does not refresh BI analytics.
|
||||
|
||||
## Conclusion
|
||||
|
||||
This is a frontend freshness bug, not an operator mistake and not a current monthly-worker/database publication
|
||||
failure. A BI page opened before V02 was published can keep rendering the cached 8.1 payload until the page reloads or
|
||||
the month selector changes. No code, runtime, report or business data was changed during diagnosis.
|
||||
|
||||
## Repair implemented
|
||||
|
||||
`arr_web/static/app.js` and `arr_web/static/h5.js` now check the selected month's lightweight `updated_at` metadata
|
||||
every five seconds while visible. They reload the full analytics payload only when the selected month changes or its
|
||||
`updated_at` changes, pause while hidden, check immediately when visible again and preserve the last good snapshot on
|
||||
transient background failures. The desktop monthly-history timer remains four seconds and is independent from BI.
|
||||
|
||||
Focused verification passes: BI visual/static contract 5/5, Web contract 19/19, both JavaScript syntax checks and
|
||||
`git diff --check`. No API, report or business data behavior changed.
|
||||
@@ -0,0 +1,47 @@
|
||||
# Booking source and company-report retry semantics
|
||||
|
||||
## Metadata
|
||||
|
||||
- Date: 2026-08-03
|
||||
- Status: Confirmed in source; live acceptance remains pending the controlled port-8766 restart
|
||||
- Scope: Repeated identical Booking Excel uploads, company-report reruns and subset-source reruns
|
||||
- Confidence: Fact for the current source contracts and tests; live runtime is still the stale PID 11176
|
||||
- Source: `booking_ingestion/excel_review_postgres.py`, `company_reports/repository.py`, `company_reports/core.py`,
|
||||
`company_reports/publishing.py` and focused unittest suites
|
||||
- Last verified: 2026-08-03
|
||||
|
||||
## Confirmed behavior
|
||||
|
||||
1. A byte-identical `.xlsx` is identified by its file SHA-256. In the real PostgreSQL review repository, if that
|
||||
artifact has already been activated, a repeated upload returns `BOOKING_EXCEL_SOURCE_ALREADY_ACTIVATED` and does not
|
||||
create another extraction batch. If the workbook has the same Booking rows but different XLSX bytes, it is a new
|
||||
upload and must go through review/activation again; row-level semantic deduplication is not used at upload time.
|
||||
2. A company report can be generated again after a successful publication. The report reservation identity includes the
|
||||
month/period cutoff, Finance daily-version pins, Booking parse-version pins and processor/rule/schema identity. An
|
||||
unchanged identity reuses the complete archive/result pair; a changed identity creates a new version and retains the
|
||||
earlier files. Partial, corrupt or semantically conflicting publication state fails closed.
|
||||
3. A subset workbook is not a duplicate of its previous superset. After review is complete and the subset is activated,
|
||||
`booking.current_source_batch` points to it and a report rerun reads that active Booking snapshot. Finance facts remain
|
||||
the report row boundary; Group Codes missing from the subset produce rows with blank Booking Room details rather than
|
||||
silently deleting the Finance rows. Upload alone does not switch the active source, and an open review draft blocks
|
||||
company-report generation until it is activated or discarded.
|
||||
|
||||
## Runtime boundary
|
||||
|
||||
The current 8766 listener is still PID `11176`, started on 2026-08-02 before the duplicate-publication fix. The source
|
||||
contracts above are therefore the intended behavior after a controlled restart; the failed August retry is not valid live
|
||||
acceptance of the fixed publisher.
|
||||
|
||||
## User-visible reuse status
|
||||
|
||||
The current publication outcome contains the artifact and result metadata but no `reused`/`publication_disposition`
|
||||
field. `CompanyRunResult` and the public company-job projection likewise expose success, version number and download URL,
|
||||
not whether the publisher reused an existing archive/result pair. Consequently, after a successful retry the operator sees
|
||||
`成功` and a normal download link; the downloaded bytes can be the original archived file, but the page does not identify
|
||||
which prior file/version was reused.
|
||||
|
||||
## Verification
|
||||
|
||||
- Focused source-upload/publisher/integration unittest: 14 tests passed, 2 optional ArtifactTool integration tests
|
||||
skipped because the local dependency is unavailable.
|
||||
- No source activation, report rerun, output repair or business-data mutation was performed.
|
||||
@@ -0,0 +1,51 @@
|
||||
# Company-report source and publish-failure copy
|
||||
|
||||
## Metadata
|
||||
|
||||
- Date: 2026-08-03
|
||||
- Status: Implemented; source provenance module subsequently removed from the end-user result/history surface, and the
|
||||
publish-failure copy was simplified after live operator feedback
|
||||
- Scope: Desktop company-channel job detail/history copy, source-module removal and company-result error explanation
|
||||
- Confidence: Fact for source/error contracts and focused verification; no live report state was changed
|
||||
- Source: `arr_web/static/app.js`, `arr_web/static/index.html`, `arr_web/static/i18n.js`, publisher tests and company job evidence
|
||||
- Last verified: 2026-08-03
|
||||
- Stale trigger: changes to company job source metadata, publication idempotency or localized status copy
|
||||
|
||||
## Problem
|
||||
|
||||
The page displayed `来源 Excel（提交时）` with `历史任务未记录来源`, which described an internal capture detail but did
|
||||
not tell an operator whether the source was missing, old or actually used. It also displayed only
|
||||
`正式表格发布失败`, which did not explain whether this was a normal duplicate-version reuse.
|
||||
|
||||
## Implemented behavior
|
||||
|
||||
- The final end-user surface no longer renders the job-detail source card or history source column. The report page is
|
||||
explicitly database-snapshot oriented; the upload/review current-source panel is retained where the operator actually
|
||||
manages the Excel input.
|
||||
- Server-side jobs still retain the safe source summary for audit and troubleshooting. Removing the UI module does not
|
||||
backfill or delete old source metadata.
|
||||
- `COMPANY_REPORT_PUBLISH_FAILED` now appears as `正式 Excel 保存失败`. The job summary and company row use the direct
|
||||
action copy `数据已处理，但正式 Excel 保存失败。请重启报表服务后再生成。`; internal duplicate-reuse rules are
|
||||
no longer shown in the operator-facing error.
|
||||
- The wording is available in Chinese, English and Thai, and the added detail is escaped before it is inserted into the
|
||||
result table.
|
||||
|
||||
## Verification
|
||||
|
||||
- Focused unittest: Web/company regression passes 29 tests.
|
||||
- `node --check arr_web/static/app.js` and `node --check arr_web/static/i18n.js` pass.
|
||||
- Runtime static scan confirms no source-card/history-column reference remains; the upload/review source references
|
||||
remain intentionally.
|
||||
- `git diff --check` passes.
|
||||
- The project virtualenv has no `pytest` module; the repository's unittest runner was used instead.
|
||||
- A live read after the operator-reported retry found job `253ce23d…` still failed for all five companies at `publish`
|
||||
with row counts `54/18/7/1/15`. The listener remains PID `11176`, started `2026-08-02 12:20`, while the current
|
||||
publisher source was modified on `2026-08-03 11:19`; the reported restart therefore did not replace the effective
|
||||
port-8766 process.
|
||||
- No Web restart, source activation, report rerun, output backfill, database write or business-data mutation occurred.
|
||||
|
||||
## Semantics retained
|
||||
|
||||
The publisher remains idempotent for a complete semantically identical archive/result pair. Incomplete, corrupt or
|
||||
semantically conflicting publication state still fails closed; the UI reports the safe action without translating every
|
||||
publish failure into a duplicate explanation.
|
||||
@@ -21,13 +21,16 @@ without exposing workbook rows, guest data or full artifact hashes.
|
||||
source identity.
|
||||
- New job records expose only source batch id/type, safe filename, activation time, disposition and summary counts.
|
||||
- The company page shows the current active workbook and summary counts.
|
||||
- Job detail and history show the source Excel captured at submission; old job files without this field render
|
||||
`历史任务未记录来源`/an equivalent localized label.
|
||||
- The server still captures the source Excel summary at submission, but the end-user job detail/history no longer
|
||||
renders that provenance. The upload/review panel remains the place where the active Excel source is managed.
|
||||
- Re-uploading the same active workbook clears the selected file, refreshes the current-source and report-history views,
|
||||
and shows an informational message. It does not create a second extraction or automatically generate a report.
|
||||
- Report generation remains an explicit month/period action. Upload controls stay blocked while a report job is active, so
|
||||
the submission-time source remains the trace shown for that job.
|
||||
|
||||
The later UI simplification intentionally removes only the result/history display; it does not remove the server-side
|
||||
source metadata or change the report's database-backed input behavior.
|
||||
|
||||
## Verification
|
||||
|
||||
- `./.venv/bin/python -m unittest tests.test_arr_web_company_reports tests.test_arr_web`: 28 tests passed.
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
# Company report source label and live runtime
|
||||
|
||||
## Read-only evidence
|
||||
|
||||
- `booking.current_source_batch` points to accepted Booking Excel batch 7 with 72 source rows. No reviewing extraction
|
||||
draft is open.
|
||||
- The latest August `01-10` jobs contain `source: null`, and all five company results report
|
||||
`COMPANY_REPORT_PUBLISH_FAILED` at stage `publish`. The reports reached publish after building company row counts
|
||||
54/18/7/1/15, so the subset was parsed and reached report generation.
|
||||
- Batch 7 was activated before the failed jobs were created. The company-report repository reads
|
||||
`booking.v_current_room_items`, so the active source boundary is the new batch; the missing source field is trace
|
||||
metadata, not proof of an old-data report.
|
||||
- The live Web PID on port 8766 started on 2026-08-02 12:20. The current `arr_web/company_jobs.py` and
|
||||
`company_reports/publishing.py` were modified on 2026-08-03, after that process started. Refreshing the page cannot
|
||||
load the new Python code; a controlled Web restart is needed for future jobs to persist source metadata and use the
|
||||
current duplicate-publication handling.
|
||||
|
||||
## Conclusion
|
||||
|
||||
The message does not by itself mean the report used the previous Excel. In this case the new subset is active, but the
|
||||
currently running Web process is stale and saved new job records without the source summary. The same stale process
|
||||
also rejected the new `01-10` publication attempts at the publish boundary. Existing null metadata should not be
|
||||
backfilled by guessing; future jobs should be checked after a controlled restart and one authorized rerun.
|
||||
|
||||
## Post-restart retry recheck
|
||||
|
||||
The operator reported restarting port 8766 and reran `2026-08/01-10`. The newest job `253ce23dc7544d34a6e0fb9405ac4d9a`
|
||||
was created at `2026-08-03T07:09:33Z` and finished at `07:10:28Z`; it again built the five row sets
|
||||
`54/18/7/1/15`, then failed all five companies at `publish` with `COMPANY_REPORT_PUBLISH_FAILED`. Earlier job
|
||||
`c5358fa293184f5f9bad85178afada22` succeeded at `06:25Z` with the same row counts and complete archive/result pairs.
|
||||
|
||||
The effective listener is still PID `11176`, started `2026-08-02 12:20:56 +0800`, under detached Screen
|
||||
`arr2-web-8766`. The current `company_reports/publishing.py` was modified at `2026-08-03 11:19:58 +0800`, after that
|
||||
process started. The deterministic reservation and complete-pair reuse fix is therefore present in source but not loaded
|
||||
by the process that handled the retry. A browser refresh only reloads static assets; it cannot reload imported Python
|
||||
modules or replace the port listener.
|
||||
|
||||
The operator-facing publish copy was shortened to an actionable message: `数据已处理，但正式 Excel 保存失败。请重启
|
||||
报表服务后再生成。`
|
||||
|
||||
No service restart, source activation, job backfill or report regeneration was performed by this recheck.
|
||||
|
||||
## Latest retry recheck
|
||||
|
||||
The operator's next `2026-08/01-10` attempt created job `087dceca79e64bebb719ad06e7813465` at
|
||||
`2026-08-03T07:28:52Z` and finished at `07:29:49Z`. It again built row counts `54/18/7/1/15`; every company failed only
|
||||
with `COMPANY_REPORT_PUBLISH_FAILED` at `publish`. The result is therefore the same stale-runtime retry signature, not a
|
||||
new source-validation, Booking-match or workbook-build failure.
|
||||
|
||||
At `2026-08-03 15:37 +0800`, `*:8766` was still owned by PID `11176` under Screen `11172.arr2-web-8766`, both started
|
||||
`2026-08-02 12:20:56 +0800`. The current publisher and company-job modules were modified after that start time. The
|
||||
existing launcher `--check` passes, so configuration is ready; the running listener itself has simply not been
|
||||
replaced.
|
||||
|
||||
The earlier successful `c5358fa...` job's five versioned XLSX files match both their result JSON artifact hashes and
|
||||
their current-download copies. The failed retries contain no artifact descriptors and did not overwrite the successful
|
||||
files. Current source's six publisher regressions pass. The safe repair is to stop the exact old listener, verify port
|
||||
8766 is free, start one new Screen session through the launcher, verify its PID/start time and authenticated company
|
||||
readiness, then run one explicitly authorized retry. No archive deletion, metadata backfill or file repair is needed.
|
||||
|
||||
This check remained read-only: no service restart, report generation, artifact mutation, source activation or database
|
||||
write was performed.
|
||||
|
||||
## Controlled restart and live acceptance
|
||||
|
||||
After explicit operator confirmation, the exact stale listener PID `11176` was stopped with SIGINT. Port 8766 was
|
||||
verified free, then one detached `arr2-web-8766` session was started through the existing launcher. The new listener was
|
||||
PID `54127`, started `2026-08-03 15:46:29 +0800`, after the current publisher and company-job module modifications.
|
||||
`/healthz` returned 200 and authenticated `/api/health` reported database, processing, monthly, download,
|
||||
company-report and company-source readiness all true.
|
||||
|
||||
Exactly one new `2026-08/01-10` task was submitted: `05cc547d62854df281b0c9cac60e6f6d`. It finished at
|
||||
`2026-08-03T07:48:37Z` with `succeeded`; all five companies succeeded with row counts `54/18/7/1/15`. The job now
|
||||
records non-null active Booking batch-7 source metadata, confirming the new Python process loaded the source-traceability
|
||||
code.
|
||||
|
||||
The five version numbers and artifact descriptors exactly match the earlier successful job `c5358fa...`. Every archive
|
||||
XLSX, current-download XLSX and result JSON hash matches its registered artifact identity; all five authenticated HTTP
|
||||
download routes returned 200 and matched those hashes. Archive mtimes remained at the original 14:24–14:25 publication
|
||||
times, proving the retry reused the existing publication pair without overwriting it.
|
||||
|
||||
The temporary authenticated session was logged out and its temporary files removed. No Booking source activation,
|
||||
Finance fact change, artifact deletion or repair occurred. The stale-listener incident is resolved; detached-session
|
||||
reboot supervision remains a separate deployment gap.
|
||||
@@ -0,0 +1,42 @@
|
||||
# Daily commit waiting for monthly worker
|
||||
|
||||
## User question
|
||||
|
||||
The operator reported that a Daily Report upload appeared complete but the monthly report did not update.
|
||||
|
||||
## Read-only checks
|
||||
|
||||
- Port 8766 returned `ready`; the active process list showed the ARR Web service but no independent
|
||||
`monthly_reports.worker` / `arr-monthly-worker` process.
|
||||
- The newest `opera_daily` processing run is run 39. It finished at 2026-08-03 10:49:01 +08 with `accepted` status and
|
||||
no failure code.
|
||||
- Finance daily version 14 is `active` for business date 2026-08-01 and contains 111 retained rows. Its retained
|
||||
`ARRIVAL` scope is 2026-08-01, so the automatic worker would derive an August 2026 monthly request.
|
||||
- Outbox event 20 (`processing-run:39:accepted`, `arr.daily_version_committed`) is still `pending`, has zero publish
|
||||
attempts and no error code, and has been available since 2026-08-03 10:49:01 +08.
|
||||
- `reporting.monthly_runs` contains only July versions. July V04 is active with `as_of_date=2026-07-27`; no August
|
||||
version is present.
|
||||
- A second read-only check found a backlog of five daily-commit events, IDs 16–20, all `pending` with zero attempts
|
||||
and no error. Events 16–19 derive the July request through 2026-07-27; event 20 derives the August request through
|
||||
2026-08-01. Recovery should let the worker consume them in order and retain its idempotent publication behavior.
|
||||
- The controlled database file, Python runtime, Node binary and artifact-tool module pass preflight checks. The focused
|
||||
monthly suite passes 27 tests with two optional skips. No worker was started and no monthly publication or database
|
||||
write was performed.
|
||||
- The monthly page polls every four seconds only while its tab is visible. That can delay page display, but it cannot
|
||||
explain the missing database row because event 20 has not been consumed.
|
||||
|
||||
## Recovery
|
||||
|
||||
- The user authorized recovery. A detached `arr2-monthly-worker` Screen session was started with the controlled
|
||||
`booking_test` configuration and the verified Node/artifact-tool paths.
|
||||
- The worker consumed outbox events 15–20 in order. Each is now `published` with one attempt and no error code.
|
||||
- July V05 is `active` at `as_of_date=2026-07-27` with 987 rows; older July V04 is `superseded`. August V01 is
|
||||
`active` at `as_of_date=2026-08-01` with 111 rows.
|
||||
- The August registered workbook/result pair exists under the controlled project output, the XLSX passes ZIP
|
||||
integrity, the result JSON reports `success`, and Web `/healthz` remains `ready`.
|
||||
|
||||
## Conclusion
|
||||
|
||||
The Daily Report was correct; the missing monthly update was caused by the absent worker process. The report is now
|
||||
updated without re-uploading the source. The remaining operational gap is reboot-persistent supervision of the
|
||||
detached worker, which is intentionally separate from this recovery and from the later logging work.
|
||||
@@ -0,0 +1,20 @@
|
||||
# Daily trace does not explain company-report publication failure
|
||||
|
||||
## Evidence
|
||||
|
||||
- The supplied `arr trace` for the Daily job ends `status=succeeded`, with Finance daily version 15 active, all
|
||||
validation/database stages successful, and `arr.daily_version_committed` marked published.
|
||||
- That trace contains no company-report job id, report month/period, per-company result, publish-stage error or artifact
|
||||
publication identity. It therefore cannot explain the separate `正式表格发布失败2026-08` message.
|
||||
- The trace reports `execution_scope=unknown`, `processor_mode=unknown` and `remote_dispatch=unknown` for an
|
||||
`artifact_callback`, despite the current trace contract having explicit values. It also prints a full source hash in
|
||||
the default CLI output.
|
||||
|
||||
## Logging assessment
|
||||
|
||||
This is a scope/observability gap, not evidence that the Daily task failed. Keep the Daily trace focused on Daily and
|
||||
the outbox boundary, add monthly report id/version/as-of results to the monthly event projection, and add a separate
|
||||
company-report trace with job id, source batch, period, per-company publish stage and safe error code/message. Include
|
||||
runtime process/build identity and mask full hashes in default output.
|
||||
|
||||
No code or runtime change was made during this assessment.
|
||||
@@ -0,0 +1,24 @@
|
||||
# Monthly worker logging and observability gap
|
||||
|
||||
## Scope
|
||||
|
||||
Review of the logging and monitoring surfaces after user-authorized recovery of the independent monthly worker.
|
||||
|
||||
## Confirmed gaps
|
||||
|
||||
- The worker emits sparse JSON outcomes only when it is not idle; it does not emit startup, heartbeat, claim, duration or
|
||||
private exception-detail records. Detached Screen is the current process owner and is not durable application logging.
|
||||
- Web request logs go through the standard handler's stderr logger with path-only request data. The checked-in local
|
||||
runtime has no service-specific rotating log file.
|
||||
- Anonymous `/healthz` checks Web database/upload readiness only. Authenticated `monthly_ready` describes Web's manual
|
||||
monthly coordinator, not the separate worker process.
|
||||
- PostgreSQL outbox status and the daily task trace are durable and privacy-minimized, but they cannot prove the worker
|
||||
is alive. The trace also uses generic downstream wording for the monthly event.
|
||||
|
||||
## Recommendation
|
||||
|
||||
Add structured worker/Web logs with bounded retention, a durable worker heartbeat/status surface, monthly-specific task
|
||||
trace states, and supervisor/alert checks for stale heartbeat, old pending events, expired leases and dead events. Keep
|
||||
database lifecycle state authoritative and keep raw business/source payloads, secrets and full hashes out of logs.
|
||||
|
||||
No code, migration, supervisor or runtime change was made during this review.
|
||||
Reference in New Issue
Block a user