Reflection: Derive Report Watermarks From Business Facts

Trigger

The user corrected the proposed monthly cutoff source: the Opera XML filename has no date, and the monthly “更新至” must come from the table's ARRIVAL field.

Expected Behavior

Identify the authoritative business field before deriving a report period or label. Use the committed facts actually included in the report, not transport metadata, upload names, request time or a nearby technical date.

Actual Behavior

Earlier planning left the cutoff derivation policy open and existing code accepted a request cutoff, which allowed a 7.30 filename even though the maximum included ARRIVAL was 7.27.

Root Cause

Source identity was conflated with business time.
An internal request field was trusted without reconciling it to the generated dataset.
The output label was validated structurally but not against the report's semantic maximum date.

Evidence

User correction on 2026-07-30.
The pre-fix manual workbook was named “更新至7.30” while included rows ended at ARRIVAL=2026-07-27.
The repaired live report stores as_of_date=2026-07-27 and core validation enforces equality with max included ARRIVAL.

Lesson

For derived period labels, trace the value to the authoritative business column and assert it again against the final dataset. Filenames and wall-clock timestamps are provenance/audit data, not business watermarks unless the domain explicitly says otherwise.

Action

Encode max-included-ARRIVAL in ADR-001 and durable business rules.
Derive worker scope through daily_version_id lookups rather than payload/filename dates.
Reject any non-empty monthly report whose stored as_of_date differs from its maximum included ARRIVAL.

Promotion

Promoted to ADR-001, the data flow, business rules, migration column comment and monthly core/worker tests.

1.9 KiB Raw Permalink Blame History