46 lines
1.9 KiB
Markdown
46 lines
1.9 KiB
Markdown
# Reflection: Derive Report Watermarks From Business Facts
|
|
|
|
## Trigger
|
|
|
|
The user corrected the proposed monthly cutoff source: the Opera XML filename has no date, and the monthly “更新至”
|
|
must come from the table's `ARRIVAL` field.
|
|
|
|
## Expected Behavior
|
|
|
|
Identify the authoritative business field before deriving a report period or label. Use the committed facts actually
|
|
included in the report, not transport metadata, upload names, request time or a nearby technical date.
|
|
|
|
## Actual Behavior
|
|
|
|
Earlier planning left the cutoff derivation policy open and existing code accepted a request cutoff, which allowed a
|
|
7.30 filename even though the maximum included `ARRIVAL` was 7.27.
|
|
|
|
## Root Cause
|
|
|
|
- Source identity was conflated with business time.
|
|
- An internal request field was trusted without reconciling it to the generated dataset.
|
|
- The output label was validated structurally but not against the report's semantic maximum date.
|
|
|
|
## Evidence
|
|
|
|
- User correction on 2026-07-30.
|
|
- The pre-fix manual workbook was named “更新至7.30” while included rows ended at `ARRIVAL=2026-07-27`.
|
|
- The repaired live report stores `as_of_date=2026-07-27` and core validation enforces equality with max included
|
|
`ARRIVAL`.
|
|
|
|
## Lesson
|
|
|
|
For derived period labels, trace the value to the authoritative business column and assert it again against the final
|
|
dataset. Filenames and wall-clock timestamps are provenance/audit data, not business watermarks unless the domain
|
|
explicitly says otherwise.
|
|
|
|
## Action
|
|
|
|
- Encode max-included-`ARRIVAL` in ADR-001 and durable business rules.
|
|
- Derive worker scope through `daily_version_id` lookups rather than payload/filename dates.
|
|
- Reject any non-empty monthly report whose stored `as_of_date` differs from its maximum included `ARRIVAL`.
|
|
|
|
## Promotion
|
|
|
|
Promoted to ADR-001, the data flow, business rules, migration column comment and monthly core/worker tests.
|