fix: ignore emoji in XML upload processing
This commit is contained in:
Wyndham ARR
@@ -9,6 +9,8 @@
|
||||
|
||||
## Recall Pointers
|
||||
|
||||
- 2026-09-17：已按用户批准实现XML表情忽略，本地处理器4.1.0。原71052877自动忽略4处CESU-8皇冠后67条保留；71052708仍100条。原字节不改，其他损坏/业务校验保留，旧v3不采用新规则；122项Python/3项JS检查验证，尚未部署。见 `.project-docs/50-evidence/topics/2026-09-17-xml-emoji-cleanup.md`。
|
||||
|
||||
- ARR-owned processing decision: `.project-docs/10-decisions/ADR-004-arr-owned-programmatic-processing.md`
|
||||
- Automatic monthly trigger and formula requirement: `.project-docs/10-decisions/ADR-001-automatic-monthly-trigger-and-total-price-formula.md`
|
||||
- ARR2.0 implementation evidence: `.project-docs/50-evidence/topics/2026-07-30-arr2-programmatic-pipeline.md`
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
## Current Focus
|
||||
|
||||
Local XML-upload improvement completed: processor4.1.0 ignores recognized emoji in text and narrowly recovers CESU-8 emoji while retaining original source bytes/hashes. Original71052877 now passes with4 ignored/67 retained;71052708 remains100.122 distinct Python/3 JS checks verified, including independent replay and frozen legacy behavior. Not deployed to the remote ARR runtime. See [emoji cleanup evidence](../50-evidence/topics/2026-09-17-xml-emoji-cleanup.md).
|
||||
|
||||
ARR2.0 owns the deterministic XML-to-Finance path and the complete post-commit monthly publication path. The user
|
||||
uploads XML once. After an accepted Finance commit, a dedicated worker consumes the durable outbox event, derives the
|
||||
month and “更新至” watermark from committed `ARRIVAL` facts, publishes a validated workbook, and records a durable
|
||||
|
||||
@@ -4,6 +4,7 @@
|
||||
|
||||
| Date | Task | Outcome | Docs Updated |
|
||||
|---|---|---|---|
|
||||
| 2026-09-17 | Ignore emoji during XML upload processing | Processor4.1.0, pinned Unicode17 text cleanup/narrow CESU-8 recovery, immutable source identity, replay-verified count and3-locale notice. Original71052877 succeeds4 ignored/67 retained;71052708 retains100.122 distinct Python and3 JS checks verified; no subagents/production rollout | [Evidence](../50-evidence/topics/2026-09-17-xml-emoji-cleanup.md), processor field/result contracts, current/history/domain/memory/evidence and scoped plan |
|
||||
| 2026-09-06 | Create a project explanation/handoff document and publish the repository | Added `PROJECT_GUIDE.md` with the business purpose, architecture, six main flows, ownership boundaries, modules, configuration, local operation, migrations, testing, deployment, troubleshooting, security and handoff checklist; linked it from README and kept unresolved deployment/business acceptance items explicit. Local links, Git whitespace and Web/worker CLI help pass. No application, database, OSS, runtime or business state changed | `PROJECT_GUIDE.md`, README, current state, task history, scoped planning record |
|
||||
| 2026-08-11 | Implement Lian Tai/QBD bilingual Booking-header compatibility | Bumped the bounded parser to 2.1.0, converted readable approved header labels into the existing exact normalized allowlist, preserved legacy aliases and fail-closed ambiguity, and added Chinese/English/Thai header-error mapping. Synthetic parser/coordinator tests, all four supplied workbook replays and wrong-report negatives pass; no API, migration, upload, draft, source activation or runtime deployment occurred | README, architecture/data-flow/business rules, current state, [header-parse evidence](../50-evidence/topics/2026-08-11-company-channel-booking-header-parse-failure.md), evidence index, stale deployment item |
|
||||
| 2026-08-11 | Make the top-right control the sole Daily task-log entry | Removed concrete-row click/Enter/Space log activation plus the row's button semantics, pointer, focus and selected styling. Explicit download and manual-review controls remain; isolated browser QA proved ordinary row → no dialog, review control → focused operation panel, header `任务日志` → dialog. JavaScript syntax and all 81 Web tests pass. The source is local and production deployment is still pending | Success criteria, current state, [interaction evidence](../50-evidence/topics/2026-08-11-daily-row-task-log-entry-boundary.md), evidence index, stale deployment item, scoped planning record |
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
## Durable Rules
|
||||
|
||||
- 用户已批准 XML 业务处理自动忽略表情。原文件字节与哈希保留；仅对可识别表情做文本清理/有限错误编码恢复，其他损坏编码、结构及业务校验照常拒绝。非零清理数量须可重放核对并提示；详见处理器 [字段契约](../../arr-opera-daily-ingest/references/field-contracts.md)。本地4.1.0实现，线上部署另行执行。
|
||||
|
||||
- Human access to the desktop Finance workspace, detailed health, generic business APIs, uploads, traces and downloads
|
||||
requires an authenticated ARR Web session. The H5 mobile dashboard is an intentional anonymous read-only exception:
|
||||
only its page/assets, `/api/public/h5/months`, `/api/public/h5/analytics` and no-detail `/healthz` are public; the H5
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# Evidence Index
|
||||
|
||||
- [XML emoji cleanup](topics/2026-09-17-xml-emoji-cleanup.md) — local processor4.1.0 ignores recognized emoji and narrowly recovers CESU-8 emoji; exact original71052877 passes with4 ignored/67 retained,71052708 remains100. Original bytes unchanged;122 distinct Python/3 JS checks verified; not deployed.
|
||||
|
||||
Use this index for searchable, traceable evidence records.
|
||||
|
||||
| Date | Topic | Status | Source | Detail |
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
# XML emoji cleanup — 2026-09-17
|
||||
|
||||
## Authorization and scope
|
||||
|
||||
The user approved automatic emoji ignoring after discussing the policy, then instructed implementation. Work was independent (no subagents). Existing unrelated repository changes were preserved. This is a local code/package change, not a production rollout or a new business upload.
|
||||
|
||||
## Behavior
|
||||
|
||||
- Active processor is 4.1.0; result/structured schema versions remain 4.0.
|
||||
- Original source bytes, size and SHA-256 stay unchanged in immutable storage and in artifact identities.
|
||||
- Normal Unicode emoji sequences are removed from parsed element text/tails, including CDATA and numeric character references. A pinned Unicode17 sequence table avoids blanket Unicode-block deletion. Chinese (including supplementary characters), Thai, normal digits, monetary symbols and dates are preserved.
|
||||
- Strict parser failures receive one narrow recovery attempt: only recognized emoji encoded as CESU-8 surrogate pairs in UTF-8 text/CDATA can be normalized. Markup, attributes, unrelated invalid bytes, non-emoji surrogate pairs and lone surrogates are not repaired. Strict XML and existing business checks still decide acceptance.
|
||||
- Optional positive `result.json.input_cleanup.ignored_emoji_count` is omitted for zero cleanup. Success/review independent validation recomputes the count from the original source; altered/missing/extra metadata is rejected. The count is returned in upload/final replay receipts and shown in the upload notice in Chinese, English and Thai.
|
||||
- The retired frozen v3 compatibility entry point keeps its previous strict XML behavior and original emoji text. The final v3 projection is independently checked before returning.
|
||||
- Rule/data semantics are documented in `arr-opera-daily-ingest/references/field-contracts.md`; zip/skill archives include all source resources and the Unicode license.
|
||||
|
||||
## Exact-file acceptance
|
||||
|
||||
Full ProgrammaticUploadCoordinator + independent DeliveryValidator + in-memory repository/private temporary filesystem object store were used. No production database, OSS credentials or hotel APIs were used.
|
||||
|
||||
| Original bytes | Business date | Source rows | Ignored emoji | Excluded | Retained | Full local pipeline |
|
||||
|---|---|---:|---:|---:|---:|---:|
|
||||
| 71052877 (currently renamed .xlsx, still raw XML) | 2026-09-01 | 161 | 4 | 94 | 67 | 4.99 s |
|
||||
| 71052708 XML | 2026-09-02 | 148 | 0 | 48 | 100 | 4.59 s |
|
||||
|
||||
SHA-256 identities:
|
||||
- 71052877: `e770d925fa1d99151aa96fe667c33f98a7927f5060f6a7845417d8eabff6fad1`
|
||||
- 71052708: `bbebe9211d162f95725792d89def4309afd5247ce84e15744a728ad5c908186b`
|
||||
|
||||
Both originals and their stored source objects are byte-for-byte unchanged. Test upload names used the required XML extension; the user's desktop filename was not renamed. The four original failures occur in TRACE_TEXT at lines2520,2527,4764,4771.
|
||||
|
||||
Final rule-set SHA-256: `414afd93e7d5efa55c06b62f6bd9b98924a138adfabbb083bbafe554652f7201`.
|
||||
|
||||
## Verification
|
||||
|
||||
- 122 distinct Python checks covered processor/archive/schema contracts, programmatic upload, independent ingestion validation/service, processing, legacy direct compatibility, deployment entry points and Web status/review/log behavior. The first121-case scope passed. After the legacy-isolation change the expanded122-case scope had only the new legacy test fail: its test harness incorrectly invoked the active LocalDailyProcessor without the explicit legacy switch. The helper was corrected to exercise the existing legacy entry point, and the complete13-test emoji module passed on the final code (15.965 s); the other109 checks passed in the expanded run. No product failure remains.
|
||||
- 3 executable JavaScript tests run the actual upload handler with synthetic responses and minimal DOM adapters: count notice in all three locales, unchanged review/failure states, and absent/invalid metadata. No real browser/production request was used.
|
||||
- JS syntax checks and `git diff --check` passed. Both rebuilt package archives match their source through the processor package contract tests.
|
||||
- Edge cases include plain and compound emoji (family, flags, skin tones, keycaps), XML escapes/CDATA, UTF-8 BOM and existing UTF-16 input, ordinary multilingual/business characters, invalid bytes/surrogates/markup, unsafe declarations, mixed business dates, empty required names, pure missing-price review/final replay, and tampered cleanup metadata.
|
||||
- A flat regex was replaced with a prefix-trie pattern after exact-file acceptance exposed ~30s processing overhead. The final same-file full pipeline measured about5s including processor/independent verification; no timing assertion was added to flaky environment-sensitive tests.
|
||||
|
||||
## Publication verification
|
||||
|
||||
The user confirmed `https://git.nianxx.cn/shiyuyun/wyndham-ARR.git` and authorized pushing this fix to `origin/main`. Only the emoji fix, tests, packages, checksums and associated documentation were staged; unrelated ARR download/integration work remains local.
|
||||
|
||||
A clean shallow clone of the confirmed remote received the staged patch. Its Git index tree was verified identical to the intended commit. In that isolated tree, all 122 Python tests passed in one run (111.340 s), all 3 JavaScript upload-handler tests passed, and both JavaScript syntax checks passed. The package checksum manifest was refreshed for the changed resources and two new Unicode files; all 35 listed checksums verified. No production deployment or business upload was performed.
|
||||
|
||||
## Deployment boundary
|
||||
|
||||
The remote ARR page was not deployed/restarted or used to submit these business files. Deploy the tested package/Web changes before expecting the live page to ignore emoji. Existing open manual-price cases remain bound to their original processor/rule identity and retain the existing cancel-and-reupload requirement across rule updates.
|
||||
|
||||
## Data source
|
||||
|
||||
Pinned data: [Unicode Emoji17 test sequences](https://www.unicode.org/Public/17.0.0/emoji/emoji-test.txt), source SHA-256 `1d8a944f88d7952f7ef7c5167fef3c67995bcae24543949710231b03a201acda`. Unicode License V3 is bundled. Runtime has no new network/dependency requirement.
|
||||
Reference in New Issue
Block a user