execl问题修复v2

This commit is contained in:
andy committed 2026-09-11 12:08:51 +08:00
1 parent 07b23fce9e
commit a4d55a5c55
3 files changed
+291 -7

No files matched your search

@@ -0,0 +1,59 @@
# 名单模板尾行兼容修复
- Task ID: `20260911-roster-template-tail-01a0858b`
- Mode: Feature
- Status: Complete (local implementation and validation)
- Authorization: 用户明确要求保留原始表格,通过修改程序兼容没有旅客资料的表尾模板行。
## Concurrent Worktree Check
- Branch: `main`
- Worktree: `/Users/andy/IdeaProjects/LWLT-AIBOT`
- Base commit: `07b23fce9e96badbd266bbda682fd16cd6733e22`
- Existing changes: 未跟踪 `.idea/` 和上一任务生成的 `dist/ltjt-order-assistant-0.5.173.zip`,均保留且不涉及;已有忽略的 Finder 文件也保留。
- Other local worktrees: 无。
- Overlap result: Clear;服务端名单 normalizer 及测试无既有改动。
## Scope and constraints
- 服务端只在读取时识别表尾的无旅客资料模板行,原始 XLS 不修改、不复制进仓库。
- 有实际旅客信息、位于名单中间或无法确定用途的行继续严格校验;保留全局不安全公式/外链/宏及隐藏数据检查。
- 本次不修改插件、多关键词搜索、数据库或 canonical 项目记忆;不部署、重启、访问或写入 ERP、发送外部通知。
- 使用合成测试夹具;本地原始 XLS 复验只输出结构、计数及哈希,不输出旅客信息。
- 遵守仓库不经用户同意不使用子 Agent 的约束,本任务自行实现与复核。
## Findings
- 原文件与生产日志的大小/哈希一致。表头在第 2 行,第 3–30 行共 28 条已填写名单。
- 第 31、32 行没有姓名和护照数据,但保留序号 29/30 及性别、出生日期、有效期公式;D31/E31/D32/E32 含失效 `#REF!` 引用。
- 当前 normalizer 在识别实际数据范围前校验整张工作表必需列的公式,因此在 D31 返回 `roster_workbook_formula_not_allowed`。
- 第 33–34 行是跨越全部名单列的合并备注区;只处理空公式行仍会把该备注当成名单,因此纳入有明确标签的表尾说明识别。
## Implementation
- Normalizer 更新为 `ltjt-passenger-roster-workbook-v1.5.0`;接口、数据库及 canonical TSV 格式保持原样。
- 从表尾倒序确定数据结束行,仅裁掉连续尾部空行、正整数序号占位行、没有任何实填旅客字段且仅含已知模板公式的行,以及覆盖全部导入列、带明确备注/说明标签且没有独立附加单元格内容的合并说明行。
- 对失效身份证派生的性别/出生日期公式仅识别原模板的确切公式形式;已有同一行 `EDATE` 日期派生公式仍可作为空模板标识。任意其他公式或任一实填字段都会保留该行为数据并严格校验。
- 识别只发生于连续尾部,不能越过有旅客信息或未知内容的行。名单中间仅有序号的占位行现在明确报 `roster_workbook_incomplete_row`;中间空行、公式错误和无效序号继续阻断。
- 全工作簿的宏、外链、危险公式、超链接、隐藏行/列检查仍保留,尾部识别不会绕开这些检查。
## Verification
- 回归先行:新增 8 项测试在旧实现上有 6 项失败,重现实际模板尾部问题;修复后新增和已有名单测试合计 20/20 PASS。
- 定向验证覆盖原始布局、列重排、序号占位、跨行合并备注、全空模板、实填部分数据、名单中间占位/空行/备注、未知公式和跨工作簿安全检查。
- 原始 XLS 只读链路 PASS:生产转换器 → normalizer → Program Parser → 插件派发校验;表头第 2 行,28 条完整 13 列记录,序号 1–28 原样保留,原文件哈希不变。未访问 ERP。
- `node --run check`:PASS;`node --run build`:PASS;`node --run test:control-plane`:202/202 PASS。
- `node --run check:repo`:原工作目录 8/10,通过项包含发布文件与哈希;两项失败仅为既有 `.idea/`/Finder 文件,已保留。在 `/private/tmp/ltjt-roster-tail-verify-f4emlqgc` 的当前源码与发布物副本中,排除上述杂项及秘密后 10/10 PASS。
- `node --run test:legacy`:累计 290/290 PASS。首次 288 项通过,另两项因沙箱禁止本地 127.0.0.1 监听而失败;单独申请权限时自动审批超时,按工具允许重试一次后获准,两项 HTTP 边界测试均通过。
- `git diff --check`:PASS。源码改动只涉及一个服务端 normalizer;新增测试和本任务记录,不改动现有插件与交付物。
- 最终自行复核完成:本任务仅涉及上述源码、新增测试及任务记录三个文件,既有改动均保留;未执行部署、外部发送或 ERP 操作。
## Outcome and follow-ups
- 原表无需编辑;服务端镜像重建并部署后,可在当前等待附件的任务中重新上传同一文件。现有接收逻辑会重新校验之前 rejected 的同一 SHA-256 附件,无需更换文件名或新建任务。
- 本次不需要更新插件或数据库。部署后的附件校验日志应显示 normalizer `v1.5.0` 和 `row_count: 28`。
- 生产部署及实际 ERP 验收由用户后续执行,本任务不自动重试业务任务或写入 ERP。
## Promotion Candidates
- Target: canonical current state / 名单输入契约。集成时登记 normalizer `v1.5.0` 对连续表尾空模板与明确合并备注的有限兼容,以及中间占位行错误码。证据为本任务合成回归和用户原始文件只读全链路验证。不等同于放开任意公式、跳过不完整旅客或声明 ERP 写入成功;本任务不写共享索引或集成快照。
+69 -7
View File
@@ -13,7 +13,7 @@ const ABSOLUTE_MAX_DATA_ROWS = 5_000;
const MAX_PASSENGER_SEQUENCE = 5_000;
const MAX_HEADER_ROW = 100;
export const PASSENGER_ROSTER_WORKBOOK_VERSION = 'ltjt-passenger-roster-workbook-v1.4.0';
export const PASSENGER_ROSTER_WORKBOOK_VERSION = 'ltjt-passenger-roster-workbook-v1.5.0';
const REQUIRED_SOURCE_FIELDS = [
'序号', '姓名', '英文姓名', '性别', '出生日期', '出生地', '护照号码',
@@ -70,6 +70,7 @@ export type PassengerRosterWorkbookErrorCode =
| 'roster_workbook_formula_not_allowed'
| 'roster_workbook_formula_result_missing'
| 'roster_workbook_non_contiguous_rows'
| 'roster_workbook_incomplete_row'
| 'roster_workbook_row_limit'
| 'roster_workbook_no_data'
| 'roster_workbook_invalid_sequence'
@@ -96,6 +97,7 @@ const ERROR_MESSAGES: Record<PassengerRosterWorkbookErrorCode, string> = {
roster_workbook_formula_not_allowed: '名单必需字段包含不允许的公式',
roster_workbook_formula_result_missing: '名单必需字段公式没有可验证的缓存结果',
roster_workbook_non_contiguous_rows: '名单数据行不连续',
roster_workbook_incomplete_row: '名单中间存在未填写旅客资料的模板行',
roster_workbook_row_limit: '名单数据行数超过技术限制',
roster_workbook_no_data: '名单工作簿没有数据行',
roster_workbook_invalid_sequence: '名单序号必须是正整数',
@@ -390,10 +392,63 @@ function rowHasData(row: ExcelJS.Row, sourceColumns: readonly number[]): boolean
return false;
}
// These two legacy template formulas derive gender/birth date from a deleted
// identity-card column. Recognize the exact broken templates, not arbitrary
// formulas or their cached results, when deciding whether a tail row is unused.
const BROKEN_TEMPLATE_FORMULAS: Partial<Record<RequiredSourceField, string>> = {
性别: 'IF(OR(LEN(#REF!)=15,LEN(#REF!)=18),IF(MOD(MID(#REF!,15,3)*1,2),"男","女"),NA())',
出生日期: 'DATE(MID(#REF!,7,VLOOKUP(LEN(#REF!),{15,2;18,4},2,0)),MID(#REF!,VLOOKUP(LEN(#REF!),{15,9;18,11},2,0),2),MID(#REF!,VLOOKUP(LEN(#REF!),{15,11;18,13},2,0),2))'
};
function unusedTemplateRow(row: ExcelJS.Row, headerRow: number, sourceColumns: SourceColumnMap): boolean {
const sequenceCell = row.getCell(sourceColumns['序号']);
if (sequenceCell.type === ExcelJS.ValueType.Formula) return false;
const sequence = normalizedText(scalarValue(sequenceCell));
if (sequence && (!/^[1-9]\d*$/.test(sequence) || Number(sequence) > MAX_PASSENGER_SEQUENCE)) return false;
for (const field of REQUIRED_SOURCE_FIELDS) {
if (field === '序号') continue;
const cell = row.getCell(sourceColumns[field]);
if (cell.type === ExcelJS.ValueType.Formula) {
const template = BROKEN_TEMPLATE_FORMULAS[field];
if (!formulaAllowed(cell, headerRow, sourceColumns)
&& (!template || normalizedFormula(cell.formula) !== normalizedFormula(template))) return false;
} else if (normalizedText(scalarValue(cell)) !== '') {
return false;
}
}
return true;
}
function mergedTemplateNote(row: ExcelJS.Row, headerRow: number, sourceColumns: SourceColumnMap): boolean {
const columns = Object.values(sourceColumns);
const master = row.getCell(Math.min(...columns)).master;
if (!master.isMerged || cellRow(master) <= headerRow
|| !columns.every(column => row.getCell(column).master.address === master.address)) return false;
// A footer must span every imported column and be explicitly labelled. Any
// separate populated cell keeps the row in the roster for normal validation.
let onlyNote = true;
row.eachCell({ includeEmpty: false }, cell => {
if (cell.master.address !== master.address && cellHasData(cell)) onlyNote = false;
});
return onlyNote && /^(?:备注|说明|注意事项|温馨提示)[::]/.test(headerText(master));
}
function lastRosterRow(sheet: ExcelJS.Worksheet, headerRow: number, sourceColumns: SourceColumnMap): number {
let lastRow = sheet.rowCount;
while (lastRow > headerRow) {
const row = sheet.getRow(lastRow);
if (!mergedTemplateNote(row, headerRow, sourceColumns)
&& !unusedTemplateRow(row, headerRow, sourceColumns)) break;
lastRow--;
}
return lastRow;
}
function validateFormulasAndHiddenData(
sheet: ExcelJS.Worksheet,
headerRow: number,
sourceColumns: SourceColumnMap
sourceColumns: SourceColumnMap,
lastDataRow: number
): void {
const selectedColumns = new Set(Object.values(sourceColumns));
sheet.eachRow({ includeEmpty: false }, (row, rowNumber) => {
@@ -405,7 +460,7 @@ function validateFormulasAndHiddenData(
if (containsExternalFormulaReference(formula)) {
fail('roster_workbook_unsafe_content', { row: rowNumber, column: columnNumber });
}
if (selectedColumns.has(columnNumber) && !formulaAllowed(cell, headerRow, sourceColumns)) {
if (rowNumber <= lastDataRow && selectedColumns.has(columnNumber) && !formulaAllowed(cell, headerRow, sourceColumns)) {
fail('roster_workbook_formula_not_allowed', { row: rowNumber, column: columnNumber });
}
}
@@ -547,12 +602,13 @@ function dataRows(
sheet: ExcelJS.Worksheet,
headerRow: number,
sourceColumns: SourceColumnMap,
maxDataRows: number
maxDataRows: number,
lastDataRow: number
): ExcelJS.Row[] {
const rows: ExcelJS.Row[] = [];
let gapAfterData = false;
const selectedColumns = Object.values(sourceColumns);
for (let rowNumber = headerRow + 1; rowNumber <= sheet.rowCount; rowNumber += 1) {
for (let rowNumber = headerRow + 1; rowNumber <= lastDataRow; rowNumber += 1) {
const row = sheet.getRow(rowNumber);
const hasData = rowHasData(row, selectedColumns);
if (!hasData) {
@@ -560,6 +616,9 @@ function dataRows(
continue;
}
if (gapAfterData) fail('roster_workbook_non_contiguous_rows', { row: rowNumber });
if (unusedTemplateRow(row, headerRow, sourceColumns)) {
fail('roster_workbook_incomplete_row', { row: rowNumber, column: sourceColumns['姓名'] });
}
rows.push(row);
if (rows.length > maxDataRows) fail('roster_workbook_row_limit');
}
@@ -654,8 +713,11 @@ export async function normalizePassengerRosterWorkbook(
if (sheet.state !== 'visible') fail('roster_workbook_hidden_data');
const header = findHeaderLayout(sheet);
const headerRow = header.row;
validateFormulasAndHiddenData(sheet, headerRow, header.columns);
const rows = dataRows(sheet, headerRow, header.columns, maxDataRows);
const lastDataRow = lastRosterRow(sheet, headerRow, header.columns);
// Content safety and hidden-data checks still cover the entire workbook,
// including discarded placeholders and footnotes.
validateFormulasAndHiddenData(sheet, headerRow, header.columns, lastDataRow);
const rows = dataRows(sheet, headerRow, header.columns, maxDataRows, lastDataRow);
const date1904 = workbook.properties.date1904 === true;
const seenSequences = new Set<number>();
const canonicalRows = rows.map((row) => canonicalRow(row, date1904, seenSequences, header.columns));
@@ -0,0 +1,163 @@
import assert from 'node:assert/strict';
import test from 'node:test';
import ExcelJS from 'exceljs';
import { parseProgramInput } from '../src/program-parser.js';
import { normalizePassengerRosterWorkbook, PassengerRosterWorkbookError } from '../src/passenger-roster-workbook.js';
const headers = ['序号', '姓名', '英文姓名', '性别', '出生日期', '出生地', '护照号码', '签发地', '签发日期', '有效期', '电话', '备注'];
const genderFormula = 'IF(OR(LEN(#REF!)=15,LEN(#REF!)=18),IF(MOD(MID(#REF!,15,3)*1,2),"男","女"),NA())';
const birthFormula = 'DATE(MID(#REF!,7,VLOOKUP(LEN(#REF!),{15,2;18,4},2,0)),MID(#REF!,VLOOKUP(LEN(#REF!),{15,9;18,11},2,0),2),MID(#REF!,VLOOKUP(LEN(#REF!),{15,11;18,13},2,0),2))';
type Mutation = (sheet: ExcelJS.Worksheet) => void;
function addPlaceholder(sheet: ExcelJS.Worksheet, row: number, sequence: number): void {
sheet.getCell(`A${row}`).value = sequence;
sheet.getCell(`D${row}`).value = { formula: genderFormula, result: { error: '#REF!' } };
sheet.getCell(`E${row}`).value = { formula: birthFormula, result: { error: '#REF!' } };
sheet.getCell(`J${row}`).value = { formula: `EDATE(I${row},10*12)-1`, result: 36500 };
}
function addNote(sheet: ExcelJS.Worksheet, firstRow: number, lastRow = firstRow): void {
sheet.mergeCells(`A${firstRow}:L${lastRow}`);
sheet.getCell(`A${firstRow}`).value = { richText: [{ text: '备注:' }, { text: '合成模板用房说明' }] };
}
async function workbook(...mutations: Mutation[]): Promise<Buffer> {
const book = new ExcelJS.Workbook();
const sheet = book.addWorksheet('Sheet1');
sheet.addRow(['合成名单模板']);
sheet.addRow(headers);
for (let sequence = 1; sequence <= 28; sequence++) {
const row = sequence + 2;
sheet.addRow([sequence, `合成游客${sequence}`, `SYNTHETIC ${sequence}`, '男', '1990-01-02', '合成城市',
`SYNTH${sequence}`, '合成城市', '2025-01-02',
{ formula: `EDATE(I${row},10*12)-1`, result: new Date('2035-01-01T00:00:00Z') },
sequence === 1 ? '13800000001' : '', sequence === 1 ? '领队' : '']);
}
for (const mutate of mutations) mutate(sheet);
return Buffer.from(await book.xlsx.writeBuffer());
}
const normalize = (content: Buffer) => normalizePassengerRosterWorkbook({ content, fileName: 'synthetic.xlsx', contentType: 'application/octet-stream' });
async function rejects(content: Buffer, code: string, row?: number): Promise<void> {
await assert.rejects(() => normalize(content), (error: unknown) => {
assert.ok(error instanceof PassengerRosterWorkbookError);
assert.equal(error.code, code);
if (row !== undefined) assert.equal(error.row, row);
return true;
});
}
test('roster tail: 28 passengers survive stale template formulas and a merged footer unchanged', async () => {
const expected = await normalize(await workbook());
const content = await workbook(sheet => {
addPlaceholder(sheet, 31, 29);
addPlaceholder(sheet, 32, 30);
addNote(sheet, 33, 34);
});
const result = await normalize(content);
assert.equal(result.headerRow, 2);
assert.equal(result.rowCount, 28);
assert.equal(result.canonicalTsv, expected.canonicalTsv);
const parsed = await parseProgramInput({
rawText: `导入独立团名单\n单号:SYNTH-ORDER-001\n名单内容:\n${result.canonicalTsv}`,
receivedAt: '2026-09-11T03:21:08Z'
});
assert.equal(parsed.status, 'agent_parse_passed');
assert.equal((parsed.operation as any).data.passenger_list.row_count, 28);
assert.deepEqual((parsed.operation as any).data.passenger_list.leader_contact, { sequence: 1, name: '合成游客1', phone: '13800000001' });
});
test('roster tail: sequence-only placeholders and separated trailing notes are not passengers', async () => {
for (const mutate of [
(sheet: ExcelJS.Worksheet) => { sheet.getCell('A31').value = 29; sheet.getCell('A32').value = 30; },
(sheet: ExcelJS.Worksheet) => { addNote(sheet, 32, 33); },
(sheet: ExcelJS.Worksheet) => { addPlaceholder(sheet, 32, 29); addNote(sheet, 34); }
]) {
assert.equal((await normalize(await workbook(mutate))).rowCount, 28);
}
});
test('roster tail: recognition follows semantic columns after reordering', async () => {
const expected = await normalize(await workbook());
const result = await normalize(await workbook(sheet => {
addPlaceholder(sheet, 31, 29);
addNote(sheet, 32, 33);
for (let row = 2; row <= 31; row++) {
const first = sheet.getCell(`A${row}`).value;
sheet.getCell(`A${row}`).value = sheet.getCell(`L${row}`).value;
sheet.getCell(`L${row}`).value = first;
}
}));
assert.equal(result.canonicalTsv, expected.canonicalTsv);
});
test('roster tail: any supplied passenger field keeps a row subject to strict formula validation', async () => {
for (const [column, value] of [['B', '合成姓名'], ['C', 'SYNTH NAME'], ['F', '合成出生地'],
['G', 'SYNTH-PASSPORT'], ['H', '合成签发地'], ['I', '2025-01-01'], ['K', '13800000001'], ['L', '领队']]) {
await rejects(await workbook(sheet => {
addPlaceholder(sheet, 31, 29);
sheet.getCell(`${column}31`).value = value;
}), 'roster_workbook_formula_not_allowed', 31);
}
await rejects(await workbook(sheet => {
sheet.getCell('D30').value = { formula: genderFormula, result: { error: '#REF!' } };
addPlaceholder(sheet, 31, 29);
}), 'roster_workbook_formula_not_allowed', 30);
});
test('roster tail: unrecognized formulas and notes cannot be silently discarded', async () => {
for (const formula of ['"男"', 'IF(A3=1,"男","女")', 'IFERROR(#REF!,"男")']) {
await rejects(await workbook(sheet => {
addPlaceholder(sheet, 31, 29);
sheet.getCell('D31').value = { formula, result: '男' };
}), 'roster_workbook_formula_not_allowed', 31);
}
await rejects(await workbook(sheet => { sheet.getCell('A31').value = '未知尾部内容'; }), 'roster_workbook_invalid_sequence', 31);
await rejects(await workbook(sheet => {
addNote(sheet, 31);
sheet.getCell('M31').value = '附加实际内容';
}), 'roster_workbook_invalid_sequence', 31);
});
test('roster tail: placeholders or notes inside the roster never hide later passengers', async () => {
await rejects(await workbook(sheet => {
sheet.getRow(15).values = [];
addPlaceholder(sheet, 15, 13);
}), 'roster_workbook_formula_not_allowed', 15);
await rejects(await workbook(sheet => {
sheet.getRow(15).values = [13];
}), 'roster_workbook_incomplete_row', 15);
await rejects(await workbook(sheet => {
sheet.getRow(15).values = [];
}), 'roster_workbook_non_contiguous_rows', 16);
await rejects(await workbook(sheet => {
sheet.getRow(15).values = [];
addNote(sheet, 15);
}), 'roster_workbook_invalid_sequence', 15);
});
test('roster tail: empty templates do not create phantom passengers', async () => {
await rejects(await workbook(sheet => {
for (let row = 3; row <= 30; row++) {
sheet.getRow(row).values = [];
addPlaceholder(sheet, row, row - 2);
}
addNote(sheet, 31);
}), 'roster_workbook_no_data');
});
test('roster tail: active content and hidden required data remain blocked across the whole sheet', async () => {
for (const column of ['D', 'M']) {
await rejects(await workbook(sheet => {
addPlaceholder(sheet, 31, 29);
sheet.getCell(`${column}31`).value = { formula: 'WEBSERVICE("https://invalid.example")', result: '' };
}), 'roster_workbook_unsafe_content', 31);
}
await rejects(await workbook(sheet => {
addPlaceholder(sheet, 31, 29);
sheet.getCell('M31').value = { text: '链接', hyperlink: 'https://invalid.example' };
}), 'roster_workbook_unsafe_content');
await rejects(await workbook(sheet => { addPlaceholder(sheet, 31, 29); sheet.getRow(31).hidden = true; }), 'roster_workbook_hidden_data', 31);
await rejects(await workbook(sheet => { addPlaceholder(sheet, 31, 29); sheet.getColumn('D').hidden = true; }), 'roster_workbook_hidden_data');
});