feat: sync latest ARR implementation
This commit is contained in:
Wyndham ARR committed
1 parent
d6f8a747fa
commit
bf7939dd1a
185 files changed
+17527
-2260
No files matched your search
@@ -1,55 +1,13 @@
|
||||
# ARR remote processing orchestration
|
||||
# ARR2 deterministic processing
|
||||
|
||||
`arr_processing` is the ARR-owned orchestration boundary between a registered XML source and the existing validated ingestion pipeline. The runner never sees an OSS credential, local Agent path, XML bytes, guest data or database credential. The input-message builder resolves one ARR-owned committed object and emits only the non-secret OSS routing that the existing `fetch_oss_file` provider needs.
|
||||
The active ARR2 path uses two modules:
|
||||
|
||||
## Request
|
||||
- `policy.py` loads the frozen processor version and rule-set SHA-256 from `arr-opera-daily-ingest`.
|
||||
- `local.py` runs `process_daily.py` in an isolated directory with a bounded timeout, requires process exit status to
|
||||
agree with both JSON artifacts, and permits only basename-confined declared XLSX outputs.
|
||||
|
||||
`ProcessingRequest` serializes exactly six values as canonical JSON:
|
||||
The caller stores the success/failure artifacts and passes a canonical `DeliveryEnvelope` to `arr_ingestion`. Processor
|
||||
stdout/stderr, temporary paths and source bytes never enter the public response or database.
|
||||
|
||||
- contract `arr-opera-daily-request-1`;
|
||||
- opaque `job_id` and `source_file_id`;
|
||||
- attempt number;
|
||||
- approved processor version and rule-set SHA-256.
|
||||
|
||||
The database request remains the six-field opaque identity. Immediately before dispatch, `OssProcessingMessageBuilder` resolves the registered immutable source and builds `arr-opera-daily-program-input-2`: one `oss_attachments` item containing the canonical filename, bucket, endpoint host, exact object key, SHA-256 and byte size, plus one short-lived attempt-bound `arr_submit_processing_result` grant. The grant is excluded from `ProcessingRequest.to_dict()`, its canonical hash and repr; it exists only in the remote program message. No AccessKey, XML bytes, URL, local path, database credential or MCP transport bearer enters that message. `OpenAgentProcessingTransport` sends the canonical JSON through the text Session/Run API with a stable dispatch idempotency key.
|
||||
|
||||
## Runner semantics
|
||||
|
||||
`ProcessingRunner` reserves one request identity, correlates one remote run, and supports start, bounded polling, cancellation and separately delivered callbacks. Retryable 408/429/5xx/network failures reuse the same idempotency key. A 409 is accepted only if the shared ledger already has the exact correlated run; an uncorrelated active run remains a safe retryable error.
|
||||
|
||||
The in-memory ledger is for tests/local slices. `PostgresProcessingState` is the production-facing implementation against migration 008: reservation identity, `remote_run_id`, status and callback replay are read from the shared ARR processing tables rather than process memory. A submission that is rejected, exhausts its transport retries, or returns an invalid start response is recorded as a terminal failed attempt/run; an identical replay is not silently submitted again.
|
||||
|
||||
Remote `success` is not an ingestion decision. Empty `final_content`, unsigned JSON, free-form text, the old internal finalizer object with `local_path`, a mismatched run, or a terminal remote failure can never construct a `DeliveryEnvelope`.
|
||||
|
||||
## Authenticated result
|
||||
|
||||
The trusted runtime/callback adapter—not the Agent—must register each generated file and produce the path-free result defined by `prompts/arr_opera_daily_processing_result.schema.json`:
|
||||
|
||||
- contract `arr-opera-daily-result-1`;
|
||||
- delivery/job/attempt/run correlation;
|
||||
- processor, rule-set and result Schema identities;
|
||||
- business success/failure shape;
|
||||
- opaque handles plus filename, SHA-256, exact byte size and MIME for `daily_report`, `result_json`, `structured_result_json` and `exception_report`.
|
||||
|
||||
`SignedResultCodec` authenticates this payload with HMAC-SHA256, key ID, UTC issue time and nonce. Keys must be at least 32 bytes, injected from platform secret management, rotated by key ID, and unavailable to the Agent. The default acceptance window is ten minutes. Callback replay is idempotent only for identical signed bytes; the same delivery ID with different signed bytes is a conflict.
|
||||
|
||||
## Output registration
|
||||
|
||||
`ProcessingOutputRegistrar` reads every opaque output handle through a controlled `RemoteFilePort`. The deployed implementation, `PrefixRemoteFilePort`, resolves a handle only below `ARR_AGENT_OUTPUT_PREFIX` in the configured private OSS bucket. For every handle it:
|
||||
|
||||
1. materializes into a fresh private directory with the role size cap;
|
||||
2. rejects missing, symlinked, non-regular, oversized or hash-mismatched bytes;
|
||||
3. uploads through the staged/committed `ManagedObjectStore` path;
|
||||
4. converts only committed identities to the strict ARR `DeliveryEnvelope 1.0`.
|
||||
|
||||
ARR then runs the existing `DeliveryValidator` and PostgreSQL transaction. The signed remote result never bypasses independent replay and never writes Finance facts by itself.
|
||||
|
||||
## Runtime writeback adapter
|
||||
|
||||
`python -m arr_processing.runtime_writeback` is the trusted post-Skill adapter. It accepts only an approved output root and the frozen `FROZEN_AGENT_RESULT`, rejects symlinks/path escape and filename mismatches, rehashes all files, publishes them under deterministic opaque handles, creates a stable delivery ID, signs the canonical result with the independent callback HMAC key, and posts the exact same signed bytes on retryable failures.
|
||||
|
||||
The runtime adapter receives OSS and callback credentials from deployment secret injection. They are not placed in the Agent prompt or frozen result. Its contract and required variables are documented in [`AGENT_WRITEBACK_CONTRACT.md`](../AGENT_WRITEBACK_CONTRACT.md).
|
||||
|
||||
## Current production boundary
|
||||
|
||||
The text Open API submit/get/cancel contract, direct OSS attachment-shaped input, PostgreSQL callback state, OSS V2 output exchange, runtime-side signer/callback and ARR-side independent validation/commit path are implemented and tested. Deployment still needs an API-enabled published Agent Profile, the actual OSS runtime permissions, an independent HMAC key, a public callback URL and invocation of the writeback adapter after the Skill finishes. Live end-to-end connectivity must not be claimed until both readiness flags are true and one non-sensitive vertical-slice job has committed.
|
||||
Older remote-run, signature and callback modules remain as ARR1 compatibility/audit code. The ARR2 Web entrypoint does
|
||||
not import them, root requirements do not install their HTTP client, and Compose exposes no corresponding service.
|
||||
@@ -0,0 +1,469 @@
|
||||
"""Privacy-minimized persistence and projection for SuperAgent trace SSE events."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import threading
|
||||
from datetime import datetime, timezone
|
||||
from hashlib import sha256
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Mapping, Optional, Protocol, Union
|
||||
|
||||
from agent_integration.client import OpenAgentEvent
|
||||
from arr_ingestion.contracts import OPAQUE_ID_RE
|
||||
|
||||
|
||||
TRACE_RECORD_VERSION = "arr-agent-trace-event-1"
|
||||
MAX_TRACE_EVENTS = 10_000
|
||||
MAX_TRACE_FILE_BYTES = 8 * 1024 * 1024
|
||||
MAX_TRACE_SUMMARY_CHARS = 1_200
|
||||
|
||||
_LEVELS = frozenset({"info", "success", "warning", "error"})
|
||||
_CODE_RE = re.compile(r"^[A-Z][A-Z0-9_]{0,95}$")
|
||||
_SAFE_DETAIL_KEYS = frozenset(
|
||||
{
|
||||
"attempt_no",
|
||||
"remote_run_id",
|
||||
"remote_status",
|
||||
"trace_event",
|
||||
"invalid_records",
|
||||
"capture_error",
|
||||
}
|
||||
)
|
||||
_SENSITIVE_ASSIGNMENT_RE = re.compile(
|
||||
r"(?i)\b(submission[_ -]?grant|access[_ -]?key(?:[_ -]?(?:id|secret))?|"
|
||||
r"api[_ -]?key|secret|token|password|authorization|cookie)\b"
|
||||
r"\s*[:=]\s*(?:[\"']?)[^\s,;}\]]+"
|
||||
)
|
||||
_BEARER_RE = re.compile(r"(?i)\bbearer\s+[A-Za-z0-9._~+/=-]+")
|
||||
_URL_RE = re.compile(r"(?i)\b(?:https?|oss)://[^\s<>{}\[\]\"']+")
|
||||
_PATH_RE = re.compile(
|
||||
r"(?<![A-Za-z0-9._-])/(?:Users|home|private|tmp|var|opt|etc)(?:/[^\s,;}\]]+)+"
|
||||
)
|
||||
_EMAIL_RE = re.compile(r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b")
|
||||
_LONG_NUMBER_RE = re.compile(r"(?<!\d)(?:\+?\d[\d ()-]{7,}\d)(?!\d)")
|
||||
_XML_FRAGMENT_RE = re.compile(r"<[^>\r\n]{1,240}>")
|
||||
|
||||
|
||||
class AgentTraceStoreError(RuntimeError):
|
||||
"""Raised when the local trace store cannot safely persist a record."""
|
||||
|
||||
|
||||
class AgentTraceSink(Protocol):
|
||||
def append(self, job_id: str, record: Mapping[str, Any]) -> None:
|
||||
...
|
||||
|
||||
|
||||
class AgentTraceReader(Protocol):
|
||||
def read(self, job_id: str) -> List[Dict[str, Any]]:
|
||||
...
|
||||
|
||||
|
||||
def _utc_now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
def _timestamp(value: Any, *, fallback: Optional[datetime] = None) -> str:
|
||||
if isinstance(value, datetime):
|
||||
parsed = value
|
||||
elif isinstance(value, str):
|
||||
try:
|
||||
parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
except ValueError:
|
||||
parsed = fallback or _utc_now()
|
||||
else:
|
||||
parsed = fallback or _utc_now()
|
||||
if parsed.tzinfo is None:
|
||||
parsed = parsed.replace(tzinfo=timezone.utc)
|
||||
return parsed.astimezone(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def sanitize_trace_summary(value: Any) -> Optional[str]:
|
||||
"""Keep the server's operational summary while removing common secret/PII forms."""
|
||||
|
||||
if not isinstance(value, str):
|
||||
return None
|
||||
text = " ".join(value.replace("\x00", " ").split())
|
||||
if not text:
|
||||
return None
|
||||
text = _BEARER_RE.sub("Bearer [REDACTED]", text)
|
||||
text = _SENSITIVE_ASSIGNMENT_RE.sub(lambda match: f"{match.group(1)}=[REDACTED]", text)
|
||||
text = _URL_RE.sub("[URL]", text)
|
||||
text = _PATH_RE.sub("[PATH]", text)
|
||||
text = _EMAIL_RE.sub("[EMAIL]", text)
|
||||
text = _LONG_NUMBER_RE.sub("[NUMBER]", text)
|
||||
text = _XML_FRAGMENT_RE.sub("[XML]", text)
|
||||
return text[:MAX_TRACE_SUMMARY_CHARS]
|
||||
|
||||
|
||||
def trace_run_id(event: OpenAgentEvent) -> Optional[str]:
|
||||
"""Return the run id only from the documented trace run-start event."""
|
||||
|
||||
payload = event.data
|
||||
if event.event != "trace" or not isinstance(payload, Mapping):
|
||||
return None
|
||||
if payload.get("event") != "run.started":
|
||||
return None
|
||||
run_id = payload.get("run_id")
|
||||
if not isinstance(run_id, str) or OPAQUE_ID_RE.fullmatch(run_id) is None:
|
||||
return None
|
||||
return run_id
|
||||
|
||||
|
||||
def project_trace_event(
|
||||
event: OpenAgentEvent,
|
||||
*,
|
||||
attempt_no: int,
|
||||
sequence: int,
|
||||
now: Optional[datetime] = None,
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""Project one raw SSE event into the strict local diagnostic contract."""
|
||||
|
||||
fallback = now or _utc_now()
|
||||
if event.event == "end":
|
||||
return _record(
|
||||
sequence=sequence,
|
||||
timestamp=_timestamp(None, fallback=fallback),
|
||||
level="info",
|
||||
code="AGENT_TRACE_STREAM_ENDED",
|
||||
title="SuperAgent trace 流已结束",
|
||||
message="SuperAgent 已关闭本次运行的 trace 连接。",
|
||||
details={"attempt_no": attempt_no, "trace_event": "end"},
|
||||
)
|
||||
if event.event != "trace" or not isinstance(event.data, Mapping):
|
||||
return None
|
||||
|
||||
payload = event.data
|
||||
name = payload.get("event")
|
||||
if not isinstance(name, str) or not name:
|
||||
return None
|
||||
if name in {"message.delta", "message.final"}:
|
||||
# Assistant text can contain result payloads, grants, object routes or PII.
|
||||
return None
|
||||
|
||||
nested = payload.get("data")
|
||||
summary = sanitize_trace_summary(
|
||||
nested.get("summary") if isinstance(nested, Mapping) else None
|
||||
)
|
||||
run_id = payload.get("run_id")
|
||||
if not isinstance(run_id, str) or OPAQUE_ID_RE.fullmatch(run_id) is None:
|
||||
run_id = None
|
||||
status = payload.get("status")
|
||||
if not isinstance(status, str) or not status or len(status) > 32:
|
||||
status = None
|
||||
details: Dict[str, Any] = {
|
||||
"attempt_no": attempt_no,
|
||||
"trace_event": name,
|
||||
"remote_run_id": run_id,
|
||||
"remote_status": status,
|
||||
}
|
||||
timestamp = _timestamp(payload.get("ts"), fallback=fallback)
|
||||
|
||||
if name == "run.started":
|
||||
return _record(
|
||||
sequence=sequence,
|
||||
timestamp=timestamp,
|
||||
level="info",
|
||||
code="AGENT_RUN_STARTED",
|
||||
title="SuperAgent 运行已启动",
|
||||
message="已建立包含任务 trace 的 SuperAgent 运行流。",
|
||||
details=details,
|
||||
)
|
||||
if name == "task.updated":
|
||||
if summary is None:
|
||||
return None
|
||||
return _record(
|
||||
sequence=sequence,
|
||||
timestamp=timestamp,
|
||||
level="info",
|
||||
code="AGENT_TASK_UPDATED",
|
||||
title="Agent 任务状态更新",
|
||||
message=summary,
|
||||
details=details,
|
||||
)
|
||||
if name == "step.updated":
|
||||
if summary is None:
|
||||
return None
|
||||
return _record(
|
||||
sequence=sequence,
|
||||
timestamp=timestamp,
|
||||
level="info",
|
||||
code="AGENT_STEP_UPDATED",
|
||||
title="Agent 执行步骤更新",
|
||||
message=summary,
|
||||
details=details,
|
||||
)
|
||||
if name == "run.completed":
|
||||
succeeded = status in {"success", "completed", "succeeded"}
|
||||
return _record(
|
||||
sequence=sequence,
|
||||
timestamp=timestamp,
|
||||
level="success" if succeeded else "warning",
|
||||
code="AGENT_RUN_COMPLETED",
|
||||
title="SuperAgent 运行已结束",
|
||||
message=summary or "SuperAgent 已报告运行终态；业务成功仍以 ARR 数据库提交为准。",
|
||||
details=details,
|
||||
)
|
||||
if name in {"run.failed", "error"}:
|
||||
return _record(
|
||||
sequence=sequence,
|
||||
timestamp=timestamp,
|
||||
level="error",
|
||||
code="AGENT_RUN_FAILED",
|
||||
title="SuperAgent 运行失败",
|
||||
message=summary or "SuperAgent trace 报告运行失败。",
|
||||
details=details,
|
||||
)
|
||||
if name in {"run.cancelled", "run.canceled"}:
|
||||
return _record(
|
||||
sequence=sequence,
|
||||
timestamp=timestamp,
|
||||
level="warning",
|
||||
code="AGENT_RUN_CANCELLED",
|
||||
title="SuperAgent 运行已取消",
|
||||
message=summary or "SuperAgent trace 报告运行已取消。",
|
||||
details=details,
|
||||
)
|
||||
if summary is None:
|
||||
return None
|
||||
return _record(
|
||||
sequence=sequence,
|
||||
timestamp=timestamp,
|
||||
level="info",
|
||||
code="AGENT_TRACE_EVENT",
|
||||
title="Agent trace 事件",
|
||||
message=summary,
|
||||
details=details,
|
||||
)
|
||||
|
||||
|
||||
def capture_failure_record(
|
||||
*,
|
||||
attempt_no: int,
|
||||
sequence: int,
|
||||
error: BaseException,
|
||||
now: Optional[datetime] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""Create a safe local event without serializing the exception message."""
|
||||
|
||||
error_name = type(error).__name__
|
||||
if not re.fullmatch(r"[A-Za-z][A-Za-z0-9_]{0,95}", error_name):
|
||||
error_name = "Exception"
|
||||
return _record(
|
||||
sequence=sequence,
|
||||
timestamp=_timestamp(now or _utc_now()),
|
||||
level="warning",
|
||||
code="AGENT_TRACE_CAPTURE_FAILED",
|
||||
title="SuperAgent trace 采集中断",
|
||||
message="远端运行可能仍在继续；当前日志连接未能完整消费。",
|
||||
details={
|
||||
"attempt_no": attempt_no,
|
||||
"trace_event": "capture.failed",
|
||||
"capture_error": error_name,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _record(
|
||||
*,
|
||||
sequence: int,
|
||||
timestamp: str,
|
||||
level: str,
|
||||
code: str,
|
||||
title: str,
|
||||
message: str,
|
||||
details: Optional[Mapping[str, Any]] = None,
|
||||
) -> Dict[str, Any]:
|
||||
value: Dict[str, Any] = {
|
||||
"version": TRACE_RECORD_VERSION,
|
||||
"id": f"agent.{sequence:06d}.{code.lower()}",
|
||||
"timestamp": timestamp,
|
||||
"stage": "agent",
|
||||
"level": level,
|
||||
"code": code,
|
||||
"title": title,
|
||||
"message": message,
|
||||
}
|
||||
safe_details = _validated_details(details or {})
|
||||
if safe_details:
|
||||
value["details"] = safe_details
|
||||
return value
|
||||
|
||||
|
||||
def _validated_details(value: Mapping[str, Any]) -> Dict[str, Any]:
|
||||
output: Dict[str, Any] = {}
|
||||
for key, item in value.items():
|
||||
if key not in _SAFE_DETAIL_KEYS or item is None or item == "":
|
||||
continue
|
||||
if key in {"attempt_no", "invalid_records"}:
|
||||
if isinstance(item, int) and not isinstance(item, bool) and item >= 0:
|
||||
output[key] = item
|
||||
elif key == "remote_run_id":
|
||||
if isinstance(item, str) and OPAQUE_ID_RE.fullmatch(item) is not None:
|
||||
output[key] = item
|
||||
elif key == "capture_error":
|
||||
if isinstance(item, str) and re.fullmatch(r"[A-Za-z][A-Za-z0-9_]{0,95}", item):
|
||||
output[key] = item
|
||||
elif isinstance(item, str) and re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9._:-]{0,127}", item):
|
||||
output[key] = item
|
||||
return output
|
||||
|
||||
|
||||
def validate_trace_record(record: Mapping[str, Any]) -> Dict[str, Any]:
|
||||
required = {
|
||||
"version",
|
||||
"id",
|
||||
"timestamp",
|
||||
"stage",
|
||||
"level",
|
||||
"code",
|
||||
"title",
|
||||
"message",
|
||||
}
|
||||
optional = {"details"}
|
||||
if not isinstance(record, Mapping) or not required <= set(record) <= required | optional:
|
||||
raise AgentTraceStoreError("trace record field set is invalid")
|
||||
if record.get("version") != TRACE_RECORD_VERSION or record.get("stage") != "agent":
|
||||
raise AgentTraceStoreError("trace record version or stage is invalid")
|
||||
if record.get("level") not in _LEVELS:
|
||||
raise AgentTraceStoreError("trace record level is invalid")
|
||||
code = record.get("code")
|
||||
if not isinstance(code, str) or _CODE_RE.fullmatch(code) is None:
|
||||
raise AgentTraceStoreError("trace record code is invalid")
|
||||
timestamp = _bounded_text(record.get("timestamp"), 64, "timestamp")
|
||||
try:
|
||||
parsed_timestamp = datetime.fromisoformat(timestamp.replace("Z", "+00:00"))
|
||||
except ValueError as error:
|
||||
raise AgentTraceStoreError("trace record timestamp is invalid") from error
|
||||
if parsed_timestamp.tzinfo is None:
|
||||
raise AgentTraceStoreError("trace record timestamp is invalid")
|
||||
title = sanitize_trace_summary(_bounded_text(record.get("title"), 160, "title"))
|
||||
message = sanitize_trace_summary(
|
||||
_bounded_text(record.get("message"), MAX_TRACE_SUMMARY_CHARS, "message")
|
||||
)
|
||||
if title is None or message is None:
|
||||
raise AgentTraceStoreError("trace record text is invalid")
|
||||
output: Dict[str, Any] = {
|
||||
"version": TRACE_RECORD_VERSION,
|
||||
"id": _bounded_text(record.get("id"), 180, "id"),
|
||||
"timestamp": parsed_timestamp.astimezone(timezone.utc).isoformat(),
|
||||
"stage": "agent",
|
||||
"level": str(record["level"]),
|
||||
"code": code,
|
||||
"title": title,
|
||||
"message": message,
|
||||
}
|
||||
details = record.get("details")
|
||||
if details is not None:
|
||||
if not isinstance(details, Mapping):
|
||||
raise AgentTraceStoreError("trace record details are invalid")
|
||||
safe_details = _validated_details(details)
|
||||
if safe_details:
|
||||
output["details"] = safe_details
|
||||
return output
|
||||
|
||||
|
||||
def _bounded_text(value: Any, maximum: int, label: str) -> str:
|
||||
if not isinstance(value, str) or not value or len(value) > maximum:
|
||||
raise AgentTraceStoreError(f"trace record {label} is invalid")
|
||||
return value
|
||||
|
||||
|
||||
class JsonlAgentTraceStore:
|
||||
"""Append-only local trace store; raw SSE payloads are never written."""
|
||||
|
||||
def __init__(self, root: Union[str, Path]) -> None:
|
||||
self.root = Path(root)
|
||||
self._lock = threading.RLock()
|
||||
|
||||
def append(self, job_id: str, record: Mapping[str, Any]) -> None:
|
||||
path = self._path(job_id)
|
||||
value = validate_trace_record(record)
|
||||
line = (
|
||||
json.dumps(
|
||||
value,
|
||||
ensure_ascii=False,
|
||||
allow_nan=False,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
)
|
||||
+ "\n"
|
||||
).encode("utf-8")
|
||||
try:
|
||||
with self._lock:
|
||||
if self.root.is_symlink() or (
|
||||
self.root.exists() and not self.root.is_dir()
|
||||
):
|
||||
raise AgentTraceStoreError("agent trace root is invalid")
|
||||
self.root.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
flags = os.O_WRONLY | os.O_CREAT | os.O_APPEND
|
||||
if hasattr(os, "O_NOFOLLOW"):
|
||||
flags |= os.O_NOFOLLOW
|
||||
descriptor = os.open(path, flags, 0o600)
|
||||
try:
|
||||
remaining = memoryview(line)
|
||||
while remaining:
|
||||
written = os.write(descriptor, remaining)
|
||||
if written <= 0:
|
||||
raise OSError("trace append made no progress")
|
||||
remaining = remaining[written:]
|
||||
finally:
|
||||
os.close(descriptor)
|
||||
except (OSError, ValueError) as error:
|
||||
raise AgentTraceStoreError("cannot append agent trace record") from error
|
||||
|
||||
def read(self, job_id: str) -> List[Dict[str, Any]]:
|
||||
path = self._path(job_id)
|
||||
try:
|
||||
with self._lock:
|
||||
if self.root.is_symlink():
|
||||
raise AgentTraceStoreError("agent trace root is invalid")
|
||||
if not path.exists():
|
||||
return []
|
||||
if path.is_symlink() or not path.is_file():
|
||||
raise AgentTraceStoreError("agent trace path is invalid")
|
||||
metadata = path.stat()
|
||||
if metadata.st_size > MAX_TRACE_FILE_BYTES:
|
||||
return [self._read_warning(metadata.st_mtime, 1, "file.too_large")]
|
||||
modified_at = metadata.st_mtime
|
||||
lines = path.read_text(encoding="utf-8").splitlines()
|
||||
except AgentTraceStoreError:
|
||||
raise
|
||||
except (OSError, UnicodeError) as error:
|
||||
raise AgentTraceStoreError("cannot read agent trace records") from error
|
||||
|
||||
records: List[Dict[str, Any]] = []
|
||||
invalid = 0
|
||||
for line in lines[:MAX_TRACE_EVENTS]:
|
||||
try:
|
||||
payload = json.loads(line)
|
||||
records.append(validate_trace_record(payload))
|
||||
except (ValueError, TypeError, AgentTraceStoreError):
|
||||
invalid += 1
|
||||
if len(lines) > MAX_TRACE_EVENTS:
|
||||
invalid += len(lines) - MAX_TRACE_EVENTS
|
||||
if invalid:
|
||||
records.append(self._read_warning(modified_at, invalid, "records.invalid"))
|
||||
return records
|
||||
|
||||
def _path(self, job_id: str) -> Path:
|
||||
if not isinstance(job_id, str) or OPAQUE_ID_RE.fullmatch(job_id) is None:
|
||||
raise AgentTraceStoreError("job id is invalid")
|
||||
return self.root / f"{sha256(job_id.encode('utf-8')).hexdigest()}.jsonl"
|
||||
|
||||
@staticmethod
|
||||
def _read_warning(modified_at: float, count: int, reason: str) -> Dict[str, Any]:
|
||||
return _record(
|
||||
sequence=MAX_TRACE_EVENTS + 1,
|
||||
timestamp=datetime.fromtimestamp(modified_at, tz=timezone.utc).isoformat(),
|
||||
level="warning",
|
||||
code="AGENT_TRACE_RECORD_INVALID",
|
||||
title="本地 Agent trace 记录不完整",
|
||||
message="部分本地 trace 记录无法安全读取，已从任务日志中忽略。",
|
||||
details={
|
||||
"trace_event": reason,
|
||||
"invalid_records": count,
|
||||
},
|
||||
)
|
||||
@@ -0,0 +1,155 @@
|
||||
"""ARR-owned execution of the frozen Opera daily processor."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import subprocess
|
||||
from dataclasses import dataclass
|
||||
from datetime import date
|
||||
from pathlib import Path
|
||||
from typing import Dict, Mapping, Optional
|
||||
|
||||
from arr_ingestion.contracts import IngestionError
|
||||
from arr_ingestion.validation import ProcessorPolicy, strict_json_file
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LocalProcessingOutput:
|
||||
status: str
|
||||
business_date: Optional[date]
|
||||
artifacts: Mapping[str, Path]
|
||||
exit_code: int
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LocalDailyProcessor:
|
||||
"""Run one isolated deterministic process and expose only its artifact paths."""
|
||||
|
||||
policy: ProcessorPolicy
|
||||
timeout_seconds: int = 240
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if (
|
||||
not isinstance(self.timeout_seconds, int)
|
||||
or isinstance(self.timeout_seconds, bool)
|
||||
or not 10 <= self.timeout_seconds <= 900
|
||||
):
|
||||
raise ValueError("local processor timeout is invalid")
|
||||
if not (self.policy.skill_root / "scripts" / "process_daily.py").is_file():
|
||||
raise ValueError("ARR daily processor is unavailable")
|
||||
|
||||
def run(self, source_xml: Path, output_dir: Path) -> LocalProcessingOutput:
|
||||
if not source_xml.is_file():
|
||||
raise IngestionError("SOURCE_NOT_FOUND", "registered source XML is unavailable")
|
||||
output_dir.mkdir(parents=True, exist_ok=False, mode=0o700)
|
||||
result_path = output_dir / "result.json"
|
||||
structured_path = output_dir / "structured-result.json"
|
||||
command = [
|
||||
self.policy.python_binary,
|
||||
str((self.policy.skill_root / "scripts" / "process_daily.py").resolve()),
|
||||
"--xml",
|
||||
str(source_xml.resolve()),
|
||||
"--output-dir",
|
||||
str(output_dir.resolve()),
|
||||
"--result-json",
|
||||
str(result_path.resolve()),
|
||||
"--structured-result-json",
|
||||
str(structured_path.resolve()),
|
||||
]
|
||||
try:
|
||||
completed = subprocess.run(
|
||||
command,
|
||||
stdin=subprocess.DEVNULL,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
check=False,
|
||||
timeout=self.timeout_seconds,
|
||||
)
|
||||
except subprocess.TimeoutExpired:
|
||||
raise IngestionError(
|
||||
"PROCESSOR_TIMEOUT",
|
||||
"deterministic XML processing exceeded its time limit",
|
||||
) from None
|
||||
except OSError:
|
||||
raise IngestionError(
|
||||
"PROCESSOR_UNAVAILABLE",
|
||||
"deterministic XML processing could not start",
|
||||
) from None
|
||||
|
||||
result = strict_json_file(result_path, "processor result")
|
||||
structured = strict_json_file(structured_path, "structured processor result")
|
||||
status = result.get("status")
|
||||
if (
|
||||
status not in {"success", "failed"}
|
||||
or structured.get("status") != status
|
||||
or (completed.returncode == 0) != (status == "success")
|
||||
):
|
||||
raise IngestionError(
|
||||
"PROCESSOR_RESULT_INVALID",
|
||||
"processor exit status and result artifacts do not agree",
|
||||
)
|
||||
|
||||
raw_date = result.get("business_date")
|
||||
business_date: Optional[date]
|
||||
if raw_date is None:
|
||||
business_date = None
|
||||
elif isinstance(raw_date, str):
|
||||
try:
|
||||
business_date = date.fromisoformat(raw_date)
|
||||
except ValueError:
|
||||
raise IngestionError(
|
||||
"PROCESSOR_RESULT_INVALID", "processor business date is invalid"
|
||||
) from None
|
||||
else:
|
||||
raise IngestionError(
|
||||
"PROCESSOR_RESULT_INVALID", "processor business date is invalid"
|
||||
)
|
||||
|
||||
outputs = result.get("outputs")
|
||||
if not isinstance(outputs, dict):
|
||||
raise IngestionError(
|
||||
"PROCESSOR_RESULT_INVALID", "processor output manifest is invalid"
|
||||
)
|
||||
artifacts: Dict[str, Path] = {
|
||||
"result_json": result_path,
|
||||
"structured_result_json": structured_path,
|
||||
}
|
||||
if status == "success":
|
||||
artifacts["daily_report"] = self._output_path(
|
||||
output_dir, outputs.get("daily_report"), ".xlsx"
|
||||
)
|
||||
if outputs.get("exception_report") is not None or business_date is None:
|
||||
raise IngestionError(
|
||||
"PROCESSOR_RESULT_INVALID", "successful processor output is incomplete"
|
||||
)
|
||||
else:
|
||||
artifacts["exception_report"] = self._output_path(
|
||||
output_dir, outputs.get("exception_report"), ".xlsx"
|
||||
)
|
||||
if outputs.get("daily_report") is not None:
|
||||
raise IngestionError(
|
||||
"PROCESSOR_RESULT_INVALID", "failed processor output is inconsistent"
|
||||
)
|
||||
return LocalProcessingOutput(
|
||||
status=str(status),
|
||||
business_date=business_date,
|
||||
artifacts=artifacts,
|
||||
exit_code=completed.returncode,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _output_path(output_dir: Path, value: object, suffix: str) -> Path:
|
||||
if (
|
||||
not isinstance(value, str)
|
||||
or not value
|
||||
or Path(value).name != value
|
||||
or not value.lower().endswith(suffix)
|
||||
):
|
||||
raise IngestionError(
|
||||
"PROCESSOR_RESULT_INVALID", "processor output filename is invalid"
|
||||
)
|
||||
candidate = (output_dir / value).resolve()
|
||||
if candidate.parent != output_dir.resolve() or not candidate.is_file():
|
||||
raise IngestionError(
|
||||
"PROCESSOR_RESULT_INVALID", "processor output artifact is unavailable"
|
||||
)
|
||||
return candidate
|
||||
+204
-10
@@ -3,7 +3,9 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
from typing import Any, Callable, Mapping, Optional
|
||||
import threading
|
||||
import time
|
||||
from typing import Any, Callable, Iterator, Mapping, NoReturn, Optional, Set
|
||||
|
||||
from agent_integration.client import (
|
||||
OpenAgentAPIError,
|
||||
@@ -11,6 +13,13 @@ from agent_integration.client import (
|
||||
OpenAgentTransportError,
|
||||
)
|
||||
from agent_integration.service import AgentResponseError, OpenAgentService
|
||||
from arr_processing.agent_trace import (
|
||||
AgentTraceSink,
|
||||
AgentTraceStoreError,
|
||||
capture_failure_record,
|
||||
project_trace_event,
|
||||
trace_run_id,
|
||||
)
|
||||
from arr_processing.contracts import PROCESSING_REQUEST_VERSION, ProcessingRequest
|
||||
from arr_processing.errors import ProcessingTransportError
|
||||
from arr_processing.runner import RemoteRunSnapshot
|
||||
@@ -24,34 +33,79 @@ class OpenAgentProcessingTransport:
|
||||
service: OpenAgentService,
|
||||
*,
|
||||
message_builder: Optional[Callable[[ProcessingRequest], str]] = None,
|
||||
trace_store: Optional[AgentTraceSink] = None,
|
||||
) -> None:
|
||||
self._service = service
|
||||
self._message_builder = message_builder or (lambda request: request.message())
|
||||
self._trace_store = trace_store
|
||||
self._trace_threads: Set[threading.Thread] = set()
|
||||
self._trace_threads_lock = threading.Lock()
|
||||
self._closed = False
|
||||
|
||||
def submit(self, request: ProcessingRequest, idempotency_key: str) -> RemoteRunSnapshot:
|
||||
conversation_id = self.conversation_id(request.job_id)
|
||||
try:
|
||||
message = self._message_builder(request)
|
||||
metadata = {
|
||||
"contract_version": PROCESSING_REQUEST_VERSION,
|
||||
"processing_kind": "opera_daily",
|
||||
}
|
||||
if self._trace_store is not None:
|
||||
try:
|
||||
return self._submit_traced(
|
||||
request,
|
||||
conversation_id=conversation_id,
|
||||
message=message,
|
||||
idempotency_key=idempotency_key,
|
||||
metadata=metadata,
|
||||
)
|
||||
except (
|
||||
OpenAgentAPIError,
|
||||
OpenAgentTransportError,
|
||||
OpenAgentProtocolError,
|
||||
AgentResponseError,
|
||||
) as error:
|
||||
self._raise_mapped(error, stream_submit=True)
|
||||
response = self._service.send_message(
|
||||
conversation_id,
|
||||
self._message_builder(request),
|
||||
message,
|
||||
message_id=idempotency_key,
|
||||
external_subject_id=conversation_id,
|
||||
metadata={
|
||||
"contract_version": PROCESSING_REQUEST_VERSION,
|
||||
"processing_kind": "opera_daily",
|
||||
},
|
||||
metadata=metadata,
|
||||
)
|
||||
except (OpenAgentAPIError, OpenAgentTransportError, OpenAgentProtocolError, AgentResponseError) as error:
|
||||
except (
|
||||
OpenAgentAPIError,
|
||||
OpenAgentTransportError,
|
||||
OpenAgentProtocolError,
|
||||
AgentResponseError,
|
||||
) as error:
|
||||
self._raise_mapped(error)
|
||||
return self._snapshot(response)
|
||||
|
||||
def close(self, timeout_seconds: float = 2.0) -> None:
|
||||
"""Best-effort drain before the shared HTTP client is closed at shutdown."""
|
||||
|
||||
if timeout_seconds < 0:
|
||||
raise ValueError("trace shutdown timeout must not be negative")
|
||||
with self._trace_threads_lock:
|
||||
self._closed = True
|
||||
threads = list(self._trace_threads)
|
||||
deadline = time.monotonic() + timeout_seconds
|
||||
for thread in threads:
|
||||
thread.join(max(0.0, deadline - time.monotonic()))
|
||||
|
||||
def get(self, request: ProcessingRequest, remote_run_id: str) -> RemoteRunSnapshot:
|
||||
try:
|
||||
response = self._service.get_run(
|
||||
self.conversation_id(request.job_id),
|
||||
remote_run_id,
|
||||
)
|
||||
except (OpenAgentAPIError, OpenAgentTransportError, OpenAgentProtocolError, AgentResponseError) as error:
|
||||
except (
|
||||
OpenAgentAPIError,
|
||||
OpenAgentTransportError,
|
||||
OpenAgentProtocolError,
|
||||
AgentResponseError,
|
||||
) as error:
|
||||
self._raise_mapped(error)
|
||||
return self._snapshot(response, expected_run_id=remote_run_id)
|
||||
|
||||
@@ -61,7 +115,12 @@ class OpenAgentProcessingTransport:
|
||||
self.conversation_id(request.job_id),
|
||||
remote_run_id,
|
||||
)
|
||||
except (OpenAgentAPIError, OpenAgentTransportError, OpenAgentProtocolError, AgentResponseError) as error:
|
||||
except (
|
||||
OpenAgentAPIError,
|
||||
OpenAgentTransportError,
|
||||
OpenAgentProtocolError,
|
||||
AgentResponseError,
|
||||
) as error:
|
||||
self._raise_mapped(error)
|
||||
return self._snapshot(response, expected_run_id=remote_run_id)
|
||||
|
||||
@@ -70,6 +129,122 @@ class OpenAgentProcessingTransport:
|
||||
digest = hashlib.sha256(job_id.encode("utf-8")).hexdigest()
|
||||
return "arrproc_" + digest[:48]
|
||||
|
||||
def _submit_traced(
|
||||
self,
|
||||
request: ProcessingRequest,
|
||||
*,
|
||||
conversation_id: str,
|
||||
message: str,
|
||||
idempotency_key: str,
|
||||
metadata: Mapping[str, Any],
|
||||
) -> RemoteRunSnapshot:
|
||||
events = iter(
|
||||
self._service.stream_message(
|
||||
conversation_id,
|
||||
message,
|
||||
message_id=idempotency_key,
|
||||
external_subject_id=conversation_id,
|
||||
metadata=dict(metadata),
|
||||
include_trace=True,
|
||||
)
|
||||
)
|
||||
sequence = 0
|
||||
for event in events:
|
||||
sequence += 1
|
||||
self._capture_trace(request, event, sequence)
|
||||
remote_run_id = trace_run_id(event)
|
||||
if remote_run_id is None:
|
||||
continue
|
||||
self._start_trace_drain(request, events, sequence)
|
||||
return RemoteRunSnapshot(remote_run_id, "running")
|
||||
raise AgentResponseError("trace stream ended before run.started")
|
||||
|
||||
def _start_trace_drain(
|
||||
self,
|
||||
request: ProcessingRequest,
|
||||
events: Iterator[Any],
|
||||
sequence: int,
|
||||
) -> None:
|
||||
thread = threading.Thread(
|
||||
target=self._drain_trace,
|
||||
args=(request, events, sequence),
|
||||
name=f"arr-agent-trace-{request.attempt_no}",
|
||||
daemon=True,
|
||||
)
|
||||
with self._trace_threads_lock:
|
||||
if self._closed:
|
||||
try:
|
||||
close = getattr(events, "close", None)
|
||||
if callable(close):
|
||||
close()
|
||||
finally:
|
||||
self._append_trace(
|
||||
request.job_id,
|
||||
capture_failure_record(
|
||||
attempt_no=request.attempt_no,
|
||||
sequence=sequence + 1,
|
||||
error=RuntimeError("transport closed"),
|
||||
),
|
||||
)
|
||||
return
|
||||
self._trace_threads.add(thread)
|
||||
try:
|
||||
thread.start()
|
||||
except RuntimeError as error:
|
||||
with self._trace_threads_lock:
|
||||
self._trace_threads.discard(thread)
|
||||
close = getattr(events, "close", None)
|
||||
if callable(close):
|
||||
close()
|
||||
self._append_trace(
|
||||
request.job_id,
|
||||
capture_failure_record(
|
||||
attempt_no=request.attempt_no,
|
||||
sequence=sequence + 1,
|
||||
error=error,
|
||||
),
|
||||
)
|
||||
|
||||
def _drain_trace(
|
||||
self,
|
||||
request: ProcessingRequest,
|
||||
events: Iterator[Any],
|
||||
sequence: int,
|
||||
) -> None:
|
||||
try:
|
||||
for event in events:
|
||||
sequence += 1
|
||||
self._capture_trace(request, event, sequence)
|
||||
except Exception as error: # best-effort diagnostics must not affect the run
|
||||
self._append_trace(
|
||||
request.job_id,
|
||||
capture_failure_record(
|
||||
attempt_no=request.attempt_no,
|
||||
sequence=sequence + 1,
|
||||
error=error,
|
||||
),
|
||||
)
|
||||
finally:
|
||||
with self._trace_threads_lock:
|
||||
self._trace_threads.discard(threading.current_thread())
|
||||
|
||||
def _capture_trace(self, request: ProcessingRequest, event: Any, sequence: int) -> None:
|
||||
projected = project_trace_event(
|
||||
event,
|
||||
attempt_no=request.attempt_no,
|
||||
sequence=sequence,
|
||||
)
|
||||
if projected is not None:
|
||||
self._append_trace(request.job_id, projected)
|
||||
|
||||
def _append_trace(self, job_id: str, record: Mapping[str, Any]) -> None:
|
||||
if self._trace_store is None:
|
||||
return
|
||||
try:
|
||||
self._trace_store.append(job_id, record)
|
||||
except (AgentTraceStoreError, OSError, ValueError, TypeError):
|
||||
pass
|
||||
|
||||
@staticmethod
|
||||
def _snapshot(
|
||||
response: Mapping[str, Any],
|
||||
@@ -92,7 +267,11 @@ class OpenAgentProcessingTransport:
|
||||
return RemoteRunSnapshot(run_id, status.lower(), final_content)
|
||||
|
||||
@staticmethod
|
||||
def _raise_mapped(error: BaseException) -> None:
|
||||
def _raise_mapped(
|
||||
error: BaseException,
|
||||
*,
|
||||
stream_submit: bool = False,
|
||||
) -> NoReturn:
|
||||
if isinstance(error, OpenAgentAPIError):
|
||||
if error.active_run_conflict:
|
||||
raise ProcessingTransportError(
|
||||
@@ -100,6 +279,11 @@ class OpenAgentProcessingTransport:
|
||||
retryable=True,
|
||||
active_run_conflict=True,
|
||||
) from None
|
||||
if stream_submit and (error.status_code == 408 or error.status_code >= 500):
|
||||
raise ProcessingTransportError(
|
||||
"PROCESSING_REMOTE_SUBMISSION_AMBIGUOUS",
|
||||
retryable=False,
|
||||
) from None
|
||||
if error.retryable:
|
||||
code = (
|
||||
"PROCESSING_REMOTE_RATE_LIMITED"
|
||||
@@ -109,9 +293,19 @@ class OpenAgentProcessingTransport:
|
||||
raise ProcessingTransportError(code, retryable=True) from None
|
||||
raise ProcessingTransportError("PROCESSING_REMOTE_REJECTED", retryable=False) from None
|
||||
if isinstance(error, OpenAgentTransportError):
|
||||
if stream_submit:
|
||||
raise ProcessingTransportError(
|
||||
"PROCESSING_REMOTE_SUBMISSION_AMBIGUOUS",
|
||||
retryable=False,
|
||||
) from None
|
||||
raise ProcessingTransportError(
|
||||
"PROCESSING_REMOTE_UNAVAILABLE", retryable=True
|
||||
) from None
|
||||
if stream_submit:
|
||||
raise ProcessingTransportError(
|
||||
"PROCESSING_REMOTE_SUBMISSION_AMBIGUOUS",
|
||||
retryable=False,
|
||||
) from None
|
||||
raise ProcessingTransportError(
|
||||
"PROCESSING_REMOTE_PROTOCOL_INVALID", retryable=False
|
||||
) from None
|
||||
@@ -0,0 +1,35 @@
|
||||
"""Neutral discovery of the frozen ARR deterministic processor identity."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from arr_ingestion.validation import ProcessorPolicy
|
||||
|
||||
|
||||
def load_processor_policy(project_root: Path) -> ProcessorPolicy:
|
||||
skill_root = (project_root / "arr-opera-daily-ingest").resolve()
|
||||
script = skill_root / "scripts" / "process_daily.py"
|
||||
module_name = "_arr_runtime_process_daily"
|
||||
specification = importlib.util.spec_from_file_location(module_name, script)
|
||||
if specification is None or specification.loader is None:
|
||||
raise ValueError("ARR daily processor identity is unavailable")
|
||||
module = importlib.util.module_from_spec(specification)
|
||||
prior = sys.modules.get(module_name)
|
||||
sys.modules[module_name] = module
|
||||
try:
|
||||
specification.loader.exec_module(module)
|
||||
processor_version = str(module.PROCESSOR_VERSION)
|
||||
rule_set_sha256 = str(module.rule_set_sha256())
|
||||
finally:
|
||||
if prior is None:
|
||||
sys.modules.pop(module_name, None)
|
||||
else:
|
||||
sys.modules[module_name] = prior
|
||||
return ProcessorPolicy(
|
||||
processor_version=processor_version,
|
||||
rule_set_sha256=rule_set_sha256,
|
||||
skill_root=skill_root,
|
||||
)
|
||||
@@ -23,6 +23,7 @@ _SAFE_START_FAILURE_CODES = frozenset(
|
||||
"PROCESSING_REMOTE_PROTOCOL_INVALID",
|
||||
"PROCESSING_REMOTE_RATE_LIMITED",
|
||||
"PROCESSING_REMOTE_REJECTED",
|
||||
"PROCESSING_REMOTE_SUBMISSION_AMBIGUOUS",
|
||||
"PROCESSING_REMOTE_UNAVAILABLE",
|
||||
}
|
||||
)
|
||||
|
||||
@@ -4,7 +4,7 @@ from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Dict, Protocol
|
||||
from urllib.parse import urlparse
|
||||
from urllib.parse import quote, urlparse
|
||||
|
||||
from arr_ingestion.contracts import ArtifactRef, ROLE_CONTRACTS
|
||||
from arr_processing.contracts import ProcessingRequest, canonical_json_bytes
|
||||
@@ -13,7 +13,7 @@ from arr_storage.aliyun_oss_v2 import AliyunOssConfig
|
||||
from arr_storage.contracts import valid_object_key
|
||||
|
||||
|
||||
PROGRAM_INPUT_VERSION = "arr-opera-daily-program-input-2"
|
||||
PROGRAM_INPUT_VERSION = "arr-opera-daily-program-input-3"
|
||||
|
||||
|
||||
class ProcessingSourceResolver(Protocol):
|
||||
@@ -23,12 +23,12 @@ class ProcessingSourceResolver(Protocol):
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class OssProcessingMessageBuilder:
|
||||
"""Resolve the ARR-owned source object and expose only non-secret OSS routing.
|
||||
"""Resolve one ARR-owned source to its public-read HTTPS object URL.
|
||||
|
||||
``fetch_oss_file`` owns the download and its credential provider. The Agent
|
||||
receives the same attachment-shaped descriptor used by the existing hotel
|
||||
runtime: bucket, endpoint host, exact object key, hash and size. AccessKey,
|
||||
Secret, local paths and signed URLs never enter the message.
|
||||
The Agent receives an attachment-shaped descriptor containing the exact URL
|
||||
that ``fetch_oss_file`` must use, plus non-secret provenance and integrity
|
||||
fields. AccessKey, Secret, local paths and signed URLs never enter the
|
||||
message.
|
||||
"""
|
||||
|
||||
config: AliyunOssConfig
|
||||
@@ -37,6 +37,7 @@ class OssProcessingMessageBuilder:
|
||||
def message(self, request: ProcessingRequest) -> str:
|
||||
source = self.sources.source_for_attempt(request.job_id, request.attempt_no)
|
||||
self._validate_source(request, source)
|
||||
endpoint_host = self._endpoint_host()
|
||||
if request.submission_grant is None:
|
||||
raise ProcessingError(
|
||||
"PROCESSING_SUBMISSION_GRANT_MISSING",
|
||||
@@ -60,8 +61,12 @@ class OssProcessingMessageBuilder:
|
||||
"source_ref": {"source": "oss_attachments", "index": 0},
|
||||
"oss": {
|
||||
"bucket": self.config.bucket,
|
||||
"endpoint": self._endpoint_host(),
|
||||
"endpoint": endpoint_host,
|
||||
"object_key": source.object_key,
|
||||
"url": self._public_object_url(
|
||||
endpoint_host,
|
||||
source.object_key,
|
||||
),
|
||||
},
|
||||
}
|
||||
],
|
||||
@@ -101,12 +106,22 @@ class OssProcessingMessageBuilder:
|
||||
)
|
||||
|
||||
def _endpoint_host(self) -> str:
|
||||
expected_host = f"oss-{self.config.region}.aliyuncs.com"
|
||||
if self.config.endpoint:
|
||||
parsed = urlparse(self.config.endpoint)
|
||||
if parsed.scheme != "https" or not parsed.hostname:
|
||||
if (
|
||||
parsed.scheme != "https"
|
||||
or parsed.hostname != expected_host
|
||||
or parsed.username is not None
|
||||
or parsed.password is not None
|
||||
or parsed.port is not None
|
||||
):
|
||||
raise ProcessingError(
|
||||
"PROCESSING_SOURCE_INVALID",
|
||||
"OSS endpoint is unavailable",
|
||||
"public OSS endpoint is unavailable",
|
||||
)
|
||||
return parsed.netloc
|
||||
return f"oss-{self.config.region}.aliyuncs.com"
|
||||
return expected_host
|
||||
|
||||
def _public_object_url(self, endpoint_host: str, object_key: str) -> str:
|
||||
encoded_key = quote(object_key, safe="/")
|
||||
return f"https://{self.config.bucket}.{endpoint_host}/{encoded_key}"
|
||||
Reference in new issue
Block a user