test: integrate reliable event recovery and receiving evidence

This commit is contained in:
2026-10-02 20:14:17 +08:00
parent d0df738bba
commit 4b531c7e22
17 changed files with 1527 additions and 37 deletions
+7 -1
View File
@@ -1,6 +1,6 @@
# 后端脚本管理与全量 HTTP 验收索引
`event-delivery-fault-proxy.py` 是 TEST 环境定向事件 ACK 丢失工具,不创建或重放事件。运行者显式传入 `--upstream` 完整消费者地址、`--event-type`,并只选 `--event-id` 或 `--loading-plan-id`;`EVENT_FAULT_CONTROL_TOKEN` 从外部环境注入,保护固定本机监听的控制端口。默认 PASS;经鉴权切为 FAULT 后,仅在选定事件已真实转发且收到匹配的 APPLIED/DUPLICATE ACK 时向发送方返回 503,其他事件直通。日志只保留事件标识、摘要和结果,不保留正文或凭据。它会影响实际投递,仅限明确授权的测试环境;生产者配置、重试、DEAD 原事件人工恢复及最终恢复直连由运行者负责,工具退出不自动恢复服务配置。`--log` 可显式指定脱敏证据文件,失败保留原事件与日志,禁止改摘要或借其他事件重跑。`event-delivery-fault-proxy.test.mjs` 使用本机替身服务验证行为,不访问共享环境;工具离线通过不能替代真实重试验收。
`event-delivery-fault-proxy.py` 是 TEST 环境定向事件 ACK 丢失工具,不创建或重放事件。运行者显式传入 `--upstream` 完整消费者地址、`--event-type`,并只选 `--event-id`、`--loading-plan-id` 或 `--container-booking-id`;柜次选择精确匹配 payload.sourceContainerBookingId,用于海外卸柜回写,不接受混合选择条件。`EVENT_FAULT_CONTROL_TOKEN` 从外部环境注入,保护固定本机监听的控制端口。默认 PASS;经鉴权切为 FAULT 后,仅在选定事件已真实转发且收到匹配的 APPLIED/DUPLICATE ACK 时向发送方返回 503,其他事件直通。同一选择范围内锁定首个事件后不改选其他事件 ID,原 ID 改摘要或载荷直接拒绝。日志只保留事件标识、摘要和结果,不保留正文或凭据。它会影响实际投递,仅限明确授权的测试环境;生产者配置、重试、DEAD 原事件人工恢复及最终恢复直连由运行者负责,工具退出不自动恢复服务配置。`--log` 可显式指定脱敏证据文件,失败保留原事件与日志,禁止改摘要或借其他事件重跑。`event-delivery-fault-proxy.test.mjs` 使用本机替身服务验证行为,不访问共享环境;工具离线通过不能替代真实重试验收。
`admin-test-env-acceptance.mjs` 与 `client-test-env-acceptance.mjs` 仅接受统一 TEST Gateway 和 Auth 会话;移除环境文件中的 `ADMIN_OVERSEAS_GATEWAY`,不再传 `--overseas-gateway` 或非 test 环境。平台、客户、PDA 身份仍分别登录并验证权限及客户隔离。海外资源接口使用 `/api/platform/v1/overseas-warehouse-resources/**`,仓库主档仍归 Warehouse。跨服务模块的重复 GET 不证明事件重放;真实重试/幂等证据缺失会报告部署阻断,详见模块 README 和下方事件人工重试入口。离线合同不会执行真实 HTTP;环境执行仍需本轮写入授权。
@@ -351,5 +351,11 @@ Dry-run 只校验两个统一 SQL 的入口标记,打印当前 SHA-256、行
部署离线合同:`overseas-deployment-contract.test.mjs` 现验证统一TEST拓扑,保留历史文件名便于调用;`deploy-built-service-rollback.test.mjs`以mock Docker验证失败与同契约回滚,无真实环境操作。唯一执行清单见 `../deploy/TEST单机部署执行清单.md`。
# 事件人工重试运维入口
`fba-core-regression-evidence.mjs` 是 Admin/F40 共用的内部只读取证器,不是独立 CLI。只在明确的 TEST Jenkins 节点读取 `firstleg-test-order`、`firstleg-test-lastleg` 对应数据库的一致性只读快照,原始库存、位置、流水及事件身份严格按本轮客户/仓库/柜次过滤。连接凭据只在执行节点内存中处理,不执行 SQL 写入;调用者保存独立报告,失败不能用空结果代替。直接合同为 `fba-core-regression-evidence.test.mjs`,覆盖只读 SQL 边界、完整作用域、长 ID、零事件保留与敏感错误脱敏;离线合同不代替真实数据库取证。
海外卸柜反向故障验收从 Admin 总入口显式传 `--modules phase4-overseas-pda --verify-unload-event-recovery true`,按正式依赖准备新客户、新柜及实收,首个本轮柜完成前设置独立故障 hook;普通收货默认不启用。目标事件固定为 `OverseasContainerUnloaded`,生产者为 Lastleg,消费者为 Firstleg,事件平台作用域固定 0、库存仍按本轮实际客户隔离。按原事件验证 ACK 丢失、DEAD、人工 retry 与重复确认;在恢复服务配置后,用相同只读库存取证器采集 `AFTER_EVENT_RECOVERY`,核对原始库存、位置、托盘组成、收货完整键及流水零漂移。任一方向恢复失败保留任务恢复记录并判失败;不得与独立部署或共享环境造数并行,不修改业务 SQL。报告沿用显式新 `--output`,旧失败报告和业务样本保留,不自动清理。此项不覆盖签收完成回写、设备和车辆新版链路。
Admin 全量中的 `phase4-cross-region-events` 使用内部 `test-event-fault-executor.mjs` 和 `test-event-fault-runtime.py`,不是额外 CLI 入口。它们仅在显式 `CHONGCHONG_ENV=test` 和已授权环境中运行:在首个本轮计划完成前启动已有 ACK 丢失代理,短暂重建同镜像 Firstleg 容器,保留数据卷、端口和网络身份,只变更事件目标与本次故障退避参数。仅选定的装柜事件被抑制 ACK;其他事件直通。核对 12 次失败、DEAD、受保护原事件 retry、DUPLICATE 及业务订单零新增,结束或失败都恢复原镜像配置并检查健康,随后停止本任务代理。不修改业务 SQL、载荷、事件 ID 或 hash,不重放已发布事件。恢复失败必须保留阻断和任务元数据;独立后端发布或其他环境造数不得与此窗口并行。凭据仅在执行节点内存与权限600的临时恢复文件中使用,恢复完成后移除临时文件中的凭据。离线合同为 `test-event-fault-executor.test.mjs`。
`event-outbox-operations.mjs`:单测试环境的内部运维脚本。只连接显式指定的生产服务地址,不通过公网 Gateway,不自动重试或跟随重定向。凭据只从进程环境 `EVENT_OPERATIONS_TOKEN` 读取,另须 `CHONGCHONG_ENV=test`;不要把真实凭据放入参数或仓库。`--service firstleg|lastleg|finance --base <服务源地址> --entry <Outbox主键>` 只读查询状态和摘要,不返回业务负载。追加 `--retry --request-id <稳定UUID> --operator <操作人> --reason <原因> --hash <sha256摘要>` 会写入目标测试服务的 Outbox 与审计表,执行前须已有该环境操作授权。仅 DEAD 原事件可重试,事件身份、负载及摘要保持不变;重复提交使用同一个请求编号,参数变化会被拒绝。三个服务须先部署 `event_delivery_retry_audit` 表,未设置至少32字节的运维凭据时入口关闭。脚本将脱敏结果输出到标准输出,不持久化凭据;离线验证为 `node --test scripts/event-outbox-operations.test.mjs`。
+80 -2
View File
@@ -383,6 +383,7 @@ export function parseArguments(argv) {
routeMode: values.get('route-mode') || 'fixed',
quiet: values.get('quiet') === 'true',
returnMainChainOnly: values.get('return-main-chain-only') === 'true',
verifyUnloadEventRecovery: values.get('verify-unload-event-recovery') === 'true',
manualSplitFixtureReport: values.get('manual-split-fixture-report') || '',
manualSplitPlanId: values.get('manual-split-plan-id') || '',
manualPreloadSourceReport: values.get('manual-preload-source-report') || '',
@@ -394,6 +395,9 @@ export function parseArguments(argv) {
}
export function validateExecutionOptions(options) {
if (options.verifyUnloadEventRecovery && !options.modules.includes('phase4-overseas-pda')) {
throw new Error('卸柜事件恢复验收必须包含正式 phase4-overseas-pda 收货链')
}
const unavailable = options.modules.filter((module) => !AVAILABLE_MODULES.has(module))
if (unavailable.length > 0) throw new Error(`未注册的 HTTP 验收模块: ${unavailable.join(',')}`)
for (const [name, value] of [['gateway', options.gateway], ['username', options.username], ['password', options.password]]) {
@@ -1424,7 +1428,7 @@ export class AcceptanceRun {
const attempt = (this.phase4ProducerAttempts.get(label) || 0) + 1
this.phase4ProducerAttempts.set(label, attempt)
const variantMarker = `${matrixMarker}-P4-${label}${attempt === 1 ? '' : `-R${attempt}`}`
const contextKeys = ['orderDestinationType', 'orderFulfillmentType', 'phase4ProducerMatrixMarker']
const contextKeys = ['orderDestinationType', 'orderFulfillmentType', 'phase4ProducerMatrixMarker', 'waitSplitRoute']
const createdKeys = [
'phase3Loading', 'phase3LoadingCompletedProducer', 'phase3ContainerDocuments', 'orchestrationFixtures', 'oceanBooking',
'phase3CustomsSupplier', 'phase3Customs', 'serviceCenterFlow', 'portSchedule',
@@ -1557,6 +1561,11 @@ export class AcceptanceRun {
if (!hasOperationsFbaProducer()) {
await this.runPhase4ProducerVariant('FBA-OPERATIONS', 'FBA', 'STANDARD', true)
}
const hasRouteFbaProducer = () => (this.createdResources.phase4InboundProducerFixtures?.containers || [])
.some((item) => /-P4-FBA-ROUTE(?:-R\d+)?:/.test(String(item?.resourceKey || '')))
if (!hasRouteFbaProducer()) {
await this.runPhase4ProducerVariant('FBA-ROUTE', 'FBA', 'STANDARD', true)
}
if (Number(readiness().overseasContainerCount || 0) < 1) {
await this.runPhase4ProducerVariant('OVERSEAS-PRIMARY', 'OVERSEAS', 'ONE_CLICK_DROPSHIP', true)
}
@@ -1576,6 +1585,8 @@ export class AcceptanceRun {
/-P4-FBA-SECOND(?:-R\d+)?:/.test(String(item?.resourceKey || '')))
const operationsFbaReady = containers.find((item) =>
/-P4-FBA-OPERATIONS(?:-R\d+)?:/.test(String(item?.resourceKey || '')))
const routeFbaReady = containers.find((item) =>
/-P4-FBA-ROUTE(?:-R\d+)?:/.test(String(item?.resourceKey || '')))
const awaitingVariants = containers.filter((item) => ['SECONDARY']
.some((label) => new RegExp(`-P4-${label}(?:-R\\d+)?:`).test(String(item?.resourceKey || ''))))
const shippingMatrixReady = dropshipReady?.shippingReady === true
@@ -1583,6 +1594,7 @@ export class AcceptanceRun {
&& standardFbaReady?.shippingReady === true && Boolean(standardFbaReady?.carrierScheduleQueryId)
&& secondStandardFbaReady?.shippingReady === true && Boolean(secondStandardFbaReady?.carrierScheduleQueryId)
&& operationsFbaReady?.shippingReady === true && Boolean(operationsFbaReady?.carrierScheduleQueryId)
&& routeFbaReady?.shippingReady === true && Boolean(routeFbaReady?.carrierScheduleQueryId)
&& overseasReady?.shippingReady === true && Boolean(overseasReady?.carrierScheduleQueryId)
&& awaitingVariants.length >= 1
&& awaitingVariants.every((item) => item.shippingReady === false
@@ -1830,6 +1842,11 @@ export class AcceptanceRun {
}
}
}
if (this.options.verifyUnloadEventRecovery) {
this.addStep('phase4-overseas-pda', '显式卸柜故障场景必须实际完成且只验证本轮一个柜', null,
unloadRecoveryWasVerified(this), '不以未触发故障或普通收货成功代替事件恢复证据',
{ scenario: 'reverse-unload-event-recovery', deploymentBlocker: true })
}
const reportSurface = JSON.stringify({
steps: this.steps, modules: this.modules, scenarios: this.scenarios,
deploymentBlockers: this.deploymentBlockers, coverageGaps: this.coverageGaps,
@@ -1870,11 +1887,72 @@ export class AcceptanceRun {
}
}
export function unloadRecoveryWasVerified(run) {
const records = Object.entries(run.createdResources?.phase4OverseasPda?.receivingCompletionEvidence || {})
.filter(([, row]) => row.eventRecovery)
if (records.length !== 1) return false
const [bookingId, row] = records[0]
return row.status === 'VERIFIED' && row.eventRecoveryInventoryUnchanged === true
&& row.eventRecovery.restored === true && row.eventRecovery.marker === run.marker
&& row.eventRecovery.containerBookingId === bookingId
}
export async function configureReadOnlyEvidence(run, environment, createReaders, faultFactories) {
if (!run.options.modules.some(name => ['phase4-overseas-pda', 'phase5-root-fulfillment', 'manual-state-matrix'].includes(name))) return
const factory = createReaders || (await import('./fba-core-regression-evidence.mjs')).createTestEvidenceReaders
const readers = factory(environment)
await readers.preflight()
run.context.phase4ReceivingEvidenceReader = readers.readReceiving
run.context.f40ReadOnlyOutboxEvidence = readers.readCompletion
const faults = []
if (run.options.modules.includes('phase4-cross-region-events')) {
const { createTestFaultExecutor } = faultFactories || await import('./test-event-fault-executor.mjs')
const fault = createTestFaultExecutor(environment)
run.context.beforeLoadingComplete = fault.prepare
run.context.directDeliveryReplayEvidenceReader = fault.readProof
faults.push(fault)
}
if (run.options.verifyUnloadEventRecovery) {
const { createTestUnloadFaultExecutor } = faultFactories || await import('./test-event-fault-executor.mjs')
const fault = createTestUnloadFaultExecutor(environment)
faults.push(fault)
let selected = null
run.context.beforeReceivingComplete = async scope => {
if (selected) return
selected = { marker: run.marker, customerId: scope.customerId, containerBookingId: scope.bookingId }
await fault.prepare(selected)
}
run.context.afterReceivingComplete = async scope => {
if (!selected || scope.bookingId !== selected.containerBookingId || scope.customerId !== selected.customerId) return null
return fault.readProof(selected)
}
}
if (faults.length) return {
cleanup: async () => {
const errors = []
for (const fault of [...faults].reverse()) {
try { await fault.cleanup() } catch { errors.push('TEST事件故障验收恢复失败,保留原任务恢复记录') }
}
if (errors.length) throw new Error(errors.join('; '))
},
}
}
async function main() {
const options = validateExecutionOptions(parseArguments(process.argv.slice(2)))
const timestamp = new Date().toISOString().replace(/\D/g, '').slice(0, 14)
const outputPath = resolve(options.output || `reports/admin-test-env-acceptance-${timestamp}.json`)
const report = await new AcceptanceRun(options).run()
const run = new AcceptanceRun(options)
const fault = await configureReadOnlyEvidence(run, process.env)
let report
try { report = await run.run() }
finally {
try { await fault?.cleanup() }
catch (error) { run.addStep('phase4-cross-region-events', '故障验收后必须恢复原服务配置与健康状态', null,
false, String(error.message), { scenario: 'direct-delivery-replay', deploymentBlocker: true }) }
}
report.summary = buildAcceptanceSummary(run.steps, run.modules, run.coverageGaps)
report.success = report.fatal === null && report.summary.failed === 0
await mkdir(dirname(outputPath), { recursive: true })
await writeFile(outputPath, `${JSON.stringify(report, null, 2)}\n`, 'utf8')
console.log(`REPORT=${outputPath}`)
+2
View File
@@ -356,3 +356,5 @@ node --test scripts/admin-test-env-modules/merchant-qualification-fixture-contra
国内拖车自动完成与应付回归由 `phase3-outbound-dispatch.mjs` 在真实离仓后只读核验:订舱全部柜数、逐柜离仓、唯一 `AUTO_COMPLETE`、按拖车编号追溯柜次应付、金额守恒及 Finance 来源入池(有界等待 Outbox)。该模块的原有确认/撤销边界操作仍会写目标环境,执行须获得对应环境授权。`manual-state-ocean-trucking.mjs` 只构造可人工操作的状态,不伪造已完成拖车;自动终态由上述离仓链路覆盖。
`event-bridge-pending-recovery.mjs` 为纯离线证据校验器,文件名保留为索引入口;无事件桥调用或历史事件常量。合同覆盖 Compose DNS、三服务内部凭据版本一致、有限重试/DEAD、原事件摘要、ACK 丢失后 DUPLICATE 与单次业务效果。合成合同通过不代表环境故障演练通过;`phase4-cross-region-events` 保留真实重放证据阻断项。环境验收需本轮 marker 事件逐次投递记录、Inbox 与业务计数;仅使用受保护 `event-outbox-operations.mjs` 对 DEAD 原事件执行另行授权重试,不重放 SENT,不改表或伪造新事件。
`direct-delivery-replay-proof.mjs` 是事件模块的内部证据校验器,不是独立执行入口。要求本轮原装柜事件的 ID/hash、12 次失败至 DEAD、人工恢复审计、持久 Inbox 及 DUPLICATE、恢复前后业务订单身份集合零新增和环境恢复同时成立;单纯重复 GET 或历史故障报告不能满足条件。离线合同为 `direct-delivery-replay-proof-contract.test.mjs`。
@@ -0,0 +1,36 @@
import assert from 'node:assert/strict'
import test from 'node:test'
import { validDirectDeliveryReplayProof } from './direct-delivery-replay-proof.mjs'
const eventId = 'd54a246f-c71c-462f-ab5b-a44f0e23d29a'
const payloadHash = `sha256:${'a'.repeat(64)}`
const scope = { marker: 'current-run', customerId: '7', loadingPlanId: '99', containerBookingId: '66' }
function proof() {
return { ...scope, readOnly: true, restored: true,
producer: { eventId, payloadHash, entryId: '88', publishStatus: 'PUBLISHED', eventType: 'FirstLegLoadingPlanCompleted' },
consumer: { eventId, payloadHash, status: 'PROCESSED', count: 1 },
attempts: Array.from({ length: 13 }, (_, i) => ({ attemptId: `attempt-${i}`, eventId, entryId: '88',
targetName: 'lastleg-service/internal/lastleg/events', outcome: i === 12 ? 'DUPLICATE' : i === 11 ? 'DEAD' : 'RETRY' })),
proxyRecords: Array.from({ length: 13 }, (_, i) => ({ eventId, payloadHash, matched: true,
outcome: i === 0 ? 'APPLIED' : 'DUPLICATE', errorCode: i === 12 ? null : 'ACK_LOSS_INJECTED' })),
retryAudit: { eventId, payloadHash, entryId: '88', previousFailures: 12, requestId: 'retry-1' },
beforeRetry: { eventId, payloadHash, publishStatus: 'DEAD', failedAttempts: 12, businessOrderIds: ['11', '12'] },
afterRetry: { eventId, payloadHash, publishStatus: 'PUBLISHED', businessOrderIds: ['12', '11'] } }
}
test('only current original event finite failure, authenticated recovery and persistent duplicate consumption together pass', () => {
assert.equal(validDirectDeliveryReplayProof(proof(), scope), true)
})
test('old evidence, GET stability, incomplete attempts, hash drift and business duplication fail closed', () => {
const changes = [
p => { p.marker = 'old' }, p => { p.attempts = [] }, p => { p.proxyRecords = [] },
p => { p.consumer.count = 2 }, p => { p.attempts[12].outcome = 'APPLIED' },
p => { p.retryAudit.payloadHash = `sha256:${'b'.repeat(64)}` },
p => { p.afterRetry.businessOrderIds.push('13') }, p => { p.restored = false },
p => { p.attempts[11].outcome = 'RETRY' }, p => { p.beforeRetry.failedAttempts = 11 },
p => { p.attempts[12].targetName = 'lastleg-service/internal/finance/events' },
]
for (const change of changes) {
const p = proof(); change(p)
assert.equal(validDirectDeliveryReplayProof(p, scope), false)
}
})
@@ -0,0 +1,44 @@
// This checks live, scoped evidence supplied by the TEST fault executor.
// It never accepts historical reports or initiates event replay itself.
const identifier = value => typeof value === 'string' && /^[1-9][0-9]{0,19}$/.test(value)
const eventIdentifier = value => typeof value === 'string'
&& /^(?:[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}|[0-9A-HJKMNP-TV-Z]{26})$/i.test(value)
const hash = value => typeof value === 'string' && /^sha256:[0-9a-f]{64}$/i.test(value)
const sortedIds = rows => [...rows].sort()
export function validDirectDeliveryReplayProof(proof, scope) {
if (!proof || proof.readOnly !== true || proof.marker !== scope.marker
|| !identifier(scope.loadingPlanId) || !identifier(scope.customerId)
|| proof.loadingPlanId !== scope.loadingPlanId || proof.customerId !== scope.customerId
|| proof.containerBookingId !== scope.containerBookingId) return false
const { producer, consumer, attempts, proxyRecords, beforeRetry, afterRetry, retryAudit } = proof
if (!eventIdentifier(producer?.eventId) || !hash(producer?.payloadHash)
|| !identifier(producer?.entryId) || producer.publishStatus !== 'PUBLISHED'
|| producer.eventType !== 'FirstLegLoadingPlanCompleted'
|| consumer?.eventId !== producer.eventId || consumer?.payloadHash !== producer.payloadHash
|| consumer?.status !== 'PROCESSED' || consumer?.count !== 1) return false
if (!Array.isArray(attempts) || attempts.length !== 13
|| new Set(attempts.map(row => row.attemptId)).size !== attempts.length
|| attempts.some(row => !row.attemptId || row.eventId !== producer.eventId
|| row.entryId !== producer.entryId || row.targetName !== 'lastleg-service/internal/lastleg/events')
|| attempts.at(-1)?.outcome !== 'DUPLICATE'
|| attempts[11]?.outcome !== 'DEAD'
|| attempts.slice(0, 11).some(row => row.outcome !== 'RETRY')) return false
if (!Array.isArray(proxyRecords) || proxyRecords.length !== 13
|| proxyRecords.some(row => row.matched !== true || row.eventId !== producer.eventId
|| row.payloadHash !== producer.payloadHash)
|| !proxyRecords.some(row => row.outcome === 'APPLIED' && row.errorCode === 'ACK_LOSS_INJECTED')
|| !proxyRecords.some(row => row.outcome === 'DUPLICATE' && row.errorCode === 'ACK_LOSS_INJECTED')) return false
if (retryAudit?.eventId !== producer.eventId || retryAudit?.payloadHash !== producer.payloadHash
|| retryAudit?.entryId !== producer.entryId || !retryAudit.requestId
|| retryAudit.previousFailures !== 12 || beforeRetry?.publishStatus !== 'DEAD'
|| beforeRetry?.failedAttempts !== 12 || afterRetry?.publishStatus !== 'PUBLISHED'
|| beforeRetry.eventId !== producer.eventId || afterRetry.eventId !== producer.eventId
|| beforeRetry.payloadHash !== producer.payloadHash || afterRetry.payloadHash !== producer.payloadHash) return false
const before = beforeRetry.businessOrderIds, after = afterRetry.businessOrderIds
return Array.isArray(before) && Array.isArray(after) && before.length > 0
&& before.every(identifier) && after.every(identifier)
&& new Set(before).size === before.length && new Set(after).size === after.length
&& JSON.stringify(sortedIds(before)) === JSON.stringify(sortedIds(after))
&& proof.restored === true
}
@@ -229,6 +229,10 @@ export async function run(context) {
packageCount: Number(item?.packageCount), picked: Number(item?.pickedPackageCount) })),
})
const operationsBeforeSignature = JSON.stringify(operationsBefore || [])
if (typeof context.context?.beforeLoadingComplete === 'function') {
await context.context.beforeLoadingComplete({ marker: context.marker, customerId: tenantId,
loadingPlanId, containerBookingId })
}
const completeAttempts = [1, 2].map((index) => ({
body: {
expectedTaskVersion: Number(taskBefore.version),
@@ -1,3 +1,5 @@
import { validDirectDeliveryReplayProof } from './direct-delivery-replay-proof.mjs'
const MODULE = 'phase4-cross-region-events'
const OPTIONS = { scenario: 'phase4-cross-region-events', deploymentBlocker: true }
@@ -139,7 +141,18 @@ export async function run(context) {
success(writtenBackAgain) && frozen.every((field) => (replaySnapshot[field] ?? null) === (snapshot[field] ?? null))
&& traced(writtenBackAgain),
evidence(writtenBackAgain, `containerId=${id(domesticMatches[0].id)}, sourceVersion=${snapshot.emptyReportSourceVersion ?? 'missing'}`), OPTIONS)
blockingGap(context, '真实投递重试与同事件幂等证据',
const reader = context.context?.directDeliveryReplayEvidenceReader
if (typeof reader !== 'function') blockingGap(context, '真实投递重试与同事件幂等证据',
'两次GET仅证明读取稳定;需按event-outbox-operations.mjs的受保护内部运维流程,在本轮事件上验证故障后有限重试/DEAD及同eventId+payloadHash重复消费,保存逐次尝试与业务计数证据。不得重放SENT、改表或伪造新事件。',
'missing-direct-delivery-replay-evidence')
const scope = { marker: context.marker, loadingPlanId: id(phase3.loadingPlanId),
customerId: id(phase3.tenantId), containerBookingId: id(phase3.containerBookingId) }
const proof = await reader(scope)
context.createdResources.directDeliveryReplayProof = proof
const valid = validDirectDeliveryReplayProof(proof, scope)
context.assert(MODULE, '本轮原事件12次失败DEAD人工恢复DUPLICATE且业务零新增', null, valid,
`loadingPlanId=${scope.loadingPlanId}, attempts=${proof?.attempts?.length || 0}, restored=${proof?.restored === true}`, OPTIONS)
if (!valid) blockingGap(context, '真实投递重试证据不完整或身份漂移',
'本轮事件身份/摘要、12次有限重试、DEAD、人工恢复审计、重复消费及业务计数和恢复直连必须同时核平',
'invalid-direct-delivery-replay-evidence')
}
@@ -6,7 +6,54 @@ import { fileURLToPath } from 'node:url'
import { expandOverseasCartonCandidates, resolveCartonFixtures, run, uniqueContainerProducer,
crossContainerCartonProbe, normalizeReceivingEvidence, receivingEvidenceValid,
receivingCompletionProof, captureReceivingEvidence, probeCartonResolveNegatives } from './phase4-overseas-pda.mjs'
receivingCompletionProof, captureReceivingEvidence, receivingEventRecoveryInventoryUnchanged,
probeCartonResolveNegatives, awaitReceivingCandidate } from './phase4-overseas-pda.mjs'
function candidateContext(responses) {
const calls=[],steps=[]
return {calls,steps,createdResources:{},addStep:(_module,name,_response,passed)=>steps.push({name,passed}),
request:async(method,path,_body,token)=>{calls.push({method,path,token});return responses[Math.min(calls.length-1,responses.length-1)]}}
}
const candidateResponse = data => ({http:200,body:{code:'200',data},requestId:'offline-request',traceId:'offline-trace'})
const exactCandidate = {containerNo:'TSTU6403402',sourceContainerBookingId:'2105838069516079106',operable:true,totalPackages:1}
const exactProducer = {containerNo:exactCandidate.containerNo,containerBookingId:exactCandidate.sourceContainerBookingId}
test('original r2 async shipping gate race waits same booking and container until visible without writing',async()=>{
const context=candidateContext([candidateResponse([]),candidateResponse([]),candidateResponse([exactCandidate])]),delays=[]
const result=await awaitReceivingCandidate(context,'offline-picker',exactProducer,{attempts:3,delay:async ms=>delays.push(ms)})
assert.deepEqual(result.candidate,exactCandidate)
assert.deepEqual(delays,[1000,1000])
assert.equal(context.calls.length,3)
assert.ok(context.calls.every(call=>call.method==='GET'&&call.token==='offline-picker'&&call.path.endsWith('/candidates')))
const proof=context.createdResources.phase4OverseasPda.candidateWaitEvidence[0]
assert.equal(proof.outcome,'READY');assert.deepEqual(proof.attempts.map(v=>v.exactMatches),[0,0,1])
assert.ok(proof.attempts.every(v=>v.requestId&&v.traceId))
})
test('same container from another booking, other container and nonoperable original cannot satisfy wait',async()=>{
for(const candidate of [{...exactCandidate,sourceContainerBookingId:'999'},
{...exactCandidate,containerNo:'ANOTHER'},{...exactCandidate,operable:false}]) {
const context=candidateContext([candidateResponse([candidate])]);let waits=0
const result=await awaitReceivingCandidate(context,'offline',exactProducer,{attempts:2,delay:async()=>waits++})
assert.equal(result.candidate,null);assert.equal(result.reason,'TIMED_OUT')
assert.equal(context.calls.length,2);assert.equal(waits,1)
}
})
test('ambiguous exact candidates and authentication or malformed replies stop without masking server errors',async()=>{
for(const response of [candidateResponse([exactCandidate,exactCandidate]),{http:401,body:{code:'401'}},candidateResponse({records:[exactCandidate]})]) {
const context=candidateContext([response]);let waits=0
const result=await awaitReceivingCandidate(context,'offline',exactProducer,{attempts:3,delay:async()=>waits++})
assert.equal(result.candidate,null);assert.equal(context.calls.length,1);assert.equal(waits,0)
assert.ok(['AMBIGUOUS','QUERY_FAILED'].includes(result.reason))
}
})
test('candidate waits are per invocation with no cross-mode cache and enforce exact scope/bounds',async()=>{
const context=candidateContext([candidateResponse([exactCandidate]),candidateResponse([])])
assert.ok((await awaitReceivingCandidate(context,'first',exactProducer,{attempts:1})).candidate)
assert.equal((await awaitReceivingCandidate(context,'second',exactProducer,{attempts:1})).candidate,null)
for(const producer of [{containerNo:'TSTU6403402'},{}])await assert.rejects(awaitReceivingCandidate(context,'offline',producer))
await assert.rejects(awaitReceivingCandidate(context,'offline',exactProducer,{attempts:31}))
assert.equal(context.calls.length,2)
})
test('single-container continuation retains malformed/unknown probes without requiring unshipped cancelled CHILD; default chain still blocks missing real other container', async () => {
const requests = [], gaps = [], checks = []
@@ -73,6 +120,28 @@ test('receiving completion freezes all raw balances when group MIN represents tw
assert.equal(stages.AFTER_COMPLETE.positions.length, 2)
})
test('unrelated canonical UNBOUND stock remains in all snapshots but new receiving still requires a real pallet', () => {
const { scope, stages } = receivingProofFixture()
const unboundPosition = { ...stages.AFTER_COMPLETE.positions[0], id: '301', balanceId: '30',
palletId: '0', palletNo: 'UNBOUND', availablePackages: 1 }
for (const snapshot of Object.values(stages)) {
const balance = { ...snapshot.balances[0], id: '30', inboundOrderId: '30',
inboundCode: 'UNBOUND-SOURCE', skuCode: 'UNRELATED', availablePackages: 1, inTransitPackages: 0 }
snapshot.balances.push(balance)
snapshot.httpGroups.push({ ...balance })
snapshot.positions.push({ ...unboundPosition })
snapshot.palletComposition.push({ ...unboundPosition })
assert.equal(receivingEvidenceValid(snapshot, scope), true)
}
assert.equal(receivingCompletionProof(scope, stages).passed, true)
stages.AFTER_COMPLETE.stockReceipts[0].warehousePalletId = '0'
assert.equal(receivingEvidenceValid(stages.AFTER_COMPLETE, scope), false)
const malformed = structuredClone(stages.BEFORE_COMPLETE)
malformed.positions[0].palletNo = 'UNKNOWN'
malformed.palletComposition[0].palletNo = 'UNKNOWN'
assert.equal(receivingEvidenceValid(malformed, scope), false)
})
test('receiving proof rejects missing raw source, wrong customer/warehouse, prefix source, missing bucket, ledger key, duplicate receipt and operation mismatch', () => {
const mutations = [
row => row.balances.pop(), row => row.positions[0].customerId = 'OTHER',
@@ -111,6 +180,21 @@ test('normalization is deterministic, preserves zero buckets, and excludes token
assert.equal(normalizeReceivingEvidence({ ...after, stockReceipts: undefined }), null)
})
test('reverse event recovery requires complete owned inventory evidence and every immutable fact unchanged', () => {
const { scope, stages } = receivingProofFixture()
const before = stages.AFTER_FRESH_KEY_REJECTION
assert.equal(receivingEventRecoveryInventoryUnchanged(scope, before, structuredClone(before)), true)
assert.equal(receivingEventRecoveryInventoryUnchanged(scope, before, null), false)
assert.equal(receivingEventRecoveryInventoryUnchanged({ ...scope, customerId: 'OTHER' }, before, before), false)
for (const mutate of [row => row.balances[0].availablePackages++, row => row.stockReceipts.pop(),
row => row.positions[0].warehouseLocationCode = 'ANOTHER', row => row.stockInLedger[0].sourceEventId = 'OTHER',
row => row.receivingOperations.push({ ...row.receivingOperations[0], id: '999' })]) {
const changed = structuredClone(before)
mutate(changed)
assert.equal(receivingEventRecoveryInventoryUnchanged(scope, before, changed), false)
}
})
test('missing read-only receipt/key reader is a blocker and retains an unproven boundary without inventing HTTP', async () => {
const checks = [], target = { stages: {} }
const context = { context: {}, addCoverageGap() {}, assert(m, n, r, passed) { checks.push(passed) },
@@ -166,7 +250,7 @@ test('physical OVERSEAS and dropship probes still require another real loaded pr
})
test('receiving selects one current booking and supplies mandatory booking identity', () => {
assert.match(source, /id\(item.sourceContainerBookingId\) === preferredBookingId/)
assert.match(source, /item.containerNo === containerNo && id\(item.sourceContainerBookingId\) === bookingId/)
assert.match(source, /matchingCandidates.length === 1/)
assert.match(source, /if \(!longId\(sourceContainerBookingId\)\)/)
assert.match(source, /sourceContainerBookingId=\$\{encodeURIComponent\(sourceContainerBookingId\)\}/)
@@ -314,7 +398,8 @@ test('PDA module covers the container-number receiving contract and excludes out
assert.match(source, /standardFbaProducers = \(Array\.isArray\(matrix\?\.containers\) \? matrix\.containers : \[\]\)/)
assert.match(source, /item\?\.fulfillmentType !== 'ONE_CLICK_DROPSHIP'/)
assert.match(source, /FBA-OPERATIONS/)
assert.match(source, /\.slice\(0, 3\)/)
assert.match(source, /FBA-ROUTE/)
assert.match(source, /\.slice\(0, 4\)/)
assert.match(source, /const producers = \[dropship, \.\.\.standardFbaProducers,/)
assert.doesNotMatch(source, /创建当前柜实收托盘|创建当前柜实收库位|async function createPallet/)
assert.match(source, /retained as acceptance evidence/)
@@ -417,7 +502,7 @@ test('real-device deployed FBA resolve A0404 is retained until server-side inbou
test('PDA receiving lifecycle is ordered from candidate claim through photos, mode-specific receipts, pallet and complete', () => {
const orderedTokens = [
"'查询当前客户待收柜号'", "'选择柜号并建立WAIT_SIGN会话'",
'const candidates = await awaitReceivingCandidate', "'选择柜号并建立WAIT_SIGN会话'",
'uploadPhotos(context, token)', "'绑定车牌封签开柜三角色照片并签收'",
'resolveGeneratedPallet(context, inbound)', "`解析${inbound.destinationType}实物托盘`", 'receiveFba(context, token',
'receiveOverseas(context, token', "'回查柜内逐入仓码实收进度'",
@@ -464,7 +549,7 @@ test('missing OVERSEAS_WAREHOUSE_PDA_OPERATOR overseas authority is a deployment
assert.ok(steps.some((step) => step?.passed === false && step?.options?.deploymentBlocker))
})
test('empty candidate response cannot become a zero-business-request pass', async () => {
test('missing original producer identity blocks before candidate lookup and still cleans temporary identities', async () => {
const calls = []; const gaps = []; const steps = []
const response = (data, requestPath, method = 'GET') => ({ method, path: requestPath, http: 200,
body: { code: '200', data }, requestId: 'req-contract', traceId: 'trace-contract' })
@@ -509,9 +594,9 @@ test('empty candidate response cannot become a zero-business-request pass', asyn
addCoverageGap(...args) { gaps.push(args) },
}
await run(context)
assert.ok(calls.some((call) => call.requestPath.endsWith('/candidates')))
assert.equal(calls.some((call) => call.requestPath.endsWith('/candidates')),false)
assert.ok(steps.some((step) => step.name.includes('正常空态') && step.passed))
assert.ok(steps.some((step) => step.name.includes('PDA柜号候选覆盖前置') && !step.passed && step.options.deploymentBlocker))
assert.ok(steps.some((step) => step.name.includes('本轮原柜候选身份覆盖前置') && !step.passed && step.options.deploymentBlocker))
assert.equal(gaps.length, 1)
assert.equal(context.createdResources.phase4OverseasPda.finalStatus, null)
assert.equal(context.createdResources.phase4OverseasPda.pickerMembershipIds.length, 2)
@@ -99,7 +99,8 @@ export function receivingEvidenceValid(snapshot, scope) {
|| !snapshot.httpGroups.every(row => owned(row) && bucketsValid(row, RECEIVING_BALANCE_BUCKETS))
|| !snapshot.httpLedger.every(owned)
|| !snapshot.positions.every(row => owned(row) && bucketsValid(row, RECEIVING_POSITION_BUCKETS)
&& snapshot.balances.some(balance => balance.id === row.balanceId) && longId(row.palletId) && nonBlank(row.palletNo)
&& snapshot.balances.some(balance => balance.id === row.balanceId)
&& ((longId(row.palletId) && nonBlank(row.palletNo)) || (row.palletId === '0' && row.palletNo === 'UNBOUND'))
&& longId(row.warehouseLocationId) && nonBlank(row.warehouseLocationCode))
|| !sameEvidence(snapshot.positions, snapshot.palletComposition)) return false
if (!snapshot.balances.every(balance => RECEIVING_POSITION_BUCKETS.every(field =>
@@ -116,6 +117,7 @@ export function receivingEvidenceValid(snapshot, scope) {
|| !snapshot.httpGroups.every(row => groups.has(balanceGroupKey(row)) && RECEIVING_BALANCE_BUCKETS.every(field => row[field] === groups.get(balanceGroupKey(row))[field]))) return false
if (!snapshot.stockReceipts.every(row => row.customerId === scope.customerId && scope.inboundCodes.includes(row.inboundCode)
&& longId(row.planId) && longId(row.taskId) && longId(row.allocationId) && longId(row.lineId) && longId(row.orderId)
&& longId(row.warehousePalletId) && nonBlank(row.palletNo)
&& nonBlank(row.skuCode) && Number.isSafeInteger(row.receivedPackages) && row.receivedPackages > 0)) return false
if (snapshot.inventoryOperations.length !== snapshot.stockReceipts.length || snapshot.stockInLedger.length !== snapshot.stockReceipts.length) return false
for (const receipt of snapshot.stockReceipts) {
@@ -220,6 +222,11 @@ export async function captureReceivingEvidence(context, scope, stage, target) {
}
}
export function receivingEventRecoveryInventoryUnchanged(scope, before, after) {
return receivingEvidenceValid(before, scope) && receivingEvidenceValid(after, scope)
&& sameEvidence(before, after)
}
export function expandOverseasCartonCandidates(inbounds, producerCartonMarks = []) {
const marks = (Array.isArray(producerCartonMarks) ? producerCartonMarks : [producerCartonMarks])
.map(id).filter(nonBlank)
@@ -339,7 +346,17 @@ export async function createPicker(context, index, overseasWarehouseId, access)
}
context.addStep(MODULE, `海外PDA操作员${index}安全投影已收敛`, projectionResponse, projectionReady,
`attempts=${projectionAttempt + 1}; ${trace(projectionResponse)}`, OPTIONS)
return { membershipId, username, token: projectionReady ? token : null }
return { membershipId, username, token: projectionReady ? token : null,
refreshToken: login.body?.data?.refreshToken, password,
userUpdate: { displayName: payload.displayName, phone, email: payload.email, tenantId: '0',
roleIds: payload.roleIds, overseasWarehouseId, domesticWarehouseId: null } }
}
export function pickerSessionHandoff(picker) {
if (!picker) return null
const { membershipId, username, token, refreshToken, password, userUpdate } = picker
return { membershipId, username, token, refreshToken, password,
userUpdate: userUpdate ? { ...userUpdate, roleIds: [...userUpdate.roleIds] } : undefined }
}
export async function disablePicker(context, picker, index) {
@@ -827,7 +844,37 @@ async function receiveDropship(context, token, producer, inbounds) {
return accepted === expectedPending
}
export async function runReceiving(context, token, secondaryToken, merchantToken, preferredProducer = null, runAuthorizationProbes = true) {
export async function awaitReceivingCandidate(context, token, producer, { attempts = 30, intervalMs = 1000,
delay = ms => new Promise(resolve => setTimeout(resolve, ms)) } = {}) {
const containerNo = id(producer?.containerNo), bookingId = id(producer?.containerBookingId)
if (!nonBlank(containerNo) || !longId(bookingId) || !Number.isInteger(attempts) || attempts < 1 || attempts > 30
|| !Number.isInteger(intervalMs) || intervalMs < 0 || intervalMs > 1000) throw Error('Invalid exact receiving candidate scope or wait bound')
const proof = { containerNo, sourceContainerBookingId:bookingId, attempts:[], outcome:'PENDING' }
const resource = context.createdResources.phase4OverseasPda ||= {}
;(resource.candidateWaitEvidence ||= []).push(proof)
let result
for (let attempt = 1; attempt <= attempts; attempt++) {
result = await pda(context, token, 'GET', `查询本轮原柜候选收敛第${attempt}次`, '/api/pda/v1/last-leg/receiving-containers/candidates')
const rows = Array.isArray(result.data) ? result.data : []
const matchingCandidates = rows.filter(item => item.containerNo === containerNo && id(item.sourceContainerBookingId) === bookingId)
proof.attempts.push({ attempt, http:result.response?.http, code:code(result.response), requestId:result.response?.requestId,
traceId:result.response?.traceId, candidateCount:rows.length, exactMatches:matchingCandidates.length,
sameNumberCandidates:rows.filter(item => item.containerNo === containerNo).map(item => ({
containerNo:item.containerNo, sourceContainerBookingId:id(item.sourceContainerBookingId),
operable:item.operable, blockReason:item.blockReason })) })
if (!success(result.response) || !Array.isArray(result.data)) { proof.outcome='QUERY_FAILED'; break }
if (matchingCandidates.length > 1) { proof.outcome='AMBIGUOUS'; break }
if (matchingCandidates.length === 1 && matchingCandidates[0].operable === true) {
proof.outcome='READY'
return {...result,candidate:matchingCandidates[0]}
}
if (attempt < attempts) await delay(intervalMs)
}
if (proof.outcome === 'PENDING') proof.outcome='TIMED_OUT'
return {...result,candidate:null,reason:proof.outcome}
}
export async function runReceiving(context, token, secondaryToken, merchantToken, preferredProducer = null, runAuthorizationProbes = true, candidateWait = {}) {
if (runAuthorizationProbes && context.token) {
const denied = await context.request('GET', '/api/pda/v1/last-leg/receiving-containers/candidates', undefined, context.token, {})
context.addStep(MODULE, '平台管理员访问令牌不能冒充仓库PDA操作员', denied, forbidden(denied) && traced(denied), trace(denied), OPTIONS)
@@ -844,20 +891,19 @@ export async function runReceiving(context, token, secondaryToken, merchantToken
blockerGap(context, '可重复的完整柜号验收', `账号已有未完成柜 ${before.body?.data?.containerNo || 'unknown'},本轮不覆盖或篡改遗留现场会话`, 'dirty-fixture', before)
return
}
const candidates = await pda(context, token, 'GET', '查询当前客户待收柜号', '/api/pda/v1/last-leg/receiving-containers/candidates')
if (!Array.isArray(candidates.data) || candidates.data.length === 0) {
blockerGap(context, 'PDA柜号候选', `当前客户没有由三期装柜完成事件生成的待收入仓计划;${trace(candidates.response)}`, 'producer-chain', candidates.response)
return
}
const preferredContainer = id(preferredProducer?.containerNo)
|| context.createdResources?.phase3Loading?.containerNo
const preferredBookingId = id(preferredProducer?.containerBookingId)
const matchingCandidates = candidates.data.filter(item => item.containerNo === preferredContainer
&& (!preferredBookingId || id(item.sourceContainerBookingId) === preferredBookingId))
const candidate = matchingCandidates.length === 1 ? matchingCandidates[0] : null
const preferredBookingId = id(preferredProducer?.containerBookingId || context.createdResources?.phase3Loading?.containerBookingId)
if (!nonBlank(preferredContainer) || !longId(preferredBookingId)) {
blockerGap(context, '本轮原柜候选身份', '缺少本轮containerNo或containerBookingId,禁止仅按柜号匹配历史柜', 'producer-chain')
return
}
const candidates = await awaitReceivingCandidate(context,token,
{containerNo:preferredContainer,containerBookingId:preferredBookingId},candidateWait)
const candidate = candidates.candidate
if (!candidate) {
blockerGap(context, `本轮柜号${preferredContainer || 'missing'}候选`,
'三期装柜完成生产者已落地,但海外候选尚未出现该柜;禁止回退处理历史或其他柜', 'producer-chain', candidates.response)
`本轮原柜候选未在有界等待内唯一可操作:${candidates.reason};禁止回退处理历史或其他柜`, 'producer-chain', candidates.response)
return
}
const containerNo = id(candidate?.containerNo)
@@ -1015,6 +1061,9 @@ export async function runReceiving(context, token, secondaryToken, merchantToken
const receivingProof = { scope, stages: {}, proof: null, status: 'UNPROVEN' }
context.createdResources.phase4OverseasPda.receivingCompletionEvidence ||= {}
context.createdResources.phase4OverseasPda.receivingCompletionEvidence[sourceContainerBookingId] = receivingProof
if (typeof context.context.beforeReceivingComplete === 'function') {
await context.context.beforeReceivingComplete(scope)
}
await captureReceivingEvidence(context, scope, 'BEFORE_COMPLETE', receivingProof)
const completed = await pda(context, token, 'POST', '完成当前柜并写入库存与卸柜事件', '/api/pda/v1/last-leg/receiving-containers/current/complete', completeBody)
await captureReceivingEvidence(context, scope, 'AFTER_COMPLETE', receivingProof)
@@ -1033,6 +1082,20 @@ export async function runReceiving(context, token, secondaryToken, merchantToken
receivingProof.status = receivingProof.proof.passed && Object.values(receivingProof.commandResults).every(Boolean) ? 'VERIFIED' : 'UNPROVEN'
for (const [check, passed] of Object.entries(receivingProof.proof.checks)) context.assert(MODULE, `收货完成库存守恒 ${check}`, null, passed,
`bookingId=${sourceContainerBookingId}, proofStatus=${receivingProof.status}`, OPTIONS)
if (typeof context.context.afterReceivingComplete === 'function') {
if (receivingProof.status !== 'VERIFIED') throw new Error('收货完成库存证据未闭合,禁止继续故障恢复验收')
const eventRecovery = await context.context.afterReceivingComplete(scope)
if (eventRecovery) {
receivingProof.eventRecovery = eventRecovery
const afterRecovery = await captureReceivingEvidence(context, scope, 'AFTER_EVENT_RECOVERY', receivingProof)
const unchanged = eventRecovery.restored === true && receivingEventRecoveryInventoryUnchanged(scope,
receivingProof.stages.AFTER_FRESH_KEY_REJECTION.evidence, afterRecovery)
receivingProof.eventRecoveryInventoryUnchanged = unchanged
context.assert(MODULE, '原卸柜事件DEAD恢复后库存位置托盘组成及不可变流水零漂移', null, unchanged,
`bookingId=${sourceContainerBookingId}`, OPTIONS)
if (!unchanged) throw new Error('卸柜事件恢复后库存证据变化或服务恢复未验证')
}
}
const after = await context.request('GET', '/api/pda/v1/last-leg/receiving-containers/current', undefined, token, {})
context.addStep(MODULE, '完成后current返回404正常空态', after, notFound(after), trace(after), OPTIONS)
context.createdResources.phase4OverseasPda.finalStatus = completed.data?.status || null
@@ -1110,8 +1173,8 @@ export async function run(context) {
}
context.context.phase4OverseasPdaOperations = {
marker: context.marker,
primary: primary ? { membershipId: primary.membershipId, username: primary.username, token: primary.token } : null,
secondary: secondary ? { membershipId: secondary.membershipId, username: secondary.username, token: secondary.token } : null,
primary: pickerSessionHandoff(primary),
secondary: pickerSessionHandoff(secondary),
overseasWarehouseId: sharedWarehouse.id,
merchantToken,
completedFixtures: [],
@@ -1127,9 +1190,9 @@ export async function run(context) {
const standardFbaProducers = (Array.isArray(matrix?.containers) ? matrix.containers : [])
.filter((item) => item?.destinationType === 'FBA')
.filter((item) => item?.fulfillmentType !== 'ONE_CLICK_DROPSHIP')
.filter((item) => /-P4-(?:FBA-STANDARD|FBA-SECOND|FBA-OPERATIONS)(?:-R\d+)?:/.test(String(item?.resourceKey || '')))
.filter((item) => /-P4-(?:FBA-STANDARD|FBA-SECOND|FBA-OPERATIONS|FBA-ROUTE)(?:-R\d+)?:/.test(String(item?.resourceKey || '')))
.sort((left, right) => String(left?.resourceKey).localeCompare(String(right?.resourceKey)))
.slice(0, 3)
.slice(0, 4)
// The sealed dropship pallet proves the indivisible handover. A separate standard FBA
// fixture supplies multiple cartons for reversible operations. The second standard FBA
// fixture remains available for the multi-destination dispatch and delivery split proof.
+13 -5
View File
@@ -32,10 +32,11 @@ def encoded(value):
class State:
def __init__(self, event_type, event_id, loading_plan_id, log_path=None):
def __init__(self, event_type, event_id, loading_plan_id, log_path=None, container_booking_id=None):
self.event_type = event_type
self.selector_id = event_id
self.plan_id = loading_plan_id
self.booking_id = container_booking_id
self.log_path = log_path
self.lock = threading.Lock()
self.mode = "PASS"
@@ -65,9 +66,13 @@ class State:
return None, False, None, 0, self.mode
identity = {"eventId": event_id, "payloadHash": payload_hash.lower()}
payload = event.get("payload")
qualifies = event.get("eventType") == self.event_type and (
event_id == self.selector_id if self.selector_id else
isinstance(payload, dict) and str(payload.get("loadingPlanId")) == self.plan_id)
if self.selector_id:
selected = event_id == self.selector_id
elif self.booking_id:
selected = isinstance(payload, dict) and str(payload.get("sourceContainerBookingId")) == self.booking_id
else:
selected = isinstance(payload, dict) and str(payload.get("loadingPlanId")) == self.plan_id
qualifies = event.get("eventType") == self.event_type and selected
envelope_hash = hashlib.sha256(json.dumps(event, sort_keys=True, ensure_ascii=False,
separators=(",", ":")).encode("utf-8")).hexdigest()
with self.lock:
@@ -254,6 +259,7 @@ def main():
selector = parser.add_mutually_exclusive_group(required=True)
selector.add_argument("--event-id")
selector.add_argument("--loading-plan-id")
selector.add_argument("--container-booking-id")
parser.add_argument("--bind", default="127.0.0.1")
parser.add_argument("--port", type=int, default=0)
parser.add_argument("--control-port", type=int, default=0)
@@ -270,9 +276,11 @@ def main():
parser.error("event-id must be a UUID or ULID")
if args.loading_plan_id and not re.fullmatch(r"[1-9][0-9]{0,19}", args.loading_plan_id):
parser.error("loading-plan-id must be a positive decimal identifier")
if args.container_booking_id and not re.fullmatch(r"[1-9][0-9]{0,19}", args.container_booking_id):
parser.error("container-booking-id must be a positive decimal identifier")
if not re.fullmatch(r"[A-Za-z][A-Za-z0-9]{0,79}", args.event_type) or not 0 < args.timeout <= 120:
parser.error("invalid event type or timeout")
state = State(args.event_type, args.event_id, args.loading_plan_id, args.log)
state = State(args.event_type, args.event_id, args.loading_plan_id, args.log, args.container_booking_id)
data = ThreadingHTTPServer((args.bind, args.port), data_handler(state, args.upstream, args.timeout))
control = ThreadingHTTPServer(("127.0.0.1", args.control_port), control_handler(state, token))
for server in (data, control):
+42 -3
View File
@@ -1,5 +1,5 @@
import assert from 'node:assert/strict'
import { spawn } from 'node:child_process'
import { spawn, spawnSync } from 'node:child_process'
import { createServer } from 'node:http'
import { mkdtemp, readFile, rm } from 'node:fs/promises'
import { tmpdir } from 'node:os'
@@ -19,7 +19,7 @@ const event = (id = targetId, planId = '2100000000000000001') => ({ eventId: id,
payload: { loadingPlanId: planId, syntheticPrivateField: secret } })
const close = server => new Promise(resolve => server.close(resolve))
async function fixture(selector = ['--loading-plan-id', '2100000000000000001']) {
async function fixture(selector = ['--loading-plan-id', '2100000000000000001'], eventType = 'LoadingCompleted') {
const calls = []
const applied = new Set()
let invalidAck = false
@@ -40,7 +40,7 @@ async function fixture(selector = ['--loading-plan-id', '2100000000000000001'])
const directory = await mkdtemp(join(tmpdir(), 'event-fault-offline-'))
const log = join(directory, 'evidence.jsonl')
const child = spawn(process.env.PYTHON || 'python', [script, '--upstream',
`http://127.0.0.1:${consumer.address().port}/internal/events`, '--event-type', 'LoadingCompleted',
`http://127.0.0.1:${consumer.address().port}/internal/events`, '--event-type', eventType,
...selector, '--log', log], { env: { ...process.env, EVENT_FAULT_CONTROL_TOKEN: controlToken },
stdio: ['ignore', 'pipe', 'pipe'] })
let output = ''; let errors = ''
@@ -179,3 +179,42 @@ test('explicit event-ID selector requires exact event type and forwards unrelate
assert.equal(status.attempt, 1)
assert.equal(status.eventId, targetId)
})
test('unloading selector pins only the exact source booking and preserves reverse-event ACK recovery', async t => {
const bookingId = '2100000000000000066'
const proxy = await fixture(['--container-booking-id', bookingId], 'OverseasContainerUnloaded')
t.after(() => proxy.stop())
await proxy.control('/mode', { mode: 'FAULT' })
const unloaded = { ...event(), eventType: 'OverseasContainerUnloaded',
payload: { sourceContainerBookingId: bookingId, containerNo: 'TEST-CONTAINER' } }
for (const unrelated of [
{ ...unloaded, eventId: otherId, payload: { sourceContainerBookingId: '2100000000000000067' } },
{ ...unloaded, eventId: otherId, payload: { loadingPlanId: bookingId } },
{ ...unloaded, eventId: otherId, eventType: 'OverseasDeliveryCompleted' },
]) assert.equal((await proxy.send(unrelated)).status, 200)
assert.equal((await proxy.send(unloaded)).status, 503)
assert.equal((await proxy.send(unloaded)).status, 503)
assert.equal((await proxy.send({ ...unloaded, eventId: samePlanOtherId })).status, 200)
const before = proxy.calls.length
assert.equal((await proxy.send({ ...unloaded, payload: { sourceContainerBookingId: '2100000000000000067' } })).status, 409)
assert.equal(proxy.calls.length, before)
await proxy.control('/mode', { mode: 'PASS' })
const recovered = await proxy.send(unloaded)
assert.equal(recovered.status, 200)
assert.equal(JSON.parse(recovered.raw).data.outcome, 'DUPLICATE')
const status = await (await proxy.control('/status')).json()
assert.equal(status.eventId, targetId)
assert.deepEqual(status.records.filter(row => row.errorCode !== 'TARGET_EVENT_HASH_CONFLICT').map(row => row.outcome),
['APPLIED', 'DUPLICATE', 'DUPLICATE'])
})
test('booking selector rejects malformed identities and mixed selection before starting listeners', () => {
for (const args of [['--container-booking-id', '0'], ['--container-booking-id', '../66'],
['--container-booking-id', '66', '--loading-plan-id', '77'], ['--container-booking-id', '66', '--event-id', targetId]]) {
const result = spawnSync(process.env.PYTHON || 'python', [script, '--upstream', 'http://127.0.0.1:1/internal/events',
'--event-type', 'OverseasContainerUnloaded', ...args], { encoding: 'utf8', timeout: 5000,
env: { ...process.env, EVENT_FAULT_CONTROL_TOKEN: controlToken } })
assert.equal(result.status, 2)
assert.ok(!result.stdout.includes('"ready"'))
}
})
+185
View File
@@ -0,0 +1,185 @@
// Internal explicit TEST evidence queries; no network or environment reads on import.
const stages = new Set(['BEFORE_COMPLETE', 'AFTER_COMPLETE', 'AFTER_SAME_KEY_REPLAY', 'AFTER_FRESH_KEY_REJECTION', 'AFTER_EVENT_RECOVERY'])
const numeric = new Set(['availablePackages', 'holdPackages', 'allocatedPackages', 'pickedPackages', 'inTransitPackages', 'receivingPackages', 'procurementPackages', 'receivedPackages', 'beforePackages', 'deltaPackages', 'afterPackages', 'packageCount'])
const fields = {
sources: 'id customerId overseasWarehouseId sourceContainerBookingId containerNo sourceInboundId inboundCode sourceRegion packageCount',
sessions: 'id sourceContainerBookingId containerNo overseasWarehouseId',
balances: 'id customerId overseasWarehouseId destinationWarehouseId destinationWarehouseCode inboundOrderId inboundOrderNo inboundCode skuCode availablePackages holdPackages allocatedPackages pickedPackages inTransitPackages receivingPackages procurementPackages',
positions: 'id balanceId customerId overseasWarehouseId destinationWarehouseCode palletId palletNo batchNo warehouseLocationId warehouseLocationCode status availablePackages holdPackages allocatedPackages pickedPackages receivingPackages',
stockReceipts: 'id customerId planId taskId allocationId lineId orderId inboundCode skuCode warehousePalletId palletNo warehouseLocationId warehouseLocationCode batchNo receivedPackages inventoryBalanceId inventoryPositionId inventoryOperationKey',
inventoryOperations: 'id customerId idempotencyKey operationType requestHash status',
stockInLedger: 'id customerId balanceId positionId overseasWarehouseId inboundOrderId inboundCode palletId palletNo batchNo skuCode businessType businessOrderNo sourceEventId idempotencyKey statusFrom statusTo beforePackages deltaPackages afterPackages',
receivingOperations: 'id sessionId action idempotencyKey requestHash',
}
const snake = value => value.replace(/[A-Z]/g, letter => '_' + letter.toLowerCase())
function identifier(value) {
if (typeof value !== 'string' || !/^[1-9][0-9]{0,19}$/.test(value)) throw Error('Invalid receiving evidence identifier')
return value
}
// UTF-8 hex literals avoid quote, backslash and SQL-mode ambiguity.
const literal = value => `(CONVERT(0x${Buffer.from(value, 'utf8').toString('hex')} USING utf8mb4) COLLATE utf8mb4_bin)`
function validateScope(scope) {
for (const name of ['customerId', 'warehouseId', 'bookingId']) identifier(scope?.[name])
if (!stages.has(scope.stage) || typeof scope.containerNo !== 'string' || !scope.containerNo.trim()
|| !Array.isArray(scope.inboundCodes) || !scope.inboundCodes.length
|| scope.inboundCodes.some(code => typeof code !== 'string' || !code.trim() || code.length > 128)
|| new Set(scope.inboundCodes).size !== scope.inboundCodes.length
|| !Array.isArray(scope.sourceInboundIds) || scope.sourceInboundIds.length !== scope.inboundCodes.length
|| new Set(scope.sourceInboundIds).size !== scope.sourceInboundIds.length) throw Error('Invalid receiving evidence scope')
scope.sourceInboundIds.forEach(identifier)
for (const code of scope.inboundCodes) if (!Number.isSafeInteger(scope.expectedPackagesByInbound?.[code])
|| scope.expectedPackagesByInbound[code] <= 0) throw Error('Invalid receiving evidence quantity')
}
export function buildReceivingEvidenceSql(scope) {
validateScope(scope)
const owned = alias => `${alias}.customer_id=${literal(scope.customerId)} AND ${alias}.overseas_warehouse_id=${literal(scope.warehouseId)}`
const source = `o.customer_id=${literal(scope.customerId)} AND o.source_container_booking_id=${literal(scope.bookingId)}`
const orderIds = `SELECT o.id FROM ll_order o WHERE ${source}`
const session = `s.source_container_booking_id=${literal(scope.bookingId)} AND s.container_no=${literal(scope.containerNo)} AND s.overseas_warehouse_id=${scope.warehouseId}`
const taskIds = `SELECT t.id FROM ll_inbound_pda_task t JOIN ll_inbound_plan p ON p.id=t.plan_id JOIN ll_order o ON o.id=p.source_order_id WHERE ${source} AND BINARY t.customer_id=BINARY o.customer_id AND BINARY p.customer_id=BINARY o.customer_id`
const query = (name, alias, from, where) => {
const pairs = fields[name].split(' ').map(field => {
const column = `${alias}.${snake(field)}`
return `'${field}',${numeric.has(field) ? column : `CAST(${column} AS CHAR)`}`
}).join(',')
return `SELECT JSON_OBJECT('collection','${name}','row',JSON_OBJECT(${pairs})) FROM ${from} WHERE ${where} ORDER BY ${alias}.id;`
}
return [
'SET NAMES utf8mb4 COLLATE utf8mb4_bin;',
'SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ;',
'START TRANSACTION WITH CONSISTENT SNAPSHOT, READ ONLY;',
query('sources', 'o', 'll_order o', source),
query('sessions', 's', 'll_receiving_container_session s', session),
query('balances', 'b', 'll_inventory_balance b', owned('b')),
query('positions', 'p', 'll_inventory_position p', owned('p')),
query('stockReceipts', 'r', 'll_inbound_stock_receipt r', `r.customer_id=${literal(scope.customerId)} AND r.order_id IN (${orderIds})`),
query('inventoryOperations', 'i', 'll_inventory_operation i', `i.customer_id=${literal(scope.customerId)} AND i.operation_type=${literal('STOCK_IN')} AND EXISTS (SELECT 1 FROM ll_inbound_pda_task t WHERE t.id IN (${taskIds}) AND i.idempotency_key LIKE (CONCAT('PDA_STOCK_IN:',t.id,':%') COLLATE utf8mb4_bin))`),
query('stockInLedger', 'l', 'll_inventory_ledger l', `${owned('l')} AND l.business_type='STOCK_IN' AND l.inbound_order_id IN (${orderIds})`),
query('receivingOperations', 'r', 'll_receiving_container_operation r', `r.action='COMPLETE' AND r.session_id IN (SELECT s.id FROM ll_receiving_container_session s WHERE ${session})`),
'COMMIT;',
].join('\n')
}
export function receivingEvidenceFromRows(scope, rows) {
validateScope(scope)
const result = Object.fromEntries(Object.keys(fields).map(name => [name, []]))
if (!Array.isArray(rows)) throw Error('Receiving evidence rows missing')
for (const item of rows) {
if (!Object.hasOwn(result, item?.collection) || !item.row) throw Error('Invalid receiving evidence collection')
result[item.collection].push(item.row)
}
const sources = result.sources
if (sources.length !== scope.inboundCodes.length
|| new Set(sources.map(row => row.inboundCode)).size !== sources.length
|| new Set(sources.map(row => row.sourceInboundId)).size !== sources.length
|| sources.some(row => row.customerId !== scope.customerId || row.overseasWarehouseId !== scope.warehouseId
|| row.sourceContainerBookingId !== scope.bookingId || row.containerNo !== scope.containerNo
|| row.sourceRegion !== 'FIRSTLEG' || !scope.sourceInboundIds.includes(row.sourceInboundId)
|| !scope.inboundCodes.includes(row.inboundCode) || row.packageCount !== scope.expectedPackagesByInbound[row.inboundCode])
|| result.sessions.length !== 1 || result.sessions.some(row => row.sourceContainerBookingId !== scope.bookingId
|| row.containerNo !== scope.containerNo || row.overseasWarehouseId !== scope.warehouseId)) throw Error('Receiving evidence source/session scope mismatch')
delete result.sources
delete result.sessions
for (const rows of Object.values(result)) rows.sort((a, b) => String(a.id).localeCompare(String(b.id)))
result.palletComposition = result.positions.map(row => ({ ...row }))
return result
}
export function createReceivingEvidenceReader(executeReadOnly) {
if (typeof executeReadOnly !== 'function') throw Error('Receiving evidence transport missing')
return async scope => receivingEvidenceFromRows(scope, await executeReadOnly(buildReceivingEvidenceSql(scope)))
}
const transactionStart = ['SET NAMES utf8mb4 COLLATE utf8mb4_bin;',
'SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ;',
'START TRANSACTION WITH CONSISTENT SNAPSHOT, READ ONLY;']
export function assertReadOnlySql(sql) {
if (typeof sql !== 'string' || /--|\/\*|\b(?:INSERT|UPDATE|DELETE|REPLACE|DROP|ALTER|TRUNCATE|CALL|EXECUTE|PREPARE|INTO|OUTFILE|DUMPFILE|LOAD_FILE|SLEEP|BENCHMARK|GET_LOCK)\b/i.test(sql))
throw Error('Read-only SQL rejected')
const lines = sql.split('\n').filter(Boolean)
if (!transactionStart.every((line,i) => lines[i] === line) || lines.at(-1) !== 'COMMIT;'
|| lines.length < 5 || lines.slice(3,-1).some(line => !/^SELECT [^;]+;$/.test(line))) throw Error('Read-only transaction required')
return sql
}
export function buildCompletionEvidenceSql(scope) {
identifier(scope?.customerId); identifier(scope?.rootOrderId)
if (scope.eventType !== 'FIRST_LEG_ORDER_FULFILLMENT_COMPLETED' || !Array.isArray(scope.sourceOrderIds)
|| scope.sourceOrderIds.length !== 2 || new Set(scope.sourceOrderIds).size !== 2
|| !scope.sourceOrderIds.includes(scope.rootOrderId)) throw Error('Invalid completion evidence scope')
scope.sourceOrderIds.forEach(identifier)
return [...transactionStart,
`SELECT JSON_OBJECT('orderId',CAST(o.id AS CHAR),'customerId',CAST(o.tenant_id AS CHAR),'rootOrderId',CAST(o.root_order_id AS CHAR),'eventType','FIRST_LEG_ORDER_FULFILLMENT_COMPLETED','count',COUNT(x.id),'publishedCount',COALESCE(SUM(x.publish_status='PUBLISHED'),0),'pendingCount',COALESCE(SUM(x.publish_status='PENDING'),0),'failedCount',COALESCE(SUM(x.publish_status NOT IN ('PENDING','PUBLISHED')),0)) FROM ord_order o LEFT JOIN ord_outbox_event x ON x.aggregate_id=o.id AND x.tenant_id=o.tenant_id AND x.event_type='FIRST_LEG_ORDER_FULFILLMENT_COMPLETED' WHERE o.tenant_id=${scope.customerId} AND o.root_order_id=${scope.rootOrderId} AND o.id IN (${scope.sourceOrderIds.join(',')}) GROUP BY o.id,o.tenant_id,o.root_order_id ORDER BY o.id;`,
'COMMIT;'].join('\n')
}
export function completionEvidenceFromRows(scope, rows) {
buildCompletionEvidenceSql(scope)
if (!Array.isArray(rows) || rows.length !== 2 || new Set(rows.map(row => row.orderId)).size !== 2
|| rows.some(row => !scope.sourceOrderIds.includes(row.orderId) || row.customerId !== scope.customerId
|| row.rootOrderId !== scope.rootOrderId || row.eventType !== scope.eventType
|| ['count','publishedCount','pendingCount','failedCount'].some(key => !Number.isSafeInteger(row[key]) || row[key] < 0)))
throw Error('Completion evidence scope or counts mismatch')
return {readOnly:true,customerId:scope.customerId,rootOrderId:scope.rootOrderId,orders:rows}
}
export const EVIDENCE_ENV_KEYS = ['JENKINS_URL','JENKINS_USER','JENKINS_API_TOKEN','FBA_CORE_JENKINS_AGENT']
export function validateEvidenceConfig(env) {
if (EVIDENCE_ENV_KEYS.some(key => typeof env[key] !== 'string' || !env[key].trim())) throw Error('Missing TEST evidence configuration')
const url = new URL(env.JENKINS_URL)
if (!['http:','https:'].includes(url.protocol) || url.username || url.password || url.search || url.hash
|| !/^[A-Za-z0-9_.-]{1,128}$/.test(env.FBA_CORE_JENKINS_AGENT)) throw Error('Invalid TEST evidence configuration')
}
const base64Expression = value => '['+Buffer.from(value).toString('base64').match(/.{1,4000}/g).map(part=>`'${part}'`).join(',')+"].join('')"
// Credentials stay in the executor's memory. No deploy, SQL mutation, or arbitrary SQL entry point.
export function buildEvidenceRemoteScript(service, sql, agent) {
assertReadOnlySql(sql)
if (!['order','lastleg'].includes(service) || !/^[A-Za-z0-9_.-]{1,128}$/.test(agent)) throw Error('Invalid TEST evidence target')
const prefix=service.toUpperCase(),container=`firstleg-test-${service}`
const python = `import json,os,subprocess,urllib.parse,base64,re
try:
c=json.loads(subprocess.check_output(['docker','inspect','${container}'],text=True,stderr=subprocess.DEVNULL))[0]
if c.get('Name')!='/${container}' or c.get('Config',{}).get('Labels',{}).get('com.docker.compose.project')!='firstleg-test': raise RuntimeError('target')
e=dict(v.split('=',1) for v in c['Config']['Env'])
u=urllib.parse.urlsplit(e['${prefix}_DB_URL'].removeprefix('jdbc:'))
if not u.hostname or not u.path.lstrip('/'): raise RuntimeError('database')
childEnv=dict(os.environ,MYSQL_PWD=e['${prefix}_DB_PASSWORD'])
host='127.0.0.1' if u.hostname=='host.docker.internal' else u.hostname
sql=base64.b64decode('${Buffer.from(sql).toString('base64')}').decode('utf-8')
p=subprocess.run(['mysql','--host='+host,'--port='+str(u.port or 3306),'--user='+e['${prefix}_DB_USERNAME'],'--batch','--raw','--skip-column-names',u.path.lstrip('/')],input=sql,capture_output=True,text=True,env=childEnv,timeout=40)
if p.returncode:
code=re.search(r'ERROR ([0-9]{3,5})',p.stderr)
print('FBA_CORE_JSON='+json.dumps({'ok':False,'error':'MYSQL_'+(code.group(1) if code else 'UNKNOWN')}))
else: print('FBA_CORE_JSON='+json.dumps({'ok':True,'rows':[json.loads(line) for line in p.stdout.splitlines() if line.strip()]}))
except Exception:
print('FBA_CORE_JSON='+json.dumps({'ok':False,'error':'REMOTE_QUERY_FAILED'}))
`
const groovy=`def p=new ProcessBuilder(['python3','-c',new String(java.util.Base64.decoder.decode(${base64Expression(python)}),'UTF-8')]).redirectErrorStream(true).start(); def output=p.inputStream.getText('UTF-8'); p.waitFor(); print(output)`
return `def c=jenkins.model.Jenkins.instance.getComputer('${agent}'); if(c==null || c.channel==null) throw new RuntimeException('TEST evidence agent unavailable'); print(hudson.util.RemotingDiagnostics.executeGroovy(new String(java.util.Base64.decoder.decode(${base64Expression(groovy)}),'UTF-8'),c.channel));`
}
export function createTestEvidenceReaders(env, fetcher = globalThis.fetch) {
validateEvidenceConfig(env)
async function execute(service, sql) {
const script=buildEvidenceRemoteScript(service,sql,env.FBA_CORE_JENKINS_AGENT)
let response
try {
response=await fetcher(new URL('scriptText',env.JENKINS_URL.replace(/\/?$/,'/')),{
method:'POST',headers:{Authorization:'Basic '+Buffer.from(env.JENKINS_USER+':'+env.JENKINS_API_TOKEN).toString('base64'),
'Content-Type':'application/x-www-form-urlencoded'},body:new URLSearchParams({script}),signal:AbortSignal.timeout(55000)})
} catch { throw Error('TEST evidence transport failed') }
if (!response.ok) throw Error('TEST evidence HTTP '+response.status)
let raw
try { raw=await response.text() } catch { throw Error('TEST evidence response failed') }
const lines=raw.split(/\r?\n/).filter(line=>line.startsWith('FBA_CORE_JSON='))
if(lines.length!==1) throw Error('TEST evidence JSON missing or ambiguous')
let result
try { result=JSON.parse(lines[0].slice('FBA_CORE_JSON='.length)) } catch { throw Error('TEST evidence JSON invalid') }
if(result?.ok!==true || !Array.isArray(result.rows)) throw Error(/^MYSQL_[0-9]{3,5}$/.test(result?.error)?result.error:'TEST evidence query failed')
return result.rows
}
return {
readReceiving: createReceivingEvidenceReader(sql => execute('lastleg',sql)),
readCompletion: async scope => completionEvidenceFromRows(scope,await execute('order',buildCompletionEvidenceSql(scope))),
preflight: async () => {
for(const service of ['order','lastleg']) {
const rows=await execute(service,[...transactionStart,"SELECT JSON_OBJECT('testProbe',1);",'COMMIT;'].join('\n'))
if(rows.length!==1 || rows[0].testProbe!==1)throw Error('TEST evidence preflight failed')
}
},
}
}
@@ -0,0 +1,60 @@
import test from 'node:test'
import assert from 'node:assert/strict'
import { assertReadOnlySql, buildReceivingEvidenceSql, receivingEvidenceFromRows, buildCompletionEvidenceSql,
completionEvidenceFromRows, buildEvidenceRemoteScript, createTestEvidenceReaders } from './fba-core-regression-evidence.mjs'
const env=()=>({JENKINS_URL:'https://ci.example.invalid/',JENKINS_USER:'offline-ci',JENKINS_API_TOKEN:'offline-api-secret',FBA_CORE_JENKINS_AGENT:'java-test-agent'})
const scope=()=>({customerId:'2105209721220861954',warehouseId:'193000000000200003',bookingId:'2105220190013755394',
stage:'BEFORE_COMPLETE',containerNo:'TEST-ONE',inboundCodes:['IN-1'],sourceInboundIds:['2105220155339444226'],
expectedPackagesByInbound:{'IN-1':1}})
const completion=()=>({customerId:'90',rootOrderId:'100',sourceOrderIds:['100','101'],eventType:'FIRST_LEG_ORDER_FULFILLMENT_COMPLETED'})
const completionRows=()=>['100','101'].map(orderId=>({customerId:'90',rootOrderId:'100',orderId,
eventType:'FIRST_LEG_ORDER_FULFILLMENT_COMPLETED',count:orderId==='100'?1:0,publishedCount:orderId==='100'?1:0,pendingCount:0,failedCount:0}))
test('both queries use consistent read-only transactions and exact scope without losing long identifiers',()=>{
const sql=buildReceivingEvidenceSql(scope())
assert.equal(buildReceivingEvidenceSql({...scope(),stage:'AFTER_EVENT_RECOVERY'}),sql)
assert.throws(()=>buildReceivingEvidenceSql({...scope(),stage:'UNKNOWN_RECOVERY'}))
assert.equal(assertReadOnlySql(sql),sql)
assert.match(sql,/START TRANSACTION WITH CONSISTENT SNAPSHOT, READ ONLY/)
assert.ok(sql.includes(Buffer.from(scope().customerId).toString('hex')))
assert.doesNotMatch(sql,/customer_id=2105209721220861954/)
assert.match(sql,/CAST\(b.id AS CHAR\)/)
assert.match(assertReadOnlySql(buildCompletionEvidenceSql(completion())),/LEFT JOIN ord_outbox_event/)
})
test('read-only guard rejects mutation, multiple statements, lock or unsafe select effects',()=>{
const sql=buildCompletionEvidenceSql(completion())
for(const statement of ['DELETE FROM ord_order;','SELECT 1 INTO OUTFILE \'x\';','SELECT SLEEP(1);',
'SELECT 1; DROP TABLE x;','SET SESSION autocommit=1;'])assert.throws(()=>assertReadOnlySql(sql.replace(/SELECT .+;/,statement)))
assert.throws(()=>assertReadOnlySql('SELECT 1;'))
assert.throws(()=>buildEvidenceRemoteScript('prod',sql,'test'))
})
test('invalid and injected IDs fail before transport; cancelled child zero evidence cannot disappear',()=>{
assert.throws(()=>buildReceivingEvidenceSql({...scope(),bookingId:'1 OR 1=1'}))
assert.throws(()=>buildCompletionEvidenceSql({...completion(),sourceOrderIds:['100','100']}))
assert.equal(completionEvidenceFromRows(completion(),completionRows()).orders[1].count,0)
for(const mutate of [r=>r.pop(),r=>r[1].customerId='91',r=>r[1].count=null,r=>r[1].orderId='100']) {
const rows=completionRows();mutate(rows);assert.throws(()=>completionEvidenceFromRows(completion(),rows))
}
})
test('receiving rows require exact source and session; empty or foreign scope fails',()=>{
assert.throws(()=>receivingEvidenceFromRows(scope(),[]))
const s=scope(),rows=[{collection:'sources',row:{id:'1',customerId:s.customerId,overseasWarehouseId:s.warehouseId,
sourceContainerBookingId:s.bookingId,containerNo:s.containerNo,sourceInboundId:s.sourceInboundIds[0],inboundCode:'IN-1',sourceRegion:'FIRSTLEG',packageCount:1}},
{collection:'sessions',row:{id:'2',sourceContainerBookingId:s.bookingId,containerNo:s.containerNo,overseasWarehouseId:s.warehouseId}}]
assert.deepEqual(receivingEvidenceFromRows(s,rows).balances,[])
rows[0].row.customerId='99';assert.throws(()=>receivingEvidenceFromRows(s,rows))
})
test('transport frames valid empty rows and suppresses raw remote and connection secrets',async()=>{
let calls=0
const transport=createTestEvidenceReaders(env(),async()=>{calls++;return {ok:true,text:async()=> 'FBA_CORE_JSON='+JSON.stringify({ok:true,rows:[{testProbe:1}]})}})
await transport.preflight();assert.equal(calls,2)
for(const output of ['secret-leak','FBA_CORE_JSON={bad','FBA_CORE_JSON={"ok":false,"error":"password=secret-leak"}']) {
const reader=createTestEvidenceReaders(env(),async()=>({ok:true,text:async()=>output}))
await assert.rejects(reader.preflight(),error=>!error.message.includes('secret-leak'))
}
const reader=createTestEvidenceReaders(env(),async()=>{throw Error('offline-api-secret')})
await assert.rejects(reader.preflight(),/TEST evidence transport failed/)
const script=buildEvidenceRemoteScript('order',buildCompletionEvidenceSql(completion()),'java-test-agent')
assert.equal(script.includes(env().JENKINS_API_TOKEN),false)
})
@@ -2,10 +2,69 @@ import test from 'node:test'
import assert from 'node:assert/strict'
import { readFileSync } from 'node:fs'
import { resolve } from 'node:path'
import { AcceptanceRun, parseArguments, validateExecutionOptions } from './admin-test-env-acceptance.mjs'
import { AcceptanceRun, configureReadOnlyEvidence, unloadRecoveryWasVerified, parseArguments, validateExecutionOptions } from './admin-test-env-acceptance.mjs'
const source = readFileSync(resolve('scripts/admin-test-env-acceptance.mjs'), 'utf8')
test('full CLI evidence preflight attaches readers and fails before business writes when unavailable', async () => {
const run = { options: { modules: ['manual-state-matrix'] }, context: {} }
const readReceiving = () => {}, readCompletion = () => {}
let checked = false
await configureReadOnlyEvidence(run, {}, () => ({ readReceiving, readCompletion,
preflight: async () => { checked = true } }))
assert.equal(checked, true)
assert.equal(run.context.phase4ReceivingEvidenceReader, readReceiving)
assert.equal(run.context.f40ReadOnlyOutboxEvidence, readCompletion)
const unavailable = { options: run.options, context: {} }
await assert.rejects(configureReadOnlyEvidence(unavailable, {}, () => ({
preflight: async () => { throw Error('evidence unavailable') } })), /unavailable/)
assert.deepEqual(unavailable.context, {})
})
test('explicit unloading fault uses a separate first-container hook and restores both directions on failure', async () => {
const calls = []
const run = { marker: 'TEST-UNLOAD', options: { modules: ['phase4-overseas-pda', 'phase4-cross-region-events'],
verifyUnloadEventRecovery: true }, context: {} }
const fault = name => ({ prepare: async scope => calls.push([name, 'prepare', scope]),
readProof: async scope => ({ name, scope, restored: true }),
cleanup: async () => { calls.push([name, 'cleanup']); if (name === 'reverse') throw Error('recovery failed') } })
const cleanup = await configureReadOnlyEvidence(run, {}, () => ({ readReceiving() {}, readCompletion() {}, preflight: async () => {} }),
{ createTestFaultExecutor: () => fault('forward'), createTestUnloadFaultExecutor: () => fault('reverse') })
assert.equal(calls.length, 0)
await run.context.beforeReceivingComplete({ customerId: '7', bookingId: '66' })
await run.context.beforeReceivingComplete({ customerId: '8', bookingId: '77' })
assert.deepEqual(calls, [['reverse', 'prepare', { marker: 'TEST-UNLOAD', customerId: '7', containerBookingId: '66' }]])
assert.equal(await run.context.afterReceivingComplete({ customerId: '8', bookingId: '77' }), null)
const proof = await run.context.afterReceivingComplete({ customerId: '7', bookingId: '66' })
assert.equal(proof.name, 'reverse')
await assert.rejects(cleanup.cleanup(), /恢复失败/)
assert.deepEqual(calls.slice(-2), [['reverse', 'cleanup'], ['forward', 'cleanup']])
})
test('unloading recovery is opt-in and cannot run against unrelated modules', () => {
const args = ['--modules', 'phase4-overseas-return-followups', '--gateway', 'http://localhost:9000',
'--username', 'contract', '--password', 'contract']
assert.equal(parseArguments(args).verifyUnloadEventRecovery, false)
assert.throws(() => validateExecutionOptions(parseArguments([...args, '--verify-unload-event-recovery', 'true'])), /收货链/)
})
test('requested unloading fault cannot pass without exact current-run recovery and inventory proof', () => {
const run = { marker: 'CURRENT', createdResources: {} }
assert.equal(unloadRecoveryWasVerified(run), false)
const row = { status: 'VERIFIED', eventRecoveryInventoryUnchanged: true,
eventRecovery: { restored: true, marker: 'CURRENT', containerBookingId: '66' } }
run.createdResources.phase4OverseasPda = { receivingCompletionEvidence: { '66': row } }
assert.equal(unloadRecoveryWasVerified(run), true)
for (const change of [r => r.eventRecoveryInventoryUnchanged = false, r => r.status = 'UNPROVEN',
r => r.eventRecovery.restored = false, r => r.eventRecovery.marker = 'OLD', r => r.eventRecovery.containerBookingId = '77']) {
const changed = structuredClone(run)
change(changed.createdResources.phase4OverseasPda.receivingCompletionEvidence['66'])
assert.equal(unloadRecoveryWasVerified(changed), false)
}
run.createdResources.phase4OverseasPda.receivingCompletionEvidence['77'] = structuredClone(row)
assert.equal(unloadRecoveryWasVerified(run), false)
})
test('F40 optional chain always creates its customer and rejects absent domestic PDA credentials', () => {
const options = parseArguments(['--modules', 'phase5-root-fulfillment',
'--gateway', 'http://localhost:9000', '--username', 'contract-user', '--password', 'contract-password',
+157
View File
@@ -0,0 +1,157 @@
// Internal single TEST fault lifecycle. Uses the existing ACK-loss proxy and
// authenticated original-DEAD retry endpoint, never direct business SQL writes.
import { readFileSync } from 'node:fs'
import { createHash } from 'node:crypto'
import { isDeepStrictEqual } from 'node:util'
import { validateEvidenceConfig } from './fba-core-regression-evidence.mjs'
const runtime = readFileSync(new URL('./test-event-fault-runtime.py', import.meta.url), 'utf8')
const proxySource = readFileSync(new URL('./event-delivery-fault-proxy.py', import.meta.url)).toString('base64')
const encode = value => Buffer.from(value).toString('base64')
const expression = value => '[' + encode(value).match(/.{1,4000}/g).map(part => `'${part}'`).join(',') + "].join('')"
export function renderTestFaultAction(action, scope, agent) {
return renderAction(action,scope,agent,'LOADING')
}
export function renderTestUnloadFaultAction(action, scope, agent) {
return renderAction(action,scope,agent,'UNLOAD')
}
function renderAction(action,scope,agent,direction) {
const keys=direction==='UNLOAD'?['marker','customerId','containerBookingId']:['marker','customerId','loadingPlanId','containerBookingId']
if (!['prepare', 'activate', 'proxy-ready', 'health', 'status', 'recover', 'proof', 'restore', 'finish'].includes(action)
|| !/^[A-Za-z0-9_.-]{1,128}$/.test(agent) || !/^[A-Za-z0-9_-]{1,96}$/.test(scope?.marker || '')
|| keys.filter(key=>key!=='marker').some(key => !/^[1-9][0-9]{0,19}$/.test(scope?.[key] || ''))
|| (direction==='UNLOAD'&&Object.keys(scope).some(key=>!keys.includes(key)))) {
throw Error('Invalid TEST fault action or scope')
}
const config = encode(JSON.stringify({ action, scope, direction, ...(action === 'prepare' ? { proxySource } : {}) }))
const groovy = `def p=new ProcessBuilder(['python3','-c',new String(java.util.Base64.decoder.decode(${expression(runtime)}),'UTF-8'),'${config}']).redirectErrorStream(true).start();def output=p.inputStream.getText('UTF-8');p.waitFor();print(output)`
return `def c=jenkins.model.Jenkins.instance.getComputer('${agent}');if(c==null || c.channel==null)throw new RuntimeException('TEST fault agent unavailable');print(hudson.util.RemotingDiagnostics.executeGroovy(new String(java.util.Base64.decoder.decode(${expression(groovy)}),'UTF-8'),c.channel))`
}
export function createTestFaultExecutor(env, fetcher = globalThis.fetch) {
return createExecutor(env,fetcher,'LOADING')
}
export function createTestUnloadFaultExecutor(env, fetcher = globalThis.fetch) {
return createExecutor(env,fetcher,'UNLOAD')
}
const canonical=value=>Array.isArray(value)?value.map(canonical):value&&typeof value==='object'
?Object.fromEntries(Object.keys(value).sort().map(key=>[key,canonical(value[key])])):value
export function mysqlDatetimeMillisFromInstant(value) {
const match=typeof value==='string'&&value.match(/^(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2})(?:\.(\d{1,9}))?Z$/)
if(!match)return NaN
const seconds=Date.parse(match[1]+'Z')
const nanos=Number((match[2]||'').padEnd(9,'0'))
// DATETIME(3) rounds fractional seconds; Date.parse alone truncates them.
return seconds+Math.floor((nanos+500000)/1000000)
}
export function assertTestUnloadFaultProof(proof,scope) {
const require=(passed,code)=>{if(!passed)throw Error('Invalid unload fault proof: '+code)}
require(proof?.readOnly===true&&Object.keys(scope).every(key=>proof[key]===scope[key]),'SCOPE')
const before=proof.beforeRetry,after=proof.afterRetry,row=proof.producer,env=row?.envelope
require(row&&env&&/^[1-9][0-9]{0,19}$/.test(row.entryId)&&/^(?:[0-9a-fA-F-]{36}|[0-9A-HJKMNP-TV-Z]{26})$/.test(row.eventId),'IDENTITY')
require(row.eventType==='OverseasContainerUnloaded'&&row.customerId==='0'&&row.aggregateId===scope.containerBookingId
&&row.eventKey===`OverseasContainerUnloaded:${scope.containerBookingId}`&&row.aggregateType==='CONTAINER_BOOKING','PRODUCER')
require(env.eventId===row.eventId&&env.eventType===row.eventType&&Number(env.eventVersion)===1&&env.tenantId==='0'
&&env.aggregateId===scope.containerBookingId&&env.aggregateType==='CONTAINER_BOOKING'
&&env.payload?.sourceContainerBookingId===scope.containerBookingId&&Boolean(env.payload.containerNo),'ENVELOPE')
const hash='sha256:'+createHash('sha256').update(JSON.stringify(canonical(env.payload))).digest('hex')
require(row.payloadHash===hash&&env.payloadHash===hash,'HASH')
require(before?.publishStatus==='DEAD'&&before.failedAttempts===12&&row.publishStatus==='PUBLISHED'
&&before.entryId===row.entryId&&before.eventId===row.eventId&&before.payloadHash===hash
&&isDeepStrictEqual(before.envelope,env)&&after?.publishStatus==='PUBLISHED'&&after.entryId===row.entryId
&&after.eventId===row.eventId&&after.payloadHash===hash&&isDeepStrictEqual(after.envelope,env),'ORIGINAL_RETRY')
const inbox=proof.consumer
require(inbox?.count===1&&inbox.status==='PROCESSED'&&inbox.eventId===row.eventId&&inbox.payloadHash===hash
&&inbox.eventType===row.eventType&&inbox.aggregateId===scope.containerBookingId&&isDeepStrictEqual(inbox.payload,env.payload),'INBOX')
const attempts=proof.attempts||[],proxy=proof.proxyRecords||[]
require(attempts.length===13&&new Set(attempts.map(a=>a.attemptId)).size===13&&attempts.every(a=>a.eventId===row.eventId&&a.entryId===row.entryId&&a.serviceName==='lastleg')
&&attempts.slice(0,11).every(a=>a.outcome==='RETRY')&&attempts[11].outcome==='DEAD'&&attempts[12].outcome==='DUPLICATE','ATTEMPTS')
const matches=proxy.filter(p=>p.matched===true)
require(matches.length===13&&matches.every((p,i)=>p.eventId===row.eventId&&p.payloadHash===hash&&p.attempt===i+1)
&&matches.slice(0,12).every((p,i)=>p.errorCode==='ACK_LOSS_INJECTED'&&p.outcome===(i===0?'APPLIED':'DUPLICATE'))
&&matches[12].outcome==='DUPLICATE'&&matches[12].errorCode==null,'REAL_ACK_LOSS')
const audit=proof.retryAudit
require(audit&&proof.retryAuditCount===1&&audit.requestId===proof.retryRequestId&&audit.entryId===row.entryId
&&audit.eventId===row.eventId&&audit.payloadHash===hash&&audit.previousFailures===12,'RETRY_AUDIT')
const container=before.container
require(container&&container.id===scope.containerBookingId&&container.containerNo===env.payload.containerNo
&&isDeepStrictEqual(container,after.container)&&container.version!=null&&container.emptyReportSourceVersion!=null,'CONTAINER_UNCHANGED')
const time=value=>Date.parse(String(value).replace(' ','T').replace(/Z$/,'')+'Z')
require(time(container.unloadingCompletedAt)===mysqlDatetimeMillisFromInstant(env.payload.unloadingCompletedAt)
&&time(container.unloadingPhotoUploadedAt)===mysqlDatetimeMillisFromInstant(env.payload.unloadingPhotoUploadedAt)
&&time(container.estimatedEmptyReportedAt)-time(container.unloadingCompletedAt)===72*3600000
&&(env.payload.actualEmptyReportedAt==null?container.actualEmptyReportedAt===null:time(container.actualEmptyReportedAt)===Date.parse(env.payload.actualEmptyReportedAt))
&&String(container.emptyReportSourceVersion)===String(env.aggregateVersion),'CONTAINER_FACTS')
return proof
}
function createExecutor(env,fetcher,direction) {
validateEvidenceConfig(env)
if (env.CHONGCHONG_ENV !== 'test') throw Error('TEST fault requires CHONGCHONG_ENV=test')
let scope = null, finished = false, preparedEvidence=null, restoration=null
async function action(name) {
const script = renderAction(name, scope, env.FBA_CORE_JENKINS_AGENT,direction)
const response = await fetcher(new URL('scriptText', env.JENKINS_URL.replace(/\/?$/, '/')), {
method: 'POST', redirect: 'error', headers: { Authorization: 'Basic ' + encode(env.JENKINS_USER + ':' + env.JENKINS_API_TOKEN),
'Content-Type': 'application/x-www-form-urlencoded' }, body: new URLSearchParams({ script }), signal: AbortSignal.timeout(55000) })
if (!response.ok) throw Error('TEST fault HTTP ' + response.status)
const rows = (await response.text()).split(/\r?\n/).filter(row => row.startsWith('TEST_EVENT_JSON='))
if (rows.length !== 1) throw Error('TEST fault evidence missing or ambiguous')
const result = JSON.parse(rows[0].slice('TEST_EVENT_JSON='.length))
if (!result?.ok) throw Error('TEST fault action failed: ' + (/^[A-Z_0-9]{1,80}$/.test(result?.error) ? result.error : 'UNKNOWN'))
return result.data
}
async function wait(name, check, attempts = 120, tolerateStartup = false) {
for (let index = 0; index < attempts; index++) {
let value
try { value = await action(name) } catch (error) {
if (!tolerateStartup || !String(error.message).endsWith('TEST_EVENT_ACTION_FAILED')) throw error
}
if (value && check(value)) return value
await new Promise(resolve => setTimeout(resolve, 1000))
}
throw Error('TEST fault observation timed out: ' + name)
}
async function cleanup() {
if (!scope || finished) return restoration
await action('restore')
await wait('health', row => row.healthy)
restoration=await action('finish')
if(direction==='UNLOAD'&&!(/^sha256:[0-9a-f]{64}$/.test(preparedEvidence?.originalImageId||'')
&&/^[0-9a-f]{64}$/.test(preparedEvidence?.originalFingerprint||'')
&&restoration?.restored===true&&restoration.healthy===true&&restoration.proxyStopped===true
&&restoration.producerName==='firstleg-test-lastleg'&&restoration.originalImageId===preparedEvidence?.originalImageId
&&restoration.currentImageId===preparedEvidence?.originalImageId&&restoration.originalFingerprint===preparedEvidence?.originalFingerprint
&&restoration.currentFingerprint===preparedEvidence?.originalFingerprint))throw Error('Unload restoration evidence mismatch')
finished = true
return restoration
}
return {
async prepare(next) {
if (scope) {
if(direction==='UNLOAD'&&!isDeepStrictEqual(scope,next))throw Error('TEST unload fault scope drift')
return // Pin only the first actual current-run plan; other events pass through.
}
if(direction==='UNLOAD')renderAction('prepare',next,env.FBA_CORE_JENKINS_AGENT,direction)
scope = direction==='UNLOAD'?{...next}:{ marker: next.marker, customerId: next.customerId,
loadingPlanId: next.loadingPlanId, containerBookingId: next.containerBookingId }
preparedEvidence=await action('prepare')
await wait('proxy-ready', row => row.ready, 10, true)
await action('activate')
await wait('health', row => row.healthy)
},
async readProof(expected) {
if (!scope || (direction==='UNLOAD'?!isDeepStrictEqual(scope,expected):Object.keys(scope).some(key => expected?.[key] !== scope[key]))) throw Error('TEST fault scope drift')
try {
await wait('status', row => row.producer?.publishStatus === 'DEAD' && row.producer.failedAttempts === 12, 180)
await action('recover')
await wait('status', row => row.producer?.publishStatus === 'PUBLISHED', 90)
const proof = await action('proof')
if(direction==='UNLOAD')assertTestUnloadFaultProof(proof,scope)
await cleanup()
return { ...proof, restored: true, ...(direction==='UNLOAD'?{restoration}:{}) }
} finally { if(direction==='UNLOAD')await cleanup() }
},
cleanup,
}
}
+221
View File
@@ -0,0 +1,221 @@
import assert from 'node:assert/strict'
import test from 'node:test'
import { createHash } from 'node:crypto'
import { spawnSync } from 'node:child_process'
import { createTestFaultExecutor, createTestUnloadFaultExecutor, assertTestUnloadFaultProof, renderTestFaultAction } from './test-event-fault-executor.mjs'
const scope = { marker: 'TEST-ROOT', customerId: '7', loadingPlanId: '99', containerBookingId: '66' }
const environment = { CHONGCHONG_ENV: 'test', JENKINS_URL: 'https://jenkins.example.invalid/',
JENKINS_USER: 'offline', JENKINS_API_TOKEN: 'offline-secret', FBA_CORE_JENKINS_AGENT: 'java-test-agent' }
function requestedAction(options) {
const script = options.body.get('script')
const chunks = [...script.matchAll(/'([A-Za-z0-9+/=]+)'/g)].map(match => match[1]).join('')
const groovy = Buffer.from(chunks, 'base64').toString('utf8')
const argument = groovy.match(/'UTF-8'\),'([A-Za-z0-9+/=]+)'\]\)/)?.[1]
return JSON.parse(Buffer.from(argument, 'base64').toString('utf8'))
}
function transport(calls, failActivate = false) {
let recovered = false
return async (_url, options) => {
assert.equal(options.redirect, 'error')
const { action } = requestedAction(options)
calls.push(action)
if (action === 'recover') recovered = true
const data = action === 'proxy-ready' ? { ready: true } : action === 'health' ? { healthy: true }
: action === 'status' ? { producer: { publishStatus: recovered ? 'PUBLISHED' : 'DEAD', failedAttempts: 12 } }
: action === 'proof' ? { ...scope, restored: false } : {}
const result = action === 'activate' && failActivate ? { ok: false, error: 'DOCKER_409' } : { ok: true, data }
return { ok: true, text: async () => 'TEST_EVENT_JSON=' + JSON.stringify(result) }
}
}
test('scope and actions are bounded and renderer never embeds caller credentials', () => {
const rendered = renderTestFaultAction('status', scope, environment.FBA_CORE_JENKINS_AGENT)
assert.ok(!rendered.includes(environment.JENKINS_API_TOKEN))
for (const change of [{ ...scope, customerId: '7;DROP' }, { ...scope, marker: '../other' }]) {
assert.throws(() => renderTestFaultAction('status', change, environment.FBA_CORE_JENKINS_AGENT))
}
assert.throws(() => renderTestFaultAction('arbitrary', scope, 'java-test-agent'))
assert.throws(() => createTestFaultExecutor({ ...environment, CHONGCHONG_ENV: 'prod' }))
})
const unloadScope = { marker: 'TEST-ROOT', customerId: '7', containerBookingId: '66' }
const originalImageId = 'sha256:' + 'a'.repeat(64), originalFingerprint = 'b'.repeat(64)
function unloadProof(nanoseconds = false) {
const payload = { actualEmptyReportedAt: null, containerNo: 'OFFLINE66', sourceContainerBookingId: '66',
unloadingCompletedAt: nanoseconds ? '2026-10-02T01:00:00.136827515Z' : '2026-10-02T01:00:00Z', unloadingPhotoUploadedAt: '2026-10-02T00:59:00Z' }
const payloadHash = 'sha256:' + createHash('sha256').update(JSON.stringify(payload)).digest('hex')
const eventId = '00000000-0000-4000-8000-000000000001', entryId = '123'
const envelope = { eventId, eventType: 'OverseasContainerUnloaded', eventVersion: 1, tenantId: '0',
aggregateId: '66', aggregateType: 'CONTAINER_BOOKING', aggregateVersion: 3, payloadHash, payload }
const row = { entryId, eventId, payloadHash, envelope, eventType: envelope.eventType, customerId: '0',
aggregateId: '66', aggregateType: 'CONTAINER_BOOKING', eventKey: 'OverseasContainerUnloaded:66', publishStatus: 'PUBLISHED' }
const container = { id: '66', containerNo: 'OFFLINE66', version: '9', emptyReportSourceVersion: '3',
unloadingCompletedAt: nanoseconds ? '2026-10-02 01:00:00.137' : '2026-10-02 01:00:00', unloadingPhotoUploadedAt: '2026-10-02 00:59:00',
estimatedEmptyReportedAt: nanoseconds ? '2026-10-05 01:00:00.137' : '2026-10-05 01:00:00', actualEmptyReportedAt: null }
return JSON.parse(JSON.stringify({ ...unloadScope, readOnly: true, producer: row,
beforeRetry: { ...row, publishStatus: 'DEAD', failedAttempts: 12, container }, afterRetry: { ...row, container },
consumer: { count: 1, status: 'PROCESSED', eventId, payloadHash, eventType: envelope.eventType, aggregateId: '66', payload },
attempts: Array.from({ length: 13 }, (_, i) => ({ attemptId: `attempt-${i}`, entryId, eventId, serviceName: 'lastleg', outcome: i < 11 ? 'RETRY' : i === 11 ? 'DEAD' : 'DUPLICATE' })),
proxyRecords: Array.from({ length: 13 }, (_, i) => ({ eventId, payloadHash, attempt: i + 1, matched: true,
outcome: i ? 'DUPLICATE' : 'APPLIED', errorCode: i < 12 ? 'ACK_LOSS_INJECTED' : null })),
retryRequestId: '00000000-0000-4000-8000-000000000002', retryAuditCount: 1,
retryAudit: { requestId: '00000000-0000-4000-8000-000000000002', entryId, eventId, payloadHash, previousFailures: 12 } }))
}
function unloadTransport(calls, proof = unloadProof(), finish = {}) {
let recovered = false
return async (_url, options) => {
const config = requestedAction(options), { action } = config
assert.equal(config.direction, 'UNLOAD')
assert.deepEqual(config.scope, unloadScope)
calls.push(action)
if (action === 'recover') recovered = true
const data = action === 'prepare' ? { originalImageId, originalFingerprint }
: action === 'proxy-ready' ? { ready: true } : action === 'health' ? { healthy: true }
: action === 'status' ? { producer: { publishStatus: recovered ? 'PUBLISHED' : 'DEAD', failedAttempts: 12 } }
: action === 'proof' ? proof : action === 'finish' ? { restored: true, healthy: true, proxyStopped: true,
producerName: 'firstleg-test-lastleg', originalImageId, currentImageId: originalImageId,
originalFingerprint, currentFingerprint: originalFingerprint, ...finish } : {}
return { ok: true, text: async () => 'TEST_EVENT_JSON=' + JSON.stringify({ ok: true, data }) }
}
}
test('unload factory pins exactly three scope fields and verifies restoration before success', async () => {
const calls = [], executor = createTestUnloadFaultExecutor(environment, unloadTransport(calls))
await assert.rejects(executor.prepare({ ...unloadScope, eventType: 'Other' }), /Invalid/)
await executor.prepare(unloadScope)
await assert.rejects(executor.prepare({ ...unloadScope, containerBookingId: '67' }), /scope drift/)
await assert.rejects(executor.readProof({ ...unloadScope, eventType: 'Other' }), /scope drift/)
const proof = await executor.readProof(unloadScope)
assert.equal(proof.restored, true)
assert.equal(proof.restoration.currentImageId, originalImageId)
await executor.cleanup()
assert.deepEqual(calls, ['prepare', 'proxy-ready', 'activate', 'health', 'status', 'recover', 'status', 'proof', 'restore', 'health', 'finish'])
})
test('unload proof rejects changed original event, fabricated ACK, nonunique audit and changed business facts', () => {
assertTestUnloadFaultProof(unloadProof(), unloadScope)
for (const mutate of [
p => { p.producer.customerId = '7' }, p => { p.producer.eventKey = 'other' },
p => { p.producer.envelope.tenantId = '7' }, p => { p.producer.envelope.payload.containerNo = 'CHANGED' },
p => { p.beforeRetry.entryId = '124' }, p => { p.beforeRetry.envelope.aggregateVersion = 4 },
p => { p.beforeRetry.failedAttempts = 11 }, p => { p.afterRetry.eventId = 'OTHER' },
p => { p.consumer.count = 2 }, p => { p.consumer.status = 'RECEIVED' }, p => { p.consumer.payload = {} },
p => { p.attempts.pop() }, p => { p.attempts[0].serviceName = 'firstleg' },
p => { p.proxyRecords[0].outcome = 'DUPLICATE' }, p => { p.proxyRecords[0].errorCode = null },
p => { p.proxyRecords[12].attempt = 1 }, p => { p.retryAuditCount = 2 },
p => { p.retryAudit.requestId = 'OTHER' }, p => { p.afterRetry.container.version = '10' },
p => { p.beforeRetry.container.estimatedEmptyReportedAt = p.afterRetry.container.estimatedEmptyReportedAt = '2026-10-04 01:00:00' },
]) {
const proof = unloadProof(); mutate(proof)
assert.throws(() => assertTestUnloadFaultProof(proof, unloadScope), /Invalid unload fault proof/)
}
})
test('unload timestamps follow DATETIME(3) rounding rather than allowing time drift', () => {
const proof=unloadProof(true)
assertTestUnloadFaultProof(proof,unloadScope)
for(const row of [proof.beforeRetry,proof.afterRetry]) {
row.container.unloadingCompletedAt='2026-10-02 01:00:00.138'
row.container.estimatedEmptyReportedAt='2026-10-05 01:00:00.138'
}
assert.throws(()=>assertTestUnloadFaultProof(proof,unloadScope),/CONTAINER_FACTS/)
})
test('invalid unload proof still restores and incomplete restoration never becomes success', async () => {
const invalid = unloadProof(); invalid.consumer.count = 0
const calls = [], executor = createTestUnloadFaultExecutor(environment, unloadTransport(calls, invalid))
await executor.prepare(unloadScope)
await assert.rejects(executor.readProof(unloadScope), /INBOX/)
assert.deepEqual(calls.slice(-3), ['restore', 'health', 'finish'])
for (const change of [{ restored: false }, { healthy: false }, { proxyStopped: false },
{ currentImageId: 'sha256:' + 'c'.repeat(64) }, { currentFingerprint: 'c'.repeat(64) }]) {
const failed = createTestUnloadFaultExecutor(environment, unloadTransport([], unloadProof(), change))
await failed.prepare(unloadScope)
await assert.rejects(failed.readProof(unloadScope), /restoration evidence mismatch/)
}
})
test('Python reverse SQL routing, scope isolation and full configuration recovery fail closed offline', () => {
const result = spawnSync('python', ['-c', String.raw`
import ast, copy, importlib.util, io, pathlib, sys
from unittest.mock import patch
sys.dont_write_bytecode = True
path = pathlib.Path('scripts/test-event-fault-runtime.py')
ast.parse(path.read_text(encoding='utf-8'))
spec = importlib.util.spec_from_file_location('fault_runtime', path)
m = importlib.util.module_from_spec(spec); spec.loader.exec_module(m)
scope = {'marker':'TEST-ROOT','customerId':'7','containerBookingId':'66'}
assert m.target_setting({'direction':'UNLOAD'}) == ('FIRSTLEG_SERVICE_URL','http://firstleg:9202')
assert m.scope_directory(scope, 'UNLOAD') != m.scope_directory(dict(scope, loadingPlanId='99'))
try: m.scope_directory(dict(scope, eventType='Other'), 'UNLOAD'); assert False
except RuntimeError as error: assert str(error) == 'INVALID_SCOPE'
queries=[]
with patch.object(m, 'inspect', side_effect=lambda name: name), patch.object(m, 'read_sql', side_effect=lambda name, sql: queries.append((name, sql)) or []):
assert m.producer(scope, 'UNLOAD') is None
assert queries[0][0] == 'firstleg-test-lastleg'
assert "customer_id='0'" in queries[0][1] and "event_key='OverseasContainerUnloaded:66'" in queries[0][1]
assert "JSON_EXTRACT(payload_json,'$')" in queries[0][1] and "publishStatus',status" in queries[0][1]
row={'entryId':'123','eventId':'00000000-0000-4000-8000-000000000001','payloadHash':'sha256:'+'a'*64,'publishStatus':'DEAD','failedAttempts':12,'envelope':{'eventType':'OverseasContainerUnloaded'}}
retry_state={'direction':'UNLOAD','operationsRequestId':'00000000-0000-4000-8000-000000000002'}
retry_container={'Config':{'Env':['EVENT_OPERATIONS_TOKEN=offline']},'NetworkSettings':{'Networks':{'firstleg-test-network':{'IPAddress':'172.20.0.3'}}}}
with patch.object(m,'producer',return_value=row),patch.object(m,'control',return_value=row),patch.object(m,'container_facts',return_value={'version':'9'}),patch.object(m,'business_ids',side_effect=AssertionError('reverse must not query inventory or orders')),patch.object(m,'private_write'),patch.object(m,'inspect',return_value=retry_container),patch.object(m.urllib.request,'urlopen',return_value=io.BytesIO(b'{"code":"200"}')) as http:
assert m.recover(scope,pathlib.Path('/offline'),retry_state)['retrySubmitted']
assert http.call_args.args[0].full_url == 'http://172.20.0.3:9303/internal/lastleg/event-operations/123/retry'
assert retry_state['beforeRetry']['container']=={'version':'9'}
assert retry_state['beforeRetry']['envelope']==row['envelope']
row['publishStatus']='PUBLISHED'
queries=[]
with patch.object(m,'producer',return_value=row),patch.object(m,'control',return_value={'records':[]}),patch.object(m,'container_facts',return_value={'version':'9'}),patch.object(m,'business_ids',side_effect=AssertionError('reverse must not query inventory or orders')),patch.object(m,'inspect',side_effect=lambda name:name),patch.object(m,'read_sql',side_effect=lambda name,sql: queries.append((name,sql)) or []),patch.object(m,'private_write') as save:
proof=m.proof(scope,retry_state)
save.assert_called_once_with(m.scope_directory(scope,'UNLOAD')/'proof.json',proof)
assert proof['retryAuditCount']==0 and proof['retryRequestId']==retry_state['operationsRequestId']
assert [item[0] for item in queries]==['firstleg-test-lastleg','firstleg-test-lastleg','firstleg-test-business']
assert "WHERE entry_id='123' AND event_type='OverseasContainerUnloaded'" in queries[1][1]
assert 'fl_container_event_inbox' in queries[2][1]
container={'Name':'/firstleg-test-lastleg','Id':'container123456789','Image':'sha256:'+'a'*64,
'Config':{'Image':'lastleg:test','Env':['FIRSTLEG_SERVICE_URL=http://firstleg:9202','A=1'],'Labels':{'com.docker.compose.project':'firstleg-test'}},
'HostConfig':{'Binds':['source:/data:rw'],'PortBindings':{'9303/tcp':[{'HostPort':'9303'}]}},
'Mounts':[{'Type':'bind','Source':'/source','Destination':'/data','RW':True}],
'NetworkSettings':{'Networks':{'firstleg-test-network':{'IPAddress':'172.20.0.3','Aliases':['lastleg-service','container123']}}},
'State':{'Health':{'Status':'healthy'}}}
fingerprint=m.fingerprint(container)
multi=copy.deepcopy(container)
multi['NetworkSettings']['Networks']['firstleg-test-network']['MacAddress']='02:42:ac:15:00:04'
multi['NetworkSettings']['Networks']['second-test-network']={'IPAddress':'172.22.0.7','MacAddress':'02:42:ac:16:00:07','Aliases':['lastleg']}
legacy=copy.deepcopy(multi); legacy['Config']['MacAddress']='02:42:ac:16:00:07'
assert m.fingerprint(legacy)==m.fingerprint(multi)
changed=copy.deepcopy(legacy); changed['Config']['MacAddress']='02:42:ac:99:00:01'
assert m.fingerprint(changed)!=m.fingerprint(multi)
changed=copy.deepcopy(legacy); changed['NetworkSettings']['Networks']['second-test-network']['MacAddress']='02:42:ac:16:00:08'
assert m.fingerprint(changed)!=m.fingerprint(multi)
state={'direction':'UNLOAD','restored':False,'image':container['Config']['Image'],'originalImageId':container['Image'],
'originalFingerprint':fingerprint,'originalBody':m.recreation(container,m.env_of(container)),'proxyPid':123,'controlToken':'offline'}
for mutate in [lambda c:c['HostConfig'].update(Binds=['other:/data:rw']),lambda c:c['NetworkSettings']['Networks']['firstleg-test-network'].update(IPAddress='172.20.0.4'),lambda c:c.update(Image='sha256:'+'b'*64),lambda c:c['State']['Health'].update(Status='unhealthy')]:
changed=copy.deepcopy(container); mutate(changed)
with patch.object(m,'inspect',return_value=changed), patch.object(m,'private_write') as write:
try: m.finish(pathlib.Path('/offline'),copy.deepcopy(state)); assert False
except RuntimeError as error: assert str(error).startswith('RESTORED_')
write.assert_not_called()
with patch.object(m,'inspect',return_value=container),patch.object(m.Path,'exists',return_value=False),patch.object(m,'private_write'):
restored=m.finish(pathlib.Path('/offline'),copy.deepcopy(state))
assert restored['proxyStopped'] and restored['currentFingerprint']==fingerprint and restored['originalImageId']==restored['currentImageId']
with patch.object(m,'inspect',return_value=container),patch.object(m.Path,'exists',return_value=True),patch.object(m.Path,'read_bytes',return_value=b'python3\x00'+str(pathlib.Path('/offline')/'proxy.py').encode()+b'\x00'),patch.object(m.os,'kill'),patch.object(m.time,'sleep'),patch.object(m,'private_write') as write:
try: m.finish(pathlib.Path('/offline'),copy.deepcopy(state)); assert False
except RuntimeError as error: assert str(error)=='PROXY_NOT_STOPPED'
write.assert_not_called()
`], { encoding: 'utf8', cwd: new URL('..', import.meta.url), timeout: 10000 })
assert.equal(result.status, 0, result.stderr || result.error?.message)
})
test('only original scoped event is recovered and restoration precedes a successful proof', async () => {
const calls = [], executor = createTestFaultExecutor(environment, transport(calls))
await executor.prepare(scope)
await executor.prepare({ ...scope, loadingPlanId: '100' })
const proof = await executor.readProof({ containerBookingId: '66', loadingPlanId: '99', customerId: '7', marker: 'TEST-ROOT' })
assert.equal(proof.restored, true)
await executor.cleanup()
assert.deepEqual(calls, ['prepare', 'proxy-ready', 'activate', 'health', 'status', 'recover',
'status', 'proof', 'restore', 'health', 'finish'])
})
test('failed activation retains task ownership for explicit restoration, never retries activation', async () => {
const calls = [], executor = createTestFaultExecutor(environment, transport(calls, true))
await assert.rejects(executor.prepare(scope), /DOCKER_409/)
await executor.cleanup()
assert.equal(calls.filter(name => name === 'activate').length, 1)
assert.deepEqual(calls.slice(-3), ['restore', 'health', 'finish'])
})
+430
View File
@@ -0,0 +1,430 @@
"""Internal TEST executor. Receives one bounded task action from Jenkins.
Never edits business SQL. Docker configuration is captured before the scoped
fault and restored to the same image, mounts, ports and network identity.
Only the original DEAD event can be retried through its protected HTTP API.
"""
import base64
import copy
import hashlib
import http.client
import json
import os
import re
import secrets
import signal
import socket
import subprocess
import sys
import tempfile
import time
import urllib.parse
import urllib.request
import uuid
from pathlib import Path
class DockerConnection(http.client.HTTPConnection):
def connect(self):
self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
self.sock.settimeout(self.timeout)
self.sock.connect('/var/run/docker.sock')
def docker(method, path, body=None):
conn = DockerConnection('localhost', timeout=20)
data = None if body is None else json.dumps(body).encode()
conn.request(method, path, data, {'Content-Type': 'application/json'})
response = conn.getresponse()
raw = response.read()
conn.close()
if response.status >= 300:
raise RuntimeError('DOCKER_' + str(response.status))
return json.loads(raw) if raw else None
def inspect(name):
return docker('GET', '/containers/' + urllib.parse.quote(name, safe='') + '/json')
def env_of(container):
return dict(item.split('=', 1) for item in container['Config']['Env'])
def require_test(container, name):
if container['Name'] != '/' + name or container['Config']['Labels'].get('com.docker.compose.project') != 'firstleg-test':
raise RuntimeError('NOT_TEST_CONTAINER')
def recreation(container, environment, full=False):
body = copy.deepcopy(container['Config'])
body['Env'] = [key + '=' + value for key, value in environment.items()]
body['HostConfig'] = copy.deepcopy(container['HostConfig'])
if body['HostConfig'].get('AutoRemove') or any(mount['Type'] not in ('bind', 'volume') for mount in container['Mounts']):
raise RuntimeError('UNSUPPORTED_CONTAINER_STORAGE')
# Preserve named volumes explicitly, never create a replacement anonymous volume.
if any(mount['Type'] == 'volume' and not mount.get('Name') for mount in container['Mounts']):
raise RuntimeError('UNNAMED_VOLUME')
if body['HostConfig'].get('VolumesFrom'):
raise RuntimeError('UNSUPPORTED_VOLUME_REFERENCE')
endpoints = {}
for name, network in container['NetworkSettings']['Networks'].items():
aliases = [alias for alias in network.get('Aliases') or [] if alias != container['Id'][:12]]
endpoints[name] = {'Aliases': aliases, 'IPAMConfig': {'IPv4Address': network['IPAddress']}}
if full:
ipam = copy.deepcopy(network.get('IPAMConfig') or {})
ipam['IPv4Address'] = network['IPAddress']
if network.get('GlobalIPv6Address'):
ipam['IPv6Address'] = network['GlobalIPv6Address']
endpoints[name]['IPAMConfig'] = ipam
for key in ('Links', 'DriverOpts', 'GwPriority', 'MacAddress'):
if network.get(key) is not None:
endpoints[name][key] = copy.deepcopy(network[key])
body['NetworkingConfig'] = {'EndpointsConfig': endpoints}
return body
def replace(container, body):
name = container['Name'].lstrip('/')
current = inspect(name)
if current['Id'] != container['Id']:
raise RuntimeError('CONCURRENT_CONTAINER_CHANGE')
docker('POST', '/containers/' + current['Id'] + '/stop?t=10')
docker('DELETE', '/containers/' + current['Id'] + '?v=false')
created = docker('POST', '/containers/create?name=' + name, body)
docker('POST', '/containers/' + created['Id'] + '/start')
return created['Id']
def private_write(path, value):
fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
with os.fdopen(fd, 'w') as file:
json.dump(value, file)
def scope_directory(scope, direction='LOADING'):
if direction not in ('LOADING', 'UNLOAD'):
raise RuntimeError('INVALID_DIRECTION')
marker = scope.get('marker', '')
if not re.fullmatch(r'[A-Za-z0-9_-]{1,96}', marker):
raise RuntimeError('INVALID_SCOPE')
keys = ('customerId', 'containerBookingId') if direction == 'UNLOAD' else ('customerId', 'loadingPlanId', 'containerBookingId')
if direction == 'UNLOAD' and set(scope) != {'marker', *keys}:
raise RuntimeError('INVALID_SCOPE')
for key in keys:
if not re.fullmatch(r'[1-9][0-9]{0,19}', scope.get(key, '')):
raise RuntimeError('INVALID_SCOPE')
identity = 'UNLOAD:' + marker if direction == 'UNLOAD' else marker
return Path(tempfile.gettempdir()) / ('chongchong-event-' + hashlib.sha256(identity.encode()).hexdigest()[:24])
def unload(state):
return state.get('direction', 'LOADING') == 'UNLOAD'
def producer_name(state):
return 'firstleg-test-lastleg' if unload(state) else 'firstleg-test-business'
def target_setting(state):
return ('FIRSTLEG_SERVICE_URL', 'http://firstleg:9202') if unload(state) else ('LASTLEG_SERVICE_URL', 'http://lastleg-service:9303')
def fingerprint(container):
body = recreation(container, env_of(container), full=True)
# Docker may add the legacy Config.MacAddress after explicit per-network MAC
# restoration. It is redundant only when it equals an actual preserved MAC;
# every network MAC remains in the fingerprint and unrelated values still fail.
network_macs = {endpoint.get('MacAddress') for endpoint in body['NetworkingConfig']['EndpointsConfig'].values()}
if body.get('MacAddress') and body['MacAddress'] in network_macs:
body.pop('MacAddress')
body['Env'].sort()
for endpoint in body['NetworkingConfig']['EndpointsConfig'].values():
endpoint['Aliases'].sort()
body['ResolvedMounts'] = sorted(container['Mounts'], key=lambda mount: mount['Destination'])
body['NetworkIds'] = {name: network.get('NetworkID') for name, network in container['NetworkSettings']['Networks'].items()}
return hashlib.sha256(json.dumps(body, sort_keys=True, separators=(',', ':')).encode()).hexdigest()
def control(state, mode=None):
url = 'http://127.0.0.1:' + str(state['controlPort']) + ('/mode' if mode else '/status')
request = urllib.request.Request(url, None if mode is None else json.dumps({'mode': mode}).encode(),
{'X-Event-Fault-Control-Token': state['controlToken'], 'Content-Type': 'application/json'})
with urllib.request.urlopen(request, timeout=5) as response:
return json.load(response)
def read_sql(container, sql):
# SQL is generated here from fixed templates and decimal/UUID/hash identities.
if re.search(r'\b(INSERT|UPDATE|DELETE|ALTER|DROP|TRUNCATE|CALL|INTO|OUTFILE)\b', sql, re.I):
raise RuntimeError('NON_READ_ONLY_SQL')
e = env_of(container)
prefix = 'FIRSTLEG' if container['Name'] == '/firstleg-test-business' else 'LASTLEG'
url = urllib.parse.urlsplit(e[prefix + '_DB_URL'].removeprefix('jdbc:'))
host = '127.0.0.1' if url.hostname == 'host.docker.internal' else url.hostname
transaction = 'SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ; START TRANSACTION WITH CONSISTENT SNAPSHOT, READ ONLY;\n' + sql + '\nCOMMIT;'
result = subprocess.run(['mysql', '--host=' + host, '--port=' + str(url.port or 3306),
'--user=' + e[prefix + '_DB_USERNAME'], '--batch', '--raw', '--skip-column-names', url.path.lstrip('/')],
input=transaction, text=True, capture_output=True, timeout=20,
env=dict(os.environ, MYSQL_PWD=e[prefix + '_DB_PASSWORD']))
if result.returncode:
code = re.search(r'ERROR (\d+)', result.stderr)
raise RuntimeError('MYSQL_' + (code.group(1) if code else 'UNKNOWN'))
return [json.loads(line) for line in result.stdout.splitlines() if line.strip()]
def producer(scope, direction='LOADING'):
if direction == 'UNLOAD':
rows = read_sql(inspect('firstleg-test-lastleg'),
"SELECT JSON_OBJECT('entryId',CAST(id AS CHAR),'eventId',JSON_UNQUOTE(JSON_EXTRACT(payload_json,'$.eventId')),'payloadHash',JSON_UNQUOTE(JSON_EXTRACT(payload_json,'$.payloadHash')),'eventType',event_type,'customerId',customer_id,'aggregateId',aggregate_id,'aggregateType',aggregate_type,'eventKey',event_key,'envelope',JSON_EXTRACT(payload_json,'$'),'publishStatus',status,'failedAttempts',retry_count) FROM ll_event_outbox WHERE customer_id='0' AND aggregate_id='"
+ scope['containerBookingId'] + "' AND event_type='OverseasContainerUnloaded' AND event_key='OverseasContainerUnloaded:" + scope['containerBookingId'] + "';")
else:
rows = read_sql(inspect('firstleg-test-business'),
"SELECT JSON_OBJECT('entryId',CAST(id AS CHAR),'eventId',JSON_UNQUOTE(JSON_EXTRACT(payload_json,'$.eventId')),'payloadHash',payload_hash,'eventType',event_type,'publishStatus',publish_status,'failedAttempts',retry_count) FROM fl_inbound_outbox WHERE tenant_id=" + scope['customerId']
+ " AND aggregate_id=" + scope['loadingPlanId'] + " AND event_type='FirstLegLoadingPlanCompleted';")
if len(rows) > 1:
raise RuntimeError('AMBIGUOUS_EVENT')
return rows[0] if rows else None
def container_facts(scope):
rows = read_sql(inspect('firstleg-test-business'),
"SELECT JSON_OBJECT('id',CAST(id AS CHAR),'containerNo',container_no,'version',CAST(version AS CHAR),'unloadingCompletedAt',CAST(unloading_completed_at AS CHAR),'unloadingPhotoUploadedAt',CAST(unloading_photo_uploaded_at AS CHAR),'estimatedEmptyReportedAt',CAST(estimated_empty_reported_at AS CHAR),'actualEmptyReportedAt',CAST(actual_empty_reported_at AS CHAR),'emptyReportSourceVersion',CAST(empty_report_source_version AS CHAR),'updatedAt',CAST(updated_at AS CHAR),'updatedBy',CAST(updated_by AS CHAR),'updatedByName',updated_by_name) FROM fl_container_booking WHERE id=" + scope['containerBookingId'] + ";")
if len(rows) != 1:
raise RuntimeError('CONTAINER_SCOPE_MISMATCH')
return rows[0]
def business_ids(scope):
rows = read_sql(inspect('firstleg-test-lastleg'), "SELECT JSON_OBJECT('id',CAST(id AS CHAR)) FROM ll_order WHERE customer_id='"
+ scope['customerId'] + "' AND source_container_booking_id='" + scope['containerBookingId'] + "' ORDER BY id;")
return [row['id'] for row in rows]
def prepare(scope, directory, config):
directory.mkdir(mode=0o700)
original = inspect(producer_name(config))
require_test(original, producer_name(config))
consumer_name = 'firstleg-test-business' if unload(config) else 'firstleg-test-lastleg'
consumer = inspect(consumer_name)
require_test(consumer, consumer_name)
old_env = env_of(original)
setting, direct = target_setting(config)
if old_env.get(setting) != direct:
raise RuntimeError('PRODUCER_NOT_DIRECT')
original_body = recreation(original, old_env, full=unload(config))
token = secrets.token_urlsafe(32)
# Reserve task-specific ports and let the existing proxy select exactly this plan.
sockets = [socket.socket() for _ in range(2)]
for sock in sockets:
sock.bind(('127.0.0.1', 0))
ports = [sock.getsockname()[1] for sock in sockets]
for sock in sockets:
sock.close()
proxy = directory / 'proxy.py'
proxy.write_bytes(base64.b64decode(config['proxySource']))
ip = consumer['NetworkSettings']['Networks']['firstleg-test-network']['IPAddress']
log = open(directory / 'proxy-output.log', 'wb')
upstream = ':9202/internal/firstleg/v1/container-events' if unload(config) else ':9303/internal/lastleg/events'
selector = ['--event-type', 'OverseasContainerUnloaded', '--container-booking-id', scope['containerBookingId']] if unload(config) else ['--event-type', 'FirstLegLoadingPlanCompleted', '--loading-plan-id', scope['loadingPlanId']]
process = subprocess.Popen(['python3', str(proxy), '--upstream', 'http://' + ip + upstream, *selector,
'--bind', '0.0.0.0', '--port', str(ports[0]), '--control-port', str(ports[1]),
'--log', str(directory / 'proxy-evidence.jsonl')],
env=dict(os.environ, EVENT_FAULT_CONTROL_TOKEN=token), stdout=log, stderr=log, start_new_session=True)
state = {'scope': scope, 'originalBody': original_body, 'originalId': original['Id'], 'image': original['Config']['Image'],
'direction': config.get('direction', 'LOADING'), 'originalImageId': original['Image'], 'originalFingerprint': fingerprint(original),
'proxyPid': process.pid, 'proxyPort': ports[0], 'controlPort': ports[1], 'controlToken': token,
'operationsRequestId': str(uuid.uuid4()), 'prepared': False, 'restored': False}
private_write(directory / 'state.json', state)
return {'created': True, 'originalImageId': state['originalImageId'], 'originalFingerprint': state['originalFingerprint']}
def activate(scope, directory, state):
snapshot = control(state, 'FAULT')
if snapshot.get('eventId'):
raise RuntimeError('EVENT_ALREADY_PINNED')
original = inspect(producer_name(state))
require_test(original, producer_name(state))
setting, direct = target_setting(state)
if original['Id'] != state['originalId'] or original['Config']['Image'] != state['image'] or env_of(original).get(setting) != direct:
raise RuntimeError('CONCURRENT_PRODUCER_CHANGE')
if unload(state) and (original['Image'] != state['originalImageId'] or fingerprint(original) != state['originalFingerprint']):
raise RuntimeError('CONCURRENT_PRODUCER_CHANGE')
body = copy.deepcopy(state['originalBody'])
e = env_of(original)
e[setting] = 'http://host.docker.internal:' + str(state['proxyPort'])
e.update(EVENT_MAX_ATTEMPTS='12', EVENT_BASE_DELAY_SECONDS='5', EVENT_MAX_DELAY_SECONDS='5', EVENT_JITTER='0')
if len(e.get('EVENT_OPERATIONS_TOKEN', '')) < 32:
e['EVENT_OPERATIONS_TOKEN'] = secrets.token_urlsafe(32)
body['Env'] = [key + '=' + value for key, value in e.items()]
state['prepared'] = True
private_write(directory / 'state.json', state)
replace(original, body)
return {'activated': True}
def recover(scope, directory, state):
row = producer(scope, state.get('direction', 'LOADING'))
if not row or row['publishStatus'] != 'DEAD' or row['failedAttempts'] != 12:
raise RuntimeError('ORIGINAL_EVENT_NOT_DEAD')
pinned = control(state)
if pinned['eventId'] != row['eventId'] or pinned['payloadHash'] != row['payloadHash']:
raise RuntimeError('PINNED_EVENT_MISMATCH')
state['beforeRetry'] = {**row, **({'container': container_facts(scope)} if unload(state) else {'businessOrderIds': business_ids(scope)})}
private_write(directory / 'state.json', state)
control(state, 'PASS')
container = inspect(producer_name(state))
e = env_of(container)
ip = container['NetworkSettings']['Networks']['firstleg-test-network']['IPAddress']
body = {'requestId': state['operationsRequestId'], 'operator': 'admin-full-test',
'reason': scope['marker'] + ': scoped ACK-loss recovery', 'expectedPayloadHash': row['payloadHash']}
endpoint = ':9303/internal/lastleg/event-operations/' if unload(state) else ':9202/internal/firstleg/event-operations/'
url = 'http://' + ip + endpoint + row['entryId'] + '/retry'
request = urllib.request.Request(url, json.dumps(body).encode(),
{'X-Event-Operations-Token': e['EVENT_OPERATIONS_TOKEN'], 'Content-Type': 'application/json'})
with urllib.request.urlopen(request, timeout=15) as response:
result = json.load(response)
if result.get('code') != '200':
raise RuntimeError('OPERATIONS_RETRY_REJECTED')
return {'retrySubmitted': True}
def restore(directory, state):
if state['restored']:
return {'restored': True}
try:
control(state, 'PASS')
except Exception:
if not unload(state):
raise
# A dead proxy must not prevent restoring the producer's direct target.
if state['prepared']:
try:
current = inspect(producer_name(state))
except RuntimeError as error:
if str(error) != 'DOCKER_404':
raise
created = docker('POST', '/containers/create?name=' + producer_name(state), state['originalBody'])
docker('POST', '/containers/' + created['Id'] + '/start')
state['prepared'] = False
private_write(directory / 'state.json', state)
return {'restoreStarted': True}
if current['Id'] == state['originalId'] or (unload(state) and current['Image'] == state['originalImageId'] and fingerprint(current) == state['originalFingerprint']):
if unload(state) and not current['State'].get('Running'):
docker('POST', '/containers/' + current['Id'] + '/start')
state['prepared'] = False
private_write(directory / 'state.json', state)
return {'restoreStarted': True}
if current['Config']['Image'] != state['image'] or env_of(current).get(target_setting(state)[0]) != 'http://host.docker.internal:' + str(state['proxyPort']):
raise RuntimeError('CONCURRENT_PRODUCER_CHANGE')
replace(current, state['originalBody'])
state['prepared'] = False
private_write(directory / 'state.json', state)
return {'restoreStarted': True}
def proof(scope, state):
row = producer(scope, state.get('direction', 'LOADING'))
if not row or row['publishStatus'] != 'PUBLISHED' or not state.get('beforeRetry'):
raise RuntimeError('RECOVERY_NOT_PUBLISHED')
event_id, entry_id = row['eventId'], row['entryId']
if not re.fullmatch(r'[0-9a-fA-F-]{36}|[0-9A-HJKMNP-TV-Z]{26}', event_id) or not re.fullmatch(r'[1-9][0-9]{0,19}', entry_id):
raise RuntimeError('INVALID_EVENT_IDENTITY')
event_type = 'OverseasContainerUnloaded' if unload(state) else 'FirstLegLoadingPlanCompleted'
attempts = read_sql(inspect(producer_name(state)),
"SELECT JSON_OBJECT('attemptId',attempt_id,'serviceName',service_name,'entryId',entry_id,'eventId',event_id,'targetName',target_name,'outcome',outcome) FROM event_delivery_attempt WHERE entry_id='"
+ entry_id + "' AND event_type='" + event_type + "' ORDER BY started_at,attempt_id;")
audit = read_sql(inspect(producer_name(state)),
"SELECT JSON_OBJECT('requestId',request_id,'entryId',entry_id,'eventId',event_id,'payloadHash',payload_hash,'previousFailures',previous_failures) FROM event_delivery_retry_audit WHERE request_id='"
+ state['operationsRequestId'] + "';" if not unload(state) else
"SELECT JSON_OBJECT('requestId',request_id,'entryId',entry_id,'eventId',event_id,'payloadHash',payload_hash,'previousFailures',previous_failures) FROM event_delivery_retry_audit WHERE entry_id='" + entry_id + "' AND event_type='OverseasContainerUnloaded';")
if unload(state):
consumers = read_sql(inspect('firstleg-test-business'),
"SELECT JSON_OBJECT('eventId',event_id,'payloadHash',payload_hash,'status',status,'eventType',event_type,'aggregateId',aggregate_id,'payload',JSON_EXTRACT(payload_json,'$')) FROM fl_container_event_inbox WHERE event_id='" + event_id + "';")
else:
consumers = read_sql(inspect('firstleg-test-lastleg'),
"SELECT JSON_OBJECT('eventId',event_id,'payloadHash',payload_hash,'status',status) FROM ll_loading_event_inbox WHERE event_id='"
+ event_id + "' AND customer_id='" + scope['customerId'] + "';")
consumer = {**(consumers[0] if len(consumers) == 1 else {}), 'count': len(consumers)}
result = {**scope, 'readOnly': True, 'producer': row, 'consumer': consumer, 'attempts': attempts,
'proxyRecords': control(state)['records'], 'beforeRetry': state['beforeRetry'],
'afterRetry': {**row, **({'container': container_facts(scope)} if unload(state) else {'businessOrderIds': business_ids(scope)})},
'retryAudit': audit[0] if len(audit) == 1 else None, 'retryAuditCount': len(audit), 'retryRequestId': state['operationsRequestId'], 'restored': False}
private_write(scope_directory(scope, state.get('direction', 'LOADING')) / 'proof.json', result)
return result
def finish(directory, state):
if state['restored']:
return state.get('restoration', {'restored': True})
current = inspect(producer_name(state))
require_test(current, producer_name(state))
if current['State'].get('Health', {}).get('Status') != 'healthy' or current['Config']['Image'] != state['image']:
raise RuntimeError('RESTORED_SERVICE_NOT_HEALTHY')
if sorted(current['Config']['Env']) != sorted(state['originalBody']['Env']):
raise RuntimeError('RESTORED_CONFIGURATION_MISMATCH')
if unload(state) and (current['Image'] != state['originalImageId'] or fingerprint(current) != state['originalFingerprint']):
raise RuntimeError('RESTORED_CONFIGURATION_MISMATCH')
cmdline = Path('/proc/' + str(state['proxyPid']) + '/cmdline')
if cmdline.exists():
if str(directory / 'proxy.py').encode() not in cmdline.read_bytes().split(b'\x00'):
raise RuntimeError('PROXY_PROCESS_OWNERSHIP_MISMATCH')
os.kill(state['proxyPid'], signal.SIGTERM)
if unload(state):
for _ in range(50):
if not cmdline.exists() or not cmdline.read_bytes():
break
time.sleep(0.1)
else:
raise RuntimeError('PROXY_NOT_STOPPED')
result = {'restored': True}
if unload(state):
result.update(healthy=True, proxyStopped=True, producerName=producer_name(state),
originalImageId=state['originalImageId'], currentImageId=current['Image'],
originalFingerprint=state['originalFingerprint'], currentFingerprint=fingerprint(current))
state['restoration'] = result
state['restored'] = True
# Proof is already captured. Remove credentials from the retained task metadata.
state.pop('originalBody', None)
state.pop('controlToken', None)
private_write(directory / 'state.json', state)
return result
def main(config):
scope, action = config['scope'], config['action']
direction = config.get('direction', 'LOADING')
directory = scope_directory(scope, direction)
if action == 'prepare':
return prepare(scope, directory, config)
state = json.loads((directory / 'state.json').read_text())
if state['scope'] != scope or state.get('direction', 'LOADING') != direction:
raise RuntimeError('TASK_SCOPE_MISMATCH')
if action == 'activate':
return activate(scope, directory, state)
if action == 'health':
current = inspect(producer_name(state))
return {'healthy': current['State'].get('Health', {}).get('Status') == 'healthy'}
if action == 'proxy-ready':
return {'ready': control(state)['mode'] == 'PASS'}
if action == 'status':
return {'producer': producer(scope, direction), 'proxy': control(state)}
if action == 'recover':
return recover(scope, directory, state)
if action == 'proof':
return proof(scope, state)
if action == 'restore':
return restore(directory, state)
if action == 'finish':
return finish(directory, state)
raise RuntimeError('INVALID_ACTION')
if __name__ == '__main__':
try:
value = main(json.loads(base64.b64decode(sys.argv[1])))
print('TEST_EVENT_JSON=' + json.dumps({'ok': True, 'data': value}))
except Exception as error:
message = str(error)
safe = message if re.fullmatch(r'[A-Z_0-9]{1,80}', message) else 'TEST_EVENT_ACTION_FAILED'
print('TEST_EVENT_JSON=' + json.dumps({'ok': False, 'error': safe}))