#25: "Keep your hands out of the scene!"
Last time the watchman went home with a new notebook and a promise: fewer departures. Before I let it near a second night, I asked a simple question. How do I actually know the new rules are good?
The honest answer was embarrassing. Every rule the robot used to decide “same object” or “this has changed” had been tuned on two recordings of one scene. And the same recording served as the exam. Like a student who writes his own test, sits it, and gives himself an A.
So we set the target: zero defects. Not “good on average” — no mistakes on the cases the robot is supposed to handle. And to check against that you need cases nobody tuned anything on.
Five recordings
I recorded five scenes, every one to a script, with a clock. The same bench as before, the same seven moves, to see how much the result changes from run to run. A table of awkward materials: a glass, a black sock, a phone face down, two identical mugs. A scene of tricky changes: identical cups swapped (the robot can’t see that at all, and it shouldn’t pretend it can), a box moved, a book turned, and my hand held over a tennis ball for five seconds without touching anything. A scene of things touching each other. And one hour of nothing at all, with nobody in the room.
That last hour weighs 74.5 gigabytes. Of nothing. Recorded very carefully.
Then I sat with the sheets again. Every time the robot said “this changed” or “this left”, I looked at the frames before and after and wrote down whether it was true.
151 marks. 27 true. Plenty of room for improvement, I’d say…
The hand
Most of the false ones had one author: me. Or rather my hand.
When a hand reaches into the scene, the camera sees it — of course it does. The robot cuts the picture into pieces, and for a moment the piece in front of the mug has the depth of my fingers. Those points went straight into the mug’s little cloud of cubes. The mug grew — and a mug that has grown no longer looks like the mug the robot remembers, so it reported a change. To the mug. Which I hadn’t touched. And to its neighbors, which I hadn’t touched either.
There was a second, sneakier version in the empty hour. Nobody in the room, and still eleven false changes. Nine of them followed a box that had grown by a single two-centimeter layer between the moment the robot learned what “normal” looked like and the moment it started comparing. It was comparing today’s object against yesterday’s slightly smaller one. Of course they differed.
The camera added its own touch: when a hand comes in, the auto exposure darkens the whole frame, so every object in the room suddenly looks a bit different too. Thank you, camera. Improving under ideal conditions is for the weak!
Nearer than anything you’ve seen there
The fix for the hand comes from people who build maps for robots moving through crowds. The idea is lovely in its simplicity: for every pixel, remember the farthest thing you have seen there while it held still. If the depth in that pixel is suddenly much nearer than that — something is in front of it, moving. No need to know it’s a hand. No need to know anything about hands at all.

The picture above is that rule at work: everything in red is what the robot decided is moving. It is my arm. Exactly my arm.
What moves is now never part of what stands still: it doesn’t add cubes to the mug, and its pixels don’t count when the robot decides whether the mug changed. And the box the robot compares against is now frozen at the moment it starts learning, until it decides. That alone took the empty hour from eleven false changes to one, and the tricky-changes scene from 84 marks to 30. And not one real change lost along the way.
A diet
The next step was to give the robot a proper 3D map of the room, and NVIDIA has a library for exactly that. Everything would have been great, except it didn’t fit. Who would have doubted it? With it running the board was down to 194 megabytes of free memory and pushing more than a gigabyte into swap.
The fattest thing on the board was the toolkit the vision models run in — PyTorch, which loads its whole kitchen to cook one dish. So the models moved to TensorRT, which is just the dish. Perception went from almost three gigabytes to about one and a half. And the answers stayed the same: bit for bit on the shapes, and the same pick on 70 of 70 pictures for the names.
Night is a mysterious time of day
The names moved too: the robot now asks for an object’s name when it is born, a minute later, ten minutes later and so on — and never while something is moving around it, because a hand in the picture is a very confident answer to the question “what is this?”.
So everything was ready for the big exam: all five recordings, played back through the real code, twice, overnight. I started it and went to bed.
First result, an hour later: the bench scene, seven staged changes. The robot found objects, named objects, wrote everything down neatly.
Changes caught: zero. Not seven, not four. Zero.
Remember the night watchman from last time? The part of the robot that keeps the notebook and writes down what changed. He had slept through the whole shift.
By morning I knew why. Once a second he is supposed to make a round: take a fresh depth frame, take the note from the hand-spotter saying which pixels are moving, blank those out, and compare. The notes come four times a second. The depth frames come thirty times a second, and he kept only the last five of them, a sixth of a second’s worth. By the time he got round to a note, its frame was often already gone. On half his rounds he found nothing to compare, shrugged, and went back to his chair.
Half the rounds doesn’t sound like a disaster. But the watchman is careful: he trusts only what he has seen three times in three seconds, and he needs thirty of those before he even knows what “normal” looks like. With every other round missing, he almost never finished learning. And a watchman who doesn’t know what the room is supposed to look like can’t tell you that it changed.
The fix: pin each note to its frame the moment the note arrives. A few lines. The exam is running again as I write this.