#27: "It's a wall! No, it's a bicycle!"
Last time I went to bed with the exam running. A piece of the wall must lie flat in the wall’s plane, so a cup on a table is a cup. And a thing put down in front of the armchair’s old gap is a new thing, not the gap coming back. Both fixes went into the robot’s code. All that was left was to check that the towel-folding was still gone. Of course, it wasn’t.
The fix that brought it back
At one in the morning the quiet corner had 110 merges in fifteen minutes. The evening before it had 4. Same corner, same recording, same nothing happening.
The culprit was my own answer to the second question. “A thing in front of the gap’s old place is new” sounds simple. But in front of what, exactly? The code compared the new thing with how far away the armchair is. And how far away is an armchair? The code took the middle of all its depths, and the armchair, together with everything merged into it over time, was three meters across. Its “middle” sat twenty centimeters behind the spot where the gap actually was.
So every time the gap showed up, at exactly the gap’s own depth, the robot measured it against the armchair’s average, decided it was twenty centimeters in front, and announced: new object! Merge. New object! Merge.
Like asking whether a cup stands in front of the sofa by measuring to the middle of the whole sofa.
The fix: when the robot remembers a place, it now remembers how deep that place was. A thing in front of the place is new; the armchair’s average has nothing to do with it. 110 merges went back to 4. That it turned up before the commit and not after is the only reason this section isn’t longer.
Two hours of a wall
Then I wanted to see the robot live, not on recordings. The camera was looking at a wall at the time, so the robot got a wall. Two markers on stands, the arm in the corner, a lot of white.

Two hours. Four objects. Zero changes, zero merges, zero anything. The most peaceful two hours of its life and possibly of mine too.
Which is nice, but a wall is not much of an exam. So back to the recordings, and to the one I had been ignoring: an hour of an empty room. Almost nothing false gets reported there, so nobody looked at the notebook. In the notebook: 300 births and 300 merges an hour of an empty room.
The bicycle that couldn’t decide
The robot sorts what it sees into two kinds. Walls, floors and other big flat things are structure: the background, never an object to keep track of. Everything else is a thing.
And in that room there is a frame leaning against the wall, with a mat thrown over it.

Through the gaps in the frame the camera sees the wall. So on some frames the robot looks at that outline, sees mostly wall, and says: structure, a wall, 2.3 meters away. On the next frame it sees the frame itself, a thing, 1.35 meters away. Same outline. Two answers.
Whichever answer comes first becomes the object. The other one is a stranger, so it gets born as a new object. A few seconds later the depths blur together, the robot notices the two have one outline, and merges them. Then the frame flips again. A wall! No, a frame! A wall! Three hundred times an hour.
It even depends on the run. Play the same recording twice and once the bicycle wins, once the wall wins. One run had 715 births in the hour instead of 340, just because the bicycle got there first.
Fixes drowning in noise
I tried two fixes straight away: let a wall reading join a thing with the same outline, or make walls wait three seconds before being born, like everything else already does. An hour of replay each. Results: 290 and 376. Against 340 to 410 for the unchanged code on different runs.
In other words, nothing. When the same code gives anywhere from 340 to 715, a fix that gives 290 has proven exactly nothing. The perception side wobbles from run to run, and that wobble was louder than either fix.
So instead of another hour of replay I recorded twenty minutes of exactly what the robot’s object layer receives, frame by frame. Now any version of the rules gets the same input, and a test takes minutes instead of an hour. No more wobble, just the rules.
On identical input it was clear in one evening. Walls and things must never merge: if one outline reads both ways, let there be a wall object and a thing object, and each reading goes to its own. And walls wait the same three seconds as everyone else, so they’re judged on a settled look and not on two frames.
Births in the empty room: from 340 an hour to 68. Then a full run of all the recorded scenes, and the sheets again. False marks went from 56 to 50, and the robot still caught as many of my staged actions as before.
The price: a thing that sometimes looks like a wall is now stored twice, once as a wall. I can live with a frame that has a wall for a twin.
Meanwhile, in the spec
While the robot was busy with the empty room, I checked the plan for this whole stage. It promised that a quiet scene writes “hundreds of rows a day.”
It writes seven thousand. Before I could panic, I went back to where “hundreds” came from: my own rule, written six weeks ago, before any of this code existed. One row per object every ten minutes. For ten objects that’s already 1,440 a day. For the 27 in the quiet corner, about 3,900 - and roughly as many again from naming. The robot is doing exactly what I told it to; I just never did the multiplication.
My team was quicker to panic than me. One of them first announced that the store would eat 78 gigabytes a year, and then remembered that the robot already cleans up after itself every hour and keeps only the last few confirmations of each object. So I measured it: an hour of the empty room, with the cleanup running every five minutes instead of every hour. The pictures behaved. They leveled off at about 3 megabytes and stayed there. The rows didn’t. My team, slightly embarrassed by now, did the arithmetic again and came back with a hundred megabytes a day.
So who was filling the notebook? Our old friend. The frame still folded now and then, every fold counts as a change, and a changed object gets its name asked again. “Frame seat.” A few minutes later, “frame seat.” Eighteen times in one hour. Every answer kept forever, with its mask and its points, because the cleanup only knew how to throw out old confirmations. My own spec, meanwhile, said a name is worth keeping when it’s new. “Frame seat” for the eighteenth time can’t be a new one.
Now the cleanup keeps a name only when something points at it: the first time a name was given, and the latest. One more hour of the empty room on the fixed code: 7.9 megabytes at the end of it, and from its last half hour the store kept two new objects and one name, besides the confirmations.
And the hundred megabytes a day? A misreading, again. In its first hour every object is still filling its five confirmation slots, one every ten minutes, so the store grows - and then stops, once the slots are full. My team had drawn a straight line through the first hour and called it a trend.
So the plan now promises what the rule actually does: one confirmation per object every ten minutes with its name, nothing else on a quiet scene, and a store that levels off under its ceiling. With that, this whole stage - the robot’s memory of what’s in the room - is closed. Two things go to the to-do list: the frame still folds a few hundred times a day, and the robot still sometimes marks something I didn’t touch.
The robot now remembers what it sees, writes down only what changed, and forgets the rest on schedule. I know a lot of people who would settle for that…