#26: "Nothing moved! Stop folding the armchair!"

Last time the exam was running again as I wrote, and the watchman finally had his notes pinned to the right frames. Since then he has sat a lot of exams. And the more he passes, the more boring the next question gets: fine, he catches what I do to the scene. What does he do when I do nothing at all?

An hour of nothing, again

So I built a corner of the room and left it alone. A table with a book, a box, a cup, an armchair under a sheet with a towel on it. Nobody touches anything. The robot just watches.

The robot's view of the quiet corner

The good news first: in a whole quiet hour nothing false was reported. The watchman kept his mouth shut.

The less good news was in his notebook. In fifteen minutes of a scene where nothing moves, the robot had merged two of its objects into one 158 times. Ten times a minute. Of nothing.

The towel that keeps getting folded

It took a while to see what was going on, because each single step was perfectly reasonable.

At the very edge of the picture there is a strip of something with the wall behind it, cut by the frame. Sometimes the robot sees that strip whole, sometimes as two halves. The whole strip is an object it already knows. A half is something new, so after three seconds of looking at it the robot says “Bingo! I caught it - new object!”. A moment later another rule notices that the new object lies entirely inside the old one, at the same depth: a duplicate. Merge it. Three seconds later the half shows up again. New object! Duplicate. Merge.

The armchair had its own version: a dark gap at its edge, twenty centimeters deeper than the armchair itself. Born, merged, born, merged, every twelve seconds.

Like someone folding the same towel over and over, and every time proudly announcing that there is one more towel in the house.

And it isn’t harmless. Every merge counts as the object changing, so the armchair gets renamed and its “what normal looks like” is learned again from scratch. A watchman whose idea of normal is reset every twelve seconds will never notice anything happen to that armchair.

Two small rules

The fixes are small. A piece of a flat wall, lying flat in the wall’s own plane, is part of the wall - don’t make it a separate object. And when two objects get merged because one kept showing up inside the other, remember where that inside one lived. If something is born there again, it’s the same thing again.

158 merges went down to 4. Then I played every recorded scene through the new rules, and sat down with the sheets once more.

Is the edge still there?

One of the sheets got me stuck.

A mug is put down in front of the edge of a bench. The robot says: the bench edge is gone - and I’m pretty sure the robot has no idea about quantum mechanics! I can see the bench is right there and its edge should be in place as well. Except… can I? On the “after” frame the mug covers almost all of it. I know the edge is still there because I was in the room and no one moved the bench. The camera doesn’t know anything of the kind.

So the reading got a new answer next to “true” and “false”: hidden. If more than about a third of the object is covered by something placed in front of it, the picture can’t tell what happened to it - and a robot that reports a change there is claiming something it never saw. That is a defect of its own kind, counted apart. Not a lie, exactly. Gossip.

The scoreboard was cheating too

To compare the old rules with the new ones I count how many of my staged actions the robot caught. “Caught” meant: a true mark landed in the time window of the action.

Then I looked at one action the old rules caught and the new ones didn’t: a bottle and a tape roll swapped. The old rules’ “catch” was a true mark on an object 89 centimeters away from both. Right time, wrong place. A goalkeeper credited with a save because he happened to be on the pitch.

So I went through every staged action once, on the sheets, and wrote down which objects it actually touched. Forty actions. Now an action is caught only by a true mark on something it touched, at the place it touched it.

The old rules: 22 of 40. The new ones: 21. The same code run twice wobbles by one action per recording, so that is a draw. And false marks: 58 against 58. The towel-folding is gone and nothing got worse.

My team reads the code

The new rules went into the robot’s real code, and my team reviewed it. Twice, by two reviewers who knew nothing about why it was written the way it was.

The second one asked a very polite question. You say a piece of a flat thing at the same depth is part of the flat thing. Is a floor a flat thing? Yes. Is a table top, if it’s big enough? Yes. So a cup standing on the table, at the table’s depth…

…is a piece of the table. My towel fix would have quietly eaten every cup I put on a table. And the second rule had the same problem from the front: put something down in front of the armchair, right where the gap used to be, and the robot would never notice it arrived.

Both rules now ask one more question. A piece of the wall has to lie flat in the wall’s plane - a cup sticking ten centimeters up out of the table doesn’t. And the gap’s old place is protected only behind the armchair, where the gap was; a thing put in front of it is born.

The exam is running again as I write this. It’s becoming a tradition.

All in all

I knew from the very start this wouldn’t be easy. I just didn’t think it would be THIS not easy! Teaching a robot to see, and to understand what it sees, is turning out to be one of the hardest parts of the whole project. No mechanics, no hardware design. Pure algorithms, math and optimization. Dozens of hours of tests. Runs, and more runs, and picking through the results. A hypothesis, a refutation, another hypothesis, and another miss… Which makes the moment all the sweeter when a hypothesis finally holds and the robot gets a tiny bit “smarter”. One more step toward a life of its own!

Get the next one by email