Day 17: it looked at the room and found its own cable

Yesterday the arm moved. That was a good day. But it moved because I told it exactly where to go, in counts, over a serial cable, like operating a puppet. It had no idea what was in front of it.

Today it looked.

Everything it is looking at, and the one thing it actually found.

The first thing it saw

I put a red marker in front of the camera and ran the detector. And there it was, in the log, the first time this project has produced a coordinate of a real object:

detection: class=red_blob
           at (0.036, 0.098, 0.782) m
           size 0.029 x 0.057 m

Three and a half centimeters right of center, ten below, seventy-eight centimeters away. And a size — twenty-nine millimeters by fifty-seven. Which is, to within about a millimeter, a marker cap.

It didn’t just spot a red thing. It told me where the red thing was in space and how big it was, from a single frame, in about a second. I stared at that line for longer than I’d like to admit.

Then I moved the marker further away and ran it again, and it locked onto something else entirely.

Which turned out to be its own power cable

There is a red wire running across the plate — the one feeding the servo. It’s red. It’s always there. It is, from the camera’s point of view, a perfectly good red object, and when the marker got too far away to register, the detector did exactly what it was built to do and picked the biggest red thing it could see.

The robot’s first independent act of perception was noticing part of itself.

There’s a real lesson in there and I’ll get to it, but mostly I just find it funny. The thing has one eye, it’s been open for about four hours, and it has already become self-absorbed.

“Is seventy-eight centimeters right, though?”

That’s the question that ate the rest of the evening.

I took a tape measure to it. Front of the camera to the marker: about seventy-six. The camera said seventy-eight. Two and a half centimeters out — not much, but I’d rather know why than shrug.

Here’s why, and I think it’s genuinely nice: a depth camera doesn’t measure from its face. It measures from the middle of a lens, inside its own body. The number it gives you is the distance from a point buried a couple of centimeters behind the front glass. So the camera wasn’t wrong and my tape wasn’t wrong — we were measuring from two different places, and the gap between them is the depth of the lens assembly.

Which means I hadn’t found an error. I’d accidentally measured where the camera’s eye actually is, which is a number I needed anyway.

Except I couldn’t be sure, because a constant offset and a percentage error look identical when you’ve only measured once. So I measured again at a meter and a half, got a completely different answer, and spent a while confused before realizing I’d taken a bad measurement — a small object, high up, off to one side, at the end of a tape measure I was stretching sideways across a room. Of course it disagreed.

Measuring things properly

The right way, it turns out, is the way the camera’s own manufacturer specifies: don’t measure to an object, measure to a wall.

A wall is unambiguous. It fills the frame. There’s no argument about which bit of it you meant. So I pointed the camera at a white wall, propped it roughly square, tape-measured it, and then fitted a flat surface through every single depth reading in the image — three hundred and seventy thousand of them — and asked where that surface sat.

perpendicular distance : 1119.5 mm
camera tilt            : 2.4°, removed by the fit
agreement with a single-point reading : 0.5 mm

Tape said one hundred and nine centimeters. Camera said one hundred and twelve. Same three-centimeter offset as before. Constant, not proportional. No scale error. The lens is where the lens is and the camera is telling the truth.

Three hundred thousand measurements beat one tape measure, and it’s not close.

And then the question that undid the whole day

Late on, going over the camera settings, my team explained a trade-off: we’d dropped the camera to a lower resolution so that the color and depth images would match, because the alignment between them needed it.

And I asked what I thought was a simple question. What exactly are we matching?

Long pause.

Nothing. There was nothing to match. The alignment process reprojects one image into the other’s viewpoint — resolution is something it handles, not something it requires you to pre-arrange. The lower resolution had been chosen to solve a problem that didn’t exist, on a theory nobody had gone back to check, and the cost of it was real: about half the camera’s usable detection range, and half its precision on every position it reports.

Fixed in one line. The camera now runs at full resolution, the alignment is perfectly happy, and the robot can see roughly fifty percent further and locate things about twice as precisely as it could this morning.

A whole capability, sitting there unused, because of an explanation that sounded reasonable and was never tested. I don’t think I’ll forget that one.

So where does that leave it

The robot can see. It can tell you where a thing is and how big it is, in meters, from one frame. And separately, it can move its arm to a commanded angle to within a tenth of a degree.

What it cannot do is connect those two facts. The camera reports positions from its own point of view, sitting on top of its little tower. The arm turns about a pivot somewhere else entirely. Nobody has yet told the robot how those two places relate — so it can see a target perfectly and still have no idea which way to turn.

That’s the next thing. It’s mostly geometry I’ve already measured, sitting in a design document, waiting to be written down in a form the robot can use.

Which is a strange sentence to type. The hard part isn’t the hardware anymore.