Day 22: the robot and its own doctor
What do children and adults have in common? Everyone gets sick. How do children differ from adults? Adults tough it out, whereas children cry loudly. I “got lucky”: I have a child who toughs it out, so I have to hunt for his ailments in the corners with a flashlight.
What happens if the power goes out ten minutes after start-up? Or the camera software freezes? Or the part that decides where to point just quietly dies in the middle of the night?
Correct — nothing happens. Silence. Peace. Everything is fine. And that’s exactly the problem. Everything else keeps running, keeps reporting, and the arm stays exactly where it was, looking perfectly healthy… and dead.
So today I called a doctor for my robot.
Rounds, once a second
The idea is old and simple, the kind a hospital figured out long before computers. Every part of the robot now reports its pulse once a second: I’m alive, here’s my state, here’s how fast I’m working. The doctor listens to all of them.
If a pulse stops for three seconds, the patient is dead or frozen, and the doctor restarts it. If the pulse is there but the numbers are bad for ten seconds, the doctor tries a gentler treatment first: put it to sleep, wake it up again. Twice. If that doesn’t help, restart. If the same patient keeps collapsing, the doctor gives up, leaves it stopped, and says so out loud rather than pretending.
One rule I was very strict about: the doctor may put the arm to sleep, but it never wakes the arm up. The muscle moves only when I say it can. A doctor who decides on its own that the patient should go for a run is not a doctor I want near my robot.
On the bench it worked. I killed one of the patients on purpose: back in about six seconds. I froze one: back in under ten. And then I switched off the servo’s power supply, and the arm went limp by itself, and the doctor wrote it down. Yesterday’s lesson one is now somebody else’s job.
First, do no harm
Of course it didn’t go like that the first time.
As the saying goes, “every doctor has his own graveyard,” and my doctor was no exception to the rule. The very first thing my doctor did was kill the detector. Not a sick detector, a healthy one. It asked it to do something that only makes sense in a different state, and the detector, instead of saying “no, thank you,” simply died on the spot. So the doctor learned to ask the patient how it feels before prescribing anything.
Then it got too enthusiastic. It saw bad numbers and treated the detector. Then again. And again. Forty-six times, without once giving the treatment a chance to work. So now every treatment is followed by a probation period, during which the doctor watches and doesn’t touch. And there’s a warm-up after every start, because the detector is slow for its first few seconds, the way anybody is slow in the morning. Let the patient wake up before you measure its pulse.
And then my tests, the careful, isolated, well-behaved tests, reached out into the real robot. A test pretended to be the servo, and the real doctor, listening on the same channel, believed it. It saw a servo in trouble and restarted the real one, which was perfectly fine and sitting on the bench minding its own business.
The good news is the direction: the arm let go of its grip and waited. Nothing moved that shouldn’t have. But a doctor who restarts the wrong patient because somebody in the corridor said the right name needed two more rules. Tests now live in their own separate world. And the doctor now checks who it’s actually about to operate on before it picks up the knife.
The waiting room
Meanwhile, something turned up that none of us was looking for.
Every so often, the arm would lag behind the cap by a third of a second. Not always. Only when the robot was busy. We had a candidate for this, a video stream I watch it through, and we were ready to blame it. It turned out to be a waiting room.
The detector kept a queue of the last ten camera frames. As long as it kept up, the queue stayed empty and nobody knew it was there. But the moment it slowed down even a little, the queue filled, and from then on every frame it processed had been waiting in line for about a third of a second, and the hand pointed precisely and unerringly at the spot where the cap had once been, unaware that it had long since moved to a new location.
As strange as it may sound, the solution was to make the waiting room smaller: two frames instead of ten. If you’re late, you’re not seen; you go home; the next one is fresher anyway — OMG, why can’t we apply a rule like that to the real world?!? — anyway, the average delay went from 92 milliseconds to 13.
A patient doesn’t get healthier from waiting in a queue. Neither does a camera frame.
Tonight
The robot is now running alone. The whole thing: the camera, the detector, the arm, the doctor, and a little later the model that names things, all together for ten hours, with something writing down its temperature, its memory and its pulse every thirty seconds.
I’m going to sleep. The doctor is on the night shift. Tomorrow we’ll read the chart.