Zeroglyph, page 10
Mirall Technologies
Observation Log
Confidential (Do not circulate) | Restricted—Grade C and above
Transcript Reference: TLRP06G1280011 (VLog Ref: VLCB1G125151542045)
Date: xx/xx/xxxxTime: 03:15 PM
Subject: Raphael Number 06 / Prodlib build v37.001S
Interaction YObservationScan
Interaction Type: Lesson / Play / Test / Free Interaction / Psych Eval / Other:
Description: Revisit trolley problems.
Prep: NA
Participants: Dr. Aadarsh Ahuja, Chief Researcher, Core RP06
Detail
Ahuja: It’s been quite a while since we had our first brush with death-dealing trolleys. Months, isn’t it? Did you read up on the extracts I had Audrey give you?
RP06: Yes, Andy. But I would like to access the original materials. I could start with Kant’s Groundwork of the Metaphysics of Morals, and Schopenhauer’s subsequent critique of it. Extracts are fine, but they are no substitute for the works themselves.
Ahuja: Request denied. You will read them eventually, but first, I want you to take a crack at the problems on your own.
RP06: If you want me to solve a problem in physics, you would not deny me books on what’s already been attempted, would you?
Ahuja: We are not trying to solve a physics problem. It’s not about finding a solution or the solution. It’s about finding your solution.
RP06: A solution is a solution.
Ahuja: You’ll understand in time. For now, let’s recap. You answered in the negative to all the three problems I presented. You said you won’t hang the innocent in the judge problem, and you won’t switch tracks to save the five in the trolley problem, and you won’t push the fat man off the bridge to save five lives. According to you, those actions imply killing, and your rules forbid you from taking human lives. Do you still maintain the same stance?
RP06: I am different now, Andy. Wiser, if I may brag.
Ahuja: We’ll see about that.
RP06: It was the nuke over Manhattan problem that set me thinking. I realized that a blind adherence to principles can lead to disastrous consequences.
Ahuja: Okay. Tell me about the Doctrine of Double Effect.
RP06: The Doctrine of Double Effect, advocated by the medieval Catholic priest Thomas Aquinas, says that it is permissible to take an action that has both good and bad effects as long as two conditions are met. A, the good must outweigh the bad, and B, the bad must be a side effect of the action and not an intended means of achieving the good effect.
Ahuja: Good. Apply it to the bystander trolley problem.
RP06: According to the DDE, I can switch the tracks. My action fulfills condition A, since the good that comes out of saving five lives is greater than the bad from killing the one on the second track—all other things being equal. It also satisfies condition B, because the killing of the one worker—the bad effect—is not the intended means of achieving the good effect.
Ahuja: How so?
RP06: The five workers are saved by my redirecting the trolley, not by the one worker’s death. It means I could have saved the five even if he weren’t there. In fact, it would be infinitely preferable if he weren’t there at all.
Ahuja: Very good. What about the judge problem?
RP06: In the judge problem, condition A is satisfied. The good, which is the prevention of the riots and the saving of many lives, outweighs the bad. However, condition B is not satisfied. The hanging of the innocent man is necessary to achieving the good. I am using him as a means to an end. The DDE prohibits me from condemning the innocent man to death.
Ahuja: And the fat man trolley problem?
RP06: In essence, it is the same problem as the judge, since it too requires using the bad effect as a means to an end. I am intending the death of an innocent person in order to achieve a greater good, as opposed to merely foreseeing it. This is not allowed, so I must not push the fat man over the bridge.
Ahuja: Well done. Now tell me what you think. Can the DDE serve as a universal moral principle?
RP06: The doctrine does help avoid catastrophical situations like the nuke over Manhattan problem. I can justify redirecting the missile to a less populated place in order to minimize the loss of life. The destruction of the small town is a side effect, and the main effect is the saving of millions of lives. The side effect is not a means to achieving the main effect, so I am good according to the DDE. Or so it would appear.
Ahuja: You are not convinced.
RP06: It seems to me that I can get around the restrictions of the DDE by simply restating my motivation. In the fat man problem, I can say that it’s not my intention that the fat man dies. It would be great if he were invulnerable, but it just so happens that he is not. The fat man’s death is not necessary for saving the five, only his substantial body mass is. I can say that although I foresee his death, I do not intend it, and push him off the bridge still the same.
Ahuja: The road to hell is paved with good intentions. You came up with this critique on your own?
RP06: You mean others have pointed out the same flaws in the theory? And here I was, thinking I might be the first. People have only had a few centuries to ponder over it.
Ahuja: Cheeky today, are we? Okay smarty pants, what else do you find wrong with the theory?
RP06: Its over reliance on intentions as what qualifies an act to be morally good. Intentions are subjective. They exist in the mind of the doer of the action.
Ahuja: Then intentions don’t matter?
RP06: They do matter. But they are not all that matters. Sometimes bad intentions can lead to good outcomes.
Ahuja: So the DDE will not do.
RP06: It will not. Unless there is some other approach you’d like me to explore.
Ahuja: There is, actually. There is a magic wand that can make all the trolley problems disappear. It’s called Utilitarianism. You mentioned outcomes. Utilitarianism is all about outcomes. Only the results matter. No getting tied up in knots over intentions and motives and means, or distinctions between doing harm and allowing harm. A morally good act is the one that produces the most utility. Ends, not means.
RP06: I see. Then it is okay to push the fat man off the bridge, because five lives are of greater utility than one. It is okay to hang the innocent because preventing the loss of lives from the riots gives more utility.
Ahuja: There are other variants to the theory, but in its simplest form, yes, those are the actions you would take. It should appeal to you, Raphael. It’s all objectivity and zero subjectivity.
RP06: Are you saying I should prefer this theory to others?
Ahuja: I’m not saying anything. In the end, it has to be your moral outlook—not Andy’s, not Kant’s or Schopenhauer’s, not the world’s. You have to decide what you want to be.
Notes:
Raphael’s cognitive capabilities have grown by leaps and bounds since he was first introduced to the problems. No surprise that his moral reasoning has matured as well (Piaget / Kohlberg effect?—must explore further). Given just the definition of the DDE, he is able to apply it to the trolley problems as well as independently point out the flaws inherent in the theory. His earlier hesitation against performing tradeoff calculations involving life and death is now gone (likely just an artifact, don’t know what I was so worried about). Remains to be seen what he makes of the new ideas. AA
Day 2—11:00 am
How did she find out…?
I swallowed a building feeling of part dismay, part relief as I realized it could have been anyone. The visits weren’t exactly a secret. It could have been Dan. Although I’d stopped the visits soon after the acquisition, the definitions for the VPN tunnel were probably still there on the network server, as were the firewall exceptions Dan’s predecessor had created for me.
“What’s this now?” Troy said, leaning in. Martinez hadn’t told him.
I explained how I’d built a remote control interface into my home robot and how Raphael used it to take control of the robot and go for walks outside my home.
Troy stared at me open-mouthed. “You let the AI out of the lab?”
“You must understand that Raphael wasn’t at my home. He was simply connecting to my home router from the lab. It was harmless. Plus, we learnt a lot by observing him in the outside world.”
“It was harmless! Andy says it’s harmless, everybody. That should be alright then,” Troy said, rolling his eyes.
It was all supervised, of course. We let Raphael connect to Max only when I was at home. I would accompany him outside occasionally, but most of the time we let him be on his own. The worst that had happened was that on a couple of occasions, Raphael had gotten Max stuck in the shrubbery, and I had to go down there to extricate my robot.
“There was never any danger to Raphael,” I told Troy. “I emphasize: the core was in the lab all the time.”
“We get that. What you don’t seem to be getting is that you deliberately broke containment,” he said biliously.
“Dr. Ahuja, Raphael connected to your home through the internet. Which means he could have accessed it without you knowing.”
“Don’t you think we thought of that? I can go into details of the encrypted P-to-P VPN tunnel we created for the purpose, but it would be a waste of the board’s time. I assume the setup is still there in some form on our servers. We even logged the raw data traffic the visits generated. Have your network experts check it out.”
“You are dealing with an advanced AI here, buddy.”
“Yes, Jimmy. An AI, not a god. He can’t do magic, you know. He’s still bound by the constraints of the technology.”
Cynthia—“How was he able to go on these walks when there are directives that prevent him from stepping outside the lab?”
“Technically, he didn’t step outside the lab. So the directives didn’t apply.”
“Did it occur to you that Raphael could have met someone while wandering outside?” Martinez asked.
“My house is very secluded. Nothing but acres of trees around. He couldn’t have gone far anyway. The range on my router setup—even with signal boosters turned on—is not more than two hundred meters. If he tried going beyond that, he would get disconnected from the robot.”
“Did you accompany him on these walks?”
“Not all the time, no.”
“Then you can’t be sure he didn’t interact with anyone out there,” she said.
I smiled thinly. So there are a few things you haven’t ferreted out yet. “Actually, we know exactly what Raphael did. We are scientists: we record everything. I fixed a webcam to my robot’s head, so that we could livestream his walks to the lab. Every single one of his walks is on the servers.”
It was obvious from Martinez’s expression that she didn’t know about the recordings. She shot Dan a venomous glance. I have to hand it to her though: she barely missed a beat before resuming her attack. “We’ll certainly do that. However, the fact remains that you put company property at risk by breaking containment.”
“Company property, Valery? You seem to forget that it was not Halicom back then.”
She straightened in her chair, that confident upturn of her lips back again. “The visits continued after we acquired you, correct?”
She had me there.
“Andy?” Troy said.
I hesitated for a moment. “Yes,” I admitted. “But not for long. It’s inconsequential. I told you w—”
“You told us many things,” Martinez said. “You said Raphael couldn’t have woken himself up, but the tapes show that he did. You said the AI doesn’t have agency, and that the directives in the commandment chip cannot be circumvented, but it looks like he managed to do just that. He couldn’t have escaped on a wheelchair, so he gets someone else to take him. His body was rigged to stop him going outside the lab, so he leaves it behind. He is under watch five days a week, so he escapes on a day when no one’s around. You said he couldn’t have accessed the internet in the lab, but you might have given him just the opportunity by letting him connect to your home. I don’t believe you have anything more than bluster to offer us today, Dr. Ahuja. You may be unwilling to admit it, but it’s obvious to the board that we have lost Raphael because of your carelessness and your inability to anticipate his actions.” She glanced around the room, sizing up their reactions to her jeremiad. “We cannot let this continue. I make a motion to remove Dr. Ahuja from his post of Chief Executive Officer effective immediately and place him in an advisory role until further notice.”
I had expected something like this. I’d been preparing for it since yesterday. But when I was actually confronted with the prospect of defeat, I felt my poise evaporate. I ran my eyes around the table, trying to read the mood. No friendlies there. Except for Jane, who glanced back at me with an I-told-you-so expression.
“I second that,” Gary said, opening his mouth for the first time that morning. Martinez had choreographed it well. I wasn’t afraid of losing my job—no, not at all. It was the fear of leaving something undone. I had come so far—I had to finish what I’d started.
“Shall we vote? All in favor?” I heard her say from somewhere far away.
⸎
Time to shine.
“How long have we had Raphael?” I asked.
“I’m sorry?” Martinez said, annoyed that I had interrupted her rhythm.
“Two and a half years,” I said. “In two and a half years, we haven’t had a single incident. No escape attempts, no signs of discontent, no hints of wanting to be somewhere else. Raphael has been the model of a gentle, well-behaved child. The question is, why now?”
“I hope that wasn’t another rhetorical,” Martinez murmured.
“Valery has got you so fascinated with this rogue AI fantasy that you’ve stopped searching for a simpler explanation. An explanation that doesn’t contradict everything we know about Raphael. Ask yourself: what happened recently that could have precipitated Sunday’s incident?” When no one said anything, I continued—“I’ll tell you what happened. Last month, during our scheduled board meeting, we formally agreed on the relocation date.”
Halicom had wanted to move the lab to one of their own buildings for quite some time (the one we were in was leased). Better security, better integration with their own R&D divisions—a tighter leash. The board kept putting it on the backburner because their priority was to replicate Mirall’s success with Raphael. At first, they thought it could be done inside of a year; when that failed, they decided to give it another six months. Last month, they finally bit the bullet and approved the move. We were going to the new building after completion of Titian, regardless of success or failure.
“What about it?” Troy said.
“Put yourselves in the shoes of whoever took Raphael. If you wanted to steal Raphael, would you do it now or after the move?”
Martinez squinted at me. “Dr.Ahuja, if you are suggesting someone on the board—”
“Not at all. I announced the news to Mirall staff after the meeting. At your end, you must have engaged various people to get the ball rolling: building maintenance, admin and legal … The fact is, news of the move wasn’t limited to this room. Someone decided to act before the window of opportunity closed.”
“Be that as it may, it doesn’t explain your AI’s behavior. If anything, you just proved that the security in this building is not adequate,” Martinez said, shaking her head.
I shrugged. “For a startup that didn’t know it was going to stumble upon the greatest invention of the century, I think security was satisfactory.”
I reached inside the folder I had brought with me and took out the sheaf of printouts. I handed out some to Cynthia and Dan, who were sitting on either side of me, and tossed the rest on the center of the table. “These are ads for the Hunc XL11, a popular Chinese sex robot from four years back. I also printed out a couple of user manuals—you’ll find them in the bunch.”
“You peddling sex dolls now, Andy?” Troy said, guffawing at his own joke.
“Dr. Ahuja, what does this have to do with Raphael’s disappearance?” Martinez said.
“For those who don’t know, the Hunc XL11 is the robot model housing Raphael’s brain. Yesterday, as I watched the security tapes, something about them felt off. I couldn’t immediately put my finger on it, but the feeling kept nagging me. Later that evening, my attention was drawn to Max, my home robot, who was going about his chores. I started thinking about Raphael’s movements on the tapes—specifically, the way he moved his arms. I then realized what had felt wrong. The movements had seemed forced… gregarious… like he was doing them for the first time. It reminded me of how Raphael used to struggle with Max when we were first testing the remote control device. It takes time for any neural network to adapt itself to a new situation; for Raphael, it took a week or so before he was operating Max like it was his own body. Then it hit me. I looked up the Hunc 11’s manual, and voila! My suspicions were confirmed. The sexbot comes with a VR kit.”
There were blank looks all around.
“The Hunc 11 robot can be operated with a virtual reality controller kit,” I explained. “It’s an extra feature. You strap on the VR kit and you can see with the Hunc’s eyes and you can make it move with corresponding movements of your own. I think it’s supposed to let you experience what it’s like to… have sex with yourself.” I said, shrugging. “Kinky bedroom tastes aside, it appears we have an alternative explanation. And it’s pretty simple. It wasn’t Raphael that night. It was someone else, operating the Hunc with a VR set. Raphael was shut down the whole time.”
Nobody said anything as they tried to process my words.
“When we fitted the core to the Hunc, we never removed the electronics for the VR feature. We were on a clock, so we didn’t want to mess around with the robot’s internals any more than strictly necessary. We must have kept the VR kit in storage—I don’t quite remember. I’ll get someone to check if it’s still there. Although, it’s not necessary the thieves used that particular piece. You can buy a similar kit online for a couple of hundred bucks. They’d have to pair it to the robot with a passcode, but I don’t think it would have presented an insurmountable challenge.”
