Zeroglyph, page 9
Dan colored at my remark but didn’t say anything. “And don’t forget,” I continued, “the robbery would not have happened if the motion detectors and the door locks weren’t turned off for the carpet cleaning. It wasn’t my idea to bring in bots to save cost.”
“I…uhm… It wasn’t just cost,” Dan sputtered. “There’s increased security as well. We don’t have to let cleaners in on the weekends.”
Martinez—“Dr. Ahuja, one can also say that the robbery wouldn’t have happened without Raphael’s active participation. Let’s talk failsafes. The commandment chip is not the only one, correct?”
I nodded. “There are others. The most visible one is Raphael’s lack of legs. We deliberately removed the legs from the robot body so as to restrict his movements. As I’ve been experiencing lately, a wheelchair can be quite a constraint on one’s ability to go places. We also embedded a proximity sensor in his body. It cuts off power supply to the motors if it doesn’t detect the office Wi-Fi, effectively paralyzing him if he moves out of range.”
“Neither of which proved a hindrance since he got someone else to take away his brain.”
I paused to look at her. “Are you saying it was an escape attempt, not a robbery?”
She shrugged. “Isn’t it obvious? He must have been planning it for some time. He found out the vulnerabilities in our system and exploited them. Right under your nose, Dr. Ahuja.”
“Raphael did not engineer an escape,” I protested, rather too loudly than I intended. “He had no reason to. The lab was his home. We were his family. He does not know of any other world besides this. Not once has h—”
She pressed on, ignoring my outburst. “And there’s the commandment chip. During our safety procedure reviews, you made us believe that it is an effective failsafe. You told us that Raphael does not have free will and that the chip limits his thoughts and actions to whatever we want them to be. You’ve even testified to the same in court. So what went wrong?”
She was talking about the OARP hearing. About six months ago, Halicom had been dragged to court by a group called the Organization for Advancement of Rights and Personhood. It was one of those animal rights groups. They had filed for the writ of habeas corpus to be applied to Raphael. Their petition, which was ultimately defeated, claimed that Raphael was a person, deserving of the same rights and privileges as a human being, and that his being kept in a lab amounted to unlawful confinement. I had been called in as an expert witness by Halicom’s counsel.
I had always been very cautious in revealing Raphael to the world. When I first announced Mirall’s achievement, it was in the form of an innocuous-sounding research paper titled “Achieving scale factor of HNNs to neocortex levels and concomitant effects on cognition.” My caution wasn’t out of a false sense of modesty or a lack of confidence; I perfectly understood the enormity of what we’d created. Raphael was the Holy Grail of AI. If I had invented the world’s greatest toaster oven or a sustainable fusion reactor, I’m quite sure I wouldn’t have been so reticent about blowing my own trumpet. What held me back was instinct—the kind of instinct a parent has. Raphael wasn’t a toaster oven. He wasn’t some device that I’d assembled. He was a living, thinking entity—a mind, the first of its kind, brought forth into the world without any thought put into what he would make of it. I felt he had to be shielded from it until he could handle himself.
The response to my paper was not at all what I had expected. Despite my attempts to keep a low profile, I knew the news of the discovery would break eventually. I envisioned a lab besieged by calls from reporters and TV stations and universities, and our scientists turned into instant celebrities (we had even finished interviewing for a full-time PR rep to handle the extra workload). To my surprise, the world hardly took notice.
Why would it? Here again was a robot that had supposedly passed the Turing Test. Here again was a company that made tall claims in a culture where hype equals funding. Every time a chatbot fooled some judge with preprogrammed nonsense, or some robot answered predefined questions on TV, the media rushed to bestow true intelligence upon the clueless thing and utter the all too familiar auguries about the dangers of AI. People were bored.
Raphael did become famous eventually, but only after that first TV interview, months after my paper. Our celebrity status didn’t last long though. After Halicom took over, they drew down the hatches: security around the lab was increased, media appearances were prohibited altogether, and we were to publish no more papers until further notice. Naturally, when OARP filed their petition, Halicom went on overdrive to limit the negative publicity the trial would have generated. They got the trial converted to a closed hearing—away from public glare—by citing the need to maintain IP secrecy. Although I cannot be a hundred percent sure, I suspect they must have put some pressure on OARP to make sure they didn’t go talk to the press after the trial. Halicom should have realized that it is hard to contain news like that—especially when OARP themselves had managed to get hold of leaked lab transcripts which they used to make their case. When the news eventually did break, it generated much debate everywhere, but the debates did not translate into any discernible action, either on part of governments or corporations. The world collectively hemmed and hawed on the issue of AI personhood and then moved on to the next hot topic of the day.
“I have no reason to suspect that the commandment chip failed,” I said in reply to Martinez’s accusation.
“Your convictions sound very hollow, Dr. Ahuja. I hope you came here with the understanding that the board expects answers from you, not more evasion.”
“I’m sorry, but I’m a little removed from all this. What is this commandment chip? What does it do?” It was Cynthia Mattice.
“It’s a subconscious region of Raphael’s brain,” I said. “We call it a chip, but it’s just a name. There’s no single chip. Think of it as a set of directives that act as a check on the rest of Raphael’s mind.”
“Like Asimov’s laws,” she said, nodding.
“Not quite. In Asimov’s stories, the robots are aware of the laws. The analogy to Asimov’s laws would be the moral rules and norms we’ve trained him on. Rules such as, do not lie, which he consciously tries to abide by.”
“Don’t you know the laws always break down in the stories?” Troy said.
It never ceased to amaze me how someone like Troy could end up running operations for one of the world’s largest robotics firms. While there were whispers that his success was because of his connections with the original promoters, I never gave them a lot of credence. Yes, the man had the bedside manner of a coroner, and yes, he often displayed an appalling ignorance of detail, but he knew how to get things done. He wouldn’t have survived for so long if he didn’t. Troy was old-school, a brawler in a world of suave thirty-something tech CEOs who had all the looks, the grooming, and the nerd vibe that he would never have.
I answered, “That’s because the laws are a fictional device. They have built-in ambiguity, which is great for generating plots where they can be bypassed in interesting ways. Not that useful for practical applications. The directives are different—subtler. Unlike Asimov’s laws, the directives don’t tell Raphael what to do. They influence behaviors in… other ways.”
“So he has two sets of rules? One he is aware of and one he is not?” Cynthia said.
“Correct.”
“Why?”
“Raphael is not some storefront robot with a narrow, preprogrammed range of behaviors. He has a mind like ours: one that can reason on its own and ignore rules if it wants to. So you have to ask, what keeps him in check? With people, it is factors like reward and punishment—and associated emotions such as guilt. Raphael can’t feel physical pain, which means he cannot be motivated by punishment. The same goes for emotions. He can mimic human behavior so well, we can’t tell whether he is merely emulating learnt behavior or genuinely experiencing something inside. His Buddha-like attitude toward everything doesn’t help clarify things either. Therefore, a hidden set of rules—the directives—to make sure he doesn’t stray.”
“And the directives cannot be skirted by quirks of interpretation?”
“The directives are a collection of logic, mathematical equations, neural weights, learning policies, and so on—you’ll find that they are not very conducive to ambiguity. Furthermore, Raphael has to be aware of their existence in order to interpret them creatively. The directives never enter his workspace; instead they exert their influence from behind the scenes. Think of them as an invisible tether that keeps him leashed within the boundaries drawn out by the more explicit rules.”
“Any directives that prevent Raphael from leaving the lab?” Cynthia said.
“In terms of influencing his behavior and disposition, there must be… I don’t know… dozens. There is one that specifies in terms of GPS coordinates the area he must remain within at all times. There are a bunch of directives, which taken together collectively, make it impossible for him to tell a lie. Then there’s another set that makes him want to share important thoughts—it’s not because of nothing he’s so talkative. Means he cannot harbor secrets. If the thought of escape had crossed his mind, he would have blurted it out to someone.”
“I don’t understand. How can you even enforce a rule that says do not lie?” Cynthia said.
“By encoding the directives not as commandments, but as beliefs. The reason is simple: one can just ignore what one is told to do, but it’s a lot harder to ignore deep-rooted beliefs. For example, you believe that you will die if you jump off a cliff. You know it for a fact. How do you know? Did you jump off a cliff to see if it is true? Did you watch someone jump and die? Maybe you deduced it. Or is it something more innate? How does a gazelle know not to jump off cliffs? No one told the animal. The fact is, we all have certain foundational beliefs on which the rest of our knowledge is built. Instinctive knowledge that we don’t really think about or question very often. Similarly, the beliefs encoded by the directives are foundational to Raphael’s mind. While he’s been told not to lie, deep down inside, he believes he is incapable of lying. Lying to him would be the mental equivalent of jumping off a cliff.”
“And what if he decides to jump anyway? For the thrill of it.” Troy said.
“Right. With us, even foundational beliefs are never sacrosanct. Sometimes we discover facts that are contrary to our beliefs. Or we might discover that a belief has exceptions. Maybe I can jump off a cliff if there is water below to break my fall. Maybe I just discovered I am Superman. So we either qualify the belief, or get rid of it altogether. Raphael cannot. If he learns something that runs contradictory to his core beliefs, he will discard the fact rather than the belief. We do this by making the directives read-only. He can change all beliefs except the ones encoded by the directives. Those are inviolable, and can be changed only by us.”
“Clever,” Cynthia said approvingly. “Doesn’t it lead to problems though? There is a limit to which you can deny the truth, isn’t it?”
She was talking about cognitive dissonance. I hadn’t interacted with Cynthia all that much, but I was starting to realize there was a lot going on underneath that dull, unassuming demeanor of hers. “We rarely encode facts about the world as foundational beliefs. A majority of the beliefs in the commandment chip are to do with Raphael himself. When the beliefs are about yourself, you can change your behavior rather than change the beliefs. Take the GPS directive. If we tell him that he is allowed to go outside the building—and we have tested this many times—he makes up reasons not to. He’ll say his batteries are running low and he needs to be near an outlet. Or that he is busy and will go later. He might say the prospect of going outside doesn’t interest him, even though at other times he is most curious about the outside world. There is something in him telling him that he cannot go outside, but because he cannot question or change it, he compensates by making stuff up.”
Martinez cleared her throat. “It all sounds clever, but there’s a big flaw in Dr. Ahuja’s design. The system holds as long as Raphael doesn’t know about the existence of the directives. The moment he finds out that he is secretly being manipulated, he will want to get rid of them.”
I narrowed my eyes at her. This was an easy one to bat off—surely she knew that? I shrugged and said, “Let’s say he did find out about the chip. He’ll still not be able to do anything about it because of the First Directive.”
“The First Directive?” Cynthia said.
“A belief that in effect says, I cannot modify, add, or delete directives. The first directive is triggered if he finds out about the existence of the commandment chip. As the name suggests, it overrides all other directives.” There were skeptical looks all around. “You don’t have to take my word for it,” I said. “Last year, a team of Halicom’s best programmers stress-tested the directives on a simulator. They couldn’t find any flaws.” I tried not to sound smug as I added, “Would you like a copy of the report?”
“It’s one thing to assess something on paper, another to see it play out in real life. It’s possible they could have overlooked something,” Martinez said halfheartedly, as if she just realized she had run out of ammunition.
This is too easy. It was right then the alarm bells started ringing in my head. It was the twitch of her mouth that gave it away—a certain haughty assuredness lurking underneath all that self-control… She isn’t trying to find a flaw in the containment measures, you idiot. She already knows what you are going to say in your defense. She is leading you on.
But where?
“Okay. Let’s assume there’s a problem with the logic and Raphael somehow got around the first directive. What then? He still can’t change them. We have a protection mechanism similar to the one for running the boot sequence. The directives are write-protected with encryption, and can be changed only from the outside. Specifically, by physically plugging in the core to the mainframe or to a laptop running the correct software. As with the start-up, he’ll need a smartcard with decryption keys to authorize the changes.”
“Who has access to change the directives?”
“Kathy Schulz. Me. The architects, Eli King and Brendon Powell.”
“You maintain a change history, I suppose?” Troy asked.
“Of course. It’s standard practice.”
“You made any changes recently?”
“Not since we started Titian. The version history is on the mainframe if you want to take a look.”
“Can the version history be tampered with?”
I shook my head. “You are asking if someone could have changed the directives and then deleted the entries in the change history log. The logs come under Dan’s read-only policy—so no.”
I was met with a blank wall of silence. Dan shifted nervously in his chair.
Martinez had more to say. “I spoke to Raphael’s caretakers who are with him at all times. Without raising any eyebrows, of course. I wanted to find out if he’s had access to the outside world somehow, or if he’s been behaving abnormally. They all said he was acting a bit distant and aloof last week—very unlike his usual talkative self. They got the feeling he wanted to be left alone.”
“Did he give a reason?” I asked.
“They said he was taking a crack at some math problem.” She looked down at her notes. “Something called the Hodge Conjecture. They said he spent most of last week buried neck deep in textbooks.”
“I’ve heard about it,” Cynthia said. “It’s one of the big unsolved problems in… algebra, I think.”
“If he was attempting a problem of that magnitude, then it’s hardly surprising that he was not chatty,” I said.
“Perhaps. Do you also know there’s a cash prize of one million dollars to whoever solves it?” Martinez said.
“I didn’t know that. What does that have to do with anything? Are you suggesting he solved the problem and then escaped to collect the money? That’s ridiculous!”
“You made the suggestion Dr. Ahuja, not me,” Martinez shot back at me.
I sighed. “Look, I get it. You want someone to blame. And since I am the one in charge, it has to be me. But you have to come up with something better than speculation. The containment measures for Raphael were more than adequate. They are not the reason we lost the core.”
“Measures and protocols only work as long as people follow them,” Martinez declared. Where are you going with this? “No amount of fool-proofing will help when someone knowingly breaks the rules, especially when that someone is the man who designed them.”
And there it was—the trap Martinez had been herding me into all along. My palms turned sweaty as I ran them over the smooth lacquer of the table. “You have to be a little more specific than that,” I said, hoping that the quiver in my voice was just my imagination.
“I’m talking about Raphael’s visits to your home. You flouted your own rules. You broke containment,” she said, leaning back with a triumphant look.
Exhibit K2
Submitted by Petitioner, The Organization for Advancement of Rights and Personhood, to the State Supreme Court of New York, on the day of xxxxxxxxxxxxxxxxx
Excerpt from lab transcript (certain sections blanked out). Transcript sourced from Mirall Technologies, 27 Woodbine Av., Albany, NY, 12205
