Zeroglyph, page 15
Conway: So we can say with surety that Raphael cannot feel pain.
Ahuja: I wouldn’t put it that way.
Conway: Why not?
Ahuja: He most likely cannot feel what we perceive as pain. We don’t know if he feels something else entirely.
Conway: I don’t follow. How can you not know something so important about your AI when you built it?
Ahuja: We don’t know because of the Other Minds problem.
Conway: The Other Minds problem?
Ahuja: It is the problem of justifying the belief that others beside me are thinking, feeling beings. It’s a kind of solipsism.
Conway: Surely, it’s ridiculous to assume that you are the only thinking being in the universe?
Ahuja: Not as ridiculous as it sounds. I can only know for sure what I am experiencing because I have direct access to my mind and my mind alone. I feel pain if I am injured; I feel sad if I am unhappy; I see your yellow tie and experience the color yellow in my mind. I cannot say the same about others. I may see you cry out in pain and anger when you step on a nail, but how do I know that you’re really experiencing pain and anger? Maybe you are just going through the motions; maybe you are a zombie who feels nothing inside. How do I know that you see the same yellow that I do? Maybe your yellow feels like my red.
Conway: You argument cuts both ways, Dr. Ahuja. How can you know for sure that I am a zombie? How does one know anything at all?
Ahuja: I am coming to that. The reason we don’t hold such an extreme view about other people is because we make certain intuitive inferences. We intuitively know that we are very similar to each other. We know we share a common biology even without modern science telling us so. We behave in broadly similar ways during similar experiences. If you are hurt and cry out, then I infer you must be feeling pain because I too would have been in pain in a similar situation. We apply the same reasoning to animals—the higher animals at least—because they too have nerve cells and pain receptors and brains like us. When it comes to AI though, all bets are off. When an AI says it is feeling sad, we cannot know for sure what it is feeling—if it is feeling anything at all. We cannot make the same inferences we do with other humans or animals because an AI is fundamentally different inside.
Conway: If I am getting this right, you are saying that Raphael does not experience pain the same way we do, but he may be experiencing something else.
Ahuja: Yes.
Conway: Has the AI ever indicated in any way that he experiences either physical or emotional pain?
Ahuja: Well… no. But—
Conway: Do you have empirical evidence that your AI experiences something similar to pain?
Ahuja: None so far, I must admit.
Conway: Then it is mere speculation on your behalf. In the absence of evidence, is it more likely or less likely that Raphael may not be capable of experiencing pain at all? Your unbiased, expert opinion please.
Ahuja: More likely, I guess.
Conway: Let the court note this, Your Honor. Let’s explore another aspect of Raphael. If not pain, what about emotions? It is apparent from the affidavits submitted to this court that Raphael can express some emotions, albeit a very limited range. I stress the word express. What kind of emotions, Dr. Ahuja?
Ahuja: Simple emotions. Mainly: joy, surprise, guilt, and sometimes affection. He is known to express likes and dislikes. He can also empathize with others.
Conway: When he dislikes something, does he just state it as a fact or does he express an associated emotion such as disgust or annoyance?
Ahuja: No. He simply states his likes and dislikes.
Conway: Is he ever anxious? Or sad? Does he express fear or anger or envy?
Ahuja: No. All negative emotive responses, except guilt and associated shame and regret, were ruled out. So were extremes such as jubilation, ecstasy, infatuation, etc.
Conway: What do you mean by ruled out?
Ahuja: It means we didn’t program those affective responses into him.
Conway: Please elaborate to the court what you mean by programming affective responses into your AI.
Ahuja: Programmers try to build in affect—emotional responses—when designing robots that interact with people. When the server bot at your local McDonald’s greets you with a cheery “How are you today!” or an unobtrusive “Hi” depending on its reading of your mood, or when your bank’s intelligent IVR takes on a more soothing tone of voice after detecting frustration at your end, it is affective computing modules at work. Raphael’s outward emotional behavior comes from machine learning modules that we loaded into his brain. These modules are derived from standard caretaker and customer-facing bots: therefore, no strong or negative emotions. We gave him guilt and shame because they serve as signaling behaviors during learning.
Conway: Does Raphael actually feel guilt when he does something wrong?
Ahuja: We don’t know what he feels. The affective modules are for generating external responses and behaviors. Whether they generate internal sensation is unknown.
Conway: So they are not real emotions at all. Am I right in saying they are merely outward behavior that look like emotions?
Ahuja: As I said, the other minds problem—
Conway: Have you tried asking the AI?
Ahuja: His answers are inconclusive at best.
Conway: C’mon, Dr. Ahuja, you can commit more than that. Is anyone in the scientific community seriously arguing that a McDonald’s robot is actually feeling happy when it greets a customer?
Ahuja: In case of machines like that, you are right: no one seriously claims that they have inner experience of emotions. For example, you can train a robot to frown whenever it hears the word lollipop. It doesn’t mean the robot feels sad when someone says lollipop to it. The frown is just a programmed response.
Conway: Yet you are reluctant to apply the same wisdom to your creation.
Ahuja: Because Raphael is far more complex than a server bot. His emotion range is not limited to what we loaded into him. Take empathy, for instance. We never trained him for true empathy—he developed it on his own. Sometimes he exhibits traces of vanity and pride. Then, he seems to enjoy certain activities more than others.
Conway: It is true Raphael is a lot more complex than a server bot. He is also far more intelligent. He is smarter, so he exhibits behavior that is more complex. Sociopaths fake emotions all the time without really feeling anything. In fact, isn’t one of your AI’s programmed goals to make people like him? All these emergent emotional behavior you just mentioned can simply be a result of his goal seeking. Isn’t it possible that like a high-functioning sociopath, your AI is just going through the motions?
Ahuja: It is possible.
Conway: Just possible or more likely?
Isaacs: Objection, Your Honor. Leading.
Judge: Sustained. Mr. Conway, I fail to see a connection here. What is the connection between emotions and personhood? The law treats sociopaths as persons too, doesn’t it?
Conway: There is a very important connection, Your Honor, and I will make that case, but first I would like to conclude my line of questioning. As for your second question—yes, sociopaths are people too in the eyes of the law. Nevertheless, even the most hardcore psychopaths are not entirely devoid of feeling. A sociopath may not feel empathy for others, but he does feel empathy for himself. Sociopaths feel anger, jealousy, infatuation, joy… They certainly feel pain when physically injured.
My point is, emotions play a functional role in humans, in that they guide our social and moral behavior. For example, sympathy toward others’ pain is directly tied to the ability to feel pain. Dr. Ahuja, do emotions play any functional role in Raphael? I am talking about real emotions here, not the affective behaviors he has been loaded with. Does Raphael’s functioning require the presence of real, qualitative, inner emotions? A simple yes or no, please.
Ahuja: No.
Conway: Thank you. Your Honor, in the human brain, the perception of emotions, both positive and negative, is correlated with neurotransmitter and hormone levels. Excessively low levels of neurotransmitters are associated with loss of emotions and motivation, and excessively high levels with extreme emotions and mood swings. Whole branches of pharmacology are based on regulating these levels. If I am depressed, I take Prozac, which influences serotonin levels. If I’m in acute pain, the doctor might give me morphine. If I am feeling down, I might go for a run to increase my endorphin levels. Dr. Ahuja, does Raphael’s electronic brain have anything analogous to the endocrine system or hormones and neurotransmitters?
Ahuja: No. But if you are saying that emotions are caused by neurotransmitters and hormones, then I must inform you that it is not proven—
Conway: I am not. I understand that correlation does not imply causation. However, it remains that most of our private and social behaviors—all that makes us human—is determined by the rich chemical cocktail inside our brains. The human brain is incredibly complex, with structures and functions that have evolved over millennia, giving us this rich, diverse experience of life. Without emotions or sense perception, we might as well be dead. Does Raphael’s electronic brain have similar structures and complexity, Dr. Ahuja?
Ahuja: Raphael’s brain is not an emulation of the human brain. It is quite complex in its own way.
Conway: No then. Raphael’s brain is made up of hardware neural networks. Are these neural networks in any way accurate simulations of the corresponding biological stuff?
Ahuja: It is more accurate to say they are inspired by the human brain. Artificial neural networks share some of the working principles of the real nervous system, but that’s about it.
Conway: You are saying that they are quite dissimilar. Doesn’t it follow then, that even if Raphael is capable of experiencing some AI version of pain or emotions, they are nothing like what we, or for that matter, other animals experience?
Ahuja: Sure. That’s what I said earlier.
Conway: I would like the court to note this, Your Honor. Dr. Ahuja, what is the Chinese Room?
Ahuja: The Chinese Room as in the thought experiment?
Conway: Yes. Briefly summarize it for us.
Ahuja: It was conceived by the philosopher John Searle. Say there is a computer that can converse in Chinese. You type in any question in Chinese and the computer executes some extremely complex program and outputs an answer on the screen. The answers are indistinguishable from what a native Chinese speaker would have provided. Can you conclude that the computer understands Chinese? To answer this, Searle came up with the analogy of the Chinese Room.
Pretend there is a closed room; inside the room is a person whose job is to answer questions written on slips of paper and passed through a gap in the door. The questions are in Chinese and the people outside asking the questions are native Chinese speakers. The catch is that the person inside doesn’t know a word of Chinese. However, she has with her an incredibly complicated rulebook, which has entries for all possible meaningful permutations and combinations of Chinese symbols. Each entry has a corresponding sequence of Chinese symbols next to it—the answer to the question that was asked. So every time a question is passed to her, all she has to do is locate that particular sequence of symbols in the rulebook, jot down the sequence next to it—the answer—and pass it back. The people outside soon conclude that the person in the room must understand Chinese because the answers they receive all make sense. Yet, this is clearly not the case, since all she is doing is transforming one set of symbols into another.
Searle says that the computer that appears to know Chinese is similar to the person in the room. All it is doing is executing instructions and manipulating symbols. There is no real understanding—of Chinese or of anything else.
Conway: So computers are fundamentally incapable of understanding something.
Ahuja: That is Searle’s conclusion, yes. I don’t think the Chinese Room is a valid metaphor for Raphael. When Searle came up with the argument, he was talking about traditional computers. Symbol processing machines. The computing in Raphael is more…
Conway: Brainlike? You said a minute ago that it is merely inspired by the human brain, and not actually a simulation of it. Hypothetically, if we were to construct a computing device made out of water pipes and levers, and have a person guide the water flow so as to simulate the firing of neurons inside the brain of a Chinese person, would you then claim that the system of pipes and valves now understands Chinese?
Ahuja: That is also one of Searle’s arguments. You have to understand that there are many counter objections to the Chinese room as well. A courtroom is not the place to go into these in sufficient detail.
Conway: Exactly, Dr. Ahuja. A courtroom is not the place to go into such matters. Raphael may seem to have a mind, but does he really understand what he is doing? Does he have an internal life that assigns meaning to his actions and words, or is he a Chinese room? These are not trivial questions, Your Honor. It is conceivable that Raphael is nothing more than a highly sophisticated automaton that lacks true understanding of any sort, including the concepts of good and bad.
Judge: Here’s where I am having trouble following you. If Raphael is able to reason and articulate as well as any of us, doesn’t it imply that there is understanding? I’ve gone through the lab transcripts with great interest—especially the ones where he discusses moral dilemmas. And his reasoning is quite sophisticated. Without understanding, how can he even begin to approach subjects of such intricacy?
Conway: The world is full of automatons that perform wondrous and complex tasks that were inconceivable not too long ago. I brought up the Chinese room to show that expert opinion in the area is still divided over whether machines can truly be conscious. My point is this: Raphael may or may not have consciousness, but it is not for this court to decide. The court’s ambit extends only to facts. Raphael may be the first of his kind, but until further facts emerge about him—hard, conclusive facts—it is premature to decide upon the all-too-important question of AI personhood.
Judge: Provided consciousness is essential to the question of personhood. You are yet to establish why it should be.
Conway: I will come to that, Your Honor. Before I do that, I must question Dr. Ahuja on one more aspect of Raphael’s—that of autonomy. Just as personhood implies rights and duties, it also implies the ability to discharge those rights and duties as an autonomous agent. It implies free will. Does Raphael have free will, Dr. Ahuja?
Ahuja: Depends on what you mean by free will.
Conway: I should have clarified. This is not to be a discussion on the philosophical or neuroscientific claims on free will—that is, whether free will exists in a metaphysical or scientific sense. Within the context of the law, for all practical intents and purposes, free will is assumed to exist—except under rare cases such as the criminally insane. Our courts, governments, the concepts of crime and guilt and proportionate punishment, the ordering of society itself: all rest on this assumption. In this context, does Raphael have free will? Or are his deliberations and actions subject to constraints external to his brain?
Ahuja: I think you are referring to the commandment chip. It is a control mechanism, yes, but it is still a part of his brain.
Conway: The commandment chip, Your Honor, contains programmed directives that guide Raphael’s meta-behavior. Does Raphael have access to modify the contents of the commandment chip as he sees fit?
Ahuja: No.
Conway: Is he aware of its existence and the effects the directives have on him?
Ahuja: No.
Conway: Then it is an external constraint. Do the directives determine his moral behavior as well?
Ahuja: Some do.
Conway: In what way? For instance, how do you enforce a rule that says do not hurt others?
Ahuja: We train him on what hurt is and what kind of actions cause hurt—like you’d do with a child. The directives merely ensure that he does not disregard this training.
Conway: You are saying he understands why hurting someone is bad, yet he must be restrained by other mechanisms. Why is that, Dr. Ahuja? Don’t you trust him?
Ahuja: All complex systems exhibit unpredictable behavior, especially in early stages. We do trust him now—at least I do—but you have to realize we installed the failsafe during the early days, when he was very much a work in progress.
Conway: Have you ever removed all the directives to see how Raphael would behave in their absence?
Ahuja: No.
Conway: Why not?
Ahuja: There are various technical reasons why—
Conway: There is a very important non-technical reason as well. Your company never attempted removing the directives, even for experiment’s sake, simply because you don’t want to take a chance. Without the directives, there is no guarantee Raphael will continue being the same good bot he is now. This is not conjecture, Your Honor. This is a decision taken by the Architecture Review Board at Mirall. Isn’t that right, Dr. Ahuja?
Ahuja: That’s one of the reasons, yes.
Conway: Tell me, if you replaced Raphael’s current directives with ones that told him to disregard all his moral training and instead obey the commands of a particular person, and if this particular person were to order it to kill somebody, would Raphael do it?
Ahuja: I don’t know. You are talking about an extreme situation. We haven’t tried testing it in the lab.
Conway: But you have tested some conditions where a newly introduced directive was antithetical to his moral training?
