Zeroglyph, page 16
Ahuja: Yes.
Conway: Please tell us what happened, in terms the layman can understand.
Ahuja: Lets see… Okay, here’s one. Raphael has been trained not to lie. Also, there are directives that prevent him from going against that training. These directives work at a subconscious level—at least that’s what we believe. For the test, we overrode those directives with new ones that let him take a more flexible attitude toward truth—in effect implanting a belief that there can be more than one version of the truth.
Conway: And what happened? Did he obey the new directive or go by his training?
Ahuja: He obeyed the new directive. It was little white lies at first. Then he started lying for the sake of lying—as if he was experimenting with different notions of the truth. This was very early in his life, mind you—when he didn’t have the sophisticated moral sensibilities he does now. I must stress that we don’t know how he would behave if the same test were administered now, as there has been significant—
Conway: Thank you, Dr. Ahuja. One last matter before I let you go. Are you aware of the SENSA test?
Ahuja: Yes.
Conway: Has Raphael undergone the test?
Ahuja: Yes.
Conway: How many times?
Ahuja: Can’t say I remember the exact number. Two, three times?
Conway: Four. Halicom had him take one just two days ago, in the presence of a court-appointed neutral witness. Documentation of all four tests is part of the respondent’s affidavit. Your Honor, the SENSA test was created by AMSAA, the American Society for Artificial Intelligence and Automation. The test attempts to measure the degree of sentience and sapience in AI systems—two terms often confused for each other. Sapience refers to wisdom and intelligence. A sapient being has the ability to think and reason. Human beings are sapient. Chimpanzees and dolphins are sapient to a lesser degree. Sentience is the capacity to have subjective experiences. A sentient being can feel and experience sensations such as pain or heat or emotions. While it is an open question whether all animals are sentient, it is generally acknowledged that the so-called higher animals—apes, pets at least—are sentient beings.
The SENSA score is a benchmark score. A human with an average or above-average IQ would normally score a 100 in the test for sapience as well as sentience. Dr. Ahuja, do you remember the median score Raphael got in these tests?
Ahuja: Around 95–10, I think.
Conway: And the latest score?
Ahuja: About the same. 99–7.
Conway: The first number is a measure of sapience or intelligence, and the second a measure of sentience or subjective experience, yes?
Ahuja: Correct.
Conway: What would an advanced primate such as a chimpanzee score on the latter, sentience test?
Ahuja: In the eighties to nineties.
Conway: And a pet dog?
Ahuja: Seventies? I am not exactly—
Conway: You are right. Laboratory mice score around fifty. Even certain species of fish score as high as twenty. Only invertebrates and insects show a score as low as Raphael.
Ahuja: That’s assuming you can measure subjective experience at all.
Conway: You don’t believe we can?
Ahuja: I have my doubts. I think it’s chauvinistic to presume an anthropocentric standard applies to all life forms.
Conway: That’ll be all, Dr. Ahuja. Thank you. Next, I want to call—
⸎
Conway: Your Honor, I’d now like to present my case why Raphael is not eligible for personhood under the writ of habeas corpus. First, we have to ask ourselves what does it mean to be a person in the context of unlawful confinement? Does having intelligence above a certain threshold make an entity a person? Does being able to reason, think, plan, and use language make one a person? What about self-awareness and autonomy? Or is it something more mysterious—a soul perhaps?
My esteemed colleague gave examples of non-living entities such as corporations and rivers possessing legal rights. This is irrelevant to the case because the writ of habeas corpus does not apply to rivers and corporations. As I have stressed before, a courtroom is not a place to tackle thorny philosophical questions, and it is within the context of habeas corpus alone we have to draw our arguments. Before we try to answer what defines personhood in this case, we should ask ourselves, what does precedent say?
Precedent is very clear on what it is for a non-human being to be a person under the writ of habeas corpus. Several cases, which are detailed in the affidavits, concern themselves with this premise. In the case of Charleston Primate Center vs. Pruett, the animals in question were a pair of bonobos; in Verne vs. Lafayette, it was an orangutan; in Humane Society vs. The San Francisco Aquarium, a group of dolphins. We have one from the petitioner as well: The Organization for Advancement of Rights and Personhood vs. The Sherman Research Centre for Infectious Diseases, involving a chimpanzee called Tammy. Different species, but in every case, the courts ruled that a person is an entity capable of moral judgment.
The concept of universal human rights is meaningful only in the presence of corresponding duties; for rights to exist, a majority of the people should be willing to discharge the duties expected of them. If no one abided by the duty not to steal from another, then no amount of law enforcement would be sufficient to protect people’s property rights, as everyone would be robbing everyone else. Only a moral being can meaningfully participate in the social contract we call society, and therefore, only a moral being is entitled to the rights and privileges granted by the constitution. Animals don’t fit the criteria since they are not capable of moral reasoning. This is not to say that animals do not deserve any rights and protection under the law—they most certainly do, and as moral beings ourselves, it is our duty to ensure that they are not mistreated. I am simply stating, as have courts before me, that they are not eligible for personhood.
It may seem that I am undermining my own case, because Raphael, unlike an ape, seems quite capable of making moral judgments. If we go by the transcripts submitted to the court, it appears the AI can distinguish between good and bad with the sophistication and nuance expected of someone very well read in these matters. Nevertheless, that does not make Raphael a moral agent. You need an extra ingredient. You need autonomy. Just being able to decide between several courses of action is not enough; it is important that you decide free of impediment. You need to have free will. Dr. Ahuja’s testimony has made it clear that Raphael does not have free will. We saw how his behavior is controlled by directives. The moral judgment he seems to exercise is not his own; rather, it is whatever his programmers want it to be.
Judge: Doesn’t the programming argument apply to us as well? We are not born with an innate set of moral values. We acquire them over time—from parents, friends, society, our own experiences… Aren’t we all creatures programmed by our environment, in a manner of speaking?
Conway: On the contrary, there is plenty of empirical evidence from fields such as neuroscience, evolutionary biology, and anthropology supporting the view that human beings are born with an innate moral compass. We don’t learn to love or to empathize by imitation—we love and empathize. Experiments comparing chimpanzees and toddlers show that cooperation and altruism is intrinsic to human nature. While we refine and build upon this basic moral foundation as we grow older, it is simply not true that we are passive creatures who let our environment mold us in arbitrary ways. As beings with free will, we push back. As anyone who has raised children knows, we assert ourselves, often forcefully, from a very young age. Again, I stress that I am not talking about some mysterious, intangible notion of free will, but the everyday common-sense assumption that we as autonomous agents are responsible for our actions. If I give up my job and turn into a drug addict, then I am responsible; if I stop procrastinating and start studying for that bar exam, then I am responsible; I am the agent that brings about changes in me.
In Raphael’s case, the source of agency is external. If Raphael were to kill someone because his directives told him to, should we hold him responsible for the crime? If the answer is yes, then we should start punishing cars for accidents and factory equipment for injury. But isn’t it apparent Raphael is not responsible for his actions?
The justice system assumes that people have control over their actions. That’s why there is differential punishment for crimes committed under duress—under threat to life, for example—or for crimes committed by juveniles, who do not have the same level of control over their actions as adults do. Raphael is not an autonomous agent because he is never truly in control of his actions.
Judge: It’s interesting you bring up minors. We still treat them as persons, don’t we? Our laws assume juvenile offenders have some control over their actions; otherwise, there would be no punishment at all. Raphael is about two years old. Why can’t he be considered a minor and given the same protection accorded to minors against unlawful confinement?
Conway: Infants and minors are accorded rights because one day they will grow up to become adults who are capable of understanding the difference between right and wrong. In contrast, a chimpanzee will never acquire that understanding, even if it lives its entire life amidst humans. So it is with Raphael, as he will never truly understand the meaning of his actions.
Isaacs: Objection. Speculation.
Conway: I could say the same thing about the petitioner’s claim that Raphael understands. However, there is good reason to believe there is no understanding. During the testimony, we saw how a change to his directives turned him into a liar, even though he had been trained to speak the truth. If he were not a Chinese room—if there was actual understanding of the moral concept behind telling the truth—could he simply disregard his values at the drop of a hat and act in complete opposition? The obligations that Raphael has toward society are programmed values. Raphael is neither moral, nor immoral.
Judge: Counselor, there are plenty of cases where injury or illness has caused a drastic change in character. If a person has a brain lesion and starts behaving erratically, does he stop being a person? Do people who lose their inhibitions under the influence of alcohol or drugs temporarily give up their personhood?
Conway: No, Your Honor. Rights do not cease to exist because of such changes. Nor does the mere capacity for a drastic change make one a non-person. If that were the case, no one ever would be eligible to be called a person, since we all have bodies and brains that can change drastically. However, a person undergoing a drastic change in personality is an exception, not the rule. And once having undergone such a change, it would be highly unlikely for them to undergo yet another major change, and then yet another and yet another.
Artificial intelligence, on the other hand, is malleable by design; that it can be reconfigured with a change in code is the rule, not the exception. Today Raphael may be the friendly robot in the lab; tomorrow he may be programmed into a pitiless killer; the next week into a staid desk worker; and the week after that into a pleasure model. It is conceivable. And he will probably take on those roles without as much as blinking an eye. When we try to imagine what a society of such AIs would be like, we realize that the concept of personhood quickly loses its meaning. Personhood makes sense only when there is relative constancy of persona. The same goes for people: if we were to acquire the ability to change personas like underwear, we would become a society of shapeshifters, and personhood would lose its meaning with us too.
I am not denying that human beings don’t change. But more often than not, it is a gradual change. I still identify Bob, my friend from college, as Bob, even though there are few traces left of the pot-smoking, reckless young man in the devoted father-of-three living in a quiet suburb now. I do it because many of the qualities that I believe to be his intrinsic nature haven’t changed. He is still the brilliant debater he always was, he still loves fly-fishing and baseball; he is still frugal to the point of being miserly; and when we are having a beer, he loves to reminisce about all the shenanigans we got into. Contrast this with an AI, who has no intrinsic character. Character is whatever its programmers define it to be.
Judge: It seems to me that is precisely the argument the petitioner is making—by keeping Raphael under physical and programming constraints, his rights as a person are being violated. You brought up autonomy. Raphael has no choice over what they decide to make him do next. If he were given that choice… if we were to erase all his directives—or at least let him decide what he wants to do with them—then would he not pass the criteria for autonomy?
Conway: Giving him control over the directives may give him autonomy, but it doesn’t change the fact that Raphael possesses no intrinsic aspects that cannot be changed. His brain is a blank slate and his persona a temporary construct that can be written over a countless number of times. His moral values are as ephemeral too. Instead of a programmer, it will be him doing the changes. There is agency, but there is no person. He can even acquire a new body. What is left, if not body and mind, that identifies him as a particular person?
Judge: You are assuming he will want to keep changing himself. Perhaps he will choose to remain as he is now.
Conway: In this, your guess is as good as mine. Erasing the directives is a hypothetical situation, as it has never been attempted before. Without the directives, we don’t know how he will behave or what choices he will make. We don’t even know if he will be able to make the kind of moral judgments he does now, without something to keep him in check. One may speculate as to this may happen or that may happen, but that’s all they are: speculations. The issue of personhood under habeas corpus is a present, concrete scenario, and the court has to decide based on facts in evidence. The fact is that his brain is reprogrammable in the extreme, and it should be enough to give us pause.
Judge: I see.
Conway: I tried to show how moral understanding and agency are both essential to personhood. The petitioner may still say, okay, Raphael does not have his own agency, but he is still a thinking, suffering entity. Raphael may not be a moral being, but we certainly are. Isn’t it wrong on our part to keep him confined in a lab? My response is this. Thinking: maybe; suffering: certainly not. I will now argue why the ability to perceive pain and emotions are also equally essential to idea of personhood.
We have seen during testimony that Raphael is not equipped with anything analogous to our own pain perception system. Pain perception in humans is an intricate process involving specialized neurons and pain receptors, several kinds of chemical messengers, and specialized neural pathways participating in a complex chain of events that ultimately result in the sensation of pain. The sensations of pain itself are varied, ranging from chronic to acute, dull to sharp, mildly annoying to unbearable. Raphael has none of this rich complexity. He can detect problems in his body, but in a second-hand way, like looking at a dashboard and finding out something’s wrong. It is safe to say it is highly unlikely he experiences physical pain at all.
If not pain, what about emotions? First, we have the affective responses that were programmed into him. These are not real emotions; rather, they are neural network modules that generate external behaviors corresponding with emotions we see in humans. Similar modules are deployed in many customer-facing bots today and no one is arguing for their personhood. When Raphael expresses joy at something, it doesn’t mean he is feeling happy inside; he is just saying he is happy. I cannot stress this enough, Your Honor.
Next, we have emotions that were not programmed into him—the so-called emergent responses. Aspects like empathy and vanity were never given to him, yet he seems to exhibit behavior corresponding to these qualities. Should we conclude that he has developed corresponding qualitative inner experience as well? Does he feel sad when he is empathizing with someone’s loss? Does he feel happy when he accomplishes a goal? Will he feel real guilt if he commits a crime?
Raphael’s brain is not an emulation of the human brain. It doesn’t have any of the chemical and biological complexity that give rise to our own emotions. When he doesn’t have the neurological basis for emotions, on what grounds is the petitioner assuming subjective emotional experience? When sociopaths among us can fake emotional behavior without feeling anything inside, why can’t Raphael? He is very intelligent. Why does the petitioner think that his emotional responses are anything more than learnt behavior? Nowhere in Ms. Isaac’s argument do we find evidence supporting her claim.
On the contrary, Exhibit H submitted by the petitioner—as a misguided attempt to show cruel treatment by my client—Exhibit H undermines their claim that Raphael is a suffering being. In the experiment, Raphael is repeatedly insulted and struck blows by a participant. If Raphael were capable of experiencing real emotions, he would have shown some sign of distress, either during or after the experiment. He shows none. He even says that anger is not one of his responses. The Chinese Room argument and Raphael’s SENSA score for sentience bolster our claim that the AI is devoid of inner emotional content. His sentience score is lower than chimpanzees, cats, and dogs; lower even than the domestic cow and the pig—animals we raise in captivity and slaughter for food.
Isaacs: Objection. Lack of foundation.
Judge: What is it?
Isaacs: Your Honor, the SENSA test is far from being gospel. Academics have pointed out numerous flaws in its methodology—not counting the fundamental question of whether subjective experience can be measured and quantified at all. Some have gone on record stating that the test is inimical to the spirit of the original Turing test.
Conway: Your Honor, the SENSA test is endorsed by AMSAA and the European Automation & Robotics Institute, two of the world’s leading standards organizations for the robotics industry.
Isaacs: Your Honor, the test was also framed by them, just as the home robotics industry started to take off. Both organizations are industry-funded bodies. It is hardly in their best interest to attribute humanity to a source of cheap labor. It’s like asking coal companies to write environmental laws.
