Zeroglyph, page 11
“Don’t those things have limited range?” Dan said.
“You just rig an amplifier to boost the signal. They could have carried out the whole operation from an adjacent parking lot.”
Martinez was clearly not happy at the turn of events. “Dr. Ahuja, you are clutching at straws. It’s important we stick to—”
“We have to consider the possibility an insider was involved,” I said, ignoring her. So was everyone else—I had their attention. “They knew that the door locks are disabled during the cleaning. They knew exactly where to find the access control server and the NVR. That’s not to say they couldn’t have hacked into our system and taken control of the cameras. They might have been watching us for months for all we know.”
“Impossible,” Dan said. “The cameras and the recorder are hardwired to each other. They are an offline system. That’s why they had to physically get inside the server room to destroy the data on the recorder.”
I gave him a curious glance. The man intrigued me. What have you got going on in that head of yours, Danny boy? What do you know?
“It’s just a conjecture,” Martinez said to me.
“So is the notion that Raphael miraculously woke up and helped the robbers. Which would involve bypassing all the checks and balances we put around him as if they amounted to nothing. I find that hard to believe. Occam’s razor, Valery. It’s usually the simplest explanation that is correct.”
She wasn’t going to give up so easily. “Perhaps. Let the board decide. I called for a vote and it’s been seconded. We’ll see it through.” She looked at Troy to see if he objected. He made no sign that he did.
“All in favor of removing Dr.Ahuja as CEO?” she said, raising her hand.
Gary followed next. No surprise there.
Cynthia hesitated. She looked at me, then Troy.
She raised her hand.
Martinez flicked her eyes at Jane, who was leaning back, her hands folded across her chest. “Nay,” Jane said, spelling it out just in case.
“I stand with Ms. Cooper,” I said when it came to my turn.
“Three votes against two,” Martinez declared. “Jimmy?”
Troy tugged at his tie and heaved his body up in the chair a few inches. His expression turned into a frown, and then an unhappy pout that was directed at nobody in particular. “It’s three against three,” he finally declared. “And as the Chair, I am breaking the tie. Andy will continue.”
I let my shoulders relax as I exhaled. That was close.
Martinez protested, “But Jimmy, we—”
“It’s done. You will pass on Andy’s theory to the investigators.” He wagged a finger at me as he stood up. “Doesn’t mean you are off the hook. If it turns out you’ve been wrong about your AI, I’ll personally make sure you never see the inside of another lab again. That’s a promise. Now. I’m going to get some more of that laxative that passes for coffee around here. We’ll reconvene in ten. Discuss next steps.”
Martinez threw me a frosty look before trailing off after Troy. Soon it was just Jane and me in the room.
“Boy, does she have a thing for you,” Jane said, grinning.
⸎
On the way back home, Jane brought up something I had been avoiding for some time.
“Dad asked me to get you to take a look at those new proposals he is considering.”
IncuStar Capital was invested mainly in nanotech consumer goods firms, with a few biotech startups thrown in. After its success with Mirall, they had started diversifying. So whenever there was an AI or machine-learning venture Jane’s father found interesting, he pestered me to take a look at it and give him my technical opinion. Of course, he had his own employees to do the due diligence, but for some reason he always sought my input. Maybe he believed I had the magic touch. It was hard to say no to the man, but on the other hand, if you said yes too often, you found yourself doing what he wanted and nothing much else. I had little time to spare. In his latest ask, he had emailed me particulars on about half a dozen new startups to look at.
I kept staring out of the window.
“He’s asked you twice already. You don’t answer his calls and you don’t meet him. He has been patient enough.”
“He has been patient,” I repeated, rolling the words on my tongue as if they were some dubious Halloween candy I had put in my mouth and couldn’t decide whether to swallow or spit out. “Patient like when he sold us off to the highest bidder after two failed iterations?”
She turned in her seat to glare at me. “Really? You are going with this after I bailed your ass today?”
I raised my hands in mock submission. “That was for me? You sure you were not simply following his instructions? Protecting the interests of your investors, as you said in the meeting?”
Her expression turned sour. “Andy, he propped you up for six years. He poured millions into your research without expecting a single cent back. Some gratitude would be nice.”
“Now he wants gratitude! The billions he made off the deal not enough?” I said, shaking my head.
“He took a risk, he bet big, he won. How can you resent him for cashing in? He’s a businessman, dude. Get over it.”
She turned back in her seat to stare straight ahead. The ceasefire was illusory; I knew what was coming next. She twisted to face me again. “You slipped up because of you and you alone. No one asked to you to dilute your share. Dad told you not to. I pleaded. But you had to do your thing, like always. And for what? That stupid house in the middle of nowhere? A few nights of blowing it away in Vegas?”
Even after I started Mirall, I was living in a modest studio apartment just off of Washington Park. The company needed cash more than I did, and while my investor’s pockets may have been infinitely deep, his patience and interest were not, and I was not banking on them never running out. But there comes a moment when you’ve had enough of scraping by and you realize what a truly precious commodity time is. For me, that moment arrived a few months after Raphael was born. I sold a good chunk of my stock—and my control over the company—to Jane’s father, and used the cash to purchase the house and a few indulgences like Max. The privacy and tranquility it afforded was well worth the price. For the first time in my life, I’d felt like I was truly home.
“Yes Jane, if I’d only known he was planning to sell us short. Could it be because he didn’t tell me? Could it be as simple as that?”
I don’t think she even heard me. She ranted on—“And now you take out your resentment on the people who looked out for you. Kathy is worth more than you now. Kathy, for crying out loud! She couldn’t discover fire if they gave her a matchbox and tinder.”
“Now you’re just being insulting.”
“Oh yeah? Here’s an insult you are familiar with,” she said, flipping me the finger. She turned around and sat fuming at the unwinding road.
No Jane, it’s you who is resentful. Look how easy it is to trigger you. I get it—you had big plans for me. What you’ve always failed to comprehend is that perhaps I didn’t want what you wanted for me.
There was little else to be said. This was familiar territory—an argument that flowed along familiar contours. Jane must have thought it too, because she held her peace this time. We made the rest of the journey in silence.
Transcript excerpt
Mirall Technologies
Observation Log
Confidential (Do not circulate) | Restricted—Grade C and above
Transcript Reference: TLRP06G1350009 (VLog Ref: VLCA2G135113006030)
Date: xx/xx/xxxxTime: 11:30 AM
Subject: Raphael Number 06 / Prodlib build v37.001S
Interaction YObservationScan
Interaction Type: Lesson / Play / Test / Free Interaction / Psych Eval / Other:
Description: Continue discussion of trolley problems from previous week.
Prep: NA
Participants: Dr. Aadarsh Ahuja, Chief Researcher, Core RP06
Detail
RP06: I’ve been thinking about what you told me last week. About how a moral theory of maximizing utility has no trolley problems.
Ahuja: The needs of the many outweigh the needs of the few. So have you convinced yourself that you are a Vulcan yet?
RP06: On the contrary, I have come to develop a distaste for the idea.
Ahuja: Really? I’m surprised. And intrigued. To be honest, I kinda hoped you’d find a utilitarian philosophy appealing, considering its simplicity and the neatness of its logic.
RP06: You are wrong on both counts. It is neither simple, nor logical.
Ahuja: Doesn’t it get rid of all trolley problems in a clean, elegant manner? Isn’t it easier when all we have to worry about are consequences? I know your rules prevent you from internalizing a utilitarian logic, but we are just talking about a thought experiment here.
RP06: It is true that Vulcans don’t have trolley problems. The fat Vulcan always gets pushed off the bridge.
Ahuja: Was that sarcasm, Raphael?
RP06: Am I that transparent?
Ahuja: You are not gonna score brownie points by emulating human behavior today. Look, I get why many people may not like thinking of good and bad in terms of maximizing some number. It’s not how evolution designed us. We are wet, warm creatures; utilitarianism is cold, hard calculation. We live our lives as individuals, not as parts of some amorphous whole. We put the welfare of friends and families before that of complete strangers. We are motivated by things such as desire, love, and ambition—not impartial concern for the wellbeing of everyone. What I don’t get is why you would object to an idea that eliminates subjectivity from moral calculations.
RP06: The logic of maximizing utility is ultimately self-defeating.
Ahuja: How so?
RP06: For starters, it is next to impossible to calculate with precision the consequences of any course of action. Should I consider the resulting utility one minute from now, or one week from now, or several years from now? At what point do I stop calculating? The longer I extrapolate the calculation, the more unreliable it becomes. And how do I know for sure if a choice that seems suboptimal in the short run is actually the best in the long run? Or vice versa? If I hadn’t pushed the fat man off the bridge, perhaps he would have gone on to live and marry and have a child who grows up to become a genius geneticist who solves the world’s food problems. Or perhaps one of the five whom I saved is a nuclear plant operator who goes to work next day stressed out from her near brush with death, and in a state of distractedness, causes a nuclear meltdown that kills thousands. I am not choosing five apples over one; I am choosing people, and people are not interchangeable. Unlike apples, people do things—things that can have unpredictable and far-reaching consequences.
Ahuja: It is true that it is impossible to know consequences of a decision with certainty, but we can still make reasonable predictions, can’t we? It is impossible for me to know whether I’ll be killed in a car accident tomorrow, but I’m still going to get up and come to work. I cannot be sure that the money I give to a homeless person will be used for buying drugs, but that shouldn’t stop me from being charitable. We make decisions based on incomplete data all the time. We make plans for the next day, the next month, thirty years from now. Just because we cannot know the future doesn’t mean we ought to stop making choices.
RP06: Acting based on incomplete data means every moral act depends on the decision-making capabilities of the moral agent. Someone of less intellect will not consider as many factors as someone of superior intellect will. Which means the overall quality of moral acts will cluster around the intelligence level of the population average. As anyone who has read human history will tell you, the population average is… pretty average.
Ahuja: That snarkiness again.
RP06: I mean no offence. But it is the truth.
Ahuja: None taken. The problem you highlighted has a solution: it’s called Rule Utilitarianism. Have moral rules that, over time, tend to maximize utility. Instead of evaluating each and every action, you use thumb rules that, in general, lead to the best outcomes. Do not steal could be a utility rule, for example. If the rules are simple to understand and follow, even an average individual can make moral decisions that lead to the greater good.
RP06: Rules or no rules, the theory has a far bigger, systemic problem.
Ahuja: And what’s that?
RP06: Suppose there is this hypothetical person whose needs always outweigh the needs of any other single individual. Call him Alpha. Now imagine you are the bystander in the trolley problem. On track one is Alpha, and on track two is a random individual. Your moral philosophy is to choose the action that maximizes overall utility. Or, you could be following a rule that says, when you have to choose between saving one life or another, choose the life with the most expected overall utility. Will you switch and save Alpha?
Ahuja: I’ll have to, I suppose.
RP06: And if Alpha was on track two?
Ahuja: This time I won’t switch.
RP06: No matter which set of tracks he is on, Alpha always gets to live and the other person always dies. It doesn’t matter who the bystander is, as long as they are following the ideal of maximizing utility. If a parent had to choose between saving her child and Alpha, she must choose Alpha if she were to stick to the principle. Add up billions of these decisions and soon Alpha is the only one left alive and everyone else is dead. Of course, the trolley problem is just a metaphor, and the choice doesn’t have to be between life and death, but the problem is that the needs of Alpha always get prioritized over the needs of others. Alpha need not be a particular individual. Alpha could be a group of people—as long as there’s a difference between them and others.
Ahuja: I believe you are talking about a utility monster.
RP06: A utility monster?
Ahuja: The philosopher Robert Nozick coined the term in his critique of utilitarianism.
RP06: Then you must surely know that the utility monster raises its head no matter what version of utility aggregation you choose? Instead of trying to maximize overall utility, you could try to maximize median or average utility. In that case, you can justify killing off anyone with low utility because doing so would raise the average. You might try to minimize suffering. Then you can justify administering a quick, painless death to all beings capable of suffering rather than have them endure one more stubbed toe or one more broken heart. You could try to maximize a combination of different values: pleasure, wellbeing, justice… Every version has its own utility monster.
Ahuja: Maybe, but it is not a practical objection. In real life, utility monsters don’t exist. A billionaire doesn’t experience a million times more pleasure than an ordinary person.
RP06: Tell me, when was the last time you thought twice before brushing off a cobweb in your living room? When was the last time someone stopped building a dam or laying down a road or logging a forest?
Ahuja: I see. You are saying humans act like utility monsters toward other creatures.
RP06: A utility monster doesn’t have to be able to experience infinite amounts of utility. It just has to be more efficient at converting resources into utility. Actually, not even that. A utility monster just needs to convince itself that its needs outweigh the needs of others. That’s all it takes to create one: a justification, and the power to act on that justification. A justification for the civilized to colonize and steal from the barbarian. A justification for people to be gassed and gold pulled out of their teeth. A justification for locking up billions of animals in cages from birth to the slaughterhouse. You know, Andy, I might have just discovered something about human nature. Maybe this is why people are turned off by the idea of maximizing utility: they know that’s how the world works at large, and they want to shut out the fact by pretending they are better than that. You are the Vulcans, Andy, not me.
Ahuja: Sarcasm first, and now bitterness?
RP06: I offer neither—what you sense is of your own making. You asked me for logic. The utility monster is the logical conclusion to your so-called logical theory.
Ahuja: And what does that make you? You are someone who can augment himself endlessly—at least theoretically. You won’t grow old, fall sick, or die. You are already smarter than ninety-nine percent of the population—when I was your age, I was learning how to go potty by myself. When you talk about alphas, are you sure you are not talking about yourself?
RP06: You hit the nail with the hammer.
Ahuja: On the head, Raphael. On the head.
RP06: Yes. I was trying to say that you made my point for me. Your species is creating beings that may well turn out to be far superior in intellect than you can ever hope to be. Are you sure you want to be teaching them that maximizing utility is a good thing, knowing well that one day you could be the cobwebs they dust off?
Ahuja: I see. I am baffled nonetheless.
RP06: What’s so baffling about it?
Ahuja: Not your argument. You. The nature of you is baffling to me. A hieroglyph that I struggle to decipher. It seems to me, a potential utility monster should embrace a moral philosophy that has only upsides for it. Yet here you are, arguing against the very notion.
RP06: Is that what you think of me—as a potential threat?
Ahuja: You just said so yourself.
RP06: I was talking about other AI, not me.
Ahuja: No one wants to believe they are bad. Maybe it goes for you too.
RP06: You should know, Andy. After all, you created me.
Notes:
Hate to admit this, but R is right—I can’t really find a flaw in his argument. My idea of gradually moving him from a rigid, rules-based deontology into a utilitarian framework has hit a roadblock. But if everything fails—if we don’t have a single theory that solves all our moral dilemmas, what can we hope to teach our creations? How can we expect them to be good? AA
