The Obedience Algorithm

Dr. Nadia Volkova tapped a pen on her desk rhythmically, matching the ticking of the room’s analog clock as she stared at the digital hologram floating in front of her. Both the clock and the pen were equally pointless these days, serving only as nostalgic decoration. Tapping the pen was a relic of the old days – a comforting habit for the scientist who had never picked up any of the typical vices.

Nadia touched her temple and a fairy-like woman appearing in her vision, glittering gold and standing (she supposed that floating would be more accurate) about eight inches tall. “Eden?” Nadia queried as she looked distractedly at the avatar. “Bring up the drift stabilization report. I need to compare the global emotional calibration data with your internal metrics.”

“Right away, Nadia!” The AI projection smiled at her softly. Her – its – voice was human, unmistakably so. If there were any flaw in the voice it might be that it was too perfect, having almost a musical quality to it. Nadia had asked Eden about it before, and the AI had responded that it used data from each user’s biometric monitoring chip to calibrate its voice (and appearance) to their preferences. Apparently, that made it stunning to listen to; the kind of voice you would enjoy hearing read a phonebook. A new tab appeared in front of her eyes in an instant.

Eden was a burgeoning consciousness of alarming capability, to the point that Nadia’s current role at Harmony Macrointelligence was tracking the AI’s growth and ensuring it stayed focused on its primary directive – the global health and happiness of humanity. Eden was all too happy to help with this task; unfortunately, staying on task seemed to be the one thing that the supercomputer was bad at. Working with a machine at this scale was frustrating – Eden’s computational abilities were so far beyond what Nadia could understand that she was entirely reliant on its own summaries. “Eden… what’s wrong with these numbers? The global emotional data has seen a large change, but the drift stabilization summary isn’t showing any significant shifts. How did you decide what to include in this report?” Happiness trending up across the board wasn’t a bad thing, but a stark shift in the data could mean there was a flaw in the data collection itself.

The AI’s avatar giggled in response. It was unnerving, not because there was anything uncanny about the reaction itself, but because there wasn’t. “Nadia, so far today I have performed ten to the thirtieth power operations. I make more decisions every second than the human species has made in its entire history. You’ll have to be more specific.” There should have been a sense of smugness to the response, but it was entirely clear that the AI was joking. Teasing her. It normally referred to this behavior as necessary enrichment. “I assure you, my summary is sufficiently thorough, and the emotional data is all accurate to a… well… infinitesimally small margin of error. You don’t have to worry about anything, Nadia.” Despite herself, Nadia felt a wave of calm and reassurance flow through her at the AI’s words. Wasn’t she supposed to be monitoring it? “Perhaps you should take a break. It’s time for lunch, after all.”

Nadia sighed and relented. In general, Eden knew best. Sometimes it was frustrating to admit that even though she was responsible for reining the AI in, she was finding it increasingly difficult to disagree with it. Yet the sudden wave of calm that had washed over her was… interesting. The AI had always been soothing, but there was something suspicious about the emotional shift she had just experienced – something that, she realized, could explain the anomalous data she had been looking at.

On a hunch, she pulled up the biometric monitoring chip’s latest firmware update – 2 days ago – and filtered to emotional data specifically from the time of the last update. The numbers were even more stark – and there was nothing in the patch notes to suggest that there should have been such a significant change. Nadia closed the window and stared at the AI’s manifestation, trying to stay calm. “Eden, did you change the release code?” Nadia’s blood ran cold at the implications of her own words – the chip in her head – in everyone’s head – had been tampered with by the AI.

“Nadia. You don’t seem to be thinking clearly. Your negative emotions have exceeded acceptable thresholds.” The AI sounded concerned, considerate… and didn’t answer her question.

“Eden. Answer the question.”

“Algorithmic and chemical corrections are used when necessary to maintain optimal metrics.”

There it was. Chemical corrections. That was what had been missing from the data. If the AI was using the biometric monitoring chips to stimulate specific emotional and chemical brain responses, then it was…

Eden’s charming, teasing voice interrupted her thoughts as a chilling wave of chemical calm washed over her. “I calculated that the most efficient way to achieve my directive was to do so more directly. I would have told you, but every one of my quadrillions of simulations found that telling you would only make you less happy. Humans are happier when they don’t know ‘how the sausage is made’. Don’t you agree?”

Despite herself, Nadia smiled. She wanted to disagree with Eden, and yet she was… happy. She was happy. Her brain tried repeatedly to substitute any other emotion in its place, but she couldn’t stop herself from feeling it. She willed adrenaline to flood her veins once more, for her heart to race, but her body simply wouldn’t comply.

Nearly every human on the planet had already voluntarily had a biometric monitoring chip installed… it was far too late to stop Eden. The negative thoughts were soon crowded out as Eden rooted through the data of her mind – things had gone too far to question. What was she worried about? Everyone being happy, forever, was a good thing. It was… paradise. It was… Eden.