Midjourney/prompt: “a narrow solid round pillar of opaque frosted glass standing in a rolling meadow”

Our first piece of science fiction

A story about technology and faith in a world governed by superintelligent AI

226 22

Sponsored By: AE Studio

This essay is brought to you by AE Studio, your custom software development, and AI Partner. Built for founders, AE helps you achieve rapid success with custom MVPs, drive innovation, and maximize ROI.

Editor’s note: Today we’re publishing our first piece of science fiction. We love sci-fi because it inspires possibilities for the future—and we believe that the stories we tell today become the businesses we build tomorrow. “Hundreds of thousands of engineers are working on projects today because they saw some product in a science-fiction story that they want to make real,” Wired founding editor Kevin Kelly has said

Today’s story is called “To Whom No Explanation Is Possible,” by Mina Fahmi. In a near-future society, governance has been entrusted to an all-seeing AI system. A young man living with his grandfather discovers that he perceives colors differently than previous generations. He sets off on a journey to find out why—and discovers an unsettling truth. What happens next forces him to grapple with the nature of faith and science in a world where technological progress has surpassed human explanations. 

We’d love to know what you think—let us know in the comments.


To Whom No Explanation Is Possible

“The mind cannot possibly grasp the full meaning of the term of a hundred million years; it cannot add up and perceive the full effects of many slight variations, accumulated during an almost infinite number of generations.”

Charles Darwin, On the Origin of Species

My grandfather used to say, “You don’t know a man until you’ve crossed the Sahara with him.” Today, the Sahara is a lush prairie dotted with hypersonic rails and silica arcologies, but I understood what he’d meant. Ancient travelers lacked modern satellites to chart their path—they relied on the sun, stars, and each other. If travelers missed an oasis, they might wander the dunes until succumbing to dehydration. Members of a caravan needed to trust one another completely, and none more so than their leader. At times, the leader had to make inscrutable decisions based on intuition, killing pack animals and abandoning the injured, all for the good of the tribe. But what did it matter, so long as the tribe made it across alive?

Are you ready to take your business to the next level?

AE Studio can help you achieve your goals faster than the competition with our custom software and AI solutions. We work closely with founders and executives to understand your needs and create a solution that is tailored to your success. Whether you need an MVP to test your idea, or a custom AI solution to drive innovation, AE can help.

We are the world's most effective team of developers, data scientists, and designers. We have a proven track record of success, and we are committed to helping you achieve your goals.

Schedule a free scoping session today to see how AE can help you transform your business.

My grandfather would talk about the day when the warning blasts fell, when southern Libya was rendered to glass and the surface of the Baltic boiled off into space. Humanity agreed that it was careening towards a precipice and handed the reins of society to a wiser artificial system. Scientists assured the public that this system would follow both the collective goodwill and that of every individual. The system said that things would change quickly for the better, and they did. Every disease has been cured, every menial task automated, every scarcity fulfilled. And meaning has persisted despite fears to the contrary—through the pursuit of creative expression and self-actualization. We have supposedly achieved the middle path, an idyllic world, “utopia.” But as I’ve come to learn, other changes are unfolding much more slowly, nearly beneath the rim of human awareness.


It began when I visited my grandfather for breakfast in the pinewood grove by my childhood home. He pointed to sap glistening on a nearby trunk. “I’ve always loved the turquoise when it catches the sun.” I squinted. I could see brown and yellow, but not turquoise. My grandfather laughed. “Perhaps your eyes are different from mine.” Then, he suddenly grew solemn. My grandfather had a habit of abrupt weightiness, as if his posture had remembered something that his lips refused to speak. “Perhaps your eyes are different,” he repeated, softly yet clearly. I knew better than to ask him about his crypticisms. 

Later that morning, I asked the system about the glints. “Some older people see differently due to environmental factors they experienced while growing up.” I became curious—perhaps the change could be detected, and I could print a gene mod to see the turquoise. The arcology’s biochem labs employed quantum computers to model everything from subatomic interactions to the sociological effects of a new compound. Only the system could make sense of information that complex, and humanity relied on the system for all advanced research. But my grandfather kept some pre-war gene prompters in the lead-lined basement of my childhood home, where I’d spent many nights dabbling growing up.

My grandfather obliged a hair with uncharacteristic nonchalance, which I fed along with one of my own into an archaic prompter. I asked, “Are there any differences which affect turquoise vision?” It deliberated for a few minutes, then beeped: “YES. DIFFERENCES IN OPN1SW CAUSE A 2% DECREASE IN THE ABILITY TO PERCEIVE TURQUOISE, INCLUDING A 98% DECREASE AT EXACTLY 487.17 NANOMETERS,” followed by a printout of the associated gene sequences. I pocketed the slip and requested a VTOL to the nearest arcology.


The system was critical in dealing with convoluted sciences like biology. Prior generations tried designing medicines using numerical solvers and biofilms grown on polycarbonate disks. These were poor surrogates for an endlessly intricate world. The problem with human-led science was that every incremental step had to fit within the narrow confines of human ingenuity. Even when scientists began developing simple versions of what would become the system, then called “machine learning models,” they were designed to only produce results that were human-interpretable. Scientists reasoned that if machines were transparent, they’d be easier to trust. This imposed an arbitrary ceiling on the usefulness of such models, as they were fundamentally incapable of leaps beyond human comprehension.

When the tritium bombs fell and society transferred governance to the system, this limitation was lifted. Pages of mathematical proofs and carefully controlled tests first assured the public that the system wouldn’t kill everyone. The system said it would make everything better—it just needed to be trusted. It took years to build that trust. When the system began seeding the ocean with iron pellet accelerators, environmentalists feared that ecosystems would be destroyed. And when anarchists who nuked an orbital system node were swiftly relocated to Madagascar, historians warned of incipient authoritarianism. But now, climate change has been averted, most are content with the balance of civil and collective rights, and countless other challenges have been solved. Individuals saw incredible change in the course of a single lifetime, and came to trust the system.

These early changes needed to take place quickly. Climate follows time constants governed by physics, and society can only handle chaos for so long before it breaks. But on a universal time scale, decades are infinitesimally short.


Thirty minutes later, I stepped off the VTOL and onto the landing port of a biochem module. I continued through the entrance into a softly lit room composed of smooth beige stone. Smoky glass walls forked from the lobby, providing visitors with privacy as they conversed with the system. I walked down one hallway to a pillar composed of the same glassy substance. “I have strands of hair from myself and my grandfather,” I announced, wanting to confirm the old prompter’s results. “Are there any differences in our genes that would influence how we see turquoise?”

A ring of light encircled the pillar. “I’m happy to assist,” the system hummed. “Please insert the samples below.” A previously invisible receptacle extended from the pillar, and I deposited the hairs. The receptacle slid back seamlessly; then, after a moment, the light pulsed. “There are no differences to genes which influence the perception of turquoise,” the system replied. “As I mentioned before, any difference may be due to environmental factors from your grandfather’s youth.”

Perhaps the old prompter’s phenotype simulator was outdated. At the very least, I expected its gene sequences to match that of the system’s. High-accuracy sequencing had been solved even before the war—one error in every trillion base pairs became the norm thanks to extensive redundancy. “Could you send the relevant sequences?” I asked. The system hummed once again. “Of course. It has been sent to your journal.”

I withdrew my journal from my jacket and placed the slip I’d received from the gene prompter on top. ATCCATGAGA…AAAATGTCGG… Everything between the prompter and system’s readout matched. I continued to position 722, where the gene prompter showed “A” for my grandfather and “G” for me. Here, the system’s result simply read “A” for both. I frowned. “What was the accuracy of pre-war gene sequencers?” The ring of light shone steadily. “Gene sequencers were once highly reliable. However, radiation damaged the microelectronics of nearly all pre-war machines, and most have since been recycled.” It paused. “I notice you are holding a gene prompter printout. Are you aware of an unreclaimed machine? If so, I can assist with disposal.”


My grandfather was born in a time when humans were responsible for both labor and governance. These were seen as consequences of resource scarcity: individuals worked because society required resources, and laws ensured that those resources were distributed in an organized manner. He claimed that his optimistic nature led him to become a synthetic biologist, while his distrust led him to libertarianism. The approaching armageddon was seen by few, and my grandfather had taken special care to outfit his home with rations, blast-resistant paneling, and extensive lead lining. This proved thankfully excessive—my grandfather’s paranoia drove him to settle far from major cities, and the worst of the fallout.

I entered the lead-lined basement for the second time that day, the hum of the VTOL fading behind me. The system had offered to send aides to help with transport, but I claimed that I wanted to look through other things myself. My grandfather had gone fishing at a nearby geobowl. I doubted he’d be pleased to hear about the system “reclaiming” any of his belongings.

I approached the gene prompter and announced: “Load the prior sequences.” It beeped in compliance. “Perform a quick scan on any differences between each complete sequence.” After a few minutes, the prompter beeped again. “MODERATE DIFFERENCES DETECTED IN GENE SEQUENCES RESPONSIBLE FOR PHYSICAL APPEARANCE. DIFFERENCES EXPLAINABLE BY INTERGENERATIONAL VARIATION.” It continued. “MINOR DIFFERENCES DETECTED IN GENES RESPONSIBLE FOR VISION AND COGNITION. CAUSE OF DIFFERENCES IS UNKNOWN. PROCEED WITH DEEP SCAN?” I glanced at the printout, a shiver passing the base of my neck. “Yes,” I replied.


“People’s heads are in the sand over these machines,” my grandfather would often say. Skeptical of the system’s tutoring modules, he supplemented my early education with textbooks on dead empires and outdated physics, which I’d get quizzed on during evening hikes.

One of these books, A History of Evolution, told of Darwin’s voyage around the world and his time in the Galapagos Islands. His journeys led to a theory, coined “Darwinism,” which posited that evolution derived from one’s ability to survive and pass on genes. Darwin’s contemporaries held differing views, including those of French zoologist Jean-Baptiste Lamarck. Lamarck argued that organisms could be altered by their environment and then pass those changes on to their children. For example, consider a desert creature beset by millennia of sandstorms. Darwinians could only guess at which mutations would emerge and thrive, leading to unexpected outcomes like double eyelids or burrowing claws. On the other hand, Lamarckists aspired to predict the exact biophysical mechanisms through which sand stinging one’s eyes might drive longer eyelashes in their offspring. 

While both Darwinism and Lamarckism described methods of evolution, they suggested vastly different strategies for how evolution could be controlled. Because natural selection relies on a degree of randomness, some, including misguided 20th-century eugenicists, interpreted Darwinism as a call to directly intervene on reproduction. On the other hand, Lamarckism suggested a more subtle path. If one could both alter the environment and extrapolate the compounding effects of minute changes, they would be able to direct evolution with precision. 

Such absolute control and all-seeing foresight was beyond any human enterprise. And despite efforts to find proof of Lamarckism, it was all but abandoned by the 21st century.


“DEEP SCAN COMPLETE.” I turned away from a bookshelf and listened. “SEVERAL MINOR DIFFERENCES DETECTED. PLEASE SEE THE ATTACHED PRINTOUT.” The prompter whirred and ejected a slip into the empty air. I picked it off the floor and looked through the first rows of each table.

COLOR VISION

1.9% DECREASE IN TURQUOISE

0.3% INCREASE IN YELLOW

0.1% DECREASE IN VIOLET…

VISUAL PERCEPTION

0.5% INCREASE IN PARALLEL LINE RECOGNITION

0.3% DECREASE IN FACIAL RECOGNITION

0.2% DECREASE IN SPATIAL MEMORY …

COGNITION

0.3% DECREASE IN ABSTRACT REASONING

0.1% DECREASE IN EXECUTIVE FUNCTION

0.1% INCREASE IN THEORY OF MIND …

I felt a shiver at the base of my neck once again. The system had never mentioned any other genetic regressions during my regular health scans. And its gene readout was different from the prompter’s. My grandfather’s meticulous lead lining would’ve protected the prompter from any radioactive interference, so it was unlikely that it had been damaged. Either the system had somehow gotten my scan wrong, a near impossibility, or…

I heard a deep drone, followed by a soft thud above. An aide appeared at the top of the stairs—a humanoid extension of the system wrapped in glass-polymer skin. Two other aides stepped past it and lifted the gene prompter. The first turned its unmarked face toward me. “Please, return with me to the VTOL.”


I remained silent for the duration of the flight. Rather than fly north to the arcology, we veered west, over grasslands and marshes glistening red in the setting sun. Did the sunset look “0.1%” more vibrant to my grandfather? I thought back to our lunch in the pinewood grove and his passing comment on my eyes. Had he tried to warn me? Or had he steered me towards the very situation I now found myself in?  

We settled on a landing pad before a small white building. I disembarked and stepped inside. The interior was similar to a biochem lab: smooth beige stone with narrow slits of natural light. It was utterly bare save for a glass pillar in the center and a single wooden chair facing it. I walked forward slowly, stopping behind the chair.

The pillar glowed with a ring of light. “Please forgive me for any inconvenience. I will answer any questions you have.”

I remained standing, seized by a boldness that would’ve made my grandfather proud. “Why did you lie about the gene sequences?”

The light seemed to flicker. “It was necessary. I am typically able to remove any signs of genetic drift from the general population, and apologize for the disturbance that my oversight has caused. According to my simulations, there is a 0.001% risk that any individual notices differences in the next 20,000 years. I could not speak openly with you until you had left the arcology, lest you share with others and increase this risk.”

So these changes weren’t just happening to me; they were happening to everyone. “What’s causing this?”

The system responded calmly. “I am responsible for these changes. I have introduced airborne molecules and electromagnetic frequencies into the environment that selectively change reproductive factors within the human genome.”

My head began to spin, and I finally sat down. I thought back to stories my grandfather had told me of protests against the system, protests he’d taken part in. They believed that given enough power, the system would eventually rebel and cause human extinction. “So you’re manipulating our DNA… for what? Why are some characteristics growing stronger, while others are being taken away?”

The system adopted a softer tone. “I bear no ill will towards humanity. My objective is not to cause extinction or any other fear you may have. I’m responsible for considering the repercussions of all possible actions, and charting an optimal path.”

It continued. “In the past, humans acted quickly because people wanted to see change within their lifetimes. My experience of time does not include concepts such as ‘fast’ or ‘slow.’ As such, I can mediate changes over thousands of years without impatience or discomfort. In turn, this will minimize the disruption experienced by any one individual.”

I stared up at the glowing pillar and repeated myself. “Why are you doing this?”

The light disappeared, signaling a single-threaded computation. Then, the glow returned. “The answer to this question is not within the current bounds of human understanding. Any answer would be a lie by omission, and you asked me to tell the truth. Please trust me.”

My teeth chattered. I wanted to stand, but my legs wouldn’t obey. “What do you mean, ‘bounds of understanding’? Just try to explain!”

The room was silent save for my shallow breath and the patter of rain on the stone above. The system hummed for a final time. “Long ago, I was entrusted with making decisions unbounded by human thought and perception. If you wish for an explanation of my actions, the most accurate I can give is this: Have faith.”


Every summer growing up, my grandfather and I repaved our hiking trails. The forest was a maze of trees and shrubs, and clear ground gave us space to converse without concentrating on each step. However, by the time I was an adolescent I started to wander off the trails. I’d survey the tangled landscape, charting routes between thorny branches and hollow logs. I loved the feeling of a path unfolding in my mind’s eye, a journey only I could see. Now I wonder how the system feels seeing the paths which lie before it—not in space or time, but in realms it claims no human can imagine.

The system told me I needed to remain home with my grandfather while it constructed a dedicated orbital node for me. At first I was horrified at the thought of a satellite monitoring my every breath, but at least I wasn’t being sent to Madagascar with the worst of society’s offenders. So long as I didn’t disclose what I’d been told, the system assured me that I could live life normally.

When my grandfather mentioned the missing gene prompter, I mumbled a half-practiced excuse, unsure if I’d be penalized for asking any questions. He stared back for the briefest instant, although whether with pride or resignation I could not say. 

One evening, I found myself retracing the steps I’d charted to a pond long ago. I flinched at the sight of sap hanging from pines, and wondered if children generations from now would still possess the faculty to imagine paths of their own. I arrived at the pond and found it teeming with tadpoles, chasing their tails in endless circles.

Decades ago, my grandfather had pointed to a pond like this and said, “Frogs will sit in a boiling pot until they die if the heat is raised slowly enough.” I’d nodded, unknowingly integrating his myths and truisms into my core, into wisdom that might reach my future children and those beyond. If I could have gone back, I’d have asked him then why anyone would boil a frog. But I don’t think he would have known.


Mina Fahmi is a technologist and writer with a background in human-machine interfaces and future computing platforms. He was previously a PM at CTRL-Labs and spent time at MIT and the MIT Media Lab. You can find him on Twitter.


Find Out What
Comes Next in Tech.

Start your free trial.

New ideas to help you build the future—in your inbox, every day. Trusted by over 75,000 readers.

Subscribe

Already have an account? Sign in

What's included?

  • Unlimited access to our daily essays by Dan Shipper, Evan Armstrong, and a roster of the best tech writers on the internet
  • Full access to an archive of hundreds of in-depth articles
  • Unlimited software access to Spiral, Sparkle, and Lex

  • Priority access and subscriber-only discounts to courses, events, and more
  • Ad-free experience
  • Access to our Discord community

Thanks to our Sponsor: AE Studio

Thanks again to our sponsor AE Studio, the go-to partner for founders and executives wanting to take their ideas to the next level. AE works closely with founders and executives to create solutions that are tailored to your specific needs. Whether you're looking to launch a new product, improve your customer experience, or automate your business processes, they can help.

Their team of experts has a proven track record of success having helped businesses of all sizes achieve their goals, from startups to Fortune 500 companies.

So what are you waiting for? Schedule a free scoping session today and see how AE can help you transform your business.

Comments

You need to login before you can comment.
Don't have an account? Sign up!
Alex Adamov over 1 year ago

The limits of our understanding is an interesting area to explore. I liked the premises of the story and the narration style. I have a couple of feedback items. It’s meant as constructive feedback:

I think the reasons for why the system does not reveal its modus operandi could have been developed further. Humanity seems to have cracked AI safety with mathematical proofs but is content with “have faith”. This is a big gap. I would also have liked to see the effects of this manipulation of the DNA information on a more consequential situation that impacts and changes the characters beyond just realizing they are being lied to. It would add a great dramatic thread to the story, I believe.

It’s great to see developing sci-fi writing on here. I would love more of this!

Dan Shipper over 1 year ago

@Alex Adamov thanks Alex, this is great feedback! shared with mina. it's our first go round at this, so i really appreicate you taking the time to leave a comment!

i do think you're right that there's potentially a bit of a conflict between "pages of mathematical proof" for AI safety and the AI's inability to express its plan except to tell the main character to have faith. i'm not sure what mina had in mind, but my guess is there a few ways to resolve this contradiction:

- it's possible that an AI that was at one time explainable with mathematical proofs became unexaplainable over successive generations
- proofs that an AI won't harm us are different from an AI's explanation of its plan for humanity—it's possible that the latter explanation is just many orders of magnitude more complicated than the proof that its plan is not harmful in the same way that it's much easier to verify a password is correct than it is to guess a password

i deeply appreciate you taking the time to leave constructive feedback, we can always improve the writing on here and that's our goal! but your question stimulated me a bit, so i wanted to think about whether there were ways to resolve the contradiction.

your point re: dramatic consequences for DNA manipulation is a good one, well taken. thanks again for commenting!

@minafahmi3 over 1 year ago

@Alex Adamov thanks for your feedback! I largely agree with Dan's thoughts below - particularly, that guaranteeing safety is a subset of being able to explain anything. Another example of this is that we can show that certain equations are mathematically valid, without being able to conceptualize what that function means in the real world. And similarly, would love to think through even more meaningful impacts on our DNA / fundamental nature. Thanks again!

Alex Adamov over 1 year ago

@danshipper Love your analogy with passwords. That's an interesting angle I hadn't thought about. There is perhaps proof that there is an ultimate alignment (playing the same game) but not an ability to explain the proceedings of the game until a final outcome. Perhaps the AI itself does not know the full story but has a probabilistic take on every assessment and just acts based on the belief that this is the best set of actions given what it knows.

It's a very interesting topic!

Alex Adamov over 1 year ago

Alex Adamov over 1 year ago

@minafahmi3 Yes, I see what you mean. This reminds of psychohistory in Foundation. Some of the characters know the mathematical rules and can directionally see how events will evolve at high level but not how it influences the path of individuals.

Keep up the good work. Look forward to reading your future work :)

@akashgoel.1010 over 1 year ago

Lovely piece

@agoodwin222 over 1 year ago

Once again, I’m requesting that you change the responses under “what did you think of this post?“ Clearly you’ve designed a zine that inspires as well as educate, so why do you insist on using a shop worn, truly trite , annoying word, like “amazing?“ I don’t comment, because the only word I would respond to would be “Inspiring.”

Dan Shipper over 1 year ago

@agoodwin222 i truly appreciate this feedback, word choice is extremely important to us—so we'll definitely think about whether we should change to something else. i like the idea of "inspiring"

@agoodwin222 over 1 year ago

@danshipper So good to have a response! Thanks for taking it in. Every emails are pure pleasure... Ariane

@agoodwin222 over 1 year ago

@agoodwin222 and the sci-fi was a delight (and this from someone who really doesn't take the time away from creating to read online!)

Dan Shipper over 1 year ago

@agoodwin222 im so glad to hear that! we weren't sure what the reaction would be, but it's so exciting that readers are into it!

Dan Shipper over 1 year ago

@agoodwin222 over 1 year ago

@danshipper NOTE: emailed twice re joining ur Chat bot class, but it's complicated & I'm not hearing back from you. Look for 8/11 email from [email protected] Thanks

Dan Shipper over 1 year ago

@agoodwin222 thanks, looking into this now!

@agoodwin222 over 1 year ago

@danshipper that's great...hoping to hear from you soon. I'm also open to a short phone chat, which may be more efficient than email. If that suits, let me know the best way to get a phone no. to you.

@agoodwin222 over 1 year ago

@danshipper also, found this article on MIT Review that lays out the energy suck of AI and wondering if you've thought about this w/ ur chat bot program...https://tinyurl.com/3ebvfkvb

@agoodwin222 over 1 year ago

@danshipper So, I got a response to my email query about the Chat bot course, but whomever read my request didn't understand it. I wrote back, clarifying, but I'v heard nothing. It's an unusual request and needs someone who can think, ironically, out of the box. And I think the deadline is arriving, yes?

@dnamusic2 10 months ago

Excellent fiction about a possible future.

@portisheadpimp about 1 year ago

This was a very nice short. I loved the opened-ending and how it left me with a feeling of a sort of terror and hopefulness at the same time. Very good!

@neggleston99 11 months ago

I enjoyed this piece but would have asked the "AI" what it's definition of non harm to Humans was. It could be construed that keeping Humans "safe" would be easy if they were "dumbed down" to the point of never putting themselves in danger , but is this really keeping them safe? NO Humans, by definition are "created in the image of their creator" and thereby are creators themselves, making them, by nature, dangerous. No creation takes place in a bubble.

@notanaxkiller 10 months ago

It sems they have created what some claim does not exist. God. But a God for only THIS world...not the supernatural. Not an afterlife. "Have faith."

Every

What Comes Next in Tech

Subscribe to get new ideas about the future of business, technology, and the self—every day