AI Doomsday For People Who Don’t Wear Fedoras

The crux of the doom argument

If you simplify the doom arguments, they all spring from one fundamental problem:

It’s dangerous to build something smarter than you without fully understanding how it thinks.

This is a real concern, and it reflects the current state of things in AI (in the sense that we don’t completely understand what we’re building).

We do know a lot: a vast amount of math and complicated tricks to make it work, and work better. But we don’t understand how it actually thinks. We haven’t built AI with a theory of how its intelligence works. Instead, it’s mostly linear algebra and trial and error stacked together.

This actually isn’t uncommon in the history of technology—we often understand things only after they work. An easy example is fire: we used flint to generate sparks for thousands of years before we understood anything about friction. Another example is steam engines. We had only a rudimentary understanding of the laws of thermodynamics when they were developed.

If you build something through trial and error, then the only way you can control it is through trial and error. This is the process of RLHF (reinforcement learning through human feedback) and related techniques. Basically, we try to get the model to do bad things—and if it does we change the model to make those bad things less likely to happen in the future.

The problem is, trial and error only work if you can afford to make an error. Researchers like Eliezer Yudkowsky argue one error with this alignment process leads to the end of humanity.

The rest of the doomer problems flow from this basic issue. If, through trial and error, you’ve built an AI that thinks you find:

It’s hard to know if you’ve successfully aligned it because they “think” so differently than us
They are not guaranteed to be nice
Even it doesn’t explicitly intend to harm humans it could kill us all as a side effect of pursuing whatever goal it does have

In order to judge these arguments, I think it’s important to start from the beginning. How is it possible to build intelligence without understanding it? We built the software ourselves, shouldn’t we know how it works?

How is it possible to build intelligence without understanding it?

We usually understand how our software works because we have to code every piece of it by hand.

Traditional software is a set of explicit instructions, like a recipe, written by a programmer to get the computer to do something.

An easy example is the software we use to check if you’ve entered your email correctly on a website. It’s simple to write this kind of software because it’s possible to come up with an explicit set of instructions to tell if someone has entered their email correctly:

Does it contain one and only one “@” symbol?
Does it end with a recognized TLD like .com, .net, or .edu?
Does everything before the @ symbol contain only letters, numbers, or a few allowed special characters like “-”?

And so on. This “recipe” can grow to contain millions of lines of instructions for big pieces of software, but it is theoretically readable step by step.

This kind of programming is quite powerful—it’s responsible for almost all of the software you see in the world around you. For example, this very website is written in this way.

But, over time, we’ve found that certain types of problems are very difficult to code in this way.

For example, think about writing a program to recognize handwriting. Start with just one letter. How might you write a program that recognizes the letter “e” in an image? Recognizing handwriting is intuitive for humans, but it gets very slippery when you have to write out how to do it. The problem is there are so many different ways to write an “e”:

You can write it in capitals or lowercase. You could make the leg of the “e” short and stubby, or as long as an eel. You can write a bowl (the circular enclosed part of the “e”) that looks domed like a half-Sun rising over the morning sea or one that looks ovular like the eggish curve of Marc Andreeson’s forehead.

For this kind of problem, we need to write a different kind of software. And we’ve found a solution: we write code that writes the code for us.

Basically, we write an outline of what we think the final code should look like, but that doesn’t yet work. This outline is what we call a neural network. Then, we write another program that searches through all of the possible configurations of the neural network to find the one that works best for the task we’ve given it.

The process by which it adjusts or “tunes” the neural network, backpropagation through gradient descent, is a little like what a musician does when they tune a guitar: They play a string, and they can tell if the note is too high or too low. If it’s too high, they tune it down. If it’s too low, they tune it up. They repeat this process over and over again until they get the string in tune.

Most AI systems you interact with are programmed like this. The problem is because the code wasn’t tuneed by humans, it's really hard to dig into it and understand how it thinks step-by-step. (If you want a slightly more technical overview of this I highly recommend Andrej Kaparthy’s article, Software 2.0.)

This helps us to understand why alignment is such a difficult problem: we train these things via trial and error, and so we have to align them that way too.