Is AI Progress Hitting a Wall?

Was this newsletter forwarded to you? Sign up to get it in your inbox.

---

A wave of recentarticles proclaims the death of deep learning. Leaked reports suggest OpenAI's new model Orion finished training without showing nearly the improvement that GPT-4 achieved over GPT-3. Critics like Gary Marcus are already writing gloating eulogies.So, is AI progress slowing down? No. Let me tell you why.

My nephew is 2 years old. Over the last year or so he’s rapidly become much more mobile. First, he learned to crawl, sticking his butt into the air and pushing with his knees to zoom across the room. Soon after that—and much to his parents’ chagrin—he began pulling himself up the sides of chairs and coffee tables. Then in the blink of an eye he was walking, unsteadily at first, but with increasing confidence. Then he was running! And waving his arms in the air as he ran, like a clumsy ballerina.

In the past month, though, this rapid progress began to decline. He’s no longer getting nightly mobility upgrades. You can still see progress, but it’s measured in weeks rather than days. It’s more subtle, too. Visible in a slightly more fluid and confident gait, or the dexterity to ride a scooter.

So yes, as far his movement goes, he’s mostly done making exponential progress. However, this is not cause for alarm. No one is sending concerned messages to the family group chat wondering if his growth is somehow stunted.

Why? He’s now growing quickly along an entirely new dimension: learning to say no. He doesn’t want to listen to the music his parents have on; he wants to hear “Baby Shark.” He doesn’t want to eat what his parents have made for him; he wants noodles. He doesn’t want me to read him his favorite book; he wants Mama.

For my nephew, it’s a completely new paradigm of exponential improvement—one he’s transitioned to smoothly from the previous one.

A is quite similar. 

It is factually accurate to say that AI progress may be slowing along one dimension: pre-training. (This is the practice of wresting exponential performance improvements from language models by training them with more data and more compute.) But if that’s all you say, you’re fundamentally missing the point. It would be like saying my nephew’s mobility growth is slowing down without also mentioning the rapid progress in his language skills and sense of self.

It does appear that pre-training is reaching some kind of diminishing marginal returns (though even this hasn’t been totally maxed out, according to sources I’ve talked to). But, regardless of which perspective you believe, these headlines miss the bigger picture.