How New Ideas Happen

“The first step to a new cosmogony is a step back” — Paul Feyerbend, Against Method

Raise your hand if you’ve murdered an idea.

I have. It’s a very strange habit.

Our ideas are like infants: they are small and precious. But while humans are biologically equipped to be parents to our children we are not biologically equipped to be parents to our ideas.

Rather than care for and nurture them we often kill them shortly after they’re born.

We do this because we conclude that the idea isn’t good enough after all. We look at what’s already out there and conclude that our idea can never measure up. We often justify this from a rational perspective:

“I’m just being realistic,” we say. Then we ball up the idea and toss it into the trash. We cross it out with bold red ink in our notebook. We let it collect dust on our hard drive. We wish it away like an old ex; we block it on social media and think to ourselves, “I’m glad I escaped that.”

What’s interesting about this is that we’re actually correct. Our ideas are bad.

The mistake is thinking bad ideas aren’t worth pursuing.

1

One of my most consistent intellectual obsessions for the last 10 years is mapping how the history of science and the history of business interact.

I first started getting into it in college. It was an interesting time for me: I was a full-time student majoring in philosophy and I was also running my last business, Firefly.

Being in college and running a business felt great to me. I was doing sales calls with large clients who had no idea I was a student, I was spending late nights at the office coding the product, and in between I was going to class, taking tests, and writing essays.

I actually really liked this split: working incredibly hard in the real world and then using school as a way to step back and reinterpret what was happening to me from a different, academic perspective. And while I think it’s easy to say that the ivory tower has nothing to offer to entrepreneurship — and vice versa — this weird marriage of two worlds early in my career actually helped me a lot.

One of the classes that was most helpful for me was philosophy of science. In philosophy of science we ask questions like: What is science? What is a scientific fact and how is it different from other kinds of facts? How does science progress towards the truth? Does it progress towards the truth at all?

These questions felt really relevant to me as I was running my company. We’d talk about running an experiment on the product and then I’d say to myself, What actually is an experiment?

It turns out, there’s actually a lot to be learned in the philosophy of science that applies to business. In fact, if we look deeply at the history of science we’ll find that a lot of the theories about the way science progresses looks strikingly similar to theories about how businesses win.

In short, both fields thrive on new ideas. And understanding more about how new ideas work in science and in business might help us become better parents to our own new ideas. It might help us stop killing them so early.

Because if we look back at how ideas work we’ll find that some of the biggest ideas in history looked bad at first. And often were bad at first.

But that actually didn’t matter too much — because bad ideas have a habit of becoming good.

Let’s start with science.

2

Copernicus was a Polish astronomer and mathematician who lived during the Renaissance and is credited with discovering heliocentrism — the theory that the earth revolves around the sun.

There are a bunch of Greek and Arabic astronomers that also posited heliocentrism before Copernicus but we just kind of ignore that for some reason. My apologies to them.

You might think that when Copernicus came up with heliocentrism that it was a good idea. You might think that Copernicus had a lot of good empirical evidence for it, and that’s why he was confident enough to push forward with his research and publish his findings rather than throwing them out.

But that’s actually wrong. Here’s what actually happened.

When Copernicus discovered heliocentrism he had very little evidence to back it up. And positing heliocentrism actually created many more problems than it solved.

Copernicus’ model attempted to replace the Ptolemaic model of astronomy that had been developed by the Greeks and had been used to great effect for more than 2,000 years. This model was supported by the Church, but, contrary to popular belief, it didn’t actually come from the Church.

Given the tools available to observe the heavens back then, the Ptolemaic model of the universe was actually very well supported by empirical evidence, and could be used to make good predictions.

So Copernicus was going up against what, at the time, were the best scientific facts available. (The definition of science back then is a little hazy, but the Ptolemaic model is at the very least empirical.)

But he did it because the overwhelming evidence showed he was right, didn’t he? Well, actually no he didn’t.

We don’t 100% know why Copernicus posited heliocentrism, but it wasn’t because he had a ton of evidence for it, or that he could predict anything significantly better with it. Copernicus likely rejected geocentrism more because of philosophical issues with the geocentric model than empirical ones. In other words, it was largely a matter of taste.

From the ancient world to the Renaissance pretty much every Western astronomer held the same philosophical position: heavenly bodies should move at uniform speed through space. That meant that they shouldn’t slow down or speed up — they should move in more or less perfect orbits through the heavens. This idea came from Aristotle, and it turned much later out to be basically right.

But it was actually very hard to justify: from our earthbound perspective, that’s not at all what we observe. If you go out and watch the night sky you won’t see planets moving in standard orbits. You’ll see them bob and weave, speed up, slow down and even reverse course over time.

So astronomers through the ages had to come up with fairly complicated systems to explain why we saw the planets move in strange patterns but they were really moving in perfect uniform-velocity orbits.

Ptolemy’s model was the best theory we could come up with for a while. It used epicycles (which are orbits within orbits) and equants. It’s fairly complicated and off-topic for this essay, but you can think of an equant as a way to say: “Each planet looks to us like it’s moving at different speeds, but there is another point in space that we could look at it from where it would look like it was moving at the same speed.”

From what we can tell from the historical record, Copernicus didn’t like the idea of an equant. He wanted to preserve the Aristotelian notion of the uniform movement of heavenly bodies, but he wanted to get rid of the equant.

And he found that heliocentrism was a way to do it.

But even Copernicus was pretty unsure about whether or not his idea was any good. In fact, he didn’t publish it at all until shortly before his death — and only because his friends urged him to do so.

We can’t know for sure why Copernicus held this idea back. But here’s a good guess: it was a gigantic change to our model of the universe but the benefit of the change was fairly small. It didn’t greatly simplify or improve on the Ptolemaic model. It didn’t help make better predictions, and it didn’t get rid of epicycles. It just got rid of equants.

And in exchange for that benefit the heliocentric model created a ton of problems.

For example, it introduced what we‘ll call the tower problem.

The philosopher Bertrand Russell explains the gist of the tower problem in his book The History of Western Philosophy:

“If the earth is continually rotating from west to east, a body dropped from a height ought not to fall to a point vertically below its starting-point, but to a point somewhat further west, since the earth will have slipped away a certain distance during the time of the fall.”

Basically, if the earth is moving, when I drop something from a great height, why does it appear to drop straight down onto the ground?

To modern eyes the answer is obvious. If everything is spinning at the same rate, the object will appear to drop down in a straight line because of inertia. But to someone from the Renaissance this was a genuine problem with the heliocentric model. One that Copernicus had no answer for.

It would take another 100 years and the discoveries of other scientists like Kepler, who discovered that heavenly bodies move in elliptical rather than circular orbits, and Galileo who used a telescope to make better observations, to create a heliocentric model that could hope to compete with the Ptolemaic model on empirical grounds.

3

Maybe all of this is specific to Copernicus though. In general, in the history of science, scientists only follow theories that have good empirical grounding right?

Wrong! In the 70s a movement of philosophers and historians of science, among them Thomas Kuhn and another philosopher named Paul Feyerabend, noticed something odd: throughout the history of science many theories that became widely adopted were rightly panned by the community when they were first put forward.

In his classic book The Structure of Scientific Revolutions Thomas Kuhn examined how new scientific theories came to overturn old ones. He called the moments when old theories are overthrown by newer ones paradigm shifts. And that’s where we get the term from today.

Kuhn’s theory of scientific change is complex and nuanced. But what’s most interesting for our purposes is what he says about how paradigm shifts happen.

We would assume that the scientists that discover new paradigms do so because the overwhelming evidence supports their case. We would assume that their new theories have extraordinary explanatory power, and coherence and that’s why they are pursued. We would assume that those ideas looked good from the beginning.

Instead, that’s not exactly what Kuhn found.

Scientists that make paradigm shifting discoveries start off by finding little inconsistencies. They notice something small in their research that can’t be explained by the current paradigm. It’s usually ignored by other scientists — or assumed that the existing paradigm will eventually grow to explain it.

But these problems stick around, and some scientists get obsessed with them. They obsess over them to the point that they construct entirely new theories that are inconsistent with the old established theories to explain the phenomenon.

And they often pursue these theories despite, not because of, the evidence.

Kuhn wrote: “The [scientist] who embraces a new paradigm at an early stage must often do so in defiance of the evidence.”

It would be really easy for a scientist, like Copernicus, to throw out their ideas early on. Many do. But some don’t. They persist on something else — something very un-science-y.

Kuhn says, “[A scientist] must, that is, have faith that the new paradigm will succeed with the many large problems that confront it, knowing only that the older paradigm has failed with a few.”

Strange. And interesting.

4

Does this have any relevance to you?

You’re not a scientist. Does the fact that new science often persists on the faith of a small group of people obsessed with resolving inconsistencies that most people wouldn’t notice really mean you shouldn’t throw out your bad ideas?

Well, one way to think about the generalizability of these kinds of lessons is to see how they apply to other fields. If we can find the same kinds of patterns in other places, it might convince us that we’re on to something.

As I noted at the top of the essay I think there’s a lot of parallels between how this kind of discovery happens in science, and how new ideas develop and grow in the business world.

In particular, there’s a lot of similarities between Clay Christensen Disruption Theory — which explains how new companies disrupt incumbents — and Kuhn’s theory of paradigms.

One deals with how new ideas replace old ones. The other deals with how new companies replace incumbents.

5

Usually when we think about how startups compete we think about it like this:

A startup can beat a big company if it makes a product that’s substantially better than the big company’s product.

It’s kind of like our previous theory for how scientific change works: a new scientific theory replaces an old scientific theory if it can help us make better predictions or explain more phenomena more simply than an old one.

Just like what Kuhn found when he looked at the history of science, when Christensen went to study the history of business, he actually discovered that this isn’t actually how small companies end up beating big ones.

If a startup simply makes a product that’s simply better than a big company, the big company will either copy the product and kill the startup, or buy the startup before it gets too big to be threatening.

Being better isn’t the way to win.

The reason for this is simple: big companies are run by smart people. Business is a long game, played over many iterations — and if something looks like a good idea to an incumbent they’ll be able to react accordingly.

In order for a startup to beat an incumbent, a new product often has to look like a bad idea.

6

How do you make a product that looks bad? In Christensen’s parlance you change the dimensions of performance.

Christensen came up with his theory of disruption by studying the hard drive industry in the 80s. He wanted to understand how new hard drive makers ended up beating established ones.

Back in those days hard drives were sold based on their diameter — 8.5 inch, 5.5 inch — etc. The dimensions of performance though were not based on diameter but based on cost and space.

What Christensen found is that over time, the hard drive makers that were used to creating a drive at a certain diameter — let’s say 5.25 inches — were eventually disrupted by hard drive makers that started making smaller diameter drives.

Why is that?

One answer might be that hard drive makers that specialize in 5.25 inch drives don’t have the technology know-how to make 3.5 inch drives.

But the opposite is actually true. Incumbents were often in the best position to make smaller drives, and were even sometimes the first ones to develop the technology to do so.

But those projects got canceled before they saw the light of day. The reason is that when an incumbent hard drive maker looked at making smaller diameter drives they were:

Going to be more expensive for their customers
Going to offer lower storage space

To a large incumbent, this was a very bad value proposition indeed. None of their customers would want a product that cost more and offered less space. There were usually a small group of customers who wanted it on the periphery — but not nearly enough of them to justify the project. So they killed it.

It’s kind of like a scientist who comes up with a new theory, but realizes that it doesn’t explain nearly as much as the old theory. What’s the use? It’s easy to throw it out.

But what Christensen found is that new firms had none of the same concerns as old firms. They could make a smaller diameter drive and sell it to different customers — customers that cared mostly about the smaller diameter, and didn’t care as much about the price or the space constraints. Like, for example, customers who were building smaller computers like laptops rather than large mainframe servers.

To a startup, this small group of customers was a gold mine. They could use them to start the business and improve their technology.

Eventually, they would improve their smaller diameter drives enough to swing into the larger markets dominated by the incumbents. And when they did they’d have a triple threat:

They were able to offer smaller diameter drives
That cost less
And provided more space

And this is what would finally kill the incumbent.

So what do we do with this?

7

First, do you see the parallels?

New ideas often solve a problem that only a small group of people care about.

In Copernicus’s case it was removing equants from the model of the universe.

In Christensen’s case it was hard drive makers producing a smaller diameter drive.

These new ideas often get rejected by the establishment for good reasons: they’re often not better, and they introduce more problems than they solve. Few people care about equants, and few people care about the diameter of hard drives.

But over time things change.

An obsession with equants turns into one of the most impactful paradigm shifts in scientific history. An obsession with hard drive diameter turns into a new class of drives that enables everything from laptops to cell phones.

The ideas that seemed bad at first, grow into something else. They explain more, and they apply to more people. They get to be mature enough that they begin to threaten — and eventually overtake the prevailing ideas and products of the time.

And that has a lot of implications for how we treat our own ideas.

First, new ideas require some faith. We often have to persist despite the evidence, not because of it.

Second, measuring new ideas by the yardstick of the ideas that they will eventually replace is a bad idea. New ideas can never measure up to old ideas. New ideas are small and fragile — all they have to do is be better in some small way, not in a large all-encompassing way, in order to be pursued.

Finally, we are the ones with the power to kill or grow our ideas. And the only thing we can be sure of is an idea will go nowhere if we toss it in the trash.

We should be gentle to our ideas. We should nurture them and let them grow.

Most of them will probably be bad. But being bad isn’t a bad thing.

Just ask Copernicus.

The key is to remember to look up at the night sky, observe the wandering planets, and allow yourself to trust in your dreams.

How did you feel about this post?

Amazing | Good | Meh | Bad

Further reading

If you want to read more about Disruption Theory check out Nathan Baschez’s Everything bundle series: Disrupting Disruption.

If you want to read more about the history and philosophy of science here are a few places to start:

The Structure of Scientific Revolutions, Thomas Kuhn

Against Method, Paul Feyerabend

The Metaphysical Foundations of Modern Science, E.A. Burtt