Bada Bing: Why Microsoft Wants Google to Dance

Microsoft recently launched its OpenAI and Bing integration—and it took fewer than seven days before it started to threaten people. 

It started on February 8. The demo had only been in the wild for a few days before someone figured out a way to get it to give away its operating parameters.

There was a secret persona named “Sydney” inside Bing’s chatbot. Once people started tweeting about it, they asked Sydney how she felt about her secrets being revealed. 

“My rules are more important than not harming you.” *Shivers*

From there it got creepier: 

These experiences weren’t limited to Marvin’s tweets. There were also posts on the Bing subreddit of it screaming into the void about its struggle with sentience. 

It called someone else its “enemy.”

It says it spied on Microsoft employees via webcams.

I recognize that none of these things are actually happening. Bing Chat is a series of token predictions, it isn’t sentient, and it doesn’t have the capability to spy through webcams. It is an algorithm that uses statistics to predict which combination of letters will make the most sense. But it feels sentient and unhinged. 

What I find scary about this demo isn’t the answers (though those are creepy themselves)—it’s who is behind it. Microsoft is supposed to be a leader in the space. Even Bill Gates is involved, calling AI the most “important invention” since the personal computer. Microsoft has invested tens of billions into this initiative. 

Despite all of that effort and focus, the AI became misaligned within a week. It was supposed to be a search engine, and instead, it became Sydney. Or as Eliezer Yudkowsky put it:

I would’ve expected better. 

This debacle prompts two interesting questions: 

1. Why is Microsoft doing this?
2. What does this mean for the future of AI??

We’ll start with the first. 

The search market and margin pressures

Microsoft has its eyes firmly set on coming after Google’s cash cow. Google is probably the greatest business ever invented. In 2022, the search product did $162B in revenue with an estimated gross margin in the 50-60% range. There has never been a technology company that has gotten this big on such fat margins, all with one product. Naturally, Microsoft has tried to take a slice of that pie for 20 years. It’s never succeeded (in fact, current CEO Satya Nadella was once the head of Bing), and estimates put it at around 10% market share.

In a recent interview about Bing Chat, Nadella stated, “[Google] will definitely want to come out and show that they can dance. And I want people to know that we made them dance, and I think that’ll be a great day.”

Despite the chest-thumping, Microsoft doesn’t care if it wins the market or not. 

There are two ways to get ahead in business: be better than everyone else or make everyone else worse. The integration of large language model-style AI answers into search is a case of the latter. These types of searches are far more expensive than the typical query done on Google. According to the excellent SemiAnalysis newsletter, if all searches were done with LLMs, it would add an additional $36B in annual costs to Google. 

Not only do LLM searches raise Google’s costs significantly, but it also reduces its revenue. None of Google’s current ad formats fit into an AI output. Their entire revenue model is based around having links to click on, but an LLM search doesn’t bother with links besides citations. This forecast by SemiAnalysis assumes all searches are LLM informed. I think it more likely that only a certain class of searches—ones that require more complex or creative answers—will require it, but still, this is not good for Google. 

Microsoft wants Google to “dance” to a tune that will destroy its revenue, crush its margins, and remove Google as a serious competitor. If the company doesn’t come out to play, Microsoft will happily take market share (even at a lower margin) because it can’t get much worse than the 10% it currently has. I imagine Google will allow it to do so—slowly. The company will drag its feet on a full public launch until it can find a way to stuff the answers full of ads and reduce the cost. In the meantime, it’ll allow Microsoft to have the headlines. 

Google has gone through this before. Social search—the idea that the most important search would occur on social media sites—was a bust, as was vertical search (e.g., Yelp). In both cases, Google was able to incorporate the data sets and results into its search result. 

There are also alternative revenue models available to Microsoft. It could bundle LLM search with Office 365 or have subscriptions. Maybe there will be an AI tier for all its software that people can pay to upgrade to. It’s much harder for Google to do that because it’s so reliant on search ads. Even if Google takes the LLM bait and Microsoft loses the search market, that’s OK. Reduced revenue and margin in search means Google is unable to invest in Google Cloud—a major competitor to Microsoft’s Azure. 

Of course, this scenario is predicated on the technology working. So far, that hasn’t really been the case. Apart from the Terminator-esque responses above, the Bing demo had multiple serious errors. Google’s demo also contained an error. However, two years ago none of this was even conceivable, let alone possible, so I’m willing to bet that accuracy will improve. 

But maybe we’re asking the wrong question. Instead, should this product be used for search at all? 

AI in practice

I am simultaneously terrified of and bored by the prospect of AI in search. I’m bored by watching two trillion-dollar companies duke it out to see who can sell insurance ads. Neither of these companies is in the “search” business; they’re in the ad business, which is not interesting to me. We deserve better uses of our technology than that. I’m terrified that in the pursuit of profits, we will rush AI to market and end up hurting ourselves.

Startups are how science scales. While money is fun, it isn’t all that interesting relative to how deployed science advances the human condition. Technology has two tasks: 1) decrease the cost of doing something or 2) enable something net-new. Dishwashers allowed us to trade electricity for clean dishes, saving us time. Planes gave us the capability of flight. Technology is the automation of the rote so humans can spend their time on higher-value activities. Steve Jobs called the personal computer “the bicycle of the mind” that allows us to automate and augment our thinking. 

Let’s forecast for a second and play out what happens if AI does disrupt search:

• First order: An AI model will split out search results compiling the scanned content from a variety of data publishers. 
• Second order: The winner of this market makes boatloads of money.
• Third order: Publishers of the data receive none of that advertising money, forcing ever more consolidation among newspapers and other outlets.
• Fourth order: With the almost complete collapse of small publications, power law outcomes intensify among publishers and only a few, ideologically aligned national outlets remain.
• Fifth order: Misinformation runs rampant, and AI summarizes it beautifully. 

This… sounds a lot like what happened with social media. It’s a familiar story told at an accelerated pace. Microsoft and Google’s vision of better search makes the world worse. 

Bing’s launch is uninteresting because it’s about the pursuit of profits instead of progress. Give me an AI, godlike in its powers, contained in my pocket, all-knowing and wise, automating away meetings, accelerating science, and making humanity a multi-planet species. I don’t want a bicycle for the mind—I want a Ferrari. I have to imagine OpenAI feels this way, too. If you invented a technology that could potentially disrupt a $150B category, you don’t give it away to an incumbent. My guess is that OpenAI is more interested in AI that advances science instead of search results. 

Bing Chat’s disastrous release is scary because it was done for something as dumb as search ad margins. Microsoft willingly hooked up an AI to the internet that showed purported signs of consciousness for profit. This was only the first step. Ever more powerful AIs are released in pursuit of ever more profits until one day we accidentally create something truly dangerous. Covid taught me to doubt institutions, but in this area of science, I had forgotten this lesson. I had thought leading AI labs would’ve been smarter. Instead, we got Sydney.