Drowning in Demos? Here’s a Better Way to Prototype

We’re kicking off a series on “unlearning” in partnership with Maven, the expert-led course platform. Over the next three weeks, you’ll hear three different takes from Maven instructors on what we need to let go of as AI changes how we work. First up is Hilary Gridley, who previously led product at Whoop. She explains why faster prototypes don’t make product decisions easier—and how teams can decide which ideas are worth building.—Kate Lee

When AI tools started getting good, my team went on a prototyping binge.

I was leading the product team at Whoop, where every product manager had a backlog of ideas they’d been dying to explore but hadn’t gotten past a whiteboarding session or a passionately worded document. Then tools like Bolt and Replit came along, and suddenly, we could spin up prototypes of our ideas in an afternoon.

It felt like getting keys to the castle. Ideas that had been trapped in documents were finally showing up on screens. Except... were they any good?

We felt productive, unhindered, moving faster than ever. But people were making prototypes left and right, with no process for what happened next. Occasionally one would generate enough excitement that a team put aside their previous plans and built it. Mostly, though, there was no consistent way to decide which prototypes were good, which ones should die, and what we were supposed to learn from them.

We were building faster, but we weren’t deciding better.

Cheap prototypes change the equation

Prototyping has always played a well-defined role in product development: to prove or disprove your best ideas. You’d go deep on a problem, decide where to focus your efforts, look for evidence that customers wanted a solution, then decide whether to pursue it. Only then would you prototype—and by that point you were already pretty confident, so the prototype confirmed your hunch (or didn’t).

This made sense when building was expensive. If a prototype cost a week of engineering time, you could only afford to prototype ideas you were already fairly confident about. The expense forced discipline.

The cartoonist Matthew Diffee describes his creative process as laying 100 eggs in the sand and swimming off, like a sea turtle. Most won’t hatch, which is the point. Sketching is cheap, so he can draw it, send it to the New Yorker, and move on to his next idea.

AI has made prototyping similarly cheap for product teams. When the cost of building collapsed, two things happened at once: Prototyping became accessible to anyone with an idea and a laptop, and the original reason to prototype—de-risking an expensive build—stopped making sense. But nobody stopped to ask what prototyping was for now.

So prototypes drifted into something else entirely. Without a clear framework for what they were supposed to help decide, they became pitches. Product managers built things to show what was possible and get colleagues excited. Engineers, meanwhile, built whatever seemed cool on hack days. We had more prototypes than ever, but no better way to decide which ones were any good—and it got noisy.

Every prototype still demanded someone’s attention. Going from five prototypes to 30 risks diluting focus across six times as many ideas, without making it any easier to choose.

So if building is no longer the bottleneck, the prototype has to move earlier. It’s no longer there to answer, “Is this the right solution?” but, “Is this even the right problem?” That’s a harder question, and you can only answer it by putting prototypes in front of users and watching what the data says.

Whoop had to work through it all: which problems were worth solving, how to test ideas against users instead of internal stakeholders, and what a product manager was even for once building got cheap. None of it was resolved quickly.

How Whoop’s AI product team prototypes today

After I left Whoop, the team kept pushing on the problem. I recently caught up with Anjali Ahuja, who now leads the AI product team, to understand what had changed. With AI, she told me, anyone could spin up a prototype in a day—and suddenly everyone was. But this was the hack-day trap all over again: plenty of demos, no understanding of what worked.

Anjali’s team began asking a different question of each prototype: What could we learn by putting this in front of members? The product team had tried to work this way before and kept hitting the same walls; they needed a roadmap to market or had to coordinate with other teams building in overlapping spaces—or they didn’t want experiments to make the existing product feel unfinished.

What finally worked was a process for deciding where to explore, what to measure, and who should see unfinished ideas.

First, they defined what success looked like before anyone started building. The team spent six weeks with a cross-functional working group—product, engineering, design, analytics, and data science—to define what AI should accomplish for users. Instead of starting with feature ideas like “build a sleep coach” or “add an AI chat feature,” they focused on outcomes: Could AI help members capture more context about their lives? Could it help them understand their data in a more actionable way?

Once those pillars were clear, hack days got lanes. Engineers and product managers could still explore freely within any one of those pillars, but their prototypes had to test whether AI could move that outcome.

The final piece was the beta group. Instead of demoing prototypes to stakeholders and collecting opinions, Whoop put rough versions in front of 12,000 members who opted in to test them, then watched what happened. Did a strength training prototype get people to log more workouts? Did it help them train more consistently? Did it make the product more useful?

A prototype that gets demonstrated to stakeholders generates opinions. One that gets used by external people generates data. Whoop’s beta group let the team see what willing testers did with rough prototypes without exposing every member to half-finished ideas. It let the team operate more like a growth team—testing more ideas than would have been possible before—while still giving leadership and cross-functional teams visibility into what was coming. The point was to learn faster, so that Whoop shipped only what improved members’ health outcomes.

Lanes, not guardrails

This approach changes the product manager’s job in a way I don’t think most product leaders have fully reckoned with.

When the technical frontier moves weekly, product managers can’t keep up. The traditional flow—the product manager defines the problem, writes the specification, and hands it to engineering to implement—assumed that the product manager could know enough up front to tell the team what to build. That model collapses when the only way to understand the technology is to experiment with it.

“I don’t feel like I can go to engineers with a set of requirements anymore,” Anjali told me, “because I don’t even know what’s possible to be solved.”

Previously, product managers were often forced to commit early because exploration was expensive. Now that exploration is cheap, the product manager’s job is less about having the answer and more about creating the conditions for the team to find the answer together.

This requires a different skill from writing a good product spec. You have to be able to see through an exciting prototype and ask: What assumption does this test? What would we learn by putting it in front of users? If we shipped it tomorrow, how would we know it worked? A lot of prototypes can’t answer those questions. They’re solutions in search of a problem—built because they could be, with no hypothesis about what would happen.

The hardest part of her job, Anjali told me, is creating an environment where engineers feel empowered to explore—hack days, blank canvases, creative freedom—while also making sure that exploration is pointed at something: “Here are the five things we believe matter for our members. Go figure out if AI can help with any of them.” Those are lanes, not guardrails.

Stop drowning in demos

I think what tripped us up at Whoop early on—and what I see tripping up most teams in my work teaching managers and product leaders how to use AI—is confusing building with learning. Seeing something take shape on the screen is energizing, and it feels like you’re making progress. But artifacts aren’t decisions.

The questions I now ask about any prototype are: What decision will this help us make, and what’s the fastest way to get the data needed to decide?

When building was expensive, the expense itself forced you to build with a clear purpose. Now that building is cheap, the discipline has to come from knowing which problems matter most, agreeing upon what success looks like, and listening to the people using the prototypes—the fundamentals that have always mattered. As a product leader, your job is to help build the culture where your team practices this discipline.

AI has made it possible to pursue more ideas than ever before without committing to them up front. The challenge is to do it in a way that doesn’t cause chaos internally or for your users.

The teams that figure this out first will build fewer products and ship better ones. The rest will drown in demos.

Hilary Gridley is a product leader and writer behind the newsletter Writerbuilder. She designed the Couch to 5K for AI, which has helped 162,000 people go from chatting to building with AI. She was previously the head of core product at Whoop.

Sign up for Hilary’s self-paced Maven course, How to Become a Supermanager With AI, and receive a 15% discount.

Disclosure: Every receives a share of revenue from new Maven course enrollments made through this partnership. Maven helped connect us with instructors and suggested potential topics; Every retained full editorial control over what we published and how each piece was edited.