What to Do After You Launch

Let’s say you just launched a product to a group of users for the first time. (I’m speaking purely hypothetically, of course 😉.) Launch went pretty well: good signups, hearty praise, and some promising early data. Or maybe you’re not sure how well it went.

Now what?

In the movies, what happens next is a montage where you work tirelessly day and night—on what, exactly, it’s not clear—and all the graphs go up and to the right, until you find yourself in the conference room of a law firm in Palo Alto with some partner telling you to that a $65 million settlement with the Winklevoss twins is just a “speeding ticket,” because that’s how rich you are.

In real life you have to do things to make the graph go up and to the right. Often it’s not exactly clear what to do.

You can think of the post-launch phase of a startup as one big optimization problem. You have a bunch of goals—engagement, retention, growth—and a lot of levers you can pull to achieve them. The challenge is figuring out which goals you should focus on, and which levers actually work.

The good news is that there are some frameworks you can use to think through this problem. Many founders before you have been in the same spot. I’m interested in this because it’s what I’m going currently through with Lex. But I hope it’s helpful for anyone else in a similar situation.

Retention: leaky until proven sticky

Most products do not retain users. Therefore, you should assume yours doesn’t, until the data conclusively proves otherwise.

This doesn’t mean you should get sad about it or worry, but it does mean it should be your top priority. In order to grow a business, almost no matter what type of business you have, you’re going to need regular usage.

After your product has had people using it for a week or so, create a retention analysis and find out how many people are still using the product seven days later. Don’t worry too much about the percentage yet—higher is obviously better, but what matters most is whether anyone cares enough to continue using it. You’re looking for a line that seems to go flat at some point, rather than following a gradual slope toward zero:

Basically you want this:

Not this:

To create a graph like this, I recommend using Amplitude to track all the core actions that users take in your product. (This is not an ad; they just have the best product.) 

I have a few Amplitude tips that I had to figure out the hard way. (This might be getting a bit in the weeds for some of you, but it’s useful to know.) There are many different ways to look at retention, and it took me a while to figure out what they mean.

• N-day retention means, for each day, what percentage of users came back on that exact day. You can use strict calendar dates or 24-hour windows custom to each user that start on the moment they signed up (good for normalizing time zones).
• Unbounded retention means, for each day, what percentage of users came back on that day or at any later date. So if a user came back on day 4, even if they weren’t around on days 2-3, they would still appear as “retained” on those days.

For most products it’s better to look at N-day retention by default, because it gives you a better sense of how many people you will end up with as sticky users at the bottom of your funnel.

Another helpful nuance is to look at retention with different actions as the starting point. By default most people start with the “New User” event, which includes anyone who makes it into the product. But you can fiddle around and see which features or milestones cause users to stick around. In the case of Lex, users who perform the “Use AI” action are much more likely to stick around than those that don’t. Facebook famously tried to get users to find 7 friends in 10 days. The point isn’t that the exact definition matters that much, but it should be directionally correct and help you understand what the core value is.

Talk to users, but have the right conversation

Once you’ve spent a bit of time looking at your retention data, take a step back and ask yourself if there seems to be a real group of people that are sticky users of the product. Either way, your next move will be to talk to users, but the type of conversation you’ll have and who you’ll want to talk to will be different.

If there is a group of users where the data indicates true product-market fit, congrats! Talk to them, understand the context of their lives that makes your product valuable, learn how they found out about the product, listen to the words they use to describe the reason they keep coming back, and then do everything possible to find more people like them and orient your product and marketing to speak more clearly to them.

If there isn’t a core group of passionate users, that’s OK, too. Talk to a broader set of people than just your most engaged users. You should now be looking for more fundamental things you can be doing differently. The easiest way is to talk to users about what they hoped your product would do (i.e., the reason they signed up). There’s going to be a lot of noise, but if you talk to enough people you’ll start noticing themes that will help you become more targeted in your questions.

Setting up these meetings might seem like a pain, but it’s easy. Create a Calendly link with 15-minute meeting slots and email it out. If you want to get systematic about it, you could write code to automate this.

When you’re on the calls, take copious notes and record them. Create transcripts using Otter. Now you have a searchable record that will be useful far into the future.

Here’s an example interview script, but you probably want to follow the flow of conversation to some extent and leave time for lots of follow-ups (that’s where the good stuff is).

1. How’d you initially hear about Product?
2. What was your first impression? What stood out?
3. What other types of products did you think it was in the same category as?
4. What did you do/what product did you use before you started using Product?
5. What expectation made you interested enough in Product to sign up?
6. In your initial session using Product, were your expectations met?
7. (If applicable) What triggered you to come back to Product a second and third time?
8. How disappointed would you be if you could no longer use Product?

This is somewhat generic, but it gives you a starting point. Ideally you will have identified some key questions that are specific to the risks and uncertainties unique to your product, and spend time on that, too.

For instance, with Lex a key risk is that the product is built on top of GPT-3, which anyone can access and build on top of. So we quickly need to find a way to create durable value that is less easy to copy. We have a few theories about how we’ll be able to do that, but it’s good to do a sanity check of some assumptions with customers.

Wrangling ambiguity

Now comes the hard part. Once you start having conversations, there’s a chance that some clear themes will pop out at you, but more often than not you’ll feel a frustrating sense of ambiguity.

In Superhuman founder Rahul Vohra’s piece about the company’s journey to finding product-market fit, he wrote about how he surveyed his users, asking, “How can we make Superhuman better for you?” 

The word cloud of responses makes their next move laughably obvious:

Most founders won’t be so lucky.

Here’s how Lex’s word cloud looks for the same question:

Not exactly obvious!

This is part of the reason why it’s so much more useful to talk to users than to just email them a Google form with a few survey questions. When you have a conversation with users, you build a much better model of what people are doing with your product, which allows you to inhabit their perspective. Once you can do this, product improvements become more obvious and intuitive. The best way of doing it is to be a user of your product yourself, but if you only ever build for yourself, at some point the product will go off the rails because what’s obvious to you will often be incomprehensible to other people.

This is also where mentors come in handy. Have someone who has been proven to build great products use yours and give their feedback. Ask them if they’d be willing to record a Loom as they sign up and use your product for the first time, and tell them you’d appreciate it if they didn’t hold anything back.

Lots of people will do this, for free, just because they’re not used to someone asking for genuine feedback. Ninety-nine percent of entrepreneurs who reach out to me don’t really care what I think, they just want me to tweet about their product or introduce them to investors. You can stand out simply by being focused on the fundamentals and writing a short, compelling cold email or Twitter DM.

Common problems

The trap most people fall into when analyzing retention is jumping to the conclusion that the solution is to bombard users with more aggressive notifications. Implementing this strategy in the context of a mediocre product will only lead to sadness.

The most common problem early products have is that they’re complicated and confusing to new users. Maybe the core engine of value is there, but it’s weighed down by a ton of drag. If you ever find yourself building an elaborate onboarding sequence, it’s a sign of failure. (Nobody likes those.)

To figure out if you have this problem, create a funnel analysis in Amplitude that starts with “New User” and ends with the event that correlates to a “Oh wow, this is great” feeling. For Lex it’s the first time you use the AI. Here’s how our funnel looks right now:

This is a reasonably good starting point, but I think we can do better. If your product is in a similar range, this probably isn’t a huge blocker for your growth, and other parts of the funnel might be better places to focus your energy.

But if your product only activates 20% or 30% of users who sign up, maybe the problem isn’t the core value your product offers, but that it takes too much effort to actually experience it. 

What’s next

What comes next is the fun grind that is every tech startup: gradually keep making all parts of your system suck less. Refine your marketing to more efficiently reach the right type of customer. Refine your onboarding to get people to experience the value more quickly. Refine your core product experience to create even deeper value in users’ lives. And then do it over and over again.

The most important thing is not to over- or under-react to problems. It can be hard to see middling data and sustain the motivation to keep going, but sometimes that’s exactly what’s needed. Other times you may need to look yourself in the eye and be honest with yourself about whether you really are creating the right product. Sometimes you’ll need to go back to square one. 

No matter what happens, keep the faith. Everything will work out in the end. The uncertainty can eat up your insides, trust me I know. But it’s important to have a sort of realistic confidence: that no matter what, even if this iteration is not the one, even if this company or team is not the one, you are going to get there if you keep going. There’s a lot in this world that you can’t control, but the one thing you do control is your own motivation and courage. The good news is that with enough time and determination, amazing things are possible.

Godspeed!