Product Building: Rapid Experimentation

Aleph
8 min readJun 6, 2023
Stable Diffusion images a young woman conjuring product magic!
Stable Diffusion imagines a young woman conjuring product magic!

I had the honor of talking to some of the folks on the Eden team a little much earlier in the year about product building. I’ve been thinking a bit and decided to share slides from the talk alongside some of my thoughts on how I’ve thought about this — particularly at Helicarrier. I’m hoping it will be useful to some Product/Engineering/Growth people hustling at startups. PS: The Eden facility is an impressive work of operational excellence and they were gracious enough to feed me a nice plate of pasta and peppered Tilapia :)

Problems

Definitely the most important thing a product person — particularly at a startup — should focus on is Problems. We very easily fall for the temptation to be results and solutions-oriented and put solutions first. But this is obviously putting the cart before the horse. Deeply understanding the problem we are tasked with solving is the most important step in a product person’s endeavor.

Of course, the grand unified problem is that of finding problem-market fit. This is what we’re all on the hunt for, particularly in early stage startups. Two images come to mind: the first is Marc Andressen’s quote and the second is this.

“The customers are buying the product just as fast as you can make it — or usage is growing just as fast as you can add more servers. Money from customers is piling up in your company checking account. You’re hiring sales and customer support staff as fast as you can.”

We need to find problems that are so critical and core to a subset of humans/users. As examples, it’s clear Google and Twitter have PMF; Crude oil has PMF. As problem-oriented product people, we are insatiable until we arrive at a point where na dem dey rush us. That’s the bar to set and that’s what keeps us up at night.

There are a handful of tools that guide in understanding the problem space in startups and I talk about them as follows.

User Obsession

The most important is user obsession. We need to be moral agents towards our users: we have to deeply empathize with them. As a product person, you have to abstract away everything that defines you as a human being and enter into the mind of your potential user. You need to understand their problem as well as they do — no, better than they do. Understanding problems isn’t just a thing Product Managers should do. It has to be a deeply cultural thing across the organization. For instance, every few months, everyone in the organization should dedicate a day to doing just customer support: responding on Front and Zendesk, calling users etc. These Support Days are particularly useful: you have to experience the pain users feel; everyone on the team needs to be exposed to users’ problems. This kind of exposure drives a feeling of self-disgust and shame that motivates us to make the products better everyday.

Rapid Experimentation

Another critical tool is experimentation. Frankly, no one really knows what the answer to most of these problems are. Even the greatest minds working on problems probably have wrong intuitions most of the time. What we need to do is lean into one of humanity’s greatest tools: tinkering. We tinker, change and evolve and are in fact often wrong because we’re always trying new things; it’s fine to be wrong.

We need to be mobile and agile; avoid decision paralysis and commit with incomplete information. Having strict and long processes around decision-making and building isn’t ideal. It makes it harder to course correct since people are overinvested and fall for sunk-cost fallacy. We need to be swift and nimble. There is so much to learn on the journey and so much of our intuition we need to evolve.

One of the important things with experimentation is having a way to prove ourselves wrong. You should start out with a thesis and before doing anything ask the critical question: what needs to happen to prove to myself that I’m wrong? If you do it halfway into the process or after the process, you trick yourself and you start moving the goalpost because you’re emotionally invested and there is all of this sunk cost: if I spend just a bit more time or invested just a bit more energy maybe that will move the dial. No, you want to avoid that kind of situation. Pre-commit to what you expect the results to be. Your more objective self and your team’s more objective-self is pre-experiment. Ask yourself: how do I demonstrate to myself that I’m wrong?

The odds of success in a startup is 1%. The odds of striking gold is 1%. So what you’re doing is saying how can I quickly discard all of the bad ideas so I can find the good one?

MVPs

The other framework in thinking here is the MVP-mindset. This isn’t just a useful mindset when building full-fledged products but also when building features. There’s so much to be said about MVPs but I will highlight just a few:

  • There is no such thing as perfection. Nothing that exists hits perfection early. For instance, Amazon Prime started it was free two-day shipping, then same-day, before they got to hours. Language translation, AI, and the iPhone are all things that started little by little (the first iPhone did not have an App Store in it). Builders kept tinkering and iterating over time.
  • Plan well and account for Murphy’s Law. This reduces the odds of being blind-sided and doesn’t negatively impact customer trust. You don’t have the luxury to gamble with trust. You can deploy solutions to small group of people. This means when things are wrong, only a small group is affected and rollbacks are easier.
  • Talk to users. Your intuition is wrong. What you think users want is generally far from what they actually want. Your opinions and anecdotal data are minuscule in the grand scheme of things . You refine your intuition by talking to a diverse set of people: very active users, passively active users, churned users etc. You need to figure out what talking to users looks like. Make sure Engineers talk to users. Make sure everyone across all the teams has access to talk to users. You also need to meet users at their pain points. Aleph is a bad user for instance: if you ask Aleph to tell you when Aleph has a problem, they will have a problem and not tell you. They will just move on with their life and use a competitor. You need to be able to figure out that Bad User Aleph had a problem even if they don’t reach out.

Culture

Now all of these tools are useless if they aren’t deeply ingrained in the team’s DNA. User obsession, experimentation and MVP mindset are useless if they are only in a Product Manager’s arsenal. You have to drive it across the whole organization. It’s the Product Manager’s responsibility to push for this if it doesn’t already exist. It’s non-negotiable.

In addition, the team culture has to be incredibly collaborative and non-political. Product leaders and team leaders should exemplify this and reward positive behavior. Don’t steal someone’s work or credit. Give very very direct feedback. Don’t focus on being nice although it’s a nice-to-have. Give it in good faith.

Getting Started — Practical Steps

One of the valuable questions I got at Eden was how to think about getting started. If you’re already a team of 20–30 and feel like a slowly moving organization, how do you pivot?

The first thing is the MVP mindset. Start with small teams: what’s a small team of 3–4 people who can rapidly experiment? Experiment with rapid experimentation. It’s meta ;) This team needs to have full autonomy; almost like a startup within a startup. Ideally, the team should report directly to a co-founder so there is no bureaucracy and there is reduced friction. Yes, even at a 20-person company, you already start having bureaucracies.

Some other useful things to consider are:

  • Build the foundations and infrastructure well: Sometimes pause to rebuild the Engineering foundation. You can already tell when this step back is needed: generally you start seeing a spike in customer support issues (often with flat growth). There is always this tension between Product people and Engineering People about building new things versus making Engineering improvements. This isn’t strange. It’s often healthy and ubiquitious, just work on reasonable compromises. One week or one sprint delays are hardly what kill a startup. They shouldn’t happen often but they can happen every couple of months.
  • Make low cost experiments. Generally speaking, your experimentation windows should be happening within a time horizon of days to weeks — especially in software startups like fintechs. Bezos is the one that has time to be thinking in 7 years. You’re a startup, you don’t have that time. You acquire data, and iterate. It’s also important to note that this makes it much easier for people to disagree and commit. Since mistakes aren’t defining and don’t lead to a lot of burn. Don’t be emotionally attached to things. Try to be dissociated. Cut things that don’t work: amputate like a 19th century surgeon and keep it moving.
  • Internalize learnings. Don’t try to make the same mistake 2–3 times. This is an obvious principle that everyone talks about but in practice isn’t obeyed. It’s actually harder than it sounds.

Risks

In all of this, there are a lot of risks to watch out for as you iterate and rapidly experiment. These risks of course exist in general with product building.

  • Users lie: users are obsessed with making your life more difficult as a product person. You ask them if reducing the price of a subscription will make them subscribe and they lie. Anything you ask is generally filled with lies. There are all sorts of reasons users lie: sometimes they don’t want to hurt your feelings so they give hopeful answers; other times, they are imagining a hypothetical version of themselves. Either ways, these aren’t useful for good decision making. Users are generally good at describing their problems (although even for this, I would rather you observed them instead of asking questions). Users are bad at answering any other type of questions.
  • Engineers lie: I love Engineers with all my heart (as I used to be one myself) but they also lie and you need to account for this in your product-building process. They underestimate the difficulty of problems, they overestimate their ability etc. The Engineer knows you care about the feature so they offer blindly optimistic timelines. Discount heavily for this in your planning.
  • Bad communication: this is a massive risk for all organizations. You need effective communication so everyone is better sold and can effectively make decisions even on the micro-level and there is less of a need to micromanage. Top-down product decision-making is generally bad. Optimize for writing a lot and making these public/accessible across the org: it’s easy to have things in your head and miscommunicate or under-communicate. Lateral comms is very good: it helps teammates plan better and not stay blocked; it also improves trust and cohesion.
  • Overstretching: one thing we had to be careful about learning in Helicarrier is that rapidly building things meant you could be spread too thin. We are always open to killing features, and unemotional about shutting down product lines. Experiments can lead to so much happening simultaneously. Always take a step back to look at the big picture and kill invalidated theses.

If there is anything to take away from all of this it’s: have an MVP-first mindset across all things, drive experimentation culturally and take a problem-first approach to things. You still have a maybe 10% odds of winning but you’re at least set up for a chance at success with the most effective tools and framework.

I am also sharing the slides I used for the Eden talk. Feel free to enjoy :)

--

--

Aleph

Cofounder, Helicarrier. Custodian, Aleph Fellowship.