Too often, I see PMs, designers, and researchers lost in their craft and “doing it right,” endlessly discussing underlying assumptions, formulating the most crystal clear and measurable unbiased hypothesis, and crafting the perfectly reliable experiment — lost in their documents without getting out of the building or making a decision. Time is too often spent pondering rather than getting their hands dirty, deciding or doing.
In product experimentation, this focus on design quality is counterproductive. While these factors are certainly important, there are other elements that are even more critical to success: experiment frequency and tenacity to keep up the habit.
Having conducted more than a thousand product experiments over the course of the past 10 years, this is what I’ve seen. From being a contributing part of experimentation and product teams that carry out experiments on a daily basis to managing groups of product teams in CPO and CTO positions, I have implemented the same recipe again and again to untrap product teams to make experimentation work for them.
Doing it too perfectly to make a move
In my role as CTO and CPO at Lenus eHealth, one team was paralyzed trying to design the perfect experiment to test an important assumption for one of the company’s most important bets.
As time passed with experiments and no releases, I became increasingly more inquisitive about what was going on. On paper, everything was in order. I found documents of beautifully crafted hypotheses all carefully mapped to customer opportunities and desired business outcomes. Almost too beautifully crafted.
It became clear to me that the team had spent the majority of their time getting everything perfect due to several factors:
- The pressure from working on the most important project of our entire product team.
- Pressure from the very effective recruitment strategy I had implemented that included the promise of doing things right and keeping very high standards in regards to product discovery.
- Team members had read much about doing experimentation and product discovery and some had even attended my masterclass on the topic before starting their position, but none had real experience doing continuous discovery.
They were too afraid to get it wrong. So afraid that they ended up in an analysis paralysis that lasted for more than a month. Finally, features were released only to be replaced half a year later as we finally got real feedback from the real world. Not from the opinions found behind desks — but from the harsh reality of our customers.
It’s better to make a move than make it perfect
It is more important to consistently test and iterate than it is to design the perfect experiment. By making experimentation a frequent and persistent habit, companies can benefit from the cumulative effect of these efforts that grow enormous over the course of months or years.
But more importantly is building up a “product sense.” An instinct or gut feeling takes over when your experiment muscles are trained with high frequency experimentation that is often just as on point as the most rationally and perfectly thought-out experiment.
As teams are consistently testing and iterating, they begin to develop a sense of what is likely to work and what isn’t. This intuition is valuable in guiding future experimentation, innovation, or decision efforts — many times this built-up product sense can guide strategy and replace “experimenting for perfect certainty.”
Another team in the same company came with a more pragmatic approach to product discovery, where a series of no-code tools and third-party integrations made mashing up both experiments and working solutions a breeze.
As they grew more confident working together as a product team, their shared intuition about what would resonate with customers became stronger, allowing them to make strategic decisions faster and more confidently. Several factors set them up for success:
- The product manager had a background in a venture studio, was an alumni of intensive growth-hacking courses, and was more focused on getting strategy right than building it right. The tech lead was more passionate about creating something worthwhile for customers than building something fancy. The design lead, who was junior at the time, was eager to learn and didn’t feel too proud to let others in on research activities.
- The product team collaborated intensively with the sales team and had them co-run experiments, which fueled discussions and alignment.
- When releases were ready for the wider public and not just behind a feature flag for a few customers, stakeholders had been so involved in its creation that they did the product marketing.
While quality is certainly important, it should not come at the expense of keeping the cycle time low and consistently getting out the building and putting things out on the world.
Let strategic intent guide your decisions
Successful product teams rely on having a clear and robust strategic intent defined, that serves as the foundation for decision-making. Some may refer to it as a “mission statement,” “vision,” “product strategy,” or “desired outcome.” The terminology may vary, but the underlying concept remains the same: providing your product team with a clear understanding of why they exist and what success looks like.
The specific name you give to your strategic intent is not as crucial as ensuring that it effectively conveys your product team’s purpose and vision of success. By providing a clear and inspiring strategic intent, you will empower your team to make informed decisions and foster a culture of collaboration and innovation aligned with the company’s objectives.
Co-creating strategic intent with stakeholders
This was the case for the second product team explained above. Co-creation with stakeholders allowed the team to explore, question, reframe, and refine the strategic intent iteratively — together with the business stakeholders. This built-up clarity and alignment allowed them to make informed decisions faster and build trust among both team members and stakeholders to move from the problem space to the solution space faster – they moved from discovery interviews and co-creation to testing and implementation, faster.
This is in contrast to the first team who ended up dwelling too long in the problem space, asking “is this problem worth solving?” The latter team was more effective in fulfilling their strategic intent by a multitude.
This strategic intent should be the driving force behind your actions, informing your choices more than experiments and product research alone.
Align teams around the intent and let them define how to achieve it. Share on XExperiments and product research serve as tools to help bring your strategic intent to life. They allow you to translate the intent into actionable plans, ensuring your product development process remains focused and efficient. This approach ensures that your experiments are grounded in your overall objectives and not simply attempts to identify what those objectives should be.
As the second product team gained confidence and clarity in their strategic intent, they were able to put it down in writing much faster, more precisely, and more concisely. By comparing all new opportunities and initiatives to their defined strategic intent, they were able to speed up their decision-making even more, and in turn evangelize it to the rest of the organization, allowing product marketing, sales, and even the CEO to capitalize on their good work.
Experiments serve to make the intent come true, not to figure it out. Share on X
That being said, it’s essential to remain open to the possibility that some of your experiments may reveal challenges or roadblocks in achieving your strategic intent. As you gather more evidence pointing to a contrary case, it’s crucial to adapt and allow the experiments to inform and refine your intent. This feedback loop creates a dynamic and responsive product development process that can adapt to new information and shifting market conditions.
While the influence of strategic intent on research and experimentation is typically more significant, particularly in cases where a profitable business and product-market fit have already been established, it’s vital to recognize that this relationship is bidirectional. Striking the right balance between intent and experimentation ensures that your product development process remains agile and responsive while staying true to your overall strategic vision.
Consistency trumps perfection
Instead of striving for the elusive “perfect” experiment, focus on consistently testing and iterating. This approach not only accelerates the learning process but also fosters an adaptive mindset within your team, enabling them to respond effectively to new information and changing market conditions. Over time, this iterative approach will yield valuable insights, build intuition, and refine your team’s product sense, ultimately driving more effective decision-making and better outcomes.
Embrace the journey of continuous improvement, and never lose sight of the strategic intent that guides your team’s actions. By doing so, you’ll create an environment that propels your organization towards its goals.
Lessons Learned
- Trust your gut feeling, if it’s backed by evidence.
- Make courageous decisions early.
- Experiments serve to make the intent come true, not to figure it out.
