How to Build an Experimentation Culture

Consistent growth is based on experimentation. 9 out of 10 A/B tests are not successful – or, in other words, they didn’t prove the hypothesis they were based on. The good news is that the one that was successful will bring meaningful business results, and that’s all you should care about.

Thanks to the army of tools and services that are widely available, experimentation, research, and testing are not just for the big guys anymore. Every startup could adopt a culture of experimentation form the very early days. This is much harder than finding the right toolset to establish the testing mechanic, because it often involves changing ingrained habits and establishing a new company culture.

You need a consistent, repeatable process for generating growth ideas, successful testing, and smooth implementation. How do you make experimentation sustainable?

We’ve learned how some of the biggest companies in the world are building and implementing their experimentation culture. Here’s how Booking.com, Hubspot, AirBnb, and Spotify are doing it.

1. Failure doesn’t exist in experimentation

The best way to kill employees’ morale and their hunger for experimentation is to shame and blame them that their hypothesis was wrong. By “shame & blame”, we mean not only pointing a finger and saying “I told you it wouldn’t work” but also passive-aggressive behavour like not approving their new test or growth ideas, mentioning what else could have been done during the time they spent testing, etc.

The basis of building a successful experimentation culture is adopting the mindset that everyone at your organisation is learning every day. Being wrong or running a test that doesn’t lead to meaningful product improvements is the only way you’re learning and moving closer to the right answers.

Failure is when people prefer to be on the safe side, do their job, not strive for growth, and then they’ve never been part of a “failed experiment”. This is a very dangerous place, as growth will become non-existent.

2. Establish a robust testing process

Finding the best format for your organisation is the hardest part, but once you develop the processes that work for you, you just have to make sure that everyone is on board. You need a standardised format – easy to understand by everyone and always bringing meaningful results. For example, rules like always testing one thing at a time, having a strict time frame for tests, using the same toolset, etc. will keep things consistent and will give people the autonomy and confidence to embrace the experimentation culture.

You should also keep a knowledge base of test results, accessible to everyone. At Hubspot, they have an internal wiki for sharing the findings from their tests.

3. Make it part of the on-boarding

Startups are privileged when it comes to building a strong company culture, and we see so many successful examples. You don’t have to fight ingrained habits and try to change 20-year-old processes. However, in times of rapid growth, culture is at the biggest risk, as you can’t sit 1:1 with everyone who joins the company.

Making sure that the experimentation culture and the rules of the testing game are part of the onboarding process and are clearly communicated with new employees during their first week will help a lot to pass the ball. Booking.com has tackled this challenge by creating a community of peer-to-peer reviews of internal experiments. They usually pair more experienced employees with new starters, so people can learn from each other and see experiments in practice. For smaller companies, this is also a great approach to build team culture without enforcing it.

4. Keep it moderate, keep it meaningful

Data is addictive, and some people are prone to losing themselves in large data sets, analysing and looking for potential loopholes. And they will find them, because the more you experiment, the more you discover and improve your product. For example, Booking.com run 1000 experiments at any time and continuously iterates on their tests to create more tests. Their motto is “Experiment on everything, every day”, because experimentation works, and this is surely a successful approach when you have 17,000 employees.

However, the only thing that is more dangerous for a small company than not experimenting at all is over-experimenting. Startups have limited resources and manpower, and tests should always be focused on achieving wider business outcomes. Sometimes, empathy is the sharpest weapon for startups, so testing and experimentation must always be “within your means”. Give freedom and enforce experimentation, but also set clear boundaries on how far they can go. Adobe is a great example of giving the opportunity and setting boundaries. They offer a $1,000 prepaid credit card to enable employees to start funding the test they’re running.

5. Start small. Test one thing at a time. Always.

Knowing the fundamentals of A/B testing and experimentation is a must, but this is not the focus of this post. There’s one golden rule that all experiments are based on: start with a global, high-level problem, then drill it down to small, measurable tests. Each test must have a clear hypothesis, one variable you’re measuring, and a clear strategy for interpreting results. There’s no point in implementing an experimentation culture if the rules of the game are not clear to all employees. You must invest in teaching your team how to accurately and productively run experiments.

When experimentation becomes part of the decision-making process, you successfully remove the right and wrong opinion-based decisions and the dangerous HiPPO syndrome (the Highest Paid Person’s Opinion). It’s the only path to scalable growth, so you’d better adopt it sooner than later.

If you enjoyed this article, you’ll love our Instagram content. Follow us below: 👇