One of the challenges as you build your startup is to maintain a culture of experimentation. Sometimes, companies get lucky with one new thing and then in order to continue to grow, they become more and more risk averse. No one wants to get blamed for killing the goose that laid the golden eggs. Ideas are suggested and just as quickly discarded. “We tried that once. It didn’t work.” Or even worse: “How will it look?”
If you want to scale your company over a long period of time you need to build a culture that is constantly learning and willing to try new things. And that means you must be willing to experiment.
Unfortunately, as companies grow they often become more conservative and risk averse. Ironically, it is sometimes the founders and early executives who become the most risk averse. It likely took massive amounts of trial and error experimentation to build the company’s first successful product. It’s understandable that people don’t want to break what made the company successful. It’s as if those early hardships exhausted their supply of experimental will.
Unfortunately, technologies, markets, customer expectations, and competition change over time. If you’re not trying new things, it’s easy to get left behind when there’s a major shift in the market.
I remember one very innovative company I worked with that had great success with their early product-led growth (PLG) strategy. Developers signed up, as they used the product more and more, they hit limits in the product plan and then expanded to the Enterprise edition. Unfortunately, the executives continued to rely on this strategy long after it was clear that the results were declining. They so much wanted to protect their trusted go-to-market strategy that they failed to invest in Enterprise lead generation and sales enablement. The result was lackluster growth, which ultimately precipitated a change in management.
You need to create an environment where people are able to run experiments, and make evidence based decisions with no stigma of failure.
What Is An Experiment?
Sometimes the toughest obstacles in a company are internal. I have found no better way to overcome resistance than the phrase “I’d like to try an experiment.” It’s hard to argue against.
An experiment is a way of testing a hypothesis. It should be time and resource bound. Limitations are good as they help you focus on the essential hypothesis. There should be clearly defined metrics for success.
It’s not uncommon to debate how many different product editions a company should have. Maybe you have a low-end version and a higher-priced Enterprise offering. Would sales increase if there was a middle version priced between these two?
Everyone has an opinion on pricing, but the only one that matter is the customer’s. Either having a new edition increases sales or it doesn’t. You can ruminate as long as you want, but the best way to decide is to run a test.
Be clear what time and resources are required. How long do you want to run the test? Who sees the new offering? What happens when they click to buy? Figure out the minimum amount of work to test the hypothesis.
You must also define what success would look like. Keep in mind you’re not trying to accurately predict the results of the experiment. If you could predict it, you wouldn’t need to run the experiment!
Instead set a bar above which you can say the hypothesis is true and worth pursuing further. If sales went up 10% in the experiment, would you introduce the new edition? Probably. If it’s under 5% it might not be statistically significant. And if the sales went down, that’s a clear answer also.
If the experiment doesn’t work, at least you found out with only a modest expenditure of effort. Don’t stop experimenting just because one test didn’t yield great result. Maybe there’s a different experiment that might yield a more meaningful increase. It takes time and creativity to get great results.
The most frustrating thing to me as an executive was when a team would try something and when asked if it worked the answer was “it’s not clear.” If you can’t measure the results and know whether it worked, it’s not an experiment.
Trial and Error
I remember at Duo Security when we were trying to improve the user interface. Our research had shown that people were often confused with what action to take when reporting an illegal login. Each time we went back to the engineers, asked them to improve it. Then we’d test it with users in a walk through. People would see the language, read it out loud and then panic and back out. It took at least three attempts before we figured out the optimal language that got users to do the right thing. None of us could predict correctly how people would interpret the language. Only through experimentation could we determine the best path.
We were fortunate at Duo to have a culture of experimentation. No one was angry or got blamed when an experiment didn’t work. It simply meant we had not found the right solution… yet. And this was true in marketing, in sales, in engineering, everywhere.
In fact, Duo had a long history of fostering experimentation with Duo Hack Days. These were typically one-day events held every few months, where people worked on speculative projects. It originally began in Engineering, as a way to build and test prototype features or user interface experiments. But over time, we expanded the scope to include every function in the company.
The idea was to tackle projects that might otherwise not get attention by pulling together colleagues from across multiple departments and ship something by end of day. It could be a prototype of a feature, an experimental UI design, a marketing campaign, a piece of marketing content, whatever. In short, anything that helped the business.
Some of the things that shipped included:
A new Chrome browser extension
Customer TCO report calculator
More customizable UI messages
Customer hall of fame
Vertical industry content
and more…
Some projects were just for fun, like “Duo Employees Read Tweets About Their Work” video which helped share the enthusiasm our customers had for our product. But even these helped people share a sense of accomplishment. Hack Day wasn’t only for engineers.
As Pete Baker, the former creative director at Duo, pointed out, Duo Hack Day brought together different teams to work together. Creating a TCO calculator required efforts form Engineering, Sales Operations, Marketing and others to pull together to define, spec and create something in a single day. People learned more about areas outside of their own responsibility and expertise and it helped foster appreciation and collaboration that lasted far beyond the day’s events. An experiment with a decent prototype could short-circuit almost any formal approval process.
“It reminded people how to hustle, to form and test a hypothesis, and to not always be precious about process. You had one day (5 hours) to work with people you had very little history with, to formulate leadership and participation organically, throw together a proof of concept, and, crucially, to present your ideas, while letting a lot of things fall of the table in the process.”
—Pete Baker
Not everything that was created on a Duo Hack Day worked out. There were some prototypes that seemed promising, but ultimately didn’t ship because they proved not to be that impactful or created confusion with our existing customers.
Duo Hack Days showed employees that ideas were always welcome and always worth testing. That had a significant positive impact in the company. Even though Hack Days were special occasions, the celebration of ideas and the emphasis on collaboration pervaded the culture and became a core part of our fabric.
Special thanks to Pete Baker for his recollections on Duo Hack Day. Pete runs a creative agency called FinalFinal which helps companies innovate through design.