Eric Ries Stresses Value of Metrics and Continual Experimentation
2 comments February 6th, 2009
Full house this morning with a great conversation led by Eric Ries. Eric recapped several of his key experiences at IMVU, stressing the importance of continual experimentation, tracking cohorts of users (based on when they signed up for your service/site/application) to judge the impact of different improvements. When asked how to find a big market he advised “you always have to start in a niche, by definition; if you are lucky you will find a big market after a number of mistakes and experiments.” He told the IMVU story two ways, the cleaned up version where they executed their plan perfectly and built a profitable and growing business, and the “as it happened” versions where they were wrong about many things (e.g. IM interoperability as key feature) and only by continual ongoing experimentation and conversation with their few customers were they able to make adjustments to find a significant opportunity. Some other insights Eric offered (from my notes):
- Your investors are not your customers. You have to satisfy your customers for enduring success.
- You can bootstrap if customer acquisition cost is less than customer payout. It’s essential to understand both.
- Read “Founders at Work” by Jessica Livingston to get a sense of the amount of change to features and business model that most startups had to make to succeed.
Mar 2: Howard from SharedGain added a number of good quotes in his comment I am promoting into the post
- Use cohort analysis for measuring retention. Use time series for looking at groups of individuals that become members or customers at a certain period and consider them as a class to be measured over time. Each group coming in is treated as a canonical group. Different classes will have different attributes of the systems applied to them, and then the continuous A/B split testing can be done against different classes to see what works most effectively. The split tests for behaviors are looked at 1, 3 and 5 days later. The rapidity of change and analysis allows quick turns for the testing and A/B analysis. Then adjustments can be made on a regular, short-cycle basis to improve the retention cohort curve.
- Pick a few key metrics. Use funnel graphs and retention graphs.
- Some sites–like dating sites–are utilities, and usage drops off naturally over time. However, on-line social network sites like Facebook have word of mouth and viral elements, so people get sucked back in by their peers and friends if they go away for a while.
- Some statistics offered by Google Analytics are not meaningful, e.g. unique visitors or total visits. Seasonal factors, like day of week can confound the analytics.
- If can be effective to acquire new prospects using PPC, and expand the budget to bring more in, as the understanding of customer value and the stickiness of new customers increases.
- Customer development is the key—customers pay money, users don’t pay. The businessman chooses to be “right” or to be “successful.” Talk to customers and see what they like. Make adjustments to improve total involvement, and number of users and customers. There is a paradox about taking care of customers and listening to them. Current customers will complain about changes, while the new customers that are being attracted won’t say much. Therefore look at the aggregate customer behavior to determine what is working, but don’t listen too much to the noisy older customers that don’t like change. Many will stay with the site despite the changes in the site/offering, but you will loose some while gaining many more as you adjust to generate satisfaction for the larger and growing number of customers.
- Rapid deployment of modifications based on assessments from continuous A/B split-testing will evolve customers and the product to growing success.