Comments on: Eric Ries Stresses Value of Metrics and Continual Experimentation http://www.bootstrappersbreakfast.com/blog/2009/02/06/eric-ries-stresses-value-of-metrics-and-continual-experimentation/ Join Other Entrepreneurs Who Eat Problems For Breakfast Mon, 26 Jul 2010 22:01:36 +0000 http://wordpress.org/?v=2.9.2 hourly 1 By: BootstrappersBreakfast » 40% Discount For Eric Ries’ “Lean Startup” Workshop Jun-18-09 http://www.bootstrappersbreakfast.com/blog/2009/02/06/eric-ries-stresses-value-of-metrics-and-continual-experimentation/comment-page-1/#comment-8524 BootstrappersBreakfast » 40% Discount For Eric Ries’ “Lean Startup” Workshop Jun-18-09 Sun, 14 Jun 2009 02:50:46 +0000 http://www.bootstrappersbreakfast.com/blog/2009/02/06/eric-ries-stresses-value-of-metrics-and-continual-experimentation/#comment-8524 [...] This will bring the price of the one day workshop down to $1200 (from $2000, neither are exactly bootstrapper friendly prices but Eric’s visit to the February 6th Bootstrapper Breakfast–documented in “Eric Ries Stresses Value of Metrics and Continual Experimentation“–was very well received. [...] [...] This will bring the price of the one day workshop down to $1200 (from $2000, neither are exactly bootstrapper friendly prices but Eric’s visit to the February 6th Bootstrapper Breakfast–documented in “Eric Ries Stresses Value of Metrics and Continual Experimentation“–was very well received. [...]

]]> By: Howard http://www.bootstrappersbreakfast.com/blog/2009/02/06/eric-ries-stresses-value-of-metrics-and-continual-experimentation/comment-page-1/#comment-3698 Howard Tue, 03 Mar 2009 05:16:54 +0000 http://www.bootstrappersbreakfast.com/blog/2009/02/06/eric-ries-stresses-value-of-metrics-and-continual-experimentation/#comment-3698 Use cohort analysis for measuring retention. Use time series for looking at groups of individuals that become members or customers at a certain period and consider them as a class to be measured over time. Each group coming in is treated as a canonical group. Different classes will have different attributes of the systems applied to them, and then the continuous A/B split testing can be done against different classes to see what works most effectively. The split tests for behaviors are looked at 1, 3 and 5 days later. The rapidity of change and analysis allows quick turns for the testing and A/B analysis. Then adjustments can be made on a regular, short-cycle basis to improve the retention cohort curve. Pick a few key metrics. Use funnel graphs and retention graphs. Some sites like dating sites are utilities, and usage drops off naturally over time. However, OSN sites like Facebook have WOM and viral elements, so people get sucked back in by their peers and friends if they go away for a while. Some statistics like Google Analytics are meaningless. Unique visitors or total visitations are not meaningful. Seasonal factors, like day of week can confound the analytics. If is often effective to buy new prospects using PPC, and expand the budget to bring more in, as the understanding of customer value and the stickiness of new customers increases. Customer development is the key—customers pay money, users don’t pay. The businessman chooses to be “right” or to be “successful.” Talk to customers and see what they like. Make adjustments to improve total involvement, and number of users and customers. There is a paradox about taking care of customers and listening to them. Current customers will complain about changes, while the new customers that are being attracted won’t say much. Therefore look at the aggregate customer behavior to determine what is working, but don’t listen too much to the noisy older customers that don’t like change. Many will stay with the site despite the changes in the site/offering, but you will loose some while gaining many more as you adjust to generate satisfaction for the larger and growing number of customers. Rapid deployment of modifications based on assessments from continuous A/B split-testing will evolve customers and the product to growing success. Use cohort analysis for measuring retention. Use time series for looking at groups of individuals that become members or customers at a certain period and consider them as a class to be measured over time. Each group coming in is treated as a canonical group. Different classes will have different attributes of the systems applied to them, and then the continuous A/B split testing can be done against different classes to see what works most effectively. The split tests for behaviors are looked at 1, 3 and 5 days later. The rapidity of change and analysis allows quick turns for the testing and A/B analysis. Then adjustments can be made on a regular, short-cycle basis to improve the retention cohort curve.

Pick a few key metrics. Use funnel graphs and retention graphs.

Some sites like dating sites are utilities, and usage drops off naturally over time. However, OSN sites like Facebook have WOM and viral elements, so people get sucked back in by their peers and friends if they go away for a while.

Some statistics like Google Analytics are meaningless. Unique visitors or total visitations are not meaningful. Seasonal factors, like day of week can confound the analytics.

If is often effective to buy new prospects using PPC, and expand the budget to bring more in, as the understanding of customer value and the stickiness of new customers increases.

Customer development is the key—customers pay money, users don’t pay. The businessman chooses to be “right” or to be “successful.” Talk to customers and see what they like. Make adjustments to improve total involvement, and number of users and customers. There is a paradox about taking care of customers and listening to them. Current customers will complain about changes, while the new customers that are being attracted won’t say much. Therefore look at the aggregate customer behavior to determine what is working, but don’t listen too much to the noisy older customers that don’t like change. Many will stay with the site despite the changes in the site/offering, but you will loose some while gaining many more as you adjust to generate satisfaction for the larger and growing number of customers.

Rapid deployment of modifications based on assessments from continuous A/B split-testing will evolve customers and the product to growing success.

]]>