Because new features add overhead to products (generally making them more complicated), a new feature has to provide so much benefit to customers that it’s worth incurring this overhead. There is no such thing as a “neutral” new feature. “The same” means worse.
When our product changes fail to actually improve business metrics—customer retention, usage, or sales, we should have the courage to admit it. In fact, failing to make the product better is an extremely powerful moment, one in which we have the opportunity to learn something important about ourselves and our customers. If we think a feature makes the product better but our customers disagree (not by what they say but by how they behave), then something about our mental model is flawed.
In one of the startups I founded, we had an extended period where we were really focused on the conversion rate of new customers into paying customers (it was a freemium business model). Our vanity metrics were looking good—up and to the right. The graphs even had the shape of the classic hockey stick. But cohort analysis revealed that something was wrong.
We divided our customers into cohorts, looking at the new customers who joined each day as a distinct group. Then we could ask, “How did today’s customers compare to yesterday’s?” And, much to our frustration, the conversion rates were almost exactly the same. It was easy to get conspiratorial. It felt like each group had set up a conference call with the previous group. “How many of you bought the product? One out of a hundred? OK, good, we’ll do that, too.”
The frustrating part was this pattern stayed constant for months, even though we were making the product “better” almost every day. The fact that customer behavior wasn’t changing revealed that we were wrong. We weren’t making the product better, we were making it worse.
The antidote to this problem is to stop using vanity metrics and start measuring progress more rigorously.
More here: http://techcrunch.com/2011/09/11/are-you-building-the-right-product/