A/B Tests are great if you’re focused on the right issues. However, many teams focus on the wrong problem. Even worse, they’ll focus on the wrong concept altogether, opting for incremental improvements instead of dramatic changes for the better.

More often than not, A/B tests are used to are a dog and pony show for the executives than to solve real problems.

Jared Spool has written a great article about this over at UIE:

In our study, we watched more than a dozen of the presenters’ company’s own customers attempt to buy products. While many were successful, a surprising number weren’t, even though this company is the biggest in its industry (and hailed by many as the most successful). Their site looks slick, but when folks sat down to use it for its primary goal, it’s design put up a ton of frustrating obstacles.

In many cases, the users thought they ordered the product they wanted, only to discover upon receipt that it wasn’t at all what they wanted. As we watched those shoppers make their orders, we could see that they would not get what they wanted.

…

The A/B tests they presented showed they were applying a ton of effort to optimize things that weren’t close to the things we saw preventing sales on their site. If the message was that A/B testing helps, I didn’t get that because I saw them futzing around with tweaking insignificant button text when there were huge deficiencies in the design that they still haven’t resolved.