On important matters, people often put more effort into figuring out the right decision. But the right decision is different depending on who puts in the effort. And what if right is defined as the option that won the experimental split test rather than what seems to be best for the overall system?
Part of product testing (and even product concept testing) depends on generating demand data by running variations in front of customers. Run lots of variations, find the ones that perform better based on the metrics you value, and repeat.
I recently read a description of large-scale product experimentation for a financial services company. The business’ product experimentation had direct financial outcomes — namely improved customer retention and customer lifetime value. But there were costs too. Continue reading “Problems or Puzzles”
