Previous Puzzle: Guns & Banjos
Previously in Things I asked how a banjo and gun website could see conversion in its individual sections improving, but overall conversion decline. Xuan pointed out that visits to some part of the site not counted in either section could see an increase in non-converting visits and so drive the effect, but this is a loophole – even without such a section, the problem can still arise.
The answer here lies in the part of the data not revealed in the setting: the underlying traffic trends. In particular, if you have one particular section (or segment) that converts at a low rate, and the mix of your traffic shifts towards that group, then that will tend to pull down the overall average conversion, even if conversion within the groups is going up.
To take a simple example, consider the following figures for this theoretical guns and banjos site, with figures in the form [before mix shift | after mix shift]:
Banjo section visits [10,000 | 20,000] – lots more traffic
Banjo section sales [100 | 220]
Banjo section conversion [1.0% | 1.1%] – conversion increases!
Gun section visits [1,000 | 1,000] – same traffic as ever
Gun section sales [100 | 110]
Gun section conversion [10% | 11%] – conversion increases!
Overall conversion before:
(100 + 100) / (10,000 + 1,000) = 1.82%
Overall conversion after:
(220 + 110) / (20,000 + 1,000) = 1.57% – overall conversion has decreased!
In practice you will rarely see an effect this strong. On the other hand, you could well see mix shifts drive a significant part of an overall shift in conversion rate. So watch out for that.
Puzzle: Mix Shift Visualisation
This is a really tough one, but I know the Things recipients are up to the task.
Given that mix shifts as described above can be difficult to spot, how might you visualise your website’s data in such a way that any such effects became obvious?
(I have an answer for this, but I feel sure a better answer must exist).
Everyone seems to be linking to this surprisingly weakly curated collection of Steve Jobs quotes. One part that I did like expressed something I’ve been thinking for a while:
Steve Jobs: I have a very optimistic view of individuals. As individuals, people are inherently good. I have a somewhat more pessimistic view of people in groups.
Or as I’ve tried to distill it: humans don’t scale.
These guys make cool stuff, even if their website is slightly too clever for its own good.
It’s too easy for imagination to be constrained by what we already know exists. That’s why I like things like this video, which help remind us that many more things are possible. It’s not particularly well edited and there are no particular highlights, so feel free to skip through and enjoy the general sensation of weirdness, which is after all what we generally look for on the internet, and indeed in life. Or maybe that’s just me.