A message to all businesses out there looking for experimentation vendors: I feel for you. The selection available is unreal. It's actually the same issue with most SaaS products. Not just in the sense of "I want a, say, behavioural analytics tool - which one do I go for?" but even in the sense of "what martech platforms should I have that make up my tech stack?"
The point of experimentation is to leverage statistics to better understand potential changes to the business/website, and make better decisions under uncertainty.
So "what statistical methods is this tool using?" is a vastly underrated criterion in software selection.
If you buy a testing tool that uses basic frequentist stats (because these tools are often, but not always, cheaper), you're going to want somebody with a solid understanding of stats running the thing. Even then, you run the risk of outsiders peeking at experiment results and running away with the wrong conclusions.
You'd need to make sure tests are sufficiently powered, cross significance thresholds, control for multiple testing problems and familywise error rate/false discovery rate....
On the other hand, a more expensive tool will likely have an elegant stats engine under the hood to make sure results are readily trustworthy and prevent you from drawing the wrong conclusions. Optimizely and VWO do a great job of this, to name just two examples.
If you have budget, invest in a tool with a thoughtful statistical approach. Without one, you'll make wrong decisions over time that will be far more costly than the increased investment in software.
Thanks, David. I think your methodology is a smart one. Know thyself, shop around, and have them prove that they know how to solve YOUR challenges (not theirs). How much of this could be reversed? To find out, I "wrote" an article called, "Experimentation agency selection is really, really hard." :-) Don't be mad!
The point of experimentation is to leverage statistics to better understand potential changes to the business/website, and make better decisions under uncertainty.
So "what statistical methods is this tool using?" is a vastly underrated criterion in software selection.
If you buy a testing tool that uses basic frequentist stats (because these tools are often, but not always, cheaper), you're going to want somebody with a solid understanding of stats running the thing. Even then, you run the risk of outsiders peeking at experiment results and running away with the wrong conclusions.
You'd need to make sure tests are sufficiently powered, cross significance thresholds, control for multiple testing problems and familywise error rate/false discovery rate....
On the other hand, a more expensive tool will likely have an elegant stats engine under the hood to make sure results are readily trustworthy and prevent you from drawing the wrong conclusions. Optimizely and VWO do a great job of this, to name just two examples.
If you have budget, invest in a tool with a thoughtful statistical approach. Without one, you'll make wrong decisions over time that will be far more costly than the increased investment in software.
Thanks, David. I think your methodology is a smart one. Know thyself, shop around, and have them prove that they know how to solve YOUR challenges (not theirs). How much of this could be reversed? To find out, I "wrote" an article called, "Experimentation agency selection is really, really hard." :-) Don't be mad!
https://docs.google.com/document/d/1Rj2U0P2mfN5X8J6Ri5--amV8vumkU7CiaK_gGymaAU8/edit?usp=sharing
Tool, Tech, Platform, Vendor, Suite = agency
Features = services