The Multi-armed Bandit: New and much improved A/B testing tools

The A/B test tool on ActBlue, which allows you to test out contribution form titles and pitches, among other variables, has gotten a significant upgrade, just in time for campaign season.

The old A/B testing tool worked great, but it also forced you to wait around for both test variations to get enough traffic to gain statistical significance. If one version was performing way better than the second one, that meant you were losing out on potential contributions in order to gain valuable insight.

This is how most A/B testing tools work, and it’s a good system. But with the new ActBlue testing tools, which use a more advanced statistical algorithm than typical A/B testing, you can still achieve statistical significance without having to sacrifice a ton of traffic to a losing form.

As the test runs and one variation begins performing better, we’ll start sending more traffic to that form, roughly in proportion to how they’re trending. You can see the traffic allocation listed just above each variation on the “A/B Test” tab of your contribution form. The traffic allocation will change continuously as donations come in. It’s important to note that if a variation is receiving 75% of the traffic, that does not necessarily mean it’s conversion rate is 3X as high as the other variation(s). If you’re curious what it actually does mean and want to talk complicated stats, you can get in touch with us here.

If there was a false positive and the losing form starts doing better, the traffic allocation will begin to reverse. The test will continue to run indefinitely until you click “Make Winner.” The A/B testing tool will eventually send 100% of volume to the winner if you don’t make either version the winner manually.

The new A/B testing tool makes your tests more efficient, which means you can try out more of them. If you have radically different language you want to try on a form, alongside three more standard pitches, there’s little risk. If it doesn’t work out, we’ll send fewer and fewer people to that losing form.

We wanted to give special thanks to Jim Pugh from ShareProgress for sharing notes on the multi-armed bandit method used in their software and helping us out with building this tool (and for hanging out in the ActBlue office for a week)!

As always, let us know what tests you’re running and what’s working for you at!