A/B Testing Is Not Analytics

What are analytics, anyway? At recent conferences and networking events, we have heard different answers to this question—many of which, unfortunately, are missing the point.

Here are the two most common misconceptions:

Misconception #1: “We use analytics – our campaign reports are accurate and timely.”

Campaign reporting is important, but it is a single piece of the puzzle (as is segment or source-code reporting). Strategic analytics go deeper, examining trends at the donor level for patterns of giving and preferences. They unlock your donors’ long-term value and giving potential—providing critical information to indicate the health of your file, orient your program strategy, and reveal new opportunities for targeting and cultivation. Additionally, analytics not only incorporate the data your organization is tracking in-house, but also can overlay additional data, such as an individual’s consumer history or the median income of her zip code.

There are three relevant categories of strategic analytics, each of which answers different and equally important questions. They are: descriptive (what has happened in the past?), predictive (what will happen in the future?) and prescriptive (what strategy should you pursue?). Campaign- and source- level analytics are typically limited to the descriptive category—important, but incomplete.

(For more on the three categories of analytics, see this primer from Harvard Business Review.)

Misconception #2: “We use analytics – we employ A/B testing for our campaigns.”

Too often, A/B testing is confused with strategy – and testing results are confused with analytics. This is a dangerous misconception because it threatens to undermine valid strategic planning for your fundraising program.

A/B testing needs a strategy to drive it—to prioritize goals, to safeguard long-term objectives, and to set your program’s tolerance for risk. Strategic analytics serve three functions here: first, they tell the story of your program to-date (descriptive); second, they predict the trajectory of your program (predictive); and, third, they recommend a strategic course of action (prescriptive).

Analytics should inform your testing parameters—revealing the most significant metrics for your program, predicting risk/reward dynamics, and identifying target audiences for specific techniques. On the back-end, strategic analytics should assess and refine your testing approach for the next round of improvements.

Both A/B testing and campaign-level analytics are essential for the smart execution of direct marketing strategies. They tell us what’s working and what’s not, they enable message and creative development, and they hold us accountable to our budgets and goals. But they are not – I repeat, not – the same thing as strategic analytics.