Knowing When to Test – and How to Do It Effectively
If You’re Not Testing, You’re Not Learning
Since many of our new clients have come from somewhere else, we’ve had the opportunity to see what others have done – to address their biggest concerns and produce the annual net income increases that they’ve needed.
Three problems are alarmingly common:
- Most organizations are not doing enough testing – to grow their lists, to renew and upgrade existing donors, and to generate the revenue that’s needed.
- The most important investment decisions are not getting the attention they deserve. Most testing is small in scope and potential. And these typical “tweak tests” (using slight variations of the same old letters and formulaic emails) almost never deliver powerful growth or save a program in decline.These nonprofits are missing systematic exploration of their biggest marketing questions and investment opportunities. As a result, they lack the directional guidance they need to answer fundamental investment decisions, such as: “Will we bring in more loyal donors and more reliable long-term net income by investing the next increment of money in recruiting canvass, event, Facebook, or direct mail donors?
- The third dirty little secret about testing is … the vast majority of email “tests” we have seen from others are not statistically significant.
Usually, due to the small number of respondents in these email campaigns or flaws in their test design, these so-called “winning ideas” are anecdotal at best. The perceived differences in response rates might be due to chance and should not be relied upon for decisions about what to do in the future. There is a strong possibility the apparent “losing idea” could actually have been equally or more effective. Worse, organizations are led to believe that they should never do X in the future, because Y was better. Any statistician worth their salt will tell you that’s not much better than flipping a coin.
Make Sure to Test the Most Important Things
All too often, the Big Things – that can change the impact of an organization – are not being adequately explored.
The Big Things not getting the attention they deserve fall into two categories:
- The fundraising elements that have already been discovered to be the most determinative of success for similar organizations; and
- The emerging technologies and more far-reaching improvements that have the potential of being the biggest game changers.
In either case, if these kinds of ideas are never tested, the mere knowledge of them will do little good for you.
This is exactly why Avalon has an in-house analytics team that is fluent in both statistics and fundraising. Our specialty is helping clients decide what to test, and how to interpret findings accurately – for smarter, faster, and more strategic innovation.