Planning to conduct A/B tests on your social sector program?

With recent massive cuts to foreign aid, it’s no wonder the international development sector is buzzing about A/B tests and other tools that can help decision-makers maximize the impact of their investments on a relatively quick time horizon (including IDinsight). But, with any tool, there’s a right place and a right time, and a wrong place and a wrong time. This post covers some pitfalls to avoid when deciding whether to run an A/B test on your program.

What are A/B Tests?

Also known as nimble RCTs, A/B tests are essentially randomized evaluations that (i) compare two versions of a program (rather than a program vs. pure control)¹, (ii) tend to be ‘rapid’ by focusing on short-term outcomes, and (iii) tend to be cheaper by relying on data that is generated for program monitoring rather than independent primary data collection. While impact evaluations are typically used to prove whether a program is impactful, A/B tests are used to streamline programs and optimize cost-effectiveness. 

Recent demonstration cases from Youth Impact, Rocket Learning, and others have shown the power of A/B tests to drive program improvements, but this tool is not a one-size-fits-all. Good A/B tests may not be possible for some organizations, and focusing on program theory and collecting robust quantitative and qualitative data on implementation may be a more cost-effective way to strengthen program design. 

Here are some pitfalls to avoid if you’re going to A/B test your program:

1. Optimizing on short-term outcomes can undermine long-run impact

Many major tech successes are built on extensive A/B testing. Companies like Google, Amazon, and Netflix run thousands of simultaneous tests to continuously optimize their products, contributing significantly to their market dominance. The reason that A/B tests work so well in the tech sector is that the outcome of interest –– user engagement –– occurs within seconds and is extremely cheap to collect in large volumes. 

The analog to user engagement in the social sector is take-up. Take-up is well-suited for A/B tests because it can be measured shortly after you start implementation, and if your intervention has been proven to work, there is a strong theoretical link between increasing take-up and improving downstream outcomes.² If you administer vaccines or distribute bednets or run other evidence-based interventions, A/B testing to increase take-up can be a great tool to expand your organization’s reach thereby increasing your impact.

But many organizations are keen to use A/B tests for more than just increasing take-up. The vision is that, with sufficient rigorous evidence of impact in hand, A/B tests can be used to strengthen program design, impact, and cost-effectiveness. One challenge with using A/B tests in this way for social sector programs is that outcomes can take months or even years to manifest. The effects of new agricultural inputs won’t be measurable until the next harvest. An asset transfer or livelihoods intervention will take a year or longer to increase household income or consumption. Teacher training will require at least one term before it starts to influence student learning outcomes.

So the key to successful A/B testing of social sector programs is finding a good proxy outcome: one that is predictive of downstream outcomes, that can be measured shortly after the program is delivered, and that is easy and cheap to collect (which, as discussed below, isn’t a given). If you have such a proxy outcome in hand, you can compare the impact of version A and version B of the program, and make inferences about which version will be more effective on the unmeasured downstream outcomes. But selecting a good proxy requires care. Optimizing on something that is partly predictive of downstream outcomes might not actually increase your program’s impact.

Consider the example of a university seeking to improve employment outcomes for its graduates. We recently conducted an impact evaluation of a blended university program in Rwanda and found that a competency-based curriculum –– featuring online courses from top international programs combined with in-person coaching and career prep services –– led to positive effects on employment rates and income six years later. 

Now suppose that you wanted to run some A/B tests on this program to increase its effectiveness or cost-effectiveness further, but you didn’t want to run tests that took several years for labor market outcomes to manifest in order to figure out which version of the program was most impactful. A reasonable approach might be to use exam scores as a proxy for downstream labor market impacts, and compare how different program versions perform in terms of increasing exam scores.³

But what happens if you start optimizing program design based on exam scores? It could quickly lead to counterproductive ‘improvements’. Maybe you eliminate a position in career services to add another member to the academic faculty. Maybe you get rid of internships so that students can take extra classes through the summer. Maybe you reduce funding for CV support so that you can increase funding for online study tools. All of these things might increase exam scores relative to the previous version of the program. But do they increase impact on the downstream outcomes like income? Maybe, but maybe not. They may improve academic skills while crowding-out the activities in the non-academic channels of impact. 

A/B testing without careful consideration of a program’s theory of change runs the risk of triggering Goodhart’s Law: “When a measure becomes a target, it ceases to be a good measure.” A better approach to strengthening the university program would be to keep constant the activities that are central to the theory of change – career support, for instance – and optimize around those.

2. Not giving enough time to work out the kinks of new program designs

Another pitfall in running A/B tests of social sector programs is that constantly updating intervention designs can compromise implementation fidelity. Any time that you implement a new program for the first time, there will be a myriad of issues that need to be fixed. That’s why funders caution implementing organizations not to dive into an external evaluation until they’ve successfully implemented the program for several cycles. 

What happens if you rush to test the status quo version A against a new version B? Participants who are used to the status quo program may be confused when shown a slightly tweaked program. Program officers may be unclear about which version they are supposed to deliver. With multiple different training scripts or program inputs circulating in the organization, some staff or participants will get the wrong version. It takes time to work out these types of kinks when updating the design of a program. 

Although speed is a key selling point of A/B tests, it’s imperative to take the time to pilot new versions before testing them. Otherwise the results of your A/B test will conflate the efficacy of the new program model with problems that arise when adapting implementation, which could prevent you from making changes that would improve the program in the long-run.

3. Acting on false positives or false negatives

When you assign individuals to two arms of an impact evaluation, you’ll never be able to achieve perfect balance: there will always be some difference in outcomes between the two groups simply because people are not exactly the same. The bigger the sample, the smaller the chances of large differences between groups – but you’ll never eliminate the possibility that an evaluation result is due to the randomness of who was assigned to each group rather than due to the intervention that each group is receiving. Typically researchers are happy if they can design an experiment where the probability of a ‘false positive’ – where the intervention isn’t having an impact, and the difference in outcomes that are due to chance – is less than 5%. Researchers tend to be more tolerant of ‘false negatives’ – where an intervention is impactful but you don’t get conclusive evidence from your evaluation – and design trials that limit the chances of false negatives to 10-20%.

For most A/B tests, you will likely not be observing large differences in outcomes, and therefore, you will need large study samples (hundreds or even thousands of individuals) to find statistically significant effects with acceptably low rates of false positives and negatives. This is rarely a problem for tech companies, who can amass thousands of website views within hours. It’s a bigger challenge for social sector organizations, who may not have thousands of participants available for A/B tests.⁴ 

But even if you have enough participants, if you run test after test, eventually you are nearly guaranteed to generate some false positives & false negatives. In other words, with data coming from A/B tests you are occasionally going to make the wrong choice about which one is the more effective design. 

Maybe some wrong choices in program design are tolerable; progress is not always linear, but at least your program design is trending in the right direction. But some  choices can be more consequential, particularly if design decisions build on each other; in those cases, getting reliable evidence requires an impact evaluation. For instance, suppose that you’re implementing a remedial curriculum for early-grade literacy, and you’re faced with the decision to deliver the curriculum through in-school volunteers or through teachers. Get this decision right and you will unlock the path to further impact; get it wrong and you will be plowing ahead with the wrong basic design. For the most consequential design decisions, A/B tests can’t substitute for good theory and well-powered large-scale impact evaluations.

4. Failing to account for one of the biggest costs of running impact evaluations: research expertise

A common claim of A/B tests is that they can “generate rapid results at low cost”, and thus A/B tests can comfortably fit within the R&D budgets of most organizations. The assumption is that an organization can leverage its own monitoring data with minimal additions or changes, so it won’t need to commission expensive primary data collection activities. However, some organizations may need to overhaul their monitoring systems in order to collect outcome data as frequently and comprehensively as would be required to run successful A/B tests. While upgrading monitoring systems is useful in its own right, it’s unlikely to make A/B testing a ‘low-cost’ venture.

Even if the data being used by A/B tests is automatically generated by the organization’s monitoring systems, it still requires research expertise to process and analyze that data, and to make statistical inferences to inform the right program decisions. Perhaps some organizations have this expertise in-house, but I suspect that many would need to hire external consultants or research organizations to support them. 

Research staff time is often the largest line item in evaluation budgets. So even in the best case scenario, where an organization already collects the exact data that would be needed to run A/B tests, a series of A/B tests could still cost close to a full-blown impact evaluation. Ideally, some data cleaning and analysis code can be reused test-after-test, so research costs come down over time. But you’re still talking about hundreds of thousands of dollars to set up and maintain the experimentation system. Can relatively marginal improvements in program design justify that cost? Maybe, if you’re talking about programs with budgets in the millions, that would be running A/B tests again and again. Otherwise, maybe not.

When A/B tests do and don’t work

I’m not arguing that A/B tests have no place in the social sector; they’re a powerful tool that can improve the design and cost-effectiveness of programs. But there are conditions that need to be satisfied for A/B tests to yield usable evidence:

When A/B tests work: A checklist for viability

• Proxy variables: 
  • Is there strong, contextually relevant evidence linking the proxy variable to long-term outcomes?
  • Do proxy variables capture all significant impact channels? If not, is there other evidence (e.g. qualitative interviews) that can be collected to capture unmeasured impact channels and unintended consequences?
• Theory-driven tests: 
  • Are the A/B tests grounded in a well-articulated theory of change?
  • Is there a clear hypothesis about how each variation affects outcomes? More tests is not necessarily better.
• Implementation fidelity: 
  • Have all program variants been adequately piloted?
  • Have communication guidelines been drafted and shared with staff and existing participants about who will receive each version of the program?
• Sample size and statistical rigor: 
  • Is the A/B test sample size large enough to detect meaningful, likely modest, differences in outcomes between program versions?
  • Are appropriate statistical methods – including multiple hypothesis corrections – used to correctly interpret results from successive A/B tests?
• Costs to run A/B tests: 
  • Do the anticipated gains in program cost-effectiveness justify the fixed and variable evaluation costs of A/B testing?

If these conditions are met, an organization might be ready to use A/B tests to optimize program design. If not, what can you do? Other evidence tools – notably examining the theory of change, landscape reviews, and process evaluation – can provide meaningful insights to inform program improvements (of course, these tools are not without their own pitfalls!). The most consequential changes to program design ––for instance, if you’re transitioning from implementing the program directly to supporting government to implement the program at scale ––should still undergo a full impact evaluation (measuring the impact of the program on long-term outcomes against a pure control group in a large sample). Impact evaluations themselves provide several opportunities for improving program design, including through testing multiple program variants simultaneously, and including midline data collection rounds to allow for learning & pivoting. 

A/B tests are a powerful tool for improving program design, if these pitfalls can be successfully navigated. Do you have examples of organizations in the social sector using A/B tests on non-digital programs to improve their effectiveness or cost-effectiveness? Please share any examples that you come across!