Informing specific decisions with data and evidence

Executive Summary

Impact evaluations of development interventions have increased dramatically over the past 20 years,¹ expanding from a research tool of academics that has recently been awarded with a Nobel Prize² to a decision-making tool by policy-makers. Despite this expansion in use cases, the methodological approach to design and analyze impact evaluations has remained mostly constant. This standard approach tends to test whether a program works, i.e. whether its effect is different than zero. Conclusions from this test implicitly assume that consumers of the research are an academic audience that is interested in generalizable knowledge and skeptical of any evaluation results. Therefore, the standard approach requires a relatively high level of certainty to convince the reader that results are “true.”

We argue that in cases where the purpose of the evaluation is to inform a specific decision, researchers should consider alternative approaches to design and analyze impact evaluations. The unifying feature of the alternative approaches to design and analyze impact evaluations we discuss is that they explicitly consider the specific decision makers’ circumstances and decision framework. While this approach isn’t necessarily new, we hope to provide practitioners with an accessible and practical treatment of the subject.

Specifically, we outline two approaches. In the first, we retain the standard frequentist statistical approach to impact evaluations but outline how certain “default” parameters can be modified to take specific decision frameworks into account. For instance, in certain cases, decision-makers may be okay implementing a policy even with relatively high uncertainty as to its effectiveness. Second, we show how Bayesian analysis may more directly account for a decision maker’s beliefs and preferences. We give an overview of a Bayesian approach to evaluation and illustrate how to implement it in practice, including hands-on guidance regarding sample size calculations, analysis, and interpretation of results. Finally, we discuss how an evaluator can choose between frequentist or Bayesian approaches.

The high-level takeaways from this paper are as follows:

• There are a number of common circumstances where the standard frequentist approach to impact evaluation is not ideal for decision-making. These scenarios include testing for “noninferiority” of a new policy vs the status quo, testing two policies head to head, and making a decision based on a certain effect size threshold.
• When designing an evaluation for decision-making in the frequentist framework, the researcher should consider carefully the decision-relevant level of certainty (size) of the test, which may be different from the standard .05. They also may want to consider whether a one-sided test or multiple hypothesis tests are appropriate.
• Bayesian analysis has distinct advantages in a decision framework, as it can quantitatively incorporate the decision makers’ prior beliefs into the estimation. It also allows evaluators to make easily understood probabilistic statements such as “the probability of program A being better than program B is 60%.”
• Bayesian analysis tends to give the same results as frequentist analysis in conditions where priors are uninformative and models are fairly simple or sample sizes are sufficiently large. Gains to Bayesian analysis arise primarily in circumstances where interpretation of the conventional approach seems less intuitive, when priors elicited from decision-makers or existing studies from similar contexts contain relevant information, and where sample sizes are restricted.
• In both frequentist and Bayesian analysis, taking the needs of the decision-maker into account can, in many circumstances, result in studies with a smaller sample size, while maintaining desired levels of Type 1 error and statistical power. In the frequentist approach, this can result from moving to one-sided tests or accepting lower levels of certainty to drive decisions. In the Bayesian approach, this can result from using tailored decision rules that better reflect the decision maker’s utility function in designing the study.
• Following good research practices, such as the pre-registration of the analytical approach, allows evaluators to maintain high levels of rigor for these adapted approaches to decision-focused evaluations.

The intention of this document is to serve as a guide to those designing evaluations for decision-makers with the intent of allowing for more directed evaluations to ensure maximum policy impact.