Closing the gap between data and decisions: Two practical tools

For those of us in the monitoring, evaluation, research, and learning space, the importance of data and evidence-driven decision-making is obvious. Policies, programs, and government services are more effective when informed by data and evidence. Yet, decisions are often made without the benefit of data, and data are frequently collected without a clear decision in mind. Even when data are available, they often get buried in the information overload that includes anecdotes, outdated expert opinions, misinformation, and disinformation. Within this treacherous landscape, how can we bridge the gap between the production of data and evidence and decision-making?

The Problem: decisions without data, data without decisions

Policies are too often created, programs launched, and implemented at scale with limited evidence. They often have no impact and waste resources in the process. Similarly, problems that are unknown or underestimated persist. The reverse problem also exists: data and evidence are produced without a clear path to informing decisions. Evaluations are conducted, reports are written, and these valuable insights end up collecting dust on a shelf or buried in academic journals, reaching only a limited audience. Organizations like the Abdul Latif Jameel Poverty Action Lab (J-PAL) and Innovations for Poverty Action (IPA) were established to tackle this gap from the supply side, identifying effective programs and then translating research findings into actionable policy recommendations that are accessible and easy to understand. IDinsight was founded to close this gap from the demand side, working with policymakers and decision-makers to generate data and evidence they explicitly request. Today, J-PAL, IPA and IDinsight operate on both sides, and for many programs and polices the bridge has been built, and lives have improved. However, the policy successes are few and far between relative to the magnitude of both problems and policy failures. The disconnect remains widespread.

How data and evidence improve decision-making

Ideally, data and evidence improve decision-making by providing new information, which can shift prior beliefs, reduce uncertainty, or both. When faced with policy choices, decision-makers must navigate unknowns: Which development goals are the most off track? Which intervention will positively impact the outcomes we care about? Is it worth the investment? Are there better, more cost-effective approaches? Can this program be implemented with fidelity? At scale? What are the weakest links in the theory of change, and how do I know if they’re broken, rendering the program ineffective? What strategies can be used to fix those broken links?

With enough uncertainty or incorrect prior beliefs, we can choose the wrong path, making decisions that harm, or at least fail to benefit our cause. Or, without sufficient awareness of the likely truth, we may fail to recognize that action must be taken, leaving the problem unresolved. These known-unknowns or unknown-unknowns risk the perpetuation of suffering and the waste of precious resources.

Reliable information, backed by data and evidence, can significantly reduce uncertainty.

Closing the gap: Two practical tools

Two key barriers to evidence use are 1) not making the right evidence investments in the first place and 2) behavioral barriers that make indecision the default. Here, I introduce tools to address each: The “Value of Information” approach and the “Response Framework”.

1. Value of Information Approach

With unlimited resources, one could explore every unknown and theoretically make uncertainty disappear. This is obviously not going to happen, nor should it. Firstly, and most obviously, we don’t have unlimited resources. And secondly, even if we did, the benefit of reducing uncertainty—making every decision with a higher probability of success—is diminishing, and for many decisions, and at some point for all decisions, not necessarily worth the cost. The Value of Information (VOI) approach enables us to compare the investment costs in evaluation to the expected value of improved decisions, treating evidence generation as an investment. Here’s how it works:

1. Formalizing current default decisions and underlying assumptions
2. Defining decision rules that would lead to alternative choices
3. Acknowledging uncertainty around current assumptions
4. Quantifying the cost of making the wrong decision
5. Determining how different levels of evidence investment would reduce uncertainty
6. Comparing investment costs against the expected value of improved decisions

For example, if our teacher training program is projected to cost $10 million at scale, we might spend $500,000 on a rigorous evaluation during the pilot. If we believe there is a 30% chance that the program doesn’t work, and the study can reveal this, the VOI calculation would compare the evaluation cost ($500,000) against the potential savings from avoiding a failed program ($3 million in expected value). In this case, the evaluation represents a worthwhile investment, with an expected value of $ 2.5 million. Beyond making yes-or-no decisions to invest or not invest in evidence (and therefore eliminate or live with uncertainty), we can also decide on the size of the investment and the extent of uncertainty we can tolerate. A $100,000 investment might only give you a 90% chance of revealing a failure of that same program. Now, the potential savings from conducting this evaluation are 90% of $3 million, or $2.7 million, relative to the $100,000 cost. That’s $2.6 million in expected value. Depending on your risk tolerance, the cheaper study with more uncertainty might be a better investment.

This example can be extended to many possible investments and decisions—different nodes of uncertainty, different evaluation types and methodologies, binary or multi-option decisions, sequential and conditional evaluations and decisions, and much more.

2. The Response Framework

The Response Framework is a structured approach to ensure that data and evidence production are directly linked to decision-making. It involves mapping learning questions—derived from scrutinizing a theory of change or facing a policy challenge—to indicators, metrics, and specific decisions. These decisions are tied to decision rules, such as metric targets or performance scores, and are assigned to specific decision-makers with clear timelines. 

For a teacher training program intended to promote student learning, we may collect data and evidence on teacher knowledge, attitudes, and practices, as well as classroom dynamics and student learning outcomes. We may examine constraints that limit teacher training attendance, such as access to transportation or intrinsic motivation. We may also examine constraints to student learning in case classroom practices aren’t the binding constraint. (The binding constraint could instead be teacher or student absenteeism, student misbehavior, undernutrition, or parasitic infection reducing cognitive capacity, for example) We may review costs for different program elements at various stages. We may want this information before designing or launching the program, or during implementation of the pilot, or after we expect the pilot’s impacts to materialize, or before the program is scaled, or at scale. Each of these pieces of evidence at different points in time would likely inform different decisions. The table below illustrates a small segment of this program’s Response Framework, where data reduces uncertainty related to teacher compliance, and where decisions stemming from the discovery of non-compliance are predetermined.

By listing the specific steps—the triggers, decisions, the decision-makers, the timeline—the default decision gets closer to action. Without the plan, the default decision is often the path of least resistance: to interpret results that may justify action, but still do nothing about it. The response framework is that plan.

Surmounting barriers in a world of constraints

Let me be clear: the underutilization of evidence is not the only barrier to good policies. Evidence is just one input into policy decisions, which also have to navigate political, financial, and administrative constraints. But to maximize the likelihood of making evidence-informed decisions,  I’m suggesting that:

• Data producers recognize that our evidence is itself an intervention. To be good stewards of our limited financial resources, we have a responsibility to ensure that our intervention (i.e. evidence-production) is cost-effective.
• Decision-makers, as humans, can fall victim to decision inertia. While inertia might not be the only constraint to good decision-making, simply making a plan on the actions we intend to take can help overcome at least one behavioral barrier.

The Value of Information approach and Response Framework offer practical solutions to overcome those behavioral barriers and close the evidence-decision gap. 

In a world of limited resources and pressing challenges, we can’t afford to make uninformed decisions or generate unused evidence. Bridging this gap isn’t just good practice—it’s essential for delivering the results our communities deserve.