Why (and how) we are measuring our own impact

Click here to read IDinsight’s 10-year retrospective. 

There’s a common saying in American English: “You’re going to get a taste of your own medicine.” It derives from one of Aesop’s fables, “The Quack Toad,” which is so short you can read it here:

An old Toad once informed all his neighbors that he was a learned doctor. In fact he could cure anything. The Fox heard the news and hurried to see the Toad. He looked the Toad over very carefully.

“Mr. Toad,” he said, “I’ve been told that you cure anything! But just take a look at yourself, and then try some of your own medicine. If you can cure yourself of that blotchy skin and that rheumatic gait, someone might believe you. Otherwise, I should advise you to try some other profession.”

Over the years, the “taste of your own medicine” has been interpreted to mean that people prescribing an unpleasant solution should suffer the fate of having to take it themselves. But we’ve lost the deeper meaning that Aesop offered as the moral to that tale:

“Those who would mend others, should first mend themselves.”

Across IDinsight, we dedicate hours, days, weeks, months, and even years – our very life energy – to squeeze every ounce of value from data and evidence so that our clients can more effectively improve the lives and livelihoods of people who depend on NGOs and government programs. We believe that looking carefully at both qualitative and quantitative data can yield insights that help leaders make better decisions about where to allocate resources and how to do their work.

That is medicine: sometimes more like a vaccine, preventing a not-so-great idea from absorbing resources that could better be used elsewhere; sometimes more like a treatment, solving a problem so that high-impact efforts can proceed. But medicine it is.

As advised by Aesop, we are now taking our own medicine.

We’ve spent nine months collecting information on the 140+ projects we’ve done over ten years. We’ve asked people who worked on those projects what methods we used, how large and consequential the questions we tackled were, and what the NGOs, philanthropies, and governments did with the findings and recommendations we generated. 

In some cases, the responses made us realize that we’d worked hard with little to show for it. Ouch. In many others, though, we learned that the work had paid off in real-world improvements in lives and livelihoods. We’ve summarized these findings in our microsite, and are reflecting among ourselves on all the details.

Interesting as the results may be to our supporters, the primary objective of this exercise was not to aggregate and report to funders the total contribution we’ve made over a decade. The point was to help us meet our ambitious goals for social impact by working smarter. And working smarter means benefiting from accumulated experience, carefully analyzed.

Like any organization, we make lots of decisions that affect our ability to make a difference in the world or not.

For example, we make choices about which NGOs and government agencies we seek to work with. We make decisions about which methods to apply in a given project – and which technical capabilities to build in the organization over time. We make decisions about which sectoral expertise to invest in, and whether to place priority on long-term projects with more open-ended objectives or shorter, urgent ones. 

Every time we face one of those decisions, we wish for a crystal ball to see into the future: how is this going to turn out? While there are no crystal balls, we do have rearview mirrors: We can systematically learn from experience to detect patterns that can help us make the best choices about which projects to take on and how to serve partners most effectively. 

The in-house team that (bravely) took on the retrospective assessment of ten years of impact was charged with helping us answer questions that are crucial to our work. For example:

• Relative to the early focus of IDinsight, decision-focused impact evaluations, how effective have other services (like the application of data science methods like machine learning) been in generating social impact? What does that tell us about the services we should offer in the future?
• Can we identify features of projects that make it especially likely (or unlikely) that we’ll succeed in helping our partners improve the impact of their programs? What does that tell us about the projects we should seek out – and the ones we should stay away from?
• As IDinsight grew from 5 to 200 team members, did we become more (or less) efficient at contributing to social impact? What does that tell us about how we should structure ourselves in the future?

Tackling this took courage because of the analytic design complications.

To get a large enough data set, we went all the way: collecting information about each and every one of the projects IDinsight has completed over ten years. To obtain comparable information about both the features of the projects and the estimated contribution, the team developed a detailed rubric, and interrogated the respondents (their colleagues) to get data we have confidence in. To be able to make comparisons across projects with different aims, the team boldly proposed a synthetic measure, combining direct contributions and indirect influence. And, most courageously of all, we talked about “impact” despite the absence of a counterfactual. (Actually, that’s not true. In the end, we couldn’t bear to talk about impact so we ended up with “contribution.”)

As you might imagine, this exercise required a level of staff time and energy that would not have been available without dedicated funding. This undertaking was made possible with a portion of the funding we received a year ago from MacKenzie Scott and Dan Jewett. This is just one example of the ways we are using those resources to take our work to a new level of depth, effectiveness, and value to our partners.

While the retrospective analysis won’t replace the judgment calls each IDinsight team member makes, it enhances our ability to do the right things, the right way. It is this taste of our own medicine that will make us stronger every day – better able to serve our partners, and better able to use the resources entrusted to us by our funders.