AI systems are increasingly being used in public service delivery settings; helping health workers triage patients, guiding users through government services, providing agricultural advice, and answering questions on a wide range of topics at scale. As this shift happens, evaluation becomes urgent and unavoidable for organizations building, funding, and deploying AI.
IDinsight is working to fill in this critical gap. We have developed sector-wide shared standards for how to evaluate AI applications, and are working with partners to evaluate their AI portfolios.
IDinsight, in collaboration with the Center for Global Development and The Agency Fund, and supported by the Gates Foundation, has developed a living AI evaluation playbook.
Our end goal is to create a resource that helps organizations evaluate their AI systems rigorously and sustainably for greater impact, grounded in the realities that social sector organizations face.
A public, four-level framework for rigorously evaluating generative AI tools in global development.
Most AI tools in the social sector get evaluated one of two incomplete ways: tech teams check whether the model performs, impact evaluators check whether outcomes improved, and the two rarely talk to each other.
The AI Evaluation Playbook closes that gap. With input from 30 cross-disciplinary experts (computer science, economics, gender studies, development), the Playbook lays out how to conduct evaluations for the four connected levels of evaluation:
These serve as a practical blueprint for implementers as they move through stages of maturity of their product from piloting to scale.
Use the AI Evaluation Playbook to design better evaluations, catch failures early, and build an evidence base around AI tools that deliver
Building QA standards for AI teacher tools in LMIC education, helping governments and funders spot what works before scaling.
Partners: FabAI, funded by Gates Foundation
This project measures the quality, safety, and effectiveness of AI-powered EdTech tools, specifically AI tutors and teacher copilots for foundational literacy and numeracy. FabAI leads benchmark development, while IDinsight contributes RCT capacity and geographic reach, with each organization building skills in the other’s area of expertise along the way. The work is product-agnostic: the goal is to inform LLM developers, product builders, and EdTech buyers and users about what is working, not to endorse specific tools.
Read more ->
Building a learning agenda to measure Claude's impact across health, education, economic mobility, and life sciences.
Partners: Anthropic
Through its Beneficial Deployments team, Anthropic gives free or discounted access to Claude and dedicated technical support to select nonprofits, governments, businesses, and other institutions working on global health, education, life sciences, and economic mobility. IDinsight is advising on a learning agenda and associated M&E framework to help the team assess the impact of these deployments – how partners adopt and use Claude, what end-outcomes are enabled by the AI solutions created, and what’s driving these results. AI’s rapid progress opens enormous potential for social good, but realizing it is not a given. Rigorous and nimble MEL that can keep up with the fast pace of AI deployments ensures the team learns whether the benefits are materializing, for whom, and what to adjust when they aren’t.
Building the evidence base on what works in AI-for-development across IDRC's portfolio of AI4D grantees in Africa.
Partners: International Development Research Centre (IDRC) and UK Foreign, Commonwealth & Development Office (FCDO)
This initiative supports AI4D as a strategic learning partner as part of the AI Evidence Alliance for Social Impact, generating evidence on whether AI innovations deliver social impact and for whom. We are co-developing a learning agenda and a funders’ theory of change, running targeted evaluations of selected AI4D grantees, and facilitating knowledge sharing across the AI for Development Funders Collaborative — with cross-cutting themes of gender equity, social inclusion, and local engagement throughout.
Strengthening M&E for Google.org's AI Collaborative across food security and wildfire response.
Partners: Google.org
This project strengthens monitoring and evaluation for Google.org’s AI Collaborative across two domains: food security and wildfires. The work spans building collaborative-wide M&E systems and providing targeted evaluation support to individual grantees, addressing two core questions: whether the Collaborative model is amplifying grantee impact, and whether grantee activities are improving outcomes in each domain.
Learn more ->
Building evaluation capacity and rigorous evidence for GitLab Foundation's Economic Opportunity Fund grantees.
Partners: GitLab Foundation
This project helps GitLab Foundation’s Economic Opportunity Fund grantees build real evaluation capacity and generate evidence on whether their AI tools are shifting intermediate (level 3) outcomes. The work starts with portfolio-wide workshops on theory of change, measurement, and experimentation, then goes deep with a handful of high-potential grantees to co-design and execute level 3 evaluations on whether tools are shifting knowledge, attitudes, and behaviors.
A public, living resource bridging technical rigor and real-world usability in AI evaluation methods.
Partners: Centre for Global Development (CGD), The Agency Fund, and the Gates Foundation
This project creates a public, living resource that bridges the gap between technical rigor and real-world usability of evaluation methods in the context of AI. Developed in partnership with technical experts and implementing organizations, the playbook is a practical, sector-wide reference to help organizations rigorously evaluate AI solutions at any level, reducing duplicative effort and raising the quality of AI evaluation practice across the social sector. It is rooted in a four-level framework covering the model, user, product, and impact of an AI solution.
See the playbook here ->
A living repository mapping how AI is being deployed across the social sector, worldwide.
This project builds a comprehensive, living repository mapping how AI is being deployed across the social sector, spanning sectors, geographies, and stages of maturity. By creating and maintaining a credible, centralized resource, the repository helps funders identify gaps and avoid duplication, while giving nonprofits a practical starting point to learn from peers. The approach combines automated web crawling, human verification, and crowdsourcing.
Visit the site here ->
24 July 2026
25 August 2026
25 February 2026
9 December 2025
19 September 2025
28 August 2025
5 August 2025