Evaluating Learning and Enrollment Gains for the Educate Girls Development Impact Bond
Client: Children’s Investment Fund Foundation (CIFF)
Partners: Educate Girls, Instiglio, UBS Optimus Foundation
Dates of service: 2012 – 2018
IDinsight service: Randomized evaluation
IDinsight contacts: Jeffery McManus, Kate Sturla
Media Coverage: The Economist, The Brookings Institute, World Finance, Devex
Additional Resources: Technical Report
Nearly three million girls, aged 6 to 14, are out of school in India. Even if enrolled in school, many students are not acquiring foundational skills like reading and basic arithmetic that can help them progress in school and life.
Educate Girls, a Mumbai-based NGO, trains community volunteers to encourage families to enroll their girls in school by making door-to-door household visits and to deliver a child-friendly supplementary curriculum in classrooms to improve basic reading and math skills. Educate Girls currently operates in over 12,000 villages and 21,000 primary schools across 15 districts in Rajasthan and Madhya Pradesh.
A Development Impact Bond (DIB) has been set up to finance Educate Girls’ program in Bhilwara district, Rajasthan. DIBs are a new tool for financing social programs in which an investor provides capital up-front to a service provider and earn a return from a donor based on the effectiveness of the program. By shifting the focus from inputs to outcomes and incentivizing service providers to improve programs, DIBs hold the potential to make social programs more cost-effective. The lessons from this world’s first DIB are informing the viability and design of future results-based financing models.
The Educate Girls Development Impact Bond (EG DIB) is a joint project between the Children’s Investment Fund Foundation (CIFF), Educate Girls, the UBS Optimus Foundation, Instiglio, and IDinsight (collectively, the “Working Group”). UBS Optimus, acting as the investor, financed EG’s project implementation, while CIFF paid for educational outcomes as evaluated by IDinsight. Instiglio is managing the project.
IDinsight designed and conducted a three-year impact evaluation of Educate Girls’ program in Rajasthan to determine financial returns in the EG DIB. We measured two outcomes: learning gains of boys and girls in grades 3-5 and enrollment of out-of-school girls.
Learning gains, which made up approximately 80% of the final DIB payments, were evaluated in a village-clustered randomized control trial, the gold standard of scientific evidence. The evaluation involved a sample of ~12,000 students across 332 schools in 282 villages. Half of these villages were randomly assigned to receive Educate Girls’ program while the other half represented the comparison group. Learning gains were calculated as an aggregate treatment effect (instead of an average treatment effect) to incentivize Educate Girls helping low performers stay in school.
We assessed students on basic literacy and math competencies using the Annual Status of Education Report (ASER) testing tool; a student’s score on ASER determined her “learning level,” which is scored out of 16 points and forms the basis of the learning metric. Student assessments were conducted at the beginning of the project (baseline) and the end of the three following school years. IDinsight teams visited both treatment and control schools in the same geographic cluster on the same day. If students were not present at school on the day of the assessment, our surveyors assessed them at their homes.
Enrollment of out-of-school girls, which contributed the remaining ~20% of the payments, was measured using a simple pre-post evaluation in the 141 villages where treatment schools are located. Due to the cost of conducting a census of all households, the project partners decided against estimating enrollments in control villages. Thus, unlike the learning estimates, the enrollment estimates do not reflect a causal effect of EG’s program.
The enrollment outcome was calculated as percent of out-of-school girls enrolled out of all eligible out-of-school girls identified in the program villages by the end of the program. At the beginning of each school year, Educate Girls submitted a list of all out-of-school girls ages 7-14 living in treatment villages. IDinsight validated that list by visiting a sample of girls in the area and verifying their eligibility. At the end of each school year, Educate Girls reported successful enrollments and IDinsight validated all enrollments by visiting the schools and verifying the enrollment in the school register.
In addition to determining final payments, midline results from IDinsight’s evaluation enabled Educate Girls to fully realize the flexibility in programming provided by DIB set-up and make effective, data-driven course corrections along the way.
IDinsight submitted a pre-analysis plan for the evaluation at the Registry for International Development Impact Evaluations (RIDIE). For more information on the evaluation design, please refer to the pre-analysis plan or the DIB design document.
Educate Girls surpassed the DIB targets for both learning gains and enrollment. Students in program villages gained an additional 8,940 ASER learning levels relative to students in control villages, surpassing the learning target set by the Development Impact Bond by 60%. The effects of Educate Girls’ program on learning gains were large and statistically significant over the three-year program: Students in EG schools gained on average an additional 1.08 learning levels, or 28% and 0.31 standard deviations, compared to students in control schools. By the end of the three-year project, Educate Girls had enrolled 768 out-of-school girls, representing 92% of all identified out-of-school school girls eligible for enrollment. Educate Girls thus exceeded the enrollment target of 79% by 16%.
The success of the world’s first ever DIB has enabled Educate Girls to seek additional funding to scale up its proven program. IDinsight is utilizing the experience and knowledge to participate in and advise several other DIBs, including the Village Enterprise DIB for poverty alleviation in sub-Saharan Africa.