Roundup from E2A 2019: Innovations in Health Data and Measurement

Strengthening health systems’ capacity

Low- and middle-income countries often have low-quality healthcare systems. In fact, the Lancet Global Health Commission argues that “poor-quality care is now a bigger barrier to reducing mortality than insufficient access.” This year’s Evidence to Action symposium, organized by the Center for Effective Global Action (CEGA), featured promising new research to improve health systems and outcomes worldwide.

Bilal Siddiqi (CEGA) presented results from a randomized control trial (RCT) aiming to improve the performance of health clinics in Sierra Leone, wherein 2010, the government launched a free healthcare initiative to combat their high maternal and under-five mortality rates. To improve implementation, they tested two programs implemented by the Government of Sierra Leone in collaboration with the World Bank and three non-governmental organizations. The first included a community monitoring arm with scorecards rating local health problems (including health outcomes and health workers’ performance like absenteeism), meetings for health workers and communities to develop joint action plans, and follow-up meetings to monitor progress. The second used non-financial rewards, involving competitions among clinics to improve health outcomes and worker performance, along with outreach and accountability mechanisms like posters and public awards ceremonies. (presentation slides, video)

Both programs led to more patients using clinics and higher patient satisfaction. In addition, the community monitoring group demonstrated large improvements in child health, including a decrease in under-five mortality from 39 to 24 children in 1000.

During the 2014 Ebola outbreak, more than a year after the initial study concluded, researchers returned and observed that the positive impact on clinic functioning persisted: both resulting in increased reporting of Ebola cases by 60 per cent, and large effects on patient survival (especially for the community monitoring arm). The authors were able to rule out other factors that could have contributed to these results, concluding patient’s improved trust in local health systems increased their likelihood of seeking care.

What stands out about this?

Both programs were light-touch — low-cost and relatively easy to implement– and community monitoring still led to impressive improvements in health outcomes, including a reduction in under-five mortality (by 15 in 1000). As a comparison, this impact is higher than the distribution of Long-Lasting Insecticide-Treated Nets (by about 5.5 in 1000 based on a meta-analysis¹), which is considered by charity evaluator GiveWell to be one of the most cost-effective interventions in global health. An earlier study testing a version of the community monitoring intervention in Uganda also found significant improvements in health outcomes (e.g. a reduction in under-five mortality of 17 in 1000 ²), though a similar study in Uganda found no impact on utilization rates or health outcomes despite a positive impact on treatment quality and patient satisfaction³. This looks like a potentially promising intervention for strengthening health systems that deserves further research to understand where it works and why.

A number of other presentations at the symposium focused on improving healthcare quality. Pius Akankwasa (Office of the Prime Minister, Uganda) presented a pilot project on a fingerprint-based monitoring system to reduce health worker absenteeism (found to be 37 per cent in Uganda in 2006⁴. The pilot faced some challenges, including infrastructure problems (like erratic power supply and poor network connectivity in remote districts) that hinder smooth implementation, negative reactions from health workers (vandalism or theft of equipment), and delayed or lack of action by district supervisors. In India, a similar fingerprint monitoring system in government health clinics showed some success but also had complications that limited its effectiveness⁵. This is a potentially promising intervention, and it will be interesting to see how the Uganda team explores ways to resolve implementation challenges. (presentation slides, video)

Reshaping supply chain incentives to improve health outcomes

Challenges ensuring the quality of pharmaceutical products have imposed great costs on health in the developing world. In health markets traditionally characterized by low quality and low patient trust, technologies that reduce the cost to verify quality can improve consumer welfare and reward producers and sellers of high-quality products. Bright Simmons shared the story of mPedigree, the technology company he founded to fill in this gap. mPedigree uses mobile technology to verify the origin of pharmaceutical products and can predict exposure to improper handling (like changes in temperature) in the transportation process. The company is now serving consumers, retailers, producers, and governments in 14 countries across Africa and Asia. It has expanded into sectors beyond pharmaceuticals, including a partnership with the Common Market for Eastern and Southern Africa to verify seed quality. (presentation slides, video)

Maria Dieci (UC Berkeley) presented another supply chain innovation aimed at reducing malaria misdiagnosis. She partnered with Maisha Meds, a nonprofit in Kenya that provides digital tools to manage pharmacy sales and inventory, to test whether different incentives for pharmacists and consumers can help improve targeted malaria treatment. If successful, these interventions have the potential to reduce diagnosis and treatment mismatch as well as unnecessary antibiotic use, which is a major problem in global health. (presentation slides, video)

Measuring the long-term impact of health interventions

Tracking a program’s long-term outcomes is the best way to understand its impact and cost-effectiveness. However, in the context of development research, this is rarely done due to data limitations, for example, the dearth of high-quality administrative data that can be linked to program participation, or the cost of tracking and surveying respondents long after the program ends.

Ted Miguel (UC Berkeley and CEGA) presented an exceptional study on the long-term outcomes of a deworming study conducted in Western Kenya between 1998 and 2003⁶, which subsequently contributed to the founding of Deworm the World that implements deworming programs in multiple countries. Twenty years after the program began, they found increases in income and consumption of 6 to 14 per cent among children who received three additional years of deworming during primary school, consistent with findings from the earlier ten-year follow-up⁷. The authors were able to measure long-term outcomes thanks to meticulous tracking of respondents over decades using the Kenya Life Panel Survey. Follow-up with the children of the original study participants has been planned, so stay tuned for even longer-term results. (presentation slides, video)

Julius Rüschenpöhler (CEGA) presented on the Long-term Impact Discovery Project, a CEGA initiative to track the long-term impact of global health interventions to inform policy and funding decisions. Specifically, they look at RCTs of childhood health interventions that have the potential to transform lives. After examining the designs of many studies (including power, attrition, feasibility for long-term follow-up etc.) on cash transfers, malaria treatments, nutrition supplementation, and diarrhoea treatments, they recommended 28 studies for follow-up. These unique opportunities to study the long-term impact of development interventions have significant potential to inform governments’ and philanthropists’ approaches. As researchers and funders continue to make progress, they are also helping measurement for long-term impact gain more prominence in development research and influencing the design of future studies. (slides, video)

High-income countries typically have administrative data that enables better outcome tracking over the long term. Hilary Hoynes (UC Berkeley) presented a study on the long-term impact of childhood exposure to the United States’ Food Stamp program using variation from the county-level roll-out of the program between 1961 and 1975⁸. It shows that access to these resources from conception to age five leads to an increase in human capital (0.06 standard deviation in a human capital index — a bit smaller than that of Head Start). It also increases neighbourhood quality and reduces mortality (0.4 percentage point or 11 per cent — a bit higher than having access to Medicaid over the same period) and the likelihood of being incarcerated (0.5 percentage point). The study puts a human capital investment lens on safety net programs, similar to this analysis on the long-term rate of returns of various US social policies⁹. (presentation slides, video)

Reducing bias and increasing diversity in medicine

Marcella Alsan (Harvard) presented a study on the effect of diversity in the physician workforce on the demand for preventive care among African-American men¹⁰. They conducted an RCT and found that black male patients selected more preventative care services (particularly invasive services) and communicated better with the doctor when randomly assigned to a black male doctor, compared to when matched with a male doctor of a different race. Since black men have the lowest life expectancy of any major demographic group in the US, and most of the disadvantage is due to chronic diseases amenable to prevention, such interventions can be effective in improving their health outcomes. The authors calculate that the results imply black doctors could reduce the black-white male gap in cardiovascular mortality by 19 per cent. (slides, video) Asha Vitatoe (Mentoring in Medicine & Science) explained how the study findings bolster the work that her organization does to bring more diversity into the healthcare workforce. (presentation slides, video)

Ziad Obermeyer (UC Berkeley) presented a study illustrating the importance of training machine learning (ML) algorithms on data from ethnically diverse patients to reduce bias and improve diagnostic accuracy. When trained on X-ray images of knees to predict the self-reported pain level of patients from a range of ethnicities, machine learning algorithms provide more accurate diagnoses than radiologists, and performance is hurt when images from ethnically diverse patients are removed. (presentation slides, video)

Cash benchmarking, naps, and words

Craig McIntosh (UC San Diego) presented a cash benchmarking study¹¹ where a combined nutrition, water, sanitation and hygiene program in Rwanda was compared to cash transfers of different amounts: one equivalent to the program cost (141 USD per household), and one larger (532 USD per household). They found that only the larger cash transfer had an impact on outcomes including consumption, assets, dietary diversity, child mortality, and child anthropometric outcomes, while the nutrition intervention and small cash transfer produced little-to-no impact on health outcomes but had some impact on savings, assets and debt. Jeanine Condo, former Director of the Rwanda Biomedical Centre, provided commentary on health programming in Rwanda and the implications of the study. Cash transfers have been demonstrated to be a relatively effective development tool and can be implemented with relative ease and flexibility, especially via mobile money. As a result, cash can serve as a useful benchmark, providing a way to measure whether development programs perform better than simply giving recipients the cash equivalent to its cost. USAID, CEGA, and GiveDirectly have partnered on a number of cash benchmarking studies, which is a great precedent in the development space. (presentation slides, video)

Mattie Toma (Harvard) presented a study that tested interventions to improve sleep quality in Chennai, India, on a sample of low-income individuals who slept 5.6 hours a day on average¹². They found that encouraging daytime naps (averaging 16 min) improved a range of outcomes, including attention, productivity, and psychological wellbeing, while increased nighttime sleep (by an average of 35 min) did not. (presentation slides, video)

Monica Ellwood-Lowe (UC Berkeley) presented a study that examined the impact of financial concerns (induced naturally by either how distant the last paycheck was, or induced experimentally in the lab) on the amount that parents speak with their children, measured by an electronic word counting technology, summarized by my colleague Crystal Huang here. (presentation slides, video)

The studies presented here are not only fascinating but have policy relevance and potential to save and improve lives around the world. We look forward to seeing more such research coming out of the CEGA community, including innovations by and collaborations with practitioners and policymakers from countries where the results will apply.