Context as a public good

LLMs are getting incredibly good – unless you’re one of the 6 billion people living outside a high-income country. For the global majority, the same models are unreliable – they hallucinate and fail significantly more often. New models might do better but are not going to automatically solve the problem. 

We already know the fix; it’s augmenting the LLM at inference time. Instead of expecting an LLM to remember every fact for every context, we equip them with definitive knowledge sources. Numerous impactful solutions have been built using this Retrieval Augmented Generation (RAG) architecture. 

In this article, I discuss how we can do better. I make the case for a public good that captures the context of the people and places LLMs too often leave behind. We need a shared, trusted, knowledge base every AI can draw on – continuously updated and widely available. I show how this will not only increase efficiency and quality of AI solutions, but unblock innovation by reducing the barriers faced by social sector AI solution builders.

From Vanilla LLMs to RAG

LLMs are not designed to store knowledge. They compress language patterns in data and use it to produce the next most probable word (or token). The fact that they are so good at facts is an incredible emergent property but there are few guarantees.

The “reverse curse”¹ should give us pause when thinking about the nature of this knowledge. “Hallucinations” are just a more insidious and a much harder to detect version of this.

Implementers have known about this unreliability for a while. Lucky for us, information retrieval is a well studied area. We can shift the task of selecting relevant information to a deterministic, verifiable system and provide these relevant snippets to the LLM to generate the answer. And so the RAG architecture was born.

A sidebar on RAG RAG stands for Retrieval Augmented Generation. It is three steps, and it is worth seeing them separately because they fail in different ways. Say a nurse asks an app how much amoxicillin to give a six month old with pneumonia. Retrieval: The system searches a collection of documents in the context layer e.g. the national treatment guidelines, and pulls out the passages most likely to answer the question. No LLM involved. This is ordinary search, and it either finds the right page or it doesn’t.  Augmentation: Those passages are added to the prompt, alongside the nurse’s question. The model is now sitting an open-book exam instead of a closed-book one. Generation: The LLM reads the question and the passages, and writes the answer in plain language. Almost all the attention goes to Generation, because that is the part that feels like magic. But if Retrieval hands over the wrong page, or the right guideline was never in the collection to begin with, no amount of model quality saves you. A better LLM will just write a more fluent wrong answer.

When your favourite chat product does a web search, it’s doing the same thing – it’s relying on the search engine’s information retrieval system. But the web is also full of misinformation and trolls, and pretty useless for queries specific to low-income country contexts. Leaving it to the LLMs to sift through it can lead to unexpected behaviours and expose it to new vectors of misinformation.

Where we are today

So we ended up in this world:

Today, each organization builds its own curated database of content, private only to them, and struggles through the same problems of correctness and completeness but also chunk-size, embedding and retrieval technique, model choice etc.

LLMs are great and getting better at instruction following and, as a result, at the “Generation” part of RAG². Most app developers just need to upgrade their LLM to the one just released and they’d get better Generation. Unless the problem was in the Retrieval – the correct answer was not provided to the LLM or the correct answer was never there in the collection.

What if we could have a similar model for Retrieval? One organization (or government) works super hard on building a high quality context layer either by sector or geography – all their academic standards and curriculum, their health guidelines, their benefits eligibility criteria³. They make it available as a data or “context layer”, possibly through MCP or an API + agent skills, for anyone building AI solutions for their geography or sector. Implementers can then focus on their end users – the UI/UX, the user journey etc. 

So the world would look like this:

Like with LLMs, if they want to improve their RAG performance, they just need to upgrade to the latest version of the “context layer”. 

This is already happening in Education. Chan Zuckerberg Initiative (CZI), through Learning Commons, has encoded academic standards and curriculum from every state in the US. An ed tech provider building a solution for Texas doesn’t need to stress about ingesting and retrieving the right academic standard. They can offload that hard work to Learning Commons. Oak National has similarly encoded curriculum standards for the UK and makes it freely available.

IDinsight is working on building this for education for multiple LMICs in partnership with Gates, Anthropic, and CZI. We should have our first set of ingested curriculums for a few sub-Saharan countries in the next few months⁴.

A similar effort is needed for other sectors. Imagine a curated knowledge base of agricultural best practices and local terminology for each region that any AI agronomist can build on. Or a database of WHO or Ministry of Health guidelines and recommendations that any local health app can localize.

What do you get for it?

Apart from the efficiency gains, we get better quality control. The content itself comes from trusted experts e.g. WHO for health, FAO or CGIAR for agriculture, who already validate their guidelines for correctness and completeness⁵. Every AI solution that builds on the layer starts from that high baseline instead of working on its own knowledge base from scratch. Or worse, searching the open web and hoping for the best. 

It also lowers the barrier to entry. LLMs over APIs let non-technical NGOs experiment with AI without standing up an ML team; a ready-made context layer does the same for domain knowledge. An organization no longer needs both engineering muscle and a validated knowledge base to ship something credible — it can offload the second and focus on its users.

The other benefit is that we can finally build AI solutions that center the individual. A farmer in India may use FarmerChat for agri advice. When she is pregnant, uses ARMAAN’s Pregnancy Care Chatbot for maternal health. And, after the birth of her child, yet another app to explore benefits she may be eligible for. The reason this exists is because expertise is fragmented by sector. A common context layer available as a public resource means we can create a single app that supports the individual’s life journey – it is the one-stop-shop for all resources the person may need as they step through life.

The world could look like this:

For countries where Claude and ChatGPT have high penetration, this platform already exists. Teachers in the US can now go on Claude and create lesson plans and it’s grounded in their state curriculum thanks to the Learning Commons connector. 

There is a fourth benefit that increases as models get better: the layer provides provenance. An answer that is traceable to a signed, dated source from a named authority is very different from an answer a model produced – even if they agree. This is a key part of what the AI governance field calls assurance and a curated context layer makes it possible. As AI systems become more capable, our tasks shift to steering and evaluating. When a clinical decision support tool suggests a diagnosis, the clinician’s job is to check it. And that is only possible if we can trace it back to a specific, named guideline in the context layer. We need to be able to audit decisions to build trust, a fundamental requirement for meaningful scale.

A sidebar on Knowledge Graphs One way to organize this information is in a Knowledge Graph (KG). When the relationships (“verbs”) between entities (“nouns”) matter as much as the entities themselves, it is probably the best way. Education is an example of this. Concepts build on other concepts, student progress though grades, and develop skills at different stages of their education. I suspect health is similar, at least for triage. Symptoms are associated with conditions, and conditions with treatments. Triaging is often algorithmic and requires multiple things to be true (and maybe some to be not true). We started encoding IMNCI guidelines in a knowledge graph and in early synthetic-data validation saw promising accuracy gains. And we are not alone in this bet. The Zitnik lab has done the herculean task of mapping over 17 thousand diseases with 4 million relationships that take underlying science (e.g. biological pathways, drug interactions) into account. Intelehealth has also been experimenting with Knowledge Graphs for health and seeing large improvements in the diagnostic accuracy. A KG is the right tool when structure carries the meaning but may be an overkill when it doesn’t. Triage and curricula are relationship-heavy so a graph makes sense. But plenty of what a context layer needs to hold, for example agricultural best practice, may not be graph-shaped at all. And since graphs are costly to maintain, we’d want to make sure the juice is worth the squeeze.

Sustainability and ownership

Public goods investments start off with a lot of enthusiasm and funding. But they often lose momentum once the novelty has worn off and funding has shifted to other more glamorous initiatives. As much as I’d love to work on this, IDinsight is not the forever home for such a public good. 

The exact implementation governance architecture between these three roles is an open design question. But here’s one proposal of a model that might work.

One owner of the context layer can be organizations who are already producing guidelines, like WHO, CGIAR, Ministries of Education, etc since they are already maintaining these guidelines forever. A machine-readable context layer would just ride along on updates they’re doing anyway. And their name is what makes the layer trustworthy in the first place. The entity best placed to keep it current is usually the same one that makes it credible. Another owner can be country governments. Kenya could maintain a data or context layer that has its health guidelines, its education standards and curriculum, the eligibility criteria and application process for social benefits etc. Maybe the current energy around sovereign AI can be harnessed to invest in country capabilities to maintain this.

Frontier labs or their philanthropic or beneficial deployment arms can fund this context layer just like any other infrastructure investment as it improves their products in these markets⁶. They are effectively cross-subsidizing it using their LLM products. 

I acknowledge that this does not feel completely satisfying. I am sure there are novel market-based solutions that I have not thought of and governance challenges that we need to work through. I’d love to hear your thoughts. But these may also be questions to tackle once we have proven that the context layer does indeed deliver on its promise.

Conclusion

The AI-for-good space is about to grow substantially with this new wave of philanthropy. Now is the time to invest in public goods that lift all boats – before we build a tonne more, each one curating its knowledge from scratch, each at wildly different quality.

Like most public goods, this one comes with hard, unresolved questions about who builds it and who keeps it alive. But those are worth wrestling with, because the upside is unusually clear. LLMs will keep getting better, but they are never going to get better at knowing Malawi’s social protection policies on their own. A shared context layer is the rare bet that compounds: as LLMs improve, so does the value they can draw from it.

—————————————————————————————————————————

Special thanks to the many people who reviewed this article and provided critical feedback, especially Poornima Ramesh, Han Sheng Chia, Sam Carter, Oliver Hanney, and Sharif Kazemi.