Lessons from on-the-ground data collection: How SurveyStream helped us streamline field operations

In many of IDinsight’s large-scale primary data collection projects, our teams found themselves juggling spreadsheets, Slack messages, and late-night WhatsApp messages just to track basic metrics like how many surveys were completed, which teams were behind schedule, or where data quality was slipping. We needed something better. That’s why we built SurveyStream.

SurveyStream is an open-source field operations management tool designed to provide real-time visibility into data collection across geographies and teams. Built with the needs of teams running data collection projects, it streamlines performance tracking, quality assurance, surveyor assignment and messaging for fieldwork-heavy projects.

As DataDelta’s big innovation push is winding down, we’re sharing what we have learned and the platform itself to help others overcome similar challenges in field operations.

The status quo before SurveyStream

The structure is decentralized, with field teams (surveyors + supervisors) and central teams (associates + field managers) relying heavily on manual workflows, fragmented tools, and informal communication. This can lead to frequent misalignment, delays in assignments, and difficulty in scaling operations or maintaining data quality at large volumes.

What is SurveyStream?

At its core, SurveyStream integrates with SurveyCTO to power data systems that allow users to:

• Track surveyor performance across indicators and time
• Monitor enumerator productivity through data submission patterns in near real-time
• Detect potential data quality issues automatically
• Flag outliers and generate feedback loops for supervisors
• Assign targets to surveyors and schedule automated messages

The system is modular and was designed to work in diverse operational contexts – from rural health monitoring to urban opinion surveys.

So far, SurveyStream has supported dozens of projects across Africa and Asia. Sharing a few examples below: 

India: Managing a survey of over 60,000 households

The Government of Telangana, through the Telangana Development Planning Society (TGDPS), commissioned the collection of large-scale and representative data in the health, nutrition, and education sectors across 33 districts of Telangana state in India. A total of approximately 750 field staff were employed to survey 61,215 households. A survey of this size required extensive field operations, and SurveyStream was used to conduct field assignments, monitor productivity and data quality, and automate communications with surveyors. The operational load on the field teams was significantly reduced due to using SurveyStream.

Kenya: Monitored data quality for a household survey

For a data collection project in Kilifi County in Kenya, SurveyStream data quality monitoring was used for high-frequency checks (HFCs), outlier detection, duration and missing variable checks, audio audits, and for protocol violations. The dashboards allowed field supervisors to continuously monitor their teams without direct involvement from the survey managers. They also allowed the field supervisors to triangulate daily productivity and nudge teams to clear submission backlogs in case of mismatches between reports from daily debriefs, SurveyCTO numbers, and the SurveyStream dashboards. 

Philippines: Streamlined administrative tasks

For an IDinsight data collection project with the Department of Health in the Philippines, surveyors were using SurveyCTO to upload photos of their receipts as part of their expense reimbursement process. A field administrator needed to review 200-300 receipt photos per day to approve expenses over a 10-week period of data collection. Using the SurveyStream media audits feature reduced the field admin’s effort on reviewing photos from 8 hours per day to 3 hours per day, a reduction of 63%.

What we learned

1. Build with, not for

All of SurveyStream’s features – from automated messaging to data quality dashboards – emerged through deep collaboration with field teams, not top-down design.

For example, automated emails began as a simple COVID-era solution to reach phone surveyors and has now evolved into a robust system that shares near real-time updates on assignments, financial reimbursements, and surveyor performance, all in local languages and at scale.

Similarly, the data quality (DQ) feature started with users configuring high-frequency checks on each survey question. This was a time-consuming process. There were also issues related to alignment on definitions, e.g, how is the denominator defined for each metric. Such issues would lead to the deprioritization of the configuration of the DQ feature amid busy pre-data collection work. 

Working closely with teams, the DataDelta team developed a leaner approach: standard definitions and best-practice defaults, which translated into dashboard templates, pre-defined metrics, and one-click configurations for several HFC checks on SurveyStream. We have now made it fast and intuitive to monitor data quality from day one.

From low usage and engagement early on, to becoming a default first layer of DQ monitoring, this transformation underscores a simple truth: tools only succeed when built with users, not for them.

2. Modularity is a superpower. 

SurveyStream was developed to support a wide range of surveys conducted across regions, sectors, and project types. No two surveys are alike: each IDinsight project team has its own goals, operational constraints, and contextual quirks. A one-size-fits-all solution was not going to work.

Instead of building a rigid tool, we approached SurveyStream as a set of building blocks. Each feature was designed to be independently configurable and interoperable. This feature-by-feature architecture gave us the flexibility to tailor SurveyStream to the specific needs of each project. We enable only the necessary components for each survey, reducing complexity while maintaining consistency. 

The result was that we had a solid foundation, and we could deploy SurveyStream across many types of primary data collection projects. If required, we could easily build light-touch customizations. This gave our teams the confidence that the tool would flex to meet their needs rather than the other way around.

Whether you’re designing internal tools or platforms for cross-functional teams or varied use cases, invest in modularity. Think in terms of discrete, reusable components that can be combined or built upon, depending on the use case. 

3. Scaling custom tools is hard. 

Another key lesson: as tool adoption grows, so do user demands and expectations. Feature requests, customization requests, and user support needs can escalate rapidly, often outpacing initial resource planning.

For example, midway through our roadmap, we were suddenly tasked with supporting a project that involved processing a massive volume of survey data. The architecture we had originally designed was optimized for smaller datasets, but it quickly became clear that our backend could not efficiently handle the scale and complexity of the queries being run. As a result, we had to revisit several foundational decisions, including our choice of database. Specifically, we moved from PostgreSQL to Amazon Redshift to better support long-running, complex analytical queries across large datasets. This shift required not only technical changes but also updated data models, new performance monitoring processes, and retraining internal users.

When building in-house tools, plan for scale early, even if your initial use case is small. Make sure you have a strategy for maintaining the tool over time, know how user support will be handled as adoption grows, and know how infrastructure can evolve to support higher volumes and more complex needs.

What’s next?

As we have reached the end of our roadmap for active development, the SurveyStream platform is now available for everyone to use and build upon. The full documentation is available here and contains videos, live demos, and step-by-step walkthroughs for SurveyStream configuration. If you’re exploring similar challenges or want to partner on tech-for-good solutions, please reach out: inquire@idinsight.org