Our Data on Demand team has now conducted two rounds of a four-day phone survey spanning eight states in North and East India on the impacts of COVID-19 on people’s lives capturing around 6,000 respondents each round. Our team in Kenya is also conducting a large-scale phone survey of roughly 600 respondents on a project with Jacaranda Health. This post is the first part of a three-part series on our experiences with remote data collection management at scale, which we hope will be useful for others to learn from. It can be read in both English and Hindi.
Snapshot of the DoD team training surveyors virtually using Skype.
During COVID-19, it is more critical than ever to collect rapid, accurate data on health, social behaviour, and economic well-being. But since it is impossible to do in-person surveys, all data must be collected remotely. Therefore, our IDinsight teams have shifted to using phone surveys as our primary data collection tool. We are collecting high-quality data to better inform policymakers on the impacts of coronavirus and lockdown on the lives of people across many regions.
Since December 2019, our Data on Demand team has been experimenting with phone surveys to understand best practices (such as the best way to maximize respondent reach and deciding survey length). As part of our engagement to understand the effects of COVID-19 in India, we launched our largest phone survey to date. We surveyed roughly 6,000 respondents in 27 districts across 8 states about the health and economic effects of COVID-19. We have now completed two rounds of this survey. Any survey of this scale requires myriad management considerations; made even more difficult by physical distance.
In this three-part series, we explain the different workstreams and considerations involved in managing a phone survey at scale. With around ten days to prepare for each round, we have been able to hire surveyors, remotely train the entire field team, and manage survey operations during our five-day data collection periods. We are able to achieve a quick turnaround in these surveys because of the investments we made during in-person surveying that laid the foundations for our field team set-up.
In April of 2020, IDinsight began conducting phone surveys in Kenya as we support Jacaranda Health’s impact evaluation of its SMS intervention to promote exclusive breastfeeding among new mothers. Jacaranda Health has designed a package of SMS messages to promote and educate mothers on exclusive breastfeeding. IDinsight has partnered with Jacaranda Health to refine the final design and independently collect data. As part of this engagement, IDinsight will survey ~3,000 mothers across Kenya by August 2020.
In two parts of this three-part series, we detail the considerations involved in managing phone surveys at scale with a small team. The Kenya team was able to conduct interactive real-time training, hands-on data quality checks, as well as managing enumerators remotely in the first round of surveying.
We hope to translate what we learned from this experience into actionable next steps to further streamline the next few surveys we run at scale. We hope that these blog posts can contribute to the growing set of resources on these topics and are useful to other organizations exploring using phone surveys for remote data collection at scale. In this post, we share our experiences with hiring and remote training of surveyors.
To run our surveys in India, we used our expected time frame, sample size, survey length, and surveyor productivity (which we calculated from previous phone survey pilots to be roughly 6 completed surveys/day) to estimate how many surveyors we needed to hire. We ended up hiring 351 surveyors and 32 State and District Coordinators to help lead and run the survey. Our hiring process was unique and expedited because we tapped into our field team network that had previously conducted in-person surveys with us. In this section, we describe our experience hiring through our network, provide tips for hiring a new field team, and share important phone-survey specific considerations.
Over the past two years, we have completed three rounds of in-person surveying in 27 of India’s highest-need districts across eight states. We surveyed roughly 27,000 households each round to collect data on health, nutrition, financial inclusion, infrastructure, sanitation, and agriculture indicators. For these in-person surveys, we hired a group of experienced surveyors as State and District Coordinators. These State (one in each of the eight states) and District (one in each of the 27 districts) Coordinators were local to each district and had previous experience in data collection. In our in-person survey work, the District Coordinators were crucial in our ability to hire surveyors as they helped streamline recruiting and interviewing surveyors who were local to each district (and therefore spoke the local languages).
For our phone surveys, we asked District Coordinators to reach out to the surveyors we had previously worked with to inform them of our upcoming survey and gather interest. Through this process, we were able to hire 12 surveyors in each district who had worked with us before to conduct the surveys and 1 experienced monitor per district to implement data quality measures. In addition to expediting the speed of hiring, we have found that using local networks is beneficial in contexts in which there is great language variability as surveyors can be hired from the regions they are surveying from. Even through our existing networks, we had cases in which surveyors struggled to communicate with respondents due to language barriers based on different dialects. Trying to hire a diverse set of surveyors from the areas which are in the survey sample can help avoid these refusals. One drawback of using our existing network of surveyors is that most of our in-person surveyors are men because we had hired surveyors on the basis of having a motorcycle or independent means of transportation.
Given that surveyors will not need to travel to different locations, teams will face fewer budgetary constraints during hiring. Since phone surveys are conducted remotely, when hiring a new team, you can be more deliberate in hiring a diverse set of surveyors based on survey needs. In the past, we have used the following steps when hiring:
Overall, teams should budget sufficient time for hiring by considering the networks they have to spread the hiring form and the capacity they have to interview/shortlist surveyors. Doing so can be beneficial to ensuring that the hired surveyors will do high-quality work.
For our remote surveys during COVID-19, there are two key requirements for hiring that may not be applicable to other in-person or remote survey contexts:
If surveyors have experience with conducting phone surveys or have assisted in phone backchecks (calling households who have been surveyed in-person to confirm what was said), that is an added bonus, but not a necessity.
After hiring, we were tasked with training surveyors and District Coordinators remotely. Below are some lessons learned from training large scale teams remotely and smaller teams more directly to ensure high levels of engagement.
We had some experience with remote training from our pilot phone surveys using a combination of video-sharing, Skype, and screen-sharing. After these pilots, we sent feedback forms to our District Coordinators, which included a section on what went well/what did not go well with training. We used this feedback to quickly pull together a training plan.
For our training, we divided our 27 District Coordinators into four groups, sent them training materials (detailed below) ahead of time, and trained them over Skype. Afterwards, the District Coordinators led their own training sessions with their respective surveyors over Skype. Overall, our trainings have been completed in 4–5 days: 2 days for District Coordinators and 2–3 days for surveyors. We sent all of the materials in different languages to our District Coordinators, who then shared these with surveyors. Doing so ahead of training allowed for a good amount of buffer time for surveyors to download the material (given the unpredictability of network stability), understand the material, and come prepared to training with questions.
We created a schedule that divided the topics we needed to cover into different modules. We included breaks and time at the beginning/end of each day to debrief, review, and ask questions.
Since we consolidated time for training preparation into tight timelines, we were a bit rushed putting together training materials, which lowered the quality of our training in our first round. We have learned to plan ahead so that there is sufficient time to create training materials. Well-thought-out videos that explain phone survey systems take some time to make, but can only be made after the content (SurveyCTO form, trackers, protocol sheet) is finalized. Had we spent more time on training during our first round, we would have been able to include more practice time. We also should have included more discussion time for less probable, yet plausible scenarios of what to do when a respondent is reached (such as if a child picks up the phone) but not available to help ensure that our protocols were standardized. We tried to correct for this in our second round and trained surveyors using a protocols cheat sheet, which we later sent them to reference.
The main challenges we foresee in each training session include software- or network-related issues, the lack of systems in place to answer surveyor questions in real-time, and training delays. To account for software-related issues, we make sure to ask surveyors to download Skype and practice using it before the training. We also allot buffer time in our training schedule to account for network issues. To efficiently answer questions and concerns, we create a spreadsheet in which District Coordinators can share their concerns (or any concerns surveyors messaged them). We aim to address concerns on the sheet and messaged District Coordinators with answers to common questions. To account for training delays, we try to space out our schedule and send reminders about each session to help ensure that training fits within our time constraints.
After hiring, the team in Kenya was able to schedule virtual training meetings on Zoom with all enumerators. All surveyors were fluent in English and Swahili which enabled the team to conduct one training for all surveyors. The team created a training schedule and provided training modules. Due to the small size, we were able to walk through the training materials with surveyors while on the Zoom call, allowing enumerators to ask questions in real-time. We were also able to create Zoom breakout rooms to allow enumerators to practice surveying with smaller groups, the training facilitator was also able to move from room to room to supervise the activities and answer questions. Doing so allowed us to directly engage with surveyors to ensure high levels of understanding of the survey and relevant protocols.
Our training is centred around training materials of the following types:
In our surveys in India with a large field team, we also held a practice section at the end in which we paired up surveyors and asked them to call each other and fill out survey forms. This enabled surveyors to practice asking questions over the phone while using SurveyCTO on the same device. Since we have integrated SurveyCTO with a software called Exotel (which can be used to record two sides of a phone call), our District Coordinators were able to listen to surveyors and provide feedback. Practice also allowed us to test our trackers, dashboards, and systems with live data prior to the start of data collection.
For our survey in Kenya, because our field team was smaller, we were able to implement very focused training sessions with many opportunities for practice. Part of the training included piloting the survey with surveyors and actual respondents. IDinisght provided airtime for enumerators to engage in this training. A member of the IDinsight team would join the call after the enumerator acquired consent from the respondent. The purpose of having a three-way call was to allow the member of the IDinsight team to take notes on the interview process and provide feedback to the surveyor trainees in an effort to improve survey quality.
Learn more from our survey teams about how to ensure productivity and data quality for a phone survey at scale and daily practices to optimize remote survey teams
5 June 2020
20 October 2020
8 June 2020
20 September 2020