How to hire and train a phone survey team

Our Data on Demand team has now conducted two rounds of a four-day phone survey spanning eight states in North and East India on the impacts of COVID-19 on people’s lives capturing around 6,000 respondents each round. Our team in Kenya is also conducting a large-scale phone survey of roughly 600 respondents on a project with Jacaranda Health. This post is the first part of a three-part series on our experiences with remote data collection management at scale, which we hope will be useful for others to learn from. It can be read in both English and Hindi.

During COVID-19, it is more critical than ever to collect rapid, accurate data on health, social behaviour, and economic well-being. But since it is impossible to do in-person surveys, all data must be collected remotely. Therefore, our IDinsight teams have shifted to using phone surveys as our primary data collection tool. We are collecting high-quality data to better inform policymakers on the impacts of coronavirus and lockdown on the lives of people across many regions.

COVID Surveys in India

Since December 2019, our Data on Demand team has been experimenting with phone surveys to understand best practices (such as the best way to maximize respondent reach and deciding survey length). As part of our engagement to understand the effects of COVID-19 in India, we launched our largest phone survey to date. We surveyed roughly 6,000 respondents in 27 districts across 8 states about the health and economic effects of COVID-19. We have now completed two rounds of this survey. Any survey of this scale requires myriad management considerations; made even more difficult by physical distance.

In this three-part series, we explain the different workstreams and considerations involved in managing a phone survey at scale. With around ten days to prepare for each round, we have been able to hire surveyors, remotely train the entire field team, and manage survey operations during our five-day data collection periods. We are able to achieve a quick turnaround in these surveys because of the investments we made during in-person surveying that laid the foundations for our field team set-up.

Surveys in Kenya

In April of 2020, IDinsight began conducting phone surveys in Kenya as we support Jacaranda Health’s impact evaluation of its SMS intervention to promote exclusive breastfeeding among new mothers. Jacaranda Health has designed a package of SMS messages to promote and educate mothers on exclusive breastfeeding. IDinsight has partnered with Jacaranda Health to refine the final design and independently collect data. As part of this engagement, IDinsight will survey ~3,000 mothers across Kenya by August 2020.

In two parts of this three-part series, we detail the considerations involved in managing phone surveys at scale with a small team. The Kenya team was able to conduct interactive real-time training, hands-on data quality checks, as well as managing enumerators remotely in the first round of surveying.

We hope to translate what we learned from this experience into actionable next steps to further streamline the next few surveys we run at scale. We hope that these blog posts can contribute to the growing set of resources on these topics and are useful to other organizations exploring using phone surveys for remote data collection at scale. In this post, we share our experiences with hiring and remote training of surveyors.

Hiring for Large Scale Surveys

To run our surveys in India, we used our expected time frame, sample size, survey length, and surveyor productivity (which we calculated from previous phone survey pilots to be roughly 6 completed surveys/day) to estimate how many surveyors we needed to hire. We ended up hiring 351 surveyors and 32 State and District Coordinators to help lead and run the survey. Our hiring process was unique and expedited because we tapped into our field team network that had previously conducted in-person surveys with us. In this section, we describe our experience hiring through our network, provide tips for hiring a new field team, and share important phone-survey specific considerations.

Our Data on Demand Network

Over the past two years, we have completed three rounds of in-person surveying in 27 of India’s highest-need districts across eight states. We surveyed roughly 27,000 households each round to collect data on health, nutrition, financial inclusion, infrastructure, sanitation, and agriculture indicators. For these in-person surveys, we hired a group of experienced surveyors as State and District Coordinators. These State (one in each of the eight states) and District (one in each of the 27 districts) Coordinators were local to each district and had previous experience in data collection. In our in-person survey work, the District Coordinators were crucial in our ability to hire surveyors as they helped streamline recruiting and interviewing surveyors who were local to each district (and therefore spoke the local languages).

For our phone surveys, we asked District Coordinators to reach out to the surveyors we had previously worked with to inform them of our upcoming survey and gather interest. Through this process, we were able to hire 12 surveyors in each district who had worked with us before to conduct the surveys and 1 experienced monitor per district to implement data quality measures. In addition to expediting the speed of hiring, we have found that using local networks is beneficial in contexts in which there is great language variability as surveyors can be hired from the regions they are surveying from. Even through our existing networks, we had cases in which surveyors struggled to communicate with respondents due to language barriers based on different dialects. Trying to hire a diverse set of surveyors from the areas which are in the survey sample can help avoid these refusals. One drawback of using our existing network of surveyors is that most of our in-person surveyors are men because we had hired surveyors on the basis of having a motorcycle or independent means of transportation.

Hiring tips for building a new field team

Given that surveyors will not need to travel to different locations, teams will face fewer budgetary constraints during hiring. Since phone surveys are conducted remotely, when hiring a new team, you can be more deliberate in hiring a diverse set of surveyors based on survey needs. In the past, we have used the following steps when hiring:

1. Create a Google Form with information about the organization and specific survey (duration, purpose etc.), a section to collect personal and professional details (to capture previous experiences, qualifications, and language skills), and a section with a few questions that aim to gain insights into what surveyors would do in particular scenarios.
2. Circulate this Google Form using social networking platforms like Facebook and Whatsapp, and share the form with other team members who have previously worked with your organization. You can also share it online through online newspapers and websites of local NGOs or local research organizations.
3. Constrain your list of surveyors who applied based on the project requirements. This could mean looking through the set of short-answer questions to narrow down surveyors who naturally think in line with survey protocols your organization follows; this could mean looking for surveyors with previous experience using software like SurveyCTO or ODK Collect; this could mean shortlisting surveyors of a particular gender (ex: hiring more female surveyors to administer questions targeted at women) etc.
4. Conduct a round of interviews to understand if the surveyor fits within the survey context (such as speaking the local language/dialect fluently) and is comfortable with the survey conditions. Because of regional language variations, it is important to have someone local to the region conduct these interviews. In our case, our District Coordinators took the lead in interviewing surveyors.
5. Leverage existing networks of high-quality enumerator candidates. Due to IDinsight’s experience in conducting surveys in Kenya, the team had direct access to quality candidate surveyors to circulate the job advertisement. They were also able to access candidates from internal teams who had to put data collection on hold due to COVID-19.

Overall, teams should budget sufficient time for hiring by considering the networks they have to spread the hiring form and the capacity they have to interview/shortlist surveyors. Doing so can be beneficial to ensuring that the hired surveyors will do high-quality work.

Special considerations when hiring for phone surveys

For our remote surveys during COVID-19, there are two key requirements for hiring that may not be applicable to other in-person or remote survey contexts:

1. Surveyors should have a working smartphone. Smartphones serve two purposes: firstly, they can be used to conduct phone surveys and secondly, they can be used to input data (through an application like SurveyCTO). In other remote survey contexts, teams may be able to distribute smartphones or tablets to surveyors in person, but this is not possible during COVID-19.
2. Surveyors should have access to the internet and/or network connection for the duration of training and data collection. Apart from the need for a network connection to facilitate phone calls for the survey, we ask that surveyors have a stable network connection for training (since we use Skype and share screens), communication and coordination via WhatsApp, and form submissions (since surveyors need to be able to submit forms throughout the day so that we can create assignments consistent with our callback protocols).

If surveyors have experience with conducting phone surveys or have assisted in phone backchecks (calling households who have been surveyed in-person to confirm what was said), that is an added bonus, but not a necessity.

Training

After hiring, we were tasked with training surveyors and District Coordinators remotely. Below are some lessons learned from training large scale teams remotely and smaller teams more directly to ensure high levels of engagement.

Training a Large Field Team Remotely: Lessons from India

We had some experience with remote training from our pilot phone surveys using a combination of video-sharing, Skype, and screen-sharing. After these pilots, we sent feedback forms to our District Coordinators, which included a section on what went well/what did not go well with training. We used this feedback to quickly pull together a training plan.

For our training, we divided our 27 District Coordinators into four groups, sent them training materials (detailed below) ahead of time, and trained them over Skype. Afterwards, the District Coordinators led their own training sessions with their respective surveyors over Skype. Overall, our trainings have been completed in 4–5 days: 2 days for District Coordinators and 2–3 days for surveyors. We sent all of the materials in different languages to our District Coordinators, who then shared these with surveyors. Doing so ahead of training allowed for a good amount of buffer time for surveyors to download the material (given the unpredictability of network stability), understand the material, and come prepared to training with questions.

We created a schedule that divided the topics we needed to cover into different modules. We included breaks and time at the beginning/end of each day to debrief, review, and ask questions.

Since we consolidated time for training preparation into tight timelines, we were a bit rushed putting together training materials, which lowered the quality of our training in our first round. We have learned to plan ahead so that there is sufficient time to create training materials. Well-thought-out videos that explain phone survey systems take some time to make, but can only be made after the content (SurveyCTO form, trackers, protocol sheet) is finalized. Had we spent more time on training during our first round, we would have been able to include more practice time. We also should have included more discussion time for less probable, yet plausible scenarios of what to do when a respondent is reached (such as if a child picks up the phone) but not available to help ensure that our protocols were standardized. We tried to correct for this in our second round and trained surveyors using a protocols cheat sheet, which we later sent them to reference.

The main challenges we foresee in each training session include software- or network-related issues, the lack of systems in place to answer surveyor questions in real-time, and training delays. To account for software-related issues, we make sure to ask surveyors to download Skype and practice using it before the training. We also allot buffer time in our training schedule to account for network issues. To efficiently answer questions and concerns, we create a spreadsheet in which District Coordinators can share their concerns (or any concerns surveyors messaged them). We aim to address concerns on the sheet and messaged District Coordinators with answers to common questions. To account for training delays, we try to space out our schedule and send reminders about each session to help ensure that training fits within our time constraints.

Training a Smaller Field Team Remotely: Lessons from Kenya

After hiring, the team in Kenya was able to schedule virtual training meetings on Zoom with all enumerators. All surveyors were fluent in English and Swahili which enabled the team to conduct one training for all surveyors. The team created a training schedule and provided training modules. Due to the small size, we were able to walk through the training materials with surveyors while on the Zoom call, allowing enumerators to ask questions in real-time. We were also able to create Zoom breakout rooms to allow enumerators to practice surveying with smaller groups, the training facilitator was also able to move from room to room to supervise the activities and answer questions. Doing so allowed us to directly engage with surveyors to ensure high levels of understanding of the survey and relevant protocols.

Training Materials

Our training is centred around training materials of the following types:

• Videos: We make videos to describe our survey management (call-back protocols, tracker, assignments etc.) and a video describing the questionnaire in its SurveyCTO format. In these videos, we display what our surveyors would see on their own devices and use a voice-over screen recording to carefully explain each aspect of our protocols, systems, and the survey itself.
• Checklists: We make checklists with general reminders and detailed step-by-step instructions of tasks to complete in each time slot. We also include up to date information on COVID-19, tips for engaging with respondents, and protocols for different scenarios, including cases in which respondents were distressed.
• Quizzes: We make quizzes on SurveyCTO to test surveyors on survey content and our protocols. After training sessions, we would ask surveyors to take the quizzes and then would discuss the questions and correct answers.

Training Simulations & Survey Practice

In our surveys in India with a large field team, we also held a practice section at the end in which we paired up surveyors and asked them to call each other and fill out survey forms. This enabled surveyors to practice asking questions over the phone while using SurveyCTO on the same device. Since we have integrated SurveyCTO with a software called Exotel (which can be used to record two sides of a phone call), our District Coordinators were able to listen to surveyors and provide feedback. Practice also allowed us to test our trackers, dashboards, and systems with live data prior to the start of data collection.

For our survey in Kenya, because our field team was smaller, we were able to implement very focused training sessions with many opportunities for practice. Part of the training included piloting the survey with surveyors and actual respondents. IDinisght provided airtime for enumerators to engage in this training. A member of the IDinsight team would join the call after the enumerator acquired consent from the respondent. The purpose of having a three-way call was to allow the member of the IDinsight team to take notes on the interview process and provide feedback to the surveyor trainees in an effort to improve survey quality.

Learn more from our survey teams about how to ensure productivity and data quality for a phone survey at scale and daily practices to optimize remote survey teams