This post is aimed at research practitioners experienced at administering in-person surveys and interested in beginning phone surveys.
After operationalizing a callback procedure and a tracking system that helped us increase respondent reach, we began to analyze data on-call times (automatically calculated in SurveyCTO as a variable entitled ‘start time) and survey length (automatically calculated in SurveyCTO as a variable entitled ‘duration’) to ascertain if there were any additional ways to reach more respondents. Our findings are below:
Call times: not always bright and early
In our context, most surveys were completed in the mornings. In the first two pilots, field officers had discretion to call, and for the latter pilots, we equally distributed call slots. As the graph illustrates, in our context, most surveys were completed in the morning. However, since our phone surveys were conducted in the winter, field teams expressed more members from more agricultural households were available in the latter half of the morning since they were not tending to crops in the early morning.
We hope to continue monitoring when calls are completed to see if there are seasonal patterns we should be aware of. The data on-call times helps us inform callback protocols to ensure surveyors are calling respondents at the most optimal times.
In-person surveys sometimes take hours, but how long would respondents stay on the phone? The literature is quite mixed on the appropriate length of surveys, but reaches a consensus that phone surveys should be shorter than in-person surveys.
Understanding the optimal survey duration is extremely important for two reasons:
The third consideration in some contexts may be the cost of phone plan/reimbursements for respondents. In our case, respondents pay for their own phone plans, but in India, phone penetration is very high and airtime is much cheaper compared to other countries.
In the first three pilots, we were primarily interested in understanding how to ensure most respondents picked up the phone, so we did not invest significant time in developing questionnaires. Instead, we asked a few simple questions sourced from teammates. We sought to test optimal survey length by implementing an experiment in two of our pilots.
We constructed a questionnaire with three sections of the same approximate length and randomized how many sections each respondent was tasked to answer. After introducing the survey and asking for consent, this would mean that surveyors were administering a survey of different lengths. We collected data on the appropriate survey lengths and found that the ideal survey in our context was roughly 10–15 minutes. Our surveyors also consistently noted that around the ten-minute mark, respondents began becoming more agitated and started asking surveyors how much longer the survey would last.
Of the phone surveys we have completed that were not refusals, the majority were completed in 10–15 minutes. From all of our pilots, we have not yet been able to consistently reach respondents for longer than 20 minutes.
We continue experimenting with survey length to build our knowledge of optimal survey duration and improve our understanding of the contexts in which longer phone surveys may be possible. For example, the permissible survey length (without causing fatigue) could be negatively correlated with the mental effort required for participating and could be increased through the use of incentives.
Building out phone calling protocols with an understanding of the optimal call times and adjusting survey length for the optimal duration can help increase respondent reach. We will continue to collect data on call times and duration to understand how to best reach respondents in India.
If you’re experimenting with phone surveys or have feedback about our processes, please comment below or on social media, or reach out to our team, firstname.lastname@example.org and email@example.com, with your thoughts and suggestions.
12 July 2020
28 January 2020
3 November 2020
15 July 2019