In this post, our team at IDinsight explores the positives and negatives of including the “I don’t know” option in a survey. We reflect on a debate that’s been ongoing for decades and share some real-life challenges we have encountered in our work.
Surveyor conducts an interview with a d.light owning respondent in southeastern Uganda. ©IDinsight/Amy Chen.
It is standard survey best-practice to give respondents the option to answer “I don’t know” to questions. That is, unless “I don’t know” isn’t a reasonable response, say if the question is asking a person his/her name. For many researchers, including “I don’t know” as an answer option is a reflex, alongside a “refuse” option, for most closed-ended survey items, that is, questions with fixed answer choices.
But “I don’t know” responses can compromise analysis if the respondent uses it for key questions that might influence the survey’s outcome. We recommend deciding to include “I don’t know” in a questionnaire on a case-by-case basis, after careful consideration.
In this post, we share our internal debate and some ideas on making the decision to force respondents to give a substantive answer by not allowing ‘I don’t know’ as an acceptable response.
A social enterprise called d.light provides solar-powered energy products for homes and small businesses in under-resourced communities. They wanted to understand whether purchasing one of their home solar-powered systems gave households a positive financial return on their investment and if it influenced other outcomes like a family’s productivity (e.g. working, studying) or improved their health.
To answer these questions, d.light commissioned an impact evaluation. In addition to the key outcomes mentioned above, our IDinsight team saw that there might be other variables affecting these results, like total number of light hours used in the home the previous day, and from which sources. We decided to measure these as well.
When the results came in, a challenge arose. Thirteen percent of respondents said “I don’t know” when asked the number of hours they used at least one light source. Twenty eight percent of respondents answered, “I don’t know” to at least one outcome variable (their return-on-investment, productive time, or improved health).
This is a mood-dimmer because:
If you scour the internet you can find that debate around addressing this challenge usually falls into two camps. Anti-DK (anti-don’t know) folks think that sometimes eliminating the ‘don’t know’ answer choice produces sounder data and clearer analysis. Pro-DK (pro don’t-know) folks think a ‘don’t know’ answer choice must always be included in the set of answer choices for a close-ended survey item — when ‘don’t know’ is a plausible answer — for the soundest data.
Anti-DK: Conducting analysis without information about key outcome variables can destroy a piece of research. It is very common that a respondent won’t know the exact answer to a question, but in general their best estimate is acceptable — and preferred to missing data — since they have information about the household that can be informative.
Pro-DK: We don’t like destroying research; that’s why we need to “include all reasonable response possibilities as explicit response options”1. Sometimes the most accurate or truthful answer is, legitimately, “I don’t know.” The respondent could have reasonably been away from the house most of the day prior to the survey and not be aware of how long the solar lights were used. The best guess estimate may be meaningless.
It is the surveyor’s role to prompt the respondent to thoughtfully estimate the answer. Respondents say “don’t know” for a variety of reasons, including, according to Converse and Schuman 19742:
We need to be diligent about training surveyors with the tools and confidence to sufficiently draw out real (if only estimated) answers for the first four types of “I don’t know.” We could, in addition, script specific prompts for survey items that directly affect key outcomes.
As you may have guessed, the two sides of this debate have strong feelings about surveyors stepping in.
Anti-DK: But it is a general rule of (closed-ended) surveying to give as little discretion to surveyors as possible. If there’s a “don’t know” option, it will be overused to avoid pushing respondents beyond the minimum possible response3
Pro-DK: Doesn’t that come back to surveyor incentives?
[Applause for both sides of the debate.]
Including “don’t know” in a survey is a difficult decision that needs to be made on a case-by-case basis. At IDinsight, we haven’t taken a definitive stance on how to guide these case-by-case decisions.
Please weigh in in the comments — because we… don’t know!
What are the analysis options you can use when you have high percentages of “don’t know” responses? What are their downsides?
We work with business and social enterprise leaders to streamline operations, quantify social impact, and to pilot, perfect, and scale-up programs.
28 November 2019
14 July 2020
11 July 2019
30 November 2019