Using the Likert Scale to understand open defecation in India

Summary

IDinsight aims to use the right evidence tool in each context to find effective solutions to social challenges. We wanted to understand why people were engaging in open defecation instead of using latrines. Answering this question would help inform how policymakers implement a cost-effective approach to reducing open defecation. To understand mental barriers to latrine use, we created a survey using the Likert scale — an approach that helps measure attitudes — and piloted five different versions to get the most honest answer from participants. We also designed and tested sanitation-related “nudges” to reduce the mental barriers towards using latrines that were identified in the survey. This article explores how our team piloted the Likert scale to address social biases potentially influencing participants’ answers and to more deeply understand why people weren’t using latrines.

The question: why are people engaging in open defecation instead of using latrines — is both critical to accurately answer and difficult to measure. Open defecation is linked to harmful and sometimes fatal health problems such as diarrhoea, cholera, and typhoid. Despite these effects, people sometimes still engage in open defecation even when they have latrines. This tendency cannot be addressed without knowing the reasons why someone might do this. If someone engages in open defecation because he feels it is cleaner than using a latrine, then a policy that provides free access to a toilet cleaning brush might make him more likely to use the latrine. But if he engages in open defecation because he thinks toilets are primarily for the women in his family, then providing a brush probably won’t be effective.

As we’ll detail below, the true reasons people choose to engage in open defecation are difficult to measure because open defecation is seen as a “bad” social behaviour and people being surveyed might bias their answers towards what they think surveyors or their community want to hear. Questions need to be framed in a way that will make people comfortable giving their true feelings — often, this requires testing different ways of asking about the same thing. We found that, qualitatively, the most effective way to ask these questions in our context was to frame questions as full sentences about peoples’ feelings towards latrine-related matters:

[Select the response you most agree with] It is difficult to clean and maintain the latrine.

1. Yes, it is very difficult.

2. It is difficult but not very difficult.

3. It is not difficult at all.

शौचालय की साफ़ सफाई और उसका ध्यान रखना मुश्किल होता है.

1. हाँ, बहुत मुश्किल होता है

2. मुश्किल होता है, लेकिन ज्यादा नहीं

3. बिलकुल भी मुश्किल नहीं होता है

Open Defecation in India

Open defecation is a critical development challenge facing India, with rates of open defecation as high as 39% nationally as recently as 2014. The combination of open defecation with unsafe water and hygiene practices have been linked to a number of deleterious public health outcomes, including diarrhoea, cholera, dysentery, and hepatitis A, typhoid, and polio. To address this, in 2014 the Government of India launched the Swachh Bharat Mission (SBM), and while latrine coverage and use has gone up significantly over the past decade, the problem of open defecation still remains. In particular, there seems to be a gap between having access to a latrine and using the latrine. Even with access to latrines, many people might not use them due to a range of difficulties (mental barriers), such as anxiety around emptying the latrine pit (as stipulated by Coffey and Spears), the perception that latrines are primarily meant for women’s use, or location of the latrine.

The IDinsight team wanted to understand mental barriers in order to reduce them, with the theory that leveraging cost-effective “nudges” — such as providing toilet cleaning liquid, air freshener, or a radio to make latrine use a more pleasant experience — could increase latrine use. If we could suggest solutions to addressing barriers, the government expressed interest in scaling up a cost-effective approach. It is worth noting that the nudges did not work to reduce open defecation. However, this project did help us more accurately understand people’s perceptions about latrines, reducing the extent of social desirability bias. We still believe this process is a valuable one for others out there looking to use the Likert scale in situations where social biases may inhibit participants from reporting their true feelings.

Using the Likert scale to understand latrine use

The process for understanding and gauging people’s true impressions of latrine usage is an important one. As part of this, we asked a total of 20 Likert scale questions related to barriers and open defecation. Most of these questions were about behaviours a person might think that their community feels are “bad”. Because of this, they are likely to tell researchers the socially “good” thing rather than the truth. This is called social desirability bias. For the sample question in this post, “It is difficult to clean and maintain the latrine,” social desirability bias might prompt a respondent to say it is not difficult, even if it is, because the respondent perceives that saying it is difficult to clean a latrine is not as socially acceptable of an answer.

A typical Likert scale has five responses. In this typical form, a sample question we used in our survey would read [Note: we never used this version, so Hindi is not included]:

Version 0: It is difficult to clean and maintain the latrine.

1. Strongly Agree

2. Somewhat Agree

3. Neither Agree nor Disagree

4. Somewhat Disagree

5. Strongly Disagree

From this standard Likert scale question, our first change was to trim the options down to three. Before our survey began, one of our team members attended a session about a maternal and child care survey in Madhya Pradesh which recommended trimming our options so respondents don’t have to make granular judgments on their level of agreement. The “neutral” option was removed because we wanted people to have an opinion and take a position. The “somewhat agree” is useful because it provides the respondent a soft way to agree to a socially undesirable thing, such as defecating in the open. We removed the “somewhat disagree” option because we felt that people do not need a gradated option for disagreement because disagreement already enables the respondent to be in line with the socially acceptable response. Thus we first tested this version during piloting:

Version 1: I feel that I have to do a lot of work to clean and maintain my latrine.

1. Completely Agree

2. Somewhat agree

3. Disagree

मुझे लगता है कि मुझे हमारे शौचालय को साफ करने के लिए, और साफ बनाए रखने के लिए, बहुत काम करना पड़ता है।पूरा सही है।

1. पूरा सही है।

2. थोड़ा सा सही है।

3. सही नहीं है।

However, in the pilot, our team found that respondents who couldn’t read or write had issues with this. People especially had difficulty differentiating between “completely” and “somewhat” agree. As there were twenty repeated questions during our one-hour survey using the same Likert scale answers, respondents seemed irritated by the repetition.

To increase comprehension and interest, we added a stoplight visual. The surveyor would guide the respondent along the stoplight with their finger.

Version 2: Same as Version 1 but with a stoplight visual

We used this version for our baseline survey. However, our surveyors found that respondents tended to answer whatever the surveyor’s finger landed on first. We also realised that survey respondents, mostly rural farmers, didn’t have a prior association with red, amber, and green in the way urban respondents or our team would. Since respondents were still tired and confused, we let go of the spotlight visual.

However, this meant that we were back to our original problem — respondents still had trouble relating each option to the question.

In response, we created a final version of the Likert scale questions that reiterated a full statement three times.

Version 3: [Select the response you most agree with] It is difficult to clean and maintain the latrine.

4. Yes, it is very difficult.

5. It is difficult but not very difficult.

6. It is not difficult at all.

शौचालय की साफ़ सफाई और उसका ध्यान रखना मुश्किल होता है.

4. हाँ, बहुत मुश्किल होता है

5. मुश्किल होता है, लेकिन ज्यादा नहीं

6. बिलकुल भी मुश्किल नहीं होता है

Respondents could understand each statement better and no longer had as much trouble differentiating between “agree” and “somewhat agree”.

When we asked a surveyor who worked with us on baseline and endline whether this approach worked, he said that the endline version was preferable, but the extremes were still better understood than the middle option (“It is difficult but not very difficult”). Endline surveyors agreed that they needed to probe further to help the respondent understand the middle option. They needed to read the middle option again, or emphasize different words between the first and second options (“you like it VERY MUCH” vs. “you like it BUT”). Surveyors also reported having to pay attention to the respondent’s body language for the first two options to try to get at their true answer.

The only reported downside of Version 3 was that the surveyor had to speak more than in previous options, since they had to read four full sentences for each of the seven Likert Scale questions.

Overall, the survey team, surveyors, and respondents felt more satisfied with the last version than the first version.

Quantitatively, data for the Likert Scale questions were correlated with/predictive of reported open defecation for both baseline and endline. That is, responses to the Likert Scale questions about different barriers from the literature (pit-emptying, gender, experience) to using latrines did predict whether a person reported being more or less likely to engage in open defecation. For example, a person who didn’t like emptying their pit was more likely to report engaging in open defecation than a person who liked emptying their pit. The predictiveness improved slightly at endline — this quantitative success, along with the qualitative bettering of experience for the people surveyed, leads us to believe that the Likert scale iteration process was a success.

Barrier salience at baseline

Barrier salience at endline

In various situations, different versions of Likert Scale questions might be most appropriate to get at respondents’ true feelings, which are vital to understand so that they effectively inform policies.

Have you similarly piloted the Likert Scale question? What did you find? We would love to know if any of you have similar or different process learnings.