Causal Inference – Quiz 4

Quiz Question Explanations

Question 1

Attrition reduces statistical power by reducing the sample size at endline.

Attrition may threaten external validity if the individuals who remain in the study are not representative of the target population. For instance, in an intervention for schoolchildren, if students who live far away from schools drop out of the evaluation, then the treatment effect estimates may not represent the impact of the program for children who live far from schools.

Attrition may also threaten the internal validity of the evaluation if there is differential attrition across treatment and control groups. For instance, if an intervention for schoolchildren increases attendance at school, then tests conducted at school may have higher rates of completion among students in the treatment group than in the control group. This may lead to imbalanced treatment and control samples. In this particular case, it would be advisable to conduct some tests at home in order to include students who were absent on the day of the test, and restore attrition balance across T & C groups.

Question 2

One approach to dealing with spillovers is to minimize the potential for interaction between treatment and control groups by assigning treatment at a higher-level. For instance, for an intervention with farmers, you may want to assign villages or market catchment areas to T & C, rather than individual farmers, in order to reduce the potential for knowledge sharing or input sharing across T & C groups. The drawback of such an approach are that your sample size requirements will increase the higher your ‘cluster’ level.

Another technique for minimizing interaction between T & C groups is to put buffers around evaluation units and exclude any units within those buffers from participating in the evaluation. For instance, selecting one village to participate in the evaluation may exclude any other villages that share a secondary school or market from participating in the evaluation as well. This approach may only be viable if you have a large enough sampling frame to draw from for your evaluation.

Spillovers can be positive or negative and so they may lead to overestimate or underestimate the impact of a program. An example of a positive spillover is deworming: if one child receives a deworming pill, then it reduces the chances that that child and any neighboring children become infected with intestinal worms. If neighboring children are included in the control group, then the comparison between T & C would underestimate the direct impact of deworming. An example of a negative spillover is local inflation: if a bunch of households receiving cash transfers leads to higher prices in the nearby market, then non-treated households may face higher expenditures and lower consumption levels.

Spillovers do not involve treatment units refusing treatment or control units getting treatment, and so they are distinct from non-compliance. Instead, spillovers are the indirect effects that non-treated units experience due to units in the treatment group receiving their treatment.

Question 3

Example 1 is an example of differential attrition rather than spillovers.

Example 2 is an example of non-compliance rather than spillovers.

Example 3 is an example of a negative spillover.

Examples 4 & 5 are examples of positive spillovers.

Question 4

The intention-to-treat (ITT) effect is the effect of the program on individuals who were assigned to the treatment group. The comparison may include individuals who did not participate in the program despite being in the treatment group, and individuals who did participate in the program despite being in the control group.

The local average treatment effect (LATE) is the effect of the program on individuals who ‘complied’ with their treatment assignment (i.e. if they were in the treatment group then they got treated, and if they were in the control group then they did not get treated). While LATE reflects the direct effect of the program, it is only a valid treatment effect estimate for compliers, and may not reflect the impact of the program for non-compliers.

The term for people who get the treatment regardless of treatment assignment is ‘always-takers’, and the term for people who never get the treatment regardless of treatment assignment is ‘never-takers’. Compliers are people who only get the treatment if assigned to the treatment group, and do not get the treatment if assigned to the control group. Defiers are people who only get the treatment if assigned to the control group, and do not get the treatment if assigned to the treatment group. Defiers are generally uncommon but you do need to make a ‘no defiers’ assumption if you want to estimate the LATE.

Dropping non-compliers is problematic because you can generally only drop observed non-compliance, which fails to account for all non-compliers. For instance, if you want to drop never-takers, while you can identify them in the treatment group (because they did not take up the program), you cannot distinguish them in the control group from compliers. With always-takers it is the opposite problem. As a result, dropping non-compliers from one group but not the other may introduce bias in your treatment effect estimates. It is better to apply an instrumental variable strategy to estimate the LATE.