Causal Inference – Quiz 2

Quiz Question Explanations

Question 1

Randomization does not ensure perfect balance across observables and unobservables, but it does ensure that any difference between treatment and control groups is due to chance. If the sample is large enough, then the probability of large imbalances in outcomes or relevant covariates is small. If the researcher wants to ensure balance in baseline outcomes or other covariates, then she should stratify.

Randomization is generally the most rigorous design for attributing differences in outcomes to the impact of a program. But it is not necessarily the best design for predicting the impact of a program in a certain population. This depends more on whether the sample of individuals in the evaluation resembles the population of interest.

Randomized designs actually tend to require less data collection than quasi-experimental designs. Whereas a baseline is often helpful but not necessary in a randomized design, in a quasi-experimental design a baseline can be critical for fulfilling a causal identification assumption (such as the assumption that changes in an outcome are similar between T & C, even if levels differ). Moreover, matching design often require collecting data from a larger pool of potential treatment and comparison units at baseline, and then identifying the best matches from that pool for endline data collection.

Randomized designs are not ‘usually’ unethical, but may be unethical if the program in question is likely to benefit participants, there are sufficient resources to scale it up to all participants, but the program is withheld from some for the purposes of maintaining a control group. However, these conditions are often not met (and if they are, then it’s not clear that any kind of impact evaluation – RCT or non-experimental – would be warranted).

Question 2

Since the client does not want to deny the program to eligible women if they apply, it will not be possible to have a pure control group, as with an individual-level RCT, cluster-level RCT, or factorial design.

An encouragement design may be feasible, since it does not deny participation to any applicants. What would be randomized in this case is some kind of encouragement to eligible women to apply. Perhaps women in the treatment group receive a flyer or a call about the program, or if there are any program fees those are waived for women in the treatment group. Women in the control group would not receive this encouragement, but could still apply. As long as the encouragement leads to higher program take-up in the treatment group than in the control group, it could be used to back out the effect of the program.

However, there are two considerations to keep in mind with an encouragement design. First, if the difference in take-up between the T & C groups is small, then you will need a very large sample size to have sufficient power to detect treatment effects. Second, treatment effect estimates will only be valid for ‘compliers’, or the types of people who take up treatment because of the encouragement. For people who would take up the treatment regardless of encouragement, or for those who would never take up the treatment, you will not be able to get valid impact estimates with an encouragement design.

A stepped wedge design may also be appropriate in this circumstance, since it would ensure that all eligible women eventually receive the program, but in a randomized order. This will allow you to compare women who receive the program early with those who receive it later to estimate treatment effects. Two considerations to keep in mind. First, a stepped wedge design does require temporary withholding of the program to women randomized to later groups. If it’s necessary to enroll a woman in the program as soon as she applies, then a stepped wedge design may not be feasible. Second, it is only possible to measure short-term effects in a stepped wedge design, as long as there are some women who have not been treated. Once all women have received the treatment, there will no longer be a control group for comparison.

Question 3

An individual-level RCT would be challenging for this program since the computer lab is being built at the school; it would be difficult to restrict usage to certain students within a school and not to others. Even if restricted usage were possible, we might be concerned about spillover effects from one student to another within the same school. For both of these reasons, a clustered RCT (clustered at the school-level) would be more appropriate.

A factorial design doesn’t really make sense unless we wanted to test other interventions as well. An encouragement design also wouldn’t make sense in this set-up. A stepped wedge design would perhaps be feasible, but if it is not a problem to maintain a pure control group, then it would be better to do that rather than a stepped wedge design in order to measure longer-term effects, simplify program roll-out, and simplify the evaluation.

Question 4

A clustered RCT clearly has less statistical power than an individual-level RCT of the same sample size, since a clustered RCT will require adjustments for intracluster correlation. A factorial design will also likely have less power since you would need to allocate the sample to multiple treatment arms, instead of two only two treatment arms. An encouragement design has less power because of incomplete take-up; treatment effect estimates will rely on those who actually take up the program, which will only be a subset of the full sample. To detect effects with this subsample requires a larger overall sample.

On the other hand, a stratified RCT may have more power than a simple RCT with the same sample size, by eliminating sources of imbalance in the stratification covariates.