Causal Inference – Quiz 1

Quiz Question Explanations

Question 1

The two statements are equivalent, and both describe the key assumption for a pre-post estimate to represent the causal impact of a program: no change over time in the outcome in the absence of the program. If the outcome would have changed, and if that change could not be perfectly predicted, then it is not possible to use pre-post data to recover the causal impact of a program.

Question 2

If there is an inflection point between pre-program trends and post-program trends, it is more credible that the program caused the inflection. However, it is not conclusive, since other factors may have changed at the same time as the program was introduced.

Depending on the outcome, it may be valid to assume that a linear trend approximates pre-program outcomes, though this is not the case for all outcomes. If the researcher has access to a lot of pre-program data points, it would be a good idea to try different functional forms and identify the best-fit, rather than to assume linearity.

Question 3

Scenario 1: Teachers who sign up for the program may differ from those who do not. For instance, teachers who sign up may be more motivated to invest in their professional development, and thus may have better outcomes even in the absence of the program.

Scenario 2: The poorest households may have different outcomes than the average household in a village (and by definition would have different poverty outcomes)

Scenario 3: The control group has been influenced by knowledge spillovers from the treatment group. A comparison of treatment and control will now be a comparison of the direct program impact with the spillover effects, which would likely be an underestimate of the program impact.

Scenario 4: Since this data is being collected at baseline, and since there are no stated differences in respondent reluctance across treatment and comparison groups, there is unlikely to be selection bias.

Scenario 5: The youth employment training program is intended to increase employment. Thus there may be a difference in respondent availability between treatment and control groups. If response rates are higher for the control group than for the treatment group, this may introduce selection bias.

Question 4

A key issue is how regions were selected to receive the treatment. If that selection process is correlated with outcomes, and if you can’t control for those differences with available data, then impact estimates may be biased.

Even if regions are similar on outcomes and observables at baseline, another concern in a non-randomized design is the treated regions may experience different ‘shocks’ than control regions, such as other programs being implemented, different economic, political, or environmental factors.

A regression model does not have a restriction on the number of control variables, or any inherent ethical considerations. Control variables should be selected to reduce differences between the treatment and comparison groups and reduce residual variance in outcomes.