Topic 12 RCT and IV designs

Learning Goals

  1. Use d-separation ideas to understand the assumptions behind instrumental variables estimation
  2. Implement a simulation of data arising from an IV design
  3. Use simulation to understand properties of IV estimation
  4. Evaluate the validity of potential instruments in research settings
  5. Relate the IV approach to imperfect randomized trials





Warm-up

Question: For studying the effect of tenofovir on risk of developing COVID-19…

  • What might a possible instrumental variables study design look like?
  • What might a possible regression discontinuity design look like?
  • What might a possible interrupted times series design look like?





Discussion

There are 4 core assumptions underlying an IV analysis:

  1. Relevance: The instrument is associated with the treatment
  2. Exclusion restriction: The instrument only affects the outcome through the treatment variable \(A\) (no direct effect of instrument on \(Y\))
  3. Randomized assignment: The instrument and the outcome do not have common causes. The instrument is as if randomly assigned.
  4. Monotonicity: There are no “defiers”. When there are no defiers, the IV estimate ends up generalizing only to the compliers and is thus called a complier average causal effect (CACE) or a local average treatment effect (LATE).
    Let \(A^{IV}\) be the potential outcome for the treatment variable depending on the value of the instrument \(IV\).
    Four compliance types or principal strata:
    • Always-takers: Individuals who always take treatment, regardless of what their IV assigned. i.e., \(A^{IV=0} = 1, A^{IV=1} = 1\)
    • Never-takers: Individuals who never take treatment, regardless of what their IV assigned. i.e., \(A^{IV=0} = 0, A^{IV=1} = 0\)
    • Compliers: Individuals who will take treatment when assigned and not when not assigned. i.e., \(A^{IV=0} = 0, A^{IV=1} = 1\)
    • Defiers: Individuals who will take treatment when assigned and not when not assigned. i.e., \(A^{IV=0} = 1, A^{IV=1} = 0\)

The first three assumptions are instrumental conditions. The fourth assumption affects the population to which the IV estimate generalizes.

For more detail about assumption 4, see Chapter 16 of our WHATIF book.





Exercises

A template Rmd is available here.

IV assumptions

Using d-separation ideas, explain why the first three assumptions are needed for instrumental variables estimation.



Simulation planning

We will be implementing a simulation study to understand statistical properties of IV estimation. Before we jump in, let’s plan:

  • How will we set up a setting where we know the true value of the causal effect? (Think about simulating DAGs.)
  • We will also be thinking of uncertainty quantification. That is, we will be looking at properties of confidence intervals for effect estimates.



Simulation: setting up helper functions

Functions are a great way to organize code, repeat tasks, and avoid errors by modularizing key chunks of a workflow. We will construct several functions to help with our simulation.

First, complete the get_sim_data() function below. (There are several missing pieces.) This function simulates data according to the DAG shown in our video.

  • Note that it takes 3 arguments. How are these arguments used within the function?
  • If you need to simulate any exogenous variables (variables with no causes), simulate them with a 50% probability of equaling 1.
library(dplyr)

get_sim_data <- function(n, p_A_IV0, p_A_IV1) {
    p_AfromIV <- dplyr::case_when(
        IV==1 ~ p_A_IV1,
        IV==0 ~ p_A_IV0
    )
    p_AfromC <- dplyr::case_when(
        C==1 ~ 0.5,
        C==0 ~ 0.2
    )
    p_A <- p_AfromIV*p_AfromC
    A <- rbinom(n, size = 1, prob = p_A)

    A_IVdo0 <- rbinom(n, size = 1, prob = p_A_IVdo0)
    A_IVdo1 <- rbinom(n, size = 1, prob = p_A_IVdo1)

    p_YfromA <- dplyr::case_when(
        A==1 ~ 0.6,
        A==0 ~ 0.3
    )
    p_YfromC <- dplyr::case_when(
        C==1 ~ 0.7,
        C==0 ~ 0.4
    )
    p_Y <- p_YfromA*p_YfromC
    Y <- rbinom(n, size = 1, prob = p_Y)

    Y_Ado0 <- rbinom(n, size = 1, prob = p_Y_Ado0)
    Y_Ado1 <- rbinom(n, size = 1, prob = p_Y_Ado1)

    sim_data <- data.frame(IV, C, A, A_IVdo0, A_IVdo1, Y, Y_Ado0, Y_Ado1)
    sim_data
}

Next, complete the functions below to obtain the ACE (\(P(Y^{a=1}=1)-P(Y^{a=0}=1)\)) and the CACE.

  • These functions take as input a data argument. We will be supplying the results of get_sim_data(), so you can assume that the sim_data variable names are present.
  • Examples of using filter():
    • filter(A==1) filters down to the treated
    • filter(A==1 | C==1) filters down to the rows where A = 1 OR C = 1
    • filter(A==1 & C==1) filters down to the rows where A = 1 AND C = 1
get_ace <- function(data) {
}

get_cace <- function(data) {
    data_subs <- data %>%
        filter()
}

Next complete the get_iv_estimate() function below to implement the instrumental variables estimate of the causal effect of treatment:

get_iv_estimate <- function(data) {
    component1 <- data %>%
        group_by(IV) %>%
        summarize(???) %>%
        pull(???) %>%
        diff()
}

The bootstrap_iv_estimate() function below performs 1000 bootstrapping iterations to get 1000 IV estimates. Add comments to document what each piece of code does.

bootstrap_iv_estimate <- function(data) {
    replicate(1000, {
        data_resampled <- data %>% sample_n(size = nrow(data), replace = TRUE)
        get_iv_estimate(data_resampled)
    })
}



Running the simulation

Now let’s actually run the simulation study! The for-loop below loops over different values of a parameter \(p\) for \(p \in \{0.05,0.15,0.25,0.35,0.45\}\).

Fill in the question marks below. When you run this code, it will take a little while.

set.seed(394)
for (p in seq(0.05,0.45,0.1)) {
    cat("p=", p, "\n")
    sim_data <- get_sim_data(n = 1e5, p_A_IV0 = p, p_A_IV1 = 1-p)
    ace <- ???
    cace <- ???
    cat("ACE (whole pop):", ace, "\n")
    cat("CACE:", cace, "\n")
    iv_estim <- get_iv_estimate(sim_data)
    boot_iv_results <- bootstrap_iv_estimate(sim_data)
    boot_ci <- quantile(boot_iv_results, probs = ???) # 95% CI
    cat("IV results:\n")
    cat("    Estimate:", iv_estim, "\n")
    cat("    95% bootstrap CI:", boot_ci, "\n")
    cat("    CI width:", diff(boot_ci), "\n\n")
}

Form some conclusions from the results:

  • There is a notion of an instrument being “strong” or “weak”, which relates to how strongly the instrument influences/predicts the treatment \(A\). Does increasing \(p\) correspond to a stronger or weaker instrument?
  • What do you notice about how well the IV estimate predicts the average causal effect? The complier average causal effect?
  • What do you notice about how the confidence intervals change with changing instrument strength?





Instruments in practice

Some examples of instruments used in health services research are below. In light of the assumptions underlying IV estimation, do you think that these instruments could be valid? (Validity of an instrument refers to the assumptions being met.) What threats to validity do you foresee?

  • Randomized encouragement designs: Physicians or patients are randomly assigned to receive encouragement (e.g., promotional materials) to use certain treatments.

  • Preference-based instruments: individual physicians, hospitals, or care providers have varied and “random” preferences for one treatment over others.

  • Distance to a specialty care provider: A special health service may only be offered at certain locations. Distance to such a specialty care provider might be random across patients and also impact whether or not a patient receives the special health service.

This article provides more examples and is also a great review article about instrumental variables in general. It has great discussion about considerations for reporting results from IV analyses and also discusses a generalization of our IV estimator (also called the Wald estimator): two-stage least squares. If you are curious about any of these ideas, ask the instructor!





IV estimation and RCTs

The issue of noncompliance in randomized trials was raised in the video. Noncompliance occurs when an individual does not comply with the treatment that was randomly assigned.

  • Compared to the usual causal diagram depicting a (perfect) randomized trial, how does a randomized trial with noncompliance compare?
  • How could instrumental variables estimation be useful in this setting?
  • In an IV analysis, to whom does the effect estimate generalize, and why might this be a concern in practice?