5: Experiments

class: center, middle, inverse, title-slide

.title[
# 5: Experiments
]
.subtitle[
## Empirical Methods
]
.author[
### <large>J. Seawright</large>
]
.institute[
### <small>Northwestern Political Science</small>
]
.date[
### Sept. 30, 2025
]

---

class: center, middle

pre[class] {
  max-height: 200px;
}
</style>

---
### Experiments

-   The fundamental problem of causal inference is that we can't compare
    a world with the cause to a world without the cause.

-   But if we take two large random samples, we know they are pretty
    much the same as each other.

---
### In-Class Example

---
### Experiments

-   So, we can apply the cause to one group but not the other.

---
### Experiments

-   There are two prerequisites:

-   We need to be able to take two large random samples of
        independent cases.

-   We need to be able to manipulate the cause that we're interested
        in.

---
### Example: Do Media Retractions Work?

---

---

---
### What do we need to assume for the previous slide's results to be causal?

---
### Causal Inference with Experiments

-   Measure the dependent variable somehow.

-   Look for a difference on the dependent variable between the
    treatment group and the control group or between different treatment
    groups.

---
### Evaluating Causal Inferences

-   Internal validity: does the treatment seem to cause the dependent
    variable for the cases under study?

-   External validity: does the causal claim in question seem to
    generalize appropriately to the desired population, and does the
    experiment seem to capture the desired concepts?

---
### Internal Validity

-   As long as the randomization procedure was done correctly, an
    experiment always ranks high on internal validity.

---
### External Validity

-   Most experiments are much weaker on external validity than they are
    on internal validity.

-   The subjects for an experiment may not be a random sample of the
        relevant population.

-   The experimental treatment may differ from the real-world
        independent variable.

-   Our descriptive inferences about the dependent variable may not
        be done well.

---
### Questions to Ask about Experiments

To figure out about internal validity (AKA confounding variables,
whether causation moves in the right order), ask:

1.  Was the treatment randomly assigned?

2.  Was the sample size large?

3.  Was treatment compliance pretty good?

4.  Was there evidence of fraud, manipulation, or other failures?

---
### Questions to Ask about Experiments

To figure out about external validity (AKA whether we can generalize to
the population of interest), ask:

1.  Is the set of experimental subjects representative of the population
    of interest?

2.  Is the experimental treatment similar enough to the independent
    variable in the theory?
    
3.  Is the dependent variable measured well enough?

---
<img src="images/Laterzo1.png" width="90%" style="display: block; margin: auto;" />

---
<img src="images/Laterzo2.png" width="90%" style="display: block; margin: auto;" />

---
<img src="images/Laterzo3.png" width="90%" style="display: block; margin: auto;" />

---
<img src="images/Laterzo4.png" width="90%" style="display: block; margin: auto;" />

---

D. Hangartner, G. Gennaro, S. Alasiri, N. Bahrich, A. Bornhoft, J. Boucher, B.B. Demirci, L. Derksen, A. Hall, M. Jochum, M.M. Munoz, M. Richter, F. Vogel, S. Wittwer, F. Wuthrich, F. Gilardi, & K. Donnay, Empathy-based counterspeech can reduce racist hate speech in a social media field experiment, Proc. Natl. Acad. Sci. U.S.A. 118 (50) e2116310118, [https://doi.org/10.1073/pnas.2116310118](https://doi.org/10.1073/pnas.2116310118) (2021).

---
<img src="images/Hangartner1.png" width="70%" style="display: block; margin: auto;" />

---
<img src="images/Hangartner2.png" width="90%" style="display: block; margin: auto;" />

---
<img src="images/Hangartner3.png" width="90%" style="display: block; margin: auto;" />

---
<img src="images/Hangartner4.png" width="90%" style="display: block; margin: auto;" />

---
### Oct. 2 Class