class: center, middle, inverse, title-slide .title[ # 7: Sampling and survey research ] .subtitle[ ## Empirical Methods ] .author[ ### <large>J. Seawright</large> ] .institute[ ### <small>Northwestern Political Science</small> ] .date[ ### Oct. 7, 2025 ] --- class: center, middle <style type="text/css"> pre { max-height: 400px; overflow-y: auto; } pre[class] { max-height: 200px; } </style> --- ### Stages of Survey Research - Select a set of individuals to answer the survey. - Ask each individual a set of pre-selected questions. - Analyze the results. --- ### Define the Population --- ### Choose a Sampling Rule - Sometimes we can study all of the cases in a population. - Sometimes we need to focus more narrowly. - We need a systematic way of deciding which cases to focus on --- a **sampling rule**. - We hope that our sampling rule will give us correct results: we want our samples to be like the population. --- ### Samples of Convenience A **sample of convenience** is a sampling rule where the researchers simply select the cases that are easily available. --- <img src="images/conveniencesample.jpg" width="90%" style="display: block; margin: auto;" /> --- In October of 2016, news began to circulate online that the major polling firms were producing badly biased results. --- Bill O'Reilly contributed to this conversation, according to various online news sources, by hiring graduate students to "call 250 people from each party's voting rolls in all 50 states." --- <img src="images/oreillypoll.jpg" width="90%" style="display: block; margin: auto;" /> --- <img src="images/trumpvoterfraud.jpg" width="90%" style="display: block; margin: auto;" /> --- But, oops, O'Reilly never actually hired anyone to do a giant telephone survey. <img src="images/oreillywebpoll.jpg" width="90%" style="display: block; margin: auto;" /> --- ### Quota Sampling A **quota sample** is where the researchers modify a sample of convenience by making sure the sample is like the population in specific, predetermined ways. --- <img src="images/deweydefeatstruman.jpg" width="90%" style="display: block; margin: auto;" /> --- ### Random Sampling A **random sample** is a sample where each case has a known probability of selection. --- ### Random Sampling The *law of large numbers* tells us that, with a large enough sample, random sampling basically guarantees that our sample will be like the population. --- ### Random Sampling Kinds of random samples: - Simple random sample. - Cluster sample. - Stratified random sample. --- ### What Can Go Wrong? Non-response; some people just don't want to participate in surveys. - Non-response is usually non-random, which means that it spoils most of our random samples. - What can we do to deal with non-response? --- ### The New World Response rates have basically collapsed in surveys in much of the world. --- <img src="images/govtsurveyresponserates.jpg" width="90%" style="display: block; margin: auto;" /> --- <img src="images/privatesectorresponserates.png" width="90%" style="display: block; margin: auto;" /> --- <img src="images/nonresponsecorrelations.jpg" width="90%" style="display: block; margin: auto;" /> --- <img src="images/Stantcheva1.png" width="90%" style="display: block; margin: auto;" /> --- <img src="images/Stantcheva2.png" width="90%" style="display: block; margin: auto;" /> --- <img src="images/Stantcheva3.png" width="90%" style="display: block; margin: auto;" /> --- <img src="images/Stantcheva4.png" width="90%" style="display: block; margin: auto;" /> --- ### What about 2016 and 2020? --- ### Was 2016 a Polling Miss? <img src="images/surveys2016.png" width="90%" style="display: block; margin: auto;" /> --- ### Was 2016 a Polling Miss? <img src="images/fivethirtyeight2016.png" width="70%" style="display: block; margin: auto;" /> --- ### Was 2016 a Polling Miss? <img src="images/reuters2016.png" width="90%" style="display: block; margin: auto;" /> --- ### Was 2020 a Polling Miss? <img src="images/surveys2020.png" width="90%" style="display: block; margin: auto;" /> --- <img src="images/surveytaskforce.png" width="90%" style="display: block; margin: auto;" /> --- <img src="images/surveyreport1.png" width="90%" style="display: block; margin: auto;" /> --- <img src="images/surveyreport2.png" width="90%" style="display: block; margin: auto;" /> --- <img src="images/surveyreport3.png" width="90%" style="display: block; margin: auto;" /> --- <img src="images/surveyreport4.png" width="90%" style="display: block; margin: auto;" /> --- <img src="images/surveyreport5.png" width="90%" style="display: block; margin: auto;" /> --- <img src="images/surveyreport6.png" width="90%" style="display: block; margin: auto;" /> --- <img src="images/surveyreport7.png" width="90%" style="display: block; margin: auto;" /> --- How much does question wording matter? --- <img src="images/Bruinsma1.png" width="90%" style="display: block; margin: auto;" /> --- <img src="images/Bruinsma2.png" width="90%" style="display: block; margin: auto;" /> --- There's no way to generate a perfectly neutral survey question, but the key is to ask things as simply as possible.