American Political Science Review

Research Article

Are Survey Experiments Externally Valid?

JASON BARABASa1 c1 and JENNIFER JERITa1 c2

a1 Florida State University

Abstract

Researchers use survey experiments to establish causal effects in descriptively representative samples, but concerns remain regarding the strength of the stimuli and the lack of realism in experimental settings. We explore these issues by comparing three national survey experiments on Medicare and immigration with contemporaneous natural experiments on the same topics. The survey experiments reveal that providing information increases political knowledge and alters attitudes. In contrast, two real-world government announcements had no discernable effects, except among people who were exposed to the same facts publicized in the mass media. Even among this exposed subsample, treatment effects were smaller and sometimes pointed in the opposite direction. Methodologically, our results suggest the need for caution when extrapolating from survey experiments. Substantively, we find that many citizens are able to recall factual information appearing in the news but may not adjust their beliefs and opinions in response to this information.

Correspondence:

c1 Jason Barabas is Associate Professor, Department of Political Science, Florida State University, 531 Bellamy Building, Tallahassee, FL 32306 (jason.barabas@fsu.edu).

c2 Jennifer Jerit is Associate Professor, Department of Political Science, Florida State University, 531 Bellamy Building, Tallahassee, FL 32306 (jjerit@fsu.edu).

Footnotes

The authors appreciate the helpful comments they received from Scott Allard, Charles Barrilleaux, Bill Berry, John Bullock, Brian Gaines, Cindy Kam, Gary King, Jon Krosnick, Jim Kuklinski, Skip Lupia, Cherie Maestas, Rose McDermott, Betsy Sinclair, Mark Souva, John Transue, Mathieu Turgeon, Gabor Toka, Paul Quirk, Will Shadish, Joe Young, and participants in colloquia at Florida State University, the University of North Carolina at Chapel Hill, Texas A&M University, and Vanderbilt University. Earlier versions of this article were presented at the Visions in Methodology Conference at Ohio State University, and at annual meetings of the American Political Science Association, the Canadian Political Science Association, and the International Society of Political Psychology. Ben Gaskins provided valuable research assistance. Data, replication code, and an online Appendix are available at http://polisci.fsu.edu/people/faculty/index.htm.

Metrics