Approaches toward External Validity of Randomized Controlled Trials in the Social Sciences

Social scientists and decision makers have been interested in whether and how well they can generalize the results of randomized controlled trials (RCTs) to new populations, places, and contexts. This question of external validity is a foundation of evidence-based decision making because study populations and sites are often different from the real-world populations and sites to which an intervention might be scaled up. This research develops statistical approaches to improve the external validity of RCTs in the social sciences. The research makes two methodological contributions. The first is a new framework to design RCTs for external validity. Specifically, this project develops an algorithm to select experimental sites such that researchers can credibly estimate generalizable causal effects. Site selection is essential because experimental results in the social sciences are often heterogeneous across places, and biased selection of experimental sites can lead to low generalizability and replicability. The second contribution is a statistical tool to quantify the robustness to external validity bias. This new measure of external robustness allows researchers to evaluate external validity even when RCTs were conducted without external validity considerations. These two methods complement each other and together provide a unified pipeline to improve the external validity of RCTs in the social sciences. The project is co-funded by the Science of Science: Discovery, Communications, and Impact program; the Accountability, Institutions and Behavior program; and the Office of Integrative Activities. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.