Economic Policy Group co-organizers, John Cochrane and Valerie Ramey, hosted a talk on “Quantifying Non-Sampling Variation: College Quality and the Garden of Forking Paths.”
Presenter: Jeffrey Smith, the Paul T. Heyne Distinguished Chair in Economics and the Richard Meese Chair in Applied Econometrics at the University of Wisconsin–Madison.
Moderator: John Cochrane, the Rose-Marie and Jack Anderson Senior Fellow at the Hoover Institution
Wednesday, October 29, 2025
Hoover Institution, Stanford University
SUMMARY
Empirical economics papers report standard errors that quantify the uncertainty associated with sampling variation but rarely consider non-sampling variation in a systematic way. Nonsampling variation arises from researcher study design choices related to measurement of key variables, functional form, tuning parameters, model selection procedures, and so on. This paper documents the current state of play regarding non-sampling variation and describes approaches from inside and outside of economics to quantify such variation more systematically. We provide a worked example in the form of an analysis of the labor market and educational effects of college quality on degree completion and earnings using data from the National Longitudinal Survey of Youth 1997 cohort (NLSY-97). In our analysis, we consider multiple ways to create our college quality index, multiple ways to deal with item non-response in our conditioning variables, multiple ways to code our earnings outcome measures, and multiple ways to choose a specification for our conditioning variables. We find that in this context, in which sampling variation matters a lot due to the relatively small sample size of the NLSY-97 and relatively high residual variance of our outcomes, these dimensions of non-sampling variability imply uncertainty in our parameter estimates on a par with the sampling variation.
To read the slides, click the following link
https://www.hoover.org/sites/default/files/2025-10/Jeffrey%20Smith%20Hoover%20EPWG%20Forking%20Paths%20Talk%2010-2025.pdf
__________
The opinions expressed on this channel are those of the authors and do not necessarily reflect the opinions of the Hoover Institution or Stanford University.
© 2025 by the Board of Trustees of Leland Stanford Junior University.
🔔 Subscribe for more discussions: @HooverInstitution
👍 Like and share this video!
🌐 Follow us on social media:
• Facebook: https://www.facebook.com/HooverInstStanford
• LinkedIn: https://www.linkedin.com/company/hoover-institution-at-stanford-university
• Instagram: https://instagram.com/hooverinstitution
• TikTok: https://www.tiktok.com/@hooverinstitution
• X: https://www.twitter.com/HooverInst
Trump meeting with Xi
US and China agree to trade framework
