Analytic error as an important component of total survey error

Abstract

"We describe the results of a meta-analysis that sought to quantify the frequency with which researchers do not use correct analytic techniques for complex sample survey data when publishing secondary analyses of survey data arising from complex samples. The analytic errors that may result from incorrect analyses of these data represent an important but understudied aspect of the total survey error (TSE) paradigm, and can negate the positive data quality aspects introduced by dedicating substantial resources to the minimization of other important sources of survey error (e.g., nonresponse error and measurement error). We also sought to explore whether characteristics of the journals in which these articles were published (e.g., impact factor, presence of statisticians an the editorial boards, analytic guidelines for authors, etc.) were related to the prevalence of various errors. The results of the meta-analysis suggest that several types of apparent analytic errors are in fact quite prevalent, including inappropriate subpopulation analyses and a failure to use appropriate software for survey data analysis. Analysts also failed to incorporate weights or compute standard errors reflecting sample design features more often than would be desirable, and we find that descriptions of analysis results and inferences may tend to mislead readers about the scope of the inferences (i.e., population vs. sample). We also find that most peer-reviewed journals, including those with large impact factors, fail to emphasize the use of specialized analysis methods for secondary analysts of complex sample survey data in their guidelines for authors. These findings suggest that academic journals and survey organizations could do much more to emphasize the types of analyses that would be appropriate for a given complex sample survey data set." (Text excerpt, IAB-Doku) ((en))

Cite article

West, B., Sakshaug, J. & Kim, Y. (2017): Analytic error as an important component of total survey error. Results from a meta-analysis. In: P. P. Biemer, E. de Leeuw, S. Eckman, B. Edwards, F. Kreuter, L. E. Lyberg, N. C. Tucker & B. T. West (Hrsg.) (2017): Total Survey Error in Practice, p. 489-510.