Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty
Beschreibung
"This study explores how researchers’ analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions. Researchers’ expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team’s workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation. The idiosyncratic nature of how researchers’ results and conclusions varied is a previously underappreciated explanation for why many scientific hypotheses remain contested. These results call for greater epistemic humility and clarity in reporting scientific findings." (Author's abstract, IAB-Doku) ((en))
Zitationshinweis
Breznau, Nate, Eike Mark Rinke, Alexander Wuttke, Nora Storz, Christof Wolf, Zsófia S. Ignácz, Fabian Winter, Sophia Hunger, Andrey Tibajev, Oshrat Hochman, Hannah Werner, Natalia Malancu, Nico Sonntag, Alexander Langenkamp, Fieke Wagemans, Magdalena Hirsch, Bogdan Voicu, Luis Maldonado, Stefan Vogtenhuber, Philipp Lutscher, Arno Van Hootegem, Philipp M. Lersch, Jolanda van der Noll, Dafina Kurti Sinatra, Meta van der Linden, John Kuk, Jasper Van Assche, Jan-Philipp Kolb, Brenton M. Wiernik, Lea-Maria Löbel, Friedolin Merhout, Kaspar Burger, Gunnar Otte, Elena Damian, Olav Nygård, Roxanne Connelly, Daniel Nüst, Maureen A. Eger, Marcel Neunhoeffer, Simon Ellerbrock, Cristóbal Moya, Katharina Burgdorf, Julia Schulte-Cloos, Amie Bostic, Leticia Micheli, Andrea Bohman, Christopher Prosser, Timo Gnambs, Samuel Merk, Jan Paul Heisig, Jonathan Mijs, Tobias Gummer, Jonathan Mellon, Martin Groß, Daniel Meierrieks, Amélie Godefroidt, Cecil Meeusen, Theresa Gessler, Kyle McWagner, Konstantin Gavras, Patricia McManus, Andrea Forster, Oscar J. Mayorga, Max Grömping, Jochen Mayerl, Sandy Schumann, Nadia Steiber, Anne-Kathrin Stroppe, Claudia Schmiedeberg, Sebastian Jungkunz, Daniel Meyer, Dieter Stiers, Jürgen Schneider, Nils Steiner, Louis Raes, Alexander Staudt, Alexander Schmidt-Catran, Willem Sleegers, Katja M. Schmidt, Julian Seuring, Regine Schmidt, Reinhard Schunck, Elmar Schlueter, Nadja Wehl, David Schieferdecker, Stefan Zins, Merlin Schaeffer, Nan Zhang, Ariela Schachter, Nora Huth, Gregor Sand, Mikael Hjerm, Robin Samuel, Zuzanna Brzozowska, Guido Ropers, Joan E. Madia, Jonathan Rogers, Manuel Kleinert, Arne Roets, Marta Kołczyńska, Miguel R. Ramos, Kathrin B. Busch, Kevin Ralston, Christian S. Czymara, Martijn Schoonvelde, Juan Carlos-Castillo, Mathias Kauff, Anna O. Pechenkina, Nils Jungmann, Nathan Chan, Sebastian Hellmeier, Andreas Hadjar, Achim Edelmann, Laura Jacobs, Vernon Gayle, Chris Gaasendam, Hung H. V. Nguyen, Anna Forke, Janna Teltemann, Muna Adem, Dragana Stojmenovska, Giacomo Vagni, Bastian Jaeger, Henning Silber, Jannes Jacobsen, Sebastian Sternberg, Licia Bobzien, Conrad Ziller, Johannes N. Blumenberg, Andreas Hövermann, Christian Hunkler, Helge Marahrens, Matthias Mader, Katharina Blinzler, Julia Klinger, Thomas Biegert, Pablo Christmann, Felix S. Bethke, Thijs Bol, Anna Berthold, Stefanie Heyne, Carl Berning, Erich Striessnig, Julian Bernauer, Paul Martinez, Verena Benoit, Tomasz Żółtak, Sharon Baute, Jule Adriaans, Markus Baumann, Alejandro Ecker, Paul C. Bauer, Stefan Gruber, Gerrit Bauer, Brian Tung, Dave Balzer, Yuki Yamada, Oke Bahnsen, Nicole Martin, Flavio Azevedo, Jürgen Schupp, Daniel Auer, Fabian Ochsenfeld, Henrik K. Andersen, Katharina Kunißen & Amalia Alvarez-Benjumea (2022): Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty. In: Proceedings of the National Academy of Sciences of the United States of America, Jg. 119, H. 44. DOI:10.1073/pnas.2203150119