Many analysts, one dataset: making transparent how variations in analytical choices affect results

Silberzahn, R. and Uhlmann, E.L. and Martin, D.P. and Anselmi, P. and Aust, F. and Awtrey, E. and Bahník, Š. and Bai, F. and Bannard, C. and Bonnier, E. and Carlsson, R. and Cheung, F. and Christensen, G. and Clay, R. and Craig, M.A. and Dalla Rosa, A. and Dam, L. and Evans, M.H. and Flores Cervantes, I. and Fong, N. and Gamez-Djokic, M. and Glenz, A. and Gordon-McKeon, S. and Heaton, T.J. and Hederos, K. and Heene, M. and Hofelich Mohr, A.J. and Högden, F. and Hui, K. and Johannesson, M. and Kalodimos, J. and Kaszubowski, E. and Kennedy, D.M. and Lei, R. and Lindsay, T.A. and Liverani, S. and Madan, C.R. and Molden, D. and Molleman, E. and Morey, R.D. and Mulder, L.B. and Nijstad, B.R. and Pope, N.G. and Pope, B. and Prenoveau, J.M. and Rink, F. and Robusto, E. and Roderique, H. and Sandberg, A. and Schlüter, E. and Schönbrodt, F.D. and Sherman, M.F. and Sommer, S.A. and Sotak, K. and Spain, S. and Spörlein, C. and Stafford, T. and Stefanutti, L. and Tauber, S. and Ullrich, J. and Vianello, M. and Wagenmakers, E.-J. and Witkowiak, M. and Yoon, S. and Nosek, B.A. (2018) Many analysts, one dataset: making transparent how variations in analytical choices affect results. Advances in Methods and Practices in Psychological Science . ISSN 2515-2459

Full text not available from this repository.


Twenty-nine teams involving 61 analysts used the same dataset to address the same research question: whether soccer referees are more likely to give red cards to dark skin toned players than light skin toned players. Analytic approaches varied widely across teams, and estimated effect sizes ranged from 0.89 to 2.93 in odds ratio units, with a median of 1.31. Twenty teams (69%) found a statistically significant positive effect and nine teams (31%) observed a nonsignificant relationship. Overall 29 different analyses used 21 unique combinations of covariates. We found that neither analysts' prior beliefs about the effect, nor their level of expertise, nor peer-reviewed quality of analysis readily explained variation in analysis outcomes. This suggests that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. Crowdsourcing data analysis, a strategy by which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective analytic choices influence research results.

Item Type: Article
Keywords: Crowdsourcing science; Data analysis; Scientific transparency; Open data; Open materials
Schools/Departments: University of Nottingham, UK > Faculty of Science > School of Psychology
Identification Number:
Depositing User: Eprints, Support
Date Deposited: 15 Nov 2017 14:53
Last Modified: 04 May 2020 19:08

Actions (Archive Staff Only)

Edit View Edit View