Elliot Gould Elliot Gould Shinichi Nakagawa Shinichi Nakagawa Simon C. Griffith Simon C. Griffith Peter A. Vesk Peter A. Vesk Fiona Fidler Fiona Fidler Daniel G. Hamilton Daniel G. Hamilton Robin N. Abbey-Lee Robin N. Abbey-Lee Jessica K. Abbott Jessica K. Abbott Luis A. Aguirre Luis A. Aguirre Carles Alcaraz Carles Alcaraz Irith Aloni Irith Aloni Drew Altschul Drew Altschul Kunal Arekar Kunal Arekar Jeff W. Atkins Jeff W. Atkins Joe Atkinson Joe Atkinson Christopher M. Baker Christopher M. Baker Meghan Barrett Meghan Barrett Kristian Bell Kristian Bell Suleiman Kehinde Bello Suleiman Kehinde Bello Iván Beltrán Iván Beltrán Bernd J. Berauer Bernd J. Berauer Michael Grant Bertram Michael Grant Bertram Peter D. Billman Peter D. Billman Charlie K. Blake Charlie K. Blake Shannon Blake Shannon Blake Louis Bliard Louis Bliard Andrea Bonisoli-Alquati Andrea Bonisoli-Alquati Timothée Bonnet Timothée Bonnet Camille Nina Marion Bordes Camille Nina Marion Bordes Aneesh P. H. Bose Aneesh P. H. Bose Thomas Botterill-James Thomas Botterill-James Melissa Anna Boyd Melissa Anna Boyd Sarah A. Boyle Sarah A. Boyle Tom Bradfer-Lawrence Tom Bradfer-Lawrence Jennifer Bradham Jennifer Bradham Jack A. Brand Jack A. Brand Martin I. Brengdahl Martin I. Brengdahl Martin Bulla Martin Bulla Luc Bussière Luc Bussière Ettore Camerlenghi Ettore Camerlenghi Sara E. Campbell Sara E. Campbell Leonardo L. F. Campos Leonardo L. F. Campos Anthony Caravaggi Anthony Caravaggi Pedro Cardoso Pedro Cardoso Charles J. W. Carroll Charles J. W. Carroll Therese A. Catanach Therese A. Catanach Xuan Chen Xuan Chen Heung Ying Janet Chik Heung Ying Janet Chik Emily Sarah Choy Emily Sarah Choy Alec Philip Christie Alec Philip Christie Angela Chuang Angela Chuang Amanda J. Chunco Amanda J. Chunco Bethany L. Clark Bethany L. Clark Andrea Contina Andrea Contina Garth A. Covernton Garth A. Covernton Murray P. Cox Murray P. Cox Kimberly A. Cressman Kimberly A. Cressman Marco Crotti Marco Crotti Connor Davidson Crouch Connor Davidson Crouch Pietro B. D’Amelio Pietro B. D’Amelio Alexandra Allison de Sousa Alexandra Allison de Sousa Timm Fabian Döbert Timm Fabian Döbert Ralph Dobler Ralph Dobler Adam J. Dobson Adam J. Dobson Tim S. Doherty Tim S. Doherty Szymon Marian Drobniak Szymon Marian Drobniak Alexandra Grace Duffy Alexandra Grace Duffy Alison B. Duncan Alison B. Duncan Robert P. Dunn Robert P. Dunn Jamie Dunning Jamie Dunning Trishna Dutta Trishna Dutta Luke Eberhart-Hertel Luke Eberhart-Hertel Jared Alan Elmore Jared Alan Elmore Mahmoud Medhat Elsherif Mahmoud Medhat Elsherif Holly M. English Holly M. English David C. Ensminger David C. Ensminger Ulrich Rainer Ernst Ulrich Rainer Ernst Stephen M. Ferguson Stephen M. Ferguson Esteban Fernandez-Juricic Esteban Fernandez-Juricic Thalita Ferreira-Arruda Thalita Ferreira-Arruda John Fieberg John Fieberg Elizabeth A. Finch Elizabeth A. Finch Evan A. Fiorenza Evan A. Fiorenza David N. Fisher David N. Fisher Amélie Fontaine Amélie Fontaine Wolfgang Forstmeier Wolfgang Forstmeier Yoan Fourcade Yoan Fourcade Graham S. Frank Graham S. Frank Cathryn A. Freund Cathryn A. Freund Eduardo Fuentes-Lillo Eduardo Fuentes-Lillo Sara L. Gandy Sara L. Gandy Dustin G. Gannon Dustin G. Gannon Ana I. García-Cervigón Ana I. García-Cervigón Alexis C. Garretson Alexis C. Garretson Xuezhen Ge Xuezhen Ge William L. Geary William L. Geary Charly Géron Charly Géron Marc Gilles Marc Gilles Antje Girndt Antje Girndt Daniel Gliksman Daniel Gliksman Harrison B. Goldspiel Harrison B. Goldspiel Dylan G. E. Gomes Dylan G. E. Gomes Megan Kate Good Megan Kate Good Sarah C. Goslee Sarah C. Goslee J. Stephen Gosnell J. Stephen Gosnell Eliza M. Grames Eliza M. Grames Paolo Gratton Paolo Gratton Nicholas M. Grebe Nicholas M. Grebe Skye M. Greenler Skye M. Greenler Maaike Griffioen Maaike Griffioen Daniel M. Griffith Daniel M. Griffith Frances J. Griffith Frances J. Griffith Jake J. Grossman Jake J. Grossman Ali Güncan Ali Güncan Stef Haesen Stef Haesen James G. Hagan James G. Hagan Heather A. Hager Heather A. Hager Jonathan Philo Harris Jonathan Philo Harris Natasha Dean Harrison Natasha Dean Harrison Sarah Syedia Hasnain Sarah Syedia Hasnain Justin Chase Havird Justin Chase Havird Andrew J. Heaton Andrew J. Heaton María Laura Herrera-Chaustre María Laura Herrera-Chaustre Tanner J. Howard Tanner J. Howard Bin-Yan Hsu Bin-Yan Hsu Fabiola Iannarilli Fabiola Iannarilli Esperanza C. Iranzo Esperanza C. Iranzo Erik N. K. Iverson Erik N. K. Iverson Saheed Olaide Jimoh Saheed Olaide Jimoh Douglas H. Johnson Douglas H. Johnson Martin Johnsson Martin Johnsson Jesse Jorna Jesse Jorna Tommaso Jucker Tommaso Jucker Martin Jung Martin Jung Ineta Kačergytė Ineta Kačergytė Oliver Kaltz Oliver Kaltz Alison Ke Alison Ke Clint D. Kelly Clint D. Kelly Katharine Keogan Katharine Keogan Friedrich Wolfgang Keppeler Friedrich Wolfgang Keppeler Alexander K. Killion Alexander K. Killion Dongmin Kim Dongmin Kim David P. Kochan David P. Kochan Peter Korsten Peter Korsten Shan Kothari Shan Kothari Jonas Kuppler Jonas Kuppler Jillian M. Kusch Jillian M. Kusch Malgorzata Lagisz Malgorzata Lagisz Kristen Marianne Lalla Kristen Marianne Lalla Daniel J. Larkin Daniel J. Larkin Courtney L. Larson Courtney L. Larson Katherine S. Lauck Katherine S. Lauck M. Elise Lauterbur M. Elise Lauterbur Alan Law Alan Law Don-Jean Léandri-Breton Don-Jean Léandri-Breton Jonas J. Lembrechts Jonas J. Lembrechts Kiara L’Herpiniere Kiara L’Herpiniere Eva J. P. Lievens Eva J. P. Lievens Daniela Oliveira de Lima Daniela Oliveira de Lima Shane Lindsay Shane Lindsay Martin Luquet Martin Luquet Ross MacLeod Ross MacLeod Kirsty H. Macphie Kirsty H. Macphie Kit Magellan Kit Magellan Magdalena M. Mair Magdalena M. Mair Lisa E. Malm Lisa E. Malm Stefano Mammola Stefano Mammola Caitlin P. Mandeville Caitlin P. Mandeville Michael Manhart Michael Manhart Laura Milena Manrique-Garzon Laura Milena Manrique-Garzon Elina Mäntylä Elina Mäntylä Philippe Marchand Philippe Marchand Benjamin Michael Marshall Benjamin Michael Marshall Charles A. Martin Charles A. Martin Dominic Andreas Martin Dominic Andreas Martin Jake Mitchell Martin Jake Mitchell Martin April Robin Martinig April Robin Martinig Erin S. McCallum Erin S. McCallum Mark McCauley Mark McCauley Sabrina M. McNew Sabrina M. McNew Scott J. Meiners Scott J. Meiners Thomas Merkling Thomas Merkling Marcus Michelangeli Marcus Michelangeli Maria Moiron Maria Moiron Bruno Moreira Bruno Moreira Jennifer Mortensen Jennifer Mortensen Benjamin Mos Benjamin Mos Taofeek Olatunbosun Muraina Taofeek Olatunbosun Muraina Penelope Wrenn Murphy Penelope Wrenn Murphy Luca Nelli Luca Nelli Petri Niemelä Petri Niemelä Josh Nightingale Josh Nightingale Gustav Nilsonne Gustav Nilsonne Sergio Nolazco Sergio Nolazco Sabine S. Nooten Sabine S. Nooten Jessie Lanterman Novotny Jessie Lanterman Novotny Agnes Birgitta Olin Agnes Birgitta Olin Chris L. Organ Chris L. Organ Kate L. Ostevik Kate L. Ostevik Facundo Xavier Palacio Facundo Xavier Palacio Matthieu Paquet Matthieu Paquet Darren James Parker Darren James Parker David J. Pascall David J. Pascall Valerie J. Pasquarella Valerie J. Pasquarella John Harold Paterson John Harold Paterson Ana Payo-Payo Ana Payo-Payo Karen Marie Pedersen Karen Marie Pedersen Grégoire Perez Grégoire Perez Kayla I. Perry Kayla I. Perry Patrice Pottier Patrice Pottier Michael J. Proulx Michael J. Proulx Raphaël Proulx Raphaël Proulx Jessica L Pruett Jessica L Pruett Veronarindra Ramananjato Veronarindra Ramananjato Finaritra Tolotra Randimbiarison Finaritra Tolotra Randimbiarison Onja H. Razafindratsima Onja H. Razafindratsima Diana J. Rennison Diana J. Rennison Federico Riva Federico Riva Sepand Riyahi Sepand Riyahi Michael James Roast Michael James Roast Felipe Pereira Rocha Felipe Pereira Rocha Dominique G. Roche Dominique G. Roche Cristian Román-Palacios Cristian Román-Palacios Michael S. Rosenberg Michael S. Rosenberg Jessica Ross Jessica Ross Freya E. Rowland Freya E. Rowland Deusdedith Rugemalila Deusdedith Rugemalila Avery L. Russell Avery L. Russell Suvi Ruuskanen Suvi Ruuskanen Patrick Saccone Patrick Saccone Asaf Sadeh Asaf Sadeh Stephen M. Salazar Stephen M. Salazar Kris Sales Kris Sales Pablo Salmón Pablo Salmón Alfredo Sánchez-Tójar Alfredo Sánchez-Tójar Leticia Pereira Santos Leticia Pereira Santos Francesca Santostefano Francesca Santostefano Hayden T. Schilling Hayden T. Schilling Marcus Schmidt Marcus Schmidt Tim Schmoll Tim Schmoll Adam C. Schneider Adam C. Schneider Allie E. Schrock Allie E. Schrock Julia Schroeder Julia Schroeder Nicolas Schtickzelle Nicolas Schtickzelle Nick L. Schultz Nick L. Schultz Drew A. Scott Drew A. Scott Michael Peter Scroggie Michael Peter Scroggie Julie Teresa Shapiro Julie Teresa Shapiro Nitika Sharma Nitika Sharma Caroline L. Shearer Caroline L. Shearer Diego Simón Diego Simón Michael I. Sitvarin Michael I. Sitvarin Fabrício Luiz Skupien Fabrício Luiz Skupien Heather Lea Slinn Heather Lea Slinn Grania Polly Smith Grania Polly Smith Jeremy A. Smith Jeremy A. Smith Rahel Sollmann Rahel Sollmann Kaitlin Stack Whitney Kaitlin Stack Whitney Shannon Michael Still Shannon Michael Still Erica F. Stuber Erica F. Stuber Guy F. Sutton Guy F. Sutton Ben Swallow Ben Swallow Conor Claverie Taff Conor Claverie Taff Elina Takola Elina Takola Andrew J. Tanentzap Andrew J. Tanentzap Rocío Tarjuelo Rocío Tarjuelo Richard J. Telford Richard J. Telford Christopher J. Thawley Christopher J. Thawley Hugo Thierry Hugo Thierry Jacqueline Thomson Jacqueline Thomson Svenja Tidau Svenja Tidau Emily M. Tompkins Emily M. Tompkins Claire Marie Tortorelli Claire Marie Tortorelli Andrew Trlica Andrew Trlica Biz R. Turnell Biz R. Turnell Lara Urban Lara Urban Stijn Van de Vondel Stijn Van de Vondel Jessica Eva Megan van der Wal Jessica Eva Megan van der Wal Jens Van Eeckhoven Jens Van Eeckhoven Francis van Oordt Francis van Oordt K. Michelle Vanderwel K. Michelle Vanderwel Mark C. Vanderwel Mark C. Vanderwel Karen J. Vanderwolf Karen J. Vanderwolf Juliana Vélez Juliana Vélez Diana Carolina Vergara-Florez Diana Carolina Vergara-Florez Brian C. Verrelli Brian C. Verrelli Marcus Vinícius Vieira Marcus Vinícius Vieira Nora Villamil Nora Villamil Valerio Vitali Valerio Vitali Julien Vollering Julien Vollering Jeffrey Walker Jeffrey Walker Xanthe J. Walker Xanthe J. Walker Jonathan A. Walter Jonathan A. Walter Pawel Waryszak Pawel Waryszak Ryan J. Weaver Ryan J. Weaver Ronja E. M. Wedegärtner Ronja E. M. Wedegärtner Daniel L. Weller Daniel L. Weller Shannon Whelan Shannon Whelan Rachel Louise White Rachel Louise White David William Wolfson David William Wolfson Andrew Wood Andrew Wood Scott W. Yanco Scott W. Yanco Jian D. L. Yen Jian D. L. Yen Casey Youngflesh Casey Youngflesh Giacomo Zilio Giacomo Zilio Cédric Zimmer Cédric Zimmer Gregory Mark Zimmerman Gregory Mark Zimmerman Rachel A. Zitomer Rachel A. Zitomer Timothy H. Parker Timothy H. Parker Hannah S. Fraser Hannah S. Fraser Springer 2025 Although variation in effect sizes and predicted values among studies of similar phenomena is inevitable, such variation far exceeds what might be produced by sampling error alone. One possible explanation for variation among results is differences among researchers in the decisions they make regarding statistical analyses. A growing array of studies has explored this analytical variability in different fields and has found substantial variability among results despite analysts having the same data and research question. Many of these studies have been in the social sciences, but one small “many analyst” study found similar variability in ecology. We expanded the scope of this prior work by implementing a large-scale empirical exploration of the variation in effect sizes and model predictions generated by the analytical decisions of different researchers in ecology and evolutionary biology. We used two unpublished datasets, one from evolutionary ecology (blue tit, Cyanistes caeruleus, to compare sibling number and nestling growth) and one from conservation ecology (Eucalyptus, to compare grass cover and tree seedling recruitment). The project leaders recruited 174 analyst teams, comprising 246 analysts, to investigate the answers to prespecified research questions. Analyses conducted by these teams yielded 141 usable effects (compatible with our meta-analyses and with all necessary information provided) for the blue tit dataset, and 85 usable effects for the Eucalyptus dataset. We found substantial heterogeneity among results for both datasets, although the patterns of variation differed between them. For the blue tit analyses, the average effect was convincingly negative, with less growth for nestlings living with more siblings, but there was near continuous variation in effect size from large negative effects to effects near zero, and even effects crossing the traditional threshold of statistical significance in the opposite direction. In contrast, the average relationship between grass cover and Eucalyptus seedling number was only slightly negative and not convincingly different from zero, and most effects ranged from weakly negative to weakly positive, with about a third of effects crossing the traditional threshold of significance in one direction or the other. However, there were also several striking outliers in the Eucalyptus dataset, with effects far from zero. For both datasets, we found substantial variation in the variable selection and random effects structures among analyses, as well as in the ratings of the analytical methods by peer reviewers, but we found no strong relationship between any of these and deviation from the meta-analytic mean. In other words, analyses with results that were far from the mean were no more or less likely to have dissimilar variable sets, use random effects in their models, or receive poor peer reviews than those analyses that found results that were close to the mean. The existence of substantial variability among analysis outcomes raises important questions about how ecologists and evolutionary biologists should interpret published results, and how they should conduct analyses in the future. PDFDocument eng 2026-01-07T10:00:49.940251Z 2025 Analytical Heterogeneity Metascience Replication Crisis Reproducibility Many-analyst 2739794 b application/pdf http://creativecommons.org/licenses/by-nc-nd/4.0/