Don't hate the replicator, hate the game

What It Covers

University of Ottawa economist Abel Brodeur created the Replication Games, a hackathon-style event where teams of social scientists spend seven hours checking whether recently published academic papers hold up under scrutiny, addressing a two-decade replication crisis affecting psychology, medicine, and economics research worldwide.

Key Questions Answered

• •P-Hacking Detection: Researchers manipulate datasets by running hundreds of analytical variations until results cross the 5% statistical significance threshold, a practice called p-hacking. Abel and colleagues identified this by scraping published journals and finding an abnormal clustering of results just above that threshold, published in their 2016 paper "Star Wars: The Empirics Strike Back."
• •Enforcement Probability Over Severity: Academic behavior changes not based on punishment severity but on the perceived likelihood of getting caught. Abel's strategy deliberately creates the impression that any paper could be audited at any time, mirroring how IRS audit rates influence taxpayer compliance even when actual enforcement remains relatively rare.
• •Replication Games Structure: Teams of three to five researchers receive a paper's full data and code package, then spend seven hours completing two phases: first, confirming the original code reproduces identical numerical results; second, running robustness checks by altering model specifications, excluded variables, or dataset subsets to test whether conclusions survive reasonable analytical alternatives.
• •Scale Through Crowdsourcing: Abel built a replication infrastructure essentially at zero cost by offering co-authorship credit on a published meta-paper as compensation. Recruiting researchers through social media and academic networks, he scaled from an unplanned 80-person Oslo event in October 2022 to over 50 games replicating approximately 300 papers within roughly two years.
• •Institutional Legitimacy as a Tool: To obtain data and code from reluctant researchers, Abel created the Institute for Replication with a professional website and recruited prominent economists to its board. This manufactured credibility generated email responses that informal individual requests had failed to produce, demonstrating that perceived institutional authority unlocks cooperation in academic settings.