Don't hate the replicator, hate the game
Episode
36 min
Read time
2 min
Topics
Marketing, Psychology & Behavior, Science & Discovery
AI-Generated Summary
Key Takeaways
- ✓P-Hacking Detection: Researchers manipulate datasets by running hundreds of analytical variations until results cross the 5% statistical significance threshold, a practice called p-hacking. Abel and colleagues identified this by scraping published journals and finding an abnormal clustering of results just above that threshold, published in their 2016 paper "Star Wars: The Empirics Strike Back."
- ✓Enforcement Probability Over Severity: Academic behavior changes not based on punishment severity but on the perceived likelihood of getting caught. Abel's strategy deliberately creates the impression that any paper could be audited at any time, mirroring how IRS audit rates influence taxpayer compliance even when actual enforcement remains relatively rare.
- ✓Replication Games Structure: Teams of three to five researchers receive a paper's full data and code package, then spend seven hours completing two phases: first, confirming the original code reproduces identical numerical results; second, running robustness checks by altering model specifications, excluded variables, or dataset subsets to test whether conclusions survive reasonable analytical alternatives.
- ✓Scale Through Crowdsourcing: Abel built a replication infrastructure essentially at zero cost by offering co-authorship credit on a published meta-paper as compensation. Recruiting researchers through social media and academic networks, he scaled from an unplanned 80-person Oslo event in October 2022 to over 50 games replicating approximately 300 papers within roughly two years.
- ✓Institutional Legitimacy as a Tool: To obtain data and code from reluctant researchers, Abel created the Institute for Replication with a professional website and recruited prominent economists to its board. This manufactured credibility generated email responses that informal individual requests had failed to produce, demonstrating that perceived institutional authority unlocks cooperation in academic settings.
What It Covers
University of Ottawa economist Abel Brodeur created the Replication Games, a hackathon-style event where teams of social scientists spend seven hours checking whether recently published academic papers hold up under scrutiny, addressing a two-decade replication crisis affecting psychology, medicine, and economics research worldwide.
Key Questions Answered
- •P-Hacking Detection: Researchers manipulate datasets by running hundreds of analytical variations until results cross the 5% statistical significance threshold, a practice called p-hacking. Abel and colleagues identified this by scraping published journals and finding an abnormal clustering of results just above that threshold, published in their 2016 paper "Star Wars: The Empirics Strike Back."
- •Enforcement Probability Over Severity: Academic behavior changes not based on punishment severity but on the perceived likelihood of getting caught. Abel's strategy deliberately creates the impression that any paper could be audited at any time, mirroring how IRS audit rates influence taxpayer compliance even when actual enforcement remains relatively rare.
- •Replication Games Structure: Teams of three to five researchers receive a paper's full data and code package, then spend seven hours completing two phases: first, confirming the original code reproduces identical numerical results; second, running robustness checks by altering model specifications, excluded variables, or dataset subsets to test whether conclusions survive reasonable analytical alternatives.
- •Scale Through Crowdsourcing: Abel built a replication infrastructure essentially at zero cost by offering co-authorship credit on a published meta-paper as compensation. Recruiting researchers through social media and academic networks, he scaled from an unplanned 80-person Oslo event in October 2022 to over 50 games replicating approximately 300 papers within roughly two years.
- •Institutional Legitimacy as a Tool: To obtain data and code from reluctant researchers, Abel created the Institute for Replication with a professional website and recruited prominent economists to its board. This manufactured credibility generated email responses that informal individual requests had failed to produce, demonstrating that perceived institutional authority unlocks cooperation in academic settings.
Notable Moment
During the very first replication game in Oslo, the opening paper contained a dataset where 75% of subjects shared identical age, gender, and location — a massive copy-paste duplication error. Since the paper studied inequality, having nearly identical subjects statistically eliminated the variation driving its core findings entirely.
Episode Transcript
This message comes from Insperity. Excellence takes drive, work, perseverance. Tiger Woods brings it to the course. Insperity brings it to your business. Want to be the best? Work with the best. Insperity, how UHR matters. Learn more at insperity.com/tiger. This is Planet Money from NPR. Alexei Horowitz Gazzi. Mary Childs. Yes. You and I took a little trip up to scenic Montreal, one of the jewels of French Canada, for a little Planet Money mission. Yes. We did. And even though it's a little bit sad that that mission did not entail joining the maple harvest or, you know, like, infiltrating a poutine cartel Next time. Dare I say next time, it did have much bigger implications for anybody and everybody whose life is impacted by science, which I think is basically all of us. I think that's right. Yeah. We were there to meet a guy named Abel Brodure. Abel is this very energetic economics professor in his late thirties at the University of Ottawa, and we found him bounding around the halls of this modernist school building in Downtown Montreal. He was getting ready to host an event he's become sort of famous for, something called the replication games. It's getting exciting now. How are you feeling? I'm, feeling good. It it's the beginning of the event, so this is the moment I'm full of energy and, full of enthusiasm. In seven hours from now, it's gonna be a different conversation. Abel is gonna be tired in seven hours because at a replication game, he is running around between 16 teams of three to five people in a kind of hackathon. People will work all day to replicate recently published social science papers to reproduce the results and see if the findings hold up. Because ever since technology has made it easy to crunch data, we've been able to go back and check old research. And turns out, it wasn't great. Rerunning an old study today a lot of the time does not yield the same result. The research no longer proves its conclusion. And the same thing often happens when we reconduct whole experiments. Altogether, these problems have become known as the replication crisis. A lot of people across academia have been trying to fix this so we can trust research, so we can actually know what we know. And this event, the replication games, is part of the Bell's attempt to help solve this crisis. The idea is to change norms through monitoring And just giving a small percentage, small chance that we will monitor Yeah. Can massively change the behavior of everyone. You know, change the way they behave, change the way they code, change the way they do research. So that's the goal. After a few minutes, we head into a big lecture hall where Abel takes center stage. Alright, folks. We're gonna get started. Welcome to the rotation games. Thanks for being here in Montreal with us. Let's get started. Today, we …
Get the full transcript (6,770 words) + summary by email — free
One-time email with the complete transcript and AI summary of this episode. No account needed.
One email, no spam. We’ll also show you what SignalCast does.
You just read a 3-minute summary of a 33-minute episode.
Get Planet Money summarized like this every Monday — plus up to 2 more podcasts, free.
Pick Your Podcasts — FreeKeep Reading
More from Planet Money
Cost-cutting, quiet guilt and the inflation generation
Sep 9 · 24 min
Eye on AI
"According to NASA's Definition of Life, I'm Not Alive" - Why Nobody Can Define Life | Dr. Kate Adamala
Jul 29
More from Planet Money
Trump drinks Venezuela’s milkshake
Sep 5 · 25 min
The Indicator
What Iran teaches us about why wars start
Jun 30
Books, tools, and gear mentioned in this episode
SignalCast may earn commission on purchases via these links. As an Amazon Associate, SignalCast earns from qualifying purchases.
Books
“Abel and colleagues identified this by scraping published journals and finding an abnormal clustering of results just above that threshold, published in their 2016 paper "Star Wars: The Empirics Strike Back."”
More from Planet Money
We summarize every new episode. Want them in your inbox?
Cost-cutting, quiet guilt and the inflation generation
Trump drinks Venezuela’s milkshake
Love in the time of Palantir
Dolly Parton’s “9 to 5,” the true story behind the anthem (bonus)
Branded fruit: How produce got “Honeycrisp-ified"
Similar Episodes
Related episodes from other podcasts
Eye on AI
Jul 29
"According to NASA's Definition of Life, I'm Not Alive" - Why Nobody Can Define Life | Dr. Kate Adamala
The Indicator
Jun 30
What Iran teaches us about why wars start
The Indicator
Jun 18
Are we in a new era of permanently higher prices?
Accidental Tech Podcast
May 21
692: A Thinking Hitch
Freakonomics Radio
May 6
Was Adam Smith Really a Right-Winger? (Update)
Explore Related Topics
This podcast is featured in Best Finance Podcasts (2026) — ranked and reviewed with AI summaries.
You're clearly into Planet Money.
Every Monday, we deliver AI summaries of the latest episodes from Planet Money and 192+ other podcasts. Free for one show.
Start My Monday DigestNo credit card · Unsubscribe anytime