Skip to main content
Machine Learning Street Talk

Why Humans Are Still Powering AI [Sponsored]

24 min episode · 2 min read
·
Jack Bevan

Episode

24 min

Read time

2 min

Topics

Career Growth, Relationships, Investing

AI-Generated Summary

Key Takeaways

  • Human data routing: Prolific uses a three-layer quality system to match humans to AI tasks: ID verification at onboarding, researcher feedback loops that re-rank participants by task performance, and network analysis that identifies clusters of participants gaming the system. High-quality data comes from properly incentivized participants, not lowest-cost labor.
  • Prisoner's dilemma incentive design: Single-shot relationships incentivize cheating among data contributors. Prolific counters this with repeated multi-touch engagements, direct peer-to-peer messaging between researchers and participants, and transparent feedback on work impact — converting a transactional dynamic into a long-term relationship that behaviorally discourages gaming.
  • Segment-by-segment marketplace scaling: Prolific treats audience segments like Uber treats cities — each new expert demographic requires bootstrapping from an atomic network to a scaled one. A US general population may be fully scaled while a niche medical specialty still needs targeted recruitment, requiring continuous marginal-user growth strategies per segment.
  • Jevons paradox in human data: As synthetic data and LLM-as-judge tools reduce the per-unit cost of human data collection, total demand for human expertise rises to compensate. Even if human data's proportional share of AI training shrinks, absolute volume and strategic value increase due to explosive overall model demand.
  • Agent-human collaboration as next frontier: Future AI workflows will embed human expert review as a discrete step inside agentic pipelines — similar to deep research agents pausing to route a verification task to a domain specialist via a platform like Prolific. Choosing correct optimization targets matters more than optimizing efficiently toward wrong ones.

What It Covers

Phelim Bradley, CEO of Prolific, a human data infrastructure platform, explains why frontier AI models depend fundamentally on verified human expertise for training, evaluation, and post-training feedback — and why this dependency grows larger as AI scales, not smaller, despite widespread assumptions about full automation.

Key Questions Answered

  • Human data routing: Prolific uses a three-layer quality system to match humans to AI tasks: ID verification at onboarding, researcher feedback loops that re-rank participants by task performance, and network analysis that identifies clusters of participants gaming the system. High-quality data comes from properly incentivized participants, not lowest-cost labor.
  • Prisoner's dilemma incentive design: Single-shot relationships incentivize cheating among data contributors. Prolific counters this with repeated multi-touch engagements, direct peer-to-peer messaging between researchers and participants, and transparent feedback on work impact — converting a transactional dynamic into a long-term relationship that behaviorally discourages gaming.
  • Segment-by-segment marketplace scaling: Prolific treats audience segments like Uber treats cities — each new expert demographic requires bootstrapping from an atomic network to a scaled one. A US general population may be fully scaled while a niche medical specialty still needs targeted recruitment, requiring continuous marginal-user growth strategies per segment.
  • Jevons paradox in human data: As synthetic data and LLM-as-judge tools reduce the per-unit cost of human data collection, total demand for human expertise rises to compensate. Even if human data's proportional share of AI training shrinks, absolute volume and strategic value increase due to explosive overall model demand.
  • Agent-human collaboration as next frontier: Future AI workflows will embed human expert review as a discrete step inside agentic pipelines — similar to deep research agents pausing to route a verification task to a domain specialist via a platform like Prolific. Choosing correct optimization targets matters more than optimizing efficiently toward wrong ones.

Notable Moment

Bradley argues that AI is accelerating demand for human experts rather than eliminating them. As more people use AI tools to build and create, they hit knowledge ceilings and need genuine specialists — pulling domain experts into higher-value, more frequent work than existed before AI proliferation.

Know someone who'd find this useful?

Episode Transcript

There's a dirty secret, isn't there? In in Silicon Valley and in the tech world that there is and I don't know if people realize the extent to this. There is an absolutely huge importance on human data and human expertise and human understanding, And that is completely glossed over. Yeah. I mean, fundamentally, artificial intelligence is founded in human intelligence. And I think in the the stack of data, algorithms, and compute, I think the the human data element is often the the least spoken about, maybe the least least glamorous. People want to imagine that there's a a simple kind of input output equation, but ultimately, there's a messy layer in the in the stack of human beings who are providing their data to, you know, either, label data, provide all HF post training data, and then ultimately the evaluation and the assessment of the the model model performance all ultimately is kind of has an element of of human human data in it. I I can't tell the difference between a doctor and someone who pretends to be a doctor. The only way that you can actually tell the difference is if you have this deep abstract understanding and you know that they're breaking the rules. It's increasingly clear that Frontier models as a as a platform are going to be centralized and controlled by a relatively small number of of players, which at the moment is almost exclusively these US tech companies. So I think there is a bit of a a wake up call. What the future will hold is basically a marketplace of intelligence. So in the past, we had a marketplace of, oil, for example, or electricity. Intelligence is gonna be the new traded thing. I'm the cofounder and CEO of Prolific, and, Prolific is a human data, infrastructure company. So we make it easy for people developing frontier AI models, and running research to get access to trustworthy, high quality participants for high quality online data collection. Prior to the chat GBT moment, the the primary modes of data collection, the people were fairly fungible, right, so you're optimizing for cost and scale, maybe offshore lower cost labor, which I think created this dynamic of human data for AI being a bit of a a dirty secret. Let's talk about your core technology. Now you you've solved an interesting problem that I've tried to solve in the past. So I I I started a company called Merge, and it was a code review platform. Mhmm. And it was exactly the same thing. I realized that code review has to be done by humans, and people talk about automation and software engineering life cycle. It's mostly bullshit. It's it's actually orchestration. You need humans involved in every single step, most importantly, code review. We had a skill matrix, and we could learn their skill, and and we could you know, a a pull request would come in from one of our customers, and …

Get the full transcript (4,482 words) + summary by email — free

One-time email with the complete transcript and AI summary of this episode. No account needed.

One email, no spam. We’ll also show you what SignalCast does.

Browse all Machine Learning Street Talk transcripts →

You just read a 3-minute summary of a 21-minute episode.

Get Machine Learning Street Talk summarized like this every Monday — plus up to 2 more podcasts, free.

Pick Your Podcasts — Free

Keep Reading

Books, tools, and gear mentioned in this episode

SignalCast may earn commission on purchases via these links.

company

  • Phelim Bradley, CEO of Prolific, a human data infrastructure platform, explains why frontier AI models depend fundamentally on verified human expertise for training, evaluation, and post-training feedback.

More from Machine Learning Street Talk

We summarize every new episode. Want them in your inbox?

Similar Episodes

Related episodes from other podcasts

Explore Related Topics

This podcast is featured in Best AI Podcasts (2026) — ranked and reviewed with AI summaries.

Read this week's Investing & Markets Podcast Insights — cross-podcast analysis updated weekly.

You're clearly into Machine Learning Street Talk.

Every Monday, we deliver AI summaries of the latest episodes from Machine Learning Street Talk and 192+ other podcasts. Free for one show.

Start My Monday Digest

No credit card · Unsubscribe anytime