#325 Phelim Brady: Why AI's Future Depends on Human Judgement

What It Covers

Phelim Brady, cofounder and CEO of Prolific, explains how his human data platform connects verified global participants with AI labs and researchers for post-training evaluation. With roughly 2 million registered participants and a 50/50 split between academic research and AI work, Prolific addresses the growing demand for rigorous human judgment in model evaluation.

Key Questions Answered

• •Model Evaluation Over Benchmarks: Academic benchmarks like math Olympiad scores have become unreliable because frontier models train directly against them, saturating their usefulness. Enterprises and labs now need real-world human evaluation instead. Prolific runs structured head-to-head model comparisons using demographically controlled participant pools to generate trustworthy, context-specific performance rankings across different use cases.
• •Demographic Representation Changes Model Rankings: Prolific's "Humane" benchmark replicates chatbot arena-style model comparisons but adds census-matched sampling controlling for age, ethnicity, and political affiliation. Results show model preference rankings shift measurably depending on audience demographics, meaning enterprises should evaluate models against their specific target user population rather than relying on aggregate public leaderboards.
• •Agentic Fraud Is an Emerging Threat to Human Data: AI agents can now replicate human behavior accurately enough to infiltrate online data collection platforms. Prolific counters this through layered identity verification, periodic re-verification, behavioral analysis during task completion, and KYC-style checks — making participant authenticity a core infrastructure investment rather than a one-time onboarding step.
• •Expert Participant Segmentation Across Three Tiers: Prolific structures its AI evaluation workforce into three roughly equal segments: general consumer samples for representativeness testing, trained taskers qualified for standard AI evaluation workflows, and domain experts whose prior professional credentials are the primary selection criterion. Enterprises building regulated-domain applications in healthcare, finance, or law should target that third expert tier specifically.
• •Human Judgment Remains Necessary at the Capability Frontier: Automated AI judges cannot evaluate capabilities that do not yet exist in current models — assessing a gap requires a benchmark that exceeds the model being tested. Human evaluators remain essential wherever tasks involve ambiguity, subjective preference, or low-confidence model outputs requiring escalation, meaning evaluation budgets should preserve human review for edge cases and novel capability assessments.