Phelim Brady

AI Summary

→ WHAT IT COVERS Phelim Brady, cofounder and CEO of Prolific, explains how his human data platform connects verified global participants with AI labs and researchers for post-training evaluation. With roughly 2 million registered participants and a 50/50 split between academic research and AI work, Prolific addresses the growing demand for rigorous human judgment in model evaluation. → KEY INSIGHTS - **Model Evaluation Over Benchmarks:** Academic benchmarks like math Olympiad scores have become unreliable because frontier models train directly against them, saturating their usefulness. Enterprises and labs now need real-world human evaluation instead. Prolific runs structured head-to-head model comparisons using demographically controlled participant pools to generate trustworthy, context-specific performance rankings across different use cases. - **Demographic Representation Changes Model Rankings:** Prolific's "Humane" benchmark replicates chatbot arena-style model comparisons but adds census-matched sampling controlling for age, ethnicity, and political affiliation. Results show model preference rankings shift measurably depending on audience demographics, meaning enterprises should evaluate models against their specific target user population rather than relying on aggregate public leaderboards. - **Agentic Fraud Is an Emerging Threat to Human Data:** AI agents can now replicate human behavior accurately enough to infiltrate online data collection platforms. Prolific counters this through layered identity verification, periodic re-verification, behavioral analysis during task completion, and KYC-style checks — making participant authenticity a core infrastructure investment rather than a one-time onboarding step. - **Expert Participant Segmentation Across Three Tiers:** Prolific structures its AI evaluation workforce into three roughly equal segments: general consumer samples for representativeness testing, trained taskers qualified for standard AI evaluation workflows, and domain experts whose prior professional credentials are the primary selection criterion. Enterprises building regulated-domain applications in healthcare, finance, or law should target that third expert tier specifically. - **Human Judgment Remains Necessary at the Capability Frontier:** Automated AI judges cannot evaluate capabilities that do not yet exist in current models — assessing a gap requires a benchmark that exceeds the model being tested. Human evaluators remain essential wherever tasks involve ambiguity, subjective preference, or low-confidence model outputs requiring escalation, meaning evaluation budgets should preserve human review for edge cases and novel capability assessments. → NOTABLE MOMENT Brady describes a UK AI Security Institute study where participants held real political conversations with roughly 20 different AI models, each instructed to use specific rhetorical strategies. Researchers measured opinion change before and after, revealing measurable differences in how persuasive various models were with real people. 💼 SPONSORS None detected 🏷️ AI Evaluation, Human-in-the-Loop, Foundation Models, Data Labeling, AI Safety