How to train your data

What It Covers

Atlantic staff writer Alex Reisner joins The Vergecast to examine AI training data — what it is, where it comes from, and why companies guard it so closely. The conversation covers Common Crawl, YouTube as a data source, music datasets, synthetic data limitations, and the emerging paid creator economy for AI training.

Key Questions Answered

• •Training data defines model capability: A model's output is determined almost entirely by what it trains on — not just its architecture. A music model trained on 1950s jazz produces 1950s jazz; one trained on hip-hop produces hip-hop. Reisner argues the most accurate name for any AI model is a description of its training dataset, not a product name like GPT or Claude.
• •Competitive secrecy masks legal exposure: AI companies cite competitive advantage as the reason training data stays secret, but a second motive is equally significant — much of the data was scraped without creator consent. Authors, musicians, and video creators frequently discover their work was used only after the fact, and companies prefer to avoid that conversation entirely rather than address it proactively.
• •Data laundering via nonprofits and universities: AI companies routinely route data acquisition through academic institutions and nonprofits — such as Common Crawl and the European organization LION, which holds 12 million songs scraped from YouTube — allowing companies to claim distance from the scraping while funding those organizations directly and benefiting from the resulting datasets.
• •YouTube is the dominant scraping target due to weak technical barriers: YouTube appears repeatedly across AI training datasets because download tools work reliably and easily, and because digital rights management protections are absent compared to platforms like Spotify. YouTube states scraping violates its terms of service but has taken no effective technical or legal action to stop it over multiple years.
• •Synthetic data does not replace human-generated content: Research on "model collapse" demonstrates that training AI models on their own outputs causes rapid quality degradation. Reisner states AI companies deliberately use vague language when claiming synthetic data works. The next actual data frontier is paying human creators directly — one company has already paid creators over $10 million to produce content specifically for AI training purposes.