Skip to main content
Lenny's Podcast

Why PMs are suddenly valuable again | Tulsee Doshi (Head of Product, Gemini)

58 min episode · 2 min read
·
Tulsee Doshi

Episode

58 min

Read time

2 min

Topics

Career Growth, Startups, Fundraising & VC

AI-Generated Summary

Key Takeaways

  • ✓Model-as-UX: The personality and behavior of an AI model constitutes its own user experience layer. PMs should develop a formal behavioral spec, translate it into rubrics, then work with researchers to embed those rubrics into reward modeling and training data — not just surface-level prompt tuning after the fact.
  • ✓60-20-20 Build Framework: When building on AI models, structure your product so roughly 60% of features work reliably today, 20% are functional but imperfect with near-term model improvements visible, and 20% represent aspirational capabilities where early signal exists. This balances shippable value with positioning for model capability gains.
  • ✓Eval-Building Starting Point: PMs new to writing evals should start with as few as five to twenty prompts targeting a specific capability, then manually rate outputs themselves. This small-scale approach builds intuition for what "good" looks like before scaling to automated or large-scale benchmarks.
  • ✓PM Hiring Priorities for AI Roles: Hiring managers at frontier labs now prioritize three PM traits: daily hands-on model usage with formed opinions on quality, ability to build and interpret evals without engineering support, and adaptive roadmap thinking — treating two-month-old plans as drafts requiring constant re-interrogation given the pace of change.
  • ✓AI Slop Management: To prevent teams from drowning in AI-generated documents, separate what AI should produce from what humans should produce. AI handles data aggregation — metrics, Reddit sentiment, dashboard summaries. Humans provide the strategic insight layer. Building structured internal skills and tools to automate data retrieval creates space for higher-order thinking.

What It Covers

Tulsee Doshi, Head of Product for Google's Gemini and frontier models, covers how AI model development differs from traditional product management, what skills PMs need now, Google's competitive strategy across the AI stack, and frameworks for building products on rapidly evolving models.

Key Questions Answered

  • •Model-as-UX: The personality and behavior of an AI model constitutes its own user experience layer. PMs should develop a formal behavioral spec, translate it into rubrics, then work with researchers to embed those rubrics into reward modeling and training data — not just surface-level prompt tuning after the fact.
  • •60-20-20 Build Framework: When building on AI models, structure your product so roughly 60% of features work reliably today, 20% are functional but imperfect with near-term model improvements visible, and 20% represent aspirational capabilities where early signal exists. This balances shippable value with positioning for model capability gains.
  • •Eval-Building Starting Point: PMs new to writing evals should start with as few as five to twenty prompts targeting a specific capability, then manually rate outputs themselves. This small-scale approach builds intuition for what "good" looks like before scaling to automated or large-scale benchmarks.
  • •PM Hiring Priorities for AI Roles: Hiring managers at frontier labs now prioritize three PM traits: daily hands-on model usage with formed opinions on quality, ability to build and interpret evals without engineering support, and adaptive roadmap thinking — treating two-month-old plans as drafts requiring constant re-interrogation given the pace of change.
  • •AI Slop Management: To prevent teams from drowning in AI-generated documents, separate what AI should produce from what humans should produce. AI handles data aggregation — metrics, Reddit sentiment, dashboard summaries. Humans provide the strategic insight layer. Building structured internal skills and tools to automate data retrieval creates space for higher-order thinking.

Notable Moment

Doshi revealed that Google's skin tone evaluation system was rebuilt after discovering the widely used Fitzpatrick scale — originally designed for healthcare, not representation — skewed heavily toward lighter tones. Partnering with a Harvard professor, they replaced it with a ten-tone scale, directly improving Pixel camera and image generation outputs.

Know someone who'd find this useful?

Episode Transcript

SPEAKER_01: The model isn't predictable. SPEAKER_01: It's kind of like Snape's potions class. SPEAKER_01: You can put all the ingredients in, SPEAKER_01: but you also have to expect that there will be a little bit of magic, SPEAKER_01: and that magic is going to be surprising. SPEAKER_00: As you're hiring PMs these days, is there anything you look for more and more? SPEAKER_01: You're looking for PMs who understand the vibes of our models or other models. SPEAKER_01: I want PMs who have an opinion, who are playing with these models every day. SPEAKER_01: You ideally want to stress test the model on things that you are excited about. SPEAKER_00: A lot of people feel that Google lags behind the top two labs right now. SPEAKER_01: I actually think we do want to win the frontier conversation. SPEAKER_01: I don't think we should say that Google is not participating in the frontier. SPEAKER_00: You have this framework around how far ahead to build. SPEAKER_00: You call it a 60-20-20 rule. SPEAKER_01: Traditional startup wisdom. SPEAKER_01: You're just like, build an MVP that works. SPEAKER_01: But I think you want to actually build features that aren't necessarily always working today. SPEAKER_00: It feels like every AI assistant launching these days is taking all this juicy SPEAKER_00: Google data, my Gmail, Google Calendar, my Docs. SPEAKER_00: I'm curious just why Google hasn't launched something awesome yet. SPEAKER_01: The team is working on some amazing things, SPEAKER_01: but I think you are right that there is more work we need to do here on this front. SPEAKER_00: Tulsi, thank you so much for being here. SPEAKER_00: Welcome to the podcast. SPEAKER_01: Thanks. I'm excited to be here. SPEAKER_00: You are head of product for Google's Frontier models, which includes Gemini, which people know. SPEAKER_00: But there's also a bunch of other models that people may not be familiar with. SPEAKER_00: Share the other models that are under your umbrella. SPEAKER_01: Yeah, so we have a set of models we kind of call our GenMedia models. SPEAKER_01: These are our models that focus on video editing and generation, SPEAKER_01: image editing and generation, if you've used Nano Banana, Lyria, our music model. SPEAKER_01: We also have Gemma, our suite of open models. SPEAKER_01: And then one set of models that actually falls under our Gemini umbrella, SPEAKER_01: but I feel like we don't talk about as much as our audio models. SPEAKER_01: So dialogue, audio understanding, text to speech, translation. SPEAKER_01: All a part of the portfolio of how we build FredShare models. SPEAKER_00: Okay, so I want to spend a little time on models. SPEAKER_00: Everyone sees models launch. SPEAKER_00: There's like a model launch every week, it feels like. SPEAKER_00: You have a really unique lens in being able to see these models from research SPEAKER_00: to pre-training, post-training launch. SPEAKER_00: What's something that you think people maybe misunderstand SPEAKER_00: about how models are created or something that may surprise people SPEAKER_00: …

Get the full transcript (12,641 words) + summary by email — free

One-time email with the complete transcript and AI summary of this episode. No account needed.

One email, no spam. We’ll also show you what SignalCast does.

Browse all Lenny's Podcast transcripts →

You just read a 3-minute summary of a 55-minute episode.

Get Lenny's Podcast summarized like this every Monday — plus up to 2 more podcasts, free.

Pick Your Podcasts — Free
Part of this week's recap (Nov 2 – Nov 8)

Keep Reading

Books, tools, and gear mentioned in this episode

SignalCast may earn commission on purchases via these links. As an Amazon Associate, SignalCast earns from qualifying purchases.

Gear

  • by Google

    “Partnering with a Harvard professor, they replaced it with a ten-tone scale, directly improving Pixel camera and image generation outputs.”

More from Lenny's Podcast

We summarize every new episode. Want them in your inbox?

Similar Episodes

Related episodes from other podcasts

Explore Related Topics

This podcast is featured in Best Product Management Podcasts (2026) — ranked and reviewed with AI summaries.

Read this week's Startups & Product Podcast Insights — cross-podcast analysis updated weekly.

You're clearly into Lenny's Podcast.

Every Monday, we deliver AI summaries of the latest episodes from Lenny's Podcast and 192+ other podcasts. Free for one show.

Start My Monday Digest

No credit card · Unsubscribe anytime