Production-Grade AI Systems with Fred Roma

What It Covers

Fred Roma, SVP of Product and Engineering at MongoDB, discusses building production-grade AI applications with Kevin Ball. They explore the data layer challenges in AI development, including vector search, embedding models, re-ranking, schema evolution, and MongoDB's Voyage AI acquisition for accurate embeddings and cost-effective information retrieval.

Key Questions Answered

• •Simplified AI Stack Integration: Production AI applications require LLMs, vector search, embedding models, re-rankers, and caching layers. MongoDB consolidates these into a unified data platform, eliminating the need to stitch together separate solutions, manage multiple identity providers, or create complex data transfer pipelines between disconnected systems for operational data and AI retrieval.
• •Hybrid Search Strategy: Combining keyword search with semantic vector search delivers optimal accuracy. For example, searching Nike red shoes should match the exact Nike keyword while allowing semantic flexibility for red shoes to include burgundy sneakers. MongoDB's aggregation pipeline enables developers to configure weighted combinations and custom ranking algorithms within a single query operation.
• •Context-Aware Embeddings: Voyage AI's context models preserve document-level information when creating chunk embeddings, enabling the system to distinguish between current documentation and outdated support tickets. This prevents AI applications from surfacing technically accurate but contextually irrelevant information, reducing hallucinations by understanding temporal and structural context beyond isolated text fragments.
• •Multi-Modal Document Processing: Voyage's multimodal embedding models accept PDFs with mixed text and images directly, eliminating preprocessing pipelines that extract and separately process different content types. This approach preserves spatial relationships and context that get lost when breaking documents apart, improving accuracy while dramatically simplifying developer workflows and reducing infrastructure complexity.
• •Cost-Performance Trade-offs: Embedding sizes directly impact storage and query costs. Voyage models offer variable embedding lengths and formats including binary representations, allowing developers to optimize for speed in e-commerce applications or accuracy in legal and financial use cases. Re-rankers add compute-intensive precision for top results after fast initial retrieval identifies candidate documents.