Building AI Factories: How Red Hat and NVIDIA Turn Enterprise Data Into Intelligence - Ep. 293

What It Covers

Red Hat CTO Chris Wright and NVIDIA VP Justin Boitano outline how enterprises build AI factories — five-layer technology stacks converting raw data into business intelligence — covering infrastructure sizing, agentic deployment, security guardrails, and a practical first-ninety-day roadmap toward full-scale AI transformation.

Key Questions Answered

• •Five-Layer AI Factory Stack: Structure enterprise AI investment across five distinct layers: data center power and cooling, rack-scale GPU infrastructure, software orchestration (Kubernetes/Linux), model delivery, and agentic applications. Enterprises that skip layers or treat these independently create fragmented environments with high failure rates. Addressing all five layers systematically before scaling reduces costly rework.
• •Hybrid Model Architecture for Cost Reduction: Use frontier models only for the planning stage of agentic workflows, while open models handle search and summarization tasks on-premises. NVIDIA's newer blueprints demonstrate a 30x cost reduction using this split architecture, making enterprise-scale agentic search economically viable for knowledge workers across large organizations.
• •Dev-to-Prod Separation as Non-Negotiable: Separate AI development environments from production data until agents pass functional verification, QA, and penetration testing. Agents must inherit role-based access controls matching the requesting user's existing permissions before promotion to production. This mirrors decades of software engineering best practice and prevents unauthorized data exposure at scale.
• •Evals as a Core Infrastructure Component: Build evaluation frameworks into the AI factory stack from day one, not as an afterthought. Evals measure output quality against defined business outcomes and guide iterative refinement of prompting, data sourcing, and problem scoping. Without continuous evals, enterprises cannot distinguish genuine productivity gains from superficially impressive but low-value demonstrations.
• •Treat Agents as Digital Employees with Least-Privilege Access: As agents operate more autonomously, scope their system permissions the same way contractors receive access — starting with minimum necessary privileges and requiring explicit approval to expand. Connecting agents to business systems often reveals existing user permissions are already over-scoped, making a security team audit a recommended first step.