[State of Post-Training] From GPT-4.1 to 5.1: RLVR, Agent & Token Efficiency — Josh McGrath, OpenAI

What It Covers

Josh McGrath from OpenAI's post-training team discusses the evolution from GPT-4.1 to 5.1, focusing on token efficiency improvements, RLVR methodology shifts, and the new shopping model. He covers the technical challenges of scaling reinforcement learning, the importance of data quality over optimization methods, and future directions for context windows and agent capabilities.

Key Questions Answered

• •Token Efficiency Over Speed: GPT-5.1 achieved similar benchmark performance to GPT-5 while dramatically reducing token consumption. This metric matters more than wall-clock time because it determines how many tool calls an agent can make within reasonable serving constraints. The team optimizes for tokens used rather than minutes elapsed, fundamentally changing how they measure model capability and user experience quality.
• •RLVR Signal Quality: The shift from RLHF to RLVR represents a move toward higher-quality training signals rather than optimization improvements. Both use policy gradient methods, but RLVR provides verifiable rewards like mathematical correctness versus subjective human preferences. The innovation lies in data source quality, not gradient variance reduction. This spectrum of signal trustworthiness determines how much optimization can be applied without degradation.
• •Post-Training Infrastructure Complexity: Running reinforcement learning at scale involves exponentially more moving parts than pretraining. Each task requires different grading setups, creating multiple potential failure points during training runs. Researchers spend late nights debugging across codebases they don't own, jumping between internal and external systems. Understanding distributed systems and ML research equally becomes essential for pushing the frontier in post-training work.
• •Context Window Utilization: Graph walk evaluations reveal models can now perform complicated transformations across entire context windows, not just retrieve single points. This capability continues climbing, addressing earlier context rot concerns. The team focuses on perfect utilization of existing windows rather than expanding to billions of tokens, though agent workflows with multiple search calls may eventually require massive context capacity.
• •Skill Gap in ML Systems: The industry lacks people who excel at both distributed systems engineering and machine learning research. Educational programs optimize for one or the other, but frontier work requires seamlessly switching between fixing infrastructure bottlenecks and designing learning algorithms. Projects shift bottlenecks multiple times, making this hybrid skill set the hardest position to fill in post-training teams.