[NeurIPS Best Paper] 1000 Layer Networks for Self-Supervised RL — Kevin Wang et al, Princeton

What It Covers

Princeton researchers Kevin Wang and team achieved NeurIPS Best Paper by scaling reinforcement learning networks to 1000 layers using self-supervised learning objectives, challenging the field's conventional shallow architecture approach.

Key Questions Answered

• •Self-Supervised RL Objective: The breakthrough required shifting from traditional value-based RL to contrastive representation learning that classifies whether future states belong to the same trajectory, converting RL into a scalable classification problem similar to language models.
• •Architectural Recipe for Depth: Scaling depth alone failed initially. Success required combining residual connections, layer normalization, and specific architectural components together. Critical performance jumps occurred only when depth exceeded 50-64 layers with these modifications in place.
• •Parameter Efficiency Trade-offs: Scaling network depth grows parameters linearly while scaling width grows them quadratically. Depth scaling proved more sample-efficient and parameter-efficient, achieving state-of-the-art performance on goal-conditioned RL tasks with single H100 GPU training runs.
• •JAX GPU Acceleration Enables Scale: Using JAX-based GPU-accelerated environments allows collecting thousands of parallel trajectories simultaneously. Performance improvements only manifest after 50 million transitions, making this data throughput essential for training deep networks in RL settings.