Why Models Are AI’s Next Training Dataset with Damian Borth - #772

What It Covers

Damian Borth, professor at University of St. Gallen, presents weight-space learning — a framework treating trained neural network weights as training data for new models. Starting in 2021, his lab built autoencoder architectures that analyze, compress, and generate neural network weights, potentially replacing pretraining by sampling models on demand from a foundation model of neural networks.

Key Questions Answered

• •Weight-Space Autoencoding: Train an encoder-decoder on populations of neural networks to learn a compressed latent manifold of weights. From this manifold, you can predict model accuracy without test data and generate new model weights. Early results on small homogeneous model zoos achieved R-squared accuracy prediction outperforming handcrafted feature baselines from Google researchers.
• •Scaling via Windowed Tokenization: Rather than encoding entire model weights at once, process weights as windowed sequences with position encodings indicating layer, block, and absolute position. This approach, developed with Michael Mahoney at UC Berkeley, enables the method to scale beyond toy networks to ResNet-scale architectures and beyond, unlocking training on diverse open-weight repositories.
• •GPU Hour Reduction via Knowledge Transfer: By using an ImageNet-trained ViT as an encoder prompt to generate remote sensing foundation models, Borth's lab matches the performance of TelIFM — trained for 12,000 GPU hours — using only 350 GPU hours, a 30x reduction. This demonstrates cross-domain knowledge transfer through weight generation rather than data-based pretraining.
• •Dataset Prompting for Privacy-Preserving Model Generation: An in-review paper proposes replacing model prompts with dataset embeddings — a single aggregated vector representing a private dataset — to generate task-specific weights without exposing raw data. This mirrors CLIP's aligned text-image space, enabling regulated institutions like central banks to receive custom model weights without sharing proprietary training data.
• •Hugging Face as Training Corpus: Download open-weight models from Hugging Face, strip weights into flat sequences, apply normalization and diversity-based filtering, and train a backbone on roughly 2,000 quality-filtered models out of 20,000 candidates. The resulting model generates weights across architectures including ViTs, ResNets, and EfficientNets, and can cross domains — producing a GPT-2-scale language model initialized from computer vision weights.