Introduction
In many deep learning applications, finding the "perfect" model requires balancing competing objectives. Whether it is maximizing image quality while minimizing file size, or balancing reconstruction accuracy with latent space organization, practitioners rarely rely on a single, fixed loss function. Instead, they train a fleet of separate models with different loss weightings to find the optimal trade-off.
You Only Train Once (YOTO) introduces a paradigm shift: rather than training multiple specialized models, YOTO trains a single model conditioned on loss weights. This allows one set of parameters to navigate the entire trade-off space at inference time, rivaling the performance of specialized models with a fraction of the training and storage overhead.
The Problem with Fixed Loss Weights
In tasks with competing objectives, the loss function is typically a weighted sum of different terms. Common examples include:
- Image Compression: Balancing minimal file size (rate) against maximum visual quality (distortion).
- $\beta$-VAE: Balancing accurate reconstruction against a well-organized latent space (KL divergence).
- Style Transfer: Balancing adherence to a target style against the structural retention of the original content.
Traditionally, handling these trade-offs requires training multiple models with different weightings ($\lambda$) and selecting the best one, or deploying an entire suite of models to offer users different options. This approach multiplies training time, consumes massive storage, and prevents dynamic fine-tuning once a model is deployed.
The YOTO Approach
The core innovation of YOTO is treating the loss weights not as fixed hyperparameters, but as input variables to the network.
- Loss-Conditional Architecture: The model, denoted as $F(x, \theta, \lambda)$, takes both the input data $x$ and the loss parameters $\lambda$ as inputs.
- Conditioning via FiLM: To inject the loss parameters into the network, YOTO utilizes Feature-wise Linear Modulation (FiLM). Small Multi-Layer Perceptrons (MLPs) process $\lambda$ and output scaling and shifting factors that dynamically modify the feature maps of the main convolutional layers.
- Training on a Distribution: During training, instead of using a fixed $\lambda$, a new $\lambda$ is sampled from a distribution (typically log-uniform) for every data point or mini-batch.
- Theoretical Foundation: The authors provide a mathematical proof demonstrating that, given sufficient model capacity, a single conditional model can represent the collection of optimal solutions for all possible loss weights within the training distribution.
Key Insights and Performance
YOTO exploits the inherent redundancy across models trained for different loss weights. Because these models likely share the same low-level feature detectors, YOTO forces a single model to learn these shared features while using the conditioning signal to "steer" the final output.
- The Capacity-Performance Trade-off: There is a "free lunch" threshold. While a YOTO model of the exact same size as a specialized model might slightly underperform, a YOTO model with roughly 1.5x the width can often match or beat an entire suite of specialized models.
- Log-Uniform Sampling: Because loss weights often vary across orders of magnitude (e.g., from 0.1 to 100), sampling them on a log scale ensures the model learns effectively across the entire range, rather than over-focusing on the largest values.
- Minimal Overhead: Remarkably, training a YOTO model is only about 8% slower than training a single standard model, yet it entirely replaces the need for dozens of separate training runs.
Experimental Results
The authors evaluated YOTO across three distinct domains: $\beta$-VAE (CIFAR-10, Shapes3D), Learned Image Compression (Kodak, Tecnick), and Fast Style Transfer (ImageNet, Pointillism).
- $\beta$-VAE: YOTO successfully recovered the entire "Pareto frontier" (the curve of optimal trade-offs). On the Shapes3D dataset, the outputs from YOTO were qualitatively indistinguishable from models trained on specific fixed weights.
- Style Transfer: A single YOTO model enabled users to slide a "content-style" bar in real-time. Quantitatively, the total loss achieved by YOTO closely matched the loss of specialized models trained for specific points on that continuum.
- Compression: While YOTO showed a small performance gap in high-compression regimes for shallow networks, wider YOTO models effectively recovered the rate-distortion performance of state-of-the-art specialized compression models.
Limitations and Future Directions
While highly effective, YOTO has a few limitations. In image compression, the performance gap between YOTO and fixed models was slightly larger than in other tasks. The authors hypothesize that the relatively shallow networks typically used in compression might lack the "representational room" needed to handle complex conditioning.
Additionally, while log-uniform sampling worked well for these tasks, choosing the optimal distribution of losses for training remains an open research question, particularly for problems with many competing objectives. Future work may also explore more sophisticated conditioning methods beyond FiLM to close the minor performance gaps observed in certain tasks.
Summary updated