Training modern vision models is a battle against the quadratic cost of pixels. As we move from standard $224 \times 224$ images to high-resolution $1024 \times 1024$ inputs, the computational requirements don't just grow—they explode. However, a recurring insight across diverse architectures suggests that we don't need high resolution for the entire training process. By treating image resolution as a curriculum—starting with coarse, low-resolution inputs and progressively increasing to fine-grained, high-resolution data—researchers are achieving state-of-the-art performance with a fraction of the traditional compute cost.
The Coarse to Fine Intuition
The fundamental reason progressive resolution works is that neural networks learn visual concepts in a hierarchical fashion. In the early stages of training, a model is primarily learning "what" is in an image: global structures, large-scale shapes, and basic semantic compositions. These features are perfectly visible even in heavily downsampled images.
As Tero Karras and colleagues demonstrated in the seminal Progressive Growing of GANs for Improved Quality, Stability, and Variation, starting with $4 \times 4$ images allows a generator to stabilize its global structure before it ever has to worry about the texture of a single hair. This "coarse-to-fine" strategy prevents the model from being overwhelmed by high-frequency noise early on, leading to more stable convergence and a significant reduction in total training time.
The Efficiency of the Inverse Scaling Law
In the context of foundation models like CLIP, this efficiency becomes even more dramatic. Xianhang Li and Cihang Xie identified what they call An Inverse Scaling Law for CLIP Training. They discovered that larger encoders actually require fewer visual tokens to learn effectively. By pre-training on low-resolution inputs and concluding with a brief high-resolution fine-tuning phase, their CLIPA framework trained models up to 33x faster than standard methods.
Similarly, Runze Li showed in RECLIP: Resource-efficient CLIP by Training with Small Images that training on tiny $64 \times 64$ images for the majority of the schedule captures nearly all the necessary semantic concepts. The final high-resolution stage acts as a "refinement" phase, where the model learns to map its established concepts to fine-grained spatial details.
Implementation Strategies Across Modalities
This curriculum approach has been successfully adapted across the entire spectrum of computer vision:
- Vision Transformers: Hugo Touvron proved in DeiT III: Revenge of the ViT that even "vanilla" ViTs can outperform complex modern architectures by simply leveraging a resolution-increasing training recipe.
- Multimodal Models: Google’s PaliGemma: A versatile 3B VLM for transfer, authored by Andreas Peter Steiner and Lucas Beyer, uses short training phases at increasingly higher resolutions to master detail-oriented tasks like document understanding without the cost of full-scale high-res pre-training.
- Generative Models: In the realm of diffusion, Keyang Xu introduced "Greedy Growing" in Greedy Growing Enables High-Resolution Pixel-Based Diffusion Models. This method isolates the "core" semantic alignment on low-resolution data before expanding the architecture to handle high-resolution rendering, enabling 8B-parameter models to generate $1024 \times 1024$ images efficiently.
- Specialized Domains: Even in medical imaging, Umberto A. Gava utilized a two-stage procedure in Neural network-derived perfusion maps: A model-free approach to computed tomography perfusion in patients with acute ischemic stroke, pre-training on downsampled data to ensure the model captures global blood flow patterns before fine-tuning for high-fidelity spatial details.
Why It Works: The Gradient Noise Perspective
From an optimization standpoint, progressive resolution training functions as a form of adaptive batching. In the early stages of training, gradients are often noisy as the model searches for the general direction of the loss landscape. Using low-resolution images allows for much larger batch sizes and faster iterations, effectively "averaging out" this noise at a lower cost.
As training progresses and the model nears convergence, the gradients become more precise. This is the optimal time to introduce high-resolution data, as the model is now stable enough to benefit from the subtle, fine-grained signals that high-resolution pixels provide. By aligning the complexity of the input with the maturity of the model, progressive resolution curricula ensure that every FLOP is spent where it provides the most utility.