Training vision models at low resolution first and full resolution later is now a standard way to cut pre-training cost. Fewer pixels means fewer tokens, and a Vision Transformer's attention cost grows with the square of the token count. Every published version of this trick resizes the whole batch on a fixed calendar, for example "75% of epochs at 112 pixels, then 224." That ignores something obvious. A centered dog on grass looks the same to the model at 112 and at 224 pixels. A crowded street scene with small objects does not. We propose to decide resolution per image. For each image we measure how much the model's representation changes when that image is shown at low versus full resolution, which we call its resolution gap. We then hand the scarce full-resolution token budget to the images with the largest gaps, and refresh this allocation as training proceeds.
The idea joins two lines of work that have stayed separate. Curriculum learning chooses which samples to emphasize by difficulty. Progressive resolution training chooses how much detail to show, but treats all samples alike. Our curriculum ranks samples by how much detail they need, which no prior work does. The bet is that, at a fixed compute budget, this allocation recovers most of the accuracy lost by stage-wise schedules. It should also avoid the instability caused by an abrupt switch to full resolution, because high-resolution images enter training gradually rather than all at once.
Core Claim
At a training-FLOP budget matched to a two-stage low-then-high schedule, assigning resolution per image by measured resolution gap should produce a better self-supervised ViT than either the stage-wise schedule or random mixed-resolution training. The gains should show most on tasks that need fine detail (semantic segmentation, corruption robustness), and should come without the patch-noise and optimizer-restart fixes that stage-wise schedules need. The claim is falsifiable in a specific way. If random per-image resolution assignment at the same token budget matches gap-ranked assignment, then image content does not matter, and the contribution reduces to "mixed resolution beats staged resolution."
Relation to Prior Work
The closest work is FastDINOv2. It pre-trains DINOv2 ViT-B/16 at 112 pixels for 75% of epochs and at 224 for the rest. It adds Gaussian noise patches and an AdamW state reset to counter a high-frequency bias that appears at the switch. It reports 76.2% ImageNet linear probe against 77.8% for full-cost training, and 56.7% on ImageNet-C. This paper both motivates our proposal, by documenting the transition shock, and supplies our training recipe and main baseline. We differ in that no batch-wide switch ever happens.
Three other systems schedule resolution or token granularity on fixed calendars. RECLIP trains CLIP at 64 pixels and then fine-tunes briefly at 224. CLIPA shows larger encoders tolerate smaller training images. As-ViT linearly moves from coarse to fine tokenization (82.5% vs 83.5% ImageNet with 56.2% fewer FLOPs). UniViTAR unlocks native variable resolution only in a late, fixed stage. All four apply one resolution decision to every image at a given step. We make that decision per image and base it on the model's current state.
Two works supply the feedback idea we reuse. iFNO grows Fourier modes when an energy-explained ratio drops below a threshold. That rule is batch-global and specific to neural operators, but it shows that feedback-driven growth can beat fixed schedules. Decomposing Densification in Gaussian Splatting schedules resolution from image Fourier energy. That suggests a static, model-free proxy for "needs detail," and we test this proxy directly as a competitor.
From curriculum learning we borrow the sample-selection view of Self-Paced Learning. We also borrow the warning from Self-Paced Curriculum Learning that purely student-driven selection latches onto outliers unless it is constrained by a teacher-defined feasible region. That warning constrains our design and is why we add a scheduled floor on full-resolution coverage.
Finally, Pixel-Level Self-Paced Learning concentrates early loss on hard, high-error pixels in super-resolution. That is the opposite direction from easy-first curricula. Our allocation is "hard-first" in its own sense: detail goes to the images that need it, while the easy majority stays cheap. We test the reversed allocation explicitly to see which direction helps.
Research Program
The method has three parts.
Scoring. For image $x_i$ and candidate resolution $r$, define the resolution gap as $$g_i(r) = 1 - \cos\big(f_\theta(x_i^{(r)}),\ f_\theta(x_i^{(224)})\big),$$ where $f_\theta$ is the frozen exponential-moving-average teacher that DINOv2 already maintains, and $x_i^{(r)}$ is the image resized to $r$ pixels. Computing $g_i$ costs one gradient-free forward pass at full resolution. To keep overhead small, we score a rotating slice of about 5% of the dataset each epoch and keep scores that are slightly stale. Images that have never been scored get a prior from their high-frequency spectral energy, the same quantity the splatting work uses.
Allocation. Resolutions come from {112, 160, 224}. Each step has a fixed token budget equal to FastDINOv2's average tokens per step at that point in training. Within that budget, a greedy rule upgrades the images with the largest estimated gap until the budget is spent.
Constraint. Following the self-paced curriculum idea, a scheduled floor sets the minimum fraction of each batch at full resolution. The floor rises smoothly from about 5% to about 50%. This guarantees every image eventually appears at full resolution, keeps the model's full-resolution statistics from drifting, and caps the influence of images whose gap stays high because they are noisy rather than informative.
Implementation. Batches are split into same-resolution micro-batches, which needs no packing machinery. Positional embeddings are interpolated, following the FastDINOv2 codebase. As an analysis, we track how gap scores evolve. We expect the gap distribution to shrink over training and to correlate with object count and object scale. We also log spectral profiles before and after training to check whether the transition-induced high-frequency bias disappears.
Baselines
The naive reference points are two. The first is full-resolution DINOv2 ViT-B/16 throughout, whose reported linear probe is 77.8% according to FastDINOv2. The second is fixed 112-pixel training throughout, which FastDINOv2 reports degrades at 224 evaluation. Both anchor the accuracy range and the cost range.
The strongest existing method is FastDINOv2's 75/25 schedule with noise patching and optimizer restart. Its reported results are 76.2% ImageNet-1K linear probe and 56.7% ImageNet-C FastDINOv2. We rerun it ourselves on the same data, augmentations, epoch count and hardware, so the comparison is not against a transcribed number. Beating it means a higher linear probe and higher ADE20K at equal or lower measured training FLOPs, with the margin larger than the seed-to-seed spread over three seeds. We also run an oracle-tuned split: the best of a sweep over 50/50, 75/25 and 90/10 at each model size. This represents a practitioner willing to pay for tuning.
Three ablations isolate the key ingredient, which is ranking images by the model's own measured gap. The first, random mixed resolution, assigns resolutions uniformly at random under the identical per-step token budget and floor schedule. It tests whether content-aware ranking matters at all. The second, static spectral allocation, ranks images by Fourier high-frequency energy only and never consults the model. It tests whether model feedback adds anything over an image statistic. The third, reversed allocation, gives full resolution to the smallest-gap images. It tests the direction question raised by PSPL. A fourth control, loss-ranked self-paced allocation, ranks by per-image DINO loss instead of resolution gap. It tests whether "difficulty" in the classic sense is the same thing as "needs detail."
All variants use the same budget, the same number of seeds, and the same small tuning budget for the floor schedule (three settings).
Why not just…?
Why not just tune the stage split? A careful sweep might recover much of the gain. CLIPA's finding that the tolerable resolution shifts with encoder size An Inverse Scaling Law for CLIP Training implies the best split must be re-tuned for every model and budget. Even an oracle split still produces the batch-wide transition shock that FastDINOv2 had to patch. The oracle-split baseline settles this directly. If it matches our method at ViT-S and ViT-B, our claim narrows to saving the cost of the sweep, and we would report it that way.
Why not just mix resolutions randomly? This is the real threat. Random mixing already removes the abrupt switch, and that removal alone may account for most of the benefit. The random-mixed ablation at an identical token budget decides it. If gap-ranked allocation does not beat random allocation on ADE20K and ImageNet-C by more than seed variance, the content-aware part of the proposal fails. In that case we would recommend smooth random mixing as the simpler method.
Why not just rank images by their frequency content, with no model in the loop? It is cheap and grounded in the splatting work's use of Fourier energy. Texture-rich but semantically simple images, such as grass or fabric, carry high spectral energy and yet may embed identically at 112 pixels. The model-based gap should waste less budget on them. The static-spectral ablation tests this, and we also report the rank correlation between the two scores. A correlation near 1 would make the feedback machinery unnecessary.
Why not just drop tokens by masking instead of resizing? CLIPA compared random masking with resizing at equal token counts and found resizing better, because masking destroys global scene context An Inverse Scaling Law for CLIP Training. We therefore keep resizing as the token-reduction operator and do not rerun this comparison.
Proposed Experiments
Development runs use ViT-S/16 on ImageNet-100, which FastDINOv2 also uses, so all ablations and the split sweep fit a modest budget. Final runs use ViT-B/16 on ImageNet-1K under the FastDINOv2 recipe and FLOP budget. We report six things: ImageNet-1K linear probe; ADE20K semantic segmentation mIoU with a frozen backbone and linear head (FastDINOv2 gives no ADE20K number on Lacuna, so we measure its baseline ourselves); ImageNet-C accuracy and mean corruption error Benchmarking Neural Network Robustness; ImageNet-P flip probability, which no progressive-resolution paper reports; measured training FLOPs including the scoring passes; and wall-clock time.
Three checks are decisive. The first is the ranking test: gap-ranked must beat random-mixed and static-spectral allocation on ADE20K at matched FLOPs. Dense prediction on cluttered scenes is where per-image detail should matter most. The second is the stability test: without noise patching or an optimizer reset, our method's training loss should show no spike, and its ImageNet-C score should match or exceed FastDINOv2's reported 56.7%. If the transition shock is really an artifact of batch-wide switching, removing the switch should remove the need for the patches. The third is the transfer test: the floor schedule tuned at ViT-S must work unchanged at ViT-B, while the oracle split re-tuned at each size must not beat it.
Failing the first check reduces the work to a negative result about content-aware allocation. Failing the third means adaptivity buys nothing over tuning. Both outcomes are informative and both are publishable.
References
- FastDINOv2: Frequency Based Curriculum Learning Improves Robustness and Training Speed
- RECLIP: Resource-efficient CLIP by Training with Small Images
- An Inverse Scaling Law for CLIP Training
- Auto-scaling Vision Transformers without Training
- VITRIX-UniViTAR: Unified Vision Transformer with Native Resolution
- Incremental Spatial and Spectral Learning of Neural Operators for Solving Large-Scale PDEs
- Decomposing Densification in Gaussian Splatting for Faster 3D Scene Reconstruction
- Self-Paced Learning for Latent Variable Models
- Self-Paced Curriculum Learning
- Pixel-Level Self-Paced Learning For Super-Resolution
- Benchmarking Neural Network Robustness to Common Corruptions and Perturbations