For years, the deep learning community observed a striking dichotomy: while natural language processing models scaled predictably according to precise power laws—where adding parameters, compute, and data consistently drove down cross-entropy loss—computer vision models frequently hit a performance ceiling. Traditional Convolutional Neural Networks (CNNs) often suffered from saturation when pushed to billion-scale datasets.
However, a profound architectural convergence has occurred. By restructuring visual data into discrete tokens, continuous sequences, or multi-resolution hierarchies, and swapping localized inductive biases for global self-attention, computer vision has unlocked its own empirical scaling laws. When vision models are scaled across parameters, data volume, and compute, their performance improves along predictable power-law trajectories, mirroring the emergent capabilities of Large Language Models (LLMs).
Overcoming the Saturation of Inductive Biases
Why did traditional vision architectures stall where transformers excel? CNNs rely on strong inductive biases like translation invariance and locality. While highly efficient in data-scarce regimes, these biases ultimately restrict a model's capacity to absorb information from massive datasets. Once a dataset grows large enough, these rigid architectural constraints prevent the model from learning complex, high-dimensional representations.
Dong Huk Park et al. demonstrated this phenomenon in Billion-Scale Pretraining with Vision Transformers for Multi-Task Visual Representations, showing that while CNNs and Vision Transformers (ViTs) perform comparably on smaller datasets, ViTs maintain a steep, unsaturated improvement curve as data volumes reach the billion-image mark. By replacing localized convolutions with global self-attention, the transformer architecture dynamically allocates its capacity across the entire image, allowing it to continuously scale without hitting a performance plateau.
Performance
^
| / Vision Transformers (Unsaturated Scaling)
| /
| /
| ------------/--
| / / \
| / / \ CNNs (Performance Saturation)
| / /
+--------------------------------------------> Data & Parameter Scale
This scalability becomes even more pronounced as the complexity and fidelity of the visual data increase:
- High-Resolution Content: Fengchuang Xing et al. observed in StarVQA: Space-Time Attention for Video Quality Assessment that the performance gap between transformers and CNNs widens significantly on high-definition video content, proving that global attention mechanisms are fundamentally better suited for high-fidelity demands.
- High-Diversity Datasets: In large-scale biological identification tasks, Alexis Joly et al. showed in Overview of LifeCLEF 2023: Evaluation of AI Models for the Identification and Prediction of Birds, Plants, Snakes and Fungi that the self-attention mechanisms of ViT foundation models capture subtle semantic nuances across highly diverse datasets far better than traditional convolutional networks.
- Data-Scarce Regimes: Even when labeled data is scarce, the underlying capacity of the architecture allows for massive scaling. Zhaowei Cai demonstrated in Semi-supervised Vision Transformers at Scale that large-scale transformer models continue to yield accuracy gains in semi-supervised regimes where convolutional networks plateau.
- Real-World Variance: When modeling complex spatial environments, single-author work by Zinan Lv in A Siamese Transformer with Hierarchical Refinement for Lane Detection confirmed that attention mechanisms scale more robustly to capture high-dimensional real-world variance as dataset complexity grows.
Autoregressive Vision and Next-Scale Prediction
The realization that vision can scale like language has led to a structural shift: treating images as sequential "visual sentences." Early attempts at visual autoregressive (AR) modeling predicted images token-by-token or pixel-by-pixel (raster-scan). However, this approach suffered from a severe computational bottleneck and struggled to capture global context because images, unlike text, contain massive spatial redundancies.
The breakthrough came from shifting from token-by-token prediction to a coarse-to-fine next-scale prediction framework. As proposed by Keyu Tian et al. in Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction, the model predicts entire multi-resolution scales of an image autoregressively. This structural adjustment allows the model to learn global dependencies first before refining fine details, unlocking a clear power-law scaling behavior identical to LLMs.
[Scale 1: 4x4 Coarse] ---> [Scale 2: 8x8 Medium] ---> [Scale 3: 16x16 Fine]
| | |
v v v
(Global Structure) (Emergent Geometry) (Texture & Detail)
This predictable scaling behavior forms the foundation of modern Large Vision Models (LVMs). Yutong Bai et al. showed in Sequential Modeling Enables Scalable Learning for Large Vision Models that treating images, videos, and annotations uniformly as sequential data allows a 3-billion parameter transformer to scale smoothly, unlocking emergent capabilities like in-context visual prompting.
A critical component of this scaling behavior is the stability of the underlying discrete latent space. The authors of Stabilize the Latent Space for Image Autoregressive Modeling: A Unified Perspective (Yongxin Zhu et al.) noted that stabilizing tokenization allows GPT-style models to scale effectively across both understanding and generation tasks. When properly stabilized, the validation loss of these autoregressive models correlates linearly with downstream performance metrics like prompt adherence and aesthetic quality, a property highlighted by Jian Han et al. in Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis.
As reviewed by Jing Xiong et al. in Autoregressive Models in Vision: A Survey, this power-law scaling implies that the historical quality gap between autoregressive and diffusion models can be systematically closed simply by scaling up compute and parameters.
Unified Modalities and Generalist Capacity
In language, scaling laws work because a single text interface can represent almost any reasoning task. To replicate this generalist capability in vision, researchers are moving away from task-specific heads toward unified multimodal sequences where pixels, text, depth maps, and segmentation masks are mapped into a shared token space.
David Mizrahi et al. demonstrated in 4M: Massively Multimodal Masked Modeling that scaling the variety of modalities alongside parameter count consistently improves performance across all tasks simultaneously. Instead of competing for capacity, diverse modalities act as mutually reinforcing signals when scaled inside a massive transformer backbone.
Input Tokens (RGB + Text + Depth)
|
v
+-----------------------------------------+
| Scaled Transformer Backbone (High Cap.) | --> Shared Representation Space
+-----------------------------------------+
|
v
Output Tokens (Segmentation + Image + Text)
This generalist scaling is further supported by Haiyang Wang et al. in GiT: Towards Generalist Vision Transformer Through Universal Language Interface, which shows that increasing model capacity allows a vanilla ViT with a universal language interface to handle everything from object detection to captioning without specialized architectures. Pushing the boundaries of this approach, Yuxin Fang et al. scaled a vanilla ViT to 1 billion parameters in EVA: Exploring the Limits of Masked Visual Representation Learning at Scale, proving that massive-scale masked pretraining creates a highly versatile "multi-modal pivot" capable of transferring high-performance representations across highly disparate visual tasks.
Extending Scaling Laws to Perception
Crucially, these scaling laws are not confined to autoregressive generation or standard image classification; they extend directly to dense perceptual tasks and alternative generative paradigms.
While diffusion models are traditionally optimized for visual quality, Rahul Ravishankar et al. revealed in Scaling Properties of Diffusion Models For Perceptual Tasks that discriminative perceptual tasks (like depth estimation and optical flow) performed by diffusion models also exhibit predictable power-law scaling relative to model size and computational budget. This means that visual understanding is just as systematically improvable through scale as generation.
Ultimately, the transition to transformer-based visual scaling works because it unifies the core mechanism of learning across modalities. By removing localized architectural constraints and treating visual data as scalable sequences, the vision domain inherits the exact same predictable, power-law performance gains that revolutionized natural language processing.
Key Foundational Work
Billion-Scale Pretraining with Vision Transformers for Multi-Task Visual Representations (2021) — contributed 2 concept elements to this direction.
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning (2018) — contributed 2 concept elements to this direction.
Pre-Trained Image Processing Transformer (2021) — contributed 1 concept elements to this direction.