In natural language processing, the paradigm of "one model to rule them all" has long been established. A single large language model can summarize text, write code, and translate languages without any structural modifications. Computer vision, however, has historically remained a land of specialists. For decades, the field has been fragmented into task-specific architectures: one model for depth estimation, another for semantic segmentation, and a third for object detection.
This architectural rigidity creates a massive engineering burden. In practical applications like robotics, managing a suite of specialized models requires individual fine-tuning and complex integration logic, as noted by Zijun Long et al. in their work on industrial vision systems (RoboLLM). To move toward true general-purpose visual intelligence, contemporary research is focused on breaking down these task-specific barriers and building unified generalist vision models.
The Root of Fragmentation: Structural Heterogeneity
Why did computer vision fragment into specialized architectures while language processing unified? The answer lies in the structural heterogeneity of visual tasks.
In language, both inputs and outputs are fundamentally sequences of discrete tokens. In vision, different tasks demand radically different output formats:
- Classification requires a single discrete label or probability distribution.
- Object Detection requires a variable number of continuous bounding box coordinates ($x, y, w, h$).
- Segmentation requires dense, pixel-level binary or categorical masks.
- Depth Estimation requires continuous, real-valued dense matrices.
As Thomas E. Huang et al. observe in the Video Task Decathlon, this divergence makes it incredibly difficult to design a shared representation that serves all requirements without compromising accuracy. Historically, this forced researchers to append highly specialized, task-specific "heads" (such as region proposal networks or multi-scale convolutional decoders) to a shared backbone network. While early unified networks like UberNet, proposed by Iasonas Kokkinos, proved that a single backbone could simultaneously output low-, mid-, and high-level predictions, they still relied on a fragmented collection of task-specific output branches.
To eliminate this rigidity, modern generalist architectures bypass specialized heads entirely by unifying the interface of computer vision. They achieve this through three primary strategies: sequence-to-sequence tokenization, unified generative modeling, and agglomerative distillation.
Strategy 1: Unifying via Sequence-to-Sequence Tokenization
The most direct way to emulate the success of language models is to format all visual outputs as sequences of discrete tokens. If bounding boxes, keypoints, and class labels can be translated into a unified "vocabulary," a standard Transformer encoder-decoder can handle any task without architectural modifications.
[Image Input] ──> [Transformer Encoder] ──> [Transformer Decoder] ──> [Unified Sequence Output]
├── "dog" (Class)
├── [42, 12, 85, 94] (Box)
└── [Token_32, Token_88] (Mask)
For instance, Zhiyang Chen et al. developed Obj2Seq, a framework that treats object detection and classification as a sequence generation problem. Instead of using specialized regression heads, the model simply "spells out" the coordinates and labels as discrete tokens. This sequence-based approach was extended to visual object tracking by Xin Chen et al. in SeqTrack, which ditches complex classification and regression branches in favor of a pure sequence-to-sequence framework that reads out bounding boxes token by token.
At scale, this concept enables Large Vision Models (LVMs). Yutong Bai et al. demonstrated in Sequential Modeling Enables Scalable Learning for Large Vision Models that images, videos, and annotations can all be treated as "visual sentences." By tokenizing both pixels and labels into a unified sequence, a single 3-billion parameter Transformer can solve diverse tasks via in-context visual prompting. Similarly, David Mizrahi introduced the 4M framework, which maps dozens of distinct modalities—from RGB pixels and text to depth maps and neural feature maps—into a single sequence of discrete tokens, creating a flexible "any-to-any" generalist engine.
Strategy 2: Unifying via Generative Diffusion
An alternative to tokenization is to cast all vision tasks as an image generation problem. If a model can generate high-fidelity images, it can also generate visual representations of abstract tasks—such as rendering a grayscale image where pixel intensity represents depth, or a color-coded image representing segmentation masks.
Charlie Nash et al. pioneered this with Transframer, a unified generative framework that treats video prediction, novel view synthesis, and semantic segmentation as a single conditional frame prediction problem. By framing perception as generation, the model requires no task-specific components.
With the advent of diffusion models, this generative unification has accelerated:
- OneDiffusion: Developed by Duong H. Le et al., One Diffusion to Generate Them All treats text-to-image generation, multi-view synthesis, depth estimation, and object detection as sequences of "views" within a single flow-matching framework.
- InstructCV: Yulu Gan et al. demonstrated in InstructCV that text-to-image diffusion models can be repurposed as vision generalists. By casting vision outputs as images and training the model with natural language instructions, it can execute segmentation or depth estimation zero-shot based purely on human phrasing.
Strategy 3: Agglomerative Distillation
Rather than training a generalist from scratch using tokens or generation, a third strategy fuses existing, highly specialized foundation models into a single backbone. The computer vision ecosystem has already produced exceptional "specialist" foundation models: CLIP excels at language alignment, DINOv2 at dense feature extraction, and SAM at high-precision segmentation.
To eliminate the computational burden of running these models concurrently, Mike Ranzinger et al. introduced AM-RADIO. This "agglomerative" framework distills the knowledge of CLIP, DINOv2, and SAM into a single, unified vision backbone. As further refined by Greg Heinrich et al. in RADIOv2.5, this approach creates a multi-task representation that retains both global semantic understanding and local pixel precision, outperforming individual teacher models while running up to 10x faster.
Why It Works: Cross-Task Synergy
The shift toward generalist architectures is not just driven by a desire for simplicity; it works because of cross-task synergy.
In a fragmented paradigm, a depth estimation model is blind to semantic boundaries, and an object detector is blind to continuous 3D geometry. However, visual tasks are deeply interconnected. Learning the boundaries of an object inherently helps determine its depth, and understanding the semantic context of a scene helps localize its components.
When a model is forced to solve multiple tasks through a single, unified interface, it must learn a shared latent representation where these distinct domains complement one another. For example, Feng Li et al. demonstrated in Mask DINO that object detection and image segmentation actively assist each other when unified under a single Transformer framework, leading to state-of-the-art results in both domains.
Furthermore, this shared representation unlocks powerful in-context learning capabilities. As shown by Amir Bar et al. in Visual Prompting via Image Inpainting, a generalist model trained purely on a visual cloze task (filling in missing image regions) inherently learns enough underlying visual structure to solve completely novel tasks—such as edge detection or segmentation—zero-shot, simply by providing an example input-output pair within the visual prompt.
Conclusion
The transition from task-specific rigidity to unified generalist models represents a fundamental shift in computer vision. By resolving structural heterogeneity through sequence tokenization, generative framing, or agglomerative distillation, modern architectures eliminate the need for fragmented pipelines. The resulting models are not only more computationally efficient to deploy, but they also tap into cross-task synergies that allow them to understand the visual world more holistically, paving the way for versatile, human-like visual intelligence.
Key Foundational Work
Learning in Computer Vision and Image Understanding (1993) — contributed 2 concept elements to this direction.