In natural language processing, a single foundational architecture can summarize text, translate languages, and write code without altering its underlying structure. Computer vision, by contrast, has historically been a domain of specialists. A typical vision pipeline relies on distinct architectural "heads" tailored to specific tasks: bounding box regressors for object detection, discrete label maps for semantic segmentation, and continuous scalar fields for depth estimation.
As noted by Sheng Jin et al. in UniFS: Universal Few-Shot Instance Perception with Point Representations, this mathematical heterogeneity of output formats presents a major hurdle to building true generalist vision models.
A powerful paradigm has emerged to resolve this fragmentation: visual task unification. Instead of engineering specialized network heads or forcing spatial data into discrete language tokens, this approach reformulates diverse computer vision problems as image-to-image translation tasks within a unified, standard three-channel RGB pixel space. By treating every task as a request to "paint" the answer, a single general-purpose architecture can solve an array of discriminative and generative vision problems.
The Pixel as a Universal Interface
The core intuition behind visual task unification is that almost any computer vision output can be rendered visually. A semantic segmentation mask can be represented as a colored overlay; a depth map can be encoded as a grayscale gradient; and object detection can be expressed by drawing bounding boxes directly onto an image canvas.
[Input Image] ---> [ Unified Generalist Model ] ---> [ RGB Output Canvas ]
|
+---> Segmentation (Color Masks)
+---> Depth (Grayscale Gradients)
+---> Detection (Painted Boxes)
Mapping heterogeneous tasks into a shared pixel space offers a fundamental advantage over token-based language interfaces: it preserves spatial precision and structural relationships. In Images Speak in Images: A Generalist Painter for In-Context Visual Learning, Xinlong Wang et al. demonstrate that defining a standard RGB output space allows a unified encoder-decoder architecture to handle varied vision problems through a consistent interface. This design eliminates the need for task-specific heads and prevents the spatial quantization errors that occur when converting dense visual coordinates into discrete text tokens.
This pixel-to-pixel mapping applies across diverse granularities and domains. For instance, Dezhi Peng et al. demonstrate in UPOCR: Towards Unified Pixel-Level OCR Interface that fragmented document tasks—such as text removal, stroke segmentation, and forgery detection—can be unified under a single RGB-to-RGB translation framework.
Even when dealing with non-visual data or downstream machine vision tasks, the pixel space serves as an effective optimization target. Zhiheng Lyu in PixelWorld: Towards Perceiving Everything as Pixels explores rendering textual and structural layouts into a shared pixel space to preserve spatial context without separate tokenization pipelines. Similarly, Xin Li et al. show in Task-driven Semantic Coding via Reinforcement Learning that diverse machine vision objectives can be compressed into a standardized pixel-wise importance map, allowing a single compression framework to serve multiple downstream applications.
Repurposing Generative Models for Perception
If visual perception tasks can be framed as image generation problems, then powerful generative architectures—such as diffusion models and autoregressive transformers—can be repurposed as vision generalists.
This insight allows researchers to exploit the massive structural and semantic priors embedded within pre-trained generative models. In InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists, Yulu Gan et al. repurpose text-to-image diffusion models to perform segmentation, detection, and depth estimation by following natural language instructions. By treating computer vision as an instruction-guided image-to-image generation problem, a subsequent work by Yulu Gan et al., InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists, achieves strong generalization to unseen categories and human phrasing simply by training the model to output binary masks or annotated images.
Similarly, Canyu Zhao et al. introduce DICEPTION: A Generalist Diffusion Model for Visual Perceptual Tasks, which maps perception outputs into a standard RGB space to handle multiple modalities without task-specific heads.
To bridge generative and discriminative tasks completely, Weifeng Lin et al. develop PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructions. This framework encodes non-pixel outputs like bounding boxes into visual representations that a Diffusion Transformer learns to generate, which are then converted back into their original formats via post-processing.
[Task: Bounding Box] ---> Encode as Pixels ---> [ Diffusion Transformer ] ---> Decode Pixels ---> [ Final Box Coordinates ]
Beyond diffusion, autoregressive models can also leverage this paradigm. Xiaolong Wang et al. introduce ARGenSeg: Image Segmentation with Autoregressive Image Generation Model, which treats image segmentation as an autoregressive token generation task. By outputting dense masks directly as visual tokens, a single backbone handles interactive segmentation and anomaly detection without specialized architectural modifications.
In-Context Visual Learning
One of the most profound consequences of visual task unification is the emergence of in-context learning for computer vision. Because the input and output spaces are uniformly composed of pixels, a model can infer a task at test time simply by looking at example image pairs provided in its input context.
In SegGPT: Towards Segmenting Everything In Context, Xinlong Wang et al. treat segmentation as an image-to-image "coloring" problem. The model uses a general-purpose encoder-decoder structure to process visual prompts and queries identically. Given a prompt image and its corresponding colored mask, the model can segment novel objects or track targets in a query image without any fine-tuning or weight updates.
Prompt Input: [ Image of Apple ] ---> Prompt Output: [ Red Masked Apple ]
Query Input: [ Image of Pear ] ---> Model Output: [ Red Masked Pear ] (Inferred Task: Segment Object)
This capability is highly valuable in specialized domains where training data is scarce or highly variable. For example, Sucheng Ren et al. introduce the Medical Vision Generalist: Unifying Medical Imaging Tasks in Context, which unifies medical segmentation, cross-modal synthesis, and denoising into a conditional image-to-image generation framework. By leveraging visual in-context learning, the model adapts to entirely new modalities and anatomical regions using only a few example pairs.
Flexible Conditionals and Shared Primitives
Framing vision as an image-to-image pipeline also redefines how we manipulate and reconstruct visual data. Instead of building isolated architectures for style transfer, image editing, or smoothing, these tasks can be viewed as different configurations of a shared underlying representation.
In Composer: Creative and Controllable Image Synthesis with Composable Conditions, Lianghua Huang et al. show that diverse manipulation tasks can be unified by decomposing images into independent conditional factors (such as sketches, depth maps, and color palettes). By selecting which source factors to retain, modify, or replace, a single model performs a wide variety of editing tasks without specialized retraining.
This concept extends down to low-level vision primitives and rasterization. Jiaolong Yang et al. demonstrate in A Generic Deep Architecture for Single Image Reflection Removal and Image Smoothing that an edge-to-image pipeline serves as a versatile primitive capable of generalizing across distinct low-level challenges like reflection removal and smoothing.
At the representation level, the authors of WIPES: Wavelet-based Visual Primitives propose a unified visual primitive integrated into a differentiable rasterizer. By capturing both low- and high-frequency details uniformly, this representation demonstrates generality across 2D image fitting, static novel view synthesis, and dynamic scene reconstruction.
Why Visual Task Unification Works
Visual task unification succeeds because it aligns the mathematical structure of the task with the architectural strengths of deep vision networks:
- Architectural Simplicity: It replaces a complex ecosystem of task-specific heads, custom loss functions, and specialized anchor boxes with a general-purpose encoder-decoder or diffusion backbone.
- Shared Optimization Targets: By mapping all outputs to a standard RGB pixel space, the model optimizes for a single, universal objective—such as mean-squared error, pixel-wise cross-entropy, or diffusion denoising.
- Spatial Alignment: Unlike language-token representations, an RGB canvas preserves the exact spatial coordinates and pixel-level correspondences inherent to visual perception.
- Data and Prior Scaling: It allows discriminative perception tasks to directly benefit from the massive scale, semantic understanding, and generative priors developed during large-scale image synthesis pre-training.
Key Foundational Work
Task-driven Semantic Coding via Reinforcement Learning (2021) — contributed 1 concept elements to this direction.
A Generic Deep Architecture for Single Image Reflection Removal and Image Smoothing (2017) — contributed 1 concept elements to this direction.