Unified Vision via Image to Image Translation

This direction is split into other directions: visual task unification. Its last version stays readable and citable here.

Version 1 of 130 papers40 conceptsfrozen 2026-10-04

For decades, computer vision has been a field of specialists. If you wanted to detect objects, you built a model with a bounding box head; if you wanted to estimate depth, you designed a regression head; if you wanted to segment pixels, you used a classification head. This fragmentation created a "zoo" of task-specific architectures that could not easily share knowledge or adapt to new problems without extensive retraining.

A new direction in research seeks to collapse these boundaries by treating every computer vision task—from low-level restoration to high-level semantic understanding—as a simple image-to-image translation problem. By representing all inputs and outputs as standard RGB images, researchers are building "generalist" models that can solve almost any visual task within a single, unified framework.

The Universal Interface: Everything is an Image

The core intuition behind this shift is that the pixel is a universal representational unit. Traditionally, we treated depth maps as single-channel continuous values and segmentation masks as discrete integer labels. However, as Xinlong Wang, Yue Cao, and their colleagues demonstrate in Painter, these heterogeneous data types can be mapped into a standard three-channel RGB space.

When a depth map is visualized as a grayscale or colormapped image, or a segmentation mask is rendered as a set of colored regions, they become "images" that a standard Vision Transformer (ViT) can process. This standardization allows a single architecture to bridge the gap between recognition, processing, and generation. As Qianru Sun and Xue Han argue in their work on In-Context Translation, once data is unified in a shared visual space, specialized output heads become unnecessary.

This "visual canvas" approach extends even to non-pixel outputs. In PixWizard, Weifeng Lin shows that coordinates for bounding boxes or keypoints can be rendered as visual overlays. By training a model to "draw" these boxes or skeletons onto a canvas, a generative model can perform discriminative tasks like object detection using the same machinery it uses for image editing.

In-Context Learning for Pixels

Perhaps the most powerful consequence of this unification is the ability to perform in-context visual learning. In Natural Language Processing, a model like GPT-4 can learn a new task simply by being shown a few examples in its prompt. By framing vision as image-to-image translation, models like SegGPT bring this capability to pixels.

In this paradigm, the model is fed a "prompt" consisting of an example input image and its corresponding output (e.g., a photo and its segmentation mask), followed by a new query image. The model's task is to "fill in the blanks" or perform "inpainting" to produce the output for the query. Because the model learns the relationship between the example pair, it can adapt to new object categories or unseen tasks at inference time without any weight updates. Yihao Liu and Xiangyu Chen further generalize this in PromptGIP, showing that a single model can handle over 15 diverse image processing tasks by inferring the user's intent from these visual examples.

Generative Models as Generalist Perceptors

While early generalist models used standard Transformers, recent work has pivoted toward Diffusion Transformers (DiT). Diffusion models are naturally suited for image-to-image translation because they are designed to refine pixel values iteratively.

Researchers are now repurposing the generative power of models like Stable Diffusion for "dense" perception tasks. Canyu Zhao and the authors of DICEPTION demonstrate that by performing diffusion directly in pixel space, a model can maintain the high spatial precision required for tasks like edge detection or depth estimation—precision that is often lost in latent-space models.

This approach also enables complex scene composition. In Canvas-to-Image, Yusuf Dalva and Kfir Aberman show that by placing different control signals—like a pose skeleton, a bounding box, and a reference subject—onto a single RGB canvas, a model can leverage its pre-trained spatial reasoning to synthesize a coherent image that satisfies all constraints simultaneously.

Domain-Specific Unification

The benefits of the image-to-image framework are particularly evident in specialized fields like Optical Character Recognition (OCR) and microscopy, which have historically relied on highly fragmented pipelines.

In the OCR domain, Dezhi Peng and Zhenhua Yang introduced UPOCR, which unifies text removal, stroke segmentation, and forgery detection into one RGB-to-RGB interface. Similarly, in bioimaging, Jianxu Chen developed MMV_Im2Im, a toolbox that treats segmentation and restoration as identical transformation tasks. These works prove that specialized architectures are not a prerequisite for high performance; rather, a consistent data pipeline allows a single model to capture shared features across disparate functional requirements.

By moving away from task-specific "heads" and toward a unified pixel-based language, the field is moving closer to a true "foundation model" for vision—one that can see, process, and create within a single, continuous visual world.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_unified_vision_via_image_to_image_transl_88d74a61,
  title = {Unified Vision via Image to Image Translation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unified-vision-via-image-to-image-translation/rdr_88d74a610c8a4f6693b87ba83953d744},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_unified_vision_via_image_to_image_transl_88d74a61_v1,
  title = {Unified Vision via Image to Image Translation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unified-vision-via-image-to-image-translation/rdr_88d74a610c8a4f6693b87ba83953d744/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}