In natural language processing, large language models (LLMs) exhibit "in-context learning"—the ability to solve a new task simply by observing a few examples in a text prompt. Traditionally, computer vision has lacked this flexibility, requiring expensive fine-tuning or specialized architectures for every new task, from segmentation to depth estimation.
Recent research has introduced a paradigm shift: Visual In-Context Learning (V-ICL). By framing diverse vision tasks as image-to-image completion problems, models can now adapt to novel tasks at inference time using only visual examples.
The Unified Image-to-Image Interface
The core intuition behind V-ICL is that most vision tasks can be reformulated as a transformation from one image to another. For example, semantic segmentation is a mapping from an RGB image to a category-colored mask, and edge detection is a mapping to a sketch.
Wang et al. et al. proposed Painter (Images Speak in Images), a generalist model that treats these diverse tasks as a simple "inpainting" problem. Instead of using task-specific heads, the model is trained to predict missing pixels in an output image given an input image. By providing a "visual prompt"—a pair of images showing an example transformation—the model identifies the underlying logic and applies it to a new query image without any weight updates.
This approach was further refined in SegGPT (SegGPT: Towards Segmenting Everything In Context), where Wang et al. et al. demonstrated that segmentation could be treated as a "coloring" problem. By attending to the context provided in the prompt, the model can segment novel object categories or follow video targets zero-shot.
Task Specification as Visual Analogy
How does a model know which task to perform? V-ICL relies on the concept of visual analogy completion.
Bar et al. et al. introduced a grid-based prompting mechanism (Visual Prompting via Image Inpainting). By arranging an example input-output pair alongside a query image in a $2 \times 2$ grid, the model is forced to "fill in the blank" for the fourth quadrant. This structure allows the model to infer the intended transformation—such as "find the edges" or "segment the cat"—purely from the spatial relationship between the provided images.
Dedhia et al. et al. argue in Im-Promptu (Im-Promptu: In-Context Composition from Image Prompts) that for this to work, the model must move beyond pixel-level processing and view the world as a collection of objects. This allows the model to recognize the latent rule governing the transformation in the prompt and compose a new scene that follows that same rule.
Scaling Vision like Language
To achieve robust in-context abilities, models require massive pre-training on sequential data. Bai et al. and Geng developed a Large Vision Model (LVM) (Sequential Modeling Enables Scalable Learning for Large Vision Models) that treats images and annotations as "visual sentences." By training on 420 billion tokens using a next-token prediction objective, the model learns to complete sequences of visual examples, enabling zero-shot generalization to tasks like 3D reasoning and edge detection.
Similarly, Yang et al. et al. suggest that video can serve as the unified interface for real-world decision-making (Video as the New Language for Real-World Decision Making). In this view, vision tasks are reformulated as next-frame predictions, where the "prompt" is the history of the scene and the "output" is the future state or the desired transformation.
Generalizing to Specialized Domains
The power of V-ICL lies in its ability to handle "out-of-distribution" data where labels are scarce.
- Medical Imaging: Ren et al. et al. showed that a Medical Vision Generalist (Medical Vision Generalist: Unifying Medical Imaging Tasks in Context) can adapt to new modalities (like MRI or CT) and anatomical regions using only a few example images, outperforming specialized models.
- Anomaly Detection: Pang et al. et al. developed InCTRL (Toward Generalist Anomaly Detection via In-Context Residual Learning), which identifies defects in industrial parts or medical scans by comparing a query image against a few "normal" reference images provided as prompts.
- Image Editing: Meng et al. et al. demonstrated in InstructGIE (InstructGIE: Towards Generalizable Image Editing) that visual prompts are often more effective than text for capturing complex stylistic changes, as the model can directly observe the "before-and-after" relationship.
By treating vision as a pattern-completion task, V-ICL bypasses the need for task-specific engineering, moving the field closer to a truly general-purpose visual assistant.
Key Foundational Work
Visual Prompting via Image Inpainting (2022) — contributed 3 concept elements to this direction.
Images Speak in Images: A Generalist Painter for In-Context Visual Learning (2023) — contributed 2 concept elements to this direction.
Sequential Modeling Enables Scalable Learning for Large Vision Models (2023) — contributed 2 concept elements to this direction.