Visual In-Context Learning via Image Inpainting

73 concepts54 papers1 proposal

In natural language processing, large language models (LLMs) exhibit "in-context learning"—the ability to solve a new task simply by observing a few examples in a text prompt. Traditionally, computer vision has lacked this flexibility, requiring expensive fine-tuning or specialized architectures for every new task, from segmentation to depth estimation.

Recent research has introduced a paradigm shift: Visual In-Context Learning (V-ICL). By framing diverse vision tasks as image-to-image completion problems, models can now adapt to novel tasks at inference time using only visual examples.

The Unified Image-to-Image Interface

The core intuition behind V-ICL is that most vision tasks can be reformulated as a transformation from one image to another. For example, semantic segmentation is a mapping from an RGB image to a category-colored mask, and edge detection is a mapping to a sketch.

Wang et al. et al. proposed Painter (Images Speak in Images), a generalist model that treats these diverse tasks as a simple "inpainting" problem. Instead of using task-specific heads, the model is trained to predict missing pixels in an output image given an input image. By providing a "visual prompt"—a pair of images showing an example transformation—the model identifies the underlying logic and applies it to a new query image without any weight updates.

This approach was further refined in SegGPT (SegGPT: Towards Segmenting Everything In Context), where Wang et al. et al. demonstrated that segmentation could be treated as a "coloring" problem. By attending to the context provided in the prompt, the model can segment novel object categories or follow video targets zero-shot.

Task Specification as Visual Analogy

How does a model know which task to perform? V-ICL relies on the concept of visual analogy completion.

Bar et al. et al. introduced a grid-based prompting mechanism (Visual Prompting via Image Inpainting). By arranging an example input-output pair alongside a query image in a $2 \times 2$ grid, the model is forced to "fill in the blank" for the fourth quadrant. This structure allows the model to infer the intended transformation—such as "find the edges" or "segment the cat"—purely from the spatial relationship between the provided images.

Dedhia et al. et al. argue in Im-Promptu (Im-Promptu: In-Context Composition from Image Prompts) that for this to work, the model must move beyond pixel-level processing and view the world as a collection of objects. This allows the model to recognize the latent rule governing the transformation in the prompt and compose a new scene that follows that same rule.

Scaling Vision like Language

To achieve robust in-context abilities, models require massive pre-training on sequential data. Bai et al. and Geng developed a Large Vision Model (LVM) (Sequential Modeling Enables Scalable Learning for Large Vision Models) that treats images and annotations as "visual sentences." By training on 420 billion tokens using a next-token prediction objective, the model learns to complete sequences of visual examples, enabling zero-shot generalization to tasks like 3D reasoning and edge detection.

Similarly, Yang et al. et al. suggest that video can serve as the unified interface for real-world decision-making (Video as the New Language for Real-World Decision Making). In this view, vision tasks are reformulated as next-frame predictions, where the "prompt" is the history of the scene and the "output" is the future state or the desired transformation.

Generalizing to Specialized Domains

The power of V-ICL lies in its ability to handle "out-of-distribution" data where labels are scarce.

By treating vision as a pattern-completion task, V-ICL bypasses the need for task-specific engineering, moving the field closer to a truly general-purpose visual assistant.

Key Foundational Work

Visual Prompting via Image Inpainting (2022) — contributed 3 concept elements to this direction.
Images Speak in Images: A Generalist Painter for In-Context Visual Learning (2023) — contributed 2 concept elements to this direction.
Sequential Modeling Enables Scalable Learning for Large Vision Models (2023) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_visual_in_context_learning_3cf16587,
  title = {visual in-context learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/visual-in-context-learning/rdr_3cf165870bae417e8c018f5f093a7bd4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_visual_in_context_learning_3cf16587_v1,
  title = {visual in-context learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/visual-in-context-learning/rdr_3cf165870bae417e8c018f5f093a7bd4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake