In natural language processing, Large Language Models (LLMs) changed the paradigm from "fine-tune for every task" to "prompt for any task." By providing a few examples in a text prompt, a model like GPT-4 can infer a new rule and apply it immediately. Researchers are now bringing this same "in-context learning" (ICL) capability to computer vision, enabling a single model to solve tasks ranging from medical image segmentation to weather forecasting without a single weight update.
The Visual Sentence
The core intuition behind visual in-context learning is to treat computer vision tasks as a sequence of "visual sentences." In a typical language prompt, you might say: "Apple -> Red; Banana -> Yellow; Cherry -> ?" The model completes the pattern with "Red."
In the vision domain, researchers like Yutong Bai and Amir Bar have demonstrated that images, videos, and even annotations can be tokenized into a continuous stream. Their Large Vision Model (LVM) Bai et al. treats a task as an analogy: an example pair (Input Image A, Output Image A') is followed by a query (Input Image B). The model’s job is to predict the next tokens in the sequence, effectively generating Output Image B'.
This framing allows a model to solve diverse tasks—like edge detection, depth estimation, or object segmentation—simply by "reading" the transformation logic from the provided examples. Yuchao Gu and colleagues further extended this in UniVid Gu et al., showing that video generation models can serve as universal backbones for these visual sentences because they are already pre-trained to understand how pixels evolve over time.
Vision as Inpainting
If a model can fill in missing parts of an image, it can solve any vision task. This is the insight behind Painter, developed by Xinlong Wang and Chunhua Shen Wang et al..
Painter unifies diverse task outputs—such as heatmaps for keypoint detection or 3-channel masks for segmentation—into a standard image format. During inference, the model is given a large composite image containing an example pair and a query image with a blank space. By "inpainting" the missing pixels, the model performs the task. Because the model learns to coordinate pixels based on the provided context, it can generalize to entirely unseen object categories or artistic styles just by looking at the example pair.
Generative and Structural Priors
While early models focused on simple pixel completion, newer approaches leverage the power of diffusion models to handle more complex, high-fidelity transformations.
- Context Diffusion Sinha et al. improves this by separating the layout of a scene from its visual style. This allows the model to generate new images that follow a specific style or object identity provided in the context, even without explicit text instructions.
- X-Prompt Sun et al. uses learnable tokens to compress visual context, allowing auto-regressive models to perform dense predictions (like segmentation) and image editing (like object removal) within the same framework.
- US-Diffusion Ji et al. introduces structural priors to these diffusion pipelines, significantly speeding up the inference process while maintaining the ability to generalize to new tasks in a zero-shot manner.
Reasoning Beyond Pixels
Interestingly, visual in-context learning isn't limited to raw pixels. Mu Cai and Yong Jae Lee explored using Scalable Vector Graphics (SVG) as a bridge Cai et al.. By representing images as code, they found that LLMs like GPT-4 could perform "visual analogy prompting." The model identifies the delta between two vector states (e.g., a circle moving left) and applies that same code transformation to a new object.
This flexibility has led to specialized generalist models in fields like meteorology and medicine. WeatherGFM Zhao et al. applies ICL to weather data, allowing a single model to switch between forecasting and satellite-to-radar translation. Similarly, Neuralizer Dalca and Czolbe allows clinicians to perform neuroimage analysis on new modalities by simply providing a few example slices, eliminating the need for expensive task-specific retraining.
Why It Works: Latent Rule Inference
The success of these models suggests that large-scale training on sequential or paired visual data teaches the model a "meta-skill": the ability to infer latent rules from examples. Whether it is a spatial boundary in a reasoning task Dirac et al. or a geometric registration in 3D point clouds Jiang et al., the model isn't just memorizing tasks. It is learning to map the relationship between $A \to A'$ and applying that mapping to $B$.
As Fei Zhang noted in the ConText framework Zhang et al., these models even begin to exhibit emergent behaviors, such as following human-drawn visual hints (like circles or boxes) to define task boundaries, even when such markers weren't explicitly part of the training set. This move toward flexible, context-aware vision marks a shift away from rigid architectures toward models that can truly "see" and adapt on the fly.