Imagine you are looking at a photo of a crowded farmers' market with a friend. Your friend points toward a table and asks, "How much is that one?" In isolation, the phrase "that one" is completely meaningless. To understand the question, your brain instantly combines two streams of information: the linguistic context of the conversation and the visual context of the physical scene.
When artificial intelligence attempts to have a conversation about the visual world, it encounters this exact problem. In multi-turn dialogues, long narrations, or multi-image documents, language is filled with ambiguous pronouns ("it", "they"), deictic expressions ("this one", "that building"), and underspecified nouns. Multimodal Coreference Resolution (MCR) is the task of identifying when different linguistic expressions and visual regions refer to the same underlying entity.
Without MCR, models treat every dialogue turn or sentence as an isolated island, leading to a complete breakdown in tracking context. Understanding why and how MCR works requires looking at how modern architectures bridge the gap between evolving textual histories and static or sequential visual data.
Why Text-Only and Standard Vision Models Fail
Traditional natural language processing systems are highly adept at text-only coreference resolution. If a text reads, "The doctor walked into the room. She sat down," a model can easily link "She" to "doctor". However, as shown by the authors of Who are you referring to? Coreference resolution in image narrations, text-only models fail completely when an image contains multiple instances of the same entity type—such as two female doctors in the same room. Without visual grounding, linguistic cues are fundamentally insufficient to resolve the ambiguity.
Conversely, standard Visual Question Answering (VQA) models are built for single-turn, explicit queries (e.g., "What color is the car?"). When forced into a multi-turn conversation, these models struggle because they lack an explicit mechanism to track how references evolve over time. As noted by Satwik Kottur et al. in Visual Coreference Resolution in Visual Dialog Using Neural Module Networks, relying on implicit memory states or bag-of-words representations turns the model's reasoning into an uninterpretable black box, making it impossible to verify if the model has truly linked a pronoun to a visual entity or is simply exploiting statistical patterns.
The "Refer then Find" Paradigm
One of the most intuitive breakthroughs in solving MCR is the decoupling of linguistic resolution from visual grounding. When a model encounters a query like "What color is it?", trying to map the pronoun "it" directly to a complex image containing dozens of objects creates an incredibly noisy attention map.
To solve this, Gi-Cheon Kang et al. introduced Dual Attention Networks in Dual Attention Networks for Visual Reference Resolution in Visual Dialog. The core intuition is a two-step cognitive process:
- REFER: The model first looks backward into the textual dialogue history to resolve the linguistic ambiguity. It determines that "it" refers to, for example, "the backpack mentioned in turn 2".
- FIND: Armed with this clarified textual description, the model then looks at the image to ground the phrase.
By separating the problem into sequential steps, the visual search space is drastically reduced. The model is no longer searching the image for a vague pronoun; it is searching for a specific, resolved noun phrase.
[Ambiguous Query: "What color is it?"]
│
▼
┌─────────────────────────────┐
│ 1. REFER (Text History) │ ──► Resolves "it" to "the backpack"
└─────────────────────────────┘
│
▼
┌─────────────────────────────┐
│ 2. FIND (Visual Grounding) │ ──► Locates "the backpack" bounding box
└─────────────────────────────┘
Tracking Context via Graphs and Recursion
As conversations grow longer, an antecedent might be buried several turns back in the dialogue history. Models need a way to dynamically trace these threads without getting overwhelmed by historical noise.
Recursive Visual Attention
The authors of Recursive Visual Attention in Visual Dialog proposed a mechanism that executes a search query through dialogue history only when the current question is detected to be ambiguous. It builds a recursion tree, shifting its attention back through previous turns until it finds the target entity, and then maps that target directly to its corresponding spatial region in the image.
Context-Aware Graph Inference
Instead of treating dialogue as a linear sequence, another powerful approach models the interaction as a dynamic graph. In Iterative Context-Aware Graph Inference for Visual Dialog, Dan Guo 0001 et al. demonstrated how to represent fine-grained visual objects and textual tokens as nodes in a shared graph. Through iterative message passing, the features of a visual object node are continuously updated by the textual nodes that describe it across multiple turns. This prevents "identity fragmentation"—ensuring that even if an object is called "the car" in turn 1, "it" in turn 2, and "the vehicle" in turn 3, the model maintains a single, unified representation of that visual entity.
Scaling to Multi-Image and Interleaved Contexts
The challenge of MCR intensifies when moving from a single image to environments containing multiple images or interleaved text-and-image sequences, such as photo albums, virtual shopping interfaces, or scientific documents.
In a multi-image setting, users frequently employ cross-image references (e.g., "the second photo" or "the same shirt from earlier"). As Satwik Kottur et al. identified in Navigating Connected Memories with a Task-oriented Dialog System, models frequently fail because they do not maintain a persistent history of visual embeddings alongside text embeddings, causing them to lose track of entities once they disappear from the immediate screen view.
To anchor references across interleaved multi-modal prompts, Haozhe Zhao introduced an "Image Declaration" scheme in MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning. By assigning unique, explicit proxy tokens to individual images within a prompt, the architecture introduces a structured syntax that allows the underlying language model to treat images as discrete, addressable entities. When the text later refers to "this product," the proxy tokens act as structural anchors, resolving cross-image referential ambiguity and ensuring the model grounds its reasoning in the correct visual asset.
Similarly, in specialized domains like scientific literature, cross-modal coreference acts as a bridge between disparate document elements. As shown by Vincent Fan et al. in OpenChemIE: An Information Extraction Toolkit For Chemistry Literature, resolving coreferences between textual compound labels in a paragraph and molecular structures inside a diagram allows a system to successfully aggregate information that is distributed across different modalities within the same document.
Summary of Core Datasets and Benchmarks
The evolution of MCR has been heavily driven by diagnostic datasets designed to isolate coreference failures from general language understanding:
- Flickr30k Entities: A foundational dataset introduced by Bryan A. Plummer et al. that explicitly maps noun phrases in captions to bounding boxes, establishing the baseline for coreference chains.
- CLEVR-Dialog: A massive diagnostic dataset by Satwik Kottur et al. utilizing synthetic scenes with perfect scene graphs to pinpoint exactly how models fail at tracking object attributes and counts across multiple rounds.
- SIMMC (Situated Interactive MultiModal Conversations): A framework by Seungwhan Moon et al. grounded in virtual shopping assistants, requiring models to map ambiguous user intents (e.g., "the blue one") directly to database object identifiers using visual metadata.
Key Foundational Work
The Representation of Multimodal User Interface Dialogues Using Discourse Pegs (1992) — contributed 1 concept elements to this direction.
Understanding Natural Language with Diagrams (1990) — contributed 1 concept elements to this direction.