Multimodal referring expression comprehension based on image and text: a review

Lian Wang, Peihan Miao, Wei Su, Xi Li, Naye Ji, Yanbing JiangJournal of Image and Graphics 2023 · DOI

Abstract: Referring Expression Comprehension (REC) is a multimodal task at the intersection of computer vision and natural language processing. It aims to uniquely locate a target object in an image based on a natural language description. This review provides a comprehensive taxonomy of REC methods, categorizing them by visual representation granularity: region-based, grid-based, and patch-based. We examine the evolution from complex two-stage pipelines to efficient one-stage Transformer architectures, discuss mainstream benchmarks like RefCOCO, and highlight future challenges in reasoning, interpretability, and real-time performance.

Introduction

Referring Expression Comprehension (REC) bridges the semantic gap between human language and visual content. Unlike standard object detection, which identifies general categories, REC identifies a specific instance based on attributes, spatial relationships, or actions (e.g., "the small white dog sitting to the left of the red car"). This capability is foundational for advanced AI systems, including autonomous robotics, augmented reality, and interactive dialogue systems.

Methodology and Taxonomy

The core challenge of REC is twofold: achieving deep semantic understanding of complex expressions and performing relational reasoning across visual objects. The field has evolved through three primary approaches to visual representation.

Visual Representation Granularity

Methods are categorized by how they "see" and process the image:

  1. Regional Convolution Granularity (Two-stage): These methods use pre-trained detectors (like Faster R-CNN) to generate object proposals. A second stage then ranks these proposals against the text description. While accurate, they are computationally expensive and limited by the initial detector's performance.
  2. Grid Convolution Granularity (One-stage): These bypass proposal generation by treating the image as a grid of features. Models like FAOA and RCCF map text directly to spatial coordinates, achieving near real-time speeds (up to 40 FPS).
  3. Image Patch Granularity (ViT-style): Inspired by Vision Transformers (ViT), these methods divide the image into fixed-size patches. This allows for a unified Transformer architecture where both image patches and text tokens are processed in a shared multimodal space.
Visual representation granularities: (a) Region-based, (b) Grid-based, and (c) Patch-based.

Taxonomy of REC Methods

The evolution of REC can be traced through various fusion strategies. Early methods relied on modular networks or attention mechanisms to link modalities. Modern approaches have shifted toward Transformer-based fusion, which allows for dynamic "cross-talk" between visual and linguistic features, leading to superior alignment.

Comprehensive taxonomy of REC methods categorized by visual granularity and fusion strategies.

Datasets and Performance

Standard benchmarks include ReferItGame, RefCOCO, RefCOCO+, and RefCOCOg. These vary in complexity; for instance, RefCOCO+ forbids location words, forcing models to rely on appearance, while RefCOCOg features long, complex descriptions.

Key results in the field show a clear trend:

  • State-of-the-Art: Models like OFA (One For All) currently lead benchmarks, achieving over 90% accuracy on RefCOCO by leveraging massive multimodal pre-training.
  • Efficiency: One-stage models like RCCF have reduced inference time from ~320ms (two-stage) to ~25ms, making them viable for real-world applications.
  • Architectural Gains: Transitioning from grid-based to patch-based Transformers (e.g., TransVG++) has yielded accuracy improvements of 5% to 10%.

Key Insights and Future Directions

Despite rapid progress, several challenges remain:

  • Reasoning vs. Overfitting: Many models overfit to simple dataset biases. Performance often drops on complex datasets like Cops-Ref, suggesting models still struggle with true multi-step logical reasoning.
  • Interpretability: Most models function as "black boxes." Future work must improve visualization of the model's internal reasoning process.
  • Dimensional Expansion: The field is expanding beyond 2D images into Video REC (tracking objects over time) and 3D REC (locating objects in point clouds for robotics).
  • Weak Supervision: Reducing the reliance on expensive, manually labeled bounding boxes through weakly supervised learning remains a major research goal.

Summary updated

Abstract

指代表达理解(referring expression comprehension,REC)作为视觉—语言相结合的多模态任务,旨在理解输入指代表达式的内容并在图像中定位其所描述的目标对象,受到计算机视觉和自然语言处理两个领域的关注。REC任务建立了人类语言与物理世界的视觉内容之间的桥梁,可以广泛应用于视觉理解系统和对话系统等人工智能设备中。解决该任务的关键在于对复杂的指代表达式进行充分的语义理解;然后利用语义信息对包含多个对象的图像进行关系推理以及对象筛选,最终在图像中唯一地定位目标对象。本文从计算机视觉的视角出发对REC任务进行了综述,首先介绍该任务的通用处理流程。然后,重点对REC领域现有方法进行分类总结,根据视觉数据表征粒度的不同,划分为基于区域卷积粒度视觉表征、基于网格卷积粒度视觉表征以及基于图像块粒度视觉表征的方法;并进一步按照视觉—文本特征融合模块的建模方式进行了更细粒度的归类。此外,本文还介绍了该任务的主流数据集和评估指标。最后,从模型的推理速度、模型的可解释性以及模型对表达式的推理能力3个方面揭示了现有方法面临的挑战,并对REC的发展进行了全面展望。本文希望通过对REC任务现有研究以及未来趋势的总结为相关领域研究人员提供一个全面的参考以及探索的方向。
BibTeX
@article{wang2023multimodal,
  author = {Lian Wang and Peihan Miao and Wei Su and Xi Li and Naye Ji and Yanbing Jiang},
  title = {{Multimodal referring expression comprehension based on image and text: a review}},
  journal = {Journal of Image and Graphics},
  year = {2023},
  doi = {10.11834/jig.221024},
}
Figure