Bridging the Gap between Expression and Scene Text for Referring Expression Comprehension (Student Abstract)

Yuqi Bu, Jiayuan Xie, Liuwu Li, Qiong Liu, Yi CaiAAAI 2022 · DOI

Abstract

Referring Expression Comprehension (REC) involves pinpointing specific objects in an image based on natural language descriptions. Despite the prevalence of scene text (e.g., brand names, jersey numbers) as distinguishing features in human communication, current REC models often treat these as raw pixels rather than semantic information. This paper introduces a novel framework that utilizes Optical Character Recognition (OCR) to extract scene text and bind it to specific visual regions. By aligning these "text-bound" features with user expressions, the proposed model bridges the semantic gap between textual queries and visual data. Experimental results on the RefCOCO and RefCOCOg benchmarks show that our method outperforms existing state-of-the-art one-stage models, particularly in scenarios where textual identifiers are key to disambiguation.

Introduction

The task of Referring Expression Comprehension (REC) requires a model to ground a natural language phrase (e.g., "the man in the red hat") to a specific bounding box in an image. While standard models effectively align visual attributes like color and shape, they frequently struggle when the distinguishing feature is text within the scene. For instance, identifying "the Burger King" in a crowded food court or "runner number 1" in a marathon is trivial for humans but difficult for models that do not "read" the image.

Scene text like 'Burger King' serves as a crucial semantic anchor for object localization in complex environments.

Existing methods typically treat scene text as raw visual patterns. This creates a semantic mismatch: the user's query contains high-level concepts (words), while the model only sees low-level pixels. This paper addresses this gap by explicitly incorporating OCR-derived semantic information into the visual representation.

Methodology

The proposed framework consists of a three-stage pipeline designed to integrate scene text into the REC process:

  1. Feature Extraction and OCR: The system employs a one-stage visual encoder for grid-based feature extraction. Simultaneously, an OCR engine identifies text strings and their spatial coordinates within the image.
  2. Scene Text Binding Module: This core component maps extracted text to its corresponding spatial grid. When text is detected within a region, its high-level semantic features are fused with the local visual features (e.g., color and shape), creating a multi-modal representation of that region.
  3. Expression-Guided Alignment: The model uses an iterative process to align the user's input expression with these enriched visual features. This allows the model to refine its focus and accurately localize the target object by matching the query's text directly to the bound scene text.

Experimental Results

The model was evaluated on two standard benchmarks: RefCOCO, which focuses on shorter expressions for people and objects, and RefCOCOg, which contains longer, more descriptive phrases.

Quantitative comparison of the proposed method against state-of-the-art one-stage REC models on RefCOCO and RefCOCOg.

As shown in the results table, our approach consistently outperformed baselines such as SSG, 1-BERT, and ReSC. On RefCOCO testA (focused on people), the model achieved 79.18% accuracy. The improvement was even more pronounced on RefCOCOg, reaching 66.04% accuracy, a ~2% increase over the ReSC baseline. These gains suggest that as expressions become more complex, the ability to anchor them to scene text becomes increasingly critical.

Conclusion

This research demonstrates that treating scene text as a semantic anchor significantly enhances the performance of REC models. By bridging the gap between the user's textual expression and the text embedded within the visual scene, the proposed framework provides a more robust way to identify objects in man-made environments. Future work will focus on handling more complex interactions between multiple text instances and improving robustness against OCR errors.

Summary updated

BibTeX
@inproceedings{bu2022bridging,
  author = {Yuqi Bu and Jiayuan Xie and Liuwu Li and Qiong Liu and Yi Cai},
  title = {{Bridging the Gap between Expression and Scene Text for Referring Expression Comprehension (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i11.21597},
}
Figure