Expressing Objects Just Like Words: Recurrent Visual Embedding for Image-Text Matching

Tianlang Chen, Jiebo LuoAAAI 2020 · DOI

This research, presented at AAAI-20, introduces a novel framework for bridging the semantic gap between visual and textual modalities. By treating image objects as a sequence—analogous to words in a sentence—the authors demonstrate a significant advancement in image-text matching accuracy.

The Contextual Challenge in Image-Text Matching

In image-text retrieval tasks, such as searching for an image using a natural language query, state-of-the-art models typically rely on "bottom-up attention." These models identify individual objects (e.g., a person, a dog, a frisbee) and calculate their independent similarity to words in a sentence.

However, this "bag of objects" approach often fails to capture complex semantic relationships. For example, a query like "two people riding skis together" requires understanding the joint relationship between the people and the equipment. A standard model might assign a high score to an image containing two separate people and skis on opposite sides of a mountain because it lacks the mechanism to process the "togetherness" or the specific flow of the scene. The Dual Path Recurrent Neural Network (DP-RNN) addresses this by processing objects sequentially to capture inter-object context.

The DP-RNN Architecture

The DP-RNN model moves beyond independent object detection by introducing a recurrent lens to visual processing. The architecture consists of four primary components:

  1. Recurrent Visual Embedding: This core innovation reorders visual detections. For every object detected via Faster R-CNN, the model identifies its most related word in the text query and sorts the objects based on the positions of those words in the sentence.
  2. Bi-Directional GRU: The reordered object features are fed into a bi-directional Gated Recurrent Unit (GRU). This allows the model to extract high-level features representing the "flow" of the image, capturing how objects relate to one another within the specific context of the query.
  3. Multi-attention Cross Matching: The model employs a sophisticated attention mechanism that evaluates both word-to-image and object-to-text similarities, using self-attention to weight the importance of specific words or objects.
  4. Pair Early-Selection Strategy: To manage the $O(s^2)$ computational complexity of reordering objects for every possible pair, the authors implement a fast, simplified matching score to filter out irrelevant pairs before applying the more intensive recurrent processing to "hard" candidates.
A comparison between the SCAN baseline and DP-RNN, showing how DP-RNN tracks image-word similarity across a sequence of specific object crops to better match the query.

Key Insights: Dynamic Visual Grammar

A striking insight of this work is that images do not have a fixed sequence; rather, their "grammar" is dynamic and defined by the narrative of the text query. By passing objects through a GRU, the representation of an object (e.g., a "person") is refined based on previously processed objects (e.g., "skis"). This mimics the human cognitive process of scanning an image to verify a description.

As shown in the comparison below, the DP-RNN maintains a more consistent and contextually aware similarity score across the sequence of words compared to baseline models like SCAN. By mapping both modalities into a shared embedding space where object sequences and word sequences are mathematically comparable, the model achieves a deeper level of cross-modal alignment.

A comparison between the SCAN baseline and DP-RNN, showing how DP-RNN tracks image-word similarity across a sequence of specific object crops to better match the query.

Performance and Results

The DP-RNN was evaluated on the MS-COCO and Flickr30K benchmarks, achieving state-of-the-art results:

  • Flickr30K: The model achieved a Recall@1 (R@1) of 70.2% for sentence retrieval and 55.5% for image retrieval, outperforming previous leaders like SCAN and PFAN.
  • MS-COCO (1K test): The R@1 for sentence retrieval improved to 75.3%, a notable increase over the 72.7% baseline.
  • Qualitative Success: The model proved particularly adept at handling "tricky" semantic relationships, such as distinguishing between "a man standing near a boat" and "a man parking a boat."

Limitations and Future Directions

Despite its success, the model remains dependent on the quality of the initial object detector; if a crucial object is missed by the Faster R-CNN, the recurrent chain is broken. Additionally, the recurrent processing adds computational overhead compared to non-recurrent baselines. Future work may explore more complex ordering mechanisms, such as 3D spatial relationships or hierarchical scene graphs, to further refine the visual sequence.

Summary updated

BibTeX
@inproceedings{chen2020expressing,
  author = {Tianlang Chen and Jiebo Luo},
  title = {{Expressing Objects Just Like Words: Recurrent Visual Embedding for Image-Text Matching}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6631},
}
Figure