Retrieval over Classification: Integrating Relation Semantics for Multimodal Relation Extraction

Lei Hei, Tingjing Liao, Peiyingxin, Yiyang Qi, Jiaqi Wang, Ruiting Li, Feiliang RenEMNLP 2025

TL;DR

This paper introduces ROC, a novel framework for multimodal relation extraction that shifts from traditional classification to a retrieval-based approach. By leveraging natural language descriptions of relations and a type-aware encoder, ROC achieves state-of-the-art performance and offers improved interpretability and robustness.

What Problem Does This Paper Address?

Relation Extraction (RE) is crucial for transforming unstructured text into structured data, powering applications like knowledge graphs. While recent work has extended RE to multimodal scenarios (text + images), existing methods primarily rely on classification. This approach has two main drawbacks:

  1. Limited Semantic Expressiveness: Discrete labels struggle to capture nuanced semantic differences between similar relations (e.g., "Peer" vs. "Couple").
  2. Neglect of Structural Constraints: Classification models often overlook important structural cues like entity types and positions, which are vital for accurate relation inference.

Multimodal RE faces additional challenges due to the inherent ambiguity of natural language and the need to jointly reason over visual and textual information. Existing multimodal approaches, while improving fusion and alignment, still fall back to classification, inheriting its limitations.

Methods

The core innovation of ROC is its reformulation of multimodal relation extraction as a retrieval task driven by relation semantics, moving away from traditional classification.

  • Type-Aware Multimodal Entity Pair Encoder: This component integrates explicit entity type information (obtained via Stanford NER) and positional cues. These structural constraints are embedded as "semantic prompts" to guide the model. A Transformer-based encoder then fuses textual and visual features to create a rich representation of the entity pair.
  • Relation Semantic Encoder: Instead of discrete labels, relation types are represented by natural language descriptions generated using GPT-40 and manually verified. These descriptions are then encoded into a unified semantic space by an independent BERT encoder. This allows for finer-grained semantic modeling and better differentiation between similar relations.
  • Contrastive Semantic Retrieval Strategy: The model is trained using a contrastive learning approach. It aims to maximize the similarity between an entity pair's representation and its corresponding relation description (positive sample) while minimizing similarity to other relation descriptions in the batch (negative samples). This aligns entity pairs with relation semantics in a shared space.
  • Retrieval-based Inference: During inference, the model calculates the similarity between the entity pair representation and all relation semantic embeddings, selecting the relation with the highest similarity. This makes the process more interpretable and flexible, as it doesn't rely on fixed category labels.

Datasets

The paper evaluates ROC on two widely used multimodal relation extraction datasets:

  • MNRE (Zheng et al., 2021b): Focuses on social media contexts and object-entity relation extraction.
  • MORE (He et al., 2023): A cross-modal dataset specifically designed for object-entity relation extraction.

Key Insights and Ideas

  • Retrieval over Classification is a Paradigm Shift: Moving from classification to retrieval for multimodal RE is a significant departure that unlocks better semantic understanding and robustness.
  • Natural Language Descriptions Enhance Expressiveness: Using GPT-40 to generate descriptive sentences for relations, rather than just labels, allows the model to capture subtle semantic nuances and distinguish between similar relations more effectively.
  • Structural Constraints are Crucial: Incorporating entity types and positions as "semantic prompts" significantly constrains the search space, leading to more accurate and efficient relation prediction.
  • Transformer Architectures Excel in Multimodal Fusion: The study highlights that Transformer-based visual encoders (like ViT) are more suitable for multimodal fusion than CNN-based encoders (like ResNet) due to their architectural consistency with text encoders, facilitating better semantic alignment.
  • Dataset Characteristics Matter for Model Design: The optimal depth of the encoder depends on the dataset's image-text alignment. Loosely aligned data benefits from shallower fusion, while tightly aligned data benefits from deeper, more complex interactions.

Key Results

  • State-of-the-Art Performance: ROC achieves state-of-the-art results on both the MNRE and MORE datasets, particularly excelling in F1 scores.
  • MNRE: ROC achieved an F1 score of 91.22%, outperforming other methods in recall and F1, despite a slightly lower accuracy than CAMRE.
  • MORE: ROC achieved an F1 score of 71.97%, outperforming the second-best model APOLLO by 2.95 percentage points.
  • Robustness: The retrieval-based approach demonstrated stronger robustness and interpretability compared to classification-based methods.
  • Ablation Studies: Removing components like the entity encoder, position encoding, or relation embedding led to significant performance drops, confirming the importance of each module. Notably, removing the relation semantic encoder resulted in a substantial decrease in F1 score on the MORE dataset.
  • Comparison with MLLMs: ROC outperforms large multimodal language models (MLLMs) in F1 scores on both datasets, demonstrating a more balanced performance between precision and recall, and offering superior practicality due to its lightweight design.

Limitations and Future Work

  • Zero-shot MLLMs: The paper notes that zero-shot multimodal LLMs struggle with this task.
  • Fine-tuned LLMs and Description Reformulation: The authors plan to systematically verify if fine-tuned LLMs can significantly enhance task performance when class labels are reformulated as natural language descriptions.
  • Generalizability: The generalizability of replacing discrete labels with semantic descriptions across diverse domains, tasks, and datasets needs further evaluation.
  • Future Work: The paper suggests exploring prompt templates for extending natural language descriptions and investigating the impact of different encoder depths and architectures. The authors also highlight the potential for further research into the interpretability of attention mechanisms in multimodal relation extraction.

Summary updated

BibTeX
@inproceedings{hei2025retrieval,
  author = {Lei Hei and Tingjing Liao and Peiyingxin and Yiyang Qi and Jiaqi Wang and Ruiting Li and Feiliang Ren},
  title = {{Retrieval over Classification: Integrating Relation Semantics for Multimodal Relation Extraction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure