M^3EL: A Multi-task Multi-topic Dataset for Multi-modal Entity Linking

Fang Wang, Shenglin Yin, Xiaoying Bai, Minghao Hu, Tianwei Yan, Yi LiangAAAI 2025 · OpenReview

TL;DR

Researchers have introduced M³EL, a large-scale, multi-topic, multi-task dataset for multi-modal entity linking (MEL) to overcome the limitations of existing datasets. They also propose a modality-augmented training strategy that significantly improves MEL model performance, demonstrating the dataset's value as a pre-training corpus.

What Problem Does This Paper Address?

Multi-modal Entity Linking (MEL) is crucial for integrating information across different modalities like text and images, aiming to link mentions to entities in a knowledge graph. However, current MEL datasets are often too small, lack diverse topic coverage, and don't support a wide enough range of multi-modal tasks. This limits the ability of current models to generalize and perform effectively in real-world scenarios. The authors highlight that existing models struggle with these limitations, showing suboptimal performance on their new, more comprehensive dataset.

Methods

The paper introduces a novel dataset construction pipeline for M³EL. Key aspects include:

  • Data Collection: Leveraging diverse Kaggle datasets and specific sources like Wikipedia Movie Plots, Wiki-Wiki, People Wikipedia Data, Books Datasets, and AIDA-CoNLL-Test for different topics.
  • Data Filtering and Cleaning: Removing non-English expressions, single-character mentions, and instances with unretrievable images.
  • Mention Processing: Extracting mentions and contextual information from various sources, including using GPT-4 for semantic understanding on common topic data.
  • Image Crawling: Employing a framework that uses Wikidata, Wikipedia, and Wikimedia to find relevant images, enhanced by a pre-trained CLIP model for filtering.
  • Candidate Construction: A crucial step where for each mention, a refined subset of potential entities (candidates) is generated from Wikidata. This mimics real-world challenges where the correct entity might not be the most obvious one.
  • Modality-Augmented Training Strategy: This novel strategy augments the standard CLIP training by also augmenting textual data with entity description information. This aims to improve the model's handling of semantic ambiguities and enhance its overall performance on multi-modal tasks.

Datasets

The primary contribution is the introduction of the M³EL dataset, which features:

  • Scale: 79,625 instances and 318.5K images.
  • Topics: 5 diverse topics (movies, common knowledge, person, books, sports).
  • Tasks: 9 distinct multi-modal linking tasks, covering various combinations of text and image inputs and outputs (e.g., Tm-Te, Tm-Ie, Im-Te, (I+T)m-(I+T)e).

The authors also use other datasets for comparison and benchmarking, including:

  • DiffusionDB (for Im-Te tasks)
  • WIKIPerson (for Im-Te, Im-Ie, and Im-(I+T)e tasks)
  • WIKIDiverse (for Tm-Te, Im-Ie, (I+T)m-Ie, and (I+T)m-(I+T)e tasks)

Key Insights and Ideas

  • Dataset Scale Matters: Existing MEL datasets are too small to effectively train robust multi-modal models. M³EL's significantly larger scale is a critical step forward.
  • Task Diversity is Key: Covering a wider range of multi-modal task combinations (9 vs. fewer in prior work) is essential for models to generalize across different MEL scenarios.
  • Text Augmentation Enhances CLIP: Augmenting textual inputs in CLIP-like models, beyond just image augmentation, can significantly improve performance by better capturing semantic nuances and handling ambiguity.
  • Candidate Set Realism: The method of generating candidate entity sets, which may not always place the correct entity first, better reflects the practical challenges of MEL.
  • Surface Form Mismatch is a Challenge: A significant percentage of mentions (41.3%) have surface forms that differ from their ground-truth entities, highlighting the need for robust linking mechanisms.

Key Results

  • Existing MEL models perform poorly on M³EL, with accuracies ranging from 49.4% to 75.8%. This underscores the limitations of current datasets and models.
  • The proposed CLIPND model, trained with the modality-augmented strategy on M³EL, shows significant improvements, with an average accuracy increase of 9.3% to 25% across various tasks compared to baseline models.
  • The CLIPND model achieved state-of-the-art performance on M³EL, reaching up to 90.2% accuracy in some configurations.
  • The study found that using both mention names and descriptions as textual context leads to optimal performance, while using only sentences can sometimes be better than full context due to noise in longer texts.

Limitations and Future Work

  • Limitations: The authors acknowledge that while M³EL is large, it still presents a challenging benchmark, and further improvements are needed. They also note that some models perform worse on datasets with more ambiguous textual information.
  • Future Work: The authors plan to continue updating M³EL. They aim to explore more complex real-world tasks, such as dynamic MEL in audio or video, and investigate the integration of M³EL with large language models.

Summary updated

Abstract

Multi-modal Entity Linking (MEL) is a fundamental component for various downstream tasks. However, existing MEL datasets suffer from small scale, scarcity of topic types and limited coverage of tasks, making them incapable of effectively enhancing the entity linking capabilities of multi-modal models. To address these obstacles, we propose a dataset construction pipeline and publish M^3EL, a large-scale dataset for MEL. M^3EL includes 79,625 instances, covering 9 diverse multi-modal tasks, and 5 different topics. In addition, to further improve the model's adaptability to multi-modal tasks, We propose a modality-augmented training strategy. Utilizing M^3EL as a corpus, train the CLIP_ND model based on CLIP (ViT-B-32), and conduct a comparative analysis with an existing multi-modal baselines. Experimental results show that the existing models perform far below expectations (ACC of 49.4%-75.8%), After analysis, it was obtained that small dataset sizes, insufficient modality task coverage, and limited topic diversity resulted in poor generalization of multi-modal models. Our dataset effectively addresses these issues, and the CLIP_ND model fine-tuned with M^3EL shows a significant improvement in accuracy, with an average improvement of 9.3% to 25% across various tasks. Our dataset publicly available to facilitate future research.
BibTeX
@inproceedings{wang2025m,
  author = {Fang Wang and Shenglin Yin and Xiaoying Bai and Minghao Hu and Tianwei Yan and Yi Liang},
  title = {{M^3EL: A Multi-task Multi-topic Dataset for Multi-modal Entity Linking}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=Mw1dRcgCu6},
}
Figure