MEL: A Multi-task Multi-topic Dataset for Multi-modal Entity Linking

Fang Wang, Shenglin Yin, Xiaoying Bai, Minghao Hu, Tianwei Yan, Yi LiangarXiv 2024 · OpenReview

TL;DR

This paper introduces $M^3EL$, a large-scale, multi-task, and multi-topic dataset for multi-modal entity linking (MEL), designed to overcome the limitations of existing datasets. They also propose a modality-augmented training strategy, leading to a new model called $CLIP_{ND}$, which significantly improves MEL performance.

What Problem Does This Paper Address?

Multi-modal Entity Linking (MEL) is crucial for integrating information across different modalities (like text and images) to correctly identify entities. However, current MEL datasets are often too small, lack diversity in topics and tasks, and are insufficient for training robust multi-modal models. This limits the ability of these models to generalize and perform well in real-world applications. The authors aim to address this by creating a more comprehensive dataset and a training strategy that leverages it effectively.

Methods

The core of the paper involves two main contributions:

  1. $M^3EL$ Dataset Construction: The authors developed a pipeline to build their dataset. This involved:
  2. Raw Data Acquisition: Gathering data from sources like Kaggle, Wikipedia, and Wikidata across various topics (movies, common knowledge, people, books, sports).
  3. Data Filtering and Cleaning: Removing non-English text, single-character mentions, and instances without associated images.
  4. Mention Processing: Extracting mentions and their contextual information from different data sources, often using LLMs like GPT-4 for semantic understanding.
  5. Candidate Construction: For each mention, a candidate set of potential entities is generated from Wikidata. This is crucial because directly searching millions of entities is computationally infeasible, and this step mimics real-world challenges where the correct entity might not be the top result or even present.
  6. Image Crawling: A framework was used to collect relevant images for mentions, leveraging CLIP for filtering and supplementary crawlers for missing images.
  7. Modality-Augmented Training Strategy: Standard CLIP training uses image augmentation but keeps text constant. This paper proposes augmenting the textual input as well, specifically by incorporating entity description information. This is denoted as $augr(x_T)$, where $x_T$ is the textual input. This strategy aims to provide richer, more precise textual context, helping the model handle semantic ambiguities better.

Datasets

  • $M^3EL$ (New Dataset): This is the primary contribution. It contains 79,625 instances, 318,500 images, covers 5 topics (movies, common, person, books, sports), and supports 9 different multi-modal tasks.
  • Existing Datasets for Benchmarking: The authors also evaluated their approach on other MEL datasets to compare performance:
  • DiffusionDB (subset)
    • WIKIPerson
    • WIKIDiverse

Key Insights and Ideas

  • Dataset Scale and Diversity Matter: The paper strongly suggests that existing MEL datasets are too small and lack the diversity in topics and tasks needed for robust model generalization. $M^3EL$'s scale and breadth are presented as a solution.
  • Text Augmentation is Key for MEL: Simply augmenting images in CLIP-like models isn't enough. Incorporating richer textual information, like entity descriptions, significantly boosts performance by improving the model's understanding of semantic nuances.
  • Candidate Set Realism: The way candidates are constructed for mentions in $M^3EL$ (including potentially incorrect ones) better reflects real-world entity linking challenges than simply providing perfect candidate lists.
  • Contextual Information Matters: The experiments show that using more detailed textual context (like full sentences or descriptions) generally improves performance, but overly long or noisy text can be detrimental.

Key Results

  • Baseline Performance on $M^3EL$ is Low: Existing MEL models perform poorly on the $M^3EL$ dataset, with accuracies ranging from 49.4% to 75.8%, highlighting the dataset's challenging nature and the limitations of current approaches.
  • $CLIP_{ND}$ Achieves Significant Improvements: The proposed $CLIP_{ND}$ model, trained with the modality-augmented strategy on $M^3EL$, shows substantial gains. It achieved an average improvement of 9.3% to 25% across various tasks compared to baselines.
  • Text Augmentation Benefits: Adding auxiliary text information (entity names and descriptions) to baseline models improved their accuracy by an average of 1.2%.
  • Context Length Impact: Models performed better with shorter, more focused textual context ($S_1$) than with longer contexts ($S$), suggesting that precise information is more valuable than sheer volume.

Limitations and Future Work

  • Limitations:
    • The paper notes that while $M^3EL$ is large, it still presents a challenging benchmark, implying that achieving human-level performance is still a distant goal.
  • The authors mention that some models performed poorly on the DiffusionDB dataset due to ambiguous mentions, indicating that certain types of textual ambiguity remain difficult to resolve.
  • Future Work:
    • The authors plan to continue updating $M^3EL$ to further support MEL research.
  • They aim to explore more complex real-world tasks, such as dynamic entity linking in audio or video.
  • Investigating the capabilities of $M^3EL$ with large language models is also a future direction.

Summary updated

Abstract

Multi-modal Entity Linking (MEL) is a fundamental component for various downstream tasks. However, existing MEL datasets suffer from small scale, scarcity of topic types and limited coverage of tasks, making them incapable of effectively enhancing the entity linking capabilities of multi-modal models. To address these obstacles, we propose a dataset construction pipeline and publish $M^3EL$, a large-scale dataset for MEL. $M^3EL$ includes 79,625 instances, covering 9 diverse multi-modal tasks, and 5 different topics. In addition, to further improve the model's adaptability to multi-modal tasks, We propose a modality-augmented training strategy. Utilizing $M^3EL$ as a corpus, train the $\textit{CLIP}_{\textit{ND}}$ model based on $\textit{CLIP} (\textit{ViT}-\textit{B}-\textit{32})$, and conduct a comparative analysis with an existing multi-modal baselines. Experimental results show that the existing models perform far below expectations (ACC of 49.4%-75.8%), After analysis, it was obtained that small dataset sizes, insufficient modality task coverage, and limited topic diversity resulted in poor generalisation of multi-modal models. Our dataset effectively addresses these issues, and the $\textit{CLIP}_{\textit{ND}}$ model fine-tuned with $M^3EL$ shows a significant improvement in accuracy, with an average improvement of 9.3% to 25% across various tasks. Our dataset is available at https://anonymous.4open.science/r/M3EL.
BibTeX
@misc{wang2024mel,
  author = {Fang Wang and Shenglin Yin and Xiaoying Bai and Minghao Hu and Tianwei Yan and Yi Liang},
  title = {{MEL: A Multi-task Multi-topic Dataset for Multi-modal Entity Linking}},
  year = {2024},
  eprint = {2410.18096},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.18096},
}
Figure