MERL: Multimodal Event Representation Learning in Heterogeneous Embedding Spaces

Linhai Zhang, Deyu Zhou, Yulan He, Zeng Yang

Introduction

In the field of Natural Language Processing (NLP), representing "events"—structured as (subject, predicate, object) triples—is fundamental for high-level reasoning tasks. Whether predicting narrative progression or analyzing market trends, understanding these triples is key. However, textual descriptions are inherently abstract and often ambiguous. A triple like (she, attend, baby) evokes a vastly different mental image than (she, attend, meeting), despite sharing the same syntactic structure. Conversely, (she, attend, baby) and (he, take care, kid) describe similar real-world events using entirely different vocabularies.

The Challenge: Abstraction vs. Instantiation

The core difficulty in multimodal event learning lies in the hierarchical relationship between text and vision. A textual triple is an abstraction—a general category encompassing many possible scenarios. An image, by contrast, is a concrete instantiation—a specific, detailed snapshot of a single occurrence.

The MERL framework models a single abstract textual triple as a distribution that encompasses multiple concrete visual instantiations.

Most existing models attempt to map both text and images into the same type of embedding space, typically as fixed-length vectors. This "point-to-point" mapping ignores the inherent difference in granularity. It forces a general concept to occupy the same semantic "volume" as a specific example, leading to poor generalization and an inability to capture the "spread" of an event category.

The MERL Framework

To address this, the authors propose MERL (Multimodal Event Representation Learning). MERL moves away from homogeneous vector spaces, instead modeling the relationship between text and images through heterogeneous embeddings and statistical alignment.

1. Text as Gaussian Distributions

Rather than a single vector, MERL maps a text triple to a Gaussian distribution. This distribution is defined by two components:

  • The Mean (μ): Derived through a tensor-based composition of the subject, predicate, and object. It represents the central semantic "location" of the event.
  • The Variance (σ²): Calculated via an additive path, representing the "spread" or the variety of ways an event can be visually manifested.

2. Component-Aware Image Encoding

To ensure the model focuses on the relevant parts of an image, the image encoder uses an attention mechanism pre-trained on the ImSitu dataset. This allows the model to specifically locate the visual counterparts of the subject, predicate, and object, fusing them into a single point embedding.

3. Coordination via Statistical Hypothesis Testing

The "bridge" between the Gaussian distribution (text) and the point embedding (image) is not a simple distance metric like cosine similarity. Instead, MERL employs statistical hypothesis testing based on the likelihood-ratio test. The model treats the image point as a sample and calculates the likelihood that it was drawn from the text's Gaussian distribution. This approach naturally handles the "one-to-many" relationship between abstract concepts and concrete examples.

Key Results and Insights

The MERL framework demonstrates significant improvements over both unimodal and standard multimodal baselines:

  • Event Similarity: On the Multimodal Hard Similarity task, MERL achieved 52.2% accuracy, outperforming its unimodal counterpart (44.3%) and traditional textual baselines.
  • Script Event Prediction: When integrated into a Scaled Graph Neural Network (SGNN), MERL reached a state-of-the-art accuracy of 55.51% on the Multiple Choice Narrative Cloze (MCNC) dataset.
  • Cross-modal Retrieval: In image retrieval tasks, MERL achieved a Recall@10 of 69.4%, a substantial 13% improvement over Hierarchical Order Embeddings.

Conclusion

MERL provides a mathematically elegant solution to the problem of event abstraction. By recognizing that text triples are "containers" for visual realities, and modeling them as such using Gaussian distributions, the framework achieves a more robust and nuanced alignment between language and vision. Future work aims to expand this into a fully Bayesian framework to better account for the inherent uncertainty in multimodal data.

Summary updated

Abstract

Previous work has shown the effectiveness of using event representations for tasks such as script event prediction and stock market prediction. It is however still challenging to learn the subtle semantic differences between events based solely on textual descriptions of events often represented as (subject, predicate, object) triples. As an alternative, images offer a more intuitive way of understanding event semantics. We observe that event described in text and in images show different abstraction levels and therefore should be projected onto heterogeneous embedding spaces, as opposed to what have been done in previous approaches which project signals from different modalities onto a homogeneous space. In this paper, we propose a Multimodal Event Representation Learning framework (MERL) to learn event representations based on both text and image modalities simultaneously. Event textual triples are projected as Gaussian density embeddings by a dual-path Gaussian triple encoder, while event images are projected as point embeddings by a visual event component-aware image encoder. Moreover, a novel score function motivated by statistical hypothesis testing is introduced to coordinate two embedding spaces. Experiments are conducted on various multimodal event-related tasks and results show that MERL outperforms a number of unimodal and multimodal baselines, demonstrating the effectiveness of the proposed framework.
BibTeX
@inproceedings{zhang2021merl,
  author = {Linhai Zhang and Deyu Zhou and Yulan He and Zeng Yang},
  title = {{MERL: Multimodal Event Representation Learning in Heterogeneous Embedding Spaces}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i16.17695},
}
Figure