Combining Tradition with Modernness: Exploring Event Representations in Vision-and-Language Models for Visual Goal-Step Inference

Chong Shen, Carina SilbererACL 2023

TL;DR

This paper explores how to improve visual goal-step inference (predicting the correct image for a given instruction step) by integrating traditional, linguistically-inspired event representations into modern vision-and-language models. They found that combining these structured event details with full sentence context significantly boosts performance, suggesting a path towards more robust procedural understanding in AI.

What Problem Does This Paper Address?

Understanding how to perform tasks, or procedural knowledge, is a fundamental aspect of intelligence. While AI models are getting better at understanding language and images separately, bridging the gap to understand sequences of actions needed to achieve a goal remains a challenge. The Visual Goal-Step Inference (VGSI) task aims to address this by asking models to select the correct image that depicts a step towards a given textual goal. Existing state-of-the-art models often treat text and images as independent entities or use complex transformer architectures without explicitly encoding the core "event" information (who did what to whom). This paper argues that older, linguistically-grounded methods that focus on extracting key event components (like actions and participants) can provide valuable structure that modern models might miss, leading to better procedural understanding.

Methods

The core of this work lies in exploring different ways to inject "event knowledge" into existing vision-and-language models, specifically the CLIP model. They investigate two main approaches for representing events:

  • Relational Event Representation: A simpler (subject, verb, object) tuple.
  • Multi-Argument Event Representation: A more detailed (subject, verb, object, prepositional phrase) tuple.

These event representations are then integrated into the model in two ways:

  1. EVENT: The original sentence describing the goal or step is replaced by its event representation.
  2. SENTENCE+EVENT: The event representation is appended to the original sentence.

They also explore different ways to extract sentence embeddings from CLIP's text encoder, specifically looking at embeddings from the first, middle, and last layers, hypothesizing that different layers capture different linguistic information (lexical, syntactic, semantic). Finally, they investigate the use of contextualized vs. local embeddings for event components and experiment with different ways to represent the implicit subject in imperative sentences common in instructional texts.

Datasets

The primary dataset used is wikiHow-VGSI (Yang et al., 2021). This dataset consists of instructional articles from wikiHow, providing goals, step headlines (sentences describing steps), and corresponding images. The authors performed some cleaning on the dataset, removing non-English entries and parsing issues, resulting in 53,186 goals and 772,221 step images.

Key Insights and Ideas

  • Tradition Meets Modernity: Linguistically-inspired event representations, though "older," can significantly enhance the performance of modern deep learning models for procedural understanding.
  • Structure is Key: Explicitly encoding the core components of an event (action, participants) provides valuable structure that complements the rich but less structured information in full sentences.
  • Context Matters: Appending event information to the full sentence (SENTENCE+EVENT) is more effective than replacing the sentence with just the event representation (EVENT). This suggests that the full sentence provides crucial context that the event representation alone lacks.
  • Layered Understanding: The last layers of CLIP's text encoder, which tend to capture semantic information, are most crucial for this task, followed by middle (syntactic) layers. Lexical information from early layers is less impactful.
  • Implicit Subject Handling: Using a special token like </startoftext|> as a placeholder for the implicit subject in imperative sentences is more effective than other methods, likely because its position within the attention mechanism helps infer subject information.

Key Results

The paper reports several key findings:

  • Multi-argument event representations consistently outperform relational ones. For example, in the strict setting with contextualized embeddings and last-layer representations, SENTENCE+EVENTmult,last4 achieved 26.4% accuracy, significantly higher than EVENTrel,last4 (16.7%).
  • Appending event information to sentences (SENTENCE+EVENT) is superior to using events alone (EVENT). The best SENTENCE+EVENT model achieved 26.4% accuracy, while the best EVENT model reached 21.2%.
  • Contextualized embeddings for event components yield better results than local embeddings. The improvement was substantial, with accuracy gains of 3.71% (weak setting) and 13.73% (strict setting) on average.
  • Using the last 4 layers of CLIP's text encoder (LAST4) for sentence embeddings generally leads to the highest accuracy.
  • The best model, SENTENCE+EVENTmult,last4, achieved 26.4% accuracy in the strict setting, which is still a significant gap from human performance (84.5%), but represents a notable improvement over previous methods.

Limitations and Future Work

  • Simple Event Representations: The authors acknowledge that they explored "very simple structured event representations." Future work could explore richer representations that include predicate-argument structures, event types, and argument roles, as well as general graph-based approaches and scene graphs.
  • Dataset Bias: The wikiHow articles used in the dataset might reflect biases present in their human authors.
  • Model Size vs. Embeddings: While they showed that adding event embeddings to sentences improved performance, they suggest further work to disentangle the exact contribution of increased model parameters versus the improved embeddings.
  • Human Performance Gap: The significant gap between the best model's performance and human performance indicates that current models still struggle with deeper semantic and causal relationships between goals and steps.

Summary updated

BibTeX
@inproceedings{shen2023combining,
  author = {Chong Shen and Carina Silberer},
  title = {{Combining Tradition with Modernness: Exploring Event Representations in Vision-and-Language Models for Visual Goal-Step Inference}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure