Visually Grounded Interpretation of Noun-Noun Compounds in English

Inga Lang, Lonneke Plas, Malvina Nissim, Albert GattACL 2022

TL;DR

This paper explores how incorporating visual information can improve the interpretation of English noun-noun compounds (NNCs). By combining linguistic word embeddings with visual features from images, they found modest but significant improvements in classifying the semantic relationships between NNC constituents, suggesting that visual grounding is a promising avenue for NNC understanding.

What Problem Does This Paper Address?

Noun-noun compounds (NNCs) like "beach house" are ubiquitous in English, but understanding the implicit relationship between their constituent nouns (e.g., "beach" modifies "house" to indicate location) is a complex task for Natural Language Processing (NLP). While previous work has focused on linguistic representations (like word embeddings), research in cognitive science suggests that grounding these representations in other modalities, particularly vision, can lead to richer and more human-like understanding. This paper addresses the gap of exploring how visual information can specifically enhance NNC interpretation, framing it as a relation classification problem.

Methods

The authors frame NNC interpretation as a classification task, aiming to identify the semantic relationship between the modifier and head nouns. They experiment with two main approaches for combining constituent word representations:

  • Simple Concatenation: Directly joining the vector representations of the modifier and head nouns.
  • Full Additive (FullAdd) Model: A compositional model that learns to combine vectors by multiplying them with learned matrices, aiming to approximate the compound's overall meaning.

They compare purely linguistic approaches (using pre-trained word2vec embeddings) with visuo-linguistic approaches. For the latter, they extract visual features from images associated with the NNC constituents using a pre-trained ResNet model. These visual vectors are then combined with the linguistic vectors, either through concatenation or the FullAdd model. The final classification is performed using an SVM classifier.

Datasets

  • Tratz (2011) Dataset: A revised version of a noun compound dataset, featuring 19,158 distinct NNCs labeled with 37 fine-grained and 12 coarse-grained relation labels. The dataset was annotated using Amazon's Mechanical Turk and includes various splits (random, lexical full, lexical mod, lexical head) to test generalization to unseen constituents.
  • ImageNet: Used to extract visual feature embeddings for NNC constituents. The authors developed a heuristic to select image synsets that likely represent the most common or "imageable" sense of a word.

Key Insights and Ideas

  • Visual Grounding Offers Modest Gains: While not a dramatic leap, incorporating visual features consistently improved NNC interpretation performance across various configurations, suggesting that visual information is indeed beneficial.
  • Concatenation Outperforms FullAdd: The simpler vector concatenation method for combining constituent representations generally yielded better results than the more complex FullAdd compositional model, especially when visual features were included.
  • Concreteness Doesn't Guarantee Better Visual Performance: Counter-intuitively, higher concreteness of NNC constituents did not consistently lead to better performance with visual features. In fact, a negative correlation was observed in some cases, suggesting that abstract concepts might be easier to classify visually in certain contexts.
  • Image Availability Matters: The number of available images for a constituent in ImageNet showed a moderate correlation with concreteness, indicating that more concrete words tend to have more visual representations, but this didn't directly translate to improved classification success.

Key Results

  • In their best-performing setup (concatenation of linguistic and visual vectors), the authors observed modest but statistically significant increases in F1 scores across most experimental settings on the Tratz (2011) dataset.
  • For example, in the coarse-grained random split using concatenated vectors, adding ResNet10 vectors improved the F1 score from 74.1% (linguistic only) to 75.3% (visuo-linguistic).
  • The paper also presents results from experiments using BERT, showing that visuo-linguistic embeddings can also improve BERT's performance on NNC classification.

Limitations and Future Work

  • Limited Visual Coverage: Not all words or NNC constituents have readily available images in ImageNet, limiting the scope of visuo-linguistic analysis. The authors acknowledge that finding images for abstract or polysemous words is challenging.
  • Heuristic Image Selection: The method for selecting images is a heuristic and might not always capture the most relevant visual aspects for NNC interpretation.
  • Future Directions: The authors suggest exploring more sophisticated image selection methods, developing ways to represent abstract or non-imageable words visually, and investigating multimodal encoders. They also propose testing their approach on different datasets and exploring visual grounding for NNC generation tasks.

Summary updated

BibTeX
@inproceedings{lang2022visually,
  author = {Inga Lang and Lonneke Plas and Malvina Nissim and Albert Gatt},
  title = {{Visually Grounded Interpretation of Noun-Noun Compounds in English}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure