VectorWeavers at SemEval-2016 Task 10: From Incremental Meaning to Semantic Unit (phrase by phrase)

Andreas Scherbakov, Ekaterina Vylomova, Fei Liu, Timothy Baldwin

TL;DR

This paper presents VectorWeavers, a system for identifying minimal semantic units (phrases) within sentences and assigning them "supersenses" (semantic categories). They leverage word embeddings and parser-based features, employing a novel incremental neural network to build phrase representations compositionally, demonstrating promising results for this challenging NLP task.

What Problem Does This Paper Address?

The paper tackles the complex problem of Multiword Expression (MWE) identification and sense disambiguation. MWEs are phrases where the meaning isn't simply the sum of its parts (e.g., "kick the bucket"). Identifying these units and understanding their specific meaning (their "supersense," like "N.PERSON" or "V.CHANGE") is crucial for deeper language understanding. Existing methods often struggle with the idiomatic nature of MWEs and the sheer variety of semantic categories. This work aims to improve both MWE boundary detection and supersense labeling by exploring the utility of word embeddings and syntactic structure.

Methods

The VectorWeavers system employs a three-block neural network architecture:

  • Incremental Vector Calculator: This is the core novelty. It uses a recursive neural network (RNN) approach, but instead of processing the entire sentence recursively, it incrementally builds a vector representation for candidate MWEs, phrase by phrase. This allows for compositional meaning construction. It combines word embeddings, character-level hashing (for unknown words), and other word features.
  • Two-Layer Perceptron for Sense Classification: Takes the incremental vector as input to predict the supersense of the identified MWE.
  • Two-Layer Perceptron for MWE Classification: Also uses the incremental vector to decide if a phrase is an MWE.

Key features used include:

  • Word Embeddings: Pre-trained 300-dimensional word2vec vectors from Google News.
  • Word Hash Sum: A fallback for unknown words.
  • Distance-Based Features: Derived from parse trees, capturing hierarchical relationships between words.
  • Heuristic Word Features: A collection of miscellaneous features like capitalization, punctuation, etc.

Datasets

The system was trained and evaluated on the DiMSUM dataset for SemEval-2016 Task 10. This dataset contains English sentences annotated with MWEs and their supersenses.

Key Insights and Ideas

  • Compositionality is Key: The incremental, phrase-by-phrase vector computation is crucial for capturing the meaning of MWEs, especially those with mixed semantic compositionality.
  • Word Embeddings are Powerful: Word embeddings proved to be a valuable resource for guessing word senses and contributed significantly to MWE recall.
  • Syntax Matters: Syntactic structure, as captured by parser-based features, strongly correlates with MWE boundaries and significantly improves both precision and recall in MWE detection.
  • Feature Importance Varies: While word embeddings and parser features are vital, heuristic word features had less impact than expected.
  • Sense Ambiguity is a Challenge: The system's frequent mispredictions, even with good MWE identification, highlight the difficulty of global sense coherence and the need for richer contextual information.

Key Results

The system achieved competitive results in the SemEval-2016 Task 10. Specifically:

  • MWE Identification: The combined system achieved an F1 score of 49.77% (official run). Ablation studies showed that removing distance features significantly hurt recall.
  • Supersense Tagging: The system achieved an F1 score of 51.62% (official run). Word embeddings were found to be critical for supersense information.
  • Overall Performance: The combined MWE and supersense F1 score was 49.94%.

Limitations and Future Work

  • Lack of Global Sense Coherence: The system's reliance on local context and word embeddings leads to harsh mispredictions when global sense coherence is needed.
  • Limited Context Awareness: The system struggles with ambiguity when broader context is required, suggesting a need for more sophisticated context modeling.
  • Hysteresis Bias: The MWE classifier exhibits a bias towards not joining words initially but then excessively extending predicted MWEs, possibly due to imbalanced training data.
  • Future Directions: The authors suggest exploring deeper neural networks, incorporating more sophisticated context modeling (e.g., attentional mechanisms), and further investigating the parser-based approach for MWE boundary detection. They also note the potential for a shared computation for MWE and sense learning to cause collisions, suggesting that providing distance features to the MWE classifier (as opposed to the incremental vector calculator) might improve MWE prediction at the cost of sense prediction.

Summary updated

BibTeX
@inproceedings{scherbakov2016vectorweavers,
  author = {Andreas Scherbakov and Ekaterina Vylomova and Fei Liu and Timothy Baldwin},
  title = {{VectorWeavers at SemEval-2016 Task 10: From Incremental Meaning to Semantic Unit (phrase by phrase)}},
  booktitle = {SemEval @ NAACL},
  year = {2016},
  url = {https://openreview.net/forum?id=FVmqJgcx58P},
}
Figure