Evaluating Composition Models for Verb Phrase Elliptical Sentence Embeddings

Gijs Wijnholds, Mehrnoosh SadrzadehNAACL 2019 · DOI

Abstract

This research investigates the efficacy of various mathematical representations for sentences containing verb phrase ellipsis (VPE), such as "Cats chase dogs, and children do too." By bridging formal logic and distributional semantics, the authors demonstrate that explicitly resolving ellipsis through non-linear "copying" operations significantly outperforms standard linear composition models. The findings suggest that structural resolution is essential for capturing the true semantics of elliptical language in vector space models.

Introduction: The Challenge of Ellipsis

In natural language, repetitive information is frequently omitted for brevity. In the sentence "Cats chase dogs, children do too," the phrase "chase dogs" is elided in the second clause. While humans intuitively fill in this gap, standard distributional semantics—which typically represents sentence meaning by combining word vectors—struggles with such structures.

Most existing compositional models are linear, processing each word once. However, ellipsis is inherently non-linear because the elided resource (the verb phrase) must be utilized twice to compute the full meaning. This study investigates whether explicitly resolving this ellipsis by "copying" the verb phrase vector into the elided slot produces superior sentence embeddings compared to treating the sentence as a simple sequence of words.

Methodology

The researchers bridge the gap between formal logic and vector space models by using lambda calculus to formally represent the "copying" operation required for ellipsis. These logical forms are then mapped into concrete vector operations. The study evaluates four classes of models:

  1. Linear Vector Models: Simple addition or multiplication of all words without ellipsis resolution.
  2. Non-Linear Vector Models: Models where the elided verb and object are explicitly "copied" and integrated into the final representation.
  3. Tensor-Based Models: Higher-order mathematical structures (matrices and tensors) representing verbs to allow for sophisticated interactions between subjects, verbs, and objects.
  4. Holistic Sentence Encoders: Modern "black box" models like Universal Sentence Encoder (USE) and InferSent, tested with both raw and manually resolved text.

Benchmarking Elliptical Semantics

To evaluate these models, the authors developed two new benchmarks by extending existing datasets:

  • GS2011 (Verb Disambiguation): Modified to include elliptical versions of ambiguous verbs (e.g., "Man draws sword and artist does too").
  • KS2013 (Sentence Similarity): Extended to 432 elliptical entries to test overall semantic similarity capture.
  • Human Baselines: Approximately 19,000 new human similarity judgments were collected via Amazon Mechanical Turk to serve as ground truth.

Results and Analysis

The Power of Explicit Resolution

Across nearly all tests, models that explicitly resolved ellipsis (non-linear) outperformed those that did not (linear). This confirms that "filling in the blanks" is a necessary step for generating high-quality vector representations of elliptical sentences.

Performance of various compositional models on the verb disambiguation task (GS2011).

As shown in the results for the GS2011 disambiguation task, the Best Tensor model in a count-based (CB) vector space achieved the highest Spearman correlation of 0.5385. This significantly outperformed simple linear additive and multiplicative models. Interestingly, treating the coordinator "and" as an addition operation generally worked better than multiplication, likely because multiplication in vector spaces can lead to excessive sparsity.

Compositional Models vs. Holistic Encoders

Compositional models, particularly tensor-based ones, demonstrated a clear advantage over holistic encoders on the verb disambiguation task. Disambiguation requires a deep understanding of how a verb interacts with its specific subject and object—a structural nuance that "black box" encoders often fail to capture.

Performance of holistic sentence encoders on the disambiguation task.

In contrast to the 0.5385 achieved by the tensor model, the best-performing holistic encoder, InferSent 2 (IS2), only managed a correlation of 0.3841. However, on the broader KS2013 similarity task, holistic encoders like InferSent 1 performed exceptionally well (0.8022), suggesting they are better suited for broad semantic similarity than fine-grained grammatical resolution.

Conclusion

The study proves that structural "copying" is essential for the semantic modeling of ellipsis. While holistic encoders excel at general similarity, they lack the grammatical precision of explicit compositional models. Future work should focus on integrating these structural copying rules directly into the architecture of neural sentence encoders to combine deep grammatical accuracy with broad semantic coverage. Additionally, scaling these elliptical datasets will be crucial for training more robust neural models in the future.

Summary updated

BibTeX
@inproceedings{wijnholds2019evaluating,
  author = {Gijs Wijnholds and Mehrnoosh Sadrzadeh},
  title = {{Evaluating Composition Models for Verb Phrase Elliptical Sentence Embeddings}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/n19-1023},
}
Figure