Skip-Prop: Representing Sentences with One Vector Per Proposition

Rachel Rudinger, Kevin Duh, Benjamin Van DurmeIWCS 2017 · OpenReview

Abstract

Rachel Rudinger, Kevin Duh, Benjamin Van Durme. IWCS 2017 — 12th International Conference on Computational Semantics — Short papers.

2017.

Introduction

In modern Natural Language Processing (NLP), sentences are typically represented as fixed-size vectors, known as embeddings. While efficient, this "one vector per sentence" approach creates a significant information bottleneck. As sentences increase in length and complexity, compressing their entire semantic content into a single vector leads to degraded performance in tasks like machine translation and question-answering.

Conversely, token-level representations (one vector per word) preserve detail but are computationally expensive for large-scale analysis, requiring the system to attend to every token. Skip-Prop proposes a "middle-ground" granularity: representing a sentence as a set of vectors, where each vector corresponds to an underlying proposition (predicate-argument structure).

Methodology

The Skip-Prop framework involves three primary stages:

  1. Proposition Extraction: The system uses PredPatt to identify predicate-argument structures. For example, a sentence describing two distinct actions is decomposed into two propositions.
  2. Linearization: These extracted structures are converted back into simple token sequences (e.g., "Company buys subsidiary").
  3. Model Architecture: The authors adapt the "Skip-Thought" framework. While standard Skip-Thought uses a single encoder, Skip-Prop employs multiple encoders (one per proposition) and an attention mechanism in the decoder to aggregate these vectors during training.
Comparison of sentence representation granularities: single-vector (left), token-level (middle), and Skip-Prop (right).

The model was trained on 100,000 sentence triples from the NYT Gigaword dataset using two objectives: a Skip-Thought objective (predicting surrounding sentences) and an autoencoder objective (reconstructing the original sentence).

Key Insights and Results

The Granularity Sweet Spot

Analysis of the NYT dataset reveals that while an average sentence contains approximately 23 tokens, it typically contains only 3.5 propositions. By shifting the representation from tokens to propositions, the model reduces the number of vectors to manage by nearly 85% while maintaining interpretable semantic units.

Correlation between tokens and propositions per sentence in the NYT Gigaword dataset.

Quantitative and Qualitative Performance

  • Perplexity: Skip-Prop achieved a per-token perplexity of 206.29, outperforming the single-vector Skip-Thought model (216.00). While it did not surpass the token-level attention model (199.16), it achieved competitive results with significantly fewer vectors.
  • Targeted Search: Qualitative tests on the SICK corpus demonstrated that Skip-Prop allows for more precise semantic retrieval. Because it de-couples meaning into individual actions, it can isolate specific facts (e.g., corporate ownership) within complex sentences that might otherwise distract a single-vector model.

Limitations and Future Work

Skip-Prop's performance is currently tied to the accuracy of upstream tools like PredPatt and Universal Dependency parsers. Errors in these tools can lead to missing information in the final embedding. Future research may explore more sophisticated proposition representations beyond simple linearization and evaluate Skip-Prop on downstream applications such as summarization and complex question-answering.

Summary updated

Abstract

Rachel Rudinger, Kevin Duh, Benjamin Van Durme. IWCS 2017 — 12th International Conference on Computational Semantics — Short papers. 2017.
BibTeX
@inproceedings{rudinger2017skip,
  author = {Rachel Rudinger and Kevin Duh and Benjamin Van Durme},
  title = {{Skip-Prop: Representing Sentences with One Vector Per Proposition}},
  booktitle = {IWCS},
  year = {2017},
  url = {https://openreview.net/forum?id=4WKVghGxjdF},
}
Figure