Abstract
We consider the problem of learning general-purpose, paraphrastic sentence embeddings, revisiting the setting of Wieting et al. (2016b). While they found LSTM recurrent networks to underperform word averaging, we present several developments that together produce the opposite conclusion. These include training on sentence pairs rather than phrase pairs, averaging states to represent sequences, and regularizing aggressively. These improve LSTMs in both transfer learning and supervised settings. We also introduce a new recurrent architecture, the Gated Recurrent Averaging Network, that is inspired by averaging and LSTMs while outperforming them both. We analyze our learned models, finding evidence of preferences for particular parts of speech and dependency relations.
Introduction
In the pursuit of universal sentence embeddings—vector representations that capture semantic meaning across diverse tasks—a surprising trend emerged. Previous research (Wieting et al., 2016b) indicated that simple word averaging (AVG) models consistently outperformed more complex Long Short-Term Memory (LSTM) networks on Semantic Textual Similarity (STS) tasks. This "back-to-basics" result challenged the assumption that sequence-aware models are inherently superior for semantic representation. This paper investigates whether the LSTM's underperformance was a fundamental architectural limitation or a consequence of suboptimal training conditions, such as data granularity and insufficient regularization.
Methodology
The authors propose three key advancements to enhance the performance of recurrent networks for paraphrastic embeddings:
1. LSTMAVG Architecture
Traditional LSTMs often use only the final hidden state to represent a sequence. For long sentences, this can lead to the "forgetting" of early information. The authors introduce LSTMAVG, which averages all hidden states across the sequence, ensuring a more balanced representation of the entire sentence.
2. Aggressive Regularization
To bridge the gap between sequence models and bag-of-words models, the authors employ robust regularization techniques:
- Word Dropout: Randomly removing entire word embeddings during training.
- Scrambling: Randomly shuffling the word order in training sentences. While counter-intuitive for sequence models, this forces the LSTM to focus on semantic content rather than relying solely on specific syntactic structures, mimicking the robustness of averaging models.
3. Gated Recurrent Averaging Network (GRAN)
The authors introduce GRAN, a hybrid architecture. In GRAN, an LSTM generates a gate for each word, which scales the word's original embedding before it is averaged. This allows the model to dynamically weight the importance of specific words, effectively "paying attention" to salient terms while filtering out noise.
Datasets and Training
The models were trained on two primary sources: the Paraphrase Database (PPDB), containing noisy phrase pairs, and SimpWiki, a dataset of aligned sentence pairs from Simple English Wikipedia and standard English Wikipedia. Evaluation was conducted across 22 SemEval STS datasets (2012–2015) and the SICK (Sentences Involving Compositional Knowledge) dataset.
Results and Analysis
The experimental results demonstrate that recurrent models can indeed surpass averaging baselines when properly configured.
The Impact of Data and Architecture
Training on sentence-level data (SimpWiki) rather than short phrases (PPDB) significantly improved LSTM performance. Furthermore, the transition from a standard LSTM to LSTMAVG provided a substantial boost in Pearson correlation scores. The addition of scrambling and dropout allowed the BiLSTMAVG to achieve a score of 68.7 on the 2016 STS task, finally exceeding the 68.4 baseline of the AVG model.
GRAN Performance
The GRAN architecture proved to be the most effective, consistently outperforming both AVG and standard LSTMs. In supervised settings, GRAN reached an average score of 83.4, compared to 82.2 for the AVG model.
Error Analysis: Handling Word Overlap
A critical insight from the error analysis is that AVG models are often misled by high word overlap in sentences with different meanings (e.g., "the act of preserving" vs. "the act of decreasing"). Recurrent models, particularly GRAN, are significantly better at detecting these subtle semantic shifts that simple averaging misses.
Conclusion
This work demonstrates that recurrent networks are highly effective for learning sentence embeddings when trained with appropriate data granularity and aggressive regularization. The success of LSTMAVG and GRAN suggests that incorporating "averaging" properties into sequence models allows them to retain the benefits of bag-of-words approaches while leveraging contextual information. Future work may involve exploring higher-quality paraphrase sources and evaluating these embeddings on a broader range of downstream NLP tasks.
Summary updated