Abstract
This paper investigates whether the "context-is-king" paradigm, central to word-level NLP, holds for unsupervised sentence representation learning. By evaluating various content-oriented and context-oriented models on Dialog Act (DA) tagging, the researchers demonstrate that context-based training provides negligible benefits. Instead, increasing the dimensionality of sentence vectors emerges as the most consistent factor in improving performance, suggesting that internal sentence content is often sufficient for capturing functional meaning.
Introduction
In modern Natural Language Processing, the meaning of a word is often defined by its context. This principle has been extended to sentence-level models, such as Skip-Thought vectors, which are trained to predict neighboring sentences. However, unlike words, sentences possess internal structure and compositionality. This study addresses a fundamental question: Does a sentence representation require contextual information to be effective, or is its internal content sufficient? The authors use Dialog Act (DA) tagging—a task where meaning often depends on conversational flow—as a rigorous test for the necessity of contextual signals.
Methodology
The study compares two primary representation philosophies:
- Content-Oriented Models: These models rely solely on internal sentence words. Examples include Latent Semantic Analysis (LSA), Latent Dirichlet Allocation (LDA), averaged GloVe embeddings, and LSTM Auto-encoders. A random baseline (averaging arbitrary word vectors) was also included.
- Context-Oriented Models: These include models trained to predict surrounding utterances, either by projecting content-oriented vectors into a "context space" or using end-to-end architectures like Skip-Thought.
To evaluate these representations, the authors employed linear probing with ridge regression on the Switchboard Dialog Act (SwDA) corpus. This approach ensures that performance reflects the quality of the representation itself rather than the complexity of the classifier.
Results
The most significant finding was the "Dimensionality Dominance." Increasing vector dimensionality from $D=6$ to $D=25$ consistently improved accuracy by 5-10%, far outweighing any gains from contextual training.
As shown in the correlation matrices above, adding contextual training objectives alters the internal feature correlations of the representations. Interestingly, for many models, this "contextualization" did not translate to better DA tagging. The gap between the best content-only model (GloVe at $D=25$, 57.29%) and the best context-aware model (Skip-Thought at $D=25$, 57.54%) was a mere 0.25%.
Key Insights
- The Context Blind Spot: The assumption that context-based learning is inherently superior for sentences may be a misconception. For models like LSA or GloVe, forcing contextual prediction often yielded no improvement or even degraded performance.
- Random Baselines: High-dimensional random vectors outperformed the majority-class baseline, suggesting that providing enough "slots" to keep lexical features separable is a critical component of representation quality.
- End-to-End Context: While contextual projections of existing models were largely ineffective, the end-to-end Skip-Thought model achieved the highest overall score, suggesting that contextual benefits may require specific architectural integration.
Discussion and Conclusion
The study concludes that for tasks like Dialog Act tagging, the internal content of a sentence is the primary driver of its representation's utility. While context is vital for disambiguating individual words, the structural complexity of sentences allows them to carry significant functional meaning independently. Future research should investigate whether these findings hold at the much higher dimensionalities used by modern Large Language Models or for more linguistically nuanced tasks such as sarcasm detection.
Summary updated