Learning Distributed Representations of Sentences from Unlabelled Data

Felix Hill, Kyunghyun Cho, Anna Korhonen

Abstract

Unsupervised methods for learning distributed representations of words are ubiquitous in today's NLP research, but far less is known about the best ways to learn distributed phrase or sentence representations from unlabelled data. This paper is a systematic comparison of models that learn such representations. We find that the optimal approach depends critically on the intended application. Deeper, more complex models are preferable for representations to be used in supervised systems, but shallow log-linear models work best for building representation spaces that can be decoded with simple spatial distance metrics. We also propose two new unsupervised representation-learning objectives designed to optimise the trade-off between training time, domain portability and performance.

Introduction

While unsupervised methods for learning word embeddings, such as Word2Vec or GloVe, are well-established, there is far less consensus on how to best learn representations for larger blocks of text like phrases or sentences. As deep learning becomes central to NLP tasks like machine translation and image captioning, representing sentences as continuous, dense vectors has become common practice. However, it remains unclear which architectures or training objectives yield the most universally useful sentence representations when learning from unlabelled data. This research addresses this gap by systematically comparing existing models and introducing novel architectures designed for efficiency and robustness.

Proposed Methods

The study compares several existing models—including SkipThought, ParagraphVector, and neural machine translation models—against baseline bag-of-words (BOW) approaches. To address the limitations of current models, the authors introduce two novel text-based models:

  • Sequential Denoising Autoencoders (SDAE): Unlike SkipThought, which requires ordered narrative text to predict adjacent sentences, SDAEs learn from unordered data. They corrupt input sentences by randomly deleting and swapping words, then train an LSTM-based encoder-decoder to reconstruct the original sentence, forcing the model to learn robust internal semantics.
  • FastSent: Designed as a highly efficient alternative to SkipThought, FastSent is a log-linear bag-of-words model. Given a BOW representation of a sentence, it predicts the words in adjacent sentences. This exploits inter-sentence context signals at a fraction of the computational cost of deeper models.

Experimental Results

Supervised Evaluation

The models were evaluated on six supervised tasks, including paraphrase identification (MSRP), sentiment analysis (MR, CR), and question classification (TREC).

Table 1: Performance of various models on supervised classification tasks.

As shown in Table 1, deep, non-linear models like SkipThought and the novel SDAE generally performed best when their representations were fed into supervised classifiers. SkipThought vectors were top performers on three of the six tasks, while the SDAE model achieved the best results on paraphrase identification (MSRP), outperforming SkipThought by over 3 percentage points in accuracy.

Unsupervised Evaluation

Unsupervised performance was measured using semantic textual similarity (STS) and relatedness (SICK) datasets, testing how well cosine distances between vectors align with human judgments.

Table 2: Performance of models on unsupervised semantic similarity tasks (STS 2014 and SICK).

In contrast to supervised tasks, shallow log-linear models like FastSent or simple CBOW performed better for unsupervised similarity metrics. As seen in Table 2, the dictionary-based DictRep model proved to be the most robust "all-rounder," winning 4 out of 6 STS categories. Among models trained purely on raw text, FastSent was the top performer on the SICK dataset.

Key Insights and Discussion

  • Application Dictates Architecture: Deep models excel as features for supervised systems, but shallow models create better representation spaces for direct spatial queries (e.g., cosine similarity).
  • Efficiency Gains: FastSent achieved near state-of-the-art unsupervised performance while training in just 2 hours on a single CPU core, whereas SkipThought required 2 weeks on a GPU.
  • The Power of Dictionaries: Training a model to map dictionary definitions to pre-trained word embeddings (DictRep) yielded highly reliable representations across both task types, making it an excellent choice when the downstream application is unknown.
  • Context vs. Content: Models predicting adjacent sentences (SkipThought, FastSent) group text by general concepts, while those predicting internal words (SDAE, CBOW) group by word overlap.

Conclusion

This systematic comparison reveals that there is no single "best" sentence representation; the choice depends heavily on whether the task is supervised or unsupervised. Future work may explore multi-task learning—training architectures simultaneously on various supervised tasks—to further induce the general world knowledge required for advanced language understanding.

Summary updated

Abstract

Unsupervised methods for learning distributed representations of words are ubiquitous in today's NLP research, but far less is known about the best ways to learn distributed phrase or sentence representations from unlabelled data. This paper is a systematic comparison of models that learn such representations. We find that the optimal approach depends critically on the intended application. Deeper, more complex models are preferable for representations to be used in supervised systems, but shallow log-linear models work best for building representation spaces that can be decoded with simple spatial distance metrics. We also propose two new unsupervised representation-learning objectives designed to optimise the trade-off between training time, domain portability and performance.
BibTeX
@inproceedings{hill2016learning,
  author = {Felix Hill and Kyunghyun Cho and Anna Korhonen},
  title = {{Learning Distributed Representations of Sentences from Unlabelled Data}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2016},
  doi = {10.18653/v1/n16-1162},
}
Figure