SentEval: An Evaluation Toolkit for Universal Sentence Representations

Alexis Conneau, Douwe Kiela

TL;DR

SentEval is a standardized evaluation toolkit designed to bring order to the "wild west" of universal sentence representations. By providing a centralized suite of tasks and a fixed evaluation pipeline, it allows researchers to fairly compare how well different sentence encoders capture general-purpose linguistic information across classification, inference, and similarity tasks.

What Problem Does This Paper Address?

In the wake of the success of word embeddings like Word2Vec and GloVe, the NLP community shifted its focus toward "universal" sentence representations—vectors that capture the meaning of entire sentences regardless of the specific task. However, evaluating these representations became a fragmented mess.

Researchers were using different datasets, inconsistent preprocessing steps, and varying downstream classifier architectures. A model might look superior simply because it used a more complex classifier or a different tuning strategy, rather than having better embeddings. This lack of standardization made it nearly impossible to track real progress in the field. SentEval was created to solve this by establishing a "community consensus" benchmark.

Methods

SentEval operates as a "black box" evaluation suite. A researcher provides their sentence encoder, and the toolkit handles the rest. The core philosophy is to use the embeddings as fixed features for downstream tasks.

The toolkit implements two main types of evaluations:

  1. Supervised Downstream Tasks: For tasks like sentiment classification or natural language inference (NLI), SentEval fits a standard classifier (either Logistic Regression or a Multi-Layer Perceptron with one hidden layer) on top of the provided embeddings. Crucially, it uses fixed hyperparameters (like batch size and optimizer) to ensure that differences in results stem from the embeddings themselves, not the classifier tuning.
  2. Unsupervised Similarity Tasks: For Semantic Textual Similarity (STS), the toolkit simply calculates the cosine similarity between two sentence vectors and correlates this with human judgments.

The interface is designed for ease of use in Python, requiring users to implement only two functions: a prepare function for preprocessing and a batcher function that converts a list of sentences into embeddings.

Datasets

SentEval aggregates a wide variety of established benchmarks, categorized into three main groups:

  • Binary and Multi-class Classification: Includes sentiment analysis (MR, SST), product reviews (CR), subjectivity (SUBJ), opinion polarity (MPQA), and question-type classification (TREC). Table 1: Classification tasks overview
  • Entailment and Semantic Similarity: Includes the SNLI and SICK datasets for Natural Language Inference (NLI), and the STS benchmarks (2012–2016) for sentence similarity. Table 2: NLI and Similarity tasks overview
  • Image-Caption Retrieval: Uses the COCO dataset to rank images by their relevance to a query caption (and vice versa).
Table 1: Classification tasks overview
Table 2: NLI and Similarity tasks overview

Key Insights and Ideas

  • Standardization is a Research Catalyst: By removing the "cumbersome wheel reinvention" of building evaluation pipelines, SentEval allows researchers to iterate faster on the actual representation learning models.
  • The "Transfer" Gap: One of the most important insights highlighted by the toolkit is the gap between "universal" representations and task-specific models. While universal embeddings are highly desirable for their generalization, they often still lag behind models trained specifically for a single task (the SOTA results in Table 3).
  • Fixed vs. Tuned Hyperparameters: The authors argue that while tuning hyperparameters on a validation set is standard, fixing the architecture of the evaluation pipeline is essential for fairness, especially on small datasets where minor setup changes lead to wildly different outcomes.

Key Results

The paper provides a comprehensive benchmark of existing models, including GloVe and fastText (using bag-of-words averages), SkipThought, and the authors' own InferSent.

Table 3: Transfer test results for various baseline methods

Key findings from the benchmarking include:

  • InferSent (trained on NLI data) consistently outperformed other transfer methods across most classification and NLI tasks, showing the power of supervised pre-training for universal representations.
  • Simple Baselines are Stubborn: Continuous bag-of-words (BoW) models remain surprisingly competitive, particularly on semantic similarity tasks (STS), as shown in Table 4. Table 4: Evaluation on semantic textual similarity benchmarks
  • Embedding Size Matters: For methods with large embedding sizes (like the 4096-D vectors of InferSent), the authors found that a simple Logistic Regression was often just as effective as a more complex MLP, suggesting the information is easily accessible in the high-dimensional space.
Table 4: Evaluation on semantic textual similarity benchmarks

Limitations and Future Work

The authors acknowledge that SentEval is a living project. While it captures the current community consensus, the "best" way to evaluate sentence embeddings is still evolving.

Future directions include:

  • Linguistic Probing: Moving beyond downstream performance to "probing tasks" that test for specific linguistic properties (e.g., does the vector know the sentence length or the tense of the main verb?).
  • Task Expansion: Adding more diverse tasks as the field matures to ensure the "universal" tag remains accurate.
  • Computational Efficiency: While the toolkit supports CPU evaluation via scikit-learn, larger datasets like SNLI still practically require a GPU, which may be a barrier for some researchers.

Summary updated

Abstract

We introduce SentEval, a toolkit for evaluating the quality of universal sentence representations. SentEval encompasses a variety of tasks, including binary and multi-class classification, natural language inference and sentence similarity. The set of tasks was selected based on what appears to be the community consensus regarding the appropriate evaluations for universal sentence representations. The toolkit comes with scripts to download and preprocess datasets, and an easy interface to evaluate sentence encoders. The aim is to provide a fairer, less cumbersome and more centralized way for evaluating sentence representations.
BibTeX
@inproceedings{conneau2018senteval,
  author = {Alexis Conneau and Douwe Kiela},
  title = {{SentEval: An Evaluation Toolkit for Universal Sentence Representations}},
  booktitle = {LREC},
  year = {2018},
  url = {https://openreview.net/forum?id=JlAiNXIH3J},
}

Loading related content...

Figure