Efficient Nearest Neighbor Emotion Classification with BERT-whitening

Wenbiao Yin, Lin ShangEMNLP 2022 · OpenReview

Abstract: Retrieval-based methods have been proven effective in many NLP tasks. Previous methods use representations from the pre-trained model for similarity search directly. However, the sentence representations from the pre-trained model like BERT perform poorly in retrieving semantically similar sentences, resulting in poor performance of the retrieval-based methods. In this paper, we propose kNN-EC, a simple and efficient non-parametric emotion classification (EC) method using nearest neighbor retrieval. We use BERT-whitening to get better sentence semantics, ensuring that nearest neighbor retrieval works. Meanwhile, BERT-whitening can also reduce memory storage of datastore and accelerate retrieval speed, solving the efficiency problem of the previous methods. kNN-EC average improves the pre-trained model by 1.17 F1-macro on two emotion classification datasets.

Introduction

While Large Language Models (LLMs) like BERT are excellent at understanding text, their raw "sentence embeddings"—the mathematical vectors representing a sentence—have a hidden flaw: they are anisotropic. This means the vectors occupy a narrow, cone-shaped region in space rather than being spread out evenly.

In this cramped space, sentences that are semantically different can end up very close to each other, making simple similarity searches (like k-Nearest Neighbors) perform poorly. Previous attempts to fix this required adding complex, trainable parameters. We introduce kNN-EC, a "plug-and-play" method that boosts emotion classification by combining standard deep learning models with a whitened kNN retrieval mechanism.

Methodology

The kNN-EC framework operates in three main stages:

  1. Datastore Creation: Every training sentence is passed through a pre-trained model (like RoBERTa) to generate a vector representation.
  2. BERT-whitening: Using a simple linear transformation (Singular Value Decomposition), we "whiten" the vectors. This process centers the data at zero and ensures uniform variance in all directions, making the space isotropic. Crucially, we also perform dimension reduction here, keeping only the most important components.
  3. Inference & Interpolation: For a new sentence, the system calculates its whitened vector and finds the $k$ most similar neighbors in the datastore. The final prediction is a weighted average of the deep learning model’s output and the distribution of the retrieved neighbors.
Figure 1: The architecture of the kNN-EC framework, illustrating the flow from model representation through whitening to final prediction interpolation.

Experimental Setup

The model was evaluated on two widely recognized emotion classification benchmarks:

  • GoEmotions: A large-scale dataset of Reddit comments labeled with Ekman-style emotions (joy, anger, fear, sadness, disgust, and surprise).
  • ISEAR: A dataset containing personal reports of emotional events across seven categories.

Key Insights and Results

Whitening is Essential for Retrieval

Without whitening, the kNN component is highly unstable and often performs worse than the base model. Whitening transforms the "messy" BERT space into one where distance actually correlates with emotional meaning. Visualization via t-SNE shows that before whitening, categories like "sadness" and "surprise" overlap significantly. After whitening, these clusters become distinct, explaining the jump in accuracy.

Figure 4: t-SNE visualization of sentence embeddings without (left) and with (right) BERT-whitening. Whitening leads to more distinct emotional clusters.

Efficiency via Geometry

By reducing the dimensionality of the vectors during the whitening process (e.g., to 128 or 256 dimensions), we reduced the memory footprint of the datastore by over 10x while actually improving classification performance. This solves the "storage bloat" usually associated with retrieval-augmented models.

Performance Gains

kNN-EC improved F1-macro scores by an average of 1.17 points across various models. On GoEmotions, improvements ranged from 1.21 to 1.65 points. Unlike raw kNN, the whitened results were consistently superior to the base model.

Conclusion and Limitations

kNN-EC offers a simple, non-parametric way to make retrieval-based emotion classification both accurate and efficient. While the study focused on specific models like BERT and RoBERTa, future work will explore generalizability to larger generative models. Additionally, as with any retrieval-based method, the system remains sensitive to inherent biases in the underlying training data stored in the datastore.

Summary updated

Abstract

Retrieval-based methods have been proven effective in many NLP tasks. Previous methods use representations from the pre-trained model for similarity search directly. However, the sentence representations from the pre-trained model like BERT perform poorly in retrieving semantically similar sentences, resulting in poor performance of the retrieval-based methods. In this paper, we propose kNN-EC, a simple and efficient non-parametric emotion classification (EC) method using nearest neighbor retrieval. We use BERT-whitening to get better sentence semantics, ensuring that nearest neighbor retrieval works. Meanwhile, BERT-whitening can also reduce memory storage of datastore and accelerate retrieval speed, solving the efficiency problem of the previous methods. kNN-EC average improves the pre-trained model by 1.17 F1-macro on two emotion classification datasets.
BibTeX
@inproceedings{yin2022efficient,
  author = {Wenbiao Yin and Lin Shang},
  title = {{Efficient Nearest Neighbor Emotion Classification with BERT-whitening}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=SkeaGWnatD},
}
Figure