Abstract: Retrieval-based methods have been proven effective in many NLP tasks. Previous methods use representations from the pre-trained model for similarity search directly. However, the sentence representations from the pre-trained model like BERT perform poorly in retrieving semantically similar sentences, resulting in poor performance of the retrieval-based methods. In this paper, we propose kNN-EC, a simple and efficient non-parametric emotion classification (EC) method using nearest neighbor retrieval. We use BERT-whitening to get better sentence semantics, ensuring that nearest neighbor retrieval works. Meanwhile, BERT-whitening can also reduce memory storage of datastore and accelerate retrieval speed, solving the efficiency problem of the previous methods. kNN-EC average improves the pre-trained model by 1.17 F1-macro on two emotion classification datasets.
Introduction
While Large Language Models (LLMs) like BERT are excellent at understanding text, their raw "sentence embeddings"—the mathematical vectors representing a sentence—have a hidden flaw: they are anisotropic. This means the vectors occupy a narrow, cone-shaped region in space rather than being spread out evenly.
In this cramped space, sentences that are semantically different can end up very close to each other, making simple similarity searches (like k-Nearest Neighbors) perform poorly. Previous attempts to fix this required adding complex, trainable parameters. We introduce kNN-EC, a "plug-and-play" method that boosts emotion classification by combining standard deep learning models with a whitened kNN retrieval mechanism.
Methodology
The kNN-EC framework operates in three main stages:
- Datastore Creation: Every training sentence is passed through a pre-trained model (like RoBERTa) to generate a vector representation.
- BERT-whitening: Using a simple linear transformation (Singular Value Decomposition), we "whiten" the vectors. This process centers the data at zero and ensures uniform variance in all directions, making the space isotropic. Crucially, we also perform dimension reduction here, keeping only the most important components.
- Inference & Interpolation: For a new sentence, the system calculates its whitened vector and finds the $k$ most similar neighbors in the datastore. The final prediction is a weighted average of the deep learning model’s output and the distribution of the retrieved neighbors.
Experimental Setup
The model was evaluated on two widely recognized emotion classification benchmarks:
- GoEmotions: A large-scale dataset of Reddit comments labeled with Ekman-style emotions (joy, anger, fear, sadness, disgust, and surprise).
- ISEAR: A dataset containing personal reports of emotional events across seven categories.
Key Insights and Results
Whitening is Essential for Retrieval
Without whitening, the kNN component is highly unstable and often performs worse than the base model. Whitening transforms the "messy" BERT space into one where distance actually correlates with emotional meaning. Visualization via t-SNE shows that before whitening, categories like "sadness" and "surprise" overlap significantly. After whitening, these clusters become distinct, explaining the jump in accuracy.
Efficiency via Geometry
By reducing the dimensionality of the vectors during the whitening process (e.g., to 128 or 256 dimensions), we reduced the memory footprint of the datastore by over 10x while actually improving classification performance. This solves the "storage bloat" usually associated with retrieval-augmented models.
Performance Gains
kNN-EC improved F1-macro scores by an average of 1.17 points across various models. On GoEmotions, improvements ranged from 1.21 to 1.65 points. Unlike raw kNN, the whitened results were consistently superior to the base model.
Conclusion and Limitations
kNN-EC offers a simple, non-parametric way to make retrieval-based emotion classification both accurate and efficient. While the study focused on specific models like BERT and RoBERTa, future work will explore generalizability to larger generative models. Additionally, as with any retrieval-based method, the system remains sensitive to inherent biases in the underlying training data stored in the datastore.
Summary updated