Linear Layer Extrapolation for Fine-Grained Emotion Classification

Mayukh Sharma, Sean O'Brien, Julian J. McAuley

Introduction

While Large Language Models (LLMs) have achieved high proficiency in general sentiment analysis, they frequently struggle with fine-grained emotion classification. Distinguishing between subtle nuances—such as grief versus remorse or anger versus disgust—remains a significant challenge. Researchers from UCSD identify two primary causes for this: a "safe label bias," where models default to high-frequency generic labels like "neutral" under uncertainty, and "late-emerging nuance," where complex emotional reasoning only crystallizes in the final layers of a Transformer.

To address these hurdles, this work introduces a method that "contrasts" the final expert layer of a model with its earlier "amateur" layers. By reinterpreting this contrast as linear layer extrapolation, the model's predictions can be dynamically adjusted to favor nuanced emotional categories over generic ones.

Methodology

The core of the proposed approach is an adaptation of the Decoding by Contrasting Layers (DoLa) technique, evolved into a process termed Linear Layer Extrapolation.

  1. Amateur Identification: For a given input, intermediate layers are evaluated to identify an "amateur" layer that exhibits the highest divergence (e.g., Cosine Distance) from the final layer's predictions.
  2. Contrastive Classification: The method penalizes the amateur layer's predictions while boosting the final expert layer's output.
  3. Linear Extrapolation: Rather than applying a static penalty, the scores from the amateur and expert layers are treated as points on a line. This line is extrapolated to a hypothetical future layer ($\ell_t$), deriving a dynamic contrastive strength that enhances stability and robustness across diverse inputs.
Evolution of classifier scores across Transformer layers, illustrating how nuanced emotions like pessimism emerge and diverge from generic categories in later stages.

This layer-wise evolution demonstrates that a model often initially classifies a sentence as "neutral" or "sad," only identifying specific nuances like "disapproving" or "pessimistic" in the final few layers. By contrasting these stages, the method effectively "flips" generic predictions to specific, correct ones.

Experimental Results

The method was evaluated on four emotion-heavy datasets—goEmotions, SuperTweetEval (tweetEmotion & tweetHate), and EmpatheticDialogues—using FLAN-T5 and DeBERTa architectures.

Performance metrics across datasets and models. (a) Recall on goEmotions, (b) F1-score on tweetEmotion, and (c) Sensitivity analysis of the extrapolation parameter.

Key Findings

  • Performance Gains: The extrapolation method consistently outperformed standard classification baselines. Notably, on the tweetHate dataset using DeBERTa-XL, the F1-score improved significantly from .599 to .685.
  • Recall Improvement: The most substantial gains were observed in Recall, indicating that the model became significantly better at identifying rare or nuanced emotions that were previously overlooked in favor of the "neutral" label.
  • Neutral-to-Specific Transitions: In the goEmotions dataset, approximately 70-80% of successful corrections involved the model moving from an initial "neutral" prediction to a specific, correct emotion such as curiosity or disapproval.

Discussion and Limitations

By framing layer contrast as a geometric extrapolation problem, the researchers achieved greater performance stability, making the system less sensitive to the specific choice of the amateur layer. However, several limitations remain. The method's effectiveness depends on a shifting probability distribution across layers; in datasets like EmpatheticDialogues, where model confidence remained "flat" throughout the network, gains were minimal. Additionally, while the dynamic approach reduces hyperparameter sensitivity, the optimal search space for amateur layers still requires careful calibration. Future research will explore whether these extrapolation benefits scale to massive 70B+ parameter models.

Summary updated

Abstract

Certain abilities of Transformer-based language models consistently emerge in their later layers. Previous research has leveraged this phenomenon to improve factual accuracy through self-contrast, penalizing early-exit predictions based on the premise that later-layer updates are more factually reliable than earlier-layer associations. We observe a similar pattern for fine-grained emotion classification in text, demonstrating that self-contrast can enhance encoder-based text classifiers. Additionally, we reinterpret self-contrast as a form of linear extrapolation, which motivates a refined approach that dynamically adjusts the contrastive strength based on the selected intermediate layer. Experiments across multiple models and emotion classification datasets show that our method outperforms standard classification techniques in fine-grained emotion classification tasks.
BibTeX
@inproceedings{sharma2024linear,
  author = {Mayukh Sharma and Sean O'Brien and Julian J. McAuley},
  title = {{Linear Layer Extrapolation for Fine-Grained Emotion Classification}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.1161},
}
Figure