Abstract
Large Language Models (LLMs) are often treated as black boxes regarding their internal emotional processing. This paper presents a method to predict the full probability distribution of a sentence's final sentiment word-by-word. By training quantile regression models on GPT-2's hidden states, we analyze emotional dynamics—such as the sentiment-reversing effect of the word "but"—and demonstrate fine-grained control over text generation by targeting specific emotional intensities.
Introduction
While LLMs generate remarkably human-like text, standard sentiment analysis typically provides only a static "point estimate" after a sentence is complete. This fails to capture the dynamic nature of human thought, where expectations are built and subverted in real-time. We address two primary gaps: understanding real-time emotional trajectories and achieving precise steering of emotional intensity in generated text.
Methods
The core of our approach involves augmenting a base model, GPT-2 Large, with specialized "quantile models."
Quantile Regression Architecture
Instead of predicting a single mean sentiment, we trained single-layer neural networks to predict 10 equally spaced quantiles (from 5% to 95%) of the final sentiment distribution. These models utilize the hidden states of the LLM at each token as input.
Distributional Prediction and Steering
As a sentence progresses, the model predicts how it might conclude. For a prefix like "I think this is...", the initial distribution is broad, reflecting many possibilities. As more tokens are added, the distribution "collapses" toward the final observed score. To steer generation, we re-weight the LLM's next-token probabilities based on how much of a token's predicted distribution aligns with a target emotional "tail" (e.g., the top 5% of admiration).
Key Insights
The "But" Effect
A significant finding is the model's ability to quantitatively capture how the word "but" acts as a sentiment reverser. Analysis shows that the "outermost" quantiles often begin to anticipate an emotional flip even before the word "but" is explicitly generated, suggesting that the LLM's internal states encode an imminent shift in tone.
Judgment and Variance
Certain words, termed "peak variance" tokens, do not necessarily change the mean sentiment but cause a spike in the predicted distribution's variance. Intensifiers like "really," "extremely," or "honestly" signal that a strong judgment is forthcoming, though the direction (positive or negative) remains uncertain until the following tokens appear.
Results and Discussion
The quantile models demonstrated high calibration, with valence predictions deviating less than 1% from perfect calibration. Furthermore, the steering mechanism successfully produced text with specific emotional nuances, such as generating highly determined rhetoric when targeting the upper quantiles of "determination."
Inverting the model to infer user bias revealed that Reddit users with self-reported depression exhibited a "pessimistic" bias, where their text was more accurately described by models targeting lower (more negative) sentiment quantiles. This suggests a link between LLM internal states and real-world psychological states, mirroring concepts from Distributional Temporal Difference (TD) learning in reinforcement learning.
Limitations and Future Work
While GPT-2 provided a robust baseline, its tendency toward repetitive text is a limitation. Future work will involve scaling this approach to modern architectures like Llama 3 and applying it to long-form narratives to track emotional arcs. Additionally, there is potential for real-time monitoring in therapeutic settings to identify cognitive biases as they emerge in natural language.
Summary updated