The Inner Sentiments of a Thought

Chris Gagne, Peter DayanarXiv 2023 · DOI · arXiv

Abstract

Large Language Models (LLMs) are often treated as black boxes regarding their internal emotional processing. This paper presents a method to predict the full probability distribution of a sentence's final sentiment word-by-word. By training quantile regression models on GPT-2's hidden states, we analyze emotional dynamics—such as the sentiment-reversing effect of the word "but"—and demonstrate fine-grained control over text generation by targeting specific emotional intensities.

Introduction

While LLMs generate remarkably human-like text, standard sentiment analysis typically provides only a static "point estimate" after a sentence is complete. This fails to capture the dynamic nature of human thought, where expectations are built and subverted in real-time. We address two primary gaps: understanding real-time emotional trajectories and achieving precise steering of emotional intensity in generated text.

Methods

The core of our approach involves augmenting a base model, GPT-2 Large, with specialized "quantile models."

Quantile Regression Architecture

Instead of predicting a single mean sentiment, we trained single-layer neural networks to predict 10 equally spaced quantiles (from 5% to 95%) of the final sentiment distribution. These models utilize the hidden states of the LLM at each token as input.

Overview of the quantile model architecture and the resulting distribution of predicted end scores.

Distributional Prediction and Steering

As a sentence progresses, the model predicts how it might conclude. For a prefix like "I think this is...", the initial distribution is broad, reflecting many possibilities. As more tokens are added, the distribution "collapses" toward the final observed score. To steer generation, we re-weight the LLM's next-token probabilities based on how much of a token's predicted distribution aligns with a target emotional "tail" (e.g., the top 5% of admiration).

Key Insights

The "But" Effect

A significant finding is the model's ability to quantitatively capture how the word "but" acts as a sentiment reverser. Analysis shows that the "outermost" quantiles often begin to anticipate an emotional flip even before the word "but" is explicitly generated, suggesting that the LLM's internal states encode an imminent shift in tone.

Visualization of the "But" effect, showing how valence quantiles shift across tokens and the correlation between median valence before and after the conjunction.

Judgment and Variance

Certain words, termed "peak variance" tokens, do not necessarily change the mean sentiment but cause a spike in the predicted distribution's variance. Intensifiers like "really," "extremely," or "honestly" signal that a strong judgment is forthcoming, though the direction (positive or negative) remains uncertain until the following tokens appear.

Results and Discussion

The quantile models demonstrated high calibration, with valence predictions deviating less than 1% from perfect calibration. Furthermore, the steering mechanism successfully produced text with specific emotional nuances, such as generating highly determined rhetoric when targeting the upper quantiles of "determination."

Inverting the model to infer user bias revealed that Reddit users with self-reported depression exhibited a "pessimistic" bias, where their text was more accurately described by models targeting lower (more negative) sentiment quantiles. This suggests a link between LLM internal states and real-world psychological states, mirroring concepts from Distributional Temporal Difference (TD) learning in reinforcement learning.

Limitations and Future Work

While GPT-2 provided a robust baseline, its tendency toward repetitive text is a limitation. Future work will involve scaling this approach to modern architectures like Llama 3 and applying it to long-form narratives to track emotional arcs. Additionally, there is potential for real-time monitoring in therapeutic settings to identify cognitive biases as they emerge in natural language.

Summary updated

Abstract

Transformer-based large-scale language models (LLMs) are able to generate highly realistic text. They are duly able to express, and at least implicitly represent, a wide range of sentiments and color, from the obvious, such as valence and arousal to the subtle, such as determination and admiration. We provide a first exploration of these representations and how they can be used for understanding the inner sentimental workings of single sentences. We train predictors of the quantiles of the distributions of final sentiments of sentences from the hidden representations of an LLM applied to prefixes of increasing lengths. After showing that predictors of distributions of valence, determination, admiration, anxiety and annoyance are well calibrated, we provide examples of using these predictors for analyzing sentences, illustrating, for instance, how even ordinary conjunctions (e.g., "but") can dramatically alter the emotional trajectory of an utterance. We then show how to exploit the distributional predictions to generate sentences with sentiments in the tails of distributions. We discuss the implications of our results for the inner workings of thoughts, for instance for psychiatric dysfunction.
BibTeX
@misc{gagne2023inner,
  author = {Chris Gagne and Peter Dayan},
  title = {{The Inner Sentiments of a Thought}},
  year = {2023},
  eprint = {2307.01784},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.01784},
}
Figure