Ranking Reader Emotions Using Pairwise Loss Minimization and Emotional Distribution Regression

Kevin Hsin-Yih Lin, Hsin-Hsi ChenEMNLP 2008

Abstract

Traditional sentiment analysis typically focuses on identifying a single dominant emotion intended by a writer. However, reader reactions to content are often diverse and distributed across multiple emotional categories. This paper addresses the challenge of ranking reader emotions by treating them as a distribution rather than a single label. We propose two distinct machine learning frameworks: Pairwise Loss Minimization (PLM), which optimizes the relative order of emotion pairs, and Emotional Distribution Regression (EDR), which predicts the specific percentage of readers for each emotion. Evaluated on a large-scale dataset of Chinese news articles, our results show that EDR achieves high accuracy in predicting the primary emotion, while PLM excels in overall ranking quality.

Introduction

Most existing research in sentiment analysis focuses on "writer-emotion"—the emotion the author intends to express. In contrast, "reader-emotion" captures the actual affective response of the audience. A critical observation is that reader reactions are rarely unanimous; a single news article can evoke a complex profile of emotions such as "Heartwarming," "Happy," and "Sad" simultaneously, in varying proportions.

Figure 1: An example of the emotional distribution of readers for a news article, showing that reactions are distributed across multiple categories rather than concentrated on a single label.

The goal of this work is to move beyond simple classification and instead solve a ranking problem: predicting the relative popularity of different emotions for any given text.

Methodology

We propose two frameworks to model the ranking of reader emotions:

Pairwise Loss Minimization (PLM)

PLM treats the problem as a preference learning task. Inspired by Ranking SVMs, it focuses on the relative order between pairs of emotions (e.g., is "Happy" more prevalent than "Angry"?). The model minimizes a loss function based on the number of misordered pairs. We train binary classifiers for all possible emotion pairs and aggregate their outputs using a "Merge Pairwise Orders" algorithm to produce a final ranked list.

Emotional Distribution Regression (EDR)

EDR takes a direct approach by attempting to predict the exact percentage of readers associated with each of the eight emotional categories. We employ Support Vector Regression (SVR) to estimate these values independently. The final ranking is obtained by sorting the predicted percentages in descending order.

Feature Engineering

The models utilize three primary feature sets extracted from Chinese news text:

  • Character Bigrams: Capturing local context through pairs of consecutive characters.
  • Segmented Words: Utilizing the Stanford NLP Chinese word segmenter.
  • Metadata: Incorporating non-textual information such as the publishing agency, category (e.g., sports, politics), and time of publication.

Experimental Setup

The study utilizes a dataset from Yahoo! Kimo News consisting of 37,416 articles. Each article includes reader votes across eight categories: happy, sad, angry, surprising, boring, heartwarming, awesome, and useful. The dataset was split into 25,975 articles for training and 11,441 for testing. The "ground truth" for each article is derived from an average of 626 reader votes.

Results and Discussion

We evaluated the performance of PLM and EDR against a Nearest Neighbor (NN) baseline using three metrics: Accuracy at position $k$ (ACC@k), Normalized Discounted Cumulative Gain (NDCG), and Successive Accuracy (SACC).

Figure 3: Performance comparison of the Nearest Neighbor (NN) baseline, Emotional Distribution Regression (EDR), and Pairwise Loss Minimization (PLM) across ACC@, NDCG@, and SACC@ metrics.

As shown in Figure 3, both proposed methods significantly outperform the baseline. EDR demonstrated the highest precision for the top-ranked emotion, achieving an ACC@1 of 75.1%. However, PLM proved more robust in capturing the "long tail" of the distribution, maintaining higher ranking accuracy across the entire list of emotions.

A key finding was the correlation between emotional ambiguity and model performance. Emotions that are conceptually similar to humans (e.g., "Awesome" and "Surprising") were more frequently misranked by the models. Furthermore, the combination of textual features with metadata consistently yielded the best results, highlighting the importance of context in reader response.

Conclusion

This research demonstrates that reader emotions can be effectively ranked using pairwise and regression-based machine learning techniques. While EDR is optimal for identifying the primary reader response, PLM offers a more accurate representation of the full emotional profile. Future work will explore hybrid models that combine these strengths and investigate the integration of emotional rankings into information retrieval systems to enable "vibe-based" content discovery.

Summary updated

BibTeX
@inproceedings{lin2008ranking,
  author = {Kevin Hsin-Yih Lin and Hsin-Hsi Chen},
  title = {{Ranking Reader Emotions Using Pairwise Loss Minimization and Emotional Distribution Regression}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2008},
}
Figure