An Improvement of Example-based Emotion Estimation Using Similarity between Sentence and each Corpus

Kenichi Mishina, Seiji Tsuchiya, Motoyuki Suzuki, Fuji RenJournal of Natural Language Processing 2010 · DOI

Abstract

発話文を感情ごとに分類したコーパスを構築し,入力文と最も類似度が高い発話文を含むコーパスの感情を推定結果として出力する用例ベースの感情推定手法が提案されている.従来手法ではコーパスを構築する際,発話テキストの収集者が個人個人で発話文の分類先を決定しているため,分類先を決定する基準が個々によってぶれてしまう.これにより,例えば“希望”のコーパスの中に喜びの発話文が混じるといったことが起こり,推定成功率を下げてしまう.本稿ではこの問題を解決するため,コーパスごとにおける入力文の形態素列の出現回数を用いて,入力文とコーパスの類似度を定義する.そしてこの類似度を従来手法に導入した新たな類似度計算式を提案する.これにより,誤って分類されてしまった発話文の影響を緩和することができる.評価実験では従来手法と比べて成功率が 21.5 ポイント向上し,提案手法の有効性が確認できた.

Introduction

Example-based emotion estimation typically works by comparing an input sentence to a large database (corpus) of pre-tagged sentences. If an input is found to be most similar to a sentence tagged as "Joy," the system labels it accordingly. However, this approach faces a significant challenge: human subjectivity. Emotions are ambiguous, and different taggers often disagree on labels. For instance, a sentence like "I'm looking forward to it" might be tagged as "Hope" by one person and "Joy" by another. This "noisy label" problem degrades performance when using standard similarity measures like BLEU scores, which are sensitive to individual mislabeled examples.

The RECARE Method

To address this, the researchers developed RECARE (Robust Emotion CAtegorization with Rough Emotion Corpora). The method moves beyond simple sentence-to-sentence matching by introducing a "corpus-wide" perspective to the similarity calculation.

Overview of the example-based emotion estimation process.

RECARE introduces two major innovations:

  1. Corpus-wide Frequency Penalty ($FP_n$): Instead of just matching an input against one specific sentence, the system analyzes the distribution of the input's N-grams across the entire emotion corpus. If a phrase appears frequently across the whole "Joy" corpus but rarely in "Anger," it receives a higher weight for Joy. This acts as a statistical "sanity check" against individual tagging errors.
  2. Arithmetic Mean Integration: Standard BLEU scores use a geometric mean of N-gram matches. If a single high-order N-gram (like a 4-gram) doesn't match, the entire score can drop to zero. RECARE switches to an arithmetic mean, allowing the system to retain valuable information from shorter, more common word combinations even when long, specific phrases don't find a perfect match.
Example of unigram counts used for similarity calculation.

Experimental Results

The researchers constructed a custom corpus of 3,352 sentences across four emotions: Joy, Anger, Disgust, and Hope. The proposed RECARE method achieved an emotion estimation accuracy of 81.8%, a massive 21.5-point improvement over the 60.3% achieved by the traditional BLEU-based method.

Key insights from the evaluation include:

  • Robustness to Noise: In stress tests where the database was intentionally "poisoned" with mislabeled sentences, RECARE's performance remained stable, maintaining a 20+ point lead over traditional methods.
  • Handling Sparsity: By avoiding the "zero-frequency problem" of geometric means, the system proved much more practical for the short, messy text typical of natural communication.
  • Superiority over SVM: While Support Vector Machines (SVM) often overfit on long word sequences, RECARE remained stable even with high-order N-grams ($N=6$), capturing specific emotional expressions without losing generalizability.

Conclusion and Future Work

By leveraging aggregate corpus statistics, RECARE effectively neutralizes the noise caused by inconsistent human tagging. Future research will focus on expanding the system to handle more complex emotional spectrums and implementing multi-label outputs to better reflect the nuance of human speech.

Summary updated

Abstract

発話文を感情ごとに分類したコーパスを構築し,入力文と最も類似度が高い発話文を含むコーパスの感情を推定結果として出力する用例ベースの感情推定手法が提案されている.従来手法ではコーパスを構築する際,発話テキストの収集者が個人個人で発話文の分類先を決定しているため,分類先を決定する基準が個々によってぶれてしまう.これにより,例えば“希望”のコーパスの中に喜びの発話文が混じるといったことが起こり,推定成功率を下げてしまう.本稿ではこの問題を解決するため,コーパスごとにおける入力文の形態素列の出現回数を用いて,入力文とコーパスの類似度を定義する.そしてこの類似度を従来手法に導入した新たな類似度計算式を提案する.これにより,誤って分類されてしまった発話文の影響を緩和することができる.評価実験では従来手法と比べて成功率が 21.5 ポイント向上し,提案手法の有効性が確認できた.
BibTeX
@article{mishina2010improvement,
  author = {Kenichi Mishina and Seiji Tsuchiya and Motoyuki Suzuki and Fuji Ren},
  title = {{An Improvement of Example-based Emotion Estimation Using Similarity between Sentence and each Corpus}},
  journal = {Journal of Natural Language Processing},
  year = {2010},
  doi = {10.5715/jnlp.17.4\_91},
}
Figure