Entity-level Sentiment Analysis in Contact Center Telephone Conversations

Xue-yong Fu, Cheng Chen, Md Tahmid Rahman Laskar, Shayna Gardiner, Pooja Hiranandani, Shashi Bhushan TnEMNLP 2022

TL;DR

This paper introduces an Entity-level Sentiment Analysis (ELSA) system tailored for noisy English telephone conversations from contact centers. They propose two novel approaches, a DistilBERT-based model and a CNN-based model with heuristics, and demonstrate their effectiveness in identifying sentiment towards specific entities like products or companies, a crucial task for business insights.

What Problem Does This Paper Address?

Businesses rely heavily on contact center interactions to understand customer sentiment towards their products and services. While Aspect-Based Sentiment Analysis (ABSA) exists, it often focuses on fine-grained aspects (e.g., "battery life") rather than the overall sentiment towards a named entity (e.g., "iPhone"). This paper tackles the challenge of performing ELSA directly on noisy, real-world telephone transcripts. These transcripts, generated by Automatic Speech Recognition (ASR) systems, are prone to errors, disfluencies (like "um" and "uh"), and linguistic variations, making traditional NLP techniques less effective. Furthermore, there's a lack of publicly available datasets specifically for ELSA in this domain, necessitating the creation of a new benchmark.

Methods

The authors propose two main approaches:

  • DistilBERT-based Model: This approach leverages the lightweight DistilBERT model. It involves two stages:
  • NER Tagging: A DistilBERT-based Named Entity Recognition (NER) model, trained on business conversation data, identifies product and organization entities. A special _NE_ tag is inserted before these entities in the input to help the sentiment model.
  • Entity Sentiment Detection: Another DistilBERT model is fine-tuned for sentiment classification. Crucially, this model is augmented with an additional prediction layer to simultaneously extract opinion words associated with the identified entities. To enhance performance, a transfer learning strategy is employed: the model is first fine-tuned on the Stanford Sentiment Treebank (SST) dataset for general sentence sentiment analysis, and then further fine-tuned on their custom ELSA dataset.
  • CNN-based Sentiment Model Supplemented with Heuristic Rules: This method uses a two-step process:
  • General Sentiment Analysis: A multi-class CNN model classifies the utterance's sentiment and extracts keywords indicative of that sentiment.
  • Heuristic-based Opinion Word Extraction: Linguistic heuristics are applied to identify opinion words from the extracted keywords that are most likely associated with the detected entities. These heuristics categorize potential opinion words based on their part of speech (verb-based, adjective-based, noun-based) and syntactic patterns.

Datasets

The paper introduces a newly constructed dataset for ELSA in contact center conversations. Due to the infrequency of entity-level sentiment events, they employed a sampling strategy:

  • Initially, 13,000 utterances were sampled using pre-existing NER and sentiment analysis models to identify those containing at least one named entity and a positive or negative sentiment.
  • An additional 10,000 utterances with at least one entity but neutral sentiment were sampled to balance the dataset.
  • The resulting 23,000 utterances were manually annotated by independent annotators to determine entity sentiment (positive, neutral, negative) and identify opinion terms.

The DistilBERT-based model also utilizes the Stanford Sentiment Treebank (SST) dataset for initial transfer learning.

Key Insights and Ideas

  • Noisy Data is a Major Hurdle: The inherent noise and disfluencies in ASR-generated transcripts pose significant challenges for ELSA, requiring robust models.
  • Transfer Learning Boosts Performance: Fine-tuning a general sentiment model (like DistilBERT on SST) before training on the specific ELSA task significantly improves performance, suggesting that learning general sentiment cues is beneficial.
  • Joint Opinion Word Extraction is Key: The ability of the DistilBERT model to jointly extract opinion words alongside sentiment classification is a critical feature for providing richer business insights.
  • Heuristics Struggle with Recall: While heuristic-based approaches can achieve high precision, they often falter in recall due to their inability to capture linguistic variations effectively.

Key Results

  • DistilBERT + SST Outperforms: The DistilBERT + SST model achieved the best overall F1 scores for both entity sentiment classification (Ent) and opinion word extraction (OP).
  • DistilBERT Dominates CNN: Both DistilBERT variations significantly outperformed the CNN + Heuristics model in terms of F1 scores for both tasks. For opinion word extraction, DistilBERT + SST achieved a 38.48% higher F1 score than CNN + Heuristics.
  • Precision vs. Recall Trade-off: The CNN + Heuristics model showed impressive precision but poor recall, especially for opinion word extraction (16% recall), leading to lower F1 scores.
  • Robustness Findings: Both models performed poorly on longer utterances (> 45 tokens), indicating challenges in modeling long-term dependencies. The DistilBERT + SST model was also more sensitive to the number of entities in an utterance, with its F1 score dropping when more than one entity was present.

Limitations and Future Work

  • Domain and Language Specificity: The current models are trained on English business telephone conversations and may not generalize well to other domains, input types (e.g., written text), or languages.
  • NER Bias: The NER component of the DistilBERT model exhibits bias towards more frequent entities, potentially missing rarer ones and impacting overall performance.
  • Future Directions: The authors plan to extend their methods to other applications within entity recognition for telephone transcripts and explore improvements for utterances with multiple entities. They also aim to investigate how to improve model performance on such complex inputs.

Summary updated

BibTeX
@inproceedings{fu2022entity,
  author = {Xue-yong Fu and Cheng Chen and Md Tahmid Rahman Laskar and Shayna Gardiner and Pooja Hiranandani and Shashi Bhushan Tn},
  title = {{Entity-level Sentiment Analysis in Contact Center Telephone Conversations}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
}
Figure