Sentiment and Emotion-Aware Multi-Modal Complaint Identification

Apoorva Singh, Soumyodeep Dey, Anamitra Singha, Sriparna SahaAAAI 2022 · DOI

Introduction

In the rapidly evolving landscape of e-commerce, identifying customer complaints accurately is crucial for maintaining brand reputation and customer satisfaction. Traditional Complaint Identification (CI) systems have historically relied almost exclusively on textual analysis. However, text alone can be deceptive; a customer might post a seemingly neutral statement like "The package arrived today" while attaching a photograph of a crushed box.

This research addresses the limitations of unimodal systems by proposing a framework that understands the interplay between verbal statements, visual evidence, and the underlying emotional state of the customer. By treating complaint identification as a "speech act" tied to breaches of expectation, the authors explore how auxiliary cues like anger or disappointment can significantly enhance detection accuracy.

Methodology: The MCI Framework

The authors introduce a Multi-modal Complaint Identification (MCI) framework, a multi-task adversarial deep learning model. This system is designed to simultaneously identify complaints, recognize emotions, and classify sentiment.

The architecture consists of several sophisticated components:

  • Feature Extraction: Semantic meanings are captured from text using SBERT (Sentence-BERT), while visual features are extracted via a pre-trained ResNet-152 model.
  • Dual-Attention Mechanism: To bridge the gap between modalities, the model employs two types of attention:
  • Inter-segment Inter-modal Attention: Aligns specific text segments with relevant image regions (e.g., linking the word "broken" to a visual crack).
  • Contextual Inter-modal Attention: Evaluates the broader context of the review to distinguish between mixed sentiments within a single post.
  • Adversarial Learning: An adversarial loss function is implemented to separate shared features (useful across all tasks) from task-specific features, ensuring the model remains focused on the primary goal of complaint identification.

The CESAMARD Dataset

A significant contribution of this work is the introduction of the CESAMARD (Complaint, Emotion, and Sentiment Annotated Multi-modal Amazon Reviews Dataset). This benchmark contains 3,962 reviews scraped from Amazon India across five diverse domains, including Electronics, Fashion, and Books.

Distribution of reviews across emotion labels in the CESAMARD dataset.

Each record in the dataset is manually annotated for complaint status, three sentiment classes, and six basic Ekman emotions (anger, disgust, fear, happiness, sadness, and surprise). This represents the first publicly available multi-modal dataset specifically designed for CI that incorporates such fine-grained emotional labeling.

Analysis and Insights: The Power of Multi-modality

The study highlights the "Modality Mismatch" advantage. In cases where text and images provide conflicting information, the multi-modal approach succeeds where text-only models fail. For instance, a sarcastic or neutral text paired with a photo of a defect is correctly flagged as a complaint.

An example of modality mismatch where text and image combined reveal a complaint.

Furthermore, the research demonstrates that sentiment alone is often too broad. By incorporating specific emotions, the model gains a "finer-grained" understanding of user intent. The multi-task synergy suggests that training for emotion and sentiment recognition actually improves the model's primary performance in identifying complaints.

Results and Future Directions

The proposed MCI model achieved a weighted F1-score of 89.07% and an Accuracy of 89.64%, significantly outperforming the previous state-of-the-art text-only models (83.18% F1-score). Ablation studies confirmed that removing either the image component or the auxiliary tasks led to a measurable decline in performance.

Despite these successes, challenges remain regarding data imbalance (non-complaints outnumber complaints) and the detection of irony. Future work aims to expand this framework to include audio and video modalities to better serve the growing trend of video-based product reviews on social media.

Summary updated

BibTeX
@inproceedings{singh2022sentiment,
  author = {Apoorva Singh and Soumyodeep Dey and Anamitra Singha and Sriparna Saha},
  title = {{Sentiment and Emotion-Aware Multi-Modal Complaint Identification}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i11.21476},
}
Figure