Deciphering the Complaint Aspects: Towards an Aspect-Based Complaint Identification Model with Video Complaint Dataset in Finance

Sarmistha Das, Basha Mujavarsheik, R E Zera Lyngkhoi, Sriparna Saha, Alka MauryaWACV 2025

TL;DR

This paper introduces "Solution 3.0," a novel multimodal model for identifying specific financial complaint aspects within video data. They also release "MulComp," a new dataset of annotated financial video complaints, addressing a critical gap in current research and enabling more nuanced customer service analysis.

What Problem Does This Paper Address?

In today's digital landscape, financial institutions face a deluge of customer feedback, often in the form of video complaints. While these videos offer rich insights into customer grievances, extracting specific complaint aspects (e.g., "customer service," "transaction issues") from this multimodal data (text, audio, video) is a significant challenge. Existing methods often rely on unimodal data or focus on general complaint detection, failing to capture the granular details needed for effective problem resolution and service improvement. This lack of specialized datasets and models hinders the ability of organizations to truly understand and address customer pain points.

Methods

The core of this work is the "Solution 3.0" model, designed for multimodal aspect-based complaint identification. Here's a breakdown of its key components:

  • Multimodal Input Processing: The model accepts video frames and audio transcripts. Audio is processed using Whisper to generate text, and video is segmented.
  • CLIP-based Feature Extraction: A pre-trained CLIP model is used to extract embeddings from both text and image modalities. Crucially, the CLIP encoders are frozen during training, leveraging their powerful pre-existing representations.
  • Image Segment Encoder with Contextual Attention (ISEC): A novel component that enhances image embeddings by incorporating positional information and a contextual attention mechanism. This helps capture relationships within image segments.
  • Multimodal Fusion: The text and image embeddings (after ISEC processing) are concatenated to create a unified multimodal representation.
  • Transformer-based Multitasking: This fused representation is fed into a transformer encoder with multiple attention heads and blocks. The final layer performs a multitask classification, simultaneously identifying complaint aspects and whether each aspect is a complaint or non-complaint. This is framed as a multilabel classification problem.

Datasets

The paper introduces the MulComp (Multimodal Complaint) dataset.

  • Size: 433 publicly accessible financial video complaint instances.
  • Annotation: Meticulously annotated at the utterance level with five distinct financial aspects and associated complaint/non-complaint labels.
  • Creation: Curated from YouTube and Twitter, rigorously verified for authenticity and financial sector affiliation.
  • Comparison to Existing: The authors note that existing financial complaint datasets are primarily unimodal (e.g., X-FinCORP), limiting their applicability to multimodal analysis. MulComp aims to fill this gap.

Key Insights and Ideas

  • Multimodality is Key: The research strongly suggests that combining audio, video, and text provides a richer understanding of financial complaints than any single modality alone.
  • Aspect-Level Granularity Matters: Simply knowing that a complaint exists isn't enough; understanding what aspect of the service or product is problematic is crucial for actionable insights.
  • Frozen CLIP for Efficiency: Leveraging pre-trained, frozen CLIP encoders is an effective strategy for extracting robust multimodal features without the computational cost of training them from scratch.
  • ISEC's Importance: The proposed Image Segment Encoder with Contextual Attention (ISEC) significantly boosts performance, highlighting the value of specialized modules for handling multimodal fusion.
  • Multitasking for Efficiency: Performing aspect identification and complaint classification in parallel within a single model streamlines the process and potentially improves overall accuracy.

Key Results

  • Superior Performance: Solution 3.0 significantly outperforms several strong baselines (including SOTA_1, SOTA_2, ViLBERT, T5, Gemma, Mistral, GPT-3.5, and GPT-4o) across various metrics (Macro-F1, Micro-F1, Hamming Loss) on the MulComp dataset. For instance, it achieves a Micro-F1 score of 76.56 for Aspect Classification (AC) on Transactions, outperforming the next best by a notable margin.
  • Modality Impact: Interestingly, the audio-only modality often performed better than video-only for Aspect Classification (AC) tasks, achieving high scores in aspects like Customer Service and Miscellaneous. However, the multimodal approach generally yielded the best overall results.
  • ISEC Effectiveness: Removing the ISEC mechanism led to a significant performance drop of 4-5%, underscoring its importance.
  • Generalizability: Solution 3.0 showed strong performance on various multimodal and unimodal datasets beyond finance, indicating its potential for broader NLP applications.
  • Qualitative Analysis: Human evaluation confirmed Solution 3.0's ability to identify more aspects than several other models, closely matching human annotator performance.

Limitations and Future Work

  • Language Dependency: The model was trained and evaluated exclusively on English-language data. Future work should involve training on different languages and code-mixed settings to broaden applicability.
  • Sarcasm Handling: The model struggles with sarcasm, a common challenge in text analysis, due to the limited availability of curated sarcastic samples.
  • Data Imbalance: While efforts were made to balance the dataset, some aspects (like "Claimed Benefit") showed discrepancies, potentially impacting performance on those specific categories.
  • Further Refinement: The authors suggest further refinement of training data and model architecture to address occasional misclassifications, particularly for aspects like "Miscellaneous" and "Service Type."

Summary updated

BibTeX
@inproceedings{das2025deciphering,
  author = {Sarmistha Das and Basha Mujavarsheik and R E Zera Lyngkhoi and Sriparna Saha and Alka Maurya},
  title = {{Deciphering the Complaint Aspects: Towards an Aspect-Based Complaint Identification Model with Video Complaint Dataset in Finance}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure