TL;DR
This paper introduces a novel method to automatically identify crucial segments and expressions within lengthy contact center conversations. By analyzing changes in a classifier's accuracy over time, they pinpoint segments that significantly influence business outcomes, enabling more efficient and insightful text mining for business improvement.
What Problem Does This Paper Address?
Contact centers generate vast amounts of conversational data (transcripts, emails, etc.) that hold valuable business insights. However, manually analyzing this data is time-consuming and requires significant workload, often only covering a small fraction of the total volume. While text mining can help, it's critical to identify the "important" parts of these conversations – specific segments and expressions – to focus on. Without this, analysis can be superficial or lead to incorrect conclusions. The challenge lies in automatically pinpointing these crucial conversational elements from lengthy, often redundant transcripts, especially when subtle differences in dialogue can lead to vastly different business outcomes (e.g., a car reservation being picked up or not).
Methods
The core of the proposed method involves Trigger Segment Detection. The authors model conversations as a sequence of turns and create chronologically cumulative data sets (Dk). They then train a classifier (SVM in their experiments) to distinguish between different business outcomes (e.g., "picked-up" vs. "not picked-up"). By plotting the classifier's accuracy as more of the conversation is included (from D1 to Dk), they identify "trigger segments". These are portions of the conversation where the accuracy curve shows a significant increase or decrease, indicating the presence or absence of features that influence the outcome.
Once trigger segments are identified, the paper focuses on Effective Expression Extraction. Within these segments, they use a combination of metrics:
- χ² statistic: To identify expressions that are statistically representative of a particular outcome class.
- New Expression Metric (new(w)): To detect expressions that appear for the first time in a trigger segment and have a significant impact on the outcome, distinguishing them from expressions that have already been influential earlier in the conversation.
These extracted expressions are then used to define viewpoints, which are essentially key associations between entities that provide valuable insights into business outcomes.
Datasets
The primary dataset used in this paper consists of 914 recorded calls from a car rental help desk, which were manually transcribed. These calls were categorized into three types: Reservation Calls (further split into picked-up and not picked-up), Unbooked Calls, and Service Calls. For their experiments, they focused on 137 reservation calls, using 100 for training and 37 for trigger segment detection.
Key Insights and Ideas
- Conversational Flow is Key: The method leverages the inherent structure of business conversations (opening, request, confirmation, closing) to identify critical points.
- Accuracy Gradients Reveal Importance: The most striking insight is that monitoring the change in classification accuracy over the course of a conversation, rather than just the final accuracy, is a powerful way to detect influential segments.
- Early Segments are Highly Informative: The experiments showed that even the first few turns of a conversation can be highly predictive of the outcome, but simply using the entire conversation doesn't necessarily lead to better results if the influential features aren't properly utilized.
- Subtle Differences Matter: The method successfully identified subtle differences in conversations that led to opposite outcomes, which would be difficult for humans to spot manually.
- Viewpoints Emerge from Data: Instead of relying solely on domain expert-defined viewpoints, this method allows for data-driven identification of viewpoints based on identified trigger segments and expressions.
Key Results
- The method successfully identified trigger segments in car rental reservation calls that were highly correlated with the outcome (picked-up vs. not picked-up). For example,
seg(1,2)(early turns) andseg(10,15)were identified as significant. - In
seg(1,2), expressions like "make" and "reservation" were correlated with "pick up," while "rate" and "check" were correlated with "not picked up." This suggests customer intention at the start of the call is impactful. - In
seg(10,15), discount-related phrases and mentions of good rates by the agent were found to influence the outcome. - Applying these insights to a real car rental center led to actionable insights. For instance, customers with a "strong start" (booking_customer) were more likely to pick up their reserved car, while those with a "weak start" (rates_customer) were less likely.
- Mentioning discount phrases to "rates_customer" significantly improved the "pick up" ratio by 0.476.
- A pilot group of agents trained on these insights showed a 4.75% increase in pick-up ratio, compared to a 2.08% increase in a control group, suggesting a meaningful improvement.
Limitations and Future Work
- Manual Transcription: The experiments used manually transcribed calls. Future work aims to use noisy output from automatic speech recognition (ASR) systems to make the process more scalable.
- Semantic Segment Information: While the method doesn't strictly require pre-defined semantic segments (like "opening" or "enquiries"), using them can also help in trigger segment detection. The paper notes that their method works even without labeled segmentation information.
- Feature Selection: The paper discusses the use of Information Gain for feature selection to improve classification accuracy, noting that it helps reduce noisy features and obtain similar trigger segments.
- Generalizability: While tested on car rental data, the robustness of the method across different domains and business outcomes would be an interesting area for future exploration.
Summary updated