HotSpots: Visualizing Edits to a Text

Srinivas Bangalore, David SmithEMNLP 2008

Abstract

In large-scale customer service, email templates are essential for efficiency, yet template authors often lack visibility into how agents customize these responses. This paper presents HotSpots, a visualization tool that uses heatmaps to highlight frequently edited sections of email templates. By aligning templates with final agent responses and employing a regression model to predict Average Handling Time (AHT) based on edit complexity, HotSpots enables data-driven refinement of customer service responses. The tool identifies "hot spots" requiring revision and discovers emerging templates from common agent-authored variations, ultimately reducing AHT without compromising quality.

Introduction

Customer service centers rely heavily on asynchronous communication like email to manage workloads. To maintain speed and consistency, agents use pre-written templates. However, a significant gap exists between template design and actual usage. If agents consistently rewrite large portions of a template, the intended efficiency gains are lost. This paper addresses the lack of feedback for template authors by introducing HotSpots, a system to track, visualize, and analyze real-world template modifications.

Methods

The authors propose two primary technical contributions: the HotSpots visualization and an AHT prediction model.

The HotSpots Visualization

The tool aligns a master template with agent-sent responses using a sentence-level version of the Levenshtein (edit) distance. Sentences are color-coded based on their modification frequency: red indicates edits in over 50% of cases, while orange, green, and blue represent lower frequencies. A centroid display allows authors to hover over a sentence and see the most common agent modifications, calculated via $k$-median clustering using $n$-gram overlap as the distance metric.

The HotSpots interface showing color-coded template versions and a popup displaying common agent modifications.

AHT Modeling and Template Extraction

To quantify the impact of these edits, the authors developed a linear regression model to estimate Average Handling Time (AHT). This model incorporates features such as customer input length, response length, and various edit types (insertions, deletions, substitutions, and block-level changes).

The linear regression equation used to model Average Handling Time (AHT) based on edit complexity features.

By clustering agent responses that deviate significantly from the template, the system can identify "emerging templates"—common response patterns that agents are forced to write manually because a suitable template doesn't exist.

Key Results

  • Operational Deployment: The HotSpots tool was successfully deployed in a live customer care environment, receiving "extremely positive" qualitative feedback from analysts.
  • AHT Prediction: The linear regression model achieved a correlation coefficient of 0.3822. While not high enough for perfect individual prediction, it proved sufficient for ranking and identifying which types of edits contribute most to handling delays.
  • Template Discovery: In a test case of 2,005 responses, the system identified a distinct cluster of 206 responses that required significantly higher AHT (1140 seconds vs. 971 seconds for the standard cluster). This quantitative gap proved that these cases represented a specific sub-issue requiring its own dedicated template.

Conclusion

HotSpots provides a novel feedback loop for text-based workflows by treating templates as evolving assets and leveraging agent behavior as a source of truth. By seeing where agents "struggle" through heavy editing, authors can perform surgical improvements to templates rather than guessing why efficiency is low. Future work includes exploring more advanced semantic metrics like BLEU and refining the granularity of the visualization to the phrase level to provide even more precise feedback.

Summary updated

BibTeX
@inproceedings{bangalore2008hotspots,
  author = {Srinivas Bangalore and David Smith},
  title = {{HotSpots: Visualizing Edits to a Text}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2008},
}
Figure