Relative Rank Statistics for Dialog Analysis

Juan M. HuertaEMNLP 2008

Abstract

Analyzing human conversation presents unique challenges due to data sparseness, acoustic noise, and rapid topic shifts. This paper introduces the Relative Rank Differential (RRD), a non-parametric statistic designed to identify semantically salient words by tracking their frequency rank changes relative to a universal language model. Experimental results demonstrate that RRD significantly outperforms traditional methods like TF-IDF in robustness to Automatic Speech Recognition (ASR) errors and efficiency in handling short dialog segments.

Introduction

Traditional text analysis tools often rely on parametric assumptions that fail when applied to the dynamic and noisy nature of spoken dialog. Conversations are typically short, leading to sparse data, and are often transcribed via ASR systems that introduce significant word error rates. This research proposes a non-parametric approach to establish semantic saliency—identifying words that characterize the topic of a conversation without requiring complex probability distributions or large amounts of data.

Methodology: Relative Rank Differential (RRD)

The core innovation is the Relative Rank Differential (RRD). Rather than relying on raw frequency counts, RRD compares the rank of a word in a specific dialog segment ($d^s$) against its rank in a "universal" dictionary ($d^u$) representing general language usage.

The RRD statistic for a word $w$ is defined as: $$c(w) = \frac{|r_{d^u}(w) - r_{d^s}(w)|}{(r_{d^u}(w))^\alpha}$$ where $r(w)$ denotes the rank and $\alpha$ is a scaling parameter used to de-emphasize extremely rare words that might be noise. The intuition is that "salient" words are those that make significant jumps in rank—for instance, a technical term like "airbag" moving from a low universal rank to a top rank in a specific call.

For real-time applications, the system utilizes a sliding window and a FIFO queue to update ranks dynamically as the conversation progresses, allowing for live topic tracking and indexing.

Experimental Evaluation

The RRD method was evaluated using datasets from a car manufacturer's contact center, including a clean, hand-transcribed corpus and an ASR-transcribed corpus with a 24% Word Error Rate (WER).

Handling Data Sparseness

A primary advantage of RRD is its ability to extract meaningful signals from very little data. In retrieval tasks, RRD achieved nearly 90% accuracy using only 5% of a document's content. In contrast, TF-IDF required 20% of the document to reach comparable performance levels.

Retrieval accuracy as a function of the percentage of the dialog available.

Robustness to Noise

RRD demonstrates exceptional resilience to the errors common in ASR. As shown in the results below, RRD maintains high retrieval accuracy even as simulated noise ($N$) increases. At a high noise level of $N=0.4$, RRD maintained 70.9% accuracy, whereas standard TF-cosine performance dropped to 32.5%.

Retrieval accuracy across varying levels of simulated noise (N).

Conclusion

The Relative Rank Differential provides a robust, non-parametric framework for dialog analysis. By focusing on rank stability rather than raw counts, it overcomes the limitations of traditional methods in noisy and sparse environments. Future work includes expanding the feature scope beyond single words and integrating RRD into more complex semantic indexing pipelines.

Summary updated

BibTeX
@inproceedings{huerta2008relative,
  author = {Juan M. Huerta},
  title = {{Relative Rank Statistics for Dialog Analysis}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2008},
}
Figure