Modeling Dialogues with Hashcode Representations: A Nonparametric Approach

Sahil Garg, Irina Rish, Guillermo A. Cecchi, Palash Goyal, Sarik Ghazarian, Shuyang Gao, Greg Ver Steeg, Aram Galstyan

Introduction

Modern open-domain dialogue generation is currently dominated by deep neural networks. While powerful, these models are data-hungry, computationally expensive to train, and often inflexible when incorporating specialized objective functions. These challenges are particularly acute in domains like automated psychotherapy, where training data is scarce, sensitive, and characterized by highly imbalanced interaction patterns between patients and therapists.

To address these limitations, this paper introduces a novel nonparametric framework that utilizes binary hashcode representations for dialogue modeling. By compressing utterances into hashcodes and optimizing them via mutual information, the proposed approach offers a highly efficient alternative that excels in specialized, small-data environments.

The Proposed Framework

The authors propose a three-stage approach that decouples semantic inference from textual generation. This separation allows the model to first determine the "essence" of a response in a compressed, noise-free space before generating the final text.

The three-stage nonparametric framework for dialogue modeling, involving representation learning, inference via decision trees, and mapping back to text.

1. Representation Learning via Hashing

Text utterances are mapped into compressed binary vectors (hashcodes) using kernelized Locality Sensitive Hashing (LSH). The authors explore three variants: LSH-RkNN (k-Nearest Neighbors), LSH-RMM (Max-Margin/SVM), and LSH-RLSTM.

2. Optimization and Inference

To capture the semantic alignment—or "working alliance"—between speakers, the model maximizes a novel lower bound on the Mutual Information (MI) between the hashcodes of consecutive utterances. Following this optimization, an ensemble of Random Forest classifiers is trained to predict the therapist's hashcode based on the patient's input.

3. Textual Response Generation

The predicted hashcode is mapped back to text. This is achieved either through retrieval from the training set or by using a "kernelized hashing of N-grams" generative language model for novel responses.

Experimental Evaluation

The framework was evaluated on three distinct datasets: a Depression Therapy dataset (~42,000 samples), Larry King TV interview transcripts (~8,200 samples), and a large-scale Twitter Dialogue Corpus (~749,000 samples).

Performance and Efficiency

The hashing-based models demonstrated significant improvements in both computational efficiency and response quality. While training a neural baseline on the Twitter dataset required over two weeks on a 1000-core GPU, the LSH model achieved training in just 5 to 6 hours on a standard 16-core CPU.

Quantitative comparison of the proposed LSH models against standard neural baselines (LSTM, HRED, VHRED) using embedding-based topic similarity metrics.

As shown in the results table, the hashing approaches consistently outperformed neural baselines like LSTM, HRED, and VHRED across standard metrics such as Average, Greedy, and Extrema topic similarity. Furthermore, human evaluators preferred the LSH-RkNN model, rating its responses as "appropriate" 28.7% of the time, compared to 3.7% for the LSTM baseline and 13.5% for VHRED.

Conclusion

This work demonstrates that nonparametric, hashcode-based models can provide a robust and efficient alternative to deep learning for dialogue systems, particularly in data-constrained domains. By leveraging Mutual Information to align speaker representations, the model effectively captures the collaborative dynamics of therapeutic dialogue. Future research will focus on enhancing response diversity and integrating clinical psychological metrics directly into the learning process.

Summary updated

Abstract

We propose a novel dialogue modeling framework, the firstever nonparametric kernel functions based approach for dialogue modeling, which learns hashcodes as text representations; unlike traditional deep learning models, it handles well relatively small datasets, while also scaling to large ones. We also derive a novel lower bound on mutual information, used as a model-selection criterion favoring representations with better alignment between the utterances of participants in a collaborative dialogue setting, as well as higher predictability of the generated responses. As demonstrated on three real-life datasets, including prominently psychotherapy sessions, the proposed approach significantly outperforms several state-of art neural network based dialogue systems, both in terms of computational efficiency, reducing training time from days or weeks to hours, and the response quality, achieving an order of magnitude improvement over competitors in frequency of being chosen as the best model by human evaluators.
BibTeX
@inproceedings{garg2020modeling,
  author = {Sahil Garg and Irina Rish and Guillermo A. Cecchi and Palash Goyal and Sarik Ghazarian and Shuyang Gao and Greg Ver Steeg and Aram Galstyan},
  title = {{Modeling Dialogues with Hashcode Representations: A Nonparametric Approach}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i04.5813},
}
Figure