Introduction
Modern open-domain dialogue generation is currently dominated by deep neural networks. While powerful, these models are data-hungry, computationally expensive to train, and often inflexible when incorporating specialized objective functions. These challenges are particularly acute in domains like automated psychotherapy, where training data is scarce, sensitive, and characterized by highly imbalanced interaction patterns between patients and therapists.
To address these limitations, this paper introduces a novel nonparametric framework that utilizes binary hashcode representations for dialogue modeling. By compressing utterances into hashcodes and optimizing them via mutual information, the proposed approach offers a highly efficient alternative that excels in specialized, small-data environments.
The Proposed Framework
The authors propose a three-stage approach that decouples semantic inference from textual generation. This separation allows the model to first determine the "essence" of a response in a compressed, noise-free space before generating the final text.
1. Representation Learning via Hashing
Text utterances are mapped into compressed binary vectors (hashcodes) using kernelized Locality Sensitive Hashing (LSH). The authors explore three variants: LSH-RkNN (k-Nearest Neighbors), LSH-RMM (Max-Margin/SVM), and LSH-RLSTM.
2. Optimization and Inference
To capture the semantic alignment—or "working alliance"—between speakers, the model maximizes a novel lower bound on the Mutual Information (MI) between the hashcodes of consecutive utterances. Following this optimization, an ensemble of Random Forest classifiers is trained to predict the therapist's hashcode based on the patient's input.
3. Textual Response Generation
The predicted hashcode is mapped back to text. This is achieved either through retrieval from the training set or by using a "kernelized hashing of N-grams" generative language model for novel responses.
Experimental Evaluation
The framework was evaluated on three distinct datasets: a Depression Therapy dataset (~42,000 samples), Larry King TV interview transcripts (~8,200 samples), and a large-scale Twitter Dialogue Corpus (~749,000 samples).
Performance and Efficiency
The hashing-based models demonstrated significant improvements in both computational efficiency and response quality. While training a neural baseline on the Twitter dataset required over two weeks on a 1000-core GPU, the LSH model achieved training in just 5 to 6 hours on a standard 16-core CPU.
As shown in the results table, the hashing approaches consistently outperformed neural baselines like LSTM, HRED, and VHRED across standard metrics such as Average, Greedy, and Extrema topic similarity. Furthermore, human evaluators preferred the LSH-RkNN model, rating its responses as "appropriate" 28.7% of the time, compared to 3.7% for the LSTM baseline and 13.5% for VHRED.
Conclusion
This work demonstrates that nonparametric, hashcode-based models can provide a robust and efficient alternative to deep learning for dialogue systems, particularly in data-constrained domains. By leveraging Mutual Information to align speaker representations, the model effectively captures the collaborative dynamics of therapeutic dialogue. Future research will focus on enhancing response diversity and integrating clinical psychological metrics directly into the learning process.
Summary updated