Abstract
Search-based dialogue systems often struggle with the trade-off between the high accuracy of cross-encoders and the computational efficiency of bi-encoders. We present Triple-Encoders, a framework that contextualizes dialogue history using a Hebbian-inspired "fire together, wire together" learning objective. By projecting utterances into dual latent subspaces and utilizing vector algebra for interaction, Triple-Encoders achieve state-of-the-art performance in next-utterance selection while maintaining linear $O(n)$ complexity. Our results across DailyDialog, MDC, and PersonaChat show significant improvements over existing bi-encoders and robust zero-shot generalization.
Introduction
Selecting the optimal response in multi-turn dialogues requires a deep understanding of context. Traditional models either re-encode the entire history at every turn (Cross-Encoders), which is computationally prohibitive ($O(n^2)$), or encode utterances independently (Bi-Encoders), which often misses nuanced contextual shifts. Triple-Encoders bridge this gap by enabling "weight-less contextualization." By allowing independently encoded utterances to interact through simple vector operations, the model captures the rich, contextualized understanding of a cross-encoder with the speed of a bi-encoder.
Methodology
Triple-Encoders build upon the Curved Contrastive Learning (CCL) framework with several architectural innovations:
Dual Latent Subspaces
Utterances are projected into two distinct latent subspaces, denoted by special tokens [B1] and [B2]. These subspaces represent the relative temporal order of utterances, allowing the model to distinguish between different turns in the dialogue history.
Hebbian-Inspired Learning Objective
The core training principle is "fire together, wire together." The model is optimized so that the mean-pooled average of co-occurring utterances ($u_i$ and $u_j$) in a context is geometrically close to the representation of the correct response ($u_k$). This encourages the model to learn how combinations of utterances predict future turns.
Contextualized Mixtures and Hard Negatives
At inference, the model computes "distributed utterance mixtures" via vector addition and averaging. This simulates interaction between history turns without the need for joint transformer encoding. To ensure the model learns specific interactions, training incorporates "triplet negatives"—combinations of valid context utterances with random "intruder" utterances.
Experimental Setup
The model was evaluated on four key benchmarks:
- DailyDialog: High-quality human-to-human conversations.
- MDC (Multi-Domain Context): Task-oriented dialogues focusing on goal-directed speech.
- PersonaChat: Used for zero-shot evaluation to test generalization across diverse personalities.
- Children Book Test (CBT): A non-dialogue sequence modeling task.
Key Results
Triple-Encoders demonstrate a significant performance leap over standard bi-encoders.
Performance Gains
The model outperformed standard CCL bi-encoders by 31.46% on DailyDialog and 46% on the task-oriented MDC dataset. These results indicate that the geometric approach to context is highly effective for both casual and goal-oriented dialogue.
Zero-Shot Robustness
In zero-shot evaluations on PersonaChat, Triple-Encoders surpassed the single-vector ConveRT model once the dialogue history exceeded five turns. This suggests that distributed representations are more resilient to long-range dependencies than single-vector bottlenecks.
Efficiency
Despite these gains, the model maintains $O(n)$ complexity. In a live setting, only the newest utterance needs to be encoded; contextualization is then performed through fast, parallelizable matrix operations on existing vectors.
Conclusion
Triple-Encoders represent a shift in how context is modeled in dialogue systems. By treating context as a geometric landscape with multiple local maxima rather than a single summary vector, the model achieves high accuracy without sacrificing efficiency. Future work will explore extending this framework to general NLP tasks such as long-form document retrieval and reasoning.
Summary updated