Abstract: Search-based dialog models typically re-encode the dialog history at every turn, incurring high cost. Curved Contrastive Learning, a representation learning method that encodes relative distances between utterances into the embedding space via a bi-encoder, has recently shown promising results for dialog modeling at far superior efficiency. While high efficiency is achieved through independently encoding utterances, this ignores the importance of contextualization. To overcome this issue, this study introduces triple-encoders, which efficiently compute distributed utterance mixtures from these independently encoded utterances through a novel hebbian inspired co-occurrence learning objective in a self-organizing manner, without using any weights, i.e., merely through local interactions. Empirically, we find that triple-encoders lead to a substantial improvement over bi-encoders, and even to better zero-shot generalization than single-vector representation models without requiring re-encoding.
Introduction
In search-based dialog systems, the objective is to select the most appropriate response from a vast pool of candidates. Traditionally, researchers have relied on two primary architectural paradigms, both of which present significant trade-offs:
- Single-vector models (e.g., ConveRT): These models re-encode the entire dialog history at every turn. While contextually rich, this approach suffers from $O(n^2)$ computational complexity and attempts to compress complex histories into a single vector, often losing critical nuance.
- Bi-encoders (e.g., CCL): These encode each utterance independently, achieving $O(n)$ complexity. However, they lack contextualization, treating past remarks as isolated events and ignoring the semantic interactions between turns.
Triple-Encoders aim to bridge this gap by providing a "weight-less" way to contextualize dialog history, inspired by the Hebbian principle: "representations that fire together, wire together."
Methodology
Triple-Encoders build upon Curved Contrastive Learning (CCL), which uses cosine similarity to represent temporal distance between turns. The "Triple" architecture introduces a second "before" space, utilizing three specialized tokens: [B1], [B2], and [A].
Pairwise Mixtures and Hebbian Learning
Instead of comparing a single past utterance to a candidate, the model processes pairs of past utterances ($u_i$ and $u_j$). These are encoded with [B1] and [B2] respectively and then mean-pooled. The model is trained using a novel Hebbian Co-occurrence objective: these averaged "mixtures" must align with the correct future response (encoded in the [A] space). This forces the model to "wire" representations together if they frequently co-occur in sequences.
Weight-less Composition
Crucially, this contextualization occurs through simple vector algebra (averaging) rather than additional neural layers or weights. At inference time, the system only needs to encode the latest utterance; all previous vectors are reused, maintaining the linear efficiency of bi-encoders while simulating the performance of more complex models.
Experimental Setup
The model was evaluated across several diverse benchmarks:
- DailyDialog: Human-to-human multi-turn conversations.
- MDC (Microsoft Dialogue Challenge): Task-oriented dialogs testing goal-directed speech.
- PersonaChat: Used for zero-shot evaluation to test generalization to unseen topics.
- Children Book Test (CBT): A non-dialog dataset used to demonstrate the general applicability of the "fire together, wire together" logic to text sequences.
Key Results and Insights
Performance Gains
- Sequence Modeling: On DailyDialog, Triple-Encoders outperformed standard CCL bi-encoders by 31.46% in average rank.
- Task-Oriented Dialog: The improvement was even more pronounced on the MDC dataset, with a 46% performance boost over curved bi-encoders.
- Zero-Shot Generalization: On PersonaChat, Triple-Encoders improved as context length grew, eventually outperforming the popular ConveRT model, which typically plateaus.
Emergent Latent Properties
- The "Wiring" Effect: Training with triplets actually improves the underlying bi-encoder. Even without using mixture logic at inference, the model performs better because co-occurrence training forces the embedding space to become more linearly additive.
- Local Maxima: Unlike traditional models seeking one "global maximum," Triple-Encoders create a latent space with multiple "local maxima." This allows candidate responses to interact with different distributed mixtures of history, leading to richer representations.
Limitations and Future Work
While Triple-Encoders show significant promise, the current model was trained with a fixed window size ($w=5$), which may limit very long-range dependency handling. Future research will explore Triple-Encoders for broader sequential tasks like story generation and document modeling. The authors view this as a foundational step toward self-organizing representations in NLP, moving away from "black box" weights toward geometrically structured latent spaces.
Summary updated