Triple-Encoders: Representations That Fire Together, Wire Together

Justus-Jonas Erker, Florian Mai, Nils Reimers, Gerasimos Spanakis, Iryna Gurevych

Abstract

Search-based dialogue systems often struggle with the trade-off between the high accuracy of cross-encoders and the computational efficiency of bi-encoders. We present Triple-Encoders, a framework that contextualizes dialogue history using a Hebbian-inspired "fire together, wire together" learning objective. By projecting utterances into dual latent subspaces and utilizing vector algebra for interaction, Triple-Encoders achieve state-of-the-art performance in next-utterance selection while maintaining linear $O(n)$ complexity. Our results across DailyDialog, MDC, and PersonaChat show significant improvements over existing bi-encoders and robust zero-shot generalization.

Introduction

Selecting the optimal response in multi-turn dialogues requires a deep understanding of context. Traditional models either re-encode the entire history at every turn (Cross-Encoders), which is computationally prohibitive ($O(n^2)$), or encode utterances independently (Bi-Encoders), which often misses nuanced contextual shifts. Triple-Encoders bridge this gap by enabling "weight-less contextualization." By allowing independently encoded utterances to interact through simple vector operations, the model captures the rich, contextualized understanding of a cross-encoder with the speed of a bi-encoder.

Comparison of ConveRT, CCL, and Triple Encoders in terms of encoding and search space characteristics.

Methodology

Triple-Encoders build upon the Curved Contrastive Learning (CCL) framework with several architectural innovations:

Dual Latent Subspaces

Utterances are projected into two distinct latent subspaces, denoted by special tokens [B1] and [B2]. These subspaces represent the relative temporal order of utterances, allowing the model to distinguish between different turns in the dialogue history.

Hebbian-Inspired Learning Objective

The core training principle is "fire together, wire together." The model is optimized so that the mean-pooled average of co-occurring utterances ($u_i$ and $u_j$) in a context is geometrically close to the representation of the correct response ($u_k$). This encourages the model to learn how combinations of utterances predict future turns.

Contextualized Mixtures and Hard Negatives

At inference, the model computes "distributed utterance mixtures" via vector addition and averaging. This simulates interaction between history turns without the need for joint transformer encoding. To ensure the model learns specific interactions, training incorporates "triplet negatives"—combinations of valid context utterances with random "intruder" utterances.

Experimental Setup

The model was evaluated on four key benchmarks:

  • DailyDialog: High-quality human-to-human conversations.
  • MDC (Multi-Domain Context): Task-oriented dialogues focusing on goal-directed speech.
  • PersonaChat: Used for zero-shot evaluation to test generalization across diverse personalities.
  • Children Book Test (CBT): A non-dialogue sequence modeling task.

Key Results

Triple-Encoders demonstrate a significant performance leap over standard bi-encoders.

Performance comparison (average rank of correct response) across Daily Dialog, MDC, and PersonaChat datasets as a function of history length.

Performance Gains

The model outperformed standard CCL bi-encoders by 31.46% on DailyDialog and 46% on the task-oriented MDC dataset. These results indicate that the geometric approach to context is highly effective for both casual and goal-oriented dialogue.

Zero-Shot Robustness

In zero-shot evaluations on PersonaChat, Triple-Encoders surpassed the single-vector ConveRT model once the dialogue history exceeded five turns. This suggests that distributed representations are more resilient to long-range dependencies than single-vector bottlenecks.

Efficiency

Despite these gains, the model maintains $O(n)$ complexity. In a live setting, only the newest utterance needs to be encoded; contextualization is then performed through fast, parallelizable matrix operations on existing vectors.

Conclusion

Triple-Encoders represent a shift in how context is modeled in dialogue systems. By treating context as a geometric landscape with multiple local maxima rather than a single summary vector, the model achieves high accuracy without sacrificing efficiency. Future work will explore extending this framework to general NLP tasks such as long-form document retrieval and reasoning.

Summary updated

Abstract

Search-based dialog models typically re-encode the dialog history at every turn, incurring high cost. Curved Contrastive Learning, a representation learning method that encodes relative distances between utterances into the embedding space via a bi-encoder, has recently shown promising results for dialog modeling at far superior efficiency. While high efficiency is achieved through independently encoding utterances, this ignores the importance of contextualization. To overcome this issue, this study introduces triple-encoders, which efficiently compute distributed utterance mixtures from these independently encoded utterances through a novel hebbian inspired co-occurrence learning objective in a self-organizing manner, without using any weights, i.e., merely through local interactions. Empirically, we find that triple-encoders lead to a substantial improvement over bi-encoders, and even to better zero-shot generalization than single-vector representation models without requiring re-encoding. Our code (https://github.com/UKPLab/acl2024-triple-encoders) and model (https://huggingface.co/UKPLab/triple-encoders-dailydialog) are publicly available.
BibTeX
@inproceedings{erker2024triple,
  author = {Justus-Jonas Erker and Florian Mai and Nils Reimers and Gerasimos Spanakis and Iryna Gurevych},
  title = {{Triple-Encoders: Representations That Fire Together, Wire Together}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.290},
}
Figure