Uni-Encoder: A Fast and Accurate Response Selection Paradigm for Generation-Based Dialogue Systems

Chiyu Song, Hongliang He, Haofei Yu, Huachuan Qiu, Pengfei Fang, Zhenzhong Lan

Abstract

Sample-and-rank is a key decoding strategy for modern generation-based dialogue systems. It helps achieve diverse and high-quality responses by selecting an answer from a small pool of generated candidates. The current state-of-the-art ranking methods mainly use an encoding paradigm called Cross-Encoder, which separately encodes each context-candidate pair and ranks the candidates according to their fitness scores. However, Cross-Encoder repeatedly encodes the same lengthy context for each candidate, resulting in high computational costs. Poly-Encoder addresses the above problems by reducing the interaction between context and candidates, but with a price of performance drop. In this work, we develop a new paradigm called Uni-Encoder, that keeps the full attention over each pair as in Cross-Encoder while only encoding the context once, as in Poly-Encoder. Uni-Encoder encodes all the candidates with the context in one forward pass. We use the same positional embedding for all candidates to ensure they are treated equally and design a new attention mechanism to avoid confusion. Our Uni-Encoder can simulate other ranking paradigms using different attention and response concatenation methods. Extensive experiments show that our proposed paradigm achieves new state-of-the-art results on four benchmark datasets with high computational efficiency. For instance, it improves R10@1 by 2.9% with an approximately 4X faster inference speed on the Ubuntu V2 dataset.

Introduction

Modern open-domain dialogue systems often employ a "sample-and-rank" strategy. A generative model produces a set of diverse candidate responses, and a separate ranking model selects the most appropriate one. The effectiveness of the dialogue system is thus heavily dependent on the ranker's performance.

Historically, researchers have navigated a trade-off between three paradigms:

  1. Bi-Encoders: These encode the context and candidates separately into vector representations. While computationally efficient, they lack the fine-grained interaction necessary for high accuracy.
  2. Cross-Encoders: These concatenate the context with a single candidate and process them together. This allows for full self-attention across the pair, yielding high accuracy, but it is computationally expensive because the context must be re-encoded for every candidate in the pool.
  3. Poly-Encoders: These attempt to bridge the gap by using multiple context codes to interact with candidate embeddings, offering a middle ground in terms of speed and performance.

The Uni-Encoder aims to provide the full-attention benefits of a Cross-Encoder while matching the computational efficiency of a Poly-Encoder.

The Uni-Encoder Paradigm

The Uni-Encoder architecture introduces three key innovations to the standard Transformer-based ranking setup.

One Pass, Multiple Candidates

Instead of processing context-candidate pairs individually, Uni-Encoder concatenates the context with the entire pool of candidates into a single sequence. This allows the model to encode the context exactly once for the entire candidate set.

Shared Positional Embeddings

To ensure that the model treats all candidates fairly, Uni-Encoder employs shared positional embeddings. In a standard sequence, each token has a unique position ID. In Uni-Encoder, every candidate response starts with the same position ID immediately following the context.

Uni-Encoder input representation with shared positional embeddings for multiple candidates.

As shown in Figure 1, the position embeddings for $r_1$ and $r_2$ are identical ($E_6, E_7, E_8$), preventing the model from developing a positional bias toward candidates that appear earlier in the concatenated string.

Arrow Attention

To prevent information "bleeding" between different candidates in the same sequence, the authors developed a specialized attention mask called Arrow Attention.

Comparison of attention masks for different ranking paradigms, highlighting Uni-Encoder's Arrow Attention.

In this mechanism (visualized in Figure 2c), the context can attend to itself, and each candidate can attend to both the context and itself. Crucially, candidates are forbidden from attending to other candidates. This "arrow-like" structure allows the model to perform full cross-attention for each context-candidate pair simultaneously without interference.

Experimental Evaluation

The Uni-Encoder was evaluated on four major dialogue benchmarks: the Ubuntu Dialogue Corpus (V1 & V2), PersonaChat, and the Douban Conversation Corpus. These datasets cover a range of scenarios, from technical support logs to persona-driven conversations.

Key Results

The Uni-Encoder achieved state-of-the-art (SOTA) results across all tested datasets:

  • Accuracy: On the Ubuntu V2 dataset, Uni-Encoder improved the R10@1 metric by 2.9% compared to the standard Cross-Encoder.
  • Efficiency: The model achieved an approximately 4x speedup in inference time compared to Cross-Encoders for typical candidate pool sizes.
  • Scalability: The efficiency gains become more pronounced as the candidate pool size increases, maintaining high performance even with 100 candidates.

Discussion and Conclusion

The Uni-Encoder demonstrates that context recomputation is the primary bottleneck in response ranking. By encoding the context once and branching attention to multiple candidates, the model eliminates significant latency. Furthermore, qualitative analysis suggests that Uni-Encoder is better at selecting specific, meaningful responses over generic "safe" answers.

While Uni-Encoder is highly effective for the candidate pools typical of generation-based systems (10–20 candidates), its sequence-length constraints mean it is not yet a replacement for retrieval systems handling thousands of candidates. However, its efficiency and accuracy make it an ideal candidate for tasks like Reinforcement Learning from Human Feedback (RLHF), where it can serve as a high-performance reward model.

In summary, Uni-Encoder successfully unifies the strengths of previous paradigms, offering a fast, accurate, and flexible framework for modern dialogue systems.

Summary updated

Abstract

Sample-and-rank is a key decoding strategy for modern generation-based dialogue systems. It helps achieve diverse and high-quality responses by selecting an answer from a small pool of generated candidates. The current state-of-the-art ranking methods mainly use an encoding paradigm called Cross-Encoder, which separately encodes each context-candidate pair and ranks the candidates according to their fitness scores. However, Cross-Encoder repeatedly encodes the same lengthy context for each candidate, resulting in high computational costs. Poly-Encoder addresses the above problems by reducing the interaction between context and candidates, but with a price of performance drop. In this work, we develop a new paradigm called Uni-Encoder, that keeps the full attention over each pair as in Cross-Encoder while only encoding the context once, as in Poly-Encoder. Uni-Encoder encodes all the candidates with the context in one forward pass. We use the same positional embedding for all candidates to ensure they are treated equally and design a new attention mechanism to avoid confusion. Our Uni-Encoder can simulate other ranking paradigms using different attention and response concatenation methods. Extensive experiments show that our proposed paradigm achieves new state-of-the-art results on four benchmark datasets with high computational efficiency. For instance, it improves R10@1 by 2.9% with an approximately 4X faster inference speed on the Ubuntu V2 dataset.
BibTeX
@inproceedings{song2023uni,
  author = {Chiyu Song and Hongliang He and Haofei Yu and Huachuan Qiu and Pengfei Fang and Zhenzhong Lan},
  title = {{Uni-Encoder: A Fast and Accurate Response Selection Paradigm for Generation-Based Dialogue Systems}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.388},
}
Figure