Global-Selector: A New Benchmark Dataset and Model Architecture for Multi-turn Response Selection

Chiyu Song, Hongliang He, Huachuan Qiu, Haofei Yu, Zhenzhong Lan

Abstract

Response selection is a critical component of generation-based dialogue systems, aiming to identify the optimal response from a candidate pool. Current state-of-the-art methods typically rely on the Cross-Encoder paradigm, which separately encodes each context-response pair. However, this approach is computationally expensive due to redundant context encoding and fails to account for relationships between candidates. We propose the Uni-Encoder (also known as the Panoramic-Encoder), a new paradigm that encodes all candidates and the context simultaneously. By utilizing a tailored Arrow Attention mechanism and shared positional embeddings, our method achieves state-of-the-art performance with significantly lower computational complexity.

Introduction

Modern open-domain chatbots often utilize a "sample-and-rank" strategy: a generative model creates a pool of candidate responses, and a ranking model selects the best one. While Cross-Encoders provide high accuracy by allowing full attention between context and response, they are prohibitively slow for real-time applications because they re-process the long conversation context for every single candidate. Faster alternatives like Bi-Encoders and Poly-Encoders exist but often sacrifice significant accuracy. The Uni-Encoder addresses this trade-off by achieving full-attention accuracy without redundant computation.

Methods

The Uni-Encoder architecture processes the context and all candidate responses together in a single forward pass through a Transformer. This unified approach is made possible by two key innovations:

1. Shared Positional Embeddings

In a standard Transformer, concatenating multiple candidates would assign them sequential position IDs, incorrectly implying a temporal order. To fix this, Uni-Encoder assigns the exact same positional embeddings to all candidates, treating them as parallel, valid continuations of the dialogue.

Uni-Encoder input representation showing the integration of token, segment, and shared positional embeddings for multiple candidates.

2. Arrow Attention

To prevent candidates from attending to one another—which would cause representation "bleeding" and degrade ranking performance—the authors introduced Arrow Attention. This custom mask allows the context and candidates to interact fully while strictly isolating candidates from each other. This framework is flexible enough to simulate other paradigms (Bi-, Poly-, and Cross-Encoders) by simply adjusting the attention mask.

Comparison of attention masks: (a) Diagonal for Bi-Encoders, (b) Light-Arrow for Poly-Encoders, (c) Arrow for Uni-Encoders, and (d) Square for Cross-Encoders.

Datasets and Evaluation

The model was evaluated on four standard benchmarks:

  • PersonaChat: English dialogues conditioned on specific personas.
  • Ubuntu Dialogue Corpus (V1 & V2): Large-scale technical support conversations.
  • Douban Conversation Corpus: An open-domain Chinese social network dataset with multiple valid responses per context.

Key Results

  • State-of-the-Art Accuracy: Uni-Encoder outperformed previous benchmarks across all datasets. On Ubuntu V2, it improved the R10@1 metric by an absolute 2.9% over the previous Cross-Encoder SOTA.
  • Efficiency Gains: By avoiding context re-computation, Uni-Encoder demonstrated an approximately 4x faster inference speed compared to traditional Cross-Encoders.
  • Improved Learning: The architecture bypasses memory bottlenecks, allowing for larger batch sizes and more effective use of in-batch negative samples during training.

Conclusion

Uni-Encoder represents a significant step forward in response selection, providing the accuracy of full-interaction models with the efficiency required for production systems. While best suited for generation-based systems with moderate candidate pools (10–100 responses), its architecture offers a promising template for more efficient reward models in Large Language Model (LLM) alignment pipelines.

Summary updated

Abstract

Response selector is an essential component of generation-based dialogue systems and it aims to pick out an optimal response in a candidate pool to continue the dialogue. The current state-of-the-art methods are mainly based on the encoding paradigm called Cross-Encoder, which separately encodes each context-response pair and ranks the responses according to their fitness scores. However, Cross-Encoder repeatedly encodes the same lengthy context for each response, resulting in high computational costs. Moreover, without considering the relationship among the candidates, it is difficult to figure out which candidate is the best response purely based on the fitness score per candidate. We aim to address these problems through a new paradigm called Panoramic-Encoder. The proposed method encodes all candidates and the context at once and realizes the mutual interaction using a tailored candidate attention mechanism (CAM). It also enables the integration of some effective training techniques, such as the in-batch negative training, which cannot be used in Cross-Encoders. Extensive experiments across four benchmark datasets show that our new method significantly outperforms the current state-of-the-art with lower computational complexity.
BibTeX
@inproceedings{song2023global,
  author = {Chiyu Song and Hongliang He and Huachuan Qiu and Haofei Yu and Zhenzhong Lan},
  title = {{Global-Selector: A New Benchmark Dataset and Model Architecture for Multi-turn Response Selection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.388},
}
Figure