Attend, Select and Eliminate: Accelerating Multi-turn Response Selection with Dual-attention-based Content Elimination

Jianxin Liang, Chang Liu, Chongyang Tao, Jiazhan Feng, Dongyan Zhao

Introduction

In the realm of retrieval-based dialogue systems, such as those used in industrial chatbots, the primary objective is to select the most appropriate response from a candidate pool based on a multi-turn conversation history. While Pre-trained Language Models (PLMs) like BERT have significantly advanced the state-of-the-art in accuracy, their computational intensity remains a major bottleneck. Processing long dialogue contexts through numerous transformer layers is both slow and resource-heavy.

The researchers behind ASE (Attend, Select and Eliminate) observed that much of a dialogue history is often redundant or irrelevant to a specific candidate response. However, standard models treat every token with equal importance throughout the entire computation. ASE addresses this by progressively dropping unimportant words as they pass through the transformer's layers, achieving significant speedups with minimal impact on performance.

The ASE Framework

The ASE framework is built upon three core pillars designed to bridge the gap between efficiency and accuracy:

  1. Dual-attention-based Content Elimination (DualA): Traditional acceleration methods often rely on self-importance (how much a word attends to its neighbors). DualA goes further by calculating importance based on both self-attention and the interaction between the context and the candidate response. If a word in the history does not interact significantly with the potential response, it is flagged for elimination.
  2. Retention Configuration Search: Determining the optimal number of tokens to keep at each layer is a complex optimization problem. The authors utilize an evolutionary search algorithm to identify the "Pareto Frontier"—the best schedule for token retention (e.g., maintaining more tokens in early layers to preserve global context while thinning them out in deeper layers).
  3. Once-for-all Self-Distillation: Eliminating tokens during inference creates a discrepancy since the model was originally trained on full sequences. To mitigate this "training-inference gap," a post-training strategy is employed where a "student" model (with elimination enabled) learns to mimic the output of a "teacher" model (the original full model).
The overall architecture of the ASE framework, illustrating the pipeline from fine-tuning to self-distillation.

Experimental Evaluation

The framework was rigorously tested on three standard multi-turn response selection benchmarks:

  • Ubuntu Corpus: Technical support logs (English).
  • Douban Corpus: Social media conversations (Chinese).
  • E-commerce Corpus: Customer service dialogues (Chinese).

Key Results and Insights

The experiments yielded several critical insights into the nature of dialogue processing:

  • Efficiency and Performance: ASE achieved speedups ranging from 1.4x to 2.3x compared to base BERT models. Remarkably, on the Ubuntu and E-commerce datasets, ASE-equipped models occasionally outperformed the original full models at specific speed settings. For instance, on the Douban dataset, the researchers observed a 2.0% improvement in P@1 while running significantly faster.
  • Dynamic Relevance: The "importance" of a word is not static; it depends on the response being evaluated. The Dual-attention mechanism successfully captures this dynamic relevance.
  • Recency Bias: The model naturally learned to prioritize the most recent turns in a dialogue, confirming that the end of a conversation typically holds the most relevant information for response selection.
  • Comparison with Baselines: ASE consistently outperformed other popular acceleration techniques, such as PoWER-BERT and L-Adaptive, demonstrating a superior trade-off between speed and accuracy across all tested benchmarks.
Performance vs. Speedup Ratio across Ubuntu, Douban, and E-commerce datasets, showing ASE's superiority over baselines.

Conclusion and Future Work

ASE demonstrates that full representations of every word are not necessary in the deeper layers of a transformer to make accurate matching decisions. By the time data reaches the final layers, the model only requires the "essence" of the conversation.

While the evolutionary search for retention configurations is currently computationally expensive, future work aims to combine token-level elimination with layer-wise reduction (skipping entire layers) to further push efficiency. Additionally, the researchers plan to explore how ASE handles extremely long-form dialogues where pruning becomes even more critical for real-time performance.

Summary updated

Abstract

Although the incorporation of pre-trained language models (PLMs) significantly pushes the research frontier of multi-turn response selection, it brings a new issue of heavy computation costs. To alleviate this problem and make the PLM-based response selection model both effective and efficient, we propose an inference framework together with a post-training strategy that builds upon any pre-trained transformer-based response selection models to accelerate inference by progressively selecting and eliminating unimportant content under the guidance of context-response dual-attention. Specifically, at each transformer layer, we first identify the importance of each word based on context-to-response and response-to-context attention, then select a number of unimportant words to be eliminated following a retention configuration derived from evolutionary search while passing the rest of the representations into deeper layers. To mitigate the training-inference gap posed by content elimination, we introduce a post-training strategy where we use knowledge distillation to force the model with progressively eliminated content to mimic the predictions of the original model with no content elimination. Experiments on three benchmarks indicate that our method can effectively speeds-up SOTA models without much performance degradation and shows a better trade-off between speed and performance than previous methods.
BibTeX
@inproceedings{liang2023attend,
  author = {Jianxin Liang and Chang Liu and Chongyang Tao and Jiazhan Feng and Dongyan Zhao},
  title = {{Attend, Select and Eliminate: Accelerating Multi-turn Response Selection with Dual-attention-based Content Elimination}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.422},
}
Figure