Abstract
This work shows how to improve and interpret the commonly used dual encoder model for response suggestion in dialogue. We present an attentive dual encoder model that includes an attention mechanism on top of the extracted word-level features from two encoders, one for context and one for label respectively. To improve the interpretability in the dual encoder models, we design a novel regularization loss to minimize the mutual information between unimportant words and desired labels, in addition to the original attention method, so that important words are emphasized while unimportant words are de-emphasized. This can help not only with model interpretability, but can also further improve model accuracy. We propose an approximation method that uses a neural network to calculate the mutual information. Furthermore, by adding a residual layer between raw word embeddings and the final encoded context feature, word-level interpretability is preserved at the final prediction of the model. We compare the proposed model with existing methods for the dialogue response task on two public datasets (Persona and Ubuntu). The experiments demonstrate the effectiveness of the proposed model in terms of better Recall@1 accuracy and visualized interpretability.
Introduction
Retrieval-based dialogue systems, such as those powering "Smart Reply" features, typically rely on dual-encoder architectures. These models compare a dialogue context against candidate responses to identify the most relevant match. Despite their efficiency and performance, dual-encoders often act as "black boxes," making it difficult to understand why a specific response was suggested. This research addresses two primary limitations: noisy attention, where models fixate on semantically "cheap" filler words, and information entanglement, where deep encoding layers obscure the contribution of individual words to the final decision.
Methodology
The authors propose the Attentive Dual-Encoder (ADE), which introduces three significant modifications to the standard architecture to enhance both accuracy and interpretability.
Attentive Dual-Encoder (ADE)
The model incorporates a pairwise word-level attention mechanism. Unlike standard encoders that produce a single vector for a whole sentence, the ADE evaluates how every word in the context relates to every word in the candidate response.
Non-attention Regularization (REG)
To refine the attention maps, the authors introduce a novel regularization term based on Mutual Information (MI). This term penalizes the model if words that are not attended to still carry significant predictive information about the correct response. By using a neural network to estimate and minimize this MI, the model is forced to concentrate its predictive power on the words it explicitly highlights.
Residual Word Embeddings (WE)
Deep encoders like Transformers often lose fine-grained word information as it becomes "entangled" in higher layers. The authors found that by layer ten of a standard Transformer, word-level features may retain less than 10% of their original information. To counteract this, they add a residual connection that feeds raw word embeddings directly into the final feature layer, ensuring the final decision remains grounded in specific word meanings.
Experimental Results
The model was evaluated on two benchmarks: the Ubuntu Dataset (1.4 million technical support dialogues) and the Persona Dataset (162k trait-driven dialogues).
As shown in the results, the full configuration (ADE + WE + REG) significantly outperformed the baseline dual-encoder (DE) across both datasets. On the challenging Ubuntu dataset, Recall@1 improved from 7.6% to 15.6%. On the Persona dataset, performance rose from 35.2% to 38.1%.
Qualitative analysis further confirmed the model's success. While baseline models often highlighted generic words like "I" or "am," the proposed model correctly identified specific keywords (e.g., "math," "science") that logically linked the context to the response.
Conclusion
This work demonstrates that interpretability and performance are not mutually exclusive. By treating interpretability as a training constraint through mutual information regularization and residual connections, the authors developed a model that is both more accurate and more transparent. While the approach introduces some training overhead due to the MI estimation network, the resulting "cleaner" attention maps provide valuable insights into the logic of dialogue retrieval systems. Future work may explore extending these techniques to multi-party conversations and generative architectures.
Summary updated