Improve Retrieval-based Dialogue System via Syntax-Informed Attention

Tengtao Song, Nuo Chen, Ji Jiang, Zhihong Zhu, Yuexian Zou

Abstract

Retrieval-based dialogue systems must accurately rank potential responses based on multi-turn conversation histories. While pre-trained language models like BERT excel at semantic representation, they often struggle with the underlying syntactic structure of complex dialogues. We present Syntax-Informed Attention (SIA), a plug-and-play module that incorporates both intra-sentence and inter-sentence syntactic dependencies into the Transformer architecture. By restricting attention to the "grammatical backbone" of utterances, SIA filters noise and enhances the model's understanding of "who-did-what-to-whom" across turns. Evaluations on Ubuntu, Douban, and E-Commerce datasets demonstrate that SIA consistently improves performance, achieving new state-of-the-art results when combined with specialized models like BERT-FP.

Introduction

In multi-turn retrieval-based dialogue systems, the primary challenge is capturing the logical flow across several utterances. Standard self-attention mechanisms in models like BERT are "dense," meaning every token can attend to every other token. While powerful, this can lead the model to focus on spurious correlations or noisy tokens (like punctuation) rather than the core grammatical structure.

Existing syntax-aware methods often focus on single-sentence processing. However, dialogue is inherently sequential; the subject of an inquiry in the first turn often relates to an action or object mentioned several turns later. To bridge this gap, the authors propose SIA to model syntactic relationships both locally (within a sentence) and globally (across the dialogue history).

Methods: Syntax-Informed Attention (SIA)

The SIA module functions by creating a specialized "mask" for the Transformer’s self-attention layers. This mask dictates which tokens are allowed to interact based on their positions in a dependency tree.

  1. Dependency Parsing: Every utterance in the dialogue is processed using an off-the-shelf parser (e.g., spaCy) to generate a dependency tree, identifying parent-child relationships between words.
  2. Intra-sentence Masking: Within a single utterance, a token is restricted to attending only to its syntactic ancestors. This forces the model to prioritize the core grammatical structure over irrelevant modifiers.
  3. Inter-sentence Masking: To capture the dialogue flow, "important" tokens—defined as those near the root of the dependency tree—are allowed to attend to each other across different sentences.
  4. Model Integration: SIA is implemented as two stacked multi-head self-attention layers. Following observations that BERT’s middle layers capture the most syntactic information, the SIA module is integrated into the 6th layer of a standard 12-layer BERT encoder.
The SIA architecture integrates dependency parsing into the Transformer's attention mechanism to create a syntax-informed mask.

Experimental Setup

The proposed method was evaluated on three diverse multi-turn response selection benchmarks:

  • Ubuntu Corpus V1: Technical troubleshooting dialogues in English.
  • Douban Conversation Corpus: Open-domain Chinese dialogues.
  • E-Commerce Corpus: Real-world customer service interactions from Taobao.

Key Results

The SIA-enhanced models demonstrated significant improvements over vanilla BERT and other competitive baselines. On the E-Commerce dataset, adding SIA to BERT resulted in a 4.5% absolute improvement in the $R_{10}@1$ metric (the probability that the correct response is the top-ranked choice). Furthermore, when integrated with BERT-FP—a model already optimized for dialogue—SIA pushed performance to new state-of-the-art levels on the Ubuntu and E-Commerce benchmarks.

Comprehensive results across three corpora show that SIA consistently improves retrieval performance over standard BERT and specialized dialogue models.

Discussion and Insights

Ablation studies highlighted that the inter-sentence component of SIA provides the most significant performance boost, confirming the importance of modeling the "backbone" connections across the entire dialogue history. By treating syntax as a noise filter, SIA prevents the model from attending to irrelevant tokens, such as punctuation or distant modifiers that do not contribute to the core semantic meaning.

Limitations and Future Work

Despite its success, SIA has limitations. It currently utilizes only the structure of the dependency tree without distinguishing between specific relationship types (e.g., subject vs. object). Additionally, the system's efficacy is tied to the accuracy of the underlying dependency parser, which may struggle with the informal or slang-heavy language common in dialogues. Future work aims to incorporate more fine-grained syntactic labels to further refine the attention mechanism.

Conclusion

Syntax-Informed Attention offers a practical, modular way to enhance the structural understanding of retrieval-based dialogue systems. By focusing on the grammatical core of conversations, it provides a robust mechanism for filtering noise and capturing long-range dependencies in multi-turn interactions.

Summary updated

BibTeX
@inproceedings{song2023improve,
  author = {Tengtao Song and Nuo Chen and Ji Jiang and Zhihong Zhu and Yuexian Zou},
  title = {{Improve Retrieval-based Dialogue System via Syntax-Informed Attention}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=mEPtDVAHdB},
}
Figure