Improving Retrieval-Based Dialogue System Via Syntax-Informed Attention

Tengtao Song, Nuo Chen, Ji Jiang, Zhihong Zhu, Yuexian Zou

TL;DR

Researchers have developed Syntax-Informed Attention (SIA), a plug-and-play module that helps AI models better understand the structure of a conversation. By forcing the model to focus on the "grammatical backbone" of sentences—both within a single turn and across multiple turns—SIA significantly improves the accuracy of retrieval-based dialogue systems, particularly in complex multi-turn scenarios.

What Problem Does This Paper Address?

In retrieval-based dialogue systems (like those powering customer service bots), the goal is to pick the best response from a pre-defined pool based on the preceding conversation. While large language models like BERT are excellent at understanding general word meanings, they often struggle with the "who-did-what-to-whom" logic of a sentence, especially when a conversation spans several turns.

Existing methods that try to incorporate syntax (the rules of sentence structure) usually focus on a single sentence at a time. However, a dialogue is a sequence of interconnected utterances. If a model only looks at the internal structure of one sentence, it misses how the "subject" of the first turn might relate to the "action" in the third turn. The authors aim to bridge this gap by modeling both intra-sentence (local) and inter-sentence (global) syntactic dependencies.

Methods

The authors propose a new mechanism called Syntax-Informed Attention (SIA). It works by creating a "mask" for the Transformer's self-attention mechanism, essentially telling the model which words it is allowed to "look at" based on their grammatical relationship.

  1. Dependency Parsing: First, they use an off-the-shelf parser (spaCy) to turn every sentence in the dialogue into a dependency tree, identifying which words are "parents" or "ancestors" of others.
  2. Intra-sentence Mask (Local): Within a single sentence, a word is only allowed to attend to its syntactic ancestors. This prevents the model from getting distracted by irrelevant modifiers and forces it to focus on the core grammatical structure.
  3. Inter-sentence Mask (Global): To handle the flow of conversation, they identify "important" tokens—those near the top of the dependency tree (the "backbone" words). These important tokens are allowed to attend to each other across different sentences in the dialogue.
  4. Integration: SIA is implemented as two stacked multi-head self-attention layers. Interestingly, they don't add this to the very end of the model. Based on prior research suggesting that BERT’s middle layers contain the most syntactic information, they "plug" SIA into the 6th layer of a 12-layer BERT model.

Datasets

The model was evaluated on three major benchmarks for multi-turn response selection:

  • Ubuntu Corpus V1: A large English dataset focused on technical troubleshooting.
  • Douban Conversation Corpus: A Chinese open-domain dialogue dataset.
  • E-Commerce Corpus: A Chinese dataset collected from Taobao, involving real-world customer-service interactions.

Key Insights and Ideas

  • Syntax as a Noise Filter: The "aha moment" here is that more attention isn't always better. By restricting what the model can see (via the SIA mask), the researchers actually made the model smarter. It stops correlating unrelated words (like a question mark and a random adjective) and starts focusing on the relationship between the "snack" mentioned in turn one and the "walnuts" mentioned in turn two.
  • The Power of the Global View: The ablation study revealed that the inter-sentence component was the heavy lifter. While local syntax helps, the ability for "backbone" words to talk to each other across the entire dialogue history provided the biggest performance boost.
  • Plug-and-Play Efficiency: Because SIA is a modular addition to existing Transformer architectures, it can be added to state-of-the-art models (like BERT-FP) with very few additional parameters, making it a practical upgrade for existing systems.

Key Results

The SIA-enhanced models consistently outperformed standard baselines across all three datasets:

  • Significant Gains: On the E-Commerce dataset, SIA improved BERT’s $R_{10}@1$ (the probability that the correct response is the top-ranked choice out of 10) by an impressive 4.5%.
  • State-of-the-Art (SOTA) Improvement: When added to BERT-FP (a specialized version of BERT for dialogue), SIA pushed the performance even further, achieving new SOTA results on the Ubuntu and E-Commerce benchmarks.
  • Interpretability: Visualization of the attention maps showed that SIA successfully ignored "noisy" tokens (like punctuation) and focused heavily on the semantic core of the dialogue.

Limitations and Future Work

  • Coarse Syntax: Currently, the model only looks at the structure of the dependency tree (who is connected to whom). It does not yet distinguish between the types of connections (e.g., the difference between a "subject" relationship and an "adjective" relationship).
  • Dependency on Parsers: The system's performance is capped by the accuracy of the external dependency parser. If the parser makes a mistake on a complex or slang-heavy sentence, the SIA mask will be flawed.
  • Future Direction: The authors plan to incorporate more fine-grained syntactic labels into the attention mechanism to further refine how the model weighs different grammatical roles.

Summary updated

Abstract

Multi-turn response selection is a challenging task due to its high demands on efficient extraction of the matching features from abundant information provided by context utterances. Since incorporating syntactic information like dependency structures into neural models can promote a better understanding of the sentences, such a method has been widely used in NLP tasks. Though syntactic information helps models achieved pleasing results, its application in retrieval-based dialogue systems has not been fully explored. Meanwhile, previous works focus on intra-sentence syntax alone, which is far from satisfactory for the task of multi-turn response where dialogues usually contain multiple sentences. To this end, we propose SIA, Syntax-Informed Attention, considering both intra- and inter-sentence syntax information. While the former restricts attention scope to only between tokens and corresponding dependents in the syntax tree, the latter allows attention in cross-utterance pairs for those syntactically important tokens. We evaluate our method on three widely used benchmarks and experimental results demonstrate the general superiority of our method on dialogue response selection.
BibTeX
@inproceedings{song2023improving,
  author = {Tengtao Song and Nuo Chen and Ji Jiang and Zhihong Zhu and Yuexian Zou},
  title = {{Improving Retrieval-Based Dialogue System Via Syntax-Informed Attention}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=mEPtDVAHdB},
}
Figure