Injecting linguistic knowledge into BERT for Dialogue State Tracking

Xiaohan Feng, Xixin Wu, Helen Meng

Abstract

Dialogue State Tracking (DST) is essential for task-oriented chatbots to understand user goals. While Transformer models like BERT have improved performance, they lack transparency and require extensive labeled data. This paper presents a framework that injects unsupervised linguistic knowledge from a Convex Polytopic Model (CPM) into BERT's attention layers. Results on the MultiWoZ benchmark show improved accuracy, faster convergence, and better interpretability.

Introduction

Dialogue State Tracking (DST) serves as the core of task-oriented dialogue systems, responsible for maintaining the state of a conversation by identifying user goals (e.g., "find a cheap hotel"). Despite the success of Transformer-based architectures, two primary challenges persist: the "black box" nature of these models and their high dependency on large-scale labeled datasets. This research proposes a method to bridge this gap by leveraging unsupervised geometric modeling to extract interpretable semantic patterns and injecting them into BERT.

Methodology

The researchers introduce a two-stage pipeline combining unsupervised geometry with deep learning. The first stage involves knowledge extraction via the Convex Polytopic Model (CPM). Unlike standard neural embeddings, CPM interprets a text corpus as a point cloud in high-dimensional space and constructs a "polytope"—a geometric shape—that encloses these points. The vertices of this polytope represent "pivotal concepts" or semantic anchors, such as patterns related to "booking" or "locations."

The proposed framework for injecting CPM-extracted features into a neural model for DST.
Visualization of a 3-D MVS polytope enclosing a corpus point cloud.

In the second stage, these CPM-extracted features (composition coefficients and word-word correlation matrices) are fused into the self-attention layers of a BERT-based model, specifically the TripPy architecture. This "context-guided attention" mechanism directs the model's focus toward semantically significant tokens based on broader dataset patterns.

Evaluation and Results

The model was evaluated on the MultiWoZ dataset (versions 2.1 through 2.4). The CPM-enhanced model achieved a Joint Goal Accuracy (JGA) of 66.16% on MultiWoZ 2.4, a significant improvement over the 64.75% baseline. Furthermore, the model demonstrated increased efficiency, converging in an average of 16.33 iterations compared to the baseline's 20.33. Ablation studies confirmed that both CPM coefficients and guided attention are necessary for optimal performance.

Interpretability and Discussion

A key finding is the alignment between geometric vertices and human linguistic logic. Vertices often correlate with specific dialogue slots, such as "departure" or "price range." Using Integrated Gradients (IG), the authors visualized that the model prioritizes contextually crucial tokens more effectively than the baseline. This suggests that the geometric constraints of the CPM help the neural model learn more robust and logical representations.

Conclusion

This work demonstrates that injecting unsupervised linguistic knowledge into BERT enhances both performance and transparency in DST. Future research will investigate extending this framework to the decoder side for response generation and improving robustness in noisy real-world environments.

Summary updated

BibTeX
@inproceedings{feng2024injecting,
  author = {Xiaohan Feng and Xixin Wu and Helen Meng},
  title = {{Injecting linguistic knowledge into BERT for Dialogue State Tracking}},
  booktitle = {IEEE Access},
  year = {2024},
  url = {https://openreview.net/forum?id=6cDJ0bba7Q},
}
Figure