BORT: Back and Denoising Reconstruction for End-to-End Task-Oriented Dialog

Haipeng Sun, Junwei Bao, Youzheng Wu, Xiaodong He

Abstract

A typical end-to-end task-oriented dialog system transfers context into dialog state, and upon which generates a response, which usually faces the problem of error propagation from both previously generated inaccurate dialog states and responses, especially in low-resource scenarios. To alleviate these issues, we propose BORT, a back and denoising reconstruction approach for end-to-end task-oriented dialog system. Squarely, to improve the accuracy of dialog states, back reconstruction is used to reconstruct the original input context from the generated dialog states since inaccurate dialog states cannot recover the corresponding input context. To enhance the denoising capability of the model to reduce the impact of error propagation, denoising reconstruction is used to reconstruct the corrupted dialog state and response. Extensive experiments conducted on MultiWOZ 2.0 and CamRest676 show the effectiveness of BORT. Furthermore, BORT demonstrates its advanced capabilities in the zero-shot domain and low-resource scenarios.

Introduction

In Task-Oriented Dialog (TOD) systems—the technology powering digital assistants for tasks like booking hotels or restaurants—accuracy is paramount. Modern systems have shifted toward "end-to-end" architectures that map user natural language directly to structured dialog states and then to natural language responses. However, these systems frequently suffer from a "house of cards" effect: a minor error in an early turn cascades through the conversation, leading to total system failure by the end of the session.

This error propagation is exacerbated by the discrepancy between training and inference. During training, models typically observe "oracle" (perfect) conversation histories. In real-world deployment, they must rely on their own previously generated, and potentially flawed, outputs. To address this, we introduce BORT (Back and Denoising Reconstruction), a framework designed to mitigate error propagation by forcing the model to verify its own logic through reconstruction tasks.

Methodology

BORT is built upon a T5-based encoder-decoder framework, augmented with two primary architectural "checks and balances" and a specialized preprocessing strategy.

Back Reconstruction (BR)

To ensure that generated dialog states are accurate and sufficient, BORT employs Back Reconstruction. The model is tasked with a reverse challenge: reconstructing the original user utterance using only the generated dialog state. If a state is missing critical information (e.g., forgetting a "cheap" price constraint), it becomes impossible to recover the original sentence. This objective forces the encoder to produce high-fidelity representations where the dialog state serves as an adequate summary of the user's intent.

Denoising Reconstruction (DR)

To improve robustness against its own mistakes, BORT utilizes Denoising Reconstruction. During training, the system intentionally "corrupts" the input by randomly deleting or masking tokens in the previous dialog states and responses. The model must then learn to reconstruct the original, clean version. This acts as a "noise-canceling" mechanism, teaching the model to identify and look past minor errors in the conversation history during real-world inference.

User Utterance Delexicalization (UD)

To bridge the gap between varied natural language and structured system outputs, we introduce a strategy to delexicalize user utterances. By replacing specific entities (e.g., "02:15") with placeholders (e.g., "[taxi_arriveby]"), the model more easily learns the underlying patterns of task-oriented communication.

Experimental Results

We evaluated BORT on two gold-standard benchmarks: MultiWOZ 2.0 (a large-scale multi-domain dataset) and CamRest676 (a smaller restaurant-specific dataset).

State-of-the-Art Performance

On the MultiWOZ 2.0 benchmark, BORT achieved a Combined Score of 108.3, significantly outperforming previous leading models such as SUMBT+LaRL (106.7) and UBAR (101.5). Notably, BORT reached a Joint Goal Accuracy of 54.0% for Dialog State Tracking, a substantial improvement over existing T5-based baselines.

Low-Resource and Zero-Shot Capabilities

BORT demonstrates exceptional performance in data-scarce environments:

  • Low-Resource: When trained on only 5% of the available data, BORT achieved a combined score of 68.9, whereas the standard DAMD baseline struggled at 47.7.
  • Zero-Shot Domain Adaptation: In tests where the model was trained on all domains except one (e.g., excluding the Taxi domain), BORT showed significantly higher "out-of-the-box" performance on the unseen domain compared to its peers, suggesting superior domain transferability.

Conclusion

BORT addresses the critical issue of error propagation in end-to-end TOD systems through the dual strategies of Back and Denoising Reconstruction. By forcing the model to verify the adequacy of its dialog states and training it to be resilient against history corruption, BORT achieves new state-of-the-art results. These findings highlight the importance of self-verification mechanisms in building reliable, multi-turn conversational AI, particularly in low-resource and zero-shot scenarios.

Summary updated

Abstract

A typical end-to-end task-oriented dialog system transfers context into dialog state, and upon which generates a response, which usually faces the problem of error propagation from both previously generated inaccurate dialog states and responses, especially in low-resource scenarios. To alleviate these issues, we propose BORT, a back and denoising reconstruction approach for end-to-end task-oriented dialog system. Squarely, to improve the accuracy of dialog states, back reconstruction is used to reconstruct the original input context from the generated dialog states since inaccurate dialog states cannot recover the corresponding input context. To enhance the denoising capability of the model to reduce the impact of error propagation, denoising reconstruction is used to reconstruct the corrupted dialog state and response. Extensive experiments conducted on MultiWOZ 2.0 and CamRest676 show the effectiveness of BORT. Furthermore, BORT demonstrates its advanced capabilities in the zero-shot domain and low-resource scenarios.
BibTeX
@inproceedings{sun2022bort,
  author = {Haipeng Sun and Junwei Bao and Youzheng Wu and Xiaodong He},
  title = {{BORT: Back and Denoising Reconstruction for End-to-End Task-Oriented Dialog}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=Xbx5C2fylI},
}
Figure