Universal Post-Processing Networks for Joint Optimization of Modules in Task-Oriented Dialogue Systems

Atsumoto Ohashi, Ryuichiro HigashinakaAAAI 2025 · DOI

TL;DR

This paper introduces UniPPN, a novel universal post-processing network that jointly optimizes all modules in task-oriented dialogue systems. By treating module output modification as a sequence-to-sequence task and employing a module-level reinforcement learning approach, UniPPN significantly improves dialogue system performance and applicability compared to existing methods.

What Problem Does This Paper Address?

Task-oriented dialogue systems typically comprise several interconnected modules, such as Natural Language Understanding (NLU), Dialogue State Tracking (DST), Dialogue Policy, and Natural Language Generation (NLG). While these modules are often optimized individually, improving the overall system's task completion capability requires optimizing their interactions. Previous approaches using Post-Processing Networks (PPNs) have shown promise by optimizing module outputs rather than the modules themselves. However, these methods faced limitations: they could only handle a subset of modules, and existing PPNs (BinPPN for structured outputs and GenPPN for natural language) had different architectures and training algorithms, preventing joint optimization. Furthermore, GenPPN's reliance on detailed dialogue act feedback limited its applicability to systems that don't explicitly output dialogue acts. This paper addresses the need for a unified, flexible, and jointly optimizable post-processing mechanism that can enhance the performance of diverse task-oriented dialogue systems, including those with non-trainable modules or end-to-end architectures.

Methods

The core contribution is the Universal Post-Processing Network (UniPPN). UniPPN is a language-model-based network designed to treat the post-processing of outputs from any module as a sequence-transformation task. This allows it to handle both structured (e.g., NLU, DST) and natural language (e.g., NLG) outputs uniformly.

UniPPN is optimized using a two-stage process:

  1. Imitation Learning (IL): To pre-train UniPPN, the authors generate "pseudo-post-processing demonstration data." This involves simulating dialogues, identifying module outputs, and creating pairs of original outputs and desired modified outputs. A key aspect here is the "learning to copy" mechanism, where the model learns to output a special "copy" token when no modification is needed, preventing unnecessary processing.
  2. Reinforcement Learning (RL): After IL, UniPPN is integrated into the dialogue system and fine-tuned using RL. The authors introduce a module-level Markov Decision Process (MDP). Instead of treating each turn as a single step, this MDP considers each module's post-processing step as a distinct time step. This fine-grained approach allows for more accurate value and advantage estimation for each module, leading to more stable joint learning, even with sparse end-of-dialogue rewards. The RL objective is to maximize the expected cumulative discounted reward, with gradients computed for each module's post-processing action.

Datasets

The primary dataset used for training and evaluation is MultiWOZ (Budzianowski et al. 2018), a large-scale, multi-domain dataset for task-oriented dialogue. The authors also mention using an agenda-based user simulator (Schatzmann et al. 2007) for generating dialogues and evaluation.

Key Insights and Ideas

  • Unified Post-Processing: The idea of treating all module outputs as sequences and using a single network (UniPPN) for post-processing is a significant simplification and generalization over previous PPNs.
  • Module-Level RL: The module-level MDP is a crucial innovation. It allows for more precise credit assignment in RL, which is essential for optimizing complex, modular systems where individual module contributions to the final task success are hard to isolate.
  • Flexibility and Applicability: UniPPN's ability to handle arbitrary sequence transformations makes it applicable to a much wider range of dialogue systems, including those with non-trainable modules (rule-based, API-based) and end-to-end systems that don't explicitly output dialogue acts.
  • "Learning to Copy": Incorporating a mechanism for the model to learn when not to modify an output is a practical insight that improves efficiency and prevents over-processing.

Key Results

  • Improved Task Completion: UniPPN consistently outperformed both the original dialogue systems and systems augmented with conventional PPNs (BinPPN&GenPPN) across various pipeline and end-to-end systems on the MultiWOZ dataset. For example, in the SYSRULE system, UniPPN improved success rate to 90.62 points from a baseline of around 84.
  • Broader Applicability: UniPPN was successfully applied to systems where conventional PPNs could not be used, such as SYSLAVA (which uses a word-level policy) and end-to-end systems (SYSPPTOD, SYSLLM), showing significant improvements.
  • Human Evaluation: Human evaluations confirmed UniPPN's effectiveness, showing it outperformed the original system and +BinPPN&GenPPN in task completion metrics (success rate, turns). While subjective metrics (LU, RA, OS) showed no significant difference, this was expected as UniPPN's reward is task-completion focused.

Limitations and Future Work

  • Training Cost: The authors acknowledge that reducing the overall training cost of UniPPN is a future direction. This includes minimizing the number of dialogue experiences needed for convergence and optimizing model size.
  • Multimodal Dialogue Systems: The current work focuses on text-based dialogue. Future work aims to extend UniPPN's capabilities to support post-processing of outputs from modules in multimodal dialogue systems.

Summary updated

BibTeX
@inproceedings{ohashi2025universal,
  author = {Atsumoto Ohashi and Ryuichiro Higashinaka},
  title = {{Universal Post-Processing Networks for Joint Optimization of Modules in Task-Oriented Dialogue Systems}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i23.34681},
}
Figure