Abstract
Task-oriented dialog (TOD) systems rely heavily on external Knowledge Bases (KBs) to provide accurate information. However, a significant "memory gap" often exists during training: the KB state available at training time frequently differs from the KB state that existed when the original dialog was recorded. These Dialog-KB Inconsistencies lead models to learn spurious patterns or memorize incorrect responses. We present the Dialog-KB Arbitration Framework (DKAF), a novel approach that reconstructs contemporary KB snapshots from dialog transcripts. By arbitrating between the current KB and the dialog history, DKAF enables more robust training, leading to significant performance gains in dynamic environments.
Introduction
Task-oriented dialog agents, such as those used for restaurant reservations or flight bookings, must query external databases to fulfill user requests. Standard research paradigms assume that for every training dialog, a perfect snapshot of the KB—exactly as it appeared during the conversation—is available. In practice, KBs are highly dynamic; availability changes, and entities are added or removed. When developers only have access to the latest KB version, the resulting inconsistencies confuse machine learning models, forcing them to reconcile contradictory information between the dialog transcript and the database.
The Problem: Dialog-KB Inconsistencies
Dialog-KB Inconsistencies occur when a training transcript shows an agent recommending an entity that the current KB identifies as unavailable, or vice versa. These contradictions prevent models from generalizing effectively. Instead of learning the underlying logic of task fulfillment, models may resort to memorizing specific, outdated instances or failing to understand why certain entities were selected or rejected by the original agent.
The DKAF Framework
DKAF addresses these inconsistencies by "arbitrating" the KB to match the dialog context before the agent model begins learning. The framework employs a three-stage cascade to reconstruct the historical KB state:
- Row Insertion (RI): If a dialog mentions an entity missing from the current KB, the RI module predicts the missing relationships using distant supervision and re-inserts the entity into the local training snapshot.
- Row Deletion (RD): When the KB suggests an entity is available but the dialog indicates it was not, the RD module removes the contradictory row. This module is trained via Reinforcement Learning (RL) using a "Masked Entity Modeling" reward, which incentivizes the deletion of rows that make the original agent's response more probable.
- Row Completion (RC): Newly inserted rows often lack latent attributes (e.g., ratings or specific amenities). The RC module uses RL to deduce these missing values based on the agent's selection criteria within the dialog.
Following this arbitration, any standard end-to-end TOD model (e.g., SimpleTOD) can be trained on the corrected, consistent data.
Experimental Methodology
To evaluate DKAF, the authors developed two new benchmarks by injecting realistic inconsistencies into existing datasets:
- inc-bAbI: An inconsistent variant of the synthetic bAbI restaurant dataset.
- inc-BiTOD: An inconsistent version of the human-generated, multi-domain BiTOD dataset.
To ensure the inconsistencies mirrored real-world dynamics, the researchers utilized check-in data from the Yelp dataset to simulate realistic fluctuations in restaurant availability over time.
Key Results and Insights
DKAF demonstrates substantial improvements over standard training methods and rule-based baselines:
- Performance Gains: On the
inc-bAbIdataset, dialog accuracy increased from 90.6% to 99.2%. On the more complexinc-BiTOD, DKAF improved the KB Entity F1 score by 7.64 points. - The Dialog as Ground Truth: A central finding is that the dialog itself contains sufficient information to reconstruct the world state at the time of the conversation, obviating the need for exhaustive historical KB logs.
- Superiority over Rule-Based Systems: Simple rule-based arbitration (e.g., deleting all unmentioned entities) proved too aggressive, removing "negative" information necessary for the model to learn why certain options were not chosen.
- Robustness: Incremental analysis showed that while standard models' performance degrades sharply as inconsistency levels rise, DKAF remains stable, maintaining high accuracy even in highly volatile data environments.
Conclusion and Future Work
DKAF provides a scalable solution for training TOD systems on legacy or dynamic data where historical KB snapshots are unavailable. While currently focused on English-language dialogs and simulated fluctuations, future work will explore the framework's performance in multi-lingual settings and its application to "in-the-wild" datasets where ground truth snapshots are entirely absent. By bridging the memory gap, DKAF moves TOD agents closer to reliable deployment in the ever-changing real world.
Summary updated