Abstract
Qingfu Zhu, Wei-Nan Zhang, Ting Liu, William Yang Wang. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
2020.
Introduction
Open-domain dialogue generation remains a significant challenge in AI due to the inherent data insufficiency of existing datasets. For any given prompt, such as "What are you up to this Friday?", there are countless valid responses. However, standard training sets typically provide only a single ground-truth example. This limitation often leads models to generate generic, "safe" responses like "I don't know" or "That's cool," failing to capture the diversity of human conversation.
To address this, researchers have introduced COPT (Counterfactual Off-Policy Training). This framework allows models to learn from "what might have been" by imagining alternative responses within the same environmental context, thereby generating higher-quality synthetic data for training.
Methodology: Counterfactual Reasoning with SCMs
The core of COPT lies in treating dialogue generation as a Structural Causal Model (SCM). In this framework, a response $Y$ is determined by two primary components:
- The Scenario ($U$): Latent factors representing the unobserved context, such as user profiles or mood.
- The Causal Mechanism ($f$): The deterministic function (the model) that maps the prompt $X$ and scenario $U$ to a response.
The COPT training process involves three counterfactual steps:
- Abduction: The model infers the latent scenario ($u$) based on an observed response from the dataset.
- Action: An intervention is performed where the model's current behavior policy is replaced with a target policy.
- Prediction: The model generates a counterfactual response, answering the question: "Given the same scenario, what would the response be under the new policy?"
By utilizing the Gumbel-Max Trick, this process is made differentiable and efficient. This "off-policy" approach allows the model to explore high-reward regions of the response space that are often missed by standard sampling methods.
Experimental Setup
The framework was evaluated using the DailyDialog dataset, which consists of high-quality multi-turn daily conversations. The dataset was split into 11,118 training dialogues and 1,000 dialogues each for validation and testing. To provide sufficient context, dialogues were processed into instances containing up to three continuous utterances.
Key Results and Insights
COPT was integrated into existing adversarial models like REGS and StepGAN, demonstrating consistent performance gains:
- Relevance: StepGAN with COPT achieved a significant improvement in BLEU-1 scores, rising from 34.76 to 43.83.
- Diversity: There was a marked increase in response diversity (Distinct-1 and Distinct-2 metrics), reducing the prevalence of repetitive, generic phrases.
- Human Preference: In human evaluations, COPT-trained models were preferred over standard REGS models 32.2% of the time (with 43.8% ties) and outperformed the baseline HRED model 63.1% of the time.
A key insight from this work is that synthetic responses grounded in specific inferred scenarios (hindsight) provide a much cleaner learning signal than those generated from scratch. This allows the generator to find responses that a discriminator "judge" finds more human-like.
Limitations and Future Work
While effective, COPT introduces additional computational overhead, increasing training time per epoch (e.g., from 951s to 1,244s for StepGAN). Additionally, the quality of counterfactual reasoning is tied to the initial behavior policy; a poor starting model may lead to nonsensical inferred scenarios. Future research will explore scaling COPT to much larger datasets, such as Reddit or Twitter, to further test its robustness.
Summary updated