TL;DR
This paper explores how to jointly optimize Natural Language Generation (NLG) components, specifically focusing on surface realization. They compare Hidden Markov Models (HMMs) and Bayesian Networks (BNs) within a Hierarchical Reinforcement Learning (HRL) framework, finding that while BNs excel in isolation, HMMs offer a scalable and cost-effective alternative in a joint system.
What Problem Does This Paper Address?
Traditional Natural Language Generation (NLG) systems often break down the generation process into distinct stages: content selection (what to say), utterance planning (how to structure it), and surface realization (converting it to words). However, research suggests that decisions made in these stages are interdependent. For instance, the level of detail in an instruction (surface realization) might depend on how familiar the user is with the environment (content selection/utterance planning). This paper addresses the challenge of optimizing these interconnected decisions. Specifically, it investigates how different graphical models, HMMs and BNs, perform when used for surface realization within a joint NLG framework, where all stages learn and adapt together, as opposed to being optimized in isolation. The goal is to improve user satisfaction and the naturalness of generated language.
Methods
The authors employ a Hierarchical Reinforcement Learning (HRL) framework to train their NLG agent. This framework allows for the optimization of multiple decision-making subtasks (content selection, utterance planning, and surface realization) in an integrated manner. For the surface realization component, they compare two graphical models:
- Hidden Markov Models (HMMs): These models represent surface realization as a sequence of states (phrases/sentences) that generate observable semantic symbols. They are trained using the Baum-Welch algorithm on a corpus of human instructions.
- Bayesian Networks (BNs): These models represent the dynamics between semantic concepts and their realisations as a directed acyclic graph. Each node is a random variable with dependencies on its parents, allowing for a more structured representation of context. The parameters are estimated using Maximum Likelihood Estimation (MLE).
The HRL agent is trained using a reward function that combines user satisfaction, task success, and a measure of repetition/variation in surface forms.
Datasets
The paper uses the GIVE corpus for training and evaluating the HMMs and BNs. This corpus contains navigation instructions and user reactions to those instructions, allowing for the simulation of user interaction and the calculation of rewards.
Key Insights and Ideas
- Joint learning is crucial: Treating NLG decisions jointly, rather than in isolation, significantly improves performance, especially for models that struggle with context-awareness.
- BNs excel in isolation: Bayesian Networks, due to their structured representation of dependencies, perform best when surface realization is optimized independently.
- HMMs are scalable for joint systems: While HMMs perform worse in isolation, they become a competitive and more scalable alternative when integrated into a joint HRL framework. This is attributed to the HRL agent learning to compensate for the HMMs' inherent lack of context-awareness.
- Context-awareness matters: The conditional independence assumptions in HMMs (Markov, stationary, observation independence) limit their ability to capture context, whereas BNs' structured dependencies allow them to better model the situation.
- Trade-offs in graphical models: BNs offer better performance but are more data-intensive and less scalable. HMMs are simpler and more scalable but less performant in isolation.
Key Results
- User Satisfaction: In the joint learning framework, both BNs and HMMs achieve comparable user satisfaction, outperforming a greedy baseline by 6%. BNs achieve an average reward of -11.53, while HMMs achieve -11.64, a marginal difference.
- Isolation Performance: When trained in isolation, BNs achieve a good performance, but HMMs' performance deteriorates significantly to an average reward of -12.12, nearly as low as the greedy baseline.
- Naturalness: The paper measures naturalness using Kullback-Leibler (KL) divergence. BNs achieve a KL score of 2.83 and HMMs a score of 2.80 when compared to human data in the joint model. The difference in naturalness between HMMs and BNs in the joint model is not statistically significant.
Limitations and Future Work
- Limitations of HMMs: HMMs' conditional independence assumptions limit their context-awareness and accuracy, leading to poorer performance in isolation.
- Limitations of BNs: BNs are more data-intensive and less scalable, which can be problematic for large-scale, real-world applications.
- Future Work: The authors suggest testing the hierarchical approach on larger NLG tasks like text or paragraph generation. They also propose comparing other graphical models, such as dynamic BNs, and conducting comprehensive human evaluations.
Summary updated