Improving Multi-party Dialogue Generation via Topic and Rhetorical Coherence

Yaxin Fan, Peifeng Li, Qiaoming ZhuEMNLP 2024 · OpenReview

Abstract

Previous studies on multi-party dialogue generation predominantly concentrated on modeling the reply-to structure of dialogue histories, always overlooking the coherence between generated responses and target utterances. To address this issue, we propose a Reinforcement Learning approach emphasizing both Topic and Rhetorical Coherence (RL-TRC). In particular, the topic- and rhetorical-coherence tasks are designed to enhance the model’s perception of coherence with the target utterance. Subsequently, an agent is employed to learn a coherence policy, which guides the generation of responses that are topically and rhetorically aligned with the target utterance. Furthermore, three discourse-aware rewards are developed to assess the coherence between the generated response and the target utterance, with the objective of optimizing the policy. The experimental results and in-depth analyses on two popular datasets demonstrate that our RL-TRC significantly outperforms the state-of-the-art baselines, particularly in generating responses that are more coherent with the target utterances.

Introduction

Generating responses in multi-party dialogue (MPD) environments—such as IRC channels or Slack—presents unique challenges compared to dyadic conversations. In these settings, multiple "entangled" conversation threads often occur simultaneously. While existing models have made strides by focusing on the reply-to structure (identifying which previous message a new one addresses), structure alone is insufficient for generating high-quality responses.

A model might correctly identify the target message but produce a response that is off-topic or logically inconsistent. As shown in Figure 1, a coherent response must align with the target utterance both topically and rhetorically to be effective.

Example of multi-party dialogue history and the importance of coherence in responses.

Methodology: The RL-TRC Framework

To address these limitations, the researchers introduced RL-TRC (Reinforcement Learning for Topic and Rhetorical Coherence). Built upon a BART-base architecture, this framework integrates two auxiliary tasks and a reinforcement learning policy to guide response generation.

The RL-TRC architecture, featuring topic and rhetorical coherence tasks and a discourse-aware reward system.

Coherence Tasks

  1. Topic Coherence Task: The model extracts keywords from the dialogue history and calculates Pointwise Mutual Information (PMI) to construct a topic coherence matrix. This allows the model to predict relevant keywords for the response.
  2. Rhetorical Coherence Task: Using a discourse parser, the model identifies 16 types of logical relations (e.g., Comment, Contrast, Question-answer pair). The model learns to predict the appropriate rhetorical relationship between the target message and the response.

Reinforcement Learning and Rewards

An RL agent learns a policy to select semantic "triggers" that guide the decoder. The policy is optimized using three discourse-aware rewards:

  • Topic Reward ($R_{tc}$): Measures the alignment between generated keywords and the predicted topic.
  • Rhetorical Reward ($R_{rc}$): Uses KL-divergence to ensure the logical flow matches human-like discourse.
  • Reply-to Reward ($R_{rt}$): A global reward assessing whether the generated response clearly points back to the correct target utterance in the history.

Experimental Results

The model was evaluated on two popular benchmarks derived from Ubuntu IRC logs: the Hu and Ou5 datasets.

Quantitative Performance

RL-TRC achieved new state-of-the-art results across various automated metrics. Notably, on the Ou5 dataset, RL-TRC achieved a BLEU-4 score of 2.93, a significant improvement over the previous best of 1.63 (MADNet). Similar gains were observed in METEOR and ROUGE-L metrics across both datasets.

Human Evaluation

Human evaluators assessed the models based on Fluency (Flu), Informativeness (Infor), and Relevance (Rel). RL-TRC outperformed baselines in Relevance (0.73 vs. 0.63 for MADNet), indicating that the topic and rhetorical modeling successfully produced more contextually appropriate responses.

Discussion and Insights

A key finding is that Topic Coherence currently plays a more vital role than Rhetorical Coherence in MPD performance. Ablation studies showed a sharper performance drop when topic components were removed. However, the effectiveness of rhetorical modeling is currently limited by the accuracy of existing discourse parsers (~65%).

Furthermore, the "Reply-to Reward" proved to be a powerful global signal. By rewarding responses that make the target utterance easily recognizable, the model learns to produce more distinctive and relevant content.

Conclusion

RL-TRC moves beyond structural modeling in multi-party dialogues by explicitly incorporating topic and rhetorical coherence. By leveraging reinforcement learning with discourse-aware rewards, the model generates responses that are not only grammatically correct but also logically and topically aligned with the conversation flow. Future work will explore scaling this framework to larger language models and improving discourse parsing accuracy.

Summary updated

Abstract

Previous studies on multi-party dialogue generation predominantly concentrated on modeling the reply-to structure of dialogue histories, always overlooking the coherence between generated responses and target utterances. To address this issue, we propose a Reinforcement Learning approach emphasizing both Topic and Rhetorical Coherence (RL-TRC). In particular, the topic- and rhetorical-coherence tasks are designed to enhance the model’s perception of coherence with the target utterance. Subsequently, an agent is employed to learn a coherence policy, which guides the generation of responses that are topically and rhetorically aligned with the target utterance. Furthermore, three discourse-aware rewards are developed to assess the coherence between the generated response and the target utterance, with the objective of optimizing the policy. The experimental results and in-depth analyses on two popular datasets demonstrate that our RL-TRC significantly outperforms the state-of-the-art baselines, particularly in generating responses that are more coherent with the target utterances.
BibTeX
@inproceedings{fan2024improving,
  author = {Yaxin Fan and Peifeng Li and Qiaoming Zhu},
  title = {{Improving Multi-party Dialogue Generation via Topic and Rhetorical Coherence}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=xz3OzamSFR},
}
Figure