Topic-switch adapted Japanese Dialogue System based on PLATO-2

Donghuo Zeng, Jianming Wu, Yanan Wang, Kazunori Matsumoto, Gen Hattori, Kazushi IkedaarXiv 2023 · OpenReview

Abstract

Large-scale open-domain dialogue systems such as PLATO-2 have achieved state-of-the-art scores in both English and Chinese. However, little work explores whether such dialogue systems also work well in the Japanese language. In this work, we create a large-scale Japanese dialogue dataset, Dialogue-Graph, which contains 1.656 million dialogue data in a tree structure from News, TV subtitles, and Wikipedia corpus. Then, we train PLATO-2 using Dialogue-Graph to build a large-scale Japanese dialogue system, PLATO-JDS. In addition, to improve the PLATO-JDS in the topic switch issue, we introduce a topic-switch algorithm composed of a topic discriminator to switch to a new topic when user input differs from the previous topic. We evaluate the user experience by using our model with respect to four metrics, namely, coherence, informativeness, engagingness, and humanness. As a result, our proposed PLATO-JDS achieves an average score of 1.500 for the human evaluation with human-bot chat strategy, which is close to the maximum score of 2.000 and suggests the high-quality dialogue generation capability of PLATO-2 in Japanese. Furthermore, our proposed topic-switch algorithm achieves an average score of 1.767 and outperforms PLATO-JDS by 0.267, indicating its effectiveness in improving the user experience of our system.

Introduction

While large-scale dialogue models like PLATO-2 have achieved state-of-the-art results in English and Chinese, Japanese-specific implementations have historically lagged behind. A significant challenge in transformer-based chatbots is "topic adaptability." In multi-turn conversations, models typically concatenate all previous turns as input. However, if a user suddenly switches topics—for instance, from "food" to "hobbies"—the model often remains anchored to the previous context, leading to incoherent or irrelevant responses. This "stickiness" of context hinders natural interaction. Researchers from KDDI Research, Inc. address this by developing PLATO-JDS, a system optimized for the fluid, pivot-heavy nature of real-world Japanese conversation.

Methodology

The researchers’ approach involves two primary components: model adaptation and a novel switching logic.

PLATO-JDS Architecture

The system adapts the PLATO-2 framework, utilizing a curriculum learning strategy divided into two stages:

  1. Stage 1: One-to-One Mapping. The model is trained on simple prompt-response pairs to learn general language characteristics and basic conversational flow.
  2. Stage 2: One-to-Many Mapping. Using discrete latent variables, the model learns to capture the inherent diversity of human speech, where a single prompt can elicit multiple valid responses.

Topic-Switch Algorithm

To manage context effectively, a BERT-based binary classifier, termed the "topic discriminator," acts as a gatekeeper. Before generating a response, it analyzes the current user input against the previous dialogue context.

The overall architecture of the topic-switch adapted Japanese dialogue system, illustrating the flow from topic discrimination to response generation via PLATO-2.

As shown in the figure above, if the discriminator predicts the topic remains consistent (using a threshold of 0.61), the system utilizes the full dialogue history. If a topic change is detected, the system "switches" by ignoring the previous history and generating a response based solely on the new input. This prevents "stale" context from polluting the generation process.

The Dialogue-Graph Dataset

The foundation of PLATO-JDS is Dialogue-Graph, a newly curated, large-scale Japanese dataset comprising 1.656 million dialogue turns. Unlike traditional linear datasets, Dialogue-Graph employs a tree structure.

The tree structure of the Dialogue-Graph dataset, showing branching paths for various topics such as food, clothes, and hobbies.

Starting from 50 initial topics (e.g., family, travel, cooking), the graph branches out, facilitating "one-to-many" training scenarios. Data was collected via a crowd-driven asynchronous system where workers generated utterances based on specific topics and preceding turns. This structure allows the model to understand that a single statement can lead to many different, equally valid conversational paths.

Experimental Results

The system was evaluated using both "bot-bot" simulations and "human-bot" interactions across four metrics: coherence, informativeness, engagingness, and humanness.

  • Human Evaluation: The topic-switch version achieved an average score of 1.767 (on a 0-2 scale), significantly outperforming the base PLATO-JDS model (1.500).
  • Coherence and Informativeness: In human-bot chats, the switching algorithm improved coherence scores from 1.600 to 1.733 and informativeness from 1.467 to 1.800.
  • Lexical Diversity: Automatic metrics (Distinct-1/2) indicated that the topic-switch model produced more diverse and less repetitive language.
  • Topic Handling: The adapted system managed an average of 6.5 topics per session, compared to 4.2 for the base model, demonstrating its superior ability to follow shifting human interests.

Discussion and Conclusion

The results suggest that "less context" can sometimes lead to a better user experience by preventing hallucinations caused by irrelevant historical data. The 0.61 threshold for the BERT discriminator provided a high recall (99.75%) for identifying pivots.

However, limitations remain. The system is currently trained on 50 specific topics, which, while broad, do not encompass the full spectrum of open-domain conversation. Additionally, the binary "yes/no" switch could be refined into a more nuanced "soft" transition. Despite these challenges, PLATO-JDS represents a significant advancement in Japanese conversational AI, proving that large-scale models can be successfully adapted to handle complex topic dynamics.

Summary updated

Abstract

Large-scale open-domain dialogue systems such as PLATO-2 have achieved state-of-the-art scores in both English and Chinese. However, little work explores whether such dialogue systems also work well in the Japanese language. In this work, we create a large-scale Japanese dialogue dataset, Dialogue-Graph, which contains 1.656 million dialogue data in a tree structure from News, TV subtitles, and Wikipedia corpus. Then, we train PLATO-2 using Dialogue-Graph to build a large-scale Japanese dialogue system, PLATO-JDS. In addition, to improve the PLATO-JDS in the topic switch issue, we introduce a topic-switch algorithm composed of a topic discriminator to switch to a new topic when user input differs from the previous topic. We evaluate the user experience by using our model with respect to four metrics, namely, coherence, informativeness, engagingness, and humanness. As a result, our proposed PLATO-JDS achieves an average score of 1.500 for the human evaluation with human-bot chat strategy, which is close to the maximum score of 2.000 and suggests the high-quality dialogue generation capability of PLATO-2 in Japanese. Furthermore, our proposed topic-switch algorithm achieves an average score of 1.767 and outperforms PLATO-JDS by 0.267, indicating its effectiveness in improving the user experience of our system.
BibTeX
@misc{zeng2023topic,
  author = {Donghuo Zeng and Jianming Wu and Yanan Wang and Kazunori Matsumoto and Gen Hattori and Kazushi Ikeda},
  title = {{Topic-switch adapted Japanese Dialogue System based on PLATO-2}},
  year = {2023},
  eprint = {2302.11280},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.11280},
}
Figure