Restatement and Question Generation for Counsellor Chatbot

John Lee, Baikun Liang, Haley FongACL 2021

TL;DR

This paper introduces a novel approach to generating empathetic and engaging responses for counsellor chatbots by adapting text summarization and question generation models. By fine-tuning a state-of-the-art summarization model (BertSum) on both domain-specific and large-scale open-domain datasets, they achieve significant improvements in generating restatements and questions, crucial for effective online counseling.

What Problem Does This Paper Address?

The increasing demand for mental health support has led to the development of AI-powered chatbots for counseling. A key challenge in these interactions is for the chatbot to effectively understand and engage the user. This involves not only recognizing emotional states but also encouraging the user to share more by using "encouragers" like restatements (paraphrasing or summarizing key points) and questions (probing for elaboration). While existing dialogue systems have explored various response generation techniques, there's a specific need for models that can generate high-quality, contextually relevant restatements and questions tailored for the sensitive domain of counseling. Furthermore, obtaining sufficient domain-specific data for training such models is a significant hurdle.

Methods

The authors tackle this problem by framing restatement and question generation as NLP tasks related to summarization and question generation. They explore several models, including:

  • DialoGPT: A pre-trained generative model for Chinese chitchat, fine-tuned on their datasets.
  • mT5: A multilingual variant of the T5 model, known for its strong performance in question generation, also fine-tuned.
  • BertSum: A state-of-the-art abstractive summarization model that uses a BERT-based encoder. This model is particularly interesting as it's adapted for restatement generation, which shares similarities with summarization.
  • Global Encoding: A framework that uses global information to improve text representations for summarization.

The core of their approach involves fine-tuning these pre-trained models on their custom datasets. A key innovation is the augmentation of a small, manually annotated in-domain dataset with a large-scale, automatically mined open-domain dataset. This strategy aims to leverage the broad knowledge of open-domain data while retaining the specificity of counseling dialogues.

For interleaving restatements and questions, they propose methods using the best performing models (BertSum+ and DialoGPT) and introduce a confidence threshold mechanism to decide whether to generate a restatement or a question, aiming for a balanced conversational flow.

Datasets

The paper utilizes two primary datasets:

  • Manual dataset: A newly introduced dataset of Cantonese post-reply pairs specifically curated for loneliness, academic anxiety, and test anxiety. This dataset was manually annotated by 10 undergraduate students, with human annotators marking target phrases and composing gold restatements and questions. It contains 12,634 post-restatement pairs and 9,036 post-question pairs.
  • Automatically mined dataset: This dataset was created by extracting pairs from large existing corpora (LCCC dataset for Mandarin dialogues and Cantonese discussion forums). Two methods were used:
  • Extraction: Identifying common strings in post-reply pairs to generate restatements and looking for question-like patterns to generate questions.
  • Matching: Reusing restatements and questions from the manual dataset for similar target phrases found in the open-domain data.

Key Insights and Ideas

  • Summarization models are surprisingly effective for restatement generation: The paper demonstrates that adapting summarization models like BertSum can yield high-quality restatements, even outperforming models specifically designed for question generation in this task. This highlights the overlap in linguistic skills required.
  • Data augmentation is crucial: Combining a small, high-quality in-domain dataset with a large, automatically mined open-domain dataset significantly boosts performance, addressing the common challenge of data scarcity in specialized domains.
  • A confidence threshold for response type selection is effective: Instead of randomly choosing between restatement and question, using a confidence threshold based on model scores allows for a more controlled and potentially more natural conversational flow, balancing the frequency of each response type.
  • Domain-specific nuances matter: While generic models can be adapted, the paper shows that fine-tuning with domain-specific data (even if limited) is essential for achieving optimal performance in the counseling context, where subtle linguistic cues are important.

Key Results

  • Restatement Generation: BertSum+, fine-tuned on both manual and automatically mined data, achieved the best ROUGE-L score of 0.7142, surpassing even an "Oracle Retrieval" baseline.
  • Question Generation: While generally scoring lower than restatements, BertSum+ also performed strongly, achieving a ROUGE-L score of 0.4665.
  • Interleaving Responses: The BertSum+ (threshold) model achieved a ROUGE-L score of 0.7013 for response generation (either restatement or question), demonstrating the effectiveness of the confidence threshold approach. This was only a slight drop from the ceiling performance, suggesting its practical utility.

Limitations and Future Work

  • Limited scope of manual dataset: The manual dataset, while valuable, is limited to Cantonese and specific topics (loneliness, academic/test anxiety). Future work could expand this to other languages and a wider range of mental health concerns.
  • Static response selection: The current approach doesn't fully account for the progression of a counseling session. Future research could incorporate session context to dynamically select replies.
  • Measuring counseling outcomes: The paper focuses on linguistic quality metrics (BLEU, ROUGE). A crucial next step would be to measure the correlation of these chatbot responses with actual counseling outcomes.
  • Beyond single-turn generation: The current work focuses on single-turn response generation. Future research could explore multi-turn dialogue coherence and context management.

Summary updated

BibTeX
@inproceedings{lee2021restatement,
  author = {John Lee and Baikun Liang and Haley Fong},
  title = {{Restatement and Question Generation for Counsellor Chatbot}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure