Self-Teaching Prompting for Multi-Intent Learning with Limited Supervision

Cheng Chen, Ivor TsangICLR 2024 · OpenReview

TL;DR

This paper introduces "Self-Teaching Prompting" (STP), a novel method to improve multi-intent learning with limited labeled data by leveraging Large Language Models (LLMs). STP enables LLMs to iteratively refine their predictions by learning from their own consistent outputs, significantly boosting accuracy in complex intent classification tasks.

What Problem Does This Paper Address?

Accurately classifying multiple intents within a single utterance is crucial for sophisticated dialogue systems. However, training such models typically requires large, meticulously annotated datasets, which are expensive and time-consuming to create. While Large Language Models (LLMs) like ChatGPT offer a promising avenue for generating these annotations, they suffer from two main drawbacks: response randomness and a static prompting approach that doesn't learn from past outputs. This leads to inconsistent and suboptimal performance, especially in domains demanding high-quality annotations.

Methods

The core of this paper is the Self-Teaching Prompting (STP) method, designed to overcome the limitations of LLM-based annotation. STP operates in an iterative manner over several epochs:

  • Hints: To combat response randomness, STP initially uses a small set of clean, labeled samples as "hints." These hints are examples with high contextual similarity to the input utterance, guiding the LLM.
  • Consistent-Sample Selection: For each input utterance, STP queries the LLM (ChatGPT) multiple times with varying "temperature" parameters (controlling randomness). It then identifies samples where the LLM consistently predicts the same label across these different temperature settings. These consistent predictions form "additional hints" for the next iteration.
  • Iterative Refinement: In each epoch, the LLM's predictions are refined by incorporating these newly identified consistent samples into the set of hints. This process allows the LLM to learn from its own more reliable outputs, gradually improving its accuracy and consistency over multiple epochs.

Datasets

The paper evaluates STP on two established multi-intent datasets:

  • MixATIS: Contains 13,162 utterances for training, 756 for validation, and 828 for testing.
  • MixSNIPS: A larger dataset with 39,776 utterances for training, 2,198 for validation, and 2,199 for testing.

Additionally, the authors explore the generalizability of STP by incorporating multilingual and multi-domain datasets, including:

  • Legal Dataset: Features 1,385 training samples across four intents.
  • Healthcare Datasets (IHQID-WebMD and 1mg): These datasets represent Indian hospital query data in various Indic languages. For evaluation, the English and Gujarati versions were used, each with 306 training samples and four intents.

Key Insights and Ideas

  • LLMs as Self-Correcting Annotators: The most significant insight is that LLMs can be trained to "self-teach" by iteratively refining their prompts based on their own consistent outputs. This moves beyond static prompting to a dynamic, learning-based annotation process.
  • Temperature as a Randomness Control: The strategic use of varying temperature parameters in LLM generation is key to identifying consistent predictions, which are then leveraged to improve future predictions.
  • Iterative Improvement is Crucial: The experiments demonstrate a clear trend of increasing accuracy with each epoch of STP, highlighting the value of iterative refinement.
  • Cost-Effectiveness: Despite the iterative process, STP is shown to be cost-effective, achieving higher accuracy with comparable or lower computational costs than other advanced prompting methods.

Key Results

STP consistently outperforms baseline methods, including Chain of Thought, Self-Consistency, Tree of Thoughts, and Few Shots, across the evaluated datasets.

  • On MixSNIPS, STP achieves a matching ratio of 0.7812 in its third epoch, significantly higher than Random-CoT Prompt (0.581) and Chain of Thought Prompting (0.4257).
  • On the Medical Dataset (English), STP reaches an intent accuracy of 0.8418 by Epoch 4, with a low standard deviation of 0.002718, indicating high consistency.
  • The paper also highlights that STP is cost-effective, achieving a 7.91% higher intent accuracy than the Few Shot method and a 6.43% improvement over the Tree of Thoughts method on the Medical Dataset, at a significantly lower cost.

Limitations and Future Work

  • Computational Cost: The iterative nature of STP, involving multiple epochs and temperature configurations, inherently increases computational resource usage compared to non-iterative methods. However, the authors argue this is justified by the improved accuracy.
  • Adaptability: The paper suggests that STP can be adjusted by decreasing the number of epochs and temperatures in resource-constrained environments, indicating a degree of flexibility.
  • Future Work: While not explicitly detailed, the success across diverse datasets (legal, medical, multilingual) suggests that further exploration into STP's adaptability to even more complex and varied domains would be a natural next step. The paper also implies that optimizing the selection of "hints" and the criteria for "consistent samples" could lead to further improvements.

Summary updated

Abstract

Multi-intent learning with limited supervision involves predicting multiple intentions of utterances using only a few annotated samples. The primary motivation for this task stems from the high costs and cumbersome processes associated with annotating large datasets. To mitigate this, we propose utilising Large Language Models (LLMs) for annotation assistance. Although LLMs show promise, they struggle with response randomness, and their previous prompts is static and do not learn from their outputs. To address this, we propose `self-teaching prompting' (STP), a method that enables Large Language Models (LLMs) to iteratively learn from their consistent samples and refine their predictions over time. Our experiments with multi-intention datasets demonstrate that STP significantly enhances response accuracy.
BibTeX
@inproceedings{chen2024self,
  author = {Cheng Chen and Ivor Tsang},
  title = {{Self-Teaching Prompting for Multi-Intent Learning with Limited Supervision}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=DeoamI1BFh},
}
Figure