TL;DR
This paper introduces "ICL Markup," a novel method that uses soft-token tags, inspired by HTML, to structure in-context learning (ICL) prompts. By learning these tags in a parameter-efficient "warm-up" phase, ICL Markup reduces arbitrary prompt engineering choices and improves LLM performance on various classification tasks, including intent detection and text classification, without requiring further fine-tuning.
What Problem Does This Paper Address?
Large Language Models (LLMs) have shown remarkable adaptability to new tasks through in-context learning (ICL). This approach involves providing the LLM with natural language instructions and a few examples of the desired task. While flexible and data-efficient, ICL is plagued by a multitude of arbitrary choices users must make when crafting prompts. These choices, such as how to format instructions or separate examples, can significantly impact model performance, leading to a lack of robustness. This paper aims to address this challenge by providing a structured and streamlined way to create effective ICL prompts.
Methods
The core innovation is "ICL Markup," which draws inspiration from markup languages like HTML. Instead of relying on arbitrary natural language phrases to structure prompts, ICL Markup introduces dedicated "soft-token tags" that are added to the LLM's vocabulary. These soft-tokens are trainable parameters that are learned in advance through a parameter-efficient fine-tuning process, termed a "warm-up" phase.
The process involves:
- Defining a Markup Template: A template is created using these soft-token tags to structure the ICL prompt (e.g., for classification tasks, tags like
<classification>,<options>,<demo>,<input>, and<label>are used). - Warm-up Phase (Meta-Learning): The soft-token tags are trained on a set of related tasks. This is a parameter-efficient fine-tuning process where only the parameters associated with the soft-token tags are updated, not the entire LLM.
- Usage on New Tasks: Once learned, these soft-token tags and their associated template can be used for ICL on new, unseen tasks without any further fine-tuning of the LLM. This allows for the transfer of learned prompt structures across tasks.
Datasets
The paper utilizes several datasets for training and evaluation:
- Huffington Post News dataset: Used for few-shot news headline classification.
- Intent Detection Datasets: HWU64, SNIPS, ATIS, BANKING77, and CLINC150 were used for training ICL Markup tags and for few-shot and open-world intent detection evaluations.
- LEDGAR dataset: A legal text classification dataset used to evaluate the transferability of learned soft-token tags to a task with a different objective.
Key Insights and Ideas
- Arbitrary choices in ICL are a significant bottleneck: The paper demonstrates that small, seemingly arbitrary changes in prompt phrasing can lead to substantial performance variations, highlighting the need for a more structured approach.
- Soft-token tags can generalize across tasks: The learned soft-token tags, trained in a "warm-up" phase, can be effectively applied to new, unseen tasks without additional fine-tuning, showcasing their meta-learning capabilities.
- ICL Markup reduces prompt engineering complexity: By abstracting away arbitrary formatting decisions into learned soft-token tags, ICL Markup simplifies the process of applying ICL, making it more accessible and consistent.
- Improved robustness and accuracy: The proposed method not only improves average performance but also significantly reduces performance variability across different prompt configurations.
- Potential for open-world classification: The approach shows promise in handling out-of-scope (OOS) predictions, a critical aspect for real-world applications.
Key Results
- News Headline Classification: ICL Markup improved Flan-T5-XL's accuracy from a mean of 68.9% (or 70.9% with the best prompt) to 76.8% on the Huffington Post dataset, outperforming previous state-of-the-art results. It also reduced performance variance.
- Intent Detection: ICL Markup brought Flan-T5-XL's few-shot performance on par with fine-tuned models like PTR, and in some cases, outperformed ChatGPT baselines. For open-world intent detection, ICL Markup outperformed previous baselines in 5 out of 6 configurations.
- Legal Text Classification: While not a few-shot task, learned intent detection tags still helped improve Flan-T5's performance on the LEDGAR dataset, demonstrating cross-objective transfer.
- Robustness to "None of the Above" (NOTA): The "anneal" initialization strategy for soft-token tags led to better task accuracy and NOTA usage in intent detection experiments, particularly when dealing with many classes.
Limitations and Future Work
- Model Size: The experiments were conducted on LLMs up to 3 billion parameters, which are considered small compared to the state-of-the-art. Scaling experiments to larger models is a natural next step.
- Task Scope: The current work focuses on classification tasks. Expanding ICL Markup to other NLP tasks and exploring its application beyond classification is a key area for future research.
- Warm-up Task Diversity: Building a more diverse pool of warm-up tasks could lead to more general-purpose tags that can adapt to a wider range of objectives and tasks.
- Full Fine-tuning: While the current method avoids fine-tuning the entire LLM, exploring scenarios where full parameter fine-tuning is combined with learned tags could be beneficial.
- Explicit NOTA Tag: Introducing a dedicated
<nota>tag could further enhance the model's ability to handle out-of-scope predictions.
Summary updated