ICL-Markup: Structuring In-Context Learning using Soft-Token Tags

Marc-Etienne Brunet, Ashton Anderson, Richard Zemel

TL;DR

This paper introduces "ICL-Markup," a novel method that uses learned "soft-token tags" to structure in-context learning (ICL) prompts, inspired by HTML. This approach reduces arbitrary prompt engineering decisions and improves performance on various downstream tasks, including intent detection and text classification, without requiring additional fine-tuning on new tasks.

What Problem Does This Paper Address?

Large Pretrained Language Models (PLMs) are incredibly versatile, and In-Context Learning (ICL) allows us to adapt them to new tasks with just a few examples. However, crafting effective ICL prompts is a complex and often arbitrary process. Engineers face numerous choices regarding how to format instructions, demonstrations, and labels, leading to significant variability in performance. This "prompt engineering" burden makes ICL less accessible and robust. The paper aims to address this by providing a structured, learnable way to create ICL prompts, reducing the need for manual trial-and-error.

Methods

The core innovation is "ICL-Markup," which uses special "soft-token tags" (also called tunable tokens) to define the structure of ICL prompts. These tags are learned in advance during a parameter-efficient fine-tuning phase, referred to as a "warm-up" process. This warm-up uses existing ICL tasks to train the soft-token weights. Once learned, these tags can be used in prompt templates for new, unseen tasks without any further fine-tuning of the PLM or the tags themselves. The approach is inspired by markup languages like HTML, where tags define structure and content. For classification tasks, they define tags like <classification>, <options>, <demo>, <input>, and <label> to delineate different parts of the prompt. This method essentially meta-learns prompt structures.

Datasets

The experiments utilize several datasets:

  • Huffington Post News dataset: Used for news headline classification.
  • BANKING77 and CLINC150: Standard datasets for intent detection.
  • BANKING77-OOS, CLINC-Single-Domain-OOS-banking, and CLINC-Single-Domain-OOS-credit-cards: Variants of the above datasets that include out-of-scope (OOS) examples, crucial for evaluating robustness.
  • HWU64, SNIPS, and ATIS: Used in conjunction with BANKING77 or CLINC150 for the "warm-up" phase of learning the ICL-Markup tags.
  • LEDGAR: A legal text classification dataset used to evaluate the transferability of learned tags to a different domain and objective.

The paper does not introduce new benchmark datasets but rather applies the proposed method to existing ones, often focusing on challenging aspects like open-world intent detection.

Key Insights and Ideas

  • Structuring ICL reduces arbitrary choices: By using learned soft-token tags, the paper demonstrates that prompt engineering for ICL can be made more systematic and less dependent on human intuition. This streamlines the process for engineers.
  • Meta-learning prompt structures is effective: The "warm-up" phase, where soft-token tags are learned on related tasks, allows the model to generalize these structural elements to new, unseen tasks without additional fine-tuning. This is a form of meta-learning for prompt design.
  • Soft-token tags offer flexibility and robustness: The learned tags can adapt to different task formats and improve performance, even reducing performance variability compared to manual prompt engineering.
  • Open-world intent detection benefits significantly: The method shows promise in handling out-of-scope (OOS) intents, a critical challenge for real-world applications like virtual assistants.

Key Results

  • Improved accuracy: ICL-Markup consistently improves performance across various tasks. For instance, on the Huffington Post dataset, Flan-T5-XL with ICL-Markup outperforms a strong baseline (PBML) in all tested configurations.
  • Reduced performance variability: The method significantly reduces the variance in performance seen with manual prompt engineering, especially for smaller models.
  • Strong intent detection performance: On intent detection tasks, ICL-Markup pushes Flan-T5-XL performance slightly above a rule-based prompt tuning method (PTR) and significantly outperforms naive ChatGPT baselines.
  • Effective OOS detection: For open-world intent detection, ICL-Markup outperforms previous baselines in a majority of configurations, demonstrating its utility in handling unseen intents.
  • Cross-domain transfer: Learned tags from intent detection tasks show improvements on the LEDGAR legal text classification dataset, indicating good generalization. For example, on LEDGAR, ICL-Markup improves Flan-T5-XL's performance.

Limitations and Future Work

  • Model size: The experiments were primarily conducted on relatively smaller PLMs (up to 3B parameters). The authors acknowledge that these are small compared to state-of-the-art models, and further investigation on larger models is warranted.
  • Scope of tasks: The current work focuses exclusively on classification tasks. The authors suggest that ICL-Markup has broader applications and future work could explore its use in other NLP tasks.
  • Soft-token representation: While the paper uses ordered sets of soft-tokens, the optimal size and structure of these sets for different tasks could be a direction for future research.
  • Initialization strategy: The paper notes that random initialization of soft-token tags can lead to more robustness in certain scenarios (like OOS threshold tuning), suggesting that exploring different initialization and learning strategies for soft-tokens could be beneficial.

Summary updated

Abstract

Large pretrained language models (PLMs) can be rapidly adapted to a wide variety of tasks via a text-to-text approach, where the instruction and input are fed to the model in natural language. Combined with in-context learning (ICL), this paradigm is impressively flexible and powerful. However, it also burdens engineers with an overwhelming amount of choices, many of them arbitrary. Inspired by markup languages like HTML, we contribute a method of using soft-token (a.k.a tunable token) tags to compose prompt templates. This approach reduces arbitrary decisions and streamlines the application of ICL. Our method is a form of meta-learning for ICL; it learns these tags in advance during a parameter-efficient fine-tuning ``warm-up'' process. The tags can subsequently be used in templates for ICL on new, unseen tasks without any additional fine-tuning. Our experiments with this approach yield promising initial results, improving PLM performance in important enterprise applications such as few-shot and open-world intent detection, as well as text classification in news and legal domains. Our method is a form of meta-learning for ICL; it learns these tags in advance during a parameter-efficient fine-tuning "warm-up" process. The tags can subsequently be used in templates for ICL on new, unseen tasks without any additional fine-tuning. Our experiments with this approach yield promising initial results. Improving PLM performance in important enterprise applications such as few-shot and open-world intent detection, as well as text classification in a legal domain.
BibTeX
@inproceedings{brunet2023icl,
  author = {Marc-Etienne Brunet and Ashton Anderson and Richard Zemel},
  title = {{ICL-Markup: Structuring In-Context Learning using Soft-Token Tags}},
  booktitle = {R0-FoMo @ NeurIPS},
  year = {2023},
  url = {https://openreview.net/forum?id=lfIXPclVHj},
}
Figure