ICXML: An In-Context Learning Framework for Zero-Shot Extreme Multi-Label Classification

Yaxin Zhu, Hamed Zamani

Abstract

This paper focuses on the task of Extreme Multi-Label Classification (XMC) whose goal is to predict multiple labels for each instance from an extremely large label space. While existing research has primarily focused on fully supervised XMC, real-world scenarios often lack supervision signals, highlighting the importance of zero-shot settings. Given the large label space, utilizing in-context learning approaches is not trivial. We address this issue by introducing In-Context Extreme Multilabel Learning (ICXML), a two-stage framework that cuts down the search space by generating a set of candidate labels through in-context learning and then reranks them. Extensive experiments suggest that ICXML advances the state of the art on two diverse public benchmarks.

Introduction

Extreme Multi-Label Classification (XMC) involves assigning a small subset of relevant labels to an input from a massive pool, often containing hundreds of thousands or millions of candidates. While supervised XMC is well-established, the zero-shot setting—where no labeled training data is available—remains a significant challenge. Traditional zero-shot methods typically rely on lexical matching or dense retrieval, which often struggle with the "vocabulary gap" between inputs and labels.

Large Language Models (LLMs) offer a potential solution due to their extensive internal knowledge. However, they face a "prompt bottleneck": the massive label space cannot fit within a standard context window. To overcome this, we propose ICXML (In-Context Extreme Multi-label Learning), a framework that leverages LLMs for both candidate generation and reranking in a zero-shot manner.

Methodology: The ICXML Framework

ICXML employs a "Generate-then-Rerank" pipeline designed to bridge the gap between LLM conceptual generation and specific label identifiers.

Overview of the ICXML framework architecture.

1. Demonstration Generation

To provide context without ground-truth data, ICXML generates "pseudo-demonstrations." This is achieved through two strategies:

  • Content-based: The LLM generates potential inputs based on the test instance, which are then paired with labels retrieved from the label space.
  • Label-centric: The system retrieves labels similar to the test instance, and the LLM "back-generates" corresponding inputs.
Examples of Content-based and Label-centric pseudo-demonstration generation.

2. Candidate Label Shortlisting

The LLM is prompted with these pseudo-demonstrations to generate a raw list of potential labels. Because these generated strings may not exactly match the dataset's official identifiers, a zero-shot retriever (TAS-B) maps the LLM's output to the nearest actual labels, creating a manageable shortlist.

3. Label Reranking

Finally, the LLM performs listwise reranking. It receives the test instance and the condensed shortlist, selecting the top-10 most relevant labels. This stage allows the model to reason across the entire candidate set simultaneously.

Experimental Evaluation

The framework was evaluated on two large-scale benchmarks: LF-Amazon-131K (~131,000 labels) and LF-WikiSeeAlso-320K (~312,000 labels).

Key Results

ICXML consistently outperformed state-of-the-art zero-shot baselines:

  • LF-Amazon-131K: Achieved a Precision@1 (P@1) of 0.234, significantly higher than the best baseline (RTS at 0.187).
  • LF-WikiSeeAlso-320K: Reached a P@1 of 0.278, surpassing the TAS-B soft-matching baseline (0.237).

Ablation studies highlighted that LLM-based reranking is a critical performance driver, far superior to heuristic or specialized ranking models. Furthermore, scaling the reranking stage to GPT-4 showed continued performance gains, particularly on complex datasets like Wikipedia.

Conclusion

ICXML demonstrates that synthetic supervision via pseudo-demonstrations can effectively enable in-context learning for extreme classification tasks. By combining LLM generation with semantic retrieval and listwise reranking, the framework bypasses the context window limitations of current models. Future work will explore multi-modal XMC and strategies to reduce the inference costs associated with multi-stage LLM pipelines.

Summary updated

Abstract

This paper focuses on the task of Extreme Multi-Label Classification (XMC) whose goal is to predict multiple labels for each instance from an extremely large label space. While existing research has primarily focused on fully supervised XMC, real-world scenarios often lack supervision signals, highlighting the importance of zero-shot settings. Given the large label space, utilizing in-context learning approaches is not trivial. We address this issue by introducing In-Context Extreme Multilabel Learning (ICXML), a two-stage framework that cuts down the search space by generating a set of candidate labels through incontext learning and then reranks them. Extensive experiments suggest that ICXML advances the state of the art on two diverse public benchmarks.
BibTeX
@inproceedings{zhu2024icxml,
  author = {Yaxin Zhu and Hamed Zamani},
  title = {{ICXML: An In-Context Learning Framework for Zero-Shot Extreme Multi-Label Classification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=ULQ5PW1DHA},
}
Figure