ICXML: An In-Context Learning Framework for Zero-Shot Extreme Multi-Label Classification

Yaxin Zhu, Hamed Zamani

Abstract

This paper focuses on the task of Extreme Multi-Label Classification (XMC) whose goal is to predict multiple labels for each instance from an extremely large label space. While existing research has primarily focused on fully supervised XMC, real-world scenarios often lack supervision signals, highlighting the importance of zero-shot settings. Given the large label space, utilizing in-context learning approaches is not trivial. We address this issue by introducing In-Context Extreme Multi-label Learning (ICXML), a two-stage framework that cuts down the search space by generating a set of candidate labels through in-context learning and then reranks them. Extensive experiments suggest that ICXML advances the state of the art on two diverse public benchmarks.

Introduction

Extreme Multi-Label Classification (XMC) is the task of assigning multiple relevant labels to an instance from an enormous pool—often reaching hundreds of thousands or millions of possibilities. Common applications include tagging Wikipedia articles or recommending products on Amazon. While supervised XMC has seen great success, it relies on massive amounts of human-annotated data, which is often unavailable in real-world "zero-shot" scenarios.

Large Language Models (LLMs) are excellent at zero-shot reasoning but face a "prompt bottleneck": the entire label space cannot fit into a prompt, and free generation often leads to "hallucinated" labels that do not exist in the official set. ICXML bridges this gap by combining the creative generation of LLMs with the grounding of traditional retrieval.

Methods

ICXML utilizes a two-stage "Generate-then-Rerank" framework to navigate massive label spaces without labeled training data.

The ICXML framework consists of two main stages: candidate label shortlisting and label reranking.

Stage 1: Candidate Label Shortlisting

To help the LLM understand the task, ICXML generates "pseudo-demonstrations" using two strategies:

  • Content-based: Generating similar inputs for the test case.
  • Label-centric: Retrieving likely labels and generating synthetic inputs for them.

The LLM is prompted with these examples to produce "raw" labels. Since these might not match the exact dataset labels, a zero-shot retriever (TAS-B) maps these outputs to the nearest actual labels, creating a condensed shortlist.

Stage 2: Label Reranking

The LLM is then provided with the original test instance and the condensed shortlist to perform a listwise reranking, selecting the most relevant labels from the grounded candidates.

Experimental Setup

The framework was evaluated on two large-scale public benchmarks:

  • LF-Amazon-131K: A product recommendation dataset with approximately 131,000 labels.
  • LF-WikiSeeAlso-320K: A Wikipedia article title tagging dataset with approximately 312,000 labels.

Key Results and Insights

ICXML sets a new state-of-the-art for zero-shot XMC. On LF-Amazon-131K, it achieved a Precision@1 (P@1) of 0.234, outperforming the previous best zero-shot method, RTS (0.187). On LF-WikiSeeAlso-320K, it reached a P@1 of 0.278, surpassing the TAS-B baseline (0.237).

Performance comparison of ICXML against baselines across different demonstration settings.

Analysis revealed several key insights:

  • The Value of Pseudo-Demonstrations: Even synthetic demonstrations significantly boost performance over free generation. As shown in the results, ICXML consistently outperforms random text or label baselines across various settings.
  • Domain Sensitivity: Label-centric demonstrations performed better on Wikipedia (where titles are strong identifiers), while Content-based demonstrations were more effective for Amazon product data.
  • Scaling with Model Power: Switching from GPT-3.5 to GPT-4 for the reranking stage showed a massive performance jump (e.g., P@10 on Wikipedia increased from 0.353 to 0.427), suggesting the framework scales well with LLM capabilities.

Conclusion and Limitations

ICXML demonstrates that LLMs can effectively serve as semantic bridges in XMC by narrowing the search space through grounded generation. However, the multi-stage LLM inference process is computationally expensive compared to traditional retrieval. Future work will explore multi-modal XMC and domain-specific applications where general-purpose LLMs might require further adaptation.

Summary updated

Abstract

This paper focuses on the task of Extreme Multi-Label Classification (XMC) whose goal is to predict multiple labels for each instance from an extremely large label space. While existing research has primarily focused on fully supervised XMC, real-world scenarios often lack supervision signals, highlighting the importance of zero-shot settings. Given the large label space, utilizing in-context learning approaches is not trivial. We address this issue by introducing In-Context Extreme Multi-label Learning (ICXML), a two-stage framework that cuts down the search space by generating a set of candidate labels through in-context learning and then reranks them. Extensive experiments suggest that ICXML advances the state of the art on two diverse public benchmarks.
BibTeX
@inproceedings{zhu2024icxml,
  author = {Yaxin Zhu and Hamed Zamani},
  title = {{ICXML: An In-Context Learning Framework for Zero-Shot Extreme Multi-Label Classification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=ULQ5PW1DHA},
}
Figure