TL;DR
Hierarchical Text Classification (HTC) is crucial for organizing vast amounts of data, but traditional fine-tuning of large language models (LLMs) is computationally expensive. This paper introduces prefix-tuning for HTC, significantly reducing the number of trainable parameters (under 1%) while achieving comparable performance to full fine-tuning. They further enhance this by incorporating contrastive learning for better prefix initialization, leading to improved results.
What Problem Does This Paper Address?
Many real-world applications, like e-commerce product categorization or customer review organization, rely on Hierarchical Text Classification (HTC). This involves assigning text to labels organized in a tree-like structure. While Pre-trained Language Models (PLMs) have revolutionized NLP, fine-tuning them for HTC presents a significant challenge. Specifically, for local HTC models (where separate classifiers are trained for different levels of the hierarchy), the model size can explode as the hierarchy deepens. Even for global HTC models (a single classifier for all flattened labels), fine-tuning requires storing and updating a massive number of parameters. This becomes increasingly problematic as PLMs themselves grow larger. The paper aims to address this by exploring parameter-efficient fine-tuning techniques for HTC.
Methods
The core innovation of this paper is the application and adaptation of prefix-tuning for HTC. Unlike traditional fine-tuning where all PLM parameters are updated, prefix-tuning keeps the PLM frozen and only trains a small set of "prefix" vectors. These vectors are prepended to the input and guide the PLM to produce task-specific representations.
The authors explore two main strategies:
- Local HTC with Prefix-Tuning: They investigate how to effectively transfer knowledge between prefix vectors learned at different levels of the hierarchy. Three approaches are proposed:
- Non-transferring: Prefix vectors for each level are trained independently.
- Copy-transferring: Prefix vectors from a higher level are used to initialize the corresponding parts of the prefix vectors at a lower level.
- Transform-transferring: Prefix vectors from a higher level are transformed using a neural network to initialize the prefix vectors at a lower level.
- Global HTC with Prefix-Tuning and Contrastive Learning: For global HTC, they propose a contrastive learning (CL) based self-training step before the standard supervised fine-tuning of prefix vectors. This CL step helps initialize the prefix vectors by encouraging similar representations for augmented versions of the same input, leveraging the success of methods like SimCSE.
Datasets
The paper evaluates their methods on two widely used datasets:
- Web of Science (WoS): A benchmark dataset for HTC with abstracts of scientific papers.
- Amazon Beauty: An industry dataset containing product reviews, notable for potentially having multiple label paths per instance.
Key Insights and Ideas
- Prefix-tuning is a viable and efficient alternative to full fine-tuning for HTC: The ability to achieve comparable performance with less than 1% of trainable parameters is a significant breakthrough for deploying HTC models in resource-constrained environments.
- Hierarchical information transfer is beneficial for local HTC: Simply training prefix vectors independently (non-transferring) is less effective. Transferring learned prefix vectors from higher levels to lower levels (copy-transferring and transform-transferring) significantly improves performance.
- Direct copying of higher-level prefix vectors is more effective than transformation: The "copy-transferring" approach, where higher-level prefix vectors are directly used to initialize lower-level ones, yielded the best results among the local HTC strategies.
- Contrastive learning pre-training significantly boosts global HTC performance: Adding a CL step before supervised fine-tuning for global HTC models leads to improved results, especially on datasets with many labels.
Key Results
- On the WoS dataset, prefix-tuning (global, BCE loss) achieved a micro-F1 score of 86.84% and a macro-F1 of 79.77%, using only 0.46% of the parameters compared to CLS-tuning (100% parameters).
- For local HTC on WoS, the "copy-transferring" approach with prefix vector lengths of 10 (top) and 20 (bottom) achieved the highest micro-F1 of 87.24% and macro-F1 of 79.98%, again with 0.46% parameters. Increasing the bottom prefix vector length to 30 further improved performance to 87.34% micro-F1.
- On the Amazon Beauty dataset, global SPP-tuning with CL pre-training achieved a micro-F1 of 88.14% and a macro-F1 of 62.07%, outperforming SPP-tuning without CL (88.00% micro-F1, 61.36% macro-F1) and showing competitive results against CLS-tuning.
Limitations and Future Work
- Limitations: The paper notes that while macro-F1 on Amazon Beauty with CL pre-training was slightly lower than CLS-tuning, this warrants further investigation, especially for labels with sparse training instances. The effectiveness of the "copy-transferring" approach for local HTC might be sensitive to the specific hierarchy structure.
- Future Work: The authors plan to extend their work to address long-tailed label distributions, which are common in real-world applications. They also suggest exploring how to integrate structural encoders (like GCNs or tree LSTMs) to better represent and utilize label hierarchy information within the prefix-tuning framework.
Summary updated