Abstract
Neural text retrievers have significantly advanced information retrieval but face challenges in parameter storage and zero-shot generalization. This study explores the efficacy of Parameter-Efficient (PE) tuning, particularly P-Tuning v2, in creating robust neural retrievers. By updating only 0.1% of a model's parameters, the authors achieve in-domain performance comparable to full fine-tuning while substantially improving generalization across diverse domains and topics. These gains are linked to superior confidence calibration and enhanced robustness to varying query lengths, offering a scalable solution for high-performance, generalizable search.
Introduction
Modern neural retrievers like DPR and ColBERT excel at semantic search but are hindered by two primary issues. First, parameter inefficiency requires storing a full copy of a large language model for every specific task, leading to prohibitive storage costs. Second, weak generalization often results in poor performance when these models are applied to "zero-shot" scenarios outside their training domain, where traditional keyword-based methods like BM25 frequently remain superior. This research investigates if PE tuning can mitigate these drawbacks.
Methodology
The study compares standard full fine-tuning (100% parameter update) against several PE methods designed to minimize the number of trainable parameters:
- Adapters: Small bottleneck layers inserted within Transformer blocks.
- BitFit: Tuning only the bias terms.
- Prompt Tuning: Adding trainable virtual tokens to the input.
- P-Tuning v2: The primary focus, which applies trainable prefix vectors to every layer of the Transformer, rather than just the input.
By utilizing P-Tuning v2, the researchers updated only approximately 0.1% of the total model parameters.
Evaluation Framework
Performance was evaluated across three distinct settings:
- In-Domain: Standard OpenQA benchmarks including Natural Questions and TriviaQA.
- Cross-Domain: The BEIR benchmark, encompassing 15 diverse datasets such as SciFact (biomedical) and ArguAna (argument retrieval).
- Cross-Topic: A new dataset, OAG-QA, containing 18,000 query-paper pairs across 22 scientific disciplines and 87 topics, sourced from academic Q&A platforms.
Results and Analysis
The experimental results highlight a significant advantage for parameter-efficient approaches in generalization.
Key Findings:
- Generalization through Sparsity: Tuning fewer parameters prevents the model from overfitting to specific training data quirks, such as vocabulary or query length distributions.
- Improved Calibration: Unlike full fine-tuning, which often leads to overconfident but incorrect predictions on out-of-domain data, PE tuning maintains better confidence calibration.
- Robustness: Because the core model and positional embeddings remain frozen, PE-tuned models are more resilient to variations in query length.
Performance Metrics:
- Efficiency: P-Tuning v2 matches full fine-tuning in-domain with 1000x fewer updated parameters.
- Zero-Shot Gains: On BEIR, P-Tuning v2 improved DPR performance by 5.2% absolute over full fine-tuning.
- Academic Retrieval: In the OAG-QA dataset, P-Tuning v2 outperformed full fine-tuning in 20 out of 22 disciplines.
Discussion and Future Work
While P-Tuning v2 offers clear benefits in efficiency and generalization, it presents certain optimization challenges. It typically converges more slowly than full fine-tuning and exhibits higher sensitivity to hyperparameters like the learning rate. Consequently, while storage is saved, training time may increase. Future research could explore whether these calibration and generalization advantages extend to other NLP tasks such as text generation or complex classification at scale.
Summary updated