Abstract
This study finds that existing information retrieval (IR) models show significant biases based on the linguistic complexity of input queries, performing well on linguistically simpler (or more complex) queries while underperforming on linguistically more complex (or simpler) queries. To address this issue, we propose EqualizeIR, a framework to mitigate linguistic biases in IR models. EqualizeIR uses a linguistically biased weak learner to capture linguistic biases in IR datasets and then trains a robust model by regularizing and refining its predictions using the biased weak learner. This approach effectively prevents the robust model from overfitting to specific linguistic patterns in data. We propose four approaches for developing linguistically-biased models. Extensive experiments on several datasets show that our method reduces performance disparities across linguistically simple and complex queries, while improving overall retrieval performance.
Introduction
When users interact with search engines, the system's ability to retrieve relevant documents should ideally remain independent of the query's linguistic complexity. However, research reveals that standard IR models—ranging from traditional lexical tools like BM25 to modern neural rankers like Dense Passage Retrieval (DPR)—are significantly biased by linguistic complexity.
Some models excel at simple queries but fail on complex ones, while others show the opposite trend. For instance, BM25 performs well on simple queries in biomedical contexts (NFCorpus) but improves as queries become more complex in financial datasets (FIQA). This performance disparity suggests that models often overfit to superficial linguistic patterns rather than deep relevance signals, leading to an inequitable user experience.
Methodology
The authors propose EqualizeIR, a two-stage framework designed to "equalize" performance across the complexity spectrum.
Stage 1: Training a Biased Weak Learner ($f_B$)
The first stage involves intentionally creating a model susceptible to linguistic shortcuts. The authors propose four strategies to "weaken" this model:
- Linguistically Biased Data: Artificially amplifying specific linguistic constructs.
- Weaker Architecture: Utilizing models with fewer parameters, such as TinyBERT.
- Less Training Time: Using an "undercooked" model that hasn't fully generalized.
- Limited Data: Training on a small subset of data to restrict the model's exposure.
Stage 2: Training a Robust Model ($f_R$) with Guidance
The robust model is trained by combining its predictions with those of the frozen weak learner. By identifying where the weak learner is overconfident or failing, the robust model learns to ignore biased patterns. If the weak learner is confident and correct, the robust model's loss is lowered (treating it as an "easy" example). Conversely, if the weak learner is confident but incorrect, the robust model is forced to prioritize that "hard" example.
Evaluation and Results
The framework was evaluated using the BEIR benchmark, focusing on four diverse datasets: MS MARCO (general), NFCorpus (biomedical), FIQA-2018 (finance), and SciFact (scientific fact-checking).
Key Findings
- Superior Accuracy: EqualizeIR consistently outperformed standard baselines. It improved average NDCG@10 by 0.03 to 0.15 absolute points across various datasets.
- Enhanced Equity: The framework significantly reduced the Coefficient of Variation ($c_v$), meaning the performance gap between simple and complex queries was narrowed. On the SciFact dataset, EqualizeIR achieved an NDCG@10 of 0.70, vastly outperforming the standard DPR model's 0.40 while maintaining much lower performance variance.
- Effective Weak Learners: Among the strategies for creating biased learners, using a weaker model (TinyBERT) and less data proved most effective at capturing a broad range of implicit biases.
Conclusion
EqualizeIR demonstrates that explicitly modeling biased, superficial patterns is a powerful way to build more robust primary models. While the current study focused on lexical and syntactic complexity in dense retrieval models, future work may expand this to discourse and pragmatics. Despite the computational overhead of training two models, the gains in both overall performance and linguistic equity mark a significant step forward for robust information retrieval.
Summary updated