EqualizeIR: Mitigating Linguistic Biases in Retrieval Models

Jiali Cheng, Hadi Amiri

Abstract

This study finds that existing information retrieval (IR) models show significant biases based on the linguistic complexity of input queries, performing well on linguistically simpler (or more complex) queries while underperforming on linguistically more complex (or simpler) queries. To address this issue, we propose EqualizeIR, a framework to mitigate linguistic biases in IR models. EqualizeIR uses a linguistically biased weak learner to capture linguistic biases in IR datasets and then trains a robust model by regularizing and refining its predictions using the biased weak learner. This approach effectively prevents the robust model from overfitting to specific linguistic patterns in data. We propose four approaches for developing linguistically-biased models. Extensive experiments on several datasets show that our method reduces performance disparities across linguistically simple and complex queries, while improving overall retrieval performance.

Introduction

Information Retrieval (IR) models are the backbone of modern search engines. However, their performance is often inconsistent across different query styles. This paper identifies a phenomenon termed "linguistic bias," where models systematically favor or disadvantage queries based on their linguistic complexity—such as vocabulary sophistication or syntactic structure. For instance, a model might perform exceptionally well on simple, keyword-like queries but fail on complex, multi-clause questions, or vice versa. This performance disparity creates an "unlevel playing field" where certain writing styles are systematically disadvantaged.

The EqualizeIR Framework

To address this, we propose EqualizeIR, a framework designed to mitigate linguistic biases by leveraging the concept of a "weak learner." The core idea is to intentionally train a model to capture dataset-specific biases and then use that knowledge to guide the training of a more robust, unbiased model.

Overview of the EqualizeIR framework, showing the two-step training process and strategies for creating biased weak learners.

The framework operates in two distinct phases:

  1. Training a Biased Weak Learner ($f_B$): We intentionally constrain a model to ensure it latches onto superficial linguistic patterns rather than deep semantics. The authors propose four strategies for this: using less training data, employing smaller architectures (e.g., TinyBERT), reducing training time, or using explicitly biased data (e.g., repeating tokens to amplify specific linguistic features).
  2. Training the Robust Model ($f_R$): The main retrieval model is trained with guidance from $f_B$. By comparing the predictions of the robust model with the biased weak learner, we can regularize the learning process. If $f_B$ is highly confident in a correct prediction, it suggests the example is "easy" due to bias, and $f_R$ is encouraged to focus on more difficult signals. Conversely, if $f_B$ makes a confident error, it signals a misleading bias that $f_R$ must learn to ignore.

Experimental Evaluation

We evaluated EqualizeIR using the BEIR benchmark, focusing on four diverse domains: MS MARCO (general), NFCorpus (biomedical), FIQA-2018 (finance), and SciFact (scientific). We quantified query complexity using 45 different lexical and syntactic metrics.

Our analysis revealed that bias is highly context-dependent. For example, the traditional BM25 model shows a bias toward simple queries in medical contexts but favors complex queries in financial data. This highlights the need for a flexible debiasing framework like EqualizeIR that doesn't rely on pre-defined rules or specific linguistic features.

Results

EqualizeIR significantly improves both the absolute performance and the consistency of retrieval models across varying levels of linguistic complexity.

Performance comparison (NDCG@10) across varying linguistic complexity for DPR and EqualizeIR on four benchmark datasets.

As shown in the results, EqualizeIR (applied to a Dense Passage Retrieval base) consistently achieves higher NDCG@10 scores compared to standard baselines. More importantly, the performance curves across linguistic complexity are much flatter for EqualizeIR, indicating a reduction in bias. Quantitatively, the Coefficient of Variation ($c_v$)—a measure of performance disparity—dropped significantly. For instance, on the FIQA dataset, EqualizeIR achieved a $c_v$ of 0.72, compared to 0.82 for standard DPR and 1.26 for BM25.

Conclusion and Future Work

EqualizeIR provides a novel and effective way to ensure equitable performance in IR models by "trapping" and then neutralizing linguistic biases. While highly effective, future work should expand the scope of complexity to include discourse and semantic biases. Additionally, optimizing the training overhead associated with the two-model approach remains a key area for practical deployment. The success of the "Less Capable Model" strategy suggests that architectural constraints are a powerful tool for identifying implicit dataset biases.

Summary updated

Abstract

This study finds that existing information retrieval (IR) models show significant biases based on the linguistic complexity of input queries, performing well on linguistically simpler (or more complex) queries while underperforming on linguistically more complex (or simpler) queries. To address this issue, we propose EqualizeIR, a framework to mitigate linguistic biases in IR models. EqualizeIR uses a linguistically biased weak learner to capture linguistic biases in IR datasets and then trains a robust model by regularizing and refining its predictions using the biased weak learner. This approach effectively prevents the robust model from overfitting to specific linguistic patterns in data. We propose four approaches for developing linguistically-biased models. Extensive experiments on several datasets show that our method reduces performance disparities across linguistically simple and complex queries, while improving overall retrieval performance.
BibTeX
@inproceedings{cheng2025equalizeir,
  author = {Jiali Cheng and Hadi Amiri},
  title = {{EqualizeIR: Mitigating Linguistic Biases in Retrieval Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  url = {https://openreview.net/forum?id=Pu81KRIylA},
}
Figure