The Trade-off between Performance, Efficiency, and Fairness in Adapter Modules for Text Classification

Minh Duc Bui, Katharina von der WenseNAACL 2024

TL;DR

This paper investigates whether parameter-efficient methods like adapters and LoRA, which speed up NLP model training, also impact fairness. They found that while these methods maintain performance and significantly boost efficiency, their effect on fairness is mixed and highly dependent on the base model's existing biases, posing a risk of amplifying existing unfairness.

What Problem Does This Paper Address?

Modern NLP research often focuses on optimizing models for a single objective, like performance or efficiency, sometimes neglecting other crucial aspects like fairness. This "SQUARE ONE" approach can lead to suboptimal outcomes, especially when efficiency-enhancing techniques might inadvertently introduce or exacerbate biases. Specifically, this paper addresses the gap in understanding how parameter-efficient fine-tuning methods, such as adapter modules and LoRA, affect the fairness of NLP models, particularly in sensitive tasks like occupation classification and toxic text detection.

Methods

The researchers compared two main approaches:

  • Full Fine-tuning: Training all parameters of a pre-trained language model (LM).
  • Adapter Modules & LoRA: Parameter-efficient fine-tuning techniques that only update a small subset of parameters. Adapters (Houlsby et al., 2019) insert small trainable modules into the LM architecture, while LoRA (Low-Rank Adaptation) (Hu et al., 2021) injects trainable low-rank matrices into specific layers.

They evaluated these methods across three dimensions: performance (accuracy), efficiency (training time), and fairness (using metrics like Equalized Odds and TPR Gap).

Datasets

The study utilized three text classification datasets:

  • Jigsaw (2019): A large dataset of public comments for toxic text detection, annotated for identity groups like religion, race, and gender.
  • HateXplain (Mathew et al., 2022): A dataset of tweets for toxic text detection, also annotated for identity groups.
  • BIOS (De-Arteaga et al., 2019): A dataset of online biographies used for occupation classification, with annotations for profession and gender.

Key Insights and Ideas

  • Efficiency Gains are Real: Adapter modules and LoRA offer substantial speedups (around 30% per epoch) and reduce training time significantly, confirming prior work.
  • Performance is Largely Preserved: These methods only cause a minor decrease in performance (less than 1% on average), with some cases even showing slight improvements.
  • Fairness is the Wildcard: The impact of adapters and LoRA on fairness is not consistent. It's highly dependent on the pre-existing biases of the base model.
  • Low Bias Base Models are Safer: When the original fine-tuned model has low bias, adapters and LoRA tend to not introduce additional bias and can even reduce variance.
  • High Bias Base Models are Risky: If the base model already exhibits significant bias, adapters and LoRA can lead to unpredictable outcomes, potentially amplifying these biases. This highlights the need for careful, case-by-case evaluation.

Key Results

  • Performance: Adapter modules and LoRA resulted in an average performance decrease of less than 1% across models and tasks. For example, ROBERTa+LoRA on Jigsaw saw a 1.7% decrease, while ROBERTa+Adapters on BIOS showed a slight increase.
  • Efficiency: Training time was reduced by approximately 30% per epoch on average, comparable to previous findings.
  • Fairness:
    • On Jigsaw, adapter modules slightly decreased Equalized Odds (EO) for most models. GPT-2+LoRA showed a notable 2.7% decrease in EO for race.
  • On HateXplain, a steady fairness decrease was observed for religion, with ROBERTalarge+LoRA and ROBERTalarge+Adapters showing the largest drops (4.9% and 3.6%). However, improvements were seen for race and gender.
  • On BIOS, BERT and ROBERTabase with adapter modules showed a strong decrease in fairness, with ROBERTabase+LoRA exhibiting the highest decrease (3.5%).

Limitations and Future Work

  • Scope: The study was limited to text classification tasks and a restricted set of identity groups (religion, race, gender). Future work could explore other tasks and a broader range of identities.
  • Inference Overhead: While adapters improve training efficiency, they do add some computational overhead during inference, which might be a concern for real-time or resource-constrained applications.
  • Model Size: The study did not experiment with the largest and most recent language models (e.g., LLaMA). While the authors believe the main conclusion would hold, further investigation with larger models is warranted.
  • Other Dimensions: The research focused on performance, efficiency, and fairness, but other important dimensions like multilinguality and interpretability were not explored.
  • Abstract Metrics: The reliance on abstract fairness metrics (like EO and TPR Gap) has limitations, and practitioners should be cautious and conduct thorough user impact assessments.
  • Computational Resources: The study utilized approximately 1500 GPU hours, highlighting the environmental impact of such research. The authors aim for efficient resource use.

Summary updated

BibTeX
@inproceedings{bui2024trade,
  author = {Minh Duc Bui and Katharina von der Wense},
  title = {{The Trade-off between Performance, Efficiency, and Fairness in Adapter Modules for Text Classification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure