BiasRV: Uncovering Biased Sentiment Predictions at Runtime

Zhou Yang, Muhammad Hilmi Asyrofi, David Lo

Abstract

Sentiment analysis (SA) systems, though widely applied, have been demonstrated to produce biased results. While existing research focuses on generating test cases to reveal unfairness before deployment, there is a lack of tools for monitoring biased predictions at runtime. This paper introduces BiasRV, the first tool designed to raise alarms when a deployed SA system makes a biased prediction on live input. By dynamically generating gender-discriminatory mutants and applying a high-efficiency two-step heuristic, BiasRV detects biased predictions with significantly reduced computational overhead.

The Challenge of Runtime Fairness

Sentiment analysis models often harbor hidden biases; for instance, a model might label a movie review as "positive" when the protagonist is named "James" but "negative" when the name is changed to "Julissa." While researchers have developed tools to uncover these biases before deployment using pre-defined templates, these methods struggle with the unpredictable nature of real-world user queries. BiasRV addresses this gap by providing a runtime "guardrail" that ensures individual predictions remain fair once a model is live.

BiasRV Architecture and Methodology

BiasRV operates as a wrapper around existing SA systems, treating them as black boxes. The system follows a multi-stage process to verify the fairness of each incoming query.

The BiasRV architecture for runtime bias detection in Sentiment Analysis systems.
  1. Dynamic Template Extraction: BiasRV uses Natural Language Processing (NLP) techniques, such as Named Entity Recognition (NER) and Coreference Resolution, to identify "protected tokens" (names and pronouns) in a live query. It then creates a template by replacing these tokens with placeholders.
  2. Mutant Generation: Using the extracted template, the tool generates "mutants"—alternative versions of the text that differ only in gender information. For example, it might generate 30 versions with female names and 30 with male names.
  3. Distributional Fairness Verification: A prediction is considered fair if the proportion of positive sentiment results for male mutants is nearly equal to the proportion for female mutants (within a threshold $\alpha$, typically 0.10).

Efficiency through Heuristics

Running dozens of extra model inferences for every user query is computationally expensive. To make BiasRV practical for production environments, the authors implemented a two-step heuristic:

  • Step 1 (Sampling): The system samples a small number (e.g., 4) of mutants from each gender. If the SA system predicts the same sentiment for all of them, it is assumed to be unbiased.
  • Step 2 (Full Check): If there is a conflict in the initial sample, BiasRV performs a full check of all generated mutants to calculate the distributional fairness property.

Experimental Results

The researchers evaluated BiasRV using 25,000 IMDB movie reviews and a fine-tuned BERT model. Key findings include:

  • Overhead Reduction: The two-step heuristic reduced the computational overhead of mutant analysis by 73.81% compared to exhaustive checking.
  • Detection Accuracy: Despite the optimization, the heuristic only missed 6.7% of biased predictions.
  • Real-world Utility: In the IMDB test set, BiasRV identified 15 specific instances of clear biased predictions that would have otherwise gone unnoticed.

Limitations and Future Work

BiasRV currently relies on the accuracy of underlying NLP components for NER and coreference resolution. Future work aims to expand the tool's scope beyond binary gender to include race, religion, and occupation-based biases, as well as further optimizing the mutation engine to reduce initial template generation time.

Summary updated

Abstract

Sentiment analysis (SA) systems, though widely applied in many domains, have been demonstrated to produce biased results. Some research works have been done in automatically generating test cases to reveal unfairness in SA systems, but the community still lacks tools that can monitor and uncover biased predictions at runtime. This paper fills this gap by proposing BiasRV, the first tool to raise an alarm when a deployed SA system makes a biased prediction on a given input text. To implement this feature, BiasRV dynamically extracts a template from an input text and from the template generates gender-discriminatory mutants (semantically-equivalent texts that only differ in gender information). Based on popular metrics used to evaluate the overall fairness of an SA system, we define distributional fairness property for an individual prediction of an SA system. This property specifies a requirement that for one piece of text, mutants from different gender classes should be treated similarly as a whole. Verifying the distributional fairness property causes much overhead to the running system. To run more efficiently, BiasRV adopts a two-step heuristic: (1) sampling several mutants from each gender and checking if the system predicts them as of the same sentiment, (2) checking distributional fairness only when sampled mutants have conflicting results. Experiments show that compared to directly checking the distributional fairness property for each input text, our two-step heuristic can decrease overhead used for analyzing mutants by 73.81% while only resulting in 6.7% of biased predictions being missed. Besides, BiasRV can be used conveniently without knowing the implementation of SA systems. Future researchers can easily extend BiasRV to detect more types of bias, e.g. race and occupation.
BibTeX
@inproceedings{yang2021biasrv,
  author = {Zhou Yang and Muhammad Hilmi Asyrofi and David Lo},
  title = {{BiasRV: Uncovering Biased Sentiment Predictions at Runtime}},
  booktitle = {ESEC/SIGSOFT FSE},
  year = {2021},
  doi = {10.1145/3468264.3473117},
}
Figure