Modeling and Simultaneously Removing Bias via Adversarial Neural Networks

John Moore, Joel Pfeiffer, Kai Wei, Rishabh K. Iyer, Denis Charles, Ran Gilad-Bachrach, Levi Boyles, Eren ManavogluarXiv 2018 · OpenReview

TL;DR

This paper introduces a novel Adversarial Neural Network (ANN) architecture to tackle the pervasive "feedback loop" bias in machine learning systems, particularly in online advertising. By creating a bias-invariant representation of data, their method effectively models and removes confounding factors like ad position without requiring costly online exploration.

What Problem Does This Paper Address?

In many real-world machine learning applications, especially in online advertising, the predictions made by a deployed model directly influence the data used to train future models. This creates a "feedback loop" where the training data becomes biased towards the model's own predictions. For instance, in paid search, ads shown in higher positions (which tend to have higher click-through rates) dominate the training data. This overrepresentation of successful features makes it difficult to accurately estimate the true potential of less-seen ads or ads in lower positions. Existing solutions often involve complex reinforcement learning frameworks or data re-weighting, which can be computationally expensive or require significant exploration, impacting revenue. The authors aim to address this by developing a method that can learn from this biased data and produce a representation that is invariant to the bias itself.

Methods

The core of the paper is a novel Adversarial Neural Network (ANN) architecture. This ANN comprises four networks:

  • Base Network: Learns a representation of the input features.
  • Prediction Network: Uses the Base Network's output to predict the target variable (e.g., click-through rate).
  • Bias Network: Attempts to predict the confounding bias factor (e.g., ad position) using the Base Network's output.
  • Bypass Network: Linearly models the bias factor and incorporates it into the final prediction.

The key innovation lies in the adversarial training process. The Prediction and Bypass networks are trained to minimize a combined loss function that includes a standard prediction loss and a regularization term penalizing the correlation between the Base Network's output and the bias factor. Simultaneously, the Bias Network is trained to maximize this correlation, effectively trying to predict the bias from the Base Network's representation. This adversarial game forces the Base Network to produce a representation that is uncorrelated with the bias, thus creating a bias-invariant feature representation. A "bypass" structure is also introduced to explicitly model the bias linearly, allowing the model to retain information about the bias while still learning a bias-invariant representation.

Datasets

  • Synthetic Data: The authors generate synthetic datasets to simulate the feedback loop bias present in online advertising systems. This allows for controlled experiments to evaluate their method.
  • Real-world Paid Search Auction Data: The method is evaluated on a large-scale dataset from a major search engine, consisting of up to 500 million training samples. This dataset is described as a "Feedback Loop (FL) dataset."
  • HeldOut (RUS) Dataset: A randomly sampled dataset from the same distribution as the FL data is used for testing to evaluate generalization performance on unbiased data.

Key Insights and Ideas

  • Feedback Loop Bias is Pervasive: The paper highlights how the natural operation of deployed ML systems can inherently create biased training data, a problem that is difficult to overcome with traditional methods.
  • Ad Position as a Proxy for Bias: The authors identify ad display position as a significant confounding factor that summarizes much of the click-through rate bias in online advertising.
  • Adversarial Training for Bias Invariance: The core idea of using adversarial networks to learn a representation that is invariant to a specific confounding factor (bias) is a powerful concept, extending beyond just encryption.
  • Bypass Network for Linear Bias Modeling: Explicitly modeling the bias linearly through a bypass network allows the model to retain useful information about the bias while still achieving bias invariance in the main representation.
  • Trade-off Between Bias Removal and Prediction Accuracy: The paper demonstrates a trade-off where increasing the adversarial component (to remove bias) can sometimes slightly increase error on the biased training data but significantly improves performance on unbiased test data.

Key Results

  • Improved AUC on Unbiased Data: The proposed ANN model achieved significant gains in AUC (up to 19%) on the unbiased RUS testing set compared to a baseline model with no adversarial bias removal (λ=0), while only incurring minor losses on the biased FL training set.
  • Comparable Performance to Naive Methods: In some synthetic evaluations, the ANN's mean squared error on the bias term reached levels comparable to a naive approach that simply predicts the average CTR, indicating effective bias modeling.
  • Effectiveness Across Synthetic and Real-World Data: The method showed consistent improvements across both synthetic simulations of feedback loops and real-world paid search data.
  • Bypass Network Benefits: The inclusion of the bypass network showed slight improvements in AUC and higher MSEs on the RUS dataset, suggesting it helps in better disentangling the bias.

Limitations and Future Work

  • Trade-off Management: While the paper shows a beneficial trade-off, managing this trade-off effectively across different applications and bias levels might require careful tuning of the hyperparameter λ.
  • Generalizability of Bias Factor: The method relies on identifying a specific confounding factor (ad position). Its effectiveness might depend on the ability to accurately identify and quantify such bias factors in other domains.
  • Computational Cost: While avoiding online exploration, adversarial training can still be computationally intensive.
  • Future Work: The authors suggest exploring other confounding factors beyond ad position and applying the ANN architecture to other domains where feedback loop bias is a significant issue. Further investigation into the optimal tuning of λ for different scenarios is also implied.

Summary updated

Abstract

In real world systems, the predictions of deployed Machine Learned models affect the training data available to build subsequent models. This introduces a bias in the training data that needs to be addressed. Existing solutions to this problem attempt to resolve the problem by either casting this in the reinforcement learning framework or by quantifying the bias and re-weighting the loss functions. In this work, we develop a novel Adversarial Neural Network (ANN) model, an alternative approach which creates a representation of the data that is invariant to the bias. We take the Paid Search auction as our working example and ad display position features as the confounding features for this setting. We show the success of this approach empirically on both synthetic data as well as real world paid search auction data from a major search engine.
BibTeX
@misc{moore2018modeling,
  author = {John Moore and Joel Pfeiffer and Kai Wei and Rishabh K. Iyer and Denis Charles and Ran Gilad-Bachrach and Levi Boyles and Eren Manavoglu},
  title = {{Modeling and Simultaneously Removing Bias via Adversarial Neural Networks}},
  year = {2018},
  eprint = {1804.06909},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1804.06909},
}
Figure