Label-Noise Robust Logistic Regression and Its Applications

Jakramate Bootkrajang, Ata Kabán

Introduction

A fundamental assumption in supervised learning is the integrity of training labels. Most classification algorithms treat the provided labels as absolute "ground truth." However, in real-world applications—ranging from medical diagnostics where experts may disagree to crowdsourced datasets where workers might be negligent—labels are frequently "noisy." This noise can significantly degrade the performance of standard classifiers.

Presented at ECML PKDD 2012, this research addresses this flaw by proposing a "label-noise robust" version of Logistic Regression (rLR). Instead of treating labels as fixed constants, the authors model the true label as a hidden variable, allowing the system to learn the classifier weights and the noise characteristics simultaneously.

The Robust Logistic Regression Framework

The core innovation of the rLR model is the introduction of a latent variable $y$ representing the true, unobserved label, while $\tilde{y}$ represents the observed, potentially noisy label.

The Gamma Matrix ($\Gamma$)

To bridge the gap between true and observed labels, the model introduces a transition matrix, $\Gamma$. The parameters $\gamma_{jk}$ within this matrix represent the probability that a true label $j$ has been flipped to an observed label $k$. By incorporating these probabilities directly into the likelihood function, the model accounts for the possibility of error during the training process.

Joint Optimization and Convergence

The authors developed a dual-track learning algorithm that alternates between two primary tasks:

  1. Weight Updates: Updating the classifier weights ($w$) using Conjugate Gradients.
  2. Noise Estimation: Updating the flipping probabilities ($\gamma_{jk}$) using a novel multiplicative update rule.

Unlike previous attempts at modeling label noise that lacked robust algorithmic solutions, the authors provide a formal proof that this alternating update scheme is guaranteed to converge to a local optimum. Furthermore, the multiplicative updates are more computationally efficient than the standard Expectation-Maximization (EM) algorithm, as they utilize updated parameter values immediately, reducing memory overhead and accelerating convergence.

Handling High-Dimensionality and Small Samples

In fields like genomics, datasets often feature thousands of variables but very few samples. Standard cross-validation is unreliable in these contexts because the validation labels themselves are likely noisy. To combat this, the authors implemented a Bayesian-regularized Generalized Lasso. This sparsity-promoting regularization allows the model to handle high-dimensional data effectively without relying on potentially flawed validation sets.

Empirical Evaluation

The rLR model was tested across a diverse range of datasets to evaluate its versatility:

  • Synthetic Noise: Using UCI datasets (Boston, Liver, and Iris), the authors injected asymmetric noise to test the model's resilience. rLR consistently outperformed standard Logistic Regression, SVMs, and "Depuration" (a common data-cleaning technique).
  • Biomedical Application: On the Colon Cancer dataset (2000 genes), the model successfully identified eight samples as likely mislabeled. These findings were later corroborated by biological literature, demonstrating the model's utility as an automated "data auditor."
  • Web-Scale Data: In a WebSearch image task ("bikes" vs. "not bikes") where labels were automatically harvested from Google Image Search, rLR achieved a generalization error of 15.67%, a significant improvement over the 18.09% error rate of standard Logistic Regression.

Qualitative Insights: Class Topology

One of the most compelling aspects of the rLR framework is that the $\Gamma$ matrix provides more than just a correction factor; it reveals the underlying relationships between classes. By treating the $\Gamma$ matrix as an adjacency matrix, the authors could visualize a "class topology." For example, in the Newsgroups dataset, the model automatically discovered the proximity between "atheism" and "religion" based on their high confusion rates, effectively "unearthing" the conceptual structure of the data.

Conclusion and Future Directions

The label-noise robust Logistic Regression offers a powerful alternative to traditional data cleaning. By treating noise as a learnable parameter rather than a nuisance to be removed, the model preserves informative data points that might otherwise be discarded.

While the current model is linear and the objective function is non-convex—requiring multiple restarts to ensure a global optimum—it sets a strong foundation for future work. The authors suggest that extending this latent variable framework to non-linear classifiers, such as kernel-based methods or deep neural networks, is the logical next step in developing truly robust machine learning systems.

Summary updated

Abstract

The classical problem of learning a classifier relies on a set of labelled examples, without ever questioning the correctness of the provided label assignments. However, there is an increasing realisation that labelling errors are not uncommon in real situations. In this paper we consider a label-noise robust version of the logistic regression and multinomial logistic regression classifiers and develop the following contributions: (i) We derive efficient multiplicative updates to estimate the label flipping probabilities, and we give a proof of convergence for our algorithm. (ii) We develop a novel sparsity-promoting regularisation approach which allows us to tackle challenging high dimensional noisy settings. (iii) Finally, we throughly evaluate the performance of our approach in synthetic experiments and we demonstrate several real applications including gene expression analysis, class topology discovery and learning from crowdsourcing data.
BibTeX
@inproceedings{bootkrajang2012label,
  author = {Jakramate Bootkrajang and Ata Kabán},
  title = {{Label-Noise Robust Logistic Regression and Its Applications}},
  booktitle = {ECML/PKDD},
  year = {2012},
  doi = {10.1007/978-3-642-33460-3\_15},
}
Figure