Attribute Noise Robust Binary Classification (Student Abstract)

Aditya Petety, Sandhya Tripathi, N. Hemachandra

Abstract

This research investigates the impact of "attribute noise"—errors in input features—on binary classification performance. While label noise is a well-studied phenomenon, the effects of corrupted categorical features are less understood. We analyze the robustness of the standard 0-1 loss and the squared loss surrogate under two distinct noise models: Symmetric Dependent (Sy-De) and Asymmetric Independent (Asy-In). Our findings reveal a significant trade-off: while 0-1 loss is robust to independent noise in low-dimensional spaces, squared loss demonstrates superior robustness to dependent noise, offering a computationally efficient alternative for training on corrupted datasets.

1. Introduction

In machine learning, data corruption typically manifests as either label noise or attribute noise. While significant research has addressed incorrect labels, attribute noise—where the input features themselves are corrupted—poses unique challenges, particularly for categorical data. For binary features, a single bit flip can drastically alter a data point's position in the feature space, potentially leading a classifier to make incorrect predictions. This paper aims to identify loss functions that are "robust" to such noise, meaning a classifier trained on noisy data performs as effectively on clean data as one trained directly on clean data.

2. Methodology

We define two mathematical models for feature corruption:

  1. Symmetric Dependent (Sy-De): All features of a data point flip simultaneously with the same probability, simulating systemic failures such as power surges affecting all sensors in a network.
  2. Asymmetric Independent (Asy-In): Each feature flips independently with its own probability, representing individual component failures.

We analyze two loss functions within the Empirical Risk Minimization (ERM) framework: the non-convex 0-1 Loss ($l_{0-1}$) and the convex Squared Loss ($l_{sq}$) surrogate. The study combines theoretical proofs (Theorems 1 and 2) with empirical validation on real-world datasets.

3. Experimental Setup

The theoretical findings were evaluated using three binary classification datasets from the UCI Machine Learning Repository:

  • Vote: Congressional voting records (232 samples, 16 features).
  • SPECT: Cardiac imaging data (267 samples, 22 features), noted for its class imbalance.
  • KR-vs-KP: Chess endgame positions (3196 samples, 35 features).

4. Results

Our analysis yields several key insights into the relationship between noise structure and loss function robustness.

4.1. Theoretical Robustness

  • Theorem 1 (Sy-De Robustness): We prove that for origin-passing linear classifiers, squared loss is theoretically robust to symmetric dependent noise (for noise rates $p < 0.5$). This is a significant result as squared loss is computationally easier to optimize than 0-1 loss.
  • Theorem 2 (Asy-In Robustness): We demonstrate that 0-1 loss is robust to independent noise specifically within 2-dimensional feature spaces.

4.2. Empirical Validation

Experiments on the UCI datasets confirm these theoretical predictions. Under increasing noise rates, performance metrics such as Average Accuracy and Average AM (Arithmetic Mean of sensitivity and specificity) show predictable degradation patterns.

Average Accuracy and AM across different noise rates for the Vote, SPECT, and KR-vs-KP datasets.

As illustrated in the figure above, the performance of squared loss remains relatively stable or degrades gracefully at low-to-moderate noise levels. For instance, on the KR-VS-KP dataset, accuracy remains above 90% even as noise rates approach 20%. The SPECT dataset, while starting from a lower baseline due to class imbalance, follows a similar trend of gradual degradation.

5. Discussion and Conclusion

This research highlights that robustness to attribute noise is not an inherent property of a loss function alone, but rather emerges from the interaction between the loss function and the specific noise structure. Squared loss emerges as a particularly effective tool for handling dependent attribute noise due to its theoretical robustness and computational efficiency, as it involves solving a linear system rather than an NP-hard optimization problem.

Future work should address the limitations of this study, including extending the 0-1 loss robustness proofs to higher-dimensional data and investigating the robustness of other common surrogates like Hinge or Logistic loss. Additionally, expanding the scope from binary features to general discrete or continuous attributes is a necessary next step for broader practical application.

Summary updated

BibTeX
@inproceedings{petety2020attribute,
  author = {Aditya Petety and Sandhya Tripathi and N. Hemachandra},
  title = {{Attribute Noise Robust Binary Classification (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i10.7221},
}
Figure