Inverse Feature Learning: Feature learning based on Representation Learning of Error

Behzad Ghazanfari, Fatemeh Afghah, MohammadTaghi Hajiaghayi

TL;DR

Inverse Feature Learning (IFL) reimagines classification error not as a scalar loss value used strictly for backpropagation, but as a rich, high-level feature space. By intentionally assigning data points to incorrect classes during a "trial" phase and measuring the resulting disruption to cluster structures, IFL generates powerful, class-independent features. This approach significantly boosts downstream classification accuracy and allows models to learn new classes incrementally without full retraining.

What Problem Does This Paper Address?

In traditional supervised learning, "error" is treated as a byproduct—a single scalar value calculated via a loss function to tell a network how far off its predictions are. This scalar is used to update weights and is then discarded. The authors argue that this conventional approach throws away a goldmine of structural information about how an instance relates to different classes.

Furthermore, standard deep learning architectures suffer from a lack of modularity: features are deeply entangled across classes. If you want to add a new class to a trained model, you usually have to retrain the entire network from scratch to prevent "catastrophic forgetting."

This paper introduces Inverse Feature Learning (IFL) to address these gaps by turning the generation and analysis of error into a proactive feature engineering tool.

Methods

The core philosophy of IFL is inspired by human decision-making: evaluating a choice by simulating its consequences. The framework operates in three distinct layers, wrapped in an "inner folding" mechanism (similar to cross-validation) to prevent data leakage during training:

  1. Layer 1 (Clustering & Baselines): Training samples are grouped by their true labels. The algorithm applies K-means clustering to each group to establish a baseline representation of each class, extracting metrics like cluster centroids, cluster "confidence" (density), and cluster means.
  2. Layer 2 (Error Generation): A target sample is intentionally assigned to every available class, one by one—even the wrong ones. The algorithm then observes how the addition of this sample alters the established clusters.
  3. Layer 3 (Error Calculation): The model quantifies the disruption caused by the sample. It calculates two feature sets: the distance of the sample to the existing clusters (Feature Set 1) and the structural changes in the clusters before vs. after the sample was added (Feature Set 2).

These calculated "error representations" are multiplied by a unique class ID to keep them distinct and are appended to the original data features. Finally, an ensemble of boosting decision trees uses this expanded feature set for final classification.

Fig. 1: Block diagram of the proposed inverse feature learning method. The figure demonstrates the feature learning process for both training and test sets. The upper part of the figure depicts the inner-folding for the training set during r rounds. The one fold of the inner-test in each run is highlighted by green. The lower part of the figure demonstrates the feature learning process for the test data set.

Datasets

The authors evaluated IFL using nine popular binary and multi-class benchmarks from the UCI Machine Learning Repository:

  • Cryotherapy (90 instances, 7 features)
  • Diabetes (768 instances, 8 features)
  • Heart (270 instances, 13 features)
  • Credit (30,000 instances, 23 features)
  • Segment (1,500 instances, 19 features)
  • Ionosphere (351 instances, 34 features)
  • Glass (214 instances, 9 features)
  • Spam (4,601 instances, 57 features)
  • Magic (19,020 instances, 10 features)

No new benchmarks were introduced, allowing for direct comparison against established baselines.

Key Insights and Ideas

  • Error as a Signature: The "aha!" moment of the paper is that putting a sample into the wrong class creates a unique, predictable "error signature." By mapping these signatures, the model learns the complex topological boundaries between classes far better than it could by looking at the raw data alone.
  • Retraining-Free Class Addition: Because IFL extracts features on a per-class basis through independent clustering, the features learned for Class A do not depend on Class B. To add a new class, you only need to cluster the new class's data and run the trial phase. The existing feature representations remain untouched, enabling true incremental learning.

Key Results

The authors compared IFL against standard classifiers (Naive Bayes, KNN, Decision Trees, and standard Ensembles) as well as state-of-the-art deep representation learning methods (such as Stacked Autoencoders and Deep Belief Networks).

  • Outperforming Baselines: IFL consistently outperformed all baseline classifiers across all nine datasets in both F1-score and Accuracy. For instance, on the Cryotherapy dataset, IFL achieved an accuracy of 96.67%, compared to the standard ensemble baseline of 87.91%.
  • Beating Deep Learning: When stacked against deep learning models on the Segment dataset, IFL achieved an accuracy of 98.07%, outperforming the best deep learning baseline, $DrELM^r$, which scored 95.79%. It also remained highly competitive on the Diabetes dataset.
TABLE II: The comparison of accuracy and F1 score of the baseline classifiers and the inverse feature learning. We have evaluated the performance of data sets with several popular classifiers libraries in MATLAB and sklearn including SVMs and logistic regressing but only the best results among different classifiers are reported. In the proposed method, the learned features are independent of the classifier, hence using a better classifier can achieve higher accuracy over our learned features as it does over other sets of features.

Limitations and Future Work

The primary limitation of the current implementation lies in its reliance on K-means clustering. K-means assumes clusters are spherical and of similar density, which isn't always true for complex real-world data.

While the authors experimented with more advanced techniques like spectral clustering (e.g., DBSCAN)—which yielded even higher accuracy—these methods rely on a graph Laplacian matrix. This introduces a massive memory overhead, making the current framework unscalable for massive, large-scale datasets. Future work will focus on integrating more scalable, non-spherical clustering algorithms into the IFL pipeline.

Summary updated

Abstract

This paper proposes inverse feature learning as a novel supervised feature learning technique that learns a set of high-level features for classification based on an error representation approach. The key contribution of this method is to learn the representation of error as high-level features, while current representation learning methods interpret error by loss functions which are obtained as a function of differences between the true labels and the predicted ones. One advantage of such learning method is that the learned features for each class are independent of learned features for other classes; therefore, this method can learn simultaneously meaning that it can learn new classes without retraining. Error representation learning can also help with generalization and reduce the chance of over-fitting by adding a set of impactful features to the original data set which capture the relationships between each instance and different classes through an error generation and analysis process. This method can be particularly effective in data sets, where the instances of each class have diverse feature representations or the ones with imbalanced classes. The experimental results show that the proposed method results in significantly better performance compared to the state-of-the-art classification techniques for several popular data sets. We hope this paper can open a new path to utilize the proposed perspective of error representation learning in different feature learning domains.
BibTeX
@article{ghazanfari2020inverse,
  author = {Behzad Ghazanfari and Fatemeh Afghah and MohammadTaghi Hajiaghayi},
  title = {{Inverse Feature Learning: Feature learning based on Representation Learning of Error}},
  journal = {IEEE Access},
  year = {2020},
  doi = {10.1109/access.2020.3009902},
}
Figure