Inverse Feature Learning: Feature Learning Based on Representation Learning of Error

Behzad Ghazanfari, Fatemeh Afghah, Mohammad Taghi Hajiaghayi

Abstract

In traditional machine learning, error is typically treated as a scalar penalty to be minimized via loss functions. This paper introduces Inverse Feature Learning (IFL), a novel paradigm where error is reimagined as a rich, high-level representation. By intentionally "mislabeling" data during a trial phase and observing the resulting structural disruptions in class clusters, IFL extracts powerful new features. Experimental results across nine UCI datasets demonstrate that IFL consistently outperforms standard classifiers and remains competitive with state-of-the-art deep learning models, particularly in scenarios with limited data or complex class distributions.

Introduction: When "Error" Becomes the Teacher

In the world of machine learning, "error" is usually something we try to squash. We define a loss function, calculate the difference between a prediction and the truth, and use that single scalar value to nudge our model weights. However, this "scalar view" misses the nuance of why a point might not fit a class.

In their paper "Inverse Feature Learning: Feature learning based on Representation Learning of Error," Ghazanfari et al. propose a paradigm shift. Instead of treating error as a byproduct, they treat it as a feature. By intentionally mislabeling data during a trial phase and observing how it disrupts class structures, they extract a powerful new set of features that help models generalize better and even learn new classes without retraining.

The Inverse Feature Learning Framework

The core of IFL is a three-layer "trial and analysis" framework designed to generate impactful new features that significantly outperform traditional classification methods.

  1. Inner Folding: To generate features for the training set itself, the authors use a partitioning strategy. They treat a portion of the training data as "inner-test" samples to see how they interact with the rest of the "inner-training" data.
  2. Layer 1 (Clustering): The training data is grouped by label. Each group is then clustered (using K-means). This establishes a "baseline" representation for each class, defined by cluster centroids, means, and "confidence" (the ratio of points in a cluster).
  3. Layer 2 (Error Generation): This is the "Trial." A test instance is intentionally assigned to every available class, one by one. The model observes how the class clusters change. Does the new point sit near the center? Does it shift the mean significantly?
  4. Layer 3 (Error Calculation): The model calculates specific metrics of this "error," such as the distance to the closest instance in the class, the shift in cluster confidence, and the distance to cluster centers.
  5. Feature Embedding & Classification: These new "error features" are added to the original data. To keep classes distinct, the features are multiplied by a class ID. Because this creates a large, complex feature space, the authors use Gradient Boosting Decision Trees (GBDT) as the final classifier.
Overview of the Inverse Feature Learning pipeline, illustrating the flow from raw data through inner folding and the three-layer feature extraction process.

Experimental Evaluation

The authors evaluated IFL on nine popular datasets from the UCI Machine Learning Repository, covering both binary and multi-class problems across medical, physical, and technical domains. These datasets were chosen to test the model's ability to handle varying numbers of instances (from 90 to 30,000) and features (from 7 to 57).

Key Results

The experimental results show that IFL consistently outperforms baseline classifiers such as Naive Bayes, KNN, and Decision Trees, and is highly competitive with state-of-the-art deep learning models.

  • Superior Accuracy: On the Segment dataset, IFL achieved 98.07% accuracy, outperforming deep learning baselines like Stacked Auto-Encoders (95.68%) and DrELM (95.30%).
  • Small Data Robustness: On the tiny Cryotherapy dataset (90 instances), IFL hit 96.67% accuracy, a significant jump over the Ensemble baseline (87.91%).
  • Generalization: The added error features acted as a form of regularization, helping the model avoid overfitting even when the original feature set was limited.
Comparison of F1 scores and Accuracy between Inverse Feature Learning and baseline classifiers across nine diverse datasets.

Discussion and Insights

The most profound insight of this work is that the way a data point fails to fit into a class provides critical relational information that standard models—which only look at the point in isolation—completely miss. By pretending a point belongs to Class A, then Class B, and so on, the model extracts features that describe its relationship to the entire data distribution.

Furthermore, IFL supports incremental learning. Because features are learned per-class, a new class can be added to the system by simply clustering its data and running the trial phase, without needing to retrain the representations for existing classes. This human-inspired method mimics how we often analyze the potential results of a choice before making it.

Limitations and Future Work

While powerful, the "trial" phase requires running calculations for every instance against every class, introducing computational overhead. Additionally, the current reliance on K-means assumes spherical clusters; future work may explore more advanced clustering techniques like DBSCAN. The authors hope to extend this "error representation" perspective to computer vision and natural language processing in the future.

Summary updated

Abstract

This paper proposes inverse feature learning as a novel supervised feature learning technique that learns a set of high-level features for classification based on an error representation approach. The key contribution of this method is to learn the representation of error as high-level features, while current representation learning methods interpret error by loss functions which are obtained as a function of differences between the true labels and the predicted ones. One advantage of such learning method is that the learned features for each class are independent of learned features for other classes; therefore, this method can learn simultaneously meaning that it can learn new classes without retraining. Error representation learning can also help with generalization and reduce the chance of over-fitting by adding a set of impactful features to the original data set which capture the relationships between each instance and different classes through an error generation and analysis process. This method can be particularly effective in data sets, where the instances of each class have diverse feature representations or the ones with imbalanced classes. The experimental results show that the proposed method results in significantly better performance compared to the state-of-the-art classification techniques for several popular data sets. We hope this paper can open a new path to utilize the proposed perspective of error representation learning in different feature learning domains.
BibTeX
@article{ghazanfari2020inverse,
  author = {Behzad Ghazanfari and Fatemeh Afghah and Mohammad Taghi Hajiaghayi},
  title = {{Inverse Feature Learning: Feature Learning Based on Representation Learning of Error}},
  journal = {IEEE Access},
  year = {2020},
  doi = {10.1109/access.2020.3009902},
}
Figure