FIB: A Method for Evaluation of Feature Impact Balance in Multi-Dimensional Data

Xavier F. Cadet, Sara Ahmadi-Abhari, Hamed HaddadiarXiv 2022 · DOI · OpenReview

Introduction

Standard evaluation metrics for machine learning models, such as Mean Squared Error (MSE) or Accuracy, primarily quantify the magnitude of failure. However, they remain largely indifferent to the distribution of error across individual features. In multi-dimensional contexts—such as medical diagnostics or robotics—a model that makes small errors across all features may be preferable to one that is perfect on most but fails catastrophically on a single critical dimension. Researchers from Imperial College London have introduced the Feature Impact Balance (FIB) score to address this "distribution-blindness" by quantifying whether a model's errors are concentrated or evenly spread.

The MSE Blind Spot

Consider two models predicting a vector of physiological data. Model A exhibits minor errors across all vitals, while Model B is highly accurate on all but one feature, where it fails significantly. Traditional metrics like MSE might assign both models identical scores if the total error magnitude is the same. This lack of granularity can be problematic in downstream tasks or transfer learning, where a holistic representation of the data is essential. The FIB score provides a mathematically grounded way to measure this "error balance," ranging from 0 (high imbalance) to 1 (perfect balance).

Comparison of MSE and FIB metrics. While MSE aggregates error into a single scalar, FIB quantifies the distribution of error across features.

The FIB Metric

The FIB score calculation follows a four-step pipeline:

  1. Internal Error (IE): Determine the absolute difference between the predicted and ground truth vectors.
  2. Feature Impact (FI): Normalize the error vector so its elements sum to 1, representing the relative contribution of each feature to the total error.
  3. Feature Impact Imbalance (FII): Measure the deviation of this distribution from a perfectly balanced state (where each feature contributes $1/M$).
  4. Normalization: Scale the result to ensure comparability across different datasets and dimensionalities.

Feature Grouping for High-Dimensional Data

A significant challenge in high-dimensional data (e.g., MNIST images with 784 pixels) is that the FIB score naturally tends toward 1.0 because it is statistically improbable for a single feature to carry the entire error. To maintain sensitivity, the authors introduce Feature Grouping. By aggregating features into "buckets" or groups, the metric becomes capable of detecting imbalances at a coarser, more meaningful level.

The effect of feature grouping on FIB sensitivity. Without grouping (a), high-dimensional data often yields near-perfect scores. Grouping (b) reveals the underlying error imbalance.

Experimental Results

The researchers validated the FIB score using the Iris, SARCOS, and MNIST benchmarks, focusing on AutoEncoders (AE) and Variational AutoEncoders (VAE).

  • Training Dynamics: FIB scores are dynamic throughout the learning process. Models often begin with high balance (failing equally at everything), experience a drop as they learn "easy" features, and eventually equilibrate as they tackle more complex dimensions.
  • Sensitivity: On the MNIST dataset, calculating FIB without groups resulted in a misleadingly perfect score of 0.999. Implementing just two groups caused the score to drop by over 0.5, revealing significant error concentration in specific image regions.
  • Transfer Learning: A key finding is the correlation between high FIB scores and downstream performance. Models that distribute errors more evenly tend to develop more robust representations, leading to better performance in subsequent classification or regression tasks.

Conclusion and Future Work

The FIB score serves as a vital complement to traditional intensity-based metrics. While a high FIB score does not guarantee a "good" model (as the total error magnitude could still be high), it provides essential insight into the quality of the learned representation. Future work includes integrating FIB directly into loss functions to encourage balanced learning and applying the metric to multimodal data to detect modality bias.

Summary updated

Abstract

Errors might not have the same consequences depending on the task at hand. Nevertheless, there is limited research investigating the impact of imbalance in the contribution of different features in an error vector. Therefore, we propose the Feature Impact Balance (FIB) score. It measures whether there is a balanced impact of features in the discrepancies between two vectors. We designed the FIB score to lie in [0, 1]. Scores close to 0 indicate that a small number of features contribute to most of the error, and scores close to 1 indicate that most features contribute to the error equally. We experimentally study the FIB on different datasets, using AutoEncoders and Variational AutoEncoders. We show how the feature impact balance varies during training and showcase its usability to support model selection for single output and multi-output tasks.
BibTeX
@misc{cadet2022fib,
  author = {Xavier F. Cadet and Sara Ahmadi-Abhari and Hamed Haddadi},
  title = {{FIB: A Method for Evaluation of Feature Impact Balance in Multi-Dimensional Data}},
  year = {2022},
  eprint = {2207.04500},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2207.04500},
}
Figure