When evaluating a machine learning classifier, our first instinct is often to look at accuracy—the proportion of correct predictions out of the total. However, accuracy carries a dangerous assumption: it treats every sample with equal weight. In the real world, data is rarely distributed evenly. If 95% of a dataset consists of a single majority class, a naive model can simply guess that majority class every time, achieving an impressive 95% accuracy while remaining entirely useless.
To build models that truly understand the "long tail" of rare but critical events—such as extreme weather, rare biological structures, or online hate speech—we must decouple our evaluation metrics from the underlying class distribution. Class-balanced evaluation metrics solve this by shifting the unit of equality from the individual sample to the category itself.
The Illusion of Global Aggregates
Standard accuracy and micro-averaged metrics pool all classification decisions together globally. This design choice means that performance on frequent classes inevitably masks poor performance on rare ones.
For example, in environmental modeling, a dataset might be overwhelmingly dominated by "No Drought" instances. As noted by Weiwei Jiang et al. in An evaluation of machine learning and deep learning models for drought prediction using weather data, standard metrics fail to reflect how well a system catches rare, extreme drought events. Similarly, Afshin Rahimi et al. demonstrate in STAVICTA Group Report for RepLab 2014 Reputation Dimension Task that relying on raw accuracy can be deeply misleading when evaluating skewed text categories, as a model can score highly simply by favoring the dominant class.
When the cost of missing a minority instance is high, global aggregates create a false sense of security. To fix this, we must compute performance independently for each class before combining them.
The Mechanism of Macro-Averaging
Macro-averaging is the foundational strategy for enforcing class balance. The core intuition is straightforward: calculate a chosen metric (such as precision, recall, or $F_1$-score) for each class independently, and then take the unweighted arithmetic mean of those scores.
As explained by Anders Johannsen et al. in Disambiguating Explicit Discourse Connectives without Oracles, macro-averaging treats every category as equally important, regardless of its frequency in the corpus. This prevents common categories from dominating the final metric and forces the evaluation to reflect the model's ability to handle rare, long-tail instances.
Consider a dataset with one majority class of 10,000 samples and one minority class of 10 samples. Under macro-averaging, the model's $F_1$-score on the 10 minority samples accounts for exactly 50% of the final score. Ziqian Luo highlights this effect in Deep Learning-Based Strategy for Macromolecules Classification with Imbalanced Data from Cellular Electron Cryotomography, demonstrating that macro-averaged metrics provide a truer reflection of model utility because they prevent high performance on frequent classes from obscuring failures on rare ones.
Mathematical Nuances in Macro Aggregation
While the concept of macro-averaging is simple, its mathematical execution requires care. A common trap in computing the macro $F_1$-score is deciding when to perform the average. One could calculate the macro-average of precision and the macro-average of recall, and then take their harmonic mean. Alternatively, one can calculate the $F_1$-score for each class first, and then take the arithmetic mean of those $F_1$-scores.
The order of operations matters immensely. In Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Study, Joakim Edin et al. show that calculating the macro $F_1$ as the arithmetic mean of individual per-class $F_1$ scores is vital. This approach ensures that the final metric is not artificially suppressed by labels that happen to be absent from a specific evaluation slice, providing a more stable and accurate benchmark for highly unbalanced datasets.
Averaging Precision/Recall First (Flawed):
Per-Class Outputs ──> Macro Precision ──┐
├──> Harmonic Mean ──> Macro F1
Per-Class Outputs ──> Macro Recall ────┘
Averaging Per-Class F1 Scores (Robust):
Per-Class Outputs ──> Per-Class F1s ──> Arithmetic Mean ──> Macro F1
For even stricter evaluation, researchers sometimes move away from additive averages entirely. Varvara Logacheva et al. explore this in Metrics for Evaluation of Word-level Machine Translation Quality Estimation, proposing a multiplicative aggregation ($F_1$-mult) that multiplies the class-specific $F_1$ scores together. This functions as a logical conjunction: because the scores are multiplied rather than added, a model cannot compensate for a terrible minority-class score with a perfect majority-class score. A low score on any single class drags the entire metric down, forcing the model to maintain high precision and recall across all categories simultaneously.
Alternative Formulations of Balance
Depending on the structure of the task, class balance can be enforced through several other mathematical formulations:
- Unweighted Average Recall (UAR) / Balanced Accuracy: Instead of balancing $F_1$ scores, UAR calculates the recall (accuracy within the class) for each category independently and averages them. As noted by De-Chuan Zhan et al. in Aligning model outputs for class imbalanced non-IID federated learning, this arithmetic mean of class-specific accuracies prevents majority classes from masking poor performance on rare classes. Lukas Christ et al. leverage this in This Paper Had the Smartest Reviewers - Flattery Detection Utilising an Audio-Textual Transformer-Based Approach to ensure that models are penalized heavily if they fail to recognize minority emotional expressions, even if they perfectly classify the majority baseline.
- Symmetric Balanced Accuracy (SBA): Traditional metrics can lose their desirable axiomatic properties when scaled from binary to multi-class settings. To address this, the authors of Good Classification Measures and How to Find Them introduce Symmetric Balanced Accuracy, providing a mathematically rigorous framework that prevents multi-class models from gaming the evaluation by over-predicting dominant categories.
- Pairwise Separability ($AUC_\mu$): Area Under the ROC Curve (AUC) is a powerful threshold-independent metric for binary classification, but extending it to multi-class data with severe skew is challenging. In AUCμ: A Performance Metric for Multi-Class Machine Learning Models, Ross Kleiman introduces a multi-class AUC formulation that treats every distinct pair of classes with equal weight, completely independent of their sample sizes. This ensures that the metric remains invariant to data skew and accurately captures how well a model separates confusing class boundaries in the long tail.
By choosing metrics that evaluate performance on a per-class basis rather than a per-sample basis, machine learning practitioners can design and optimize models that remain robust, fair, and effective—even when the data itself is profoundly stacked against the minority.