This paper, presented at IJCAI-07, explores a fundamental question in machine learning: how do we determine if one evaluation metric is superior to another, and can we leverage this to build more effective models? The authors propose a formal framework for constructing "finer" evaluation measures that provide a more granular landscape for optimization algorithms.
The Problem of Metric Coarseness
In machine learning, standard metrics like Accuracy or AUC (Area Under the ROC Curve) are often "coarse." Two distinct models might yield identical accuracy on a test set, even if one provides significantly better probability estimations. This coarseness creates "flat plateaus" in the optimization landscape, making it difficult for algorithms like gradient descent or greedy search to identify the best direction for improvement. The goal is to create measures that are consistent (they do not contradict the original metric) but finer (they can distinguish between models that the original metric sees as identical).
The Two-Level Measure Framework
The core contribution is the Two-Level Measure, denoted as $f:g$.
- The Logic: To compare two models $a$ and $b$, the primary metric $f$ is evaluated first. If $f(a) > f(b)$, then $a$ is better. If $f(a) = f(b)$, the secondary metric $g$ is used as a tie-breaker.
- Formal Properties: The authors mathematically prove that this construction is consistent with $f$ and finer than both $f$ and $g$.
- The Hybrid
AUC:acc: The study specifically focuses on using AUC as the primary metric and Accuracy as the tie-breaker. - Optimization Strategy: The authors modified the training of Artificial Neural Networks (ANNs) to monitor the finer $f:g$ metric rather than just minimizing Mean Squared Error, using it to navigate out of local plateaus during training.
Experimental Evaluation
The approach was validated using 20 real-world datasets from the UCI Machine Learning Repository. These datasets vary significantly in size and complexity, providing a robust testbed for the proposed metrics.
Key Results and Insights
The experimental results confirmed that optimizing for a finer metric during training leads to superior final models.
- Consistency and Discriminancy: Artificial dataset tests confirmed that
AUC:accis 100% consistent with AUC while being significantly more discriminant. - Superior ANN Training: As shown in the detailed results table, the ANN optimized for
AUC:acc($ANN_{AUC:acc}$) consistently outperformed models optimized for standard Accuracy or AUC alone across nearly all datasets. - Win-Loss Record: The summary of head-to-head comparisons highlights the robustness of the approach. When evaluated by Accuracy, the
AUC:acc-optimized model won 11 times and lost only once against the standard Accuracy-optimized model. A similar trend was observed when evaluating by AUC.
Conclusion
By constructing finer evaluation measures like AUC:acc, researchers can provide optimization algorithms with a more informative fitness landscape. This not only captures more "truth" about a model's underlying probability distribution but also results in better final predictive performance. Future work may involve applying this framework to complex evaluation tasks in Information Retrieval and Natural Language Processing.
Summary updated