Using the Pearson’s correlation coefficient as the sole metric to measure the accuracy of quantitative trait prediction: is it sufficient?

Shouhui Pan, Zhongqiang Liu, Yanyun Han, Dongfeng Zhang, Xiangyu Zhao, Jinlong Li, Kaiyi WangFrontiers in Plant Science 2024 · DOI

Abstract

How to evaluate the accuracy of quantitative trait prediction is crucial to choose the best model among several possible choices in plant breeding. Pearson's correlation coefficient (PCC), serving as a metric for quantifying the strength of the linear association between two variables, is widely used to evaluate the accuracy of the quantitative trait prediction models, and generally performs well in most circumstances. However, PCC may not always offer a comprehensive view of predictive accuracy, especially in cases involving nonlinear relationships or complex dependencies in machine learning-based methods. It has been found that many papers on quantitative trait prediction solely use PCC as a single metric to evaluate the accuracy of their models, which is insufficient and limited from a formal perspective. This study addresses this crucial issue by presenting a typical example and conducting a comparative analysis of PCC and nine other evaluation metrics using four traditional methods and four machine learning-based methods, thereby contributing to the improvement of practical applicability and reliability of plant quantitative trait prediction models. It is recommended to employ PCC in conjunction with other evaluation metrics in a targeted manner based on specific application scenarios to reduce the likelihood of drawing misleading conclusions.

Introduction

In the field of plant breeding, quantitative trait prediction—estimating complex traits like yield or height from genomic and environmental data—is essential for selecting superior genotypes. For decades, the Pearson Correlation Coefficient (PCC) has been the primary metric for assessing model accuracy. However, PCC only measures the linear association between predicted and observed values. It is scale-invariant, meaning it captures trends but ignores absolute error and bias. As machine learning (ML) and deep learning models become more prevalent, their non-linear nature exacerbates the limitations of PCC, potentially leading to the selection of models that appear accurate in correlation but fail significantly in absolute prediction.

Methods

The researchers conducted a multi-faceted evaluation:

  1. Simulated Stress Test: A controlled scenario with 10 data points and four hypothetical models was used to visualize how high PCC can mask high absolute error.
  2. Empirical Benchmarking: Eight prediction methods were tested, including four traditional statistical models (e.g., rrBLUP, Bayesian LASSO) and four ML methods (e.g., Random Forest, Deep Neural Networks).
  3. Multi-Metric Comparison: Models were evaluated using ten metrics: PCC, error-based metrics (MAE, MSE, RMSE), fit-based metrics ($R^2$), ranking-based metrics (SRCC, NDCG), breeding-specific "hit rates" (THR@P%, BHR@P%), and a proposed hybrid metric (CICE).
  4. Datasets: Seven diverse real-world breeding datasets were used, covering rice (IRRI), wheat (Wheat487, Wheat599), maize (G2F_2017, USNAM, CNGWAS), and millet (Millet827).

Results

The simulated stress test clearly demonstrates the "correlation trap." While Model A shows a strong alignment with the diagonal (low error), Model B maintains a higher correlation despite a significant shift in scale.

Comparison of prediction accuracy metrics for Model A (PCC=0.8345, MAE=1.28) and Model B (PCC=0.8785, MAE=11.4).

The most extreme case is seen in Model D, which achieves the highest PCC (0.9229) among the simulated set but also the highest Mean Absolute Error (MAE=26.6). This highlights how a model can perfectly capture a trend while being practically useless for absolute value prediction.

Model D demonstrates a high PCC (0.9229) but a very high MAE (26.6), illustrating the scale-invariance trap of correlation metrics.

Empirical results from the IRRI rice dataset mirrored these findings. A Deep Learning model (DNNGP) achieved a superior PCC of 0.351 compared to traditional models, yet it was the worst-performing model when evaluated by $R^2$ and MAE. Furthermore, models with high PCC did not consistently yield the best "hit rates" (THR@10%), which is the primary concern for breeders selecting the top performers.

Discussion and Conclusion

PCC is a valuable but incomplete metric. Its scale-invariance makes it a poor proxy for absolute accuracy, especially in ML models that may capture rankings well but fail on scale. The study suggests that breeders should adopt a targeted evaluation strategy, pairing PCC with error-based metrics (like MAE) and ranking-based metrics (like NDCG) depending on the specific breeding objective. Relying solely on PCC risks drawing misleading conclusions and selecting models that may underperform in real-world field applications.

Summary updated

Abstract

How to evaluate the accuracy of quantitative trait prediction is crucial to choose the best model among several possible choices in plant breeding. Pearson's correlation coefficient (PCC), serving as a metric for quantifying the strength of the linear association between two variables, is widely used to evaluate the accuracy of the quantitative trait prediction models, and generally performs well in most circumstances. However, PCC may not always offer a comprehensive view of predictive accuracy, especially in cases involving nonlinear relationships or complex dependencies in machine learning-based methods. It has been found that many papers on quantitative trait prediction solely use PCC as a single metric to evaluate the accuracy of their models, which is insufficient and limited from a formal perspective. This study addresses this crucial issue by presenting a typical example and conducting a comparative analysis of PCC and nine other evaluation metrics using four traditional methods and four machine learning-based methods, thereby contributing to the improvement of practical applicability and reliability of plant quantitative trait prediction models. It is recommended to employ PCC in conjunction with other evaluation metrics in a targeted manner based on specific application scenarios to reduce the likelihood of drawing misleading conclusions.
BibTeX
@inproceedings{pan2024using,
  author = {Shouhui Pan and Zhongqiang Liu and Yanyun Han and Dongfeng Zhang and Xiangyu Zhao and Jinlong Li and Kaiyi Wang},
  title = {{Using the Pearson’s correlation coefficient as the sole metric to measure the accuracy of quantitative trait prediction: is it sufficient?}},
  booktitle = {Frontiers in Plant Science},
  year = {2024},
  doi = {10.3389/fpls.2024.1480463},
}
Figure