Practical guidelines for validation of supervised machine learning models in accelerometer‐based animal behaviour classification

Oakleigh Wilson, David S. Schoeman, Andrew P. Bradley, Christofer J. ClementeJournal of Animal Ecology 2025 · DOI

TL;DR

This paper highlights a critical issue in animal behavior classification using machine learning: widespread improper validation practices that mask model overfitting. The authors propose practical guidelines and a theoretical framework to help researchers ensure their models generalize reliably to new, unseen data, a crucial step for advancing the field.

What Problem Does This Paper Address?

Biologging devices, particularly accelerometers, have revolutionized our understanding of animal behavior by providing rich, continuous data. Machine learning (ML) offers powerful tools to analyze this data and classify fine-scale behaviors. However, a significant challenge in ML, known as overfitting, is prevalent in this field. Overfitting occurs when a model becomes too specialized to the training data, essentially memorizing it rather than learning generalizable patterns. This leads to models that perform exceptionally well on the data they were trained on but fail miserably when applied to new, real-world scenarios.

The problem this paper addresses is the lack of standardized, rigorous validation protocols in accelerometer-based animal behavior classification. Without proper validation, researchers cannot confidently determine if their models are truly effective or if their high performance metrics are simply a result of overfitting. This can lead to misleading conclusions and hinder the reliable application of ML in ecological research.

Methods

The core of this paper is a systematic review of 119 studies that used supervised ML for animal behavior classification from accelerometer data. The authors meticulously analyzed the validation methods employed in these studies. Beyond the review, they provide a theoretical overview of overfitting and propose practical guidelines for optimal validation techniques. These guidelines are grounded in established ML principles but tailored to the specific challenges of time-series data from biologging devices.

Datasets

The paper does not introduce new datasets. Instead, it analyzes the validation practices applied to datasets within the 119 reviewed studies. These datasets consist of accelerometer data collected from various animal species, labeled with specific behaviors. The authors highlight that the quality and representativeness of these labeled datasets are crucial for effective validation.

Key Insights and Ideas

  • Overfitting is rampant and often hidden: The review found that a staggering 79% of studies did not validate their models sufficiently to detect potential overfitting. This means many published results might be overly optimistic.
  • "Gold standard" validation is often overlooked: The ideal validation involves truly independent test sets that mimic real-world application. However, logistical and ethical challenges in animal research often make this difficult, leading to the adoption of less robust methods.
  • Data leakage is a subtle but critical issue: Especially with time-series data, splitting data randomly can lead to "data leakage," where information from the test set inadvertently influences the training process, masking overfitting.
  • Hyperparameter tuning on test sets is a trap: Optimizing model parameters using the test set itself is a form of data leakage that inflates performance estimates. A dedicated validation set or nested cross-validation is crucial.
  • Accuracy alone is insufficient: For imbalanced datasets (common in behavior classification), accuracy can be misleading. Metrics like recall, precision, F1-score, and MCC provide a more nuanced understanding of model performance.
  • Reporting needs standardization: The lack of clear and consistent reporting of validation methods (e.g., data splitting, tuning procedures) makes it difficult to assess the reliability of published findings.

Key Results

The systematic review revealed that:

  • 79% (94 out of 119) of studies did not adequately validate their models to identify potential overfitting.
  • Nearly half of the studies (47%) relied on random data splitting for validation, which is particularly problematic for time-series data.
  • While 70% of studies reported model accuracy above 80%, the review suggests this high accuracy may be inflated due to insufficient validation.
  • Only 25% of studies followed "gold standard" ML validation methods (independent training, validation, and testing sets without model tuning on the test set).

Limitations and Future Work

The authors acknowledge that their review cannot definitively prove overfitting in all cases, as suboptimal validation doesn't automatically mean a model is overfit. However, it strongly suggests that the potential for masked overfitting is high.

Future work should focus on:

  • Developing and adopting standardized reporting guidelines: The authors advocate for adherence to frameworks like the Data Optimisation Model Evaluation (DOME) to ensure transparency and reproducibility.
  • Conducting dedicated studies on optimal validation techniques: More research is needed to determine the most appropriate validation strategies for various animal behavior classification scenarios.
  • Encouraging a critical approach to ML results: Researchers should treat ML models as experiments, rigorously questioning whether reported performance is generalizable and trustworthy, especially when performance metrics are exceptionally high.
  • Developing methods for transferring captive-trained models to free-roaming individuals: This is a significant challenge for expanding ML applications in wild populations.

Summary updated

Abstract

Supervised machine learning has been used to detect fine-scale animal behaviour from accelerometer data, but a standardised protocol for implementing this workflow is currently lacking. As the application of machine learning to ecological problems expands, it is essential to establish technical protocols and validation standards that align with those in other 'big data' fields. Overfitting is a prevalent and often misunderstood challenge in machine learning. Overfit models overly adapt to the training data to memorise specific instances rather than to discern the underlying signal. Associated results can indicate high performance on the training set, yet these models are unlikely to generalise to new data. Overfitting can be detected through rigorous validation using independent test sets. Our systematic review of 119 studies using accelerometer-based supervised machine learning to classify animal behaviour reveals that 79% (94 papers) did not validate their models sufficiently well to robustly identify potential overfitting. Although this does not inherently imply that these models are overfit, the absence of independent test sets limits the interpretability of their results. To address these challenges, we provide a theoretical overview of overfitting in the context of animal accelerometry and propose guidelines for optimal validation techniques. Our aim is to equip ecologists with the tools necessary to adapt general machine learning validation theory to the specific requirements of biologging, facilitating reliable overfitting detection and advancing the field.
BibTeX
@article{wilson2025practical,
  author = {Oakleigh Wilson and David S. Schoeman and Andrew P. Bradley and Christofer J. Clemente},
  title = {{Practical guidelines for validation of supervised machine learning models in accelerometer‐based animal behaviour classification}},
  journal = {Journal of Animal Ecology},
  year = {2025},
  doi = {10.1111/1365-2656.70054},
}
Figure