How the Selection of Training Data and Modeling Approach Affects the Estimation of Ammonia Emissions from a Naturally Ventilated Dairy Barn—Classical Statistics versus Machine Learning

Sabrina Hempel, Julian Adolphs, Niels Landwehr, David Janke, Thomas AmonSustainability 2020 · DOI

TL;DR

This study investigates how different data sampling strategies and machine learning models impact the accuracy of estimating ammonia emissions from dairy barns. The research demonstrates that advanced machine learning techniques, particularly gradient boosting and random forests, significantly outperform traditional statistical methods, especially when combined with well-designed temporal sampling of measurement data.

What Problem Does This Paper Address?

Accurately quantifying ammonia emissions from agricultural sources, particularly livestock operations, is crucial for environmental protection efforts, including mitigating climate change and reducing air pollution. Naturally ventilated dairy barns, a common housing system, present a challenge for emission monitoring due to their open structure and the difficulty in precisely measuring air exchange rates. Current methods often rely on intermittent measurements and statistical extrapolation, which can introduce significant uncertainties. The paper aims to address this by systematically evaluating the effectiveness of various temporal sampling strategies and comparing classical statistical approaches with modern machine learning techniques for estimating these emissions. The goal is to identify optimal measurement and modeling strategies that provide reliable and accurate emission estimates.

Methods

The researchers employed a comprehensive approach to compare different modeling strategies and data sampling techniques. They analyzed nearly 40 weeks of hourly ammonia emission data from a naturally ventilated dairy barn.

Regression Approaches: Eight different regression methods were tested, including:

  • Classical Statistics: Ordinary Multilinear Regression (LinReg) and Ridge Regression (Ridge) for linear modeling with and without regularization.
  • Machine Learning:
    • Ensemble Methods: Gradient Boosting (GradBoost) and Random Forests (RandFor), both based on decision trees.
  • Artificial Neural Networks (ANN): Including a simple ANN with a fixed architecture (fixAnn) and more complex ANNs with tunable hyperparameters.
  • Other ML Models: Support Vector Regression (SVM) and Gaussian Processes (GausProc).

Temporal Sampling Scenarios: The study simulated 27 different scenarios for collecting training data. These scenarios varied in the number of measurement periods (3, 4, or 6), the duration of each period (1, 7, or 14 days), and the distribution of these periods across seasons (summer, winter, and transition).

Evaluation Protocol: A nested cross-validation protocol was used. An outer loop estimated the generalization error (extrapolation error), while an inner loop optimized model hyperparameters. Four evaluation measures were used: Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), R-squared (R²), and Total Absolute Error (TAE).

Datasets

The study utilized a dataset of nearly 40 weeks of hourly ammonia emission values collected from a naturally ventilated dairy barn in Mecklenburg-Western Pomerania, Germany. This dataset was collected from November 2016 to September 2017. The paper does not introduce a new benchmark dataset but rather uses this existing, rich dataset for its comprehensive analysis.

Key Insights and Ideas

  • Machine Learning Dominance: Machine learning models, particularly gradient boosting and random forests, consistently outperformed traditional linear regression methods in accurately predicting ammonia emissions. This highlights the power of these algorithms in capturing complex, non-linear relationships in environmental data.
  • Temporal Sampling is Crucial: The way measurement data is sampled throughout the year significantly impacts prediction accuracy. Simply having more data points isn't enough; their distribution across seasons and the duration of each sampling period are critical.
  • Robustness of Ensemble Methods: Gradient boosting and random forests proved to be more robust to variations in sampling strategies and less sensitive to outliers compared to linear regression.
  • Hyperparameter Tuning Matters: For models like ANNs and Ridge Regression, proper hyperparameter tuning was essential for achieving good performance.
  • TAE vs. Other Metrics: The "Total Absolute Error" (TAE), which focuses on the aggregated annual emission, revealed different optimal sampling strategies compared to metrics like MAE, RMSE, or R², which assess hourly prediction accuracy. This suggests that the choice of evaluation metric should align with the specific goal of the emission estimation (e.g., annual inventory vs. real-time control).

Key Results

  • Error Below 20%: On average, the tested measurement protocols and models achieved prediction errors well below 20% for the estimated emission values.
  • Best Performing Models: Gradient boosting and random forests provided the most accurate and robust emission value predictions, with the second-smallest model errors.
  • Optimal Sampling Scenario: The scenario with the best overall performance involved four measurement periods, each lasting 14 days, with one period in summer and three in the transition periods.
  • Minimum Sampling Duration: At least seven days of hourly measurements per period were advisable for robust predictions.
  • Number of Periods: Scenarios with four or six measurement periods consistently outperformed those with only three.
  • Linear Regression Limitations: Ordinary multilinear regression, especially with short (one-day) measurement periods, often failed to project reasonable emission values and was more sensitive to noise. Regularization improved its performance but sometimes led to underestimation.
  • Gaussian Process Variance: While Gaussian processes showed the lowest variance in predicted emissions and low test/extrapolation errors, their predicted mean values were systematically too low.

Limitations and Future Work

  • Single Barn Study: The study was conducted on a single naturally ventilated dairy barn. While the findings are likely generalizable, further studies on different barn types and management practices would strengthen the conclusions.
  • Data Availability: The study relied on a dataset of nearly 40 weeks of continuous hourly measurements, which is often not feasible in practice. The simulated scenarios aimed to address this, but real-world constraints might differ.
  • Focus on Ammonia: The study specifically focused on ammonia emissions. Future work could extend this analysis to other relevant pollutants like methane or nitrous oxide.
  • Further Temporal Sampling Studies: The authors suggest that more in-depth studies on the effects of temporal sampling are advisable to better understand trends in model accuracy when changing the number and duration of measurement periods. This could involve exploring more complex sampling patterns or adaptive sampling strategies.

Summary updated

Abstract

Environmental protection efforts can only be effective in the long term with a reliable quantification of pollutant gas emissions as a first step to mitigation. Measurement and analysis strategies must permit the accurate extrapolation of emission values. We systematically analyzed the added value of applying modern machine learning methods in the process of monitoring emissions from naturally ventilated livestock buildings to the atmosphere. We considered almost 40 weeks of hourly emission values from a naturally ventilated dairy cattle barn in Northern Germany. We compared model predictions using 27 different scenarios of temporal sampling, multiple measures of model accuracy, and eight different regression approaches. The error of the predicted emission values with the tested measurement protocols was, on average, well below 20%. The sensitivity of the prediction to the selected training dataset was worse for the ordinary multilinear regression. Gradient boosting and random forests provided the most accurate and robust emission value predictions, accompanied by the second-smallest model errors. Most of the highly ranked scenarios involved six measurement periods, while the scenario with the best overall performance was: One measurement period in summer and three in the transition periods, each lasting for 14 days.
BibTeX
@inproceedings{hempel2020how,
  author = {Sabrina Hempel and Julian Adolphs and Niels Landwehr and David Janke and Thomas Amon},
  title = {{How the Selection of Training Data and Modeling Approach Affects the Estimation of Ammonia Emissions from a Naturally Ventilated Dairy Barn—Classical Statistics versus Machine Learning}},
  booktitle = {Sustainability},
  year = {2020},
  doi = {10.3390/su12031030},
}

Loading related content...

Figure