Analyzing Reinforcement Learning Algorithms for Nitrogen Fertilizer Management in Simulated Crop Growth

Michael Vogt, Benjamin RosmanEPiC 2022 · DOI

TL;DR

This paper evaluates four reinforcement learning (RL) algorithms to optimize nitrogen fertilizer application in a simulated winter wheat environment. The researchers demonstrate that the Dueling Double Deep Q-Network (Dueling DDQN) successfully learns "cautious" strategies that significantly reduce fertilizer waste while maintaining crop yields comparable to standard farming practices, even across decades of varying weather conditions.

What Problem Does This Paper Address?

Nitrogen fertilizer is a double-edged sword in modern agriculture. While it is essential for boosting crop yields to feed a growing global population, its excessive use is an environmental disaster. Over-application leads to soil acidification, groundwater contamination, and the release of nitrous oxide ($N_2O$)—a greenhouse gas 300 times more potent than $CO_2$.

The challenge lies in the fact that "optimal" fertilization is a moving target. It depends on complex interactions between soil health, crop growth stages, and unpredictable weather. Traditional "standard practice" (applying fixed doses at fixed times) is rigid and often inefficient. However, testing new strategies in the real world is slow, expensive, and risky. This paper explores whether RL agents can navigate these complexities in a simulation to find more sustainable, adaptive management strategies.

Methods

The authors framed fertilizer management as a Markov Decision Process (MDP) and compared four model-free RL architectures:

  • DQN (Deep Q-Network): The baseline deep RL algorithm.
  • Double DQN (DDQN): An evolution of DQN designed to reduce the "overestimation bias" (where the agent thinks an action is better than it actually is).
  • Dueling DDQN: A further refinement that uses two neural streams to separately estimate the value of being in a state and the "advantage" of taking a specific action.
  • PPO (Proximal Policy Optimization): A policy-based actor-critic method known for its stability and data efficiency.

The agents were tasked with deciding every week whether to apply one of seven discrete doses of nitrogen. The reward function was specifically designed to penalize excessive fertilizer use while rewarding high grain yields at the end of the season.

Datasets

The study utilized CropGym, an RL environment built on the LINTUL-3 crop growth model within the Python Crop Simulation Environment (PCSE).

  • Training Data: 25 years of historical weather data from the Netherlands (1983–2016).
  • Evaluation/Testbed: Six specific years (1984, 1994, 2004, 2014, 2019, 2020) were used to test the agents' ability to generalize across different decades and climate conditions.

Key Insights and Ideas

  • The Overestimation Trap: Standard DQN and the "Reactive" baseline (applying fertilizer whenever soil nitrogen dropped) performed poorly. They fell into a trap of applying extreme amounts of fertilizer, overestimating the marginal gain in yield and ignoring the heavy environmental "penalties" in the reward function.
  • Generalization is the Real Test: Weather varies wildly year-to-year. A strategy that works in a wet 1984 might fail in a dry 2014. The paper highlights that an agent's success isn't just about high yield in one year, but its ability to adapt its strategy to the specific needs of the current season.
  • The "Aha" Moment for Dueling DDQN: The Dueling DDQN agent was the only one to realize that in certain years (like 2014), the crop required very little additional nitrogen to reach its maximum potential. By withholding fertilizer in those years, it achieved the highest rewards, proving it had learned a more nuanced understanding of the environment than its competitors.

Key Results

  • Dueling DDQN was the winner: It showed the best generalized strategy, maintaining yields while being the most "cautious" with fertilizer application.
  • Efficiency Gains: The Double DQN and Dueling DDQN agents substantially reduced fertilizer amounts compared to the reactive baseline and standard practice, often achieving "sufficient" yields with a fraction of the chemical input.
  • PPO Performance: While PPO outperformed the basic DQN, it struggled to generalize as effectively as the Dueling DDQN across all test years, particularly in years where very little fertilizer was needed.

Limitations and Future Work

  • Simulated Simplicity: The study relies on the LINTUL-3 model. While robust, simulations cannot perfectly capture the chaotic biological and chemical realities of a physical field.
  • Nutrient Scope: The research focused exclusively on nitrogen. Real-world management requires balancing nitrogen with irrigation, phosphorus, and potassium.
  • Future Directions: The authors suggest moving toward more detailed "process-based" models and exploring multi-agent RL or multi-nutrient optimization to create a more holistic tool for digital agriculture.

Summary updated

Abstract

Establishing intelligent crop management techniques for preserving the soil, while providing next-generational food supply for an increasing population is critical. Nitrogen fertilizer is used in current farming practice as a way of encouraging crop development; however, its excessive use is found to have disastrous and long-lasting effects on the environment. This can be reduced through the optimization of fertilizer application strategies. In this work, we apply a set of reinforcement learning algorithms – the DQN, Double DQN, Dueling DDQN, and PPO – to learn novel strategies for reducing this application in a simulated crop growth setting. We provide an analysis of each agent’s ability and show that the Dueling DDQN agent can learn favourable strategies for minimizing nitrogen fertilizer application amounts, while maintaining a sufficient yield comparable to standard farming practice.
BibTeX
@inproceedings{vogt2022analyzing,
  author = {Michael Vogt and Benjamin Rosman},
  title = {{Analyzing Reinforcement Learning Algorithms for Nitrogen Fertilizer Management in Simulated Crop Growth}},
  booktitle = {EPiC},
  year = {2022},
  doi = {10.29007/1v4x},
}
Figure