Catching a Baseball: A Reinforcement Learning Perspective Using a Neural Network

Rajarshi Das, Sreerupa DasAAAI 1994

Abstract

The "Outfielder Problem" is a classic puzzle in experimental psychology: how does a baseball player decide where to run to catch a fly ball? This paper explores this problem through Reinforcement Learning (RL), specifically using an Adaptive Heuristic Critic (AHC) model. The study demonstrates that while popular trigonometric hypotheses like "Optical Acceleration Cancellation" provide a strong basis for movement, they are insufficient on their own. By incorporating the ball's perpendicular velocity ($v_p$), a neural network agent successfully learns to catch fly balls in a realistic simulation with aerodynamic drag, exhibiting emergent behaviors similar to those of human athletes.

The Outfielder Problem

When a fly ball is hit, an outfielder must make rapid, continuous decisions about their movement. This presents a "temporal credit assignment" problem: the player receives immediate visual cues but only knows the success of their actions seconds later when the ball is caught or missed.

Geometry of the outfielder problem, showing the ball's trajectory and the elevation angle $\phi$.

A prominent theory in psychology suggests that fielders use "Optical Acceleration Cancellation" (OAC). According to this hypothesis, players run to keep the second derivative of the tangent of the ball's elevation angle ($\phi$) near zero ($d^2(\tan \phi)/dt^2 \approx 0$). If the tangent increases at a constant rate, the fielder is on track to meet the ball. This paper tests whether this cue is computationally sufficient for an agent to learn the task in a complex environment.

Methods: The Adaptive Heuristic Critic

The researchers employed an Adaptive Heuristic Critic (AHC) model, an early form of Actor-Critic architecture. The system consists of two neural networks:

  1. Action Network (The Agent): Determines the fielder's instantaneous acceleration based on current visual inputs.
  2. Evaluation Network (The Critic): Predicts the future reward (utility) of the current state, helping the agent learn from its successes and failures.

The state space included the elevation angle ($\phi$), the second derivative of its tangent ($d^2(\tan \phi)/dt^2$), the fielder's velocity, and a binary indicator of the ball's relative position. Crucially, the authors introduced the ball's perpendicular velocity ($v_p$) as an additional input to resolve visual ambiguities during the final descent. The simulation environment was physically grounded, accounting for aerodynamic drag, which makes the ball's path non-parabolic and the catching task significantly more difficult than in a vacuum.

Key Insights: Beyond Trigonometry

The study revealed that while $d^2(\tan \phi)/dt^2$ is a useful heuristic early in the flight, it becomes extremely volatile as the ball nears the ground. Small deviations in position lead to massive spikes in this value, which can confuse a learning agent.

By adding $v_p$, the agent gained the necessary "discriminating ability" for the final catch phase. Interestingly, the trained network developed human-like strategies. For example, if a fielder starts too close to the batter for a deep fly ball, the agent learns to initially run backward—a common behavior in professional outfielders—before settling under the ball. This suggests that the RL model captured fundamental aspects of human athletic strategy.

Key Results

The performance of the RL agent was evaluated across several models. Model A, which utilized both the trigonometric cue and $v_p$, achieved a high success rate of approximately 45 catches per 50 trials after 10,000 training iterations.

Learning curves for different models: Model A (using $v_p$) significantly outperforms models relying solely on trigonometric cues.

In contrast, models relying solely on $d^2(\tan \phi)/dt^2$ (Model B) or Brancazio’s $d^2\phi/dt^2$ hypothesis (Model C) failed to learn the task effectively. Furthermore, the trained Model A showed impressive generalization; it could catch balls with launch angles (45° to 75°) and ranges (34m to 70m) far outside the narrow parameters used during its training phase.

Limitations and Future Work

While successful, the model is currently restricted to 2D movement (balls hit directly at the fielder). Real-world catching involves 3D tracking and azimuth angles. Additionally, the model assumes the fielder can accurately perceive complex derivatives. Future research could explore the role of stereoscopic vision for depth perception in the final "catch" phase and extend the simulation to full 3D trajectories.

Summary updated

BibTeX
@inproceedings{das1994catching,
  author = {Rajarshi Das and Sreerupa Das},
  title = {{Catching a Baseball: A Reinforcement Learning Perspective Using a Neural Network}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1994},
}
Figure