Active learning meets fractal decision boundaries: a cautionary tale from the Sitnikov three-body problem

Nicolas Payot, Mario Pasquato, Alessandro A. Trani, Yashar Hezaveh, Laurence Perreault-LevasseurarXiv 2023 · DOI · arXiv

Abstract

Chaotic systems such as the gravitational N-body problem are ubiquitous in astronomy. Machine learning (ML) is increasingly deployed to predict the evolution of such systems, e.g. with the goal of speeding up simulations. Strategies such as active Learning (AL) are a natural choice to optimize ML training. Here we showcase an AL failure when predicting the stability of the Sitnikov three-body problem, the simplest case of N-body problem displaying chaotic behavior. We link this failure to the fractal nature of our classification problem's decision boundary. This is a potential pitfall in optimizing large sets of N-body simulations via AL in the context of star cluster physics, galactic dynamics, or cosmology.

Introduction

In computational astronomy, simulating the gravitational interaction of multiple bodies—the N-body problem—is a computationally intensive task. To mitigate these costs, researchers are increasingly turning to Machine Learning (ML) to predict system stability without the need for exhaustive numerical integration. Given that generating labeled data (i.e., running full simulations) is the primary bottleneck, Active Learning (AL) appears to be an ideal candidate for optimizing training. AL typically improves efficiency by allowing a model to query the most informative data points, often those near the decision boundary where uncertainty is highest.

However, chaotic systems exhibit a sensitive dependence on initial conditions, where infinitesimal changes can lead to radically different outcomes (e.g., stability vs. escape). This sensitivity manifests as a fractal decision boundary. This paper investigates whether the inherent complexity of these boundaries causes AL to fail by "getting lost" in infinite detail, ultimately performing worse than simple random sampling.

Methodology

The study focuses on the Sitnikov problem, a restricted three-body configuration where two primary masses orbit each other while a third, massless body oscillates along the z-axis. The goal is to classify initial conditions $(z, v_z)$ as either "stable" or "unstable."

Experimental Setup

  • Model Architecture: A 7-layer fully-connected neural network designed for classification.
  • Active Learning Strategy: Ranked batch-mode sampling was employed, utilizing a scoring function that balances Exploration (sampling unexplored regions) and Uncertainty Sampling (sampling near the perceived boundary).
  • Control Group: A model trained using simple random sampling (No AL).
  • Ground Truth: Generated using a high-order Runge-Kutta-Fehlberg 7(8) solver with a variable time step to ensure numerical errors remained below $10^{-12}$.

Datasets

The primary dataset consisted of approximately 46,500 Sitnikov simulations. For comparison, a more complex Generalized Three-Body Dataset (~92,800 simulations) involving a star orbiting a binary system with additional parameters (eccentricity, inclination) was also tested to evaluate the universality of the findings.

Results and Discussion

The experimental results reveal a significant performance gap in the Sitnikov problem. The model trained with random sampling (No AL) achieved an accuracy of 89.4%, whereas the AL-trained model reached only 85.9%. For context, a model trained on the entire ground truth dataset reached a "ceiling" of 92.6% accuracy.

Comparison of classification boundaries. The ground truth (top-left) shows the fractal complexity that the AL model (bottom-right) fails to capture as effectively as random sampling (bottom-left).

The Fractal Trap

The failure of AL in this context is attributed to the "Fractal Trap." In standard classification problems, the decision boundary is a relatively smooth manifold. In the Sitnikov problem, however, the boundary is fractal; zooming in only reveals further complexity. Because the AL algorithm prioritizes high-uncertainty regions, it spends its labeling budget on noisy, infinitely complex edges. This leads to two major issues:

  1. Wasted Queries: The model queries points that provide little information about the global structure of the stability regions.
  2. Neglect of the Bulk: By focusing on the messy boundaries, the model fails to adequately sample the broader parameter space, leading to poor generalization compared to random sampling.
Accuracy vs. number of generated samples per epoch. The Active Learning approach (blue) consistently underperforms compared to simple random sampling (green) as the number of samples increases.

Interestingly, AL proved successful for the generalized three-body problem. This suggests that in higher-dimensional spaces or systems where chaos is less dominant, the fractal regions may be sufficiently small to allow AL to function normally. In the "pure" chaos of the Sitnikov geometry, however, the fractal nature dominates the landscape.

Conclusion

This study serves as a cautionary tale for the application of Active Learning in physical systems governed by chaos. While AL is a powerful tool for many ML tasks, it can be "outsmarted" by the infinite complexity of fractal decision boundaries. Future work should focus on developing metrics to quantify the fractality of a problem's boundary a priori, allowing researchers to choose the most robust training strategy. For now, in the realm of chaotic N-body simulations, simple random sampling remains a surprisingly formidable baseline.

Summary updated

Abstract

Chaotic systems such as the gravitational N-body problem are ubiquitous in astronomy. Machine learning (ML) is increasingly deployed to predict the evolution of such systems, e.g. with the goal of speeding up simulations. Strategies such as active Learning (AL) are a natural choice to optimize ML training. Here we showcase an AL failure when predicting the stability of the Sitnikov three-body problem, the simplest case of N-body problem displaying chaotic behavior. We link this failure to the fractal nature of our classification problem's decision boundary. This is a potential pitfall in optimizing large sets of N-body simulations via AL in the context of star cluster physics, galactic dynamics, or cosmology.
BibTeX
@misc{payot2023active,
  author = {Nicolas Payot and Mario Pasquato and Alessandro A. Trani and Yashar Hezaveh and Laurence Perreault-Levasseur},
  title = {{Active learning meets fractal decision boundaries: a cautionary tale from the Sitnikov three-body problem}},
  year = {2023},
  eprint = {2311.18010},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.18010},
}
Figure