Abstract
Managing petabytes of physics data at CERN requires a delicate balance between expensive, high-speed disk storage and cost-effective but high-latency magnetic tapes. This paper presents a machine learning framework designed to predict future dataset popularity for the LHCb experiment. By implementing a three-stage pipeline—comprising popularity estimation, intensity prediction, and placement optimization—the system achieves up to a 40% reduction in disk space requirements while simultaneously decreasing data access wait times by 30%. This approach addresses the limitations of traditional Least Recently Used (LRU) caching in high-energy physics environments characterized by sparse and bursty data access patterns.
Introduction
The LHCb experiment at the Large Hadron Collider generates a staggering volume of data, currently managing approximately 10.5 PB on disk and 1.5 PB on tape. While magnetic tape is essential for long-term archiving due to its low cost, it is significantly slower than disk storage, creating a bottleneck for active analysis. Conversely, disk space is fast but limited and expensive.
The core challenge is a massive-scale cache management problem: determining which datasets should remain on "hot" disk storage and which can be safely relegated to "cold" tape archives. Traditional methods like Least Recently Used (LRU) often fail in high-energy physics because data usage is typically "bursty" and sparse. In such systems, the penalty for a cache miss—accidentally deleting a dataset from disk that is needed shortly after—is severe, requiring a time-consuming restoration from tape.
Methodology
The proposed framework automates storage decisions through a three-module pipeline:
1. Data Popularity Estimator
This supervised learning module utilizes a Gradient Boosting Classifier to predict the probability that a dataset will be accessed in the near future. The model incorporates metadata (file type, size, creation date) and the temporal "shape" of its usage history, such as the time elapsed since the last access and the frequency of historical access peaks.
2. Data Intensity Predictor
While the first module determines if a dataset will be used, this module forecasts how much it will be used. Standard neural networks often overfit on the sparse usage data typical of LHCb. To counter this, the authors employ non-parametric models, specifically Nadaraya-Watson kernel smoothing and rolling mean values, to generate robust forecasts of future access intensity.
3. Data Placement Optimizer
The final module integrates predictions into a loss function that balances the cost of disk storage, the cost of tape storage, and the penalty of a cache miss. This optimization determines the ideal number of replicas for each dataset. The logic follows a "Square Root Rule," where the optimal number of replicas ($Rp$) scales with the square root of the predicted usage intensity ($I$), adjusted by a cost factor ($\alpha$).
Datasets and Evaluation
The study was conducted using real-world data from the LHCb experiment, including:
- Usage History: Weekly access counters collected over a two-year period (104 data points per dataset).
- Metadata: Information from the LHCb file catalogue, including detector configuration, event type (Monte Carlo vs. real data), and file size.
- Evaluation Set: A subset of 7,375 datasets was used to benchmark the ML-based algorithm against the baseline LRU approach.
Key Insights
- Sparse Data Strategy: The authors found that complex models like Artificial Neural Networks (ANNs) were prone to overfitting due to low statistics. Non-parametric smoothing and Gradient Boosting provided more reliable results for this specific domain.
- The "Unpopularity" Metric: The model effectively identifies datasets with a high probability of remaining unused, allowing the system to confidently evict them to make room for new data without increasing cache misses.
- Mathematical Grounding: The derivation of the Square Root Rule for replicas provides a rigorous foundation for scaling storage resources based on predicted demand.
Results
The ML-based algorithm significantly outperformed the standard LRU baseline across all key metrics:
- Accuracy: LRU frequently resulted in thousands of "wrong removals" (deleting data required shortly after). In contrast, the ML algorithm reduced these errors to single digits (e.g., 8 or 9 errors across the entire test set).
- Efficiency: By maintaining a maximum of 7 replicas for the most popular data, the algorithm saved 40% of disk space while decreasing download times by 30%.
- Latency: The ML approach consistently maintained a download time ratio below 1.0, indicating superior speed compared to the existing manual and LRU-based systems.
Conclusion
This research demonstrates that machine learning can significantly optimize large-scale scientific data management. By moving beyond simple recency-based heuristics to predictive models of popularity and intensity, the LHCb experiment can achieve substantial infrastructure savings and improved researcher productivity. Future work will focus on tuning cost parameters as hardware prices evolve and integrating the Python-based implementation directly into the LHCb grid production environment.
Summary updated