A time-sensitive learning-to-rank approach for cloud simulation resource prediction

Yuhao Xiao, Yiping Yao, Kai Chen, Wenjie Tang, Feng ZhuComplex 2023 · DOI

Abstract

Predicting the computing resources required by simulation applications can provide a more reasonable resource-allocation scheme for efficient execution. Existing prediction methods based on machine learning, such as classification/regression, typically must accurately predict the runtime of simulation applications and select the optimal computing resource allocation scheme by sorting the length of the simulation runtime. However, the ranking results are easily affected by the simulation runtime prediction accuracy. This study proposes a time-sensitive learning-to-rank (LTR) approach for cloud simulations resource prediction. First, we use the Shapley additive explanation (SHAP) value from the field of explainable artificial intelligence (XAI) to analyze the impact of relevant factors on the simulation runtime and to extract the feature dimensions that significantly affect the simulation runtime. Second, by modifying the target loss function of the rankboost algorithm and training a time-sensitive LTR model based on simulation features, we can accurately predict the computing resource allocation scheme that maximizes the execution efficiency of simulation applications. Compared with the traditional machine learning prediction algorithm, the proposed method can improve the average sorting performance by 3%–48% and can accurately predict the computing resources required for the simulation applications to execute in the shortest amount of time.

1. Introduction

In cloud computing environments, running complex Parallel Discrete Event Simulations (PDES) requires an optimal resource allocation strategy. Assigning insufficient CPU cores leads to slow execution, while over-provisioning can introduce communication overhead that degrades performance. Traditional approaches utilize regression or classification models to predict exact runtimes. However, these models are often brittle; small errors in predicted time can lead to incorrect rankings of resource configurations. This paper introduces a Time-Sensitive Learning-to-Rank (TSLTR) approach that shifts the focus from absolute runtime prediction to the relative ranking of resource schemes, prioritizing the identification of the fastest configuration.

2. Methodology

The proposed TSLTR framework employs a two-stage process to optimize resource prediction, focusing on feature relevance and ranking accuracy.

2.1 Feature Extraction via SHAP

To ensure the model focuses on relevant data, the researchers used Shapley Additive Explanations (SHAP), a tool from Explainable AI (XAI). By analyzing the impact of various factors on simulation runtime, they determined that "Entity" count was the most critical factor, while File I/O had a negligible effect. This allowed for effective feature pruning, focusing the model on the top 16 most significant features.

2.2 Time-Sensitive Learning-to-Rank (TSLTR)

Most Learning-to-Rank (LTR) algorithms treat all ranking errors with equal weight. The authors modified the RankBoost algorithm by introducing a time penalty factor ($\epsilon$). This ensures that if the model fails to correctly identify the optimal (fastest) resource configuration, it receives a significantly higher penalty during training than if it misranks mediocre options. This heuristic-based approach aligns the model's loss function with the practical goal of minimizing execution time.

Overall framework of the proposed TSLTR approach, showing the pipeline from resource monitoring and SHAP-based feature extraction to model building and final resource ranking.

3. Experimental Evaluation

The model was trained and evaluated using two primary simulation environments:

  • Phold: A standard benchmark for parallel simulations where entities randomly exchange messages.
  • PODM (Public Opinion Dissemination Model): A complex system simulation modeling opinion spread across populations.

Data was collected from a real cloud environment using Docker containers on Intel Xeon Gold 6338 CPUs. The TSLTR model was compared against eight popular models, including XGBoost, Random Forest, and standard RankBoost.

4. Results and Discussion

The TSLTR model demonstrated significant improvements over traditional regression-based and standard ranking models.

  • Ranking over Regression: Treating resource allocation as a preference problem makes the model more robust to the inherent noise of cloud environments. Focusing on relative performance (e.g., 8 cores vs. 16 cores) yields better practical results than attempting to predict exact seconds.
  • Superior Accuracy: TSLTR achieved the highest Mean Reciprocal Rank (MRR) and Normalized Discounted Cumulative Gain (NDCG) across both benchmarks, improving average sorting performance by 3% to 48% over traditional methods.
  • Inference Speed: While training takes approximately 116 seconds, the actual ranking prediction takes only 0.55 seconds, enabling real-time scheduling decisions before a simulation starts.

5. Conclusion

The TSLTR approach provides a significant advancement in cloud resource prediction by prioritizing the ranking of optimal configurations. By incorporating time-sensitivity into the loss function and leveraging XAI for feature selection, the method ensures high accuracy and efficiency. Future research will explore the scalability of this approach on larger cloud platforms and the potential for dynamic, on-the-fly resource adjustment.

Summary updated

Abstract

Abstract Predicting the computing resources required by simulation applications can provide a more reasonable resource-allocation scheme for efficient execution. Existing prediction methods based on machine learning, such as classification/regression, typically must accurately predict the runtime of simulation applications and select the optimal computing resource allocation scheme by sorting the length of the simulation runtime. However, the ranking results are easily affected by the simulation runtime prediction accuracy. This study proposes a time-sensitive learning-to-rank (LTR) approach for cloud simulations resource prediction. First, we use the Shapley additive explanation (SHAP) value from the field of explainable artificial intelligence (XAI) to analyze the impact of relevant factors on the simulation runtime and to extract the feature dimensions that significantly affect the simulation runtime. Second, by modifying the target loss function of the rankboost algorithm and training a time-sensitive LTR model based on simulation features, we can accurately predict the computing resource allocation scheme that maximizes the execution efficiency of simulation applications. Compared with the traditional machine learning prediction algorithm, the proposed method can improve the average sorting performance by 3%–48% and can accurately predict the computing resources required for the simulation applications to execute in the shortest amount of time.
BibTeX
@inproceedings{xiao2023time,
  author = {Yuhao Xiao and Yiping Yao and Kai Chen and Wenjie Tang and Feng Zhu},
  title = {{A time-sensitive learning-to-rank approach for cloud simulation resource prediction}},
  booktitle = {Complex},
  year = {2023},
  doi = {10.1007/s40747-023-01045-z},
}
Figure