Requirements for numeric models as sources of synthetic data for predicting real-world data sets

Markus Schumann, Jonas Moske, Antonia Wüst, Kristian Kersting, Peter GrocheResearch Square 2025 · DOI

Abstract

In the field of forming technology, synthetic data generated by finite element (FE) simulations is increasingly being used to train machine learning (ML) models for quality prediction. However, the predictive accuracy on real-world process data is often limited by the so-called “reality gap” between simulated and measured signals. This study investigates how simulation model complexity influences the suitability of synthetic data for training ML models that generalize to real progressive deep drawing processes. Three representative simulation configurations of increasing complexity (L1–L3) are implemented and evaluated against experimental data collected under production conditions using sensor-integrated tools. The analysis covers multiple ML tasks, including classification of pre-process connector cut geometries, detection of process disturbances, separation of subtle geometry variants, assessment of feature transfer robustness, and saliency-based interpretation of signal regions. The results show that simple models (L1) enable robust classification of failures such as material damage within synthetic domains though their transferability to real data is restricted. Although they are computationally more expensive, higher complexity levels (L2 and L3) better capture the effects of pre-processing and deformation-history, improving domain alignment and supporting physically meaningful model interpretation. Saliency map analysis reveals that models trained on synthetic data emphasize different signal regions than models trained on real data. This underscores the importance of task-relevant signal fidelity. The findings provide quantitative guidance for selecting an adequate level of simulation complexity and demonstrate that the underlying principles apply across a broader spectrum of modelling configurations.

Introduction

In industrial manufacturing, particularly progressive deep drawing, collecting large-scale experimental datasets for training AI models is often prohibitively expensive. Synthetic data generated via Finite Element (FE) simulations offers a solution, but it suffers from a "reality gap"—the discrepancy between idealized simulations and noisy real-world sensor data. This paper provides a systematic framework to determine the level of simulation complexity required for specific machine learning tasks, balancing computational cost with predictive accuracy.

The Manufacturing Process

The study focuses on a progressive deep drawing process, which involves a series of sequential operations on a sheet metal strip.

The sequence of operations in the progressive deep drawing process, from pilot punching to the final deep-drawn part.

As shown in the sequence above, the process begins with pilot punching and proceeds through a connector cut, which defines the initial geometry for the subsequent deep drawing stage. The final part is then cut out from the strip. Understanding the deformation history from these early stages is crucial for accurate quality prediction.

Methodology: Simulation Complexity Levels

Three levels of FE simulation complexity were implemented to generate training data:

  • Level 1 (L1): A basic approach using rigid bodies and linear press movement, omitting the return stroke (20h computation time).
  • Level 2 (L2): Adds realism with sinusoidal press kinematics, material anisotropy, and the inclusion of the return stroke (48h computation time).
  • Level 3 (L3): A high-fidelity "digital twin" approach that includes deformable bodies and simulates the pre-process connector cut to account for deformation history (68h computation time).

ML models, including a 3-layer CNN and various tree-based classifiers, were trained on this synthetic data and tested against 3,200 real-world samples from a Bruderer BSTA 810 press equipped with piezoelectric sensors.

Bridging the Reality Gap

The primary challenge in using synthetic data is ensuring that the features learned by the model in the virtual domain transfer effectively to the physical domain.

A comparison of synthetic data generation versus the real-world process, highlighting the 'reality gap' in force curves and classification performance.

As illustrated, the force curves generated by simulations often differ from experimental measurements. The study found that while simple L1 simulations are highly effective for binary damage detection (achieving zero false negatives with expert features), they struggle with more nuanced tasks like geometry classification.

Key Insights and Results

  • Task-Relevant Fidelity: The researchers found that simulation complexity should not be maximized by default. Instead, "task-relevant signal fidelity" is key. For example, L2-trained models reached a peak of ~45.5% accuracy on the 4-class geometry problem, outperforming both the simpler L1 and the more complex L3 models.
  • The L3 Overfitting Trap: The added complexity of L3 sometimes introduced specific signal patterns that caused models to "overfit" to simulation-specific artifacts (shortcut learning) that were absent in real-world data.
  • Saliency and Interpretation: Saliency map analysis showed that models trained on real data focus heavily on the backstroke region of the force signal. Simulations that fail to accurately model tool elasticity or material recovery during this phase lead to a mismatch in model reasoning.

Conclusion

This study provides quantitative guidance for selecting simulation complexity in industrial ML applications. While low-cost models are sufficient for robust failure detection, higher-fidelity simulations are essential for nuanced geometry classification and physically meaningful interpretation. Future work will aim to further close the reality gap by incorporating machine dynamics and advanced friction models.

Summary updated

Abstract

<title>Abstract</title> In the field of forming technology, synthetic data generated by finite element (FE) simulations is increasingly being used to train machine learning (ML) models for quality prediction. However, the predictive accuracy on real-world process data is often limited by the so-called “reality gap” between simulated and measured signals. This study investigates how simulation model complexity influences the suitability of synthetic data for training ML models that generalize to real progressive deep drawing processes. Three representative simulation configurations of increasing complexity (L1–L3) are implemented and evaluated against experimental data collected under production conditions using sensorintegrated tools. The analysis covers multiple ML tasks, including classification of pre-process connector cut geometries, detection of process disturbances, separation of subtle geometry variants, assessment of feature transfer robustness, and saliency-based interpretation of signal regions. The results show that simple models (L1) enable robust classification of failures such as material damage within synthetic domains though their transferability to real data is restricted. Although the are computationally more expensive, higher complexity level (L2 and L3) better capture the effects of pre-processing and deformation-history, improving domain alignment and supporting physically meaningful model interpretation. Saliency map analysis reveals that models trained on synthetic data emphasize different signal regions than models trained on real data. This underscores the importance of task-relevant signal fidelity. The findings provide quantitative guidance for selecting an adequate level of simulation complexity and demonstrate that the underlying principles apply across a broader spectrum of modelling configurations.
BibTeX
@inproceedings{schumann2025requirements,
  author = {Markus Schumann and Jonas Moske and Antonia Wüst and Kristian Kersting and Peter Groche},
  title = {{Requirements for numeric models as sources of synthetic data for predicting real-world data sets}},
  booktitle = {Research Square},
  year = {2025},
  doi = {10.21203/rs.3.rs-8164519/v1},
}
Figure