A Temporal Convolutional Network with Multi-Head Attention Model: A Surrogate for Parallel DEVS Semiconductor Fab Simulation

Vamsi Krishna Pendyala, Hessam S. Sarjoughian, Edward J. Yellig, Bharadhwaj BalasubramaniamarXiv 2026 · arXiv

TL;DR

This paper introduces TCNA, a hybrid machine learning architecture that combines Temporal Convolutional Networks (TCN) with Multi-Head Attention to act as a fast "surrogate" for complex semiconductor factory simulations. By trading a marginal amount of accuracy for orders-of-magnitude gains in execution speed, TCNA enables real-time production scheduling and capacity planning that was previously bottlenecked by slow, high-fidelity discrete-event simulations.

What Problem Does This Paper Address?

Semiconductor fabrication is arguably the most complex manufacturing process on Earth. Wafers travel through hundreds of steps, often "re-entering" the same machines multiple times over weeks or months. Because these factories are stochastic (subject to random machine failures and variable arrival rates), managers rely on Discrete-Event Simulation (DES)—specifically the Parallel DEVS (PDEVS) formalism—to predict throughput ($TH$) and turnaround time ($TA$).

The catch? High-fidelity PDEVS simulations are computationally expensive. In a fast-paced "fab," waiting hours for a simulation to finish means you can't react quickly to a machine breakdown or a sudden change in order priority. There is a dire need for a "surrogate model"—a mathematical approximation that can mimic the simulation's output in milliseconds rather than minutes or hours.

Methods

The authors propose the Temporal Convolutional Network with Multi-head Attention (TCNA). The architecture is designed to capture two distinct types of patterns in factory data:

  1. Local Temporal Patterns: Handled by the TCN layers, which use dilated causal convolutions to look at step-to-step increments and short-term bursts of activity.
  2. Global Context: Handled by a Multi-Head Attention layer, which relates different positions in a long "look-back" window to understand overall trends, such as the factory ramping up to full capacity.

A key feature of TCNA is its use of static covariates. Information about the factory configuration—like the wafer lot size, the arrival profile (uniform vs. sinusoidal), and the maintenance policy—is embedded and concatenated with the time-series data. Finally, a gating mechanism (inspired by Gated Linear Units) allows the model to dynamically suppress irrelevant features, further sharpening its predictive focus.

Datasets

Because real-world semiconductor data is a highly guarded trade secret, the authors used synthetic data generated from the Intel MiniFab benchmark.

  • The Model: An 8-stage cascade MiniFab model implemented in PDEVS.
  • Scenarios: Six distinct scenarios (A–F) varying by repair types (MTBF vs. processing steps) and wafer arrival patterns.
  • Scale: 72 individual time series, each containing 30,000 data points spanning roughly 22 simulated days.
  • Comparison: The model was tested against several state-of-the-art architectures, including the Temporal Fusion Transformer (TFT), Autoformer, FEDformer, and N-BEATS.

Key Insights and Ideas

  • The "Regime Change" Challenge: One of the hardest things for an ML model to predict is the transition from a "cold start" (empty factory) to a steady state. The authors found that while many models (like LSTM) struggle with these sharp transitions, TCNA’s combination of local convolutions and global attention allows it to track these "non-zero events" much more accurately.
  • Efficiency vs. Accuracy: TCNA achieves accuracy comparable to the Temporal Fusion Transformer (TFT) but with a 10% smaller model size and 12% fewer parameters. In a production environment, this smaller footprint translates to lower deployment costs and faster inference.
  • Robustness to Lot Size: The paper highlights that smaller lot sizes create "choppier" data that is harder to predict. TCNA showed remarkable stability across small, medium, and large lot sizes, whereas simpler models like TCN or LSTM saw significant performance degradation as the data became more volatile.

Key Results

  • Accuracy: TCNA achieved an $R^2$ score of 0.9999 for throughput in most scenarios, consistently outperforming or matching the TFT.
  • Error Metrics: In turnaround time ($TA$) predictions, TCNA often achieved the lowest Mean Absolute Percentage Error (MAPE), frequently dipping below 1%.
  • Speedup: While the paper focuses on the ML architecture, it notes that TCNA provides orders-of-magnitude speedup over the original PDEVS simulation, transforming a slow batch process into a near-instantaneous prediction tool.
  • Ablation Study: Removing the TCN layer caused the most significant drop in performance (MAPE jumped from 0.86% to nearly 22%), proving that capturing local temporal dependencies is the "secret sauce" for this specific manufacturing domain.

Limitations and Future Work

The authors identify two main areas for improvement:

  1. Dynamic Covariates: The current model only considers "static" settings (things that don't change during a run). Future versions will incorporate "known-future" dynamic variables, such as planned maintenance schedules or scheduled wafer releases, using the "future-time embedding" branch already designed into the architecture.
  2. Beyond Point Accuracy: Standard metrics like MSE and $R^2$ don't always capture the "shape" of a discrete-event trajectory. The authors plan to move toward multi-metric assessments, including Dynamic Time Warping (DTW) and frequency-domain errors, to better evaluate how well the model captures the timing of specific factory events.

Summary updated

Abstract

Semiconductor factories process many kinds of wafers organized into lots of various sizes and configurations. Different kinds of machines, each with its own operations, process wafers for hours to days, and the wafers undergo many incremental fabrication steps (with re-entry) and flows spanning days to weeks across connected machines. Fast and accurate predictions of throughput ($TH$) and turnaround time ($TA$) dynamics for individual machines and their combinations are essential for factory operations. To use machine learning methods for this purpose, this paper presents a Temporal Convolutional Network with Multi-head Attention (TCNA) architecture. It uses simulated datasets from a benchmark Parallel DEVS (PDEVS) semiconductor fabrication factory model. The factory processes configurable wafer lots with uniform and sinusoidal wafer arrival profiles and may undergo preventive maintenance. The wafer lot size, arrival profile, and factory maintenance policy are used as static covariates. TCNA is applied to a selected set of simulated discrete-event throughput and turnaround time trajectories. The $TH$ and $TA$ time series predicted by TCNA, evaluated with the common $MAPE$, $MSE$, and $R^2$ measures, show comparable or higher accuracy relative to other model architectures, with similar parameter counts and epoch times; the TCNA model size and execution time are similar to those of other architectures such as the Temporal Fusion Transformer. Although time series predictions are less accurate than high-resolution simulations, they can be deployed at a much lower computational cost. The TCNA achieves orders-of-magnitude speedup over PDEVS simulation with an acceptable loss of accuracy, enabling frequent product scheduling, capacity planning, and maintenance planning.
BibTeX
@misc{pendyala2026temporal,
  author = {Vamsi Krishna Pendyala and Hessam S. Sarjoughian and Edward J. Yellig and Bharadhwaj Balasubramaniam},
  title = {{A Temporal Convolutional Network with Multi-Head Attention Model: A Surrogate for Parallel DEVS Semiconductor Fab Simulation}},
  year = {2026},
  eprint = {2611.20493},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20493},
}
Figure