Introduction
Predicting outcomes in complex industrial processes, such as rubber mixing, presents a significant challenge for conventional machine learning models. These environments generate multivariable time series data characterized by high levels of noise and "covariate shift"—where the statistical properties of data change over time due to machinery wear or raw material variability. Standard models, including Transformers, often struggle with these "messy" real-world scenarios because they fail to capture local dependencies and are sensitive to irrelevant features.
To address these issues, this paper introduces AdaDynaTrans, a novel attention-based model designed for robust industrial time series prediction. By integrating dynamic feature selection, position-enhanced attention, and a "worst-case" segmental training strategy, AdaDynaTrans achieves state-of-the-art accuracy in predicting Mooney viscosity, a critical metric in tire production.
Methodology: The AdaDynaTrans Architecture
The AdaDynaTrans model is composed of several specialized modules designed to handle the unique complexities of industrial data.
Dynamic Feature Selection
The model utilizes Dynamic Residual Units (DRU) and Feature Focused Units (FFU). These components act as intelligent filters, employing gating mechanisms to automatically suppress irrelevant input features. This ensures that the model focuses on the most predictive variables while adjusting its depth based on the complexity of the incoming data.
Position Enhanced Transformer (PET)
A core innovation is the Position Enhanced Transformer (PET). Standard Transformers rely on absolute position encodings, which can lead to "disorderly" attention patterns in time series. PET replaces this with a directed graph-based approach, treating time steps as nodes. This allows the model to explicitly learn the importance of the relative distance between any two points in time, significantly improving its ability to capture temporal dependencies.
Temporal Segmentation and Segmental Training (TSST)
To combat covariate shift, the authors developed the Temporal Segmentation and Segmental Training (TSST) strategy. This method uses a "maximum entropy" principle to divide training data into diverse segments with distinct distributions. By training the model to find "common knowledge" that performs well across these varied segments, it becomes more resilient to shifts in unseen data.
Static Covariate Extraction
Unlike models that treat static metadata (e.g., equipment IDs) as simple inputs, the Static Covariate Extraction Module (SCEM) extracts context vectors. These vectors are used to initialize the hidden states of temporal modules, providing a stable anchor for the model's predictions.
Experimental Evaluation
The model was rigorously tested on two datasets:
- Mooney Viscosity Dataset: An industrial dataset comprising 100,000 samples from a tire factory, featuring six temporal variables (temperature, power, etc.) and five static covariates.
- Air Quality Dataset: A public benchmark containing hourly data from Beijing to evaluate the model's generalizability.
Key Results
AdaDynaTrans consistently outperformed established baselines, including GRU, LSTM, LightGBM, and AdaRNN.
- Industrial Performance: On the Mooney viscosity task, the model reduced Mean Absolute Error (MAE) by 51% and Root Mean Square Error (RMSE) by 49.5% compared to a standard GRU baseline.
- Public Benchmark: It achieved an average improvement of 49.13% on the air quality dataset.
- Ablation Study: Removing the Temporal Variable Evolution Module (TVEM) resulted in a 27.66% spike in RMSE, highlighting the importance of modeling feature evolution over time.
Conclusion and Future Directions
AdaDynaTrans demonstrates that combining relative position awareness with adversarial-style segmental training can significantly improve the reliability of industrial time series forecasting. While the model excels in accuracy, future work will focus on enhancing interpretability for factory operators and extending the architecture to handle spatiotemporal data from distributed sensor networks.
Summary updated