Abstract
Time series forecasting (TSF) is one of the most important tasks in data science, as accurate time series (TS) predictions can drive and advance a wide variety of domains including finance, transportation, health care, and power systems. However, real-world utilization of machine learning (ML) models for TSF suffers due to pretrained models being able to learn and adapt to unpredictable patterns as previously unseen data arrives over longer time scales. To address this, models must be periodically retained or redesigned, which takes significant human and computational resources. This work presents the Online NeuroEvolution based Neural Architecture Search (ONE-NAS) algorithm, which to the authors' knowledge is the first neural architecture search algorithm capable of automatically designing and training new recurrent neural networks (RNNs) in an online setting. Without any pretraining, ONE-NAS utilizes populations of RNNs which are continuously updated with new network structures and weights in response to new multivariate input data. ONE-NAS is tested on real-world large-scale multivariate wind turbine data as well a univariate Dow Jones Industrial Average (DJIA) dataset, and is shown to outperform traditional statistical time series forecasting, including naive, moving average, and exponential smoothing methods, as well as state of the art online ARIMA strategies.
Introduction
Most machine learning models for time series forecasting (TSF) follow a "train-then-deploy" workflow. However, real-world data is rarely stationary; distributions shift, and new patterns emerge over time—a phenomenon known as concept drift. When this happens, static pretrained models become obsolete and lose accuracy. To fix this, engineers usually have to manually redesign or retrain models, which is resource-intensive. While online learning exists, it typically involves updating the weights of a fixed architecture. Finding the optimal architecture for a specific, changing data stream remains a massive challenge. This paper introduces ONE-NAS, which automates the discovery of both the network structure and its weights while the data is streaming in.
Methodology
ONE-NAS is built upon the Evolutionary Exploration of Augmenting Memory Models (EXAMM) framework but adapted for an asynchronous, online environment. The algorithm maintains a population of RNNs that evolves as data arrives, starting from a simple "seed genome" with no hidden layers and growing in complexity as needed.
Key components of the methodology include:
- NeuroEvolution (NE): In each "generation" (a fixed window of time steps), the algorithm performs mutation (adding nodes/edges) and crossover to create new candidate networks.
- Diverse Memory Cells: ONE-NAS draws from a library of cells including $\Delta$-RNN, GRU, LSTM, MGU, and UGRNN.
- Island Repopulation Strategy: To prevent the population from getting stuck in local optima, the algorithm uses multiple "islands." Periodically, the worst-performing island is wiped out and repopulated with mutations of the global best network.
- Lamarckian Weight Inheritance: Offspring networks inherit weights from their parents, significantly speeding up the training process compared to starting from scratch.
- Asynchronous Parallelism: The system uses a master-worker architecture where workers independently train and evaluate genomes, allowing the system to scale with available hardware.
Experimental Setup
The authors evaluated ONE-NAS on two challenging real-world datasets:
- Wind Turbine Engine Data: A multivariate dataset (22 parameters) from the La Haute Borne wind farm. It is noisy, non-seasonal, and contains raw, uncleaned data with outliers.
- Dow Jones Industrial Average (DJIA): A univariate dataset spanning 1885–1962 (35,701 samples). This is a classic benchmark for testing long-term adaptability in financial forecasting.
Results and Discussion
Experimental results demonstrate that ONE-NAS naturally scales its capacity to match the complexity of incoming data. A critical observation was the "burn-in" period: ONE-NAS initially performs worse than a "naive" predictor (which guesses the last seen value) but consistently surpasses baseline methods after roughly 500 generations.
On the wind turbine data, ONE-NAS with Island Repopulation significantly outperformed Naive, Moving Average, and Exponential Smoothing baselines. On the DJIA dataset, ONE-NAS outperformed modern online ARIMA variants (ARIMA-ONS and ARIMA-OGD) by multiple orders of magnitude in terms of Root Mean Squared Error (RMSE).
The Island Repopulation strategy proved essential for maintaining diversity and preventing stagnation. By forcing "extinction" events, the search was pushed into new areas of the architecture space, leading to significantly lower Mean Squared Error (MSE). Despite the complexity of evolution, the system is efficient; for the wind dataset (10-minute intervals), a generation took between 35 and 70 seconds on average, leaving ample buffer for real-time deployment.
Conclusion and Future Work
ONE-NAS represents a significant step forward in automated online learning for time series. While it faces an initial "burn-in" lag, its ability to adapt to concept drift without manual intervention is a major advantage. Future work will explore hybrid approaches to mitigate the initial lag, dynamic generation windows, and the use of population ensembles for more robust predictions and uncertainty estimates.
Summary updated