TL;DR
ONE-NAS is the first neural architecture search (NAS) algorithm designed for online time series forecasting. It evolves and trains recurrent neural network (RNN) architectures in real-time as data arrives, successfully outperforming both traditional statistical methods and state-of-the-art online ARIMA models without requiring any offline pretraining.
What Problem Does This Paper Address?
Most machine learning models for time series forecasting (TSF) follow a "train-then-deploy" workflow. However, real-world data is rarely stationary; distributions shift, and new patterns emerge over time (a phenomenon known as concept drift). When this happens, static pretrained models become "obsolete" and lose accuracy.
To fix this, engineers usually have to manually redesign or retrain models, which is resource-intensive. While online learning exists, it typically involves updating the weights of a fixed architecture. Finding the "right" architecture (e.g., the number of layers or types of memory cells) for a specific, changing data stream is a massive challenge. This paper introduces ONE-NAS, which automates the discovery of both the network structure and its weights while the data is streaming in.
Methods
ONE-NAS is built upon the EXAMM (Evolutionary Exploration of Augmenting Memory Models) framework but adapted for an asynchronous, online environment.
Key components include:
- Minimal Start: Evolution begins with a "seed genome"—a simple network with no hidden layers—and grows in complexity as needed.
- NeuroEvolution (NE): It maintains a population of RNNs. In each "generation" (a fixed window of time steps), the algorithm performs mutation (adding nodes/edges) and crossover to create new candidate networks.
- Diverse Memory Cells: Unlike standard NAS that might only use LSTMs, ONE-NAS draws from a library of cells including $\Delta$-RNN, GRU, LSTM, MGU, and UGRNN.
- Island Repopulation Strategy: To prevent the population from getting stuck in "local optima" (mediocre solutions), the algorithm uses multiple "islands" of populations. Periodically, the worst-performing island is wiped out and repopulated with mutations of the global best network.
- Lamarckian Weight Inheritance: Offspring networks inherit weights from their parents, significantly speeding up the training process compared to starting from scratch.
- Asynchronous Parallelism: The system uses a master-worker architecture where workers independently train and evaluate genomes, allowing the system to scale with available hardware.
Datasets
The authors evaluated ONE-NAS on two challenging real-world datasets:
- Wind Turbine Engine Data: A multivariate dataset (22 parameters) from the La Haute Borne wind farm. It is noisy, non-seasonal, and contains raw, uncleaned data with outliers.
- Dow Jones Industrial Average (DJIA): A univariate dataset spanning 1885–1962 (35,701 samples). This is a classic benchmark for testing long-term adaptability in financial forecasting.
Key Insights and Ideas
- Structure Follows Complexity: By evolving the architecture online, the model naturally scales its capacity to match the complexity of the incoming data. If the data becomes harder to predict, the evolution process favors more complex structures.
- The "Burn-in" Period: The authors observed that ONE-NAS initially performs worse than a "naive" predictor (which just guesses the last seen value). However, after a "burn-in" period (roughly 500 generations), the evolved networks consistently surpass baseline methods.
- Diversity is Safety: The Island Repopulation strategy was critical. Without it, the population tended to stagnate. The "extinction" events forced the search into new areas of the architecture space, leading to much lower Mean Squared Error (MSE).
- Handling Catastrophic Forgetting: Because the algorithm maintains a population of networks and trains on a mix of recent and historical data, it retains "memory" of past patterns better than simple gradient-descent-based online learners.
Key Results
- Vs. Classical Methods: On the wind turbine data, ONE-NAS with Island Repopulation significantly outperformed Naive, Moving Average, and Exponential Smoothing baselines.
- Vs. State-of-the-Art: On the DJIA dataset, ONE-NAS outperformed modern online ARIMA variants (ARIMA-ONS and ARIMA-OGD) by multiple orders of magnitude in terms of Root Mean Squared Error (RMSE).
- Efficiency: Despite the complexity of evolution, the system is fast. For the wind dataset (10-minute intervals), a generation took between 35 and 70 seconds on average, leaving plenty of "buffer" time for real-time deployment.
Limitations and Future Work
- Initial Lag: The "burn-in" period where the model is less accurate than a naive guess is a hurdle for immediate deployment. The authors suggest a hybrid approach—using a naive predictor until the NAS has matured.
- Fixed Generation Windows: Currently, the time window for a "generation" is fixed. Future work could involve dynamically adjusting this window based on data volatility.
- Ensemble Potential: The current version only uses the single "global best" network for prediction. The authors note that using the entire population as an ensemble could provide even more robust predictions and uncertainty estimates.
- Data Selection: Currently, training data is selected randomly from history. More sophisticated "active learning" or "curriculum learning" strategies could help the model learn faster.
Summary updated