Abstract
Data stream classification faces significant challenges when ground truth labels are delayed or unavailable, a phenomenon known as label latency. Traditional models often fail to adapt to concept drift without immediate feedback. This paper introduces SkipE-RNN, a self-evolving recurrent neural network that utilizes skip-connections and an autoencoder-based mapping unit to handle label latency. By identifying historical matches for current unlabeled samples, SkipE-RNN maintains high accuracy and structural flexibility in real-time environments.
Introduction
In real-world data streams—such as sensor networks, financial markets, or social media—labels are frequently delayed by hours or days, or may never be verified at all. This "Label Latency" paralyzes standard classifiers that rely on immediate feedback to detect concept drift. While semi-supervised or active learning methods exist, they often struggle with the temporal dependencies inherent in online data streams. SkipE-RNN addresses this by supervising itself "virtually" while dynamically adapting its architecture to evolving data patterns.
Methods
SkipE-RNN (Skip-Connected Evolving Recurrent Neural Network) consists of two main components: a Label Mapping Unit (M) and a Feature Learning Unit (H).
The Label Mapping Unit uses an autoencoder to find the "best match" for a current unlabeled sample from a pool of previously seen, labeled data. Instead of connecting only to the immediate previous step ($t-1$), the Feature Learning Unit (the core RNN) establishes a "skipped recurrent connection" to the state at the best-matched historical timestamp ($k$). This allows the model to exploit temporal dependencies even across long gaps.
Key innovations include:
- Teacher Forcing Policy: The model uses the label from the "best match" as a pseudo-teacher to update its weights immediately.
- Regularized Weight Updating: A regularization term penalizes the model if a delayed true label reveals that the "best match" was incorrect, pulling parameters back toward an accurate state.
- Structural Evolution: The network starts with a single hidden node and uses a "Network Significance" metric (based on bias and variance) to automatically grow new nodes when underfitting or prune redundant nodes when overfitting.
Results and Evaluation
The model was evaluated across two distinct scenarios: finite latency (labels arrive after a delay $\delta$) and infinite latency (labels never arrive after initial setup). Datasets included real-world examples like electricity pricing (ELECT) and weather data (NOAA), alongside synthetic benchmarks (HYPER, SEA, P-MNIST).
- Accuracy: SkipE-RNN outperformed vanilla RNNs by an average of 13% in finite latency tests. In infinite latency scenarios, it consistently beat the state-of-the-art SCARGC model, particularly on the MG-2C-2D dataset (87.31% vs 61.91%).
- Efficiency: Despite the overhead of structural evolution, SkipE-RNN is fast, processing samples in approximately 0.0004 seconds.
- Robustness to Delay: Performance remained stable even as delays increased. Increasing the delay from 0 to 500 steps resulted in an average accuracy drop of only about 2.5%.
- High-Dimensional Performance: On the P-MNIST dataset, SkipE-RNN achieved ~84% accuracy, significantly higher than other evolving networks like DEN (~52%) or PNN (~64%).
Conclusion
SkipE-RNN redefines recurrence as a search for similarity rather than just sequential order. By creating a "virtual supervisor" through its mapping unit, it maintains high performance in extreme latency scenarios where other models typically degrade. Its parsimonious approach—starting small and evolving only as needed—allows it to achieve superior results with fewer parameters than fixed-size models.
While the model requires a small initial set of labeled data (<2%) to seed the mapping unit, its ability to self-correct without continuous ground truth makes it a powerful tool for real-world streaming applications. Future work will explore integrating convolutional modules to handle more complex data types like video or image streams.
Summary updated