A Skip-Connected Evolving Recurrent Neural Network for Data Stream Classification under Label Latency Scenario

Monidipa Das, Mahardhika Pratama, Jie Zhang, Yew-Soon Ong

Abstract

Data stream classification faces significant challenges when ground truth labels are delayed or unavailable, a phenomenon known as label latency. Traditional models often fail to adapt to concept drift without immediate feedback. This paper introduces SkipE-RNN, a self-evolving recurrent neural network that utilizes skip-connections and an autoencoder-based mapping unit to handle label latency. By identifying historical matches for current unlabeled samples, SkipE-RNN maintains high accuracy and structural flexibility in real-time environments.

Introduction

In real-world data streams—such as sensor networks, financial markets, or social media—labels are frequently delayed by hours or days, or may never be verified at all. This "Label Latency" paralyzes standard classifiers that rely on immediate feedback to detect concept drift. While semi-supervised or active learning methods exist, they often struggle with the temporal dependencies inherent in online data streams. SkipE-RNN addresses this by supervising itself "virtually" while dynamically adapting its architecture to evolving data patterns.

Methods

SkipE-RNN (Skip-Connected Evolving Recurrent Neural Network) consists of two main components: a Label Mapping Unit (M) and a Feature Learning Unit (H).

The overall framework of SkipE-RNN, illustrating the testing and training phases within a 'Prequential Test-Then-Train' cycle.

The Label Mapping Unit uses an autoencoder to find the "best match" for a current unlabeled sample from a pool of previously seen, labeled data. Instead of connecting only to the immediate previous step ($t-1$), the Feature Learning Unit (the core RNN) establishes a "skipped recurrent connection" to the state at the best-matched historical timestamp ($k$). This allows the model to exploit temporal dependencies even across long gaps.

Detailed unfolding of SkipE-RNN over time, showing the skip connections and the structural evolution process involving unit growing and pruning.

Key innovations include:

  • Teacher Forcing Policy: The model uses the label from the "best match" as a pseudo-teacher to update its weights immediately.
  • Regularized Weight Updating: A regularization term penalizes the model if a delayed true label reveals that the "best match" was incorrect, pulling parameters back toward an accurate state.
  • Structural Evolution: The network starts with a single hidden node and uses a "Network Significance" metric (based on bias and variance) to automatically grow new nodes when underfitting or prune redundant nodes when overfitting.

Results and Evaluation

The model was evaluated across two distinct scenarios: finite latency (labels arrive after a delay $\delta$) and infinite latency (labels never arrive after initial setup). Datasets included real-world examples like electricity pricing (ELECT) and weather data (NOAA), alongside synthetic benchmarks (HYPER, SEA, P-MNIST).

  • Accuracy: SkipE-RNN outperformed vanilla RNNs by an average of 13% in finite latency tests. In infinite latency scenarios, it consistently beat the state-of-the-art SCARGC model, particularly on the MG-2C-2D dataset (87.31% vs 61.91%).
  • Efficiency: Despite the overhead of structural evolution, SkipE-RNN is fast, processing samples in approximately 0.0004 seconds.
  • Robustness to Delay: Performance remained stable even as delays increased. Increasing the delay from 0 to 500 steps resulted in an average accuracy drop of only about 2.5%.
  • High-Dimensional Performance: On the P-MNIST dataset, SkipE-RNN achieved ~84% accuracy, significantly higher than other evolving networks like DEN (~52%) or PNN (~64%).

Conclusion

SkipE-RNN redefines recurrence as a search for similarity rather than just sequential order. By creating a "virtual supervisor" through its mapping unit, it maintains high performance in extreme latency scenarios where other models typically degrade. Its parsimonious approach—starting small and evolving only as needed—allows it to achieve superior results with fewer parameters than fixed-size models.

While the model requires a small initial set of labeled data (<2%) to seed the mapping unit, its ability to self-correct without continuous ground truth makes it a powerful tool for real-world streaming applications. Future work will explore integrating convolutional modules to handle more complex data types like video or image streams.

Summary updated

Abstract

Stream classification models for non-stationary environments often assume the immediate availability of data labels. However, in a practical scenario, it is quite natural that the data labels are available only after some temporal lag. This paper explores how a stream classifier model can be made adaptive to such label latency scenario. We propose SkipE-RNN, a self-evolutionary recurrent neural network with dynamically evolving skipped-recurrent-connection for the best utilization of previously observed label information while classifying the current data. When the data label is unavailable, SkipE-RNN uses an auto-learned mapping function to find the best match from the already known data labels and updates the network parameter accordingly. Later, upon availability of true data label, if the previously mapped label is found to be incorrect, SkipE-RNN employs a regularization technique along with the parameter updating process, so as to penalize the model. In addition, SkipE-RNN has inborn power of self-adjusting the network capacity by growing/pruning hidden nodes to cope with the evolving nature of data stream. Rigorous empirical evaluations using synthetic as well as real-world datasets reveal effectiveness of SkipE-RNN in both finitely delayed and infinitely delayed data label scenarios.
BibTeX
@inproceedings{das2020skip,
  author = {Monidipa Das and Mahardhika Pratama and Jie Zhang and Yew-Soon Ong},
  title = {{A Skip-Connected Evolving Recurrent Neural Network for Data Stream Classification under Label Latency Scenario}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i04.5781},
}
Figure