Online Deep Learning from Doubly-Streaming Data

Heng Lian, John Scovil Atwood, Bojian Hou, Jian Wu, Yi He

Abstract

In modern data environments, information often arrives in a "doubly-streaming" format, where both the number of instances and the set of features describing them evolve over time. Traditional online learning models struggle with this dual volatility, often sacrificing model expressiveness for real-time adaptability. This paper presents OLD3S (Online Learning Deep models from Data of Double Streams), a novel framework that leverages Variational Autoencoders (VAEs) and an adaptive "Hedge Backpropagation" mechanism. OLD3S allows deep neural networks to dynamically adjust their architecture and preserve knowledge across feature transitions, significantly outperforming existing linear and shallow online learning methods.

Introduction

Most machine learning paradigms operate under the assumption of a static feature space. However, real-world applications—ranging from industrial IoT sensor networks to multilingual text streams—are characterized by "doubly-streaming" data. In these scenarios, data arrives continuously (sample streaming), and the attributes or sensors providing that data change (feature streaming).

The primary challenge in this domain is the onlineness vs. expressiveness trade-off. Shallow models converge quickly and adapt easily to new features but lack the capacity to model complex patterns in high-dimensional data like images. Conversely, deep models offer high expressiveness but are typically "data-hungry," converging too slowly for real-time streaming and often suffering from catastrophic forgetting when feature spaces shift. OLD3S is designed to bridge this gap by enabling deep models to learn their optimal depth on-the-fly while maintaining continuity across feature sets.

The OLD3S Framework

The OLD3S paradigm integrates three core components to handle the complexities of doubly-streaming data:

1. Variational Latent Subspace Discovery

When a feature space evolves (e.g., from set $S_1$ to $S_2$), there is typically a short overlapping period where both sets are present. While prior methods used linear mappings to bridge these spaces, OLD3S employs two independent Variational Autoencoders (VAEs). These VAEs learn a shared latent subspace, allowing the model to reconstruct "old" features from "new" ones even after the original sensors or attributes have vanished. This non-linear approach is critical for capturing the complex interplay of high-dimensional data.

2. Adaptive Model Depth (Hedge Backpropagation)

To solve the convergence problem, OLD3S utilizes an over-complete neural network architecture. Rather than fixing the depth of the model, the framework employs a Hedge Backpropagation mechanism. This system assigns weights to different layers of the network. Initially, the model relies on shallow layers for rapid convergence. As more data is processed and patterns become more intricate, the system "mobilizes" deeper layers by shifting weight toward them, effectively learning the optimal semantic depth for the current data stream.

3. Ensemble Prediction and Knowledge Reuse

During the transition between feature spaces, OLD3S maintains an ensemble consisting of the mature classifier (operating on reconstructed features) and a nascent classifier (operating on the new features). Using an "exponential experts" strategy, the model dynamically shifts its reliance to the new classifier as it gains reliability, ensuring a smooth transition without the accuracy "dips" common in traditional online learners.

Experimental Evaluation

The researchers evaluated OLD3S against state-of-the-art competitors, including FOBOS, OLSF, and FESL, across 10 diverse datasets:

  • Tabular Benchmarks: UCI magic04 and adult datasets.
  • Multilingual Text: Categorization of documents across English, French, Italian, and Spanish, where language shifts represent feature evolution.
  • Image Classification: CIFAR-10, Fashion-MNIST, and SVHN, with feature evolution simulated through spectral mapping and geometric transformations.

Key Results

  • Superior Accuracy: OLD3S consistently outperformed all baselines across all 10 datasets.
  • Efficiency in Convergence: The model achieved a 69% improvement in Averaged Cumulative Regret (ACR) compared to the baseline FOBOS, demonstrating its ability to make fewer mistakes during the learning process.
  • Robustness to Shift: While competing models showed significant performance degradation during feature transitions, OLD3S maintained a stable learning curve due to its VAE-based reconstruction and ensemble strategy.

Discussion and Insights

The success of OLD3S highlights two critical insights for the future of online learning. First, depth should be treated as a learnable semantic rather than a fixed hyperparameter; by allowing a model to grow into its complexity, we can achieve both speed and power. Second, non-linear feature mapping is essential for modern data. The failure of linear methods on image datasets proves that as dimensionality increases, the ability to reconstruct latent relationships becomes the deciding factor in model resilience.

Conclusion and Future Work

OLD3S represents a significant step forward in making deep learning viable for non-stationary, real-time environments. While the current framework requires a brief overlap period ($T_b$) to learn feature mappings, it sets the stage for more complex "infinitely evolving" streams. Future research will focus on reducing the computational overhead of the VAE components to make the framework suitable for resource-constrained edge devices and exploring its application in unsupervised learning contexts.

Summary updated

BibTeX
@inproceedings{lian2022online,
  author = {Heng Lian and John Scovil Atwood and Bojian Hou and Jian Wu and Yi He},
  title = {{Online Deep Learning from Doubly-Streaming Data}},
  booktitle = {ACM Multimedia},
  year = {2022},
  doi = {10.1145/3503161.3548355},
}
Figure