Standard deep learning operates under a luxury: time. Models repeatedly traverse the same dataset over multiple training epochs, shuffling mini-batches to smooth out non-convex loss landscapes and escape saddle points. However, real-world deployment often breaks this paradigm. In applications like real-time recommendation systems, on-device speculative decoding, or on-the-fly voice assistants, data arrives as a continuous, transient stream. The model gets exactly one look at each sample before it vanishes forever.
This strict single-pass constraint fundamentally breaks standard optimization. Without the ability to revisit past data, models face a dual crisis. First, they suffer from catastrophic forgetting, where new gradient updates overwrite previously learned representations. Second, as highlighted by Dahuin Jung et al. in New Insights for the Stability-Plasticity Dilemma in Online Continual Learning, they hit a plasticity bottleneck: a single exposure to a data point provides insufficient signal for a deep network to extract robust, generalized features.
To make single-pass learning work, researchers are redesigning how neural networks optimize, adapt their architectures, and preserve memory on the fly.
Navigating Loss Landscapes with Dynamic Architectures
Why does backpropagation fail in a single pass? In a deep network, early layers rely on gradients propagated back through later layers. When a model only sees a sample once, this highly non-linear coordination fails to converge, trapping deep architectures in poor local minima.
To bypass this, Doyen Sahoo et al. proposed Hedge Backpropagation (HBP) in Online Deep Learning: Learning Deep Neural Networks on the Fly. Instead of forcing the entire deep network to converge simultaneously, the authors attach classifiers to every hidden layer. Early on, shallow layers converge rapidly because their loss surfaces are simpler. As more data streams in, deeper layers gradually learn more complex features. By dynamically weighting these classifiers using the Hedge algorithm, the network effectively adapts its depth in real time, ensuring stable convergence from the very first pass.
Streaming Input ───► [ Layer 1 ] ───► [ Layer 2 ] ───► [ Layer 3 ]
│ │ │
▼ ▼ ▼
Classifier 1 Classifier 2 Classifier 3
│ │ │
└─────────► [ Hedge ] ◄─────────────┘
│
▼
Final Prediction
Another architectural solution is to grow the network modularly rather than modifying existing weights. In A Progressive Model to Enable Continual Learning for Semantic Slot Filling, Yilin Shen et al. introduced context-gated components that allocate new capacity for incoming data blocks. This prevents the single-pass update from corrupting previously established parameters, bypassing the stability-plasticity dilemma entirely by isolating new knowledge.
Meta-Learning and Virtual Gradients for One-Shot Optimization
If an architecture cannot be changed, the optimization step itself must become smarter. In a single-pass stream, a standard gradient step is short-sighted; it optimizes purely for the current instance, often at the expense of past tasks. Meta-learning addresses this by optimizing how the network learns.
In Look-ahead Meta Learning for Continual Learning, Gunshi Gupta et al. developed La-MAML, an algorithm that meta-learns per-parameter learning rates directly within the online stream. By looking ahead at the impact of a gradient update, it modulates the learning rates to favor weight updates that generalize across tasks.
Similarly, Kaushik Roy et al. introduced Amphibian in Amphibian: A Meta-Learning Framework for Rehearsal-Free, Fast Online Continual Learning (Trans. Mach. Learn. Res. 2025). Amphibian tracks "gradient alignments" across incoming samples using a diagonal scaling matrix. This accelerates updates in directions that mutually benefit multiple concepts while dampening updates that cause interference, maximizing the utility of the single pass.
When future data cannot be anticipated, models can simulate the future by evaluating updates against a running memory. In VERSE: Virtual-Gradient Aware Streaming Lifelong Learning with Anytime Inference, Soumya Banerjee et al. utilized "virtual gradient" updates. This mechanism evaluates how a potential weight change will impact a semantic memory maintained via exponential moving averages, ensuring the single-pass update remains anchored to historical knowledge.
Distillation and Latent Regularization
In many streaming environments, storing raw data for replay is prohibited due to privacy or hardware constraints. When restricted to a single pass, how can a model retain past knowledge without a physical buffer?
One approach is to distribute supervision across the network's internal hierarchy. In Orchestrate Latent Expertise: Advancing Online Continual Learning with Multi-Level Supervision and Reverse Self-Distillation, HongWei Yan et al. introduced MOSE. The authors treat a single network as a stack of sequential experts, using a reverse self-distillation process to pass knowledge from shallow layers to deep layers. This multi-level supervision ensures that even if a single pass fails to deeply embed a feature in the final layers, the shallow layers act as an internal buffer that preserves the representation.
Alternatively, parameter-efficient fine-tuning can isolate updates to low-rank subspaces. In Online-LoRA: Task-Free Online Continual Learning via Low Rank Adaptation, Guihong Li et al. applied online weight regularization exclusively to low-rank adapter parameters. By detecting distribution shifts through loss surface plateaus, the model knows precisely when to adapt its adapters and when to regularize them, maximizing plasticity during new concepts while freezing older, stable low-rank pathways.
Key Foundational Work
Online Deep Learning: Learning Deep Neural Networks on the Fly (2018) — contributed 2 concept elements to this direction.
Online Ranking Prediction in Non-stationary Environments (2017) — contributed 1 concept elements to this direction.