Introduction
Modern deep learning has seen a surge in time-dependent architectures, most notably Neural Ordinary Differential Equations (NODEs) and Diffusion Models. In these frameworks, "time" $t$ is a fundamental variable: NODEs interpret network depth as a continuous temporal flow, while Diffusion Models rely on timesteps to navigate varying noise levels during image generation. To integrate this temporal information, researchers typically employ timestep embeddings—vector representations of $t$ that are injected into the network's feature maps.
The Vanishing Embedding Phenomenon
A critical architectural oversight has been identified in how these embeddings interact with standard normalization layers, such as Batch Normalization (BN) or Group Normalization (GN). These layers stabilize training by centering features around a zero mean. However, if a timestep embedding is implemented as a uniform scalar offset added to a channel, the normalization process identifies this offset as part of the channel's mean and subtracts it.
This mathematical cancellation renders the model "time-blind," effectively erasing the temporal signal. The authors found that some models only function due to "edge effects"—minor variations introduced by zero-padding at image boundaries. However, relying on these accidental artifacts leads to inefficient, "stiff" ODEs that are computationally expensive to solve.
Methods for Preserving Temporal Signals
To restore time-awareness, the authors propose three primary modifications to the standard architecture:
- Positional Timestep Embedding: Instead of a uniform scalar, the embedding is injected as a spatial map ($H \times W$) where values vary across pixel locations. Because the signal is non-uniform, it cannot be fully removed by mean-subtraction during normalization.
- Zero Bias Initialization: By initializing the biases of main convolutional layers to zero while maintaining non-zero biases for the timestep branch, the relative variance of the temporal signal is preserved.
- Group-Count Adjustment in Group Norm: The authors demonstrate that the vanishing effect is most pronounced when the number of channels per group is small. Reducing the number of groups (increasing channels per group) makes the embedding more resilient.
Experimental Evaluation
The effectiveness of these solutions was validated on the CIFAR-10 and CIFAR-100 datasets across classification and generation tasks.
Neural ODEs
On CIFAR-100, implementing Positional Timestep Embeddings improved classification accuracy from 69.47% to 69.86%. More importantly, it significantly enhanced computational efficiency. Models suffering from vanishing embeddings required nearly three times the Number of Function Evaluations (NFE) to converge, whereas time-aware models remained "smooth" and fast.
Diffusion Models
In image generation tasks using Denoising Diffusion Probabilistic Models (DDPM), the combination of the three proposed fixes led to a measurable improvement in image quality. The Fréchet Inception Distance (FID) dropped from 3.238 to 3.074, indicating more realistic and higher-quality synthetic images.
Discussion and Conclusion
While this study focuses on convolutional architectures (CNNs and U-Nets), the findings highlight a fundamental tension between normalization and temporal signaling. The interaction is a subtle but impactful bottleneck in modern AI. By ensuring that timestep embeddings are "alive" through positional encoding and careful normalization settings, researchers can build models that are both more accurate and computationally efficient. These guidelines serve as a necessary standard for the design of future time-dependent neural architectures.
Summary updated