The Disappearance of Timestep Embedding in Modern Time-Dependent Neural Networks

Bum Jun Kim, Yoshinobu Kawahara, Sang Woo KimarXiv 2024 · DOI · arXiv

Introduction

Modern deep learning has seen a surge in time-dependent architectures, most notably Neural Ordinary Differential Equations (NODEs) and Diffusion Models. In these frameworks, "time" $t$ is a fundamental variable: NODEs interpret network depth as a continuous temporal flow, while Diffusion Models rely on timesteps to navigate varying noise levels during image generation. To integrate this temporal information, researchers typically employ timestep embeddings—vector representations of $t$ that are injected into the network's feature maps.

The Vanishing Embedding Phenomenon

A critical architectural oversight has been identified in how these embeddings interact with standard normalization layers, such as Batch Normalization (BN) or Group Normalization (GN). These layers stabilize training by centering features around a zero mean. However, if a timestep embedding is implemented as a uniform scalar offset added to a channel, the normalization process identifies this offset as part of the channel's mean and subtracts it.

Normalization layers can mathematically cancel out scalar timestep embeddings, rendering models 'time-blind'.

This mathematical cancellation renders the model "time-blind," effectively erasing the temporal signal. The authors found that some models only function due to "edge effects"—minor variations introduced by zero-padding at image boundaries. However, relying on these accidental artifacts leads to inefficient, "stiff" ODEs that are computationally expensive to solve.

Methods for Preserving Temporal Signals

To restore time-awareness, the authors propose three primary modifications to the standard architecture:

  1. Positional Timestep Embedding: Instead of a uniform scalar, the embedding is injected as a spatial map ($H \times W$) where values vary across pixel locations. Because the signal is non-uniform, it cannot be fully removed by mean-subtraction during normalization.
  2. Zero Bias Initialization: By initializing the biases of main convolutional layers to zero while maintaining non-zero biases for the timestep branch, the relative variance of the temporal signal is preserved.
  3. Group-Count Adjustment in Group Norm: The authors demonstrate that the vanishing effect is most pronounced when the number of channels per group is small. Reducing the number of groups (increasing channels per group) makes the embedding more resilient.
The proposed architecture incorporates both channel-wise ($\tilde{v}_t$) and positional ($\tilde{p}_t$) embeddings to ensure time-awareness survives normalization.

Experimental Evaluation

The effectiveness of these solutions was validated on the CIFAR-10 and CIFAR-100 datasets across classification and generation tasks.

Neural ODEs

On CIFAR-100, implementing Positional Timestep Embeddings improved classification accuracy from 69.47% to 69.86%. More importantly, it significantly enhanced computational efficiency. Models suffering from vanishing embeddings required nearly three times the Number of Function Evaluations (NFE) to converge, whereas time-aware models remained "smooth" and fast.

Diffusion Models

In image generation tasks using Denoising Diffusion Probabilistic Models (DDPM), the combination of the three proposed fixes led to a measurable improvement in image quality. The Fréchet Inception Distance (FID) dropped from 3.238 to 3.074, indicating more realistic and higher-quality synthetic images.

Discussion and Conclusion

While this study focuses on convolutional architectures (CNNs and U-Nets), the findings highlight a fundamental tension between normalization and temporal signaling. The interaction is a subtle but impactful bottleneck in modern AI. By ensuring that timestep embeddings are "alive" through positional encoding and careful normalization settings, researchers can build models that are both more accurate and computationally efficient. These guidelines serve as a necessary standard for the design of future time-dependent neural architectures.

Summary updated

Abstract

Dynamical systems are often time-varying, whose modeling requires a function that evolves with respect to time. Recent studies such as the neural ordinary differential equation proposed a time-dependent neural network, which provides a neural network varying with respect to time. However, we claim that the architectural choice to build a time-dependent neural network significantly affects its time-awareness but still lacks sufficient validation in its current states. In this study, we conduct an in-depth analysis of the architecture of modern time-dependent neural networks. Here, we report a vulnerability of vanishing timestep embedding, which disables the time-awareness of a time-dependent neural network. Furthermore, we find that this vulnerability can also be observed in diffusion models because they employ a similar architecture that incorporates timestep embedding to discriminate between different timesteps during a diffusion process. Our analysis provides a detailed description of this phenomenon as well as several solutions to address the root cause. Through experiments on neural ordinary differential equations and diffusion models, we observed that ensuring alive time-awareness via proposed solutions boosted their performance, which implies that their current implementations lack sufficient time-dependency.
BibTeX
@misc{kim2024disappearance,
  author = {Bum Jun Kim and Yoshinobu Kawahara and Sang Woo Kim},
  title = {{The Disappearance of Timestep Embedding in Modern Time-Dependent Neural Networks}},
  year = {2024},
  eprint = {2405.14126},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.14126},
}
Figure