TL;DR
This paper introduces "Backforward Propagation," a novel method that completely eliminates Internal Covariate Shift (ICS) by recalculating layer updates. While computationally expensive, it demonstrates that removing ICS can lead to faster convergence and potentially better performance, suggesting ICS might act as an implicit regularizer.
What Problem Does This Paper Address?
Neural network training can be hampered by "Internal Covariate Shift" (ICS). This phenomenon occurs because as a network learns, the distribution of inputs to each layer changes with every weight update. This means a layer's weights are updated based on data distributions that are constantly shifting, making the learning process less stable and often requiring smaller learning rates for convergence. Existing methods like Batch Normalization (BatchNorm) mitigate ICS by normalizing layer inputs, but they don't entirely eliminate it and can introduce their own biases. This paper aims to tackle ICS head-on by proposing a method to completely remove its effect.
Methods
The core innovation is "Backforward Propagation." Unlike standard backpropagation where gradients are computed and applied sequentially, Backforward Propagation recalculates the update for each layer by taking into account the distribution changes in all preceding layers after their updates have occurred.
The naive implementation involves performing a full forward and backward pass for each layer, updating only one layer at a time. This is computationally intensive. The authors suggest that a more efficient implementation would involve processing portions of the model or having access to the computation graph of gradients, which is not readily available in standard deep learning frameworks.
Datasets
The experiments were conducted using the MNIST dataset.
Key Insights and Ideas
- Complete ICS Elimination: The paper proposes a method that goes beyond simply mitigating ICS, aiming for its complete removal.
- ICS as a Regularizer: The experiments suggest that ICS might have a weight-regularizing effect on models. Removing it leads to larger weight norms, implying that ICS, while detrimental to convergence speed, might prevent overfitting in some scenarios.
- Computational Cost vs. Performance: The proposed method is computationally expensive, but it shows that eliminating ICS can enable faster convergence, especially at lower learning rates.
- Potential for Faster Convergence: By allowing the model to take steps in the "exact" gradient direction without the noise introduced by shifting distributions, Backforward Propagation can achieve faster convergence.
Key Results
- On a toy experiment with a single batch from MNIST, Backforward Propagation was able to reduce the loss to near zero in a single update with a learning rate of 1.0, whereas the baseline model showed a very large loss, indicating the significant impact of ICS.
- When comparing Backforward Propagation with a standard SGD baseline on MNIST using an MLP with one hidden layer, the authors observed that the baseline performed similarly to Backforward Propagation at a learning rate of 0.1.
- However, Backforward Propagation converged faster and performed better at learning rates of 0.05 and 0.01.
- The norm of weights was observed to be larger in the absence of ICS, suggesting ICS has a regularizing effect.
Limitations and Future Work
- Computational Cost: The primary limitation is the significant computational overhead of the proposed Backforward Propagation method, which greatly decreases training speed.
- Implementation Complexity: A general solution requires access to computation graphs, which is not easily available in current deep learning frameworks.
- Future Work: The authors plan to extend experiments to deeper architectures where ICS has a greater impact. They also aim to investigate applications where the computational cost might be acceptable, such as model warmup or fine-tuning. Furthermore, they intend to compare Backforward Propagation with BatchNorm and explore how BatchNorm behaves in the absence of ICS.
Summary updated