Advanced Learning Technologies for Deep Learning

Yasutoshi Ida, Sekitoshi Kanai, Yasuhiro Fujiwara, Satoshi Yagi, Yasuhiro IidaNTT 2018 · DOI

Abstract

NTT is focusing on artificial intelligence (AI) as a key technology in developing business strategies. The NTT laboratories have been engaged in research and development (R&D) of deep learning algorithms, which play a central role in AI. This article presents an overview of our R&D of novel algorithms that speed up and stabilize deep learning and touches on our collaboration with several partners to verify those algorithms.

Introduction

While deep learning has revolutionized fields like image recognition and natural language processing, it remains computationally expensive and prone to instability. Researchers at NTT have identified two specific bottlenecks:

  1. Inefficient Optimization: Popular optimizers like Adam and RMSprop adjust learning rates based on the magnitude of past updates but ignore their variance. This can lead to unstable learning or getting stuck in local minima.
  2. Gradient Explosion in RNNs: Recurrent Neural Networks (RNNs), specifically the Gated Recurrent Unit (GRU), are essential for time-series data. However, they often suffer from "gradient explosion," where gradients become so large that the model fails to learn. Standard fixes like gradient clipping are often "trial-and-error" approaches requiring manual tuning.

Methodology

Variance-Based Adaptive Learning

Instead of just looking at the absolute values of update directions, NTT's algorithm calculates the variance of those directions. If the variance is high (unstable updates), the algorithm reduces the update amount to stabilize learning. Conversely, if the variance is low (stable updates), it increases the update amount to speed up convergence and help the model "break away" from local solutions.

Overview of the training process and the variance-based update logic.

This approach is theoretically linked to Information Geometry and the Fisher information matrix. By approximating the square root of the diagonal elements of the Fisher information matrix, the optimizer can navigate the loss function more efficiently than standard methods.

Bifurcation-Aware GRU Training

The authors analyzed GRUs through the lens of dynamical systems. They discovered that gradient explosions occur when model parameters hit a "bifurcation point"—a point where the behavior of the system changes drastically.

Instead of reactive gradient clipping, NTT’s method identifies these points and uses constrained training to keep parameters within a "safe" region. This proactive approach mathematically defines the boundaries of stability so the model never crosses them.

Analysis of GRU state convergence and the bifurcation point where gradients explode.

Key Results and Discussion

The proposed algorithms were implemented and verified within popular deep learning frameworks including Chainer, TensorFlow, and Caffe.

  • Faster Convergence: The variance-based optimizer demonstrates faster convergence on loss functions compared to typical updating rules by sensing consistency in update directions.
  • Reduced Manual Tuning: The GRU stabilization method requires significantly fewer trial-and-error attempts than traditional gradient clipping because it is based on the mathematical properties of the GRU state.
  • Real-World Integration: The technology is being integrated into NTT’s corevo Computing Infrastructure (CCI), aimed at allowing non-experts to perform advanced data analysis without needing to manually tune complex hyperparameters.

While the paper focuses on GRUs and general deep networks, future work involves exploring performance on newer architectures like Transformers and moving toward providing these tools as open-source software.

Summary updated

Abstract

NTT is focusing on artificial intelligence (AI) as a key technology in developing business strategies.The NTT laboratories have been engaged in research and development (R&D) of deep learning algorithms, which play a central role in AI.This article presents an overview of our R&D of novel algorithms that speed up and stabilize deep learning and touches on our collaboration with several partners to verify those algorithms.
BibTeX
@inproceedings{ida2018advanced,
  author = {Yasutoshi Ida and Sekitoshi Kanai and Yasuhiro Fujiwara and Satoshi Yagi and Yasuhiro Iida},
  title = {{Advanced Learning Technologies for Deep Learning}},
  booktitle = {NTT},
  year = {2018},
  doi = {10.53829/ntr201808fa6},
}
Figure