Tree-CNN: A hierarchical Deep Convolutional Neural Network for incremental learning

Deboleena Roy, Priyadarshini Panda, Kaushik RoyNeural Networks 2020 · OpenReview

TL;DR

This paper introduces Tree-CNN, a novel hierarchical deep convolutional neural network architecture designed for incremental learning. Tree-CNN addresses the challenge of "catastrophic forgetting" by growing in a tree-like fashion, allowing it to learn new classes without retraining the entire network, thus offering a better trade-off between accuracy and training effort.

What Problem Does This Paper Address?

Deep Convolutional Neural Networks (DCNNs) have achieved remarkable success in computer vision tasks, but they typically rely on a fixed dataset for training. In real-world scenarios, data is constantly evolving, and models need to adapt to new information. This poses a significant challenge due to issues like "catastrophic forgetting," where learning new information leads to the loss of previously learned knowledge. Traditional methods of retraining DCNNs on new data are computationally expensive and can lead to performance degradation. The paper aims to develop a DCNN architecture that can incrementally learn new classes efficiently while retaining knowledge of previously learned classes.

Methods

The core innovation is the Tree-CNN architecture, which organizes DCNNs in a hierarchical, tree-like structure.

  • Hierarchical Structure: The network starts with a root node and branches out as new classes are introduced. Each node (except leaf nodes) contains a DCNN responsible for classifying inputs into one of its children.
  • Incremental Growth: When new classes arrive, the Tree-CNN uses a "Grow-Tree" algorithm to decide where to integrate them. This decision is based on the similarity of features between new and existing classes, determined by the softmax output of the root node.
  • Adaptive Branching: The algorithm considers three actions: adding a new class to an existing child node, merging child nodes to form a new one, or adding the new class as a new leaf node. This process is guided by user-defined thresholds and constraints like maximum children per node and tree depth.
  • Localized Retraining: Crucially, only the affected nodes (newly added or modified) are retrained, significantly reducing the computational cost compared to retraining the entire network. The root node is always retrained on all available data to maintain global context.

Datasets

  • CIFAR-10: Used for demonstrating the addition of multiple new classes.
  • CIFAR-100: Used for sequentially adding new classes in groups and for benchmarking against other incremental learning methods.

Key Insights and Ideas

  • Hierarchical Growth as a Solution to Forgetting: The tree-like structure naturally accommodates new information by adding branches, effectively isolating changes and mitigating catastrophic forgetting.
  • Feature Similarity Guides Growth: Using softmax likelihoods to group semantically similar classes into branches is an intuitive and effective way to organize the learning process.
  • Trade-off Optimization: Tree-CNN offers a compelling balance between achieving high accuracy and minimizing training effort, outperforming many fine-tuning strategies.
  • Potential for Discovering Hidden Relationships: The hierarchical grouping of classes might reveal underlying semantic relationships in the data, opening avenues for further exploration.

Key Results

  • CIFAR-10: Tree-CNN achieved comparable accuracy to fine-tuning strategies (e.g., B:II, B:III) while offering significantly lower normalized training effort (~40% less than B:V and ~30% less than B:II). It achieved an accuracy of 86.24% with a normalized training effort of 0.60.
  • CIFAR-100: Tree-CNNs (Tree-CNN-5, -10, -20) demonstrated a better trade-off between training effort and accuracy compared to fine-tuning methods. They achieved near state-of-the-art accuracy (e.g., 69.85% for Tree-CNN-5) with significantly lower training effort than some fine-tuning approaches. Compared to iCaRL, Tree-CNN yielded 10% higher accuracy.

Limitations and Future Work

  • Growing Network Size: The paper acknowledges that the Tree-CNN grows in size over time, and the implications for memory requirements during inference need to be investigated.
  • Exploration of Semantic Similarity: The authors suggest that the correlation between semantic similarity of class labels and feature similarity within branches is an interesting area for future research.
  • Label Relation Graph: The hierarchical grouping can generate a label relation graph, which could be further utilized for classification.
  • Image Similarity Measure: The final leaf nodes and their distances could be used as a measure of image similarity, enabling hierarchical classification of large datasets.

Summary updated

BibTeX
@inproceedings{roy2020tree,
  author = {Deboleena Roy and Priyadarshini Panda and Kaushik Roy},
  title = {{Tree-CNN: A hierarchical Deep Convolutional Neural Network for incremental learning}},
  booktitle = {Neural Networks},
  year = {2020},
  url = {https://openreview.net/forum?id=GVoFPf5m3q6},
}

Loading related content...

Figure