Learning Shared Knowledge for Deep Lifelong Learning using Deconvolutional Networks

Seungwon Lee, James Stokes, Eric EatonIJCAI 2019 · OpenReview

Abstract

Current mechanisms for knowledge transfer in deep networks tend to either share the lower layers between tasks, or build upon representations trained on other tasks. However, existing work in non-deep multi-task and lifelong learning has shown success with using factorized representations of the model parameter space for transfer, permitting more flexible construction of task models. Inspired by this idea, we introduce a novel architecture for sharing latent factorized representations in convolutional neural networks (CNNs). The proposed approach, called a deconvolutional factorized CNN, uses a combination of deconvolutional factorization and tensor contraction to perform flexible transfer between tasks. Experiments on two computer vision data sets show that the DF-CNN achieves superior performance in challenging lifelong learning settings, resists catastrophic forgetting, and exhibits reverse transfer to improve previously learned tasks from subsequent experience without retraining.

Introduction

In standard deep learning, models are typically trained on a single dataset for a specific task. When these models are subsequently trained on new tasks, they often suffer from catastrophic forgetting, where the network completely loses its ability to perform the original task.

Existing solutions like Hard Parameter Sharing (sharing lower layers) are often too rigid, assuming all tasks share identical low-level features. Conversely, Progressive Networks avoid forgetting by instantiating new sub-networks for each task, but this leads to quadratic growth in parameters and prevents "reverse transfer"—the ability for new knowledge to improve performance on older tasks.

Comparison of multi-task learning architectures

The authors propose a "middle ground": a shared knowledge base that is flexible enough for diverse tasks and allows bidirectional knowledge flow.

Deconvolutional Factorized CNNs (DF-CNN)

The core innovation of this work is Deconvolutional Factorization. Instead of learning convolutional filters directly for each task, the DF-CNN reconstructs them from a shared latent knowledge base ($L$).

DF-CNN architecture and filter reconstruction

The reconstruction process for each layer involves two stages:

  1. Deconvolutional Mapping: A task-specific filter ($V_t$) deconvolves the shared knowledge base ($L$) to generate a basis of convolutional filters. This stage naturally captures spatial structures. Notably, this achieves "natural sparsity" because the deconvolution operator is mathematically sparse by design.
  2. Tensor Contraction: A second task-specific tensor ($U_t$) performs a linear combination of these basis vectors to produce the final filters for the specific task, handling channel-wise transfer.

As new tasks arrive, the network initializes new task-specific tensors ($V_t, U_t$) while continuing to update the shared knowledge base ($L$), allowing the global knowledge to evolve over time.

Experimental Evaluation

The DF-CNN was evaluated on two computer vision benchmarks:

  • CIFAR-100: Split into 10 consecutive tasks of 10 classes each, using only 4% of training data to simulate data-scarce lifelong learning.
  • Office-Home: 10 tasks across different domains (Product and Real-World), each containing 13 classes.

Key Results

  • Superior Accuracy: On CIFAR-100, DF-CNN achieved a peak per-task accuracy of 37.2%, outperforming Single-Task Learning (31.2%–34.3%) and Hard Parameter Sharing (24.7%).
  • Resistance to Forgetting: DF-CNN maintained a "Catastrophic Forgetting Ratio" near 1.0, indicating it successfully retained knowledge of previous tasks.
  • Reverse Transfer: A significant finding was that learning subsequent tasks (e.g., Task 10) actually improved the model's performance on earlier tasks (e.g., Task 1) as the shared knowledge base became more refined.
  • Efficiency: The model scales linearly with the number of tasks, offering better parameter efficiency than Progressive Networks, which scale quadratically.

Conclusion

The DF-CNN architecture demonstrates that factorizing convolutional filters into a shared knowledge base via deconvolution provides a powerful mechanism for lifelong learning. By decoupling spatial and channel knowledge, the model achieves flexible transfer and resists forgetting. Future work may explore applying this deconvolutional approach to non-CNN architectures and automating the selection of knowledge base capacity.

Summary updated

Abstract

Current mechanisms for knowledge transfer in deep networks tend to either share the lower layers between tasks, or build upon representations trained on other tasks. However, existing work in non-deep multi-task and lifelong learning has shown success with using factorized representations of the model parameter space for transfer, permitting more flexible construction of task models. Inspired by this idea, we introduce a novel architecture for sharing latent factorized representations in convolutional neural networks (CNNs). The proposed approach, called a deconvolutional factorized CNN, uses a combination of deconvolutional factorization and tensor contraction to perform flexible transfer between tasks. Experiments on two computer vision data sets show that the DF-CNN achieves superior performance in challenging lifelong learning settings, resists catastrophic forgetting, and exhibits reverse transfer to improve previously learned tasks from subsequent experience without retraining.
BibTeX
@inproceedings{lee2019learning,
  author = {Seungwon Lee and James Stokes and Eric Eaton},
  title = {{Learning Shared Knowledge for Deep Lifelong Learning using Deconvolutional Networks}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=7rpsU3X5est},
}
Figure