L3DOC: Lifelong 3D Object Classification

Yuyang Liu, Yang Cong, Gan Sun, Tao Zhang, Jiahua Dong, Hongsen LiuIEEE TIP 2021 · OpenReview

TL;DR

This paper introduces L3DOC, a novel lifelong learning framework for 3D object classification. L3DOC addresses the challenge of catastrophic forgetting in 3D data by factorizing shared "point-knowledge" and using a memory attention mechanism to selectively transfer knowledge between tasks, enabling continuous learning without retraining.

What Problem Does This Paper Address?

The field of 3D object classification has seen significant advancements, with applications ranging from autonomous driving to robotics. However, current state-of-the-art models typically operate on a fixed set of object classes. In real-world scenarios, new object classes or tasks emerge over time, requiring models to adapt without forgetting previously learned information. This is known as the "catastrophic forgetting" problem. Traditional approaches like retraining from scratch are computationally expensive and inefficient. Existing lifelong learning methods, while successful in 2D domains, struggle with the unique challenges of 3D data, such as its unordered and irregular nature, which makes knowledge transfer difficult. L3DOC aims to bridge this gap by enabling continuous learning of new 3D object classes while preserving performance on older ones.

Methods

L3DOC tackles the lifelong learning challenge in 3D object classification through two core innovations:

  • Layer-Wise Point-Knowledge Factorization: This method factorizes the convolution kernels of a 3D neural network (specifically, PointNet in their implementation) into shared "point-knowledge" components and task-specific factors. This allows the model to store and reuse common geometric features across different tasks, rather than learning entirely new representations for each. The factorization is applied layer-wise to capture hierarchical knowledge.
  • Memory Attention Mechanism (MAM): To prevent catastrophic forgetting, MAM is employed. It selectively identifies and transfers relevant task-specific knowledge from previously learned tasks to the current one. This mechanism uses attention scores to weigh the importance of past knowledge, ensuring that only similar and useful information is transferred, thus mitigating the negative impact of new tasks on old ones.

The framework is designed to be end-to-end and aims to achieve this lifelong learning capability with more compact model parameters compared to traditional methods.

Datasets

The paper evaluates L3DOC on the following 3D point cloud datasets:

  • ModelNet10: A dataset containing 10 object categories.
  • ModelNet40: A larger dataset with 40 object categories.
  • 3DMNIST: A dataset of 3D representations of handwritten digits.

For lifelong learning evaluation, these datasets are split into multiple sequential tasks, with each task containing a subset of object classes.

Key Insights and Ideas

  • "Point-Knowledge" as a Cross-Task Resource: The core idea of factorizing convolution kernels to extract and store shared "point-knowledge" is crucial. This allows the model to build a reusable knowledge base from the geometric properties of 3D objects, which is fundamental for lifelong learning in this domain.
  • Selective Knowledge Transfer is Key: Simply transferring all knowledge from previous tasks can be detrimental. The memory attention mechanism's ability to intelligently select and transfer similar task-specific knowledge is vital for preventing catastrophic forgetting and maintaining performance.
  • 3D Data's Unique Challenges: The paper highlights that standard 2D lifelong learning techniques don't directly translate to 3D due to the inherent unordered and irregular nature of point clouds, necessitating specialized approaches like point-knowledge factorization.
  • Compactness and Efficiency: The proposed factorization not only aids in knowledge transfer but also leads to a more parameter-efficient model, making it more suitable for resource-constrained applications.

Key Results

  • Superior Performance: L3DOC consistently outperforms other lifelong learning models (STL-PointNet, Prog-PointNet, DF-PointNet, DEN) in terms of Average Per-Task Accuracy (APA) across ModelNet10, ModelNet40, and 3DMNIST.
  • Reduced Catastrophic Forgetting: The Catastrophic Forgetting Ratio (CFR) is significantly lower for L3DOC compared to most baseline methods, indicating its effectiveness in preserving past knowledge.
  • Faster Convergence: L3DOC demonstrates a faster Speed of Convergence (SC) on most datasets, aligning with its complexity analysis.
  • Parameter Efficiency: L3DOC reduces the number of parameters by 1.68x to 3.36x compared to STL-PointNet, while maintaining competitive accuracy.
  • Scalability: The framework is shown to be extendable to other backbone networks, such as PointNet++, with similar efficiency gains.

Limitations and Future Work

  • Parameter Tuning: The paper notes that the choice of scale parameters (n, lout, s) in the factorization architecture can affect performance and training time, suggesting that optimal tuning might be task-dependent.
  • Dataset Splits: While the paper uses sequential task splits, real-world lifelong learning scenarios can involve more complex task relationships and orderings, which could be a direction for future research.
  • Extending to Other 3D Tasks: The current work focuses on classification. Future work could explore extending L3DOC to other 3D perception tasks like segmentation or detection in a lifelong learning setting.

Summary updated

Abstract

3D object classification has been widely applied in both academic and industrial scenarios. However, most state-of-the-art algorithms rely on a fixed object classification task set, which cannot tackle the scenario when a new 3D object classification task is coming. Meanwhile, the existing lifelong learning models can easily destroy the learned tasks performance, due to the unordered, large-scale, and irregular 3D geometry data. To address these challenges, we propose a <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L</u> ifelong <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3D</u> <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">O</u> bject <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C</u> lassification ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.,</i> L3DOC) model, which can consecutively learn new 3D object classification tasks via imitating “human learning”. More specifically, the core idea of our model is to capture and store the cross-task common knowledge of 3D geometry data in a 3D neural network, named as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">point-knowledge</i> , through employing layer-wise <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">point-knowledge</i> factorization architecture. Afterwards, a task-relevant knowledge distillation mechanism is employed to connect the current task to previous relevant tasks and effectively prevent <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">catastrophic forgetting</i> . It consists of a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">point-knowledge</i> distillation module and a transforming-space distillation module, which transfers the accumulated <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">point-knowledge</i> from previous tasks and soft-transfers the compact factorized representations of the transforming-space, respectively. To our best knowledge, the proposed L3DOC algorithm is the first attempt to perform deep learning on 3D object classification tasks in a lifelong learning way. Extensive experiments on several point cloud benchmarks illustrate the superiority of our L3DOC model over the state-of-the-art lifelong learning methods.
BibTeX
@article{liu2021l,
  author = {Yuyang Liu and Yang Cong and Gan Sun and Tao Zhang and Jiahua Dong and Hongsen Liu},
  title = {{L3DOC: Lifelong 3D Object Classification}},
  journal = {IEEE Trans. Image Process.},
  year = {2021},
  url = {https://openreview.net/forum?id=eILHae6rATJ},
}
Figure