TL;DR
This paper introduces Targeted Gradient Descent (TGD), a novel fine-tuning method for convolutional neural networks (ConvNets) that allows them to adapt to new data distributions without retraining on the original dataset. TGD enables efficient online learning, significantly reducing training time and preventing artifacts on out-of-distribution data, particularly demonstrated in PET image denoising.
What Problem Does This Paper Address?
Convolutional Neural Networks (ConvNets) typically perform best when trained and tested on data from the same distribution. However, in many real-world applications, especially in medical imaging, this assumption often breaks down. Variations in patient data, imaging protocols, or reconstruction methods can lead to significant performance degradation and the generation of artifacts in ConvNet outputs when applied to "out-of-distribution" data. Retraining ConvNets from scratch with new datasets is often impractical due to the expense, time, and difficulty of collecting comprehensive datasets. Existing fine-tuning methods can sometimes lead to forgetting previously learned knowledge or require access to the original training data. This paper addresses the challenge of adapting pre-trained ConvNets to new tasks or data distributions efficiently, without losing prior knowledge and without needing the original training data, while also enabling continuous learning on new, incoming data.
Methods
The core innovation is the Targeted Gradient Descent (TGD) layer. This method aims to identify and retrain only the "meaningless" kernels in a pre-trained ConvNet that contribute less to meaningful feature extraction, while preserving the "useful" kernels that hold valuable learned knowledge.
Here's a breakdown of the TGD approach:
- Kernel Importance Assessment: The paper utilizes the Kernel Sparsity and Entropy (KSE) metric, proposed by Li et al. [16], to quantify the importance of feature maps. KSE considers both the sparsity and information richness of a kernel.
- Identifying "Meaningless" Kernels: Based on the KSE scores, a threshold ($\phi$) is used to identify feature maps that are considered "meaningless." The kernels in the preceding layer responsible for generating these meaningless feature maps are then targeted for retraining.
- Targeted Gradient Update: During backpropagation, a binary mask ($M_n$) is generated. This mask effectively zeros out the gradients for the "useful" kernels, preventing them from being updated. Only the gradients for the "meaningless" kernels are allowed to propagate, thus enabling targeted fine-tuning.
- TGD Layer Integration: The TGD mechanism is implemented as a novel layer that can be inserted after each convolutional and batch normalization layer in a ConvNet architecture. This layer is active during backpropagation but inactive during the forward pass, ensuring all kernels are utilized for feature extraction.
- Online Learning: TGD can be applied iteratively. A network fine-tuned with TGD can be further adapted to a third task, and so on, creating a TGD$^n$ network. This enables online learning, where the network continuously adapts to new data as it becomes available.
Datasets
The primary dataset used in this study is FDG PET human studies.
- v1 Images: These are PET images reconstructed using an older version of the OSEM algorithm. The v1 network was trained on 20 such studies (approximately 30,720 training slices) with various noise levels.
- v2 Images: These are PET images reconstructed using an updated OSEM algorithm, resulting in finer noise textures.
- Fine-tuning/Online Learning Data: For fine-tuning and online learning experiments, significantly smaller datasets were used, consisting of only 7 training datasets from patients with low BMI.
The paper does not introduce new benchmark datasets but rather demonstrates the effectiveness of TGD on existing medical imaging data.
Key Insights and Ideas
- Selective Retraining is Key: The core idea that not all kernels in a deep network are equally important for a new task is crucial. TGD leverages this by selectively updating only the less informative kernels, preserving the valuable knowledge encoded in the others.
- Preserving Prior Knowledge: By targeting specific kernels, TGD effectively avoids catastrophic forgetting, a common issue in fine-tuning where a network might lose its ability to perform its original task.
- Efficiency Gains: The ability to fine-tune with a drastically smaller dataset (7 studies vs. 20 studies) and significantly reduced training time is a major practical advantage.
- Enabling True Online Learning: TGD's iterative nature and its ability to adapt to patient-specific data in real-time (online learning) are critical for dynamic environments like clinical practice, where new data is constantly generated.
- Artifact Reduction: The method's success in preventing artifacts on out-of-distribution data, such as the bladder in Fig. 6, highlights its robustness and generalization capabilities.
Key Results
- Performance Parity: TGD achieved results comparable to training-from-scratch methods for PET image denoising, while using significantly less data and training time.
- Training Time Reduction: Fine-tuning a v1-net using TGD to denoise v2 PET images saved approximately 64% of computational time compared to training a v2-net from scratch.
- Artifact Prevention: In online learning experiments (TGD-N2N), the proposed method successfully alleviated artifacts around out-of-distribution objects (e.g., urinary catheters) that were present in outputs from standard v2-net and TGD-net.
- Quantitative Improvements: For low-BMI patients, TGD achieved the best lesion quantification with a small ensemble bias (-3.77%) and a low noise level (6.45% CoV).
- KSE Threshold Impact: The study found that an optimal KSE threshold ($\phi=0.3$) was crucial for balancing performance and avoiding severe artifacts.
Limitations and Future Work
- KSE Threshold Optimization: While the paper identifies an optimal KSE threshold, this threshold might need to be re-optimized for different tasks or network architectures. The paper mentions adjusting the KSE threshold for TGD-N2N results to achieve similar liver CoV for fair comparison, suggesting this parameter is task-dependent.
- Generalizability to Other Tasks: The study primarily focuses on PET image denoising. While the method is presented as a general fine-tuning technique, its performance on a wider range of computer vision tasks (e.g., segmentation, classification) would be valuable to explore.
- Computational Overhead of KSE: Calculating KSE adds a computational step. While the overall training time is reduced, the per-iteration overhead might be a consideration for extremely large networks or real-time applications where every millisecond counts.
- Future Work: The authors suggest exploring the TGD method for other medical imaging modalities and tasks. They also imply that further investigation into the optimal selection of the KSE threshold and its impact on different network architectures and datasets would be beneficial. The iterative application of TGD (TGD$^n$) also presents an avenue for exploring more complex incremental learning scenarios.
Summary updated