TL;DR
This paper introduces "Label-Assemble," a novel framework that effectively leverages multiple, partially labeled public datasets for improved disease diagnosis and detection. By learning from "negative examples" (related but not the primary class of interest), the method significantly boosts performance, especially for novel or rare diseases where positive labels are scarce.
What Problem Does This Paper Address?
Deep learning models thrive on large, meticulously labeled datasets. However, in many real-world scenarios, especially in medical imaging, we often encounter numerous smaller datasets with "partial labels." These labels can be inconsistent, heterogeneous, or even conflicting, making it challenging to combine them effectively. The core problem this paper tackles is how to unlock the full potential of these fragmented, partially labeled public datasets to build more robust and accurate computer-aided diagnosis and detection systems, particularly when dealing with novel diseases where obtaining extensive positive labels is difficult.
Methods
The "Label-Assemble" framework introduces a dynamic adapter with learnable "class queries." Unlike traditional classification that requires a fixed set of categories, this approach uses a question-answering-like mechanism to encode different visual tasks, allowing for flexible integration of partial labels across datasets. A shared CNN extracts image features, which are then processed by a prediction head that integrates these features with learned class queries.
To handle missing labels and bridge domain gaps between datasets, the framework employs:
- Pseudo-labeling: A sharpening operator generates "pseudo-labels" for unlabeled data, allowing the model to perform self-training.
- Consistency Constraints: These ensure that the model's predictions remain consistent across weakly and strongly augmented versions of the same image, further leveraging unlabeled data and mitigating domain shifts.
The overall loss function combines binary cross-entropy for annotated labels with pseudo-labeling and consistency losses for unlabeled data.
Datasets
The paper evaluates "Label-Assemble" on a diverse set of datasets:
- General Vision Datasets: MNIST, CIFAR10.
- Medical Datasets:
- COVIDx CXR-2
- CheXpert
- ChestX-ray14
- DermaMNIST
- TissueMNIST
- OrganAM NIST
- RetinaMNIST
- Two private medical datasets (JHH and HMS)
These datasets vary in size, annotation quality, and the specific diseases they cover, making them suitable for testing the framework's ability to handle heterogeneity and partial labels.
Key Insights and Ideas
- The Power of "Negative Examples": The most striking insight is that learning from classes that are not the primary class of interest (i.e., "negative examples") significantly improves the performance on the target class. This is counter-intuitive, as one might expect only labels for the target class to be beneficial.
- Pathological Similarity Matters: The performance gain from assembling labels is positively correlated with the pathological similarity between the assembled classes and the target class. This suggests a targeted approach to dataset assembly can yield even better results.
- Novel Disease Diagnosis Boost: This approach is particularly valuable for novel diseases where positive examples are rare. By leveraging related diseases from existing datasets, researchers can accelerate diagnosis and detection.
- Eliminating the Need for Full Labels: The framework demonstrates that it's possible to achieve performance comparable to models trained on fully labeled datasets, even when using a significantly smaller set of partially labeled data, thus reducing annotation costs.
Key Results
- COVID-19 Diagnosis: Assembling labels from ChestX-ray14 improved COVID-19 diagnosis accuracy from 96.3% to 99.3%.
- Pancreatic Cancer Detection (PDAC): Sensitivity for PDAC detection increased from 52.1% to 84.0% by leveraging labels for Cysts and PanNETs, while maintaining high specificity (98.0%).
- State-of-the-Art Performance: On the NIH ChestX-ray14 dataset, "Label-Assemble" achieved a new state-of-the-art mean AUC (mAUC) of 0.832 across 14 thorax diseases, outperforming previous methods on 13 out of 14 diseases.
- Competitive with Full Labels: The method performed on par with a model trained on 105,434 full labels using only 75,310 partial labels, saving significant annotation effort.
Limitations and Future Work
- Pathological Similarity Guidance: While the paper shows pathological similarity is beneficial, it suggests more investigation is needed to systematically understand and leverage this relationship.
- Pseudo-labeling Effectiveness: The paper notes that pseudo-labeling and consistency constraints can largely eliminate the requirement for similar diseases, but further exploration into the nuances of this is warranted.
- Foundation for Large-Scale Datasets: The authors position "Label-Assemble" as a step towards creating large-scale, multi-center, fully-labeled medical datasets, implying future work will focus on building and utilizing such resources.
Summary updated