Overview
This paper introduces a novel framework for merging multiple datasets that have inconsistent labeling systems. Instead of relying on the uninterpretable latent space of neural networks to bridge these datasets, the authors shift the "joining" process directly into the label space. By constructing a knowledge-driven label graph and training models to predict trajectories through this graph, the approach achieves superior performance on fine-grained classification tasks while significantly enhancing model interpretability.
The Challenge: Label Discrepancy
Most deep learning models are trained on a single, isolated dataset. However, in real-world scenarios, we often have access to multiple datasets for the same general task (e.g., two different collections of animal photos). The primary bottleneck to combining them is label discrepancy: one dataset might use coarse labels (like "Dog"), while another uses fine-grained labels (like "Siberian Husky").
Existing solutions, such as transfer learning or domain adaptation, usually operate in the "black-box" vectorial hidden space of the neural network. These methods often fail to exploit the rich semantic relationships between labels and lack transparency. This paper asks a fundamental question: How can we combine datasets directly in the label space to make the model smarter and its decisions more traceable?
Methodology: Predicting Trajectories on a Label Graph
To solve this, the authors propose a paradigm shift from predicting a single, isolated label to predicting a trajectory on a label graph.
The framework operates through several key mechanisms:
1. Knowledge-Driven Label Graph Instead of a flat list of categories, the researchers build a hierarchical graph. This graph includes labels from all datasets being merged, plus "augmented nodes." These augmented nodes represent intermediate features—such as "tabby color" or "longhair"—derived from external domain knowledge (e.g., pet breed websites).
2. Trajectory Prediction The model acts as a navigator. For a given input image or text, it predicts a path from a root node (e.g., "Animal") through intermediate augmented nodes to the final target classification.
3. Path Categorization and Training * Deterministic Paths: These represent fixed relationships (e.g., a specific breed always has short hair). They are trained using standard Maximum Likelihood Estimation (MLE) with Teacher Forcing. * Nondeterministic Paths: These represent uncertain relationships (e.g., a breed could be one of several colors). Because the exact path isn't strictly defined by the final label, these are trained using Policy Gradient (REINFORCE) to handle the ambiguity.
4. Block Softmax To prevent unrelated labels from competing against each other during training, the authors use a modified "Block Softmax." This ensures probabilities are only calculated relative to specific groups (e.g., color traits only compete with other color traits).
5. Architecture The system utilizes standard encoders (EfficientNet-B4 for images, BERT for text) coupled with a Recurrent Neural Network (RNN/GRU) decoder to sequentially generate the label paths.
Datasets and Key Results
The researchers evaluated their approach across both Computer Vision and Natural Language Processing (NLP) modalities:
- Computer Vision: They joined the fine-grained Oxford-IIIT Pet dataset with the coarse Dogs vs. Cats dataset. They also tested merging the 102 Category Flowers and 17 Category Flowers datasets.
- NLP: They used the Arxiv Dataset, leveraging its hierarchical labels (like "cs.machine-learning") to create original, augmented, and fused versions.
The "label-aug" approach consistently outperformed standard end-to-end models and traditional pseudo-labeling methods:
- Vision Accuracy: On the Oxford-IIIT Pet dataset, the proposed method achieved 94.66% accuracy, surpassing both the standard EfficientNet-B4 baseline (93.84%) and a pseudo-labeling approach (91.50%).
- NLP Performance: Using BERT with label augmentation improved F1 scores on the Arxiv Fusion dataset from 76.9 to 79.3.
- Reasoning Capability: In manual checks, the model correctly identified nondeterministic traits (like "tabby color") over 90% of the time, even when those specific traits weren't part of the original ground-truth labels for those images.
Key Insights
- Label Space as a Bridge: By adding intermediate "augmented nodes," the authors create a common language. This allows coarse datasets to provide valuable "weak supervision" for fine-grained tasks.
- Interpretability by Design: Because the model predicts a path, its decision process is transparent. If it misclassifies a cat breed, developers can check if it at least correctly identified the hair length or color pattern.
- Knowledge is Cheaper than Data: Extending a label graph using existing domain knowledge is significantly more cost-effective than manually re-annotating thousands of images to a finer granularity.
Limitations and Future Work
The system relies heavily on the quality of the manually or semi-automatically constructed label graph; flawed domain knowledge will degrade model performance. Additionally, the current work focuses strictly on classification. Extending this trajectory-based paradigm to more complex tasks like object detection or segmentation remains an open challenge. Finally, as the label graph grows larger, managing the training stability of the policy gradient for nondeterministic paths may become difficult.
Future work will likely explore automated ways to construct these graphs and investigate methods to further resolve ambiguities in complex taxonomies.
Summary updated