Joining datasets via data augmentation in the label space for neural networks

Jake Zhao, Mingfeng Ou, Linji Xue, Yunkai Cui, Sai Wu, Gang Chen

Abstract

Most modern deep learning systems restrict themselves to a single dataset for training and inference. This article explores systematic ways to join datasets created for similar purposes. Unlike previous works that conduct dataset joining in uninterpretable latent vectorial spaces, our method centers on an augmentation procedure in the label space. We address challenges like non-overlapping label sets and varying granularities by leveraging an artificially created knowledge graph, recurrent neural networks, and policy gradients. Empirical results on image and text classification justify the validity of this approach.

Introduction

In deep learning, we often encounter multiple datasets for the same general task, such as image classification, yet they are rarely compatible. For instance, one dataset might distinguish between "Cats vs. Dogs," while another identifies 37 specific breeds. This incompatibility stems from label discrepancy: inconsistent label sets, different semantic hierarchies, or varying levels of granularity.

Traditional methods typically address this within the "black box" of latent vectorial space through transfer learning or adversarial training. However, these approaches often fail to exploit the rich semantic information inherent in the labels themselves and lack transparency in the model's decision-making process.

Methodology: Trajectory Prediction on Label Graphs

The proposed framework shifts the paradigm from predicting a single discrete label to predicting a trajectory on a label graph. This graph-based approach allows for the integration of disparate datasets by mapping their semantic relationships.

1. Knowledge-Driven Label Graph

A graph is constructed where nodes represent labels from all combined datasets, supplemented by "augmented nodes" representing intermediate features (e.g., "tabby color," "long hair"). This structure bridges the gap between coarse and fine-grained labels.

The process of constructing a label graph and defining trajectories from coarse to fine-grained labels.

2. Path Categorization and Training

For any input, the model traverses the graph from a root node to a target leaf node. Paths are categorized as:

  • Deterministic Paths: Fixed hierarchical relationships (e.g., Animal → Cat → British Shorthair), trained using Teacher Forcing to maximize log-likelihood.
  • Nondeterministic Paths: Paths where intermediate nodes may have multiple valid features (e.g., a breed could be "Solid-Color" or "Tabby-Color"). These are handled using REINFORCE (Policy Gradient) to manage uncertainty where ground truth is not explicitly labeled.

3. Model Architecture

The architecture consists of an encoder—EfficientNet-B4 for images or BERT for text—coupled with an RNN-based decoder (specifically a GRU with attention) to generate the path node-by-node. A "Block Softmax" function is employed to limit competition to nodes sharing the same parent.

The proposed architecture featuring an encoder and an RNN-based decoder with attention for sequential node prediction.

Experimental Setup

The approach was evaluated across two modalities:

  • Computer Vision: Using "PetFusion" (Oxford-IIIT Pet joined with Dogs vs. Cats) and "FlowerFusion" (102 Category Flowers joined with 17 Category Flowers).
  • Natural Language Processing: Using the Arxiv dataset for hierarchical multi-label text classification.

Key Results and Insights

The "label-aug" method demonstrated significant performance gains:

  • Image Classification: On Oxford-IIIT Pet, the method achieved 94.66% accuracy, surpassing the standard EfficientNet-B4 baseline (93.84%). The joined "PetFusion" model reached 94.95% accuracy, proving that coarse data can enhance fine-grained recognition.
  • Text Classification: Using BERT, the strategy improved the F1 score from 75.5 to 77.4 on the original Arxiv dataset and to 79.3 on the fused dataset.
  • Interpretability: The model exhibits reasoning capabilities; in manual checks of nondeterministic paths, it correctly identified intermediate features (like "Tabby") over 90% of the time, even without explicit labels for those features during training.

Conclusion

By treating labels as structured knowledge rather than discrete integers, this framework enables the effective joining of datasets with inconsistent granularities. While current graph construction requires some manual effort, the resulting models offer improved accuracy and a transparent decision process, turning the "black box" of neural networks into a traceable sequence of reasoning steps.

Summary updated

Abstract

Most, if not all, modern deep learning systems restrict themselves to a single dataset for neural network training and inference. In this article, we are interested in systematic ways to join datasets that are made of similar purposes. Unlike previous published works that ubiquitously conduct the dataset joining in the uninterpretable latent vectorial space, the core to our method is an augmentation procedure in the label space. The primary challenge to address the label space for dataset joining is the discrepancy between labels: non-overlapping label annotation sets, different labeling granularity or hierarchy and etc. Notably we propose a new technique leveraging artificially created knowledge graph, recurrent neural networks and policy gradient that successfully achieve the dataset joining in the label space. Empirical results on both image and text classification justify the validity of our approach.
BibTeX
@inproceedings{zhao2021joining,
  author = {Jake Zhao and Mingfeng Ou and Linji Xue and Yunkai Cui and Sai Wu and Gang Chen},
  title = {{Joining datasets via data augmentation in the label space for neural networks}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=g3Yo5AueueU},
}
Figure