Training deep nets with imbalanced and unlabeled data

Jeff Berry, Ian Fasel, Luciano Fadiga, Diana ArchangeliInterspeech 2012

TL;DR

This paper tackles the challenge of training deep belief networks (DBNs) on imbalanced datasets, specifically for the novel task of automatically tracing tongue contours in ultrasound speech images. They introduce an entropy-based method for selecting a diverse, yet small, initial training set, significantly reducing human labeling effort and improving DBN performance compared to random sampling.

What Problem Does This Paper Address?

Training deep belief networks (DBNs) typically requires large, pre-labeled datasets. However, in many real-world applications, such as extracting tongue contours from ultrasound images for speech analysis, creating these labeled datasets is prohibitively expensive and time-consuming. The specific task of tongue tracing is further complicated by two factors: the labels are structured contours, not simple categories, and the dataset is inherently imbalanced. This means many images are very similar, while a few represent unique tongue shapes. Imbalanced data can lead DBNs to become biased towards the majority class, performing poorly on under-represented shapes. This paper aims to develop a method that drastically reduces the human effort required for labeling while achieving high accuracy in training DBNs for this imbalanced, contour-based task.

Methods

The core of the paper's contribution lies in a novel approach to selecting an initial training set for DBNs. Instead of random sampling, they employ an entropy-based diversity measure to identify images that capture a wide range of tongue shapes. This involves:

  1. Distance Scoring: Each ultrasound image is assigned a "distance score" based on its pixel intensity differences from an average image of the entire dataset. Higher scores indicate more unique or diverse tongue shapes.
  2. High-Entropy Set Selection: They select a subset of images with the highest distance scores to represent diverse shapes. To ensure common shapes aren't entirely missed, they also include a small number of the lowest-scoring images. This combined set forms a "high-entropy" training set.
  3. Bootstrapping: After training an initial DBN (specifically a translational DBN or tDBN) on this high-entropy set, the trained DBN is used to label the remaining images. These machine-generated labels are then reviewed, and the most erroneous ones are hand-corrected. These corrected labels are added to the training set, and the DBN is retrained. This iterative process, called bootstrapping, further refines the model and reduces the need for extensive initial human labeling.

Datasets

The primary dataset used in this study consists of 3,209 ultrasound images of the tongue surface, captured from 76 short video clips of Italian words spoken in isolation by a single female speaker. The labels are hand-traced contours of the tongue surface. This dataset is characterized by its imbalanced nature, with many similar tongue shapes and fewer unique ones.

Key Insights and Ideas

  • Entropy as a Diversity Metric: The paper demonstrates that using an entropy-based measure (derived from pixel intensity differences) is a powerful way to select a diverse initial training set for DBNs, effectively addressing data imbalance.
  • Bootstrapping for Efficiency: The iterative bootstrapping procedure significantly reduces the overall human labeling time by leveraging the DBN's own predictions and focusing human effort on correcting the most critical errors.
  • Imbalance is a Major Bottleneck: The study highlights that imbalanced data is a significant hurdle for DBNs, potentially leading them to suboptimal solutions, and that addressing this imbalance is crucial for practical applications.
  • Contour Tracing is Challenging: The task of tracing structured contours, rather than simple class labels, presents unique challenges for machine learning models, making the proposed method particularly valuable.

Key Results

  • The high-entropy training set resulted in a median Mean Sum of Distances (MSD) score of 1.69 pixels, compared to 2.61 pixels for a randomly selected training set. This indicates significantly more accurate tongue contour tracing.
  • A t-test confirmed that the difference in MSD scores between the high-entropy and random training sets was statistically significant (p < 0.001).
  • The bootstrapping procedure reduced the total human time required to label the entire dataset from 7.62 hours to 6.03 hours.
  • The authors estimate that their method allows the entire dataset to be labeled in less than 8 human hours, a substantial reduction from the nearly 18 hours it would take to hand-trace all images.

Limitations and Future Work

  • Optimal Training Set Size: While the study identified an optimal initial training set size of 450 images for balancing labeling time and accuracy, the paper notes that further iterations of bootstrapping did not yield significant improvements. This suggests that adding more images beyond a certain point might reintroduce imbalance issues or that the current dataset size might be limiting further gains.
  • Iterative Bootstrapping Plateau: The authors suspect that adding more images in the bootstrapping process might cause the natural imbalance of the data to surface, leading the network to a non-optimal local minimum.
  • Future Work: The paper suggests that this approach is highly promising for other real-world applications dealing with imbalanced data and structured labels, implying that further exploration of this methodology across different domains would be beneficial.

Summary updated

BibTeX
@inproceedings{berry2012training,
  author = {Jeff Berry and Ian Fasel and Luciano Fadiga and Diana Archangeli},
  title = {{Training deep nets with imbalanced and unlabeled data}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure