The log-Gabor method: speech classification using spectrogram image analysis

Harm Buisman, Eric PostmaInterspeech 2012

TL;DR

This paper explores using the log-Gabor transform, a technique borrowed from image analysis, to classify personality and likability traits from speech spectrograms. The method treats speech spectrograms as images and applies log-Gabor filters to extract spectro-temporal features, which are then fed into a Support Vector Machine (SVM) for classification. The approach shows promise, outperforming baseline methods on development sets, though performance drops on test sets, suggesting potential overfitting issues.

What Problem Does This Paper Address?

The paper tackles the challenge of automatically inferring human personality and likability traits from speech. While previous research has shown that spectrograms can reveal socially informative cues, the authors investigate a specific method – the log-Gabor transform – for extracting these cues. This method is inspired by its success in visual texture analysis, aiming to leverage its ability to capture local orientations and spatial frequencies, which are hypothesized to correspond to speech characteristics like pitch, rhythm, and articulation. The goal is to determine if this image-analysis-inspired approach can effectively translate spectro-temporal patterns into reliable classifications of speaker traits.

Methods

The core of the proposed method is the log-Gabor transform, adapted from its use in visual image analysis. The process involves several stages:

  1. Spectrogram Generation: The speech signal is first converted into a spectrogram using the Short-Time Fourier Transform (STFT). The temporal resolution is controlled by a parameter Δt.
  2. Log-Gabor Transform: This is the novel application. The spectrogram is treated as an image, and a bank of 2D logarithmic Gabor filters is applied. These filters are designed to capture specific orientations and spatial frequencies within the spectrogram, analogous to how they detect edges and textures in images. The output is an "energy image" for each filter, representing the strength of the spectro-temporal patterns matching the filter's characteristics.
  3. Feature Extraction: Raw features are extracted by summing the energy values from the filtered images across different frequency bands. To manage the potentially large number of features and their correlations, Principal Component Analysis (PCA) is used for dimensionality reduction. Features are normalized before and after PCA.
  4. Classification: The reduced and normalized features are fed into a Support Vector Machine (SVM) with a radial basis function kernel to perform the binary classification (e.g., likable vs. not likable).

Datasets

The study utilizes datasets from the INTERSPEECH 2012 Speaker Trait Challenge, specifically the Personality and Likability Sub-Challenges. These datasets consist of speech samples labeled for likability and for the presence/absence of the "big five" personality dimensions (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism). The datasets are divided into training, development, and test sets. The authors also explore the impact of gender separation by treating male and female speakers as separate subsets.

Key Insights and Ideas

  • Borrowing from Vision: The core idea of treating speech spectrograms as images and applying visual feature extraction techniques like the log-Gabor transform is a key insight. This cross-domain application suggests that fundamental spectro-temporal patterns in speech might share characteristics with visual textures.
  • Log-Gabor Filters Capture Speech Structure: The log-Gabor filters, by their design, are sensitive to oriented structures and spatial frequencies. This aligns well with how speech characteristics like harmonics (vertical orientation, related to pitch) and rhythm (horizontal orientation, related to temporal patterns) manifest in spectrograms.
  • Gender Matters: Separating speech samples by gender generally improved classification performance, indicating that gender-specific vocal characteristics can influence trait prediction. The method also demonstrated a high accuracy in estimating gender itself from speech.
  • Parameter Sensitivity: The performance of the log-Gabor method is sensitive to parameter choices, particularly the time resolution (Δt) of the spectrogram and the number of principal components (k) used for dimensionality reduction. This highlights the need for careful parameter tuning.

Key Results

  • Above Baseline on Labeled Sets: The log-Gabor method achieved above-baseline performance on the development sets for both Likability and Personality classification tasks. For example, on the development set, the average recall for Likability was 67.5% (no gender) and 74.2% (with gender separation), compared to a baseline of 58.5%.
  • Performance Drop on Test Sets: A significant drop in performance was observed on the test sets, with only Likability and Neuroticism performing above baseline. The authors attribute this to potential overfitting caused by optimizing parameters using the development set performance.
  • Gender Estimation Accuracy: The method achieved a high gender estimation accuracy of 96.1% for the Likability dataset and 99.6% for the Personality dataset using 10-fold cross-validation.

Limitations and Future Work

  • Overfitting on Test Sets: The most significant limitation acknowledged is the disappointing performance on the test sets, which is attributed to overfitting due to parameter optimization on the development set. The authors suggest that future parameter optimization should be performed within a cross-validation scheme.
  • Black-Box Nature: The current method is described as a "black-box approach." Future work aims to perform an "inverse mapping" to identify the specific speech features that contribute to successful predictions of likability and personality. This would make the method more interpretable and informative.
  • Further Investigation: Despite the limitations, the authors conclude that the log-Gabor method is a promising approach for speaker trait classification and warrants further investigation.

Summary updated

BibTeX
@inproceedings{buisman2012log,
  author = {Harm Buisman and Eric Postma},
  title = {{The log-Gabor method: speech classification using spectrogram image analysis}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure