HumanACGAN: conditional generative adversarial network with human-based auxiliary classifier and its evaluation in phoneme perception

Yota Ueda, Kazuki Fujii, Yuki Saito, Shinnosuke Takamichi, Yukino Baba, Hiroshi Saruwatari

Abstract

Standard Generative Adversarial Networks (GANs) are typically designed to mimic static datasets, often overlooking the broader range of data that humans find perceptually acceptable. This paper introduces HumanACGAN, a conditional generative model that incorporates human perception directly into the training loop. By utilizing humans as both discriminators of naturalness and classifiers of categories, the model learns to generate data—specifically phonemes—that align with human-acceptable distributions rather than just real-data distributions.

Introduction: Beyond Real-Data Distributions

Standard GANs aim to replicate a "real-data distribution." For instance, a GAN trained on cat photos attempts to produce images statistically indistinguishable from the training set. However, human perception is more flexible; we often accept "out-of-distribution" data, such as stylized cartoons or synthesized voices, as natural and recognizable. This broader range is termed the human-acceptable distribution.

While the previously proposed HumanGAN could model this distribution unconditionally, it lacked the ability to generate specific categories on demand. HumanACGAN addresses this by adding a conditional layer, allowing human guidance to ensure generated data meets standards for both naturalness and category accuracy (e.g., specific vowel sounds).

Methods: Humans in the Loop

The HumanACGAN architecture extends the Auxiliary Classifier GAN (ACGAN) by replacing the automated Discriminator and Classifier with human listeners. Since humans are non-differentiable, the authors employed a Natural Evolution Strategy (NES) to approximate gradients:

  1. Perturbation: The generator produces a sound ($x$), followed by two slightly modified versions ($x + \Delta x$ and $x - \Delta x$).
  2. Human Evaluation: Crowdsourced listeners rate these versions based on Naturalness (human-like quality) and Class Acceptability (phoneme identity).
  3. Gradient Approximation: The system estimates the "direction" for improvement by comparing human preferences between the tweaks.
  4. Update: The generator’s weights are updated using this approximated gradient, effectively mapping the human internal perception of phonemes.
The HumanACGAN training loop using human listeners for gradient approximation.

Experimental Setup and Datasets

The study focused on Japanese phoneme perception:

  • Baseline Data: The Japanese Vowel Database (JVPD), featuring 199 female speakers, was used for feature normalization.
  • Acoustic Features: 513-dimensional log spectral envelopes, fundamental frequency (F0), and aperiodicities were extracted via the WORLD vocoder.
  • Human Participants: Training signals were derived from 105 human listeners, with an additional 160 listeners participating in the final evaluation.

Key Insights

  • Expanded Acceptability: The range of sounds humans find "natural" and "correct" for specific vowels is significantly broader than the distribution of recorded speech. AI trained solely on "real" data is unnecessarily constrained.
  • Multi-Task Perceptual Optimization: Humans can effectively judge naturalness and category identity simultaneously, allowing the model to optimize for dual perceptual goals.
  • Black-Box Feasibility: The success of the NES algorithm demonstrates that subjective human judgment can serve as a mathematical objective function for training complex Deep Neural Networks (DNNs).

Results and Discussion

The HumanACGAN successfully converged, with generator outputs moving toward regions of higher human posterior probability. These "human-acceptable" maps for vowels like /i/ and /e/ were visually and statistically more expansive than the "real-data" clusters from the JVPD dataset.

Heatmaps showing the human-acceptable distribution of vowels /i/ and /e/ compared to real data.

Furthermore, the model demonstrated generalization by improving the quality of noise samples not seen during training. This proves that the generator learned general rules of human perception rather than simply memorizing specific samples.

Limitations and Future Work

Despite its success, HumanACGAN faces scalability challenges due to the cost and speed of human-in-the-loop training. The current study also relied on dimensionality reduction (PCA) to make the task manageable. Future research will focus on scaling the system to higher-dimensional data and more complex tasks, such as full text-to-speech synthesis, while addressing standard GAN issues like mode collapse.

Summary updated

Abstract

We propose a conditional generative adversarial network (GAN) incorporating humans’ perceptual evaluations. A deep neural network (DNN)-based generator of a GAN can represent a real-data distribution accurately but can never represent a human-acceptable distribution, which are ranges of data in which humans accept the naturalness regardless of whether the data are real or not. A Human-GAN was proposed to model the human-acceptable distribution. A DNN-based generator is trained using a human-based discriminator, i.e., humans’ perceptual evaluations, instead of the GAN’s DNN-based discriminator. However, the HumanGAN cannot represent conditional distributions. This paper proposes the HumanACGAN, a theoretical extension of the HumanGAN, to deal with conditional human-acceptable distributions. Our HumanACGAN trains a DNN-based conditional generator by regarding humans as not only a discriminator but also an auxiliary classifier. The generator is trained by deceiving the human-based discriminator that scores the unconditioned naturalness and the human-based classifier that scores the class-conditioned perceptual acceptability. The training can be executed using the backpropagation algorithm involving humans’ perceptual evaluations. Our experimental results in phoneme perception demonstrate that our HumanACGAN can successfully train this conditional generator.
BibTeX
@inproceedings{ueda2021humanacgan,
  author = {Yota Ueda and Kazuki Fujii and Yuki Saito and Shinnosuke Takamichi and Yukino Baba and Hiroshi Saruwatari},
  title = {{HumanACGAN: conditional generative adversarial network with human-based auxiliary classifier and its evaluation in phoneme perception}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=GeH3nmQ-ok},
}
Figure