TL;DR
HumanGAN is a novel generative adversarial network that replaces the traditional neural network discriminator with human crowdworkers. By doing so, it can model the "perception distribution"—the wide range of data humans find acceptable or natural—which is often much broader than the narrow "real-data distribution" captured by standard GANs.
What Problem Does This Paper Address?
Standard Generative Adversarial Networks (GANs) are designed to mimic a specific dataset. The discriminator's job is to distinguish between "real" training data and "fake" generated data. However, in fields like speech synthesis, "real" isn't the only thing that matters.
Humans can perceive a voice as "natural" even if it doesn't sound exactly like any specific person in a training set. This "human-acceptable" or perception distribution is typically much wider than the distribution of the training data itself. A standard GAN is fundamentally limited because it can never represent anything outside of its training samples. If a synthetic voice sounds perfectly human but isn't "real" according to the training data, a standard discriminator will reject it. HumanGAN seeks to bridge this gap by using actual human judgment to define what is "natural."
Methods
The core innovation is the human-based discriminator. Since you cannot mathematically differentiate through a human being to perform standard backpropagation, the authors treat the crowd of human workers as a "black-box" system.
To train the generator, they employ a technique called Natural Evolution Strategies (NES):
- Perturbation: The generator produces a sample, and the system creates two slightly different versions by adding small amounts of random noise (perturbations).
- Crowdsourcing: Human listeners (via a crowdsourcing platform) compare these two perturbed samples and rate which one sounds more natural on a 5-point scale.
- Gradient Approximation: The difference in these human scores is used to estimate a gradient. This allows the system to determine which "direction" in the data space makes the output sound more human.
- Update: The generator’s parameters are updated using this approximated gradient, effectively "moving" the generator toward the center of human-perceived naturalness.
Datasets
The researchers evaluated HumanGAN on speech naturalness modeling.
- Primary Dataset: The Vowel Database (JVPD), specifically focusing on the Japanese vowel /a/ uttered by 199 female speakers.
- Preprocessing: They used the WORLD vocoder to extract 513-dimensional log spectral envelopes and then used Principal Component Analysis (PCA) to reduce these to a 2-dimensional space for easier visualization and lower experimental costs.
- Evaluation Platform: Lancers, a Japanese crowdsourcing service, was used to gather judgments from 96 listeners.
Key Insights and Ideas
- Perception vs. Reality: The paper visually demonstrates that the area humans consider "natural" is significantly larger than the cluster of "real" data points. This confirms that standard GANs are inherently "too picky" for tasks involving human perception.
- The Crowd as a Function: The authors successfully formulated a way to integrate subjective human preference into a formal machine learning optimization loop. By treating the crowd as a posterior probability estimator ($D(\cdot)$), they turned "gut feelings" into a trainable signal.
- Generalization to "Open Data": One of the most important findings was that the generator didn't just learn to fool humans on the specific noise vectors used during training; it also improved its performance on entirely new, unseen noise inputs.
Key Results
- Qualitative Shift: Visualization of the training process (Fig. 4) shows the generated data points migrating from "dark" areas (low naturalness) to "bright" areas (high naturalness) over just four iterations.
- Quantitative Improvement: The "posterior probability" (a proxy for how natural the speech sounds) increased steadily across iterations.
- Wider Coverage: The HumanGAN successfully represented a distribution that covered a wider range of the feature space than the original training data, proving it can model the "perception distribution" where a basic GAN would fail.
Limitations and Future Work
- Cost and Scalability: Because every training step requires human intervention, the process is expensive and slow. The authors had to limit their experiment to a small generator and 2D data to keep costs manageable.
- Initialization Sensitivity: The model is prone to "gradient vanishing" if the initial generator output is too far away from what humans consider natural (i.e., if it sounds like pure noise, humans can't give a meaningful "better or worse" judgment).
- Mode Collapse: Like all GANs, HumanGAN faces the risk of mode collapse, where the generator learns to produce only a single "perfect" sample rather than a diverse range of natural sounds.
- Future Directions: The authors suggest exploring "human-computer discriminators" that combine the speed of DNNs with the nuance of human judgment, and scaling the system to handle higher-dimensional data.
Summary updated