Abstract
Generative Adversarial Networks (GANs) are powerful generative models but are notoriously difficult to train due to issues like vanishing gradients and mode collapse. This paper introduces a simple yet effective architectural modification to stabilize GAN training: replacing the discriminator's final fully-connected layer with a soft decision forest. By improving the numerical conditioning of the optimization process, this method yields significantly sharper images and improves Fréchet Inception Distance (FID) scores by up to 56% across standard benchmarks.
Introduction: The Challenge of GAN Stability
Generative Adversarial Networks (GANs) are renowned for their ability to synthesize realistic data, yet they remain notoriously difficult to train. Common failure modes include vanishing gradients, mode collapse, and general numerical instability. While existing research often focuses on modifying loss functions (e.g., Wasserstein loss) or applying strict regularization (e.g., Spectral Normalization), this work investigates the architectural bottlenecks of the discriminator.
The authors argue that the standard final layer of a discriminator—a fully-connected (FC) linear layer—is fundamentally ill-equipped to interpret the highly non-linear, complex correlations inherent in image data. This mismatch leads to a poorly conditioned Hessian matrix, making the optimization process fragile and prone to failure.
Methods: The Soft Decision Forest Discriminator
The proposed solution involves replacing the final FC layer of the discriminator with a Soft Decision Forest. This architectural shift introduces several key mechanisms:
- Soft Routing: Unlike traditional decision trees that use "hard" binary splits, this model employs "soft" nodes. Each node uses a sigmoid function to determine the probability of an input moving to a specific child node.
- Differentiability: The probabilistic nature of the routing ensures the entire forest is a continuous, differentiable function, allowing for end-to-end training via standard backpropagation.
- Ensembling: The discriminator's final output is the average of multiple soft decision trees. This ensemble acts as a robust interpreter of the features extracted by preceding convolutional layers.
Key Insights and Theoretical Advantages
The core advantage of the decision forest lies in its ability to model complex, joint distributions of image features that a linear layer struggles to separate. Mathematically, this structure improves the conditioning of the Hessian matrix. By lowering the "condition number" for the Gauss-Newton matrix, the training landscape becomes smoother, allowing the optimizer to take more efficient steps toward a stable solution.
Importantly, this technique is orthogonal to other stabilization methods. It can be integrated into existing frameworks like SNGAN or WGAN-GP to provide additive benefits.
Experimental Results
The researchers evaluated their approach on four diverse datasets: MNIST, CIFAR-10, Oxford Flowers (102), and CUB Birds (200). The results consistently outperformed baseline models across all metrics:
- FID Improvements: On the Oxford Flowers dataset, the forest-based WGAN-GP improved the FID score from 80.4 to 35.3—a massive leap in quality.
- Image Quality: Qualitative results on CUB Birds and Oxford Flowers showed noticeably more detail and fewer blurry artifacts compared to standard GANs.
- Convergence and Stability: Models reached higher Inception Scores (IS) faster and exhibited less erratic behavior in loss curves.
Limitations and Future Work
While effective, the method requires careful parameter matching to ensure fair comparisons with standard FC layers. Managing the memory footprint of very large forests in high-resolution models (e.g., 1024x1024) remains a challenge for future scaling. Additionally, while a better discriminator improves the generator's performance, future work could explore whether similar non-linear structures could be applied directly to the generative side of the network.
Summary updated