Introduction
Deep Generative Models (GMs) have traditionally been evaluated on their ability to replicate training data with high fidelity. While metrics like Fréchet Inception Distance (FID) serve as effective benchmarks for realism, they often fail to account for the necessity of diversity in creative and scientific applications. In domains such as architecture, drug discovery, and game design, the goal is not merely to reproduce existing patterns but to discover novel, high-quality solutions. This paper argues for a paradigm shift from simple distribution fitting toward "Active Divergence," where models are incentivized to explore the boundaries of a domain rather than its center.
The Limitations of Fidelity-Centric Objectives
Current GMs are optimized to capture the breadth of a training set (mode coverage) while maintaining realism (fidelity). However, this "distribution fitting" objective creates a mismatch for multi-solution tasks. By assigning high probability to common features, models effectively smooth over the rare or "weird" features that are often the most valuable in creative workflows. Consequently, models become "trapped" by their training data, struggling to generate artifacts that are truly novel or that represent the full expressivity of a domain.
Methodological Approach: Active Divergence and AutoML
To address these limitations, this research outlines a trajectory focused on three core areas:
- Active Divergence: This involves adjusting algorithmic objectives to consciously produce artifacts that diverge from the training distribution while maintaining a high standard of quality.
- AutoML for Computational Creativity (CC): By integrating Automated Machine Learning—such as automated hyperparameter tuning—into artistic pipelines, technical burdens can be offloaded. This increases the "creative autonomy" of the system, allowing it to explore a wider range of possibilities without constant human intervention.
- Quality Diversity (QD) Comparison: The research evaluates QD algorithms, which search for a diverse set of high-performing solutions, across different search spaces: manually-encoded parameter spaces and learned latent spaces (e.g., via Variational Auto-Encoders).
The Latent Space Bottleneck
A significant finding in this research is the "latent space bottleneck." In a 2D shape optimization study, searching for solutions within a model’s learned latent space (VAE) yielded lower diversity than searching in a manually defined parameter space. This suggests that the compression process inherent in many GMs filters out the expressivity of the domain, binding the search to the most frequent features of the training data. This tension highlights a fundamental challenge: while latent spaces offer efficiency, they may inadvertently suppress the very diversity required for innovation.
Applications and Future Directions
The implications of this work span several high-impact domains where data may be scarce or where novelty is the primary objective:
- Video Game Design: Generating diverse levels (e.g., Mario) to maintain player engagement and challenge.
- Biotechnology: Synthesizing novel nanobody libraries and small molecules where "more of the same" is insufficient for medical breakthroughs.
- Generative Art: Enhancing the autonomy of computational agents in artistic creation by allowing them to move beyond simple imitation.
The primary challenge remains the fidelity-diversity trade-off. Pushing a model toward extreme diversity can result in low-quality "noise." Future work focuses on quantifying the correlation between fidelity and diversity measures and collaborating with domain experts to test these diversity-focused GMs in real-world scenarios, ensuring that models can discover the "novel and useful" rather than just the "different."
Summary updated