How do we measure the diversity of a collection of items? Whether evaluating a text-to-image model, forming a team of LLM agents, or training a population of reinforcement learning policies, we need a metric for "variety." Traditional approaches rely on discrete categories (which require manual labels) or average pairwise distances. However, pairwise distances suffer from a fundamental flaw: a population can achieve a high average distance by clustering into a few distant groups, collapsing into a low-dimensional subspace rather than filling the available space.
To solve this, a powerful family of metrics leverages the spectrum (the eigenvalues) of a similarity matrix. By treating a collection of items as a geometric configuration or a physical system, these methods capture the global structure of the data, penalizing redundancy and rewarding coverage across independent dimensions.
The Intuition: Reading the Spectrum
Suppose we have a set of $N$ items represented as vectors in an embedding space. We can compute an $N \times N$ similarity matrix $K$ using a kernel function (such as cosine similarity), where each entry $K_{ij}$ measures how similar item $i$ is to item $j$. The eigenvalues $\lambda_1, \lambda_2, \dots, \lambda_N$ of this matrix reveal the underlying dimensionality and distribution of the collection:
- Perfect Redundancy: If all items are identical, every entry in $K$ is $1$. The matrix has a rank of 1, and its eigenvalues are $[N, 0, 0, \dots, 0]$. All variance is concentrated in a single dimension.
- Perfect Diversity: If all items are completely orthogonal (perfectly distinct), $K$ becomes the identity matrix $I$. Its eigenvalues are $[1, 1, 1, \dots, 1]$. The variance is perfectly distributed across $N$ independent dimensions.
By analyzing these eigenvalues, we can mathematically formalize diversity in two primary ways: geometrically (via the volume of the space spanned) or information-theoretically (via the entropy of the eigenvalue distribution).
The Geometric View: Volume and Determinants
One way to exploit this spectrum is to compute the determinant of the kernel matrix, $\det(K)$, which is equal to the product of its eigenvalues ($\prod_{i=1}^N \lambda_i$). Geometrically, the determinant corresponds to the squared volume of the parallelotope spanned by the item vectors in the feature space. If any two items are identical, the vectors become linearly dependent, the volume collapses to zero, and the determinant vanishes.
In Effective Diversity in Population Based Reinforcement Learning, Jack Parker-Holder et al. introduce Diversity via Determinants (DvD). The authors demonstrate that maximizing this volume forces a population of reinforcement learning agents to explore linearly independent directions rather than clustering in a low-dimensional subspace, significantly improving sample efficiency.
This geometric formulation is deeply connected to Determinantal Point Processes (DPPs), which elegantly model repulsion between items. In Modelling Behavioural Diversity for Learning in Open-Ended Games, Nicolas Perez Nieves et al. utilize the expected cardinality of a subset sampled from a DPP to quantify behavioral diversity, helping break strategic cycles in complex games. Similarly, Yaodong Yang et al. argue in Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systems that measuring policy orthogonality via DPPs is critical for preventing curriculum collapse. For offline settings, Takayuki Osa et al. show in Discovering Multiple Solutions from a Single Task in Offline Reinforcement Learning that maximizing the determinant of a behavior-embedding kernel matrix allows an algorithm to extract multiple distinct, successful strategies from an unlabeled dataset.
The Information-Theoretic View: The Vendi Score
While determinants work well for optimization, they scale poorly as an absolute metric when the population size changes. To create an interpretable, label-free diversity metric, Dan Friedman et al. introduced the Vendi Score in The Vendi Score: A Diversity Evaluation Metric for Machine Learning.
The core insight is to normalize the eigenvalues of the similarity matrix so they sum to 1:
$$p_i = \frac{\lambda_i}{N}$$
Because the eigenvalues are non-negative, $p_i$ forms a valid probability distribution representing how the dataset's "energy" is distributed across different orthogonal features. The authors then compute the Shannon entropy of this distribution, $H(p) = -\sum p_i \log p_i$, and take its exponential to find the effective number of unique elements:
$$\text{Vendi Score} = \exp\left(-\sum_{i=1}^N p_i \log p_i\right)$$
Let's look at how this resolves our two extreme cases:
- For identical items, $p = [1, 0, \dots, 0]$. The entropy is $0$, giving $\exp(0) = 1$ effective item.
- For perfectly orthogonal items, $p = [1/N, 1/N, \dots, 1/N]$. The entropy is $\log N$, giving $\exp(\log N) = N$ effective items.
This framework provides a continuous, interpretable score without requiring predefined categories. In Cousins Of The Vendi Score: A Family Of Similarity-Based Diversity Metrics For Science And Machine Learning, Amey P. Pasarkar et al. generalized this approach by replacing Shannon entropy with Renyi entropy. By adjusting a parameter $q$, researchers can tune the metric's sensitivity to rare outliers or common duplicates. To balance variety with utility, Quan Nguyen et al. developed Quality-Weighted Vendi Scores And Their Application To Diverse Experimental Design, appending a quality score to the similarity matrix so that the effective count only rewards high-performing, distinct items.
A Unified Metric Across Domains
Because spectral diversity metrics depend entirely on the similarity matrix rather than domain-specific features, they have emerged as a unified tool across machine learning. In A Unified Diversity Measure for Multiagent Reinforcement Learning, Zongkai Liu et al. demonstrate that varying the kernel or the function applied to the eigenvalues allows a single spectral framework to consolidate multiple disparate multi-agent diversity metrics.
This mathematical engine is currently being applied to solve evaluation and optimization challenges across several domains:
- Image and Visual Analysis: In Benchmarking Diversity in Image Generation via Attribute-Conditional Human Evaluation, Isabela Albuquerque et al. apply the entropy of similarity matrix eigenvalues to evaluate text-to-image models, showing that metrics built on unimodal image embeddings align closely with human diversity judgments. Similarly, Tarun Tater et al. use eigenvalue analysis in Unveiling the mystery of visual attributes of concrete and abstract concepts: Variability, nearest neighbors, and challenging categories to quantify visual variance across different depictions of semantic concepts.
- Speech and Language Generation: In MAD Speech: Measures of Acoustic Diversity of Speech, Matthieu Futeral et al. compute the entropy of embedding similarity matrices across distinct acoustic facets (such as emotion, accent, and background noise) to detect mode collapse in text-to-speech models. For text generation, Tianhui Zhang et al. show in Evaluating the Evaluation of Diversity in Commonsense Generation that embedding-based eigenvalue metrics capture semantic breadth far better than traditional word-overlap metrics like self-BLEU, which often mistake gibberish for creativity.
- System Design and Uncertainty: In AgentInit: Initializing LLM-based Multi-Agent Systems via Diversity and Expertise Orchestration for Effective and Efficient Collaboration, Chunhao Tian et al. perform eigenvalue analysis on agent descriptions to prune redundant roles and optimize multi-agent team composition. Finally, Lucas Hurley McCabe et al. leverage this spectrum in Estimating Semantic Alphabet Size for LLM Uncertainty Quantification, using spectral graph analysis of model outputs to accurately estimate the true semantic dimensionality of an LLM's response space, providing a highly effective way to identify hallucinations.
Key Foundational Work
Effective Diversity in Population Based Reinforcement Learning (2020) — contributed 3 concept elements to this direction.
Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systems (2021) — contributed 2 concept elements to this direction.
Modelling Behavioural Diversity for Learning in Open-Ended Games (2021) — contributed 1 concept elements to this direction.