Quantifying Diversity via Similarity Matrix Eigenvalues

21 concepts14 papers1 proposal

How do we measure the diversity of a collection of items? Whether evaluating a text-to-image model, forming a team of LLM agents, or training a population of reinforcement learning policies, we need a metric for "variety." Traditional approaches rely on discrete categories (which require manual labels) or average pairwise distances. However, pairwise distances suffer from a fundamental flaw: a population can achieve a high average distance by clustering into a few distant groups, collapsing into a low-dimensional subspace rather than filling the available space.

To solve this, a powerful family of metrics leverages the spectrum (the eigenvalues) of a similarity matrix. By treating a collection of items as a geometric configuration or a physical system, these methods capture the global structure of the data, penalizing redundancy and rewarding coverage across independent dimensions.

The Intuition: Reading the Spectrum

Suppose we have a set of $N$ items represented as vectors in an embedding space. We can compute an $N \times N$ similarity matrix $K$ using a kernel function (such as cosine similarity), where each entry $K_{ij}$ measures how similar item $i$ is to item $j$. The eigenvalues $\lambda_1, \lambda_2, \dots, \lambda_N$ of this matrix reveal the underlying dimensionality and distribution of the collection:

  • Perfect Redundancy: If all items are identical, every entry in $K$ is $1$. The matrix has a rank of 1, and its eigenvalues are $[N, 0, 0, \dots, 0]$. All variance is concentrated in a single dimension.
  • Perfect Diversity: If all items are completely orthogonal (perfectly distinct), $K$ becomes the identity matrix $I$. Its eigenvalues are $[1, 1, 1, \dots, 1]$. The variance is perfectly distributed across $N$ independent dimensions.

By analyzing these eigenvalues, we can mathematically formalize diversity in two primary ways: geometrically (via the volume of the space spanned) or information-theoretically (via the entropy of the eigenvalue distribution).

The Geometric View: Volume and Determinants

One way to exploit this spectrum is to compute the determinant of the kernel matrix, $\det(K)$, which is equal to the product of its eigenvalues ($\prod_{i=1}^N \lambda_i$). Geometrically, the determinant corresponds to the squared volume of the parallelotope spanned by the item vectors in the feature space. If any two items are identical, the vectors become linearly dependent, the volume collapses to zero, and the determinant vanishes.

In Effective Diversity in Population Based Reinforcement Learning, Jack Parker-Holder et al. introduce Diversity via Determinants (DvD). The authors demonstrate that maximizing this volume forces a population of reinforcement learning agents to explore linearly independent directions rather than clustering in a low-dimensional subspace, significantly improving sample efficiency.

This geometric formulation is deeply connected to Determinantal Point Processes (DPPs), which elegantly model repulsion between items. In Modelling Behavioural Diversity for Learning in Open-Ended Games, Nicolas Perez Nieves et al. utilize the expected cardinality of a subset sampled from a DPP to quantify behavioral diversity, helping break strategic cycles in complex games. Similarly, Yaodong Yang et al. argue in Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systems that measuring policy orthogonality via DPPs is critical for preventing curriculum collapse. For offline settings, Takayuki Osa et al. show in Discovering Multiple Solutions from a Single Task in Offline Reinforcement Learning that maximizing the determinant of a behavior-embedding kernel matrix allows an algorithm to extract multiple distinct, successful strategies from an unlabeled dataset.

The Information-Theoretic View: The Vendi Score

While determinants work well for optimization, they scale poorly as an absolute metric when the population size changes. To create an interpretable, label-free diversity metric, Dan Friedman et al. introduced the Vendi Score in The Vendi Score: A Diversity Evaluation Metric for Machine Learning.

The core insight is to normalize the eigenvalues of the similarity matrix so they sum to 1:

$$p_i = \frac{\lambda_i}{N}$$

Because the eigenvalues are non-negative, $p_i$ forms a valid probability distribution representing how the dataset's "energy" is distributed across different orthogonal features. The authors then compute the Shannon entropy of this distribution, $H(p) = -\sum p_i \log p_i$, and take its exponential to find the effective number of unique elements:

$$\text{Vendi Score} = \exp\left(-\sum_{i=1}^N p_i \log p_i\right)$$

Let's look at how this resolves our two extreme cases:

  • For identical items, $p = [1, 0, \dots, 0]$. The entropy is $0$, giving $\exp(0) = 1$ effective item.
  • For perfectly orthogonal items, $p = [1/N, 1/N, \dots, 1/N]$. The entropy is $\log N$, giving $\exp(\log N) = N$ effective items.

This framework provides a continuous, interpretable score without requiring predefined categories. In Cousins Of The Vendi Score: A Family Of Similarity-Based Diversity Metrics For Science And Machine Learning, Amey P. Pasarkar et al. generalized this approach by replacing Shannon entropy with Renyi entropy. By adjusting a parameter $q$, researchers can tune the metric's sensitivity to rare outliers or common duplicates. To balance variety with utility, Quan Nguyen et al. developed Quality-Weighted Vendi Scores And Their Application To Diverse Experimental Design, appending a quality score to the similarity matrix so that the effective count only rewards high-performing, distinct items.

A Unified Metric Across Domains

Because spectral diversity metrics depend entirely on the similarity matrix rather than domain-specific features, they have emerged as a unified tool across machine learning. In A Unified Diversity Measure for Multiagent Reinforcement Learning, Zongkai Liu et al. demonstrate that varying the kernel or the function applied to the eigenvalues allows a single spectral framework to consolidate multiple disparate multi-agent diversity metrics.

This mathematical engine is currently being applied to solve evaluation and optimization challenges across several domains:

Key Foundational Work

Effective Diversity in Population Based Reinforcement Learning (2020) — contributed 3 concept elements to this direction.
Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systems (2021) — contributed 2 concept elements to this direction.
Modelling Behavioural Diversity for Learning in Open-Ended Games (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_eigenvalue_based_diversity_measurement_fe0a1272,
  title = {eigenvalue-based diversity measurement},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/eigenvalue-based-diversity-measurement/rdr_fe0a12728d5d41a29f9e8e371316df62},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_eigenvalue_based_diversity_measurement_fe0a1272_v1,
  title = {eigenvalue-based diversity measurement},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/eigenvalue-based-diversity-measurement/rdr_fe0a12728d5d41a29f9e8e371316df62/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake