Abstract
Von Neumann entropy (VNE) is a fundamental quantity in quantum information theory and has recently been adopted in machine learning as a spectral measure of diversity for kernel matrices and kernel covariance operators. While maximizing VNE under constraints is well known in quantum settings, a principled analogue of the classical maximum entropy framework, particularly its decision theoretic and game theoretic interpretation, has not been explicitly developed for VNE in data driven contexts. In this paper, we extend the minimax formulation of the maximum entropy principle due to Grünwald and Dawid to the setting of von Neumann entropy, providing a game-theoretic justification for VNE maximization over density matrices and trace-normalized positive semidefinite operators. This perspective yields a robust interpretation of maximum VNE solutions under partial information and clarifies their role as least committed inferences in spectral domains. We then illustrate how the resulting Maximum VNE principle applies to modern machine learning problems by considering two representative applications, selecting a kernel representation from multiple normalized embeddings via kernel-based VNE maximization, and completing kernel matrices from partially observed entries. These examples demonstrate how the proposed framework offers a unifying information-theoretic foundation for VNE-based methods in kernel learning.
1. Introduction
In classical statistics, the Maximum Entropy Principle, famously championed by E.T. Jaynes, provides a rigorous method for making inferences based on partial information. It dictates that one should choose the probability distribution that satisfies known constraints while remaining as uncertain as possible. This "least-committal" approach prevents the introduction of unsupported assumptions into a model.
Recently, machine learning has begun utilizing von Neumann Entropy (VNE)—a quantum mechanical analogue of Shannon entropy—to quantify the diversity of data representations. In this context, normalized kernel matrices and covariance operators are treated as quantum density matrices. However, a theoretical gap remained: how can we rigorously justify maximizing VNE when faced with incomplete data or multiple candidate representations? This work addresses this by establishing a formal "Maximum VNE Principle" grounded in game theory.
2. Theoretical Minimax Framework
The authors extend the minimax formulation of maximum entropy, originally proposed by Grünwald and Dawid, to the domain of matrix-valued states. They frame the problem as a zero-sum game between "Nature" and a "Decision Maker." Nature selects an unknown density matrix from a set of constrained possibilities, while the Decision Maker chooses an action to minimize a quantum log loss.
The core theoretical contribution is the proof that maximizing VNE is mathematically equivalent to finding the minimax equilibrium of this game. This provides a robust justification for VNE as the optimal objective for spectral inference under uncertainty. Furthermore, the framework generalizes to a broader family of trace entropies, such as Rényi entropies, through the use of matrix Bregman divergences.
3. Applications in Representation Learning
The Maximum VNE Principle is applied to two distinct challenges in modern machine learning:
3.1 Kernel Mixture Selection
When multiple pretrained embeddings (e.g., OpenCLIP, DINOv2, SigLIP) are available, selecting the best one or combining them effectively is non-trivial. The authors propose using projected gradient ascent to find the optimal convex combination of their respective kernel matrices that maximizes VNE. This process naturally promotes spectral diversity, forcing the resulting mixture to capture a wider array of features than any single constituent model.
3.2 Kernel Matrix Completion
In scenarios where a kernel matrix has missing entries, the principle suggests completing the matrix by maximizing VNE subject to the observed entry constraints. To handle the computational demands of large matrices, the authors employ an order-2 Rényi entropy surrogate and a low-rank factorization approach, making the optimization feasible for practical datasets.
4. Experimental Results
The proposed methods were evaluated across several standard image benchmarks, including ImageNet-1K, CIFAR-100, and the AFHQ dataset.
4.1 Embedding Mixtures
The Max-VNE selected mixture consistently outperformed individual state-of-the-art embeddings in linear probing tasks. As shown in the table below, the mixture achieved superior accuracy across all tested datasets. Notably, on CIFAR-100, the Max-VNE mixture reached 91.5% accuracy, surpassing the best single model (DINOv2 at 89.9%). On ImageNet, it achieved 85.1% compared to SigLIP's 81.8%.
4.2 Kernel Completion
The completion method demonstrated remarkable robustness, recovering meaningful clustering structures even when 90% of the kernel matrix's off-diagonal entries were masked. On the AFHQ dataset, spectral clustering on the recovered kernel achieved a Normalized Mutual Information (NMI) of 0.93 and an accuracy of 98%. On MNIST, the method achieved 0.67 NMI and 74% accuracy from only 10% observed entries.
5. Discussion and Future Work
While the Maximum VNE Principle provides a powerful theoretical foundation, computational complexity remains a hurdle. Exact VNE calculation requires full eigendecomposition ($O(N^3)$), which limits scalability for very large datasets. Future research may focus on more efficient optimization techniques or further exploration of alternative trace entropies. Additionally, the extension of this framework to conditional VNE could offer new insights into diversity evaluation for generative AI models.
Summary updated