TL;DR
Traditional statistical distances (like Kullback-Leibler or Bhattacharyya) output a single scalar value, losing granular details about how individual features relate to one another. This paper introduces a mathematical "de-trace" operation to convert these scalar distances into full matrices, and then applies hierarchical clustering to them. The resulting visualizations form symmetrical, radiating patterns dubbed "Information Mandalas," offering an intuitive, viewpoint-invariant way to understand feature differences and potentially shedding light on the inner workings of neural networks.
What Problem Does This Paper Address?
In machine learning, we constantly measure the distance between two probability distributions to classify data or cluster observations. Tools like the Mahalanobis distance or Kullback-Leibler (KL) divergence are standard for this. However, they have a major limitation: they output a single scalar number.
While a scalar is great for representing the global distance between two images, it completely hides the local relationships between individual features (like pixel-to-pixel differences). If you want to know exactly which parts of a feature space are driving the difference between a picture of a cat and a dog, a single number won't help. The author tackles the problem of upgrading these traditional scalar metrics into a matrix format to preserve and analyze this high-dimensional, local feature data.
Methods
The author introduces a clever mathematical workaround to generate a Statistical Distance Matrix:
- The "De-Trace" Operation: The author takes the standard formulas for well-known statistical distances (Mahalanobis, Bhattacharyya, Chernoff, and KL divergence) and manipulates them into a "trace" form. (The trace of a matrix is the sum of its diagonal elements, which results in a scalar). By simply dropping the trace operator, they extract the underlying, uncollapsed matrix.
- Distance-Matrix-Imaging: To make sense of this massive matrix, the author sums up the local distances related to each individual pixel to create a 2D "distance-accumulation image." This allows us to literally see the matrix.
- Hierarchical Clustering: Finally, standard agglomerative hierarchical clustering is applied to group the pixels based on their statistical distance matrix values, revealing hidden geometric structures.
Datasets
- CIFAR-10: The author used subsets of this famous dataset (60,000 32x32 color images). Specifically, they compared Airplanes vs. Dogs (low similarity), Birds vs. Dogs (medium similarity), and Cats vs. Dogs (high similarity) to test how the matrices handle varying degrees of visual overlap.
Key Insights and Ideas
- Not all distance matrices are created equal: The Bhattacharyya ($D_B$) and Chernoff ($D_C$) matrices worked beautifully, while the Mahalanobis and KL divergence matrices looked like chaotic static. The author realized this is because $D_B$ and $D_C$ contain a specific logarithmic covariance ratio term. When the means of two distributions overlap heavily (e.g., similar images), this covariance term becomes the dominant factor in successfully separating them.
- The "Mandala" Effect: When you cluster the pixels based on the $D_B$ distance matrix, they don't group randomly. They naturally organize into concentric, symmetrical rings radiating from the center of the image. Because these patterns look strikingly similar to traditional religious Mandalas, the author coined the term "Information Mandala."
- Connections to Capsule Networks: The author notes that these statistical distance matrices behave similarly to the routing weights in Capsule Networks (CapsNets). They are viewpoint-invariant (they can distinguish an object regardless of its pose) but offer a much more visually intuitive representation of how the model is distinguishing objects.
Key Results
- Visualizing Differences: The distance-accumulation images successfully highlighted the most discriminative pixels between two classes, which naturally concentrated in the center of the images.
- Similarity Scaling: For highly dissimilar classes (Airplanes vs. Dogs), the high-value (most different) pixels were intensely concentrated in a tight central cluster. For highly similar classes (Cats vs. Dogs), the high-value pixels were fewer and spread out more broadly alongside medium-value pixels.
- Clustering: Clustering the $D_B$ matrix into 3 or 10 groups yielded perfectly symmetrical, geometric "Mandala" patterns that map out the structural differences between the object classes in a highly interpretable way.
Limitations and Future Work
- Mathematical Limitations: The "de-trace" trick doesn't work for every statistical distance. For example, the Hellinger distance cannot be converted into a complete trace form, meaning it is impossible to generate a Hellinger distance matrix using this specific method.
- Future Work: The author suggests that mapping these matrices into tree structures (via clustering) can be used to prune unimportant edges in large graphs. This could massively accelerate computation speeds and save memory in complex ML tasks. Furthermore, they propose the Information Mandala as a new form of entropy that could help researchers demystify the "black box" feature extraction processes of Convolutional Neural Networks (CNNs).
Summary updated