Information Mandala: Statistical Distance Matrix with Clustering

Xin Lu

Introduction

Traditional statistical distances, such as Kullback-Leibler (KL) divergence or Bhattacharyya distance, typically compress high-dimensional differences into a single scalar value. While useful for global classification, this compression loses granular details about how individual features relate to one another. This research introduces a mathematical "de-trace" operation to expand these scalar distances into full matrices. By applying hierarchical clustering to these matrices, the authors reveal symmetrical, radiating patterns termed "Information Mandalas," which provide a viewpoint-invariant method for understanding feature-level differences in neural networks.

Methodology: The De-Trace Operation

The core innovation lies in converting standard parametric distance formulas into a matrix format. Most statistical distances are defined using a trace operator, which sums the diagonal elements of a matrix to produce a scalar. By removing this operator, the underlying "uncollapsed" matrix is preserved.

Parametric formulas for the statistical distance matrices $D_M, D_B, D_C,$ and $D_{KL}$.

The paper focuses on four primary distances: Mahalanobis ($D_M$), Bhattacharyya ($D_B$), Chernoff ($D_C$), and Kullback-Leibler ($D_{KL}$). To visualize these massive matrices, the author employs "Distance-Matrix-Imaging," summing local distances for each pixel to create a 2D "distance-accumulation image."

Experimental Results

The method was tested using the CIFAR-10 dataset, comparing class pairs with varying degrees of visual similarity: Airplane vs. Dog (low), Bird vs. Dog (medium), and Cat vs. Dog (high).

A key finding is that not all distance matrices are equally effective for visualization. As shown below, the Bhattacharyya ($D_B$) and Chernoff ($D_C$) matrices exhibit clear structural patterns, whereas Mahalanobis ($D_M$) and KL divergence ($D_{KL}$) appear as chaotic noise.

Visual comparison of distance matrices for different class pairs. $D_B$ and $D_C$ show clear structural patterns compared to the noise in $D_M$ and $D_{KL}$.

This discrepancy arises because $D_B$ and $D_C$ incorporate a logarithmic covariance ratio. When class means overlap significantly (e.g., Cats vs. Dogs), this covariance term becomes the primary driver for successful class separation. In highly dissimilar classes like Airplanes vs. Dogs, discriminative pixels are intensely concentrated in the center, while similar classes show a broader, more diffuse spread.

The Information Mandala

When hierarchical clustering is applied to the $D_B$ matrix, pixels naturally organize into concentric, symmetrical rings. These geometric structures, dubbed "Information Mandalas," map out the structural differences between object classes in a highly interpretable way. The author notes that these matrices behave similarly to routing weights in Capsule Networks (CapsNets), offering viewpoint invariance while remaining visually intuitive.

Conclusion and Future Directions

While the "de-trace" method is powerful, it has limitations; for instance, the Hellinger distance cannot be easily converted into a trace form. Future work involves using these matrix-derived tree structures to prune unimportant edges in large graphs, potentially accelerating complex machine learning tasks. Furthermore, the Information Mandala serves as a new form of entropy visualization, helping to demystify the "black box" of feature extraction in Convolutional Neural Networks (CNNs).

Summary updated

BibTeX
@article{lu2021information,
  author = {Xin Lu},
  title = {{Information Mandala: Statistical Distance Matrix with Clustering}},
  journal = {IEEE Access},
  year = {2021},
  doi = {10.1109/access.2021.3072237},
}
Figure