Most machine learning algorithms rely fundamentally on measuring the distance between data points. Standard metrics like Euclidean or Hamming distance implicitly assume that the underlying feature space is isotropic—meaning all features are independent, un-correlated, and equally important. Geometrically, this treats similarity as a perfect sphere.
However, real-world data is rarely isotropic. Features are often highly correlated and vary across completely different scales. Treating features as independent overemphasizes redundant information and ignores the true geometric structure of the data. As noted by Katsumi Inoue et al. in Iterated Belief Change as Learning, relying on simple, uncalibrated distance metrics prevents models from accounting for these critical underlying dependencies.
The Geometry of the Mahalanobis Metric
To account for feature interactions, we can generalize the distance metric using a positive semidefinite matrix $M$. The generalized Mahalanobis distance between two vectors $x$ and $y$ is defined as:
$$d_M(x, y) = \sqrt{(x-y)^T M (x-y)}$$
To understand why this works, consider how the structure of $M$ alters the geometry of the space:
- If $M$ is the identity matrix ($I$), the formula simplifies exactly to standard Euclidean distance.
- If $M$ is a diagonal matrix, it scales each feature axis independently, acting as a feature-weighting mechanism.
- If $M$ is a full matrix, its off-diagonal terms explicitly model the cross-correlations between different variables.
As Wei Hu et al. demonstrate in Feature Graph Learning for 3D Point Cloud Denoising, optimizing these off-diagonal terms allows a system to capture complex interdependencies (such as spatial coordinates and surface normals) far better than methods that scale features independently.
Because $M$ is positive semidefinite, it can always be decomposed into $M = L^T L$, where $L$ is a linear transformation matrix. Substituting this back into the distance formula reveals the core intuition:
$$d_M(x, y) = \sqrt{(x-y)^T L^T L (x-y)} = \sqrt{(Lx - Ly)^T (Lx - Ly)}$$
Computing the Mahalanobis distance in the original space is mathematically identical to applying a linear transformation $L$—which rotates, stretches, and squashes the coordinate axes—and then computing the standard Euclidean distance in that transformed space. Kilian Q Weinberger et al. leverage this property in Distance Metric Learning for Large Margin Nearest Neighbor Classification to warp the input space, making distance-based rules highly sensitive to the relevant data manifolds while suppressing noise.
Learning the Transformation Matrix
Instead of calculating $M$ purely from empirical covariance, distance metric learning treats $M$ as a parameter matrix to be optimized based on task-specific supervision or "side-information."
In their foundational work, Distance Metric Learning with Application to Clustering with Side-Information, Eric P. Xing et al. formulated this as a convex optimization problem using pairs of similar points. The objective is to minimize the distance between points known to be similar while ensuring that dissimilar points are pushed apart. Because the formulation is convex, it avoids local optima, providing a robust way to warp data space so that downstream algorithms like K-means can isolate relevant patterns.
Subsequent research introduced distinct optimization paradigms to learn this matrix:
- Large Margin Constraints: The Large Margin Nearest Neighbor (LMNN) framework by Kilian Q Weinberger et al. uses semidefinite programming to establish a wide margin around target neighbors, actively pushing "impostor" classes away to improve nearest-neighbor classification.
- Information-Theoretic Regularization: Brian Kulis et al. introduced Information-theoretic metric learning, proving a formal mathematical equivalence between learning a Mahalanobis metric and learning a kernel matrix. This duality allows models to scale efficiently without expensive eigenvalue decompositions.
- Empirical Risk Minimization: Wei Bian in Learning a Distance Metric by Empirical Loss Minimization proved that framing metric learning as an empirical loss minimization problem guarantees statistical consistency, ensuring that the learned metric converges to the true optimal transformation as more data becomes available.
Applications Across Complex Data Manifolds
Learning a Mahalanobis metric functions as an effective linear representation learning step, projecting data into a space where Euclidean distance aligns with semantic similarity (Paramveer Dhillon et al. in Learning Better Data Representation Using Inference-Driven Metric Learning). This capability is particularly valuable in settings where standard metrics fail:
- High-Dimensional Embeddings: In large language models, dimensions within embedding spaces are often highly correlated. Thilini Wijesiriwardene et al. show in ANALOGICAL - A Novel Benchmark for Long Text Analogy Evaluation in Large Language Models that using Mahalanobis distance instead of standard cosine similarity normalizes the vector space based on covariance, yielding a more accurate measure of semantic equivalence.
- Tabular Data Clustering: Tabular datasets typically exhibit dense, overlapping distributions with high feature inter-dependencies. Hafiz Tayyab Rauf et al. demonstrate in TableDC: Deep Clustering for Tabular Data that integrating Mahalanobis distance into deep clustering latent spaces stretches the representation space to accommodate this variance, significantly improving cluster separability.
- Distribution-Aware Explanations: When generating counterfactual explanations, models must suggest realistic modifications to features. Kentaro Kanamori et al. show in DACE: Distribution-Aware Counterfactual Explanation by Mixed-Integer Linear Optimization that incorporating Mahalanobis distance into the cost function penalizes changes to a single feature that are statistically improbable without concurrent changes in correlated features, ensuring recommendations respect real-world data distributions.
Key Foundational Work
Distance Metric Learning with Application to Clustering with Side-Information (2002) — contributed 2 concept elements to this direction.