Abstract: One of the basic problems in data analysis lies in choosing the optimal rescaling (change of coordinate system) to study properties of a given data-set $Y$. The classical Mahalanobis approach has its basis in the classical normalization/rescaling formula $Y \ni y \to Σ_Y^{-1/2} \cdot (y-\mathrm{m}_Y)$, where $\mathrm{m}_Y$ denotes the mean of $Y$ and $Σ_Y$ the covariance matrix. Based on the cross-entropy we generalize this approach and define the parameter which measures the fit of a given affine rescaling of $Y$ compared to the Mahalanobis one. This allows in particular to find an optimal change of coordinate system which satisfies some additional conditions. In particular we show that in the case when we put origin of coordinate system in $ \mathrm{m} $ the optimal choice is given by the transformation $Y \ni y \to Σ^{-1/2} \cdot (y-\mathrm{m})$, where $$ Σ=Σ_Y(Σ_Y-\frac{(\mathrm{m}-\mathrm{m}_Y)(\mathrm{m}-\mathrm{m}_Y)^T}{1+\|\mathrm{m}-\mathrm{m}_Y\|_{Σ_Y}^2})^{-1}Σ_Y. $$
Introduction
In data science, the representation of data is fundamental. Before applying analytical algorithms, researchers typically normalize or rescale datasets to ensure feature comparability. The standard method for this is the Mahalanobis distance, which transforms the coordinate system based on the data's mean and covariance, effectively "sphering" the data.
However, practical applications often impose constraints that prevent simple centering at the mean. For instance, in image processing, pixel values might be anchored to a fixed range, making it computationally expensive or inconsistent to move the origin for every image block. Przemysław Spurek and Jacek Tabor address this in "Optimal Rescaling and the Mahalanobis Distance," providing a rigorous framework for finding the best coordinate system under such constraints.
Theoretical Framework: Cross-Entropy and Gaussian Subfamilies
The authors utilize information theory to define the "best" coordinate system. By treating a coordinate system as a Gaussian distribution, they identify the optimal system as the one minimizing the cross-entropy with the actual data. This approach is equivalent to Maximum Likelihood Estimation (MLE).
To handle various practical constraints, the paper defines several subfamilies of Gaussian densities, representing different types of allowed transformations.
These subfamilies include cases with fixed origins ($m$), diagonal covariances (axis-aligned scaling), or uniform scaling (proportional to identity). The mathematical proof of optimality for these cases relies on the von Neumann trace inequality to ensure global minima for the cross-entropy functions.
The Optimal Rescaling Formula
The most significant theoretical contribution is the derivation of a "correction" term for cases where the origin is fixed at a point $m$ different from the data mean $m_Y$. In such scenarios, the optimal covariance matrix $\Sigma$ is not the standard sample covariance but follows a specific adjustment:
$$\Sigma = \Sigma_Y \left(\Sigma_Y - \frac{(m - m_Y)(m - m_Y)^T}{1 + \|m - m_Y\|_{\Sigma_Y}^2}\right)^{-1} \Sigma_Y$$
This formula allows for a principled rescaling even when centering is restricted.
Visualizing Optimal Rescaling
The impact of these constraints can be visualized using synthetic 2D data. When the origin is moved or the covariance is restricted to be diagonal, the resulting "optimal" transformation changes significantly.
As seen in the figure, (a) represents the standard unconstrained Mahalanobis transformation. In (b), the origin is fixed at (0,0), leading to a different stretching of the space. In (c), the transformation is restricted to axis-aligned scaling (diagonal covariance), which fails to "sphere" the data if features are correlated.
Experimental Validation: Image Compression
The theory was validated using the "Lena" image benchmark, treating 8x8 pixel blocks as vectors in a 192-dimensional space—a setup common in JPEG compression. The analysis showed that the standard engineering heuristic of fixing the center at 0.5 is remarkably close to the theoretical optimum. Specifically, moving the origin from the mean to the center of the color space ([0.5, ..., 0.5]) only slightly increased the information "cost" (from 0 to 0.52), whereas leaving the origin at [0, ..., 0] was significantly worse (cost of 2.73).
Conclusion
This framework provides a mathematical foundation for choosing optimal coordinate systems under constraints. While currently reliant on Gaussian assumptions, it offers a principled way to evaluate and design normalization layers in neural networks or data compression codecs. Future work may extend these concepts to non-linear transformations beyond affine rescaling.
Summary updated