Designing Machine Learning Invariant to Linear Transformations

23 concepts16 papers1 proposal

When we represent data numerically, we make arbitrary choices about units, scales, and coordinate axes. If we measure a patient's height in meters versus millimeters, the numerical values stretch by a factor of one thousand. If we rotate a camera, the pixel coordinates undergo a linear transformation.

Ideally, a machine learning model should extract the same underlying semantic truths regardless of these representational shifts. However, standard algorithms are notoriously sensitive to coordinate choices. Traditional Euclidean distance-based clustering (like $K$-means) or isotropic margin classifiers (like standard Support Vector Machines) assume that the feature space is uniform and spherical. If you stretch or rotate the space, their decision boundaries and cluster assignments warp dramatically, requiring practitioners to rely heavily on manual preprocessing pipelines like standardization or principal component analysis.

A powerful alternative paradigm focuses on designing algorithms, objectives, and metrics that are intrinsically invariant to invertible linear and affine transformations. By embedding coordinate independence directly into the mathematics of the model, these techniques ensure that the system's behavior remains identical whether the data is scaled, rotated, or sheared.


The Geometry of Invariance: From Spheres to Ellipsoids

To understand why standard classifiers fail under linear transformations, consider how they bound data. A standard Support Vector Machine attempts to find a separating hyperplane maximized against a spherical margin. If the underlying data distribution is elongated—shaped more like a football than a basketball—a spherical bound forces the classifier to become overly sensitive to the highly scaled dimensions, leading to suboptimal boundaries.

Sensitive to Scaling (Spherical Bound):
   (  O  )   --> Stretched Axis -->   (   O   )  [Boundary warps]

Invariant to Scaling (Ellipsoidal Bound):
   (  O  )   --> Stretched Axis -->   (   O   )  [Boundary scales proportionally]

To resolve this, classifiers can be built using ellipsoidal geometry. As shown in Ellipsoidal Machines by Shivaswamy et al., we can replace rigid spherical bounds with adaptive ellipsoids that encapsulate the data's covariance structure. Because any invertible linear transformation maps an ellipsoid to another ellipsoid, the relative decision boundary scales proportionally with the data.

By transforming the input space so that its bounding ellipsoid becomes a unit sphere, the model achieves complete affine transformation invariance. The resulting decision boundary is determined solely by the relative distribution of the points rather than the arbitrary units of individual features. This geometric insight is extended to structured domains in Text Categorization via Ellipsoid Separation by Kharechko et al., where concentric ellipsoids define class boundaries in latent semantic spaces, ensuring that text classification remains robust across varied representational strategies.


Coordinate-Independent Clustering and Projections

The vulnerability to feature scaling is especially acute in unsupervised learning, where there are no labels to guide the alignment of disparate features. If one feature has a larger numerical range purely due to its unit of measurement, it will disproportionately dominate distance metrics.

Instead of relying on distance, we can achieve invariance by framing clustering as an optimization problem over discriminative capacity. Consider a classifier trying to separate data points into groups. A linear classifier's fundamental ability to separate two groups is completely unaffected by whether those groups are scaled or rotated; if a hyperplane can separate them in one coordinate system, an appropriately transformed hyperplane can separate them in any linearly transformed coordinate system.

The authors of DIFFRAC: a discriminative and flexible framework for clustering (Bach et al.) exploit this by utilizing a regularized square-loss discriminative cost for clustering. Because the underlying loss adapts to linear shifts, the resulting partitions remain perfectly consistent regardless of affine transformations.

Similarly, Huang et al. introduce Maximin Separation Probability Clustering in Maximin Separation Probability Clustering, which optimizes a lower bound on worst-case generalization accuracy. Because this probability is based on linear separability, the objective function is mathematically invariant to any invertible linear transformation, eliminating the need for manual feature normalization.

This discriminative philosophy also applies to dimensionality reduction and feature selection:

  • Focusing on Informative Dimensions: In Clustering with the Fisher Score, Tsuda et al. design an EM-like optimization that automatically filters out high-dimensional "nuisance dimensions." The algorithm is invariant to invertible linear transformations, ensuring that the final clusters are not artifacts of how the feature space was originally mapped or rotated.
  • Maximizing Separability: In Maximum Entropy Linear Manifold for Learning Discriminative Low-dimensional Representation, Czarnecki et al. propose a projection technique that exhibits affine transformation invariance. By ensuring the learned low-dimensional manifold is robust to scaling and translation, the method simplifies preprocessing requirements across varying data pipelines.

Overcoming Correlation in Feature Extraction and Sparse Recovery

When features are highly correlated or dictionaries are poorly scaled, standard optimization techniques like $\ell_1$ regularization (Lasso) frequently break down, selecting arbitrary or incorrect feature subsets.

To make sparse estimation robust, we can design estimators whose objective functions absorb the transformation matrix. In Sparse Estimation with Structured Dictionaries, Wipf et al. demonstrate that Type II Bayesian estimators possess a fundamental invariance to arbitrary row correlations and column scalings within a dictionary. Because the global minimum of the objective function remains stable under these transformations, the estimator effectively automates data normalization and decorrelation.

This principle is extended to the problem of identifying underlying low-dimensional subspaces in Data-Dependent Sparsity for Subspace Clustering. Here, Xin et al. utilize data-dependent, non-convex penalties derived from a Bayesian framework. The resulting subspace recovery is mathematically invariant to invertible linear transformations and diagonal scaling, allowing the model to consistently group data even in the presence of extreme outliers and heavy feature correlation.

For explicit feature extraction, we can turn to generalized eigenvalue problems. In Discriminative Features via Generalized Eigenvectors, Karampatziakis et al. construct non-linear feature mappings by analyzing the second-order variance structure between pairs of classes. Why does this guarantee invariance? The generalized eigenvalue problem solves equations of the form:

$$S_B v = \lambda S_W v$$

where $S_B$ is the between-class scatter matrix and $S_W$ is the within-class scatter matrix. If we apply an invertible linear transformation $A$ to our input data, both scatter matrices transform identically ($S \to A S A^T$). The transformation matrix $A$ effectively cancels out on both sides of the equation, leaving the resulting generalized eigenvectors—and the extracted features—completely invariant to the transformation.


Invariant Metrics for Model Evaluation

Invariance is not just a property of classifiers and clustering algorithms; it is equally vital for the metrics we use to evaluate representations. If a metric changes when features are scaled or rotated, it fails to provide a reliable measure of a model's true intrinsic capacity.

Evaluating Neural Representations

When analyzing deep neural networks, researchers often measure "Neural Collapse"—the tendency of last-layer features of the same class to cluster tightly together. In Quantifying the Variability Collapse of Neural Networks, Xu et al. introduce the Variability Collapse Index (VCI). They argue that because downstream linear probes are inherently unaffected by linear transformations of the feature space, any metric used to predict their performance must share this invariance. A metric that shifts when features are rotated or scaled cannot accurately capture the representation's true linear separability.

Out-of-Distribution Detection

A similar challenge occurs in Out-of-Distribution (OoD) detection, where models must identify when an input does not belong to the training distribution. In Fisher-Rao Sensitivity for Out-of-Distribution Detection in Deep Neural Networks, Nguyen et al. leverage Fisher-Rao sensitivity. Because this metric is invariant to linear transformations of the feature space, it operates independently of the specific coordinate system of the embeddings. This allows the exact same detection principles to be applied seamlessly across radically different neural architectures, such as ResNets and Transformers, without requiring architecture-specific hyperparameter tuning.

Information-Theoretic and Statistical Measures

On a broader statistical level, standard information-theoretic metrics often fail when coordinate systems change. As explored by Sarra et al. in Renormalized Mutual Information for Extraction of Continuous Features, robust feature extraction requires metrics that remain invariant under bijective transformations to ensure that the extracted "collective variables" are physical realities rather than artifacts of a chosen coordinate system.

Mathematically, this search for invariant statistical measures leads directly to the $f$-divergence family. In f-divergence is a generalized invariant measure between distributions, Qiao et al. prove that $f$-divergences represent the unique class of integration-type statistical measures that remain invariant under any invertible transformation of the feature space, providing a rigorous foundation for comparing probability distributions robustly.

Finally, this can be applied to distance design itself: Mahamud et al. show in Minimum Risk Distance Measure for Object Recognition that defining distance as the probability that two samples belong to different classes yields a non-metric measure that is invariant to any strictly increasing transformation of the underlying space, allowing the seamless combination of completely disparate features like color and shape.


Analytical Invariance vs. Data Augmentation

The traditional deep learning approach to achieving transformation invariance is data augmentation—feeding the model thousands of altered variations (scaled, rotated, or flipped) of the training data so it can "learn" the invariance. However, this approach is computationally expensive and rarely yields perfect invariance.

An alternative is to analytically derive invariant subspaces directly from the transformation matrices. In Flip-Invariant Motion Representation, Kobayashi demonstrates this by mathematically constructing a descriptor-level transformation for video motion representations. By hardcoding the geometric invariance into the feature space itself, the descriptors remain identical under transformations (such as horizontal flipping) without requiring any additional training data or computational cycles.

Approach Mechanism Computational Cost Invariance Guarantee
Data Augmentation Force model to learn patterns from transformed data High (Requires larger models and longer training) Empirical / Approximate
Analytical Invariance Design objectives/spaces that mathematically cancel transformations Low (Embedded directly into the algorithm) Exact / Mathematical

By shifting the burden of invariance from the training data to the mathematical structure of the algorithm, coordinate-independent machine learning provides a principled, highly efficient path toward models that see past the arbitrary choices of data representation.

Key Foundational Work

Clustering with the Fisher Score (2002) — contributed 2 concept elements to this direction.
Minimum Risk Distance Measure for Object Recognition (2003) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_linear_transformation_invariance_8c17f325,
  title = {linear transformation invariance},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-transformation-invariance/rdr_8c17f32524d64d5a8d66ea91c7dbcc5d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_linear_transformation_invariance_8c17f325_v1,
  title = {linear transformation invariance},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-transformation-invariance/rdr_8c17f32524d64d5a8d66ea91c7dbcc5d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake