Manhattan Distance for High Dimensional Feature Separation

54 concepts37 papers

When working with high-dimensional data—whether it is deep neural network embeddings, genomic profiles, or sparse word histograms—our geometric intuition often betrays us. The default choice for measuring similarity is typically the Euclidean ($L_2$) distance or its angular cousin, cosine similarity. However, as the number of dimensions grows, these metrics suffer from a phenomenon known as distance convergence, a core symptom of the "curse of dimensionality."

In high-dimensional spaces, the Euclidean distance between the nearest neighbor and the farthest neighbor collapses to a nearly uniform value. When every point appears roughly equidistant from every other point, the metric loses its discriminative power. To combat this, a growing body of research demonstrates that shifting from Euclidean geometry to Manhattan ($L_1$) geometry—or even fractional $L_p$ norms ($p < 1$)—fundamentally alters how spaces behave, restoring contrast and improving feature separation.

The Geometry of High Dimensions: Why $L_2$ Fails

To understand why the Manhattan distance works where Euclidean distance fails, consider how each metric penalizes coordinate differences. The Euclidean distance squares each coordinate-wise difference before summing them:

$$D_{L2}(x, y) = \sqrt{\sum_{i=1}^d (x_i - y_i)^2}$$

In high dimensions ($d \gg 1$), if a few dimensions exhibit large variations, their squared values dominate the entire sum. This washes out subtle but meaningful patterns distributed across the remaining thousands of dimensions.

As shown by Scetbon et al., Manhattan geometry ($L_1$) aggregates small, dense variations across many features much more robustly because it sums absolute differences linearly:

$$D_{L1}(x, y) = \sum_{i=1}^d |x_i - y_i|$$

This prevents a handful of outlier dimensions from overwhelming the metric, allowing statistical tests and retrieval models to maintain high sensitivity even as dimensionality scales.

When the dimensionality becomes exceptionally high, researchers sometimes push this principle further by adopting fractional $L_p$ norms where $p < 1$. As noted by Wang et al. and Akbas et al., fractional norms effectively "stretch" the high-dimensional space. This prevents the distance between the nearest and farthest points from converging, preserving the discriminative power needed for complex optimization and classification tasks.

Preserving Coordinate-Wise Structure and Sparsity

Beyond mitigating distance convergence, the Manhattan distance aligns naturally with the underlying structure of sparse or disentangled representations. In sparse matrices—such as word association vectors or visual word histograms—similarity is often a function of exact coordinate-wise matches rather than overall angular alignment.

For instance, early work by Rapp demonstrated that the city-block metric ($L_1$ norm) yields significantly more accurate vector comparisons than cosine similarity when identifying word translations in sparse statistical matrices. Similarly, Karypidis et al. found that selecting Manhattan distance over Euclidean distance directly improves classification performance in visual word histograms by providing better separation between sparse vector classes.

This property extends to modern deep learning architectures. In generative models, minimizing squared errors ($L_2$) across high-dimensional spatial features tends to produce blurred results because the model averages out variations to minimize the squared penalty. Cho et al. showed that utilizing $L_1$ distance for latent alignment in diffusion models preserves sharp background details far better than $L_2$.

When features are explicitly disentangled—such as separating medical intent from acoustic properties in speech—Wang et al. observed that linear coordinate differences ($L_1$) capture similarity much more effectively than straight-line or angular metrics.

Enhancing Retrieval and Associative Memory

The geometric advantages of Manhattan distance translate directly into superior performance for nearest-neighbor retrieval and associative memory systems. In high-dimensional biometric spaces, such as gait recognition, Sokolova demonstrated that the Manhattan distance provides more stable similarity measurements than Euclidean distance by reducing the impact of extreme variations in individual feature dimensions.

This stability is crucial for associative memory models and episodic control in reinforcement learning. Chateau-Laurent et al. found that employing Manhattan distance as a similarity kernel in high-dimensional associative memory consistently yields superior retrieval capacity compared to standard Euclidean distance. By altering the mathematical metric used to calculate distances, the system can better filter and group related information, a principle that dates back to early implementations of sparse distributed memory by Turk et al..

Whether calculating genomic distances between neighboring cells in single-cell data (Montemurro et al.), tracking subtle syntactic shifts in language model hidden states (Jurayj et al.), or aligning speech embeddings with neural EEG signals (Soman et al.), the core insight remains the same: when spaces become high-dimensional, the linear, coordinate-wise nature of Manhattan geometry provides a more robust, lower-variance, and highly discriminative lens than traditional Euclidean metrics.

Key Foundational Work

Automatic Identification of Word Translations from Unrelated English and German Corpora (1999) — contributed 2 concept elements to this direction.
Kanerva's Sparse Distributed Memory: An Object-Oriented Implementation on the Connection Machine (1995) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_manhattan_distance_similarity_3e2a1ab2,
  title = {Manhattan distance similarity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/manhattan-distance-similarity/rdr_3e2a1ab2feac4a50bfc45a77b4bb0157},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_manhattan_distance_similarity_3e2a1ab2_v1,
  title = {Manhattan distance similarity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/manhattan-distance-similarity/rdr_3e2a1ab2feac4a50bfc45a77b4bb0157/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake