In representation learning, the choice of how we measure similarity between two vectors dictates the geometric properties of the entire latent space. For years, the default choice across collaborative filtering, matrix factorization, and graph embedding has been the inner product (or its normalized cousin, cosine similarity).
The inner product is computationally cheap and works well for simple matching, but it suffers from a fundamental mathematical flaw: it does not satisfy the triangle inequality.
Because the inner product is not a true metric, similarity in an inner-product space is not transitive. This lack of metricity creates severe representation inconsistencies, limits a model's ability to generalize from sparse data, and introduces geometric distortions.
The Transitivity Gap
To understand why the lack of a triangle inequality is problematic, consider a simple recommendation or graph scenario. Suppose we have three entities: $A$, $B$, and $C$. If $A$ is highly similar to $B$, and $B$ is highly similar to $C$, a robust latent space should allow us to infer that $A$ and $C$ share some underlying relationship. This property is known as collaborative transitivity.
In an inner product space, this inference is mathematically unguaranteed. As noted by He et al. in their survey on neural recommendation, if two entities are both close to a third in an inner-product space, the model may completely fail to recognize their relationship to one another. Alfarhood et al. demonstrate this in CATA++, showing that scoring user-item pairs via dot products leads to unstable and inaccurate mappings because the relative geometric distances between entities are fundamentally unreliable.
Inner Product Space (Non-Metric):
(A) · (B) is high
(B) · (C) is high
(A) · (C) can still be ZERO or negative!
No geometric constraint forces them together.
This structural limitation severely hinders generalization in sparse datasets. When a model cannot rely on transitivity, it cannot effectively propagate preferences or structural roles across unobserved pairs. As He et al. argue in the context of sequential recommendation (TransRec), metric-based approaches are inherently better suited for sparse data because the triangle inequality provides a mathematical mechanism to infer connections between items that have never been explicitly co-observed.
Enforcing Metric Consistency
To resolve this issue, a prominent line of research replaces inner products with true distance metrics—most commonly Euclidean distance—giving rise to Collaborative Metric Learning (CML).
By embedding entities into a true metric space, the triangle inequality is strictly enforced:
$$d(A, C) \le d(A, B) + d(B, C)$$
If the model learns that $A$ is close to $B$ and $B$ is close to $C$, the geometry of the space forces $A$ and $C$ to be close. Xu et al. utilized this property in their Latent Dual-Metric Learning (LDME) framework, mapping user-item, user-user, and item-item relationships into a single, geometrically consistent space. Similarly, Xu et al. introduced Magnetic Metric Learning (MML) to ensure stable and reliable clustering of users and items by adhering strictly to geometric principles, while Guo et al. extended this paradigm to feature interactions by upgrading traditional factorization machines with generalized metric learning (GML-FM).
The Geometric Bottleneck of Strict Metrics
While enforcing the triangle inequality fixes the transitivity gap, it introduces a secondary problem: the geometric bottleneck.
In real-world applications, user preferences and entity relationships are often heterogeneous. Imagine a user who enjoys both Heavy Metal music and Classical Piano. A naive metric learning model tries to push this user close to both items in the latent space. However, because of the triangle inequality, pushing the user close to both items inadvertently forces Heavy Metal and Classical Piano to be positioned close to each other.
The Geometric Bottleneck:
[User]
/ \
/ \ (Strict Euclidean Distance)
/ \
[Heavy Metal]--[Classical Piano] <-- Forced to be close!
As the number of diverse interactions grows, it becomes mathematically impossible to satisfy all these distance constraints simultaneously. Liu et al. identify this in MAML, noting that strict geometric restrictions cause severe representation degradation when users exhibit diverse preferences.
To unlock the benefits of transitivity without falling victim to the geometric bottleneck, researchers have developed several sophisticated architectures:
1. Translational Spaces
Instead of pulling entities to the exact same coordinate, relationships can be modeled as vector translations ($\mathbf{A} + \mathbf{r} \approx \mathbf{B}$). In sequential recommendation, He et al. treat users as translation vectors moving from one item to the next (TransRec).
To handle multi-interest profiles, Tran et al. proposed Hierarchical Latent Relation (HLR) modeling, which uses attention networks to dynamically compute translation vectors based on context. This allows unrelated items to remain far apart in the global space while still satisfying a localized, translated metric consistency.
2. Probabilistic Clouds and Wasserstein Distance
Another approach replaces rigid point embeddings with probability distributions (typically Gaussians). Instead of measuring Euclidean distance between points, models calculate the Wasserstein distance between "probability clouds."
As demonstrated by Ma et al. in PMLAM and Liu et al. in STOSA, representing items as distributions allows the model to capture user uncertainty and flexible collaborative transitivity. The variance of the distribution acts as a buffer, absorbing the geometric tension caused by diverse interactions.
3. Contextual and Decoupled Metrics
Other frameworks dynamically alter the metric space based on context or action types. Tran et al. adopted the Mahalanobis distance for playlist continuation (AMD-ASR), allowing the model to warp the latent space dimensions depending on the specific theme of a playlist. In a different approach to multi-behavior modeling, Lee et al. introduced METAS, which decouples high-level entity similarities from specific action spaces (like clicking versus purchasing), preventing conflicting behaviors from distorting a single metric space.
Beyond Recommendation: Graphs and the Inverse Problem
The tension between metricity and transitivity extends far beyond recommender systems. In graph representation learning, structural similarity measures frequently violate the triangle inequality, leading to unstable embeddings. To combat this, Luo et al. designed ripple2vec, which enforces a strict "ripple distance" metric to ensure that nodes with similar structural roles are transitively mapped near one another. For massive networks, Weiren Yu developed SimEdge, a graph-theoretic similarity model that guarantees edge-to-edge transitivity, allowing the model to accurately surface similarities between weakly connected entities that do not share immediate neighbors.
When Transitivity is a Bug: Contradiction Retrieval
Intriguingly, understanding the transitivity of embedding spaces also unlocks solutions for tasks where transitivity is explicitly undesirable.
Consider contradiction retrieval (e.g., finding statements that contradict a given fact). Contradiction is inherently non-transitive: if Statement $A$ contradicts Statement $B$, and Statement $B$ contradicts Statement $C$, Statement $A$ definitely does not contradict Statement $C$ (they likely agree).
Standard contrastive text embeddings rely on cosine similarity, which is inherently transitive. As a result, standard models struggle with contradiction retrieval because they cannot map multiple distinct statements as contradicting the same fact without accidentally making those statements look identical to each other.
To solve this, Xu et al. developed SparseCL. By combining cosine similarity with a mathematical sparsity constraint, the authors intentionally broke the inherent transitivity of the embedding space, allowing the model to cleanly map non-transitive real-world relationships.
Summary
The shift from inner products to metric spaces represents a fundamental trade-off in geometric representation learning:
| Representation Space | Triangle Inequality | Transitivity | Primary Limitation |
|---|---|---|---|
| Inner Product / Cosine | Violated | Unregulated (Transitivity Gap) | Inconsistent representations; poor generalization on sparse data. |
| Strict Metric (CML) | Enforced | Strict | Geometric Bottleneck; flattens heterogeneous/diverse interests. |
| Advanced Metric (Translational/Stochastic) | Conditioned / Relaxed | Adaptive | Increased architectural complexity. |
By understanding and manipulating the triangle inequality—whether through translational vectors, probabilistic distributions, or mathematical sparsity—modern architectures can tailor the transitivity of a latent space to match the true logical structure of the underlying data.
Key Foundational Work
Collaborative Metric Learning (2017) — contributed 2 concept elements to this direction.
Translation-based Recommendation (2017) — contributed 1 concept elements to this direction.
Learning Kernels for Semantic Clustering: A Deep Approach (2015) — contributed 1 concept elements to this direction.