Absorbing Gradient Conflicts: Modeling Semantic Variance via Kent Distributions for Cross-Modal Hashing

Hengjie Zhu, Dayan Wu, Zihao Zhang, Xinze Liu, Jingxuan Yu, Peng Fu, Zheng Lin, Weiping WangarXiv 2026 · arXiv

TL;DR

Traditional cross-modal hashing methods represent categories as rigid, single points in an embedding space, leading to severe optimization instability and "proxy collapse" when handling multi-label data. This paper introduces Kent-based Distributional Proxy Hashing (KDPH), which models class proxies as flexible, anisotropic Kent distributions on a hypersphere. By dynamically adjusting directional variance instead of shifting centroids, KDPH successfully absorbs gradient conflicts during training with zero additional computational overhead during inference.

What Problem Does This Paper Address?

Cross-modal hashing aims to map multimodal data (like images and text) into a compact, shared Hamming space to enable ultra-fast, large-scale retrieval. Modern approaches often use "proxies"—deterministic points in the embedding space that represent entire semantic categories.

While this works well for single-label data, it breaks down in real-world multi-label scenarios where an instance possesses multiple overlapping tags (e.g., an image containing both a "car" and a "bicycle"). In a point-proxy framework, a multi-label sample exerts simultaneous, conflicting pulling forces on divergent proxies. Because these point proxies are rigid and zero-variance, they cannot accommodate these mixed signals. This results in two major training pathologies:

  1. Chaotic Proxy Oscillation: Proxies constantly shift positions, preventing the model from anchoring to a stable semantic center.
  2. Proxy Collapse: Proxies of distinct categories are forcibly dragged together to minimize aggregate loss, causing rare "tail" classes to merge into common "head" classes and destroying fine-grained decision boundaries.

Methods

To solve these challenges, the authors propose Kent-based Distributional Proxy Hashing (KDPH). Instead of static points, each class proxy is modeled as a learnable, anisotropic Kent distribution on a hypersphere ($S^{K-1}$).

Figure 2: Overview of the Kent-based Distributed Proxy Hashing (KDPH) framework. It consists of a CLIP-based feature extraction module and a Kent-based Distributional Proxy Learner. As shown in the right panel, we model class proxies as learnable anisotropic distributions rather than static points. Through the gradient decoupling mechanism by separating centroid µ and variance B, Γ, the model dynamically stretches the proxy’s semantic volume to accommodate multi-label variance, effectively mitigating proxy collapse and oscillation.

Key components of the methodology include:

  • Anisotropic Modeling: Unlike isotropic distributions (like von Mises-Fisher) which assume uniform variance in all directions, the Kent distribution allows elliptical contours. It is parameterized by a mean direction (the semantic centroid), global concentration, and orthogonal principal axes with individual shaping coefficients.
  • Gradient Decoupling: When a multi-label sample introduces conflicting gradients, the proxy absorbs the conflict by stretching its shape along specific semantic axes rather than displacing its centroid.
  • Cayley Transform: To maintain the strict orthogonality required by the Kent distribution's principal axes during backpropagation, the authors use the Cayley transform to map unconstrained matrices to a strict rotation matrix, avoiding expensive orthogonalization steps.
  • Anisotropic Energy Scoring: The authors bypass the computationally intractable normalization constant (Bessel functions) of the standard Kent probability density function by reinterpreting the log-density as a geometric compatibility score.
  • Zero Inference Overhead: The probabilistic module is strictly used to guide the embedding space during training. During inference, the model reverts to standard deterministic hashing, preserving retrieval efficiency.

Datasets

The framework was evaluated on three standard multi-label cross-modal retrieval benchmarks:

  • MIRFLICKR-25K: 24,581 image-text pairs across 24 classes.
  • NUS-WIDE: A subset of 195,834 pairs from 21 frequent categories.
  • MS COCO: 123,289 images, each with 5 captions, spanning 80 categories.

No new datasets or benchmarks were introduced in this work.

Key Insights and Ideas

  • Shape over Position: In multi-label learning, semantic variance is directional. Giving a proxy "volume" and "shape" allows it to absorb contextual variance from label co-occurrences without losing its core identity (centroid).
  • Isotropy is a Limitation: Previous probabilistic embedding methods used isotropic distributions, which assume uniform variance. Multi-label data manifolds are inherently non-uniform; hence, anisotropic models like the Kent distribution provide the necessary geometric flexibility to handle densely entangled semantics.
  • Optimization Stability via Geometry: By decoupling the gradient updates into centroid shifts and shape deformations, the training dynamics stabilize dramatically, leading to smoother convergence trajectories.

Key Results

  • State-of-the-Art Performance: KDPH consistently outperforms existing deep cross-modal hashing methods (such as DSPH, DCPH, and nivMF) across all three datasets for both Image-to-Text (I2T) and Text-to-Image (T2I) retrieval tasks.
  • Substantial Gains in Complex Scenarios: On the highly complex MS COCO dataset at 16 bits, KDPH achieves an I2T mAP improvement of 7.92% over isotropic alternatives and 6.07% over the second-best state-of-the-art method.
  • Mitigating Oscillation and Collapse: Flip-rate analysis shows that KDPH maintains exceptional stability (flip rate < 1%) in high co-occurrence classes where baselines oscillate wildly. t-SNE visualizations confirm that KDPH preserves distinct, well-separated proxy distributions instead of collapsing them into a central anchor.
Figure 1: Comparison of proxy optimization behaviors. (a)-(b) Conventional proxy learning suffers from proxy collapse, where learned proxies move toward the anchor and become poorly sep- arated, whereas KDPH preserves distinct proxy distributions. (c)- (d) Conventional methods exhibit slow and oscillatory optimization trajectories, while KDPH achieves smoother trajectories and faster convergence. Colors denote different categories.

Limitations and Future Work

Not discussed in the paper.

Summary updated

Abstract

Supervised proxy-based deep cross-modal hashing has become the dominant paradigm for large-scale retrieval. However, prevalent methods model class proxies as deterministic points in the embedding space. This rigid assumption causes severe gradient conflicts in multi-label scenarios, where gradient conflicts arising from label co-occurrence lead to severe gradient contention and optimization collapse. To resolve this, we propose Kent-based Distributional Proxy Hashing (KDPH), a novel framework that shifts proxy representation from static points to flexible anisotropic Kent distributions on the hypersphere. Unlike point proxies that must shift their positions to accommodate conflicting gradients, KDPH absorbs these conflicts by dynamically adjusting its directional variance. This allows the proxy to maintain a stable semantic mean direction while stretching to cover diverse label correlations. Furthermore, to ensure stable training of these geometric parameters, we derive a tailored loss function incorporating the Cayley transform to enforce strict orthogonality. To the best of our knowledge, KDPH is the first framework to successfully introduce the Kent distributions into cross-modal hashing. Experiments on three benchmark datasets demonstrate that KDPH mitigates proxy collapse and chaotic oscillation, significantly outperforms state-of-the-art methods. Code is available at https://github.com/Senmo996/KDPH-official-code.
BibTeX
@misc{zhu2026absorbing,
  author = {Hengjie Zhu and Dayan Wu and Zihao Zhang and Xinze Liu and Jingxuan Yu and Peng Fu and Zheng Lin and Weiping Wang},
  title = {{Absorbing Gradient Conflicts: Modeling Semantic Variance via Kent Distributions for Cross-Modal Hashing}},
  year = {2026},
  eprint = {2608.24010},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.24010},
}
Figure