Robust Self-reflective Hashing for Cross-modal Retrieval with Noisy Label

Hao Sun, Qibing Qin, Lei HuangICML 2026 · OpenReview

Cross-modal Hashing (CMH) is the engine behind efficient large-scale retrieval, allowing systems to find a specific image using a text query (or vice versa) by converting complex data into compact binary codes. However, most CMH models operate under a flawed "perfect world" assumption: that every training sample is correctly and completely labeled.

In reality, datasets are messy. Labels generated by non-experts or automated systems often contain noise—for instance, a photo of a sunset mistakenly labeled "ocean." When standard models encounter these noisy labels, they try too hard to reconcile the visual data with the incorrect text, leading to overfitting. The model essentially learns bad habits from bad data, resulting in unreliable search results during real-world use.

To solve this, researchers have developed Robust Self-reflective Hashing (RSH), a framework that prevents cross-modal retrieval models from overfitting to incorrect labels by teaching them to adaptively ignore unreliable training data.

The RSH Framework: Double Features and Self-Reflection

The RSH framework mimics human intuition: if a piece of information seems contradictory or "fuzzy," we tend to trust it less. It achieves this through two primary innovations:

1. Double Feature Representation (DFR) Instead of extracting a single feature vector for an image or text, RSH extracts two distinct features. The first captures the semantic content (what is actually in the image or text), while the second captures uncertainty (how confident the model is about this specific sample).

2. Self-reflective Similarity Metric (SSM) Traditional metrics calculate the distance between two items and force the model to minimize that distance if they share a label. SSM acts as the "brain" of the operation by evaluating the uncertainty features first. If the uncertainty is high, the metric uses an exponential decay function to "soften" the semantic discrepancy. This ensures that highly uncertain samples have a negligible impact on the model's learning process.

Importantly, RSH is designed to be plug-and-play. Developers can take a standard loss function, such as Triplet Loss, and simply swap out the traditional similarity metric for SSM to instantly make their model noise-resistant.

Key Insights: Gradient Muting and Relative Uncertainty

The core breakthrough of RSH is that uncertainty is no longer just a byproduct of the model; it is an active participant in the loss function. By allowing the model to "reflect" on the reliability of a sample, it can prioritize clean data.

Through gradient analysis, the authors demonstrate a phenomenon called Gradient Muting. High uncertainty mathematically leads to smaller gradients. In plain English, the model turns down the volume on noisy samples so they do not drown out the signal from correct ones. Furthermore, the model evaluates the relative uncertainty between pairs of images and text rather than just absolute fuzziness. This helps it navigate semantic ambiguity, where an image might legitimately belong to multiple categories.

Experimental Results: Dominating High-Noise Scenarios

The researchers evaluated RSH on three gold-standard benchmarks for cross-modal retrieval: MIRFlickr-25K, MS COCO, and IAPR TC-12. To test robustness, they intentionally corrupted these datasets by injecting symmetric label noise at rates of 20%, 50%, and 80%.

RSH consistently outperformed nine other state-of-the-art methods across all datasets:

  • High-Noise Dominance: In the most extreme scenario (80% noise, 128-bit codes), RSH outperformed the best baseline by 3.27% to 6.38% in Mean Average Precision (MAP).
  • Stability: While competing models saw their performance crash as noise increased, RSH's performance curve remained remarkably stable.
  • Beyond Labels: The model also demonstrated a lower Mean Corruption Error (MCE). This means it handles "natural" noise—like motion blur or JPEG compression—better than its peers, even though it wasn't specifically trained to handle those visual degradations.

Limitations and Future Work

While RSH demonstrates impressive results, a few areas remain open for future exploration:

  • Hyperparameter Tuning: The framework relies on two key hyperparameters ($\gamma$ and $\tau$) that control the "caution" and "softening" levels. Finding the optimal values currently requires manual adjustment, which might vary across different real-world applications.
  • Asymmetric Noise: The study focused on symmetric noise (randomly flipping labels). In the real world, noise is often asymmetric (e.g., a "dog" is more likely to be mislabeled as a "wolf" than a "car"). Testing RSH against these more insidious patterns is a logical next step.
  • Computational Overhead: Although the authors note the overhead is negligible, the double-feature extraction does add a layer of architectural complexity that might require optimization for deployment on edge devices.

Summary updated

Abstract

Cross-modal Hashing (CMH) typically assumes a perfectly complete data annotation, whereas noisy labels are unavoidable in practical scenarios. Existing CMH methods often overlook the uncertainty introduced by noise or semantic ambiguity, making models susceptible to overfitting noisy labels and yielding unreliable similarity judgments during inference. To address this issue, we propose a Robust Self-reflective Hashing (RSH) framework that prudently analyzes semantic discrepancies while accounting for uncertainty, thereby effectively mitigating interference from noisy labels. Specifically, the Double Feature Representation (DFR) method is introduced, employing semantic and uncertainty features to represent the semantic representation and fuzziness of samples. With a double feature, we propose a novel cross-modal similarity metric - the Self-reflective Similarity Metric (SSM), which judges sample similarity by integrating semantic discrepancy and fuzziness, enabling the model to adaptively weaken semantic discrepancy according to uncertainty level. The proposed method is plug-and-play, enabling seamless integration into diverse objective functions to enhance model robustness and reliability. Extensive experiments on benchmark datasets demonstrate that RSH outperforms existing methods. Code is available at https://github.com/QinLab-WFU/RSH.
BibTeX
@inproceedings{sun2026robust,
  author = {Hao Sun and Qibing Qin and Lei Huang},
  title = {{Robust Self-reflective Hashing for Cross-modal Retrieval with Noisy Label}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=jyGcO0W5xd},
}
Figure