Training machine learning models to understand the relationship between video and text typically requires massive amounts of meticulously labeled data. To scale, researchers often utilize "self-supervised" learning, leveraging "free" labels such as automatic speech recognition (ASR) transcripts from web videos. However, these labels are notoriously noisy; a chef might discuss a family recipe while the video shows them performing a specific technical task like chopping onions. When models attempt to learn from these mismatched pairs, performance degrades. In their AAAI-21 paper, Amrani et al. propose a solution: treating noise estimation as a density estimation problem.
Addressing Multimodal Noise
The core challenge in large-scale datasets like HowTo100M is multimodal noise. It is estimated that roughly 50% of video clips do not contain the objects or actions mentioned in their accompanying captions. Standard multimodal embedding models attempt to pull representations of a video ($v$) and its caption ($c$) closer in a shared space. If the pair is a mismatch, the model learns incorrect associations, harming its ability to generalize to tasks like video retrieval or question answering.
Previous attempts to mitigate this often required a small, manually cleaned "gold" subset to train a separate noise-detection model. The authors' approach avoids this requirement by relying on the internal structure of the data itself.
Methodology: Noise as Sparsity
The authors' key insight is that correctly associated pairs naturally form dense clusters in a multimodal embedding space, while wrongly associated pairs appear as sparse outliers. Their method follows a four-step process:
- Multimodal Similarity Function: They define a similarity metric $S(z_i, z_j)$ between two video-caption pairs. For two pairs to be considered similar, both their visual components and their text components must be semantically close.
- Density Estimation via k-NN: For every pair in the dataset, the local density is calculated by examining its $k$ nearest neighbors in the multimodal space.
- Noise Probability ($\hat{p}_i$): These density scores are normalized to represent the probability that a pair is "correctly associated." High density indicates a likely signal (high $\hat{p}_i$), while low density indicates likely noise (low $\hat{p}_i$).
- Soft Max Margin Ranking Loss: Instead of treating every pair equally, the ranking loss is weighted by $\hat{p}_i$. This directs the model to prioritize dense, high-confidence clusters and down-weight sparse outliers.
Key Results and Insights
The proposed method, integrated into a model built on the HTM architecture, achieved state-of-the-art results across several benchmarks without requiring any manually cleaned data:
- Video Question Answering: The model set new records on MSRVTT-QA (35.06%) and MSVD-QA (35.13%), outperforming previous baselines.
- Zero-Shot Retrieval: On the MSVD dataset, the model achieved a Recall@1 of 13.66, a notable improvement over the standard HTM-PT baseline (12.86).
- Robustness: The noise-robust training allowed the model to achieve high performance even when using simpler 2D visual features, suggesting significant potential for computational savings.
Conclusion and Future Directions
By reframing noise detection as an unsupervised density estimation problem, this research demonstrates that semantic "truth" creates observable structure in high-dimensional spaces. While the current method uses a global value for $k$, future work may explore adaptive density estimation to better handle rare but valid concepts that might currently be misidentified as noise due to their inherent sparsity. This approach paves the way for more robust learning from the vast, uncurated data available on the web.
Summary updated