Guest Editorial: Learning From Noisy Multimedia Data

Jian Zhang, Alan Hanjalic, Ramesh C. Jain, Xian-Sheng Hua, Shin'ichi Satoh, Yazhou Yao, Dan Zeng

Abstract

This special issue of IEEE Transactions on Multimedia addresses the critical challenge of "noise"—specifically label errors and domain mismatches—inherent in massive, automatically collected web datasets. It curates 19 high-impact papers that propose novel architectures, loss functions, and purification strategies to make deep learning more robust when training on the "wild" data of the internet. This editorial summarizes the core contributions, ranging from data purification and self-supervision to adversarial domain adaptation and uncertainty modeling.

1. Introduction: The Challenge of Noisy Web Data

We live in an era of exponential data growth. Platforms like TikTok, Instagram, and YouTube provide hundreds of millions of data points that serve as a goldmine for training robust machine learning models. However, because this data is typically collected automatically through web scraping or search engines, it is notoriously "noisy."

The research presented in this issue identifies two primary hurdles:

  1. Label Noise: Metadata, tags, or user-provided descriptions are often incorrect, misleading, or missing.
  2. Domain Mismatch: There is frequently a significant gap between the "web domain" (where models are trained) and the "real-world domain" (where they are deployed).

Standard deep learning models tend to "memorize" this noise, leading to poor generalization and degraded performance. This editorial introduces a collection of research aimed at filtering, adapting to, or learning through this noise across various modalities, including video, image, and text.

2. Methodological Innovations

The 19 featured papers employ a diverse array of advanced techniques to mitigate the impact of noise. These methods can be categorized into several innovative strategies:

  • Data Purification & Self-Supervision: Zhang et al. utilize a self-supervised "rotation prediction" task to identify outliers in noisy web images, effectively cleaning the dataset before formal training begins.
  • Dynamic Loss Correction: Several papers propose modifying the loss function during training. For instance, the Dynamic Training Data Dropout (DTDD) method tracks the "fluctuation score" of samples across epochs; samples that cause high model volatility are flagged as noise and excluded from training.
  • Co-Training & Label Distribution: The Co-LDL framework trains two divergent networks simultaneously. They perform a "peer-review" of each other’s samples, selecting low-loss samples for conventional training and using high-loss samples to update and "smooth" label distributions.
  • Adversarial Domain Adaptation: To bridge domain gaps, researchers utilized adversarial learning with "classification-constrained discriminators." This ensures that while features become domain-invariant, they still retain the structural information necessary for tasks like semantic segmentation.
  • Uncertainty Modeling: Using Variational Graph Auto-encoders (MVGAE), one study models the variance of latent factors to represent the noise level of specific modalities within recommendation systems.

3. Datasets and Benchmarking

The papers in this issue utilize a mix of classic benchmarks and newly introduced datasets designed to test model robustness in realistic scenarios:

  • Evolving-Image-Search (EVIS): A new benchmark containing images of vehicles and digital products from 2009 to 2020, specifically designed to test "gradual" domain adaptation over time.
  • Standard Benchmarks: Amazon Movies, Amazon Electronics, and AliShop-7C are used to evaluate recommendation systems.
  • Specialized Data: New noisy skeleton datasets for human action recognition and web-scraped recipe-image pairs for food recognition provide challenging testbeds for multi-modal learning.

4. Key Insights and Results

4.1 Theoretical and Practical Insights

  • Noise as a Signal: Rather than simply deleting noisy data, some methods treat high-loss samples as a source of information to refine the model’s understanding of label ambiguity (e.g., Label Distribution Learning).
  • The Fluctuation Metric: A key discovery in face recognition research was that noise is characterized not just by high loss, but by inconsistent predictions across training iterations.
  • Cross-Modal Filtering: In video retrieval, the text query itself can be used as a "denoiser" for the visual domain, helping the model focus on relevant temporal moments while ignoring "interference noise" from other actions in the same scene.

4.2 Quantitative Performance

While the editorial summarizes 19 different papers, the overarching quantitative theme is a significant leap over State-of-the-Art (SOTA) in noisy environments. For example, the CDN method achieved superior localization accuracy in video moment retrieval with constrained inference costs. Similarly, the DTDD method demonstrated the ability to robustly train deep models for face recognition even in the presence of heavy label noise and low-quality imagery.

5. Limitations and Future Directions

Despite the progress shown, several challenges remain for future research:

  • Computational Overhead: Many robust methods, such as co-training multiple networks, increase the computational cost of training, which may be a barrier for smaller research institutions.
  • Complex Modalities: While image and video noise are being addressed, "recipe" and complex structured text data still prove difficult to learn from noisy web pairs. A study on food recognition (Zhu et al.) found that noisy web pairs are less effective for learning recipe (text) features compared to image features.
  • Temporal Evolution: As shown by the EVIS dataset, data distributions change over time. Future work must move toward "active gradual adaptation" to keep models relevant as the web and its data distributions evolve.

Summary updated

Abstract

This special issue provides a premier forum for researchers in multimedia big data to share challenges and recent advancements in learning from noisy multimedia data. The multimedia age and its proliferation of devices and platforms is fueling exponential data growth. As computational power and deep learning algorithms rapidly evolve, the web has become a rich source of potential training data for robust machine learning, with search engines such as Google and Bing, Twitter, TikTok, Instagram, and short video sharing platforms offering large-scale data points in the hundreds of millions. The concurrent shift in the Internet to richer web data modalities such as text, audio, image, and video reveal further opportunities to leverage large-scale data for the automatic construction of a variety of datasets for model training and testing. However, the ubiquity of multimedia data means noise is a fundamental challenge, with ‘label noise’ and ‘domain mismatch’ the most critical issues in automatically collected datasets. Learning from noisy multimedia data tends towards poor performance, making it increasingly essential to address these challenges.
BibTeX
@article{zhang2022guest,
  author = {Jian Zhang and Alan Hanjalic and Ramesh C. Jain and Xian-Sheng Hua and Shin'ichi Satoh and Yazhou Yao and Dan Zeng},
  title = {{Guest Editorial: Learning From Noisy Multimedia Data}},
  journal = {IEEE Trans. Multimedia},
  year = {2022},
  doi = {10.1109/tmm.2022.3159014},
}
Figure