DENOISER: Rethinking the Robustness for Open-Vocabulary Action Recognition

Haozhe Cheng, Cheng Ju, Haicheng Wang, Jinxiang Liu, Mengting Chen, Qiang Hu, Xiaoyun Zhang, Yanfeng Wang

Introduction

Open-Vocabulary Action Recognition (OVAR) represents a significant milestone in video understanding, enabling models to identify actions based on arbitrary text descriptions. However, this capability relies heavily on the assumption of pristine text input. In real-world scenarios, user-provided labels are often corrupted by typos, incorrect tenses, or missing characters.

Recent evaluations of state-of-the-art OVAR models, such as ActionCLIP and X-CLIP, reveal a startling vulnerability: even a 5-10% character-level noise rate leads to a precipitous drop in classification accuracy. This fragility stems from textual ambiguity and the propagation of errors when using independent, vision-agnostic spell-checkers.

OVAR models are highly sensitive to noise in class names, as shown by the performance degradation across UCF101, K700, and HMDB51 datasets.

The DENOISER Framework

To address these challenges, we introduce DENOISER, a framework that treats text denoising as a joint generative-discriminative task. Instead of relying on a separate preprocessing step, DENOISER creates a feedback loop that leverages visual information to disambiguate noisy text.

Methodology

The framework operates through two alternating phases:

  1. Generative Step ($\Psi_{gene}$): This stage decodes noisy text by generating candidates based on Levenshtein Edit Distance. These candidates are then ranked using a dual-weighting scheme:
  2. Intra-modal weighting: Evaluates spelling similarity.
  3. Inter-modal weighting: Uses visual embeddings to "vote" on the most likely candidate, providing critical context that text-only models lack.
  4. Discriminative Step ($\Psi_{disc}$): A standard OVAR model uses the refined class names to classify visual samples.
The DENOISER framework iterates between a Generative Step for text proposal and an Inter-modal/Intra-modal weighting process, and a Discriminative Step for visual classification.

The iterative nature of DENOISER allows the model to progressively refine its understanding: better text leads to more accurate video classification, which in turn provides stronger visual evidence for further text denoising.

Experimental Results

DENOISER was evaluated on UCF-101, HMDB-51, and Kinetics-700 datasets using ActionCLIP and X-CLIP as base architectures.

  • Robustness Gains: On the UCF-101 dataset with 20% noise, DENOISER improved ActionCLIP's accuracy from 32.3% to 54.8%.
  • Comparison with LLMs: DENOISER consistently outperformed general-purpose language models like GPT-3.5. While LLMs possess vast linguistic knowledge, they lack the visual grounding necessary to resolve ambiguities like "palying bll" (ball vs. bell) in a video context.
  • Visual Grounding: The "visual vote" mechanism effectively filters out outliers, proving more robust than adversarial training, which often degrades zero-shot performance.

Conclusion and Future Work

DENOISER demonstrates that visual context is a powerful tool for linguistic denoising in OVAR. By integrating vision and language in an iterative loop, the framework significantly enhances model robustness against real-world input noise. Future research will explore the integration of more sophisticated language models and the application of this vision-assisted denoising to other domains like object detection and segmentation.

Summary updated

Abstract

As one of the fundamental video tasks in computer vision, Open-Vocabulary Action Recognition (OVAR) has recently gained increasing attention, with the development of vision-language pre-trainings. To enable open-vocabulary generalization, existing methods formulate vanilla OVAR to evaluate the embedding similarity between visual samples and text descriptions. However, one crucial issue is completely ignored: the text descriptions given by users may be noisy, e.g., misspellings and typos, limiting the real-world practicality. To fill the research gap, this paper analyzes the noise rate/type in text descriptions by full statistics of manual spelling; then reveals the poor robustness of existing methods; and finally rethinks to study a practical task: noisy OVAR. One novel DENOISER framework, covering two parts: generation and discrimination, is further proposed for solution. Concretely, the generative part denoises noisy text descriptions via a decoding process, i.e., proposes text candidates, then utilizes inter-modal and intra-modal information to vote for the best. At the discriminative part, we use vanilla OVAR models to assign visual samples to text descriptions, injecting more semantics. For optimization, we alternately iterate between generative-discriminative parts for progressive refinements. The denoised text descriptions help OVAR models classify visual samples more accurately; in return, assigned visual samples help better denoising. We carry out extensive experiments to show our superior robustness, and tho rough ablations to dissect the effectiveness of each component.
BibTeX
@misc{cheng2024denoiser,
  author = {Haozhe Cheng and Cheng Ju and Haicheng Wang and Jinxiang Liu and Mengting Chen and Qiang Hu and Xiaoyun Zhang and Yanfeng Wang},
  title = {{DENOISER: Rethinking the Robustness for Open-Vocabulary Action Recognition}},
  year = {2024},
  eprint = {2404.14890},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.14890},
}
Figure