Introduction
Open-Vocabulary Action Recognition (OVAR) represents a significant milestone in video understanding, enabling models to identify actions based on arbitrary text descriptions. However, this capability relies heavily on the assumption of pristine text input. In real-world scenarios, user-provided labels are often corrupted by typos, incorrect tenses, or missing characters.
Recent evaluations of state-of-the-art OVAR models, such as ActionCLIP and X-CLIP, reveal a startling vulnerability: even a 5-10% character-level noise rate leads to a precipitous drop in classification accuracy. This fragility stems from textual ambiguity and the propagation of errors when using independent, vision-agnostic spell-checkers.
The DENOISER Framework
To address these challenges, we introduce DENOISER, a framework that treats text denoising as a joint generative-discriminative task. Instead of relying on a separate preprocessing step, DENOISER creates a feedback loop that leverages visual information to disambiguate noisy text.
Methodology
The framework operates through two alternating phases:
- Generative Step ($\Psi_{gene}$): This stage decodes noisy text by generating candidates based on Levenshtein Edit Distance. These candidates are then ranked using a dual-weighting scheme:
- Intra-modal weighting: Evaluates spelling similarity.
- Inter-modal weighting: Uses visual embeddings to "vote" on the most likely candidate, providing critical context that text-only models lack.
- Discriminative Step ($\Psi_{disc}$): A standard OVAR model uses the refined class names to classify visual samples.
The iterative nature of DENOISER allows the model to progressively refine its understanding: better text leads to more accurate video classification, which in turn provides stronger visual evidence for further text denoising.
Experimental Results
DENOISER was evaluated on UCF-101, HMDB-51, and Kinetics-700 datasets using ActionCLIP and X-CLIP as base architectures.
- Robustness Gains: On the UCF-101 dataset with 20% noise, DENOISER improved ActionCLIP's accuracy from 32.3% to 54.8%.
- Comparison with LLMs: DENOISER consistently outperformed general-purpose language models like GPT-3.5. While LLMs possess vast linguistic knowledge, they lack the visual grounding necessary to resolve ambiguities like "palying bll" (ball vs. bell) in a video context.
- Visual Grounding: The "visual vote" mechanism effectively filters out outliers, proving more robust than adversarial training, which often degrades zero-shot performance.
Conclusion and Future Work
DENOISER demonstrates that visual context is a powerful tool for linguistic denoising in OVAR. By integrating vision and language in an iterative loop, the framework significantly enhances model robustness against real-world input noise. Future research will explore the integration of more sophisticated language models and the application of this vision-assisted denoising to other domains like object detection and segmentation.
Summary updated