DENOISER: Rethinking the Robustness for Open-Vocabulary Action Recognition

Haozhe Cheng, Chen Ju, Haicheng Wang, Jinxiang Liu, Mengting Chen, Xiaoyun Zhang, Yanfeng WangSubmission to NeurIPS 2024 · Rejected

TL;DR

Current open-vocabulary action recognition (OVAR) models assume perfectly typed class labels, but real-world user queries are frequently riddled with typos and misspellings that tank recognition accuracy. DENOISER introduces an alternating generative–discriminative framework that cleans up noisy text at test time by letting visual video samples help "vote" on spelling corrections, significantly outperforming text-only spell-checkers and large language models (LLMs).

What Problem Does This Paper Address?

Open-Vocabulary Action Recognition (OVAR) matches video representations to free-form text labels using vision-language models (e.g., ActionCLIP, XCLIP). In benchmark setups, class names are curated and clean. In real-world applications, however, user input contains spelling errors, omitted letters, and typos (e.g., "wal4ingm with a dog" or "boird").

The authors reveal that even slight character-level noise causes catastrophic performance drops across state-of-the-art OVAR models. Addressing this via standard uni-modal spell-checkers or LLMs fails for two reasons:

  1. Textual Ambiguity: Action class prompts are usually brief (a few compact words) and lack the sentence-level context needed for language models to disambiguate corrections (e.g., whether "boird" was intended to be "bird" or "board").
  2. Cascaded Errors: Decoupled, pipelined text-cleaning propagates errors directly into visual classification.

Methods

DENOISER solves noisy OVAR at inference time by coupling text correction and video recognition in an iterative, alternating loop:

[Noisy Text Input] 
       │
       ▼
┌────────────────────────────────────────────────────────┐
│ 1. Text Proposals:                                     │
│    Generate top-K candidate words using edit distance  │
└───────────────────────┬────────────────────────────────┘
                        │
                        ▼
┌────────────────────────────────────────────────────────┐
│ 2. Discriminative Step:                                │
│    Vanilla OVAR model assigns video samples to classes │
└───────────────────────┬────────────────────────────────┘
                        │
                        ▼
┌────────────────────────────────────────────────────────┐
│ 3. Generative Denoising Step:                          │
│    Select best candidate using joint scoring:          │
│    • Intra-modal: character edit distance score        │
│    • Inter-modal: assigned video consensus score      │
└───────────────────────┬────────────────────────────────┘
                        │
                        ▼
             [Next Iteration / Output]
  • Candidate Text Proposals: For the target word at step $i$, the model retrieves $K$ candidates from an English vocabulary minimizing Levenshtein edit distance.
  • Discriminative Step ($\Psi_{\text{disc}}$): An off-the-shelf, frozen OVAR model temporarily assigns test videos to class candidates, identifying which visual instances correspond to each semantic category.
  • Generative Denoising Step ($\Psi_{\text{gene}}$): The candidate word is chosen by maximizing a joint objective: $$\Phi_{\text{intra}} \times \prod_{v_j \in \mathcal{V}_c} \Phi_{\text{inter}}$$
  • Intra-modal Weighting ($\Phi_{\text{intra}}$): Evaluates spelling similarity via edit distance.
  • Inter-modal Weighting ($\Phi_{\text{inter}}$): Allows the assigned video clips to cast votes on the candidate phrase via visual–textual cosine similarity.
  • Dynamic Temperature Schedule: A parameter $\lambda$ linearly increases across decoding steps, steering the model to rely primarily on spelling similarity early on, then shifting weight toward visual evidence as the text becomes clearer.

Datasets

  • Downstream Video Action Benchmarks: Evaluated on UCF101 (101 classes), HMDB51 (51 classes), and Kinetics700 (tested on novel classes disjoint from Kinetics400 pre-training).
  • Noise Statistics & Simulation: Real-world error patterns and rates were quantified using the GitHub Typo Corpus (350k+ developer edits) and the TOEFL-Spell Corpus (misspellings from English learners), alongside simulated character substitutions, insertions, and deletions.

Key Insights and Ideas

  • Visual Context Disambiguates Compact Text: When a text model cannot determine whether "boird" means "bird" or "board", a 2-second video of an ocean wave or an animal instantly settles the ambiguity.
  • Test-Time Correction Trumps Adversarial Training: The authors experimented with pre-training OVAR models on noisy labels via adversarial training. This degraded zero-shot generalization on clean data and failed to generalize to unseen test distributions. Resolving noise at inference time is significantly more effective.
  • Mutual Reinforcement: Clean text yields more accurate video cluster assignments; more accurate video assignments cast cleaner votes for spelling candidate selection.

Key Results

  • Superiority over Text-Only Baselines: On UCF101 (using ActionCLIP), DENOISER achieved 61.5% Top-1 Accuracy under real noise (~5.52% noise rate), outperforming GPT-3.5 (61.2%), PySpellChecker (59.9%), and NeuSpell-BERT (56.0%). Under 10% simulated noise, DENOISER maintained 61.2% accuracy, while GPT-3.5 dropped to 58.5% and PySpellChecker fell to 55.7%.
  • Cross-Model Robustness: Under 10% synthetic noise on UCF101:
  • ActionCLIP: Recovers from 47.3% (w/o defense) to 61.2%.
  • XCLIP: Recovers from 46.4% (w/o defense) to 62.9% (close to the 68.6% clean upper bound).
  • Consistent Across Benchmarks: On HMDB51 at 10% noise, DENOISER elevated XCLIP accuracy from 31.8% to 37.3%. On Kinetics700 at 10% noise, it recovered XCLIP accuracy from 27.5% to 41.8%.

Limitations and Future Work

  • Spelling Focus vs. Semantic Noise: DENOISER is tailored for character-level orthographic errors (typos, misspellings). It is not designed to handle high-level semantic ambiguities, slang, or complete phrase substitutions.
  • Information Loss from Character Deletions: Ablations indicate that character deletion is harder to recover than substitution or insertion, as omitted characters cause irrecoverable textual information loss.
  • Inference Overhead: Computing proposals and running cross-modal similarity checks for $K$ candidates over multiple iterations increases test-time computational latency compared to a single forward pass.

Summary updated

Abstract

As one of the fundamental video tasks in computer vision, Open-Vocabulary Action Recognition (OVAR) has recently gained increasing attention, with the development of vision-language pre-trainings. To enable open-vocabulary generalization, existing methods formulate vanilla OVAR to evaluate the embedding similarity between visual samples and text descriptions. However, one crucial issue is completely ignored: the text descriptions given by users may be noisy, e.g., misspellings and typos, limiting the real-world practicality. To fill the research gap, this paper analyzes the noise rate/type in text descriptions by full statistics of manual spelling; then reveals the poor robustness of existing methods; and finally rethinks to study a practical task: noisy OVAR. One novel DENOISER framework, covering two parts: generation and discrimination, is further proposed for solution. Concretely, the generative part denoises noisy text descriptions via a decoding process, i.e., proposes text candidates, then utilizes inter-modal and intra-modal information to vote for the best. At the discriminative part, we use vanilla OVAR models to assign visual samples to text descriptions, injecting more semantics. For optimization, we alternately iterate between generative-discriminative parts for progressive refinements. The denoised text descriptions help OVAR models classify visual samples more accurately; in return, assigned visual samples help better denoising. We carry out extensive experiments to show our superior robustness, and tho rough ablations to dissect the effectiveness of each component.
BibTeX
@misc{cheng2024denoiser,
  author = {Haozhe Cheng and Chen Ju and Haicheng Wang and Jinxiang Liu and Mengting Chen and Xiaoyun Zhang and Yanfeng Wang},
  title = {{DENOISER: Rethinking the Robustness for Open-Vocabulary Action Recognition}},
  year = {2024},
  eprint = {2404.14890},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.14890},
}
Figure