Refined Semantic Enhancement towards Frequency Diffusion for Video Captioning

Xian Zhong, Zipeng Li, Shuqin Chen, Kui Jiang, Chen Chen 0001, Mang Ye

Abstract

Video captioning models frequently struggle with the "long-tail" problem, where rare but descriptive words are ignored in favor of generic, high-frequency terms. This paper introduces Refined Semantic Enhancement towards Frequency Diffusion (RSFD), a framework designed to improve the generation of low-frequency tokens. RSFD utilizes Frequency-Aware Diffusion (FAD) to treat rare words as "noise" that is diffused into semantically similar common words, alongside a Divergent Semantic Supervisor (DSS) to maintain distinct word meanings. Evaluated on MSR-VTT and MSVD benchmarks, RSFD demonstrates significant improvements in descriptive detail and consensus-based metrics like CIDEr.

1. Introduction: The Long-Tail Challenge

Video captioning—automatically describing video content in natural language—has seen great strides thanks to Transformer-based architectures. However, a persistent "long-tail" problem remains: training datasets are dominated by common words (e.g., "person," "playing," "video game").

Because rare words (low-frequency tokens like "keyboard," "minecraft," or "surfing") appear so infrequently, models struggle to learn their specific visual associations. Consequently, at test time, a model might see someone playing a piano and output the safer, more frequent caption "a person is playing a video game." This lack of semantic refinement makes captions feel generic and less useful for real-world applications like accessibility or video search.

The long-tail problem in video captioning: rare words like 'keyboard' are often replaced by generic terms like 'video game' due to insufficient training occurrences.

2. Methodology: The RSFD Framework

The authors propose the RSFD framework, which introduces two novel modules into the standard encoder-decoder architecture to address semantic refinement.

2.1 Frequency-Aware Diffusion (FAD)

Inspired by diffusion models used in image generation, FAD treats low-frequency tokens as "noise." During training, the semantics of these rare tokens are "diffused" into semantically similar high-frequency tokens. This forces the model to perceive and "absorb" the linguistic representations of infrequent words by associating them with their more common counterparts in the semantic space.

2.2 Divergent Semantic Supervisor (DSS)

While FAD helps the model learn rare words, the diffusion process can sometimes blur the distinct meaning of common words. The DSS module acts as a constraint. It uses a "sliding window" approach (Skip-gram style) to supervise the generation of a central word based on its neighbors. This ensures the model maintains distinctive semantics for every word, regardless of its frequency.

The RSFD architecture featuring Frequency-Aware Diffusion (FAD) and the Divergent Semantic Supervisor (DSS).

2.3 Smart Token Categorization

The framework distinguishes between words that are truly rare across the dataset (inter-video frequency) and those that are specific to a certain context (intra-video frequency). This categorization allows for more precise semantic enhancement.

3. Experimental Results

The model was evaluated on two gold-standard benchmarks: MSR-VTT (10,000 clips) and MSVD (1,970 clips). RSFD achieved impressive performance gains, particularly in metrics that favor descriptive detail:

  • CIDEr Scores: On MSR-VTT, RSFD reached 53.1%, a 2.0% improvement over the strong AR-B baseline. On MSVD, it reached 96.7%, a 5.3% jump.
  • Qualitative Superiority: In side-by-side comparisons, while a baseline model might describe a video as "a person is folding paper," RSFD correctly identifies the specific object: "a person is folding a paper airplane." Similarly, it correctly identifies "minecraft" or "keyboard" where other models default to "video game."
  • Efficiency: Unlike some previous attempts to solve this problem, RSFD does not require a massive external language model (like GPT) to be running in the background, making it more computationally efficient.

4. Conclusion and Future Work

RSFD effectively addresses the long-tail problem by viewing rare word learning through the lens of diffusion and contextual supervision. While the model handles semantics exceptionally well, it remains dependent on the quality of pre-trained visual encoders. Future work may involve combining this frequency-aware approach with models that better capture complex temporal interactions and object-to-object relationships to achieve even more human-like video understanding.

Summary updated

Abstract

Video captioning aims to generate natural language sentences that describe the given video accurately. Existing methods obtain favorable generation by exploring richer visual representations in encode phase or improving the decoding ability. However, the long-tailed problem hinders these attempts at low-frequency tokens, which rarely occur but carry critical semantics, playing a vital role in the detailed generation. In this paper, we introduce a novel Refined Semantic enhancement method towards Frequency Diffusion (RSFD), a captioning model that constantly perceives the linguistic representation of the infrequent tokens. Concretely, a Frequency-Aware Diffusion (FAD) module is proposed to comprehend the semantics of low-frequency tokens to break through generation limitations. In this way, the caption is refined by promoting the absorption of tokens with insufficient occurrence. Based on FAD, we design a Divergent Semantic Supervisor (DSS) module to compensate for the information loss of high-frequency tokens brought by the diffusion process, where the semantics of low-frequency tokens is further emphasized to alleviate the long-tailed problem. Extensive experiments indicate that RSFD outperforms the state-of-the-art methods on two benchmark datasets, i.e., MSR-VTT and MSVD, demonstrate that the enhancement of low-frequency tokens semantics can obtain a competitive generation effect. Code is available at https://github.com/lzp870/RSFD.
BibTeX
@inproceedings{zhong2023refined,
  author = {Xian Zhong and Zipeng Li and Shuqin Chen and Kui Jiang and Chen Chen and Mang Ye},
  title = {{Refined Semantic Enhancement towards Frequency Diffusion for Video Captioning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i3.25484},
}
Figure