Abstract
Public health monitoring of short-form video platforms is increasingly challenged by the evolution of tobacco promotion from explicit advertisements to subtle lifestyle showcases. This paper presents FLAASH (Flow-Attention Adaptive Semantic Hierarchical Fusion), a novel multi-modal framework designed to detect and analyze tobacco-related content. By combining a unique Flow-Attention mechanism with the reasoning capabilities of Large Language Models (LLMs), FLAASH effectively integrates text and video modalities to capture temporal dynamics and contextual nuances. Experimental results demonstrate that FLAASH achieves state-of-the-art performance on the Multimodal Tobacco Content Analysis Dataset (MTCAD) and general Video Question Answering (VQA) benchmarks.
Introduction
The proliferation of short-form video content on platforms like TikTok and YouTube has made manual public health surveillance nearly impossible. Tobacco promotion has adapted to these platforms, often appearing as product reviews or usage scenarios rather than blatant ads. Traditional AI models frequently fail to identify these instances due to three primary challenges:
- Modality Imbalance: Over-reliance on either text or visual pixels, leading to poor integration.
- Temporal Dynamics: Tobacco use is an action-based event that requires understanding context over time rather than in static frames.
- Contextual Nuances: Distinguishing between health warnings and product promotion requires high-level semantic reasoning.
Methodology
FLAASH addresses these challenges through a three-stage architecture:
1. Frozen Encoders
The framework utilizes a BERT-base model for text and a 3D ResNet-101 for video. These encoders are kept frozen to leverage their pre-trained knowledge while maintaining computational efficiency.
2. The FlowAHCAF Module
The core innovation is the Flow-Attention Adaptive Hierarchical Cross-Attention Fusion (FlowAHCAF) module. Inspired by flow network theory, it treats modality interaction as a conservation problem, ensuring information flows between text and video without loss or "trivial attention" on irrelevant background noise. This module operates across four hierarchical levels to capture both fine-grained textures and high-level scene concepts.
3. LLM Integration and Multi-Task Learning
Fused features are fed into an LLM (Llama-2 or Llama-3) fine-tuned via Low-Rank Adaptation (LoRA). This allows the model to act as a reasoning engine, interpreting social media slang and intent. The training objective optimizes for classification, sentiment, temporal consistency, and feature discrimination.
Datasets
The researchers evaluated FLAASH using the Multimodal Tobacco Content Analysis Dataset (MTCAD), which contains 5,730 videos from YouTube and TikTok annotated for product types and sentiment. To ensure robustness, the model was also tested on general VQA benchmarks including MSVD-QA, MSRVTT-QA, and ActivityNet-QA.
Key Results and Discussion
FLAASH significantly outperforms existing baselines in both specialized and general tasks:
- Tobacco Analysis: Achieved a classification accuracy of 0.83 on MTCAD, a 3% improvement over the previous state-of-the-art, VideoFormer.
- Generalization: Set a new benchmark on MSVD-QA with 73.5% accuracy, proving its utility beyond tobacco-specific domains.
- Ablation Studies: Removing the Flow-Attention component resulted in a 7% accuracy drop, highlighting its critical role in effective modality fusion.
- Temporal Consistency: The model achieved a score of 0.75, indicating stable and reliable analysis throughout the duration of video clips, whereas previous methods often produced flickering or inconsistent predictions.
Conclusion and Future Work
FLAASH represents a significant advancement in multi-modal video understanding for public health. While highly effective, the authors note that the computational intensity of hierarchical fusion with LLMs may limit real-time moderation capabilities. Future research will focus on multilingual support to address the global nature of social media and implementing continuous learning strategies to keep pace with rapidly evolving tobacco products and social media trends.
Summary updated