T2Vs Meet VLMs: A Scalable Multimodal Dataset for Visual Harmfulness Recognition

Chen Yeh, You-Ming Chang, Wei-Chen Chiu, Ning Yu

As generative AI makes it trivial to create highly realistic synthesized content, traditional safety filters are struggling to keep pace. While widespread access to Text-to-Image and Text-to-Video models boosts creativity, it also exponentially increases the risk of encountering inappropriate or harmful media.

Existing datasets designed to train AI in harmfulness recognition suffer from three major flaws:

  1. Narrow Scope: They often focus exclusively on specific objects (e.g., weapons or nudity) rather than broader, more complex concepts like misinformation, self-harm, or animal welfare.
  2. Lack of Context: A knife on a kitchen cutting board is benign; a knife held threateningly is not. Most current detectors lack the "common sense" to distinguish between the two.
  3. Real-World Bias: Most datasets ignore the unique artifacts and styles of AI-generated content, which is rapidly becoming a primary vector for online harm.

To address these gaps, researchers have introduced VHD11K, a comprehensive, large-scale dataset designed to train AI to recognize a full spectrum of harmful content across both real-world and AI-generated media.

The VHD11K Dataset

VHD11K consists of 10,000 images and 1,000 videos spanning 10 distinct harmful categories. To ensure the dataset reflects the modern digital landscape, the media is split evenly between real-world and synthesized content.

Real-world data was crawled from platforms like Google and YouTube. Synthesized data was generated using state-of-the-art generative models, including SDXL, HiGen, and AnimateDiff, with prompts crafted by Llama3.

Overview of the VHD11K dataset creation pipeline, showing both real and synthesized content streams.

The Multi-Agent Debate Framework

Annotating complex, context-dependent harmful content is notoriously difficult and expensive. To ensure high-quality labels without relying solely on human annotators, the researchers pioneered a novel Debate Annotation Framework.

Instead of asking a single Vision-Language Model (VLM) to classify an image, the framework formulates the annotation process as a multi-agent Visual Question Answering (VQA) task. Three different GPT-4V instances play distinct roles:

  • The Affirmative Debater: Argues why the image or video is harmful.
  • The Negative Debater: Argues why the content is safe or benign.
  • The Judge: Evaluates the arguments and makes the final verdict.
The two-round debate process featuring Affirmative, Negative, and Judge agents.

The debaters engage in a two-round discussion. This forces the AI to deeply consider the context and edge cases (e.g., distinguishing a toy gun from a real one) before the Judge makes a final decision. To keep the AI aligned with human values, the agents are provided with In-Context Learning (ICL) examples of human-labeled "tricky" cases during the debate. Finally, Llama3 is used to distill the messy debate text into clear, structured reasoning.

Key Insights and Results

The introduction of VHD11K and the debate framework yielded several critical insights into the current state of visual safety filters:

  • Human Alignment is High: The VLM debate framework achieved an impressive 82.5% accuracy when compared against human labels on the baseline SMID (Socio-Moral Image Database) dataset. This proves that multi-agent AI debates are a viable, scalable proxy for expensive human annotation.
  • Context is King: The debate format significantly reduces false positives. By forcing a "negative debater" to find reasons why an image might be safe, the system avoids the over-censorship common in simpler AI filters.
  • The Synthetic Gap is Real: Existing detectors trained primarily on real-world data performed poorly on AI-generated harm. Furthermore, most current "harmfulness detectors" showed a massive bias toward labeling everything as "unharmful" when faced with the diverse, non-traditional categories in VHD11K. For example, specialized filters like NudeNet failed almost entirely on non-nudity harms.
  • Performance Boosts: Finetuning general VLMs on the VHD11K dataset significantly improved their performance. When InstructBLIP was finetuned on VHD11K, its average accuracy on image tasks jumped from roughly 68% to 75.5%, demonstrating a much better balance between detecting actual harm and avoiding false alarms.
  • Categorization vs. Binary Classification: Interestingly, VLMs performed better at identifying which of the 10 harmful categories an image belonged to than simply answering a binary "Is this harmful: Yes/No." The extra semantic information required for categorization helps the model reason more effectively.

Limitations and Future Work

While VHD11K represents a significant step forward, the authors note a few limitations. First, using GPT-4V as a "judge and jury" for thousands of images is computationally expensive compared to smaller, dedicated models. Second, for videos, the dataset relies on extracting only 5 frames to represent the content. While effective for many cases, this sparse sampling might miss harmfulness that relies on specific, fleeting temporal actions.

Finally, "harmfulness" remains culturally subjective. While the authors utilized established frameworks, future work could involve tuning the "Judge" agent to reflect different regional, cultural, or community-specific safety standards.

Summary updated

Abstract

While widespread access to the Internet and the rapid advancement of generative models boost people's creativity and productivity, the risk of encountering inappropriate or harmful content also increases. To address the aforementioned issue, researchers managed to incorporate several harmful contents datasets with machine learning methods to detect harmful concepts. However, existing harmful datasets are curated by the presence of a narrow range of harmful objects, and only cover real harmful content sources. This restricts the generalizability of methods based on such datasets and leads to the potential misjudgment in certain cases. Therefore, we propose a comprehensive and extensive harmful dataset, **VHD11K**, consisting of 10,000 images and 1,000 videos, crawled from the Internet and generated by 4 generative models, across a total of 10 harmful categories covering a full spectrum of harmful concepts with non-trival definition. We also propose a novel annotation framework by formulating the annotation process as a multi-agent Visual Question Answering (VQA) task, having 3 different VLMs "debate" about whether the given image/video is harmful, and incorporating the in-context learning strategy in the debating process. Therefore, we can ensure that the VLMs consider the context of the given image/video and both sides of the arguments thoroughly before making decisions, further reducing the likelihood of misjudgments in edge cases. Evaluation and experimental results demonstrate that (1) the great alignment between the annotation from our novel annotation framework and those from human, ensuring the reliability of VHD11K; (2) our full-spectrum harmful dataset successfully identifies the inability of existing harmful content detection methods to detect extensive harmful contents and improves the performance of existing harmfulness recognition methods; (3) our dataset outperforms the baseline dataset, SMID, as evidenced by the superior improvement in harmfulness recognition methods. The entire dataset is publicly available: https://huggingface.co/datasets/denny3388/VHD11K
BibTeX
@inproceedings{yeh2024t,
  author = {Chen Yeh and You-Ming Chang and Wei-Chen Chiu and Ning Yu},
  title = {{T2Vs Meet VLMs: A Scalable Multimodal Dataset for Visual Harmfulness Recognition}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=0G8AXwtmy2},
}
Figure