PV-VTT: A Privacy-Centric Dataset for Mission-Specific Anomaly Detection and Natural Language Interpretation

Ryozo Masukawa, Sanggeon Yun, Yoshiki Yamaguchi, Mohsen Imani

TL;DR

The authors introduce PV-VTT (Privacy Violation Video To Text), a privacy-preserving dataset focused on detecting precursor criminal activities (like trespassing or voyeurism) using only video feature vectors to protect anonymity. To explain these context-dependent anomalies without incurring massive API costs, they propose a hierarchical Graph Neural Network (GNN) framework that extracts key reasoning paths to prompt an LLM using just a single video frame, slashing token costs by over 85% while maintaining high caption quality.

What Problem Does This Paper Address?

Most Video Anomaly Recognition (VAR) systems focus on detecting severe crimes—such as explosions, assaults, or burglaries—after they have already occurred. To truly prevent crime, AI systems need to recognize preemptive indicators or precursor activities, many of which manifest as privacy violations (e.g., casing a building with a drone, trespassing, or taking unauthorized photographs).

However, privacy violations are highly nuanced and context-dependent. For instance, determining if someone is trespassing requires understanding whether they have the property owner's permission. A simple binary alert system would trigger massive false positives. Therefore, models must be capable of reasoning and providing natural language explanations. Unfortunately, feeding entire multi-frame video clips into state-of-the-art Large Language Models (LLMs) for video description is prohibitively expensive due to token-heavy video patch processing.

Methods

The authors develop a cost-effective framework that combines the structural reasoning of GNNs with the generative power of LLMs:

  1. Mission-Specific Knowledge Graphs (KGs): For each privacy violation category, the model uses GPT-4o to generate a set of "Key Concepts," which are then expanded using semantic relationships from ConceptNet. Video frame features are integrated into this graph via a specialized "sensor node."
  2. Hierarchical GNN: A GNN processes information through three distinct abstraction layers (Sensor $\rightarrow$ Key Concepts $\rightarrow$ ConceptNet Associations $\rightarrow$ Final Embedding) to classify the video frame.
  3. Impactive Reasoning Path Finding: Once trained, the GNN is frozen. The model calculates the mathematical norm of the messages passing through each edge to identify which paths contributed most to the final classification.
  4. Cost-Effective LLM Prompting: Instead of sending an entire video sequence to an LLM, the framework selects only the single video frame with the highest anomaly score. It bundles this image with the classification label and the natural language text paths extracted from the GNN to prompt the LLM.

Datasets

The paper introduces the PV-VTT dataset:

  • Composition: 702 videos spanning 31 hours (~2.3 million frames).
  • Categories: Features 5 distinct privacy violation types: Drone Trespassing, Trespassing, Pervert (voyeurism/harassment), Peeping Tom, and Unauthorized Photographing, alongside Normal behavior.
  • Privacy-Centric Design: To protect participant confidentiality and comply with copyright laws, no raw video data is released. Instead, the dataset provides frame-level feature vectors extracted using Meta's ImageBind architecture, paired with LLM-anonymized, human-verified text descriptions.
Overview of the PV-VTT dataset structure, showcasing frame-level annotations and text descriptions.

Key Insights and Ideas

  • Feature-Only Benchmarking: PV-VTT proves that computer vision researchers can successfully train and evaluate state-of-the-art anomaly detection and captioning models using frozen multi-modal embeddings (ImageBind vectors) without ever needing access to raw video pixels.
  • GNNs as Token Filters: Instead of forcing a Vision-Language Model to process dozens of video frames—which causes an explosion of token costs—a small, specialized GNN acts as an information bottleneck. It distills the video's temporal and structural meaning into a few highly descriptive text paths.
  • Explanation as a Requirement: For subtle legal concepts like privacy, natural language explanations are not just a luxury for interpretability; they are a core requirement to validate the model's decision-making process before triggering security interventions.

Key Results

  • Anomaly Detection Accuracy: The hierarchical MissionGNN architecture achieved a 94.61% mean AUC (mAUC) on the PV-VTT dataset, outperforming standard baselines like ImageBind+MLP (93.24%).
  • Dramatic Token Reduction: The proposed single-frame + GNN path method reduced the required input length by over 8,000 tokens per video compared to standard video-based LLM prompting (1,123 tokens vs. 9,400+ tokens).
  • Uncompromised Caption Quality: Despite the 85%+ reduction in token usage, the model achieved a BLEU-4 score of 23.42 (compared to 23.00 for full-video prompting) and secured an 80% approval rating in blind human evaluations.

Limitations and Future Work

The authors note that some text descriptions in PV-VTT were generated from news manuscripts. These descriptions occasionally contain external context (such as names or background stories) that cannot be inferred purely from the visual features of the video frames. This discrepancy can confuse models attempting strict vision-to-text alignment. Future work will focus on tighter data curation to ensure all ground-truth text descriptions correspond strictly to visually verifiable actions.

Summary updated

Abstract

Video crime detection is a significant application of computer vision and artificial intelligence. However, existing datasets primarily focus on detecting severe crimes by analyzing entire video clips, often neglecting the precursor activities (i.e., privacy violations) that could potentially prevent these crimes. To address this limitation, we present PV-VTT (Privacy Violation Video To Text), a unique multimodal dataset aimed at identifying privacy violations. PV-VTT provides detailed annotations for both video and text in scenarios. To ensure the privacy of individuals in the videos, we only provide video feature vectors, avoiding the release of any raw video data. This privacy-focused approach allows researchers to use the dataset while protecting participant confidentiality. Recognizing that privacy violations are often ambiguous and context-dependent, we propose a Graph Neural Network (GNN)-based video description model. Our model generates a GNN-based prompt with image for Large Language Model (LLM), which deliver cost-effective and high-quality video descriptions. By leveraging a single video frame along with relevant text, our method reduces the number of input tokens required, maintaining descriptive quality while optimizing LLM API-usage. Extensive experiments validate the effectiveness and interpretability of our approach in video description tasks and flexibility of our PV-VTT dataset.
BibTeX
@inproceedings{masukawa2024pv,
  author = {Ryozo Masukawa and Sanggeon Yun and Yoshiki Yamaguchi and Mohsen Imani},
  title = {{PV-VTT: A Privacy-Centric Dataset for Mission-Specific Anomaly Detection and Natural Language Interpretation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure