VIoTGPT: Learning to Schedule Vision Tools towards Intelligent Video Internet of Things

Yaoyao Zhong, Mengshi Qi, Rui Wang, Yuhan Qiu, Yang Zhang, Huadong MaarXiv 2023 · DOI · arXiv

Abstract

The proliferation of Video Internet of Things (VIoT) sensors has created a massive influx of data that traditional vision models struggle to process in a unified, context-aware manner. We present VIoTGPT, a framework that transforms Large Language Models (LLMs) into intelligent agents capable of scheduling specialized vision tools. By employing ReAct instruction tuning on a curated dataset of 200,000 instruction pairs, VIoTGPT learns to autonomously decompose complex surveillance queries into executable sequences of vision algorithms. Our results demonstrate that fine-tuned open-source models can significantly outperform general-purpose LLMs like ChatGPT in multi-step tool-chaining tasks, achieving high decision accuracy and robust tool selection.

Introduction

The Video Internet of Things (VIoT) generates staggering amounts of data from ubiquitous sensors. While general-purpose "foundation models" have made strides in computer vision, they often lack the domain-specific expertise required for high-stakes surveillance, such as gait recognition or precise license plate identification. Furthermore, real-world queries are often complex and interrelated, requiring the output of one tool to inform the use of another. Existing LLMs frequently struggle with these multi-step reasoning tasks, often "hallucinating" tool names or failing to maintain context across a chain of operations.

Overview of the VIoTGPT framework showing the LLM as a central scheduler for vision tools and video knowledge bases.

Methodology: The VIoTGPT Framework

VIoTGPT positions the LLM as an "intelligent agent" that bridges the gap between human users, a video knowledge base, and a suite of specialized vision tools. The core of this approach is ReAct Instruction Tuning, which trains the model to follow a "Reason + Act" loop.

ReAct Instruction Tuning

Instead of providing a direct answer, the model is trained to generate a structured sequence:

  1. Thought: The model articulates the reasoning behind selecting a specific tool.
  2. Action: It selects a tool from a predefined set (e.g., Face Recognition, Anomaly Detection).
  3. Action Input: It identifies the necessary parameters, such as specific video files or geographic locations.
  4. Observation: It processes the tool's output and determines if further action is required or if a final answer can be formulated.
The multi-step ReAct reasoning process used by VIoTGPT to chain vision tools.

To ensure computational efficiency, the researchers utilized LoRA (Low-Rank Adaptation). This allowed for the fine-tuning of 7-billion-parameter models, such as Llama and Vicuna, on consumer-grade hardware like NVIDIA RTX 4090s.

Experimental Setup and Datasets

The backbone of VIoTGPT is the VIoT-Tool dataset, a meticulously crafted collection of 200,000 instruction pairs totaling 2.79 billion tokens. The dataset covers 11 representative vision tools categorized into:

  • Human-centric: Face recognition, Person Re-ID, Gait recognition, and Crowd counting.
  • Vehicle-centric: Vehicle Re-ID and License plate recognition.
  • Event-related: Fire/smoke detection, Anomaly detection, and Action recognition.

The knowledge base simulates a real-world distributed sensor network using a collection of videos associated with 125 different city-level names.

Key Results

The evaluation of VIoTGPT reveals significant improvements over base models and general-purpose LLMs:

  • Decision Accuracy: VIoTGPT (based on Vicuna) achieved a 94.13% accuracy in determining when to invoke a tool, whereas the base Llama model failed almost entirely.
  • Superiority in Interrelated Tasks: In complex scenarios requiring multiple tools, VIoTGPT (Vicuna) reached a "Whole Response" accuracy of 52.00%. In contrast, ChatGPT (few-shot) struggled with hallucinations and redundant tool calls, scoring only 28.25%.
  • Robust Tool Selection: The model demonstrated high proficiency in distinguishing between semantically similar tools, such as differentiating between License Plate Recognition and Vehicle Re-identification based on the user's specific intent.

Conclusion and Future Work

VIoTGPT demonstrates that fine-tuning smaller, open-source LLMs on specialized instruction sets can create highly reliable schedulers for complex IoT environments. While the model excels at tool invocation, challenges remain regarding the generalization of diverse human queries and occasional "silence" where the model fails to provide final natural language feedback. Future research will focus on scaling the framework with more powerful LLMs and expanding the toolset to incorporate a wider array of environmental sensors.

Summary updated

Abstract

Video Internet of Things (VIoT) has shown full potential in collecting an unprecedented volume of video data. How to schedule the domain-specific perceiving models and analyze the collected videos uniformly, efficiently, and especially intelligently to accomplish complicated tasks is challenging. To address the challenge, we build VIoTGPT, the framework based on LLMs to correctly interact with humans, query knowledge videos, and invoke vision models to analyze multimedia data collaboratively. To support VIoTGPT and related future works, we meticulously crafted the VIoT-Tool dataset, including the training dataset and the benchmark involving 11 representative vision models across three categories based on semi-automatic annotations. To guide LLM to act as the intelligent agent towards intelligent VIoT, we resort to the ReAct instruction tuning method based on VIoT-Tool to learn the tool capability. Quantitative and qualitative experiments and analyses demonstrate the effectiveness of VIoTGPT. We believe VIoTGPT contributes to improving human-centered experiences in VIoT applications. The project website is https://github.com/zhongyy/VIoTGPT.
BibTeX
@misc{zhong2023viotgpt,
  author = {Yaoyao Zhong and Mengshi Qi and Rui Wang and Yuhan Qiu and Yang Zhang and Huadong Ma},
  title = {{VIoTGPT: Learning to Schedule Vision Tools towards Intelligent Video Internet of Things}},
  year = {2023},
  eprint = {2312.00401},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.00401},
}
Figure