Leveraging Computer Vision and Visual LLMs for Cost-Effective and Consistent Street Food Safety Assessment in Kolkata India

Alexey Chernikov, Klaus Ackermann, Caitlin Brown, Denni TommasiAAAI 2025 · DOI

Introduction

Street food is a cornerstone of urban nutrition and employment in developing nations. However, it remains a significant vector for foodborne illnesses, contributing to hundreds of thousands of deaths annually. Monitoring food safety at scale presents a formidable logistical challenge. Traditional human-led inspections are not only expensive and slow but also inherently subjective. Local inspectors are susceptible to "survey fatigue" and cognitive biases, often overlooking critical infrastructure gaps due to familiarity or exhaustion. This research proposes an automated, scalable, and objective pipeline to assess food safety infrastructure—such as hand-washing stations and waste management systems—using mobile technology and advanced artificial intelligence.

Methodology

The researchers developed a decentralized multi-stage pipeline designed to balance computational efficiency with high-fidelity analysis. The system is architected to run partially on mobile devices and partially on local government servers to ensure data privacy and reduce operational costs.

The multi-stage pipeline for automated food safety assessment, integrating frame selection, object detection, and VLLM analysis.

The Automated Pipeline

  1. Frame Selection: To ensure high-quality input, the system utilizes optical flow algorithms to filter out blurry or low-quality frames from raw video streams.
  2. Object Detection: A lightweight YOLOv10-N model is employed to identify four critical infrastructure categories: hand-washing stations, dishwashing areas, garbage bins, and water tanks. The model was specifically tuned for high precision to prevent the Visual Large Language Model (VLLM) from generating "hallucinated" assessments based on misidentified objects.
  3. Feature Extraction and Clustering: To avoid redundant processing, MobileNetV3 extracts features from detected objects, which are then grouped via K-Means clustering. This allows the system to select the five most unique views of an object, providing the VLLM with diverse perspectives without the cost of processing every video frame.
  4. VLLM Assessment: Representative images are analyzed by fine-tuned VLLMs, specifically Google PaliGemma or Microsoft Florence-2. These models answer binary safety questions (e.g., "Is the garbage bin covered?") to determine compliance with health standards.

Data Collection and Benchmarking

The study is grounded in a unique multi-modal dataset collected in Kolkata, India, comprising 7,510 videos and 7,000 images from 244 vendors. This data was paired with traditional social science survey responses. For benchmarking, a test set of 258 images and 864 expert-verified questions was used to compare the AI's performance against human surveyors.

Results and Comparative Analysis

The empirical results demonstrate that fine-tuned VLLMs significantly outperform human surveyors in consistency and accuracy. In scenarios where humans and AI disagreed, expert reviews revealed that the AI was frequently correct, while humans failed due to oversight or fatigue.

Performance comparison across models and human surveyors, highlighting the superior accuracy and consistency of fine-tuned VLLMs.

As shown in the performance metrics, fine-tuned models like PaliGemma and Florence-2 achieved accuracies between 71% and 74%, whereas human surveyors dropped to 60% accuracy in complex "disagreement" cases. While proprietary models like ChatGPT-4o provided a high benchmark for accuracy (77%), the smaller, fine-tuned models proved highly competitive and significantly more cost-effective for large-scale government deployment. PaliGemma, in particular, demonstrated high efficiency, processing batches in roughly 154 seconds on consumer-grade hardware.

Discussion and Future Work

A key insight from this study is the value of decentralization. By performing detection and clustering on-device, the system protects vendor privacy and reduces server workloads. However, limitations remain in spatial reasoning; the AI occasionally struggles with complex visual scenes or non-standard infrastructure, such as repurposed plastic bags used as garbage bins.

Future research aims to scale this methodology to 100 cities across India. By expanding the library of detectable facilities and refining spatial reasoning capabilities, this system could provide a comprehensive, real-time map of urban infrastructure gaps, ultimately reducing the burden of foodborne illness in developing urban centers.

Summary updated

BibTeX
@inproceedings{chernikov2025leveraging,
  author = {Alexey Chernikov and Klaus Ackermann and Caitlin Brown and Denni Tommasi},
  title = {{Leveraging Computer Vision and Visual LLMs for Cost-Effective and Consistent Street Food Safety Assessment in Kolkata India}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i27.35008},
}
Figure