The 7th AI City Challenge

Milind Naphade, Shuo Wang, David C. Anastasiu, Zheng Tang, Ming-Ching Chang, Yue Yao, Liang Zheng, Mohammed Shaiqur Rahman, Meenakshi S. Arya, Anuj Sharma, Qi Feng, Vitaly Ablavsky, Stan Sclaroff, Pranamesh Chakraborty, Sanjita Prajapati, Alice Li, Shangru Li, Krishna Kunadharaju, Shenxin Jiang, Rama Chellappa

Abstract

The seventh edition of the AI City Challenge (2023) focused on two high-potential domains: retail business and Intelligent Traffic Systems (ITS). With a record-breaking 508 teams from 46 countries, the challenge introduced five distinct tracks designed to push the boundaries of real-world computer vision. These tracks covered multi-target multi-camera (MTMC) people tracking, natural-language-based vehicle retrieval, driver action classification, automated retail checkout, and motorcycle helmet violation detection. By leveraging high-fidelity synthetic data and multi-modal architectures, participating teams established new state-of-the-art baselines, demonstrating the increasing viability of AI in complex, privacy-sensitive urban environments.

Introduction

While computer vision has achieved significant success in controlled settings, "AI City" applications—such as autonomous retail and city-wide traffic management—face unique real-world hurdles. These include extreme lighting variations, the necessity of privacy-preserving tracking (avoiding facial recognition), and the "Sim-to-Real" gap, where models trained on simulated data struggle in physical environments. The 2023 AI City Challenge addressed these issues across five high-impact tracks, fostering innovation in both data generation and model architecture.

The Five Challenge Tracks

Track 1: Multi-Target Multi-Camera (MTMC) People Tracking

This track focused on tracking individuals across non-overlapping indoor cameras. To comply with privacy standards, teams were required to track subjects without using facial features, relying instead on gait, clothing textures, and temporal consistency. The dataset included over 8.6 million bounding boxes across both real and synthetic environments.

Track 2: Natural Language Vehicle Track Retrieval

Teams were tasked with identifying specific vehicles across city-wide footage using natural language descriptions (e.g., "a white SUV turning left"). This required aligning visual features with linguistic embeddings to create a searchable video index.

Track 3: Naturalistic Driving Analysis

Focusing on driver safety, this track required the classification of 16 distinct distracted driving behaviors (such as texting, eating, or reaching into the backseat). The goal was not only to identify the action but to pinpoint the exact start and end times within the video stream.

Track 4: Multi-Class Product Counting and Recognition

This track aimed to develop an automated checkout system for retail stores. Using a single overhead camera view, teams had to identify and count products as they were scanned, bridging the gap between synthetic training data and real-world retail environments.

Track 5: Motorcycle Helmet Violation Detection

A new addition for 2023, this track tasked teams with detecting helmet law violations in dense, chaotic traffic. This is a critical safety application for urban environments where motorcycle usage is high and enforcement is difficult.

The winning strategies across all tracks moved away from generic models toward highly customized, multi-stage pipelines. Key trends included:

  • Synthetic-to-Real Pipelines: For Tracks 1 and 4, teams utilized the NVIDIA Omniverse Platform to generate photorealistic synthetic data. This allowed for training on millions of perfectly labeled frames before fine-tuning on limited real-world datasets.
  • Multi-Modal Transformers: Track 2 saw the extensive use of CLIP-based architectures to align visual vehicle features with natural language embeddings.
  • Advanced Clustering and Association: For MTMC tracking, teams employed Hungarian algorithms and hierarchical clustering based on appearance and spatio-temporal constraints to maintain identity across camera "blind spots."
  • Temporal Action Localization (TAL): In driver monitoring, self-supervised pre-trained models like VideoMAE were used to localize distracted behaviors with high temporal precision.

Key Datasets and Benchmarks

The challenge introduced several massive benchmarks to the research community:

  • MTMC People Tracking Dataset: 129 cameras across 7 environments.
  • SynDD2: A synthetic dataset for distracted driving featuring 150 video clips of 16 behaviors.
  • ARC (Automated Retail Checkout): A combination of 116,500 synthetic images and real-world test videos.
  • Bike Helmet Violation Dataset: 200 videos from an Indian city, specifically annotated for riders in congested traffic.

Results and Performance

The 2023 challenge saw a 100% increase in participation over the previous year, resulting in significant performance gains:

  • Track 1: The top team (UW-ETRI) achieved an IDF1 score of 0.9536.
  • Track 2: Reached a Mean Reciprocal Rank (MRR) of 0.8263 for vehicle retrieval.
  • Track 4: Automated retail checkout reached an F1 score of 0.9792, nearing the reliability required for commercial deployment.
  • Track 5: The winning mAP of 0.8340 demonstrated the ability to handle visual noise in dense traffic.

Conclusion and Future Work

The 7th AI City Challenge proved that synthetic data is no longer a "second-class" substitute; it is now a primary driver for establishing strong baselines in domains where real-world data is scarce or sensitive. However, several challenges remain. The "Sim-to-Real" gap still accounts for a 5-10% accuracy drop, and many top-performing algorithms require manual camera calibration. Future research will focus on automating these processes and ensuring that these complex models can run in real-time on edge devices for practical urban deployment.

Summary updated

Abstract

The AI City Challenge's seventh edition emphasizes two domains at the intersection of computer vision and artificial intelligence - retail business and Intelligent Traffic Systems (ITS) - that have considerable untapped potential. The 2023 challenge had five tracks, which drew a record-breaking number of participation requests from 508 teams across 46 countries. Track 1 was a brand new track that focused on multi-target multi-camera (MTMC) people tracking, where teams trained and evaluated using both real and highly realistic synthetic data. Track 2 centered around natural-language-based vehicle track retrieval. Track 3 required teams to classify driver actions in naturalistic driving analysis. Track 4 aimed to develop an automated checkout system for retail stores using a single view camera. Track 5, another new addition, tasked teams with detecting violations of the helmet rule for motorcyclists. Two leader boards were released for submissions based on different methods: a public leader board for the contest where external private data wasn't allowed and a general leader board for all results submitted. The participating teams' top performances established strong baselines and even outperformed the state-of-the-art in the proposed challenge tracks.
BibTeX
@inproceedings{naphade2023th,
  author = {Milind Naphade and Shuo Wang and David C. Anastasiu and Zheng Tang and Ming-Ching Chang and Yue Yao and Liang Zheng and Mohammed Shaiqur Rahman and Meenakshi S. Arya and Anuj Sharma and Qi Feng and Vitaly Ablavsky and Stan Sclaroff and Pranamesh Chakraborty and Sanjita Prajapati and Alice Li and Shangru Li and Krishna Kunadharaju and Shenxin Jiang and Rama Chellappa},
  title = {{The 7th AI City Challenge}},
  booktitle = {Workshops @ CVPR},
  year = {2023},
  url = {https://openreview.net/forum?id=rsLI2GurkMZ},
}
Figure