VISTA: A Framework for Vision-Informed Semantic Traffic Awareness

Murat Arda Onsu, Poonam Lohan, Burak Kantarci, Aisha Syed, Matthew Andrews, Sean KennedyarXiv 2026 · arXiv

TL;DR

VISTA is a novel edge-intelligence framework that replaces bandwidth-heavy raw video streams with compact "future-scene" embeddings for Intelligent Transportation Systems (ITS). By combining the predictive power of V-JEPA with the reasoning capabilities of a Small Language Model (TinyLlama), VISTA enables vehicles to predict collisions and answer natural language traffic queries with 97.77% accuracy while reducing communication payloads by several orders of magnitude.

What Problem Does This Paper Address?

Modern Intelligent Transportation Systems (ITS) rely on a web of roadside cameras and sensors to keep cities moving safely. However, two major bottlenecks prevent these systems from reaching their full potential:

  1. The Bandwidth Crunch: Transmitting high-definition, raw RGB video from roadside units (RSUs) to vehicles over Vehicle-to-Everything (V2X) links is incredibly resource-intensive. In a dense urban environment with hundreds of connected cars, the network would quickly become congested.
  2. The Compute Gap: While Large Multimodal Models (LMMs) are great at "understanding" scenes, they are too massive to run on the resource-constrained hardware found inside vehicles or roadside boxes.

Current systems often rely on simple object detection (drawing boxes around cars), which lacks the "semantic" understanding needed to answer complex questions or proactively predict accidents before they happen.

Methods

The authors propose VISTA (Vision-Informed Semantic Traffic Awareness), which shifts the paradigm from sending pixels to sending meaning. The framework is split into two main stages:

1. RSU-Level Encoding (The "Eyes")

Instead of a standard video encoder, VISTA uses V-JEPA (Video Joint Embedding Predictive Architecture). V-JEPA is unique because it is trained to predict masked or future parts of a video in a latent (mathematical) space.

  • Attentive Pooler: To make the data even smaller, an attentive-pooler module compresses the patch-level outputs of V-JEPA into a single 1024-dimensional "future-scene" embedding. This vector represents what is likely to happen in the next second of traffic.

2. Vehicle-Level Reasoning (The "Brain")

The vehicle receives this tiny 4 KB embedding and processes it using TinyLlama, a Small Language Model (SLM) with 1.1 billion parameters.

  • Visual Adapter: Since TinyLlama is a text-only model, the authors developed a custom visual adapter (using Transformer layers and an MLP) to "translate" the V-JEPA embeddings into tokens that the language model can understand.
  • Natural Language Interaction: Users (or autonomous systems) can query the model in plain English, such as "Is there a collision risk in this scene?" or "How many vehicles are on the road?"

Datasets

The researchers evaluated VISTA using a high-fidelity digital twin environment:

  • Quanser Interactive Lab (Qlab): A configurable simulator that mimics urban traffic, including pedestrians, signals, and various weather conditions.
  • Urban Mobility Research Dataset: This contains 1,975 urban traffic video records (4–8 seconds long). The data was sampled at 4 fps to balance temporal detail with computational efficiency.
  • Experimental Split: 1,571 clips were used for training and 404 for testing.

Key Insights and Ideas

  • Predictive vs. Reactive: Most ITS systems look at the current frame. VISTA’s use of V-JEPA allows it to generate embeddings that are inherently future-oriented, giving the vehicle a critical head start in safety-critical scenarios.
  • Semantic Communication: This paper is a strong argument for "goal-oriented" communication. By sending a 1024-dimensional vector instead of 3.15 MB of video, the system achieves a 94.9% reduction in transmission time without losing the ability to reason about the scene.
  • The Power of Attentive Pooling: The authors found that using learnable query tokens (attentive pooling) to aggregate visual features improved vehicle-counting accuracy by over 11% compared to simple mean or max pooling.
  • Linguistic Robustness: The model doesn't just memorize templates; it showed strong generalization when tested with paraphrased queries, proving that SLMs can handle the messy reality of human language in a traffic context.

Key Results

  • Collision Prediction: Achieved 97.77% accuracy and an 89% F1 score on the test set.
  • Efficiency: The communication payload was reduced from ~3.15 MB (raw video) to just 4 KB (float32 embedding) or 2 KB (float16).
  • Latency: The total end-to-end latency (from camera capture to language model response) was clocked at 204.86 ms. This is fast enough for near-real-time safety advisories.
  • Vehicle Counting: Achieved 91% accuracy, with most errors differing by only a single vehicle, usually due to heavy occlusion.

Limitations and Future Work

  • Sim-to-Real Gap: While the digital twin is sophisticated, real-world variables like sensor noise, extreme weather, and hardware degradation remain to be tested.
  • Hallucination Risks: As with all language models, there is a risk of "hallucination"—the model providing a confident but incorrect safety assessment. The authors suggest future work should include uncertainty estimation and rule-based safety constraints.
  • Privacy: Although embeddings are more private than raw video (you can't see faces or license plates easily), they still represent sensitive data. Future iterations may explore federated learning or encrypted inference.
  • Scaling: The authors plan to validate VISTA under larger-scale and more diverse real-world traffic conditions.

Summary updated

Abstract

The Internet of Things (IoT) is transforming intelligent transportation systems by connecting roadside cameras, edge computing units, vehicles, and traffic infrastructure into a shared sensing and decision-making ecosystem. However, transmitting raw traffic video over vehicle-to-everything (V2X) links is bandwidth-intensive, while deploying large multimodal models directly on resource-constrained vehicles is often impractical. To address these limitations, this article proposes VISTA (Vision-Informed Semantic Traffic Awareness), a multimodal vision-language edge-intelligence framework that combines predictive video representation learning with lightweight language-based reasoning to provide communication-efficient traffic situational awareness. In the proposed architecture, roadside surveillance units employ the Video Joint Embedding Predictive Architecture (V-JEPA) and an attentive-pooler module to convert traffic video clips into compact future-scene embeddings. These embeddings are transmitted through V2X links to vehicle-mounted systems, where a TinyLlama small language model, enhanced with a custom visual adapter, interprets the embeddings alongside natural-language (NL) traffic queries. VISTA enables vehicles to obtain interpretable traffic information, reason about upcoming scene dynamics, and identify potential collision risks without receiving raw video streams. Experimental results in a digital-twin urban traffic environment show that VISTA achieves 97.77% accuracy and an 89% F1 score for future collision prediction while reducing the communication payload by several orders of magnitude compared with raw RGB video transmission. These results demonstrate the feasibility of embedding-based multimodal intelligence for low-latency, communication-efficient, and safety-oriented IoT-enabled transportation systems.
BibTeX
@misc{onsu2026vista,
  author = {Murat Arda Onsu and Poonam Lohan and Burak Kantarci and Aisha Syed and Matthew Andrews and Sean Kennedy},
  title = {{VISTA: A Framework for Vision-Informed Semantic Traffic Awareness}},
  year = {2026},
  eprint = {2611.20458},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20458},
}
Figure