Your Vision-Language-Action Model Already Has Attention Heads For Path Deviation Detection

Jaehwan Jeong, Evelyn Zhu, Jinying Lin, Emmanuel Jaimes, Tuan-Anh Vu, Jungseock Joo, Sangpil Kim, M. Khalid JawedarXiv 2026 · arXiv

TL;DR

Vision-Language-Action (VLA) models for robot navigation frequently suffer from visual hallucinations that cause them to drift off course. This paper reveals that the signal to detect these deviations already exists inside the frozen model: monitoring the attention entropy of just three internal "Navigation Heads" allows for real-time anomaly detection with zero extra training and near-zero computational overhead. When a deviation is detected, the system safely hands control over to a lightweight Reinforcement Learning policy to execute a shortest-path rollback recovery.

What Problem Does This Paper Address?

Vision-Language-Action (VLA) models represent a major leap forward in robotics, allowing embodied agents to map natural language instructions and visual streams directly into continuous navigation actions. However, these models inherit a chronic flaw from their Large Language Model backbones: hallucinations. In a navigation context, a visual reasoning failure or a language grounding error leads to trajectory drift, incorrect sub-goal selection, and catastrophic path deviations.

Traditionally, detecting when a robot has lost its way requires either training expensive external critic modules, relying on heavy external LLM APIs that introduce severe latency, or waiting for post-facto physical cues (like colliding with a wall or getting stuck). This paper addresses the need for an efficient, high-frequency, and training-free method to detect navigation anomalies in real time before a collision occurs.

Methods

The authors introduce a hierarchical detection-to-recovery framework that repurposes the internal mechanics of a frozen VLA model (specifically, NaVILA, built on the VILA-8B architecture).

Fig. 1: Conceptual teaser of our approach to mitigating hallucinations in VLA-based navigation. 1. The Problem: VLA models are highly vulnerable to vision-language reasoning hallucinations, which cause severe deviations from planned trajectories. 2. The Discovery: We reveal that a specific subset of internal attention heads, termed Navigation Heads (Hnav), is highly sensitive to the robot’s navigation states. This allows them to serve as a built-in anomaly detector with near-zero computational overhead. 3. Our Solution: Building on this insight, we enable real-time, training-free path deviation detection. Upon detecting a failure, the system immediately bypasses the computationally heavy VLA model and triggers a low-level Reinforcement Learning (RL) policy to execute a safe, collision-free rollback.

The pipeline consists of four main stages:

  1. Trajectory Phase Categorization: The authors design a state machine that tracks the robot's progress relative to a reference path, automatically labeling steps as Normal ($\mathcal{N}$) or Anomaly ($\mathcal{A}$).
  2. Navigation Heads ($H_{\text{nav}}$) Selection: Through offline statistical analysis, the authors isolate a sparse subset of attention heads that inherently capture the spatiotemporal causality between historical visual sequences and linguistic instructions. They score these heads based on cross-modal grounding metrics (frame energy uniformity and focus sharpness) and tracking metrics (diagonal alignment and smooth shifting). Heads are then ranked by their sensitivity to failure modes using Cohen's $d$ effect size.
  3. Attention-Based Anomaly Detection: During online deployment, the system monitors the relative Shannon entropy ($\mathcal{R}_t$) of the top-3 selected Navigation Heads. A sudden dispersion of attention indicates that the model is confused or hallucinating. If this entropy score exceeds a threshold $\tau$ for a set number of consecutive steps, an anomaly is triggered.
  4. Collision-Free Low-Level Action Policy: Upon detecting an anomaly, the system bypasses the heavy VLA model entirely. It triggers a lightweight Reinforcement Learning (RL) policy trained via Proximal Policy Optimization (PPO) in IsaacLab. Operating at 10 Hz, this policy utilizes local 2D occupancy grid costmaps to safely execute a shortest-path rollback to the last verified safe checkpoint ($\mathcal{C}_{\text{safe}}$).

Datasets

The framework was trained and evaluated using the following resources:

  • VLN-CE R2R (Room-to-Room): Used to validate ground-truth behavioral labels, perform hyperparameter grid sweeps, and select the optimal Navigation Heads.
  • RxR (Room-Across-Room): Used to test the generalizability of the attention-based detection mechanism in environments with higher linguistic and spatial complexity.
  • IsaacLab Procedural Simulation Environment: A custom $60\text{m} \times 60\text{m}$ subterranean testbed populated with procedurally generated cylindrical obstacles and wall-shaped boxes, used to train and benchmark the low-level RL policy.

Key Insights and Ideas

  • The Model Already "Knows": During successful navigation, Navigation Heads exhibit a highly structured attention progression that tracks the agent's movement through the instruction text over time. When the agent deviates, this structured alignment predictably degrades, causing a measurable spike in attention entropy.
  • Extreme Sparsity: Out of more than a thousand attention heads within the 8-billion-parameter VLA model, an ensemble of just three heads is sufficient to accurately detect path deviations.
  • Zero-Cost Monitoring: Because attention weights are already computed during the standard forward pass of the VLA model, extracting this anomaly signal requires no additional neural network layers. It introduces a marginal latency increase of less than 20 ms per step and requires 0 MB of additional GPU memory.

Key Results

  • Robust Anomaly Detection: On the R2R Val-Unseen split, the 3-head ensemble achieved a 41.9% Episode Detection Rate (EDR) with a low 9.6% False Episode Rate (FER), significantly outperforming traditional kinematics-based baselines (such as monitoring position stagnation).
  • Superior Obstacle Avoidance: The low-level RL policy achieved an average 87.3% Success Rate (SR) in dense obstacle environments, outperforming classical local planners (like APF and TEB) by nearly 30 percentage points while maintaining low collision and timeout rates.
  • Real-World Viability: The entire pipeline was successfully deployed on a physical AgileX Scout 2.0 mobile robot equipped with an NVIDIA Jetson AGX Orin. By offloading the state estimation (Fast-LIVO2) to the CPU, the system preserved the GPU's bandwidth, allowing the VLA model to run alongside real-time trajectory recording and obstacle avoidance.

Limitations and Future Work

  • Context Desynchronization: A key limitation is that while the low-level RL policy physically rolls the robot back to a safe position, it does not reset or synchronize the VLA model's internal hidden states or visual history. Consequently, the VLA model may immediately repeat the same cognitive error upon resuming control. The authors suggest that integrating active replanning or re-prompting frameworks to realign these internal states is a crucial next step.
  • Model Pruning Opportunities: The discovery of highly specialized, navigation-essential attention heads provides a clear pathway for targeted model compression. Pruning the remaining redundant heads could significantly accelerate VLA inference speeds for edge-device deployments.

Summary updated

Abstract

Vision-Language-Action (VLA) models have demonstrated strong potential for predicting semantic actions in navigation tasks, demonstrating the ability to reason over complex linguistic instructions and visual contexts. However, they are fundamentally hindered by visual-reasoning hallucinations that lead to trajectory deviations. Addressing this issue has conventionally required training external critic modules or relying on complex uncertainty heuristics. In this work, we discover that monitoring a few attention heads within a frozen VLA model can accurately detect path deviations without incurring additional computational overhead. We refer to these heads, which inherently capture the spatiotemporal causality between historical visual sequences and linguistic instructions, as Navigation Heads. Using these heads, we propose an intuitive, training-free anomaly-detection framework that monitors their signals to detect hallucinations in real time. Surprisingly, among over a thousand attention heads, a combination of just three is sufficient to achieve a 44.6 % deviation detection rate with a low false-positive rate of 11.7 %. Furthermore, upon detecting a deviation, we bypass the heavy VLA model and trigger a lightweight Reinforcement Learning (RL) policy to safely execute a shortest-path rollback. By integrating this entire detection-to-recovery pipeline onto a physical robot, we demonstrate its practical robustness. All source code will be publicly available.
BibTeX
@misc{jeong2026your,
  author = {Jaehwan Jeong and Evelyn Zhu and Jinying Lin and Emmanuel Jaimes and Tuan-Anh Vu and Jungseock Joo and Sangpil Kim and M. Khalid Jawed},
  title = {{Your Vision-Language-Action Model Already Has Attention Heads For Path Deviation Detection}},
  year = {2026},
  eprint = {2603.13782},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.13782},
}
Figure