Plan-Recognition-Driven Attention Modeling for Visual Recognition

Yantian Zha, Yikang Li, Tianshu Yu, Subbarao Kambhampati, Baoxin LiarXiv 2018 · arXiv

Abstract

In complex visual environments, human perception is often guided by top-down attention—a cognitive process that prioritizes regions of interest based on goals and anticipated actions. This paper presents a novel framework for Plan-Recognition-Driven Attention (PRDA) that bridges high-level plan recognition with low-level pixel dynamics. The core of the approach is the Pixel Dynamics Network (PDN), a convolutional architecture that predicts future pixel states conditioned on recognized agent plans. Evaluated on the VIRAT surveillance dataset, the proposed Event Recognition with Pixel Dynamics Network (ER-PDN) demonstrates the ability to anticipate motion in cluttered scenes, outperforming traditional attention models by focusing on the "intent" of the scene rather than just physical saliency.

Introduction

Human vision is not merely a passive recording of light; it is an active, goal-oriented process. When observing an agent, humans use "top-down attention" (TDA) to focus on regions relevant to the agent's likely future actions. For instance, seeing someone approach a vehicle with keys in hand naturally shifts a viewer's attention toward the car door. In contrast, most computer vision attention models are "bottom-up," driven by low-level features like color contrast or motion, or use simplistic top-down signals like class labels. These models often struggle in cluttered environments where objects are small, partially occluded, or moving erratically.

This research addresses the challenge of integrating long-term plan recognition—the sequence of actions an agent is expected to take—into visual attention mechanisms. By predicting where an agent will be and what they will do, a vision system can proactively allocate computational resources to the most informative regions of a scene.

Methods

The authors propose the Event Recognition with Pixel Dynamics Network (ER-PDN) framework. This system operates through several integrated stages:

  1. Augmented Motion Primitives (AMPs): The system first extracts motion features, including Histogram of Optical Flow (HOF), Histogram of Oriented Gradients (HOG), and Histogram of Frame Difference (HOD). These are clustered into "meta-actions" or AMPs, which form the basic vocabulary for describing agent behavior.
  2. Plan Recognition: Utilizing the Distr2Vec model, the system observes initial actions and predicts a sequence of future AMPs, effectively identifying the agent's "plan."
  3. Pixel Dynamics Network (PDN): The PDN is the central technical innovation. It is a Convolutional Neural Network (ConvNet) designed as an internal dynamics model. It takes an image patch and a predicted action (AMP) as input to predict the "offset"—the likely shift of pixels in the next state.
  4. Action Conditional Filters (ACFs): To handle the variability of different actions, the PDN generates specific filters on-the-fly based on the recognized plan. These filters convolve with the image to produce State Prediction Maps (SPMs).
  5. Translation Pooling: This specialized layer converts predicted offsets into a final PRDA map, highlighting where "object particles" are likely to move.
The ER-PDN framework integrates plan recognition with a pixel dynamics network to generate predictive attention maps.

A significant theoretical contribution of this work is the mathematical proof that the PDN architecture is equivalent to a belief update in a Partially Observable Markov Decision Process (POMDP). This provides a rigorous probabilistic foundation for treating pixel-level prediction as a form of reasoning under uncertainty.

Experimental Evaluation

The model was evaluated using the VIRAT 1.0 Ground Dataset, a benchmark for naturalistic video surveillance. The dataset is notoriously difficult due to its wide-angle views, small object sizes, and complex multi-agent interactions. The researchers used 70% of the data for training and 30% for testing. Notably, while ground-truth bounding boxes were used to learn motion primitives, they were not utilized during the event recognition task, forcing the model to rely entirely on its learned attention and recognition capabilities.

The ER-PDN was compared against a state-of-the-art baseline (Girdhar & Ramanan, 2017) that combines bottom-up and top-down attention. Quantitatively, the ER-PDN achieved a Mean Average Precision (mAP) of 29.15%, a modest but meaningful improvement over the baseline's 29.06%, especially given the lack of object tracking.

Qualitatively, the PRDA maps reveal the model's unique strengths. As shown in the visualizations, the model successfully anticipates the "intent" of a scene. For example, when individuals walk toward a vehicle, the PRDA map highlights the empty space between the people and the car—the area most relevant to the completion of their plan. Standard models, by contrast, tend to focus only on the current location of the moving agents.

Visualization of PRDA maps showing how the model anticipates future regions of interest based on agent plans.

Discussion and Conclusion

The PRDA model demonstrates that high-level cognitive constructs like plans can be effectively decomposed into low-level pixel dynamics. This allows for a more proactive form of visual attention that can "look ahead" in both space and time. By bridging the gap between classical AI planning (POMDPs) and modern deep learning (CNNs), the authors provide a scalable path for incorporating reasoning into perception.

However, several limitations remain. The model's performance is currently bottlenecked by the accuracy of the initial plan recognition; incorrect predictions lead to misplaced attention. Additionally, the dynamic generation of filters is computationally intensive. Future work may explore more efficient filter generation and the integration of an agent’s own internal goals to drive attention, moving toward more autonomous and intelligent visual systems.

Summary updated

Abstract

Human visual recognition of activities or external agents involves an interplay between high-level plan recognition and low-level perception. Given that, a natural question to ask is: can low-level perception be improved by high-level plan recognition? We formulate the problem of leveraging recognized plans to generate better top-down attention maps \cite{gazzaniga2009,baluch2011} to improve the perception performance. We call these top-down attention maps specifically as plan-recognition-driven attention maps. To address this problem, we introduce the Pixel Dynamics Network. Pixel Dynamics Network serves as an observation model, which predicts next states of object points at each pixel location given observation of pixels and pixel-level action feature. This is like internally learning a pixel-level dynamics model. Pixel Dynamics Network is a kind of Convolutional Neural Network (ConvNet), with specially-designed architecture. Therefore, Pixel Dynamics Network could take the advantage of parallel computation of ConvNets, while learning the pixel-level dynamics model. We further prove the equivalence between Pixel Dynamics Network as an observation model, and the belief update in partially observable Markov decision process (POMDP) framework. We evaluate our Pixel Dynamics Network in event recognition tasks. We build an event recognition system, ER-PRN, which takes Pixel Dynamics Network as a subroutine, to recognize events based on observations augmented by plan-recognition-driven attention.
BibTeX
@misc{zha2018plan,
  author = {Yantian Zha and Yikang Li and Tianshu Yu and Subbarao Kambhampati and Baoxin Li},
  title = {{Plan-Recognition-Driven Attention Modeling for Visual Recognition}},
  year = {2018},
  eprint = {1812.00301},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1812.00301},
}
Figure