Active Object Localization with Deep Reinforcement Learning

Juan C. Caicedo, Svetlana Lazebnik

Abstract

We present an active detection model for localizing objects in scenes. The model is class-specific and allows an agent to focus attention on candidate regions for identifying the correct location of a target object. This agent learns to deform a bounding box using simple transformation actions, with the goal of determining the most specific location of target objects following top-down reasoning. The proposed localization agent is trained using deep reinforcement learning, and evaluated on the Pascal VOC 2007 dataset. We show that agents guided by the proposed model are able to localize a single instance of an object after analyzing only between 11 and 25 regions in an image, and obtain the best detection results among systems that do not use object proposals for object localization.

Introduction

Traditional object detection typically relies on exhaustive sliding window searches or bottom-up object proposals. Sliding windows are computationally expensive, while proposal-based methods often generate thousands of candidate boxes based on low-level visual cues like edges or color. Both approaches lack the "active attention" characteristic of human vision, where high-level context is used to narrow the search space.

This paper introduces a class-specific agent that reframes object localization as a sequential decision-making process. Starting from the entire image, the agent intelligently reasons about how to move and reshape its focus—the bounding box—to tightly encapsulate a target object. By utilizing Deep Reinforcement Learning (DRL), the model learns to "zoom in" on objects efficiently, outperforming other methods that do not rely on pre-computed proposals.

Methodology

The authors cast the localization task as a Markov Decision Process (MDP) and train the agent using a Deep Q-Network (DQN).

The Agent's Action Space

The agent navigates the image using a discrete set of 9 actions:

  • Translation: Four moves (left, right, up, down) to shift the box.
  • Scaling: Two actions to make the box bigger or smaller.
  • Aspect Ratio: Two actions to make the box fatter or taller.
  • Trigger: A terminal action to signal that the object has been successfully localized.
Figure 1: The agent transforms the bounding box through a sequence of discrete actions—such as shifting and zooming—to localize the target object over several time steps.

State Representation and Reward

The state at each step is a combination of visual features and action history. Visual features are 4,096-dimensional vectors extracted from the current bounding box using a pre-trained Zeiler-Fergus CNN architecture. A binary vector recording the last 10 actions is included to provide the agent with "memory," preventing it from getting stuck in repetitive cycles.

The reward function is tied to the Intersection-over-Union (IoU) between the current box and the ground truth. Actions that increase IoU yield a $+1$ reward, while those that decrease it result in a $-1$ penalty. The "Trigger" action provides a significant $+3$ reward if the IoU exceeds 0.6, incentivizing high-precision localization.

Training via Apprenticeship Learning

To accelerate convergence, the authors employ "apprenticeship learning" (guided exploration). Initially, the agent is nudged toward actions known to improve IoU, rather than relying solely on random exploration. This helps the agent find successful search trajectories more effectively in the high-dimensional state space.

Experimental Results

The model was evaluated on the Pascal VOC 2007 and 2012 datasets, which contain 20 object categories in diverse natural scenes.

  • Accuracy: The model achieved 46.1 mAP (Mean Average Precision) on Pascal VOC 2007. This significantly outperformed other "no-proposal" methods of the era, such as MultiBox (29.2 mAP).
  • Efficiency: The agent is remarkably sparse in its search. The median number of steps to localize an object was just 11, with an average of 25 steps for more complex cases.
  • Recall: The system reaches 50% recall with only 10 proposals, a milestone that traditional methods like Selective Search require much more computation to achieve.

Discussion and Insights

Top-Down Reasoning

Unlike bottom-up methods that look for edges, this agent utilizes top-down logic. If it perceives a portion of an object (e.g., "a bit of a cow"), it understands the spatial context required to move and scale the box to capture the whole animal. This leads to dynamic search paths unique to every image and context.

Limitations

Despite its strengths, the system faces challenges with small objects, as they provide a weak signal in the initial full-image view. Additionally, in cluttered scenes, the "Inhibition of Return" mechanism—which marks found objects with a black cross to prevent re-detection—can sometimes obscure adjacent objects of the same class.

Conclusion

This work demonstrates that object localization can be solved efficiently through active, sequential reasoning. By combining CNN-based visual features with DRL, the agent achieves high precision with minimal region analysis. Future work may involve end-to-end training of the CNN and the DQN policy, as well as scaling the approach to handle thousands of object categories through hierarchical models.

Summary updated

Abstract

We present an active detection model for localizing objects in scenes. The model is class-specific and allows an agent to focus attention on candidate regions for identifying the correct location of a target object. This agent learns to deform a bounding box using simple transformation actions, with the goal of determining the most specific location of target objects following top-down reasoning. The proposed localization agent is trained using deep reinforcement learning, and evaluated on the Pascal VOC 2007 dataset. We show that agents guided by the proposed model are able to localize a single instance of an object after analyzing only between 11 and 25 regions in an image, and obtain the best detection results among systems that do not use object proposals for object localization.
BibTeX
@inproceedings{caicedo2015active,
  author = {Juan C. Caicedo and Svetlana Lazebnik},
  title = {{Active Object Localization with Deep Reinforcement Learning}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2015},
  url = {https://openreview.net/forum?id=H1ZicZfdbr},
}
Figure