Learning to View: Decision Transformers for Active Object Detection

Wenhao Ding, Nathalie Majcherczyk, Mohit Deshpande, Xuewei Qi, Ding Zhao, Rajasimman Madhivanan, Arnie Sen

Introduction

In traditional computer vision, systems are typically passive observers, processing static images or video streams provided to them. However, in the context of robotics, an agent possesses the agency to alter its perspective. If an object is occluded, poorly lit, or viewed from an ambiguous angle, a robot should ideally navigate to a superior vantage point to improve its perception. This task, known as Active Object Detection (AOD), is critical for autonomous agents operating in complex, real-world environments.

Despite its potential, AOD faces significant hurdles. Traditional Reinforcement Learning (RL) approaches often struggle with the "Markov limitation," where the agent's decisions are based solely on the current state, leading to repetitive loops or a failure to remember previously explored areas. Furthermore, the joint training of detection and navigation modules is notoriously unstable, and the data required to learn complex navigation from scratch is often prohibitively expensive to collect.

Methodology

This research introduces a novel framework that reframes active object detection as a sequence modeling problem, utilizing the Decision Transformer (DT) architecture. By treating navigation as a sequence of states, actions, and rewards, the model can leverage the long-range dependency capabilities of Transformers to overcome the limitations of standard RL.

Sequence Modeling with Decision Transformers

Unlike traditional RL agents that predict the next action based on the current state, the DT considers the entire history of the trajectory. It processes a sequence of past states, actions, and "Rewards-to-Go" (the cumulative reward the agent aims to achieve). This allows the agent to maintain a functional memory of its path, recognizing when it is stuck or when a previous viewpoint was more effective.

Two-Stage Training Pipeline

The authors employ a hybrid training strategy to ensure both stability and high performance:

  1. Stage 1 (Offline Pre-training): The model is first trained on a dataset of "expert" trajectories. These trajectories are generated using a human-like heuristic (e.g., rotating until an object is centered and then moving forward).
  2. Stage 2 (Online Fine-tuning): To move beyond the limitations of the heuristic, the model is fine-tuned through direct interaction with the environment. This stage allows the agent to discover novel strategies that the initial expert data might not have covered.

Enhancing Exploration

To prevent the model from simply mimicking sub-optimal expert behavior, the researchers introduced reward-variance sampling. This method prioritizes training on trajectories where rewards fluctuate significantly, forcing the model to learn from diverse and informative experiences. Additionally, an entropy term is added to the loss function to encourage action diversity during the online phase.

Experimental Setup

The framework was evaluated using a custom interactive dataset built within the AI2THOR simulator. The dataset encompasses 120 floor plans across four distinct indoor categories: Bathrooms, Kitchens, Living Rooms, and Bedrooms. The environment was discretized into 30cm cells and 30-degree orientation increments to facilitate efficient training.

For the perception component, the agent receives both RGB and depth images. The reward signal is derived from the confidence score of a pre-trained DETR (Detection Transformer) model. A high confidence score—indicating a clear, unobstructed view of the target object—results in a high reward, while occlusions or distant views result in lower scores.

Key Insights and Results

The proposed method, DT-Online-Expert, demonstrated substantial improvements over traditional baselines such as DQN, SAC, and purely offline DT models.

  • Surpassing the Expert: While the initial heuristic expert had a failure rate of approximately 25%, the DT-based agent reduced this to just 5%. The model learned to "sidestep" obstacles and furniture to obtain clear views, a behavior the original heuristic was unable to perform.
  • The Importance of Depth: Ablation studies revealed that depth information is more critical than color for AOD. Removing depth images significantly degraded performance, as the agent lost the spatial context necessary to navigate around physical barriers.
  • Superior Detection Metrics: In bathroom scenarios, the model achieved a Recall of 0.734 and a Precision of 0.901, markedly higher than the heuristic's Recall of 0.554 and Precision of 0.828.
  • Robust Generalization: The DT architecture showed a superior ability to handle "out-of-distribution" states compared to standard offline RL methods like CQL, suggesting better adaptability to unfamiliar environments.

Conclusion and Future Work

By treating active object detection as a sequence modeling task, this research demonstrates that Decision Transformers can effectively bridge the gap between perception and action. The combination of offline pre-training and online fine-tuning allows robots to learn sophisticated navigation strategies that exceed simple heuristics.

Future research will focus on addressing the reliability of surrogate rewards, as detection models can sometimes "hallucinate" high confidence on noisy data. Additionally, extending the framework to dynamic environments with moving obstacles and exploring end-to-end training—where the detector and navigator are optimized simultaneously—remain promising avenues for further development.

Summary updated

BibTeX
@inproceedings{ding2023learning,
  author = {Wenhao Ding and Nathalie Majcherczyk and Mohit Deshpande and Xuewei Qi and Ding Zhao and Rajasimman Madhivanan and Arnie Sen},
  title = {{Learning to View: Decision Transformers for Active Object Detection}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  doi = {10.1109/icra48891.2023.10160946},
}
Figure