Abstract
Traditional object detection systems are passive, relying on static images that may suffer from poor lighting or occlusions. This paper introduces the Student-Teacher Framework (STF), which transforms detection into an active, embodied task. STF enables an agent to dynamically adjust camera parameters—such as zoom, angle, and exposure—to optimize detection performance. By modeling camera adjustment as a sequence prediction task using a GPT-like architecture and employing a self-tuning teacher network for reward generation, STF significantly outperforms existing passive and active detectors in challenging environments.
Introduction
Standard object detectors are "passive," trained on fixed datasets without the ability to interact with their environment. In real-world scenarios, environmental factors like harsh lighting or physical obstructions often lead to detection failure. While "active vision" aims to solve this by allowing agents to interact with the scene, it typically requires expensive, manually annotated interactive data and often suffers from the instability of traditional Reinforcement Learning (RL) approaches.
The authors address these challenges by asking: Can we create a system that "learns how to see" by dynamically adjusting its viewpoint, and can it do so without needing a human to label every single camera movement?
Methods
The core of the proposed approach is the Student-Teacher Framework (STF), which treats embodied detection as a collaborative learning process involving two primary components:
- The Student Network (The Actor): Utilizing a GPT-based structure with causal self-attention, the student treats camera adjustment as a sequence modeling task. It predicts the next optimal action (e.g., zoom, move, or adjust brightness) based on the history of states, actions, and rewards. This approach captures long-term temporal context better than traditional RL, preventing "jittery" camera movements.
- The Teacher Network (The Evaluator): To facilitate learning on unlabeled data, a Siamese architecture-based teacher network compares consecutive frames. It generates a "reward" signal based on the improvement in detection confidence (the "Benefit" metric), allowing for self-supervised learning.
The framework employs a replay buffer to store trajectories, enabling the student to "self-tune" and adapt to new, unlabeled scenarios during inference without human intervention.
Datasets and Benchmarks
The STF was evaluated on two primary benchmarks:
- Small Airport (SA): 12 categories of aircraft in outdoor environments.
- Virtual Park (VP): 5 types of vehicles in various simulated conditions.
- Embodied Object Detection Replay Buffer: A new contribution containing 220,000 trajectory transformations across 76 diverse scenarios.
Key Results
The STF framework demonstrates a clear performance leap over both passive and active baselines:
- Superior Accuracy: STF outperformed the FCOS detector by 15.8% (SA) and 24.6% (VP) in mAP (mean Average Precision). It also surpassed the heavy-hitting DINO transformer detector by over 7%.
- Efficiency: STF reached higher accuracy in just 3 steps than most RL methods reached in 8 steps, proving it learns more efficient search patterns.
- Downstream Impact: In car-counting experiments, active adjustment significantly reduced "missed" and "over-counted" errors compared to standard passive detectors.
Conclusion and Future Work
By framing decision-making as sequence modeling, the STF achieves greater stability and captures long-term temporal context. The use of self-supervised rewards via the teacher network allows the system to improve continuously on unlabeled video streams. While the current action space is discrete, future work will explore continuous action spaces for smoother camera control and methods to reduce the computational overhead of the self-tuning process.
Summary updated