Learning Good Representation via Continuous Attention

Liang Zhao, Wei Xu

Abstract

In this paper we present our scientific discovery that good representation can be learned via continuous attention during the interaction between Unsupervised Learning (UL) and Reinforcement Learning (RL) modules driven by intrinsic motivation. Specifically, we designed intrinsic rewards generated from UL modules for driving the RL agent to focus on objects for a period of time and to learn good representations of objects for later object recognition tasks. We evaluate our proposed algorithm in both with and without extrinsic reward settings. Experiments with end-to-end training in simulated environments with applications to few-shot object recognition demonstrated the effectiveness of the proposed algorithm.

Introduction: The Challenge of Passive Learning

In traditional machine learning, Unsupervised Learning (UL) is often a passive process. Algorithms are typically fed static datasets and tasked with finding patterns without active exploration. This contrasts sharply with biological learning; for instance, infants actively interact with their environment, manipulating objects and observing them from multiple angles to build robust mental models.

In Artificial Intelligence, bridging this gap is difficult. Standard Reinforcement Learning (RL) agents often exhibit random exploration patterns. If an agent only glimpses an object briefly before moving away, it fails to gather the diverse visual data necessary to learn a robust representation. This paper addresses this "passive learning" gap by introducing a framework for "continuous attention," where an agent actively chooses to linger on objects to teach itself about the world.

Methodology: The RL-UL Feedback Loop

The proposed framework utilizes a dual-module system where UL and RL modules operate in a continuous feedback loop.

The interaction between RL and UL modules driven by intrinsic rewards.
  1. The RL Module (The Explorer): Based on the A3C (Asynchronous Advantage Actor-Critic) architecture, this module governs the agent's movement and gaze within the environment.
  2. The UL Module (The Teacher): This module processes the agent's visual input. Two variants were tested: an Autoencoder (AE), which reconstructs current images, and a Prediction Module, which predicts future states based on current actions.

The core innovation is the use of intrinsic rewards. When the UL module encounters a high error (indicating it has not yet mastered the current visual input), it generates a reward signal for the RL agent. This "curiosity" drives the RL agent to maintain focus on the object. As the UL module learns and its error decreases, the reward diminishes, prompting the agent to explore new areas.

Experimental Setup and Datasets

The framework was evaluated in two simulated environments:

  • XWorld2D: A grid-based navigation task used to test if intrinsic rewards help agents find goals when external rewards are sparse.
  • XWorld3D: A complex 3D environment providing first-person perspectives. This environment was critical for evaluating how well the agent could learn to recognize 3D objects from limited viewpoints.
A sequence of frames from XWorld3D showing an agent maintaining continuous attention on a target object.

Key Results and Insights

The researchers found that the UL+RL approach significantly outperformed random exploration across several metrics:

  • Enhanced Focus: In XWorld3D, the Prediction-driven agent maintained focus on objects for an average of 12.1 consecutive frames, nearly double the 7.1 frames achieved by a random agent.
  • Superior Few-Shot Recognition: The Autoencoder-driven agent achieved a 7.0% error rate in "5-way n-shot" recognition tasks (identifying new categories from few examples), outperforming the Prediction module's 14.0% error rate. This suggests that static feature reconstruction (AE) may be more effective for object recognition than dynamic state prediction.
  • Overcoming Sparse Rewards: In navigation tasks, agents utilizing intrinsic UL rewards reached higher average returns significantly faster than those relying solely on standard extrinsic RL rewards.

Conclusion

By incentivizing "surprise" through intrinsic rewards, the proposed framework allows agents to naturally develop a policy of continuous attention. This active exploration mimics human-like learning and provides the diverse data necessary for robust 3D vision and few-shot recognition. Future work will focus on addressing the "noisy TV" problem—where irrelevant background noise triggers false curiosity—and transitioning these models from simulated environments to real-world robotics.

Summary updated

Abstract

In this paper we present our scientific discovery that good representation can be learned via continuous attention during the interaction between Unsupervised Learning(UL) and Reinforcement Learning(RL) modules driven by intrinsic motivation. Specifically, we designed intrinsic rewards generated from UL modules for driving the RL agent to focus on objects for a period of time and to learn good representations of objects for later object recognition task. We evaluate our proposed algorithm in both with and without extrinsic reward settings. Experiments with end-to-end training in simulated environments with applications to few-shot object recognition demonstrated the effectiveness of the proposed algorithm.
BibTeX
@misc{zhao2019learning,
  author = {Liang Zhao and Wei Xu},
  title = {{Learning Good Representation via Continuous Attention}},
  year = {2019},
  eprint = {1903.12344},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1903.12344},
}
Figure