Visual Hide and Seek

Boyuan Chen, Shuran Song, Hod Lipson, Carl Vondrick

TL;DR

Researchers from Columbia University demonstrate that training embodied agents in a 3D "Visual Hide and Seek" game forces them to develop sophisticated internal representations of their environment. Crucially, they found that imposing "judicious weaknesses" on an agent—such as making it slower than its pursuer—actually encourages the emergence of richer features, like the ability to predict its own visibility from the predator's perspective.

What Problem Does This Paper Address?

In reinforcement learning (RL), agents often learn "reactive" policies—simple mappings from pixels to actions that solve a specific task without truly "understanding" the underlying scene dynamics. The authors wanted to see if a more complex, multi-agent task like Hide and Seek could trigger the emergence of higher-level concepts, such as object affordances and perspective-taking.

The challenge lies in the environment: unlike previous work that used top-down views or global game-engine states, this study uses a partially observable, 3D environment where agents only see through an egocentric (first-person) camera. This forces the agent to deal with occlusions and navigate based on limited visual information, mimicking the real-world constraints of biological predator-prey dynamics.

Methods

The team built a 3D simulation in the Unity engine. The game features two agents: a Hider (the learning agent) and a Seeker (a programmed opponent).

  • The Hider's Brain: A neural network trained using Proximal Policy Optimization (PPO). It takes raw pixel input from a 120-degree field of view and outputs discrete movement actions.
  • The Seeker's Logic: Usually follows a deterministic policy (move toward the hider if visible, otherwise move to the last known position or explore waypoints).
  • Systematic Interventions: To understand what drives representation learning, the authors perturbed the game by changing the hider's speed, adding auxiliary rewards for staying hidden ("visibility reward"), and introducing stochastic (randomized) maps and seeker behaviors.
  • Probing the Latent Space: To see what the agent "knows," the researchers froze the trained network and used linear regression to see if the internal features could predict ground-truth states, such as "Is the seeker visible to me?" or "Am I visible to the seeker?"

Datasets

The paper does not use a static dataset. Instead, it utilizes a custom-built 3D Unity simulation environment. The environment is a 14x14 square arena populated with various obstacles (cubes, cylinders, etc.) that create occlusions. While the environment is procedurally simple, it provides the complex visual data (pixels) necessary for training deep RL agents from scratch.

Key Insights and Ideas

  • The Disadvantage Paradox: One of the most striking findings is that making the agent "weaker" (e.g., the slowerhider variant) actually leads to better internal representations. If an agent is too fast, it can simply outrun the predator reactively. If it is slow, it must learn to leverage obstacles and predict visibility to survive.
  • Emergent Perspective-Taking: Without being explicitly told to do so, the agents learned to recognize the predator’s perspective. The models developed features that could accurately predict whether they were currently visible to the seeker, a key component of "Theory of Mind."
  • The "Sweet Spot" for Learning: The authors identified a parabolic relationship between survival time and representation quality (Figure 3). The best representations emerge when the task is difficult enough to require strategy but not so hard that the agent fails to learn entirely.
  • Visual State Transitions: By analyzing state frequencies, they found that successful hiders don't just run; they learn to "peek." For example, the visibilityreward+faster agent learned to monitor the seeker from behind or from a hidden position, transiting between "I can see him" and "He can't see me."

Key Results

  • Task Performance: All trained agents significantly outperformed random chance. The basic hider survived for an average of 500 steps (out of 1,000) with a 45% success rate of avoiding capture entirely.
  • Representation Accuracy: In downstream perception tasks, the stochasticseeker model achieved a 96.42% accuracy in recognizing the seeker and 94.58% accuracy in awareness of its own visibility based purely on its mid-level visual features.
  • Qualitative Strategy: Visualizations showed that "faster" hiders tended to be more reckless, while "slower" hiders or those with visibility rewards developed "cautious" behaviors, such as hugging walls and using obstacles as shields.

Limitations and Future Work

The primary limitation identified is a lack of memory. The current architecture uses a standard CNN without recurrent units (like LSTMs). This leads to "forgetful" failures, where an agent might get caught because it "forgot" an obstacle was behind it or where the seeker was last seen once it left the field of view.

The authors suggest that future work should integrate computational memory models to allow agents to build internal maps of the environment and track moving opponents over longer durations. They also propose that increasing the complexity of the disadvantages could lead to even richer, more human-like strategic behaviors.

Summary updated

Abstract

We train embodied agents to play Visual Hide and Seek where a prey must navigate in a simulated environment in order to avoid capture from a predator. We place a variety of obstacles in the environment for the prey to hide behind, and we only give the agents partial observations of their environment using an egocentric perspective. Although we train the model to play this game from scratch, experiments and visualizations suggest that the agent learns to predict its own visibility in the environment. Furthermore, we quantitatively analyze how agent weaknesses, such as slower speed, effect the learned policy. Our results suggest that, although agent weaknesses make the learning problem more challenging, they also cause more useful features to be learned. Our project website is available at: http://www.cs.columbia.edu/ ~bchen/visualhideseek/.
BibTeX
@inproceedings{chen2020visual,
  author = {Boyuan Chen and Shuran Song and Hod Lipson and Carl Vondrick},
  title = {{Visual Hide and Seek}},
  booktitle = {ALIFE},
  year = {2020},
  url = {https://openreview.net/forum?id=SeOh-kSPmUb},
}

Loading related content...

Figure