Task-Relevant Object Discovery and Categorization for Playing First-person Shooter Games

Junchi Liang, Abdeslam BoulariasarXiv 2018 · arXiv

Introduction

Deep Reinforcement Learning (RL) has demonstrated remarkable success in complex environments, yet standard "model-free" agents remain significantly more data-hungry than human learners. While a human player quickly identifies task-relevant entities—such as enemies or health packs—standard Deep Q-Networks (DQNs) must process millions of raw pixels to implicitly learn these associations. This high dimensionality often leads to training inefficiencies as agents struggle to distinguish meaningful objects from environmental noise. This paper proposes a method to bridge the "pixels-to-actions" gap by enabling agents to autonomously discover and categorize task-relevant objects.

Methodology

The proposed architecture extracts a symbolic understanding of the game environment without manual labeling. The pipeline consists of several primary stages designed to mimic biological visual saliency and cognitive filtering:

  1. Optical Flow Computation: Utilizing FlowNet 2.0, the system calculates pixel-wise motion between consecutive frames. This allows the agent to isolate moving entities from the static background.
  2. Segmentation and Feature Extraction: Motion data is processed via a watershed technique to group pixels into distinct segments. For each segment, the agent extracts Histogram of Oriented Gradients (HoG) features to capture the object's geometric structure.
  3. Unsupervised Clustering: These features are processed using spectral clustering, allowing the agent to categorize objects (e.g., grouping all instances of a specific enemy type) without predefined labels.
  4. Task-Relevance Filtering: To filter out irrelevant motion (e.g., environmental debris), the system performs a Pearson Correlation test between the presence of specific object clusters and changes in the agent's value function. Only clusters that consistently correlate with performance changes are retained.
  5. Semantic Integration: The identified relevant categories are converted into binary masks (semantic channels) and fed into a Deep Recurrent Q-Network (DRQN) alongside the original raw image.
The proposed pipeline for autonomous object discovery, categorization, and task-relevance filtering.

Experimental Setup

The approach was evaluated using the ViZDoom platform across four distinct scenarios designed to test different aspects of FPS gameplay: Predict Position (aiming at moving targets), Defend The Line (fixed-position defense), Center Defense (360-degree engagement), and Deadly Corridor (navigation and hazard avoidance).

Results and Analysis

The proposed method was benchmarked against a standard DRQN baseline and an "Oracle" agent, which received perfect object data directly from the game engine.

Experimental tasks and performance comparison across four ViZDoom scenarios.

The results demonstrate superior sample efficiency in "Defend The Line" and "Center Defense," where the proposed method learned significantly faster and achieved higher peak scores than the standard DRQN. Notably, the agent's performance closely tracked that of the Oracle, suggesting that the unsupervised object discovery mechanism is highly effective. However, in tasks with significant temporal delays between actions and rewards, such as "Predict Position," the performance gains were less pronounced, indicating that while object-centric views improve perception, they do not fully resolve the fundamental RL challenge of long-term credit assignment.

Discussion and Limitations

While effective, the current system relies heavily on optical flow, creating a "stationary object" problem: if both the agent and the target are immobile, the object disappears from the semantic map. Additionally, rapid camera movements can cause motion blur, leading to temporary segmentation failures. Future research may explore incorporating causal models to better understand long-term consequences and extending this object-centric approach to real-world robotics, where identifying relevant entities in unstructured environments is a critical requirement.

Summary updated

BibTeX
@misc{liang2018task,
  author = {Junchi Liang and Abdeslam Boularias},
  title = {{Task-Relevant Object Discovery and Categorization for Playing First-person Shooter Games}},
  year = {2018},
  eprint = {1806.06392},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1806.06392},
}
Figure