Abstract
We investigate reinforcement learning in the Game Of Hidden Rules (GOHR) environment, a complex puzzle in which an agent must infer and execute hidden rules to clear a 6x6 board by placing game pieces into buckets. We explore two state representation strategies, namely Feature-Centric (FC) and Object-Centric (OC), and employ a Transformer-based Advantage Actor-Critic (A2C) algorithm for training. The agent has access only to partial observations and must simultaneously infer the governing rule and learn the optimal policy through experience. We evaluate our models across multiple rule-based and trial-list-based experimental setups, analyzing transfer effects and the impact of representation on learning efficiency.
Introduction: The Quest for AI Metrology
Modern Artificial Intelligence, particularly Large Language Models (LLMs), often operates as a "black box." While their capabilities are evident, the field lacks a rigorous metrology—a formal system of measurement—to quantify intelligence or predict task difficulty. This research proposes a step toward such a science by focusing on trainability and discovery. Much like thermodynamics was developed to explain the limits of steam engines, the authors argue for a "cognodynamics" to explain the "thought engines" of today.
By utilizing the Game Of Hidden Rules (GOHR), this study measures how reinforcement learning (RL) agents infer latent logic through trial and error. The environment requires agents to simultaneously discover a governing rule (e.g., "only red pieces belong in the top-left bucket") and execute the optimal policy to clear the board, mimicking human inductive reasoning.
Methodology and Model Architecture
The researchers employed a Transformer-based Advantage Actor-Critic (A2C) algorithm. The architecture, designed to handle the sequential nature of the game and the need for rule induction, is detailed below:
A critical component of the study is the comparison between two distinct state representations:
- Feature-Centric (FC) Representation: The state is encoded as a stack of feature maps (one for each shape and color), emphasizing spatial coordinates on the 6x6 grid.
- Object-Centric (OC) Representation: Each piece is treated as a distinct entity described by a 20-dimensional vector containing properties like color, shape, and coordinates.
The GOHR Environment
The GOHR benchmark tests rule induction across 18 distinct rules, ranging from simple positional mappings to complex conditional sequences. The board consists of a 6x6 grid with four corner buckets (labeled Buckets 0 through 3). Agents must learn to move pieces to the correct bucket based on the hidden rule currently in effect.
In the FC model, the board is represented as a multi-dimensional tensor, as shown in Figure 2. While this provides parallel signals, it requires the model to manually learn the association between disparate feature maps (e.g., a "square" map and a "red" map) to identify a single "red square."
Key Insights: Representation and Generalization
The study reveals that "representation is destiny." The FC model excelled at positional rules because its input structure is inherently spatial. However, the OC model proved far superior for abstract rules involving object properties. Because the OC model treats objects as entities, it does not need to "re-learn" that a red square at one coordinate is the same type of object as a red square elsewhere.
This difference is most apparent in generalization tests. When trained on a "checkerboard" pattern (training on white squares, testing on black), FC models failed significantly, often reaching 100% error. In contrast, OC models maintained stable performance by associating behavior with object features rather than specific grid locations.
The Geometry of Difficulty
By applying Multi-Dimensional Scaling (MDS) to agent performance, the authors began mapping a "geometry of difficulty." They found that rules cluster into "families." Learning efficiency varied wildly; simple quadrant mapping was mastered in roughly 421 steps, while complex conditional rules required over 229,000 steps. Furthermore, the study demonstrated a "transfer effect": learning component rules (A and B) significantly accelerated the learning of a compound rule (A+B), though intervening unrelated tasks could degrade this benefit.
Conclusion
While the current difficulty metrics are tied to the Transformer-A2C architecture, this work lays the groundwork for a more comprehensive "periodic table" of task difficulty. Future research aims to develop a formal metric to quantify the distance between different logical rules, moving closer to a universal science of AI measurement.
Summary updated