ABCDE: An Agent-Based Cognitive Development Environment

Jieyi Ye, Jiafei Duan, Samson Yu, Bihan Wen, Cheston TanarXiv 2022 · OpenReview

Abstract

Children's cognitive abilities are sometimes cited as AI benchmarks. How can the most common 1,000 concepts (89% of everyday use) be learnt in a naturalistic children's setting? Cognitive development in children is about quality, and new concepts can be conveyed via simple examples. Our approach of knowledge scaffolding uses simple objects and actions to convey concepts, like how children are taught. We introduce ABCDE, an interactive 3D environment modeled after a typical playroom for children. It comes with 300+ unique 3D object assets (mostly toys), and a large action space for child and parent agents to interact with objects and each other. ABCDE is the first environment aimed at mimicking a naturalistic setting for cognitive development in children; no other environment focuses on high-level concept learning through learner-teacher interactions. The simulator can be found at https://pypi.org/project/ABCDESim/1.0.0/

Introduction: The Gap in AI Cognitive Training

Current AI training often resembles "animal training"—processing millions of data points until a model recognizes patterns. While effective for specific tasks, this differs fundamentally from human cognitive development. A five-year-old child recognizes far fewer object classes than ImageNet contains, yet possesses a much deeper, grounded understanding of how the world works.

Existing 3D environments, such as AI2-THOR or Habitat, typically focus on adult-centric household tasks like cooking or cleaning. There is a lack of "naturalistic" settings designed for knowledge scaffolding—the process where a teacher uses simple examples to help a learner grasp abstract concepts. ABCDE provides a sandbox where agents can learn the 1,000 most common concepts through interaction rather than just observation.

Methodology: Knowledge Scaffolding and Agents

The ABCDE environment, developed using the Nvidia PhysX Engine, ensures realistic physical interactions. The core methodology relies on two distinct agent roles to facilitate pedagogical learning:

  1. The Parent (Teacher) Agent: This agent is algorithmically programmed to demonstrate concepts. It generates natural language text, points at objects, and performs actions to show the child agent what a concept looks like (e.g., placing a ball under a table to teach prepositions).
  2. The Child (Learner) Agent: Designed for imitation learning and exploration, this agent observes the parent's motions and verbal cues to ground language in physical reality.
The Parent and Child agents designed for learner-teacher interactions within the ABCDE environment.

The system supports a large action space, including navigation (crawling, walking) and interaction (grabbing, rotating). To ensure clear visual examples, the environment allows for dynamic scene generation where objects are spawned with specific separation buffers to prevent overlaps.

Dataset and Environment Attributes

ABCDE includes a curated library of assets and multi-modal data outputs to support robust training:

  • 3D Object Assets: The environment includes 332 unique 3D objects, primarily sourced from the Google Scanned Object dataset. These are categorized into 89 static objects (furniture) and 243 interactable objects (toys).
  • Visual Data Streams: The simulator outputs multiple data streams for training, including RGB images, depth maps, instance segmentation, and surface normals.
Visual data attributes provided by the simulator, including RGB, instance segmentation, depth maps, and surface normals.

Key Insights

  • Pedagogy over Big Data: Conceptual knowledge is best conveyed via simplified, abstracted examples. An agent might learn better by seeing 50 variations of a chair used in different pedagogical contexts rather than 20,000 static images.
  • Embodied Agency: In most simulators, objects move without clear cause. ABCDE insists on an embodied teacher because children learn by observing human agency and intent.
  • Linguistic Grounding: While most benchmarks focus on nouns, ABCDE is built to teach determiners, prepositions, and verbs—the "connective tissue" of language—by tying abstract language to specific, observable states in the 3D world.

Results and Future Work

The primary result of this work is the release of the ABCDE Simulator. The system provides multi-camera support (four stationary cameras) to ensure full visual coverage of the playroom. It is designed to replicate the density of "learning events" a child experiences (roughly 240 per waking hour), allowing researchers to generate labeled video examples of high-level actions like "put-on" or "keep-inside" automatically.

Future work will involve benchmarking modern Reinforcement Learning (RL) and Large Language Models (LLMs) within this environment, expanding the vocabulary beyond the initial 1,000 concepts, and exploring human-in-the-loop controls for the Parent agent to provide more nuanced educational scaffolding.

Summary updated

Abstract

Children's cognitive abilities are sometimes cited as AI benchmarks. How can the most common 1,000 concepts (89\% of everyday use) be learnt in a naturalistic children's setting? Cognitive development in children is about quality, and new concepts can be conveyed via simple examples. Our approach of knowledge scaffolding uses simple objects and actions to convey concepts, like how children are taught. We introduce ABCDE, an interactive 3D environment modeled after a typical playroom for children. It comes with 300+ unique 3D object assets (mostly toys), and a large action space for child and parent agents to interact with objects and each other. ABCDE is the first environment aimed at mimicking a naturalistic setting for cognitive development in children; no other environment focuses on high-level concept learning through learner-teacher interactions. The simulator can be found at https://pypi.org/project/ABCDESim/1.0.0/
BibTeX
@misc{ye2022abcde,
  author = {Jieyi Ye and Jiafei Duan and Samson Yu and Bihan Wen and Cheston Tan},
  title = {{ABCDE: An Agent-Based Cognitive Development Environment}},
  year = {2022},
  eprint = {2206.04909},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.04909},
}
Figure