Generating Chunks for Cognitive Architectures

Goonmeet Bajaj, Kate Pearce, Seán Kennedy, Othalia Larue, Alexander Hough, Jayde King, Christopher W. Myers, Srinivasan ParthasarathyAAAI 2024 · DOI

Abstract

Knowledge engineering is an important task for creating and maintaining a knowledge base for cognitive models. It involves acquiring, representing, and organizing knowledge in a form that computers can use to make decisions and solve problems. However, this process can be a bottleneck for designing and using cognitive models. Knowledge engineering is a time-consuming and resource-intensive task that requires subject matter experts to provide information about a domain. In addition, models can acquire knowledge but require significant mechanisms to structure that information in a structured format appropriate for general use. Given the knowledge engineering bottleneck, we propose a solution that relies on natural language processing to extract key entities, relationships, and attributes to automatically generate chunks encoded as triples or chunks from unstructured text. Once generated, the knowledge can be used to create or add to a knowledge base within cognitive architectures to reduce knowledge engineering and task-specific models.

Introduction

In the field of cognitive science, architectures like ACT-R (Adaptive Control of Thought—Rational) are used to simulate human cognition. These models rely on two types of knowledge: procedural (how to do things) and declarative (facts and concepts).

The "Knowledge Engineering Bottleneck" refers to the grueling, manual process of acquiring, representing, and organizing this declarative knowledge. Currently, subject matter experts must hand-craft "chunks"—the basic units of declarative memory—to provide a model with the information it needs to solve a task. This process is slow, expensive, and doesn't scale. If we want cognitive models to perform complex tasks like analogical reasoning, they need a massive library of structured facts that are currently too tedious to build by hand.

Methodology: Automated Knowledge Extraction

The authors propose an automated pipeline that combines traditional Natural Language Processing (NLP) with modern Large Language Models (LLMs) to bridge the gap between messy human language and rigid symbolic structures:

  1. Data Sourcing: The system identifies relevant Wikipedia pages for a specific domain (in this study, 22 different sports).
  2. Entity Extraction: Using Spacy, the system extracts nouns to create a baseline set of "entity chunks."
  3. Triple Generation: Sentences from Wikipedia are fed into GPT-3.5-turbo to extract knowledge triples in the format: (subject, relation, object).
  4. Filtering & Verification: To ensure quality, the system filters the LLM-generated triples against the entities identified by Spacy. It further refines this by checking if the entities belong to specific types (like "Person" or "Event").
  5. Integration: Verified triples are formatted for the Structure-Mapping Engine (SME), which identifies similarities between different knowledge sets to perform analogical reasoning.
The automated pipeline for retrieving, compiling, and mapping sport-related knowledge chunks.

Key Results and Analysis

The pipeline was tested across 22 sports, including archery, badminton, and soccer. On average, the pipeline extracted 398 triples per sport.

Filtering proved essential for precision. While entity-based filtering reduced the list by about 41%, checking for specific entity types resulted in only about 20% of the original triples remaining. Although this represents a significant reduction in volume, it produced a much cleaner and more relevant knowledge base for cognitive tasks.

Statistical summary of triple generation and filtering efficiency across 22 sports.

The system successfully identified meaningful correspondences between sports. For example, in a comparison between badminton and volleyball, the model correctly mapped functional equivalents like shuttlecock ↔ ball and server ↔ player. Logically related sports yielded higher similarity scores (0.25 for Badminton vs. Volleyball) compared to less related ones (0.08 for Cricket vs. Volleyball).

Discussion and Future Work

While the hybrid pipeline significantly improves efficiency, limitations remain. Some "noisy" triples (e.g., "fans support their team") survive filtering, and the SME can struggle with linguistic variations like verb tenses.

Future directions include:

  • Developing a ranking model to prioritize the most task-relevant triples.
  • Integrating word embeddings to help the model understand that different words (like "pitcher" and "server") might play similar roles.
  • Cross-referencing extracted triples with existing knowledge graphs like ConceptNet to verify factual accuracy.

Summary updated

Abstract

Knowledge engineering is an important task for creating and maintaining a knowledge base for cognitive models. It involves acquiring, representing, and organizing knowledge in a form that computers can use to make decisions and solve problems. However, this process can be a bottleneck for designing and using cognitive models. Knowledge engineering is a time-consuming and resource-intensive task that requires subject matter experts to provide information about a domain. In addition, models can acquire knowledge but require significant mechanisms to structure that information in a structured format appropriate for general use. Given the knowledge engineering bottleneck, we propose a solution that relies on natural language processing to extract key entities, relationships, and attributes to automatically generate chunks encoded as triples or chunks from unstructured text. Once generated, the knowledge can be used to create or add to a knowledge base within cognitive architectures to reduce knowledge engineering and task-specific models.
BibTeX
@inproceedings{bajaj2024generating,
  author = {Goonmeet Bajaj and Kate Pearce and Seán Kennedy and Othalia Larue and Alexander Hough and Jayde King and Christopher W. Myers and Srinivasan Parthasarathy},
  title = {{Generating Chunks for Cognitive Architectures}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaaiss.v2i1.27683},
}
Figure