Unleashing Artificial Cognition: Integrating Multiple AI Systems

Muntasir Adnan, Buddhi Gamage, Zhiwei Xu, Damith Herath, Carlos C. N. Kuhn

TL;DR

This paper introduces OpenSI-COSMIC, an open-source AI system that integrates Large Language Models (LLMs) with specialized tools like a chess engine and a vector database. The goal is to move beyond task-specific AI towards more general cognitive abilities, demonstrated through chess. The system aims to provide not just answers, but also explanations for its decision-making process, bridging the gap between raw computation and human-like understanding.

What Problem Does This Paper Address?

Current AI systems, while excelling at specific tasks like image recognition or natural language processing, often lack "general cognition" abilities. This fragmentation limits their potential for broader applications that require seamless interaction of multiple cognitive functions. The authors argue that achieving Artificial General Intelligence (AGI) necessitates systems that can integrate various cognitive abilities, such as perception, memory, attention, reasoning, and anticipation. Evaluating these integrated cognitive abilities in AI is challenging, and existing benchmarks like the Turing Test are considered insufficient. This paper proposes a framework and a system to demonstrate and evaluate these integrated cognitive abilities in a controlled environment, using chess as a testbed.

Methods

The core of the OpenSI-COSMIC system is the integration of multiple AI components:

  • Query Analyzer: An LLM-based service that intelligently routes user queries to the most appropriate underlying service.
  • Large Language Models (LLMs): The system supports various LLMs, including fine-tuned versions using Parameter-Efficient Fine-Tuning (PEFT) and Low-Rank Adaptation (LoRA). Mistral 7B is used as the base model for fine-tuning.
  • External Knowledge Source (Vector Database + RAG): A Faiss vector database is used to store and retrieve information, augmented by Retrieval-Augmented Generation (RAG) techniques. This allows the system to access and utilize external knowledge.
  • Chess Engine (Stockfish): A powerful chess engine is integrated to provide expert-level move analysis and prediction.
  • Vector Database Update Service: Enables real-time updates to the vector database, ensuring the system has access to the latest information.

The system employs a teacher-student learning paradigm for fine-tuning the LLM. A powerful teacher model (GPT-4o) is used to generate step-by-step reasoning explanations, which then form training data for a smaller student model. This aims to transfer sophisticated reasoning capabilities to the student model.

The paper also details a scoring mechanism to evaluate five key cognitive qualities: Perception, Memory, Attention, Reasoning, and Anticipation. This involves curated question-and-answer sets and human evaluation for certain aspects.

Datasets

  • Custom Chess Reasoning Dataset: A dataset of chess games with detailed, step-by-step reasoning annotations for each move, inspired by the "chain-of-thought" approach.
  • Kaggle Lichess Dataset: Publicly available dataset used to supplement the custom dataset for broader chess scenario coverage.
  • Chess.com Puzzles: Used for evaluating reasoning and anticipation capabilities.
  • Books for RAG: Three books are used to create RAG questions for evaluating attention.

Key Insights and Ideas

  • Integration is Key: The paper strongly suggests that combining specialized AI models (like a chess engine) with LLMs is crucial for developing more general cognitive abilities. Individual models are limited, but their synergy can lead to emergent behaviors.
  • Explainability Matters: The system's ability to not only predict moves but also explain why it made that prediction is a significant step towards human-like understanding. This bridges the gap between raw computational power and interpretability.
  • Chess as a Cognitive Testbed: Using chess as a demonstration environment is effective because it requires a complex interplay of perception, memory, reasoning, and anticipation, making it a good proxy for evaluating general cognitive skills.
  • Teacher-Student Learning for Reasoning: The approach of using a powerful LLM as a teacher to train a smaller student model on reasoning tasks shows promise for transferring complex cognitive skills efficiently.
  • RAG Challenges: While RAG is powerful, the fine-tuned model can struggle with novel contexts, leading to hallucinations. This highlights the need for further fine-tuning to effectively leverage retrieved information.

Key Results

  • Query Analyzer Performance: The query analyzer achieved high accuracy in routing queries to the correct services, with GPT-4o reaching 100% and Gemma 7B Instruct at 91.25%.
  • Best-Move Prediction: The integrated system, leveraging Stockfish, demonstrated superior performance in predicting optimal chess moves compared to individual LLMs. While GPT-4o alone had a 32.5% success rate, the system accurately predicted optimal moves in 40 chess games.
  • Cognition Capability Scores: Figure 2 shows normalized scores for the five cognitive qualities across different LLMs. GPT-4o generally performed best, with the fine-tuned Mistral 7B model showing competitive results, especially when integrated into the OpenSI-COSMIC system.
  • Anticipation: When the chess engine was used for anticipation, all models achieved high scores, indicating the engine's strength in this area.

Limitations and Future Work

  • Hallucinations with RAG: The fine-tuned model exhibits confusion and hallucinations when presented with novel contexts from RAG, indicating a need for further fine-tuning to improve context utilization.
  • Limited Dataset: The chess puzzle dataset used for evaluation is acknowledged as not exhaustive, and the questions can be generalized to other domains.
  • Future Work:
    • Improved Query Analyzer: Transitioning to a fine-tuned LLM or a classification model for more context-aware query routing.
  • Automated CoT Generation: Training an LLM to automatically generate Chain-of-Thought queries to enhance reasoning.
  • Self-Correcting Mechanism: Developing a feature for the AI to identify and correct errors during code generation.
  • Distributed Architecture: Replacing the single LLM with a collection of specialized, fine-tuned LLMs deployed in a distributed system for better scalability and adaptability.
  • Further RAG Study: Dedicated research into enhancing RAG capabilities across various problem-solving tasks.

Summary updated

Abstract

In this study, we present an innovative fusion of language models and query analysis techniques to unlock cognition in artificial intelligence. The introduced open-source AI system seamlessly integrates a Chess engine with a language model, enabling it to predict moves and provide strategic explanations. Leveraging a vector database to achieve retrievable answer generation, our AI system elucidates its decision-making process, bridging the gap between machine's computational cognition and human-like understanding. Our choice of Chess as the demonstration environment underscores the versatility of our approach. Beyond Chess, our system holds promise for diverse applications, from medical diagnostics to financial forecasting. Our AI system is available at \href{https://github.com/TheOpenSI/CoSMIC.git}{https://github.com/TheOpenSI/CoSMIC.git}.
BibTeX
@inproceedings{adnan2024unleashing,
  author = {Muntasir Adnan and Buddhi Gamage and Zhiwei Xu and Damith Herath and Carlos C. N. Kuhn},
  title = {{Unleashing Artificial Cognition: Integrating Multiple AI Systems}},
  booktitle = {ACIS},
  year = {2024},
  url = {https://openreview.net/forum?id=JVp8e98b27},
}
Figure