Abstract
To address modeling problems of brain-inspired intelligence, this research focuses on a semantic-oriented framework design for multimedia and multimodal information. The Multimedia Neural Cognitive Computing (MNCC) model was designed based on nervous mechanisms and cognitive architecture. Furthermore, the semantic-oriented hierarchical Cross-modal Neural Cognitive Computing (CNCC) framework was proposed based on the MNCC model, providing a formal description and analysis. This framework effectively improves semantic processing performance for multimedia and cross-modal information, marking a significant step toward realizing brain-inspired computing.
Introduction
Despite the success of deep learning, modern AI systems often function as "black boxes" that lack general autonomous intelligence. They struggle to integrate information across different senses—cross-modal processing—in the seamless way humans do. This research addresses the gap between Brain-Inspired Intelligence (BII) and current algorithms by proposing a unified architecture capable of handling perception, attention, memory, and high-level reasoning within the context of massive, unstructured multimedia data.
The CNCC Framework: A Hierarchical Approach
The core of this research is the Cross-modal Neural Cognitive Computing (CNCC) framework. It organizes computational systems into a hierarchy that mirrors the evolutionary structure of the human brain:
- Level 1 (Perceptual): Simulates the "motor brain" (archicortex), focusing on low-level perception and saliency detection using Spiking Neural Networks (SNN).
- Level 2 (Neural): Simulates the "emotional brain" (paleocortex/limbic system), handling memory and reinforcement learning via Deep Belief Networks (DBN) and Convolutional Neural Networks (CNN).
- Level 3 (Cognitive): Simulates the "rational brain" (neocortex), enabling high-level ensemble learning and Bayesian reasoning through Probabilistic Graphical Models (PGM).
To provide a rigorous foundation, the framework utilizes the Chemical Abstract Machine (CHAM) for formal description. This metaphor treats data as "molecules" and cognitive processes as "chemical reactions," allowing for a structured transformation of information across layers.
Biology-Driven Insights
The CNCC framework is built on several key biological principles:
- Homomorphism Mapping: Establishing a direct link between biological nervous structures and functional computing modules.
- Thalamus-Cortical Projection: Modeling the thalamus as a "switch" for selective attention, allowing the system to filter noise and focus on relevant semantic features.
- Functional Modularity: Explicitly dividing the architecture into modules representing the hippocampus (memory), the limbic system (emotion), and the neocortex (logic).
Experimental Validation
The framework was validated using high-resolution remote sensing and target recognition datasets, including HRSS, UCMLU, and MSTAR. These tasks are particularly challenging due to the complexity and scale of ground objects.
The CNCC framework demonstrated exceptional performance:
- Scene Classification: Achieved an Average Precision (AP) of 84.73% on HRSS and 88.26% on UCMLU.
- Target Recognition: Reached 99.90% AP and 99.88% Overall Accuracy (OA) on the MSTAR dataset.
- Target Detection: Achieved a Probability of Detection (PD) of 91.63% on harbor-specific datasets (HRSHTD).
Conclusion and Future Work
While the CNCC framework provides a robust theoretical bridge between neuroscience and AI, challenges remain. Specifically, the "mesoscopic gap"—understanding how local networks assemble into advanced global functions—requires further study. Additionally, the mathematical complexity of the CHAM description presents implementation hurdles. Future research will focus on applying this hierarchical model to unmanned autonomous systems and advanced cross-media search engines.
Summary updated