Abstract
Brain-inspired approaches can efficiently analyze the activities of biological neural networks and solve computationally hard problems with energy efficiencies unattainable with von Neumann architectures. This paper presents a brain-inspired multimodal signal processing system with organic memristor arrays that integrate signal sensory, storage, and computation. The system utilizes four key components: a multimodal signal sensory module for signal collection and fusion; a high-density cross-point memristive synapse array fabricated from albumin protein; a general learning module for hierarchical learning, recognition, and imagination; and a peripheral circuit module for signal modulation and conversion. This system enables in situ processing of massive multimodal data streams, facilitating the integration of nanomaterials into energy-efficient neuromorphic computing systems.
The Challenge: Bypassing the Von Neumann Bottleneck
Traditional computers follow the von Neumann architecture, where the processor and memory are separate. This creates two major hurdles: the "von Neumann bottleneck," where data movement consumes massive energy, and the "memory wall," a performance mismatch between fast processors and slower memory units. Furthermore, while the human brain effortlessly integrates multiple senses, most current AI hardware is specialized for a single data type. This research addresses these issues by creating a system that senses, stores, and computes multimodal data in one place using eco-friendly materials.
Methods: Mimicking the Brain's Hierarchy
The authors built a four-part system designed to mimic the hierarchical structure of the human brain:
- Organic Memristors: The "synapses" are made from Ag/Egg Albumin/ITO. Using natural proteins makes the devices more eco-friendly and flexible. These memristors change resistance based on previous current, allowing them to "remember" and process data simultaneously.
- Multimodal Sensory Module: This includes sensors for vision, touch, sound, smell, and taste. A coupled auto-encoder/decoder fuses these signals into a single feature vector, which is then converted into "spike trains" (electrical pulses) similar to biological neuronal communication.
- Hierarchical General Learning Module: The system organizes neurons into "chunks" and uses Winnerless Competition (WLC). Instead of a single "winning" neuron, the system moves through a sequence of states, creating a memory trace for pattern recognition over time.
- In-Memory MAC Operations: Multiply-Accumulate (MAC) operations are performed directly within the memristive cross-point array using Kirchhoff’s and Ohm’s laws, eliminating the need to move data to a separate processor.
Datasets and Evaluation
The researchers evaluated the system using a custom multimodal dataset:
- Training Data: 2,000 sets of audio signals, including music fragments and specific sounds like "truck," "train," and "car."
- Multimodal Mapping: These audio signals were associated with corresponding visual, tactile, olfactory, and gustatory data to train the "imagination" capability of the hierarchical learning module.
Key Insights and Results
The use of egg albumin provides a stable resistive switching effect with a high HRS/LRS ratio of $10^4$, crucial for distinguishing between memory states. One of the most striking features is the system's ability to "imagine." After training, a single stimulus (e.g., a car horn) can trigger associated visual and tactile memory traces, reconstructing a multimodal experience.
Key results include:
- Stability: The albumin-based memristors remained stable over 900 switching cycles.
- Optimization: A 10% hydrogen peroxide ($H_2O_2$) modification of the egg albumin provided the most optimized memory cell performance at room temperature.
- Functional Success: The system successfully demonstrated "semantic imagination," reproducing complex multimodal representations from audio input alone.
Limitations and Future Work
Despite promising results, the authors note several areas for improvement. Brain-inspired in-memory computing is still in its early stages, and the physical mechanisms of organic memristors at the molecular level require deeper analysis. Additionally, while protein-based memristors are eco-friendly, their long-term durability compared to traditional silicon-based devices needs further testing. Future work will focus on creating more robust fault-diagnosis systems and improving cognition function modeling to handle real-world, noisy data.
Reference: Ji, X., Dong, Z., Lai, C. S., & Qi, D. "A Brain-inspired in-Memory Computing System for Neuronal Communication via Memristive Circuits." IEEE Communications Magazine.
Summary updated