Toward High-Accuracy, Programmable Extreme-Edge Intelligence for Neuromorphic Vision Sensors utilizing Magnetic Domain Wall Motion-based MTJ

Md. Abdullah-Al Kaiser, Gourav Datta, Peter A. Beerel, Akhilesh R. Jaiswal

Introduction

Traditional computer vision at the "extreme edge"—such as on autonomous drones or smart cameras—is fundamentally limited by the von Neumann bottleneck. The energy cost of moving massive amounts of raw pixel data from a sensor to a separate backend processor often exceeds the energy required for the computation itself. While Neuromorphic Vision Sensors (NVS) offer a more efficient alternative by recording only asynchronous light changes (events), they still face significant hurdles in local processing.

The first layer of a Spiking Neural Network (SNN) typically requires multi-bit Multiply-Accumulate (MAC) operations. Current analog CMOS implementations use capacitors to store "membrane potential," but these capacitors suffer from charge leakage. This leakage necessitates short integration windows, which limits the complexity of the features the sensor can extract and forces a rigid timing structure on what should be an asynchronous process.

The CMOS+X P2M Architecture

To address these challenges, this research introduces a Processing-in-Pixel-in-Memory (P2M) architecture using a hybrid CMOS+X approach. Here, "X" represents Magnetic Domain Wall Magnetic Tunnel Junction (MDWMTJ) technology.

Unlike standard MTJs, which are binary (switching between high and low resistance), MDWMTJs allow for a continuous range of resistance based on the position of a magnetic domain wall. This physical property allows them to function as non-volatile "accumulators." Because the state is stored magnetically, it does not leak over time, allowing the sensor to "remember" its state indefinitely without power consumption. This enables much longer integration times and more robust asynchronous processing.

Hybrid Weight Implementation and 3D Integration

Spintronic devices often suffer from a low dynamic range. To overcome this, the authors developed a hybrid weight system that combines CMOS transistor widths (providing high dynamic range) with MDWMTJ resistance (providing fine-tuned programmability).

The physical implementation utilizes a heterogeneously integrated 3D stack. The top die contains a backside-illuminated NVS array, while the bottom die houses the P2M compute elements. These layers are connected via high-density Cu-Cu hybrid bonding, minimizing the footprint and maximizing interconnect speed.

Hardware-Aware Co-design Framework

One of the primary obstacles to using emerging spintronic devices is their inherent hardware imperfections, such as non-linearity, process variation, and low tunnel-magnetoresistance (TMR). The researchers addressed this through a device-circuit-algorithm co-design framework. By using HSpice and Verilog-A to model these constraints, they were able to incorporate real-world hardware limitations directly into the neural network training process. This ensures that the trained weights are robust enough to maintain high accuracy despite the physical limitations of the MDWMTJ devices.

Experimental Evaluation and Results

The architecture was benchmarked against three standard neuromorphic datasets: NMNIST, CIFAR10-DVS, and IBM DVS128-Gesture. The results demonstrate that the hybrid system can achieve high-tier performance while significantly reducing energy overhead.

  • Energy Efficiency: By performing feature extraction in-pixel and reducing the data volume sent to the backend, the system achieved an average 45.3% reduction in backend-processor energy.
  • Accuracy: Despite the hardware constraints, the model maintained high accuracy across all tasks:
  • 97.82% on NMNIST.
  • 79.17% on CIFAR10-DVS.
  • 95.99% on IBM DVS128-Gesture.
  • Robustness: Compared to an ideal software baseline, the hardware-constrained model saw a negligible accuracy drop (ranging from only 0.28% to 1.55%), proving the effectiveness of the co-design framework.

Discussion and Future Outlook

The most significant breakthrough of this work is the combination of non-volatility and programmability at the sensor level. Most in-pixel processing is "hard-wired" during fabrication, but the use of MDWMTJs allows weights and neuron firing thresholds to be reprogrammed post-fabrication. This allows a single sensor to be optimized for various tasks, such as switching from simple face detection to complex gesture recognition.

However, challenges remain. The 3D heterogeneous integration of CMOS and spintronic materials is more complex to manufacture than standard CMOS. Additionally, while the researchers optimized for area, fitting complex MAC units under every pixel remains a challenge for scaling to ultra-high resolutions. Future work will likely focus on denser spintronic configurations and aggressive pruning techniques to further reduce the silicon footprint.

Conclusion

By integrating MDWMTJ technology directly into the pixel array, this research provides a viable path toward high-performance, programmable intelligence at the extreme edge. The hybrid CMOS+X approach successfully mitigates the leakage issues of analog CMOS while overcoming the precision limitations of spintronic devices, resulting in a highly efficient, non-volatile vision system.

Summary updated

BibTeX
@inproceedings{kaiser2024toward,
  author = {Md. Abdullah-Al Kaiser and Gourav Datta and Peter A. Beerel and Akhilesh R. Jaiswal},
  title = {{Toward High-Accuracy, Programmable Extreme-Edge Intelligence for Neuromorphic Vision Sensors utilizing Magnetic Domain Wall Motion-based MTJ}},
  booktitle = {DAC},
  year = {2024},
  doi = {10.1145/3649329.3657359},
}
Figure