Photonic neuromorphic architecture for tens-of-task lifelong learning

Yuan Cheng, Jianing Zhang, Tiankuang Zhou, Yuyan Wang, Zhihao Xu, Xiaoyun Yuan, Lu FangLight 2024 · DOI

TL;DR

This paper introduces L²ONN, a novel photonic neuromorphic architecture that enables lifelong learning for AI systems. By leveraging the inherent properties of light, L²ONN can incrementally learn tens of tasks without forgetting previous knowledge, offering significant advantages in efficiency and scalability over traditional electronic approaches.

What Problem Does This Paper Address?

The rapid growth of AI tasks and the increasing complexity of datasets are pushing the limits of current computing architectures. Traditional electronic deep neural networks face challenges with energy consumption and scalability, especially for continuous learning scenarios. A key problem in machine learning is "catastrophic forgetting," where an AI model trained on new tasks rapidly loses its ability to perform previously learned tasks. This hinders the development of truly adaptable and versatile AI systems that can learn and evolve over time, much like humans do. Existing photonic computing approaches, while promising for speed and efficiency, have also struggled with this lifelong learning challenge.

Methods

The researchers propose a reconfigurable lifelong-learning optical neural network (L²ONN) that integrates algorithm and hardware design. The core innovation lies in its ability to learn incrementally without forgetting. Key aspects of the L²ONN architecture include:

  • Sparse Photonic Neuron Activation: Instead of activating all neurons, L²ONN selectively activates sparse photonic neuron connections. This is achieved using phase-change materials (PCMs) that act as adaptive optical filters. These filters can be switched to modulate both the intensity and phase of light, allowing for task-specific sparse activations.
  • Multi-Spectrum Parallel Processing: Different tasks are encoded into coherent light fields with distinct wavelengths. This allows for parallel processing of multi-task optical features, with each wavelength carrying information for a specific task.
  • Algorithm-Hardware Co-Design: The architecture is designed to exploit the natural properties of light, such as sparsity and parallelism, rather than simply mimicking electronic neural network structures. This "light-matter interaction" approach is central to its lifelong learning capability.
  • Lifelong Learning Strategy: When learning a new task, L²ONN adaptively activates new photonic neurons, effectively expanding its knowledge base. These newly learned connections are then consolidated to prevent forgetting. The training strategy involves initially learning a dense activation map for a task, which is then pruned to a sparse one, and this sparse activation map is fixed for subsequent learning.

The paper presents both a free-space implementation and an on-chip version of the L²ONN.

Datasets

The L²ONN was evaluated on a variety of datasets:

  • Vision Classification:
    • MNIST
    • FashionMNIST
    • KMNIST
    • OracleMNIST
    • OverheadMNIST
  • Voice Recognition:
    • Vowel (12 classes)
    • Number, Word, Command (subsets of Speech Commands, 10, 15, 10 classes respectively)
    • Gender (4 classes)
    • UrbanSound (10 classes)
  • Medical Diagnosis:
    • BloodMNIST (8 classes)
    • OrganMNIST (11 classes)
    • PathMNIST (9 classes)
    • TissueMNIST (8 classes)
  • Other Classification:
    • Iris flower classifier (3 classes)
    • Red wine quality (3 classes)

The paper notes that for voice recognition tasks, audio data was pre-processed into mel-scale frequency cepstral coefficients (MFCCs). The medical diagnosis datasets are subsets of MedMNIST.

Key Insights and Ideas

  • Light as a Natural Lifelong Learner: The paper demonstrates that the inherent properties of light (sparsity, parallelism, speed) can be harnessed to create AI systems that learn continuously and efficiently, mirroring biological learning processes.
  • Overcoming Catastrophic Forgetting in Photonics: L²ONN is presented as the first photonic computing architecture to successfully avoid catastrophic forgetting, a major hurdle for AI systems that need to adapt to new information over time.
  • Algorithm-Hardware Co-Design is Crucial: The success of L²ONN stems from designing the hardware (photonic architecture) to align with the desired learning algorithm (lifelong learning), rather than trying to force an electronic paradigm onto optics.
  • Sparsity is Key to Scalability and Efficiency: The deliberate use of sparse photonic neuron activation is shown to be critical for both reducing computational load and enabling a much larger learning capacity compared to existing optical and electronic networks.
  • Learning Sequence Matters: The paper highlights that, similar to human learning, an "easy-to-hard" learning sequence leads to better performance and efficiency than a "hard-to-easy" sequence.

Key Results

  • Lifelong Learning Capability: L²ONN successfully learned tens of tasks (e.g., 5 vision classification tasks, 6 voice recognition tasks, 4 medical diagnosis tasks) with a single model, without forgetting previously learned tasks.
  • Efficiency: L²ONN achieves more than an order of magnitude higher efficiency than representative electronic ANNs (e.g., LeNet).
  • Capacity: The architecture boasts a 14x larger learning capacity than existing optical neural networks while maintaining competitive accuracy on individual tasks.
  • Accuracy: While achieving high accuracy on individual tasks, L²ONN only experienced a slight accuracy drop (at most 1.9%) compared to vanilla ONNs when learning multiple tasks.
  • Sparsity Performance: L²ONN maintains competitive accuracy even at 99% sparsity, outperforming vanilla ONNs and electronic ANNs at high sparsity levels.
  • On-Chip Validation: A fabricated on-chip L²ONN successfully demonstrated lifelong learning capabilities on classification tasks, showing high integration and compactness.

Limitations and Future Work

  • Learning Sequence Impact: While the paper shows that the learning sequence (easy-to-hard) is beneficial, further exploration into optimal learning strategies for complex, real-world scenarios could be valuable.
  • Integration with Electronic Systems: While the on-chip L²ONN is promising for terminal/edge AI, further work on seamless integration with existing electronic systems might be needed.
  • Complexity of Tasks: The paper demonstrates performance on various classification and recognition tasks. Extending L²ONN to more complex generative or reinforcement learning tasks could be a future direction.
  • Hardware Fabrication: While an on-chip version is demonstrated, further advancements in fabrication techniques for even larger and more complex photonic neuromorphic systems could unlock greater potential.

Summary updated

Abstract

Scalable, high-capacity, and low-power computing architecture is the primary assurance for increasingly manifold and large-scale machine learning tasks. Traditional electronic artificial agents by conventional power-hungry processors have faced the issues of energy and scaling walls, hindering them from the sustainable performance improvement and iterative multi-task learning. Referring to another modality of light, photonic computing has been progressively applied in high-efficient neuromorphic systems. Here, we innovate a reconfigurable lifelong-learning optical neural network (L<sup>2</sup>ONN), for highly-integrated tens-of-task machine intelligence with elaborated algorithm-hardware co-design. Benefiting from the inherent sparsity and parallelism in massive photonic connections, L<sup>2</sup>ONN learns each single task by adaptively activating sparse photonic neuron connections in the coherent light field, while incrementally acquiring expertise on various tasks by gradually enlarging the activation. The multi-task optical features are parallelly processed by multi-spectrum representations allocated with different wavelengths. Extensive evaluations on free-space and on-chip architectures confirm that for the first time, L<sup>2</sup>ONN avoided the catastrophic forgetting issue of photonic computing, owning versatile skills on challenging tens-of-tasks (vision classification, voice recognition, medical diagnosis, etc.) with a single model. Particularly, L<sup>2</sup>ONN achieves more than an order of magnitude higher efficiency than the representative electronic artificial neural networks, and 14× larger capacity than existing optical neural networks while maintaining competitive performance on each individual task. The proposed photonic neuromorphic architecture points out a new form of lifelong learning scheme, permitting terminal/edge AI systems with light-speed efficiency and unprecedented scalability.
BibTeX
@inproceedings{cheng2024photonic,
  author = {Yuan Cheng and Jianing Zhang and Tiankuang Zhou and Yuyan Wang and Zhihao Xu and Xiaoyun Yuan and Lu Fang},
  title = {{Photonic neuromorphic architecture for tens-of-task lifelong learning}},
  booktitle = {Light},
  year = {2024},
  doi = {10.1038/s41377-024-01395-4},
}
Figure