Metasurface-enabled on-chip multiplexed diffractive neural networks in the visible

Xuhao Luo, Yueqiang Hu, Xiangnian Ou, Xin Li, Jiajie Lai, Na Liu, Xinbin Cheng, Anlian Pan, Huigao Duan

TL;DR

Researchers have developed a compact, on-chip "Metasurface-based Diffractive Neural Network" (MDNN) that performs image recognition at the speed of light using visible light. By leveraging polarization multiplexing, the device can process multiple tasks simultaneously—such as identifying handwritten digits and fashion items at the same time—with a massive neuron density of over 6 million neurons per square millimeter.

What Problem Does This Paper Address?

Traditional artificial intelligence relies on electronic processors, which face bottlenecks in speed, parallelism, and power consumption. All-optical computing, which uses photons instead of electrons, offers a way out.

While "diffractive deep neural networks" (D²NNs) have been demonstrated before, they typically suffer from two major flaws:

  1. Bulkiness: They often operate in the terahertz or infrared range, requiring large components that are difficult to integrate into standard electronic devices.
  2. Single-tasking: Most existing optical networks are "single-skilled," meaning they can only perform one specific task at a time, unlike the multi-channel processing of the human brain.

This paper introduces a way to shrink these networks down to the visible spectrum and integrate them directly onto a standard CMOS imaging sensor while enabling multitasking.

Methods

The authors designed a Metasurface-based Diffractive Neural Network (MDNN). The core of the system consists of subwavelength TiO2 (titanium dioxide) nanopillars.

  • Polarization Multiplexing: This is the "secret sauce." By using rectangular nanopillars, the researchers can independently control how the metasurface interacts with x-polarized and y-polarized light. This allows them to "bake" two different neural network functions into the same physical layer.
  • Diffractive Layers as Neurons: Each nanopillar acts as an artificial neuron. Light propagating through these layers is modulated in phase, effectively performing the weighted additions and transformations required for a neural network.
  • On-Chip Integration: The metasurface is fabricated using electron-beam lithography (EBL) and atomic layer deposition (ALD), then bonded directly to a Sony IMX686 CMOS sensor using an optically clear adhesive (OCA).
  • Training: The network's phase distributions were optimized using standard deep learning techniques (error backpropagation) in Python/TensorFlow, then verified using both scalar diffraction theory and 3D full-vector FDTD (Finite-Difference Time-Domain) simulations.

Datasets

The MDNN was trained and evaluated using two classic machine learning benchmarks:

  • MNIST: A dataset of 70,000 handwritten digits (0-9).
  • Fashion-MNIST: A dataset of 70,000 images of fashion products (e.g., t-shirts, sneakers, ankle boots).

The dual-channel nature of the device allowed it to process one dataset on the x-polarization channel and the other on the y-polarization channel simultaneously.

Key Insights and Ideas

  • Multitasking at the Speed of Light: By using the vector nature of light (polarization), the researchers proved that an optical network can perform parallel, independent tasks without increasing the physical footprint of the device.
  • Extreme Density: The use of subwavelength structures allows for an incredible neuron density of $6.25 \times 10^6 / \text{mm}^2$. This is orders of magnitude higher than previous diffractive networks operating at longer wavelengths.
  • Phase Dominance: The researchers found that while metasurfaces inevitably introduce some "amplitude crosstalk" (unwanted changes in light brightness), the phase modulation is so dominant that the network's accuracy remains high.
  • CMOS Compatibility: Unlike previous designs that required specialized detectors, this MDNN works in the visible spectrum (532 nm), making it directly compatible with the sensors found in smartphones and industrial cameras.

Key Results

  • High Accuracy: In numerical simulations, the MDNN achieved over 99% accuracy for both MNIST and Fashion-MNIST classification.
  • Experimental Validation: The physical prototype's results matched the simulations with 96% consistency.
  • Energy Efficiency: In vector simulations, the network successfully focused over 80% of the input energy into the correct target detection region on the sensor.
  • Miniaturization: The experimental device used a single hidden layer with 78,400 neurons in an area of just $112 \times 112 \text{ μm}^2$.

Limitations and Future Work

While the results are a significant step forward, the authors note a few areas for improvement:

  • Experimental Complexity: The physical prototype used only one hidden layer. While they simulated up to five layers, fabricating multi-layer 3D-integrated metasurfaces remains a significant manufacturing challenge.
  • Polarization Leakage: Experimental errors were partly attributed to the linear polarizers not being 100% efficient, leading to slight crosstalk between the two channels.
  • Scaling Up: The authors suggest that future versions could use more than just two channels by incorporating other multiplexing schemes, such as wavelength (different colors of light), spatial mode, or orbital angular momentum (vortex) multiplexing.

This work paves the way for "smart" image sensors that can perform high-speed object recognition or sensing before the data even reaches a traditional electronic processor.

Summary updated

Abstract

Replacing electrons with photons is a compelling route toward high-speed, massively parallel, and low-power artificial intelligence computing. Recently, diffractive networks composed of phase surfaces were trained to perform machine learning tasks through linear optical transformations. However, the existing architectures often comprise bulky components and, most critically, they cannot mimic the human brain for multitasking. Here, we demonstrate a multi-skilled diffractive neural network based on a metasurface device, which can perform on-chip multi-channel sensing and multitasking in the visible. The polarization multiplexing scheme of the subwavelength nanostructures is applied to construct a multi-channel classifier framework for simultaneous recognition of digital and fashionable items. The areal density of the artificial neurons can reach up to 6.25 × 10<sup>6</sup> mm<sup>-2</sup> multiplied by the number of channels. The metasurface is integrated with the mature complementary metal-oxide semiconductor imaging sensor, providing a chip-scale architecture to process information directly at physical layers for energy-efficient and ultra-fast image processing in machine vision, autonomous driving, and precision medicine.
BibTeX
@inproceedings{luo2022metasurface,
  author = {Xuhao Luo and Yueqiang Hu and Xiangnian Ou and Xin Li and Jiajie Lai and Na Liu and Xinbin Cheng and Anlian Pan and Huigao Duan},
  title = {{Metasurface-enabled on-chip multiplexed diffractive neural networks in the visible}},
  booktitle = {Light},
  year = {2022},
  doi = {10.1038/s41377-022-00844-2},
}

Loading related content...

Figure