Opto-ViT: Architecting a Near-Sensor Region of Interest-Aware Vision Transformer Accelerator with Silicon Photonics

Mehrdad Morsali, Chengwei Zhou, Deniz Najafi, Sreetama Sarkar, Pietro Mercati, Navid Khoshavi, Peter A. Beerel, Mahdi Nikdast, Gourav Datta, Shaahin Angizi

Abstract

Vision Transformers (ViTs) have set new benchmarks in computer vision by capturing long-range dependencies through self-attention mechanisms. However, their deployment on edge devices is hindered by high computational complexity and memory demands. We present Opto-ViT, a hybrid electronic-photonic accelerator designed for near-sensor processing. By leveraging silicon photonics for high-speed matrix multiplications and a novel Mask Generation Network (MGNet) for region-of-interest (RoI) pruning, Opto-ViT achieves significant energy savings and throughput improvements. Experimental results demonstrate an efficiency of 100.4 KFPS/W and up to 84% energy reduction with minimal impact on accuracy.

Introduction

While traditional Convolutional Neural Networks (CNNs) rely on local receptive fields, Vision Transformers (ViTs) utilize self-attention to model global context, leading to superior performance in tasks like image classification and object detection. Despite their efficacy, ViTs are notoriously "compute-hungry" and "memory-heavy." Standard CMOS-based hardware faces significant bottlenecks when executing these models at the edge, primarily due to the energy costs of data movement and the sequential nature of electronic matrix-matrix multiplications (MatMul).

Opto-ViT addresses these challenges by integrating silicon photonics (SiPh) into the accelerator architecture. By processing data at the speed of light and implementing hardware-aware pruning, Opto-ViT bridges the gap between high-performance transformer models and the strict power constraints of edge devices.

Architecture and Methodology

1. Hybrid Electronic-Photonic Core

Opto-ViT utilizes a bifurcated architecture to maximize the strengths of different physical domains:

  • Optical Core: This unit handles the most computationally intensive operations—large-scale MatMuls. It employs Vertical-Cavity Surface-Emitting Lasers (VCSELs) and Microring Resonators (MRs) to perform calculations in the optical domain, significantly reducing latency compared to electronic alternatives.
  • Electronic Core: Non-linear operations such as Softmax, GELU activation, and Layer Normalization are handled by digital CMOS logic. These functions are more efficiently implemented in the digital domain, where high-precision control is more easily maintained.

2. MGNet: Region-Aware Pruning

One of the primary inefficiencies in standard ViTs is the uniform processing of all image patches, regardless of their informational content. Opto-ViT introduces MGNet (Mask Generation Network), a lightweight front-end network that identifies Regions of Interest (RoI) before the main ViT processing begins.

MGNet generates a binary mask that identifies "background" or irrelevant patches. Because ViT patches are processed independently, the accelerator can simply skip the computation for masked patches. This results in linear savings in both energy and time across every subsequent layer of the transformer model.

3. Pipelined Matrix Decomposition

Optical components often suffer from "tuning delay"—the time required to reconfigure optical weights. To mitigate this, the authors mathematically refactored the self-attention mechanism. By decomposing the attention formula, the system can pre-load weights and pipeline the flow of data. This allows the $Q \cdot K^T$ multiplication to proceed without waiting for the full calculation of the Key ($K$) matrix, eliminating idle time in the optical core.

4. Hardware-Aware Optimization

To ensure robustness against the physical limitations of optical hardware (such as signal noise and limited precision), the model is trained using 8-bit Quantization-Aware Training (QAT). This ensures that the neural network maintains high accuracy even when operating within the 8-bit constraints of the photonic components.

Experimental Results

The performance of Opto-ViT was evaluated using standard benchmarks, including CIFAR-10, Tiny-ImageNet, and the COCO dataset for object detection.

  • Energy Efficiency: Opto-ViT achieved a peak efficiency of 100.4 KFPS/W. For comparison, this is orders of magnitude higher than high-end FPGAs (1.42 KFPS/W) and GPUs (0.86 KFPS/W).
  • Power Savings: The RoI-aware pruning via MGNet enabled up to an 84% reduction in energy consumption by eliminating redundant computations on background patches.
  • Accuracy Preservation: Despite aggressive 8-bit quantization and patch pruning, the system maintained high performance, with less than 1.6% accuracy loss compared to full-precision, non-pruned models.
  • Throughput: The optical core's ability to operate at speeds approaching 100 GHz allows for near-instantaneous processing of high-resolution visual data.

Discussion and Future Work

While Opto-ViT represents a significant leap in edge AI acceleration, some challenges remain. The Analog-to-Digital and Digital-to-Analog converters (ADC/DAC) currently account for approximately 55% of the total energy consumption. This "conversion tax" is a primary target for future optimization. Additionally, while MGNet is highly effective, its performance is sensitive to the quality of the initial mask; future iterations may explore self-supervised mask generation to improve pruning accuracy on complex datasets like Tiny-ImageNet.

Conclusion

Opto-ViT demonstrates that the integration of silicon photonics and region-aware algorithmic pruning can overcome the traditional bottlenecks of Vision Transformers. By architecting a system that processes data near the sensor and skips redundant information, Opto-ViT provides a viable path for deploying state-of-the-art vision models on power-constrained edge platforms.

Summary updated

BibTeX
@inproceedings{morsali2025opto,
  author = {Mehrdad Morsali and Chengwei Zhou and Deniz Najafi and Sreetama Sarkar and Pietro Mercati and Navid Khoshavi and Peter A. Beerel and Mahdi Nikdast and Gourav Datta and Shaahin Angizi},
  title = {{Opto-ViT: Architecting a Near-Sensor Region of Interest-Aware Vision Transformer Accelerator with Silicon Photonics}},
  booktitle = {ICCAD},
  year = {2025},
  doi = {10.1109/iccad66269.2025.11240696},
}
Figure