Event-driven spatiotemporal processing-in-sensor with phase change memory-based optical acceleration

Abstract

This work introduces a novel hybrid electronic-optical processing-in-sensor architecture designed for low-cost, real-time frame processing at the edge. The proposed system enables event detection and integrates a TinyLSTM-based temporal inference model to analyze multiple frames in real time, extracting meaningful spatiotemporal features that trigger an address actuator for region-of-interest selection. By selectively reading out only relevant pixel regions, the architecture significantly reduces data transfer overhead and power consumption. Additionally, it harnesses the efficiency of silicon photonic (SiPh) devices to enable adaptive frame compression techniques and perform convolution operations through intrinsic, conversion-free multiply-accumulate computations. Device-to-architecture simulation results demonstrate 11.2x improvement in performance compared to the state-of-the-art SiPh accelerator achieving 37 KFPS/W. This marks a significant advancement in processing-in-sensor technology, enhancing both computational efficiency and energy savings for edge AI applications.