CMOS+X: Retinomorphic Infrared Imager with Sparsity-adaptive Machine-Learning Accelerator

Real-time signal analysis plays a critical role in applications such as autonomous navigation and robotic guidance. However, conventional imaging systems are hindered by delays and high energy costs associated with transferring large amounts of data to centralized processors. To overcome those shortcomings, this project will embed computational elementals directly in the sensor arrays to locally run machine-learning algorithms that would provide capabilities for object recognition and motion tracking in real time and at low power. Such local in-sensor computing approach would offer significant reductions in energy consumption and delays by avoiding the data transfer bottlenecks. This project will co-optimize the designs of organic infrared sensors and silicon circuits, taking inspiration from the biological retina, which is highly sensitive to dynamic changes and well-suited for motion analysis. The proposed prototype is expected to reduce energy consumption by up to 100 times compared to conventional architectures, thereby paving the way for low-power machine learning. The integration research will contribute to advancing semiconductor manufacturing technologies within the United States and support workforce training. The research team will also engage in outreach activities to promote awareness of various career paths in science and engineering and the rewards of engineering careers. The goal of this research project is to integrate retinomorphic infrared sensors and silicon circuits in order to create a prototype imaging system with machine-learning capabilities for motion analysis. The design strategy assigns complementary roles to the organic sensors and silicon circuits: the retinomorphic sensors will generate highly sparse, feature-extracted data in both temporal and spatial domains, while the circuitries will use the sparsity to boost the overall system performance and power savings. The first research objective focuses on enhancing the sensor’s signal gain and adjusting the time constant to deliver streamlined data into the silicon processor. The second objective aims to optimize sparsity-adaptive architectures that can handle a wide range of sparsity levels and implement the circuit designs using 65 nm technology. The third objective involves establishing the processing workflow to integrate organic sensor arrays onto silicon chips and evaluating the functionalities of the imager by measuring the success rate of object tracking and classification. The resulting prototype will offer valuable insights into design guidelines for effectively balancing energy use, noise and variation tolerance, and latency in smart infrared imaging systems. The proposed imager, equipped with sophisticated yet energy-efficient machine learning capabilities, will have broad applicability across various fields including navigation, biomedical imaging, security, and machine vision applications. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.