SBIR Phase I: Lightweight Learning-based Camera Image Signal Processing (ISP) for Photon-Limited Imaging

The broader impact of this Small Business Innovation (SBIR) Phase I project will result from the ability to operate digital image sensors at lower light levels than is currently possible. The technology is expected to be deployable in any mid- to low-level camera device, with potential applications across all industries that leverage camera technology. Consumer applications that would benefit from improved low-light imaging include dashboard cameras and notebook cameras for videoconferencing; military and national security applications include night vision and autonomous navigation; while the technology will also enable improved diagnostic capabilities in medical procedures such as endoscopies. The technology is expected to have a direct impact on workforce development, and deployment of the solution will drive economics in consumer electronics. This Small Business Innovation Research (SBIR) Phase I project aims to achieve photon-limited image denoising using a lightweight algorithm that has the potential to be implemented on a camera chip. Accomplishing this goal requires several technological breakthroughs, collectively leading to a new image signal processor (ISP) known as a Small and Learnable ISP Module (SLIM). The key to SLIM is to identify the bottlenecks of physics-based ISPs and replace them with customized learning-based modules. Specifically, SLIM consists of five innovations: (i) learning-based frequency demodulation, (ii) guided denoising, (iii) learned feature extraction, (iv) learned indexing, and (v) learned filtering. In Phase 1, the team proposes to optimize SLIM and implement it on a field programmable gate array (FPGA). This includes shrinking the size of the filters and streamlining the indexing scheme to further speed up SLIM, introducing new encoders to improve generalization, and optimizing the memory, communication, and processing through improved programming and real-data evaluation and demonstration. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.