In-Context Learning for Sim2Real Reconstructive Spectroscopy: Bridging Modern Machine Learning and Hardware-Software Co-Design

Optical spectroscopy plays a crucial role across various scientific fields, from chemical process analysis to material identification and fluorescence detection. Driven by the demand for portable and field-deployable tools, miniaturizing spectroscopic systems onto chip-scale platforms has become a major research focus. This project leverages cutting-edge machine learning techniques for spectral reconstruction to develop a compact, on-chip spectrometer supporting ultraviolet-visible fluorescence, chemi-/electro-luminescence, and a broad range of optical sensing applications. To overcome the major challenges of limited labeled experimental data and inherent measurement noise for training machine learning models, we will pursue two synergistic strategies: (i) training modern machine learning models on simulated data calibrated to device-specific characteristics to bridge the simulation-to-real gap via in-context learning, and (ii) co-designing the spectrometer and learning framework to jointly optimize hardware and software for accurate, miniaturized spectral analysis. By pioneering training methodologies that utilize realistic simulations, we will demonstrate the efficacy of our on-chip spectrometer across a diverse range of scientific applications. Technically, this project aims to develop an ultra-compact, chip-scale spectrometer with high reconstruction accuracy through the integrated co-design of hardware and modern machine learning algorithms. Our proposed approach will minimize system size while enabling fast, robust, and accurate spectral reconstruction. The project has two primary objectives: (i) development of device-informed Sim2Real in-context learning methods: to address data scarcity and corruption challenges in real-world datasets, we will develop advanced Sim2Real in-context learning techniques. These methods will leverage realistic simulated data—tailored to device characteristics—to train models that generalize effectively to real-world conditions. In particular, we will harness the emerging in-context learning capabilities of large language models to bridge the domain gap between simulated and real data, enhancing model robustness and reliability in deployment. (ii) Co-design of machine learning framework and compact spectrometer: we will jointly design the machine learning algorithms and the chip-scale spectrometer to maximize efficiency and reconstruction performance. This hardware-software co-design will enable compact, low-power deployment with fast and accurate inference. We will validate the proposed spectrometer system across a range of scientific applications, including use cases in healthcare and chemistry. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.