FuSe2 Topic 1: Edge Reinforcement Learning with Algorithm, Architecture and Circuit Co-Design

Reinforcement learning is a powerful machine learning approach for autonomous decision-making. It has found extensive and impressive recent applications in various domains ranging from core algorithms used in Google AlphaGo to its use in ChatGPT. However, most of these advancements require an abundance of computing resources, including both complex hardware and energy. These requirements are easily met in the context of datacenter computing, but they are often invalid for “edge” devices such as smartphones, embedded computing, and IoT (Internet of Things) systems, where resources are extremely limited. This project will develop a cross-level methodology that enables the computations of state-of-the-art reinforcement learning techniques to run on resource-constrained devices, including the inference and training of multiple neural networks. The proposed methodology and resulting computing platform have the potential to extend the success of reinforcement learning to a vast range of application domains, similar to the impact that GPUs (graphics processing units) have had on large language models. These domains include personal computational platforms, biomedical devices, health assistants for people with disabilities, precision agriculture, smart manufacturing, and many others. As a result, the benefits of this project to human society and quality of life could be significant. In addition, the research results will be incorporated into workforce development efforts. The proposed methodology involves innovation and co-design of techniques spanning different levels, including devices, circuits, system architectures and algorithms. The central idea is an innovative integration between a novel approximate computing circuit technique and flash device-based computing, each of which offers at least one order of magnitude improvement in computing efficiency compared to conventional circuit implementations. To address the limited write cycles of flash devices, the algorithmic level approach adopts the meta learning framework, which allows few-shot online adaptation training. Moreover, the use of the approximate computing technique provides fault tolerance and graceful precision degradation. At the architectural level, neural network models and their mapping on hardware will be co-designed for further resource efficiency improvement. Another essential co-design element in this methodology is the co-optimization of hardware parameters and algorithmic parameters. Both types of parameters impact the tradeoff between computational accuracy and cost, but they do so in different ways, requiring joint optimization. The proposed methodology will be used to design an overall system, which will then be validated through a silicon prototype and a robotic control system. The research outcome will significantly advance the knowledge of reinforcement learning computing on resource-constrained edge devices. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.