Collaborative Research: SHF: Small: Quasi Weightless Neural Networks for Energy-Efficient Machine Learning on the Edge

Deep Neural Networks (DNNs) have recently enabled revolutionary advances in a wide variety of tasks, however these deep networks demand large amounts of memory and computation resources. Such demands can be highly difficult (or even impractical) for systems on the edge. Although DNNs are very accurate, the energy consumed by DNNs is orders of magnitude higher than biological neural activities for similar tasks. It is important to reduce the computational and energy demands of machine learning hardware so that inferencing on the edge can become a low-cost, low-energy task. Weightless Neural Networks (WNNs) represent a distinct class of neural models which derive inspiration from the processing of input signals by the dendritic trees of biological neurons. WNNs do not use weights or multiply-add operations to determine their responses. Instead, they rely on value lookups implemented using look-up tables. This project explores small models that are more energy efficient compared to multiplication-and-addition-based deep learning models. WNNs are very promising from the perspective of energy-efficiency, and low latency, and our effort is directed at enabling a myriad of ultra-low energy edge applications otherwise impossible. This project explores low-energy machine learning hardware which combine the benefits of traditional DNNs and the computation-less weightless neural networks. Techniques used include (1) utilizing multi-layer networks and hierarchical networks to create novel weightless neural network architectures, (2) devising novel training algorithms for WNNs utilizing multi-shot training with feedback (3) exploring quasi-weightless neural networks using emerging novel memory technologies, and (4) designing systems for energy-efficient edge intelligence. The collaborative project between the University of Texas and Stanford University innovates across multiple layers of the system stack, including architecture and circuit layers. The collaborative activity between the University of Texas and Stanford involves many underrepresented communities from a STEM perspective, including minority and women, undergrads, and first-generation college students. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.