Collaborative Research: NewSpectrum: Toward Untethered Extended Reality Through Wireless Sensing and Communications Co-design

Extended reality (XR) offers users immersive experience in virtual worlds, and enables a broad range of applications (i.e., training, gaming, and medical imaging). There has been an increasing interest on the study of the deployment of XR services over next era of wireless networks (nextE), so as to provide seamless wireless connectivity for XR users to eliminate the wired connection constraints thus enabling future wireless devices to use VR services. However, the few prior studies have two major limitations: 1) They are mainly focused on network optimization for XR data transmission and are lacking in novel user behavior sensing methods, 2) Their XR sensing methods mostly rely on statically installed sensors or cameras, which also restrict the operation range of users and suffer from user movement and blockage, 3) they are restricted to either a single XR system, or multiple XR systems where each XR system consists of only one user and hence cannot be applied for multi-user XR systems. To address the aforementioned challenges, a holistic wireless XR framework is developed, which utilizes mmWave for joint XR user movement detection and XR data transmission while satisfying the joint communication, computing, sensing, and XR service requirements. If successful, this project will enable highly efficient and robust wireless enabled XR networks and applications, with significantly enhanced accuracy, resilience, and user experience. The project integrates the research insights into new modules for communication and network related courses and hosts outreach activities with the vision of advancing the participation of underrepresented minorities in STEM fields. The untethered XR project presents a cutting-edge solution for eliminating XR wired connections and limitations of XR user activity space by utilizing mmWave, machine learning, edge computing, and joint sensing and communications technologies to truly unleashing the high potential of XR via: 1) developing novel mmWave-based sensing methods which exploit complex valued channel state information and radio map information to detect the full-body movements of multiple XR users; 2) designing a novel collaborative reinforcement learning (RL) framework to produce a low-complexity and reliable collaborative learning process that enables distributed XR access points (APs) to jointly optimize XR sensing and data transmission in order to improve the quality-of-experience of XR users; 3) building an open-source software platform and hardware testbed to validate the wireless XR solutions. This project provides a rich environment and virtualized platform that facilitate educating and training students at multiple levels. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.