Collaborative Research: CNS Core: Medium: Innovating Volumetric Video Streaming with Motion Forecasting, Intelligent Upsampling, and QoE Modeling

Volumetric video is an emerging type of multimedia content. Unlike traditional videos and 360-degree videos that are two-dimensional (2D), every frame in a volumetric video consists of a three-dimensional (3D) scene or object represented by a 3D model such as a point cloud or a polygon mesh. The 3D nature of volumetric video enables viewers to move with six degrees of freedom, leading to a truly immersive viewing experience. However, compared to conventional videos, streaming volumetric videos over the Internet faces unique challenges: a lack of precise motion tracking, extremely high bandwidth consumption, a lack of quality-of-experience (QoE) models, and complex interactions among network, computation, and motion. This project proposes a holistic research agenda addressing the above challenges through four core approaches: multi-dimension, multi-modality motion sensing and prediction; deep-learning-based content upsampling; comprehensive QoE modeling; and resource/motion adaptation. The project aims at demonstrable systems and networking research with a synergy among multimedia systems, networking, wireless sensing, machine learning, computer vision, and graphics. The research team will develop prototypes and evaluate them in real-world settings. Internet video streaming is playing a key role in today's world, especially during the COVID-19 pandemic. As a key enabler of immersive telepresence, volumetric content will become popular in the near future. The techniques developed from this project will significantly reduce network resource usage and improve the user experience for volumetric content delivery over the Internet. This will benefit both producers and consumers of volumetric videos, as well as boost their applications in various domains such as education, telehealth, manufacturing, and entertainment, ultimately leading to a high impact on societies and economies. The project will also offer new education components, and provide a platform for various Broadening Participation in Computing (BPC) activities. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.