Collaborative Research: SaTC: CORE: Small: Towards the Security of Immersive Multimedia Systems

Immersive multimedia systems, such as volumetric video (VV), virtual reality (VR), and augmented reality (AR) systems, have become popular recently and attracted a broad range of user applications. While presenting a fully interactive and engaging user experience, the immersive multimedia poses unique security challenges associated with its fine-grained three-dimensional (3D) content. In particular, the 3D content (e.g., 3D human face) may be exploited by adversaries to issue biometric-based attacks (e.g., spoofing attacks against face authentication), causing significant security and societal concerns. This project aims to investigate and address such security challenges in volumetric videos by developing (1) an advanced face spoofing attack leveraging real-time environment lighting estimation and generation; and (2) an effective countermeasure injecting protective perturbations to the 3D content, which invalidates the spoofing attack while maintaining the original quality of experience. As a pilot study on the security implications of immersive multimedia, the proposed project will benefit the larger-scale adoption of VR and AR technologies in many fields of studies involving sensitive data and computations, such as teleconferencing, remote education, and healthcare. The project consists of three research thrusts. First, it demonstrates that the state-of-the-art face authentication systems, even equipped with advanced liveness detection mechanisms, can be effectively compromised by the 3D face models leveraging the proposed lighting estimation and generation approach. Second, it proposes a real-time perturbation generation mechanism to obfuscate and protect the sensitive 3D content from being exploited for spoofing attacks. Third, it develops an evaluation framework in both lab and community settings to verify the effectiveness of the proposed security approaches and expand the scope of the project to the broader communities. Overall, the project aims to fill the critical security gap in the popular immersive multimedia systems and pave the way toward larger-scale and user-facing deployments. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.