RI: Medium: From Vision to Dynamics

In the realms of health and sports, detailed analysis of human movement provides guidance for individualized performance assessment, training, and health evaluation. Data collection for biomechanical analysis of human motion is typically performed using expensive, specialized hardware that requires experiments to be carried out in a constrained lab setting. Computer vision algorithms that use video cameras for data collection enable observation in more natural environments with minimal cost, but current vision-based estimates of body pose and motion are not biomechanically accurate and lack a quantitative model of the physical forces underlying human movement. This research effort will train a modern machine learning system to perform imagery-based estimation of biomechanically accurate body pose, motion and force, turning informal computer vision into a scientific measurement tool for human movement and stability research in unconstrained, everyday settings using readily available low-cost video cameras. The results will enable novel applications in personalized training and preventative medicine, and lead to strategies for improving flexibility and stability among the elderly, reducing fall-induced mortality and improving quality of life, serving NSF's mission to promote progress of science to advance the national health. An outreach program covering topics in human and robot stability will inspire K-12 students with demonstrations that combine exercise, augmented reality, and humanoid robots. The project team will collect a unique, multi-modal human motion dataset and use it to train a novel deep learning system that combines multi-task regression with a biomechanical human motion model to estimate, from visual data alone, accurate pose and motion as well as dynamics such as muscle torques and ground reaction forces. The results will fill an existing gap between visual observation of kinematics and the inference of body dynamics, greatly improving the ability of computer vision to make inferences about balance and stability that require dynamics information. Data-driven discovery of discriminative features that are causal or correlated to stability will lead to identifying the most stability-enhancing poses/moves for a movement sequence and new measures for estimating stability of a person in motion. Guided by the scientific research, four applications will be explored: 1) Developing a vision-based stability surveillance system that can quantify and monitor changes in stability; 2) Developing a vision-based personal trainer for teaching stability-enhancing exercises; 3) Analyzing historical sports videos to reveal changes in performance styles and dynamics; and 4) Studying methods for retargeting human movements to a humanoid robot while maintaining stability. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.