Collaborative Research: FW-HTF-RM: Artificial Intelligence Technology for Future Music Performers

Nearly 300,000 people in the U.S are working in the music field. Recent progress in Artificial Intelligence (AI) has had profound impacts for music creators but has not yet had much impact on the practices of professional music performers. This project will investigate how AI technology could transform the work of future music performers for both individual practice (through developing coaching tools that analyze music performance) and collaborative practice (through developing systems that can substitute for missing performers in a group). The team will assess the tools in terms of two main questions: (1) When can AI technology provide measurable benefits to professional musicians' practice and performance? (2) What factors would affect future musicians' acceptance of AI technology in their work? The initial tools will focus on stringed instruments, but the underlying technologies are likely to be adaptable to a wide variety of performance contexts in the art and entertainment industry. The project team will also explore ways to use the tools to benefit students from disadvantaged groups; the tools may improve learning opportunities for students with limited access to music instruction. The team will recruit student researchers from groups underrepresented in STEM. This project will develop and integrate techniques from computer vision, natural language processing, and audio analysis to create two AI-enabled tools to support string music performers. The first tool, the Evaluator, aims to improve individual practice and performance. It analyzes a musician's sound and compares it to digitized music scores to detect deviations in intonation, rhythm, and dynamics. The Evaluator also analyzes captured video and compares it to a database of sample performers recorded with correct postures, allowing it to recommend better postures, which can both improve musical performance and reduce injury risks. The second tool, the Companion, aims to support common use cases when one or more musicians are missing from a group performance rehearsal. The Companion can play the part of one or several instruments to replace absent musicians, matching tempo, and style of the human musicians through audio analysis of their performance while also responding in real-time to verbal instructions. These tools will be developed and evaluated through a series of user studies, surveys, focus groups, and longitudinal deployments. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.