EAGER: Identifying and Producing Code-Switching in Languages from Spoken, Lexical and Socio-linguistic Features

This Early Grant for Exploratory Research investigates conversations between the vast number of persons in our world who speak multiple languages and who frequently switch back and forth between those languages in what is called “code-switching”. It is important for speech dialogue systems and voice assistants to not only be able to identify when, why, and to what effect code-switching occurs, but also to correctly interpret what is said and to be able to generate similarly code-switched responses when interacting with such users. Advances in speech technology in recent years have resulted in widespread use of voice assistants such as Siri, Google Assistant and Alexa. They enable vast improvement in information access by voice for languages such as English, French, German, Cantonese, Mandarin, and Spanish. However, such access is limited to monolingual speech, which for many multilingual speakers is not the most natural form of speech production. Thus, code-switched speech is rarely understood correctly and is never able to be produced in assistant responses. A major barrier to enabling naturalistic and comfortable communication for these speakers is the lack of speech technology that can not only understand code-switched input but also produce similar human-like output. This project addresses these issues by examining how spoken and written code-switching interacts with other aspects of language communication. It will explore research questions not yet studied in code-switching research including (1) whether speakers entrain, speak more similarly, on pronunciation and other strategies of code-switching in speech; (2) whether there is a quantifiable relationship between code-switching and empathy in speech, where empathy is a speaker's intention to convey that they understand another's problems and want to help address them; (3) whether the presence of named entities, such as names or geographical locations, primes code-switching; (4) which dialogue acts, such as questions or statements or backchannels, tend to be produced most often in code-switched speech; and (5) how speakers produce intonational contours when they code-switch (via choosing their intonation production to match either of the languages they are producing or by being different from both?) Statistical and machine-learning techniques will both be used to address these questions in the context of spoken and lexical-feature-tagged code-switched speech in Standard American English, Spanish, Mandarin Chinese, and Hindi. By identifying new aspects of code-switching, the project will seed further exploration of this phenomenon by the research community. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.