Technology-driven opera in Omaha empowers two silent performers to express themselves artistically
Opera in Omaha, "Sensorium Ex," features disabled performers front and center, including two nonverbal individuals. Jeff Lunden reports that the production, after five years of development, has employed AI technology to give them voice.
Jakob Jordan, now 23, is a nonverbal cast member who uses technology developed by the NYU Ability Lab. This technology makes his voice more expressive, distinct from off-the-shelf speech synthesizers.
"When I first heard the sound in my voice come to life, a new realization was born - dream the bigger dreams, you know, the ones you dismiss and hide away because they seem impossible," said Jordan.
Luke DuBois, co-director of the NYU Ability Project, and a team of technology professors and graduate students developed an AI interface to enable Jordan and another nonverbal castmate to speak in the opera.
"Even though it might still sound a little stilted, a little synthetic, it's not perfect, it's them, right? It sounds like them," DuBois shared.
The collaboration between composer Paola Prestini, librettist Brenda Shaughnessy (who has a nonverbal son), and the technological team resulted from Prestini's desire to incorporate disabled performers in her work. She approached DuBois, who agreed to create a device that would allow this nonverbal character to speak. Prestini intended this device to be open-source so that it could benefit the broader nonverbal community.
The team spent two years surveying the nonverbal community, with a mantra of "nothing about us without us," before developing sophisticated technology that combines sampling of the sounds nonverbal actors make, AI to create an expressive voice, and devices to control onstage expressions.
"Moving my hand over one of them will trigger it, 'I am me,'" explained DuBois, demonstrating how Jakob Jordan can manipulate the prerecorded sound of his voice using these devices.
"They took our sounds and incorporated them into our performances, so each showing will be a unique experience that reflects us. Thank you for hearing our voices," Jordan said.
The "Sensorium Ex" opera will premiere at the Common Sense Festival in Omaha, focusing on arts, science, and disability acceptance. The team hopes that the technology can be further developed to provide voice to individuals in the Augmentative and Alternative Communication (AAC) community who cannot speak.
- The collaboration between the team at NYU Ability Project, led by Luke DuBois, has developed an AI interface to enable nonverbal performers in the opera "Sensorium Ex" to speak, using technology originally developed for Jakob Jordan.
- The opera "Sensorium Ex" features the voices of nonverbal performers, such as Jakob Jordan, using technology that makes their voices more expressive and distinct from off-the-shelf speech synthesizers.
- Composer Paola Prestini and librettist Brenda Shaughnessy, who has a nonverbal son, have incorporated technology into their opera work to give voice to disabled performers, with the aim of making the technology open-source to benefit the broader nonverbal community.
- The "Sensorium Ex" opera, premiering at the Common Sense Festival in Omaha, focuses on arts, science, and disability acceptance, and the team hopes that the technology developed for the opera can be further developed to provide voice to individuals in the Augmentative and Alternative Communication (AAC) community who cannot speak.
