From silence to sound
Researchers from UC Berkeley and UCSF have made another breakthrough in restoring speech for people with vocal-tract paralysis. Using AI-based modeling, the researchers developed a method that synthesizes the signals from brain-computer interfaces into audible speech in near-real time, improving upon a previous approach with an 8-second delay.
“Our streaming approach brings the same rapid speech decoding capacity of devices like Alexa and Siri to neuroprostheses,” said Gopala Anumanchipalli, assistant professor of electrical engineering and computer sciences and co-principal investigator with UCSF’s Edward Chang. “Using a similar type of algorithm, we found that we could decode neural data and, for the first time, enable near-synchronous voice streaming.”
The neuroprosthesis works by sampling neural data from the motor cortex, the part of the brain that controls speech production, then uses AI to decode brain function into speech. To collect data to train their algorithm, the researchers first had Ann, their subject, look at a prompt on the screen and then silently attempt to speak that sentence.
“This gave us a mapping between the chunked windows of neural activity that she generates and the target sentence that she’s trying to say, without her needing to vocalize at any point,” said Ph.D. student Kaylo Littlejohn.
Because Ann does not have any residual vocalization, the researchers did not have target audio, or output, to which they could map the neural data. They solved this challenge by using AI to fill in the missing details.
“We used a pretrained text-to-speech model to generate audio and simulate a target,” said Ph.D. student Cheol Jun Cho. “And we also used Ann’s pre-injury voice, so when we decode the output, it sounds more like her.”
Learn more: Brain-to-voice neuroprosthesis restores naturalistic speech; A streaming brain-to-voice neuroprosthesis to restore naturalistic communication (Nature Neuroscience)