Researchers translate brain signals directly into speech.

A Brain-Computer Interface (BCI) marries the brain to Artificial Intelligence (AI), using signals recorded from the brain to enable communication or to control a neuroprosthesis. This technology is now being widely used, however, there is vast room for improvement with key biological and engineering problems remaining to be resolved. These hurdles include low-quality recordings by home users, low translation speed, rudimentary accuracy of translation and adapting applications to the needs of the user. Now, a study from researchers at Columbia University develops a system capable of translating thought into intelligible, recognizable speech. The team states this breakthrough harnesses the power of speech synthesizers and artificial intelligence and could lead to new ways for computers to communicate directly with the brain. The opensource study is published in the journal Scientific Reports.

Previous studies show when people speak or imagine speaking, distinguishable patterns of activity appear in their brains. A distinct pattern of signals also emerges when listening to someone speak, or when a person imagines listening. Reconstructing speech from the human auditory cortex creates the possibility of a speech-based neuroprosthetic with the ability to establish direct communication with the brain. However, the low quality of reconstructed speech has severely limited the utility of this method for BCI applications. The current study combines recent advances in deep-learning with the latest innovations in speech synthesis to reconstruct closed-set intelligible speech from the human auditory cortex.

The current study utilizes a vocoder, a computer algorithm used by Amazon Echo and Apple Siri, to synthesize speech after being trained on recordings of people talking. Epilepsy patients, already undergoing brain surgery, were asked to listen to sentences and numbers spoken by different people, while their patterns of brain activity were recorded via invasive electrocorticography to train the vocoder. Results show the sound produced by the vocoder in response to the patient’s brain signals was analyzed and cleaned up by virtual neural networks, AI biomimicking the structure of neurons in the biological brain. Data findings show the output from this BCI is a robotic-sounding voice reciting an accurate sequence of numbers.

To test the accuracy of the recording, the group asked the participants to listen to the recording and report what they had heard. Results show the patients could understand and repeat the sounds approximately 75% of the time. The lab states the sensitive vocoder and virtual neural networks represented the sounds the patients had originally listened to with surprising accuracy. They go on to add by monitoring someone’s brain activity, their technology can reconstruct the words a person hears with unprecedented clarity. 

The team surmises they have developed a BCI able to translate brain signals directly into speech. For the future, the researchers state they now plan to test more complicated words and sentences in the hope their system could one day be used as part of an implant translating the wearer’s thoughts directly into words. 

Source: Columbia Engineering

Get Healthinnovations delivered to your inbox: