A Brain-Computer Interface (BCI) marries the brain to Artificial Intelligence (AI), using signals recorded from the brain to enable communication or to control a neuroprosthesis. This technology is now being widely used, however, there is vast room for improvement with key biological and engineering problems remaining to be resolved. These hurdles include low-quality recordings by home users, low translation speed, rudimentary accuracy of translation and adapting applications to the needs of the user. Now, a study from researchers at Columbia University develops a system capable of translating thought into intelligible, recognizable speech. The team states this breakthrough harnesses the power of speech synthesizers and artificial intelligence and could lead to new ways for computers to communicate directly with the brain. The opensource study is published in the journal Scientific Reports.
Previous studies show when people speak or imagine speaking, distinguishable patterns of activity appear in their brains. A distinct pattern of signals also emerges when listening to someone speak, or when a person imagines listening. Reconstructing speech from the human auditory cortex creates the possibility of a speech-based neuroprosthetic with the ability to establish direct communication with the brain. However, the low quality of reconstructed speech has severely limited the utility of this method for BCI applications. The current study combines recent advances in deep-learning with the latest innovations in speech synthesis to reconstruct closed-set intelligible speech from the human auditory cortex.
The current study utilizes a vocoder, a computer algorithm used by Amazon Echo and Apple Siri, to synthesize speech after being trained on recordings of people talking. Epilepsy patients, already undergoing brain surgery, were asked to listen to sentences and numbers spoken by different people, while their patterns of brain activity were recorded via invasive electrocorticography to train the vocoder. Results show the sound produced by the vocoder in response to the patient’s brain signals was analyzed and cleaned up by virtual neural networks, AI biomimicking the structure of neurons in the biological brain. Data findings show the output from this BCI is a robotic-sounding voice reciting an accurate sequence of numbers.
To test the accuracy of the recording, the group asked the participants to listen to the recording and report what they had heard. Results show the patients could understand and repeat the sounds approximately 75% of the time. The lab states the sensitive vocoder and virtual neural networks represented the sounds the patients had originally listened to with surprising accuracy. They go on to add by monitoring someone’s brain activity, their technology can reconstruct the words a person hears with unprecedented clarity.
The team surmises they have developed a BCI able to translate brain signals directly into speech. For the future, the researchers state they now plan to test more complicated words and sentences in the hope their system could one day be used as part of an implant translating the wearer’s thoughts directly into words.
Source: Columbia Engineering
Get Healthinnovations delivered to your inbox:
The Law of Sync and Think—— entropy reaches to the maximum and a pure spontaneous process.
Regarding the basic mechanism of synchronization systems in nature and neural communications I have some thoughts to share.
In the case of metronomes, a mechanical force makes two or more otherwise not synchronized metronomes perfectly synchronized. I am trying to figure out if other natural synchronized systems share the same mechanism. In the case of synchronized fireflies, photon is the major player; and in both the cases of synchronized schools of fishes and birds, the radiation of electromagnetic fields around the body (fishes, a body length apart; birds, 3-4 body length apart) is the main player. However the photons reflected by all the bodies’ scales and feathers could be involved. (https://www.ted.com/talks/steven_strogatz_on_sync)
Because photons move in light speed and electromagnetic waves are also a kind of light, both would have momentum (p) if Einstein’s theory of relativity is applied. Then, the “mechanical force” would be established: just like a light speed “rigid network”, every individual is tightly linked to each other timely and periodically. Although at the biological terminals, like the optic nerve and fish latera line organ, may have a brief delay, the whole picture of these synchronization systems would be a light speed rigid momentum network. In this picture, a radiation from one individual is also the stimulation to all others and these two functions occur at the exact same time. This mechanism also works with synchronized metronomes if we go back to the original, simplest non-life synchronization system.
The law of synchronization is: in any synchronization system, all the individuals are connected to each other rigidly (or in a light speed momentum network), energy radiated from one individual will be propagated to and conserved in all other individuals in light speed. Energy radiated from all individuals will be fed back to each one individual at exactly the same time. Energy states of all individuals tend to be even up; entropy increase tends to be maximum when sync is established; one’s energy output is another’s energy input. The system tends to be energy conservatively beneficial and stable. Compare the randomly oscillating schools, synchronization has the lowest energy state (or the lowest energy cost), that is why all the synchronization systems are spontaneous processes and have universal appeal. Without extra energy input from outside of the system, the synchronization will not be broken.
As for how the “school” of neurons in human brain behave like a light speed rigid network and concentrate on a task or “focus” on a neuron disorder, I have the following concern:
In the University of Tokyo, I made such an observation: the traveling electrical field mediated transmission of action potentials between excitable cells with the cell-cell distance more than 10 mm (an anatomically astronomical distance in cortex). When longitudinal cells are arranged in parallel separately, the action potential generated from one cell can “jump” to other cells and cause all the cells to fire action potentials in concert. If two cells fire action potentials spontaneously and have their own rhythm, they tend to “learn” from each other, adjust their own pace, eventually lock their phases, and “remember” this common rhythm for a long while.
For traveling electric field mediated cell-to-cell communication, we need the following conditions: 1) cells must be very sensitive; 2) cells must have certain length and arranged in parallel; 3) medium between cells must be relatively insulated; 4) electric field must be traveling to produce potential difference on the longitudinal cell surface. (see the paper in attachment http://gallica.bnf.fr/ark:/12148/bpt6k56752048/f671.image).
Unlike synaptic neuronal network, which is a physiological transmission with the velocity of 0.2-120 m/s (synaptic delay period is not included), traveling electrical field mediated transmission is a physical process with the velocity of light speed because an electromagnetic wave is also a kind of light. (See the power point in attachment: traveling electrical field mediated “learning, remembering and forgetting” process). Here again, the process obeys the law of synchronization. Without extra energy input, the things that are remembered will not be easily lost. In other words, forgetting is difficult. We could tell this human nature even from Greek mythology: Lethe, also known as the Ameles potamos (river of unmindfulness). Lethe was also the name of the Greek spirit of forgetfulness and oblivion.
In a cortical circuit, the synaptic elements provide delicate and precise connections; while the other elements, traveling electrical field, may provide transient, rapid, flexible rather than fixed connections to synchronize rhythmic action potentials fired from axons which are arranged in parallel and are well insulated by dielectric media. Whether exploring this invisible “tele” bridge linked synchronization or harmony in discrete regions of the brain, would bring exotic fresh air in the fields of neurological disorder, perception, recognition, learning and memory? I believe it is time for us to address “how the brain thinks”—that would fill the gap between the brain and the mind.
The law of thinking is: The neural action potentials in human brain circuits produce clusters of traveling electrical fields. The traveling electrical field clusters with similar frequency tend to be synchronized. Integration, imagination, remembering, creation, etc. needs some effort sometimes, however, if these processes are simply synchronizations between different regions of the brain, the energy conservatively beneficial property of sync really helps human beings perform these mental activities with great ease, great appeal and great pleasure. And this law explains why most of the memories are lifetime long，why civilization, art, science, culture are popping up and going forward spontaneously.
Algae giant cells I was using can be excited, just like squids’ giant nerve, neuroscientists did pioneering electrical physiology with them. This plant cell is easy to reach its super excited state. I was in Woods Hole Marine Lab to see how neuroscientists dissected squids’ giant nerves, they may use about 3 hours to do it and then put the nerve in incubator over night before doing experiments. The physiological high sensitivity was long gone after that process. So, there is no way for them to show light speed transmission. As for real mammal nerve cells, they are too small to arrange, again they usually are far from the perfect physiological conditions for us to do experiment. And culture media are not insulators.
This beautiful and delicate physics mechanism of light speed transmission began to exist in a 1-2 billion years old plant. Since there was no oxygen on earth 3 billion years ago, plants are all living things’ ancestors. Plants turned earth green, only then did intelligent life evolve. A provocative remark: “Would have been stupid of God not to use light speed in human designs.”
I was working on synaptic transmission for the last 20 years. Glutamate receptors for instance, are found in both humans and crayfishes. Human’s receptor is not any smarter than that of the crayfish. If there is not a super highway (light speed) above the traditional networks, I just cannot imagine how people can be an intelligent life-form. We have been entangled in synaptic networks for so long, and so deep. It’s time for us to have a brand new idea to deal with consciousness and intellect.
The existence of the Myelin System (dielectric or insulation) on the surface of neurons makes perfect sense for a traveling electrical field to transmit signals across long distances between separated cells. This is a physics process. The speed of an electrical field is light speed. Anything that moves in light speed must have quantum properties: wave-particle duality. This neat and simple mechanism of light speed transmission has existed on earth even before the Cambrian explosion, why not evolution should give the best to human beings.
Every intelligent person can tell what a fresh idea striking in the brain is. I understand that the traveling electrical field between neurons in our brain is very weak, and the long-distance light speed communication does not always happen. However, when it is happening, say, when we are thinking, when we are imagining, when we have a great idea, while exactly at this moment we are not in a neuroscientist’s experimental Petri dish and they just could not record any of them.
Now I have another thought: when we grow old, our myelinated neurons become leaky and will no longer insulate and will lose its dielectric property, then “telecommunication “ will be stopped, and our thinking will be stopped, that is basically the case of Alzheimer’s disease. I hope pharmaceutical companies will get this idea and save millions of old people by producing a drug that can rebuild one’s dielectric system. We should understand why our lives are so sharply different from other animals and improve our lives right away. Quantum is ruling our thinking. As for circuits, that is for lower animals and survival functions of humans, not for thinking, imagining and creating.
What I mean by quantum is：The long-time puzzle between brain and mind might be a light speed telecommunication between remote cells that are arranged in parallel. I call this transmission “quantum shortcut”, which might have occurred in the brain 30000 years ago when our ancestors drew the first cave painting. A predicted learning, remembering and forgetting process based on the experiment of “telecommunication” between two separated cells: learning is epiphany; remembering is lasting; forgetting is ambiguousness. Ask yourself whether these are true or not? Science could be very simple and neat. If light speed communications between remote cells arranged in a parallel pattern really is involved in the human thinking process, it would be very narrow minded to play with synapse which evolved from squid and crayfish hoping to find a magic thinking molecule. Neuroscience labs are centuries far behind. Waiting for another 500 years without knowing how our own brain thinks will simply be no fun.
If this thinking temporarily is not recognized by the neuroscience field, we benefit ourselves first: we all love parallel and symmetric things because our brain feels more comfortable and feels easier to remember. I predict that all kinds of(Hz) synchronization valso happens between hemispheres (as well as between cerebral sulcus and cerebral convolution) of our brain in the speed of light to stock various information and to generate creative, imaginative ideas. If we are right-handed (I would like to point out some of left handed characters: Leonardo di Ser Piero da Vinci, Raffaello Sanzio, Michelangelo di Lodovico Buonarroti Simoni, Sir Isaac Newton and Albert Einstein) try hard to use our left hands more and try to do some exercises in front of a mirror. Then, our brain would have more balance and more efficient synchronization or coherence, we will be able to remember things better, become cleverer than normal people, and sure enough, we will have better daily lives than all others (I am not joking).