Research highlight

Neuroscience: Translating brain activity to text

Nature Neuroscience

March 31, 2020

A machine-translation algorithm that can decode neural activity and translate it into sentences with high levels of accuracy is reported in a paper in Nature Neuroscience.

Brain–machine interfaces have had limited success in decoding speech from neural activity, and their accuracy remains far below that of natural speech. Previous brain–machine interfaces could only decode fragments of spoken words or less than 40% of words in spoken phrases.

Joseph Makin and colleagues examined recent advances in machine translation and used these approaches to train recurrent neural networks to map neural signals directly to sentences. Four participants, who had been implanted with intracranial electrodes for seizure monitoring, read sentences aloud while the electrodes recorded their neural activity. The neural activity was fed into a recurrent neural network, which created a representation of regularly occurring neural features that are likely to be related to repeated features of speech such as vowels, consonants or commands to parts of the mouth. Another recurrent neural network then decoded this representation word by word to form a sentence. The authors also found that brain regions that strongly contributed to speech decoding were also involved in speech production and speech perception.

This machine-translation approach decoded spoken sentences from one patient’s neural activity with an error rate similar to that of professional-level speech transcription. Additionally, when the recurrent networks were pre-trained on neural activity and speech from one person before training on another participant, decoding results improved, suggesting that this approach may be transferable across people. However, further research is needed to investigate more fully the capabilities of this system, and to expand decoding beyond the circumscribed language of the study.

doi: 10.1038/s41593-020-0608-8

Return to research highlights

PrivacyMark System