Research press release


Nature Neuroscience

Neuroscience: Translating brain activity to text

神経活動を解読し、高い精度で文章に翻訳する機械翻訳アルゴリズムについて報告する論文が、Nature Neuroscience に掲載される。


今回、Joseph Makinたちの研究チームは、機械翻訳の最近の進歩を調べた上で、機械翻訳の手法を用いて、回帰型ニューラルネットワークの訓練を行い、神経信号を文章に直接マッピングした。今回の研究では、発作モニタリングのために頭蓋内に電極を設置された4人の参加者による実験が行われ、参加者に文章を音読させて、その際の神経活動が電極によって記録された。この神経活動の記録は、回帰型ニューラルネットワークに入力され、発話の反復的特徴(母音、子音、口の部分への命令など)に関連している可能性の高い、規則的に発生する神経特徴の表現が生成された。次に、別の回帰型ニューラルネットワークが、この表現を単語ごとに解読して、文章を作成した。また、Makinたちは、発話の解読に大きく寄与する脳領域が、発話の産出と知覚にも関与していることを発見した。


A machine-translation algorithm that can decode neural activity and translate it into sentences with high levels of accuracy is reported in a paper in Nature Neuroscience.

Brain–machine interfaces have had limited success in decoding speech from neural activity, and their accuracy remains far below that of natural speech. Previous brain–machine interfaces could only decode fragments of spoken words or less than 40% of words in spoken phrases.

Joseph Makin and colleagues examined recent advances in machine translation and used these approaches to train recurrent neural networks to map neural signals directly to sentences. Four participants, who had been implanted with intracranial electrodes for seizure monitoring, read sentences aloud while the electrodes recorded their neural activity. The neural activity was fed into a recurrent neural network, which created a representation of regularly occurring neural features that are likely to be related to repeated features of speech such as vowels, consonants or commands to parts of the mouth. Another recurrent neural network then decoded this representation word by word to form a sentence. The authors also found that brain regions that strongly contributed to speech decoding were also involved in speech production and speech perception.

This machine-translation approach decoded spoken sentences from one patient’s neural activity with an error rate similar to that of professional-level speech transcription. Additionally, when the recurrent networks were pre-trained on neural activity and speech from one person before training on another participant, decoding results improved, suggesting that this approach may be transferable across people. However, further research is needed to investigate more fully the capabilities of this system, and to expand decoding beyond the circumscribed language of the study.

doi: 10.1038/s41593-020-0608-8


メールマガジンリストの「Nature 関連誌今週のハイライト」にチェックをいれていただきますと、毎週各ジャーナルからの最新の「注目のハイライト」をまとめて皆様にお届けいたします。