Facebook's wearable BCI: A future where people will type directly from brains

In 2017, Facebook announced at its F8 conference that it was working on a system that will let people type with their brains without even saying a word. The tech giant then outlined its goal to build non-invasive, wearable sensors that will type five times faster than one can type on a smartphone today.

Well, it seems Facebook has marked its first step towards delivering on its promise. More than two years after the announcement, the company has released its first-ever report on the progress it has achieved so far to develop a fully non-invasive brain-computer interface (BCI) as a potential input solution for augmented reality (AR) glasses. The findings published in the Nature Communication journal provide insight into how the speech was decoded in real-time from the brain signals.

Today we're sharing an update on our work to build a non-invasive wearable device that lets people type just by imagining what they want to say. Our progress shows real potential in how future inputs and interactions with AR glasses could one day look. https://t.co/ilk192GwAR

— Boz (@boztank) July 30, 2019

Supported by the Facebook Reality Labs (FRL), a team of researchers at the University of California, San Francisco (UCSF) worked to develop a way to decode the spoken responses of participants (epilepsy patients) to a set of standard questions based solely on their brain activity, in real-time. Led by David Moses, a postdoctoral scholar in Chang's lab at UCSF, the researchers developed a set of machine learning algorithms equipped with refined phonological speech models, which were capable of learning to decode specific speech sounds from participants' brain activity.

Brain data was recorded while volunteers listened to a set of nine simple questions using already-implanted ECoG electrodes. The team decoded produced and perceived utterances with accuracy rates as high as 61 percent and 76 percent, respectively, much higher than current rates that stand at 7 percent and 20 percent.

"Real-time processing of brain activity has been used to decode simple speech sounds, but this is the first time this approach has been used to identify spoken words and phrases," Moses said. "It's important to keep in mind that we achieved this using a very limited vocabulary, but in future studies, we hope to increase the flexibility as well as the accuracy of what we can translate from brain activity."

What's ahead?

The study is a stepping stone for the researchers who hope to reach a real-time decoding speed of 100 words per minute with a 1,000-word vocabulary and a word error rate of less than 17 percent. For Facebook, it's a tantalizing vision that will require an enterprising spirit, hefty amounts of determination, and an open mind, even if it takes a decade.

The promise of AR lies in its ability to seamlessly connect people to the world that surrounds them — and to each other. Rather than looking down at a phone screen or breaking out a laptop, we can maintain eye contact and retrieve useful information and context without ever missing a beat, Facebook said in a blog post

While BCI as a potential input solution for AR is clearly still a long way off, the breakthrough, if achieved, will not only dramatically improve the lives of people suffering from various forms of speech impairment or speech loss but will also completely transform the way we interact via digital devices.