Last week, Synchron announced an integration with OpenAI’s ChatGPT to incorporate prompt-based interactions with their brain-computer interface.
In this application ChatGPT generates inputs by prompting users, whose implanted BCIs enable navigation and selection of responses. As Synchron founder and CEO Dr. Tom Oxley described to Bloomberg, the BCI restores people’s ability to choose how they respond, while AI transforms various inputs into a context-based experience.
As per CNBC, Synchron’s user Mark, a person living with ALS, says the feature has saved him time and energy, and reduced pressure by giving him choices of how to communicate through the platform.
In addition to restoring “faster response times, natural conversational speeds and interaction with texting” today, this integration enables Synchron to bring sound, image and video to their users for multimodal experiences in the months ahead.
I spoke to Oxley about current limits, opportunities and challenges of integrating with large language models (LLM) and what this work means for the BCI field.
BCI as a Harbor of Choice
Synchron’s mission is restoring autonomy to their users. Oxley explained their focus with this integration is restoring the ability to engage with digital environments more naturally for people with severe mobility limitations in their upper limbs.
“Generative AI is taking the inputs from the digital world in, generating prompts, and then we’re on the other side. Our deep learning models are making predictions on what the brain wants to do, so the BCI is the harbor of choice, restoring autonomy in decision making.”
The BCI use case in the demo involves controlling a platform through navigation and selection of prompts displayed onto an iPad App, built with Apple’s iOS Accessibility features. Oxley says that the GPT-4o announced in May, which works across audio, vision, and text in real time, represents a frontier beyond text.
“The reason we did this is because the multimodal ‘4o’ is something different, because it uses inputs from the environment that behave like an extension of the brain of the user. It can see, it can hear, and it can receive text inputs, so when the user goes to start engaging with the prompts, it’s got a real time information flow of everything happening in their environment.”
Powering a Personal Touch
While this integration is currently a feature that can be toggled on or off, Oxley noted “We are now committing substantial resources towards novel interaction methods with generative AI.”
He says the text-based feature is just the beginning, with vision, audio, and eventually video in the wings for additional use cases: “As this grows with different inputs, we’re going to be building interaction methods that are simple, but which unlock access to technology that we need to use our hands for.”
These could range from incorporating conversational context from family members and caregivers to powering calendar integrations and other practical applications, including eventually connecting with other consumer devices. But Oxley says the immediate focus is driven by their users.
“A huge problem we’re hearing from our patient population is that they’re isolated because they can’t engage in natural speed conversations. So the goal here is to keep up with a level of conversation, in dialogue with friends and family and caregivers. The AI bridges the gap to help keep up.”
Another key piece of this integration is a feature Synchron calls emotional categorization, to provide users with a context-based array of choices. In the demo example, a doctor’s question about back pain generates different prompts spanning various positive, neutral, or negative responses.
“When you or I get asked a question or have a conversation, we decide how to respond based on how we’re feeling. For our users, we’ve now also built some proprietary layers on top, including these emotional categorizations for selecting words and verbal prompts.”
Strategic Outlook: Clear View, Long Distance
Oxley believes multimodal LLM represent the future of BCI interactions. But he takes a pragmatist’s view, pointing to several variables that will determine product roadmap, business partnerships, and other commercial strategy in the coming years.
Non-Exclusive
This integration does not represent an exclusive partnership between Synchron and OpenAI. “This is such a fast moving field, so we’re going to remain agnostic to whatever API we use based upon what is in the best interest of our users at any point in time.”
Privacy Paramount
Synchron has repeatedly stated that they do not share any brain data with OpenAI. Given the nature of this technology, I asked whether the platform will remember users over time. “That’s the goal, but it raises a bunch of privacy challenges,” he said.
Speaking to the technical work required to configure encryption between device, local network, and cloud, he added, “We’re working through what that looks like in our first commercial offering. But the product vision is to build that knowledge base of the user locally and protect that. We’re adopting a very, very high bar for privacy.”
Long-term Play
Oxley “honestly [acknowledged] that we debated internally as to whether we should even do this,” citing costs, time, risks, perceptions and unknowns. Ultimately, GPT’s 4o multimodal capabilities provided the impetus to move forward, to better connect users like Mark to digital systems today, and unlock broader applications across a rapidly evolving consumer technology ecosystem of tomorrow.
Reflections and Ripples
As a casual user of various popular LLMs, I admitted to Oxley that when I first saw the press release, I shrugged. Despite knowing full well that off-the-shelf GPT I use differs substantially from this application, I still had trouble seeing the significance. Several neurotech executives have expressed similar scrutiny in private.
But with early stage technology, it’s good to have more questions than answers. As a skeptical optimist, here are some summary reflections and lingering thoughts about what LLM could represent for user applications in commercial neurotech markets.
Burden of Interpretation
Communicating normally is something most of us do without thinking, and thus take for granted: Analyzing a question for intent, gathering broader context based on external cues and internal feelings, and factoring that into a reply, all at conversational pace.
However, what able-bodied folks like me can’t fathom is how this can represent a hurdle or even a barrier for those living with different forms of impairment, including early BCI users. If you’ve struggled to use a remote control to browse Netflix, just imagine if that was what all communication required.
By removing even some burden of interpretation and other friction for users, LLM can substantially improve utility of BCI technology and lead to new use cases. Basic features are a good place to start. I’m excited to see what ideas and features emerge from end users, patient advocacy groups, caregiver circles, registries, and trial participants.
Form Factor
A common category of question I receive is about minimally invasive versus non-invasive neural interfaces. In this case, the question might be “Why is an implanted device even necessary to incorporate LLM?”
Leaving aside various critical distinctions between implanted and non-invasive tools, in this case the question is a good one. Cognixion, Augmental, Synaptrix, and many others are commercializing wearable or non-invasive means to decode user intent into digital signals, in some cases with advantages in upgradability, cost, and safety.
So while Synchron’s integration with OpenAI is fairly high profile, the truth is LLM are modality-agnostic. The upshot: Expect several other neurotech players to adopt similar and different approaches to leverage prompt automation and other LLM-enabled features in the months ahead.
Towards Social Robotics
Thanks to comments by Musk, enough ink has already been spilled on BCI as a portal to transhumanism, a topic Oxley has shrugged off. But a separate area in need of speculation is the role that semi-autonomous AI agents could play in future BCI markets where human privacy and safety can be protected, and LLM reliability ensured.
Beyond responsive prompts, there is tremendous value to unlock through small automations for these users (and others). If personalized agents could become capable and, again, reliable, they could help with checking in, communicating with the medical team, collecting patient-reported outcomes, and possibly handling administrative labor, from managing appointments, medications, paying bills, dealing with insurers or disability offices, initiating interpersonal requests to caregivers, say to charge a device or flag other needs, and so on.
Those examples are all text-based; the possibilities for incorporating audio, cameras, sensors, wearables, other inputs (e.g. BCI-derived neural data for additional passive sensing) could produce applications that arguably go beyond what we can even imagine today. To be clear, all indications suggest this level of reliability with LLM is likely years away. But, today’s prototype from Synchron, with a laser focus on user privacy and simple, value-add features, opens the door for those possibilities to take shape in the future.
Widget vs. Platform
Some think a reckoning is due for the frothy implantable BCI space, driven by the hype of Synchron’s main competitor. Just this week, Neuralink’s valuation hit a reported $8b, which represents a larger size than all active US neuromodulation markets segments last year ($6.6b) as per LSI’s recent analysis.
The hype factor is real, but this only makes sense if one looks at BCI as an enabling platform technology. Synchron’s integration offers a keyhole view. Think of it this way: if a commercially available API can significantly augment user functionality of a class III medical device – during active clinical trials – then BCI are not simply a point solution, but an enabling technology whose features and functions are up to the market (and not necessarily regulators.)
This integration is the latest example of how today’s BCI companies are building in public. What seems like relatively narrow use cases and target markets will expand because they only can, given the speed of innovation across AI, ML, materials science, surgical innovations, consumer technology, and beyond.
Read the full article here