about
Thought2Text: Text Generation from EEG Signal Using Large Language Models (arxiv.org)
2 points by trkaky on Apr 7, 2025 | hide | past | pdf | discuss on HN

In plain words: Brain signals from people viewing pictures are turned into visual features, then fed to a language model trained to describe images so it can write text from the EEG. On recordings from six people, its captions rated as fluent and adequate on text-quality measures.

Abstract · Thought2Text: Text Generation from EEG Signal using Large Language Models (LLMs)

Decoding and expressing brain activity in a comprehensible form is a challenging frontier in AI. This paper presents Thought2Text, which uses instruction-tuned Large Language Models (LLMs) fine-tuned with EEG data to achieve this goal. The approach involves three stages: (1) training an EEG encoder for visual feature extraction, (2) fine-tuning LLMs on image and text data, enabling multimodal description generation, and (3) further fine-tuning on EEG embeddings to generate text directly from EEG during inference. Experiments on a public EEG dataset collected for six subjects with image stimuli and text captions demonstrate the efficacy of multimodal LLMs (LLaMA-v3, Mistral-v0.3, Qwen2.5), validated using traditional language generation evaluation metrics, as well as fluency and adequacy measures. This approach marks a significant advancement towards portable, low-cost "thoughts-to-text" technology with potential applications in both neuroscience and natural language processing.

Abhijit Mishra, Shreya Shukla, Jose Torres, Jacek Gwizdka, Shounak Roychowdhury
arXiv:2410.07507 · cs.CL · submitted Oct 10, 2024 · updated Feb 10, 2025
abstract · pdf · html · Accepted to Findings of NAACL 2025

add comment on HN