PDFdigest

Eess.SP Research

ChineseEEG-2: An EEG Dataset for Multimodal Semantic Alignment and Neural Decoding during Reading and Listening

This paper presents a new dataset called ChineseEEG-2, which helps researchers understand how the brain processes language through reading and listening. It includes recordings of brain activity while participants read aloud and listened to stories.

4 min read September 30, 2026 Watch Video
Analyze with PDFdigest

Content & Liability Disclaimer

This article and its accompanying video are automated summaries derived from ChineseEEG-2: An EEG Dataset for Multimodal Semantic Alignment and Neural Decoding during Reading and Listening by Sitong Chen, Beiqianyi Li, Cuilin He, Dongyang Li, Mingyang Wu, Xinke Shen, Song Wang, Xuetao Wei, Xindi Wang, Haiyan Wu, Quanying Liu. The original research was conducted solely by the paper's authors; PDFdigest did not conduct any of the research and makes no claims of ownership over the underlying scientific work.

The video narration is generated by artificial intelligence and references the paper's authors for attribution. The video is not narrated by any of the paper's authors. This content may contain inaccuracies, omissions, or misinterpretations of the original research. First-person language (e.g., "we found", "our results") reflects the original authors' voice, not PDFdigest's. Always read the original paper for accurate, verified information before making any decisions based on this content.

This content is provided "as is" without any warranties, express or implied. Simulated systems OÜ, its officers, directors, employees, and agents shall not be liable for any direct, indirect, incidental, special, consequential, or punitive damages arising from your use of, reliance on, or access to this content, including but not limited to errors, omissions, or misinterpretations of the original research. This disclaimer applies to the fullest extent permitted by applicable law.

Key Takeaways
  1. 1 For subsequent experimental phases, gender-balanced stimulus sets were created by selecting recordings of one male and one female participant based on objective acoustic quality metrics, including clarity and articulation precision.
  2. 2 In summary, ChineseEEG-2 is designed to serve as a benchmark dataset for semantic neural decoding.
  3. 3 The Little Prince was organized into 27 content-based segments, while Garnett Dream was partitioned into 10-minute temporal segments, each preceded by a 5-second silent period to facilitate attentional preparation.
  4. 4 The visualization of the source activities corresponding to each word in the example sentence is provided.

Introduction

Understanding how the human brain processes language is a longstanding challenge in both neuroscience and artificial intelligence, particularly in how linguistic information is represented across different modalities such as speech and text. Multimodal large language models (MLLM) have accelerated research in this area, showing an increasing capacity to decode natural language from brain signals 2 .

However, further development and validation of these models depend on the availability of large-scale datasets that provide tightly aligned neural, acoustic, and linguistic data 3 .

The current scarcity of multimodal and multilingual resources poses a fundamental challenge, as existing datasets are often constrained by a single-modality focus or limited linguistic coverage 4, 5 .

Important Note

The current scarcity of multimodal and multilingual resources poses a fundamental challenge, as existing datasets are often constrained by a single-modality focus or limited linguistic coverage 4, 5 .

Important Note

Furthermore, nearly all these existing resources are limited to English, leaving a pronounced gap for Chinese and other non-English languages, restricting the development of next-generation neural decoding, brain-inspired language modeling, and the multilingual advancement of BCI applications.

Research Question

For subsequent experimental phases, gender-balanced stimulus sets were created by selecting recordings of one male and one female participant based on objective acoustic quality metrics, including clarity and articulation precision.

Methodology

Most provide only single-modality recordings (e.g., silent reading in ZuCo 11, 12 or auditory listening in Broderick et al. 13 , Brennan et al. 14 ), lacking the instance-level correspondence of linguistic content across modalities required for effective cross-modal and multi-task neural learning. Moreover, datasets such as those from Schoffelen et al. 15 and Gwilliams et al. 16 leverage MEG but lack both task diversity and scalable modality coverage.

Study Design

Using the same linguistic corpus as ChineseEEG, we recorded EEG and audio data from four participants for approximately 10.7 hours of reading aloud.

Their recordings were then played to eight additional participants in a listening task, resulting in 21.6 hours of EEG data aligned to the same semantic content.With fine-grained temporal and linguistic alignment, ChineseEEG-2 is the first large-scale token-aligned Chinese EEG benchmark supporting comparative studies across reading, speaking, and listening modalities.

How PDFdigest Helps You Understand Research

Instant Paper Analysis

Get structured summaries and key findings from dense PDFs in seconds.

Visual Explanations

Turn complex methods, figures, and results into clearer visual breakdowns.

AI-Powered Q&A

Ask focused questions and get answers grounded in the paper.

Try PDFdigest Free

Results & Findings

Many of these models employ contrastive pre-training to establish a shared representational space between audio and text, which in turn serves as a target for mapping neural signals in brain-to-language alignment. However, datasets that can support this alignment are not yet widely available.

  • Many of these models employ contrastive pre-training to establish a shared representational space between audio and text, which in turn serves as a target for mapping.
  • However, datasets that can support this alignment are not yet widely available.
  • In summary, ChineseEEG-2 is designed to serve as a benchmark dataset for semantic neural decoding.
  • Together, our resources support three main research directions:.
  • This study complied with the Declaration of Helsinki and was approved by the Institutional Review Board of the University of Macau (Approval No.
Important Note

In summary, ChineseEEG-2 is designed to serve as a benchmark dataset for semantic neural decoding.

Important Note

The Little Prince was organized into 27 content-based segments, while Garnett Dream was partitioned into 10-minute temporal segments, each preceded by a 5-second silent period to facilitate attentional preparation.

Background & Summary

The study addresses the challenge of understanding language processing in the brain and the need for large-scale multimodal datasets to support neural decoding models. Existing datasets are often limited in scope, highlighting the necessity for resources that provide aligned neural, acoustic, and linguistic data.

Participants and Task Overview

The study involved 12 healthy participants aged 18-25, who completed reading aloud and passive listening tasks. Data collection resulted in approximately 32.4 hours of EEG recordings, with careful attention to participant consent and ethical considerations.

Experimental Materials

The materials included the Chinese translations of ‘The Little Prince’ and ‘Garnett Dream’, chosen for their linguistic accessibility and cultural significance. Texts were divided into uniform-length passages to standardize reading sessions.

Source Paper Figures and Captions

Source-paper figure
Figure 1. Overview of the experiment and the modalities included in the dataset. (a) The experiment setup. Participants were instructed to sit quietly approximately 67cm from the screen and sequentially read the highlighted text. (b) The subject arrangement, only the audio from female and male subject 1 in Reading-aloud group were chosen as materials for the two groups of subjects in Passive-listening task. (c) The experimental paradigm. Participants' 128-channel EEG signals were recorded while reading the highlighted sentence or listening to the audio. (d) Data alignment in ChineseEEG-2. The corresponding audio and EEG segment to the same sentence are converted to embeddings and semantically aligned.
Figure 1. Overview of the experiment and the modalities included in the dataset. (a) The experiment setup. Participants were instructed to sit quietly approximately 67cm from the screen and sequentially read the highlighted text. (b) The subject arrangement, only the audio from female and male subject 1 in Reading-aloud group were chosen as materials for the two groups of subjects in Passive-listening task. (c) The experimental paradigm. Participants' 128-channel EEG signals were recorded while reading the highlighted sentence or listening to the audio. (d) Data alignment in ChineseEEG-2. The corresponding audio and EEG segment to the same sentence are converted to embeddings and semantically aligned.
Source-paper figure
Figure 2.Overview of the experimental pipeline followed by the participants. During the reading phase, four participants independently read two novels while their EEG signals were recorded. In the listening phase, eight participants were divided into two groups (four participants per group), and each group listened to the novel audio recordings by Female Participant 1 and Male Participant 1 while their EEG activity was recorded.
Figure 2.Overview of the experimental pipeline followed by the participants. During the reading phase, four participants independently read two novels while their EEG signals were recorded. In the listening phase, eight participants were divided into two groups (four participants per group), and each group listened to the novel audio recordings by Female Participant 1 and Male Participant 1 while their EEG activity was recorded.
Source-paper figure
sub-f1_ses-garnettdream_task-reading_run-11_eeg.eeg sub-f1_ses-garnettdream_task-reading_run-11_eeg.json sub-f1_ses-garnettdream_task-reading_run-11_eeg.vhdr sub-f1_ses-garnettdream_task-reading_run-11_eeg.vmrk sub-f1_ses-garnettdream_task-reading_run-11_events.json sub-f1_ses-garnettdream_task-reading_run-11_events.tsv onset -garnettdream_task-reading_run-11_ica_components.json { "shape ": [20, 2748204], "exclude ": [1,7,10,16,15 ] } └─ sub-f1 ├─ses-garnettdream │ │sub-04_ses-garnettdream_scans.tsv │ └─eeg └─ses-LittlePrince │sub-04_ses-littleprince_scans.tsv └─eeg │sub-f1_ses-garnettdream_scans.tsv └─eeg sub-f1_ses-garnettdream_space-CapTrak_coordsystem.json sub-f1_ses-garnettdream_space-CapTrak_electrodes.tsv sub-f1_ses-garnettdream_task-reading_run-11_bad_channels.json { "bad channels": [E1, E5, E20] } sub-f1_ses-GarnettDream_task-reading_run-f1_channels.tsv
sub-f1_ses-garnettdream_task-reading_run-11_eeg.eeg sub-f1_ses-garnettdream_task-reading_run-11_eeg.json sub-f1_ses-garnettdream_task-reading_run-11_eeg.vhdr sub-f1_ses-garnettdream_task-reading_run-11_eeg.vmrk sub-f1_ses-garnettdream_task-reading_run-11_events.json sub-f1_ses-garnettdream_task-reading_run-11_events.tsv onset -garnettdream_task-reading_run-11_ica_components.json { "shape ": [20, 2748204], "exclude ": [1,7,10,16,15 ] } └─ sub-f1 ├─ses-garnettdream │ │sub-04_ses-garnettdream_scans.tsv │ └─eeg └─ses-LittlePrince │sub-04_ses-littleprince_scans.tsv └─eeg │sub-f1_ses-garnettdream_scans.tsv └─eeg sub-f1_ses-garnettdream_space-CapTrak_coordsystem.json sub-f1_ses-garnettdream_space-CapTrak_electrodes.tsv sub-f1_ses-garnettdream_task-reading_run-11_bad_channels.json { "bad channels": [E1, E5, E20] } sub-f1_ses-GarnettDream_task-reading_run-f1_channels.tsv
Source-paper figure
Figure 3. Directory structure of the ChineseEEG-2 dataset and EEG preprocessing pipeline. (a) File structure of the dataset. The file structure encompasses experiment metadata, detailed electrode information (including location, type, and sampling rate), annotations for channels identified as defective during pre-processing, EEG data in BrainVision format, event files containing marker information, records of removed independent components during pre-processing, as well as the original text and embedding files. (b) EEG pre-processing pipeline. The pipeline includes data segmentation, band-pass filtering, interpolation of faulty channels, independent component analysis (ICA) for denoising, and re-referencing.
Figure 3. Directory structure of the ChineseEEG-2 dataset and EEG preprocessing pipeline. (a) File structure of the dataset. The file structure encompasses experiment metadata, detailed electrode information (including location, type, and sampling rate), annotations for channels identified as defective during pre-processing, EEG data in BrainVision format, event files containing marker information, records of removed independent components during pre-processing, as well as the original text and embedding files. (b) EEG pre-processing pipeline. The pipeline includes data segmentation, band-pass filtering, interpolation of faulty channels, independent component analysis (ICA) for denoising, and re-referencing.
PDFDIGEST AI

Struggling to understand complex research papers?

Upload any PDF and get instant AI-powered explanations, summaries, and visual breakdowns. Turn dense academic writing into clear, actionable insights.

Upload a Paper

Frequently Asked Questions

Multimodal large language models (MLLM) have accelerated research in this area, showing an increasing capacity to decode natural language from brain signals 2 . For subsequent experimental phases, gender-balanced stimulus sets were created by selecting recordings of one male and one female.

A total of 12 healthy participants were recruited for the study, ranging in age from 18 to 25 years (mean age: 21.9 years; 4 males, 8 females). Before the formal experiment, all participants underwent a pre-experimental session to validate the experimental procedures.

In summary, ChineseEEG-2 is designed to serve as a benchmark dataset for semantic neural decoding. The Little Prince was organized into 27 content-based segments, while Garnett Dream was partitioned into 10-minute temporal segments, each preceded by a 5-second silent period to facilitate.

The current scarcity of multimodal and multilingual resources poses a fundamental challenge, as existing datasets are often constrained by a single-modality focus or limited linguistic coverage 4, 5 . Furthermore, nearly all these existing resources are limited to English, leaving a pronounced.

This paper presents a new dataset called ChineseEEG-2, which helps researchers understand how the brain processes language through reading and listening. It includes recordings of brain activity while participants read aloud and listened to stories.

Yes. PDFDigest can turn this paper into a structured explanation, key takeaways, visual summaries, and a narrated video when available.