Research on hybrid brain-computer interface based on imperceptible visual and auditory stimulation responses_Journal of Biomedical Engineering

Authors：

PANG Zexin ¹ , WANG Yijun ² , DONG Qingpeng ³ , CHENG Zijian ³ , LI Zhaohui ¹ , ZHANG Ruoqing ¹ , CUI Hongyan ¹ ,  CHEN Xiaogang ¹

1. Institute of Biomedical Engineering, Chinese Academy of Medical Sciences and Peking Union Medical College, Tianjin 300192, P. R. China;
2. Institute of Semiconductors, Chinese Academy of Sciences, Beijing 100083, P. R. China;
3. School of Life Sciences, Tiangong University, Tianjin 300387, P. R. China;

Corresponding author：

CHEN Xiaogang, Email: chenxg@bme.cams.cn

Keywords：

Hybrid brain-computer interface; Electroencephalogram; Steady-state visual evoked potential; Auditory steady-state response; Imperceptible stimulation

DOI：

10.7507/1001-5515.202504033

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

In recent years, hybrid brain-computer interfaces (BCIs) have gained significant attention due to their demonstrated advantages in increasing the number of targets and enhancing robustness of the systems. However, Existing studies usually construct BCI systems using intense auditory stimulation and strong central visual stimulation, which lead to poor user experience and indicate a need for improving system comfort. Studies have proved that the use of peripheral visual stimulation and lower intensity of auditory stimulation can effectively boost the user’s comfort. Therefore, this study used high-frequency peripheral visual stimulation and 40-dB weak auditory stimulation to elicit steady-state visual evoked potential (SSVEP) and auditory steady-state response (ASSR) signals, building a high-comfort hybrid BCI based on weak audio-visual evoked responses. This system coded 40 targets via 20 high-frequency visual stimulation frequencies and two auditory stimulation frequencies, improving the coding efficiency of BCI systems. Results showed that the hybrid system's averaged classification accuracy was (78.00 ± 12.18) %, and the information transfer rate (ITR) could reached 27.47 bits/min. This study offers new ideas for the design of hybrid BCI paradigm based on imperceptible stimulation.

1.	Gao X, Wang Y, Chen X, et al. Interface, interaction, and intelligence in generalized brain-computer interfaces. Trends Cogn Sci, 2021, 25(8): 671-684.
2.	迟新一. 结合高频稳态视觉诱发电位的混合脑-机接口关键技术研究. 北京: 北京协和医学院, 2022.
3.	Pfurtscheller G, Allison B Z, Brunner C, et al. The hybrid BCI. Front Neurosci, 2010, 4: 30.
4.	陈小刚, 王毅军. 基于脑电的无创脑机接口研究进展. 科技导报, 2018, 36(12): 22-30.
5.	Zhang H, Xie J, Zhao C, et al. A novel spatial auditory brain-computer interface based on low-frequency periodic auditory motion stimulation paradigm. IEEE Trans Biomed Eng, 2025: 1-12.
6.	Zhang L, Liu S, Liu X, et al. Emotional arousal and valence jointly modulate the auditory response: A 40-Hz ASSR study. IEEE Trans Neural Syst Rehabil Eng, 2021, 29: 1150-1157.
7.	Zhang S, Chen Y, Zhang L, et al. Study on robot grasping system of SSVEP-BCI based on augmented reality stimulus. Tsinghua Sci Technol, 2023, 28(2): 322-329.
8.	Gao S, Wang Y, Gao X, et al. Visual and auditory brain-computer interfaces. IEEE Trans Biomed Eng, 2014, 61(5): 1436-1447.
9.	Barbosa S, Pires G, Nunes U. Toward a reliable gaze-independent hybrid BCI combining visual and natural auditory stimuli. J Neurosci Methods, 2016, 261: 47-61.
10.	郭柳君, 张雪英, 陈桂军. 基于空-频域特征的视听混合脑机接口. 计算机工程与设计, 2020, 41(6): 1755-1761.
11.	Wang F, He Y, Pan J, et al. A novel audiovisual brain-computer interface and its application in awareness detection. Sci Rep, 2015, 5: 9962.
12.	Brainard D H. The psychophysics toolbox. Spat Vis, 1997, 10(4): 433-436.
13.	Manyakov N V, Chumerin N, Robben A, et al. Sampled sinusoidal stimulation profile and multichannel fuzzy logic classification for monitor-based phase-coded SSVEP brain-computer interfacing. J Neural Eng, 2013, 10(3): 747-754.
14.	Shamsi E, Shirzhiyan Z, Keihani A, et al. Classification of the EEG evoked by auditory stimuli with a periodic carrier frequency coding in order to be used in BCI systems. Front Biomed Technol, 2016, 3(3-4): 41-48.
15.	Spüler M, Kurek S. Alpha-band lateralization during auditory selective attention for brain–computer interface control. Brain-Comput Interfa, 2018, 5(1): 23-29.
16.	Kaongoen N, Jo S. A novel hybrid auditory BCI paradigm combining ASSR and P300. J Neurosci Methods, 2017, 279: 44-51.
17.	Wang L, Noordanus E, Opstal A J V. Towards real-time detection of auditory steady-state responses: a comparative study. IEEE Access, 2021, 9: 108975-108991.
18.	Wang Y, Liu X, Cui H, et al. An auditory selective attention brain-computer interface system based on auditory steady-state response. Appl Acoust, 2025, 228: 110291.
19.	Kachenoura A, Albera L, Senhadji L, et al. ICA: A potential tool for BCI systems. IEEE Signal Process Mag, 2008, 25(1): 57-68.
20.	Delorme A, Makeig S. EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J Neurosci Methods, 2004, 134(1): 9-21.
21.	Chen X, Wang Y, Gao S, et al. Filter bank canonical correlation analysis for implementing a high-speed SSVEP-based brain-computer interface. J Neural Eng, 2015, 12(4): 046008.
22.	陈小刚. 高速率稳态视觉诱发电位脑-机接口的关键技术研究. 北京: 清华大学, 2015.
23.	Mariko T, Kenji K, Yohei I, et al. Global and parallel cortical processing based on auditory gamma oscillatory responses in humans. Cereb Cortex, 2021, 31(10): 4518-4532.
24.	肖晓琳. 基于微弱事件相关脑电特征的脑—机编解码关键技术及应用研究. 天津: 天津大学, 2020.
25.	Zhao X, Wang Z, Zhang M, et al. A comfortable steady state visual evoked potential stimulation paradigm using peripheral vision. J Neural Eng, 2021, 18: 056021.
26.	Ming G, Pei W, Gao X, et al. A high-performance SSVEP-based BCI using imperceptible flickers. J Neural Eng, 2023, 20: 016042.
27.	Nakanishi M, Wang Y, Chen X, et al. Enhancing detection of SSVEPs for a high-speed brain speller using task-related component analysis. IEEE Trans Biomed Eng, 2017, 65(1): 104-112.
28.	Liu B, Chen X, Shi N, et al. Improving the performance of individually calibrated SSVEP-BCI by task- discriminant component analysis. IEEE Trans Neural Syst Rehabil Eng, 2021, 29: 1998-2007.
29.	Xu G, Wu Y, Li M. The study of influence of sound on visual ERP-based brain computer interface. Sensors, 2020, 20(4): 1203.
30.	Syrov N, Yakovlev L, Nikolaeva V, et al. Mental strategies in a P300-BCI: visuomotor transformation is an option. Diagnostics (Basel), 2022, 12(11): 2607.
31.	Treder M S, Purwins H, Miklody D, et al. Decoding auditory attention to instruments in polyphonic music using single-trial EEG classification. J Neural Eng, 2014, 11(2): 026009.
32.	Zhou S, Allison B Z, Kübler A, et al. Effects of background music on objective and subjective performance measures in an auditory BCI. Front Comput Neurosci, 2016, 10: 105.
33.	Zhang H, Xie J, Xiao Y, et al. Steady-state auditory motion based potentials evoked by intermittent periodic virtual sound source and the effect of auditory noise on EEG enhancement. Hear Res, 2023, 428: 108670.
34.	Robinson A K, Venkatesh P, Boring M J, et al. Very high density EEG elucidates spatiotemporal aspects of early visual processing. Sci Rep, 2017, 7(1): 16248.

1. Gao X, Wang Y, Chen X, et al. Interface, interaction, and intelligence in generalized brain-computer interfaces. Trends Cogn Sci, 2021, 25(8): 671-684.
2. 迟新一. 结合高频稳态视觉诱发电位的混合脑-机接口关键技术研究. 北京: 北京协和医学院, 2022.
3. Pfurtscheller G, Allison B Z, Brunner C, et al. The hybrid BCI. Front Neurosci, 2010, 4: 30.
4. 陈小刚, 王毅军. 基于脑电的无创脑机接口研究进展. 科技导报, 2018, 36(12): 22-30.
5. Zhang H, Xie J, Zhao C, et al. A novel spatial auditory brain-computer interface based on low-frequency periodic auditory motion stimulation paradigm. IEEE Trans Biomed Eng, 2025: 1-12.
6. Zhang L, Liu S, Liu X, et al. Emotional arousal and valence jointly modulate the auditory response: A 40-Hz ASSR study. IEEE Trans Neural Syst Rehabil Eng, 2021, 29: 1150-1157.
7. Zhang S, Chen Y, Zhang L, et al. Study on robot grasping system of SSVEP-BCI based on augmented reality stimulus. Tsinghua Sci Technol, 2023, 28(2): 322-329.
8. Gao S, Wang Y, Gao X, et al. Visual and auditory brain-computer interfaces. IEEE Trans Biomed Eng, 2014, 61(5): 1436-1447.
9. Barbosa S, Pires G, Nunes U. Toward a reliable gaze-independent hybrid BCI combining visual and natural auditory stimuli. J Neurosci Methods, 2016, 261: 47-61.
10. 郭柳君, 张雪英, 陈桂军. 基于空-频域特征的视听混合脑机接口. 计算机工程与设计, 2020, 41(6): 1755-1761.
11. Wang F, He Y, Pan J, et al. A novel audiovisual brain-computer interface and its application in awareness detection. Sci Rep, 2015, 5: 9962.
12. Brainard D H. The psychophysics toolbox. Spat Vis, 1997, 10(4): 433-436.
13. Manyakov N V, Chumerin N, Robben A, et al. Sampled sinusoidal stimulation profile and multichannel fuzzy logic classification for monitor-based phase-coded SSVEP brain-computer interfacing. J Neural Eng, 2013, 10(3): 747-754.
14. Shamsi E, Shirzhiyan Z, Keihani A, et al. Classification of the EEG evoked by auditory stimuli with a periodic carrier frequency coding in order to be used in BCI systems. Front Biomed Technol, 2016, 3(3-4): 41-48.
15. Spüler M, Kurek S. Alpha-band lateralization during auditory selective attention for brain–computer interface control. Brain-Comput Interfa, 2018, 5(1): 23-29.
16. Kaongoen N, Jo S. A novel hybrid auditory BCI paradigm combining ASSR and P300. J Neurosci Methods, 2017, 279: 44-51.
17. Wang L, Noordanus E, Opstal A J V. Towards real-time detection of auditory steady-state responses: a comparative study. IEEE Access, 2021, 9: 108975-108991.
18. Wang Y, Liu X, Cui H, et al. An auditory selective attention brain-computer interface system based on auditory steady-state response. Appl Acoust, 2025, 228: 110291.
19. Kachenoura A, Albera L, Senhadji L, et al. ICA: A potential tool for BCI systems. IEEE Signal Process Mag, 2008, 25(1): 57-68.
20. Delorme A, Makeig S. EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J Neurosci Methods, 2004, 134(1): 9-21.
21. Chen X, Wang Y, Gao S, et al. Filter bank canonical correlation analysis for implementing a high-speed SSVEP-based brain-computer interface. J Neural Eng, 2015, 12(4): 046008.
22. 陈小刚. 高速率稳态视觉诱发电位脑-机接口的关键技术研究. 北京: 清华大学, 2015.
23. Mariko T, Kenji K, Yohei I, et al. Global and parallel cortical processing based on auditory gamma oscillatory responses in humans. Cereb Cortex, 2021, 31(10): 4518-4532.
24. 肖晓琳. 基于微弱事件相关脑电特征的脑—机编解码关键技术及应用研究. 天津: 天津大学, 2020.
25. Zhao X, Wang Z, Zhang M, et al. A comfortable steady state visual evoked potential stimulation paradigm using peripheral vision. J Neural Eng, 2021, 18: 056021.
26. Ming G, Pei W, Gao X, et al. A high-performance SSVEP-based BCI using imperceptible flickers. J Neural Eng, 2023, 20: 016042.
27. Nakanishi M, Wang Y, Chen X, et al. Enhancing detection of SSVEPs for a high-speed brain speller using task-related component analysis. IEEE Trans Biomed Eng, 2017, 65(1): 104-112.
28. Liu B, Chen X, Shi N, et al. Improving the performance of individually calibrated SSVEP-BCI by task- discriminant component analysis. IEEE Trans Neural Syst Rehabil Eng, 2021, 29: 1998-2007.
29. Xu G, Wu Y, Li M. The study of influence of sound on visual ERP-based brain computer interface. Sensors, 2020, 20(4): 1203.
30. Syrov N, Yakovlev L, Nikolaeva V, et al. Mental strategies in a P300-BCI: visuomotor transformation is an option. Diagnostics (Basel), 2022, 12(11): 2607.
31. Treder M S, Purwins H, Miklody D, et al. Decoding auditory attention to instruments in polyphonic music using single-trial EEG classification. J Neural Eng, 2014, 11(2): 026009.
32. Zhou S, Allison B Z, Kübler A, et al. Effects of background music on objective and subjective performance measures in an auditory BCI. Front Comput Neurosci, 2016, 10: 105.
33. Zhang H, Xie J, Xiao Y, et al. Steady-state auditory motion based potentials evoked by intermittent periodic virtual sound source and the effect of auditory noise on EEG enhancement. Hear Res, 2023, 428: 108670.
34. Robinson A K, Venkatesh P, Boring M J, et al. Very high density EEG elucidates spatiotemporal aspects of early visual processing. Sci Rep, 2017, 7(1): 16248.

Journal of Biomedical Engineering

Latest ArticlesResearch on hybrid brain-computer interface based on imperceptible visual and auditory stimulation responses

Abstract Full text Figures/Tables Video References Cited by

Format

Content