Electroencephalography (EEG) offers millisecond-level temporal resolution but limited spatial localization, whereas functional near-infrared spectroscopy (fNIRS) provides better spatial resolution but shows a hemodynamic delay of several seconds. A hybrid EEG-fNIRS brain-computer interface fuses the complementary strengths of both modalities but faces challenges from intrinsic spatiotemporal heterogeneity between electrophysiological and hemodynamic signals. To address this, this study proposes a Progressive Cross-Modal Alignment Network (PCMAN), which achieves deep EEG-fNIRS fusion and alignment through a three-layer progressive mechanism: physical alignment (a Spatiotemporal correlation Dynamic Fusion framework, SDF), semantic alignment (a Multi-Dimensional Feature Enhancement Attention network, MDFEA), and task alignment (a Cross-modal Adaptive Decoding Network, CADN). On three cognitive tasks–mental arithmetic (MA), word generation (WG), and motor imagery (MI)–PCMAN achieved accuracies of 88.18%, 81.93%, and 72.94%, outperforming state-of-the-art hybrid BCI methods.
@article{gao2026pcman,title={Progressive Cross-Modal Attention Network for Brain--Computer Interface Decoding Using EEG and Functional Near-Infrared Spectroscopy (fNIRS)},author={Gao, Yunyuan and Luo, Xiaoqi and Zhang, Zhengnan and Shao, Xu and Qin, Yanhua},journal={Applied Soft Computing},volume={199C},pages={115318},year={2026},month=jun,publisher={Elsevier},doi={10.1016/j.asoc.2026.115318},url={https://doi.org/10.1016/j.asoc.2026.115318},keywords={Hybrid brain-computer interface, EEG-fNIRS fusion, Cross-modal attention, Progressive alignment, Multimodal deep learning},}
Accepted
Neurovascular Alignment and Reliability-Aware Fusion for Continuous EEG–fNIRS Emotion Regression
Yunyuan Gao, Xiaoqi Luo, Zhengnan Zhang, and 3 more authors
IEEE Transactions on Affective Computing. IF≈9.8 · JCR Q1 · 中科院计算机 1区 TOP · Accepted on 2026-08-19. Second author (advisor Yunyuan Gao 高云园 first author). Framework: PhysioSync. , 2026
Continuous affect tracking is central to closed-loop affective brain–computer interfaces and adaptive human–computer interaction. Continuous EEG–fNIRS regression is well suited to this setting because EEG captures rapid neural dynamics and fNIRS provides complementary hemodynamic information, but direct fusion often underperforms EEG-only baselines when neurovascular coupling (NVC) introduces subject- and region-dependent delays. PhysioSync addresses this by performing HRF-like temporal alignment and residual-aware reliability control before multimodal fusion. A learnable dual-gamma kernel maps EEG-derived drive to the fNIRS timeline through finite-window causal convolution, and the resulting residuals modulate the contribution of hemodynamic evidence during fusion. A linear-complexity Mamba encoder provides long-context EEG representation. On REFED, PhysioSync reduces Avg MAE by 5.8% relative to the best fusion baseline and by 7.2% relative to EEG-only when evaluated across three within-subject folds, while showing lower error in 1–3 s trajectory forecasting. With lightweight task-specific adaptations, ENTER and ZBra-music results indicate conditional applicability to discrete classification rather than uniform transferability, with performance depending on EEG coverage and HbO/HbR chromophore decomposition.
@article{gao2026physiosync,title={Neurovascular Alignment and Reliability-Aware Fusion for Continuous EEG--fNIRS Emotion Regression},author={Gao, Yunyuan and Luo, Xiaoqi and Zhang, Zhengnan and Lu, Jiangwen and Zhang, Yingchun and Huang, Ruyue},journal={IEEE Transactions on Affective Computing},year={2026},keywords={physiological affect recognition, EEG--fNIRS fusion, continuous emotion regression, neurovascular coupling, multimodal affective BCI},}
Progressive Cross-Modal Attention Network for Brain–Computer Interface Decoding Using EEG and Functional Near-Infrared Spectroscopy (fNIRS)Applied Soft Computing. IF≈7.8 · JCR Q1 · 中科院计算机 2区 TOP. Second author (advisor Yunyuan Gao 高云园 first author). Framework: PCMAN. , Jun 2026
Neurovascular Alignment and Reliability-Aware Fusion for Continuous EEG–fNIRS Emotion RegressionIEEE Transactions on Affective Computing. IF≈9.8 · JCR Q1 · 中科院计算机 1区 TOP · Accepted on 2026-08-19. Second author (advisor Yunyuan Gao 高云园 first author). Framework: PhysioSync. , 2026