Self-Supervised Pretraining on Paired Sequences of fMRI Data for Transfer Learning to Brain Decoding Tasks

Authors: Sean Paulsen, Michael Casey

Abstract:

In this work, we present a self-supervised pretraining framework for transformers on functional Magnetic Resonance Imaging (fMRI) data. First, we pretrain our architecture on two self-supervised tasks simultaneously to teach the model a general understanding of the temporal and spatial dynamics of human auditory cortex during music listening. Our pretraining results are the first to suggest a synergistic effect of multitask training on fMRI data. Second, we finetune the pretrained models and train additional fresh models on a supervised fMRI classification task. We observe significantly improved accuracy on held-out runs with the finetuned models, which demonstrates the ability of our pretraining tasks to facilitate transfer learning. This work contributes to the growing body of literature on transformer architectures for pretraining and transfer learning with fMRI data, and serves as a proof of concept for our pretraining tasks and multitask pretraining on fMRI data.

Keywords: Transfer learning, fMRI, self-supervised, brain decoding, transformer, multitask training.

References:

[1] H. A. Bedel, I. S¸ ıvgın, O. Dalmaz, S. U. H. Dar, and T. C¸ ukur, “BolT: Fused Window Transformers for fMRI Time Series Analysis,” Feb. 2023, arXiv:2205.11578 (cs, eess). (Online). Available: http://arxiv.org/abs/2205.11578
[2] E. M. C. Hillman, “Coupling mechanism and significance of the BOLD signal: a status report,” Annual Review of Neuroscience, vol. 37, pp. 161–181, 2014.
[3] J. C. Rajapakse, F. Kruggel, J. M. Maisog, and D. Y. von Cramon, “Modeling hemodynamic response for analysis of functional MRI time-series,” Human Brain Mapping, vol. 6, no. 4, pp. 283–300, 1998.
[4] M. Kubicki, R. W. McCarley, P. G. Nestor, T. Huh, R. Kikinis, M. E. Shenton, and C. G. Wible, “An fMRI study of semantic processing in men with schizophrenia,” NeuroImage, vol. 20, no. 4, pp. 1923–1933, Dec. 2003.
[5] D. Wang, L. Shi, D. S. Yeung, P.-A. Heng, T.-T. Wong, and E. C. C. Tsang, “Support vector clustering for brain activation detection,” Medical image computing and computer-assisted intervention: MICCAI ... International Conference on Medical Image Computing and Computer-Assisted Intervention, vol. 8, no. Pt 1, pp. 572–579, 2005.
[6] J. M. Papma, M. Smits, M. de Groot, F. U. Mattace Raso, A. van der Lugt, H. A. Vrooman, W. J. Niessen, P. J. Koudstaal, J. C. van Swieten, F. M. van der Veen, and N. D. Prins, “The effect of hippocampal function, volume and connectivity on posterior cingulate cortex functioning during episodic memory fMRI in mild cognitive impairment,” European Radiology, vol. 27, no. 9, pp. 3716–3724, Sep. 2017.
[7] K. Li, L. Guo, J. Nie, G. Li, and T. Liu, “Review of methods for functional brain connectivity detection using fMRI,” Computerized Medical Imaging and Graphics: The Official Journal of the Computerized Medical Imaging Society, vol. 33, no. 2, pp. 131–139, Mar. 2009.
[8] A. Venkataraman, K. R. A. Van Dijk, R. L. Buckner, and P. Golland, “EXPLORING FUNCTIONAL CONNECTIVITY IN FMRI VIA CLUSTERING,” Proceedings of the ... IEEE International Conference on Acoustics, Speech, and Signal Processing. ICASSP (Conference), vol. 2009, pp. 441–444, Apr. 2009.
[9] S. Nishimoto, A. Vu, T. Naselaris, Y. Benjamini, B. Yu, and J. Gallant, “Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies,” Current Biology, vol. 21, no. 19, pp. 1641–1646, Oct. 2011. (Online). Available: https://linkinghub.elsevier.com/retrieve/pii/S0960982211009377
[10] O. Simon, F. Kherif, G. Flandin, J.-B. Poline, D. Rivi`ere, J.-F. Mangin, D. Le Bihan, and S. Dehaene, “Automatized clustering and functional geometry of human parietofrontal networks for language, space, and number,” NeuroImage, vol. 23, no. 3, pp. 1192–1202, Nov. 2004. (Online). Available: https://linkinghub.elsevier.com/retrieve/pii/S105381190400549X
[11] B. P. Rogers, V. L. Morgan, A. T. Newton, and J. C. Gore, “Assessing functional connectivity in the human brain by fMRI,” Magnetic Resonance Imaging, vol. 25, no. 10, pp. 1347–1357, Dec. 2007.
[12] S. Paulsen, L. May, and M. Casey, “Decoding imagined auditory pitch phenomena with an autoencoder based temporal convolutional architecture,” in BRAININFO. Nice, France: IARIA, July 2021.
[13] C. Niu, A. D. Cohen, X. Wen, Z. Chen, P. Lin, X. Liu, B. H. Menze, B. Wiestler, Y. Wang, and M. Zhang, “Modeling motor task activation from resting-state fMRI using machine learning in individual subjects,” Brain Imaging and Behavior, vol. 15, no. 1, pp. 122–132, Feb. 2021. (Online). Available: http://link.springer.com/10.1007/s11682-019-00239-9
[14] B. T. T. Yeo, F. M. Krienen, J. Sepulcre, M. R. Sabuncu, D. Lashkari, M. Hollinshead, J. L. Roffman, J. W. Smoller, L. Z¨ollei, J. R. Polimeni, B. Fischl, H. Liu, and R. L. Buckner, “The organization of the human cerebral cortex estimated by intrinsic functional connectivity,” Journal of Neurophysiology, vol. 106, no. 3, pp. 1125–1165, Sep. 2011.
[15] K. R. A. Van Dijk, T. Hedden, A. Venkataraman, K. C. Evans, S. W. Lazar, and R. L. Buckner, “Intrinsic functional connectivity as a tool for human connectomics: theory, properties, and optimization,” Journal of Neurophysiology, vol. 103, no. 1, pp. 297–321, Jan. 2010.
[16] Z. Hu and P. Shi, “Interregional functional connectivity via pattern synchrony,” 01 2007, pp. 1 – 6.
[17] X. Zhan and R. Yu, “A Window into the Brain: Advances in Psychiatric fMRI,” BioMed Research International, vol. 2015, p. 542467, 2015.
[18] N. D. Woodward and C. J. Cascio, “Resting-State Functional Connectivity in Psychiatric Disorders,” JAMA psychiatry, vol. 72, no. 8, pp. 743–744, Aug. 2015.
[19] C. H. Xia, Z. Ma, R. Ciric, S. Gu, R. F. Betzel, A. N. Kaczkurkin, M. E. Calkins, P. A. Cook, A. Garc´ıa de la Garza, S. N. Vandekar, Z. Cui, T. M. Moore, D. R. Roalf, K. Ruparel, D. H. Wolf, C. Davatzikos, R. C. Gur, R. E. Gur, R. T. Shinohara, D. S. Bassett, and T. D. Satterthwaite, “Linked dimensions of psychopathology and connectivity in functional brain networks,” Nature Communications, vol. 9, no. 1, p. 3003, Aug. 2018.
[20] L. Zou, J. Zheng, C. Miao, M. J. Mckeown, and Z. J. Wang, “3D CNN Based Automatic Diagnosis of Attention Deficit Hyperactivity Disorder Using Functional and Structural MRI,” IEEE Access, vol. 5, pp. 23 626–23 636, 2017. (Online). Available: http://ieeexplore.ieee.org/document/8067637/
[21] J. Kawahara, C. J. Brown, S. P. Miller, B. G. Booth, V. Chau, R. E. Grunau, J. G. Zwicker, and G. Hamarneh, “BrainNetCNN: Convolutional neural networks for brain networks; towards predicting neurodevelopment,” NeuroImage, vol. 146, pp. 1038–1049, Feb. 2017.
[22] J. Dakka, P. Bashivan, M. Gheiratmand, I. Rish, S. Jha, and R. Greiner, “Learning Neural Markers of Schizophrenia Disorder Using Recurrent Neural Networks,” Dec. 2017, arXiv:1712.00512 (cs). (Online). Available: http://arxiv.org/abs/1712.00512
[23] X. Li, Y. Zhou, N. Dvornek, M. Zhang, S. Gao, J. Zhuang, D. Scheinost, L. H. Staib, P. Ventola, and J. S. Duncan, “BrainGNN: Interpretable Brain Graph Neural Network for fMRI Analysis,” Medical Image Analysis, vol. 74, p. 102233, Dec. 2021.
[24] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention Is All You Need,” Dec. 2017, arXiv:1706.03762 (cs). (Online). Available: http://arxiv.org/abs/1706.03762
[25] I. Malkiel, G. Rosenman, L. Wolf, and T. Hendler, “Self-Supervised Transformers for fMRI representation,” Aug. 2022, arXiv:2112.05761 (cs, eess). (Online). Available: http://arxiv.org/abs/2112.05761
[26] S. Nguyen, B. Ng, A. D. Kaplan, and P. Ray, “Attend and Decode: 4D fMRI Task State Decoding Using Attention Models,” Jan. 2021, arXiv:2004.05234 (cs). (Online). Available: http://arxiv.org/abs/2004.05234
[27] S. Li, X. Jin, Y. Xuan, X. Zhou, W. Chen, Y.-X. Wang, and X. Yan, “Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting,” Jan. 2020, arXiv:1907.00235 (cs, stat). (Online). Available: http://arxiv.org/abs/1907.00235
[28] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” May 2019, arXiv:1810.04805 (cs). (Online). Available: http://arxiv.org/abs/1810.04805
[29] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” Jun. 2021, arXiv:2010.11929 (cs). (Online). Available: http://arxiv.org/abs/2010.11929
[30] L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, and K.-W. Chang, “VisualBERT: A Simple and Performant Baseline for Vision and Language,” Aug. 2019, arXiv:1908.03557 (cs). (Online). Available: http://arxiv.org/abs/1908.03557
[31] D. Erhan, Y. Bengio, A. Courville, P.-A. Manzagol, P. Vincent, and S. Bengio, “Why does unsupervised pre-training help deep learning?” Journal of Machine Learning Research, vol. 11, no. 19, pp. 625–660, 2010. (Online). Available: http://jmlr.org/papers/v11/erhan10a.html
[32] K. S. Kalyan, A. Rajasekharan, and S. Sangeetha, “AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing,” Aug. 2021, arXiv:2108.05542 (cs). (Online). Available: http://arxiv.org/abs/2108.05542
[33] W. D. Penny, K. J. Friston, J. T. Ashburner, S. J. Kiebel, and T. E. Nichols, Statistical parametric mapping: the analysis of functional brain images. Elsevier, 2011.
[34] M. W. Woolrich, B. D. Ripley, M. Brady, and S. M. Smith, “Temporal autocorrelation in univariate linear modeling of FMRI data,” NeuroImage, vol. 14, no. 6, pp. 1370–1386, Dec. 2001.
[35] K. A. Norman, S. M. Polyn, G. J. Detre, and J. V. Haxby, “Beyond mind-reading: multi-voxel pattern analysis of fMRI data,” Trends in Cognitive Sciences, vol. 10, no. 9, pp. 424–430, Sep. 2006.
[36] J. V. Haxby, “Multivariate pattern analysis of fMRI: the early beginnings,” NeuroImage, vol. 62, no. 2, pp. 852–855, Aug. 2012.
[37] L. May, A. R. Halpern, S. D. Paulsen, and M. A. Casey, “Imagined Musical Scale Relationships Decoded from Auditory Cortex,” Journal of Cognitive Neuroscience, vol. 34, no. 8, pp. 1326–1339, 07 2022. (Online). Available: https://doi.org/10.1162/jocn a 01858
[38] X. Song and N.-k. Chen, “A SVM-based quantitative fMRI method for resting-state functional network detection,” Magnetic Resonance Imaging, vol. 32, no. 7, pp. 819–831, Sep. 2014.
[39] Z. Wang, A. R. Childress, J. Wang, and J. A. Detre, “Support vector machine learning-based fMRI data group analysis,” NeuroImage, vol. 36, no. 4, pp. 1139–1151, Jul. 2007.
[40] S. H. Hojjati, A. Ebrahimzadeh, A. Khazaee, A. Babajani-Feremi, and Alzheimer’s Disease Neuroimaging Initiative, “Predicting conversion from MCI to AD using resting-state fMRI, graph theoretical approach and SVM,” Journal of Neuroscience Methods, vol. 282, pp. 69–80, Apr. 2017.
[41] H.-I. Suk, C.-Y. Wee, S.-W. Lee, and D. Shen, “State-space model with deep learning for functional dynamics estimation in resting-state fMRI,” NeuroImage, vol. 129, pp. 292–307, Apr. 2016.
[42] H. Huang, X. Hu, Y. Zhao, M. Makkie, Q. Dong, S. Zhao, L. Guo, and T. Liu, “Modeling Task fMRI Data Via Deep Convolutional Autoencoder,” IEEE transactions on medical imaging, vol. 37, no. 7, pp. 1551–1561, Jul. 2018.
[43] X. Wang, X. Liang, Z. Jiang, B. A. Nguchu, Y. Zhou, Y. Wang, H. Wang, Y. Li, Y. Zhu, F. Wu, J.-H. Gao, and B. Qiu, “Decoding and mapping task states of the human brain via deep learning,” Human Brain Mapping, vol. 41, no. 6, pp. 1505–1519, Apr. 2020, arXiv:1801.09858 (q-bio). (Online). Available: http://arxiv.org/abs/1801.09858
[44] D. L. K. Yamins, H. Hong, C. F. Cadieu, E. A. Solomon, D. Seibert, and J. J. DiCarlo, “Performance-optimized hierarchical models predict neural responses in higher visual cortex,” Proceedings of the National Academy of Sciences, vol. 111, no. 23, pp. 8619–8624, Jun. 2014. (Online). Available: https://pnas.org/doi/full/10.1073/pnas.1403112111
[45] N. C. Dvornek, P. Ventola, K. A. Pelphrey, and J. S. Duncan, “Identifying Autism from Resting-State fMRI Using Long Short-Term Memory Networks,” Machine learning in medical imaging. MLMI (Workshop), vol. 10541, pp. 362–370, Sep. 2017.
[46] W. Li, X. Lin, and X. Chen, “Detecting Alzheimer’s disease Based on 4D fMRI: An exploration under deep learning framework,” Neurocomputing, vol. 388, pp. 280–287, May 2020. (Online). Available: https://linkinghub.elsevier.com/retrieve/pii/S0925231220301041
[47] C. Zhao, H. Li, Z. Jiao, T. Du, and Y. Fan, “A 3D Convolutional Encapsulated Long Short-Term Memory (3DConv-LSTM) Model for Denoising fMRI Data,” Medical image computing and computer-assisted intervention: MICCAI ... International Conference on Medical Image Computing and Computer-Assisted Intervention, vol. 12267, pp. 479–488, Oct. 2020.
[48] A. W. Thomas, H. R. Heekeren, K.-R. M¨uller, and W. Samek, “Analyzing Neuroimaging Data Through Recurrent Deep Learning Models,” Apr. 2019, arXiv:1810.09945 (cs, q-bio, stat). (Online). Available: http://arxiv.org/abs/1810.09945
[49] M. Pominova, A. Artemov, M. Sharaev, E. Kondrateva, A. Bernstein, and E. Burnaev, “Voxelwise 3D Convolutional and Recurrent Neural Networks for Epilepsy and Depression Diagnostics from Structural and Functional MRI Data,” in 2018 IEEE International Conference on Data Mining Workshops (ICDMW). Singapore, Singapore: IEEE, Nov. 2018, pp. 299–307. (Online). Available: https://ieeexplore.ieee.org/document/8637478/
[50] D. C. Van Essen, S. M. Smith, D. M. Barch, T. E. J. Behrens, E. Yacoub, K. Ugurbil, and WU-Minn HCP Consortium, “The WU-Minn Human Connectome Project: an overview,” NeuroImage, vol. 80, pp. 62–79, Oct. 2013.
[51] P. Sid´en, Scalable Bayesian spatial analysis with Gaussian Markov random fields, 09 2020.
[52] T. Nakai, N. Koide-Majima, and S. Nishimoto, “”music genre fmri dataset”,” 2021.
[53] F. F. Labs, “Supercharging classification - the value of multi-task learnin,” June 2018.
[54] V. N. Salimpoor, D. H. Zald, R. J. Zatorre, A. Dagher, and A. R. McIntosh, “Predictions and the brain: how musical sounds become rewarding,” Trends in Cognitive Sciences, vol. 19, no. 2, pp. 86–91, 2015. (Online). Available: https://www.sciencedirect.com/science/article/pii/S1364661314002538
[55] T. Nakai, N. Koide-Majima, and S. Nishimoto, “Correspondence of categorical and feature-based representations of music in the human brain,” Brain and Behavior, vol. 11, no. 1, p. e01936, 2021. (Online). Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/brb3.1936
[56] F. Pestilli and L. Kitchell, “Brainlife,” 2017. (Online). Available: https://brainlife.io/
[57] O. Esteban, C. J. Markiewicz, R. W. Blair, C. A. Moodie, A. I. Isik, A. Erramuzpe, J. D. Kent, M. Goncalves, E. DuPre, M. Snyder et al., “fmriprep: a robust preprocessing pipeline for functional mri,” Nature methods, vol. 16, no. 1, pp. 111–116, 2019.
[58] V. Fonov, A. C. Evans, K. Botteron, C. R. Almli, R. C. McKinstry, and D. L. Collins, “Unbiased average age-appropriate atlases for pediatric studies.” Neuroimage, vol. 54, no. 1, pp. 313–327, Jan 2011.
[59] A. Angulo-Perkins, W. Aub´e, I. Peretz, F. A. Barrios, J. L. Armony, and L. Concha, “Music listening engages specific cortical regions within the temporal lobes: differences between musicians and non-musicians.” Cortex; a journal devoted to the study of the nervous system and behavior, vol. 59, pp. 126–137, Oct. 2014, place: Italy.
[60] V. D. Angelis, F. D. Martino, M. Moerel, R. Santoro, L. Hausfeld, and E. Formisano, “Cortical processing of pitch: Model-based encoding and decoding of auditory fMRI responses to real-life sounds,” NeuroImage, vol. 180, pp. 291–300, 2018. (Online). Available: https://www.sciencedirect.com/science/article/pii/S1053811917309278
[61] N. Staeren, H. Renvall, F. De Martino, R. Goebel, and E. Formisano, “Sound categories are represented as distributed patterns in the human auditory cortex.” Current biology : CB, vol. 19, no. 6, pp. 498–502, Mar. 2009, place: England.
[62] P. McCarthy, “Fsleyes,” Aug. 2022. (Online). Available: https://doi.org/10.5281/zenodo.7038115
[63] R. C. Craddock, G. A. James, P. E. Holtzheimer, 3rd, X. P. Hu, and H. S. Mayberg, “A whole brain fMRI atlas generated via spatially constrained spectral clustering,” Hum Brain Mapp, vol. 33, no. 8, pp. 1914–1928, Jul. 2011.