A Portable Cognitive Tool for Engagement Level and Activity Identification

Authors: T. Teo, S. W. Lye, Y. F. Li, Z. Zakaria

Abstract:

Wearable devices such as Electroencephalography (EEG) hold immense potential in the monitoring and assessment of a person’s task engagement. This is especially so in remote or online sites. Research into its use in measuring an individual's cognitive state while performing task activities is therefore expected to increase. Despite the growing number of EEG research into brain functioning activities of a person, key challenges remain in adopting EEG for real-time operations. These include limited portability, long preparation time, high number of channel dimensionality, intrusiveness, as well as level of accuracy in acquiring neurological data. This paper proposes an approach using a 4-6 EEG channels to determine the cognitive states of a subject when undertaking a set of passive and active monitoring tasks of a subject. Air traffic controller (ATC) dynamic-tasks are used as a proxy. The work found that using a developed channel reduction and identifier algorithm, good trend adherence of 89.1% can be obtained between a commercially available brain computer interface (BCI) 14 channel Emotiv EPOC+ EEG headset and that of a carefully selected set of reduced 4-6 channels. The approach can also identify different levels of engagement activities ranging from general monitoring, ad hoc and repeated active monitoring activities involving information search, extraction, and memory activities.

Keywords: Neurophysiology, monitoring, EEG, outliers, electroencephalography.

References:

[1] J. Bryson and L. Andres, "Covid-19 and rapid adoption and improvisation of online teaching: curating resources for extensive versus intensive online learning experiences", Journal of Geography in Higher Education, vol. 44, no. 4, pp. 608-623, 2020. Available: 10.1080/03098265.2020.1807478.
[2] W. Bao, "COVID ‐19 and online teaching in higher education: A case study of Peking University", Human Behavior and Emerging Technologies, vol. 2, no. 2, pp. 113-115, 2020. Available: 10.1002/hbe2.191.
[3] P. Kumari, L. Mathew, and P. Syal, “Increasing trend of wearables and multimodal interface for Human Activity Monitoring: A Review,” Biosensors and Bioelectronics, vol. 90, pp. 298–307, 2017.
[4] R. Yuvaraj, S. W. Lye, and H. J. Wee, “A real time neurophysiological framework for general monitoring awareness of air traffic controllers,” 2020 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE), 2020.
[5] H. Kiiski, M. Bennett, L. M. Rueda‐Delgado, F. R. Farina, R. Knight, R. Boyle, D. Roddy, K. Grogan, J. Bramham, C. Kelly, and R. Whelan, “EEG spectral power, but not theta/beta ratio, is a neuromarker for adult ADHD,” European Journal of Neuroscience, vol. 51, no. 10, pp. 2095–2109, 2020.
[6] S. Puma, N. Matton, P.-V. Paubel, É. Raufaste, and R. El-Yagoubi, “Using Theta and Alpha Band Power to assess cognitive workload in multitasking environments,” International Journal of Psychophysiology, vol. 123, pp. 111–120, 2018.
[7] J. LaRocco, M. D. Le, and D.-G. Paeng, “A systemic review of available low-cost EEG headsets used for drowsiness detection,” Frontiers in Neuroinformatics, vol. 14, 2020.
[8] W. Li, Q.-chang He, X.-min Fan, and Z.-min Fei, “Evaluation of driver fatigue on two channels of EEG Data,” Neuroscience Letters, vol. 506, no. 2, pp. 235–239, 2012.
[9] G. Li, B.-L. Lee, and W.-Y. Chung, “Smartwatch-based wearable EEG system for driver drowsiness detection,” IEEE Sensors Journal, vol. 15, no. 12, pp. 7169–7180, 2015.
[10] S. Majumder, B. Guragain, C. Wang, and N. Wilson, “On-board drowsiness detection using EEG: Current status and future prospects,” 2019 IEEE International Conference on Electro Information Technology (EIT), 2019.
[11] P. Aricò et al., "Adaptive Automation Triggered by EEG-Based Mental Workload Index: A Passive Brain-Computer Interface Application in Realistic Air Traffic Control Environment", Frontiers in Human Neuroscience, vol. 10, 2016. Available: 10.3389/fnhum.2016.00539.
[12] G. Borghini et al., "EEG-Based Cognitive Control Behaviour Assessment: an Ecological study with Professional Air Traffic Controllers", Scientific Reports, vol. 7, no. 1, pp. 1-16, 2017. Available: 10.1038/s41598-017-00633-7.
[13] D. Dasari, G. Shou and L. Ding, "ICA-Derived EEG Correlates to Mental Fatigue, Effort, and Workload in a Realistically Simulated Air Traffic Control Task", Frontiers in Neuroscience, vol. 11, 2017. Available: 10.3389/fnins.2017.00297.
[14] F. Dehais et al., "Monitoring Pilot’s Mental Workload Using ERPs and Spectral Power with a Six-Dry-Electrode EEG System in Real Flight Conditions", Sensors, vol. 19, no. 6, p. 1324, 2019. Available: 10.3390/s19061324.
[15] W. Matcha, N. A. Uzir, D. Gasevic, and A. Pardo, “A systematic review of empirical studies on learning analytics dashboards: A self-regulated learning perspective,” IEEE Transactions on Learning Technologies, vol. 13, no. 2, pp. 226–245, 2019.
[16] A. Pardo, F. Han, and R. A. Ellis, “Combining University student self-regulated learning indicators and engagement with online learning events to predict academic performance,” IEEE Transactions on Learning Technologies, vol. 10, no. 1, pp. 82–92, 2017.
[17] T. Alotaiby, F. El-Samie, S. Alshebeili and I. Ahmad, "A review of channel selection algorithms for EEG signal processing", EURASIP Journal on Advances in Signal Processing, vol. 2015, no. 1, 2015. Available: 10.1186/s13634-015-0251-9.
[18] M. Z. Baig, N. Aslam, and H. P. Shum, “Filtering techniques for channel selection in motor imagery EEG applications: A survey,” Artificial Intelligence Review, vol. 53, no. 2, pp. 1207–1232, 2019.
[19] J. Zhang, M. Chen, S. Zhao, S. Hu, Z. Shi, and Y. Cao, “ReliefF-based EEG sensor selection methods for emotion recognition,” Sensors, vol. 16, no. 10, p. 1558, 2016.
[20] A. Gevins et al., "Towards measurement of brain function in operational environments", Biological Psychology, vol. 40, no. 1-2, pp. 169-186, 1995. Available: 10.1016/0301-0511(95)05105-8.
[21] S. Saha, K. A. Mamun, K. Ahmed, R. Mostafa, G. R. Naik, S. Darvishi, A. H. Khandoker, and M. Baumert, “Progress in brain computer interface: Challenges and opportunities,” Frontiers in Systems Neuroscience, vol. 15, 2021.
[22] P. Arico et al., "Reliability over time of EEG-based mental workload evaluation during Air Traffic Management (ATM) tasks", 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 7242 - 7245, 2015. Available: 10.1109/embc.2015.7320063.
[23] M. Bennasar, Y. Hicks and R. Setchi, "Feature selection using Joint Mutual Information Maximisation", Expert Systems with Applications, vol. 42, no. 22, pp. 8520-8532, 2015. Available: 10.1016/j.eswa.2015.07.007.
[24] L. Fang et al., "Feature selection method based on mutual information and class separability for dimension reduction in multidimensional time series for clinical data", Biomedical Signal Processing and Control, vol. 21, pp. 82-89, 2015. Available: 10.1016/j.bspc.2015.05.011.
[25] I. Guyon and A. Elisseeff, “An introduction to variable and feature selection,” Journal of Machine Learning Research, vol. 3, no. 3, pp. 1157–1182, 2003.
[26] J. Foy and M. Foy, "Dynamic Changes in EEG Power Spectral Densities During NIH-Toolbox Flanker, Dimensional Change Card Sort Test and Episodic Memory Tests in Young Adults", Frontiers in Human Neuroscience, vol. 14, 2020. Available: 10.3389/fnhum.2020.00158.
[27] A. Gupta et al., "On the Utility of Power Spectral Techniques with Feature Selection Techniques for Effective Mental Task Classification in Noninvasive BCI", IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 51, no. 5, pp. 3080-3092, 2021. Available: 10.1109/tsmc.2019.2917599.
[28] D. Bansal and R. Mahajan, EEG-based brain-computer interfaces. Elsevier, 2019, pp. 21-71.
[29] R. Alam, H. Zhao, A. Goodwin, O. Kavehei and A. McEwan, "Differences in Power Spectral Densities and Phase Quantities Due to Processing of EEG Signals", Sensors, vol. 20, no. 21, p. 6285, 2020. Available: 10.3390/s20216285.
[30] T. O. Zander, C. Kothe, S. Jatzev, and M. Gaertner, “Enhancing human-computer interaction with input from active and passive brain-computer interfaces,” Brain-Computer Interfaces, pp. 181–199, 2010.
[31] J. M. ten Have, “The development of the NLR ATC Research Simulator (narsim): Design philosophy and potential for ATM research,” Simulation Practice and Theory, vol. 1, no. 1, pp. 31–39, 1993.
[32] Emotiv, EPOC+ User Manual, 2021. Available: https://emotiv.gitbook.io/epoc-user-manual/
[33] N.-H. Liu, C.-Y. Chiang, and H.-C. Chu, “Recognizing the degree of human attention using EEG signals from mobile sensors,” Sensors, vol. 13, no. 8, pp. 10273–10286, 2013.
[34] C.-J. Peng, Y.-C. Chen, C.-C. Chen, S.-J. Chen, B. Cagneau, and L. Chassagne, “An EEG-based attentiveness recognition system using Hilbert–Huang transform and support vector machine,” Journal of Medical and Biological Engineering, vol. 40, no. 2, pp. 230–238, 2019.
[35] H.H., Jasper, “International Federation of Societies for electroencephalography and clinical neurophysiology,” Electroencephalography and Clinical Neurophysiology, vol. 10, no. 2, p. 367, 1958.
[36] F. Faul, E. Erdfelder, A. Buchner and A. Lang, "Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses", Behavior Research Methods, vol. 41, no. 4, pp. 1149-1160, 2009. Available: 10.3758/brm.41.4.1149.
[37] J. Medeiros, R. Couceiro, G. Duarte, J. Durães, J. Castelhano, C. Duarte, M. Castelo-Branco, H. Madeira, P. de Carvalho, and C. Teixeira, “Can EEG be adopted as a neuroscience reference for assessing software programmers’ cognitive load?,” Sensors, vol. 21, no. 7, p. 2338, 2021.
[38] M. Teplan, “Fundamentals of EEG measurement,” Measurement Science Review, vol. 2, no. 2, pp. 1–11, 2002.
[39] D. F. Stegeman and M. J. Van Putten, “Recording of neural signals, neural activation, and signal processing,” Oxford Textbook of Clinical Neurology, pp. 37–45, 2017.
[40] P. J. Charles, R. J. Sclabassi and Mingui Sun, "Non-Gaussian modeling of EEG data," Proceedings of the First Joint BMES/EMBS Conference. 1999 IEEE Engineering in Medicine and Biology 21st Annual Conference and the 1999 Annual Fall Meeting of the Biomedical Engineering Society (Cat. N), 1999, pp. 1023 vol.2-, doi: 10.1109/IEMBS.1999.804176.
[41] Á. Costa, R. Salazar-Varas, A. Úbeda and J. Azorín, "Characterization of Artifacts Produced by Gel Displacement on Non-invasive Brain-Machine Interfaces during Ambulation", Frontiers in Neuroscience, vol. 10, 2016. Available: 10.3389/fnins.2016.00060.
[42] R. Srinivasan, “Methods to improve spatial resolution of EEG,” International Journal of Bioelectromagnetism, vol. 1, no. 1, pp. 102–111, 1988.
[43] E. Düzel et al., "Task-related and item-related brain processes of memory retrieval", Proceedings of the National Academy of Sciences, vol. 96, no. 4, pp. 1794-1799, 1999. Available: 10.1073/pnas.96.4.1794.
[44] H. Magen, T.-A. Emmanouil, S. A. McMains, S. Kastner, and A. Treisman, “Attentional demands predict short-term memory load response in posterior parietal cortex,” Neuropsychologia, vol. 47, no. 8-9, pp. 1790–1798, 2009.
[45] M. F. Rushworth, T. Paus, and P. K. Sipila, “Attention systems and the organization of the human parietal cortex,” The Journal of Neuroscience, vol. 21, no. 14, pp. 5262–5271, 2001.
[46] M. Behrmann, J. Geng and S. Shomstein, "Parietal cortex and attention", Current Opinion in Neurobiology, vol. 14, no. 2, pp. 212-217, 2004. Available: 10.1016/j.conb.2004.03.012.
[47] A. C. Nobre, J. T. Coull, V. Walsh, and C. D. Frith, “Brain activations during visual search: Contributions of search efficiency versus feature binding,” NeuroImage, vol. 18, no. 1, pp. 91–103, 2003.
[48] H. Kojima and T. Suzuki, “Hemodynamic change in occipital lobe during visual search: Visual attention allocation measured with NIRS,” Neuropsychologia, vol. 48, no. 1, pp. 349–352, 2010.
[49] C. Colby and M. Goldberg, "SPACE and attention in parietal cortex", Annual Review of Neuroscience, vol. 22, no. 1, pp. 319-349, 1999. Available: 10.1146/annurev.neuro.22.1.319.
[50] S. Kastner and L. G. Ungerleider, “Mechanisms of visual attention in the human cortex,” Annual Review of Neuroscience, vol. 23, no. 1, pp. 315–341, 2000.
[51] H. Stemmann and W. A. Freiwald, “Evidence for an attentional priority map in inferotemporal cortex,” Proceedings of the National Academy of Sciences, vol. 116, no. 47, pp. 23797–23805, 2019.
[52] R. H. Grabner, A. C. Neubauer and E. Stern, “Superior performance and neural efficiency: The impact of intelligence and expertise,” Brain Research Bulletin, 69, 422–439, 2006.
[53] R. H. Grabner, E. Stern and A. C. Neubauer, “When intelligence loses its impact: neural efficiency during reasoning in a familiar area,” International Journal of Psychophysiology, 49, 89¬–98, 2003.
[54] D. Nussbaumer, R. H. Grabner and E. Stern, “Neural efficiency in working memory tasks: The impact of task demand,” Intelligence, 50, 196–208, 2015.