A Psychophysiological Evaluation of an Effective Recognition Technique Using Interactive Dynamic Virtual Environments

Authors: Mohammadhossein Moghimi, Robert Stone, Pia Rotshtein

Abstract:

Recording psychological and physiological correlates of human performance within virtual environments and interpreting their impacts on human engagement, ‘immersion’ and related emotional or ‘effective’ states is both academically and technologically challenging. By exposing participants to an effective, real-time (game-like) virtual environment, designed and evaluated in an earlier study, a psychophysiological database containing the EEG, GSR and Heart Rate of 30 male and female gamers, exposed to 10 games, was constructed. Some 174 features were subsequently identified and extracted from a number of windows, with 28 different timing lengths (e.g. 2, 3, 5, etc. seconds). After reducing the number of features to 30, using a feature selection technique, K-Nearest Neighbour (KNN) and Support Vector Machine (SVM) methods were subsequently employed for the classification process. The classifiers categorised the psychophysiological database into four effective clusters (defined based on a 3-dimensional space – valence, arousal and dominance) and eight emotion labels (relaxed, content, happy, excited, angry, afraid, sad, and bored). The KNN and SVM classifiers achieved average cross-validation accuracies of 97.01% (±1.3%) and 92.84% (±3.67%), respectively. However, no significant differences were found in the classification process based on effective clusters or emotion labels.

Keywords: Virtual Reality, effective computing, effective VR, emotion-based effective physiological database.

Digital Object Identifier (DOI): doi.org/10.5281/zenodo.1130437

References:

[1] Paul Cairns, Anna Cox, Nadia Berthouze, Samira Dhoparee, and Charlene Jennett, "Quantifying the experience of immersion in games," in Workshop on the Cognitive Science of Games and Gameplay, Vancouver, 2006.
[2] Anton Nijholt, Bos, Danny Plass-Oude, and Boris Reuderink, "Turning shortcomings into challenges: Brain–computer interfaces for games," Entertainment Computing 1, vol. 1, no. 2, pp. 85–94, 2009.
[3] Mohammadhossein Moghimi, Robert J. Stone, Pia Rotshtein, and Neil Cooke, "Influencing Human Affective Responses to Dynamic Virtual Environments," Teleoperators and Virtual Environments, vol. 25, no. 2, November 2016.
[4] Gunnar Ahlberg et al., "Proficiency-based virtual reality training significantly reduces the error rate for residents during their first 10 laparoscopic cholecystectomies," The American Journal of Surgery, vol. 193, no. 6, pp. 797–804, 2007.
[5] Michael Zyda, "From visual simulation to virtual reality to games," Computer, vol. 38, no. 9, pp. 25 - 32, 2005.
[6] Neal E. Seymour et al., "Virtual Reality Training Improves Operating Room Performance," Annals of Surgery, vol. 236, no. 4, pp. 458–464, 2002.
[7] Nicole E Mahrer and Jeffrey I. Gold, "The Use of Virtual Reality for Pain Control: A Review," Current pain and headache reports, vol. 13, no. 2, pp. 100-109, 2009.
[8] Hunter G. Hoffman, Jason N. Doctor, David R. Patterson, Gretchen J. Carrougher, and Thomas A. Furness, "Virtual reality as an adjunctive pain control during burn wound care in adolescent patients," Pain, vol. 18, no. 2, pp. 305–309, 2000.
[9] A A Rizzo et al., "Virtual environments for the assessment of attention and memory processes: the virtual classroom and office," Virtual Reality, pp. 3-12, 2002.
[10] David Jack et al., "Virtual Reality-Enhanced Stroke Rehabilitation David," IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 9, no. 3, pp. 308-318, 2001.
[11] Thomas D. Parsons and Albert A. Rizzo, "Affective outcomes of virtual reality exposure therapy for anxiety and specific phobias: A meta-analysis," Journal of Behavior Therapy and Experimental Psychiatry, vol. 39, no. 3, pp. 250-261, 2008.
[12] JoAnn Difede et al., "Virtual Reality Exposure Therapy for the Treatment of Posttraumatic Stress Disorder Following," The Journal of clinical psychiatry, vol. 68, no. 11, pp. 1639-1647, 2007.
[13] Marian Joels, Zhenwei Pu, Olof Wiegert, Melly S. Oitzl, and Harm J. Krugers, "Learning under stress: how does it work?," Trends in Cognitive Sciences, vol. 10, no. 4, pp. 152-158, 2006.
[14] Mohammad Soleymani, Maja Pantic, and Thierry Pun, "Multimodal emotion recognition in response to videos," IEEE Transactions on Affective Computing, vol. 3, no. 2, pp. 211-223, April 2012.
[15] Mohammad Soleymani, Sadjad Asghari-Esfeden, Yun Fu, and Maja Pantic, "Analysis of EEG signals and facial expressions for continuous emotion detection," IEEE Transactions on Affective Computing, vol. 7, no. 1, pp. 17 - 28, May 2015.
[16] Mohammad Soleymani, Sander Koelstra, Ioannis Patras, and Thierry Pun, "Continuous emotion detection in response to music videos," in 2011 IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011), Santa Barbara, 2011, pp. 803 - 808.
[17] Sander Koelstra et al., "DEAP: a Database for Emotion Analysis Using Physiological Signals," Affective Computing, IEE Transactions, vol. 3, no. 1, pp. 18-31, January 2012.
[18] Christos A. Frantzidis et al., "On the Classification of Emotional Biosignals Evoked While Viewing Affective Pictures: An Integrated Data-Mining-Based Approach for Healthcare Applications," IEEE Transactions on Information Technology in Biomedicine , vol. 14, no. 2, pp. 309-318, March 2010.
[19] Peter J. Lang, Mark K. Gereenwald, Margaret M. Bradley, and Alfons O. Hamm, "Looking at pictures: affective, facial, visceral, and behavioral reactions," Psychophysiology, vol. 30, no. 3, pp. 261-73, May 1993.
[20] Kazuhiko Takahashi and Akinori Tsukaguchi, "Remarks on Emotion Recognition from Multi-Modal Bio-Potential Signals," in Systems, Man and Cybernetics, 2003. IEEE International Conference, vol. 2, Yamaguchi Univ., Japan, 2003, pp. 1654-1659.
[21] Mimma Nardelli, Gaetano Valenza, Alberto Greco, Antonio Lanata, and Enzo Pasquale Scilingo, "Recognizing emotions induced by affective sounds through heart rate variability," IEEE Transactions on Affective Computing, vol. 6, no. 4, pp. 385-394, October 2015.
[22] Herbon Antje, Christian Peter, Lydia Markert, Elke Van Der Meer, and Jörg Voskamp, "Emotion studies in HCI-a new approach," in HCI International Conference, vol. 1, Las Vegas, 2005.
[23] Christos D. Katsis, Nikolaos Katertsidis, George Ganiatsas, and Dimitrios I. Fotiadis, "Toward Emotion Recognition in Car-Racing Drivers: A Biosignal Processing Approach," IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans, vol. 38, no. 3, pp. 502-512, May 2008.
[24] Dongrui Wu et al., "Optimal Arousal Identification and Classification for Affective Computing Using Physiological Signals: Virtual Reality Stroop Task," IEEE Transactions on Affective Computing, vol. 1, no. 2, pp. 109-118, July 2010.
[25] Alejandro Rodríguez, Beatriz Rey, Miriam Clemente, Maja Wrzesien, and Mariano Alcañiz, "Expert Systems with Applications Assessing brain activations associated with emotional regulation during virtual reality mood induction procedures," Expert Systems with Applications, vol. 42, no. 3, pp. 1699-1709, February 2015.
[26] James A. Russell, "A Circumplex Model of Affect," Journal of Personality and Social Psychology, vol. 39, no. 6, pp. 1161-1178, 1980.
[27] Albert Mehrabian, "A Semantic Space For Nonverbal Behaviour," Consulting and Clinical Psychology, vol. 35, no. 2, pp. 248-257, October 1970.
[28] Paul Ekman and Wallace V. Friesen, Unmasking The Face. Los Altos: ISHK, 2003, ISBN: 0-13-938175-9.
[29] Guillaume Chanel, Cyril Rebetez, Mireille Bétrancourt, and Thierry Pun, "Emotion assessment from physiological signals for adaptation of game difficulty," IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans , vol. 41, no. 6, pp. 1052-1063, November 2011.
[30] Kevin P. Murphy, "Introduction," in Machine Learning: A Probabilistic Perspective.: MIT Press, 2012, vol. 1, pp. 2-12.
[31] Michael Steinbach, Vipin Kumar Pang-Ning Tan, "Introduction to Data Mining," in Introduction to Data Mining.: Addison-Wesley, 2005, pp. 65-73.
[32] T. Tamura, H. Miike K. Nakajima, "Monitoring of heart and respiratory rates by photoplethysmography using a digital filtering technique," Medical Engineering & Physics, vol. 18, no. 5, pp. 365–372, 1996.
[33] Saeid Sanei and Jonathon Chambers, "Brain Rhythms," in EEG Signal Processing. West Sussex: John Wiley & Sons, 2009, pp. 10-13.
[34] William H. Press, Saul A. Teukolsky, William T. Vetterling, and Brian P. Flannery, "Chapter 12 - Fast Fourier Transform," in Numerical recipes in Fortran (The art of scientific computing).: Cambridge University Press, 1992, vol. 1, pp. 490-529.
[35] Fredric J. Harris, "On the use of windows for harmonic analysis with the discrete Fourier transform," Proceedings of the IEEE, vol. 68, no. 1, pp. 51-83, January 1978.
[36] Hanchuan Peng, Fuhui Long, and Chris Ding, "Feature Selection Based on Mutual Information: Criteria of Max-Dependency, Max-Relevance, and Min-Redundancy," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 27, no. 8, pp. 1226-1238, August 2005.
[37] Kevin P. Murphy, "Entropy," in Machine Learning: A Probabilistic Perspective: MIT Press, 2012, vol. 1, p. 57.
[38] Kevin P. Murphy, "Support Vector Machines," in Machine Learning A Probabilistic Perspective: MIT Press, 2012, vol. 1, pp. 498-507.
[39] Kevin P. Murphy, "A Simple non-Parametric Classifier: K-Nearest Neighbors," in Machine Learning A Probabilistic Perspective.: MIT Press, 2012, vol. 1, pp. 16-18.
[40] M. Murugappan et al., "Time-Frequency Analysis of EEG Signals for Human Emotion Detection," in Springer-Verlag, Berlin, 2008, pp. 262-265.
[41] M. Rizon, M. Murugappan, R. Nagarajan, and S. Yaacob, "Asymmetric Ratio and FCM based Salient Channel Selection for Human Emotion Detection Using EEG," WSEAS Transactions on Signal Processing, vol. 4, no. 10, pp. 596-603, October 2008.
[42] Kevin P. Murphy, "Logistic Regression," in Machine Learning a Probabilistic Perspective: MIT Press, 2012, vol. 1, pp. 21-22.