Improving the Performance of Deep Learning in Facial Emotion Recognition with Image Sharpening

Authors: Ksheeraj Sai Vepuri, Nada Attar

Abstract:

We as humans use words with accompanying visual and facial cues to communicate effectively. Classifying facial emotion using computer vision methodologies has been an active research area in the computer vision field. In this paper, we propose a simple method for facial expression recognition that enhances accuracy. We tested our method on the FER-2013 dataset that contains static images. Instead of using Histogram equalization to preprocess the dataset, we used Unsharp Mask to emphasize texture and details and sharpened the edges. We also used ImageDataGenerator from Keras library for data augmentation. Then we used Convolutional Neural Networks (CNN) model to classify the images into 7 different facial expressions, yielding an accuracy of 69.46% on the test set. Our results show that using image preprocessing such as the sharpening technique for a CNN model can improve the performance, even when the CNN model is relatively simple.

Keywords: Facial expression recognition, image pre-processing, deep learning, CNN.

References:

[1] A. Lonare, and S. V. Jain. “A Survey on Facial Expression Analysis for Emotion Recognition”. International Journal of Advanced Research in Computer and Communication Engineering 2.12
[2] C. Shan, S. Gong, and P. Meowan, "Facial expression recognition based on local binary patterns: A comprehensive study", Image & Vision Computing, vol. 27, no. 6, pp. 803-816, 2009.
[3] C. Padgett, and G. Cottrell. 1996. “Representing face images for emotion classification”. In Proceedings of the 9th International Conference on Neural Information Processing Systems (NIPS'96). MIT Press, Cambridge, MA, USA, 894–900.
[4] G. B. Huang, H. Lee, and E. Learned-Miller. Learning hierarchical representations for face verification with convolutional deep belief networks. In Proc. CVPR, 2012. 4
[5] 24.K. Liu, M. Zhang, and Z. Pan. 2016. “Facial Expression Recognition with CNN Ensemble”. In International Conference on Cyberworlds. 163–166.
[6] 25.M. Shin, M. Kim and D. Kwon, "Baseline CNN structure analysis for facial expression recognition," 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), New York, NY, 2016, pp. 724-729, doi: 10.1109/ROMAN.2016.7745199.
[7] M. Dantone, J. Gall, G. Fanelli, and L. J. V. Gool. “Real-time facial feature detection using conditional regression forests”. In Proc. CVPR, 2012. 1, 2
[8] M. F. Valstar, M. Mehu, B. Jiang, M. Pantic, and K. Scherer. “Metaanalysis of the first facial expression recognition challenge”. IEEE TSMC-B, 42(4):966–979, 2012.
[9] S. Jain, C. Hu, and J. K. Aggarwal. “Facial expression recognition with temporal modeling of shapes”. In ICCV Workshops, pages 1642–1649, 2011.
[10] S. Rani, V. Tejaswi, B. Rohitha, and B. Akhil,”Pre filtering techniques for face recognition based on edge detection algorithm. J. Eng. Technol. 13–218 (2017)
[11] J. Kaur, and A. Sharma, “Review Paper 0n Edge Detecti0n Techniques in Digital lmage Pr0cessing” lnternati0nal J0urnal 0f lnn0vati0ns & Advancement in C0mputer Science ljiacs lssn 2347 – 86l6, V0lume 5, lssue ll, N0vember 2ol6.
[12] J. Prasad, and G. P. Chourasiya, and N.S. Chauhan, "Face detection using color based segmentation and edge detection," International Journal of Computer Applications (0975-8887), voL72, no.16, pp.49-54, June 2013.
[13] M. Abo-Zahhad, R. Gharieb, S. Ahmed, and A. Donko.. Edge Detection with a Preprocessing Approach. Journal of Signal and Information Processing. (2014) 5. 123-134. 10.4236/jsip.2014.54015.
[14] M. Ali, and D. Clausi, "Using the Canny edge detector for feature extraction and enhancement of remote sensing images," IGARSS 2001. Scanning the Present and Resolving the Future. Proceedings. IEEE 2001 International Geoscience and Remote Sensing Symposium (Cat. No.01CH37217), Sydney, NSW, Australia, 2001, pp. 2298-2300 vol.5, doi: 10.1109/IGARSS.2001.977981.
[15] M. Pantic, M. Valstar, R. Rademaker and L. Maat, "Web-based database for facial expression analysis", IEEE International Conference on Multimedia and Expo (ICME), pp. 1-5, 2005.
[16] M. Kamachi, M. Lyons, and J. Gyoba, The japanese female facial expression (jaffe) database, 1998.
[17] T. Kanade, J. F. Cohn and Y. Tian, "Comprehensive database for facial expression analysis", Fourth IEEE International Conference on Automatic Face and Gesture Recognition, pp. 46-53, 2000.
[18] L. Nanni, A. Lumini, and S. Brahnam.. “Ensemble of texture descriptors for face recognition obtained by varying feature transforms and preprocessing approaches”. Applied Soft Computing. 61. 10.1016/j.asoc.2017.07.057. (2017)
[19] R. Cui, M. Liu, M. Liu. Facial Expression Recognition Based on Ensemble of Mulitple CNNs
[C] Chinese Conference on Biometric Recognition. Springer International Publishing, 2016:511-518.
[20] S. Li, and A. Jain, “Handbook of Face Recognition”, Springer Publishing Company Incorporated, 2011
[21] V. Bettadapura.” Face expression recognition and analysis: the state of the art”. arXiv preprint arXiv:1203.6722(2012)
[22] Y. Liang, S. Liao, L. Wang, and B. Zou, "Exploring regularized feature selection for person specific face verification," 2011 International Conference on Computer Vision, Barcelona, 2011, pp. 1676-1683, doi: 10.1109/ICCV.2011.6126430.
[23] Z. Xie, Y. Li, X. Wang, W. Cai, J. Rao, and Z. Liu, "Convolutional Neural Networks for Facial Expression Recognition with Few Training Samples," 2018 37th Chinese Control Conference (CCC), Wuhan, 2018, pp. 9540-9544, doi: 10.23919/ChiCC.2018.8483159.
[24] Z. Yu, and C. Zhang.”Image based Static Facial Expression Recognition with Multiple Deep Network Learning”. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction (ICMI '15). Association for Computing Machinery, New York, NY, USA, 435–442.
[25] Y. Tang. 2013. Deep Learning using Linear Support Vector Machines. Computer Science (2013).
[26] S. Zhou, Y. Liang, J. Wan. 2016. Facial Expression Recognition Based on Multi-scale CNNs. In Biometric Recognition. Springer International Publishing, 128–135.
[27] X. Wang, J. Huang, J. Zhu, M. Yang, and F. Yang. “Facial expression recognition with deep learning”. In: Proceedings of the 10th International Conference on Internet Multimedia Computing and Service. New York, NY, USA: ACM, 2018. (ICIMCS ‘18), p: 10:1 10:4.