Improved Feature Extraction Technique for Handling Occlusion in Automatic Facial Expression Recognition

Authors: Khadijat T. Bamigbade, Olufade F. W. Onifade

Abstract:

The field of automatic facial expression analysis has been an active research area in the last two decades. Its vast applicability in various domains has drawn so much attention into developing techniques and dataset that mirror real life scenarios. Many techniques such as Local Binary Patterns and its variants (CLBP, LBP-TOP) and lately, deep learning techniques, have been used for facial expression recognition. However, the problem of occlusion has not been sufficiently handled, making their results not applicable in real life situations. This paper develops a simple, yet highly efficient method tagged Local Binary Pattern-Histogram of Gradient (LBP-HOG) with occlusion detection in face image, using a multi-class SVM for Action Unit and in turn expression recognition. Our method was evaluated on three publicly available datasets which are JAFFE, CK, SFEW. Experimental results showed that our approach performed considerably well when compared with state-of-the-art algorithms and gave insight to occlusion detection as a key step to handling expression in wild.

Keywords: Automatic facial expression analysis, local binary pattern, LBP-HOG, occlusion detection.

Digital Object Identifier (DOI): doi.org/10.5281/zenodo.2643966

References:

[1] Ahmed, F., Bari, H. & Hossain, E. (2014) Person-independent facial expression recognition based on compound local binary pattern (CLBP). Int. Arab J. Inf. Technol., 11(2): 195-203.
[2] Dalal, N. & Triggs, B. (2005, June) Histograms of oriented gradients for human detection. In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference, 1: 886-893. IEEE.
[3] Dhall, A., Goecke, R., Lucey, S. & Gedeon, T. (2011, November) Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark. In Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference, pp. 2106-2112. IEEE.
[4] Donato, G., Bartlett, M. S., Hager, J. C., Ekman, P. and Sejnowski, T. J. (1999) Classifying facial actions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(10): 974–989.
[5] Hablani, R., Chaudhari, N. & Tanwani, S. (2013) Recognition of facial expressions using local binary patterns of important facial parts. International Journal of Image Processing (IJIP), 7(2): 163-170.
[6] Jayasumana, S., Hartley, R., Salzmann, M., Li, H. & Harandi, M. (2015) Kernel methods on Riemannian manifolds with Gaussian RBF kernels. IEEE transactions on pattern analysis and machine intelligence, 37(12): 2464-2477.
[7] Lyons, M., Akamatsu, S., Kamachi, M. & Gyoba, J. (1998, April). Coding facial expressions with gabor wavelets. In Automatic Face and Gesture Recognition, 1998. Proceedings. Third IEEE International Conference, pp. 200-205. IEEE.
[8] Mayya, V., Pai, R.M., & Pai, M.M. (2016) Automatic facial expression recognition using DCNN. Procedia Computer Science, 93: 453-461.
[9] Mistry, V.J. & Goyani, M.M. (2013) A literature survey on facial expression recognition using global features. Int. J. Eng. Adv. Technol, 2(4): 653-657.
[10] Ojala, T., Pietikainen, M. and Maenpaa, T. (2002) “Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns,” IEEE Transaction on Pattern Analysis Analysis and Machine Intelligence, 24(7): 971-987.
[11] Otberdout, N., Kacem, A., Daoudi, M., Ballihi, L. & Berretti, S. (2018) Deep Covariance Descriptors for Facial Expression Recognition. arXiv preprint arXiv:1805.03869.
[12] P. Ekman and W. V. Friesen (1978) ”Facial Action Coding System: A Technique for the Measurement of Facial Movement,” Consulting Psychologists Press.
[13] P. Lucey, J. Cohn, T. Kanade, J. Saragih, Z. Ambadar and I. Matthews (2010) The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. In CVPR4HB10, p 94–101.
[14] P. Viola and M. Jones (2001) Robust real-time object detection In International Workshop on Statistical and Computational Theories of Vision - Modelling, Learning, Computing and Sampling.
[15] Tian, Y. & Chen, S. (2012) Understanding effects of image resolution for facial expression analysis. Journal of Computer Vision and Image Processing.
[16] Zhang, L., Verma, B., Tjondronegoro, D. & Chandran, V. (2018) Facial Expression Analysis under Partial Occlusion: A Survey. ACM Computing Surveys (CSUR), 51(2): 25.