Gesture Recognition by Data Fusion of Time-of-Flight and Color Cameras

Authors: Piercarlo Dondi, Luca Lombardi, Marco Porta

Abstract:

In the last years numerous applications of Human- Computer Interaction have exploited the capabilities of Time-of- Flight cameras for achieving more and more comfortable and precise interactions. In particular, gesture recognition is one of the most active fields. This work presents a new method for interacting with a virtual object in a 3D space. Our approach is based on the fusion of depth data, supplied by a ToF camera, with color information, supplied by a HD webcam. The hand detection procedure does not require any learning phase and is able to concurrently manage gestures of two hands. The system is robust to the presence in the scene of other objects or people, thanks to the use of the Kalman filter for maintaining the tracking of the hands.

Keywords: Gesture recognition, human-computer interaction, Time-of-Flight camera.

Digital Object Identifier (DOI): doi.org/10.5281/zenodo.1075751

References:

[1] T. Oggier, M. Lehmann, K. R., M. Schweizer, M. Richter, P. Metzler, G. Lang, F. Lustenberger, and N. Blanc, "An all-solid-state optical range camera for 3D real-time imaging with sub-centimeter depth resolution (SwissRanger)," in Proceeding of SPIE Vol. 5249, 2003, pp. 634-645.
[2] A. Kolb, E. Barth, R. Koch, and R. Larsen, "Time-of-Flight Cameras in Computer Graphics," Computer Graphics Forum, vol. 29, no. 1, pp. 141-159, 2010.
[3] P. Dondi, L. Lombardi, and M. Porta, "Human-Computer Interaction through Time-of-Flight and RGB cameras," in Proceedings of ICIAP 2011, 16th International Conference on Image Analysis and Processing, vol. 2. Springer, September 2011, pp. 89-98.
[4] R. Reulke, "Combination of distance data with high resolution images," in Proceedings of IEVM06, Image Engeeniring and Vision Metrology, 2006.
[5] S. Ghobadi, O. Loepprich, K. Hartmann, and O. Loffeld, "Hand segmentation using 2D/3D images," in Proceedings of Image and Vision Computing 07, December 2007, pp. 64-69.
[6] S. E. Ghobadi, O. E. Loepprich, F. Ahmadov, J. Bernshausen, K. Hartmann, and O. Loffeld, "Real time hand based robot control using 2D/3D images," in Proceedings of the 4th International Symposium on Advances in Visual Computing, Part II, ser. ISVC -08. Berlin, Heidelberg: Springer-Verlag, 2008, pp. 307-316.
[7] P. Breuer, C. Eckes, and S. Mller, "Hand gesture recognition with a novel IR time-of-flight range camera: a pilot study," in Proceedings of 3rd International Conference on Computer vision/computer graphics collaboration techniques (MIRAGE-07), 2007, pp. 247-260.
[8] Z. Li and R. Jarvis, "Visual interpretation of natural pointing gestures in 3d space for human-robot interaction," in Proceedings of Control Automation Robotics Vision (ICARCV), 2010 11th International Conference on, December 2010, pp. 2513-2518.
[9] A. Treskunov, S. Kim, and S. Marti, "Range camera for simple behind display interaction," in Proceedings of MVA2011 IAPR Conference on Machine Vision Applications, Nara, Japan, June 2011, pp. 160-163.
[10] E. Kollorz, J. Penne, J. Hornegger, and A. Barke, "Gesture recognition with a time of flight camera," Int. J. Intell. Syst. Technol. Appl., vol. 5, pp. 334-343, November 2008.
[11] M. B. Holte, T. B. Moeslund, and P. Fihl, "View invariant gesture recognition using the csem swissranger sr-2 camera," Int. J. Intell. Syst. Technol. Appl., vol. 5, pp. 295-303, November 2008.
[12] M. Van den Bergh and L. Van Gool, "Combining RGB and ToF cameras for real-time 3D hand gesture interaction," in Applications of Computer Vision (WACV), 2011 IEEE Workshop on, January 2011, pp. 66-72.
[13] M. Haker, M. Bhme, T. Martinetz, and E. Barth, "Deictic gestures with a time-of-flight camera," in Proceedings of Gesture in Embodied Communication and Human-Computer Interaction 8th International Gesture Workshop, GW 2009, S. Kopp and I. Wachsmuth, Eds., January 2009, pp. 110-121.
[14] T. Oggier, B. Bttgen, F. Lustenberger, G. Becker, B. Regg, and A. Hodac, "Swissranger SR3000 and first experiences based on miniaturized 3DTOF cameras," in Proceedings, 1st Range Imaging Research Day. Springer, September 2005, pp. 97-108.
[15] N. Haubner, U. Schwanecke, R. Drner, S. Lehmann, and J. Luderschmidt, "Recognition of Dynamic Hand Gestures with Time-of-Flight Cameras," in Proceedings of ITG/GI Workshop on Self-Integrating Systems for Better Living Environments 2010 (Sensyble Workshop), 2010, pp. 33-39.
[16] S. Soutschek, J. Penne, J. Hornegger, and J. Kornhuber, "3-D gesturebased scene navigation in medical imaging applications using timeof- flight cameras," in Proceedings of Computer Vision and Pattern Recognition Workshops, 2008. CVPRW -08. IEEE Computer Society Conference on, June 2008, pp. 1-6.
[17] J. Penne, S. Soutschek, L. Fedorowicz, and J. Hornegger, "Robust real-time 3D time-of-flight based gesture navigation," in Proceedings of Automatic Face Gesture Recognition, 2008. FG -08. 8th IEEE International Conference on, September 2008, pp. 1-2.
[18] P. Dondi and L. Lombardi, "Fast real-time segmentation and tracking of multiple subjects by time-of-flight camera," in Proceedings of VISAPP 2011, 6th International Conference on Computer Vision Theory and Applications, March 2011, pp. 582-587.