Method for Auto-Calibrate Projector and Color-Depth Systems for Spatial Augmented Reality Applications

Authors: R. Estrada, A. Henriquez, R. Becerra, C. Laguna

Abstract:

Spatial Augmented Reality is a variation of Augmented Reality where the Head-Mounted Display is not required. This variation of Augmented Reality is useful in cases where the need for a Head-Mounted Display itself is a limitation. To achieve this, Spatial Augmented Reality techniques substitute the technological elements of Augmented Reality; the virtual world is projected onto a physical surface. To create an interactive spatial augmented experience, the application must be aware of the spatial relations that exist between its core elements. In this case, the core elements are referred to as a projection system and an input system, and the process to achieve this spatial awareness is called system calibration. The Spatial Augmented Reality system is considered calibrated if the projected virtual world scale is similar to the real-world scale, meaning that a virtual object will maintain its perceived dimensions when projected to the real world. Also, the input system is calibrated if the application knows the relative position of a point in the projection plane and the RGB-depth sensor origin point. Any kind of projection technology can be used, light-based projectors, close-range projectors, and screens, as long as it complies with the defined constraints; the method was tested on different configurations. The proposed procedure does not rely on a physical marker, minimizing the human intervention on the process. The tests are made using a Kinect V2 as an input sensor and several projection devices. In order to test the method, the constraints defined were applied to a variety of physical configurations; once the method was executed, some variables were obtained to measure the method performance. It was demonstrated that the method obtained can solve different arrangements, giving the user a wide range of setup possibilities.

Keywords: Color depth sensor, human computer interface, interactive surface, spatial augmented reality.

References:

[1] M. K. Bekele, R. Pierdicca, E. Frontoni, E. S. Malinverni, y J. Gain, «A Survey of Augmented, Virtual, and Mixed Reality for Cultural Heritage», J Comput Cult Herit, vol. 11, n.o 2, p. 7:1–7:36, mar. 2018, doi: 10.1145/3145534.
[2] O. Bimber y R. Raskar, Spatial augmented reality: merging real and virtual worlds. AK Peters/CRC Press, 2005.
[3] P. Chen, X. Liu, W. Cheng, y R. Huang, «A review of using Augmented Reality in Education from 2011 to 2016», en Innovations in smart learning, Springer, 2017, pp. 13–18.
[4] H. Gonzalez-Jorge et al., «Metrological comparison between Kinect I and Kinect II sensors», Measurement, vol. 70, pp. 21–26, 2015.
[5] T. Shareef, «5 best cross-platform game engines for rising game developers».
[6] G. B. García, O. D. Suarez, J. L. E. Aranda, J. S. Tercero, I. S. Gracia, y N. V. Enano, Learning image processing with opencv. Packt Publishing Ltd, 2015.
[7] S. Garrido-Jurado, R. Munoz-Salinas, F. J. Madrid-Cuevas, y R. Medina-Carnicer, «Generation of fiducial marker dictionaries using mixed integer linear programming», Pattern Recognit., vol. 51, pp. 481–491, 2016.
[8] F. J. Romero-Ramirez, R. Muñoz-Salinas, y R. Medina-Carnicer, «Speeded up detection of squared fiducial markers», Image Vis. Comput., vol. 76, pp. 38–47, 2018.
[9] S. Voida, M. Podlaseck, R. Kjeldsen, y C. Pinhanez, «A study on the manipulation of 2D objects in a projector/camera-based augmented reality environment», en Proceedings of the SIGCHI conference on Human factors in computing systems, 2005, pp. 611–620.
[10] T. Motta, M. Loaiza, L. Soares, y A. Raposo, «Projection Mapping for a Kinect-Projector System», en 2014 XVI Symposium on Virtual and Augmented Reality, 2014, pp. 200–209.
[11] S. A. Coons, «Surface patches and B-spline curves», en Computer Aided Geometric Design, Elsevier, 1974, pp. 1–16.
[12] «Best Fitting Plane given a Set of Points». https://math.stackexchange.com/questions/99299/best-fitting-plane-given-a-set-of-points.