Natural User Interface Adapter: Enabling Natural User Interface for Non-Natural User Interface Applications

Authors: Vijay Kumar Kolagani, Yingcai Xiao

Abstract:

Adaptation of Natural User Interface (NUI) has been slow and limited. NUI devices like Microsoft’s Kinect and Ultraleap’s Leap Motion can only interact with a handful applications that were specifically designed and implemented for them. A NUI device just can’t be used to directly control millions of applications that are not designed to take NUI input. This is in the similar situation like the adaptation of color TVs. At the early days of color TV, the broadcasting format was in RGB, which was not viewable by blackand-white TVs. TV broadcasters were reluctant to produce color programs due to limited viewership. TV viewers were reluctant to buy color TVs because there were limited programs to watch. Color TV’s breakthrough moment came after the adaptation of NTSC standard which allowed color broadcasts to be compatible with the millions of existing black-and-white TVs. This research presents a framework to use NUI devices to control existing non-NUI applications without reprogramming them. The methodology is to create an adapter to convert input from NUI devices into input compatible with that generated by CLI (Command Line Input) and GUI (Graphical User Interface) devices. The CLI/GUI compatible input is then sent to the active application through the operating system just like any input from a CLI/GUI device to control the non-NUI program that the user is controlling. A sample adapter has been created to convert input from Kinect to keyboard strokes, so one can use the input from Kinect to control any applications that take keyboard input, such as Microsoft’s PowerPoint. When the users use the adapter to control their PowerPoint presentations, they can free themselves from standing behind a computer to use its keyboard and can roam around in front of the audience to use hand gestures to control the PowerPoint. It is hopeful such adapters can accelerate the adaptation of NUI devices.

Keywords: command line input, graphical user interface, human computer interaction, natural user interface, NUI adapter