Mobile Interaction through Video
In 1992, Tani et al. envisioned how users could interact with a real-world device located at a distance through live video. Cameras observed industrial machinery and allowed users to manipulate mechanical switches and sliders over a distance by clicking and dragging within the live video image with a mouse. This was made possible by mapping portions of the video frame to the respective parts of the remote hardware. The system was revolutionary in that it established a particularly direct type of affordance – in many ways similar to the affordance of direct touch.
While the metaphor is still interesting, the environments and usage scenarios have changed since that time. (1) The proliferation of displays on machines and computer systems has turned many spaces into multi-display environments. (2) With the presence of portable computers such as laptops or tablet PCs, the displays within these environments may be rearranged. (3) In these flexible display setups, Tani's fixed camera setup is not necessarily appropriate anymore.
We investigate how to apply "interaction through video" to these new scenarios and to what extent mobile devices can offer the required flexibility. We build on recent advances in mobile augmented reality (such as markerless tracking and camera-based pose estimation) combined with techniques for manipulating objects at a distance (such as distant pointing, input redirection and local portholes from remote displays).
Touch Projector allows users to manipulate content on displays at a distance, including those that would otherwise be unreachable. It further allows users to manipulate devices that are be incapable of touch interaction, such as a wall projection, or a laptop computer. Users aim the device with one hand and then manipulate objects by touching and dragging it in the live video using the other hand. Touch input is "projected" onto the remote display, as if it had occurred on it.
With Touch Projector, users manipulate targets using both hands in concert. The non-dominant hand holds the device and coarsely orients it, while the dominant hand interacts within the reference frame established by the non-dominant hand (cf. toolglass interaction). This combination allows interaction with large displays by moving the entire device (cf. peephole displays) as well as interaction with small displays using touch input. Touch Projector preserves immediate feedback: when content on the target display is changed, users immediately perceive these changes through the live video. This allows for a close connection of action and reaction as both occur on the mobile device.