Spatially Augmented Reality Toolkit

Using projectors as light sources we can transform the surface properties of arbitrarily complex objects. The color of a surface is normally defined by the amount of light reflected/absorbed by the particular pigment/material of the object. In spatially augmented reality we lift the properties of the surface into the light source. I.e. instead of illuminating a red object with white light, we can illuminate a white object with red light.

Rajinder Sodhi and Brett Jones are in the process of developing a toolkit for the development of projection mapping/spatially augmented reality (sar) applications. Our focus is on projection mapping for theater and new media installation pieces using tracking with embedded light sensors.

Currently, they are researching human computer interaction issues with spatially augmented content authoring. They are focusing on improving the content pipeline (which is largely non-existent). Traditional video editing and computer graphics workflows are not designed for animated texturing and easily matching physical and digital models.

For the most recent updates on this research see the research blog, Augmented Engineering.

In this demo we are calibrating a digital model of the box to the physical box using embedded light sensors. Essentially, the projector displays a pattern which photosensors embedded into the box recieve and decode. With this information, we can determine the positions of the sensors in the projectors frame of reference. With enough sensors (at least six) we can determine the parameters of the projector (the lens etc.) along with the rotation and orientation of the box. Then using a game engine, we render a 3D model of the box from the projectors frame of reference. Finally, applying an animated texture to the 3D model of the box, yeilds the demonstration above. This technique is explained extensively by Johnny Chung Lee.

Facebook Stalking (Projection Mapping)

A new media exhibit piece designed to push the boundaries of digital identity. This piece was created with spatially augmented reality/projection mapping techniques, using software developed as part of a larger research project.

As a background, this was presented as a final project in a new media art class. A facebook friend request was sent to everyone in the class in order to research their digital identities. Without their knowledge a video of their facebook photos was constructed and a projection mapped mannequin head was used to comment on their photos.

Note that the video is calibrated to a 3D digital model of the mannequin head. The projection onto the mannequin head is then computed in realtime with the software we developed.