Microsoft IllumiRoom Concept Taps Kinect's Augmented Reality Potential

By Wesley Fenlon

Your room is the TV. But you TV is still the TV. It's complicated.

At CES 2013, Microsoft showed off a new Microsoft Research project called IllumiRoom, not as a product, but as a proof-of-concept--a possible way to do something very, very different with the Kinect. IllumiRoom combines a Kinect sensor with a projector to take in the layout of a room and then project video from a game onto the walls surrounding the TV. The system could extend the content of a game beyond the boundaries of the television screen, effectively turning an entire part of a room into a giant screen. But the augmented reality potential of the technology is much cooler.

Microsoft Research just released a video overviewing IllumiRoom's potential uses. Check it out for yourself below. Extending a full game's environment outside the TV looks jarring and distracting in some cases--due to the differences in tone and brightness, the two don't really blend. The AR effects, on the other hand, look pretty cool. In one example, the video shows only gunfire and explosions in a first-person shooter extending onto the walls, and the way they fly out of the TV frame and into the living room works really well. A similar gimmick makes it snow all around the TV.

The Research team also released a white paper describing the technology of IllumiRoom and giving multiple examples of how it could be implemented. Here's an excerpt describing how IllumiRoom could be implemented:

"Our vision for a fully developed IllumiRoom system includes an ultra-wide field of view device sitting on the user’s coffee table, projecting over a large area surrounding the television. The device would be connected wirelessly to a next generation gaming console as a secondary display...

Our current proof-of-concept prototype uses a commodity wide field of view projector (InFocus IN126ST) and a Microsoft Kinect for Windows sensor. The prototype is limited by the field of view of the Kinect and projector (~57º horizontal FOV). Therefore, the system is mounted above and behind the user’s head, as they are seated on a couch in front of the television.

The Kinect sensor captures the color and geometry of the scene, and the system renders the illusions using the acquired depth map. Careful calibration of the system is required in order for the illusions to tightly match the onscreen content and the physical environment. The calibration of the IllumiRoom system is fully automatic, determining the relative pose of the projector to the depth sensor and the position of the television in the projector image. Therefore, setup only requires that the projector and depth camera are placed such that they cover the area surrounding the TV."

The use of the Kinect allows IllumiRoom to conform to furniture or other objects in a room, which means it isn't limited to a flat, white surface as much as other projectors. But there are other limitations--you'll still need a dark room, bundling a projector with a console would be expensive, and developers will have to rethink or reprogram their games to fully take advantage of IllumiRoom. On that note, though, Microsoft's video also demonstrates some ways IllumiRoom can still interpret game motion and show that on the walls in what they call Peripheral Flow.

Even if this proof-of-concept isn't destined to show up at Microsoft's May 21st Xbox reveal, we hope Microsoft Research sticks with it. IllumiRoom could be in the running for weirdest video game accessory ever if it becomes a weird product.