Virtual Reality

Start-up PrimeSense’s 3D technology may revolutionize much more than just the gaming industry

By ZIV HELLMAN
primesense311 (photo credit: Primesense LTD.)
primesense311
(photo credit: Primesense LTD.)
IT IS THE END OF THE WORKING day. You leave your desk and head for the parking lot. When you are within a few feet of your car, it recognizes you and opens its door.
As you settle into the driver’s seat, the steering wheel, mirrors, air bag, and climate control adjust to your body. At the entrance to your home, you needn’t fumble with keys because the front door also recognizes you and opens accordingly. Similarly, the air conditioner fires up automatically to your preset temperature and humidity settings.
With a hand gesture, you give the coffee machine an order to start percolating, as the TV switches itself on, asking whether you would prefer to watch the program you usually do at that hour, or instead accept a teleconference call from a friend. Choosing the call, you view your friend’s place in three-dimensional detail so fine that it feels like you are actually inside her flat.
Ten years into the 21st century, the previous paragraph still reads like it was written by a science fiction writer in the 1960s describing “the future.” But it may very soon be a commonplace reality, if the 3D sensors being developed by Tel Aviv-based start-up PrimeSense live up to the promise they have already exhibited. Those sensors can track thousands of points on a human body (and other objects), creating a detailed 3D map of a room or scene, essentially mimicking what the human brain does, for computer use. Once machines are capable of knowing in three-dimensional detail what is happening around them, they can be programmed to respond and interact with an environment, shattering the bounds of the computer keyboard and screen as the only channels of interaction.
PrimeSense’s technology has so impressed Microsoft that the software giant announced in March that it is contracting with PrimeSense to provide the 3D-sensing technology for its next generation “Kinect” gaming device.
But don’t be fooled by the game connection into thinking that 3D sensors are only a toy. PrimeSense is keen to stress that what it is producing is a technology platform that will be available for developers around the world to be used in ways not yet imaginable.
With 3D viewing technology galloping forward at lightning speed, we are another step closer to truly immersive and interactive virtual reality environments.
“People intuitively understand games because they are interactive by design,” says Uzi Breier, PrimeSense’s Executive VP for Global Sales, explaining why the games industry is the first target audience for its technology to The Report. “But we are talking about a revolution in lifestyle, going far beyond games and living-room entertainment.
“Interactive 3D environments can be used for medical and surgical applications, and in physical rehabilitation centers. They can be used in sterile ‘clean rooms’ where objects need to be manipulated, but direct human touch is forbidden – by remote reading of human hand and body actions, a robot in the clean room can be controlled to do whatever is needed. The potential robotic applications alone are immense.”
AT THE HEART OF PRIMESense’s system is a large webcam-like device that contains an infrared sensor along with a standard video camera, which connects to computer chip technology developed to process the 3D information fed into it by the sensor.
Infrared sensors have long been familiar and ubiquitous for motion detection in everything from home alarm systems to supermarket doors. But those sensors for the most part apply the simplest of binary logic: they detect whether or not motion has occurred, and no more. PrimeSense’s innovation is to gather a large amount of real-time information via infrared sensors, and then to feed that information into algorithms for three-dimensional processing, a field that has undergone tremendous progress in recent years.
Using computers for tracking motion in three dimensions is not a new idea – it has been used for industrial needs and advanced film special effects for some time – but until now required subjects to don clumsy gloves or even full-body suits connected to powerful computer processors. PrimeSense’s system will enable this technology to be readily available, while dispensing with the need for expensive accessories.
It is easy to see why the computer games industry is PrimeSense’s first target market; it is a high-revenue market comprised of technologically savvy customers who will readily appreciate the new “control-free” technology.
Consider the evolution of computer games.
The first generations of games required continual “hands-on” control through consoles and joysticks. Amajor leap was attained by systems such as Nintendo’s Wii, introduced in 2006, that enabled remote interaction using a handheld device. The remote control approach spawned new ideas in gaming and gave players the opportunity to use the remote devices as pretend tennis rackets and golf clubs, among other applications, with the game display reacting accordingly.
PrimeSense’s system carries this idea much further, to what it terms “full body-based gaming.” Where the Wii device can track two points in three dimensions, for a total of what is termed six degrees of freedom, PrimeSense’s technical literature promises to track over 300,000 degrees of freedom.
That translates into full-body gesture control and real-time motion capture, free of wearable items or controllers. A player can stand in front of the sensor and pretend to swing a racquet in his hand, or he can even bring in his own gear – a skateboard, a real tennis racquet – and it will be seamlessly captured by the PrimeSensor system and integrated into the game. Game developers can choose how much of the hundreds of thousands of points of data PrimeSense supplies for use in their games, from detailed motion-capture muscles, to tracking a simple “stick-figure” skeleton of the body constructed of a few dozen points, in order to ease the processing burden.
The capacity of detailed machine tracking of motion can enable human-machine interaction in games beyond what has been known hitherto. For example, it should be possible to create an automated home personal trainer that will track a user’s body positions and dynamics, offering pointers for improved performance. This could be a boon for aspiring amateur tennis players, golfers, prize fighters, and so on, as well as the home user who wants to get the most out of a workout.
THE NEXT MARKET, AFTER games, in PrimeSense’s sight is home entertainment, which the company interprets as including everything from intensive web interaction to couch-potato channel hopping. At the heart of this application is what PrimeSense calls “control by gesture,” in which users control consumer electronics with their bare hands.
“There is no doubt that the PC and TV are converging,” asserts Breier. “The young generation does not simply punch channels 10, 24, 42, and so forth. They want to select their own content and pick what they interact with, as they do on YouTube. This is a revolution in the attitude to access content, from a ‘lean back’ paradigm to a ‘lean forward’ paradigm.”
The idea of integrating computers and television screens isn’t new. WebTV and similar initiatives in Silicon Valley date back to the earliest days of the web takeoff in the mid-1990s, but in most homes the PC screen and the TV screen remain resolutely separate in use and perception. PrimeSense hopes that its technology will bring the two together.
“We believe the integration did not happen until today because the user interface [for accessing the Internet via the TV] has been inconvenient and unnatural,” says Breier. “I travel quite a lot in my work, and I have been in hotel rooms with Internet access through the TV, but I never use it because the interface [is clumsy]. What we are offering goes beyond GUI [graphic user interface] to NUI [natural user interface].
With our technology, with simple and intuitive hand gestures, you can control any machine.”
PrimeSense has a demo that shows how its sensors enable “device-free” control of appliances. In it, a person sitting on a livingroom sofa and facing a large-screen TV is able to command the screen to do his bidding with a flick of the wrist. The mouse, touchpad and remote control become obsolete, as PrimeSense’s system tracks the user’s hand motions in the air and reacts. It can simulate a touch screen, as a user “touches” an imaginary glass wall just in front of him.
The demo also showcases the flexibility of the system. It can be set to ignore everything beyond a pre-set distance from the sensor, or to track only hand movements. It can also be set to follow all objects in a three-dimensional scene or only people, even taking high-quality video shots and then inserting the video into a scene or a PC screen. One can imagine that when married with 3D viewing systems, teleconferencing incorporating the 3D sensors may enable participants to interact with each other in 3D virtual environments.
PrimeSense is counting on developers to take the potential for new ideas and applications in directions that have not yet been considered. Towards that end, it announced early this year the initiation of an “Experience Partner Program” to encourage collaborative development. The program provides qualifying parties access to PrimeSense’s latest sensor designs and software, along with joint promotional opportunities.
“We supply the chips and the sensors to software developers, who will create the applications, and we expect them to exceed our imaginations,” says Breier. “With this technology available to any software developer in the world to work with, who knows where they will take it. It is like the [Apple] iPhone, which has become what it is because it is a platform for thousands of developers to create applications. I recently came across a mosquito repelling app for the iPhone (it mimics the sound of natural predators of mosquitoes, and it works). I promise you that Steve Jobs, when he created the iPhone, did not have that application in mind.”
PRIMESENSE WAS FOUNDED nearly five years ago by five founders “with a passion for games.” It has been funded by premier venture capital firms, such as Genesis, Gemini and Racan, and presently employs 120 people, with branch offices in the US, Japan and Taiwan, in addition to its headquarters in Tel Aviv. The company received a big boost, in both sales potential and market visibility, with Microsoft’s decision to incorporate PrimeSense technology in its latest gaming devices, expected to be marketed in late 2010.
“PrimeSense has delivered an important component to the technology,” said Ilan Spillinger, vice president of Microsoft’s Xbox 360 hardware, in a statement issued to the press in March, “helping us deliver revolutionary controller-free entertainment experiences in the living room.”
After focusing on business opportunities in the games and living-room entertainment niches, PrimeSense is now looking to expand to possible partnerships in the mobile telephone and robotics industries. Another intriguing possibility is 3D infrared mapping for biometric identification.
Biometric identification is big business in defense and security circles, and it has begun to spill over to some home uses; laptops now routinely include an option to restrict access by way of fingerprint identification.
But most of the biometric identifying applications on the market require a user to make an active effort in order to be identified, such as placing a finger in a specified location. With 3D infrared biometrics, machines can potentially identify people by their presence in a room, just as humans recognize each other by face. This can be accomplished by using infrared scanners to construct detailed 3D maps of human faces for comparison with databases of known faces. Doors and appliances can be programmed to open and function as soon as an authorized individual is close enough to a sensor.
Electronic signatures of contracts can be authenticated by scanning the face of the individual tapping at the keyboard.
Breier, a former president of Alverien, a telecommunications company, former CEO of Optibase, which marketed video streaming technology, and prior to that CEO of software firm Emblaze, furrows his brow when asked whether the ease of biometric identification afforded by 3D infrared sensors can become a double-edged sword. Is there not a danger that people’s faces can be stolen from such biometric databases, for identity theft scams? Breier admits that this is a subject that requires careful consideration, and then responds: “We are a technology provider. Application developers will need to deal with those issues and resolve them. We believe that our devices can give ample privacy protection. But there is no escaping progress.”
He clearly believes that PrimeSense is on to something potentially revolutionary. “We are talking about a new situation,” he says. “Until now, you had to learn how to work with machines. Now, instead of learning the machine, the machine will learn you. That is what will make interacting with it natural, and what will make those interactions do things they could never do before.”