PrimeSense: Beyond Natal

The fact that the NITE middleware isn't part of Natal demonstrates that some of the really crazy stuff we've seen with it, particularly in how it maintains its multi-point skeletons even when limbs disappear outside of the field of view, is all Microsoft hoodoo.

"Project Natal: the magic is totally Microsoft," says Maizels in admiration. "What they did to the performance, to the robustness of the solution... we have to salute them."

"Being the biggest software company in the world, they did a phenomenal job on the software," adds Berenson. "It's totally their solution."

One interesting element that came up during our discussions is that while it's certain that PrimeSense tech is within Natal, the precise implementation of it remains unknown. Its reference camera does look uncannily similar to the Microsoft concept we've seen to date though.

"We cannot comment on what Natal is doing with the technology," warns Aviad Maizels. "We changed a lot to fit their requirements."

Perhaps he is referring to the removal of the SoC within the camera in order to lower costs on Natal production. This has now been repurposed into a software solution within the Xbox 360's Xenon CPU. Estimates of 10 to 15 per cent of total system power have been mooted for handling the 50MB motion control libraries, while other, more pessimistic sources have indicated that an entire core (or at least one hardware thread) of the triple-core system is reserved purely for Natal.

Technical changes apart, clearly the combination of the 3D camera with state-of-the-art software from a company that boasts one of the largest R&D budgets in the world basically means that PrimeSense's very first consumer-level product is a massive opportunity to show the world its vision for the future of user interfaces.

"We wanted big exposure to be in the trusted hands of a company that we value very much, like Microsoft," explains Adi Berenson. "We think it is the ultimate platform to expose this technology to the world. Shortly after, you'll see more and more rollouts of the technology in other domains."

Aviad Maizels mentions that the first implementations of the PrimeSense tech outside of gaming should be seen in 2011, after the Microsoft launch of Project Natal towards the end of 2010. So what can we expect? The gesture-controlled media centre promotional video on the first page of this feature gives some idea of how the Israeli company see the "zcam" changing the way that we interact with consumer electronics.

"What we believe is that if you give people a very good, intuitive, seamless interface then the willingness to adopt new technology is drastically enhanced and that is what PrimeSense wants to do in every aspect of consumer technology," shares Adi Berenson.

"So we're starting with gaming, then we'll move to other aspects of the living room. We'll go to mobility, we'll go to automotive... we really believe that in the future you'll see a nature interface being adopted everywhere in everyone's life. That's the vision."

"We wanted to have machines that understand us seamlessly," adds Aviad Maizels. "It should be a platform that sees but is not seen. In terms of automotive, if you're a little bit 'wide', it can adjust the airbag. It can see you so it can adjust your car seat. If can warn you if you're tired. It should be seamlessly integrated with other technologies."

The performance capabilities of the PrimeSensor vary according to context. As it has to support all gaming conditions, Natal might well have issues discerning fingers in its skeletal structures. However, put the camera in a car and the tight, enclosed cockpit ensures that a much higher degree of sensibility can be discerned. At close range, the PrimeSensor can even distinguish the nuances of your facial expression.

"One of the unique benefits of our technology is our relatively flexible ability to create from the same core technology and core components different flavours of the design," explains Adi Berenson.

"We can easily play with parameters like resolution, range, field of view and so forth in order to adapt it to different scenarios, all using the same core components and core technology."

The Light Coding technology means that PrimeSense can adapt its design quickly and easily with a range of parts, giving the firm a competitive advantage over other companies using time of flight tech.

"With time of flight, companies need to develop their own CMOS sensor. They spend a lot of time creating a certain resolution... creating another flavour is a hard task. We can buy off the shelf," explains Berenson

"There are many vendors offering different sensors from VGA up to 10 megapixels. We can just pick and create new design flavours. Using standard components not only allows you an easier ramp-up to mass production but is also giving you a lot of flexibility in design flavours and that's what's allowing us to operate in different domains."

Returning to the subject of implementation of the system for gaming, it's clear that Project Natal is just the first generation of this kind of technology. Microsoft itself has confirmed that it scan rate is limited to 30 frames per second, and according to Sony's R&D team, the inability of zcam technology thus far to track finger movement was a key reason they went with its wand approach for PlayStation Move.

Being a first generation product, you can expect that future iterations of Natal will improve the performance significantly.

"Technology is highly adaptable. A few years ago we only had VGA sensors in the mobile world. Now we have three megapixels, five megapixels... we get more resolution," explains Aviad Maizels. "The same kind of thing is happening to all different kinds of technology, including of course 3D sensing. You can expect higher frame rates, better resolution, better ranges."

Buoyed by the huge opportunity Natal represents for their business, the PrimeSense team are massively optimistic about the future going forward.

"We want to lead the natural interaction trend that we believe will take over our lives," explains Adi Berenson. "Ten years from now we want it to be natural for people to interact intuitively with their devices. We want people to know that PrimeSense did it and that PrimeSense is doing it."

In the here and now, the newly released data on the make-up of the reference design of the PrimeSensor offers up some intriguing insight into the potential performance of Project Natal. How will the final product stack up? Digital Foundry will be attending E3 to find out.