Sony: Why we turned down Kinect

Somebody sent out a video internally where there was a guy playing Joy Ride like this [holds up hands]. He's not moving. He's just staying like this, and he comes in third. And I was like... [whistles]. That's harsh.

But I guess so. I haven't actually analysed the games too intensely. If you just think about how you'd do it? If I want to go forward, what can I do? Well, I can put my hand forward. OK, that's pretty good. And I can move my hand to the left and right to go left and right. You get a joystick with your hand.

OK, that works, but is that something you want to do? Well, maybe for the first 10 minutes, or maybe for a short action. You can run in place, but nobody wants to run in place, and it's actually very hard to tell how fast the user wants to run.

So you lose these subtle and quick controls. We felt, what's the advantage you get? You get a magical experience. EyeToy felt magical as well, just because you have no controller. You can move and something happens. But you lose that sense of connectedness to the game. You get a ton of latency.

You sacrifice some of that for the advantage of it being intuitive. That's always a trade off. Ideally you can find an interface that is both intuitive and powerful, but it's very rare you find something that's catch-all. Even if Kinect was highly accurate and it could detect the tiniest motions, still for a sword game you'd want something in your hand because it feels weird to swing around just your hand. Your tactile feedback isn't correct. It doesn't trigger these wrist feelings. People are very physical in that sense.

For some actions, I can think of each of my finger touchings [on the palm of the hand] be a button, but that's much slower than pressing a regular button, and what's the advantage of that? So buttons are still very good for a lot of different applications. And that's just assuming perfect accuracy. In reality, the 3D cameras we surveyed and what Kinect ended up using, they're 320x240 resolution, so when you're talking about tracking fingers, or even tracking things like the rotations of your hand, you're working with 10x10 pixels. It's very hard to get anything useful out of it.

That's the long-winded answer of why we stopped that research. We basically got to a point where we felt we understood the limitations of the tech. Sony as a group wanted to do a motion controller that could work with a broad variety of games.

Motion controller is not quite the right term because what people think of motion control is you swing a gesture and an event triggers. What we were aiming for is a spatial input device, like a mouse, where it could map to a variety of actions.

We weren't trying to replace buttons by gestures. When you do that, you get some intuitiveness. You can swing a tennis racket and the character swings. That's nice. That lets people that don't usually play videogames play videogames.

But what you don't get is a competitive edge. You don't get that I can swing better. You don't have that tracking capability, so you can't analyse the player's motion. If you can do that, then you can do both. You get both an intuitive interface and you get something that's pretty powerful. That's what we're trying to do with Move. We wanted to bring a spatial input device, because that's something the DualShock's not very good at.

For example, if you want to create a level in LittleBigPlanet and you want to draw something over there, you have to slide your cursor there and then you have to rotate. With the Move you can just move it there and rotate the object quickly. It's a spatial task that's much better suited for a spatial controller.

For action games, take Ninja Gaiden, you're going for twitch movements. You want a very short throw analogue stick and buttons. That's better suited for the DualShock.

We realised what games we were good at, and what we were missing. We wanted to fill that space and never necessarily replace controllers. That was the rationale behind Move; get something that's complimentary to the DualShock rather than just try to replace it altogether.

First, it has to be higher resolution, because at this current stage the cooler things you can do you really want some precision. Some of the neat demos we don't want to disclose, we would much rather have the camera be higher-res.

The next big one is the camera must run at least at 60Hz. 30Hz is awful. When we switched from EyeToy 30 to EyeToy 60 it was miraculously better. But some people don't notice that, so that's fine.

There are a couple of technical issues we ran into. For example, rubber is hard to track. While nobody should be wearing a rubber suit, often belts and things make an empty section through your waist, which causes some tracing problems. Dark jeans, fresh denim jeans often completely bug the tracking out. Skirts are hard to track because you just have a single leg instead of two and without higher resolution it's hard to disambiguate. Baggy jeans. Shorts cause issues if you don't have higher res. Certain materials, like polyester.

Basically it's an infrared camera. Infrared light is not some sort of special light. It's just like visible light, but it's not visible. It's a different wavelength. I'm wearing a black shirt and if you look at it with a regular camera, that's black to the camera. The reason is because there's no light coming in from that colour. So there are certain objects that are infrared black. To an infrared light in an infrared camera they appear black. Those are the objects that are hard to track.

There are also some objects that are infrared reflective, just like a mirror is visible light reflective. Those tend to show up bright white on the camera, and then they come up as infinite depth. Those are the funky materials that sometimes just make the tracking bad. I don't know if that can be solved, but some cameras have fewer issues with them and some have more.

I'll be honest with you, most of the time it's covered up by the game itself. Dance Central is a good example. In Dance Central, if you see your silhouette missing parts you know something is going wrong. But Dance Central is clever, and the reason is, I think, because all they're doing is silhouette-matching. I don't think they're actually doing any skeleton-tracking. Skeleton-tracking is the hard thing. That's the neat thing about Kinect, but it's actually the hardest part.

The reason they did silhouette-tracking is because the Harmonix guys worked with EyeToy and they know the best tech is the most reliable tech. So they take the z data from the camera and they just chop it out of depth and feed that into their game.

Some of the more sophisticated Kinect games, like Adventures, try to do skeleton-tracking, and that's when you get things wigging out. That's when you're trying to do something hard. The reason Dance Central is one of the higher-rated games is because it works pretty well.

Background subtraction is one thing the EyeToy developers said was the neat and solid tech out of it. That, we were quite excited about, but we still had this irking feeling that that cost, just to get that, is a bit high.

If you look at Kung-fu Live for the Eye, that does background subtraction as well. It's not as robust due to lighting, but they do a really good job. We always felt we can get better with just visible light to get background subtraction and we didn't need to go to 3D. The really exciting stuff was the skeleton-tracking, and that's what proved to be really hard.

Cost down, 60Hz, fewer issues, higher res. And those are all very achievable, just not at this moment.

You'd have to ask PrimeSense, and you'd have to ask 3DV, and whoever else Microsoft doesn't buy out by the time that...

Of course, there are other solutions we can do. There are stereo cameras, which are also looked at. Lots of other things we can even do on our own as well, so it's not necessarily that we're dependent on that research.

I feel it's a case of early tech. The same thing happened to the Wii. The Wii started out with accelerometers. They hit that point right where accelerometers started to become cheap, but still at that point they weren't very good. When Sony looked at that idea it said, 'We're not so sure about it.'