Console Gaming: The Lag Factor

The COD masters commissioned arch-prototyper Ben Heckendorn to create a latency controller board which lights LEDs in conjunction with button presses on the joypad. The screen is still videoed with a 60FPS camera, but the board replaces the controller in the shot. In effect, the most inaccurate element of the procedure - judging when the button has been pressed - is replaced with digital precision via Heckendorn's custom creation.

"We commissioned Ben to make us the light board after a programmer saw me spending a lot of time filming myself pressing buttons in front of a CRT to test input latency," says Infinity Ward's Drew McCoy. "He, being a programmer, was obviously frustrated that such an imprecise method was used to test something that he and the rest of the engineers here at Infinity Ward spend a great deal of time and energy on - reducing input latency."

And so, with that issue resolved, the only inaccuracy now comes down to the camera itself. It won't be entirely synced with the refresh rate of the screen, so there will be ghosting, but in a worst-case scenario, results can only be affected by one frame, and that's good enough to be getting on with.

In common with games development studios including BioWare and Wolfenstein multiplayer makers Endrant Studios, I got in contact with Ben and bought one of these devices. His handmade designs aren't cheap and I paid ticket price out of my own pocket, but when the kit arrived, I knew it was worth it, and the value of this piece of equipment to the likes of Infinity Ward must be incalculable.

And from my perspective, at the very least, next time there's an argument about unresponsive controls, it can be put to the test. Alas, Ben's controller is based only on the wireless version of the 360 joypad, so it can't be used to measure PS3 games (although wired 360 pads can be interfaced with the Sony console, the internal set-up is different, so just as incompatible with Heckendorn's design as the SixAxis and DualShock 3 are). That being the case, key responsiveness tests - such as comparisons of Resident Evil 5 on PS3 and 360 - will have to wait for now.

First order of business was to try my new toy with a game with known lag issues. Enter Grand Theft Auto IV. Frame-rate analysis of the captured clip runs alongside the camera shot. The idea here is remarkably straightforward - in the Face-Off features I often remark that frame-rate drops impact responsiveness. Here we can see the game running in optimum conditions and also when it is under load, with a sizeable performance drop.

So, at 30FPS we get baseline performance of eight frames/133ms, but in the second clip where the game has dropped to 24FPS, there is a clear 12 frames/200ms delay between me pulling the trigger, and Niko beginning the shotgun firing animation. That's 200ms plus the additional delay from your screen. Ouch.

Next up, another test for one of the Xbox 360's most popular games, and with it, an opportunity to address a developer concern with another recent Eurogamer article. In the Digital Foundry versus Project Natal analysis, I measured the latency of the prototype controller. It came in at around 200ms (which I suspect might be down to a PC emulating Natal's internal processor during the demo, combined with the LCD lag) and I put this into context by mentioning the Killzone 2 measurement and also the original Gamasutra Halo 3 test - Mick West reckons it's between 133ms to 166ms depending on whether you're shooting or jumping. After the article was published, Bungie got in touch with Eurogamer editor Tom Bramwell and said that the actual lag was significantly lower. With the new controller monitor safely installed in the Digital Foundry lair, I could test this myself.

So in this test, my own results differed from Mick West's - my tests seem to show that Halo 3's firing rate is at the minimum for 30FPS: 100ms, with a measurement of 150ms for jumping. The original Gamasutra article seems to be calling for a uniform response rate, but I'm fairly sure that differing rates for different actions is totally deliberate on the developer's behalf. Think about it: you would expect instant response from pulling a trigger, but something more strenuous, like hefting a bloody great armoured suit into the air, isn't going to be quite so fast to achieve.

So how about looking at Neversoft's own game? Guitar Hero: Aerosmith should do quite nicely as a test case. Rhythm-action games like this and Rock Band, rely on ultra-low latencies between man and machine, to the point where calibration options within the game are introduced in an attempt to remove the effects of lag completely.

The Gamasutra article pegged Guitar Hero III at the very lowest possible latency: 50ms. However, I couldn't get the same level of response in any of my tests in the Aerosmith edition. I ended up with four frames as an average, again in line with the assumption that this is, by and large, the fastest latency that a 60FPS game can achieve.

Back in the Burnout Paradise technical retrospective, Criterion said it was aiming for 50ms latency, so I thought I'd give that a go next, along with a range of other 60FPS titles.

Forza Motorsport 2 claims to poll the joypad 360 times a second to ensure the smoothest response, but that still comes in at four frames, just like West's original Ridge Racer 7 test. Similarly with Criterion's Burnout Paradise, and Bizarre Creations' epic Geometry Wars 2. A hint of frame blending in the latter might suggest that Bizarre could have hit the mythical 50ms barrier... further testing would be required to see for sure, but I'm fairly sure it's an artifact of the camera.