GTA IV Face-Off Frame-Rates and FAQ

Following on from last Thursday's Grand Theft Auto IV: PS3 vs. Xbox 360 Special, author Rich Leadbetter told me that he'd found an even more precise way of measuring average frame-rates and wanted to update you on this. He also thought it would be a good idea to answer a few of the more commonly asked questions about his methodology. The Frame Rate Tests section of the piece has now been tweaked accordingly, and here's Rich's commentary on it:

The Xbox 360 vs PlayStation 3 GTAIV face-off certainly caused some controversy across forums far and wide, partly because it contained what is most likely the first ever frame-rate test for a console game; tests we're going to continue in future face-off coverage.

Most of the controversy seemed to surround the fact that 360's results would be influenced by the inclusion of frames subject to v-lock screen tear. Over the weekend I worked with notable tech experts, including the now infamous 'Quaz51' in refining the frame-rate detector. It's now extremely sensitive to the point where it can detect frame-rate even on a static screen (i.e. a car moving at 0mph).

More than that, we can now detect how many torn frames there are in any clip.

The GTA piece has now been updated with the new data, but in the interests of openness, here are the revised results, alongside the old ones.

Test1
Old 360: 31.990fps
Old PS3: 26.460fps
New 360: 31.627fps (3.77% torn frames in full capture)
New PS3: 26.504fps

Test2
Old 360: 28.624fps
Old PS3: 23.452fps
New 360: 29.233fps (4.26% torn frames in full capture)
New PS3: 23.989fps

Test3
Old 360: 35.252fps
Old PS3: 29.041fps
New 360: 35.662fps (2.37% torn frames in full capture)
New PS3: 29.523fps

Test4
Old 360: 26.076fps
Old PS3: 26.081fps
New 360: 26.015fps (6.07% torn frames in full capture)
New PS3: 25.803fps

Test5
Old 360: 26.712fps
Old PS3: 23.781fps
New 360: 26.507fps (4.9% torn frames in full capture)
New PS3: 23.781fps

Test6
Old 360: 33.798fps
Old PS3: 28.313fps
New 360: 33.713fps (2.35% torn frames in full capture)
New PS3: 28.313fps

A few common questions and answers then, to explain the results and to answer some questions that have arisen since the article was published.

1. What does the torn frame percentage actually mean?
   Regardless of the game performance, the video output of the Xbox 360 (and indeed PS3) pumps out 60 frames per second. So in the case of Test1, for example, over the course of the entire clip, 3.77 percent of the frames were torn. So it's not 3.77 percent of the measured frame-rate, but of the complete 60Hz output of the console's video processor.
2. Why bother including this stat?
   Screen tear can have a huge impact on picture quality. For example on one test I did on the Race Driver GRID demo, 360 ran at a constant 30fps while PS3 averaged at 28.76fps. Not so much difference there you might think, until you factor in that almost 40 percent of the 60Hz output of the PS3 version was torn frames, versus 0 percent on 360.
3. Why haven't the 360 scores been adjusted downwards by the torn frame percentages?
   For the same reason that the v-locked PS3 results have also changed - the scanner itself is now far more precise. Weirdly, the imprecision in the old version mostly compensated for the v-lock issue, so by and large the results haven't really changed that much.
4. If GTA IV on Xbox 360 isn't v-locked, why is the amount of torn frames so low?
   Actually it dynamically switches between two different v-lock settings (versus one on PS3) only dropping out of sync very rarely - around two percent to six percent of the time. More than that, the tearing will usually only appear in areas of the screen you don't see (the overscan area of your display), making it even less of an issue.
5. How accurate can the tests be bearing in mind changes in lighting, weather, traffic, people etc?
   The whole point of multiple tests is to produce an average, an overall trend if you will. The captures were taken with the same weather conditions and at the same time of day on both consoles. The only variable would be people and cars on the streets in the outdoor scenes. For a truly 1:1 comparison, Test1 is of the entire intro sequence which should be identical on both machines. Test3 is indoors and again should be like-for-like.

Check out the Grand Theft Auto IV: PS3 vs. Xbox 360 Special for more on how the two versions compare.