Digital Foundry vs. OnLive

Latency Measurements

It's the $64,000 question. Just how much lag is there in the OnLive experience? The company's founder, Steve Perlman, has made a number of difficult-to-believe claims about the system since its launch at GDC 2009, and one of them has been the measurement of latency he has ascribed to the system.

"The round trip latency from pushing a button on a controller and it going up to the server and back down, and you seeing something change on screen should be less than 80 milliseconds," he told the BBC. "We usually see something between 35 and 40 milliseconds."

Bearing in mind that the fastest latency we have ever seen on a local console system is 50ms (with the PS3 XMB), this quote must either be complete fantasy or else taken out of context. Somewhat. Taken at face value, OnLive is effectively claiming that gameplay over IP can be more responsive than a console game which runs at 60FPS.

Let's consider the whole latency ecosystem here. There are an array of factors that add delay to the response mechanism.

• Local controller-reading lag (client side)
• Internet lag (there and back)
• Game-specific lag
• OnLive-specific lag (including video encoding/decoding)
• Local display lag

So how do you go about testing this? Obviously, Digital Foundry has some experience in measuring latency, as we followed up original work by Neversoft co-founder Mick West in his Gamasutra article and then expanded on it, using the same Ben Heck latency controller board employed by the likes of Infinity Ward in order to get a real-life lag measurement from their games. OnLive has full support for the Xbox 360 controller, meaning our existing Ben Heck tech could be used immediately without modification.

The principle of using the board is very straightforward: point a 60FPS camera at the screen and have the latency controller monitor in the same shot. Whenever a button on the controller is pressed, an LED on the board lights up (they are hard-wired electrically). Then you simply review your camera shot, and count the number of frames between the LED lighting up and action occurring on-screen.

The only downside to this measurement is that it includes the lag of the flat-panel display. To mitigate this as much as possible we used a TN style of monitor with an advertised 5ms response time. Calibrating against the 50ms of the PS3 XMB, we found that the latency was actually just one frame or 16.67ms. So that would be the figure we would deduct from our measurements to give the true value.

Let's start with the good news. In optimum conditions, OnLive lag is impressively low: much lower than we expected and definitely the most promising element of the whole system. If you can get OnLive running at a sustained 60FPS, we measure latency at nine frames, or 150ms. Bearing in mind that many console titles operate at 133ms locally, this is very impressive indeed. Here's some footage of the system running on various games: Unreal Tournament III, Assassin's Creed II and DiRT 2.

Now, there will be some variation from game to game. Just Cause 2 and Batman: Arkham Asylum, for example, appear to run 16ms slower than a great many of the other titles with a base-line of 166ms. Also bear in mind that if OnLive runs at anything less than 60FPS, which it does very frequently, this obviously means that the time taken to beam back the response to your in-game movement can be delayed. We'll cover the performance aspect of OnLive a bit later on in this piece, but it stands to reason that if the server is stalling, your response time is going to increase.

Unreal Tournament III seems to run well and maintains its 150ms response fairly uniformly, but as you can see from the video, Assassin's Creed II can run with the same base-line but can also exceed 200ms. Similarly we see a 166ms to 200ms spread in DiRT 2. It is no surprise that both games feature some quite alarming performance drops from the advertised service standard of 60 frames per second.

So, anything between 150ms to 200ms+ latency for OnLive then. While this is just a few frames away from the current consoles' performance on 30FPS titles, the problem OnLive has is that there is something of a lag threshold. A fair few first-person titles operate at 133ms on console, but few people seem upset with the controls. Contrast with Killzone 2 at 150ms which received plenty of bad press for its "laggy" response.

While there is a tolerable level of latency, there additionally seems to be a cut-off point where it becomes intrusive, and that will vary according to the response requirements of the game plus the perception level of player himself, of course. A game like Batman: Arkham Asylum will "feel" less laggy compared to something like DiRT 2 simply because absolutely precise controls aren't really necessary.

The variable issue of display latency also has an impact. The TN screen we used for this test added just a single frame of lag. Other types of monitor can add anything up to five frames.

An additional element to factor in is the control scheme. OnLive feels significantly laggier when using keyboard and mouse compared to the Xbox 360 controller. Use of the joypad helps mitigates the feeling of numbed response, whereas mouse control is so much more precise that the latency is much more apparent and intrusive. It is at its worst on moving a mouse pointer. There has been talk about using OnLive-style servers to rent time on big-money apps like Photoshop (Gaikai had a similar demo running at E3 behind closed doors). Factoring in the latency there is going to be a big ask for people used to the precision of local response via mouse.

Perceptually then, does it work? Yes, it's playable, but there is just the odd feeling that something isn't quite right with it. During intense gameplay, OnLive is hovering right at the boundary of what is acceptable lag and often exceeds it, resulting in a variable, often unsatisfactory experience. With DiRT 2, for example, drifting is made more difficult because you don't feel you're getting the feedback as quickly as possible.

In slower-paced games, OnLive can work rather well. However, immediately switching to the console version of the game afterward simply serves to remind you that the way things are right now is, by and large, absolutely fine and that whichever way you slice it, OnLive is a downgrade.

Overall though, the latency level is probably the most pleasant surprise with this system. Let's be clear: it is most definitely not a replacement for the local experience, but if the system can be tightened up and that 150ms becomes the norm, then it's clear there is potential here for the infrastructure to find a home with certain types of game or certain types of player.