Face-Off: Gaikai vs. OnLive

OnLive doesn't do enough to convince us that cloud gaming is ready to be the next big thing, but the fact that it works as well as it does is undoubtedly a major technological achievement. The company has set the standard for "first gen" performance in this field, and it's now down to others to enter the market and compete. And that's exactly what upstart rival Gaikai has done - with intriguing results.

Although based on similar principles, the implementation is very different. OnLive launched with a full games service, while Gaikai specialises in offering playable demos with plans to expand beyond that when the time is right. OnLive uses widely spaced datacentres to address a large area, whereas Gaikai offers more servers closer to players. The technology behind the video compression is also very different, with OnLive using hardware encoders while Gaikai uses the x264 software running on powerful Intel CPUs.

Gaikai reckons its approach results in more responsive gameplay, better base visuals and superior video compression. So how can this be tested?

Right now there are only two titles available on both OnLive and Gaikai: Assassin's Creed: Brotherhood and Orcs Must Die, so we'll be using these two games to form the basis of our comparisons, while also referencing other titles to get a more comprehensive picture of how well the two services hold up in specific areas.

The internet connection used in testing is an "up to" 20mbps ADSL line in Brighton, with SpeedTest.net's London server suggesting download speeds of 16.25mbps and an upload stream of 0.87mbps.

Graphics Quality: Who's Running the More Powerful Servers?

"One of the most compelling arguments in favour of cloud gaming is that hardware is upgraded server-side, potentially allowing for better-than-consoles visuals. Right now, only Gaikai is really delivering in terms of high-end visuals."

One of the most compelling arguments in favour of cloud gaming is the ability to be able to play high-end PC games at top-end settings without the user needing to own any of the hardware required to do so. The technology can be transparently upgraded server-side in line with new releases that benefit from a boost in raw grunt at no additional cost to the player. In theory, you need never upgrade your console again.

We came away disappointed with OnLive in this regard when we first looked at the service, and in the here and now, nothing much has really changed. Far from delivering high-end, console-beating experiences, we are treated to a mixture of downgraded visuals with higher than console frame-rates.

In order to maintain the target 60FPS as closely as possible, graphics settings are dialled back to a standard that is in some cases visibly worse than what we are seeing on the current consoles. In previous analyses, we deduced that OnLive servers most likely use dual core CPUs combined with something along the lines of an NVIDIA 9800GT or 9800GTX. These conclusions were based on matching graphics between the cloud service and the original PC game, then measuring performance. As OnLive blocks off graphics settings menus, it was the only way we could compare.

Interestingly, Gaikai takes a different approach in allowing the user to tweak the video settings in some titles, while also providing an upper-end baseline for games where the user is unable to make any changes in this area. A look at the display settings menu in From Dust hosted on Gaikai reveals that an NVIDIA GeForce GTX 560Ti is used in the terminals to run the game, while the refresh rate is set up to allow for a 60FPS update - a Core i5 and GTX 560ti combo would be enough to run all of the titles featured on Gaikai at 60Hz, most of them at 1080p.

In all of the games we've tried the graphics settings are generally set to maximum (or thereabouts), with high levels of texture filtering present and with 8x multi-sampling anti-aliasing (MSAA) enabled. The batch of screenshots below give you a good idea of just how much of a visual upgrade you are getting over what is routinely available in OnLive, and on current console hardware.

The difference isn't all that large with Orcs Must Die. The basic artwork is identical across both services, with the level of anisotropic filtering (AF) being ever so slightly higher on Gaikai. However, the inclusion of MSAA makes a big difference with regards to image quality - things just look smoother and visibly cleaner. Still, there's a sense that the game's muted colour choices and general artwork don't benefit quite so much from the higher settings in play. The game still looks reasonably good on OnLive, which patently uses less powerful hardware than Gaikai.

"On both of the like-for-like examples we have for comparison, Gaikai is running the same PC code at obviously higher graphical presets than its OnLive rival."

On the other hand, the difference in Assassin's Creed: Brotherhood is absolutely huge, and there's no question that Gaikai offers up the richer graphical experience. The intricately detailed cityscapes greatly benefit from the use of higher resolution textures, increased environment detail, and much faster LOD (level of detail) streaming. Simply put, we are seeing a visibly better-looking game in which you can fully appreciate the work gone into constructing the lavish world Ezio Auditore is free to roam. Of the claims that cloud gaming offers a high-end "beyond console" visual experience, it's clearly Gaikai that gets closest to realising the ideal, while OnLive offers a diminished experience - albeit one operating at a higher frame-rate than the console game.

Improved visual quality server-side also has ramifications for video compression. Hard edges with no anti-aliasing will be harder to encode than a richer, smoother picture derived from the game running on higher graphics settings - after all, these video compressors were designed with real life footage in mind, not raw game visuals. In order to best demonstrate the difference we've put together a head-to-head video, with the settings in Assassin's Creed Brotherhood on Gaikai being dialled down to match those found in the OnLive version of the game as closely as possible. For the record, we needed to pull down pretty much every setting to the lowest level to match the make-of OnLive's visuals.

Quite clearly, by upping the level of detail and overall image quality in OnLive, the service could stand to gain a small but tangible boost in video quality without having to change the hardware encoder that handles the compression, but it would come at the cost of substantially upgrading the server hardware. However, sooner or later the company will need to do this to remain graphically competitive. The Gaikai comparison shows that OnLive is running at very low quality presets on recent games and the general migration towards next-gen console hardware will put the cloud service at an even greater disadvantage.

Performance: Only OnLive Runs at 60FPS

In terms of target frame-rates, the difference between the two services is very clear cut: OnLive runs at 60FPS and results in a visibly smoother experience compared to its new rival. On the other hand, Gaikai appears to encode the video stream at a much lower frame-rate, meaning that the action never looks as smooth, and any dips in performance stand out more noticeably.

To demonstrate this, let's take a look at a performance analysis video of Assassin's Creed: Brotherhood on both services. Encoded at 60FPS, there's almost no need to focus on the frame-rate counters at all; the difference is readily apparent when doing a simple visual comparison. Note that 60FPS playback within Adobe Flash can be a bit challenging, so use the full-screen button for best performance (or check out this article on an iPad) - the hardware accelerated video support makes all the difference.

"OnLive's most obvious advantage is temporal resolution - it aims for 60FPS on our comparison games and does a decent enough job of maintaining it, while Gaikai offers an inconsistent update closer to 30FPS."

Running at 60FPS, the action is smoother with OnLive and the visible judder that is present with the game on Gaikai is completely absent. Even when OnLive does drop frames, it's far less noticeable compared to its rival. What we describe as a 'perceptual' 60FPS is in play here, where the dips in performance are so small that they can go unnoticed by the player when engrossed in gameplay.

On the flip side, there's nothing like this at work with Gaikai, and the lower frame-rate makes even subtle dips much more noticeable. Also worth pointing out is that while we see something along the lines of a console-level 30FPS, the presentation is very different. 30Hz console games operate on a cadence of a unique frame followed by a duplicate frame, and so on. This cadence keeps things smooth. Gaikai isn't like this at all, batching two, three or more duplicate frames together sometimes, followed by a batch of unique frames. This emphasises the judder. We tried lowering graphical settings in order to increase frame-rate but it made no difference whatsoever.

With Orcs Must Die things remain much the same, but curiously the variability we would expect in latency response (which we found to be quite annoying in Assassin's Creed) is all but absent here. This gives the illusion of a smoother game when you actually play it, simply because of there's much more consistency in your control over the action.

So on the face of it, the conclusions are obvious. OnLive has a smoother frame-rate, but takes a hit in visual quality while Gaikai looks much, much better in terms of base image quality, but suffers in terms of overall smoothness. We also found that perhaps there is less room for performance issues with Gaikai in the current set-up: while things are smooth we find that gameplay remains mostly unaffected - outside of network stability and traffic - but when the frame-rate drops, the dip in controller response can be felt more obviously by the player. Our thinking is that Gaikai's move to 30FPS pays off for the firm overall, but we'd really need to see a more consistent console-style approach to the lower frame-rate.

Video Quality: Which Service Compresses Video Better?

Both OnLive and Gaikai operate on the same principles of transmitting h.264 video from server to client, albeit with some custom modifications that allow for greater efficiency in real-time encoding. However, the implementation of the tech between the two cloud services is very different.

"A combination of lower frame-rates and a more versatile encoder gives Gaikai the means to power past OnLive in terms of video encoding quality."

OnLive uses a cheap, custom-made hardware encoder installed into each one of its servers while Gaikai utilises CPU-driven x264 - the most powerful, versatile video encoder on the market, utilised by YouTube, Facebook and thousands of others (including us). Unless you've a budget in the tens of thousands, hardware encoders simply cannot match the quality and efficiency of x264, and the developers have cited other areas in the encoding set-up that gives the software solution a clear advantage.

This all sounds fine in theory, but how does this translate into the quality of the actual services on offer? Let's kick off with the first of two comparisons. Assassin's Creed: Brotherhood's mix of fast-paced aerial acrobatics, detailed scenery and more sedate moments during the cut-scenes allows us to get a good general look at how well things hold up on both services. Let's take a look at our first head-to-head video.

What we've done here is to match Assassin's Creed: Brotherhood's settings on Gaikai as closely as we can with the OnLive version - in essence this meant lowering most of the presets to the minimum.

In terms of video quality, Gaikai's advantage is clear. While OnLive often looks muddy, blurry and is filled with heavy compression artifacts during the general run of play, video quality is much more solid with Gaikai - in fast-paced scenes with lots of complex scenery, detail is maintained, and despite there being some visible artifacts on show at times, we really get the impression that we are looking at something closer to a native high definition presentation.

However, in slower scenes where depth-of-field is present and the colour scheme is more muted, there's little to separate the two services at all, and here OnLive holds its own. Less demanding scenes simply don't require huge amounts of bandwidth or advanced encoding techniques and it's here that we see visual quality between the two services close up.

The use of a more refined video encoder obviously helps a great deal to give Gaikai its advantage, but the decision to target a lower frame-rate than OnLive obviously plays its part. OnLive aims for a 60FPS update in order to minimise latency so that the player gets the fastest controller response possible. On the other hand, Gaikai appears to aim for a more manageable 30FPS, seemingly relying on the use of more local datacentres to keep the level of input lag in check. Our theory is that the games themselves may actually be running at 60FPS server-side with Gaikai, but with the video encoder not encoding every frame generated, giving a lower frame-rate client-side.

Basically, fewer unique frames means less work for the compressor to deal with when encoding the video stream. From another perspective, dropping down to 30FPS also provides double the amount of bandwidth for image quality and thus delivers overall clarity closer to the experience of gaming on local hardware.

"Orcs Must Die on OnLive doesn't match the quality of Gaikai, let alone running the game locally on PC, but it holds up well enough for the reduction in quality not to be so much of an issue."

Of course games with less in the way of fast motion and more muted colour schemes work much better with OnLive. Orcs Must Die actually fares pretty well - handing in a much better presentation overall than Assassin's Creed: Brotherhood. There are compression artifacts, and things do tend to blur together and begin to fall apart in fast moving scenes, but viewing from a few feet away on a smaller HDTV screen, the results can actually look rather good. It's not directly comparable quality-wise with Gaikai, let alone running the game locally on PC, but it holds up well enough for the reduction in quality not to be so much of an issue.

The overall conclusion we draw is that OnLive is better suited to slower moving games (a touch ironic perhaps, considering its 60Hz update), such as RPGs and adventure titles, while Gaikai is more versatile, retaining enough image quality in demanding scenes to handle faster paced action games and first-person shooters - at least in terms of raw video quality. The trade-off is that the 60FPS update is gone.

Latency: Do Closer Datacentres Make Any Difference?

We define input lag as the time taken between the player inputting a movement on the control pad and his action playing out on-screen. Our extensive testing of latency on Xbox 360 and PlayStation 3 tells us that games running at 30FPS usually operate with around 100 to 150ms of input lag, while those running at 60FPS tend to see this drop to between 50ms and 83ms. The simple fact is that higher frame-rates result in a faster controller response, leading to a more enjoyable gameplay experience.

"Higher frame-rates should give OnLive a clear latency advantage over Gaikai, but testing demonstrates that this is not the case. Gaikai's more local datacentres offer the service an important advantage over its rival."

The principle behind OnLive is basic enough: render console titles at faster frame-rates on server-side PCs, then use the latency saving to mitigate the cost of encoding and transmitting the video to the player. However, Gaikai is a lesser known beast. Our findings strongly suggest that the service is also rendering its games natively at 60FPS on its servers, but with a much lower frame-rate on the actual video encode. In theory, this introduces slightly more latency, although Gaikai claims that by having its datacentres positioned closer to the player, this additional lag is cancelled out, and the result is overall close to the native 30FPS console experience.

Our extensive tests with OnLive revealed that the amount of latency produced by the service with titles running at 60FPS measured between a very respectable 150ms and a virtually unplayable 250ms. So, how well does Gaikai hold up in comparison, given that the service sacrifices some of the gains of having a 60FPS refresh rate for better video quality?

In order to measure baseline latency with OnLive, we used a Ben Heck monitor board, which has LEDs that light up when button presses are made on a wireless 360 controller, and film the action using a 60FPS camera. We then count the number of frames between the LEDs lighting up on the monitor board with a specific action occurring in-game. Afterwards the additional lag created by our display is factored out and this gives us our latency reading.

For our comparison between OnLive and Gaikai (where the gamepad is not supported in all titles), we couldn't use this technique. Instead we used FRAPS to map action buttons on the keyboard to the benchmarking tool's capture button. The FRAPS frame-rate indicator changes colour when recording is engaged and we counted frames from this event to the action happening in-game.

While this technique doesn't give us a complete measurement, it gives us the differences in latency between Gaikai and OnLive, and comparing the OnLive measurements to the same tests using the 360 pad apparatus then allows us to calculate overall latencies for both services with some degree of precision.

As consistency appears to be an issue in both cloud services, we've produced two sets of measurements - gameplay with not much going on and in-game frame-rates at their maximum, and then we followed that up with further metrics taken from more demanding scenes where performance drops. We tried to match these as closely as possible between the two services. We also took multiple readings, as they can change significantly at any given point.

The results are rather interesting. Not only does the amount of latency present in some of the titles running on Gaikai equal that of OnLive, but there are also times where controller response is significantly faster despite the frame-rate being lower due to the 30FPS video encode. Taking a closer look at Assassin's Creed on Gaikai reveals that when the frame-rate is consistently hitting the target, the level of latency hovers around the 167ms to 183ms mark - it's playable, but inconsistent. With the game running flat-out at 60FPS, OnLive is more consistent, if slightly slower than Gaikai. When performance drops, both platforms are impacted, but it's Gaikai that seems to suffer the most.

"The cloud services seem to be within striking distance of console input lag, but the lack of any real consistency in response times is a major issue and transforms the way a game 'feels' to play."

On the other hand, when looking at Orcs Must Die we get far better latency results and as such a completely different feeling of controller response. With Gaikai, baseline latency isn't exactly wonderful at 183ms, but we found it to be consistent and it was possible to adjust to the gameplay experience. Gaikai annihilated OnLive in terms of response here with a consistent 83ms-100ms advantage - remarkable.

Other tests were equally polarising when it came to seeing how well Gaikai fared when put up against games running on the Xbox 360 and natively on PC. Sometimes we got some incredibly good results: playing Bulletstorm, we found controller response to be slightly slower than the 360 game, but sometimes it hit parity - a truly remarkable achievement. The quickest response time we measured was 133ms (identical to Bulletstorm 360) but it could drift to 150ms, in-line with The Darkness 2 on the 360 or Killzone 2 on PS3 - and we also recorded the odd 166ms measurement too. A matter of 17 or 33ms in additional lag might seem miniscule but you can definitely feel it in the inconsistency of the response.

Regardless, Bulletstorm is still playable on Gaikai, and we didn't find the additional latency to be too impactful across a general run of play. A look at Crysis 2 is far less pleasing though, with controller response being measured anywhere from 167ms, all the way up to 217ms, resulting in a virtually unplayable experience when it came to aiming and shooting with any degree of precision. Both of these shooters feature gamepad support, so we were able to measure latencies in exactly the same way between platforms.

From what we can see, OnLive's decision to target a 60FPS update definitely helps, but the geographic location of its datacentres is perhaps the cause of its disappointing performance against Gaikai. In our case, the input from the user's controller has to be beamed from the UK all the way to Luxembourg (where the server for Europe is located) before the game's output is encoded into streaming video and sent back, travelling across a multitude of network connections to do so. The fact that the server supplying OnLive to all of Europe is located so far away seems to add considerable latency to the initial lag generated locally by the game, despite the end-to-end 60Hz update.

On the other hand, Gaikai's server for supplying the UK is based in London (around 52 miles from where we tested the service) and it appears that the closeness of the server allows for levels of latency on a par, or even better than OnLive, even with the video stream being encoded at half the frame-rate.

While our findings are positive when looking at the performance of Orcs Must Die, they are hardly representative of Gaikai taken as a whole, factoring in the high level of lag found in Assassin's Creed Brotherhood or Crysis 2 at their worst. On the other hand, the best case 133ms results we got when testing out Bulletstorm are very revealing, and show that Gaikai's methodology can work very well indeed when network conditions are stable and when there are no changes in frame-rate. However, there is the suggestion that for cloud to take off as a platform, a consistency needs to be found - right now we see alarming differences in response on both services between games.

Gaikai vs. OnLive: The Digital Foundry Verdict

"It's important to remember that these are 'first gen' services - improvements in infrastructure, increased bandwidth and refinements to latency management will make all the difference."

If there's one thing that levels the two Cloud services it's the fact that they both have little in the way of network resources to utilise - current levels of internet bandwidth represent a really tight budget for transmitting h.264 HD video encoded at real-time. The net result is that OnLive and Gaikai have chosen quite different solutions in how this bandwidth is utilised, with very different results.

OnLive targets temporal resolution - they want the player to enjoy 60 frames per second gaming because it helps with the latency challenge and there's also the sense that a faster moving image makes it harder for the human eye to track video compression artifacting. Gaikai's solution appears to be quite the opposite - halving the frame-rate but effectively doubling the image quality. Combine that with more powerful servers running the games at much higher graphical settings and the result is a dramatically improved look to the games you're playing. However, we found the lack of consistency in the way frames are rendered to quite off-putting.

In terms of the lag factor, the jury's still out, but the evidence suggests that Gaikai has seized the lead here. In the past, OnLive has shown occasional glimpses of being within striking distance of local gaming response rates, but the maddening inconsistency in latency from second to second, let alone from game to game, means that the overall experience simply doesn't compare to gaming on the Xbox 360 or PlayStation 3. Gaikai's more locally-based datacentres suggest that it enjoys an advantage, and our tests with Orcs Must Die back that up, but again, the remarkable variance in response from game to game is frustrating. There's also the sense that if a game drops frames server-side in Gaikai, the impact to controller response is more noticeable than it is with OnLive.

In short, it's still early days for the cloud - both OnLive and Gaikai are still what we would term "first gen" products of this new generation in gaming and are far from mature technologies - but it is clear that there is genuine promise here. Steve Perlman's group takes honours for frame-rate, while David Perry's team command clear advantages in game spec, video encoding and overall latency. The signs are there that this can work: Bulletstorm on Gaikai running with the same latency as the Xbox 360 game - however inconsistent it may be - is as clear an indicator as any that cloud could evolve into a viable contender. The technology here is achieving miracles as is, and with advances in infrastructure and improved engineering, it's only going to get better.