3DMark's DX12 bench analysed: GTX 1060 vs RX 480

Benchmark specialist Futuremark has updated its 3DMark software with a new test designed from the ground up to support DirectX 12 - and its results are causing some controversy. AMD users in particular are somewhat surprised by the higher results gleaned from Nvidia hardware, suggesting that the tests scores are somewhat at odds with a clutch of DX12 games showing Radeon hardware in the ascendency under the new API. Other AMD fans believe that Nvidia hardware's 'preempt' style of async compute isn't asynchronous at all and believe the test is fixed, despite the developer's well-earned reputation. It led to Futuremark releasing a statement going into depth on just how its new test, dubbed Time Spy, actually works, while emphasising that its development was coordinated with partners including Nvidia, Intel, Microsoft and - yes - AMD.

"For benchmarks to be relevant and useful tools, they must be fair, impartial and unbiased," says Futuremark. "This is why 3DMark Time Spy, and all other Futuremark benchmarks, are developed with industry-leading hardware and software partners through our Benchmark Development Program using a process that's been government-vetted for fairness and neutrality. The process ensures that our benchmarks are accurate, relevant and impartial."

The focus on the async compute controversy has been somewhat unfortunate, because the scale of ambition seen in Time Spy is exceptional. Games have barely begun to scratch the surface of what new APIs like DX12 and Vulkan are capable of. Essentially, we've seen existing OpenGL and DX11 games ported over - there have been performance gains, but little to hint at the new options open to developers. Time Spy sees Futuremark attempt to create a benchmark based on the idea of a game engine built specifically to take advantage of the key benefits offered by DirectX 12. The image below shows how scene complexity in terms of vertices, triangle count, tessellation patches and compute shader invocations rises from Fire Strike to Time Spy. In many cases, we're looking at an order of magnitude increase.

So if Time Spy is a vision of a DX12-powered future, how does today's hardware hold up? We decided to test the benchmark using the mainstream graphics hardware of the moment - the Radeon RX 480 and Nvidia's freshly minted GTX 1060. It's an interesting test, not least because of the architectural match between AMD's technology and the upcoming PlayStation Neo. The end result is a 30-40fps showing for both GPUs across the two graphics tests when played out at 1080p - console game-style performance from next-gen hardware then, but crucially played out with a cutting edge piece of software designed to capitalize on the API's core strengths.

And in terms of the performance differential between the two GPUs, the graphics test shows Nvidia's GTX 1060 handing a six per cent boost over its cheaper AMD competitor, but what's also interesting is that Time Spy can run with async compute on and off. This gifts the RX 480 an 11.6 per cent boost, whereas Nvidia's card sees a lower 5.5 per cent increase. It's significant - especially for the Radeon kit - but not exactly game-changing. But remember, this is one test and one test only. Different rendering workloads will see varying performance increases with async compute - after all, the technology works by 'filling in holes', utilising dormant compute resources on the GPU. If there are fewer 'holes' to fill, it stands to reason that you'll get a smaller increase in performance.

As things stand, while there has been surprise from AMD users about Nvidia performing so well in Time Spy, the results tell us little that we didn't already know - GTX 1060 is a touch faster than RX 480, as you would expect from a generally faster card. And we know that AMD gains more from async compute, hence the bigger boost in the Time Spy results. However, you'll also note that our tests below also include the GTX 980 - effectively GTX 1060's closest equivalent from the last-gen Maxwell architecture. And there we see no boost to performance whatsoever with async compute active. Futuremark says that the Nvidia driver effectively ignores the async commands, explaining why the scores are effectively identical.

Time Spy also features Futuremark's customary CPU test and reassuringly, this duly delivers very similar results on all three graphics cards tested, though we did note a small drop on the AMD run with async compute disabled. And CPU performance is something that's worth pointing out - while there's been a lot of discussion about async compute, the principal benefit of new APIs like Vulkan and DX12 is the ability to leverage more from the GPU by making the CPU much less of a bottleneck. Revisit the table above: much of this is possible via much more efficient utilisation of multiple CPU cores - and that still works for your system whether the GPU supports async compute or not.

3DMark DirectX 12 Time Spy: GTX 1060 vs RX 480 Analysis

We aim to return to DX12 soon to focus on the available titles and to judge performance improvements - or the lack of them - with actual gameplay. And it's here where there are reasons to expect AMD to provide higher returns, certainly if the firm's own benchmarks are to be believed. So the question is really, to what extent is Time Spy actually useful if it doesn't reflect gaming performance?

Quite apart from the fact that this is a test built from the ground up for DX12 as opposed to being a 'port' to the new API, there's also the notion that 3DMark created Time Spy to work as a non-vendor specific piece of code. The likes of Microsoft, Nvidia, Intel and indeed AMD were invited to provide optimisations to the codebase, but they were only accepted if there was no degradation to performance on competing hardware. And this is a contrast to actual games, which can see definitive competitive advantages to one vendor or another - and that's not necessarily down to the API. As a case in point, Hitman's DX12 benchmark always shows a clear AMD advantage and is often mooted by some as an example of 'DX12 done right' but the fact is that it typically runs faster than equivalent Nvidia hardware on DX11 too.

But it's going to be the future titles designed with DX12 or Vulkan as their foundation that should finally see some increased visual return, as opposed to performance boosts alone. As id software's Tiago Sousa said in our Doom tech interview deep dive last week:

"I think it will be interesting to see the result of a game entirely taking advantage by design of any of the new APIs - since no game has yet. I'm expecting to see a relatively big jump in the amount of geometry detail on-screen with things like dynamic shadows. One other aspect that is overlooked is that the lower CPU overhead will allow art teams to work more efficiently - I'm predicting a welcome productivity boost on that side."