The Secret Developers: Wii U - the inside story
From reveal to launch and beyond - a third-party developer talks about the challenges of working with Nintendo and its latest console.
The Secret Developers is Digital Foundry's occasional series where game-makers come forward to talk with us - and you - about topics they are passionate about, or in the case of this article, to give you the inside story behind a particular hot topic. As the future of the Wii U looks uncertain in the face of the successful launches for both Xbox One and PlayStation 4, this "warts and all" tale from a respected third-party creator gives you some idea of how Nintendo handled the transition to the high-definition gaming era, and the challenges developers faced in bringing their games to the Wii U platform.
I was there when Nintendo first pitched the Wii U to developers, I worked on the hardware extensively and helped to produce one of the better third-party titles. Now, as the fate of the hardware looks uncertain after a second Christmas of disappointing sales, I wanted to tell the story of what it was actually like to work with the console, and with Nintendo, and perhaps give some context to the mixed fortunes of the machine and its third-party titles.
But first, let's go back to the beginning. The genesis of a new games console generally follows a standard pattern. Initially there is a prolonged period of research and development internally within a manufacturer where the goals and hardware designs are sketched out. These then go through a process of refinement with the hardware parts manufacturers, based on their technology and, obviously, cost.
Once the basic hardware design has been thrashed out, the internal software (SDK) teams get involved in writing the initial code/drivers and tests that are required to run the hardware. Once the teams are happy with the hardware, cost and timelines, the companies start to go out and talk to developers about the new hardware.
To begin with this will be first-party developers and feedback will be gathered that may, or may not, affect the design of the hardware. At this stage the hardware design can be changed, but the window of opportunity is getting smaller. The hardware parts manufacturers have to ramp up their production lines to produce the silicon, which takes time.
After initial feedback, the studio 'tours' begin, talking to select third-party publishers, the Ubisoft, Take-Two and EAs of the world, that the platform holders need to entice to make games for their consoles. Without games, and the income that they provide, the console soon starts to lose money, becoming a noose around the manufacturer's neck.
Major changes at this point are rare, unless they are things that can be altered through software changes (clock speeds, system OS time-slices, etc.) or can be 'easily' added to the hardware design, for example swapping out one set of memory modules for another of higher capacity.
That's where I come in.
"After initial feedback, the studio 'tours' begin, talking to the Ubisoft, Take-Two and EAs of the world... Without games, and the income that they provide, the console soon starts to lose money, becoming a noose around the manufacturer's neck."
The reveal and post-reveal catch-up
When I was told that Nintendo had come into the office for a meeting I could already guess as to what they were going to be talking about. Rumours had been circulating for weeks of new hardware, but nothing concrete had been said. After signing the various NDAs we all gathered in a room to hear the presentation.
It started off in the usual way with a look back on how successful the Wii had been and what their intentions were for the new hardware. They wanted a console that was the same size as the Wii and wouldn't make much noise, so "mum wouldn't mind having it in the living room". It was during this statement that quiet alarm bells started to ring in my brain, but I ignored them and continued watching the presentation. The pitch then moved on to the usual "we need your help to ensure that the Wii U is a success and you can help us (Nintendo) along the way". These words ended up having more significance than either we, or the presenters, could have envisaged.
Then the new controller was shown as a dummy prototype, complete with a glossy video showing how it could be used in games as a series of mock-ups, which looked exciting. By this point we were all considering how we could use the controller in our games. But then they revealed the internal details of the console and I realised the reason for my earlier alarm bells. If Nintendo wanted the hardware to have a small footprint and be quiet, they needed minimal fan noise, meaning that cooling was limited, which in turn meant that the CPU would have to produce a minimal amount of heat, which meant that the clock speed would have to be kept low. While I can't confirm specific details, the collective thoughts of the internet are presented for reference on Wikipedia.
So a basic comparison/calculation makes the Wii U look, on paper at least, significantly slower than an Xbox 360 in terms of raw CPU. This point was raised in the meeting, but the Nintendo representatives dismissed it saying that the "low power consumption was more important to the overall design goals" and that "other CPU features would improve the performance over the raw numbers".
"Almost immediately after the reveal the emails starting flying asking what people thought of the new console design and specification. The almost universal answer was, 'I like the new controller, but the CPU looks a bit underpowered.'"
Almost immediately after the reveal the emails starting flying asking what people thought of the new console design and specification. The almost universal answer was, "I like the new controller, but the CPU looks a bit underpowered".
Over the coming weeks people started doing other calculations trying to guess the performance of the machine - don't forget that this is a long time before development kits were available to do actual tests. Some people even built custom PC rigs with under-clocked CPUs to try and gauge performance of their code on these machine. Again, the almost universal answer was that it wasn't going to be powerful enough to run next-gen engines and it might even struggle to do current-gen (PS3 and X360) titles. But in spite of these tests the management made the decision, for various business reasons, to release a game on the Wii U. So now we had to get stuck in and try to make a game.
And so, to work
Soon after the decision was made the development kits started arriving. As is usual for early hardware they were bigger than the final design with a mixture of connectors and ports used specifically for development. So we plugged them in and flashed them to the latest system code, then tried to get a simple "hello world" type game running, which proved harder than you might think.
Having worked on other hardware consoles, I suppose that we were rather spoilt by having mature toolchains that integrated nicely with our development environment. Wii U on the other hand seemed to be trying at every turn to make it difficult to compile and run any code. Nintendo had provided an integration of their development tools into Visual Studio - the de facto standard for development - but it didn't work, not even close. So time was spent trying to get this fixed up, while reporting the issue to the platform holder. Eventually we received a solution from Nintendo via another third-party company who had also been working on this issue for a while.
So now we could make the code visible in Visual Studio and get it compiling, which was good, but the compilation times were really slow, even for minor changes. Then it had to do the link step, at which point you could happily get up, make a cup of tea, have a chat and get back to your desk before the link was complete. Link times were measured in multiple (four or more) minutes on Wii U compared to around one minute on other platforms.
This doesn't sound bad, but when you are debugging and making lots of changes, these additional times add up. If you made 10 changes to a file in a morning, you could be spending over 50 minutes waiting for the linker to complete, which is a lot of wasted time.
"As a team, we lost days of time to the compile/link/debug overheads and this negatively impacted the amount of features that we could put into our game before the release date."
Finally, when you had the code, you would deploy it to the console and start up the debugger, which was part of the toolchain that Nintendo had licensed from Green Hills Software. As a seasoned developer I've used a lot of debuggers, but this one surprised even me. Its interface was clunky, it was very slow to use and if you made the mistake of actually clicking on any code, then it would pause and retrieve all of the values for the variables that you had clicked, which might take a minute or more to come back.
All of these things made the actual development of code harder than it should have been and ate into the development time of the game. As a team, we lost days of time to the compile/link/debug overheads and this negatively impacted the amount of features that we could put into our game before the release date.
Another curious thing to note at this point was that over the course of six months we received multiple different development kits in a variety of colours, none of which revealed why they were different from the previous one. We knew that there were some hardware bugs that were being fixed, but the release notes rarely stated what had changed - we just had to take the new ones and get them working with our code again, consuming valuable development time. There have been some interesting rumours circulating of PC-style development boxes, and even the Radeon HD 4850 (running underclocked) utilised as a proxy for the Wii U's GPU. We worked on Wii U from the early days and never saw equipment like this - our kits always took the form of custom hardware that I presume was based on near-to-final silicon.
Working with Wii U
Now that the game was up and running on the console we could start developing features that would use the new controllers and make our game stand out on the platform. But soon after starting this we ran into some issues that the (minimal) documentation didn't cover, so we asked questions of our local Nintendo support team. They didn't know the answers so they said they would check with the developers in Japan and we waited for a reply. And we waited. And we waited.
After about a week of chasing we heard back from the support team that they had received an answer from Japan, which they emailed to us. The reply was in the form of a few sentences of very broken English that didn't really answer the question that we had asked in the first place. So we went back to them asking for clarification, which took another week or so to come back. After the second delay we asked why it was taking to long for replies to come back from Japan, were they very busy? The local support team said no, it's just that any questions had to be sent off for translation into Japanese, then sent to the developers, who replied and then the replies were translated back to English and sent back to us. With timezone differences and the delay in translating, this usually took a week !
Getting the game to run at its target frame-rate is a part of the development process that is less interesting in this context as it follows the standard pattern. Get the game running, optimise the code (CPU and GPU) and if it still won't perform, cut back on features until it does fit.
"Getting the game to run at its target frame-rates... follows the standard pattern. Get the game running, optimise the code and if it still won't perform, cut back on features until it does fit."
As far as the CPU optimisations went, yes we did have to cut back on some features due to the CPU not being powerful enough. As we originally feared, trying to support a detailed game running in HD put a lot of strain on the CPUs and we couldn't do as much as we would have liked. Cutting back on some of the features was an easy thing to do, but impacted the game as a whole. Code optimised for the PowerPC processors found in the Xbox 360 and PlayStation 3 wasn't always a good fit for the Wii U CPU, so while the chip has some interesting features that let the CPU punch above its weight, we couldn't fully take advantage of them. However, some code could see substantial improvements that did mitigate the lower clocks - anything up to a 4x boost owing to the removal of Load-Hit-Stores, and higher IPC (instructions per cycle) via the inclusion of out-of-order execution.
On the GPU side, the story was reversed. The GPU proved very capable and we ended up adding additional "polish" features as the GPU had capacity to do it. There was even some discussion on trying to utilise the GPU via compute shaders (GPGPU) to offload work from the CPU - exactly the approach I expect to see gain traction on the next-gen consoles - but with very limited development time and no examples or guidance from Nintendo, we didn't feel that we could risk attempting this work. If we had a larger development team or a longer timeframe, maybe we would have attempted it, but in hindsight we would have been limited as to what we could have done before we maxed out the GPU again. The GPU is better than on PS3 or Xbox 360, but leagues away from the graphics hardware in the PS4 or Xbox One.
I've also seen some concerns about the utilisation of DDR3 RAM on Wii U, and a bandwidth deficit compared to the PS3 and Xbox 360. This wasn't really a problem for us. The GPU could fetch data rapidly with minimal stalls (via the EDRAM) and we could efficiently pre-fetch, allowing the GPU to run at top speed.
Nintendo vs. online gaming
Now that the game was coming together and the hardware issues were being resolved our attention turned to the networking side of our game and its interface to the newly announced Nintendo Network. We spotted early on that there seemed to be gaps in the documentation, and the code, around the networking area, so we asked for clarification. After the usual translation delay we received word that they were still working on the code, but don't worry it would be arriving soon.
Alarm bells started ringing quietly in my head again, but I put them to one side for the time being. This is Nintendo's new network infrastructure that they are basing their console around, they should make sure that it is complete and fully tested before sharing it, so I could forgive them some delay. We had the basics so we could at least do some testing and connect multiple kits together, but a lot of the Mii and friends content was missing and there was no way to test how the existing code would behave in a "retail environment" as there was no retail "flash" for the development kits. We had to code it all in the dark and just hope that it worked.
Around this time we got the chance to talk to some more senior people in Nintendo, via a phone conference, as they were gathering feedback on our development experiences and their toolchain. This phone conference gave an interesting insight into Nintendo and how it appears to operate.
"At some point in this conversation we were informed that it was no good referencing Live and PSN as nobody in [Nintendo's] development teams used those systems (!) so could we provide more detailed explanations for them?"
The discussion started off well enough and covered off our experiences with the hardware and (slow) toolchain and then we steered them towards discussing when the online features might be available. We were told that the features, and the OS updates to support them, would be available before the hardware launch, but only just. There were apparently issues with setting up a large networking infrastructure to rival Sony and Microsoft that they hadn't envisaged.
This was surprising to hear, as we would have thought that they had plenty of time to work on these features as it had been announced months before, so we probed a little deeper and asked how certain scenarios might work with the Mii friends and networking, all the time referencing how Xbox Live and PSN achieve the same thing. At some point in this conversation we were informed that it was no good referencing Live and PSN as nobody in their development teams used those systems (!) so could we provide more detailed explanations for them? My only thought after this call was that they were struggling - badly - with the networking side as it was far more complicated than they anticipated. They were trying to play catch-up with the rival systems, but without the years of experience to back it up.
As promised, (just) before the worldwide launch we received the final networking features that we required for our game along with an OS update for the development kits that would allow us to test. So we patched up our code and tried to start testing our game.
First up we had to flash the kits to the retail mode that had the Mii and network features. This was a very complicated manual process that left the consoles in a halfway state. In the retail mode we could test our features and ensure that they worked as expected, which would be a requirement for getting through Nintendo certification, but in this mode the debugging capabilities were limited. So we could see when things went wrong, but we couldn't fully debug to find out why. As developers, we had to make a choice and hope that any issues that you found were due to the (untested) OS code and wouldn't happen in the final retail environment. What should have been simple tasks were long-winded and error prone. Simple things like sending a friends request to another user were not supported in the OS, so you had to boot a separate program on the console manually, via a debug menu, so that you could send one. But if any error occurred there was no way to debug why it had failed, it just failed.
We started to ask questions about how they could possibly launch the console, which was a matter of weeks away, with a partially developed OS. How were they going to get the OS onto all of the consoles that had been manufactured up to that point? Was it just that we got it late, but they had pushed it into the production line earlier?
Launch day came around and the answer became clear: Nintendo was late - very late - with its network systems. In fact, the only way to access their systems fully was to download a big patch on day one that added all these missing components. Without that patch a lot of the release titles would have been only semi-functional.
"We started to ask questions about how they could possibly launch the console, which was a matter of weeks away, with a partially developed OS... Launch day came around and the answer became clear: Nintendo was late - very late - with its network systems."
What happened next?
Well, we eventually released our game and it was generally well-received, so the management sat back to see what kind of sales figures we would get for all our efforts. Without going into detail it would be fair to say that the numbers we were seeing were less than impressive. In fact we would be lucky to make back all the money that we had invested in making the game in the first place, and although the management publicly supported the Wii U platform, it is unlikely that we would ever release another Wii U title.
But what about the rest of the world? How had other development studios faired? The story of what happened next is pretty well documented in the gaming press, but I'd like to highlight some interesting points that have been on my mind recently. Firstly, third-party support. Do you remember all the hype surrounding the Wii U launch? All those third parties showing videos of existing games that they were going to bring to the Wii U? Whatever happened to a lot of those games?
After the initial flurry of game titles a lot of the studios quietly backed away from their initial statements and announced, with minimal press, that they were in fact not going to make a Wii U version. The reasons behind a particular title not appearing on the Wii U are all pure speculation, but I personally think that a combination of:
- Previous development experience using the toolchain and hardware put off development teams from making another title on Wii U.
- The technical and feature support from Nintendo were lacking for third-party studios. There was a feeling internally that if you weren't a first-party development studio, you were largely ignored by Nintendo, as we were superficial to their profits. Internally developed titles would save Nintendo and we were just there to add depth to the games catalogue.
- The sales figures for the Wii U console were not looking that good soon after launch. There was a lot of confusion in the general population around the launch as most people thought that the Wii U was some kind of add-on to the Wii, they didn't know that it was a new console. This lack of awareness probably contributed to the console not getting off to the start that Nintendo would have hoped and put off studio from developing on the hardware.
- Nintendo also fell victim to bad timing. A few months after the console launched the next-gen hype train stepped up a gear as Sony announced the PlayStation 4, with Microsoft joining the fray a few months later. Don't forget that many of the larger studios would have known about the hardware months before it was announced, well before the Wii U hardware actually launched.
So, these larger studios had a choice. Would they develop a port of an existing game to a console with limited capabilities and limited market penetration? Or put their teams to work on developing new features and concepts for the "real" next-gen consoles that were going to be launched that year? When you look at it this way, the choice isn't that hard.
"Larger studios had a choice. Would they port of an existing game to a console with limited capabilities and limited market penetration? Or put their teams to work on developing new features and concepts for the 'real' next-gen consoles?"
From a first-party perspective, it seems that Nintendo itself hasn't had the easiest time. Now this is pure speculation, but from interactions with some of the development teams it seems as though Nintendo's own teams were having real troubles adapting to the new console - the main reason being the move to HD and the ability of the hardware to support it. Don't forget that until the Wii U came out, none of the first-party titles were in HD and the move from SD to HD is not as easy as you would expect. PS3 and Xbox 360 developers went through this pain early in the previous console cycle and it cost them a lot of time and money trying to adapt, with some studios failing in a big way.
Nintendo's internal teams were now facing this challenge on a new console with limited development time and a lot of pressure to deliver compelling titles. With these pressures upon them it was inevitable that some of the higher-profile titles would slip, but it's surprising how sparse the first-party line-up has been over the last year.
The future for Wii U
Can the Wii U compete in this brave new world of next-gen (current-gen?) consoles? In terms of raw performance it sits uncomfortably between the previous generation and the current one. Parts of the hardware run better than the previous generation, but other parts drag it down. If you tried to compare the Wii U against the PS4/XO, it comes off very badly indeed - it just cannot compete with the new consoles.
At a very basic level, look at the power draw taken by the next-gen consoles compared to the Wii U. The PlayStation 4 draws over 100W more from the mains than Nintendo's console, and it does so using the latest, most power-efficient x86 cores from AMD in concert with a much larger GPU that's a generation ahead and runs on a much smaller fabrication process - 28nm vs. what I'm reliably informed is the 55nm process from Japanese company Renasas.
There are some fleeting parallels between Wii U and the next-gen consoles - the combination of a low-power CPU with a much more powerful graphics chip - but the notion of next-gen titles being easily portable to the Wii U just doesn't work. The gulf in power is just too high, while the GPGPU that we'll see on Xbox One and PlayStation 4 isn't compatible with the older shader model four hardware found in the Wii U.
Doubtless, the first-party developers at Nintendo will make the hardware sing - they always do - but the situation looks grim for those of us in third-party development, with the opportunity to progress on the hardware held back by both the quality of the tools and the lack of financial reward for tailoring our code to the strengths of the hardware. So where does that leave the Wii U?
Personally I'm not sure on what will happen, but if the current trends continue, the Wii U will probably continue to sell in small quantities until a "must have" title is released, probably from a first-party studio, at which point the sales will sky rocket for a while - but even so, matching the momentum of PlayStation 4 and Xbox One seems highly unlikely. Other variables such as the recent news regarding China lifting the ban on games consoles may influence Nintendo's future direction. This huge untapped market may provide a lifeline in terms of sales, but with the low wages of the general population these sales might well come from the original Wii, rather than the more expensive Wii U.
You can never discount Nintendo, but based on my experience - and the sales of the platform - the company's facing its most testing challenge in modern times.