The big PlayStation 5 Pro tech interview with Mark Cerny and Mike Fitzgerald
"Well, it's not all going to be rainbows and path tracing."
Sony's PlayStation 5 Pro is a powerful mid-generation upgrade that delivers machine learning upscaling, improved ray tracing and more raw graphics horsepower. We've spent hours testing and discussing its capabilities, but it's always fascinating to see the other side: how it was built and how its software was designed. With that in mind, Digital Foundry's Oliver Mackenzie spoke to PS5 Pro lead system architect Mark Cerny and core technology director at Insomniac Games, Mike Fitzgerald.
This tech interview covers a lot of ground through its 30-minute runtime, including the challenges and opportunities of PSSR upscaling, the thinking behind the hardware design of the new console and how developers have approached the new options - and performance - afforded to them. This piece arrives hand in glove with a deep dive presentation into the architecture of PS5 Pro with the interview taking place on October 30th, so while the topics covered are very extensive, elements like the variable nature of PSSR implementations across different titles weren't apparent at the time of the conversation.
The discussion also touches on where Sony is looking to go in the future. As Rich postulated in his PS5 Pro review, as well as being a powerful machine in the here and now, this console is also a great opportunity for Sony and game developers alike to get to grips with the RT and ML tech that will define the next generation of games - and that's echoed in the conversation here. If you're looking for some hints as to the direction Sony is taking in PlayStation 6, there's plenty to think about in this piece. As always, the text has been lightly edited for length and clarity.
Digital Foundry: I wanted to start off here by talking about PSSR, PlayStation Spectral Super Resolution. There has been a little bit of a difference in terms of approach between developers like Naughty Dog and Square Enix in terms of the modes that they're offering. Some studios are focusing on offering many modes, while others are using PSSR as an opportunity to homogenise their existing modes into one ultimate PS5 Pro experience. Do you have any preference for either of these approaches?
Mark Cerny: My preference is that the game developers do what they feel is best for their titles, so this is really a question for Mike.
Mike Fitzgerald: When we make a game and we work as a whole studio and team on delivering it, we're making some really tough trade-offs between what we want to present at a higher frame rate and what we can present at 30fps. The nice thing about being approached with new hardware is that suddenly those compromises we made while shipping the game go away. We're able to bring together a lot of what made the fidelity mode the fidelity mode and what made the performance mode the performance mode on the PS5 Pro. After spending some time doing that and having a really nice performance pro mode together, attention starts to turn to, well, maybe some people still want to play at 30fps and we do have this nice new piece of hardware, so what other fun things can we do with it?
Digital Foundry: Yeah, it does depend a lot on the game. For the Insomniac games on PS5, you have so much RT in there that you can pack in all kinds of different configurations. Whereas I think a lot of other developers are going to be in a position where they can now deliver a good 4K image at 60fps, so maybe that's the way to go with one mode.
Mark Cerny: Yeah. I knew there would be a lot of different strategies. I was upfront about this in that first video I did, with six games that were shown and the first three do the "almighty" mode and the other three titles do something completely different.
Digital Foundry: We've seen from developers that there have been a wide range of upscaling factors that are going into this PSSR process. We've seen in a title like Alan Wake 2 that's doing 864p to 4K, while in the Insomniac titles it's more often scaling from 1440p or 1584p to 4K. Does the Sony team provide guidance with regards to those upscale factors, or is it just recommended that developers roll their own and see what kind of image quality you get?
Mark Cerny: I'll talk a little bit about the tech we're developing and then Mike can talk about how they're using it. There's a lot of research going on there. We support one mode that covers a range of 2.5:1 to 1:1. And whether that's an ideal long-term strategy or not, we're definitely talking about it. We're looking at the low end of the range and the issues we see there versus the high end of the range. So I really wouldn't take that current 2.5:1 as any sort of indication about where we'll be years from now.
Mike Fitzgerald: Yeah, I would say a lot of it depends too on what you're moving from. So the TAA that we had been leveraging as a studio - and still do - was really well suited for about 1440p up to 4K, but really struggled when it got down closer to the 1080p to 1280p range. PSSR behaves very well at that range, so for us, it's a big improvement and gives us an opportunity to spend our time differently there.
Mark Cerny: Yeah, that fur on performance RT.
Mike Fitzgerald: Yeah [laughter]. It really struggled for me playing at home in the performance modes that we had on the base PS5. And so to see that quality come back...
Digital Foundry:PSSR is your first implementation of an ML-based upscaling model, and we're seeing wide ranges in terms of upscaling factors. In the future, are you potentially looking to increase those upscaling factors or deliver better quality out of a new version of PSSR? We heard that there were quality improvements in PSSR pre-launch, so do you think that will continue in the future?
Mark Cerny: We're definitely continuing with development of these ML-based libraries for game graphics. It's just that there's so many targets to go up against. Like in that talk today, I was also going over frame generation, frame extrapolation, ray tracing, denoising and the like. So what order and what focus? Those are really good questions.
Additionally, complicating this is that we announced Project Amethyst with AMD today. So it's not just us developing the networks; we have a partner that we're doing it with. And of course, what the partner feels about that technology direction and where the focus should be is also very important.
Digital Foundry:Okay. So in that vein, on PC at the moment, we have a lot of technologies - frame generation, AI upscaling, frame extrapolation, ray reconstruction, which are doing pretty profound things to the makeup of a game image in real time in a pretty lightweight manner. Are those areas that you're exploring for future PSSR upgrades?
Mark Cerny: Definitely. Well, it wouldn't be PSSR - PSSR is PlayStation Spectral Super Resolution. Spectral is our branding for the ML libraries for graphics, just like Tempest is our branding for our 3D audio. And so it's going to be Spectral something for all of that. And yeah, we're definitely looking at what it would be that would be that Spectral something, if not Super Resolution. But Super Resolution is a very good place to start.
Digital Foundry: And in terms of PSSR and how it's implemented on a per-game basis, presumably developers will have to go and patch in to upgrade to new versions of PSSR, right? Because you wouldn't necessarily want new versions of PSSR in the mix there without a title update, or is that something you're thinking about?
Mark Cerny: Well, there are a lot of interesting possibilities there. And honestly, I don't know at this time. It's wonderful to think that the game could say it's okay to update or a user could make a decision to update. Just how all of those pieces fit together, I don't know.
Mike Fitzgerald: From a developer perspective, I think we can look at our games differently a lot of the time. And I know some developers would say, "I don't want the system to touch what I have confirmed looks good to me and players." Other developers would say, "Oh, your system is going to go and make my game look better for people without me having to do anything? Great." So there's options there.
Digital Foundry: With PSSR, you can have a pretty tremendous improvement in image quality while internal resolution stays similar. Is it a challenge to figure out that messenging, that this is a profoundly improved image despite the fact that the internal resolution is comparable? I know that we're part of this problem for sure...
Mark Cerny: You're not part of the problem. I've heard you talk. I've spent too much time on DF. But I've heard you talk about how you really shouldn't be looking at internal rendering resolution and drawing too much in the way of conclusions from that. You really need to be looking at image quality. And I was trying to communicate exactly the same message today, that we need to change the way we're talking about resolution. Maybe we shouldn't be talking about it at all. The difficulty is if you don't have a number, it's very hard to have that conversation.
Digital Foundry: Yeah, we tried to get away from it with a model that would score approximate image quality, but it's a big challenge in terms of communication. Resolution remains a big part of that discussion, for the moment at least. One thing we saw with PSSR in Ratchet and Clank in particular is that it appeared to improve the stability and resolve of RT reflections. Is that something you've seen across other titles that you're working on, and would do you expect that to carry over to other games with RT reflections and PSSR?
Mike Fitzgerald: Ray-traced reflections are an interesting challenge because they are a composite of multiple images in the frame, right? And so with the information you feed into an algorithm like PSSR, you're limited in how much you can describe about each pixel.
An example would be if you're passing the depth buffer in to tell PSSR how far things are from the camera, it can't tell it how far the window is and how far the reflecting thing is and how far the thing behind the window, right? As you turn the camera, you have some velocity information about how the window is changing, but maybe not information about how the reflecting thing in the mirror is changing. This can be really hard to encapsulate in an analytical algorithm like TAA, so that's one of the areas we were hoping to see improvement in PSSR as it potentially has more capability to recognise that type of situation or adapt to it.
Digital Foundry: With PS4 Pro, the focus was targeting 4K displays. With PS5 Pro, you're achieving an enormous improvement in the power level. Did you have a certain kind of game experience in mind to deliver to players when you were designing this console? Or was it more that we're going to engineer big performance improvements and the developers are going to show players what that looks like?
Mark Cerny: It's a mid-generation release. So really the target is better graphics and it felt like the big three were the right things to go after: larger GPU, more advanced ray tracing, and ML. Not that complex of an approach!
Personally speaking, it's been just great digging into the ML aspects of this and gaining some knowledge there. Because we know that's the future we're heading into. And so what an opportunity to gain all that knowledge with the Pro, rather than waiting for the start of a true generation and having to get everything right with no experience whatsoever.
Digital Foundry:One interesting thing about the PS5 Pro is that while GPU, memory and storage have all been improved, the CPU has stayed the same apart from a clock bump that developers can tap into. Why wasn't the CPU upgraded?
Mark Cerny: It's really just a question of where you put your resources. And I wasn't seeing - and I didn't think I would see - that many titles that couldn't run at 60. That's kind of my litmus test. I'm not asking, "do all the games run at 120?" at this point in time. I'm asking, "do all the games run at 60?" You can count the AAA exceptions on one hand, I think, for games that didn't eventually run at 60. And so it seemed if we threw pretty much all of our resources into the graphics, that that would be the best approach.
Mike Fitzgerald: Right. I would say as a developer, the day Mark comes and says, "Hey, we want to give you guys a presentation about what we're thinking about for the next hardware." That's a fun meeting for everyone and we were very happy with where the focus was here.
Gran Turismo 7: PS5 Pro Tech Review - Upgraded RT, 120Hz Support, PSVR2 + 8K - The Complete Analysis
Digital Foundry: With regards to the machine learning hardware in the console, do you see that as a solution for the fact that you don't get good cost scaling on new silicon nodes any more, of the kind that historically enabled mid-gen console refreshes?
Mark Cerny: Yeah. The strategies that we had up through PS4 Pro or the like... I wouldn't say it's reached a limit, but mostly it's about making the GPU bigger or memory faster. And so, as we look to the future, the improvements will be ray tracing. I think we're going to see a lot happening there. And then everything we can get done with ML.
Digital Foundry: The PS5 Pro seems to target similar GPU clocks as in the base PS5 model. Given that the GPU is larger, would you run up against power constraints there, and would clock speeds therefore be a bit more variable?
Mark Cerny: That's a great question. It's pretty much identical. So, PlayStation 5 capped at 2.23GHz, PlayStation 5 Pro caps at 2.35GHz, which I did not talk about today because it's just such a distraction. I don't want anybody doing flop math based on 2.35GHz because, at the end of the day, when we do the set design, we tune the set design so that the games are running at about 2.17GHz. We don't want it to get too bulky. We don't want to have the fan have to be too large or the like. So, I mean, that 2.35GHz is really a red herring. I don't think too many games will get up to that too often.
Digital Foundry: In the presentation, you positioned the PS4/PS5 and PS4 Pro/PS5 Pro as existing along separate tracks: the left-hand track is for generational leaps and the right track is for mid-gen refreshes, enhanced consoles. We're in an era where developers are shipping cross-gen more than they ever have, but does that chart reflect that you still believe fundamentally in generational leaps, going from one generation to the next generation, big changes in software, big changes in what people can deliver on that hardware?
Mark Cerny: We're not making PCs. We're making a few very specific things, right? The Pro consoles are very tightly targeted around GPU. But when a new generation starts, then it gets opened up - what are we doing with the CPU? What are we doing with the amount of memory? What are we doing with haptics? It's a whole new experience for developers. I think it's really helpful, even if we drop words like "revolution" from the conversation, that developers can ask the question, "OK, this game will ship on one and the next, but how can it be a different creature for each of those?" Not gameplay-wise, but in terms of the visual experience.
Digital Foundry: There's one other question that I had about just the kind of way that you're positioning these consoles, which is that we're talking about an era where rasterised performance improvements are maybe not as important as they used to be, and the use of upscaling technologies is changing the way that we perceive all of those metrics.
Mark Cerny: Well, it's not all going to be rainbows and path tracing, so...
Digital Foundry: [Laughter] Well, exactly. So along those lines, the PS5 Pro still has 67 percent more WGPs (work group processors) than the standard PS5. Even though this is the potential future of graphics rendering - a massive focus on AI and ray tracing - it's not completely here today, is it?
Mark Cerny: Sure. What was that like Mike? Because you had a moderate boost for the raw graphics performance, and then you had a much better boost for the ray tracing. How did you tweak the engine?
Mike Fitzgerald: Well, the nice part was that those things stacked together in a really constructive way. So we were able to use the faster RT performance to get some time to spend to drive the upscaler further. The more sizable GPU came back to cut frame-times down. So I think they stacked well for a game that was designed for the base PS5 and already had little bits of each of those pieces.
Mark Cerny: Well, I liked it because very simplistically looking at it, it's faster, and so you've just sped up your rendering, and now you have 5ms to spend on whatever. And so why not spend them on improving your RT or adding a new widget or whatever? If you're making an engine, nobody is ever coming to you and saying, "you now have 5ms to do whatever you want to do that's just gravy." Right? That doesn't happen. You're desperately trying to get to 60fps.
Digital Foundry: Yeah, I thought one compelling part of the presentation was seeing a 16ms workload on PS5 become an 11ms workload on PS5 Pro. Then all of a sudden you've got this big chunk of frame-time left over, looking very scrumptious. Then you can devote your resources to tackling RT, to doing machine learning upscaling, doing a lot more on the hardware. Can you explain the Amethyst initiative with AMD? What's the vision for your collaboration and what it will mean going forward?
Mark Cerny: There's two components. It's a deep collaboration. It's multi-year. Don't expect some massive hardware announcement immediately coming out of this. One target is more optimal architectures for machine learning. And I look at that through a very narrow lens because we're making a console. And so I really want to hear about, number one, does it support these lightweight CNNs that we need for the graphics? But I'd also like to hear about, people want to do their AI or something on it. Is it something that is portable? Is it something that's easy to create? Because you never want to lock people into just one hardware architecture when they make that stuff. So working on this generalised architecture that's particularly good at the ML needs of consoles. I suspect AMD might frame it slightly differently [laughter].
Digital Foundry: So for Sony, it's that focus on these really lightweight models that you're using in real-time rendering.
Mark Cerny: That's my personal focus. AMD, of course, is incredibly broad and they have big machinery out in the cloud and they have laptops. They're in so many different spaces. But I look at it as, yes, I'd love to see these things happen. And the other part of it is for these neural networks, they are time consuming to make. I think, with no exaggeration, PSSR is the toughest technical project I've ever been in in my life. And it's just marvelous to have a partner who also has resources and expertise that can be going into very much the same targets. And what are those targets? It is super resolution. It is. I guarantee it's possible to do better than PSSR. And that's just scratching the surface of where we can head. I mean, I mentioned frame generation and the like, but there are uses I know are out there that we haven't even started talking about yet. So it's to develop those together.
Now, to clarify, we're talking about developing components, not developing libraries. Because our customers are different and when it comes down to it, the hardware is a little bit different. I hope it rung true when I was saying super resolution on PCs is a little bit different from consoles, because we tend to use a variable upscaling ratio and PCs tend be fixed upscaling ratios. So having this ability to grab those co-developed algorithms and then integrate them separately is really going to be a boost across the board.
Digital Foundry: We're curious about how the PS5 Pro hardware handles backwards compatibility. We know the PS5 Pro doesn't have a perfect butterfly GPU configuration like on the PS4 Pro.
Mark Cerny: It's a great question. We learned a lot from PS4 Pro - going into that, I thought we had to replicate pretty much the exact timing that the game was running... and it turns out you don't need to. The timing can drift a bit and the game will still work, though there's a limit to how far that can go. We did see a few titles - like 11, out of thousands - that wouldn't respond well when did PS4 Pro to the radically different timing. But in general, the games are pretty robust when it comes to the timing. So the shorter answer is that on PS5 Pro we just didn't worry about it, and the games are working. We still need binary compatibility - all of the registers that the game was hitting need to be there. But we don't need whatever was taking a thousand cycles before to still take a thousand cycles. It's okay if it takes 600 or 900 or even occasionally 1100.
Mike Fitzgerald: Well, I think we've already seen it. We've already seen that consistency of ray tracing performance pay off even in the Spider-Man 2 patch we've done. So we're able to put in RTAO. We have some RT shadows in there and places where we were bumping up against those divergence limits. In fact, for the water reflections in Spider-Man 2, we cheated them all so that they were less divergent than would be physically accurate in the water to sort of scrape that performance back. And so we're already seeing opportunities where it's paying off for us.
Mark Cerny: I think path tracing on PS5 Pro would be a little tricky to do. But, you know, I wouldn't put it past the developers. There are some amazing engineers out there.
Mike Fitzgerald: It sounds like a challenge.
Digital Foundry: What is your kind of vision for what developers will be achieving on PS5 Pro in the future? If you're looking at PS5 Pro software in a few years, what would you be happy to see?
Mark Cerny: I'm happy with the launch line-up, to be honest. We view it very simply - we're just making tools for the developers. How the developers use those tools, that's up to them. We've seen a couple different ways of doing it, I think right now. Part of the fun will be, you know, in about three or five years, there will be a lot more different ways to use these tools that we hadn't anticipated. Look forward to it.