Getting physical with Virtual Realms

In a dark room encircled by a 360-degree screen, Patrick Moran methodically leaps through a series of blue spotlights on the floor. The Barbican curator is showing me an internal cheat code to speed through a section of Book of Sand, a specially-commissioned work named for the Jorge Luis Borges short story about a book with infinite pages. The meditative scenes projected around us are from Tequila Works' 2017 puzzler RiME, reincarnated in a new form here at the ArtScience Museum in Singapore.

Book of Sand is one of six new installations at Virtual Realms, an exhibition co-curated by the Barbican and Sega alumnus Tetsuya Mizuguchi, who adapted elements of his own game Rez into a new piece for the show. Six game developers were paired with shortlisted media artists to create what Mizuguchi terms an "experiential new artform" for collaborative group play; onedotzero's Shane Walter jokingly describes the Barbican as a sort of marriage broker armed with lists of potential matches. The idea was to get visitors to think about games as art, and push some boundaries in the process.

Each realm is defined by a theme*:

• Synesthesia: "Rezonance" by Enhance and Rhizomatiks
• Unity: "Together: the distance between (us)" by thatgamecompany and FIELD.IO
• Connection: "Wall" by Kojima Productions and The Mill
• Play: "Dream Shaping" by Media Molecule and Marshmallow Laser Feast
• Narrative: "Book of Sand" by Tequila Works and The Workers
• Everything: "Eye" by David OReilly and onedotzero

*More about each installation at the museum website

While Virtual Realms is a high-profile instance of well-known game developers working in the physical world -- a first for all of them -- it also revisits the exhausting question, "can games be art?" Playful interactivity and gamelike experiences have been part of the "established" art world for decades, from Scott Snibbe's Boundary Functions to Jamie Zigelbaum's Pixel. Lawrence Lek's 2065 -- echoing the look of an arcade racing game -- appeared at the Barbican in 2019. Ian Cheng's Emissaries, which was acquired by MoMa PS1, used Unity to create a "video game that plays itself." There's also the whole Meow Wolf scene -- massive, spectacle-like art installations that take cues from immersive theatre.

"Games as art" is not a new concept, and the least interesting angle about Virtual Realms. Its most fascinating draw is in how Covid-19 removed it from the direct gaze of its creators, potentially changing how visitors (and even museum staff) interact with each space. Each installation was designed to be played by a group of people -- a challenge, at best, under constantly-changing social distancing rules.

Virtual Realms' creators, located all over the world, have yet to see their completed installations on site, and may not be able to visit the exhibition for years; the show is scheduled to leave Singapore in January 2022 and will travel to Australia and then the UK. Moran is the only member of the Barbican team who was able to fly to Singapore. Everyone else spoke longingly about wanting to visit an exhibition that has been three years in the making; many expressed frustration, though everyone is acutely aware of pandemic restrictions.

During development, UK-based designers like John Beech from Media Molecule and onedotzero's Walter were able to visit the playtest rooms in the Barbican's basement. LA-based OReilly sent notes and sketches to Walter, who then made little 3D models of the seating layout and wheel controllers for "Eye." In Madrid, Tequila Works didn't have access to a 360-degree screen and had to make do. "[Our mockup] was a very crappy one," says CEO Raul Rubio with a laugh. "There were things we couldn't replicate ourselves…for us, it was more like we could get glimpses of it."

In Japan, Mizuguchi's Enhance studio and the Rhizomatiks team worked on the initial concept for Rezonance in person before Covid-19 hit, after which they worked in VR. "We simulated the same size of the [ArtScience Museum space] in real space, and designers and sound designers, engineers made that with… the Oculus Quest," Mizuguchi says. Rezonance requires "travellers" to interact with a multi-sensory experience while holding haptic "seeds" -- white spheres containing motion capture sensors, WiFi, lights, and an audio connection. The custom-made hardware eventually made its way to the Barbican for testing, then sent to Singapore on the Ever Given container ship, which infamously jammed up the Suez Canal in March. "Everyone was in shock, like, oh my god, that's our container," Mizuguchi recalls.

After all these trials and tribulations, and a handful of trips to the Barbican before travel bans kicked in -- co-curator Mizuguchi still hasn't seen the full-fledged exhibits in action. "I envy you," he says earnestly in our video call. "This is your interview to me… but I want to ask you about [the exhibition]."

It is deeply weird to interview a creator on something they've made but haven't fully seen, especially when you're not sure if the result would have been the same if each creator had been able to physically participate in the execution and installation of their work. With Singapore's changing rules on social distancing -- we just came out of semi-lockdown -- some of the games couldn't be played as intended. Over the last few weeks, we could only socialize in groups of two (recently increased to five). You couldn't mingle with other pairs. This meant that during launch week, Rezonance, which ideally needs four people, could only be played by two visitors at a time, with a museum docent joining in as both third and fourth, holding two seeds at once. It changed the entire experience.

For other installations, changing restrictions weren't necessarily bad. On the day before Virtual Realms' official opening, Moran told me that Book of Sand was being re-patched to accommodate solo visitors, and not just because of Covid-19. Earlier in development, Raul Rubio, a self-described antisocial person who prefers to go to museums alone, inquired about making Book of Sand a single-player experience. "The official answer back then was like, no no, the experience is for groups," says Rubio. "And the thing is that the first day, we noticed that some people wanted to be alone, and that's very human." Unfortunately, when I went back post-patch to try the solo experience, Book of Sand was down for repairs.

The most engrossing part of Virtual Realms is the rich well of emergent behavior in trying to get people to do things together. These are experiences made across different cultures, where each person brings their own learned social behavior to the space. When asked about the impact this might have on their work, some of the game developers hadn't really thought about the issue until I'd brought it up. The media artists, as veteran practitioners in hybrid and physical media, had much more experience working with human unpredictability in this context.

Ayahiko Sato from Rhizomatiks says that internal playtesting often doesn't mirror how "real" visitors might experience an installation. "When we were seeing people experience it in Japan...we were imagining how people would experience it [in Singapore]. It was more, more restrained, more relaxed. People weren't dancing around," he explains. "And then we saw the videos on YouTube [of people in Singapore] enjoying it and it's completely different. I won't say it's completely different from what we expect, because we did expect some variance because everybody's different, different countries, different cultures."

ArtScience Museum staff will also inevitably behave differently from Virtual Realms' future host museums. Where each installation is designed to invite visitors to do specific things -- touch a reactive wall, approach a glowing wheel, or move around beneath a sound-reactive light sculpture -- the main goal is to get people to explore on their own, as best as they can, within a limited timeframe. So it defeats the purpose for you to step into Book of Sand -- ready to unpack its mysteries and figure out the blue spotlights -- and have a docent immediately give you a full explanation of what to do and what to expect.

Take the briefing for Rezonance. While it's a linear sequence with three distinct acts, the docents will always tell you to move to the middle of the area and hold your seeds in a certain way, if only because you only have five minutes to do it. There's no time to experiment, which is understandable -- there's a socially distanced queuing area outside the exhibit. In thatgamecompany and FIELD.IO's "Together," which adapts elements of Sky, multiple visitors are encouraged to figure out how to create a gorgeous piece of symphonic music together. You're meant to walk into a dark room and slowly bring light and sound into the space via Kinect-rigged sensors, which is spoiled by the fact that the museum tied back the blackout curtains at the entrance. On my second visit, I watched multiple people walk in and immediately leave because the light sculpture didn't seem to be fully responsive. Upon bringing my 70-year-old mother along as a human guinea pig, she wanted to know why they didn't make a companion app or a short briefing video to remove the risk of docents over-explaining and ruining the element of discovery.

Nonetheless, all the designers and artists I spoke to were into the idea of people behaving unpredictably, because at the end of the day, that brings about new inspirations. Media Molecule's John Beech is all about the unknown unknowns. "We built [Dream Shaping] in a way that it was.... there were no tutorials and stuff, that it would just work however you decided to interact with it, which is the kind of philosophy of Media Molecule and Marshmallow Laser Feast anyway, it's that sort of explanation, freeform creativity, and expression." Of course, this also means kids are going to bang up your hardware, in this case, soft play shapes embedded with sensors -- on my first run, someone put their shape down too hard and one of the sensors yeeted onto the floor.

"Sometimes it's more like, acknowledging that... I have no clue what is going on, and I won't feel silly, or stupid, or insane for trying," says Rubio when I ask him about how people act differently because of different cultural and social factors. He believes that watching people figure things out and have fun is an important move away from the success/failure binary we associate with conventional games. "I think it's enlightening, watching kids and their grandparents all together, and they don't give a shit if it's right or wrong. That's probably the biggest thing."

Virtual Realms is expressly designed to invite, as I heard repeatedly, different audiences -- the elderly, children, people who aren't interested in games -- to engage with games in a new setting. I ask David OReilly about whether the distinction of "gamer" and "non-gamer" here actually matters. "Everyone's a gamer and every system is gamified," he says. "Everyone engaging with social media is gaming others while being gamified. We urgently need to ungamify reality and become ungamers - only then can we be truly alive (dead?)."

I walked away from Virtual Realms feeling frustrated, having learned about the constraints and conditions that shaped its final form. For all the time, work, and dedication that's been put into the exhibition, it is, like all art, flawed, and like all games, there are bugs and technical issues. It all felt like a watered-down glimpse at what each creator is powerfully capable of doing. If you're avidly into interactive experimental art and art-games, there is nothing here that reinvents the wheel, perhaps because of the fragmented way it had to be put together, and individual experiences of each installation are wildly subjective. But where other artforms had centuries to develop and "legitimize" themselves, there must also be space for experiments in mediocrity.