The Visual Effects Oscar nominations for 2020 have been announced. Five films are competing to take home the prestigious award in film, with the winner revealed live on Sunday, February 9 on ABC. The nominees for Visual Effects are Avengers: Endgame, The Irishman, The Lion King, 1917, and Star Wars: The Rise Of Skywalker.
While all are outstanding films, The Lion King is also outstanding technically. The quality of the character animation, the look of the final rendered imagery, and the innovation in making the film are not only impressive—they show us the future of cinema.
The movie set out to solve a classic VFX/animation problem—how to direct a story when the director cannot see everything they are supposed to be directing.
I spoke to Ben Grossmann, Oscar-winning and Emmy Award-winning Visual Effects Supervisor, Virtual Production Supervisor on The Lion King and cofounder of Magnopus—an AR/VR company that developed the systems used in the creation of the film—about virtual production and what it means for the film industry.
What is virtual production?
BG: As VFX have grown to be a greater part of movies and television today, there’s a growing divide between what the filmmakers can see through the camera on the live-action set, and what they have to imagine will be added digitally many months later. Virtual Production attempts to unite those two worlds in real-time.
Using game engines like Unreal and Unity, combined with high-power graphics cards, camera tracking, as well as VR and AR, filmmakers are now able to create scenes across the physical and digital worlds, together.
Often a movie flops because the director just can’t see it all together until it’s too late. The process of getting there sometimes requires thousands of VFX artists, which means a LOT of complex communication that slows down the creative exploration required to discover “what’s right” for a film.
What process is involved with virtual production?
BG: At its simplest form, we’re building a game called “filmmaking,” and we’re building the world of the film in the game. Instead of guns or jet packs, we give the players traditional film equipment like cameras, dollies, and lights.
The new approaches allow us to use VR for content that is almost entirely computer generated. Or to composite the computer graphics with live-action footage through camera tracking and AR.
When you’re using VR to make the movie, it’s as though you’re standing on a film set in the real world, even though it’s all virtual. You can pick up a tree and move it. You can grab the Sun and change the light. You can become a character and give a different performance.
Why has virtual production suddenly progressed so rapidly?
BG: A lot of this revolution is happening because of consumer technology. In the movie business, there’s only so much money you have to do R&D because there are so few people who need it. So all the solutions we developed in the past could only go so far, with limited money. Because games and virtual reality are targeting average consumers, there’s suddenly a LOT of money available to solve the problems we’ve faced.
When you think about how much time people today spend on their phones and laptops, you start to realize they’re living two lives—the physical one we’ve always known, and a digital one inside devices. All the world’s biggest companies are trying to marry those two worlds now—Apple, Google, Microsoft, Facebook. The movie business has been tackling that problem through visual effects for decades. “How do we unite the digital and physical worlds so we can tell any story we can imagine?”
What does this mean for filmmakers?
BG: This empowers the filmmakers to create shots based on what “feels” right, rather than looking at a computer and trying to guess where things should go when it’s not feeling right. And they can do it quickly without having to spend hours moving heavy equipment.
The technology puts the tools of storytelling back in the hands of the filmmakers, rather than an army of technicians.
This lets them explore ideas a lot faster with intuition and a stronger creative vision. Imagine if filmmakers had an “undo” in the real world!
What about assets?
BG: We build the assets of the film in a game engine, the same way filmmakers have been doing on movie sets for decades, but now virtually.
The physical cinematography equipment on a film set is replicated in the virtual world, and tracked so they match exactly. This allows the director of photography to put their hands on the gear and react more naturally as the performances unfold.
How are real actors put into these virtual environments?
BG: Now, we can use high resolution LED screens that transform a movie stage into a Holodeck. Then we can shoot real actors and props directly in front of the screens in any direction. What we see on the screens are high quality game engine assets you can have almost infinite control over.
When all this is done, the game engine outputs video material to go to editorial, and packages of data that can be sent to VFX—where it can be made photorealistic.
What are the benefits?
BG: Because filmmakers can see the world in real-time, they can react and make changes quickly to creatively explore, before an army of people get involved. Using VR also means they’re reacting naturally to the world they’ve created, and discovering new perspectives they hadn’t thought of until they were standing there. Because it’s multiplayer, there’s a sense of teamwork where people are talking to each other and working together, each bringing their skills to the process.
Having traditional film interfaces to computer graphics means that film crews who have worked on live action sets for decades can bring their talents to a new kind of movie.
Because you figure out the creative with less people, you’re taking a lot of the uncertainty out of post production. Instead of spending months with dozens of visual effects artists trying to figure out what a shot should be, you are doing it with a couple people in a few hours. Plus, you can edit the scene so you know exactly what the film needs to be great BEFORE you send it to the hundreds of artists who will make it feel real.
More iterations done faster with fewer people makes a better film, cheaper.
What would make virtual production easier?
BG: There is still a lot of progress to be made. We need more advancements in graphics cards, like real-time raytracing to increase the quality.
What about AI?
BG: We’ve only just begun to scratch the surface of how AI can contribute. It can help us create these worlds faster. It can become like a virtual film crew that awaits our every command. As AI gets better, it can even provide the intelligence behind the characters we create, giving them personalities and performances we can direct.
While virtual production has some very practical and financial benefits, is there anything else you think it can achieve?
BG: One of the most exciting things is when you’re standing inside a movie that you built as a game, you wonder “why don’t we invite the audience in here too? What would it be like if the audience watched the movie from the inside? What if they were characters? What would storytelling become if we did that?”
There are so few filmmakers in the world because learning the language of cinema typically requires millions of dollars of equipment a large film crews. What if every kid could have the tools in their bedroom? We all remember playing with toys when we were kids. It was an infinite world of imagination where anything was possible! What if making movies was that easy?
