The Visual Effects of War for the Planet of the Apes

By Norman Chan

We went on set last year as this epic movie was being made.

War for the Planet of the Apes hits theaters this weekend, and by most accounts, the film is a triumph (93% fresh on RottenTomatoes). I'm really excited to see it on the big screen, since it was shot in 65mm to explicitly showcase the film's landscapes and larger-than-life sets. We were on one of those sets over a year and a half ago, while the film was in mid-production. Fox invited us out to visit the shoot, on a massive outdoor prison built a few miles away from Vancouver. The Canadian winter was an ideal backdrop for this sprawling snowy set, which had its own set of railroad tracks, grungy barracks, and a towering battle-worn wall that separates soldier and simian.

The enormity and tangibility of this built-out world stands in contrast to the film's computer-generated heroes--this Planet of the Apes trilogy has anchored itself in its ability to meld the real and the digital. Actors like Andy Serkis and Steve Zahn do their best to give performances in tight-fitting performance-capture suits, but their performance is really a collaboration with the animators and special effects artists who turn mo-cap data into the characters you see on screen. On set, we sat in a roundtable interview with visual effects producer Ryan Stafford, who worked with director Matt Reeves and the effects team at Weta Digital to realize the digital characters and environments in the film. Here are some highlights from that conversation.

On compositing live and CG characters

Every time we do a shot, we do it in a variety of ways. On a traditional movie, you set up the camera, you set up your characters, you roll, get the performances you want, and you move on. Well, when we get the performances we want, we say "great, let's do that again without the actors." So we pull the actors out of the shot and we replicate the camera move as closely as possible using motion control and a lot of other tricks to get it as accurate as possible. And then we run the whole take "clean"--and if there's a human character in it, then they have to act to nothing. Sometimes we put a piece of tape up with fishing wire, and that's their eye line. And they have to re-enact the entire performance to nothing, to thin air, to a piece of tape.

The apes proportions--their anatomy--are different from humans. It's very close. We've made Caesar just a few inches shorter than Andy Serkis. And the build is similar. But where his joints are are different than Andy's. His arms are much longer. His legs are shorter, his chest is more barreled. So when we put Caesar on top of Andy in the shot, there's a lot of Andy left. And we have to paint that out. Painting out things is very expensive, it's very labor intensive, it's very complex, especially when you have very dynamic camera movements. So we do it all on a clean plate in hopes that it's as dynamic as with the actors in it so we don't have to paint anything out.

We use both. It's kind of a mixed bag. Our ideal scenario is a clean plate but we have maybe a 60% success rate with that. We still have to do a lot of cleanup with actors still in the shot. That's mostly performance driven. The reality is that you're going to get a better performance from a human character when they're acting against someone else in frame. Particularly Andy, who has such a great presence, you get a much different experience than if someone's acting against thin air. In those instances, we take the clean plate, and use that information to do an overlay, try to get as much information, and steal as information as we can.

We use a device on this movie that's new to us. It's been around for a little while, called the Technodolly. It's a technocrane that's fully repeatable in every axis. You can shoot a shot as you normally would--with dolly grips manipulating the base, it's on a track. It does all those things that are human powered, and then, when you get the performance you like, you literally hit a button, and the machine takes over and replicates everything exactly, down to the frame. So it's a huge save, especially on this movie, in terms of the accuracy. If we're doing it manually, we'd have to sometimes shoot four or fives times just to get a clean plate right.

The state of performance capture is constantly improving. The technology goes through huge developments every year and a half or so. We're benefitting greatly from a lot of new technology that's mostly software-based. We're benefiting a lot from the film The BFG. BFG had a hugely complex mo-cap shoot that was right before we landed here, so we learned a lot from their tricks and tips and new software that they had field tested for us.

The use of virtual environments

We're using a lot more real-time virtual environment playback on this film, which is aiding in our motion capture. It's not directly linked to mo-cap, but it's helping us know where we are in our worldspace. We've joined forces with the art department. We have a team of Weta artists who are fully involved with the art department. The production designer gives them creative notes, and they act as if they're just another set designer on the team. And they are building out all the virtual environments in the movie. And the idea is that once we get into the set, and there's a green screen there, we can do a real-time composite and show what that virtual environment is beyond the green screen. So it helps for framing. So on this set, we have the colonel's quarters, which is 55 feet above ground, not represented on our set. Tut in a simul-cam situation, we know exactly where that is, whether the cameras are pointing too high, too low, or whatever the case may be. And it helps that accuracy improve.

We'll a VAD--a virtual art department--artist working here in our Brain Bar that will be able to manipulate the virtual set to a certain extent. Like world rotation, and orientation, and things like that. To make sure it's positioned just so for the cameras to make sense in space. But in terms of making mountains taller or lower, that's locked in further in advance.

The Brain Bar is just a bunch of road cases full of computer equipment, and we have technicians in there all the time. We run mo-cap out of there, we run the virtual art department out of there.

On using the latest technology

Typically, when big technology advancements are made, we don't change course in the middle of a show. On Dawn of the Planet of the Apes, for example, Weta was already field testing the next version of fur rendering and all these super wonderful tech that were going to make things much more efficient and real, and we weren't able to take advantage of them. We were already mid-stream on our show, and for them to pull us off of one pipeline and put us onto another would just disintegrate everything we had done.

Right now (18 months before release), we're not quite locked in yet. We're just starting our digital asset build for our characters, so advancements Weta makes in the next four to six months we'll be able to take advantage of, but that'll quickly start to get locked down onto a pipeline train we'll take to the end.

We're able to create certain shots if there's something that's missing, like a one-off that would be too difficult to build for a reshoot.

One of the benefits of shooting clean plates is that we will beg, borrow, and steal plate from scenes that were never meant to go together, and we'll construct new scenes or new beats. There's a lot of manipulation of the footage, using digital versions of almost all of the sets. So we're able to create certain shots if there's something that's missing, like a one-off that would be too difficult to build for a reshoot. We can steal a plate from another scene or build it out digitally and create those moments.

We did a lot of that on 'Dawn' and will be on this as well where we had whole moments in the film that were completely digital. And we have a plan for that in this movie--there's a whole scene that will be completely digital that we're building now with the art department. So it's being built as though it was a hammer and nails type of set, by our virtual art department. And then it'll live entirely virtually in the film.

It all comes down to logistics and money. We have a footprint--the studio gives us a box to fit into, and our movie is growing by a certain percentage in areas that are very well known, and in others we need to find efficiencies. In certain huge sets where there's not a single human that exists in that space in the context of the story, it's actually cheaper to build it digitally than it is to build a real set. And it lives in such a space that it'll be entirely convincing digitally. We're not really stretching the envelope--we didn't want to do that, we didn't want it to look like a digital set. We want it to be photoreal. So, it's grounded. It's rock. It's a cave. We know we can do that. So we felt comfortable making that decision and applying those resources elsewhere in the film.

On cameras and not shooting in 3D

Shooting in 3D had an impact in a couple of ways. Mostly in physical production. The 3D rig weighs 95 pounds. It is a beast, and it was really hard to move that sucker around. We had two Steadicam shots in the last film because the best Steadicam operators in the world were just like "it hurts my back." It's way too difficult to move that around. We also shot that movie mostly with zoom lenses, because a lens change takes time. Because you have to change two lenses, and calibrate the interocular settings and the disparity between the lenses. All of those things had to be fined tuned. On this movie, we're shooting almost entirely on prime lenses. I think we put a zoom on the camera maybe once or twice. So it gives a very different depth of field, which offers a huge aesthetic variation to the film.

Matt loves that long lens shallow focus feeling but now we're really going for it, shooting wide open on everything. And when there's too much light to shoot wide open, we're throwing filters in front of the lens instead of closer the f-stop.

On matching digital scenes with practical ones

For the scene in the virtual set, we'll be "filming" it. We'll be doing a virtual production process where we will be capturing the performances first and then going back into the volume with film tools we're familiar with--cranes, steadicams, and things like that. But instead of a 65mm camera on top of the crane, it'll be a stick. A T-shaped stick, with a point on it, and mo-cap markers all over it. And we manipulate that the same way we manipulate a real camera. It'll literally be in an empty room. We'll be looking in a monitor like a sort of VR way--a very videogame feeling. But the camera language is the same. We hire a camera operator to do it--it's not a CG guy that knows cameras. We hire the proper camera operator that we train to use the virtual equipment, and we film the scenes the same way that we would on a physical set. So we keep that feeling--that camera language feeling--very present.

That's something in my opinion that really tips the hat--when a camera starts to suddenly look false. When it moves too smooth or too clean, or does something that a real camera shouldn't. But with this, we keep ourselves true to the physics.

We don't have our monitors set up with VR yet. It's something that we fantasize about--one of our supervisors is a VR specialist and we've been trying to find ways to incorporate it, and we have a few months left to field test it, but the jury's still out on that one. But it would be very cool!

On the set of the film! Photo credit: Doane Gregory

On using practical effects vs computer-generated effects

I work very closely with the special effects coordinator. We talk constantly. Our MO is that if it's meant to happen, it should happen for real. And if it can't for whatever reason, then let's talk about how it can happen digitally or how we can enhance or augment it. But I feel very strongly that if you want it, put it front of the camera. Because that will always give you your best result. And then we'll spitball from there.

So we have explosions in our movie and we're doing them real. We have gunfire we're doing with live squibs. Smoke, fire, all those things. The only time when we really pull it out of the equation is when it overlaps with our digital characters. So if we have a campfire and a digital character, we'll shoot the campfire for reference so we know what it needs to be, but we'll turn it off and shoot our digital character. Because getting a digital character to live behind a real fire takes a heck of a lot more manipulation than doing it in layers, and doing either a CG fire or element of a fire and layering it.