Our brains perceive depth by comparing the images that our eyes see. If you alternatively close each of your eyes, you will notice that the object you see will seem to shift left and right. An object that is closer, will seem to shift more than an object that is farther away. That's stereoscopic vision, and the core concept behind creating the illusion of three-dimensional objects and space from two 2D images. Your brain can use this information to subconsciously calculate and tell you how far away an object is supposed to be. In a similar way, photogrammetry is a photography technique using software to map and reconstruct the shape of an object, by comparing two or more photographs. The science of photogrammetry has been around for over 100 years. It was used in World War II by the Allies to construct invasion maps, discover the V2 rocket program, and later by NASA to make topographical maps of the moon for the Apollo missions. This was an expensive, laborious procedure employing a ton of people, and massive specialized cameras and plotting equipment.
Photogrammetry has come a long way since then, and it has even come along way since I first encountered it in my professional life years ago. Now you can create a 3D model from photos with just a smartphone and a few minutes of processing--what used to take a room of specially trained people many weeks to accomplish. Photogrammetry scanning pioneers like Lee Perry-Smith from Infinite Realities, and TEN24 have turned it into an art form.
From these photos, there are a few different technologies for making 3D models that are becoming easier to use, and cheap enough for anyone to do. The most popular include laser scanning with software like David 3D Scanner, using a Microsoft Kinect with software like ReconstructMe, or consumer photogrammetry with software like Autodesk’s 123D Catch, or Photoscan.
The very best 3D scanning I’ve seen has been done with a laser scanner, but photogrammetry is not too far behind. Laser scanning also takes special equipment, whether you have to make it or buy it. Using Microsoft’s Kinect for 3D scanning is neat because it gives you real time feedback, so you know when you’ve missed a spot. It’s pretty cheap, and millions of people already own one for their Xbox 360. However, because the camera in the Kinect is relatively low-res, it is not great for fine detail. I’m excited to see what people will be able to do with the Kinect 2 in the Xbox One, once Microsoft releases developer software for it.
Compared to the other 3D mapping techniques, photogrammetry with a still camera and most of the work done in computation is relatively easy. Though not as streamlined as using a closed system like Kinect, photogrammetry gives much higher-fidelity results, and makes use of equipment that is available to virtually everyone. Because it employs just a regular digital camera, the quality of photogrammetry modeling scales well as camera technology gets better. Modern digital camera sensors are extremely advanced, and because there is so much demand, they are also very inexpensive for what they do.
Today, I'm going to give you an overview of how photogrammetry works, what consumer software and hardware is available for you to try it yourself, and how to stage a lighting environment to best conduct your photogrammetry work.
123D Catch by Autodesk is free and relatively easy to use. Your photos are processed on Autodesk's servers so you don’t need a powerful computer. By the same token Catch gives you very little in the way of configuration options, and the fine print says that Autodesk actually owns any scans you make with it.
Photoscan is fast on lower detail settings, fairly easy to use, and can produce some amazing models. While $179 for the standard version isn’t cheap, compared to buying or building a laser scanner, it is very affordable. The quality of its results is also determined by your home computer, specifically memory capacity (since you're processing many high-resolution images). If you want the finest detail you can get, you will need a machine with as much RAM as you can get your hands on. That can get pricey, but if Photoscan runs out of memory while computing, you may be in for very long waits or even crashes. 256GB is not unheard of for use with this program.
Both programs work in a similar way. You have to take a series of photographs of an object from different angles, import them to the software, which then compare features across the photographs to generate a textured 3D model. Yes, it's textured and full color!
Since the programs have very similar requirements for input, we can actually do an apples-to-apples comparison of the output. This also means we can reprocess our photos as better software comes out, or even use old images that were never meant for photogrammetry. That's where some real creativity can come in, in creating 3D models from aerial photography or even video.
But all this depends on the source footage. If the photos aren't good, then it's going to put a ceiling on the quality of your 3D model, no matter how good the software is. That's why photogrammetry is really about taking good photos--which is going to be different than a "good" photo for normal photography work. If you are familiar with 3D tracking software, like Syntheyes or Mocha Pro, you will have a good idea of what will make for good footage.
I'm going to talk about photography techniques for photogrammetry in the next piece, but the basic tenets are as follows:
Keep It Clear
Peep that pixel! You want to see those small features with as much detail as possible. If that freckle or tiny screw is blurry, find out why, and eliminate everything standing in the way of maximum sharpness. This is where more megapixels actually matters. And of course, shooting in RAW helps.
Give the software only high-confidence information. If you don't need the background, mask it out.
No Information is Better than Bad Information
Give the software only high-confidence information. If you don't need the background, mask it out. If you can't track a subject's hair, cover it up. If one image isn’t aligning correctly, get rid of it. You're smarter than the software at filtering this out before it gets to work, and you want to make its job as smooth as possible.
It’s All in The Picture
If its not in the picture, then It’s not in your mesh. Get underneath your subject to take photos, get above it as well. For heads, take a few extra pictures behind the ear. Make sure you have the coverage you need to get all the details you want, because it's difficult to go back and reshoot in the exact same conditions.
Of course, you’ll need a camera to take your source photos. A smartphone camera will do, but a nice DSLR will give you better results. I have used several DSLRs, Including the Nikon D800. With it’s almost unparalleled resolution (36 megapixels), this camera is my favorite for larger and more complicated scenes that contain small details you want to capture. The Canon 5D Mk III is a great alternative with its excellent light sensitivity, allowing you to raise the ISO without introducing a lot of noise to your photos. This lets you keep your shutter speed high, and your aperture small (which is actually useful here).
I have also used a number of other cameras for photogrammetry work. Most modern cameras will work with enough light and a good technique. But the better the camera, the more leeway you'll get with your photos and the higher source quality for your software.
A tripod is extremely useful and almost necessary when shooting photos for photogrammetry. The same can be said for your subject--something to stabilize both it and your camera while you take your photos. For a person, that can be a tall stool. for a small object, a lazy susan really helps. Even if you're shooting photogrammetry source photos with a smartphone, a tripod can give you more consistent shots. A tripod with a ball head that lets you rotate your DSLR to its side is very useful for portrait orientation shooting, to full the frame with your subject and capture as much detail as possible.
As with "normal photography", lighting is important for a number of reasons. Bright, even lighting will allow you to have a small aperture to reduce the image's depth of field. Think about that for a second. Shallow depth-of-field is actually a bad thing for photogrammetry, because blurred details confuse the software. Our goal is to have high-detail, sharp, and flat imagery. That requires closing up the aperture, which means you need more light. Good lighting will also allow you to lower the ISO which will reduce grain, and it will allow you to have a high shutter speed which reduces motion blur.
The “evenness” of the lighting itself will help in several ways, too. Even lighting will reduce the effect of highlights on shiny objects. It'll be much easier to use the texture map from the scanning program because the shadows will not be “baked in" to the photo. In uneven lighting, areas with shadows will have less detail, and will not resolve as well in processing.
If you don't have access to lights, soft flat light can be achieved easily outdoors on an overcast day. Pose your subject, and if you see almost no shadow under them (in the chin and neck areas), your lighting will be very flat and even. You can even use something white or reflective to get rid of any remaining shadows, like under the chin if you are scanning a head.
Since the software is essentially tracking patterns across multiple images, we do everything we can to help it see those patterns. Sometimes, when the patterns on your object are too complex, or not complex enough, we just need to fudge it a bit.
If you happen to have a green screen or something similar, it can help you make a digital mask to cut out unwanted portions of your image. Light the screen evenly and place it far enough away so that the color of the screen does not bounce onto your subject. Test this by importing a shot of the screen onto your computer and selecting it with the magic wand tool in Photoscan or Photoshop, or a keying tool like Keylight in After Effects.
The same phenomena that helps our green screen work (i.e. being flat and featureless) will be bad for our subject. Subjects like blank white walls, or a one color plate. You can combat this by giving the software something to track. For a big white wall place pea-sized pieces of painters tape or post-it notes on the wall, which will give the software a hint of where the wall is, relative to the subject in each shot. For items like plates you can use a crayon or a grease pen. Put these marks in a place that will be easy to remove, both digitally and in real life.
As much fudging as you can do to help the software track the details of an object, some stuff is just too complicated for the software to make sense of. Unfortunately, people, (a popular scanning subject) have a big wad of this stuff on the tops of their heads--hair is really difficult to scan with photogrammetry. Until cameras have the resolution to resolve each individual hair, this is going to be a problem. Sometimes hair will scan, but usually you get a mess that is not worth fixing. It’s usually better to avoid the problem by covering it up. A stocking, bald cap, or a thin beanie will give you the shape of the persons head and scan well.
So that's a brief introduction to the software, hardware, and basic principles of photogrammetry. Despite it looking like a dark art, it's actually really accessible. Next time, I'll dispense some tips for taking photogrammetry photos, and we'll walk through the process of photographing one object and processing it in PhotoScan. Post any questions you have in the comments below!