We’ve gone over the basic concepts and photography techniques on how to capture ideal images for photogrammetry 3D scanning. Now let's get into the meat of the subject and start processing our data so we can see some results. The case study we're going to use is a replica prop from the movie Hellboy, which I found at the Tested office. I spent an afternoon photographing the prop, and processed it using PhotoScan software. Here's how that process went, and what you can learn from it.
Step 1: Inspecting Your Photos
For this photogrammetry scan, I used the turntable method to capture photos of the prop pistol, the “Samaritan” from the movie Hellboy. Since the prop is an irregular shape, I didn't put it on an actual turntable or Lazy Susan. It’s propped up on the end of a C-stand pole, which allowed it to be turned a few degrees between shots. I took one "ring" of pictures from slightly above the prop, and another one below. That gave me about 45 photos total. Click here for an example of how one full rotation of photos looked.
Since the front and the back of the gun aren’t visible from the main sequence, I took another set of photos of the front, and another of the back of the pistol.
Below is an example of how one of the photos looks up close, which shows good detail for processing. At 100% crop there are plenty of small details for the software to track.
But here is one where I’ve missed the focus. The details begin to fall off right after the hammer of the pistol. Shooting the photo at f/8 helped, giving it a wide depth of field, but the detail of this photo is much lower than the others. When you are even a little bit out of focus, you negate the advantage of a high megapixel camera. If I had to do it again, I think I would try an even smaller aperture, f/16 or above.
Step 2: Color Correction / Export
Photoscan can ingest several different RAW photo formats, but if your format isn’t fully supported, the recommended method is to save your photos as 16-bit TIFF files. This conserves the additional depth you gain from shooting RAW.
The 36 megapixel .NEF files from the Nikon D800 totaled about 1.8 GB. That's a lot of disk space just for images! Converting them to TIFF actually raised the total to about 2.2 GB. And that's with compression. Since there is so much data involved, a fast storage drive like an SSD will speed up your processing time.
Step 3: Importing into Photoscan
To import your photos into Photoscan, simply select the images in explorer and drag them to the workspace window in the upper left of the program.
Look through your photos and make sure they look like how they did when you viewed them in Photoshop or any other imaging program. Sometimes Photoscan's RAW module will only be semi-compatible with certain camera formats, and will produce large color shifts and other glitches during the import. In that case, saving as TIFF is the way to go.
Step 4: Give it a Shot / Processing
If it looks like you have a good series of photos from your capture session, processing them in Photoscan is relatively straightforward. The software includes a basic workflow menu that you can simply follow along, one step at a time. If you are lucky, you will end up with a great mesh that you can print or animate! But in my experience, this happens maybe a tenth of the time, under the most perfect conditions.
The four steps to processing are: Align Photos, Build Dense Cloud, Build Mesh, and Build Texture. I'll run through what each means.
Align Photos is the most important process for a quality output. It requires that you place each of the photos in the position where the camera was, relative to the subject, but in Photoscan's virtual space. Basically, you're helping Photoscan realize where each photo was taken, and from what angle, distance, and elevation. After you align, look at the positioning of the virtual cameras. Make sure they are where you expect them to be. If you see some cameras in a position you did not take a picture from, right click on that image and reset the alignment. Then try to align it again. If you have enough images, you can just leave it unaligned.
The most computationally intensive process is Build Dense Cloud. Start with the low or medium settings. Photoscan tends to be very RAM hungry, if you have a large number of high-res photos, and you set Build Dense Cloud to Ultra, you can consume dozens to hundreds of gigabytes of RAM. And often, the increase in quality will be subtle between these settings. When you run out of RAM, Photoscan will keep going, but slow down considerably. I have had a process take over 24 hours. Make sure you save and close other programs before you start processing!
Build Mesh will take your point cloud and turn it into 3D Geometry. You can adjust how far it will try to fill a hole in the model. If the number is very large, you may end up with large pillowy structures on your mesh. Before you build your mesh, take this time to delete any unnecessary points, which will save you some clean up time later.
Build Texture will look at the images and turn them into a texture map to put onto the new 3D model. An advanced user can export the model, clean it up manually, do the texture mapping, and re-import the model into Photoscan.
In the event that your images do not all align nicely, we can use a few techniques to help Photoscan along. One of these is “Masking”. Notice that there is a large portion of the images that we don’t need to be part of our model. We want the software to be focusing on the important stuff. To get rid of this unwanted part of the image we will have to mask it. That means selecting an area that you want Photoscan to ignore. This will nearly always need to be done for the turntable method, because the background is not staying relative to the subject as it rotates.
As you can see on the images of the front and back of the pistol I have even masked out portions of it that begin to get too blurry to be useful, and only included the parts I need for the model. I also left in the parts that will be useful for alignment to the other images.
Name the masks the same name as the original image, with a suffix like “_Mask”. So your photo would look like “IMG123.tif” and the mask would look like “IMG123_Mask.tif”. Then, in Photoscan, import with Tools>Import>Import Masks...
Out-of-Order Alignment, and Markers
You can solve separate parts of your model using different “Chunks”, and then align them back together using Markers. Photoscan seems to align images in alphabetical order, not all at once. and sometimes this order isn't the best. This means you can potentially get better results by aligning images by twos or fours, then one by one. Sometimes this will give you a nice solve, but sometimes, it will refuse to align perfectly good looking images, or images that aligned well when you tried them first.
This is where “Chunks” and “Markers” come in. In the left panel select those stubborn images (Cameras) that didn't align, and right click and choose Move Cameras > New Chunk. Align them separately, and create markers (at least three, and not too many more) on small details visible from both chunks. The marker on the detail on both chunks should have exactly the same name.
Then use “Align Chunks” and “Merge Chunks” from the workflow menu to bring the chunks together. Place markers on spots that vary widely in depth In all three axes. The wider apart the markers, the more accurate the alignment will be.
Step 5: Inspecting the Output
Photoscan has worked well into the night. Let’s see what it's given us! With a few hours of work we’ve taken some photos from a regular camera, and turned them into a digital replica of our Hellboy pistol. Though the surface is a little noisy, Photoscan has done a great job replicating the size and shape of all the features. Look for excessive noise, or splits in the model. You can re-align cameras or chunks to help solve these problems.
This is awesome if you want to print it out or use as a base to make a game model, it may even work as a full size prop, after some physical cleanup. Scans are often accurate enough to use as a guide for making replacement parts or accessories that work with the original item.
Step 6: Cleanup and Export for 3D Printing
So the model looks great, but it’s not perfect. The surface is bumpy and there is some missing detail. I did some minimal cleanup and and then exported a file to print. I used Maya to align the model to the world, reduce the polygon count, and split the model in half. There are free programs like Blender, Netfabb, and OpenSCAD that will let you do the same thing.
Let’s Check Out Our Prints!
Splitting the model in half lets it print with very little support material, because it can lay flat on the build platform. The trigger guard snapped off it could use some re-enforcement. I used a .1mm layer height, and it took about four hours to print.
It looks pretty sweet--even the embossed BPRD seal on the grip is visible. The small scale noise in the scan seems to actually help. It reduces the perceived stepping created by the layers, without interfering with the form too much. I rather like the effect.
While I think this would hold up OK at full size, the scan would allow you to quickly remodel the surface, giving you the opportunity to have a spinning cylinder, removable bullets, a hammer that pulls back and a trigger that you could squeeze.
For me, photogrammetry has gone from an amusing novelty to a viable way to create models for professional use. Being able to create lifelike 3D models with a single digital camera is amazing. Along with 3D printing, I expect to see photogrammetry to start being used In all manner of applications.
(And as a bonus, here's another 3D model I printed using photogrammetry, while at the Tested office. Let me know if you're interested in reading about how I processed and cleaned up this one!)