Everything You Need to Know About Benchmarking Android

By Ryan Whitwam

Don't waste time obsessing over the numbers. Here's what you need to know about effectively benchmarking your Android device.

The specs of Android hardware continue to increase at an almost obscene rate. Not two years ago a dual-core phone was a novelty -- the highest of high-end. Now we’re on the cusp of eight-core devices with the Galaxy S 4, and the clock speed of ARM chips is beginning to rival those of recent desktop processors. The numbers are getting incredibly large, but what does any of that mean to you? Can you just run the new 3DMark for Android and call it a day?

Benchmarking devices has been a pastime of modders and enthusiasts for years, but not all benchmarks are created equal on Android. While some have real-world applications, others are merely a distraction. Let’s take a closer look at what benchmarking can tell you.

Types of Benchmarks

Just like your PC, smartphones have numerous components that affect its overall speed, and there are tests that can be run on all of them. The first category is that of raw CPU power. A 1.5GHz quad-core chip can crunch a lot of numbers, but just how many? Apps that run mathematical calculations and spit out some numerical rating of the CPU (probably measured in MFlops or floating-point operations) are measuring the capacity of the CPU to do work.

One often overlooked aspect of smartphone design is the RAM speed. Sure, we’re all caught up worrying about how MUCH RAM your devices have, but the speed of RAM is just as important. There are few apps that measure just memory performance because it is partially dependant on the chipset, but this data is often included in comprehensive benchmarks.

Smartphones aren’t just for work anymore, and that’s why there are a glut of gaming-centric benchmarks in the Play Store. These apps are used to push the SoC, and specifically the GPU to render 3D environments with as much fidelity as possible. This often involves tests at multiple resolutions and with different engines. GPU benches may also test 2D drawing performance.

So much of what goes on inside an Android device requires the system to interact with the NAND flash storage, so of course there are ways to benchmark the speed of this component. Simply having a lot of storage isn’t going to do you any good if it’s so slow that it causes your system to lag. There are a number of comprehensive benchmarks that include I/O per-second and other measures of storage speed.

Lastly, there’s the good old browser benchmark. The nice thing about browser benchmarking Android devices is that you sometimes don’t have to install anything. Simply point the browser (be it Chrome or an OEM stock browser) at a particular website, and let it go. This kind of test is heavily dependant on CPU performance and screen resolution, but it gives data about a specific use case, so I like to think of it as a separate category.

Top Benchmarking Apps

So now we know what kinds of tests are out there, what about the apps you need to run them? The exact methods utilized by each benchmarking app vary, so I like to run a few even if they are essentially measuring the same things. A good place to start is with the AnTuTu Benchmark tool. AnTuTu is a full system test that analyzes CPU and memory performance, GPU rendering, and storage speed. I’ve found this app the be very consistent, although it doesn’t give you extremely deep information about some system aspects. It’s a good place to start at any rate.

Another multi-vector benchmark I like to run is Vellamo by Qualcomm. You might recognize Qualcomm as the company behind the Snapdragon line of ARM SoCs. This app runs two different sets of tests, one for the CPU and one for web performance (which again is heavily dependant on the CPU). One nice thing about Vellamo is that it estimates how the results will figure into user experience.

Vellamo is a Qualcomm product, and there is some concern that it favors Qualcomm parts. I have never personally worried about this -- some newer Exynos chips consistently beat Qualcomm processors in this test. The HTML5 benchmark is a standards-based test, so you can definitely take confidence in those results.

Moving on, graphics benchmarks come in all shapes and sizes. The newest (and dare I say prettiest) app in Google Play is 3DMark. This graphical testing suite has been on PCs for years, but it’s finally on Android. The app itself is small, but it will need to download a few hundred megabytes of data before it can actually do anything.

3DMark renders an OpenGL ES2.0 action sequence at 720p and 1080p. I like this test because it breaks down your graphics performance so you can see what effect your SoC is having on the results. For example, if you’re wondering how CPU (as opposed to the GPU) speed is figuring into the score, check out the physics value in the score details page.

Another pure gaming test I like to run is Epic Citadel. Not only does it offer a straightforward FPS bench of your device’s performance, but you can wander around in a beautifully rendered world. Epic Citadel runs the Unreal Engine 3, which is one of the most high-performance engines used in mobile gaming. There are still relatively few Unreal Engine games on Android, but if your phone or tablet can power through Epic Citadel, that’s a good sign.

Several of the comprehensive test packages above will analyze your storage, but Androbench is great if you want to dig deeper. This app can check the speed of your device’s internal and SD card storage (if you have one). It gives incredibly detailed results that show how a variety of system components are hitting the storage.

To wrap up your benchmarking adventure, it’s good to run a few web benchmarks if you haven’t gotten results as part of a full system test above. Even if you have, it’s probably a good idea to check out SunSpider, which determines how well your device handles JavaScript. If you run Vellamo, it includes SunSpider.

What Do the Numbers Mean?

The cold, hard truth is that some of these tests are little more than self-validation. The difference in a CPU score of a few percent is not going to make a huge difference to the way you use the phone.

Any high-end Android device is going to have a fast enough CPU to do what it needs to do. Even a mid-range device like the HTC First has the juice for a nice user experience.

More than the CPU numbers, it’s the software optimization that makes a device smooth and easy to use. If the OEM has clogged up Android with poorly implemented features, it won’t matter how high the CPU benchmarks. The same goes for RAM speed, but to a slightly lesser extent. Some phones and tablets do actually have faster RAM, which can improve the experience.

I would argue, however, that graphics benchmarks can serve a purpose. If you want to know how well a device is going to handle the hottest new 3D games, running OpenGL benchmarks is a good idea. This is the closest you can get to a real world test without playing a ton of games yourself. Most of the full system tests hit the GPU, but 3DMark is just so pretty I suggest you run it too.

The 2D drawing aspects of some GPU assays can also clue you into your device’s ability render a user interface, which relates to smoothness. Android itself is now GPU accelerated after Ice Cream Sandwich, so there is some value here.

Don’t just blindly accept what graphics benchmarks tell you, though. Nvidia’s Tegra 3 chip doesn’t perform quite as well in raw benchmarks as a newer Snapdragon does, but it may have better gaming performance in some instances. Nvidia often courts developers in order to get games coded to take advantage of Tegra chips at a very low level. So while the Tegra 3 and 4 might not be as fast in absolute terms, the actual experience you have could be better.

The storage benchmark situation is a little different. There is a real reason to check your storage speed, and it won’t even take you terribly long. A few recent Android devices have had some bugs related to storage space (the Nexus 7 being the prime example). The throughput of the NAND can slow over time, eventually dragging the system down. The exact cause isn’t clear, but it in the case of the N7 it probably had something to do with how space is allocated when things get nearly full.

A quick pass with Androbench or something similar could help diagnose the problem. With the Nexus 7, a root app could be used run a TRIM command on the internal storage, but other devices might just require you to clear off some space.

Finally, knowing a bit about web rendering and performance can be useful when the time comes to troubleshoot browser issues. If pages are loading slowly, you’ll want to know if it’s your phone, or simply network congestion. Vellamo is a good option here because it also looks at smoothness and user experience. Standards conformity will largely depend on which browser you use, and Chrome is very good in this category.

There are important things you can learn from benchmarking your Android device. You can better know what to expect out of gaming performance, storage speed, and web browsing. Raw CPU operations won’t make or break the phone, but it can be fun to push a chip for higher numbers, especially if you like overclocking.

Your experience is more dependant on software now that all the hardware is so fast, but it doesn’t hurt to know just how fast.