Skip to content
April 28, 2013

Android Benchmarks: Why it shouldn’t matter that much

A lot of people, tech-heads specifically, would look at the benchmark scores of a specific Android device before making any purchase decisions. We don’t normally do this when buying other devices with different mobile operating systems. Which brings us to the question — do Android benchmarks matter?

FIRST WORLD

Introduction to Benchmarks

First of all, let us explain what benchmarking is. Benchmarks are usually done to measure the performance of a certain device. It’s a numerical value we use to compare with other Android devices. In essence, it’s the measure of how fast a specific hardware can perform a set of tasks.

The Google Play Store is loaded with a lot of apps for such a purpose. Each benchmark too has a specific parameter to measure. The performance of your device is represented by these major components — the processor, RAM, graphics, internal storage.

While Android has a lot of benchmark tools in the Play Store compared to any other platform, some are also cross-platform.

Antutu is very consistent with its measurements of CPU speed, GPU rendering, storage write speeds and etc., while Vellamo does the same with slight differences, plus HTML5 benchmarking. Quadrant is also somewhat similar.

For the graphics, the gaming performance and all of that, there’s 3DMark, Epic Citadel and Nenamark 2. 3DMark just became available for Android, and it’s a heavy duty tool since you need to download nearly 300MB of data. It’ pretty comprehensive and is also available for iOS and Windows RT/8.

Nenamark 2, which we also use as a standard in our reviews, focuses on OpenGL|ES 2.0 benchmarking. Epic Citadel is also quite heavy, as it puts your device through Unreal Engine 3 with intensive graphics and measure frame rates.

Benchmarks can be inconsistent and unreliable. There’s also the fact that it can be manipulated. Here’s why:

People don’t usually go through all of that suite of benchmarks just to judge a specific device. A few tests or so will probably suffice but we can’t really let that be the main basis for the device’s performance, and we want to explain the logic behind this.

Here are the Antutu scores of devices running on a 1.5GHz dual-core Qualcomm MSM8260 Snapdragon, with the same GPU and RAM (1GB):

  • Sony Xperia S: 8418
  • Samsung GS2 Skyrocket (US variant): 6,336

Through that, you can see there’s discrepancy of around 2,000. Benchmarks can be quite inconsistent. The Xperia S features a higher resolution display so that’s a factor, though it doesn’t really make significant discrepancies. For reference, the HTC Sensation on a lower 1.2GHz and lower 768MB RAM scores in at 3,932 (that being qHD).

A bigger flaw is noticeable when you look at this next example. The Samsung Galaxy S2 has been very known for its Exynos processor, which raged through the benchmark charts in the past, and that pretty much started the Exynos craze. Due to network incompatibilities with US carriers, Samsung had to switch the Exynos with a Snapdragon (the SGS2 Skyrocket). A lot of people found this disappointing due to difference in benchmark scores of the variant.

Turns out, they were wrong. When the Skyrocket was tested, it got scores near to the dual-core Exynos for Quadrant, Antutu annd others. The interesting thing to note here is that HTC devices such as the Amaze 4G, which ran exactly similar specs to the Skyrocket safe for the qHD display, never went near the scores of the Skyrocket.

You can check out the specs of the devices here for confirmation.

COMPARISON

Here are the scores of the devices from the reviews of GSMArena (other sites’ benchmarks are approximately the same):

  • Amaze 4G on Quadrant: 2,793
  • Skyrocket on Quadrant: 3,224
  • Amaze 4G on Antutu: 3,641
  • Skyrocket on Antutu: 5,881

It’s very suspicious to see here that Samsung has a huge lead with the benchmarks when the configurations are almost the same. Samsung may have tinkered with a few elements to keep its title as the benchmark king. Plus, if that’s not enough, you should see another anomaly in our Starmobile Diamond review, where the Quadrant scores were suspiciously high.

Oh, and more on Quadrant; Android Central has an article and a video on how to play with the software to make your device give out higher scores than usual, without having to tinker with the CPU and all that.

Benchmark scores may also be affected by a lot of external factors — OS versions can greatly affect it (Jellybean has improved a lot of scores of older devices), background services and apps can also affect it, CPU throttling of some chipsets (like Atom Z-series) will also reflect erratic results.

Conclusion

There are still a number of people out there who would look at benchmark scores and immediately conclude that they should get that device after seeing superior numbers. We’d like to advise caution and give pause. From low-end phones to high-end ones, benchmark scores can be manipulated. After all, these are just apps, and hardware can be optimized to make sure they get good results on these benchmarks.

Even if you’re looking at the high-scorer out there (at the time of this writing) like the Samsung Galaxy S4, you can’t really tell the difference anymore as high-end Android phones have already eliminated the lag. Probably, the only thing worth looking at now is how well it works with software (as Windows Phone devices run smoothly in the UI even with low specs), and how great it is in handling graphics and power efficiency.

why

In the end, we still think people will still look at benchmarks. We advise that you don’t base your decisions solely from the numbers. Read reviews, watch hands-on videos, and ask people around you who have experienced the device. Real world experiences can never equal any glowing spec sheets and benchmark numbers.

{source}

5 Android Games You May Want To Try #2
Android 4.3 update for Xperia SP now rolling
Android 4.4.3’s alleged list of bug fixes enumerated

24 Responses to “Android Benchmarks: Why it shouldn’t matter that much”

  1. Freeje says:

    Wait until the device has had a lot of reviews then make a decision.

  2. Mr. Curious says:

    Pretty ironic in my opinion.

    Isn’t it that you guys are the ones who keep on posting several benchmarks on almost ALL phone/tablet reviews?

    ‘There are still a number of people out there who would look at benchmark scores and immediately conclude that they should get that device after seeing superior numbers.’

    It has been a signature move from this site to put multiple benchmark results on prod reviews. Ofcourse people will ‘look out’ for benchmark results when you constantly post them.

    • Zo says:

      I actually have to agree with Mr. Curious on this one.

      If reviewers keep posting benchmark scores then undoubtedly, readers and consumers would keep looking for them.

      But that’s how the ball started rolling and if the reviewers stopped posting scores, readers would start asking for them.

      What a fix we’ve found ourselves in hehe

      In my personal opinion:

      For an everyday phone: Look for a phone that has software and OS that wisely manages and maximizes the capacity of its resources.

      Then look to the scores if you use your device for heavy gaming… but that’s what my Xbox, PS3, PC rig is for… I’m happy with a few rounds of 2Fuse on my phone while waiting in the airport. :)

  3. RaGe
    Twitter: freelancerage
    says:

    paalala ito sa mga adik sa benchmarks

  4. Asd says:

    Shut the fuck up freking dork. Wala ka talagang magawa sa buhay no? Pati ba naman to kokopyahin mo? My gaaawwd. So pathetic.

  5. Marc says:

    One of the better articles that I’ve read on Yugatech in a long time. Solid, practical advice and not just pics and stats of the latest tech toys. Hope we see more of these in the future.

  6. bench says:

    Galing ng writer, idol!

  7. Raul says:

    @Bob. Sabi mo in comparing the Xperia S and the GS2 Skyrocket, “The Xperia S features a higher resolution display so that’s a factor, though not really as significant as screen resolution does.”

    What do you mean?

    • Bob Freking says:

      Screen resolution may affect the benchmark scores since that means the system has more or less pixels to push.

      Software may most likely be optimized for devices with different screen resolutions, so the discrepancy may still be little. Sorry if it caused quite a confusion.

    • Calvin says:

      bob, i think what he meant was this confusing statement:

      “The Xperia S features a higher resolution display so that’s a factor, though not really as significant as screen resolution does.”

      It seems that you’re talking about the high resolution display and compare it to screen resolution. hmm?

    • Bob Freking says:

      Updated it to erase confusion.

  8. luther says:

    forget the benchmarks, get a Life.

  9. . says:

    i don’t get it, better scores = optimized firmware = more responsive phone. tell me why i should not look at it? i get the benchmark manipulating thing but a chunk of your argument does not make sense (to me at least)

  10. rhk111 says:

    I highly disagree with this article, and I am in fact a believer in benchmarks.

    While I won’t entirely base my decision on benchmarks, I still feel they provide a good, consistent view of how a system OVERALL is performing.

    The author cites different benchmark results for phones with similar CPUs and GPUs, and uses this as an example of how inconsistent Benchmarks can be.

    However, he also fails to consider other factors like Maximum RAM, Current Available RAM and even screen resolution that affects overall the system’s performance. It is very disappointing to see this type of a mistake from a widely-read blogger …

  11. abuzalzal says:

    If you’re into games then benchmarks will absolutely matter….

    Tapos ang debate

  12. this article does not make sense! maybe u need to do more RESEARCH BOB! phone manufacturers rely on benchmarks! go back to school and learn how to research pls

  13. daniel says:

    sa sobrang bilis na ng processors at GPU ngayon… hindi mo na mahalata ang difference betweeen a high-end and low-end… nagkakatalo nalang if i-pu-push mo talaga sa limit ung system mo… which is very impossible. Will you play 10 movies simultaneously? Your quadcore might do that…. but a dual core cannot. and the frequencies too…. so benchmark doesnt really matter that much… its just 20% of the judgement :) for me.

  14. ferdinand marte says:

    The only thing that matters to me is if the device provided satisfactory performance given the price i paid for it.

    • BertSingson says:

      Very well put, pare! I recently bought my daughter an Alcatel Glory 2 for P3,900! It does not have LTE or a dual core processor but it does FB, Twitter, Instagram and runs all the apps a high schooler uses. And it does it rather smoothly to my surprise. I didn’t rely on any benchmark when I bought it but for the amount I paid for this Jelly bean device and the things it can do, I can say it is more than sulit.

  15. Phillip says:

    Parang grades lang yan nung college. Though it may be a reference sa study habits and work attitude, it won’t necessarily mean na pag mataas ang grades, perfect employee ka na for a specific job. :)

  16. DDoS says:

    Seriously?
    Oo namamanipula yan, which means na na-iimprove talaga ang performance ng isang phone.
    SIMPLE LANG.
    mataas na benchmark scores = good phone.
    ogic naman.

  17. CherryMobileSucks!! says:

    Sa palagay ko may point si Bob Freking… cherry mobile sucks

Leave a Reply

*
*

Written by

Bob Freking writes, edits, & renders for YugaTech. He is also currently studying Business Administration in UST. You can contact, follow or tweet him at @bobfreking. For all his social media links, you can go to https://tinyurl.com/bobfreking

More articles by Bob Freking :