How to Effectively Measure Broadband Speeds?

There will always be claims of product/service superiority among competing companies and the one between Smart and Globe is a classic example. The latest one is the battle between mobile broadband services Globe Tattoo and Smart Bro.

I’m sure a lot of you have seen SmartBro’s latest commercial with Mo Twister in it. If you haven’t, check this YouTube clip:

In the commercial, Smart claims that 9 out of 10 times, SmartBro is faster than the competing brand. In this case it, was implying Globe Tattoo (the blurred footage of the Tattoo USB dongle is pretty obvious). In the trials made by Smart they used Speedtest.net as their benchmark tool of choice.

Not to be outdone, Globe Telecom wants to rebut the claim and says it isn’t so. So Globe commissioned NESIC to do a broadband test for them and showed the results to the media proving otherwise.

The methodology includes both an HTPP Speed Test using 2Wire.com as well as Direct HTTP Download from Download.com and an FTP Speed Test using a 12MB file from a local server. I am sure you know what the results of the study are and who came on top.

What is more interesting is the methodology of the benchmarking they’ve done. In my opinion, using SpeedTest.net is not a reliable way to test broadband speeds, despite it being a popular tool among users. My understanding is that Speedtest.net measures the burst speed of your connection and not the sustained speed or average speed. It means that during the test window, the tool will see variable speeds (that’s while the dial meter fluctuates) and will pick up the fastest it has detected. In some sense, it is a good way to measure the maximum achievable speed of an internet connection but that’s all it does.

Here’s an analogy — imagine a Javelin Thrower in a competition is allowed 3 attempts to get the javelin the farthest. His 3 attempts were 55 feet, 125 feet and 185 feet. If SpeedTest.net were the judge, it will score the athlete a 185 feet rating. If 2Wire.com were the judge, it will give the athlete just 121.67 feet (average of 3 attempts).

So why is Globe saying that Speedtest.net is not reliable? Because it does not take into account consistency. Let’s go back to the Javelin tournament — a second athlete throws with the following distances: 145 feet, 130 feet, 155 feet. If Speedtest.net were the judge, it will give the 2nd athlete a score of 155 feet while 2Wire.com would give it 143.3 feet.

Now, which of the two athletes is the better javelin thrower?

However, there could be a number of external factors that may affect speed test for mobile broadband:

• Location of the Test Site. The distance of the cell tower from the test site can affect the performance of the benchmark as signal degrades over distance.
• Signal Strength. This is self-explanatory. Factors include distance, elevation and barriers like buildings or thick walls.
• Connection Mode. Connection mode (GPRS, EDGE, 3G, HSDPA) sometimes switches depending on which tower is servicing the connection.
• Time of the Day. Networks could get congested depending on the time of the day when a lot of users are using the network.

Rather than using benchmark tools, I’d rather propose some real-world measurements:

• Download a movie file using Bittorent. A 600MB file downloaded in about 4 hours will give you a speed of 44KB/s or 350Kbps. Longer download times will give us a better picture of the network’s consistency.
• Streaming with YouTube HD. Streaming a 10-minute HD video on YouTube will also provide live bandwidth performance of the connection. This can be augmented with a YouTube upload of a 100MB file to see upstream speeds.
• iTunes Download. What I used to do is download 3 video podcasts simultaneously on iTunes and record the incoming bandwidth using NetMeter. This will show sustained download speeds (like the one shown below).

Who has the biggest subscriber base? Who has the widest coverage? Who has the fastest mobile broadband service? Any one of them can claim these but ultimately it is the users’ commulative experience that will decide which is more reliable.