PC World tests the major services in 13 cities to see whether they deliver
By Mark Sullivan
During March and April, we spent a day testing the major 3G services in 13 cities across the United States. Verizon's service showed a combination of speed and reliability, Sprint's results lent credence to its 'most dependable' claim, and AT&T's network showed fast upload speeds in most cities.
Only after the thrill of picking out your new smart phone is over — after you’ve marveled at all the stylish new gadgetry and features, signed a service contract and finally laid your money down — do you finally get a feel for the speed and reliability of the wireless service that connects your new phone. That wireless service — so often an afterthought to smart phone buyers — is hugely important: it connects your new phone to all the fun and useful apps and services that made you lust after a smart phone in the first place.
The truth is, the 3G wireless networks of today are not perfect (they are, collectively, a work in progress), and they are not all created equal.
Because independent research on these networks is very hard to come by, PC World took a single-day, real-world snapshot of the performance of the three biggest 3G networks in the U.S. — Verizon Wireless, AT&T and Sprint — using industry-accepted testing technology and techniques. If there’s a smart phone in your future, we hope to give you some idea of the wireless service that may be in store for you, beyond the anecdotal information you hear from other users, on the web and in the media, and aside from the claims made by the wireless service providers themselves.
Now for an important note before we start reviewing our results. Wireless signal, by its nature, is extremely variable; that is, many things, such as obstruction by fixed objects (buildings, trees, etc.), weather, network load, cell tower locations, and time of day, can affect the quality of the signal. These factors can cause service from a single wireless service to vary widely from day to day and from neighborhood to neighborhood. Our results, taken together, provide a snapshot of the performance of the largest 3G networks in 13 major markets during March and early April. But they are by no means exhaustive, and your own connection speeds may differ from ours.
During March and early April, our testing partner, Novarum Inc., used Ixia ixChariot testing software to measure network performance from more than twenty fixed locations in each of the following cities: Baltimore, Boston, Chicago, Denver, New Orleans, New York City, Orlando (Florida), Phoenix, Portland (Oregon), San Diego, San Francisco, San Jose, and Seattle. In all, our testing partner ran 5443 individual tests from 283 testing locations. At each location, Novarum measured download speed, upload speed, and reliability for each provider's 3G service. (See "How We Tested and What the Ratings Mean.")
Testing results in a nutshell
In Novarum's tests for us, Verizon Wireless demonstrated a good mix of speed and reliability. Across more than 20 testing locations in each of the 13 cities we tested, Verizon had an average download speed of 951 kbps. Verizon demonstrated good reliability, too; the network was available at a reasonable and uninterrupted speed in 89.8 percent of our tests.
Sprint's 3G network delivered a solid connection in 90.5 percent of our 13-city tests. Sprint's average download speed of 808 kbps across 13 cities wasn't flashy (at that speed, a 1MB file downloads in 10 seconds), but dependability is an important asset. The Sprint network performed especially well, both in speed and in reliability, in our test cities in the western part of the United States.
The AT&T network's 13-city average download speed in our tests was 812 kbps. Its average upload speed was 660 kbps. Reliability was an issue in our experience of the AT&T system: Our testers were able to make a connection at a reasonable, uninterrupted speed in only 68 percent of their tests.
Somewhat surprisingly, our testers also found that the "bars of service" readings on their phones were rarely an accurate predictor of the quality of the ensuing connection. In most places and with most wireless providers, the "bars" did little more than indicate whether the phone had access to some service or to no service. (See "What Do Bars Say About Your Connection?")
As advertised
Do wireless providers deliver the connection speeds they promise for their 3G networks? In our tests, on average, they did. However, the services promise speeds within a wide range — if they provide a low end to the range at all — due to the wide variability of network performance from day to day and from neighborhood to neighborhood. So in practical terms, these ranges don't represent much of a commitment to consumers.
Verizon says publicly that its wireless customers can expect download speeds of up to 1.4 mbps and average upload speeds of 500 kbps to 800 kbps from its 3G network. Verizon came reasonably close to those speeds in our tests, delivering downloads at an average rate of 951 kbps across 13 cities, and uploads at an average rate of 426 kbps across the same cities. At these speeds, a 1MB file would download from the Web in 8.4 seconds or upload to the Web in 18.8 seconds. Verizon, which claims to have the "largest and most reliable" wireless network in the United States, delivered a reliability number of 89.8 percent across our tests in 13 cities.
Sprint easily delivered on its dependability promises in our tests, with a reliability rate of 90.5 percent. (The company claims to have "America's most dependable 3G network.") Sprint promises average download speeds of 600 kbps to 1.4 mbps, and average upload speeds of 350 kbps to 500 kbps. In our tests, Sprint came through on both counts, with an average download speed of 808 kbps and an average upload speed of 377 kbps. At those speeds, a 1MB file would download from the Web in 10.1 seconds, or upload to the Web in 21.6 seconds.
AT&T claims to be the "nation's fastest 3G network" and promises download speeds between 700 kbps and 1.7 mbps, and upload speeds between 500 kbps and 1.2 mbps. While we can't crown AT&T as having the nation's fastest network, we found that, on average, AT&T delivered upload and download speeds that fell within their promised ranges in our tests.
Verizon speeds fast, consistent
Verizon Wireless did very well in our speed tests in the central and eastern parts of the country, especially in the cities of Baltimore, Chicago, Denver, New Orleans, New York, and Orlando.
Verizon achieved its fastest speeds in New Orleans, with averages of 1425 kbps for downloads and 550 kbps for uploads in our tests. The company's single fastest connection speed among the 13 cities where we tested was just over 2.3 mbps (also in New Orleans). Its poorest showing in our results came in Portland, with download speeds of 622 kbps and upload speeds of 410 kbps on average.
Verizon says that since the company formed in 2000, it has invested $50 billion in its wireless network. As of January 2009, the network supported some 80 million subscribers, the company says.
Sprint fast in West Coast tests
Sprint's test results were competitive with those of Verizon Wireless in most of the cities where we took samples, especially in the West Coast cities of Portland, San Diego, San Francisco, and Seattle. Sprint's top speeds came in our Seattle tests, where it clocked an average download speed of 1005 kbps and an average upload speed of 469 kbps. Sprint's single fastest download speed in our results was a mark of 2.1 mbps in San Francisco. Sprint's worst average performance was in our New Orleans tests, with a download speed of 599 kbps and an upload speed of 240 kbps.
Sprint won't divulge precisely how much it has invested in its 3G network, but it does say that it has poured $18 billion into its wireless and wireline networks since 2006. The carrier's 3G network, which it launched in 2005, uses the same 3G wireless protocol used by Verizon Wireless-CDMA, EvDO Revision A. This may have something to do with the comparable performances of the two networks.
Sprint says that it has moved well down the 3G road, quickly shifting its subscribers away from older networks. "The vast majority of our customers use our 3G network, and the vast majority of the Sprint Mobile Broadband Network has been upgraded to faster [3G] EvDO, Revision A technology," says Sprint spokesperson Stephanie Vinge-Walsh. "Use of 2G continues to phase out; we expect those few customers still using our 2G network to upgrade as they upgrade their devices."
But unlike Verizon's, Sprint's wireless subscriber base has been shrinking. The network connected 49.3 million customers at the end of 2008, compared to 53.8 million at the end of 2007. Has the decrease in number of customers reduced the load on Sprint's network and contributed to its solid performance in our tests? Sprint's vice president of network development and engineering, Iyad Tarazi, insists that the two things are unrelated and that traffic on its 3G network has continued to increase throughout the past year.
Sprint will be the sole provider of the much-hyped Palm Pre smartphone, which it hopes will quickly increase its subscriber numbers. The iPhone certainly lit a fire under AT&T's wireless business, and the right killer device might work for Sprint, too.
AT&T speed results
In our 13-city testing, AT&T's 3G network produced download speeds that averaged 818 kbps, and upload speeds that averaged 549 kbps. The network clocked some of it best results in our tests in Boston and Chicago. In Boston, AT&T's 3G network delivered an average download speed of 1259 kbps and an average upload speed of 708 kbps in our 20 test locations in the city. AT&T's Boston network also proved available at a reasonable speed in 90 percent of our tests there. In Chicago, the AT&T network clocked an average download speed of 1148 kbps and an average upload speed of 712 kbps. In New York City, the company's network delivered an average download speed of 502 kbps and an average upload speed of 308 kbps across our 20 test locations there. AT&T delivered relatively fast upload speeds (549 kbps on average) in the 13 cities we tested.
"AT&T stands behind its claim of providing the nation's fastest 3G network, as verified by two independent third-party testing companies," says AT&T spokeswoman Jenny Bridges. "The third-party companies that measure the network performance of AT&T and its competitors conduct comprehensive drive tests throughout the year, spanning more than 1 million road miles in more than 340 markets and more than 1 million 3G data sessions combined," Bridges adds.
"Beyond this third-party testing, AT&T conducts millions of its own tests across the nation to measure performance and maximize the service quality and reliability of our network. For comparison purposes, we also test our competitors' networks, as they do ours. We believe this combined data provides the most thorough, comprehensive view of the network performance of AT&T and its competitors."
Over the past few months, AT&T has announced plans for big improvements to its 3G networks. The company announced in March that it would invest $17 billion to $18 billion in its networks in 2009, "two-thirds of which will extend and enhance the company's wireless and wired broadband networks to provide more coverage, speed and capacity."
More recently, on May 27, AT&T announced plans to increase the speed of its 3G service by upgrading its networks to the faster High Speed Packet Access (HSPA) 7.2 technology, utilizing more radio-frequency spectrum, increasing backhaul capacity, and adding 2100 new cell towers. The company says that it will begin the upgrade this year and expects to complete the process in 2011.
Of course speed isn't everything. The fastest wireless connection in the world is worth little if service gets interrupted in the middle of a session. In each city, we measured how often the AT&T, Sprint, and Verizon networks delivered poor service (similar to or slower than dial-up speed), interrupted service, or no service at all. If any of these conditions occurred during one of our tests, we labeled service at that location "faulty." Thus, if 10 out of 20 tests were faulty in a given city, we gave the operator a reliability score of 50 percent. Using that standard, we calculated an average reliability score for each service from our thousands of individual tests.
Sprint's 3G network delivered high reliability scores, especially in our testing in Boston, Chicago, Denver, Portland, San Francisco, and Seattle. Sprint's network proved reliable in 90.5 percent of our numerous tests across 13 cities. In fact, in Boston, Denver, and Seattle, Sprint's service earned perfect scores for reliability, proving available at reasonable speeds in 100 percent of our tests.
Sprint's vice president of network development and engineering, Iyad Tarazi, wasn't surprised by the reliability results. "We've invested heavily in our 3G network; we've made it the most critical part of our network," he says.
Nationwide, Tarazi says, Sprint simply has more base stations on the ground than its competitors do, which leads to higher reliability. "On the reliability side, you're seeing something very similar to our own [network testing] numbers," Tarazi says of our results.
Verizon's EvDO service was available at a reasonable speed in 89.8 percent of our 13-city tests. Verizon's network showed impressive reliability results in Baltimore, New Orleans, New York, Phoenix, and San Jose. In New Orleans and New York City, Verizon's 3G service showed perfect reliability throughout our testing locations in each city.
What do bars say about your connection?
If you've ever wondered why your phone was showing five out of five bars and yet your service was poor, you're not alone. In one of the most surprising findings of our study, we discovered that the number of bars showing on our phones was by no means an accurate predictor of our connection speed or of network service reliability. In most cases the bars were useful only for telling whether we had some signal or no signal.
Phoenix is the only city we tested where the bars-of-service reading did a good job of predicting service quality. There we saw a correlation between bars and service quality in more than 70 percent of our tests, across all carriers. In the other 12 cities where we tested, bars of service were far less meaningful. In San Francisco, for example, only 13 percent of our tests showed any correlation between bars and service quality.
Our test results suggest that the bars-of-service readings tend to be a more accurate predictor of service quality on networks that have enough capacity to handle all of the devices connected to them. Overall, the bars on Verizon devices did the best job of predicting service quality in our testing. We saw at least some correlation between number of bars and Verizon's network performance in 11 of 13 cities. Bars were a moderately useful predictor of Sprint service in our tests in only two cities—San Diego and Phoenix.
In our tests of AT&T's networks in four cities (New York, San Diego, San Francisco and San Jose), the number of bars was not predictive of the service quality we saw; in the other nine cities in our study, the number of bars proved marginally useful in predicting how well our test experience would turn out.
The problem of backhaul
Backhaul refers to a wireless operator's method of moving data back and forth from the cell tower to the wireline Internet backbone. Bars of service measure your phone's connection to the cell tower, but that's just the first part of the trip. Bars offer no clue as to how smoothly and quickly the backhaul network is connecting with the Internet. That's why your phone may show the optimum five bars even though your service seems to be taking forever.
Let's suppose that you have keyed in 'www.nytimes.com' on your smartphone. Your phone shows five bars, indicating that it has connected very quickly to the nearest cell tower. The cell tower then sends your request over the backhaul network, which connects to the system of fiber-optic and copper lines constituting the backbone of the Internet. Over this system, your request reaches the server that contains the New York Times content. But again, if the backhaul system is slow or overloaded, it creates a bottleneck in the system, lengthening your response time even though your device connects to the cell tower quickly.
Understandably, wireless carriers are investing heavily in more-efficient networks and technology to backhaul wireless data, which is rapidly increasing in volume. Perhaps the fastest and most reliable backhaul strategy uses fiber-optic cable to route wireless data back and forth between the cell tower and the Internet backbone. The carrier leases, buys, or builds new fiber-optic lines. To reach cell towers that are far away from existing fiber-optic infrastructure, mobile operators will increasingly use wireless approaches such as ethernet over microwave technology as a less expensive alternative to fiber for backhauling traffic.
Over the next year, it will become obvious that the iPhone and the Blackberry aren't the only cool smartphones on the market. New models such as the Palm Pre and a wave of new Android-based units will emerge to compete for the many consumers who haven't yet caught the smartphone bug. Mr. Smartypants is one such consumer; and when he goes shopping for a phone, he asks a lot of questions that any smart smartphone shopper should. Some of these questions are about the phone itself; others relate to the network that will connect the phone to the Internet.
Smartypants: How many cell towers do you have in town?
Store rep: Umm. I'll have to call the main office to get that.
Smartypants: About how many wireless subscribers does each cell tower support?
Store rep: Oh, yeah, I, uh...I don't think the company gives out those numbers.
Smartypants: Is the chipset in this phone optimized for the flavor of 3G service you're selling-you know, 1xEvDO Rev 0 or Rev A? HSPA or HSDPA+?
Store rep: Yes! This phone does have a very large screen!
Smartypants: What kind of wireless backhaul does the network use? Fiber?
Store rep: Well, see here, on the screen, if all five of these little bars are lit up, that means it's, uh, backhauling well...and those bars are always lit up!
Smartypants: Can this phone be converted to connect to a 4G network when one becomes available? And by the way, when will you offer 4G service here in town?
Store rep: Look, buddy—Sprint's right across the street. Go bother them.
It may seem as though Mr. Smartypants was just dipping into his knowledge base to give the store rep the needle, but all of his questions focus on issues that can dramatically affect a 3G network's performance. Wireless companies ought to be ready and willing to provide accurate answers to these questions. Why? Because their customers have a right to know what they're buying.
How we tested and what the ratings mean
We decided to test the three major 3G cellular wireless broadband providers in 13 U.S. cities that we judged to be broadly representative of the locales where most customers are likely to use these services. In each city, we randomly chose 20 test locations, evenly distributed over the metropolitan area. We performed all of our tests inside a parked car.
We created a 1-minute stress test to evaluate the quality and performance of the wireless service. We tested network delay, upload speed, download speed, and reliability, as well as the correlation between "bars of service" and network performance.
We conducted the tests using industry-standard wireless-testing software (Ixia Chariot) running on a Windows XP SP3 laptop. We tested on a laptop, rather than on a smartphone, because we needed the laptop's processing power to run Ixia's rigorous 1-minute tests, and because a laptop can test the strengths and weaknesses of the network more accurately than a cell phone can. To connect to each network, we used the latest USB modem from each vendor: AT&T's USBConnect Option Quicksilver, Sprint's Sierra Wireless USB 598, and Verizon Wireless's Novatel Wireless USB 727. All of the client adapters we used came from the respective vendors and were recommended by the outlets where we purchased them.
Test definitions
Download speed: the average speed (in kilobits per second) at which we downloaded random data from a known Internet server during a 1-minute streaming test.
Upload speed: the average speed (in kilobits per second) at which we uploaded random data to a known Internet server during a 1-minute streaming test.
Reliability: the percentage of tests for a given city in which we could detect a signal, connect at a reasonable speed (faster than dial-up), and sustain an uninterrupted connection for the duration of a 1-minute streaming test.