This post is the second in a series examining the Microsoft CDN study comparing Akamai and Limelight. The first post discusses measurement: what the study does and doesn’t look at. Now, I want to build on that foundation to explain what the study misses.

In the meantime, Akamai has responded publicly. One of the points raised in their letter is the subject of my post — why the study does not provide a complete picture of the Akamai network, and why this matters.

The paper says that the researchers used two data sets — end-user IP addresses, as well as webservers — in order to derive the list of DNS servers to use as vantage points. Webservers are generally at the core or the middle mile, so it’s the end-user IPs we’re really interested in, since they’re the ones which indicate the degree to which broader, deeper reach matters. The study says that reverse DNS lookup was used to obtain the authoritative nameserver for an IP, and the ones which responded to open-recursive queries were used.

The King methodology dates back to 2002. Since that time, open-recursive DNS servers have become less common because they’re potentially a weapon in DDoS attacks, and open-recursive authoritatives even more so because of the potential for cache poisoning attacks. So immediately, we know that the study’s data set is going to miss lots of vantage points owned by the security-conscious. Lack of a vantage point means that the study may be “blind” to users local to it, and indeed, it may miss some networks entirely.

Let’s take an example. I live in the Washington DC area; I’m on MegaPath DSL. A friend of mine, who lives a bit less than 20 miles away, is on Verizon FIOS.

Verizon FIOS customers have IP addresses that reverse-lookup to something of a scheme format that ends in verizon.net. The nameservers that are authoritative for verizon.net are not open-recursive. Moreover, the nameservers that Verizon automatically directs customers to, which are regional (for instance, DC-suburb customers are given nameservers in the ‘burbs of Reston, Virginia, plus one in Philadelphia), are not open-recursive. So that tells us right off the bat that Verizon broadband customers are simply not measured by this study.

Let me say that again. This study almost certainly ignores one of the largest providers of broadband connectivity in the United States. They certainly can’t have used Verizon’s authoritative nameservers as a vantage point, and even if they had somehow added the Verizon resolvers manually to their list of servers to try, they couldn’t have tested from them, since they’re not open-recursive.

Of course, the study doesn’t truly ignore those users per se — those users are probably close, in a broad network sense, to some vantage point that was used in the study. But note that it almost certain to be cross-AS at that point, i.e., on somebody else’s network, which means that the traffic had to cross a peering point, which is itself a bottleneck. So right off the bat, you’re not getting an accurate measure of their experience.

The original King paper (which describes the sort of DNS-based measurement used in the Microsoft study) asserts that the methodology is still reasonable for estimating end-user latency, because, from their sample data, the distance from end-user clients to the name servers has a median of 4 hops, with about 20% longer than 8 hops; as high as 65-70% of these account for 10 ms or less of latency. But that’s a significant number of hops and a depressingly low percentage of negligible-latency distances, which absolutely matters when the core of your research question is whether being at the edge makes a performance difference.

The problem can be summed up like this: Many customers are closer to an Akamai server than they are to their nameserver.

My friend and I, living less than 20 miles apart, get totally divergent results for our lookups of Akamai hosts. We’re likely served off completely different clusters. In fact, my ISP’s closest nameserver is 18 ms from me — and my closest Akamai server is 12 ms away.

It’s a near certainty that the study has complete blind spots — places where there’s no visibility from a proximate open-recursive nameserver, but a local Akamai server. Akamai has tremendous presence in ISP POPs, and there’s a high likelihood that a substantial percentage of their caches serve primarily customers of a given ISP — that’s why ISPs agree to host those servers for free and give away the bandwidth in those locations.

More critique and some conclusions to come.

Cheng Huang et.al.’s Microsoft Research and NYU collaboration on a study entitled Measuring and Evaluating Large-Scale CDNs is worth a closer look. This is the first of what I expect will be a series of posts that aims to explain what was studied and what it means.

The study charts the Akamai and Limelight CDNs, and compares their performance. Limelight has publicly responded, based on questions from Dan Rayburn.

I want to begin by talking about what this study does and doesn’t measure.

The study measures two things: latency to the CDN’s DNS server, and latency to the CDN’s content server. This is latency in the purest network sense — the milliseconds of transit time between the origin measurement point (the “vantage point”) and a particular CDN server. The study uses a modified King methodology, which means the origin measurement points are open recursive DNS servers. In plain English, that means that the origin measurement points are ordinary DNS resolvers — the servers provided by ISPs, universities, and some businesses who have their resolvers outside the firewall. The paper states that 282,700 unique resolvers were used as vantage points.

Open recursive DNS servers (I’m just going to call them “resolvers” for short) are typically at the core of networks, not at the edge. They sit in the data centers of service providers and organizations; in the case of service providers, they may sit at major aggregation points. For instance, I’m a MegaPath DSL customer; the two MegaPath-based resolvers provided to me sit at locations with ping times that average 18 ms and 76 ms away. The issues with this are particularly acute given the study’s resolver discovery methodology — open authoritatives found by a reverse DNS lookup. Among other things, this results in the large diverse networks being significantly under-represented.

So what this study emphatically does not measure is latency to the end user. Instead, think of it as latency to the core of a very broad spectrum of networks, where “the core” means a significant data center or aggregation point, and “networks” mean service provider networks as well as enterprise networks. This is going to be very important when we consider the Akamai/Limelight performance comparison.

Content delivery performance can typically broken down into the “start time” — the amount of time that passes until the first byte of content is delivered to the user — and the “transfer time”, which is how long it takes for the content to actually get delivered.

The first component of the start time is the DNS resolution time. The URL is typically a human-readable name; this has to get turned into an IP address that a computer can understand. This is where CDNs are magic — they take that hostname and they turn it into the IP address of a “good”, “nearby” CDN server to get the content from. This component is what the study is measuring when it’s measuring the CDN DNS servers. The performance of this involves:

• the network latency between the end-user and his resolver
  
• the network latency between his resolver and the CDN’s DNS server
  
• the amount of time it takes for the CDN’s DNS server to return a response to the query (the CDN has to figure out which server it wants to serve the content from, which takes some computational cycles to process; in order to cut down computational time, it tends to be a “good enough” server rather than “the optimal” server)
  

The start time has another component, which is how long it takes for the CDN content server to find the file it’s going to serve, and start spitting it out over the network to the end user. This is a function of server performance and workload, but it’s also a function of whether or not the content is in cache. If it’s not in cache, it’s got to go fetch it from the origin server. Therefore, a cache miss is going to greatly increase the start time. The study doesn’t measure this at all, of course.

The transfer time itself is dependent upon the server performance and workload, but also upon the network performance between the CDN’s content server and the end user. This involves not just latency, but also packet loss (although most networks today have very little packet loss, to the point where some carriers offer 0% packet loss SLAs). During the transfer period, jitter (the consistency of the network performance) may also matter, since spikes in latency may impact things like video, causing a stream to rebuffer or a progressive-download viewing to pause. In the end, the performance comes down to throughput — how many bytes can be shoved across the pipe, each second. The study measures latency to the content server, but it does not measure throughput, and throughput is the real-world metric for understanding actual CDN performance. Moreover, the study measures latency using a DNS packet — lightweight and singular. So it in no way reflects any TCP/IP tricks that a CDN might be doing in order to optimize its throughput.

Now, let’s take all this in the context of the Akamai/Limelight comparison that’s being drawn. The study notes that DNS resolution time is 23% higher for Limelight than Akamai, and that Limelight’s content server latency is 114% higher. However, this includes regions for which Limelight has little or no geographic coverage. For instance, in North America, where both companies have good coverage, Akamai has a DNS server delay of 115.81 ms and a content server delay of 67.24, vs. 78.64 and 79.03 respectively for Limelight. (It’s well-known that Akamai’s DNS resolution can be somewhat slower than competitors, since its much more extensive and complex network results in greater computational complexity.)

The study theorizes that it’s comparing the Akamai “as far to the edge as possible” approach vs. the Limelight (and most other current-generation CDNs) “megaPOP” approach. In other words, the question being asked is, “How much performance difference is created by not being right at the edge?”

Unfortunately, this study doesn’t actually answer that question, because the vantage points — the open recursive DNS servers — are not at the edge. They’re at the core (whether of service provider or enterprise networks). They’re at locations with fast big-pipe connectivity, and likely located in places with excellent peering — both megaPOP-friendly. A CDN like Akamai is certainly also at those same megaPOP locations, of course, but the methodology means that a lot of vantage points are essentially looking at the same CDN points of presence, rather than the more diverse set that might otherwise be represented by actual end-users. It seems highly likely that the Akamai network performance difference, under conditions where both CDNs feel they have satisfactory coverage, is underestimated by the study’s methodology.

More to come…

DataCenterKnowledge has reported that Limelight has a workaround for the Akamai patents. Limelight’s last SEC filing noted that it amended its agreement with Microsoft (to whom it has licensed its CDN technology, for Microsoft to use in building its own internal CDN), to provide them with a new version of the software that they believe is non-infringing. That suggests that they have or will have a workaround for their own network.

Also, Microsoft and NYU researchers have recently released a paper, Measuring and Evaluating Large-Scale CDNs, that charts the Akamai and Limelight networks, and offers (DNS-based) delay measurements for their DNS resolvers and content servers.

I’ll have more commentary on both topics soon, when I’ve got some more time.

Also, I have decided that I’m going to start adding stock tickers to my tags, whenever I write about something that’s likely to be of interest to investors in a particular company. Hopefully, this will help Gartner Invest clients and others with similar interests to navigate my blog.