Essential considerations of CDN performance monitoring

I wanted to share some musings around performance and Content Delivery Networks.

As most (all?) readers will know, CDN acceleration is one of the more established and widely used of the many performance-centric interventions now available. A recent study by Catchpoint systems [2012] on the top 50 (US) retail sites identified 64% of third party services being represented by one major independent CDN provider alone.

The same study pointed out that a very high proportion (some 50%) of these major retailers do not optimise their CDN utilisation through domain sharding.

This in some way leads me to my key point: CDN adoption is not a (relatively) simple matter of setting up a few test points and quoting availability and latency. It could be better than nothing, and any efforts in this area – such as the comparison portal under development by Juho Vepsäläinen and his team elsewhere should be applauded – the need certainly exists.

CDN – not “one size fits all”

CDN adoption is far from a ‘one size fits all’ decision. The market itself is both large and volatile, with the contending capabilities and claims of Independents (Akamai, CD Networks, Limelight and many others), platform based offerings (e.g. Amazon Cloudfront) and ISP based services – often themselves delivered through licencing agreements with independents, e.g. Deutsche Telecom and EdgeCast.

The broad choice is of less relevance than the inherently dynamic nature of the problem. At its most basic level, CDN usage is far from binary – a given site may well employ several CDNs either directly or indirectly (through third party content). So what are some of the core questions?

  1. What are you accelerating? Depending upon the nature of the site and its usage, all or part may be accelerated. Is CDN caching applied just too large static images (for example), or all content?
  2. Where is it going? –what are the key global markets? – CDN providers differ widely in their capacity across the globe (indeed, some are regional niche players). Which primary regional ISPs are involved in web delivery?
  3. What is the nature of your core content and to whom is it principally delivered?
  4. Web HTTP and/or Video (total download, adaptive streaming etc.), PC browser, Mobile device, other (gaming console?)

Specific considerations apply in optimising each of these areas. These may or may not be embraced by a particular product or service. As an example, delivery optimisation to mobile devices may involve test compression (to reduce overall payload), video pacing and TCP acceleration.

All of these factors should influence test design and tooling approaches.

The right strategy

In my experience, an effective CDN based strategy begins with initial selection, informed by some of the considerations above. As we have seen, determining optimal return on investment repays a thorough approach, and some form of comparative Proof of Concept trial is likely to pay major dividends. Following the purchase decision and deployment, key on-going questions remain, if best value is to be maintained.

Looking at some of the test fundamentals:

Where are the priorities for usage optimisation?

  • Is there an end user monitoring strategy in place across all key (and potential) markets?
  • Do target KPIs (absolute or relative to local competitors) exist for all markets and usage categories?
  • Is a service level agreement – with defined interventions / remedies in place with the selected provider?
  • How do these stack up – both during ‘business as usual’ and across peak demand events?

These questions should be answered by an on-going monitoring programme, supported by a best practice based test matrix. Having defined which areas (functional, geographic and demographic) would potentially benefit from the use of CDN approaches, and assuming other interventions (e.g. appliance based, FEO) have been discounted:

Is the CDN accelerating all appropriate components?

  • Some ‘entry level’ providers are restricted to caching static content such as images. If other elements are required e.g. login and search, are these also included and effectively measured?

Test design. Where does caching occur? E.g. single POP /ISP domain, via multiple hosts, at cellular network mast?

  • Testing should reflect any potential constraints implied by the design of a particular CDN provider’s network.
  • The most effective way to do this is by pre-scheduled, synthetic testing from appropriate end users (PC and/or mobile) in the country concerned.
  • A parallel testing methodology should be employed, comparing performance directly to the origin servers with that delivered locally by the CDN provider.

This approach will ensure, not only that end user performance is satisfactory, but that the extent of benefit delivered (and purchased from) the CDN is maintained. In the interests of space, one example may suffice:

Comparative monitoring of Cached content to PC end users in a key regional market (known connectivity conditions). Origin vs. local cache

The illustration shows the overall benefit delivered (some 7 seconds on average). Highlighted are illustrates a convergence between the traces (i.e. transient failure of the CDN).

The illustration shows the overall benefit delivered (some 7 seconds on average). Highlighted are illustrates a convergence between the traces (i.e. transient failure of the CDN).

Importance of test location

The image below compares the number of CDN hosts used to deliver content (i.e. be monitored) over a 1 week period by a major CDN provider. Testing from the Tier 1 ISP cloud provisions [left hand image] cached content from two CDN host groups (16 individual hosts). The image on the right shows results from the same geography when tested from a wide range of PC end users (using a variety of tertiary ISPs). 96 individual hosts were employed.

The image compares the number of CDN hosts used to deliver content (i.e. be monitored) over a 1 week period by a major CDN provider. Testing from the Tier 1 ISP cloud provisions [left hand image] cached content from two CDN host groups (16 individual hosts). The image on the right shows results from the same geography when tested from a wide range of PC end users (using a variety of tertiary ISPs). 96 individual hosts were employed.

The relevance is that failure or poor performance by any of the 80 hosts not detected by ISP-level monitoring would not be detected, whilst being paid for, and impacting all end users of that particular host.

The image illustrates the range of performance at individual CDN host level during a 5 day test window.

The image illustrates the range of performance at individual CDN host level during a 5 day test window.

Hopefully this has shed some light on the issue as I see it, but it would be great to hear what your thoughts are.

Advertisements

2 thoughts on “Essential considerations of CDN performance monitoring

    • The key thing is to be able to generate (and analyse) distributed end user (as opposed to ISP cloud based (traffic).
      Several of the external monitoring Vendors (Neustar, NCC Group, etc.) offer a ‘Private Peer’ service enabling users to install their test agent wherever they wish.
      As you will appreciate, the difficulty with ‘doing it yourself using this approach is not so much the technology as access to anything from 10’s to hundreds (or thousands) of PC based end users in the global region which you wish to test.

      We use the Compuware Gomez SaaS tooling as it offers automatic test set up and analysis from some 250,000 end users in 168 countries around the globe.
      It also has a number of cool features such as the ability to select for a single retail ISP which can be useful if chasing down a specific issue.
      There may well be others, but I have found their so-called ‘Last Mile’ capability ticks most of the boxes for CDN performance monitoring. It’s not cheap, although that aspect depends on exactly how you use it.

      Hope this helps,
      Larry

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s