Response time, availability and consistency are the “holy trinity” of external performance monitoring. Effective operations teams are required to provide assurance of all three metrics (and many more besides, of course).
Clean room testing – the best case scenario
There is a superficially attractive logic that argues that the primary requisite for understanding and managing these metrics is test tooling with optimised “clean room” characteristics. These would include:
- Tier 1 ISP connections
- “Unlimited” bandwidth (or defined connectivity from such a connection)
- “Industrial strength” high capacity test nodes
- Highly instrumented / modified test agents
Certainly, the ability to run replicable comparative testing of this nature has its place. Indeed, it is far preferable to the approach of some “bargain basement” tools running tests across tertiary ISP connections from local datacentres with little or no understanding of the above characteristics.
It is not difficult to accept that “near edge” clean room testing has value – in fact I regard it as essential in many areas, including:
- Application performance baselining across the business demand cycle
- Competitive benchmarking
- KPI and third party SLA management
Put this together with the concept that there is no point in identifying issues (such as ISP Peerage) that are outside ones span of immediate control, and we are all done aren’t we? To quote Evelyn Waugh, “up to a point, Lord Copper.” This approach would be perfectly adequate if web delivery was made in such consistent conditions.
Performance in the real world
Unfortunately, as we all know, the opposite is the case – and the busier your site, the more this applies. The big grocers, for example, find themselves getting web requests from IE5 and other browsers that time has forgotten, not to mention every kind of interactive user device. Add mobile to the mix and the situation is incrementally worse – add hundreds of user devices and wireless connectivity into the mix for good measure.
The key to all this is taking a “knowledge is power” approach based on business, not just operational risk. So by all means run near edge testing, but understand what your customers are experiencing too. This means getting as close to them as you possibly can.
- Browser versions
- PC system characteristics
- Tertiary (retail) ISPs
- Real world connection speeds (refer to Ofcom – they will be a lot lower than you may think through reading the adverts)
- Wireless carrier / Wi-Fi provision
- Mobile device characteristics including processor, memory state, battery stats & signal strength
….and plenty more – basically, anything that can affect application performance (and therefore such “minor matters” as digital revenue and brand perception – probably will.
Realistic approach to testing
A testing approach (supported by appropriate tooling) that provides visibility to these areas (not necessarily all the time, but certainly frequently enough to spot major issues and define the “offsets” from your clean room tests) will pay dividends in areas including:
- Peerage issues (often “silent” but affecting significant proportions of your entire customer base) – “fixed wire” ISP and wireless carriers
- Client side problems (especially where RIA 2.0 technology – Flex, AJAX etc. and / or multimedia content is involved)
- Mobile device-centric issue – particularly with regards to native mobile applications
- Key client intranet/extranet performance
- Provision of detailed objective data to support demands for resolution to third parties
- And so on
Understanding will enable:
- Issue identification
- Prioritisation of intervention (on basis of revenue risk, etc.) and
- Validation of resolution
…and the “anti-promotion” phone call from the CEO who has just been cornered for half an hour in the golf club by someone with website “issues” will be a thing of the past!
Final thought – end user testing requires different approaches to test design in order to maximise business value.