Follow us

How Do You Measure IT Services?

By | May 2, 2017 in ITSM

How do you measure IT services?

All the IT service providers I’ve worked with assure me that they measure the services they provide. They use metrics and KPIs to do this, and have service-level agreements (SLAs)  with their customers, which they use to document what’s been agreed.

Unfortunately, most of the metrics and KPIs that I’ve seen only measure and report the things that service providers can control, and not the things their customers actually care about. This tends to result in reports that show service providers meeting most, if not all, of their targets, even when customers are distinctly unhappy about the service. This happens so frequently that the phenomenon even has a name: the “watermelon SLA”. When you look at a watermelon from the outside, all you see is green.  Delve a bit deeper and you discover that most of the fruit is red.  In the same way, we report that everything is meeting its targets (it’s all green), but if you delve into the customer’s experience you find that it’s not very good (it’s mostly red).

So, what can service providers do that will help them to delve that bit deeper and focus on what matters?

What’s Important to Customers?

When service providers think about any service, they need to pay attention to what’s important: the value of the service, the intended outcomes, the costs, and the risks – together referred to as VOCR. Here is a brief summary of what we mean by these terms.

  • Value is a measure of the benefits that the service creates for the customer. This could be in terms of money, but it might be something like lives saved, or some other indication of the customer’s mission.
  • Outcomes are the things that the customer achieves as a result of receiving the service. For example the service may help the customer to manufacturer products, or communicate with their partners, or collect payments on a web site.
  • Costs are the money that the customer has to spend to achieve the outcomes they want.
  • Risks are possible events that could affect the ability of the customer to achieve the desired outcomes.

In my view, if every service provider produced reports showing the VOCR of their services, then customers would have no difficulty understanding what these reports mean, and judging whether the service was good value (or not).

Of course, it’s not that easy. The trouble is that value, outcomes, costs, and risks are only partially under the control of an individual service provider. For example, consider an IT company that provides a manufacturing support service to a car manufacturer. The outcome that the customer wants might be “a car comes off the production line every 40 seconds” (my thanks to James Finister for this example), but this outcome is not completely controlled by the IT service. Even if the IT service works perfectly there could be other reasons why a car doesn’t come off the production line. So, what should you measure and report in this case?

What Should You Report to Your Customers?

Under these circumstances most IT service providers would probably report things like:

  • Availability of the IT service
  • Number and severity of incidents
  • Average time to resolve incidents, subdivided by category

The information provided might be accurate, but it is very IT specific. It represents an IT service provider’s view of the world. But of course, this isn’t what the customer cares about. The metrics that matter to the customer are how many cars come off the production line (outcome), how much extra cost do they have due to IT failures (cost), how much lost production might they have next week (risk), and derived from these, how much profit can they make selling cars (value).

So, is it possible to create IT metrics and reports that actually show all of these things? Yes, it is. An ideal SLA could state the following:

“The desired outcome from this service is to support the production of one car every 40 seconds.”

The monthly report could then show whether this was achieved. Similarly there could be report sections on cost and risk, showing how well these have been managed for the customer.

Does this mean we stop measuring IT service availability, number and severity of incidents, incident resolution times? Obviously not. Providers can’t do their jobs well without this information. What I am saying is that these are not the key things to report to the customer. The main findings of any report to customers should be about value, outcomes, cost and risk; IT-centric issues should be reported within this context, as contributing factors, and at whatever level of detail suits the specific customer.

Here are two more examples of IT organizations that have great business-focused reports:

  • One client that I worked with had a key business metric stating that they should never lose any customer data (the data represented a lot of money). It didn’t matter if the data was lost because a paper document got lost before it was scanned, or because of an IT error. The thing that mattered is that every piece of customer data must be correct and accurate at all times. The reports they produced started by summarizing the status of customer data, THEN they went on to account for any risks or issues that had occurred that month. Some of these were IT related, others were down to the various business units, but the report included everything.
  • Another client was responsible for shipping parcels to end customers. Their key metrics were the percentage of parcels that arrived on time, and the percentage of parcels that were lost. As in the other examples, these things might be caused by IT failures, or they might be due to many other issues, but the IT reports focused on the business metrics, and then on how IT had contributed to these.

What about your SLAs and customer reports? Are they about IT or are they about value, outcomes, cost, and risk? Maybe it’s time to think about what your customers really care about, and create SLAs and reports that match their view of the world.

Learn about SysAid for SLAs


Stuart Rance

About Stuart Rance

Stuart is an ITSM and security consultant, working with clients all round the world. He is one of the authors of ITIL 4, as well as an author of ITIL Practitioner, ITIL Service Transition, and Resilia: Cyber Resilience Best Practice. He is also a trainer, teaching standard and custom courses in ITSM and information security management, and an examiner helping to create ITIL and other exams. Now that his children have all left home, he has plenty of time on his hands for contributing to our blog - lucky us!

11 thoughts on “How Do You Measure IT Services?”

  1. Avatar Yves Van Liefferinge


    In my view, this depends a lot on the maturity level of both supplier and customer. Too many parties, both customers and suppliers, fail to think beyond the traditional technical bits and pieces delivered as opposed to end-to-end services. To get there, it may take the SLA a few iterations of improvement to become a quality dashboard.

    Suppose you’re stuck with this basic SLA of response and resolution times and infrastructure availability, which would be you 5 key evaluation questions to take the next step in a more mature SLA, or at least to end up with an SLA that shows a better view on the quality of the services delivered?



  2. Stuart Rance Stuart Rance

    That really is a great question Yves, and gets to the heart of continual improvement. I guess the high level answer is Understand where you are now, Understand where you want to be, Then think hard about the next step that is a good cultural and technical fit for your organization and takes you nearer to the desired state. Continue iterating till you get there.

    What that first step is will be very different for every organization. One thing that I have seen work is redesigning monthly reports to subsume technical KPIs under business goals, so that you can start to change the level of the conversation.


  3. Avatar Yves Van Liefferinge


    Thanks for your advice!

    I’m currently leading an SLA improvement initiative for our outsourcing contract. My plan is to create a questionnaire / evaluation sheet allowing the managers of the different areas to do an evaluation for themselves (i.e. making them think about as-is and to-be) and come up with new, alternative and/or improved measurements.



  4. Avatar Mark Watson

    KPIs have been in the market for as long as we can remember. It is the default metric to gather qualitative and quantitative data from businesses. Even in MDM implementation, companies have seen poor implementation but yet pay the full price!


    1. Stuart Rance Stuart Rance

      I don’t fully understand the point you are making here Mark. KPIs have been in use for a long time, and the quality is very variable. Some organizations use them well, but others are far too internally focussed.


  5. Avatar Braden Martin

    I must say that’s a great article from a different perspective!
    However, how will a customer who has no knowledge about the SLAs know which MSP is right for him/her while choosing one?


    1. Stuart Rance Stuart Rance


      Choosing a managed service provider (assuming that is what you mean by MSP) is much more than just evaluating metrics and KPIs. The most important thing to think about is cultural fit. Is this an organization that you want to work with?


Leave a Reply

Your email address will not be published.