IT Focus Area: IT Operations Management
October 9, 2013
From Measurement to Improvement
A key factor in any IT improvement initiative is being able to effectively measure to improve. Many organizations start with the best of intentions by leveraging a common framework such as plan-do-check-act (PDCA) or the seven-step improvement process of the information technology infrastructure library (ITIL).
However, after a short period of time, they soon realize that the reporting and measurements being gathered didn’t help them achieve the overall improvements they initially anticipated.
While there are a number of reasons which prevent organizations from demonstrating improvements and planning future improvement efforts, there are five practical tips that can help a company move from measuring to prove to measuring to improve. These tips are based on years of experience implementing measurement strategies at large and small organizations.
1. You can't manage what you don't measure
Even the best process requires a measurement strategy to ensure the process outcome is meeting or exceeding its designed goal. In this case, the measurement strategy provides accurate and relevant measures that provide feedback and help gauge if desired expectations and outcomes are being met. Often, organizations may find themselves in a situation where analysis assumes priority over action. A common symptom usually manifests itself when an organization generates volumes of measures and reports. They spend too little or no time analyzing them to identify and plan improvement initiatives.
By pairing down a measurement strategy to just an initial set of metrics and reports, a company can see how well the process is performing. It is important to establish a specific set of goals for each process and then adjust metrics to meet these goals. Once these goals are consistently met, goals can be adjusted to a higher level of service.
2. Measure to improve not prove
By measuring to improve not prove helps a company align a measurement strategy to business or operational value. Why is your company implementing a measurement program in the first place? In most cases, companies determine how many IT changes were implemented, how many incidents occurred, or the availability of applications. The goal with measurement and reporting should be focused on presenting information to drive business value through improvement-related decisions. An organization should tie metrics and improvements to business value in one of the following three areas: effectiveness (e.g. increasing the change success rate); efficiency (e.g. making changes quickly and accurately or reducing the number of service disruptions) and governance (e.g. reducing the number of change audit issues).
When developing your measurement strategy, it is important to incorporate measurements from each of these three areas. Additionally, ensure that the measurement strategy is an integral part of an overall continual improvement program.
3. Triangulate your measurements
Triangulation is a powerful practice that facilitates validation of data through cross verification from three or more sources. A company should collect three related measures for each individual outcome not just a single data point. A wrong measurement conclusion or an undesired behavior can occur.
An example of triangulation within the context of measures is with the IT service desk and its first call resolution (FCR) rate. Many organizations cite this measure as an important gauge of how well the IT service desk is performing. In these cases, the FCR rate measures the service desk's efficiency as being able to resolve customer issues. However, upon further review, FCR on its own can be deceiving. While FCR may be a very high percentage and look good on the surface, a company doesn’t know how many times the same customer has had to call back to re-open the same issue nor understand customer satisfaction. So triangulating other measures such as the number of recurring incidents, the number of times tickets are closed or reopened, and the FCR should ensure the measurement provides a more accurate gauge of the service desk performance.
4. Collect data for a minimum of 90 days before trending
It is important to develop a measurement baseline or a point of reference to show measured changes over time. In some cases, current baseline data is not available and organizations have to create one. For example, when a new initiative or program is being rolled out and no current data exists. In this case, it is important to collect and measure data for a period of 90 days or more before interpreting any changes from your baseline data. After 90 days, trends should start to develop. However, some measures require a longer period to establish a baseline (e.g. calendar-driven events such as end of quarter and year-end sales and marketing activities).
5. No metrics are better than bad metrics
No metrics is better than having bad metrics. Why? There is a difference between good and bad metrics. An example of a bad metric: How many of a certain type of changes have occurred during a given period? This information does not align to business value or making improvement decisions. An example of a good metric: How many failed changes of a certain type caused major incidents? This metric provides an organization with a clear path toward a more stable environment (i.e. business value) and a springboard for implementing improvements. Overall, good metrics help organizations determine whether an organization is getting the expected outputs, if teams are getting better or worse, where areas of improvement exist and whether or not there is a problem. Bad metrics usually take more time and resources to collect and review than they add in value, don’t actually tell you anything, and don’t tell you what you think they’re telling you.
Move towards improving rather than proving
Going from measurement to improvement is not as cumbersome as it may initially appear. Remember: improve don’t prove. Ensure a measurement strategy is aligned with producing business value and making improvements. Focus on the metrics and measurements that provide the most value. Avoid bad metrics. When possible, triangulate and create a baseline. It is impossible to move forward with any improvement program, if an organization doesn’t know it starting point or its destination. Any IT improvement initiative is able to effectively measure to improve.