Supply Chain Benchmarking: Get the Gain Without the Pain - The Challenges of Benchmarking

By Joe Francis -- Supply Chain Management Review, 4/1/2008

Previous page: Some Basics of Benchmarking

Through experiences with these and subsequent programs, I've identified some recurring challenges with regard to instituting an effective benchmarking initiative. These include:

• Sponsorship—every benchmarking initiative needs a sponsor, the higher in the organization the better.
• Scope—selecting the supply chains to be benchmarked is critical; it's not a simple process.
• Selection of processes and metrics—focusing on strategic elements helps keep the program targeted and useful (deep metrics in a few areas rather than many metrics across numerous areas).
• Standards—standard definitions of supply chain processes (e.g., what activities are in manufacturing or procurement) enable “like-for-like” benchmarks across divisions or companies. Conversely, lack of standards make meaningful comparisons difficult if not impossible.
• Sources—identifying sources of data for metrics and having clear pointers to which processes generate transactional data necessary for calculations.
• Cost—benchmarking can be expensive, especially when outside consultants are used. It's not uncommon for the cost of a single benchmark to range between $300,000 and $500,000.
• Time—the benchmarking process can take from three to five months; set expectations accordingly.
• Deriving meaning—the benchmarking initiative must be structured so that the results produced are meaningful.

So how do supply chain managers address these challenges and conduct benchmarking that is truly effective? In the last five years, Supply-Chain Council members have asked us to build a metrics repository based on SCOR for benchmarking purposes—that is, a scorecard that would let them compare their performance against industry peers and companies outside their industry. Many members in particular wanted to be able to periodically check on a couple of key metrics to see if their performance was in order.

The Supply-Chain Council had to carefully consider how it would respond to this member need, recognizing that compiling the data necessary for even a casual check-up could be a daunting task. In 2005, we began negotiation with APQC to build a SCOR benchmarking resource for our members. In 2006, IBM sponsored the development of the SCC/APQC benchmarking system, which has the trademarked name of SCORmark. The approach we took recognized the value of process reference models such as SCOR, coupled with the value of having access to benchmarking data.

We began the SCORmark development process by asking what were the realistic expectations of a quantitative benchmarking exercise and how could we reach those expectations. We quickly agreed that more than a database of metrics information was required. Rather, we needed a system for performing benchmarking that, to the best of our ability, would resolve the key issues in the benchmarking process. There were some challenges we could not address directly—C-level sponsorship of a benchmarking program, for instance. But we decided we could address most of the other issues.

First, on the challenge of scope and focus. SCOR provides an elegant and effective three-step process for (1) identifying all the supply chains in a given business, (2) prioritizing them according to business impact, and (3) linking them to business strategy. The “supply chain” identification matrix greatly simplifies all of the discussions about “what” supply chains are present. Generated from sales and marketing segment data, and from product and supplier segment data, the matrix quickly gives a commonsense stratification of supply chains for further examination. Exhibit 1 shows such an identification matrix for the fictitious ComfyCo Air Conditioning Company. In this case, ComfyCo identified three supply chains: Big Air, Small Air, and Commercial.

The supply chain “priority matrix” simply ranks the identified supply chains according to company performance criteria (see Exhibit 2).

Then the supply chain “strategy matrix” links the prioritized supply chains to top-level company strategy (Exhibit 3), according to whether each should be superior (S), give you an advantage (A), or be at parity (P) competitively. This is a simple 1-2-3 approach to identifying what to benchmark, instead of spending weeks of agonizing discussion.

Continuing on the issue of scope, SCORmark adopted the NAICS (North American Industry Classification System) coding system. (NAICS replaces the old SIC classifications.) We added in SCOR's standard supply chain types—Make-to-Order, Make-to-Stock, or Engineer-to-Order. Thus, for any company, even complex conglomerates, once managers identify a piece of their supply chain that they want to benchmark, they can unambiguously compare it to similar supply chain types in a given region and industry.

NAICS has more than 1,175 distinct industry codes, which provides a rich set of types to identify almost any possible supply chain industry segment. For instance, a supply chain may be defined as Chemical Industry, European, Build-to-Order. It could be defined as High-Tech Manufacturing, Asia-Pacific, Engineer-to-Order. If you're a toy manufacturer, soybean grower, or IT service provider, there's a category for you to use. There are also segments for different sizes of supply chains based on revenue; so you can compare small-to-small and small-to-extra large if that's your wish. The categories, geographies, and supply chain types were “menuized” to simplify the benchmarker's task of identifying both internal and external supply chains for comparison.

The next challenge addressed was selection of metrics. This has long been a subject of intense discussion at Supply Chain Council training sessions on the SCOR model. SCOR has several hundred supply chain metrics organized by purpose (level) and categories. A purpose could be creating and measuring strategy (Level 1), diagnosing process defects (Level 2), or measuring workflow performance (Level 3). All metrics fall into one of five categories based on the metric's strategic impact. These categories are reliability, responsiveness, agility, cost, and assets. Order Cycle Time, for example, is a responsiveness category metric. Cash Cycle Time is an asset-type metric, and so on.

To understand the breakout, consider Cash Cycle Time (Level 1-strategic). This metric is composed of Days Sales Outstanding, Days of Inventory, and Days Payables Outstanding (Level 2-strategy diagnostic). Total Supply Chain Management Cost (Level 1-strategic) is composed of constituent non-COGS process costs—Plan, Source, Deliver, Return (Level 2 -strategy diagnostic), which in turn are composed of costs of each component process (level 3-process diagnostic).

To do a supply chain benchmark with SCORmark, the user needs to select at least one Level-1 strategic metric for each of the five major categories. Further, he or she must prioritize the company's strategy in each of those five categories according to whether the supply chain must achieve superiority, advantage, or parity. One (and only one) superior rating is allowed for analyzing a benchmark, two advantage, and two parity ratings. For the superior category, we would expect managers to select component metrics at Level 2 – strategy diagnostics and some at Level 3 – process diagnostics. For the advantage category, we would expect them to add some component metrics at level 2 – strategy diagnostics. This would build out a benchmark or “SCORcard” of 24 metrics: 5 (Level 1 metrics) + 3 (Level 2 superior metrics) + 10 (Level 3 superior metrics) + 6 (Level 2 advantage metrics). That number is not too big, not too small. Most important, it is sharply focused on company strategy. (See Exhibit 4 for sample metrics selection)

This process is really not as complicated as it may sound. When supply chain managers have a menu of metrics organized by category in front of them, metrics selection becomes almost cut and dried. Selecting Level 1, and then inheriting Level 2 (and Level 3) metrics is a clear and logical process. It is a deceptively simple system because SCOR already has cause-effect data on all strategic, strategy-diagnostic, and process-diagnostic metrics. Rolf Poluha wrote an excellent book¹ that actually articulates the statistical significance (that is, cause-effect relationships and correlations between metrics) for all SCOR metrics. The book provides a fantastic amount of detail for sticklers.

Compare this straightforward process to the seemingly endless benchmarking debates around:

• Metric definitions (what should “complete order” really mean?).
• Which metrics are valuable to our company (a guessing game)?
• How should we decompose the metrics once we've defined them.

I've been through these debates, and they are neither pretty nor short. At Compaq in the mid-1990s, when we were standardizing the definition of order cycle time, on-time delivery, inventory days and related metrics, it took almost a year to achieve a global consensus on how to measure and manage the data.

Responding to the standards challenge, SCOR developers have created or adopted the most widely accepted definitions of supply chain metrics in use among around 2,500 companies worldwide over the last 11 years. This greatly facilitates data gathering. For instance, order cycle time is defined as beginning with receipt of a customer order and ending with the customer acceptance of the service or material. There is no debate about interpretation. Managers do not need to undertake (or have consultants undertake) custom programs to create like-for-like comparisons. The SCOR metrics already do that. More significantly, managers do not have to embark on customized programs to gather and reclassify external data—that is, figure out how to compare their operations to those of other companies.

Another important advantage is that SCOR provides guidance for data gathering. All the SCOR strategic, strategy-diagnostic, and process-diagnostic metrics provide a specific list of process sources for the raw data necessary for calculations. With this guidance, companies can readily identify process owners who govern access to IT systems that may hold transactional data. This provides rough back-of-the-envelope planning for data gathering and quality control of the measurements, thereby speeding up the onerous data-gathering phase of the benchmark.

On the challenge of reducing the cost of benchmarking, council members and other interested parties can easily learn how to use the SCOR methodology and SCORmark workflow. (For more information on this, see accompanying sidebar.) A consultant may be valuable in providing the manpower to gather the data and manage a big benchmarking program. But managers do not need any expertise in the fundamentals of benchmarking—selecting and defining the metrics, the methodology, and how to analyze the data—outside of the SCOR framework itself. Once a company has standardized on SCOR and trained managers on how to use the model, they can easily execute the benchmarking and interpret the results. No key information “walks out the door” at the program's conclusion.

The SCORmark approach saves time and money. Users avoid the cost of customized benchmarking because the system is based on open standards shared among the SCOR community via the SCORmark system. In effect, access to the standards is part of the cost of membership to the Supply-Chain Council. In addition, SCORmark cuts the time required to conduct a benchmark (assuming that your company manages supply chains with standard metrics) to a fraction of the usual three to four months. In fact, we've seen high-quality benchmarks completed in as short a time as one day, though the norm is typically two to three weeks. APQC will need time to perform statistical validation (“quality checks”) of the benchmark data, which can take up to one week. Once your company is known to provide quality data, the statistical validation can be done within days.

Consider the implications of this capability. Instead of spending one or two quarters benchmarking and goal setting with lagging data, a company can benchmark monthly, and identify leading trends and set forward-looking goals on a continuous basis. There are no substantial additional costs in using monthly KPI data (SCOR data), and benchmarking against that data over and over.

Finally, with respect to deriving meaning from the benchmark results, we start with the fact that SCORmark is designed around the SCOR methodology. So once managers select their appropriate metrics and identify their company strategy, the benchmark not only places their company relative (by metric) to their industry or selected demographic, but also identifies the targeted improvements needed (see Exhibit 5). The analysis of this benchmark performance gap ties directly into the subsequent phases of SCOR—material flow, work and, information flow—for identifying the root causes of performance problems. It also tells you what projects you're going to need to execute to address the problem areas.

Continue to Early Experiences with the Benchmark

In this article:

• Introduction
• Some Basics of Benchmarking
• The Challenges of Benchmarking
• Early Experiences with the Benchmark

Resource Center