Supply chain benchmarking has always been costly and cumbersome. But a new benchmarking resource allows companies across all industries to readily gauge how well they're doing on key supply chain metrics. A joint effort of APQC and CSCMP, the benchmarking tool is a sort of "metrics users club." Participants input their data and in return receive a comparative report with qualitative and quantitative information that identifies improvement opportunities.

Most standardized units of measure were agreed upon centuries ago. For instance, businesses have long been speaking a common financial language. But factories, distribution centers (DCs), and logistics facilities have few shared terms for crucial operating measures. Talk about speaking in tongues! The following are three quite different ways to interpret the simple concept of order fill rate.

Company A receives an order on Monday but postpones sending it to the DC because a SKU that is part of the order is out-of-stock. When the SKU is restocked on Thursday, the order is forwarded for fulfillment on Friday. By now, the customer has waited four extra days. Nonetheless, Company A, which measures fill rates by how quickly the order is filled once it hits the DC (not from the time the order was received), proudly reports a 100-percent fill rate.

Similarly, Company B receives an order on Monday. Although its normal cycle time is two days from order to shipment, the company quotes a five-day cycle time because it is experiencing unusually high demand. When the order ships on Friday, Company B reports a 100-percent fill rate because it shipped when it said it would (but not when the customer needed it).

At Company C, the language is different again. This company runs a 24-hour operation; all orders received before noon are targeted to ship the same day. Although the company works around the clock, the cutoff for trucks leaving the DC is 9 p.m. During the peak demand period, the company gets slammed with same-day ship requests. Unfortunately, the pick/pack and ship function falls behind schedule, and the order misses the truck. The DC keeps working on the order and finally sends it to the loading dock at 10 p.m. where it sits until the next evening. Although the order languishes for nearly 24 hours until the next truck leaves, Company C's computer system shows the order as shipped because it was completed before midnight. It also reports a 100-percent fill rate.

So Company A ensures high order-fill rates by defining it in terms of DC performance, while supply chain managers at Company B believe that the company is performing well because they have reset customers' expectations. For their part, Company C's managers are happy because they have arranged the time boundaries so they score complete order-fulfillment success on paper. In each case, the company appears to "win"—but in every case the customer fails to get what they want.

Regrettably, these examples are absolutely typical of supply chain activities worldwide.

Many efforts have been made to create practical systems for measuring and tracking the performance of familiar business operations. Yet business executives still lack a standardized performance "language" that makes it easy to compare business processes and performance measurements. And they lack consistent, reliable, and affordable sources for benchmarking data that allows them to compare their companies' performance with that of others in the same industries, size groups, or geographies.

To give business leaders the necessary measurement tools, two leading organizations recently teamed up on a major research effort. The joint research was conducted by the Council of Supply Chain Management Professionals (CSCMP) and APQC, an international nonprofit research organization dedicated to process and performance improvement. This research has begun to identify new and widely applicable techniques that provide ongoing benchmarking opportunities for participants. These new techniques are attracting growing interest among supply chain executives through a kind of "users club" that APQC has set up.

There is little debate over the value of standards. In fields as diverse as stock-option accounting and e-commerce transactions, standards that win widespread compliance are successful in streamlining communications and averting conflict. They reduce the friction of misunderstandings and misinterpretations. Financial standards, for example, have long provided the necessary beacon to executives for measuring performance; today's focus on earnings before interest, tax, depreciation, and amortization (EBITDA) numbers is a good case in point. In the information technology field, the fast-growing Linux computer operating system shows how a de facto standard can emerge.

However, there can be much disagreement over what makes the best standard. Discussions are typically reopened every time there is a new development in, say, technology; a change in regulations or tariffs; or an evolution in business processes. (A good example: The shift to intercompany collaboration is requiring business to develop new kinds of performance measures.)

One of the examples of this disagreement is in the area of quality. In the 1980s and 1990s, leading academics and consultants built greater awareness of the need for managers to extend their focus beyond traditional accounting measures. The Total Quality Management (TQM) movement laid down new ways of measuring product standards and performance as well as of monitoring production processes. But even today, there is still a wide array of quality standards: from the Lean/Six Sigma methodologies spearheaded by organizations such as Motorola to the detailed recording of process procedures demanded by ISO 9000.

On the business-performance side, management tools such as the Balanced Scorecard have helped to provide a standardized prescription on what companies should measure to "balance" the financial perspective. In terms of the supply chain, practitioners have applied the Supply-Chain Operations Reference (SCOR) model as a cross-industry standard to address, improve, and communicate supply chain management practices within and between all interested parties. And recently CSCMP has published a set of Supply Chain Process Performance Standards in six handbooks,¹ which outline qualitative best practices and suggested process standards across more than 200 supply chain-related processes. While these process standards are a great start, companies must benchmark both quantitative and qualitative benchmarks to obtain a complete picture of their performance.

These well-thought-out frameworks and models have helped thousands of organizations to expand their use of performance data beyond financial elements. But there has never been a good source for measuring quantitative operational performance, according to the research firm Gartner.² While many quantitative benchmarking studies do exist, there have been few opportunities to benchmark continually against up-to-date information.

The problem is twofold. The first concern centers on basic communication. As the examples above show, organizations have neither a common language nor a standardized set of definitions and calculations for benchmarking. Organizations often find themselves comparing apples to oranges in their benchmarking efforts. This confusion is especially evident when companies try to collaborate more closely with their supply chain partners. As enterprises focus on core competencies and outsource other activities, the integration of business processes between suppliers and customers is becoming more critical.³ When each organization uses its own terminology to describe component processes and performance, it creates confusion and inefficiency.⁴

The second major obstacle has been the lack of a widely accepted source for benchmarking data. At the start of their performance-data research, CSCMP and APQC found nearly 50 sources that provide various forms of such data. But most have significant drawbacks—or come with costly strings attached. There are plenty of one-time studies, but they do not work as ongoing sources for measuring performance. There are aggregated data, but they are not granular enough to allow organizations to look at the differences among industries or regions. And there are trends-focused benchmarking data (such as the growth of the third-party logistics market), which do not include best or average performance against a certain metric. Of the few benchmarking sources that did offer promising content, almost all were managed by for-profit organizations that charge subscriber-based fees—more than $15,000 in some cases.

Since May 2004, APQC has spearheaded a research program, called the Open Standards Benchmarking Collaborative (OSBC), that includes a confidential online database of standardized process measures and benchmarks. The database provides measures for more than 100 processes and functions to help organizations compare performance across all business, health care, educational, and governmental sectors.⁵ Organizations of all industries and sizes may access and use these standards, which are identified in the APQC Process Classification Framework (PCF). (The PCF can be downloaded at no charge at www.apqc.org.) This framework serves as a high-level, generic enterprise model that allows organizations to see their activities from a cross-industry process viewpoint.

The OSBC research looks at areas such as finance and accounting, human resources, customer service, information technology, and supply chain management. APQC and CSCMP define supply chain management as the planning and management of all activities involved in sourcing and procurement, conversion, and logistics management. Importantly, it also includes coordination and collaboration with channel partners such as suppliers, intermediaries, third-party service providers, and customers. In essence, supply chain management integrates supply and demand management within and across companies.⁶

In late 2004, a working group comprised of APQC and CSCMP professionals assembled to determine which supply chain processes and measures should be used in the OSBC research. The joint working team culled through more than 300 measures, narrowing them down to 150 defined metrics to be used in the supply chain area of the OSBC database. These 150 metrics spread across the following five critical process areas and functions: new-product development, procurement, customer order management, manufacturing, and logistics. During the working group's sessions, each process category and its associated measures were scrutinized to ensure that they would be practical to apply. (See the sidebar "Five Critical Process Areas" on page 26 for a snapshot of each of the five categories.) Both APQC and CSCMP also recognize that universal practices are evolving in each of those sections of the supply chain and released updated supply chain surveys in March 2005.

The OSBC research can be thought of as a kind of "metrics users club"—or more practically, as a "credit union" where all participants have a shared interest and where they both give and take. Business leaders can share their performance data through the online OSBC database. In return, participating organizations receive a comparative report with quantitative and qualitative information. This report identifies improvement opportunities by comparing the participating organization to top performers as well as others in, for example, the same industry or the same region. All data are blinded, normalized, and reported in aggregate.

The OSBC research pushes benchmarking further than previous efforts. The "give and take" operating model provides immediate incentives for its use. It is easy to participate in and easy to access; the database is continually strengthened with volunteered and screened data, and it is continually updated.

So far more than 150 organizations have submitted their performance data to the database and received benchmarking reports. (As soon as APQC has 10 data points per industry, it can publish the results.) Participation continues to grow; APQC expects that by the end of 2005 more than 1,000 organizations will be involved, representing many industries and providing data across all five of the critical supply chain process areas.

Defense and aerospace supplier, Raytheon Co., for example, is an enthusiastic participant with a deep understanding of the roles that external benchmarking and knowledge-sharing play in creating and maintaining a competitive advantage. Raytheon is already applying OSBC research findings to its critical business decision making. Don Ronchi, vice president of Raytheon Six Sigma and supply chain and chief learning officer, recently described the company's participation: "The work of the OSBC research will make it much easier for companies like ours to draw meaningful comparisons with other world-class companies."

Participation in the APQC/CSCMP initiative is simple. It is not necessary to be a member of either organization; managers simply visit www.apqc.org/OSBCdatabase or link to this site via www.cscmp.org. They they enter performance data through surveys for one or more of the five supply chain areas categorized above. APQC then validates and analyzes the responses and, without identifying the company, adds its data to the OSBC database. APQC is well aware that quantitative information and definitions used to calculate data points vary from one organization to the next. So as part of its data-validation process, APQC identifies the various factors that are used to arrive at a specific data point. For example, if there are incongruities in the reported data, APQC calls the participating organization to determine if factors such as an internal reorganization, new accounting recognition in a specific quarter, a total company merger, or something similar could have caused the anomaly.

The company participating in the OSBC research then receives a report from APQC assessing its core operational functions and providing the opportunity to:

• Measure its operational performance relative to external peer groups.
• Identify performance gaps and evaluate improvement opportunities.
• Establish performance targets using external benchmarks.
• Identify key business-practice drivers that influence performance.
• Assess operational progress over time relative to internal and external benchmarks.

Participants can benchmark themselves using the criteria of their choice. For example, a procurement manager can gauge the company's cycle time to replace purchase orders against the replacement cycle time of others in the same industry or the same region.

Top performers are defined as organizations with performance levels within the top 20 percent when compared to all other participants. The benchmarking information demonstrates how performance or shareholder value can be affected by improving the performance relative to the top performer level, keeping all other factors the same.

It is instructive to see how the reports present the OSBC data. Let us look at two key logistics areas: order-fill rate and percentage of expedited orders.

Order-fill rate is a critical metric to most supply chain management professionals partly because timely delivery is a widely accepted indicator of customer satisfaction. The OSBC research team calculates order-fill rate as the number of completely filled annual sales orders divided by the total number of annual sales orders. Exhibit 1 on page 23 shows a representative page from an OSBC report, which records the particular organization's order-fill rate at 87 percent.

While an 87-percent fill rate may sound like a relatively high number, it is immediately obvious that the company is weak in relation to its peers—and a laggard compared to the top performers. In fact, the company's order-fill rates fell into the 17th percentile of the total benchmarked population. By presenting the benchmarking variance so starkly, the OSBC report can provide a valuable "wake-up call" to managers whose companies are lagging so far behind. The message is doubly forceful when, as in this example, there is only a small gap between the top performers and the median organizations.

A second benefit of the OSBC benchmarking report is that a company can quickly compare itself with others in specific industries, geographies, and revenue-size ranges. This comparison is important because the median or top performance for one industry may differ greatly from that of the total population. For example, in the pharmaceutical industry where fill rates are especially time-sensitive, the median score may be much higher than in the chemical industry where most product is shipped in bulk. Exhibit 1 on page 23 also shows the significant variance between median fill rates in the automotive industry compared to the all-industry figure.

Now let's turn to the example of the percentage of expedited orders. By looking at this measure, a supply chain manager can drill down to determine if his company is achieving the order-fill rate at the expense of expediting orders. For the high-tech company shown in Exhibit 2 on page 24, the report displays a good news/bad news situation. While it can take comfort in knowing that it is expediting orders less frequently than the average performers, it still has a long way to go before it will reach the level achieved by the top companies.

This report brings up a very good question: Why is there such a big difference between the top performers and median companies, especially among larger organizations? Answers begin to appear when the data is viewed from yet another angle—from the viewpoint of companies' operating practices. Exhibit 3 shows that 43.6 percent of all organizations that participated in the OSBC research have a warehouse management system (WMS). Further data slices indicate that of those organizations with WMS systems, 43.8 percent were able to process an order from multiple distribution locations as a single customer purchase order. Therefore it is reasonable to conclude that those companies' WMS capabilities help them achieve higher order completion rates. The data also suggest that larger organizations outperform others, which may be indicative that the large organizations are able to afford an advanced WMS.

While almost all organizations are interested in knowing where they stand compared to their peers, those that actually work to close performance gaps will get the biggest payoff.

IBM, for one, has taken a serious approach to benchmarking. "Performance benchmarking at the process level has allowed us to establish a solid baseline on how our business performance compares to that of similar organizations," said Oscar Evans, IBM's manager of business process outsourcing solutions. "The benchmarks provided a useful, holistic assessment of our operation in terms of cost, productivity, efficiency, and cycle-time performance."

The OSBC initiative is geared to helping companies develop practical supply chain improvement roadmaps. The report features a management dashboard that contains the top 20 key performance indicators recommended by APQC and CSCMP. (See Exhibit 4.) Organizations can use this information to determine where to focus improvement efforts by mapping current performance (represented by the small circle) against their desired performance (shown as the arrow).

Most supply chain managers have been eager to benchmark their companies' performance against that of their peers. Some have relied on the standardized guidelines presented in frameworks such as the SCOR model; others are turning to the Supply Chain Process Performance Standards developed by CSCMP. And still others have applied the broader management measures from tools such as the Balanced Scorecard. But there have been no continually updated, easily accessible, and affordable means of benchmarking broadly across industries, regions, and operating practices.

With the development and growth of the APQC and CSCMP initiative, managers now have a source of reliable, affordable, and timely data. A large part of the value of the initiative is that it is open-ended: Continued participation by growing numbers of companies strengthens the database and provides better and better opportunities to chart progress over time—and to track rates of improvement against those of other organizations. Current CSCMP president Mark Richards believes that ongoing participation in the OSBC research will allow organizations to monitor their effectiveness annually or even quarterly. "Supply chain management professionals can watch industry-specific and universal supply chain trends evolve and can better focus their continuous improvement efforts," Richards said.

The key words are "improvement efforts." Both APQC and CSCMP point out that while the OSBC research will provide valuable information, understanding the data is simply the starting point for an organization's benchmarking efforts. The crucial next steps are: to analyze that data to determine which process improvements are best-suited for implementation—and then to build implementation plans accordingly.