Outcomes, Not Outputs: Why We Need Portfolio-Level Performance Metrics

by William Walton

Portfolios are widely accepted and used by IT organizations to help manage sets of related IT assets, activities, and resources. These include projects, applications, infrastructure components, and IT services. The goals and intentions of using portfolios as management tools are all related to improving the business value delivered or derived from IT assets and capabilities by:

• Identifying opportunities to sustain or expand the IT assets and capabilities that are generating or will generate value

• Identifying redundant, obsolete, or inefficient IT assets and capabilities that indicate which associated IT resources can be consolidated, reduced, or reallocated

• Implementing controls to reduce and mitigate the relative risks (business and technological) inherent to IT investments and assets and their ability to deliver their expected value
  
  

That is, IT portfolio management is really about supporting the decisions regarding the ongoing allocation and reallocation of IT resources with the goal of optimizing the value delivered to the business. In today's turbulent economic environment, it is critically important that an IT organization have a mature IT portfolio management capability that extends across the full range of its IT projects, applications, and services. Unfortunately, IT portfolio management in most IT organizations is handicapped by the absence of consistent and relevant portfolio-level performance metrics. Consequently, IT portfolio management tends to emphasize outputs (e.g., completed projects) over outcomes (maximizing business value).

Parallels are often drawn between IT portfolios and their counterparts in the financial investment world. With an investment portfolio, success has been consistently based upon the financial return (income + appreciation) generated by the assets held within that portfolio. More recently, portfolio performance comparison tools have been developed that consider a portfolio's return within the measured context of the investment risks taken to achieve that return (i.e., portfolio return normalized to portfolio risk). Having a consistent set of portfolio performance measures allows portfolio managers to monitor portfolio performance and track it over time, as well as to assess their own performance. Also important, the presence of portfolio-level performance metrics allows portfolio management to develop and mature as a discipline.

When we look at IT portfolio management vis-á-vis investment portfolio management, the following questions come to mind:

• How do IT portfolio managers measure the performance of their IT portfolios and, by extension, their own performance over time as portfolio managers?

• What metrics should be used to measure the performance of IT portfolios -- recognizing that project portfolios are likely to have different metrics than application or service portfolios?
  
  

Without these IT portfolio metrics (i.e., without consistent and relevant feedback), how can portfolio management as a process be meaningfully assessed and, more important, improved?

In this article, I will address these questions and examine the current state of IT portfolio performance management based on existing portfolio management literature as well as interactions with IT portfolio practitioners. In addition, I will propose specific ways in which organizations can use IT portfolio management and the related portfolio metrics to better align IT resources with the strategic and operational priorities of the business.

THE ORIGINS OF IT PORTFOLIOS

For most organizations, the first IT portfolios were project portfolios, and they can be associated with the emergence of the project management office (PMO). Project portfolios are useful tools for monitoring the individual and collective status of projects and for reallocating project staff and resources based on changes in status and organizational priorities.

A second development that contributed to the use of IT portfolios was the evolution of the role of the IT steering committee from project approval to project prioritization. Early steering committees typically reviewed project proposals and provided a thumbs-up or thumbs-down decision that was sometimes based on meeting some minimum level of projected ROI -- and other times was based on the clout and rhetorical abilities of the project sponsor. A thumbs-up decision meant that the approved project was placed in a queue with other projects also awaiting project resources. Managing the queue (or backlog) became a common issue, and the responsibility for prioritizing projects was pushed back up to the IT steering committee. These prioritization decisions required the steering committee to look at projects not individually but collectively as a portfolio.

Finally, a third trend that contributed to the use of IT portfolios was the growing use of such IT management models as ITIL and COBIT, which advocate portfolios (and related service catalogs) as key components of governance and service management best practices.

Regardless of their roots within any particular organization, the goal for most PMOs and project portfolios was to complete more projects on time and on budget with fewer resources. Consequently, IT portfolio performance metrics tend to be focused on outputs (completed projects) rather than outcomes (business value). This is somewhat like having a stock portfolio whose performance is measured based on the percentage of companies whose recent earnings met analyst forecasts -- perhaps a useful indirect measure, but not a primary measure of success.

ANSWERING THE RIGHT QUESTIONS

So what's the problem? The problem is that the primary goal of IT portfolio management should not be to complete more projects on time and on budget, but rather to deliver a higher return in terms of business value for the total dollars invested in a portfolio. Look at it this way. You are the CFO for Acme Entertainment, an organization with two separate business units, each with its own IT steering committees, PMOs, and IT portfolios, and you have provided each business unit with US $1 million for IT-related projects. Twenty-four months from now, you will want to compare how well each business unit did with that $1 million in terms of contribution to their respective businesses. Upon what kind of data or metrics will you want to base that comparison?

My research strongly suggests that the comparison should be based on the answers to the following questions:

• Is the "right" set of projects being identified and approved?

• Are the "right" projects receiving the "right" priority?

• Are the prioritized projects being delivered efficiently and effectively? (That question doesn't go away!)

• Are the completed projects delivering the expected value?

• What is the total value delivered for the initial $1 million invested?
  
  

The following sections will explore possible approaches to answering each of these questions.

Is the "Right" Set of Projects Being Identified and Approved?

A challenge facing many companies (even in today's economic climate) is not necessarily the need to identify and develop projects that incrementally improve or upgrade existing systems and solutions, but rather the need to support a manageable process for identifying and developing the more innovative "wildcat" projects. ¹

It is useful to establish categories of investments or projects that correspond to the types of business impact expected and the associated risk/reward characteristics. A commonly cited taxonomy of IT investment portfolio classifications is shown in Figure 1. ²

These categories can be used to create corresponding subportfolios, and the question then becomes, how much should we allocate to each subportfolio and how does this affect the performance of the whole portfolio?

Returning to the Acme Entertainment example, we find that the two business units have allocated their project portfolio dollars in different ways (see Table 1).

Table 1 raises at least two sets of questions:

1. Why is the risk profile of the two business units so different?

Is it the result of different business conditions? Is it the result of different risk tolerances of the two IT steering committees? Or is it the result of the absence of a managed process for identifying and developing Grow and Transform projects within Business Unit 2?

If we track the portfolio allocations over time, we might begin to develop answers to these questions. If we don't, then our ability to understand and improve our portfolio management process and regularly achieve our desired outcomes becomes considerably compromised.

2. Which portfolio distribution delivers the best outcome for the business unit?

This is a much more complicated question that involves factoring in the different project success rates and the different value profiles for the different subportfolios. We would expect that the success rate for Run projects would be higher than that for Grow or Transform projects. We would also expect that the peak value profile of Run projects would be lower than that of Grow or Transform projects. This is, of course, the question addressed by Harry Markowitz in his 1952 paper defining modern portfolio theory, which showed that a diversified portfolio of high- and low-risk investments yields a higher return than a portfolio composed of only high-risk investments or low-risk investments. ³

For our IT portfolios, unfortunately, we do not have any way of answering or addressing this question of better-performing portfolios because we do not have an accepted method for assessing or measuring the collective business outcome produced by our portfolio. We are limited to deliverable-based metrics (e.g., percentage completed, percentage on time) or occasional project-level outcome statements, but we do not answer the portfolio-level performance question.

Are the "Right" Projects Receiving the "Right" Priority?

IT alignment with the business remains an important issue for most organizations, and a key element of this issue is whether the allocation of IT resources is consistent with the strategic and operational priorities of the business. A necessary question here is: What are the strategic and operational priorities of the business?

A recent Cutter E-Mail Advisor by Bob Benson and Tom Bugnitz ⁴ suggests that only 20% (or less) of business strategies and goals are actually worked on. How can IT support business strategies if 80% of those strategies do not in fact drive business activities and business managers undertake many initiatives that are not, strictly speaking, connected to the business strategy statements?

Because high-level strategy statements typically are not useful, it is necessary to use the concept of strategic intentions to identify exactly what the business leadership intends to do. The management team's strategic intentions state what they want to accomplish over a one- to three-year time frame. This provides the "hook" for the strategic IT plan and for project prioritization.

The whole process of developing strategic intentions (and associated weights and metrics) has been worthy of numerous books ⁵ and articles and won't be discussed here. Nor will I describe the process of assessing the projected impact of each project on each strategic intention. Rather, like a TV chef, I will fast-forward to the end of the process and pull the final results (or at least some of the final results), perfectly cooked, from the oven (see Table 2).

The elements of Table 2 are:

• Alignment score. Produced by summing the product of the expected contribution (on a 0-5 scale) of each project to the strategic intention times the weighted value of that strategic intention.

• Alignment profile. A statistical profile of the distribution of project alignment scores within each subportfolio. Over time, this can be used to track the strategic alignment characteristics of the portfolio and as feedback to the project identification and development process.

• Footprint. A high-level measure (on a 0-5 scale) of the expected impact of the project in terms of users and/or business processes.

• Value score. The product of the alignment score times the footprint value.

• Value profile. A statistical profile of the distribution of project value scores within each subportfolio. Over time, this can be used to track the value characteristics of the portfolio and as feedback to the project identification and development process.

Additional project and portfolio data elements that are not shown but should be included are risk levels (i.e., technical and business); project interdependencies (e.g., the success or failure of Project 1 has an impact on the risks associated with Project 3); net present value (NPV), internal rate of return (IRR), and other financial projections; and, of course, cost and schedule projections.

The important point is that we want to provide transparency for the prioritization process ⁶ that will allow us to backtest the validity of our prioritization criteria in relation to the outcomes of the individual projects and the collective portfolio and subportfolios.

Are the Prioritized Projects Being Delivered Efficiently and Effectively?

Metrics around project delivery remain important and will continue to receive the justified attention of most consultants/writers contributing to the literature of project/portfolio management. As the area is well covered, I will not give it much attention here.

The key point in Table 3 is to look at our delivery metrics at the subportfolio level. Do we understand what specific things are driving the differences in success rates for Run projects versus Grow projects for Business Unit 2? Addressing project delivery issues has an obvious impact on the value delivered for the whole portfolio.

Are the Completed Projects Delivering the Expected Value?

Here, I hope, is where things get really interesting. As projects are completed and the new systems and solutions are implemented, we need to be able to assess the actual value being delivered and how that compares to the original project value forecast.

Most organizations feel compelled to use methodologies such as NPV/IRR calculations that become much too complex and yield results that are presented with high precision but have really low accuracy. We need a simple methodology that yields a result with sufficient precision and accuracy. By "sufficient," I mean adequate for communicating relative (not absolute) value and providing directional data for portfolio management process improvements that correlate to the business outcomes we desire.

I propose employing the same methodology used during the prioritization stage to project a value score to now calculate a postimplementation value score (using business managers' assessment of the systems' support of strategic intentions and their assessment of system footprint). We can then compare the projected value scores with the postimplementation value scores and calculate a value conversion metric (Value^postimp/Value^projected).

Table 4 shows a portion of what a postimplementation scorecard should contain. As you might imagine, the postimplementation assessment data parallels much of the data included in project prioritization. In fact, we will want to understand the differences between the projected and actual values so that we can improve the project description and forecasting process as well as the development and implementation process. The value conversion factor shown in Table 4 is really a measure of how much of the total forecast value was delivered for all funded projects.

The scorecard is still organized by subportfolio categories so that performance differences can be identified and analyzed. In the example above, why is the value conversion factor for Grow projects higher than that for Run projects? Are the value scores for Run projects overforecast or underdelivered?

What Is the Total Value Delivered for the Initial $1 Million Invested?

With processes and data such as these in place, maybe Acme Entertainment's CFO can begin to have a sense of the relative performance of the IT portfolio management process. Table 5 illustrates how some of this summary value delivery data can be presented.

SO WHAT?

I certainly recognize that I have been using the term "value" in a nonfinancial way and that what I have been calling a "value score" or a "value point" is really a quantitative measure of what many would rightly consider to be qualitative assessments. And I don't think that matters.

What we sorely need is a set of portfolio-level metrics that can be developed, collected, and used over time to measure the performance of our IT portfolio management processes through the lifecycle stages of discovery, approval, funding, development, and implementation. Without good measures that focus on outcomes as much as deliverables, our IT portfolio management processes will not have the appropriate feedback data that will allow those processes to improve and develop into mature value delivery and value management practices. Consequently, too high a percentage of organizational resources will continue to be spent on low-value and failed projects.

An important element in the development of portfolio-level metrics and process improvements is the development of a true IT portfolio manager (an individual, not a committee) who manages both portfolio data and the portfolio management process itself. In addition, this individual should be responsible -- with the participation and advice of the steering committee -- for all project-funding decisions. Without a single point of responsibility, IT portfolio management will not be able to fully develop as a mature management practice.

ENDNOTES

¹ In oil and gas exploration, wildcat wells are drilled based on a large element of hope, in a frontier area where little is known about the subsurface.

² Maizlish, Bryan, and Robert Handler. IT Portfolio Management Step-by-Step. John Wiley & Sons, 2005.

³ Sharpe, William. Portfolio Theory & Capital Markets. McGraw-Hill, 2000.

⁴ Benson, Robert, and Tom Bugnitz. "For Strategic IT Planning, Focus on the Demand Side." Cutter Consortium Business-IT Strategies E-Mail Advisor, 28 January 2009.

⁵ Benson, Robert, Tom Bugnitz, and William Walton. From Business Strategy to IT Action: Right Decisions for a Better Bottom Line. John Wiley & Sons, 2004.

⁶ I am assuming a process distinction between approval, prioritization, and staging. Prioritization is the process whereby approved projects are actually funded, and we are assuming that not all approved projects actually receive funding. Staging is the scheduling process, and project work may be staged based on resource availability and other project dependencies unrelated to priority rank.

ABOUT THE AUTHOR