A Fresh Look at Software Project Estimation: Part I -- Promises Impossible to Keep

by E.M. Bennatan, Senior Consultant, Cutter Consortium

Software estimation is an area where more promises have been broken than in any other area of software development. But like addicts, we keep demanding promises because we need them to survive. As managers, there is a limit to what we can reasonably expect from software estimation, and when we cross that limit, we are, in effect, asking for short-term satisfaction in the form of promises that will not be kept.

University of Southern California researcher John Lewis questions whether software can be objectively estimated at all and asserts that there are limits that contradict some of the enthusiastic claims made by commercial software estimation advocates. ¹ Lewis believes that software development should not be viewed as a manufacturing process, but as a process more akin to research.

At Cutter Consortium, we have been observing the way software organizations estimate software successes and failures, as well as their effects on the organization. The first survey, which was conducted in 2002, ² exposed software estimation as "a tough beast to control." According to the findings, 40% of companies surveyed reported poor software schedule and budget estimation, while only 14% reported good performance. Analysis of this unimpressive report card indicated that successful software estimation requires as much of senior management as it does of the developers themselves.

The analysis of the survey concluded with a statement that the beast was alive and well: "Some have learned to restrain it; some have learned to live with it; and apparently some continue to suffer its wrath."

Now, six years later, we wondered what has changed. So Cutter again examined the beast to see whether we are now doing any better at controlling it. The latest survey studied software project estimation at more than 100 software development organizations, ³ and the analysis of the results used the same criteria that were used in the earlier survey (see sidebar).

The survey covered a broad range of organizations: 34% have fewer than 25 developers, 23% have between 25 and 100, and 43% have more than 100 developers (of them, 26% have more than 400). At 61% of surveyed companies, projects are typically small -- fewer than 5 person-years; at 32%, they are between 5 and 20 person-years (of them, at 10%, they are between 10 and 20); while the remaining 7% typically have larger projects that are more than 20 person-years (of them, at 3%, they are more than 40).

In this Executive Update, the first in a series of three comparing the results of the two software estimation surveys, we begin by exploring whether our estimating skills have improved. We then look at the proportion of project failures that are caused by overbudget and overscheduled projects, and we examine how organizations perceive their own performance in project estimation. Finally, we attempt to determine whether software organizations recognize the limitations of what various estimation methods can realistically achieve.

HAVE OUR ESTIMATES IMPROVED?

The first area we examined was comparative performance; how are projects estimated today compared to six years ago? Figure 1 shows the survey findings for schedule estimates. As a reminder, success was defined by the ±10% rule.

Just 33% of surveyed software organizations report a success rate of more than 70%, which means that more than 70% of their projects were developed on time (of which 8% report a success rate of more than 90%). It is interesting to note that the numbers are not radically different from the 2002 survey, when 35% reported a 70% success rate (of which 14% had a success rate of more than 90%). However, we do see a certain decline in the schedule estimation success rate, especially in the 90+% success group.

In the mid-to-lower end of the success scale, the differences were more profound. The current survey found that 36% of organizations had a schedule success rate of between 50% and 70% (it was just 26% in 2002), and 24% had a success rate of less than 50% (36% in 2002). Total failure (no projects were on time) was reported by 8% of organizations (3% in 2002).

The latest survey shows a small shift toward the center. This means that while schedule estimates today are not as poor on the low end of the scale as they were in 2002, they are also not as good on the high end.

We next examined the survey findings for budget estimates and compared them to the 2002 findings. According to the data presented in Figure 2, 37% of software organizations had a budget success rate of 70% or more (of which 10% had a greater than 90% success rate). This is surprisingly similar to the 2002 survey findings, where 38% had a 70+% success rate (of them 10% had a 90+% success rate).

Here, too, the mid-to-lower end of the budget success scale showed more significant differences. The new survey found that 32% had a success rate between 50% and 70% (24% in 2002), and 24% had a less than 50% success rate (34% in 2002). Total budget failure (no project completed within budget) was identical to total schedule failure, and was reported by 8% (4% in 2002). Here again, we found a shift toward the center, similar to the schedule estimate findings.

ARE ESTIMATE DISASTERS NOW LESS COMMON?

One of the most costly results of poor estimation skills is often the complete cancellation of a project. The survey examined the extent to which software organizations have abandoned or cancelled projects over the past three years, due to significant budget or schedule overruns.

Figure 3 shows that just over half (52%) of surveyed software organizations have had no projects cancelled or abandoned; in the 2002 survey, the figure was 56%. A further 29% report that up to 10% of their projects were cancelled (also 29% in 2002), and 17% state that between 10% and 25% had been abandoned (11% in 2002).

Beyond that, the numbers were minuscule. Just 2% report that the proportion of cancelled projects was between 25% and 50% (3% in 2002), and no surveyed companies had more than 50% of their projects abandoned (it was 1% in 2002).

It was quite surprising to see that project disasters have actually increased (for companies with more than 10% of project failures). In other words, the performance of poorly performing organizations has declined even further. ⁴ Other than that, the findings are very similar for both surveys.

SOFTWARE ORGANIZATIONS' SELF-PERCEPTION

To what extent are the results of the two surveys similarly reflected by the impressions of the software development organizations themselves? We examined this question, and the findings are shown in Figure 4.

Only 37% of software organizations believe that their ability to estimate software projects has improved over the last three years (down from 51% in 2002). A further 58% see no change (up from 43% in 2002), and 5% see a decline (down from 6% in 2002).

It is interesting that software organizations see less improvement of their estimation skills, compared to what they perceived six years ago, and the difference is quite significant. This finding can be interpreted in two ways; either estimation skills are leveling out, with little improvement expected in the future, or software projects are becoming more difficult to estimate.

SO WHAT ABOUT OBJECTIVE LIMITS?

Most areas of computer technology have improved immensely in the past six years. In software development, we have learned the benefits of shorter, more agile projects, and even for larger projects, we have adopted a more evolutionary approach. But while new tools, techniques, and methodologies abound, the survey findings do not indicate that they are having any appreciable effect on our ability to estimate.

Interestingly, the Cutter findings correlate rather well with data published recently by University of Southern California's Barry Boehm et al., which shows no improvement in software project estimation between 2002 and 2006 (see Figure 5). ⁵ (As we have seen, the Cutter surveys extend this observation to 2008.)

So why is our ability to estimate software not improving? Have we reached the limit of our abilities? This is a question that is raised by both the Cutter and the Boehm findings, as well as by the perception of software organizations (see Figure 4). All indicate that improvement in software estimation has come to a virtual standstill.

It is difficult, based on the survey data discussed so far, to determine the extent to which software organizations recognize the limitations on what can realistically be achieved with estimation techniques. But there is much more survey data available that can help us understand this issue.

The next two Updates in this series will discuss additional survey data and will continue to examine the question of objective limits on software estimation. We will look at the most common causes of project overruns and the methods used to resolve the most common problems. We will also report on software estimation training and the use of estimation techniques, and we will look at the effects of conflict within the organization derived from the different expectations of various groups.

Readers who would like to comment on this discussion or on their experience with software project estimation are invited to e-mail me at ebennatan@cutter.com.

ENDNOTES

¹ Lewis, J.P. "Large Limits to Software Estimation." ACM Software Engineering Notes, Vol. 26, No. 4, July 2001, pp. 54-59.

² Bennatan, E.M. "The State of Software Estimation: Has the Dragon Been Slain? Parts I-III." Cutter Consortium Agile Product & Project Management Executive Update, Vol. 3, Nos. 10-12, 2002.

³ To be statistically accurate, in some cases more than one response may have come from the same organization.

⁴ Admittedly, they are likely not the same companies as in 2002.

⁵ Boehm, Barry W., and Ricardo Valerdi. "Achievements and Challenges in Cocomo-Based Software Resource Estimation." IEEE Software, Vol. 25, No. 5, September 2008, pp. 74-83.

ABOUT THE AUTHOR