What Determines a Project Success or Failure

Chaos chronicle, project failure, project success

The reasons for the increase (1994 to 2000) in successful projects vary. First, the average cost of a project has been more than cut in half. Better tools have been created to monitor and control progress, and more highly skilled project managers are using improved management processes. The fact that there are processes is significant in itself. Most of these new projects are well within The Standish Group's criteria established in "Recipe for Success, 1998," which limits project duration to six months and project staff to six people. This article is based on information from the company's latest paper, "Extreme CHAOS 2001."

Standish Group, West Yarmouth, Mass., is a research firm that focuses on mission-critical project management applications or IT projects, and publishes a research report annually.

Canadian Problems with large IT projects
Only two of the seven projects looked at: the 2006 Census Online and My Account, My Business Account projects met all audit criteria for well-managed projects. Five of the projects were allowed to proceed with a business case that was incomplete or out-of-date or contained information that could not be supported. The majority of projects examined were undertaken even though departments lacked the appropriate skills and experience to manage the projects or the capacity to use the system to improve the way they deliver their programs.

Sainsbury's $526m Project Failure
In October 2005 giant British food retailer J Sainsbury had to write off $526m it had invested in an automated supply-chain management system. It seems that merchandise was stuck in the company's depots and warehouses and was not getting through to many of its stores. As a result, Sainsbury was forced to hire about 3000 additional clerks to stock its shelves manually. "If an ERP project costs more than $10m, your chances of coming in on time and on budget are statistically zero," Jim Johnson, chairman of Standish Group International, surveyed more than 8,000 software-application projects over the past few years. "You also have a 50/50 chance of its being canceled before it's completed after you've spent 200% of your budget."

Verdict on a $64m Project Failure
In April 2005 inter-departmental warfare played a significant role in the failure of a $64m federal IT project, according to the Auditor General, who had the last word on the long-running debacle yesterday. Among a series of criticisms the audit report notes ongoing tensions between the two agencies involved, Centrelink and the Department of Family and Community Services (FaCS). “There were tensions between FaCS and Centrelink at all levels during the project,” says the report. “It would have been surprising if there were not, as the two agencies had somewhat differing needs from the project. “Generally, work on the project progressed despite the tensions. The tensions were greater at more senior levels, where funding could not be agreed, and effective high-level governance of the project was not evident.” A memorandum of understanding covering funding and savings from the project was never concluded between the two agencies as intended, leaving responsibility for costs in dispute.

Software failure in hybrid vehicles
In May 2005 automaker Toyota recalled 160,000 of its Prius hybrid vehicles following reports of vehicle warning lights illuminating for no reason, and cars' gasoline engines stalling unexpectedly. But unlike the large-scale auto recalls of years past, the root of the Prius issue wasn't a hardware problem -- it was a programming error in the smart car's embedded code. The Prius had a software bug.

MCI 's Operational Infrastructure Failure
MCI had recently upgraded to a more scalable infrastructure, a move that reportedly caused the initial congestion and led to under-performance and complete network instability for over a week. As efforts to fix the problem repeatedly failed, MCI was forced to shut down the whole system for 24 hours. The Chicago Board of Trade was one of MCI's 3,000 customers rendered helpless by the outage. The failure disabled the electronic system that governs the board's exchange leading to an estimated loss of some 180,000 trades. At anywhere between $10,000 and $100,000 per trade, the loss of business was significant and tough to calculate.

UK Air Traffic Control Upgrade Project

UK Air Traffic Control Upgrade Project
Flights across the UK were grounded on Thursday after an air traffic control computer failure at West Drayton control centre. Nats' Flight Data Processing System failed at around 0600 BST for an hour, after overnight testing of an upgrade. Thousands of passengers have been experiencing delays as airlines work to clear the backlog of flights. Planes had to be grounded at airports including Gatwick, Heathrow, Manchester and Inverness. By mid-afternoon, delays at Heathrow and Gatwick were still 90 minutes, while at Stansted and Scottish airports the delay was about 30 minutes.

project failure at the Department for Work and Pensions (DWP)

Comair Back in Air After Outage
Comair airlines was scrambling to contain the fallout from a cancellation of over 1,000 flights on Christmas Day after its computer systems for reservations crashed, saying it expects to be running at 60% by Monday evening. Officials of the Cincinatti-based subsidiary of Delta Air Lines (Quote) blamed severe weather in the middle state regions as the cause of a surge in crew flight re-assignments that knocked out its computer reservations system. The disruption snarled airline traffic and stranded passengers up and down the eastern- and middle-region states through the weekend.

UK Government Operational Failure
A computer failure at the Department for Work and Pensions (DWP) was "blown out of proportion", says the government. The DWP admitted 80,000 staff were not able to process new pensions and benefits claims for several days, but regular payments were unaffected. DWP computer specialists have launched an investigation into the massive IT failure which took out a third of their computer network. BBC political correspondent James Hardy said on Friday: "After nearly five days of chaos and round the clock repairs, the system was finally given the all clear this afternoon."

What is Project Failure?

IT project failures are far more common than most people expect. In the last decade numerous studies and surveys on IT projects have shown that the success rate is around 25%, the failure rate is about 25%, and partial successes and failures fall somewhere in the middle. Typically, there are two types of project failure:

A project that consumes resources but fails to deliver an acceptable Return on Investment (ROI), is terminated before completion, or is poorly scoped so resource allocation is insufficient. This results in low adoption, or produces insufficient value and no learning lessons.
A project that consumes resources but fails to deliver as proposed, exceeds budget exceeds time, and doesn't meet specifications.

KPMG International's survey of 600 organizations across 22 countries revealed that 86% of respondents reported the loss of up to a quarter of their targeted benefits across their project portfolios. Nearly half of respondents reported at least one project failure in the past year, an improvement from KPMG’s 2003 survey where 57% experienced one or more project failures in the previous 12 months. 86% of projects have a business case but over 60% ignore it.
Sources: KPMG in Information Age April/May 2006.

Does Project Failure Happen Often?

There is evidence to support that project success rates are rising. The original Standish's 1994 CHAOS study found that only 16% projects met the criteria for success—completed on time, on budget, and with all the features and functions originally specified. In subsequent studies this rate has improved and project failures have decreased.

Sources: http://www.softwaremag.com/archive/2001feb/CollaborativeMgt.html and “Chaos, a recipe for success”, Standish Group, 1998

For 2004 results show that 29% of all projects succeeded (delivered on time, on budget, with required features and functions); 53% are challenged (late, over budget and/or with less than the required features and functions); and 18% have failed (cancelled prior to completion or delivered and never used). A staggering 66% of IT projects prove unsuccessful in some measure, whether they fail completely, exceed their allotted budget, aren't completed according to schedule or are rolled out with fewer features and functions than promised.

Sources: http://www.standishgroup.com/sample_research/PDFpages/q3-spotlight.pdf Chaos, 2004

IT projects are notorious for being over budget. In fact, Gopal Kapur, president of the Center for Project Management in San Ramon, Calif., estimates that 77% of projects blow their budgets, with an average cost overrun of 169%. As for the remaining 23%, Kapur doesn't have a lot of faith in those project managers. "They just lie about it," he says.
Sources: http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=95196 August 16, 2004 (Computerworld)

In 2005 organizations and governments will spend an estimated $1 trillion on IT hardware, software, and services worldwide. Of the IT projects that are initiated, from 5 to 15 percent will be abandoned before or shortly after delivery as hopelessly inadequate. Many others will arrive late and over budget or require massive reworking. Few IT projects, in other words, truly succeed.
Sources: http://www.spectrum.ieee.org/sep05/1685 September 2005 (IEEE)

Notable Project Failures

The following list of failures happened within the project itself supporting the Standish claim that close to 50% of projects are seriously challenged:

The IRS project on taxpayer compliance took over a decade to complete and cost the country an unanticipated $50 bn.
The Oregon DMV conversion to new software took eight years to complete, the budget grew by 146% ($123m) and public outcry eventually killed the entire project.
The State of Florida welfare system was plagued with numerous computational errors and $260m in overpayments!
In September 2006 Department of Homeland Security admitted project failure and closed the Emerge2 program $229m (a new financial IT system).
In May 2006 the disastrous Seasprite helicopter program for the Australian Navy, with $1bn spent, the helicopters were grounded due to software problems.
In April 2005 inter-departmental warfare played a significant role in the failure of a $64m federal IT project.
In 2005 British food retailer J Sainsbury had to write off $526m it had invested in an automated supply-chain management system.
In 2005 US Justice Department Inspector General report stated $170m FBI Virtual Case File project was a failure, after five years and $104m in expenditures. Over one 18-month period, the FBI gave its contractor nearly 400 requirements changes.
In 2005 the UK Inland Revenue produced tax payment overpayments of $3.45 bn because of software errors.
May 2005 major hybrid car manufacturer installed software fix on 160,000 vehicles. The automobile industry spends $2 to $3 bn per year fixing software problems.
July 2004 a new government welfare management system in Canada costing $200m was unable to handle a simple benefits rate increase. The contract allowed for 6 weeks of acceptance testing and never tested the ability to handle a rate increase.
In 2004 Avis cancelled an ERP system after $54.5m is spent
In 2002 the UK government wasted £698m on Pathway project, smartcards for benefits payments, & £134m overspend on magistrates' courts Libra system.

Notable Operational Failures

Most problems with IT projects happen during the implementation, after the solution is built and has undergone testing. For example:

The Hershey Foods ERP system implementation failure ($112m) led to massive distribution problems and 27% market share loss.
The FoxMeyer Drug ERP system implementation failure led to the collapse of the entire $5 bn company.

A more critical failure is after the implementation maybe days, weeks, or month into operation. These are the most expensive failures. The project has been deemed completed and therefore successful. These failures are unpredictable, unexpected and by far the most costly, because of impact on customers. With the arrival of the internet and ecommerce businesses have become increasingly more dependent on their operational systems to the point where if they are unavailable this can have a massive impact on the organization at different levels. The following list of failures happened post-project, during and after implementation:

In March 2007, US Airways struggled with a faulty reservation-and-ticketing system, and kept lines down by adding workers and asking travelers to use the Internet for check-ins.
In December 2006, a computer systems outage made it difficult for air traffic controllers in Florida to identify and track more than 200 flights in the air, allowing some planes to come too close together, according to officials.
In April 2006 Microsoft 's MSN search engine, the third most popular in the U.S., suffered an hours-long outage as queries returned an error message instead of Web page results.
In January 2006 Tokyo Stock Exchange Inc. was forced to halt trading 20 minutes earlier than normal because its system was close to capacity. In December software was questioned after an erroneous order to sell 610,000 shares of J-Com Co.
In July 2005 HSBC admitted hardware failure caused a major systems crash that hit thousands of customers for ATM, credit/debit, online services and internet, and it was the worst in its history.
In December 2004, Comair airlines had to face a cancellation of over 1,000 flights on Christmas Day after its computer systems for reservations crashed.
In November 2004, a computer failure at the Department for Work and Pensions (DWP) stopped 80,000 staff from processing new pensions and benefits claims for several days.
In October 2004, a computer failure at Waikato Hospital (NZ) left thousands of health workers out of pocket and forced the manual processing of patient records.
In October 2004, Avis Europe took a €45m hit due to problems with a new ERP system. Development halted with delays & higher costs due to implementation and design problems.
In September 2004, hundreds of flights were grounded for 3 hours at Western US airports. A computer failure knocked out radio contact between pilots and air-traffic controllers. In five instances airplanes passed very close to each other.
In August 2004, a computer crash prevented thousands of UK pensioners collecting benefits payments on the busiest day of year after the £500m Benefits Transfer system went down.
In June 2004, RBC fell behind processing salary deposits thousands of Canadian workers as millions of transactions were affected by a computer glitch that caused payroll delays.
In June 2004, an air traffic control computer failure saw massive air disruption across the UK. All flights from UK airports were grounded after a problem at the National Air Traffic Service.
In March 2003, 4,700 Kaiser Permanente patients received wrong medications because a computer glitch that caused labeling errors.
In July 2003, Orbitz LLC, an airline-owned travel site, suffered a 24 hours outage with database problem.
In March 2003, 4,700 Kaiser Permanente patients received wrong medications because a computer glitch that caused labeling errors.
In February 2003, thousands of BlackBerry users experienced service outages amid interruption on the company's network.
In February 2003, a glitch knocked Microsoft's bCentral services, leaving 15,000 small-business Web sites inaccessible for 8 hours.
In January 2003, MSN Messenger outage lasted five hours and affected 75 million customers.
Web site downtime cost UK businesses a total of £565m in 2001, and is expected to rise to £715m in 2002 according to Yankee Group. The report cites two main culprits: physical failure and human error.
In October 2001, a 12-hour outage put out most of Toronto-Dominion Bank’s elec-tronic banking channels. Millions of customers were unable to access their accounts.
In September 2001, a program glitch at Rogers Cable opened a back door into email accounts of 423,000 subscribers for 12 hours, allowing anyone to hijack them.
In September 2001, customers trying to use Citibank’s 2,000 ATMs in New York and other parts of the country continued to be thwarted by system problems.
In June 2001, the NYSE halted trading for more than an hour due to a failed software upgrade that caused half of its listings to be disabled. The exchange suspended trad-ing in all stocks. The NYSE halted trade in October 1998 with similar problems.
In March 2001, a computer malfunction temporarily shut down the flight dispatch system used by a Delta Airlines subsidiary, causing cancellations and delays affecting flights all day across the entire travel network in the eastern U. S. and Canada.
In January 2001, hardware problems shut down eBay for 11 hours. The outage was blamed on a failed data-backup system.
In December 2000, for the third time in less than two weeks, Amazon crashed for about 40 minutes after what the on-line retailer said was an internal software mix-up.
In December 2000, Walmart.com collapsed several times over the weekend as the company refined the newly designed Web site. Upgrades and site improvements trig-gered three blackouts, ranging from an hour to 90 minutes.
In November 2000, AOL suffered two outages that locked out 12,000 subscribers. A separate hardware problem also blanked email for thousands of users for 90 minutes.
In April 2000, a six-hour telecommunications outage stopped hundreds of trains on the tracks and disrupted rail traffic throughout CSX’s transportation system.
In March 2000, the Toronto Stock Exchange suffered a third blackout. The outage shut Canada’s main trading venue twice during the day, for more than 2.5 hours. These examples are just the tip of the iceberg. Most outages are unreported, as organiza-tions will avoid the harmful effects of publicity and the likely loss in customer confidence.

So Why Do Project Failures Happen?

When you examine the root causes to these problems, or failures, they tend to trace back to decisions made in all stages of the IT project. In parallel, investments in technology are not enough and need to be supported by investments and changes in processes and organization. To learn more about these type of failures and root causes read about Titanic Lessons for IT Projects.

More sources: