Attack Warning: Status of the Cheyenne Mountain Upgrade Program (Letter
Report, 09/01/94, GAO/AIMD-94-175).
Colorado's Cheyenne Mountain complex is designed to track and warn of
enemy air attacks. The Air Force program to modernize aging computer
systems at the facility is eight years behind schedule and $792 million
over budget. Air Force officials now project that the completion date
will slip by another three years and that development costs will
increase by at least $104 million. Both projections, GAO believes, may
be understated. Moreover, major subsystem incompatibilities could
prevent the upgraded system from becoming fully operational.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: AIMD-94-175
TITLE: Attack Warning: Status of the Cheyenne Mountain Upgrade
Program
DATE: 09/01/94
SUBJECT: Air defense systems
Warning systems
Systems evaluation
Systems development life cycle
Systems conversions
Air Force procurement
Cost overruns
Systems compatibility
Research and development costs
Computerized information systems
IDENTIFIER: NORAD Cheyenne Mountain Complex Computer System
NORAD Cheyenne Mountain Upgrade Program
Colorado Springs (CO)
NORAD Tactical Warning and Attack Assessment System
NORAD Communications System Segment Replacement Program
NORAD Command Center Processing and Display System
Replacement Program
NORAD Survivable Communications Integration System
**************************************************************************
* This file contains an ASCII representation of the text of a GAO *
* report. Delineations within the text indicating chapter titles, *
* headings, and bullets are preserved. Major divisions and subdivisions *
* of the text, such as Chapters, Sections, and Appendixes, are *
* identified by double and single lines. The numbers on the right end *
* of these lines indicate the position of each of the subsections in the *
* document outline. These numbers do NOT correspond with the page *
* numbers of the printed product. *
* *
* No attempt has been made to display graphic images, although figure *
* captions are reproduced. Tables are included, but may not resemble *
* those in the printed version. *
* *
* A printed copy of this report may be obtained from the GAO Document *
* Distribution Facility by calling (202) 512-6000, by faxing your *
* request to (301) 258-4066, or by writing to P.O. Box 6015, *
* Gaithersburg, MD 20884-6015. We are unable to accept electronic orders *
* for printed documents at this time. *
**************************************************************************
Cover
================================================================ COVER
Report to the Chairman, Subcommittee on Defense, Committee on
Appropriations, House of Representatives
September 1994
ATTACK WARNING - STATUS OF THE
CHEYENNE MOUNTAIN UPGRADE PROGRAM
GAO/AIMD-94-175
Cheyenne Mountain Upgrade Program
Abbreviations
=============================================================== ABBREV
ADP - x
AIMD - Accounting and Information Management Division
APCC - Alternate Processing and Correlation Center
CCPDS-R - Command Center Processing and Display System Replacement
CMU - Cheyenne Mountain Upgrade
CSSR - Communications System Segment Replacement
GAO - General Accounting Office
IOC - initial operational capability
ITW/AA - Integrated Tactical Warning and Attack Assessment
NORAD - North American Aerospace Defense Command
SCIS - Survivable Communications Integration System
SPADOC - Space Defense Operations Center
Letter
=============================================================== LETTER
B-257932
September 1, 1994
The Honorable John P. Murtha
Chairman, Subcommittee on Defense
Committee on Appropriations
House of Representatives
Dear Mr. Chairman:
On March 22, 1994, you requested that we review the status of
computer upgrades at the Air Force's Cheyenne Mountain Complex at
Colorado Springs, Colorado. These upgrades, known collectively as
the Cheyenne Mountain Upgrade (CMU) program, are intended to
modernize the systems which are the nucleus of the worldwide
Integrated Tactical Warning and Attack Assessment (ITW/AA) system.
Designed to identify and track potential enemy objects, these systems
provide critical surveillance, air defense warning, and attack
assessment information to United States and Canadian leaders.
The CMU program has experienced a series of development problems
since it began in 1981. This report, the eleventh that GAO has
issued on the program since 1988, discusses (1) the current status of
the cost and schedule for the CMU program, (2) the status of CMU
subsystems development and performance, (3) the status of the
integration of the systems, and (4) management issues affecting
future CMU development. Our work was performed between April 1994
and July 1994 in accordance with generally accepted government
auditing standards. Appendix I provides a detailed description of
our objectives, scope, and methodology for the report.
RESULTS IN BRIEF
------------------------------------------------------------ Letter :1
After a series of delayed completion schedules and increased
development cost estimates, the CMU program is 8 years behind
schedule and $792 million over budget. The Air Force recently
determined that the program will meet neither its currently scheduled
completion date of December 1995 nor target development costs of
$1.76 billion. Air Force officials now project that the completion
date will slip by an additional
3 years and that development costs will increase by at least $104
million. Both projections, GAO believes, may be understated.
Initial versions of several CMU subsystems that the Air Force has
declared operational are unreliable and do not meet users'
requirements. As a result, those subsystems must be operated in
parallel with the systems they are meant to replace. Concurrently
operating the old Cheyenne Mountain systems costs the Air Force an
additional $22 million annually.
Moreover, significant incompatibilities among the CMU component
subsystems have been identified that could prevent the overall CMU
system from becoming fully functional. Further incompatibilities
have been identified between CMU and the rest of the ITW/AA system
that could hinder integration of the worldwide ITW/AA system. In
1992, GAO reported that CMU would continue to face serious
development and integration problems until the Air Force performs the
analysis needed to define an overall CMU architecture.\1 Because the
Air Force has not yet fully addressed this issue, integration
problems continue to surface and will persist.
The Air Force recognizes these problems and has already made some
improvements to the CMU management structure to provide a more
integrated approach. It is currently exploring further changes.
Nevertheless, managing the complex schedule of developing, testing,
deploying, and finally integrating the large number of independently
developed CMU systems will pose formidable challenges. Continued
close oversight of the program is warranted to ensure that the Air
Force faces up to these challenges.
--------------------
\1 Attack Warning: Lack of System Architecture Contributes to Major
Development Problems (GAO/IMTEC-92-52, June 11, 1992).
BACKGROUND
------------------------------------------------------------ Letter :2
The North American Aerospace Defense Command (NORAD) is responsible
for warning United States and Canadian leaders of any missile, air,
or space attack. This mission is supported by the ITW/AA system,
which is designed to identify and track potential enemy objects. The
ITW/AA system consists of a worldwide network of ballistic missile,
atmospheric, and space warning systems; intelligence centers;
associated communications links; and command and control centers.
The computer and telecommunications hardware and software at the
Cheyenne Mountain Complex, the command center for NORAD, form the
nucleus of the ITW/AA system. Information from Cheyenne Mountain is
provided to national command authorities primarily through the Air
Force's Space Command.
In 1981, the Air Force began a modernization effort consisting of
five separate acquisitions to replace aging and obsolete computer
systems at the Cheyenne Mountain Complex. The five original
subsystems under development for the Cheyenne Mountain Complex are
(1) the Communications System Segment Replacement (CSSR) to process
and control most of the internal and external automated
communications at the Cheyenne Mountain Complex, (2) the Survivable
Communications Integration System (SCIS) to provide multiple
survivable communications capabilities between missile warning
sensors, command centers, and other users, (3) the Command Center
Processing and Display System Replacement (CCPDS-R) to process and
display ballistic missile warning data received from sensors located
throughout the world, (4) Granite Sentry to process and display data
for use by all air defense, command post, battle staff, and weather
support activities, and (5) the Space Defense Operations Center 4
(SPADOC 4) to process space defense and space surveillance data. A
sixth subsystem, the Alternate Processing and Correlation Center
(APCC) at Offutt Air Force Base in Nebraska, was added in 1989. The
APCC will provide backup missile warning and air defense information
should the systems at the Cheyenne Mountain Complex fail.
From 1988 to 1992, we issued 10 reports addressing a variety of
aspects of the CMU program, including assessments of individual CMU
subsystems as well as overall reviews of the CMU program. In
general, the reports discussed development problems being experienced
on subsystem projects and ways to preclude more such problems arising
in the future. In 1992, we reported that the Air Force was
continuing to develop CMU subsystems as five separate systems and
that it lacked an overall CMU architecture, thus increasing the risks
that the CMU would not meet systems requirements, operate as an
integrated unit, or be capable of evolving to meet the needs of new
missions in the future. We also noted that because of cost and
schedule overruns, the Air Force had deferred some CMU requirements,
leaving the system with less capability than originally planned. A
list of our previous reports on the CMU program appears at the end of
this report.
THE CMU PROGRAM CONTINUES TO
EXPERIENCE COST INCREASES AND
SCHEDULE DELAYS
------------------------------------------------------------ Letter :3
The five original acquisition programs for the Cheyenne Mountain
Complex were initially scheduled for completion in 1987 at a cost of
$968 million. In 1989, we reported that the program was 7 years
behind schedule and already $342 million over budget. We attributed
these problems to a cumbersome and diffuse management structure.\2
The Air Force responded by combining the five original subsystems and
the APCC backup subsystem into a single CMU program, and committing
to completion by December 1995 at a cost of $1.58 billion. However,
we reported in 1991 that this cost estimate did not include funding
for all requirements, and we estimated that costs for completing a
fully functional, mission-ready system would surpass $1.9 billion.\3
The Air Force recently determined that it will not meet either its
December 1995 schedule milestone or its last approved $1.76 billion
cost estimate. The Air Force states that it has been unable to meet
its cost and schedule milestones because it underestimated the
complexities of delivering and integrating systems to support three
parallel warning missions\4 and the difficulty and cost of subsystem,
mission, and integration testing. Air Force officials now project
that CMU will be complete in December 1998 but that costs can only be
estimated to exceed $1.864 billion, since they concede that
additional unidentified integration problems are likely to occur. As
discussed below, we believe even these cost estimates are
understated.
--------------------
\2 Attack Warning: Better Management Required to Resolve NORAD
Integration Deficiencies (GAO/IMTEC-89-26, July 7, 1989).
\3 Attack Warning: Costs to Modernize NORAD's Computer System
Significantly Understated (GAO/IMTEC-91-23, April 10, 1991).
\4 The three missions are to warn against ballistic missile, space,
and atmospheric (bomber) attacks.
MAJOR CMU SUBSYSTEMS STILL NEED
SIGNIFICANT DEVELOPMENT WORK
------------------------------------------------------------ Letter :4
Although the Air Force committed CMU development to completion by
December 1995, significant work remains to be done on some of its
subsystems. Air Force operation and certification testing has
revealed that these subsystems do not meet user requirements.
Additionally, the costs for this work are being understated because
the Air Force has commingled the costs of some development activities
with costs related to other systems.
THE AIR FORCE HAS PLACED
SOME SUBSYSTEMS IN USE
PREMATURELY
---------------------------------------------------------- Letter :4.1
The CMU subsystems are in various stages of development, test, and
operations. SCIS is the only subsystem still completely in
development; initial operational test and evaluation is scheduled to
begin at 20 sensor sites in September 1994. CCPDS-R has been
installed at Cheyenne Mountain, is being tested and monitored, but is
not yet operational. Portions of CSSR and SPADOC 4 are operational;
other parts are still being tested. Phase III of Granite Sentry is
operational; the subsequent phase is under development. Construction
of the APCC, the backup correlation center, is complete. The APCC
missile warning functions are being tested and evaluated.
The Air Force recently declared three of these subsystems (CSSR,
CCPDS-R, and SPADOC 4) to have reached initial operational capability
(IOC). However, tests conducted by the Air Force Operational Test
and Evaluation Center found that none of the three subsystems met
users' requirements. For example, the CSSR subsystem, which must
relay critical attack and warning messages to other CMU subsystems,
took too long to do this in many cases and sometimes failed to relay
messages altogether. Additionally, further independent testing by
U.S. Space Command found that only one of the three subsystems
(SPADOC 4) met Air Force standards for accuracy, reliability, and
timeliness. Passing these tests is essential before the new CMU
subsystems can be made operational and the old computer systems
removed. Until then, continuing to operate the old systems in
conjunction with the new, costs the Air Force $22 million annually.
DEVELOPMENT COSTS ARE
UNDERSTATED
---------------------------------------------------------- Letter :4.2
Because the Air Force is paying for some CMU subsystem development
work with funds designated for operations and maintenance, the cost
of developing the CMU is being understated. While a system is
officially in its development phase, any necessary fixes or changes
are to be paid for using money set aside specifically for
development. However, once a system is placed in regular use, any
fixes, changes, or deferred enhancements are to be paid for using
operations and maintenance funds. At the Cheyenne Mountain Complex,
operations and maintenance funds are designated for maintaining the
old ITW/AA systems as well as the new CMU subsystems that are being
used with them. As a result, the costs for the substantial amount of
development work that remains to be done on these CMU subsystems are
being commingled with ITW/AA costs and are being paid from operations
and maintenance funds. For example, in April 1994, the Air Force
identified 34 changes essential for the independent operation of the
SPADOC 4 subsystem that would be paid for with operations and
maintenance funds.
The Air Force was not able to provide an estimate of the associated
development costs that are being paid for with operations and
maintenance funds because they do not specifically track how much of
their operations and maintenance funds are being used for CMU
development activities.
INTEGRATION PROBLEMS REMAIN
UNRESOLVED
------------------------------------------------------------ Letter :5
During testing of the complete missile warning function, the Air
Force identified significant incompatibilities among CMU's component
subsystems, which could prevent the complete CMU system from
functioning properly. Further incompatibilities have been identified
between CMU and other, already deployed ITW/AA systems that could
hinder integration of the worldwide ITW/AA system. While parts of
the system might perform their functions successfully, the parts
might fail to work together properly, causing warning data to be lost
or inaccurately transmitted to decisionmakers.
We reported on CMU integration problems as early as 1988 and, in a
later report, recommended that the Air Force develop an overall
system architecture. Such an architecture would have specified how
subsystems interact and could have been used by developers as a
common reference point to ensure interoperability among subsystems.
Without it, we projected that the program would continue to face
integration and development problems that were continuing to surface
at that time.\5
As of April 1994, the Air Force unit responsible for ITW/AA
integration had identified (1) eight unresolved compatibility
problems with major impact to cost, schedule, and performance, (2)
four compatibility problems that have been resolved, but for which
the final cost and schedule impacts have not yet been determined, and
(3) 144 potential integration problems that will require more
analysis. The Air Force acknowledges that these 144 items may result
in additional significant problems being identified.
One example of a serious incompatibility among CMU subsystems
involves the transmittal of data between SPADOC 4 and CSSR. The
space surveillance data generated by SPADOC 4 must go through the
CSSR communications system for distribution to ITW/AA system users.
SPADOC 4 needs to transmit data at 30 frames per second; however,
CSSR can process only 3 frames per second. During tests, this
incompatibility has resulted in data either being lost or taking an
unacceptably long time to be transmitted. The Air Force anticipates
that the solution to this problem will require an upgrade to the CSSR
computers and disks that will not be fully implemented until 1996.\6
Also, the Air Force has identified compatibility problems between CMU
subsystems and the already deployed systems and sensor sites that
constitute the remainder of the ITW/AA system. For example, each
sensor site independently sends a message to the Cheyenne Mountain
Complex when it identifies a potential threat. When more than one
sensor identifies a threat, more than one message is sent to CMU.
This serves to increase confidence in the veracity of the threat.
However, under certain circumstances the SPADOC 4 and CCPDS-R
subsystems may interpret multiple incoming messages as being
extraneous duplicates and discard all but one, thus losing important
information. Other integration problems exist as well, and, in
general, the Air Force recognizes that it has underestimated the
difficulty of integrating CMU with existing ITW/AA systems.
--------------------
\5 Attack Warning: NORAD's Communications System Segment Replacement
Program Should Be Reassessed (GAO/IMTEC-89-1, November 30, 1988) and
Attack Warning: Lack of System Architecture Contributes to Major
Development Problems (GAO/IMTEC-92-52, June 11, 1992).
\6 This problem was first reported by us in Attack Warning: NORAD's
Communications System Segment Replacement Program Should Be
Reassessed (GAO/IMTEC-89-1, November 30, 1988). At that time we
noted that several of the CMU subsystems were being sized to process
different amounts of data, and we concluded that these inconsistent
design parameters could adversely affect communications among CMU
subsystems.
AIR FORCE HAS MADE IMPROVEMENTS
IN PROGRAM MANAGEMENT
------------------------------------------------------------ Letter :6
Two significant management improvements have occurred in the CMU
program since we concluded our previous reviews. In 1989, we
reported that the acquisition effort for ITW/AA systems was
characterized by a cumbersome structure, divided responsibility, poor
management continuity, and deferred problem resolution.\7 These
problems contributed to the delivery of subsystems that did not meet
specifications and could not be effectively integrated without
additional, costly changes. Four years later, in 1993, the Air Force
began taking steps to address these problems.
The first improvement is that the Air Force is no longer developing
the CMU program as individual subsystems. Instead, the program is
now managed in four blocks that correspond to the system's major
missions: two for missile warning, one for air warning, and one for
space surveillance. The block concept focuses on ensuring that all
of the subsystems within each block can work together effectively to
warn, for example, of a missile launch. This increases the
possibility that development, testing, and management will be better
coordinated throughout the life cycles of the subsystems.
Second, the Air Force has made the CMU program director responsible
for CMU during its acquisition and development phases as well as
during the subsequent operations and maintenance phases. Having one
manager responsible across all phases of the system's life cycle will
likely improve the coordination and cooperation among the many
offices involved in development, testing, and maintenance.
--------------------
\7 Attack Warning: Better Management Required to Resolve NORAD
Integration Deficiencies (GAO/IMTEC-89-26, July 7, 1989).
FURTHER PROGRAM MANAGEMENT
CHANGES ARE BEING CONSIDERED
---------------------------------------------------------- Letter :6.1
In response to the recent determination that CMU will not meet its
approved costs and scheduled milestones, the Assistant Secretary of
the Air Force for Acquisition chartered a task force of Air Force
personnel, known as the Red Team, to review the current CMU program
approach. The Red Team reviewed CMU requirements, operations,
testing, contractor performance, configuration management, and
integration. They then made specific recommendations to improve
these activities. Subsequently, the Air Force developed a revised
CMU management plan that would consolidate the implementation of
fixes and enhancements to CMU subsystems and extend the milestones
for integration and testing of the overall CMU system through
December 1998.
The revised CMU management plan defines a core program of basic
capabilities to be achieved in November 1995. After that, three
additional phases are planned through December 1998. Each phase
includes a single annual delivery of hardware and software for CMU
subsystems as well as testing of both individual subsystems and the
integrated "blocks" of subsystems that correspond to warning
missions. The last phase includes a test of the end-to-end CMU
system, which, if successful, will signal the completion of the
program and the old systems will be decommissioned. At that point,
additional development work is to be undertaken on an incremental,
"evolutionary" basis, through a single annual software and hardware
upgrade based on priorities set by the users.
GAO OBSERVATIONS
------------------------------------------------------------ Letter :7
As CMU development enters its 13th year, schedules continue to slip
and costs continue to rise. Serious subsystem development and
integration problems continue to slow CMU deployment. The continuing
absence of an overall architecture that fully describes system and
subsystem requirements, which we reported on previously, increases
the risks that CMU will not meet original systems requirements nor be
capable of evolving to meet the needs of new missions in the future.
The Air Force has already taken some steps to improve program
management and is now working on a revised management plan that will
provide more time for testing and integration of CMU subsystems. The
new plan will require additional funding and a further delay, now
projected by the Air Force to be at least $104 million and 3 years.
We will evaluate these changes in future work and assess the
likelihood that they will effectively address CMU's many complex
problems.
---------------------------------------------------------- Letter :7.1
As requested, we did not obtain comments on a draft of this report
from the Department of Defense. However, we discussed the
information contained in it with Defense officials, including the
Vice Commander, Air Force Space Command; the Commander, Cheyenne
Mountain Complex; the System Program Director, Electronic Systems
Center; and the Office of the Secretary of Defense (Acquisition) at
the Pentagon. These officials generally concurred with the facts
presented.
We are providing copies of this report to the Secretary of Defense;
the Secretary of the Air Force; the Director, Office of Management
and Budget; and other interested parties. We will also make copies
available to others upon request.
This work was conducted under the direction of David O. Nellemann,
Director for Information Resources Management/National Security and
International Affairs, who can be reached at (202) 512-6240. Other
major contributors are listed in appendix II.
Sincerely yours,
Gene L. Dodaro
Assistant Comptroller General
OBJECTIVES, SCOPE, AND METHODOLOGY
=========================================================== Appendix I
On March 22, 1994, the Chairman, Subcommittee on Defense, House
Appropriations Committee, requested that GAO ascertain the status of
the computer upgrades at Cheyenne Mountain. We agreed to examine (1)
the current status of the cost and schedule for the CMU program, (2)
the status and results of developmental and operational testing of
CMU subsystems, (3) the status of the integration of the systems, and
(4) management issues affecting future CMU development.
We performed work at Air Force Space Command, U.S. Space Command,
the Air Force Operational Test and Evaluation Center, the Space
Warning and Systems Center, the Site Activation Team of the
Electronic Systems Center, and Air Force Materiel Command's
Detachment 25 at Peterson Air Force Base in Colorado Springs,
Colorado. We also interviewed officials at the Electronic Systems
Center at Hanscom Air Force Base in Bedford, Massachusetts.
As the basis for this work, we obtained mission status briefings from
the six U.S. Air Force Space Command subsystem leads, as well as
briefings from six other Air Force and U.S. Space Command
organizations involved in the development, test, evaluation, and
maintenance of CMU. We reviewed and analyzed test result reports,
budget documents, and subsystem position papers. We obtained
briefings and additional information from the Red Team assigned by
the Assistant Secretary for Acquisition to develop ways to mitigate
CMU's projected cost and schedule increases. We also met with
officials at Air Force and Department of Defense headquarters at the
Pentagon.
We discussed the information in this report with appropriate Defense
program officials, including the Vice Commander, Air Force Space
Command; the Commander, Cheyenne Mountain Complex; the System Program
Director, Electronic Systems Center; and the Office of the Secretary
of Defense (Acquisition) at the Pentagon.
MAJOR CONTRIBUTORS TO THIS REPORT
========================================================== Appendix II
ACCOUNTING AND INFORMATION
MANAGEMENT DIVISION,
WASHINGTON, D.C.
-------------------------------------------------------- Appendix II:1
John A. de Ferrari, Assistant Director
Keith A. Rhodes, Technical Assistant Director
David A. Powner, Evaluator-in-Charge
Elizabeth L. Johnston, Assignment Manager
DENVER REGIONAL OFFICE
-------------------------------------------------------- Appendix II:2
Jamelyn A. Smith, Regional Assignment Manager
RELATED GAO PRODUCTS
============================================================ Chapter 0
Granite Sentry (GAO/IMTEC-92-84R, September 21, 1992).
Attack Warning: Status of the Survivable Communications Integration
System (GAO/IMTEC-92-61BR, July 9, 1992).
Attack Warning: Lack of System Architecture Contributes to Major
Development Problems (GAO/IMTEC-92-52, June 11, 1992).
Computer Technology: Air Attack Warning System Cannot Process All
Radar Track Data (GAO/IMTEC-91-15, May 13, 1991).
Attack Warning: Costs to Modernize NORAD's Computer System
Significantly Understated (GAO/IMTEC-91-23, April 10, 1991).
Defense Acquisition: Air Force Prematurely Recommends ADP
Acquisitions (GAO/IMTEC-90-7, March 29, 1990).
Attack Warning: Defense Acquisition Board Should Address NORAD's
Computer Deficiencies (GAO/IMTEC-89-74, September 13, 1989).
Attack Warning: Better Management Required to Resolve NORAD
Integration Deficiencies (GAO/IMTEC-89-26, July 7, 1989).
Space Defense: Management and Technical Problems Delay Operations
Center Acquisition (GAO/IMTEC-89-18, April 20, 1989).
Attack Warning: NORAD's Communications System Segment Replacement
Program Should Be Reassessed (GAO/IMTEC-89-1, November 30, 1988).