Weather Forecasting: Unmet Needs and Unknown Costs Warrant Reassessment
of Observing System Plans (Chapter Report, 04/21/95, GAO/AIMD-95-81).
Pursuant to a congressional request, GAO reviewed the National Weather
Service's (NWS) Automated Surface Observing System (ASOS), focusing on:
(1) the extent to which NWS has addressed ASOS operational problems; (2)
the costs of resolving these operational problems; and (3) whether NWS
plans for enhancing ASOS are reasonable.
GAO found that: (1) six of the eight sensors in the ASOS system do not
meet key performance specifications; (2) ASOS shortfalls are caused by
contractor failure to deliver products that meet specifications and
government failure to furnish sufficient equipment; (3) ASOS reliability
problems surfaced after deployment because NWS did not perform
reliability testing prior to deployment; (4) NWS does not have adequate
personnel or integrated information systems for it to isolate and
correct ASOS failures at Federal Aviation Administration sites; (5) ASOS
does not satisfy the weather observational needs of many users; (6) ASOS
users state that incorrect ASOS observations could risk aviation
efficiency and safety, and skew national climate research; (7) in March
1995, the National Oceanic and Atmospheric Administration resumed the
commissioning of observers on the basis of expected system and equipment
improvements; (8) although NWS has recently begun to recognize user
needs that go beyond ASOS specifications, it has not yet determined how
it will enhance or supplement ASOS to address user needs; (9) NWS cannot
reliably assess whether system enhancements will continue to be cost
beneficial without a complete cost estimate; (10) ASOS problems have
delayed NWS plans for releasing human weather observers; and (11)
although ASOS officials stated that user needs will be met before
observers are released, NWS does not have a strategy to ensure that the
information needs of all users will be met.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: AIMD-95-81
TITLE: Weather Forecasting: Unmet Needs and Unknown Costs Warrant
Reassessment of Observing System Plans
DATE: 04/21/95
SUBJECT: Weather forecasting
Communications operations
Interagency relations
Testing
Earth resources satellites
Information resources management
Meteorological research
Cost control
Systems design
Strategic planning
IDENTIFIER: NWS Automated Surface Observing System
NWS Next Generation Weather Radar
NWS Next Generation Geostationary Operational Environmental
Satellite
NOAA/NASA GOES-Next Satellite Program
NWS Advanced Weather Interactive Processing System
**************************************************************************
* This file contains an ASCII representation of the text of a GAO *
* report. Delineations within the text indicating chapter titles, *
* headings, and bullets are preserved. Major divisions and subdivisions *
* of the text, such as Chapters, Sections, and Appendixes, are *
* identified by double and single lines. The numbers on the right end *
* of these lines indicate the position of each of the subsections in the *
* document outline. These numbers do NOT correspond with the page *
* numbers of the printed product. *
* *
* No attempt has been made to display graphic images, although figure *
* captions are reproduced. Tables are included, but may not resemble *
* those in the printed version. *
* *
* A printed copy of this report may be obtained from the GAO Document *
* Distribution Facility by calling (202) 512-6000, by faxing your *
* request to (301) 258-4066, or by writing to P.O. Box 6015, *
* Gaithersburg, MD 20884-6015. We are unable to accept electronic orders *
* for printed documents at this time. *
**************************************************************************
Cover
================================================================ COVER
Report to Congressional Requesters
April 1995
WEATHER FORECASTING - UNMET NEEDS
AND UNKNOWN COSTS WARRANT
REASSESSMENT OF OBSERVING
SYSTEM PLANS
GAO/AIMD-95-81
Weather Forecasting
Abbreviations
=============================================================== ABBREV
ADF - Airline Dispatchers Federation
AOMC - ASOS Operations and Monitoring Center
ASOS - Automated Surface Observing System
ATA - Air Transport Association
AWIPS - Advanced Weather Interactive Processing System
DOD - Department of Defense
FAA - Federal Aviation Administration
GAO - General Accounting Office
GOES-Next - Next Generation Geostationary Operational Environmental
Satellite
IFR - Instrument Flight Rules
NASA - National Aeronautics and Space Administration
NATCA - National Air Traffic Controllers Association
NEXRAD - Next Generation Weather Radar
NOAA - National Oceanic and Atmospheric Administration
NWS - National Weather Service
NWSEO - National Weather Service Employees Organization
OSO - Office of Systems Operations
Letter
=============================================================== LETTER
B-260400
April 21, 1995
The Honorable Larry Pressler
Chairman
The Honorable Ernest F. Hollings
Ranking Minority Member
Committee on Commerce,
Science, and Transportation
United States Senate
The Honorable Robert S. Walker
Chairman
The Honorable George E. Brown, Jr.
Ranking Minority Member
Committee on Science
House of Representatives
This report responds to your request that we review the National
Weather Service's (NWS) Automated Surface Observing System (ASOS).
As agreed with your offices, we focused on ASOS' operational problems
and NWS' efforts to resolve them, the cost of resolving these
problems, and whether NWS' plans for implementing ASOS make sense in
light of these problems.
We are providing copies of this report to the Secretary of Commerce;
the Secretary of Transportation; the Director, Office of Management
and Budget; and interested congressional committees. Copies will
also be made available to others upon request.
Please call me at (202) 512-6253 if you or your staffs have any
questions concerning the report. Other contributors to this report
are listed in appendix III.
Joel C. Willemssen
Director, Information Resources
Management/Resources, Community,
and Economic Development
EXECUTIVE SUMMARY
============================================================ Chapter 0
PURPOSE
---------------------------------------------------------- Chapter 0:1
The historical approach of humans observing the weather is quickly
taking a backseat to sophisticated electronic sensors and
state-of-the-art computer techniques. Such progress is not occurring
without some problems, however, and given the crucial role the
weather plays each day in the life of every American, the sooner
these issues are resolved, the better.
One component of the National Weather Service's (NWS) over $4.5
billion modernization that is experiencing problems, as well as
successes, is its $351 million Automated Surface Observing System
(ASOS). Concerned with anecdotal reports of ASOS problems and
uncertain about their scope and severity, the Chairman and Ranking
Minority Member, House Committee on Science, and the Chairman and
Ranking Minority Member, Senate Committee on Commerce, Science, and
Transportation requested that GAO determine (1) what ASOS problems
exist and how effectively NWS is resolving them, (2) the cost of
resolving these problems, and (3) whether NWS' plans for implementing
ASOS make sense in light of these problems.
BACKGROUND
---------------------------------------------------------- Chapter 0:2
ASOS is intended to automate the observation and dissemination of
data on temperature and dew point, visibility, wind direction and
speed, pressure, cloud height and amount, and types and amounts of
precipitation. According to NWS, ASOS offers considerable advantages
over the current manual surface observation practices, including
uniform measurements across all sites and times, continuous
observation and reporting, and more observing sites nationwide. ASOS
is also intended to replace human observers at many airports and most
NWS weather service offices.
NWS, in cooperation with the Federal Aviation Administration (FAA)
and Department of Defense (DOD), manages the $351 million ASOS
program. NWS has contracted with AAI Corporation for a minimum of
868 ASOS units--245 for itself, 537 for FAA, and 86 for DOD. It
plans to buy and install all 868 by fiscal year 1997. Contract
options allow NWS to buy up to 832 additional units.
As of December 1994, NWS had bought 617 units, and of those, 491 had
been accepted. Of the 491 accepted units, 47 had been
commissioned.\1 No human observers have yet been released. These
observers continue to provide some of the observations that ASOS
cannot, such as thunderstorms and tornadoes, as well as observations
in the event ASOS errs or fails.
--------------------
\1 An ASOS unit is commissioned when it provides the official weather
observation. For those sites that have a human observer, this means
that ASOS, rather than the human observer, provides the official
weather observation.
RESULTS IN BRIEF
---------------------------------------------------------- Chapter 0:3
While ASOS meets many of its specified requirements, it does not meet
them all, and it does not provide certain capabilities that some
users say are critical to ensuring safe aviation, effective
weather-related decision-making, and accurate climatological
analysis. NWS has actions under way to correct shortfalls in meeting
specifications and is identifying or addressing other user concerns.
However, because of the volume and severity of these problems, NWS
temporarily halted further ASOS commissionings and has delayed the
release of human weather observers.
While NWS has corrective actions under way, it has not determined the
full range of problems that it will address, nor how much the system
enhancements or supplements needed to do so will cost. Moreover, it
has yet to establish explicit, user-based system performance and
cost/benefit criteria for releasing observers. Until NWS does so, it
runs the risk of spending money to enhance and supplement a system
that may not be cost-effective and providing weather observations
that could impair important public safety and environmental
decisions.
PRINCIPAL FINDINGS
---------------------------------------------------------- Chapter 0:4
ASOS FALLS SHORT OF KEY
SPECIFICATIONS AND
CORRECTIVE ACTIONS ARE NOT
YET COMPLETED
-------------------------------------------------------- Chapter 0:4.1
Six of the eight sensors in the ASOS system do not meet key contract
specifications for accuracy or performance. For example, the
precipitation accumulation sensor underreports rainfall amounts
during heavy downpours, and the temperature and dew point sensor
readings frequently fall short of dew point reliability requirements.
Some of these shortfalls are caused by the contractor's failure to
deliver products that meet specifications, and others result from the
failure of government-furnished equipment to meet specifications.
Also, ASOS' overall reliability during 1994 winter testing, measured
in terms of mean hours between critical system failures and errors,
was only about one-half and one-third of specified levels,
respectively. This problem did not surface until ASOS was deployed
because reliability testing prior to deployment was not performed.
This is inconsistent with good industry practice, and does not
protect the government from buying production units that fail to meet
specifications.
Further, ASOS' monitoring organization is unable to meet requirements
for isolating ASOS failures and errors and taking corrective actions
at FAA sites because an average of one-sixth of the data
communications between FAA ASOS units in the field and NWS' ASOS
monitoring organization were lost in 1994. This data communication
problem is primarily due to the current configuration of FAA's
network.
These shortfalls can result in incorrect ASOS observations or delay
needed maintenance to correct system errors or failures. Although
observers are responsible for correcting and augmenting ASOS
observations after the units are commissioned, they are no longer
responsible for taking hourly observations and may not always notice
incorrect ASOS observations. Air traffic controllers and weather
observers provided us numerous reports of incorrect ASOS observations
that were issued. For example, an airline dispatcher described a
situation in which ASOS' underreported visibilities were not caught
for 4 hours and several aircraft had to cancel their flights
needlessly. ASOS users stated that incorrect observations could risk
aviation efficiency and safety, and skew national climate research.
NWS has effectively addressed system problems in meeting
specifications in the past. Further, it is now testing the
contractor's modifications to correct current shortfalls on
contractor-developed sensors and software, and evaluating and
developing corrections to shortfalls on government-furnished
equipment. Also, NWS expects system reliability to improve once
these corrections are implemented, and it is working with FAA to
improve FAA's network configuration. However, because these
corrective actions are all ongoing, it is unknown at this point when
or whether NWS' efforts will solve the shortfalls.
Because of ASOS' problems, NWS temporarily halted commissionings in
November 1994. In March 1995, the National Oceanic and Atmospheric
Administration's (NOAA) Assistant Administrator for Weather Services
approved the resumption of commissionings on the basis of expected
improvements in the availability of spare equipment and approval of
the wind sensor modification.
NWS IS NOW ASSESSING NEEDS
THAT USERS CLAIM ASOS, AS
SPECIFIED, DOES NOT SATISFY
-------------------------------------------------------- Chapter 0:4.2
ASOS users from the aviation, meteorology, and climate communities
cite needs that the ASOS system, as specified, does not satisfy. For
example, the originally specified equipment for displaying ASOS
observations in airport towers does not satisfy FAA's requirements.
Also, professional aviation organizations state that, in order to
efficiently and safely conduct aviation operations, they need
visibility and cloud height observations that are more representative
of prevailing conditions than ASOS currently provides. Further, NWS
meteorologists identified three areas where ASOS does not provide
observations that they describe as significant to weather reporting
and that users expect. For example, they cited a need for ASOS to
identify ice pellets as solid precipitation. Additionally,
climatologists stated that they require uninterrupted, accurate
measurements of precipitation for research in local and national
environmental issues. However, ASOS studies raise concerns about
inaccurate ASOS precipitation measurements.
The ASOS program office categorizes reported user problems as either
failures of ASOS to meet specifications, or problems outside the
scope of ASOS as currently defined. In the past, they have not been
responsive to the latter. However, NWS now acknowledges that user
concerns exist that go beyond the ASOS specification, and it has
recently begun to identify and evaluate these problems.
COSTS TO RESOLVE FULL RANGE
OF ASOS PROBLEMS ARE UNKNOWN
-------------------------------------------------------- Chapter 0:4.3
Because NWS has not yet determined how it will enhance or supplement
ASOS to address needs that users say are important but are beyond the
scope of the ASOS specification, it does not know what it will cost
to do so. Without a complete estimate of ASOS' cost, NWS cannot
reliably assess whether enhancing and supplementing ASOS will
continue to be a prudent, cost-beneficial course of action.
PLANS FOR RELYING ON ASOS
LACK EFFECTIVE USER
INVOLVEMENT
-------------------------------------------------------- Chapter 0:4.4
Because of ASOS' problems, NWS has delayed plans for releasing human
weather observers. However, its strategy for defining when it can
begin removing observers does not adequately involve all ASOS users.
Specifically, NWS' strategy calls for it to first reach agreement
with the aviation community on that weather information that is
essential for aviation operations. It then calls for a 6-month
demonstration to determine the staffing levels needed to provide
information that ASOS does not provide, and then making arrangements
for non-NWS staff to provide this information.
However, this strategy ignores unmet needs that non-aviation ASOS
users, such as climatologists, have identified as important.
Although the ASOS program manager stated that all user needs will be
met before releasing observers, there is no written plan or strategy
to accomplish this and to ensure that users are active participants
in the process. As a result, NWS may run the risk of releasing
weather service observers before ASOS can meet needs that some users
describe as important.
RECOMMENDATIONS
---------------------------------------------------------- Chapter 0:5
Given the criticality of accurate and timely weather observations,
GAO recommends that the Secretary of Commerce direct the NOAA
Assistant Administrator for Weather Services to
define and prioritize, in conjunction with ASOS users, all system
corrections, enhancements, and supplements that must occur to meet
valid user needs;
reestimate ASOS' costs in light of planned system corrections,
enhancements, and supplements;
formulate, in conjunction with ASOS users, explicit system
performance and cost/benefit criteria governing the release of human
observers; and
certify to the Secretary that (1) the cost/benefit criteria have been
satisfied before investing in ASOS corrections, enhancements, and
supplements and (2) the system performance criteria have been
satisfied before releasing human observers.
AGENCY COMMENTS
---------------------------------------------------------- Chapter 0:6
GAO received oral comments on a draft of this report from senior NOAA
and FAA officials, including the NOAA Comptroller, the NOAA Assistant
Administrator for Weather Services, the FAA Weather Sensor Systems
Manager, and the FAA Product Lead of the Terminal Products Team.
These officials generally agreed with GAO's findings, although they
added that the problems and unmet needs that GAO identified are being
addressed. They also noted that not all ASOS user needs have the
same weight and urgency.
These statements are not inconsistent with our report. However, the
point remains that the ASOS problems in meeting both specified
requirements and user needs beyond the specification's scope have yet
to be resolved. Moreover, the time and money needed to resolve these
problems and when ASOS will ultimately permit the release of human
observers remain uncertain. Until NWS addresses these uncertainties
in collaboration with ASOS users, it does not know whether ASOS
corrections, enhancements, and supplements are worth the investment.
INTRODUCTION
============================================================ Chapter 1
The National Weather Service (NWS) is a component of the National
Oceanic and Atmospheric Administration (NOAA), within the Department
of Commerce. NWS' basic mission is to provide weather and flood
warnings, public forecasts, and advisories primarily for the
protection of life and property. NWS' operations also support other
agencies' missions and the nation's commercial interests. For
example, NWS provides specialized forecasts to support aviation
safety and the agricultural and marine industries. To carry out its
mission, NWS uses a variety of systems and manual processes to
collect, process, and disseminate weather data to and among its
network of field offices and regional and national centers. Many of
these systems and processes are outdated. For example, some radar
equipment currently in use dates back to 1957. These radars
experience frequent and long-lasting outages, and some replacement
parts are no longer available.
During the 1980s, NWS initiated a program to modernize its systems
and restructure its field offices. The goals of this modernization
program are to achieve more uniform weather services across the
nation, improve forecasts, provide more reliable detection and
prediction of severe weather and flooding, permit more cost-effective
operations, and achieve higher productivity. The modernization
program includes four new major system developments: the Next
Generation Weather Radar (NEXRAD), the Next Generation Geostationary
Operational Environmental Satellite (GOES-Next), the Advanced Weather
Interactive Processing System (AWIPS), and the Automated Surface
Observing System (ASOS). It also includes upgrades to existing
systems and several smaller new system development projects. The
total cost of the modernization is estimated to exceed $4.5 billion.
We have previously reported on NWS' modernization.\1 Additionally, we
designated NWS' modernization as a high-risk information technology
initiative in our 1995 report series on high-risk programs.\2
NWS plans to restructure its field offices in association with the
modernization program. This restructuring will consolidate the
current field structure of 52 Weather Service Forecast Offices, 204
smaller Weather Service Offices, 13 River Forecast Centers, and 3
National Centers into 118 Weather Forecast Offices, 13 River Forecast
Offices,\3
13 Data Collection Offices, and 9 National Centers.
Three other agencies are participating with NWS in the modernization.
The Federal Aviation Administration (FAA) and the Department of
Defense (DOD) are major users of weather data and have formed
tri-agency programs with NWS to manage and fund the development and
deployment of both NEXRAD and ASOS. Similarly, the National
Aeronautics and Space Administration (NASA) is managing the
development and procurement of GOES-Next.
Because of the importance of timely and accurate weather forecasts
and warnings to our nation's commerce and its citizens' safety, the
Congress has been keenly interested in the modernization,
particularly in preserving the existing quality of weather service.
As a result, the Congress passed Public Law 100-685 in 1988, and
Public Law 102-567, Title VII, in 1992. This legislation states that
the Secretary of Commerce shall not close, consolidate, automate, or
relocate any field office until the Secretary certifies to the
Congress that such action will not result in any degradation of
weather service to the public.
--------------------
\1 Weather Service Modernization: Despite Progress, Significant
Problems and Risks Remain (GAO/T-AIMD-95-87, Feb. 21, 1995); Weather
Forecasting: Improvements Needed in Laboratory Software Development
Processes (GAO/AIMD-95-24, Dec. 14, 1994); Weather Forecasting:
Systems Architecture Needed for National Weather Service
Modernization (GAO/AIMD-94-28, Mar. 11, 1994); Weather Forecasting:
Important Issues on Automated Weather Processing System Need
Resolution (GAO/IMTEC-93-12BR, Jan. 6, 1993).
\2 High-Risk Series: An Overview (GAO/HR-95-1, Feb. 1995).
\3 The 13 River Forecast Offices are to be co-located with Weather
Forecast Offices.
ASOS: A BRIEF DESCRIPTION
---------------------------------------------------------- Chapter 1:1
ASOS is a system of sensors, computers, display units, and
communications equipment intended to automate the ground-based
observation and dissemination of weather information nationwide.
This weather information includes data on temperature and dew point,
visibility, wind speed and direction, pressure, cloud height and
amount, and types and amounts of precipitation. Figure 1.1 depicts
the specified configuration of ASOS sensors and describes their
functions.
Figure 1.1: Specified
Configuration of ASOS Sensors
(See figure in printed
edition.)
The precipitation accumulation
sensor measures the amount of
liquid precipitation.
The temperature and dew point
sensor measures the temperature
and dew point.
The precipitation
identification sensor detects
type and intensity of rain and
snow.
The data collection package
collects sensor data and sends
data to a central processing
unit.
The wind sensor measures wind
speed and direction.
The ceilometer measures cloud
height and amount.
The freezing rain sensor
detects freezing precipitation.
(This sensor is not part of the
initial ASOS deployment.)
The visibility sensor measures
visibility.
Though not pictured above, the
pressure sensor determines
sea-level pressure and
altimeter settings. It is
housed indoors.
(See figure in printed
edition.)
According to NWS, ASOS offers considerable advantages over the
current manual surface observation practices, including uniform
measurements across all sites and times, continuous observation and
reporting, and more observing sites nationwide. ASOS is also
intended to eventually replace human observers at weather service
offices and at airports. However, ASOS is not designed to detect all
weather conditions that human observers have historically provided.
Weather conditions that ASOS does not observe include thunderstorms,
tornadoes, volcanic ash, hail, virga,\4 snow amount and depth, cloud
layers above 12,000 feet, ice pellets, drizzle, blowing snow, blowing
dust, and blowing sand. NWS plans to continue to provide this
information to its users through a combination of remote sensing
systems (e.g., radar, satellite, and lightning detection system),
additional ASOS sensors, and manual observation by non weather
service observers, such as volunteers, on-call cooperators, and/or
FAA personnel.
--------------------
\4 Virga is rain that falls high in the atmosphere and does not reach
the ground.
ASOS STATUS AND PLANS
-------------------------------------------------------- Chapter 1:1.1
NOAA, as the lead agency on the ASOS program, has contracted with AAI
Corporation to purchase a minimum of 868 units, with an option to buy
an additional 832 units. Through the base contract, NWS, FAA, and
DOD plan to procure 245, 537, and 86 ASOS units, respectively. As of
December 1994, NWS had purchased 617 units for the three agencies.
Of these, 491 have been accepted, and 47 have been commissioned. NWS
commissions a system when it provides the official weather
observation. However, human observers still augment ASOS
observations at those sites that have historically used human
observers. In these cases, the human observers continue to deliver
some of the information ASOS cannot provide (such as thunderstorms
and tornadoes), and provide correct observations should ASOS err or
fail. (Table 1.1 provides more information on the status and number
of ASOS units for each agency).
Table 1.1
Status of ASOS Units by Agency as of
December 2, 1994
Units Units
purchase Units commission
Agency d accepted ed
---------------------------- -------- -------- ----------
NWS 232 159 45
FAA 352 312 2
DOD/Navy 33 20 Not
applicable
\a\
============================================================
Total 617 491 47
------------------------------------------------------------
\a NWS' commissioning process does not apply to Navy ASOS units.
NWS plans to purchase and install all 868 units by fiscal year 1997.
In June 1993, NOAA estimated that developing and deploying the base
ASOS program through fiscal year 1996 for all three agencies would
cost about $340 million. NWS now estimates that the base program
will cost $351 million through fiscal year 1997 due to FAA changes to
the program. The three agencies spent roughly $235 million to
develop and deploy ASOS through fiscal year 1994, and plan to spend
about $116 million to procure and deploy the remaining ASOS units in
the base program through fiscal year 1997.
ASOS IS INTENDED TO REPLACE
HUMAN OBSERVERS
-------------------------------------------------------- Chapter 1:1.2
Until an ASOS site is commissioned, human observers are responsible
for providing official NWS surface weather observations. These
observations are used by meteorologists, hydrologists,
climatologists, and the aviation community to provide weather
forecasts, research climate changes, and conduct aviation operations.
Within the aviation community, these observations are necessary for
aircraft to take off, land, or be dispatched to an airport. As part
of its modernization and associated restructuring, NWS expects that
ASOS will assume most surface observing responsibilities and replace
human observers at about 175 weather offices and airports. In
addition, according to the FAA manager responsible for defining air
traffic requirements for weather sensors, FAA intends for ASOS to
replace its contract observers at about 194 sites. To date, neither
FAA nor NWS has replaced any of their human weather observers.
ASOS MANAGEMENT STRUCTURE
-------------------------------------------------------- Chapter 1:1.3
Three organizations play key roles in the ASOS program. NOAA's
System Acquisition Office is responsible for the ASOS acquisition.
NWS' ASOS program office manages the overall program. Its
responsibilities include overseeing system development and
deployment, coordinating the program with FAA and the Navy, and
investigating system enhancements. Finally, NWS' Office of Systems
Operations (OSO) is responsible for operating and maintaining ASOS
systems. Within OSO, ASOS Operations and Monitoring Center (AOMC) is
responsible for on-line monitoring of the quality of ASOS
observations and dispatching maintenance technicians for operational
ASOS units.
OBJECTIVES, SCOPE, AND
METHODOLOGY
---------------------------------------------------------- Chapter 1:2
The objectives of our review were to determine (1) what ASOS problems
exist and how effectively NWS is resolving them, (2) the cost of
resolving these problems, and (3) whether NWS' plans for implementing
ASOS make sense in light of these problems.
To determine what ASOS problems exist, we
obtained and analyzed (1) the ASOS program office's problem report
database, (2) summaries of issues compiled by an NWS task team
charged with resolving ASOS problems, (3) action plans on problematic
sensors developed by this NWS task team, (4) NWS' system performance
reports, (5) relevant FAA Unsatisfactory Condition Reports, (6)
relevant NASA Aviation Safety Reporting System reports, (7) ASOS
discrepancy reports from two weather service offices and three air
traffic control towers that representatives from the National Air
Traffic Controllers Association (NATCA) and the National Weather
Service Employees Organization (NWSEO) recommended we contact, and
(8) ad hoc problem reports and letters from weather observers, air
traffic controllers, and air dispatchers;
discussed meteorologists' requirements and system performance with
and obtained related documentation from NWS' Test and Evaluation
Branch in Sterling, Virginia; NWS' Office of Meteorology; 14 NWS
weather service offices across the country; and the NWSEO;
discussed climatologists' requirements and system performance with
and obtained related documentation from NOAA's National Climate Data
Center, NWS' Climate Data Continuity Project, the Colorado State
University Department of Atmospheric Science, and the ASOS Climate
Working Group; and
discussed aviation requirements and system performance with and/or
obtained related documentation from the National Research Council,
FAA's Air Traffic Plans and Requirements Service, NATCA, the Air
Transport Association, the Airline Dispatchers Federation, and 13
individual air traffic controllers from airport towers referred to us
by NATCA.
To determine the effectiveness of NWS' problem resolution efforts, we
reviewed policies and procedures and interviewed program officials to
document NWS' processes for identifying and resolving ASOS problems,
obtained and assessed documentation on problem resolution efforts and
discussed these efforts with NWS and FAA program officials, and
assessed a random sample of NWS' database of 499 problem reports to
identify (1) how these reports were tracked and their resolution
efforts documented and (2) how satisfied the originators of the
problem reports were with NWS' resolution efforts.\5
To determine the cost of resolving ASOS problems, we requested cost
estimates for resolving known problems. We then reviewed available
documentation on the costs to fix these problems and analyzed the
current ASOS project cost estimate to determine what costs were and
were not included.
Last, to determine whether plans for implementing ASOS make sense in
light of these problems, we discussed deployment and commissioning
plans and NWS' analysis of alternative deployment plans with program
officials, and we requested copies of any analyses performed.
Additionally, we compared NWS' deployment plans with plans for
resolving ASOS problems for inconsistencies.
In performing our work, we used our System Assessment Framework
methodology for guidance on assessing operational systems. We also
used our research on the best information management and technology
practices of leading organizations as published in our report,
Executive Guide: Improving Mission Performance Through Strategic
Information Management and Technology, (GAO/AIMD-94-115, May 1994).
We performed our work at the Department of Commerce in Washington,
D.C.; NOAA and NWS headquarters in Silver Spring, Maryland; NOAA's
Sterling Research and Development Center in Sterling, Virginia; NWS
weather offices in Colorado Springs and Denver, Colorado; FAA
headquarters in Washington, D.C.; FAA's air traffic control tower at
the Colorado Springs Airport in Colorado Springs, Colorado; Colorado
State University in Fort Collins, Colorado; the National Climate Data
Center in Asheville, North Carolina; and the Air Transport
Association, National Weather Service Employees Organization, and
National Air Traffic Controllers Association headquarters in
Washington, D.C.
Our work was performed between October 1993 and March 1995, in
accordance with generally accepted government auditing standards. We
requested written comments from the Department of Commerce and the
Department of Transportation, but none were provided within the
allotted response period. However, we obtained oral comments from
senior NOAA and FAA officials, including the NOAA Comptroller, the
NOAA Assistant Administrator for Weather Services, the FAA Weather
Sensor Systems Manager, and the FAA Product Lead of the Terminal
Products Team. Their comments are presented and addressed in chapter
6.
--------------------
\5 We utilized a random number generator to select 50 problem
reports, a sufficient sample size for making statistically
significant observations about the population with a 95-percent
confidence level.
ASOS FALLS SHORT OF SPECIFIED
REQUIREMENTS AND CORRECTIVE
ACTIONS NOT YET COMPLETED
============================================================ Chapter 2
ASOS units are not meeting all specified requirements. NWS
recognizes this and has actions under way or planned to correct the
shortfalls. Whether or not these actions prove successful, however,
remains to be seen. Until then, fielded ASOS units will provide some
inaccurate or incomplete weather observations. If not caught and
corrected by weather observers, these observations could adversely
impact aviation operations, climate research, and the general public.
Despite ASOS' inability to fully meet specified requirements, NWS
chose to accept, deploy, and rely on the system because testing prior
to the decisions did not disclose the problems. To NWS' credit, it
has historically been effective at eventually resolving ASOS
shortfalls in meeting specifications.
ASOS CURRENTLY DOES NOT MEET
SOME IMPORTANT SPECIFIED
REQUIREMENTS, BUT STEPS ARE
UNDER WAY TO ADDRESS SHORTFALLS
---------------------------------------------------------- Chapter 2:1
Specifications define the minimum system functions and performance
levels needed to satisfy user needs. As such, they provide the
baseline against which to gauge how successful a system developer is
in providing promised capabilities. Failure to satisfy
specifications means that mission needs are not fulfilled unless
system shortfalls are corrected or work-arounds are implemented.
Meeting specified requirements is thus critical to the success of any
system.
While ASOS meets many specifications, it currently does not meet some
specified requirements on most ASOS sensors, requirements for overall
system reliability, and requirements for performance monitoring of
FAA ASOS units. In some cases, this is because the contractor failed
to provide the promised capability, while in other cases, it is
because ASOS components provided to the contractor as
government-furnished equipment do not perform as required, thus
preventing the contractor from delivering what was promised. ASOS
program officials told us that most of these shortfalls did not
surface during acceptance testing, and with the exception of the
freezing rain sensor discussed below, have only been brought to light
as a result of ASOS' extended operational use during harsh weather.
NWS and the ASOS contractor are taking steps to address most of these
current shortfalls. In fact, ASOS units now being produced include
sensor upgrades to correct some problems. However, these upgrades do
not address all known problems, and earlier deployed units have not
been retrofitted with these upgrades. As a result, ASOS systems that
are currently operational provide some inaccurate or incomplete
weather observations. Unless caught and corrected by weather
observers, these weather reports are provided to the aviation
community, climatologists, hydrologists, meteorologists, and the
general public. This could adversely impact aviation safety,
environmental policy decision-making, or even an individual's
preparation for environmental conditions.
Because of ASOS' problems, the NOAA Assistant Administrator for
Weather Services decided in November 1994 to temporarily halt
commissioning additional units. NWS officials identified three areas
that needed to be addressed in order to resume commissionings. These
included (1) installing the improved wind sensors, (2) correcting the
precipitation accumulation sensor problems, and (3) improving the
availability of ASOS spare parts. Also in November 1994, FAA agreed
with its union to halt ASOS commissionings at FAA-sponsored towered
airport sites for 7 months, during which time both organizations
would evaluate ASOS' suitability for air traffic operations. This
evaluation is ongoing.
In March 1995, NOAA's Assistant Administrator for Weather Services
approved the resumption of commissionings beginning in April 1995.
This decision was based on NWS' satisfaction of two of the three
criteria. Specifically, the wind sensor modification was approved
for implementation and an adequate supply of spare parts is expected
to be delivered in April 1995. The precipitation accumulation sensor
has not yet been approved for implementation.
MOST ASOS SENSORS DO NOT
MEET SPECIFICATIONS
-------------------------------------------------------- Chapter 2:1.1
Of ASOS' eight sensors, four that were developed by the contractor
and two that were provided in limited quantities to the contractor as
government-furnished equipment do not meet contract specifications.
According to program office reports, two other contractor-developed
sensors (the visibility and pressure sensors) meet specifications.
Thus, promised capabilities have not been delivered on six of eight
ASOS sensors.
The contractor-developed sensors that do not meet specifications are
the wind, precipitation identification, precipitation accumulation,
and freezing rain sensors. Specifically, (1) the wind sensor falls
short of specifications for reporting wind speed and direction
accurately, and operating during freezing conditions, (2) the
precipitation accumulation sensor does not meet accuracy
specifications for liquid precipitation amounts, (3) the
precipitation identification sensor does not meet specifications for
operating during heavy snow and icing conditions, and (4) the
freezing rain sensor failed acceptance tests and thus has not yet
been accepted by the program office or deployed with the baseline
ASOS configuration. Program officials stated that, with the
exception of the freezing rain sensor, these problems were not
discovered during system acceptance testing.
NWS officials acknowledged that all but one of these sensors do not
yet meet specified requirements. However, they stated that the
precipitation identification sensor's problem is not a failure to
meet the specification. We disagree. We found that the
problem--snow blockage--is identified in NWS' trouble report system
as a contract deficiency, meaning that any sensor modifications to
address this problem are the contractor's responsibility.
The government-provided sensors that do not meet specifications are
the temperature and dew point sensor and the ceilometer. The
temperature and dew point sensor does not meet reliability
requirements, and the ceilometer fails more frequently than the
specification permits. The shortfalls of each of these six sensors
in meeting specifications and NWS' resolution plans are discussed in
detail in appendix I.
Inaccurate reports, according to a representative for the Airline
Dispatchers Federation, could lead a pilot to approach an airport
during underreported wind gusts, incurring unexpected safety risks.
Similarly, a University of Maryland climatologist stated that
inaccurate precipitation reports could lead researchers to
incorrectly evaluate flood and drought frequencies, intensities, and
durations, and could result in ineffectively designed irrigation
systems, dams, and other structures. Finally, a senior NWS
meteorologist stated that inaccurate dew point observations could
contribute to a forecast for clear weather, when in fact a hazardous
thunderstorm is brewing.
NWS has steps under way to address most of these sensor shortfalls,
but these actions are not yet complete. The program office, which is
responsible for the units being produced, has had the contractor
develop modifications to resolve shortfalls with the wind sensor, the
precipitation accumulation sensor, and the precipitation
identification sensor; has tested the modifications; and has
incorporated these modifications in ASOS units now being produced.
NWS' Office of System Operations (OSO), which is responsible for the
units once they are commissioned and which did not participate in the
program office testing, is testing the modifications to these three
sensors in an operational setting before approving them for
implementation on commissioned systems. In February 1995, OSO
approved the wind sensor modification for implementation on
commissioned systems, based on preliminary testing. However, OSO is
continuing to test the modification.
Additionally, the program office is currently testing modifications
to the freezing rain sensor and ceilometer and evaluating how best to
improve the wind direction problem, excessive ceilometer failures,
and the dew point sensor reliability problems.
Until NWS completes actions to resolve ASOS shortfalls, their success
cannot be ascertained. In the interim, ASOS will not provide its
full complement of promised capabilities and will sometimes produce
inaccurate or incomplete weather reports.
ASOS DOES NOT MEET OVERALL
RELIABILITY SPECIFICATIONS
-------------------------------------------------------- Chapter 2:1.2
The ASOS specification calls for mean time between system critical
failures\1 and system critical errors\2 to be 2,190 and 8,760 hours,
respectively. NWS system reliability demonstrations during winter
months in 1993 and 1994 indicated that ASOS was not meeting these
requirements. The 1994 results show mean time between critical
failures to be 1,200 hours (versus 2,190) and mean time between
critical errors to be 2,800 hours (versus 8,760). Furthermore,
critical failures and errors that were caught and corrected by
weather observers were not included in these results. Once these
errors are included, ASOS failure and error rates are 730 and 1,680
hours, respectively (see figure 2.1).
The ASOS program manager said that the January 1994 results are
outdated, and that more recent performance data show that ASOS meets
specifications with a mean time between failures on aviation-critical
observations of 3,055 hours (versus the 2,190 hours requirement).
This claim, however, is not supported by the facts. The data upon
which the claim is based are incomplete and insufficient to calculate
ASOS reliability. The specification defines system critical failures
as the loss of any of the following observations: pressure, sky
condition, visibility, wind, freezing rain, and precipitation. NWS'
recent performance data, however, do not include measurements taken
during the harshest winter weather when ASOS reliability is poorest.
Further, the data do not include failures by the precipitation
identification and precipitation accumulation sensors. Therefore,
these data alone are not indicative of ASOS' overall reliability.
Figure 2.1: ASOS Reliability
Demonstration Results
(See figure in printed
edition.)
The reason that ASOS' reliability problems were not discovered during
testing and corrected prior to system deployment and operation is
that ASOS program management repeatedly chose to defer testing of
mean time between failures. Instead, the program office relied on
the results of a model run by the contractor to predict system
reliability, rather than testing reliability. After production, NWS'
testing branch finally performed a reliability test, but these tests
did not include mean time between failures. Instead, it was decided
that reliability could be better determined by analyzing actual
maintenance actions on operational systems than by testing the
preproduction system. While we agree that continuous monitoring of
maintenance is important, it does not substitute for predeployment
testing of performance specifications as practiced by leading
information technology organizations. Further, it does not protect
the government from buying production units that do not meet
specifications for reliability.
Low system reliability increases the likelihood that wrong
information will be disseminated to system users, again potentially
risking aviation safety, impairing environmental research, and
misguiding general public decision-making. Low reliability also
results in a larger maintenance workload, lower user confidence in
ASOS, and continued dependence on human observers for quality
control. For example, one commissioned ASOS site reported that on
occasion the ceilometer took 5 days to fix. During that time, a
human observer had to take and disseminate all cloud height
observations in order for airport operations to continue.
A senior AOMC manager stated that the planned modifications on
individual sensors should improve ASOS' overall reliability. While
overall system reliability should improve as individual sensor
improvements are implemented, there is no assurance at this time that
the required reliability will be attained.
--------------------
\1 System critical failures are defined as the loss of any ASOS
observation that NWS defines as critical (pressure, sky condition,
visibility, wind, freezing rain, and precipitation). Because the
freezing rain sensor was not deployed with the rest of the ASOS
sensors, its performance was not included in these reliability
results.
\2 System critical errors involve the erroneous reporting of any
critical observation.
REQUIREMENTS FOR LONG
DISTANCE QUALITY MONITORING
OF FAA ASOS UNITS NOT BEING
MET
-------------------------------------------------------- Chapter 2:1.3
Successfully communicating information between an ASOS site and the
AOMC is critical to effective operation and maintenance of ASOS.
AOMC monitors all ASOS sites for system-detected errors or failures,
and dispatches technicians to fix problems. It is required to
provide uninterrupted monitoring of ASOS observations 24 hours a day,
365 days per year. However, AOMC has not been able to provide this
level of uninterrupted monitoring on FAA systems because an average
of 16 percent of all observations transmitted from ASOS sites to the
AOMC over FAA's wide area network were lost in 1994.\3 Comparatively,
only about 1 percent of observations transmitted from ASOS sites to
the AOMC over NWS' network are lost. Although NWS has no documented
quantitative criteria, the ASOS program manager stated that an
acceptable percentage of lost messages would be less than 5 percent.
NWS has refused to commission individual FAA sites until
communications improve. The FAA ASOS program manager noted that the
16-percent data loss is high because it is an average of new and old
ASOS sites. He stated that data communications at new sites are more
problematic and therefore raise the average.
To monitor FAA ASOS sites, the AOMC must use FAA's wide area network.
NWS and FAA officials agree that a major reason for lost data is that
the FAA network cannot support ASOS monitoring requirements in some
locations. This network is configured such that a microwave signal
must travel between as many as 30 points. This number of "hops" in
the microwave transmission causes signal degradation or loss. FAA
and NWS are working together to improve FAA communications through
changes in the network configuration.
Because of this communication shortfall, AOMC cannot adequately
monitor or respond to all ASOS problems at FAA sites. For example,
an FAA ASOS site could be inoperative or provide inaccurate
observations, and no maintenance technician would be dispatched, or
maintenance technicians would be dispatched when none are needed.
AOMC instituted a "work-around" in which it would have FAA
troubleshoot the communications line when an hourly observation is
missing, and then contact a maintenance person only if the next
hourly observation is missing as well.
--------------------
\3 Only 3 of the 31 ASOS sites transmitting data over these
communication lines are commissioned. Two are the commissioned FAA
sites, and the third is a commissioned NWS site that is using FAA
communication lines. These commissioned sites averaged a loss of
about 7 percent of the ASOS observations in 1994.
NWS HAS HISTORICALLY BEEN
EFFECTIVE IN RESOLVING ASOS
SHORTFALLS IN MEETING
SPECIFICATIONS
---------------------------------------------------------- Chapter 2:2
Successfully operating and maintaining a system requires an effective
process for reporting, tracking, and resolving instances where the
system does not meet specified requirements. Such a process includes
providing users with a vehicle for easily and quickly documenting and
transmitting their concerns to a control point that (1) centrally
records all problems, (2) controls disposition status of these
problems, and (3) provides feedback to the problem originator on
disposition actions. This process also includes active participation
by the system program office, contractor, and change control board in
evaluating the nature of the problem and deciding on corrective
actions, if any.
The NWS program office has an effective process for reporting ASOS
problems and resolving those problems it finds to be shortfalls in
meeting specifications. It provides users with a standard form,
commonly called a trouble report, for them to easily and quickly
document their concerns and transmit them to the program office. The
program office centrally records these trouble reports, tracks their
disposition status, and provides feedback to the problem originator
on disposition actions. Trouble reports are evaluated by the program
office, contractor, and change control board to determine if the
problem results from (1) a shortfall in meeting specifications, (2) a
defective system component in need of maintenance, or (3) a
misunderstanding of the system's functions highlighting an issue that
is beyond the specification. Trouble reports are closed when the
contractor fixes the shortfall, the needed maintenance is provided,
or the program office responds to the trouble report originator
explaining that the problem identified did not constitute a
specification shortfall.
Through its trouble reporting process, the ASOS program office has
established a good record in resolving ASOS problems that it
determined to be shortfalls in meeting specifications. Between July
1991 and December 23, 1994, the program office received a total of
575 trouble reports and, as of December 1994, had closed 470 of
these. Of these 470, the program office reported that shortfalls in
meeting the specification were fixed in 224 cases, the program office
provided needed maintenance in 22 cases, and the program office
responded to the trouble report originator that the problem
identified did not constitute a contract shortfall on 148 cases. The
remaining 76 cases included duplications of prior trouble reports and
reports that the originator withdrew.
Early in our review, we sampled the database of trouble reports to
determine NWS' effectiveness in resolving user reports of ASOS
failing to meet specified requirements.\4 On the basis of our sample
data, we project that about 96 percent of the full population of
reported problems in meeting the ASOS specification are being
rectified to the satisfaction of their originators. This, however,
does not address user satisfaction with NWS resolution of ASOS
problems deemed to be beyond the scope of the specification. This
question is addressed in chapter 3 of this report.
--------------------
\4 As of March 16, 1994, when we drew our sample, the number of
trouble reports that NWS had received totaled 499, as compared to the
575 reports as of December 1994.
NWS BEGINNING TO ASSESS NEEDS THAT
USERS HAVE LONG CLAIMED ASOS, AS
SPECIFIED, DOES NOT SATISFY
============================================================ Chapter 3
ASOS user groups have repeatedly cited needs that the system, as
specified, does not meet. While NWS has recently started assessing
these user concerns and plans to decide whether or not to enhance or
supplement the system to address them, NWS' track record in
effectively resolving ASOS reported problems that it deems to be
outside the scope of the specification is poor. Until it either
enhances ASOS or otherwise supplements ASOS to address user concerns,
NWS is giving its users a system that falls short of their
expectations, cannot by itself replace human observers, and,
according to these users, could result in poor weather-related
decision-making.
NWS HAS YET TO DECIDE HOW TO
ADDRESS USER CONCERNS THAT
ASOS, AS SPECIFIED, DOES NOT
SATISFY
---------------------------------------------------------- Chapter 3:1
In our 1994 report on how leading organizations improved mission
performance through strategic management of information and
technology, we reported that the success and value of a system is
largely measured by the extent to which it meets users' needs.\1
The primary users of ASOS are the aviation, meteorology, climatology,
and hydrology communities. Each relies on weather observations that
have long been provided by human observers. According to these
users, however, ASOS as specified does not fully substitute for the
human observer and, therefore, does not meet their needs. We did not
attempt to determine the validity and criticality of these unmet
needs. However, ASOS users stated that these needs are important and
not meeting them could cause adverse consequences.
NWS is now identifying, evaluating, and, in some cases, resolving
ASOS users' unmet needs. It is holding workshops with FAA and
aviation organizations to further define requirements, and it is
documenting climatologists' concerns. Additionally, because the ASOS
display equipment originally specified in the ASOS contract for use
in airport towers did not satisfy air traffic controllers' needs, NWS
has modified the specification and the contractor is now developing
new display equipment.
NWS officials stated that not all user needs can be allocated equal
weight and urgency. NWS' top priorities are life and safety issues,
such as supporting aviation operations and providing public warning
and forecast services. However, NWS officials also stated that some
user communities, such as climatological information users, could
benefit from additional observations and data and that NOAA will work
to establish requirements and formulate and justify cost-effective
means of meeting them. The ASOS program, for example, is to explore
development of sensors to obtain frequently requested data that ASOS
does not now provide. Additionally, the National Research Council is
proposing a new committee to assess the costs and benefits of climate
data needs and recommend strategies for fulfilling these needs.
Using these inputs, NWS plans to define a comprehensive set of
climatological requirements during the next 2 years.
NWS officials also noted that the ASOS program has suffered from
changing FAA requirements. For example, they stated that FAA's
requirement for tower display equipment has changed significantly and
often, resulting in several previous tower display designs.
Additionally, they stated that FAA's operational availability
requirements have changed and have yet to be adequately defined.
--------------------
\1 Executive Guide: Improving Mission Performance Through Strategic
Information Management and Technology (GAO/AIMD-94-115, May 1994).
AVIATION COMMUNITY USERS
STATE THAT ASOS DOES NOT
SATISFY NEED FOR
REPRESENTATIVE OBSERVATIONS
-------------------------------------------------------- Chapter 3:1.1
Representative observations are weather observations that accurately
describe the actual, prevailing conditions for the airport area.
Aviation users stated that it is critical that visibility and cloud
ceiling observations be representative. According to officials from
NATCA, the Airline Dispatchers Federation (ADF), and the Air
Transport Association (ATA), if visibility and ceiling observations
are not representative, inappropriate flight rules can be applied,
causing inefficient use of airspace or placing unnecessary
constraints or risks on aircraft operations. Specifically, when
reported observations are much worse than actual weather conditions,
pilots may unnecessarily avoid the airport, or be required to
undertake a more complex and costly instrument-guided approach.
Alternatively, when reported observations are much better than actual
weather conditions, pilots could decide to use an airport when they
otherwise would not.
NATCA reported in October 1994 that ASOS readings are
unrepresentative when the weather is changing rapidly or conditions
are patchy. This has led to situations, for example, in which
aircraft were routed for an instrument-guided approach because ASOS
was reporting low visibility, but the pilot could see the runways on
approach from miles away. Also, one controller described an incident
in which several small aircraft were allowed to take off based on the
ASOS report of 2,500-foot cloud ceilings. A pilot radioed in from
the air to say that ceilings were actually at 900 feet. According to
the controller, this variance could put small aircraft at risk.
The ASOS specification does not require that its observations be
representative of actual, prevailing weather conditions. In fact,
the specification never mentions representativeness. Instead, the
specification requires ASOS to perform a time average of weather
within a small area (i.e., a sample area) rather than a spatial
average as a human would when scanning the entire sky. Thus, when
weather is changing rapidly or is patchy, ASOS observations can be
accurate for the small area it is viewing, but unrepresentative of
prevailing weather conditions in the universe being sampled.
According to NWS and FAA program managers, ASOS is not, and never
was, intended to replicate the way a human observes weather
conditions, and that the aviation community must undergo a "paradigm
shift" to accept automated observations.
Neither NWS nor FAA has reliable data on the extent to which ASOS
observations are unrepresentative or the effect of unrepresentative
observations on aircraft operations. A senior NWS manager stated
that such data have not been systematically collected and analyzed
because to date both NWS and FAA management have focused on
correcting ASOS where it does not meet specifications rather than
enhancing the system to meet users' needs. Some limited data exist,
which although not conclusive, support the users' statements that
unrepresentativeness is a problem. For example, a New York weather
office recorded an average of 51 unrepresentative events per month
from November 1993 to February 1994 and an average of 32 events per
month from March to May 1994. These data are discussed in appendix
II.
The ASOS program office is drafting a plan for systematically
collecting and assessing data on unrepresentative ASOS observations,
after which it will decide whether or not to enhance or supplement
the system to address this concern. At the same time, NWS is also
collecting and assessing other aviation requirements. Through a
series of workshops with FAA, NATCA, and aviation industry officials,
NWS is identifying these users' minimum service requirements and
evaluating plans for enhancing and supplementing ASOS to satisfy any
unmet needs.
METEOROLOGISTS CITE SEVERAL
NEEDS BEYOND THE ASOS
SPECIFICATION THAT ARE NOT
BEING MET
-------------------------------------------------------- Chapter 3:1.2
NWS meteorologists at the Sterling Research and Development Center
identified three areas in which ASOS does not provide observations
that they describe as significant to weather reporting and that human
observers currently provide. These areas are (1) identifying ice
pellets as solid precipitation, (2) correctly identifying rain, snow,
or no precipitation more frequently than currently specified in ASOS'
specification, and (3) distinguishing between actual cloud layers and
invisible layers of particles in the atmosphere. According to ASOS
users, because this information is particularly important to aviation
forecasts, incorrect observations could risk aviation safety,
inconvenience airport operations, and add to aviation costs. NWS is
planning to invest resources in ASOS enhancements to address two of
these three unmet needs. Each unmet need is described below.
The ASOS specification states that solid precipitation shall be
correctly detected at least 99 percent of the time (reported as
either snow or mixed/other precipitation) and shall be correctly
identified at least 97 percent of the time. According to the ASOS
Technical Chief, solid precipitation should mean only snow. It
should not mean other forms of solid precipitation, such as ice
pellets.
ASOS satisfies the program office's narrow definition of solid
precipitation. However, ASOS tests and operational experience show
that the system misidentifies solid precipitation other than snow
(e.g., it often identifies ice pellets as rain). According to ASOS
users, failure to correctly identify other types of solid
precipitation could pose a safety problem should the system report
rain in ice pellets, hail, or icing conditions, and the problem is
exacerbated at unstaffed ASOS sites where human observers would not
be present to correct erroneous reports. The error occurs because
the sensor identifies precipitation types on the basis of their size
and velocity. The ASOS program office is evaluating alternative
technological enhancements that could permit discriminating ice
pellets and hail from other forms of solid precipitation.
The ASOS specification requires that the precipitation identification
sensor correctly detect precipitation (liquid and solid) at least 99
percent of the time, and correctly identify at least 90 percent of
liquid precipitation and 97 percent of solid precipitation.
According to the ASOS program office, ASOS meets this specification.
However, NWS' Office of Systems Operations, which is responsible for
the operation and maintenance of commissioned units, reported that
current performance levels may not satisfy aviation operational
requirements. As stated by a senior testing manager, 1 percent of 24
hours of precipitation is about 15 minutes, and if ASOS does not
correctly detect precipitation for 15 minutes while an aircraft is
approaching, a significant safety risk could be incurred. The
significance of the situation was underscored in the workshops
involving NWS, FAA, and aviation industry officials to define
aviation industry requirements for weather information. During these
workshops, industry officials stressed the importance of having more
accurate and available precipitation information. To address this
concern, the ASOS program office is assessing technical enhancements
to improve this sensor's accuracy.
ASOS users cite requirements for accurately identifying cloud
ceilings. Because of technological limitations, ceilometers alone
(i.e., without human observers) cannot currently provide this. ASOS
ceilometers falsely identify weather phenomena such as invisible
particles, moist air layers, virga, and precipitation as cloud
layers. This can result in observations of broken or overcast
ceilings when in fact there are clear or scattered conditions.
Broken and overcast conditions are defined by the Federal Aviation
Regulations as a ceiling, and ceilings determine legal limits and
flight rules for various aircraft operations.
NWS managers stated that this problem does not risk aviation safety;
rather, it inconveniences the aviation community. Moreover, they
stated that it occurs too infrequently to be a concern. In contrast,
aviation industry officials stated that even infrequent occurrences
can be costly to airlines. Despite this, program officials stated
that they have no plans for researching and developing new technology
to address the concern.
CLIMATOLOGISTS CONTEND THAT
ASOS IS NOT PROVIDING NEEDED
PRECIPITATION OBSERVATIONS
-------------------------------------------------------- Chapter 3:1.3
Officials from the National Climate Data Center, the American
Association of State Climatologists, and the ASOS Climate Working
Group, as well as various prominent climate researchers from industry
and academia, stated that climatologists need uninterrupted and
accurate measurement of precipitation. However, ASOS does not meet
this need. In February 1994, NOAA's Climate Data Continuity
Project\2 reported that ASOS' precipitation accumulation sensor
generally reports less precipitation than conventional sensors, with
the greatest underreporting occurring in summer and winter months.
The report explained that ASOS' gauge performed reasonably well in
rain events that were of light to moderate intensity but did not
perform well in snow or freezing conditions or in heavy rain events
that are typical in the summer. The project further reported that,
from the summer of 1993 through the spring of 1994, ASOS
precipitation observations were 94 and 91 percent of conventional
observations in fall and spring, respectively, but fell to 86 percent
in summer and only 75 percent in winter (see figure 3.1).
Climatologists stated that incorrect precipitation observations
affect not only weather-related decision-making, but also accurate
climatological analysis. Specifically, inaccurate precipitation
records could affect (1) businesses that rely on this information for
everyday decisions, (2) climatological analysis of flood and drought
frequencies, intensities, and durations, and (3) long-term national
climate research on global and regional precipitation, potentially
skewing national environmental policy decisions on global change. To
address this climatological need, NWS is developing specifications
and plans to procure an all-weather precipitation gauge to better
report liquid and frozen precipitation amounts.
Figure 3.1: ASOS Reports Less
Precipitation Than Conventional
Methods
(See figure in printed
edition.)
Note: ASOS precipitation as a percentage of conventional
precipitation based on data from 13 commissioned ASOS sites in the
central United States.
Source: NOAA's Climate Data Continuity Project.
According to the ASOS program manager, ASOS does not meet all
climatology needs because it was not originally intended to do so.
Rather, it was intended only to meet forecasting and aviation
requirements. The program office, however, is now assessing how best
to meet climatologists' requirements. It has started meeting with
climatology users to identify their unmet needs and concerns,
including how to supplement ASOS to ensure that needed observations,
which are currently provided by the human observers, continue once
the human observers are removed.
--------------------
\2 The Climate Data Continuity Project is a multiyear study of
systematic differences that exist between the climate record as
recorded by ASOS and the historical climate record as recorded by the
systems and people that ASOS is replacing. Understanding the
differences will help minimize the effects of discontinuities in the
historical climate record and help ensure the integrity of climate
data used in research.
ASOS AIRPORT TOWER DISPLAY
EQUIPMENT DOES NOT MEET
FAA'S NEEDS
-------------------------------------------------------- Chapter 3:1.4
The ASOS specification defines the display equipment needed by air
traffic controllers in airport towers. Originally, the specification
did not require that the ASOS display equipment integrate data from
other airport tower displays. However, after operational experience
in the towers with the ASOS displays, FAA raised several concerns,
including that the ASOS displays were forcing controllers to view too
many tower displays. As a result, FAA and NWS changed the ASOS
specification to require enhanced displays that could integrate data
from other tower displays with the ASOS data. Currently, FAA is
testing the new, integrated display equipment in several towers. If
testing is successful, FAA plans to accept and deploy the equipment.
An FAA official estimated that the redesigned displays will cost
between $10 and $12 million, depending on the number bought.
NWS' PROCESS FOR RESOLVING
PROBLEMS IS NOT EFFECTIVE FOR
CONCERNS DEEMED OUTSIDE THE
SCOPE OF THE SPECIFICATION
---------------------------------------------------------- Chapter 3:2
Successfully deploying, operating, and maintaining a system requires
an effective process for resolving user problems. As discussed in
chapter 2, NWS' process for reporting and resolving problems where
the system fails to meet specified requirements is effective.
However, NWS does not have an effective process for resolving user
concerns that are beyond the scope of the ASOS specification.
Internal NWS memoranda and NWS managers described the ASOS problem
resolution system as unresponsive to system users whose problems were
determined to be outside the scope of contract specifications. To
validate these statements, we randomly sampled 50 trouble reports and
found
20 reports that the program office closed because it found the
system's reported performance to be consistent with specified
requirements. The originators on 11 of these 20, however, were not
satisfied with the program office's actions. For example, one
technician reported that the size of the openings in the mesh screen
on the temperature and dew point sensor was too large, allowing too
many bugs to enter and thus requiring the sensor to be cleaned as
much as 12 times more frequently than the maintenance manual
required. The ASOS program office closed this trouble report on the
basis of the mesh screen size being compliant with the size stated in
the contract specification.
The ASOS program officials stressed that their priority has been in
getting the system to meet specifications. They described other user
requirements as being of lesser importance. ASOS program officials
also stated that the trouble report process was only intended to
handle shortfalls in specifications and that other processes should
be followed to seek changes or enhancements to the system. They
acknowledged, however, that users often are unable to distinguish
between specification shortfalls and system enhancements and thus
will often file trouble reports to surface their concerns. In cases
where the reported problem is not a specification shortfall, the user
is told to file a different report to request a system enhancement.
However, this approach places the burden on the user to resubmit a
concern already raised. Without a user-friendly problem reporting
process, valid concerns may not be surfaced, increasing the
possibility of NWS fielding a system that fails to satisfy valid user
requirements.
To improve its record in resolving all trouble reports, the ASOS
program office set up a new quality assurance working organization
that will review all trouble reports proposed for closure. The ASOS
program manager stated that this group will also be more responsive
to user concerns that are beyond the specification.
COST OF ADDRESSING ALL ASOS
REPORTED PROBLEMS IS NOT KNOWN
============================================================ Chapter 4
Because NWS is still assessing many of the problems facing ASOS, it
has yet to estimate the cost to resolve them, either via system
enhancements or human/system supplementation. Without reliable
estimates of what an enhanced or supplemented ASOS that fully meets
users' needs will ultimately cost, NWS does not know whether
continued investment in ASOS is cost beneficial.
The Federal Acquisition Regulation requires that the government
develop an estimate of the cost of any goods and services to be
acquired. Further, the Federal Information Resources Management
Regulation requires that a system's life cycle costs (i.e., the cost
to design, develop, deploy, operate, and maintain a system over its
expected useful life) be used in any analysis of the relative costs
and benefits of competing system alternatives. Cost estimates are
important considerations in all investment decisions related to the
goods and services being acquired, such as whether the benefits
expected from the goods and services outweigh their costs and thus
whether investment in these items is justified. Further, these cost
estimates should not be static. They are to be appropriately refined
and updated throughout the acquisition cycle.
NWS does not have a complete estimate of what it will cost to address
all user concerns with ASOS. The current cost estimate of $351
million includes the cost to purchase and deploy 868 ASOS units as
contractually specified. Under the ASOS firm, fixed-price contract,
the contractor is to absorb all costs to fix shortfalls in meeting
these contract specifications. However, the government is
responsible for (1) correcting any shortfalls in meeting specified
requirements that are due to deficient government-furnished sensors,
(2) improving or upgrading the system beyond its specified
requirements, and (3) supplementing the system with observations from
other sources (e.g., humans or other systems).
The cost of those problems that are due to deficient
government-furnished sensors, such as the dew point sensor and the
ceilometer, are included in the $351 million cost estimate. However,
other reported problems or concerns that are outside the scope of the
ASOS specification are only now being assessed, and thus the total
cost to address them, either by improving ASOS or supplementing it,
is unknown.
According to NWS officials, the ASOS 1995 and 1996 budgets include
$3.9 million for product improvement activities to correct current
sensor limitations and meet remaining commonly accepted requirements.
These requirements include a sunshine sensor, an all-weather
precipitation accumulation sensor, an ice-free wind sensor, a new dew
point sensor, and a sleet and hail detection sensor. While the cost
to produce the sensors is not yet known, NWS officials estimate that
production costs will not exceed $22 million for the 868 ASOS units
in the base program.
Supplemental costs associated with residual backup requirements at
major airports are not yet known because NWS, FAA, and various
aviation organizations are currently negotiating the required level
of backup. NWS officials stated that if a very high availability is
required, human observers would not be released, and thus expected
cost reductions would not be achieved. NWS officials estimated that
if they are unable to release observers, the cost of human
augmentation and backup in NWS alone will be $16 million for 336
staff years in fiscal year 1996, eventually rising to approximately
$33 million for 615 staff years per year.
PLANS FOR IMPLEMENTING ASOS LACK
EFFECTIVE USER INVOLVEMENT
============================================================ Chapter 5
Before relying on ASOS to provide NWS' official weather observations,
NWS should demonstrate that all valid user needs are being met. To
date, this has not occurred. Accordingly, NWS recently discontinued
ASOS commissionings and postponed plans for releasing its human
weather observers until ASOS problems are addressed. However, NWS
has not made plans to ensure that all users are effectively involved
in decisions on releasing human observers.
NWS expects ASOS to replace NWS and contract weather observers at
about 175 weather offices and airports, and FAA plans to release its
contract observers at 194 sites. Before releasing any observers,
however, NWS must ensure that either ASOS or some supplementary
observing source can meet the needs of ASOS' different user
communities.
While NWS plans for ensuring that ASOS meets user needs effectively
involve ASOS' aviation users, they do not ensure that other ASOS
users will be adequately involved. Specifically, NWS has postponed
plans to release observers while it (1) reaches agreement with the
aviation community as to what weather information is essential for
aviation operations, (2) determines, through a 6-month demonstration,
the workload associated with augmenting ASOS observations with
information ASOS cannot provide (e.g., tornadoes, thunderstorms,
hail, volcanic ash, and virga) and supplementing ASOS observations in
case ASOS fails or errs, and (3) arranges to have non-NWS staff
augment and supplement ASOS as needed, based on demonstration
results. These efforts, however, do not address unmet needs that
other ASOS users have identified as important.
The extent to which other ASOS users will be involved in decisions
regarding the release of human weather observers is not clear.
According to the ASOS program manager, human observers will not be
released until all users' needs are met either through ASOS or some
supplementary source. However, he acknowledged that this is not
confirmed in any documented plan or strategy for human observer
release. Further, it contradicts the explicit purpose of NWS'
demonstration for aviation users, which links human observer release
to meeting solely aviation user needs. Without a plan for releasing
human observers that includes explicit criteria reflecting the needs
of all ASOS users, NWS runs the risk of releasing observers before
adequate supplementary sources are in place.
CONCLUSIONS, RECOMMENDATIONS, AND
AGENCY COMMENTS AND OUR EVALUATION
============================================================ Chapter 6
CONCLUSIONS
---------------------------------------------------------- Chapter 6:1
ASOS is performing neither as intended nor expected. While many
specified requirements are being met and steps are underway to
correct instances where they are not, the system does not provide
certain information that human observers do and users say is
important to aviation safety, weather-related decision-making, and
climatological analysis.
Whether an enhanced ASOS can eventually provide the information users
say they need or whether ASOS will need to be supplemented by other
observing technology or people is not yet known. What is known is
that ASOS problems became severe enough for NWS and FAA to
temporarily halt commissioning additional ASOS units and to refrain
from implementing plans to release human observers until the problems
are solved. However, NWS has yet to determine what specific ASOS
problems must be solved before observers can be released, what the
solutions will cost, and whether expected ASOS benefits will be worth
the additional costs. Until NWS works collaboratively with ASOS
users in doing so, it runs the risk of deploying a system that is not
cost-effective, and that does not adequately support important public
safety and environmental decisions.
RECOMMENDATIONS
---------------------------------------------------------- Chapter 6:2
Given the criticality of accurate and timely weather observations, we
recommend that the Secretary of Commerce direct the NOAA Assistant
Administrator for Weather Services to
define and prioritize, in conjunction with ASOS' users, all system
corrections, enhancements, and supplements that must occur to meet
valid user needs;
reestimate ASOS' costs in light of planned system corrections,
enhancements, and supplements;
formulate, in conjunction with ASOS users, explicit system
performance and cost/benefit criteria governing the release of human
observers; and
certify to the Secretary that (1) the cost/benefit criteria have been
satisfied before investing in ASOS corrections, enhancements, and
supplements and (2) the system performance criteria have been
satisfied before releasing human observers.
AGENCY COMMENTS AND OUR
EVALUATION
---------------------------------------------------------- Chapter 6:3
NOAA and FAA officials provided oral comments on a draft of this
report. These officials generally concurred with the report's
findings. NWS officials emphasized that they are tackling ASOS'
technical problems and plan to identify, validate, and address
outstanding user needs. Additionally, they noted that not all ASOS
user needs can be allocated equal weight and urgency. They also
stated that ASOS will satisfy aviation requirements, which they
described as the most challenging, and that the essential
requirements for ASOS' contribution to other NWS operational services
will be met. NWS officials added that other users' needs will be
satisfied through a combination of ASOS product improvements and
other data sources.
These statements are not inconsistent with our report. However, the
point remains that ASOS problems in meeting both specified
requirements and user needs beyond the specification's scope have yet
to be resolved. Moreover, what it will take in terms of the time and
money needed to resolve these problems and when ASOS will ultimately
permit the release of human observers remain uncertain. Until NWS
addresses these uncertainties in collaboration with ASOS users, it
does not know whether ASOS corrections, enhancements, and supplements
are worth the investment.
ASOS FAILURES TO MEET
SPECIFICATIONS AND STATUS OF NWS
ACTIONS
=========================================================== Appendix I
WIND SENSOR
--------------------------------------------------------- Appendix I:1
ASOS' wind sensor, designed to measure wind speed and direction, and
required to operate in all weather conditions encountered throughout
the United States, reports incorrect wind speed and direction and
freezes in cold weather because water seeps into the sensor's
mechanical devices. While the ASOS program office does not have data
on how often this problem occurs, the chief of the technical division
stated that it "happens a lot." These problems were identified only
after system deployment. An NWS testing manager stated that this
problem was not identified earlier because system tests were not
conducted in winter conditions.
Since the problem surfaced, the ASOS program office, which is
responsible for systems in production, has approved a sensor
modification on systems currently under production. OSO, which is
responsible for commissioned systems, is field testing the sensor
modification during the 1994-1995 winter. In February 1995, OSO
approved the wind sensor modification for implementation on
commissioned systems, based on preliminary testing. OSO is
continuing to test the modification.
Additionally, a number of wind sensor components have become
contaminated with a residue, resulting in incorrect wind direction
observations. The ASOS contractor reported in April 1994 that its
subcontractor sampled several wind sensors and found 1 in 10 were
affected by this residue. According to an ASOS program official, the
program office has not yet determined why the residue forms, how
often this residue causes the sensors to fall short of
specifications, or its impact on wind direction. OSO is now studying
the problem.
PRECIPITATION ACCUMULATION
SENSOR
--------------------------------------------------------- Appendix I:2
ASOS' precipitation accumulation sensor, a heated tipping bucket
intended to measure liquid precipitation amounts to an accuracy
within .02 inches or 4 percent of hourly totals (whichever is
greater), is not accurate in heavy rain events. On the basis of
testing in two locations, ASOS program officials stated that the
sensor underestimates precipitation by about 6 percent, but they have
no data to determine how frequently the underestimations occur in the
field. In another study sponsored by NWS, climatologists compared
ASOS-measured and conventionally measured precipitation amounts at 13
commissioned ASOS sites and found that ASOS understated precipitation
amounts by 14.4 percent in the summer of 1993, by 6.2 percent in the
fall of 1993, and by 9.2 percent in the spring of 1994.\1
According to the ASOS user manual and climatology publications, these
inaccuracies are due to well-known limitations of the tipping bucket
technology. The contractor has modified the sensor to correct this
problem, and the ASOS program office tested and approved the
modification for ASOS units currently being produced. OSO is
operationally testing the modification. Once testing is successfully
completed, commissioned ASOS units are to be retrofitted.
--------------------
\1 ASOS reported 25.2 percent less precipitation than conventional
sensors in the winter of 1994. However, these data are not
meaningful because the specification applies only to liquid
precipitation. Because it was winter, these data are likely to
include some accumulation from frozen precipitation.
PRECIPITATION IDENTIFICATION
SENSOR
--------------------------------------------------------- Appendix I:3
ASOS' precipitation identification sensor, intended to identify
different types and intensities of precipitation, is required to
operate in all weather conditions encountered throughout the United
States. However, this sensor becomes blocked and shuts down in heavy
snow and icing conditions. The ASOS program office has no estimate
of how frequently this problem occurs. The program office tested and
approved a contractor-developed sensor modification for ASOS systems
currently in production, which program officials stated will
partially address the problem. OSO is field testing this
modification before approving it for implementation on commissioned
ASOS units. The program office is exploring further modifications to
fully address the problem.
NWS officials acknowledged that the sensor experiences problems
during heavy snow events when the lens is blocked, but stated that
the precipitation identification sensor's problem is not a failure to
meet the specification. We disagree. We found that the
problem--snow blockage--is identified in NWS' trouble report system
as a contract deficiency, meaning that any sensor modifications to
address this problem are the contractor's responsibility.
FREEZING RAIN SENSOR
--------------------------------------------------------- Appendix I:4
ASOS' freezing rain sensor failed winter tests for detecting
occurrences of freezing rain and thus has not yet been accepted by
the program office or deployed with the rest of the ASOS sensors.
According to the program office, there was no known technology for
detecting freezing rain occurrences when ASOS was specified. OSO is
operationally testing an upgraded version of this sensor. NWS
officials stated that they expect this sensor to be ready for
deployment by next winter.
TEMPERATURE AND DEW POINT
SENSOR
--------------------------------------------------------- Appendix I:5
ASOS' temperature and dew point sensor does not meet reliability
specifications for dew point measurement. Since NWS provided over
400 temperature and dew point sensors to the contractor as
government-furnished equipment, the program manager acknowledged that
NWS is responsible for the sensors' performance. The ASOS program
office does not know the magnitude of the dew point inaccuracies or
the frequency with which they occur, but acknowledges that the dew
point problems are severe enough to warrant corrective action. NWS
officials stated that NWS and the contractor are addressing these
problems. Additionally, NWS is exploring and evaluating the use of
alternative technologies for measuring dew point and humidity.
CEILOMETER
--------------------------------------------------------- Appendix I:6
The ASOS ceilometer, required to measure cloud height up to 12,000
feet, to be accurate within 100 feet or 5 percent (whichever is
greater), and to have a mean time between failures of more than 8,000
hours, experienced failures about every 4,000 hours in a June 1994
NWS report. Since NWS provided several hundred ceilometers to the
contractor as government-furnished equipment, the ASOS program office
acknowledged that NWS is responsible for the sensors' performance.
OSO is evaluating the ceilometer to determine ways to decrease
overall ceilometer failure rates. For example, the ASOS program
office and OSO are evaluating a sensor modification to address
ceilometer failures due to overheating when bright sunshine reflects
off snow. NWS officials stated that they expect to implement this
modification before next winter.
AVAILABLE DATA, THOUGH LIMITED,
SUGGEST UNREPRESENTATIVENESS IS A
PROBLEM
========================================================== Appendix II
In attempting to determine the extent to which ASOS observations are
unrepresentative of prevailing weather conditions, we contacted NWS,
FAA, and officials from various aviation and climate organizations.
Through these contacts, we obtained and assessed records of
unrepresentative observations from three individual ASOS sites, a
draft NWS assessment of unrepresentativeness, and a Colorado State
University Atmospheric Science Paper comparing ASOS and conventional
observations of cloud height and visibility. Though these documents
have limitations, they support users' statements concerning
unrepresentative ASOS observations and suggest that this may be a
problem. These documents are discussed below.
UNREPRESENTATIVE ASOS
OBSERVATIONS RECORDED AT SOME
SITES
-------------------------------------------------------- Appendix II:1
Officials from various aviation organizations\1 told us that air
traffic controllers, pilots, and weather observers sometimes
encounter situations in which ASOS observations are unrepresentative
of existing weather conditions. However, not all ASOS users
regularly record and report these occurrences, and thus data
summarizing the full extent of the concern do not exist. National
Air Traffic Controllers Association and National Weather Service
Employees Organization representatives identified six FAA and two NWS
field locations where they stated unrepresentative ASOS observations
might be logged. We interviewed officials at these field locations
and reviewed the records of three.\2
Our review of the available records at the three field offices
reinforced the officials' statements about unrepresentativeness.
Specifically, air traffic controllers at a Texas airport recorded an
average of 19 events per month from November 1992 through March 1993
in which they found that ASOS observations did not represent actual
visibility, ceiling, or precipitation conditions. Similarly, a New
York weather office recorded an average of 51 unrepresentative events
per month from November 1993 to February 1994 and an average of 32
events per month from March to May 1994. Also, a Colorado weather
office recorded an average of 9 unrepresentative events per month
from November 1993 to January 1994, increasing to an average of 17
events per month between February and April 1994. We did not assess
the significance or impact of these events because the records
generally did not provide information on the events' impact on
aviation operations.
--------------------
\1 The organizations that we contacted were the National Air Traffic
Controllers Association, Airline Dispatchers Federation, Air
Transport Association, and the National Weather Service Employees
Organization.
\2 Of the five remaining locations we contacted, air traffic
controllers in two airports provided sporadic records over limited
periods of time. Controllers in two other airports had no records to
provide. Controllers in the final airport stated that they turned
the records over to the local weather office, but the weather office
representatives were unable to locate them.
NWS' DRAFT ASSESSMENT OF
UNREPRESENTATIVENESS IS
INCONCLUSIVE
-------------------------------------------------------- Appendix II:2
NWS started collecting data on the frequency of instances in which
ASOS is unrepresentative of prevailing weather in February 1993, and
drafted an assessment of unrepresentativeness in June 1994. Using
these data, NWS reported that unrepresentative ceiling and visibility
observations are each occurring less than 1 percent of the time,
respectively, and thus is a "small but not insignificant" problem.
However, the assessment itself stated that no conclusions should be
drawn from the data because of the study's limitations.
Specifically, the study relied on observers to report
unrepresentative observations instead of systematically collecting
and comparing observations from ASOS and observers. Additionally,
the data being collected and used omitted some instances and thus
understate the magnitude of the problem. For example, air traffic
controller observed instances of unrepresentativeness that are not
validated by an NWS observer are not reported. According to
controllers that we interviewed, such validation does not always
occur. Also, the assessment's criteria for what constitutes a "good"
observation could be more permissive than that used by observers, and
according to an ASOS testing official, resulted in unrepresentative
observations being counted as representative. For example, one
weather office reported an unrepresentative event in which ASOS
observed visibilities of 3 miles while the observer observed
visibilities of 2 miles. Because this situation did not fit the
criteria for an unrepresentative event, it was counted as a good
observation. Additionally, not all study sites choose to participate
and thus their data were omitted.
The study was also limited in that its observations were drawn in all
weather conditions, and did not focus on bad and changing weather
conditions, when ASOS users say representativeness is most important
to the aviation community and when ASOS performance is poorest.
Senior FAA and NWS field managers reported that representativeness
statistics derived from all weather conditions are misleading because
the weather is good the majority of time and ASOS performs well when
the weather is good. They stated that the more meaningful data are
ASOS performance during those less frequent but critical periods when
the weather is bad.
UNIVERSITY STUDY COMPARING ASOS
AND OFFICIAL OBSERVATIONS
DEMONSTRATES OCCURRENCES AND
EFFECTS OF UNREPRESENTATIVE
OBSERVATIONS
-------------------------------------------------------- Appendix II:3
A 1993 Colorado State University Atmospheric Science Paper, entitled
A Comparison of Ceiling and Visibility Observations for NWS Manned
Observation Sites and ASOS Sites,\3 analyzed the representativeness
of visibility and ceilings by comparing about 64,000 ASOS
observations against the official weather observations reported in 16
locations.\4 The study concluded that ASOS-observed ceilings differed
by more than 1,000 feet from official human observations 7 percent of
the time, and the ASOS visibilities differed by more than one
reportable category\5 from official human observations 6 percent of
the time.
The study also targeted observations during active weather (i.e.,
rain, snow, fog, or drizzle) and found that unrepresentative
observations occur more frequently in these conditions. Of the
64,000 observations, about 9,300 were during active weather. The
study reported that in rain, snow, drizzle, or fog, the ASOS-observed
ceilings differed by more than 1,000 feet from human observations 24
percent of the time, and the ASOS visibilities differed by more than
one reportable category from human observations 39 percent of the
time.
According to aviation organization officials, not all
unrepresentative observations are significant. These officials
stated that unrepresentativeness is significant if it affects flight
rules. To determine the significance of unrepresentative ASOS
observations, the study compared ASOS and human observations in
situations where one or the other indicated the need to invoke
Instrument Flight Rules (IFR).\6 The study assumed that ceilings
between 200 and 1,000 feet and/or visibility between 1/2 and 3 miles
constituted IFR flying conditions. These conditions are generally
agreed upon in the aviation community to constitute IFR flying
conditions. Our analysis of the study data determined that of 5,893
potential IFR observations, the ASOS and officially reported
observation disagreed 24 percent of the time. That is, ASOS observed
IFR conditions when the official observation did not 11 percent of
the time, and the official observation reported IFR conditions when
ASOS did not 13 percent of the time.
--------------------
\3 Jon C. Cornick and Thomas B. McKee, Atmospheric Science Paper
No. 529, May 1993.
\4 A limitation of the study is that the systems included were ones
that were deployed and operating, but not yet commissioned. Because
of ASOS' maintenance policy, these systems may not have received the
same level of routine maintenance and calibration that commissioned
systems receive. Thus, errors documented in the study may be
attributable, at least in part, to sensor degradation due to reduced
maintenance. According to an OSO program manager, however, this
limitation is mitigated by the fact that any meaningful study of ASOS
representativeness has to be based on uncommissioned systems because
only these systems have an accompanying human observation taken each
hour to compare ASOS against. Once an ASOS is commissioned, the ASOS
observation becomes the official observation and the hourly human
observations are no longer required.
\5 ASOS reportable categories for visibility observations are less
than 1/4 mile, 1/4 mile, 1/2 mile, 3/4 mile, 1 mile, 1-1/4 miles,
1-1/2 miles, 1-3/4 miles, 2 miles, 2-1/2 miles, 3 miles, 3-1/2 miles,
4 miles, 5 miles, 7 miles, and 10+ miles.
\6 Instrument Flight Rules are the set of FAA regulations that govern
aircraft operations under certain conditions, including poor weather.
IFR conditions require the use of navigational aids and specific
takeoff, landing, and air traffic control procedures.
MAJOR CONTRIBUTORS TO THIS REPORT
========================================================= Appendix III
ACCOUNTING AND INFORMATION
MANAGEMENT DIVISION,
WASHINGTON, D.C.
------------------------------------------------------- Appendix III:1
Rona B. Stillman, Chief Scientist for Computers and Communications
Randolph C. Hite, Assistant Director
Keith A. Rhodes, Technical Assistant Director
Colleen M. Phillips, Evaluator-in-Charge