Federal Contracting: Weaknesses Exist in NSF's Process for Awarding
Contracts (Letter Report, 01/31/94, GAO/RCED-94-31).
Weaknesses exist in the National Science Foundation's (NSF) process for
awarding competitive contracts. Among the 10 competitive contracts it
reviewed, GAO discovered that NSF, when soliciting for proposals,
sometimes did not clearly describe the work needed or identify the
specific items that it would consider in evaluating the offerors'
proposals. In addition, NSF sometimes either changed or improperly
scored evaluation factors during the evaluation process. These
weaknesses were mainly due to the NSF's overall lack of emphasis of
contracting activity, including inadequate internal oversight of the
contract award process. Insufficient guidance for preparing
solicitations and evaluating offerors' proposals were also contributing
factors. In addition, GAO discovered shortcomings in NSF's process for
awarding noncompetitive contracts. Although agencies are allowed to
obtain services from a sole source rather than through competition, the
agency must justify the decision to do so in writing. The justification
documents what efforts were taken to identify other potential offerors
and what steps the agency plans to take to remover barriers to future
competition. NFS did not meet either of these requirements for 6 of 11
noncompetitive contracts it awarded during fiscal years 1990 and 1991.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: RCED-94-31
TITLE: Federal Contracting: Weaknesses Exist in NSF's Process for
Awarding Contracts
DATE: 01/31/94
SUBJECT: Contract administration
Contracting procedures
Bid evaluation
Evaluation criteria
Competitive procurement
Sole source procurement
Research and development contracts
Federal procurement
Research grants
Solicitation specifications
**************************************************************************
* This file contains an ASCII representation of the text of a GAO *
* report. Delineations within the text indicating chapter titles, *
* headings, and bullets are preserved. Major divisions and subdivisions *
* of the text, such as Chapters, Sections, and Appendixes, are *
* identified by double and single lines. The numbers on the right end *
* of these lines indicate the position of each of the subsections in the *
* document outline. These numbers do NOT correspond with the page *
* numbers of the printed product. *
* *
* No attempt has been made to display graphic images, although figure *
* captions are reproduced. Tables are included, but may not resemble *
* those in the printed version. *
* *
* A printed copy of this report may be obtained from the GAO Document *
* Distribution Facility by calling (202) 512-6000, by faxing your *
* request to (301) 258-4066, or by writing to P.O. Box 6015, *
* Gaithersburg, MD 20884-6015. We are unable to accept electronic orders *
* for printed documents at this time. *
**************************************************************************
Cover
================================================================ COVER
Report to the Chairman, Subcommittee on Investigations and Oversight,
Committee on Science, Space, and Technology, House of Representatives
January 1994
FEDERAL CONTRACTING - WEAKNESSES
EXIST IN NSF'S PROCESS FOR
AWARDING CONTRACTS
GAO/RCED-94-31
Federal Contracting
Abbreviations
=============================================================== ABBREV
GAO - General Accounting Office
NSF - National Science Foundation
Letter
=============================================================== LETTER
B-254395
January 31, 1994
The Honorable Jimmy Hayes
Chairman, Subcommittee on
Investigations and Oversight
Committee on Science, Space,
and Technology
House of Representatives
Dear Mr. Chairman:
The National Science Foundation (NSF), an independent federal agency
with a fiscal year 1994 budget of about $3 billion, promotes and
advances science in the United States, primarily through grants to
research institutions. In addition, NSF enters into contracts for
services that assist the agency in its program evaluations and in
other program activities, such as processing grant proposals.
Because of concerns about the integrity of NSF's procurement process
for awarding its competitive and noncompetitive contracts, we were
asked to determine whether weaknesses exist in this process.
Specifically, for competitive contracts, we assessed (1) whether NSF,
when soliciting for proposals, clearly described the work needed and
identified the specific items that would be considered in evaluating
the proposals and (2) whether NSF evaluated the proposals in
accordance with the factors included in the solicitations. For
noncompetitive contracts, we assessed whether NSF, as required,
properly justified procuring the services on a sole-source basis. To
address these objectives, we reviewed 10 competitive contracts that
we randomly selected from a universe of 32 competitive contracts and
all 11 noncompetitive contracts that NSF awarded during fiscal years
1990 and 1991. The estimated value of the 21 contracts we reviewed
totaled about $49.4 million. (See app. I.)
RESULTS IN BRIEF
------------------------------------------------------------ Letter :1
Weaknesses exist in NSF's process for awarding competitive contracts.
Specifically, we identified one or more weaknesses in the contract
award process for 6 of the 10 competitive contracts that we reviewed.
Nine weaknesses in the award process for the six contracts are
discussed below. For three contracts, NSF did not clearly describe
during the contract award process the work it needed; as a result,
the offerors, who are the potential contractors, were not provided
with enough information to determine what services NSF actually
needed. Also, for three contracts, NSF did not inform the offerors
about the relative importance of the factors it would use in its
evaluation of the offerors' proposals. As a result, the offerors had
to rely on their own judgment of the factors' importance to NSF in
evaluating offerors' proposals. In addition, for three contracts,
NSF's evaluations of proposals were faulty because the evaluation
factors were either changed or improperly scored during the
evaluation process. When proposals are not evaluated correctly,
contracts could be awarded that are not the most advantageous to the
government.
The weaknesses we found, in general, can be attributed to the
agency's overall lack of emphasis on contracting activity, including
inadequate internal oversight of the contract award process.
Insufficient guidance for preparing solicitations and evaluating
offerors' proposals has also contributed to the weaknesses
identified. Taken together, these weaknesses raise questions about
NSF's competitive contract award process.
Weaknesses also exist in NSF's process for awarding noncompetitive
contracts. Although federal law allows an agency to obtain services
from only one source, rather than through competition, the agency
must provide a written justification for doing so. The justification
is to, among other things, document what efforts were taken to
identify other potential offerors and state what actions, if any, the
agency plans to take to remove obstacles to future competition. NSF
did not meet either of these requirements for 6 of the 11
noncompetitive contracts that it awarded during fiscal years 1990 and
1991. As a result, NSF may have denied other potential offerors the
opportunity to compete for procurements and deprived itself of the
benefits of competition.
BACKGROUND
------------------------------------------------------------ Letter :2
Federal agencies, including NSF, are required to comply with the
Competition in Contracting Act of 1984 and the Federal Acquisition
Regulation when acquiring goods and services through contracts. When
a contract award is protested, GAO, under the authority of the act,
reviews the award for compliance with federal procurement laws and
regulations and issues a decision. NSF has developed a Proposal and
Award Manual, which was updated in November 1993, that (1) provides
the policies and procedures for soliciting procurement proposals and
for awarding contracts and (2) identifies the responsible officials
and their roles. NSF has supplemented the manual with memorandums
that provide additional guidance for awarding contracts.
As outlined in the act, the federal standard for competition in the
contracting process is called "full and open competition." Under full
and open competition, all responsible sources are permitted to
compete for a proposed procurement. When federal agencies acquire
goods and services through full and open competition, they generally
solicit proposals from offerors by distributing solicitations. The
competition act and the acquisition regulation require that a
solicitation (1) describe an agency's needed services; (2) identify
specific items, called evaluation factors, that offerors need to
address in their proposals and that will be considered in evaluating
the proposals; and (3) explain how the various factors will be
evaluated and what the relative importance of the factors will be
during the agency's contractor selection process. After the agency
receives the offerors' proposals, the agency is required to evaluate
the proposals in accordance with the factors given in the
solicitation.
When federal agencies acquire goods and services by noncompetitive
procurement rather than by full and open competition, they must
prepare a written justification that complies with the requirements
of the act and the regulation. This justification must identify (1)
the statutory basis and the reasons for making an exception to the
general requirement for competition, (2) the efforts that the agency
took to identify other suppliers, and (3) the efforts that it will
take to ensure that future procurements for the item will be
competitive.
At NSF, the Division of Contracts, Policy and Oversight is
responsible for issuing procurement solicitations, making contract
awards, and establishing the policies and procedures used in NSF's
contracting activities.\1
Contracting officers perform contracting functions within the
Division. NSF's personnel in program offices, who need the contract
services to accomplish their program missions, develop the
procurement statements of work and evaluation factors and assist in
evaluating contractors' offers.
As of April 1993, the cumulative amount that had been obligated to
fund active NSF contracts was about $1.7 billion. Services procured
under NSF contracts have ranged from processing grant applications to
evaluating NSF programs.
--------------------
\1 The Division of Contracts, Policy and Oversight was established in
May 1993 through a reorganization of NSF's Division of Grants and
Contracts, which had been responsible for performing grants and
contracts functions. All of the contracts that we reviewed were
awarded before May 1993.
WEAKNESSES EXIST IN THE
COMPETITIVE CONTRACT AWARD
PROCESS
------------------------------------------------------------ Letter :3
In 6, or 60 percent, of the 10 contracts we reviewed, one or more
weaknesses existed in the process NSF used to award the contracts.\2
These weaknesses included failures to (1) clearly explain NSF's needs
to offerors, (2) adequately explain the relative importance of the
evaluation factors provided to offerors, (3) evaluate proposals in
accordance with the evaluation factors and without scoring errors,
and (4) reject an offeror's late proposal.
--------------------
\2 On the basis of the sample, the estimated percentage of contracts
in the universe having one or more of the weaknesses is between 28.1
and 84.4 percent, at the 95- percent confidence level. (See app.
I.)
DESCRIPTIONS OF WORK DID NOT
EXPLAIN NSF'S NEEDS TO
OFFERORS
---------------------------------------------------------- Letter :3.1
NSF did not always provide to offerors clear descriptions of the work
it needed for use in preparing proposals to compete for contracts.
Specifically, statements of work for 3, or 30 percent, of the 10
competitive procurements that we reviewed were not clear. The act
and regulation require agencies to specify their needs in order to
promote full and open competition for procurements and to include any
type of descriptions or specifications needed to do this. Similarly,
both NSF's Proposal and Award Manual and internal memorandums require
solicitations to have statements of work that contain sufficient
detail to elicit responsive proposals. Furthermore, GAO has stated
that it is a fundamental principle of federal procurement law to
require that solicitations--which include statements of work--be
clear and complete in order to provide offerors with a common basis
for preparing proposals so that they can compete on an equal basis.\3
For one procurement, the statement of work stated that the objective
of the procurement was to assess the overall merits of an NSF
program. However, we do not believe the specific tasks included in
the statement of work would accomplish a program assessment. The
tasks included activities such as describing the proposed methodology
for sampling participants, designing the interview, and designing the
plan for analyzing the data. The implication was that an assessment
was to be made and a report prepared, but no specific tasks were
mentioned for compiling and analyzing data. Thus, the statement of
work did not clearly and completely describe NSF's needs to the
competing offerors.
Within 3 months after the contract was awarded, the winning offeror
told NSF program officials that the contract needed to have tasks
added so that an assessment could be done. The NSF contracting
officer reviewed the statement of work and determined that a change
had to be made to the contract to add the tasks necessary to complete
the original intent of the procurement. The additional work
increased the amount of the contract by $40,743, or from $238,004 to
$278,747.
For a second procurement, the statement of work included a provision
for additional technical support tasks that did not clearly define
the additional services requested. The statement of work stated that
the offeror should be prepared to provide additional technical
assistance to NSF staff in survey research design and analysis in
areas that were not identified. The unspecified technical support
assistance was estimated to cost $250,000, representing 20 percent of
the amount of the contract. Since the services were not specifically
identified, this open-ended provision could have meant many different
things, and the competing offerors may not have been competing with
an equal understanding of NSF's needs.
For a third procurement, the statement of work was unclear; several
vague terms and phrases were used to describe NSF's required
services. For example, the statement of work included terms and
phrases such as "adequate spare parts" and "adequate documentation."
In addition, the statement of work did not clearly indicate the need
for a minimum number of personnel and company resources. However,
these items were considered significant when the proposals were
evaluated.
Such unclear statements of work do not provide a common basis on
which offerors can compete, and the offerors may be reduced to
guessing at what services NSF wants. Furthermore, unclear and
incomplete statements of work in cases where more specificity could
have been used may lead to additional costs after a contract is
awarded for work that was not originally requested or clearly
understood. We found that contracting is not a major activity at NSF
compared to its grant activity and that the problems we identified
were the result of inadequate oversight and guidance.
NSF procurement officials in the former Division of Grants and
Contracts agreed that contracting is not a major activity at NSF
compared with the agency's grant activity. Also, the officials told
us that because contracting officers were responsible for both
contracts and grants before May 1993 and did not routinely prepare
contracts, inadequate oversight by contracting officials would have
contributed to the problems we identified. Furthermore, the
officials said that some personnel in program offices do not
routinely prepare statements of work because of the small amount of
contracting in those offices. In addition, the officials
acknowledged that NSF's guidance does not specify how statements of
work should be written.
To ensure better oversight of contracts, the Division was split into
two separate divisions in May 1993--one handling contracts and the
other handling grants.\4 Division officials believe that because the
contracting officers and other contracting personnel responsible for
contract awards will no longer handle grant awards, they will be able
to focus their attention on ensuring that contract awards comply with
applicable requirements. However, the Division has not issued any
specific guidance to its program offices on how to prepare statements
of work.
--------------------
\3 Tony Ingoglia Salami and Cheese, Inc., Comptroller General
Decision, B-244452, Sept. 23, 1991, 91-2 C.P.D. � 268.
\4 The Division of Contracts, Policy and Oversight is now handling
contracts, while the Division of Grants and Agreements is handling
grants.
IMPORTANCE OF EVALUATION
FACTORS WAS NOT PROVIDED TO
OFFERORS
---------------------------------------------------------- Letter :3.2
The relative importance of the factors used to evaluate proposals was
not clearly explained to competing offerors in 3, or 30 percent, of
the 10 competitive contracts that we reviewed. As a result, offerors
had to rely on their own judgment of the factors' importance. For
one of the procurements, additional explanations of the importance of
the evaluation factors were needed in order to comply with the
regulation, as GAO interprets it. In two other procurements,
additional explanations could have improved offerors' understanding
of the factors' importance.
The act and the regulation require agencies to issue solicitations
that clearly state all factors and their relative importance--such as
cost and technical capability-- that will be used to evaluate
proposals. The regulation requires the factors to be listed in their
order of relative importance, but it does not specifically require
that a numerical weight be assigned or that adjectival descriptions
of the factors' relative importance be included. However, in bid
protest decisions, GAO has interpreted the regulation to mean that
when one factor is assigned a predominant value over the other
factors, this importance should be disclosed to potential offerors,
and more is required than a mere statement that evaluation factors
are listed in their order of importance.\5 NSF's Proposal and Award
Manual also requires that technical evaluation criteria and their
relative importance must be included in solicitations.
Although all of the solicitations we reviewed listed the evaluation
factors, the relative importance of the factors was not explained as
precisely as possible in three of the solicitations. For one
procurement, the solicitation listed the evaluation factors as A, B,
C, and D and stated that they were listed in order of importance.
The relative importance of the factors was not disclosed in the
solicitation, thus giving the impression that the difference between
the factors was relatively small. However, the maximum value
assigned to each factor and used in evaluating the proposals was 750
points for A, 100 points for B, 75 points for C, and 75 points for D.
Therefore, factor A is at least 7.5 times greater than any other
factor--a clearly significant difference. Thus, merely listing the
factors in the order of importance was misleading to the competing
offerors, since the significant difference in the value of the
factors was not disclosed. GAO interpreted the solicitation as being
inconsistent with the regulatory requirements.
For two other procurements, the differences in the numerical values
assigned to the factors during the evaluation process were not as
large as those in the example above, but a clearer explanation of the
factors' relative importance would have helped the offerors in
identifying the most important factors. For example, these two
procurements had four evaluation factors--A, B, C, and D; factor C
was 2.5 times as important as factor D, which followed it. One of
the potential offerors who submitted questions to NSF on one of the
solicitations asked about the importance of the evaluation factors.
However, NSF did not provide any additional information and stated
that the information provided in the solicitation was sufficient for
offerors to determine the factors' importance.
In these two procurements, the relative importance of the evaluation
factors could have been described more fully. As a matter of sound
procurement policy, we believe that when evaluation factors are
materially unequal, the fullest possible disclosure of all of the
evaluation factors and their relative importance is preferred to
relying on the reasonableness of the offerors' judgment as to the
relative importance of the various evaluation factors.
When the relative importance of evaluation factors is not explained
to competing offerors, the offerors' ability to prepare proposals
that meet the requirements of the procuring agency is hindered. The
offerors must rely on their own judgment to determine the importance
of the evaluation factors; as a result, they may not spend enough
time on important factors and may spend too much time on less
important factors. These unneeded efforts to address less important
factors could lead to wasted costs and could hurt the procuring
agency, because it may not obtain proposals that address its true
needs. We found that NSF had not developed any additional guidance
for use in describing the relationship between factors.
Officials in the Division of Grants and Contracts agreed that NSF's
guidance requires disclosure only of the relative importance of the
evaluation factors in solicitations. The guidance does not include
any criteria for describing the relationship between these factors.
Furthermore, as discussed above, the officials said that because
contracting officers do not award contracts as routinely as they do
grants and because the number of contracts awarded is small,
oversight of evaluation factors may also have been inadequate.
--------------------
\5 Sperry Rand Corporation, Comptroller General Decision, B-179875,
Sept. 12, 1974, 74-2 C.P.D. � 158.
EVALUATIONS OF PROPOSALS
WERE FAULTY
---------------------------------------------------------- Letter :3.3
For 3, or 30 percent, of the 10 competitive contracts that we
reviewed, the evaluation factors, though satisfactorily described,
were either changed or improperly scored when the proposals were
evaluated. The act and the regulation require NSF, once the
competing offerors' proposals are received, to evaluate the proposals
in accordance with the factors in the solicitations and in a manner
that is fair to competing offerors. Similarly, NSF's Proposal and
Award Manual requires that specific evaluation criteria and the
relative importance of the criteria as stated in the solicitations be
used to evaluate offerors' proposals. In addition, NSF's May 8,
1992, memorandum on contracting states that the criteria used to
evaluate proposals must be the same as those given in the
solicitation; the memorandum includes a model evaluation plan and
procedures and examples of rating scales and scoring sheets for
personnel in program offices to use in evaluating proposals.
Specifically, in two procurements, we found that the factors used to
evaluate proposals were not the same as those given in the
solicitations. For one procurement, the solicitation stated that
factors A and B were of equal value, but during the actual evaluation
of proposals, factor A was given almost twice the importance of
factor B. In the other procurement, the factors in the solicitation
and the factors used in evaluating the proposals were completely
different. A factor in this solicitation, which accounted for 20
percent of the evaluation score, stated that the offerors would be
evaluated on their ability to provide remedial maintenance within 4
hours. The criterion in the evaluation plan stated that an offeror
must have a detailed plan to provide remedial maintenance 24 hours a
day, 7 days a week.
For the remaining procurement, each factor in the proposals was to be
evaluated and given an even-numbered score between 0 and 10 on the
basis of the criteria associated with the number score. For example,
a score of 2 was required when a proposal was evaluated as
inadequate, with a major weakness that could not be corrected without
substantive revisions. A score of 6 or more reflected a factor that
was adequate and met specifications without substantive revisions,
with no major weaknesses. For one offeror's proposal, the numerical
evaluation criteria related to the evaluation factors were not
consistently applied to the proposal by the evaluators who scored it.
For example, the evaluators had rated the proposal with numerical
scores of 6 or more, indicating that the proposal had no major
weaknesses, but the narrative evaluation of the proposal showed that
the evaluators believed that the proposal did have major weaknesses
and thus should have resulted in numerical scores of 4 or less.
Mistakes in the application of evaluation factors and the scoring of
proposals affect the integrity of NSF's competitive award process.
Also, faulty scoring could result in an award to an offeror who is
not the most advantageous for the government. We found that the
contracting officers did not provide adequate oversight of the
application of evaluation factors in three cases and that NSF's
current guidance does not specifically address the contracting
officers' responsibility to ensure that the numerical scoring of the
proposals is done correctly.
Officials in the Division of Grants and Contracts told us that they
could not explain the causes of the problems we found with the
proposal evaluations. They pointed out, however, that NSF's current
guidance instructs contracting officers and program offices'
personnel to ensure that the evaluation factors used to evaluate
proposals are the same as those in the solicitation. Even so, the
guidance does not specifically tell the contracting officers to
ensure that the evaluations are done correctly.
NSF ALLOWED ONE LATE OFFEROR
TO COMPETE AFTER INITIAL
PROPOSAL EVALUATIONS WERE
COMPLETED
---------------------------------------------------------- Letter :3.4
According to the Federal Acquisition Regulation, the offerors'
proposals that are received after the deadline for submission should
not be considered. However, a late proposal may be accepted if it is
received before a contract is awarded, if it meets the mailing or
other requirements specified by the regulation, and if the late
receipt was the result of the government's mishandling of the
proposal. In one procurement we reviewed, NSF accepted an offeror's
proposal 3 months after the cutoff date. Although the offeror who
submitted the late proposal was not awarded the contract, the
inclusion of the late proposal was unfair to the original offerors.
Acceptance of the late proposal was unfair because it was not
required to go through the initial evaluation and ranking process
that the original offerors' proposals went through.
Although NSF contracting personnel in the Division of Grants and
Contracts reviewing the procurement believed the acceptance of this
late proposal was improper, they continued with the procurement to
prevent delays in awarding the contract. They did so because the
procurement had been under way for over 6 months, the late proposal
already had been evaluated when the Division reviewing official found
out that the proposal was late, and the potential for a bid protest
was likely if the solicitation was canceled.
JUSTIFICATIONS FOR
NONCOMPETITIVE PROCUREMENTS DID
NOT ADDRESS ALL REQUIRED
FACTORS
------------------------------------------------------------ Letter :4
NSF's justifications for some noncompetitive awards did not document
NSF's attempts to identify other capable offerors and the efforts
made, if any, to remove obstacles to future competition, as required,
prior to making noncompetitive contract awards. The act and
regulation allow federal agencies, including NSF, to obtain services
from only one source when no other source will satisfy the agencies'
requirements. However, in doing so, agencies must provide a written
justification to support the decision. The justification should
describe the efforts made to ensure that offers are solicited from as
many potential sources as is practical. It should include any effort
that was made to identify potential offerors from the commercial
marketplace; a list of other sources that expressed an interest; and
a statement of actions, if any, that the agency plans to take to
remove obstacles to future competition. Additionally, NSF's May 8,
1992, memorandum states that noncompetitive procurements should be
documented and justified, as required by the regulation. Unlike
competitive procurements, noncompetitive procurements do not require
the issuance of solicitations that contain statements of work and
evaluation factors. In addition, proposals are not received from
multiple offerors for evaluation and scoring prior to the contract
award.
NSF's justifications for 6 of the 11 noncompetitive contracts awarded
during fiscal years 1990 and 1991 did not document if other offerors
were available and capable of meeting NSF's needs and if efforts were
made to increase future competition for the services. In each of the
six cases, the contracting officer prepared a justification stating
that it was impossible to obtain full and open competition. However,
our review of NSF's justifications and related information in the
contract files did not disclose any written documents to show that
NSF had attempted to identify other potential offerors capable of
providing the required services. In addition, the justifications did
not document NSF's efforts, if any, to remove obstacles to future
competition, as required by the regulation. When NSF, or any other
agency, awards contracts without checking for the existence of other
capable offerors, it denies potential offerors the opportunity to
compete for procurements and deprives itself of the benefits of
competition.
A contracting official in the Division of Grants and Contracts said
that for the six noncompetitive procurements we questioned, the
contracts had been awarded to the National Academy of Sciences. He
explained that since the Academy is recognized as a unique and
reputable source throughout the government, NSF did not attempt to
identify other capable offerors and did not take any actions to
remove obstacles to future competition. The contracting official
told us that NSF recently awarded a competitive contract to an
organization other than the Academy to manage an NSF program though
the contract had been awarded to the Academy for the past 40 years.
The official further explained that he did not know how many other
noncompetitive contracts are with the Academy that could be
competitive. This lack of awareness of other potential offerors is
what the act sought to eliminate when it imposed the documentation
requirements in the justifications.
Although circumstances may exist in which a sole-source award is
justified on the basis of the unique status of an offeror and the
particular needs of an agency for a specific contract, we believe
that this justification should not be assumed to apply on a wholesale
basis. Furthermore, an offeror's unique status does not relieve
agencies of their responsibility under the regulation. The act and
the regulation require that an individual justification must be made
for each noncompetitive procurement on a case-by-case basis.
CONCLUSIONS
------------------------------------------------------------ Letter :5
At the heart of the federal competitive procurement process is the
concept of fairness to all potential competitors. This concept can
be gleaned from procurement law, which requires that all responsible
sources be permitted to compete for federal procurements. To
effectively compete for competitive procurements, potential offerors
need to know what the government's needs are and what evaluation
factors are important to the government. Furthermore, competing
offerors need to know that their proposals will be evaluated in
accordance with the evaluation factors listed in the solicitation and
that their proposals will be scored correctly. Although the concept
of fairness has been defined and explained by law, regulation, and
bid protest decisions by the Comptroller General, six of NSF's
competitively awarded contracts that we reviewed had one or more
weaknesses in these key areas. Considering the number of contracts
with weaknesses and the variety of weaknesses, NSF's competitive
contract award process is open to question.
We found inadequate oversight of the process for awarding contracts
and a lack of specific guidance on key aspects of the award process
to be the two primary causes of the weaknesses we identified.
Recently, in May 1993, NSF reorganized its contracts and grants
organization to create separate offices for contracts and grants.
Specifically, this was done to provide greater oversight of the
contract award process. We believe this is a step in the right
direction. However, the need still exists for more specific guidance
on the competitive contract award process in connection with the
weaknesses that we identified.
On the majority of the noncompetitive contracts that we reviewed, NSF
did not document whether it had made an effort to identify other
sources and to remove obstacles to competition for future
procurements. Since NSF depends largely on one contractor, other
potential offerors may not have been given the opportunity to compete
for the work.
RECOMMENDATIONS
------------------------------------------------------------ Letter :6
To further improve NSF's contracting activities, we recommend that
the Director, NSF, develop detailed guidance for
program officials to use in preparing statements of work, including
a description of the type of information the statements of work
should provide and examples of weaknesses to avoid;
program and contracting personnel to use in explaining the relative
importance of evaluation factors in solicitations; and
contracting officers to use in ensuring that evaluation criteria
are scored properly.
In addition, the Director, NSF, should ensure
that contract personnel comply with existing requirements to
determine if other offerors are available before awarding
contracts on a sole-source basis and document the efforts taken
to do so and
that planned efforts to remove obstacles to future competition for
sole-source procurements are documented as required.
AGENCY COMMENTS AND OUR
EVALUATION
------------------------------------------------------------ Letter :7
NSF provided written comments on a draft of this report and agreed
with all of our recommendations. However, NSF expressed an overall
concern that the draft report did not reflect recent improvements
made to the agency's contracting process and organization since the
period covered by our audit. We agree that NSF has taken a number of
steps to strengthen its management of contracting activities,
including a major reorganization in May 1993. We acknowledge this
reorganization in the report. To further address NSF's concerns, we
added a statement recognizing that NSF revised its contracting
policies and procedures manual in November 1993. While an
improvement, the manual still falls short of addressing the specific
weaknesses we identified.
Furthermore, while NSF acknowledged weaknesses in the process for
awarding certain contracts, it disagreed that the weaknesses were
pervasive. We do not state that the weaknesses we identified were
pervasive. The results of our random sample of competitive contracts
are presented in the report as an estimate of the percentage of
contracts in the universe with the identified weaknesses. To avoid
misinterpretation, we provide the lower and upper bounds of the
estimate of contracts with weaknesses in a footnote in the report and
in appendix I. An attachment to NSF's response letter details these
and other concerns about the report's characterization of our
findings. These comments and our evaluation of them are presented in
appendix II.
---------------------------------------------------------- Letter :7.1
To accomplish our objectives, we interviewed the appropriate NSF
officials in Washington, D.C. We also reviewed pertinent
legislation, regulations, Comptroller General Decisions, NSF
guidance, reports from NSF's Office of Inspector General, and
contract files. We conducted our review from June 1992 to November
1993, in accordance with generally accepted government auditing
standards. Appendix I provides additional information on our scope
and methodology.
As arranged with your office, unless you publicly announce its
contents earlier, we plan no further distribution of this report
until 30 days after the date of this letter. At that time, we will
send copies to the Director, NSF; the Director, Office of Management
and Budget; and other interested parties. We will also make copies
available to others upon request.
This work was performed under the direction of Victor S. Rezendes,
Director, Energy and Science Issues, who can be reached on (202)
512-3841 if you or your staff have any questions. Major contributors
to this report are listed in appendix III.
Sincerely yours,
Keith O. Fultz
Assistant Comptroller General
SCOPE AND METHODOLOGY
=========================================================== Appendix I
In response to a request from the Subcommittee on Investigations and
Oversight, House Committee on Science, Space, and Technology, we
determined whether weaknesses existed in the National Science
Foundation's (NSF) process for awarding its competitive and
noncompetitive contracts. Specifically, we assessed (1) whether
NSF's solicitations for competitive procurements clearly described
the work needed and identified the specific items that would be
considered in evaluating proposals and (2) whether NSF evaluated the
proposals in accordance with the factors in the solicitations. For
the noncompetitive contracts, we determined whether NSF properly
justified procuring the services on a sole-source basis.
We used data from NSF's computerized contract award system to
identify the universe of active contracts having award values of
$25,000 or more, and were awarded in fiscal years 1990 and 1991. We
identified 44 contracts that were awarded in fiscal years 1990 and
1991. One contract was excluded from the universe of contracts
because it was a management and operations contract that was not
typical of the other contracts. We determined that 32 of the 43
remaining contracts were awarded through competition, and 11 were
awarded without competition. We determined whether the contracts
were competitive or noncompetitive from data, which we obtained by
computer from the governmentwide Federal Procurement Data System, and
NSF files.
We selected for review the 11 noncompetitive contracts with an
estimated value of about $27.8 million. We also selected a random
sample of 10 competitive contracts with an estimated value of about
$21.6 million, for a total of 21 contracts with an estimated value of
about $49.4 million.
The 10 competitive contracts that we reviewed were taken from a
universe of 32 active contracts that were awarded by NSF in fiscal
years 1990 and 1991 with a total estimated value of about $95
million. We took a simple random sample of the competitively awarded
contracts to estimate the percentage of contracts in the universe
that had the weaknesses we found. Each sample proportion has a
measurable precision that indicates how closely we can reproduce from
the sample the results that we would obtain if we were to take a
complete count of the universe of 32 contracts using the same
measurement methods. For example, when a particular weakness was
found in 30 percent of the cases sampled, we estimate that between
9.4 and 62.5 percent of the cases in the universe have that weakness.
This range, or confidence interval, is calculated at the 95 percent
confidence level, meaning that 95 times out of 100 we would obtain a
confidence interval actually containing the percent of the cases in
the universe with that weakness. Table I.1 below shows the estimates
from our sample.
Table I.1
Estimated Percentage of NSF's
Competitive Contracts With Contract
Award Weaknesses
Number Percent Lower bound Upper bound
---------- ------------ ---------------- ----------------
0 0.0 0.0 25.0
1 10.0 3.1 40.6
2 20.0 6.3 53.1
3 30.0 9.4 62.5
4 40.0 15.6 71.9
5 50.0 21.9 78.1
6 60.0 28.1 84.4
7 70.0 37.5 90.6
8 80.0 46.9 93.8
9 90.0 59.4 96.9
10 100.0 75.0 100.0
------------------------------------------------------------
Note: The lower and upper bounds are calculated at a 95-percent
confidence level.
To determine whether NSF followed the procurement requirements for
awarding competitive contracts, we examined the description of
services in the statements of work, the criteria for evaluating
proposals that NSF provided to offerors, and the results of proposal
evaluations. We analyzed the descriptions of services contained in
the solicitations to determine whether they were clearly expressed
and understandable. We determined whether the criteria and
evaluations complied with regulatory requirements. We also
determined whether the justifications for noncompetitive contract
awards were consistent with certain requirements of the act and
regulation. In addition, we discussed the awards with NSF officials.
(See figure in printed edition.)Appendix II
COMMENTS FROM THE NATIONAL SCIENCE
FOUNDATION
=========================================================== Appendix I
See comment 1.
See comment 2.
(See figure in printed edition.)
See comment 1.
See comment 3.
(See figure in printed edition.)
See comment 4.
See comment 5.
See comment 6.
(See figure in printed edition.)
The following are GAO's comments on the Director's letter dated
November 24, 1993.
GAO'S COMMENTS
--------------------------------------------------------- Appendix I:1
1. These comments state that the report does not recognize the
changes and improvements that NSF has made to its contracting
activities. As noted in the report, we reviewed contracts that were
awarded in fiscal years 1990 and 1991. This period was chosen
because complete fiscal year 1992 data were not available at the time
we began our review in June 1992. However, we did evaluate and
consider contracting guidance that NSF issued in 1991 and 1992 and
acknowledge that NSF reorganized its contracting function in May 1993
to improve management and oversight of the contract awards. To
reflect recent changes, however, we have added a statement in the
report to note that NSF revised its contracting policies and
procedures manual in November 1993--guidance issued after completion
of our audit work. But more importantly, even though the manual is
an improvement, it does not address the specific weaknesses we
identified. Additional guidance is needed, as evidenced by NSF's
agreement with our recommendation that detailed guidance is needed in
the areas of statement of work clarity, evaluation factors, and
evaluation criteria scoring.
2. These comments state that we did not find that the outcome of any
contract awards should have been different, an implication that NSF
ultimately awarded the contracts to the appropriate firm. We believe
that this statement needs to be clarified. Our objective was to
determine whether weaknesses exist in the process for awarding
contracts, not whether the outcome of the contract awards should have
been different or whether different offerors should have received the
contract awards. The appropriateness of an agency's contract awards
is generally not questioned unless an interested party, such as a
potential offeror or competing offeror, chooses to protest the award.
3. These comments question whether our statistics generally reflect
pervasive weaknesses in NSF's contracting process. NSF states that
it agrees with certain of the weaknesses we identified but believes
that other weaknesses are subject to differences in opinion about the
best contracting practices. First, we disagree with NSF's view that
certain of the contracting weaknesses that we identified can be
explained away as "legitimate differences of opinion." We identified
two types of weaknesses: those that reflect violations of
procurement regulation or law and those that reflect poor contracting
practices. Second, in connection with the statistical
characterization of the weaknesses, we present the results of our
random sample of competitive contracts as an estimate of the
percentage of contracts in the universe with the identified
weaknesses. We provide the lower bound and upper bound of the
estimate of contracts with weaknesses. On the basis of finding
weaknesses in 6 of the 10 competitive contracts in our sample, we
estimate that between 28.1 percent and 84.4 percent of the 32
contracts in our universe of contracts contained a weakness. To
avoid ministerpretation, the lower and upper bounds of estimates are
included in a footnote in the report and in appendix I.
4. These comments state that the application of the significant
evaluation factor criteria as defined in GAO's Sperry Rand decision
was improperly applied to two contract solicitations included in our
sample, thereby distorting the sample results. The decision states
that when one factor is assigned a predominant value over the other
factors, this importance should be disclosed to potential offerors.
In one of the three solicitations that we found to be deficient, one
evaluation factor was 7.5 times more important than another
evaluation factor. We concluded that the solicitation was
inconsistent with our interpretation of the procurement regulation,
which requires agencies to state the relative importance of
evaluation factors in the solicitations. NSF agreed. In the second
two solicitations--the ones NSF is challenging--an evaluation factor
was 2.5 times more important than another evaluation factor. We do
not state in the report, nor do we mean to imply, that NSF's actions
in these two cases were inconsistent with procurement regulation or
at odds with the Sperry Rand decision. Instead, as a matter of sound
procurement policy, we believe that the fullest possible disclosure
of all of the evaluation factors and their relative importance is
preferred to relying on the reasonableness of the offerors' judgment
as to the relative importance of the various evaluation factors.
Consequently, on this basis we concluded that a clearer explanation
of the factors' importance would have helped offerors.
NSF also states that our report fails to acknowledge the Federal
Acquisition Regulation, which states that the relative importance of
solicitation factors is within the broad discretion of agency
acquisition officials. This section of the regulation applies to the
weight that agencies give the evaluation factors, and we agree that
agencies need this discretion to tailor a procurement to meet their
needs. However, this issue is not a subject of our report. The
issue discussed in our report involves the information that NSF
provides to potential offerors about the importance of the evaluation
factors, not NSF's determination of the importance of the evaluation
factors.
5. This comment relates to NSF's disagreement with the report's
characterization of a statement of work requirement for additional
technical assistance (worth $250,000, or 20 percent of the contract's
value) as a statement of work weakness. NSF added that all offerors
submitted proposals on the basis of the same requirement and competed
using the same dollar amount. While this latter statement is true,
it does not adequately address the issues involved. Offerors must be
given sufficient detail in a solicitation to enable them to compete
intelligently on requirements that are free from ambiguity.
Similarly, the statement of work contained in the solicitation should
describe the agency's minimum needs accurately. For example, in
University Research Corporation, B-216461, Feb. 19, 1985, 85-1 CPD �
210, we found that a statement of work that listed "other specific
programs" as its need was too general and vague to allow offerors to
submit intelligent proposals. In the NSF contract, the statement of
work asked for "additional technical support" services that had not
been clearly defined. Therefore, we maintain that our finding is
valid.
NSF also stated that its Inspector General, in an internal review,
agreed with the use of a technical service clause when it is limited
to a dollar amount or a certain percentage of the contract and that
it has issued guidance to that effect. Although the Inspector
General did agree that the use of technical service clauses is
appropriate in some instances, in that same report, the Inspector
General stated that " well-written procurement contracts have (1)
well-structured statements of work...." Also, the Inspector General
stated that "a contract with a loosely defined technical service
clause of substantial dollar value or that represents a significant
proportion of the contract's total cost does not meet basic
procurement standards." The Inspector General's report identified a
contract in its sample of contracts that it considered to have a
modest technical service amount of $10,000 (3 percent) and others
with dollar amounts and percents of contract value ranging from
$70,000 to $300,000 (14 to 38 percent, respectively) that the
Inspector General considered significant. By comparison, the
contract we reviewed falls into the latter or "significant"
category--$250,000, or 20 percent of the contract's value.
6. The comments object to our finding of weaknesses in NSF's process
for awarding noncompetitive contracts based solely on the National
Academy of Sciences' contracts. Also, the comments state that NSF's
justifications for sole-source awards to the Academy discuss the
unique capabilities of the Academy in detail and that we neglect to
indicate that NSF has a waiver from the synopsis requirement (public
notification of proposed contract actions and contract awards) for
awards made to the Academy. We agree that the uniqueness of the
Academy is discussed in NSF's justifications for other than full and
open competition to the Academy. However, merely discussing the
Academy's unique capabilities and status in the justifications does
not satisfy the legislative and regulatory requirements to document
whether other eligible sources are available and to describe what
actions NSF has taken, or plans to take, to remove barriers to
competition. In addition, the exemption from the synopsis
requirement to advertise procurements in the Commerce Business Daily
does not exempt the Academy contracts from the justification
requirements for sole-source procurements. The fact that the
Academy's charter prohibits it from competing for contracts has
nothing to do with the federal procurement requirements.
MAJOR CONTRIBUTORS TO THIS REPORT
========================================================= Appendix III
RESOURCES, COMMUNITY, AND ECONOMIC
DEVELOPMENT DIVISION, WASHINGTON,
D.C.
Jim Wells, Associate Director
Doris E. Cannon, Assistant Director
Robert M. Antonio, Assignment Manager
Carrie M. Stevens, Senior Evaluator-in-Charge
Karen E. Bracey, Assistant Director, Design, Methodology, and
Technical
Assistance
Sara Ann W. Moessbauer, Operations Research Analyst
OFFICE OF GENERAL COUNSEL
Michael R. Golden, Assistant General Counsel
David K. Hooper, Attorney Adviser