Elections: Federal Program for Certifying Voting Systems Needs to
Be Further Defined, Fully Implemented, and Expanded (16-SEP-08,
GAO-08-814).
The 2002 Help America Vote Act (HAVA) created the Election
Assistance Commission (EAC) and, among other things, assigned the
commission responsibility for testing and certifying voting
systems. In view of concerns about voting systems and the
important role EAC plays in certifying them, GAO was asked to
determine whether EAC has (1) defined an effective approach to
testing and certifying voting systems, (2) followed its defined
approach, and (3) developed an effective mechanism to track
problems with certified systems and use the results to improve
its approach. To accomplish this, GAO compared EAC guidelines and
procedures with applicable statutes, guidance, and best
practices, and examined the extent to which they have been
implemented.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-08-814
ACCNO: A84202
TITLE: Elections: Federal Program for Certifying Voting Systems
Needs to Be Further Defined, Fully Implemented, and Expanded
DATE: 09/16/2008
SUBJECT: Best practices
Certification and accreditation
Developmental testing
Elections
Federal regulations
Federal/state relations
Information management
Operational testing
Product evaluation
Program evaluation
Program management
Reports management
Requirements definition
Software verification and validation
Standards
Strategic planning
System software
Systems analysis
Systems evaluation
Systems testing
Testing
Voting
Voting systems
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-08-814
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to [email protected].
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Chairman, Committee on House Administration, House of
Representatives:
United States Government Accountability Office:
GAO:
September 2008:
Elections:
Federal Program for Certifying Voting Systems Needs to Be Further
Defined, Fully Implemented, and Expanded:
Federal Certification of Voting Systems:
GAO-08-814:
GAO Highlights:
Highlights of GAO-08-814, a report to the Chairman, Committee on House
Administration, House of Representatives.
Why GAO Did This Study:
The 2002 Help America Vote Act (HAVA) created the Election Assistance
Commission (EAC) and, among other things, assigned the commission
responsibility for testing and certifying voting systems. In view of
concerns about voting systems and the important role EAC plays in
certifying them, GAO was asked to determine whether EAC has (1) defined
an effective approach to testing and certifying voting systems, (2)
followed its defined approach, and (3) developed an effective mechanism
to track problems with certified systems and use the results to improve
its approach. To accomplish this, GAO compared EAC guidelines and
procedures with applicable statutes, guidance, and best practices, and
examined the extent to which they have been implemented.
What GAO Found:
EAC has defined an approach to testing and certifying voting systems
that follows a range of relevant practices and statutory requirements
associated with a product certification program, including those
published by U.S. and international standards organizations, and those
reflected in HAVA. EAC, however, has yet to define its approach in
sufficient detail to ensure that certification activities are performed
thoroughly and consistently. This lack of definition also has caused
voting system manufacturers and test laboratories to interpret program
requirements differently, and the resultant need to reconcile these
differences has contributed to delays in certifying systems that
several states were intending to use in the 2008 elections. According
to EAC officials, these definitional gaps can be attributed to the
program�s youth and the commission�s limited resources being devoted to
other priorities. Nevertheless, they said that they intend to address
these gaps, but added that they do not yet have written plans for doing
so.
EAC has largely followed its defined approach for each of the dozen
systems it is in the process of certifying, with one major exception.
Specifically, it has not established an effective and efficient
repository for certified versions of voting system software, or related
procedures and tools, for states and local jurisdictions to use in
verifying that their acquired voting systems are identical to what EAC
has certified. Further, EAC officials told GAO that they do not have a
documented plan or requirements for a permanent solution. As an interim
solution, they stated that they will maintain copies of certified
versions in file cabinets and mail copies of these versions upon their
request by states and local jurisdictions. In GAO�s view, this process
puts states and local jurisdictions at increased risk of using a
version of a system during an election that differs from the certified
version.
Under its voting system testing and certification program, EAC has
broadly described an approach for tracking problems with certified
voting systems and using this information to improve its certification
program. While this approach is consistent with some aspects of
relevant guidance, key elements are either missing or inadequately
defined. According to EAC officials, while they intend to address some
of these gaps, they do not have documented plans for doing so. In
addition, even if EAC defines and implements an effective approach, it
would not affect the vast majority of voting systems that are to be
used in the 2008 elections. This is because the commission�s approach
only applies to those voting systems that it has certified, and it is
unlikely that any voting systems will be certified in time to be used
in the upcoming elections. Moreover, because most states do not
currently require EAC certification for their voting systems, it is
uncertain if this situation will change relative to future elections.
As a result, states and other election jurisdictions are on their own
to discover, disclose, and address any shared problems with these
noncertified systems.
What GAO Recommends:
GAO is making recommendations to EAC relative to establishing and
implementing plans to better define and implement its certification
program. GAO is also proposing that Congress consider expanding EAC�s
role under HAVA to include facilitating understanding and resolution of
shared problems with noncertified voting systems and providing it with
the resources to accomplish this. EAC stated that it generally agrees
with GAO�s conclusion that its certification program needs to improve
and that it accepts GAO�s recommendations. It also provided other
comments, some of which GAO used to clarify its findings, one
recommendation, and its proposal to amend HAVA.
To view the full product, including the scope and methodology, click on
[http://www.gao.gov/cgi-bin/getrpt?GAO-08-814]. For more information,
contact Randolph C. Hite at (202) 512-3439 or [email protected].
[End of section]
Contents:
Letter:
Results in Brief:
Background:
EAC's Approach Largely Follows Many Accepted Practices, but Needs to Be
Further Defined:
EAC Has Largely Followed Its Defined Certification Approach, but a Key
System Verification Capability to Support States and Local
Jurisdictions Is Not Yet in Place:
EAC's Mechanism for Tracking and Resolving Problems with Certified
Voting Systems Lacks Sufficient Definition, and Its Scope Is Limited:
Conclusions:
Recommendations for Executive Action:
Matter for Congressional Consideration:
Agency Comments and Our Evaluation:
Appendix I: Objectives, Scope, and Methodology:
Appendix II: Comments from the Election Assistance Commission:
Appendix III: GAO Contact and Staff Acknowledgments:
Table:
Table 1: EAC Certification Program Roles and Responsibilities:
Figures:
Figure 1: A Voting System Life Cycle Model:
Figure 2: EAC Voting System Testing and Certification Process:
Abbreviations:
EAC: Election Assistance Commission:
FEC: Federal Election Commission:
HAVA: Help America Vote Act:
IEC: International Electrotechnical Commission:
ISO: International Organization for Standardization:
NASED: National Association of State Election Directors:
NIST: National Institute of Standards and Technology:
NSRL: National Software Reference Library:
NVLAP: National Voluntary Laboratory Accreditation Program:
VSS: Voting Systems Standards:
VSTL: voting system test laboratory:
VVSG: Voluntary Voting System Guidelines:
United States Government Accountability Office:
Washington, DC 20548:
September 16, 2008:
The Honorable Robert A. Brady:
Chairman:
Committee on House Administration:
House of Representatives:
Dear Mr. Chairman:
Following the 2000 and 2004 general elections, we issued a series of
reports and testified on virtually every aspect of our nation's
election system, including the many challenges and opportunities
associated with various types of voting systems.[Footnote 1] In this
regard, we emphasized that the voting systems alone were neither the
sole contributor nor solution to the problems that were experienced
during the 2000 and 2004 elections, and that the overall election
system depended on the effective interplay of people, process, and
technology and involved all levels of government. Among other things,
we specifically reported in 2001 that no federal entity was responsible
for developing voting system standards and for testing and certifying
these systems against such standards, and we raised the establishment
of such an entity as a matter for congressional consideration.[Footnote
2]
Subsequently, Congress passed the Help America Vote Act (HAVA), which
created the Election Assistance Commission (EAC) and assigned it
responsibilities for, among other things, the testing, certification,
decertification, and recertification of voting system hardware and
software.[Footnote 3] In 2004, we testified on the challenges facing
EAC in meeting its responsibilities, adding that the commission's
ability to meet these challenges depended in part on having adequate
resources. In 2007, EAC established and began implementing its voting
system certification program. In view of the continuing concerns about
voting systems and the important role the commission plays in
certifying them, you asked us to determine whether EAC has (1) defined
an effective approach to testing and certifying voting systems, (2)
followed its defined approach, and (3) developed an effective mechanism
to track problems with certified systems and use the results to improve
its certification program.
To accomplish this, we reviewed EAC policies, procedures, and standards
for testing, certifying, decertifying, and recertifying voting systems
and compared them, as appropriate, with applicable statutory
requirements and leading practices, such as HAVA and guidance published
by the National Institute of Standards and Technology (NIST),
International Organization for Standardization (ISO), and International
Electrotechnical Commission (IEC). We also compared EAC actions and
artifacts for executing the voting system certification program with
its policies, guidelines, and procedures. In addition, we interviewed
officials from EAC, NIST, voting system test laboratories (VSTL),
voting system manufacturers, and the National Association of State
Election Directors (NASED).[Footnote 4]
We conducted this performance audit at EAC offices in Washington, D.C.,
from September 2007 to September 2008 in accordance with generally
accepted government auditing standards. Those standards require that we
plan and perform the audit to obtain sufficient, appropriate evidence
to provide a reasonable basis for our findings and conclusions based on
our audit objectives. We believe that the evidence obtained provides a
reasonable basis for our findings and conclusions based on our audit
objectives. Further details of our objectives, scope, and methodology
are included in appendix I.
Results in Brief:
EAC has defined an approach to testing and certifying voting systems
that follows statutory requirements reflected in HAVA and a range of
relevant practices associated with a product certification program,
including those published by U.S. and international standards
organizations. In particular, its voting system certification program
defines key roles and responsibilities; describes the steps that voting
system manufacturers are to follow; provides for testing to be
performed by accredited VSTLs; provides for interpretation of voting
system standards; includes mechanisms for VSTLs and EAC reviewers to
declare their impartiality and independence; and addresses how
manufacturer complaints, appeals, and disputes will be handled.
Nevertheless, the commission has not developed its approach in
sufficient detail to ensure that its certification activities are
performed thoroughly and consistently. For example, it has not defined
procedures or specific criteria for many of its review activities,
including the specific test procedures to be used and how decisions are
to be documented. According to EAC officials, these gaps exist because
the program is still new and evolving and resources are focused on
other commission priorities. Officials further stated that they intend
to carry out a number of activities to address these gaps; however,
they said that written plans for doing so are not yet developed. Until
these definitional gaps are fully addressed, EAC cannot adequately
ensure that its certification approach is repeatable and verifiable
across all manufacturers and systems. Moreover, these gaps have already
led to different interpretations of program requirements among EAC's
stakeholders, and the resultant need to reconcile differences has
contributed to delays in certifying systems that several states had
intended to use in the 2008 elections.
EAC has largely followed its voting system testing and certification
approach as defined. For example, it has received and reviewed
manufacturer registration applications, system certification
applications, and system test plans, as well as issued notices of
noncompliance. However, contrary to its own certification program
requirements, the commission has yet to establish a sufficient
mechanism for states and local jurisdictions to use in verifying that
the voting systems that they receive from manufacturers are identical
to the ones that were tested and certified. Specifically, EAC has yet
to designate a repository of trusted software versions for certified
systems, and it has not established plans or time frames for doing so.
Instead, EAC officials stated that they plan to store these software
builds on compact disks and place the disks in fireproof filing
cabinets in their offices, and mail copies of the disks to those states
and local jurisdictions that request them, until they can develop a
permanent solution. However, the commission has no activities planned
or under way to establish one. In addition, EAC has yet to specify
exactly what needs to be done to ensure that manufacturers provide
effective and efficient tools and processes to support states and local
jurisdictions in using the repository to verify that their respective
systems match the certified version, and has not developed plans for
ensuring that this occurs. Until such tools are in place, state and
local election officials are likely to face difficulty in determining
whether the systems that they receive from a manufacturer are identical
to the system that was tested and certified.
EAC has broadly described an approach to allow it to track and resolve
problems with certified voting systems and use the information about
these problems to improve its voting system testing and certification
program. Under this approach, EAC is to investigate reports alleging
defects with certified voting systems, and take action by, for example,
issuing a notice of noncompliance, providing the manufacturer with an
opportunity to correct the noncompliance, and decertifying a system if
the manufacturer does not take sufficient action to bring the system
into compliance. In addition, EAC is to periodically inspect the
production facilities of voting system manufacturers. However, it has
yet to specify how this broadly defined approach will be executed,
including what steps will be followed, what criteria will be used to
reach decisions, how it will know if system problems have been
corrected, and how it will use the information to improve its testing
and certification program. In addition, while EAC officials stated that
they intend to address some of these gaps, they do not have defined
plans or time frames for doing so. According to the EAC officials, this
is because the certification program is still new and other priorities
have consumed the commission's limited resources. Until these
definitional limitations are addressed, it is unlikely that EAC will be
able to effectively track and resolve problems that arise with the
systems that it has certified.
Even if EAC defines and implements an effective problem tracking and
resolution process for certified systems, these actions will not affect
the vast majority of voting systems that are to be used in the 2008
elections, and it is uncertain when this situation will change relative
to future elections. This is because of two factors. First, most of the
systems to be used will not be systems that were certified by EAC, but
instead will be systems that were qualified by the now-discontinued
NASED program, or those that were not endorsed by any national entity.
Second, HAVA does not explicitly assign EAC any responsibility for non-
EAC-certified voting systems, and, according to commission officials,
they do not have the authority or resources to identify and address
problems with these systems in the same manner that they do for EAC-
certified systems. Although EAC has established a mechanism, under its
HAVA-required information clearinghouse responsibility, to post state
and local officials' reports of problems and experiences with non-EAC-
certified systems on its Web site, this mechanism does not involve any
EAC actions to facilitate problem understanding or resolution.
Notwithstanding the EAC's efforts to establish an effective voting
systems testing and certification program, more can be done to build on
its existing program to be more effective. Accordingly, we are making
recommendations to the Chair of the EAC to develop and execute plans to
(1) establish detailed procedures, review criteria, and documentation
requirements to ensure that voting system testing and certification
reviews are conducted thoroughly, consistently, and verifiably; (2)
establish an accessible and available software repository for its
certified systems, and procedures and review criteria for manufacturer-
provided tools for using the repository; and (3) fully define and
implement certified voting system problem tracking and resolution
activities, and apply lessons learned to improve the program. We are
also providing a matter for congressional consideration aimed at
expanding EAC's role under HAVA such that, consistent with the
nonregulatory and voluntary nature of its certification program, the
commission is assigned responsibility for providing resources and
services to facilitate understanding and resolution of problems with
non-EAC-certified voting systems that will be used in future elections,
and provided with the resources needed to accomplish this.
EAC provided written comments on a draft of this report, signed by the
EAC Executive Director and reprinted in their entirety in appendix II.
In its comments, the commission described our review and report as
helpful as it works to fully implement and improve its voting system
certification program. It also stated that it agrees with the report's
conclusions that more can be done to build on the program and ensure
that certifications are based on consistently performed reviews.
Further, it stated that it generally accepts our three recommendations,
adding that it will work hard to implement them. EAC provided
additional comments on the findings that underlie each of the
recommendations, which it described as needed to clarify and avoid
confusion about some aspects of its certification program. In response
to these comments, we have modified the report, as appropriate, to
clarify our findings and one of the recommendations.
The commission also provided comments on our matter for congressional
consideration, including characterizing it as intending "to affect a
sea change in the way that EAC operates its testing and certification"
program. We agree that EAC does not have the authority to compel the
manufacturers or states and local jurisdictions to submit to its
testing and certification program and that the wording of the matter in
our draft inadvertently led EAC to believe that our proposal would
require the commission to assume a more regulatory role. In response to
EAC's comments, and in order to avoid any misunderstanding as to the
intent of this matter for congressional consideration, we have modified
its wording.
Background:
All levels of government share responsibility in the overall U.S.
election system. At the federal level, Congress has authority under the
Constitution to regulate presidential and congressional elections and
to enforce prohibitions against specific discriminatory practices in
all federal, state, and local elections. Congress has passed
legislation that addresses voter registration, absentee voting,
accessibility provisions for the elderly and handicapped, and
prohibitions against discriminatory practices.[Footnote 5] At the state
level, individual states are responsible for the administration of both
federal elections and their own elections. States regulate the election
process, including, for example, the adoption of voluntary voting
system guidelines, the state certification and acceptance testing of
voting systems, ballot access, registration procedures, absentee voting
requirements, the establishment of voting places, the provision of
election day workers, and the counting and certification of the vote.
In total, the overall U.S. election system can be seen as an assemblage
of 55 distinct election systems--those of the 50 states, the District
of Columbia, and the 4 U.S. territories. Further, although election
policy and procedures are legislated primarily at the state level,
states typically decentralize election administration, so that it is
carried out at the city or county levels, and voting is done at the
local level. As we reported in 2001, local election jurisdictions
number more than 10,000, and their sizes vary enormously--from a rural
county with about 200 voters to a large urban county, such as Los
Angeles County, where the total number of registered voters for the
2000 elections exceeded the registered voter totals in 41
states.[Footnote 6] Further, these thousands of jurisdictions rely on
many different types of voting methods that employ a wide range of
voting system makes, models, and versions.
The U.S. Election System Depends on Effective Interactions among
People, Processes, and Technology:
Voting systems are but one facet of a multifaceted, continuous election
system that involves the interplay of people, processes, and
technology. All levels of government, as well as commercial voting
system manufacturers and VSTLs, play key roles in ensuring that voting
systems perform as intended.
Electronic voting systems are typically developed by manufacturers,
purchased as commercial off-the-shelf products, and operated by state
and local election administrators. These activities can be viewed as
three phases in a system's life cycle: product development,
acquisition, and operations (see fig. 1). Spanning these life cycle
phases are key processes, including managing the interplay of people,
processes, and technologies, and testing the systems and components. In
addition, voting system standards are important through all of these
phases because they provide the criteria for developing, testing, and
acquiring the systems, and they specify the necessary documentation for
operating the systems. We discuss each of these phases after figure 1.
Figure 1: A Voting System Life Cycle Model:
This figure is a flowchart showing a voting system life cycle mode.
[See PDF for image]
Source: GAO analysis of EAC, NIST, and Institute of Electrical and
Electronics Engineers (IEEE) publications.
[End of figure]
* The product development phase includes such activities as
establishing requirements for the system, designing a system
architecture, and developing software and integrating components.
Activities in this phase are performed by the system manufacturer.
* The acquisition phase includes such procurement-related activities as
publishing a solicitation, evaluating offers, choosing a voting
technology, choosing a vendor, and awarding and administering
contracts. Activities in this phase are primarily the responsibility of
state and local governments, but include responsibilities that are
shared with the vendor, such as establishing contracts.
* The operations phase consists of such activities as ballot design and
programming, setup of systems before voting, pre-election testing, vote
capture and counting during elections, recounts and system audits after
elections, and storage of systems between elections. Responsibility for
activities in this phase typically resides with local jurisdictions,
whose officials may, in turn, rely on or obtain assistance from system
vendors for aspects of these activities.
* Standards for voting systems were developed at the national level by
the Federal Election Commission (FEC) in 1990 and 2002 and were updated
by EAC in 2005. Voting system standards serve as guidance for product
developers in building systems, a framework for state and local
governments to evaluate systems, and as the basis for documentation
needed to operate the systems.
* Testing processes are conducted throughout the life cycle of a voting
system. For example, manufacturers conduct product testing during
development of the system. Also, national certification testing of
products submitted by system manufacturers is conducted by nationally
accredited VSTLs. States and local jurisdictions also perform a range
of system tests.
* Management processes help to ensure that each life cycle phase
produces desirable outcomes. Typical management activities include
planning, configuration management, system performance review and
evaluation, problem tracking and correction, human capital management,
and user training.
Testing and Certification Are Important to Ensuring Voting System
Security and Reliability:
Testing electronic voting systems for conformance with requirements and
standards is critical to ensuring their security and reliability, and
an essential means to ensuring that systems perform as intended. In
addition, such testing can help find and correct errors in systems
before they are used in elections. If done properly, testing provides
voters with assurance and confidence that their voting systems will
perform as intended.
Testing is particularly important for electronic voting systems because
these systems have become our Nation's predominant method of voting,
and concerns have been raised about their security and reliability. As
we reported in 2005,[Footnote 7] these concerns include weak security
controls, system design flaws, inadequate system version control,
inadequate security testing, incorrect system configuration, poor
security management, and vague or incomplete voting system standards.
Further, security experts and some election officials have expressed
concerns that tests performed under the NASED program by independent
testing authorities and state and local election officials did not
adequately assess voting systems' security and reliability. Consistent
with these concerns, most of the security weaknesses that we identified
in our prior report[Footnote 8] related to systems that NASED had
previously qualified. Our report also recognized that security experts
and others pointed to these weaknesses as an indication that both the
standards and the NASED testing program were not rigorous enough with
respect to security, and that these concerns were amplified by what
some described as a lack of transparency in the testing process.
EAC's Responsibilities under HAVA include Certifying Voting Systems:
Enacted in October 2002, HAVA affects nearly every aspect of the
election system, from voting technology to provisional ballots and from
voter registration to poll worker training.[Footnote 9] Among other
things, the act authorized $3.86 billion in funding over several fiscal
years for states to replace punch card and mechanical lever voting
equipment, improve election administration and accessibility, and
perform research and pilot studies. In addition, the act established
EAC and assigned it responsibility for, among other things, (1)
updating voting system standards, (2) serving as a clearinghouse for
election-related information, (3) accrediting independent test
laboratories, and (4) certifying voting systems. EAC began operations
in January 2004. In 2004, we testified on the challenges facing EAC in
meeting its responsibilities.[Footnote 10] For example, we reported
that EAC needed to move swiftly to strengthen voting system standards
and the testing associated with these standards. We also reported that
the commission's ability to meet its responsibilities depended, in
part, on the adequacy of the resources at its disposal.
Updating standards: HAVA requires EAC to adopt a set of federal voting
system standards, referred to as the Voluntary Voting System Guidelines
(VVSG). In December 2005, the commission adopted the VVSG, which
defines a set of specifications and requirements against which voting
systems are to be designed, developed, and tested to ensure that they
provide the functionality, accessibility, and security capabilities
required to ensure the integrity of voting systems. As such, the VVSG
specifies the functional requirements, performance characteristics,
documentation requirements, and test evaluation criteria for the
national certification of voting systems. In 2007, the Technical
Guidelines Development Committee[Footnote 11] submitted its
recommendations for the next iteration of the VVSG to EAC. The
commission has yet to establish a date when the update will be approved
and issued.
Serving as an information clearinghouse: HAVA requires EAC to maintain
a clearinghouse of information on the experiences of state and local
governments relative to, among other things, implementing the VVSG and
operating voting systems. As part of this responsibility, EAC posts
voting system reports and studies that have been conducted or
commissioned by a state or local government on its Web site. These
reports must be submitted by a state or local government that certifies
that the report reflects its experience in operating a voting system or
implementing the VVSG. EAC does not review the information for quality
and does not endorse the reports and studies.
Accrediting independent test laboratories: HAVA assigned
responsibilities for laboratory accreditation to both EAC and NIST. In
general, NIST focuses on assessing laboratory technical qualifications
and recommends laboratories to EAC for accreditation. EAC uses NIST's
assessment results and recommendations, and augments them with its own
review of related laboratory capabilities to reach an accreditation
decision. As we have previously reported,[Footnote 12] EAC and NIST
have defined their respective approaches to accrediting laboratories
that address relevant HAVA requirements. However, neither approach
adequately defines all aspects of an effective program to the degree
needed to ensure that laboratories are accredited in a consistent and
verifiable manner. Accordingly, we recently made recommendations to
NIST and EAC aimed at addressing these limitations.
Certifying voting systems: HAVA requires EAC to provide for the
testing, certification, decertification, and recertification of voting
system hardware and software. EAC's voting system testing and
certification program is described in detail in the following section.
Prior to HAVA, no federal agency was assigned or assumed responsibility
for testing and certifying voting systems against the federal
standards. Instead, NASED, through its voting systems committee,
assumed this responsibility by accrediting independent test
authorities, which in turn tested equipment against the standards. When
testing was successfully completed, the independent test authorities
notified NASED that the equipment satisfied testing requirements. NASED
would then qualify the system for use in elections. According to a
NASED official, the committee has neither qualified any new or modified
systems, nor taken any actions to disqualify noncompliant systems,
since the inception of EAC's testing and certification program in
January 2007.
Overview of EAC's Voting System Testing and Certification Program:
EAC implemented its voting system testing and certification program in
January 2007. According to the commission's Testing and Certification
Program Manual, EAC certification means that a voting system has been
successfully tested by an accredited VSTL, meets requirements set forth
in a specific set of federal voting system standards, and performs
according to the manufacturer's specifications.
The process of EAC's voting system testing and certification program
consists of seven major phases. Key stakeholders that are involved in
this process include voting system manufacturers, accredited VSTLs, and
state and local election officials. These seven phases are described in
the following text and depicted in figure 2.
1. Manufacturer Registration:
All manufacturers must be registered to submit a voting system for
certification. To register, a manufacturer must provide such
information as organizational structure and contact(s); quality
assurance, configuration management, and document retention procedures;
and identification of all manufacturing and assembly facilities. In
registering, the manufacturer agrees to certain duties and requirements
at the outset of its participation in the program. These requirements
include properly using and representing EAC's certification label,
notifying the commission of any changes to a certified system,
permitting EAC to verify the manufacturer's quality control procedures
by inspecting fielded systems and manufacturing facilities, cooperating
with any inquiries and investigations about certified systems,
reporting any known malfunction of a system, and otherwise adhering to
all procedural requirements of the program manual.
Once a manufacturer submits a completed application form and all
required attachments, EAC reviews the submission for sufficiency using
a checklist that maps to the application requirements listed in the
program manual. If the application passes the review, EAC provides the
manufacturer with a unique identification code and posts the applicant
as a registered manufacturer on the commission's Web site, along with
relevant documentation.
2. Voting System Certification Application:
For each voting system that a manufacturer wishes to have certified, it
submits an application package. The package includes an application
form requiring the following: manufacturer information, accredited VSTL
selection, applicable voting system standard(s), nature of submission,
system name and version number, all system components and corresponding
version numbers, and system configuration information. The package also
includes the following documentation: system implementation
statement,[Footnote 13] functional diagram, and system overview. EAC
reviews the submission for completeness and accuracy and, if it is
acceptable, notifies the manufacturer and assigns a unique application
number to the system.
3. Test Plan Development and Approval:
Once the certification application is accepted, the accredited VSTL
prepares and submits to EAC a test plan defining how it will ensure
that the system meets applicable standards and functions as intended.
When a laboratory submits its test plan, EAC's technical reviewers
assess it for adequacy. If the plan is deemed not acceptable, the
commission provides written notice to the laboratory that includes a
description of the problems identified and the steps required to remedy
the test plan. The laboratory may take remedial action and resubmit the
test plan until it is accepted by EAC reviewers.
4. Test Plan Execution:
The VSTL executes the approved test plan and notifies EAC directly of
any test anomalies or failures, along with any changes or modifications
to the test plan as a result of testing. The laboratory then prepares a
test results report.
5. Test Report Review:
The VSTL submits the test results report to EAC's Program Director who
reviews it for completeness. If it is complete, the technical reviewers
analyze the report in conjunction with related technical documents and
the test plan for completeness, appropriateness, and adequacy. The
reviewers submit their findings to the Program Director, who either
recommends certification of the system to the Decision Authority, EAC's
Executive Director, or refers the matter back to the reviewers for
additional specified action and resubmission.
6. Initial Certification Decision and Appeal:
EAC's Decision Authority reviews the recommendation of the Program
Director and supporting materials and issues a written decision to the
manufacturer. If certification is denied, the manufacturer may request
an opportunity to correct the basis for the denial or may request
reconsideration of the decision after submitting supporting written
materials, data, and a rationale for its position. The Decision
Authority considers the request and issues a written decision. If the
decision is to deny certification, the manufacturer may request an
appeal in writing to the Program Director. The Appeal Authority, which
consists of two or more EAC Commissioners or other individuals
appointed by the Commissioners who have not previously served as the
initial or reconsideration authority, consider the appeal. The Appeal
Authority may overturn the decision if it finds that the manufacturer
has demonstrated by clear and convincing evidence that its system met
all substantive and procedural requirements for certification.
7. Final Certification:
The initial decision becomes final and EAC issues a Certificate of
Conformance to the manufacturer, and posts the system on the list of
certified voting systems on its Web site, when the manufacturer and
VSTL successfully demonstrate that the voting system under test has
been:
* Subject to a trusted build: The voting system's source code is
converted to executable code in the presence of at least one VSTL
representative and one manufacturer representative, using security
measures to ensure that the executable code is a verifiable and
faithful representation of the source code. This demonstrates that (1)
the software was built as described in the technical documentation, (2)
the tested and approved source code was actually used to build the
executable code on the system, and (3) no other elements were
introduced in the software build. It also serves to document the
configuration of the certified system for future reference.
* Placed in a software repository: The VSTL delivers the following to
one or more trusted repositories designated by EAC: (1) source code
used for the trusted build and its file signatures;[Footnote 14] (2)
disk image of the prebuild, build environment, and any file signatures
to validate that it is unmodified; (3) disk image of the postbuild,
build environment, and any file signatures to validate that it is
unmodified; (4) executable code produced by the trusted build and its
file signatures of all files produced; and (5) installation device(s)
and its file signatures.
* Verified using system identification tools: The manufacturer creates
and makes available system identification tools that federal, state,
and local officials can use to verify that their voting systems are
unmodified from the system that was certified. These tools are to
provide the means to identify and verify hardware and
software.[Footnote 15]
Figure 2: EAC Voting System Testing and Certification Process:
This figure is a flowchart of the EAC voting system testing and
certification process.
[See PDF for image]
Source: GAO analysis of EAC data.
Note: At the review and approval of manufacturer registration
applications, voting system certification applications, and test plans,
EAC may request additional clarification or information before
approval.
[End of figure]
EAC's Approach Largely Follows Many Accepted Practices, but Needs to Be
Further Defined:
To its credit, EAC has taken steps to develop an approach to testing
and certifying voting systems that follows statutory requirements and
many recognized and accepted practices. However, the commission has not
developed its approach in sufficient detail to ensure that its
certification activities are performed thoroughly and consistently. It
has not, for example, defined procedures or specific criteria for many
of its review activities, and for ensuring that the decisions made, and
their basis, are properly documented. According to EAC officials, these
gaps exist because the program is still new and evolving and resources
are limited. Officials further stated that they do not yet have written
plans for addressing these gaps. Until these gaps are addressed, EAC
cannot adequately ensure that its approach is repeatable and verifiable
across all manufacturers and systems. Moreover, this lack of definition
has caused EAC stakeholders to interpret certification requirements
differently, and the resultant need to reconcile these differences has
contributed to delays in certifying systems that several states were
planning on using in the 2008 elections.
EAC's Approach Reflects Key Practices and Statutory Requirements:
Product certification or conformance testing is a means by which a
third party provides assurance that a product conforms to specific
standards. In the voting environment, EAC is the third party that
provides assurance to the buyer (e.g., state or local jurisdictions)
that the manufacturer's voting system conforms to the federal voting
standards set forth in FEC's 2002 Voting System Standards (VSS) or
EAC's 2005 VVSG.
Several organizations, such as NIST, ISO, and IEC, have individually or
jointly developed guidance for product certification and conformance
testing programs.[Footnote 16] This guidance includes, among other
things, (1) defining roles and responsibilities for all parties
involved in the certification process, (2) defining a clear and
transparent process for applicants to follow, (3) ensuring that persons
involved in the process are impartial and independent, (4) establishing
a process for handling complaints and appeals, and (5) having testing
conducted by competent laboratories. Further, HAVA established
statutory requirements for a federal testing and certification program.
These requirements include ensuring that the program covers testing,
certification, decertification, and recertification of voting system
hardware and software.
EAC's defined voting system certification approach reflects these key
practices. Specifically:
* EAC has defined the roles and responsibilities for itself, the VSTLs,
and manufacturers in its Testing and Certification Program Manual.
These roles and responsibilities are described in table 1.
Table 1: EAC Certification Program Roles and Responsibilities:
Key players: EAC;
Roles and responsibilities: * Establish and maintain the testing and
certification program;
* Be the authority for granting and withdrawing certification;
* Provide the procedural requirement of the program;
* Review manufacturer registration applications for sufficiency;
* Review certification applications for completeness and accuracy;
* Provide technical guidance;
* Perform technical reviews of test plans, test reports, and other
technical documents;
* Resolve disputes about voting system standards and program
administration.
Key players: VSTLs;
Roles and responsibilities: * Prepare test plans and test
methodologies;
* Conduct testing to approved test plans;
* During testing, report any changes to approved test plans and all
test failures or anomalies;
* Record results of testing in a test report;
* Meet all requirements of EAC's laboratory accreditation program.
Key players: Manufacturers;
Roles and responsibilities: * Provide required information to EAC;
* Adhere to program requirements;
* Submit voting system to EAC for certification;
* Report malfunctions with certified voting systems and submit
corrective action plans to address any issues that violate EAC's
program.
Source: GAO analysis of EAC documentation.
[End of table]
* EAC's testing and certification process is documented in its program
manual. Among other things, the manual clearly defines the program's
administrative requirements that manufacturers and VSTLs are to follow.
EAC has made the program manual, along with supporting policies and
clarifications, publicly available on its Web site, and has made
program-related news and correspondence publicly accessible as they
have come available.
* EAC's certification program addresses impartiality and independence.
For example, EAC policy states that all personnel and contractors
involved in the certification program are subject to conflict-of-
interest reporting and review. In addition, the policy mandates
conflict-of-interest and conduct statements for the technical reviewers
that support the program, requires conflict-of-interest reporting and
reviews to ensure the independence of EAC personnel assigned to the
program, and requires that all VSTLs maintain and enforce policies that
prevent conflict-of-interest or the appearance of a conflict-of-
interest, or other prohibited practices.
* EAC's program manual outlines its process for the resolution of
complaints, appeals, and disputes received from manufacturers and
laboratories. These can be about matters relating to the certification
process, such as test methods, procedures, test results, or program
administration. Specifically, the program manual contains policies and
procedures for submitting a Request for Interpretation, which is a
means by which a registered manufacturer or accredited laboratory seeks
clarification on a specific voting system standard, including any
misunderstandings or disputes about a standard's interpretation or
implementation. The manual also contains policies, requirements, and
procedures for a manufacturer to file an appeal on a decision denying
certification, request an opportunity to correct a problem, and request
reconsideration of a decision. In addition, EAC provides for Notices of
Clarification, which offer guidance and explanation on the requirement
and procedure of the program. Notices may be issued pursuant to a
clarification request from a laboratory or manufacturer. EAC may also
issue a notice or interpretation if it determines that any general
clarifications are necessary.
* EAC has a VSTL accreditation program. This program is supported by
NIST's National Voluntary Laboratory Accreditation Program (NVLAP),
which is a long established and recognized laboratory accreditation
program. According to EAC's program manual, all certification testing
is to be performed by a laboratory accredited by NIST and EAC. Further,
all subcontracted testing is to be performed by a laboratory accredited
by either NIST or the American Association of Laboratory Accreditation
for the specific scope of needed testing. Finally, any prior testing
will only be accepted if it was conducted or overseen by an accredited
laboratory and was reviewed and approved by EAC.
HAVA also established certain requirements for EAC's voting system
testing and certification program.[Footnote 17] Under HAVA, EAC is to
provide for the testing, certification, decertification, and
recertification of voting system hardware and software by accredited
laboratories. EAC's defined approach addresses each of these areas.
According to program officials, EAC's certification program reflects
many leading practices because the commission consciously sought out
these best practices during program development. Officials stated that
their intention is to develop a program that stringently tests voting
systems to the applicable standards; therefore, they consulted with
experts to assist with drafting the program manual. For example,
according to EAC officials, they met with officials from other federal
agencies that conduct certification testing in order to benefit from
their lessons learned. By reflecting relevant practices, standards, and
legislative requirements in its defined approach, EAC has provided an
important foundation for having an effective voting system testing and
certification program.
EAC's Approach Does Not Sufficiently Define the Details for
Certifications to Be Performed Thoroughly and Consistently:
EAC has yet to define its approach for testing and certifying
electronic voting systems in sufficient detail to ensure that its
certification activities are performed thoroughly and consistently. It
has not, for example, defined procedures or specific criteria for many
of its review activities and for ensuring that the decisions made are
properly documented. EAC officials attributed this lack of definition
to the fact that the program is still new and evolving, and they stated
that available resources are constrained by competing priorities. Until
these details are defined, EAC will be challenged to ensure that
testing and review activities are repeatable across different systems
and manufacturers, and that the activities it performs are verifiable.
Moreover, this lack of definition is likely to result in different
interpretations of program requirements by stakeholders, which has
already resulted in the need to reconcile different interpretations and
thereby caused delays in certifying systems that several states
intended to use in the 2008 elections. In such cases, the delays are
forcing states to either not require EAC certification or rely on an
alternative system.
According to federal and international guidance,[Footnote 18] having
well-defined and sufficiently detailed program management controls help
to ensure that programs are executed effectively and efficiently.
Relative to a testing and certification program, such management
controls include, among other things, having (1) defined procedures and
established criteria for performing evaluation activities so that they
will be performed in a comparable, unambiguous, and repeatable manner
for each system and (2) required documentation to demonstrate that
procedural evaluation steps and related decisions have been effectively
performed, including provisions for review and approval of such
documentation by authorized personnel.
EAC's defined approach for voting system testing and certification
lacks such detail and definition. With respect to the first management
control, the commission has not defined procedures or specific criteria
for many of its review activities, instead it relies on the personal
judgment of the reviewers. Specifically, the program manual states that
EAC, with the assistance of its technical experts, as necessary, will
review manufacturer registration applications, system certification
applications, test plans, and test reports, but it does not define
procedures or criteria for conducting these reviews. For example:
* The program manual states that upon receipt of a completed
manufacturer registration application, EAC will review the information
for sufficiency. However, it does not define what constitutes
sufficiency and what this sufficiency review should entail. Rather, EAC
officials said that this is left up to the individual reviewer's
judgment.
* The program manual lists the information that manufacturers are
required to submit as part of their certification applications and
states that EAC will review the submission for completeness and
accuracy. While the commission has developed a checklist for
determining whether the required information was included in the
application, neither the program manual nor the checklist describe how
reviewers should perform the review or assess the adequacy of the
information provided. For example, EAC requires certification
applications to include a functional diagram depicting how the
components for the voting system function and interact, as well as a
system overview that includes a description of the functional and
physical interfaces between components. Although the checklist provides
for determining whether these items are part of the application
package, it does not, for example, provide for checking them for
completeness and consistency. Moreover, we identified issues with
completeness and consistency of these documents for approved
certification application packages. Again, EAC officials said that
these determinations are to be based on each reviewer's judgment.
* The program manual states that test plans are to be reviewed for
adequacy. However, it does not define adequacy or how such reviews are
to be performed. This lack of detail is particularly problematic
because EAC officials told us that the VSS and VVSG contain many vague
and undefined requirements. According to these officials, reviewers
have been directed to ensure that VSTLs stress voting systems during
testing, based on what they believe are the most difficult and
stringent conditions likely to be encountered in an election
environment that are permissible under the standards.
* The program manual states that EAC technical experts will assess test
results reports for completeness, appropriateness, and adequacy.
However, it does not define appropriateness and adequacy or the
procedural steps for conducting the review.
* The program manual requires VSTLs to use all applicable test suites
issued by EAC when developing test plans. However, program officials
stated that they currently do not have defined test suites and NIST
officials said that they are focused on preparing test suites for the
forthcoming version of the VVSG and not the 2005 version. As a result,
each laboratory develops its own unique testing approach, which
requires each to interpret what is needed to test for compliance with
the standards and increases the risk of considerable variability in how
testing is performed. To address this void, EAC has tasked NIST with
developing test suites for both the 2005 VVSG and the yet-to-be-
released update to these guidelines.[Footnote 19] Until then, EAC
officials acknowledge that they will be challenged to ensure that
testing conducted on different systems or at different VSTLs is
consistent and comparable.
With respect to the second management control, the commission has not
defined the documentation requirements that would demonstrate that
procedural steps, evaluations, and related decision making have been
performed in a thorough, consistent, and verifiable manner.
Specifically, while the program manual requires the Program Director to
maintain documentation to demonstrate that procedures and evaluations
were effectively performed, EAC has yet to specify the nature or
content of this documentation. For example:
* The program manual requires technical reviewers to assess test plans
and test reports prepared by laboratories and then to submit reports of
their findings to the Program Director. However, EAC does not require
documentation of how these reviews were performed beyond completion of
a recently developed checklist, and this checklist does not provide for
capturing how decisions were reached, including steps performed and
criteria applied. For example, the VVSG requires that systems permit
authorized access and prevent unauthorized access, and lists examples
of measures to accomplish this, such as computer-generated password
keys and controlled access security. While the checklist cites this
requirement and provides for the reviewer to indicate whether the test
plan satisfies it, it does not provide specific guidance on how to
determine whether the access control measures are adequate, and it does
not provide for documenting how the reviewer made such a decision.
* The program manual does not require supervisory review of work
conducted by EAC personnel. Moreover, it does not require that the
reviewers be identified.
According to EAC officials, its approach does not yet include such
details because it is still new and evolving and because the
commission's limited resources have been devoted to other
priorities.[Footnote 20] To address these gaps, EAC officials stated
that they intend to undertake a number of activities. For example, the
Program Director stated that in the near-term, the technical reviewers
will collaborate and share views on each test plan and test report
under review as a way to provide consistency, and they will use a
recently finalized checklist for test plan and test report reviews. In
the longer term, EAC intends to define more detailed procedures for
each step in its process. However, it has yet to establish documented
plans, including the level of resources needed to accomplish this.
Until these plans are developed and executed, EAC will be challenged to
ensure that its testing and certification activities are performed
thoroughly and consistently across different systems, manufacturers,
and VSTLs.
Moreover, this lack of program definition surrounding certification
testing and review requirements has already caused, and could continue
to cause, differences in how EAC reviewers, VSTLs, and manufacturers
interpret the requirements. For example, the program requires
sufficient and adequate testing, but it does not define what
constitutes sufficient and adequate. As a result, laboratory officials
and manufacturer representatives told us that EAC reviewers have
interpreted these requirements more stringently than they have, and
that reconciling these different interpretations has already caused
delays in the approval of test plans,[Footnote 21] and they will likely
prevent EAC from certifying any systems in time for use in the upcoming
2008 elections. This is especially problematic for those states that
have statutory or other requirements to use federally certified systems
and that have a need to acquire or upgrade existing systems for these
elections. In this regard, 18 states reported to us that they were
relying on EAC to certify systems for use in the 2008 elections, but
now will have to adopt different strategies for meeting their states'
respective EAC certification requirements. For example, officials for
several states said that they would use the same system as in 2006,
while officials for other states described plans to either undertake
upgrades to existing systems without federal certification, or change
state requirements to no longer require EAC certification.
EAC Has Largely Followed Its Defined Certification Approach, but a Key
System Verification Capability to Support States and Local
Jurisdictions Is Not Yet in Place:
EAC has largely followed its defined certification approach for each of
the dozen voting systems that it is in the process of certifying, with
one major exception. Specifically, it has not established sufficient
means for states and local jurisdictions to verify that the voting
systems that each receives from its manufacturer have system
configurations that are identical to those of the system that the EAC
certified, and it has not established plans or time frames for doing
so. This means that states and local jurisdictions are at increased
risk of using a version of a system during an election that differs
from the certified version. This lack of an effective and efficient
verification capability could diminish the value of an EAC system
certification.
EAC Has Largely Followed Its Defined Certification Approach:
EAC has largely executed its voting system testing and certification
program as defined. While no system has yet to complete all major steps
of the certification process discussed in the Background section of
this report, and thus receive certification, 12 different voting
systems have completed at least the first step of the process, and some
have completed several steps. Specifically, as of May 2008, EAC had
received, reviewed, and approved 12 manufacturer registration
applications, and these approved manufacturers have collectively
submitted certification applications for 12 different systems, of which
EAC has accepted 9. For these 9 systems, manufacturers have submitted 7
test plans, of which EAC has reviewed and approved 2 plans. In 1 of
these 2 cases, EAC has received a test results report, which it is
currently in the process of reviewing. At the same time, EAC has
responded to 8 requests for interpretations of standards from
manufacturers and laboratories. EAC has also issued 6 notices of
clarification, which provide further guidance and explanation on the
program requirements and procedures.
Our analysis of available certification-related documentation showed
that in each of the 12 systems submitted for certification, all
elements of each executed step in the certification process were
followed. With respect to the manufacturer registration step, EAC
reviewed and approved all 12 applications, as specified in its program
manual. For the certification application step, EAC reviewed and
approved 9 applications. In doing so, EAC issued 3 notices of
noncompliance to manufacturers for failure to comply with program
requirements. In each notice, it identified the area of noncompliance,
and described what requested relevant information or corrective
action(s) was needed in order to participate in the program. In 1 of
the cases, EAC terminated the certification application due to the
manufacturer's failure to respond within the established time frame,
which is consistent with the program manual. These actions were
generally consistent with its program manual.
EAC Has Not Established a Sufficient Means for States and Local
Jurisdictions to Verify That Their Systems Match the Certified Version:
Notwithstanding EAC's efforts to follow its defined approach, it has
not yet established a sufficient mechanism for states and local
jurisdictions to use in verifying that the voting systems that they
receive from manufacturers for use in elections are identical to the
systems that were actually tested and certified. According to EAC's
certification program manual, final certification is conditional upon
(1) testing laboratories depositing certified voting system software
into an EAC-designated repository and (2) manufacturers creating and
making available system identification tools for states and local
jurisdictions to use in verifying that their respective systems'
software configurations match that of the software for the system that
was certified and deposited into the repository. However, EAC has yet
to establish a designated repository or procedures and review criteria
for evaluating the manufacturer-provided tools, and has not established
plans or time frames for doing so. While none of the ongoing system
certifications have progressed to the point where these aspects of
EAC's defined approach is applicable, they will be needed when the
first system's test results are approved. Until both aspects are in
place, state and local officials will likely face difficulties in
determining whether the systems that they receive from manufacturers
are the same as the systems that EAC certified.
EAC's program requires the use of a designated software repository for
certified voting systems. Specifically, the certification program
manual states that final certification will be conditional upon, among
other things, the manufacturer and VSTL creating and documenting a
trusted software build,[Footnote 22] and the laboratory depositing the
build in a designated repository. In its 2005 VVSG, EAC designated
NIST's National Software Reference Library (NSRL) as its repository and
required its use. However, program officials stated that the commission
does not intend to use the NSRL as its designated repository because
the library cannot perform all required functions. While these
officials added that they may use the NSRL for portions of the
certification program, they said it will not serve as EAC's main
repository.
Nevertheless, the commission has not established plans to identify and
designate another repository, and has yet to define minimum
requirements (functional, performance, or interface) for what it
requires in a repository to efficiently support states and local
jurisdictions. As an interim measure, an EAC official stated that they
will store the trusted builds on compact disks and keep the disks in
fireproof filing cabinets in their offices. According to the official,
this approach is consistent with established program requirements
because the program manual merely refers to the use of a trusted
archive or repository designated by EAC. Under this measure, state and
local election officials will have to request physical copies of
material from EAC and wait for the materials to be physically packaged
and delivered. This interim approach is problematic for several
reasons, including the demand for EAC resources to keep up with the
requests during a time when the Executive Director told us that a more
permanent repository solution has not been a commission focus because
its limited resources have been focused on other priorities.
EAC has also not defined how it will ensure that manufacturers develop
and provide to states and local jurisdictions tools to verify their
respective systems' software against the related trusted software
builds.[Footnote 23] According to the program manual, final
certification of a voting system is also conditional upon manufacturers
creating and making available to states and local jurisdictions tools
to compare their systems' software with the trusted build in the
repository. In doing so, the program manual states that manufacturers
shall develop and make available tools of their choice, and that the
manufacturer must submit a letter certifying the creation of the tools
and include a copy and description of the tools to EAC. The commission
may choose to review the tools. Further, the 2005 VVSG provides some
requirements for software verification tools, for example, that the
tools provide a method to comprehensively list all software files that
are installed on the voting system. However, EAC has yet to specify
exactly what needs to be done to ensure that manufacturers provide
effective and efficient system identification tools and processes, and
it has not developed plans for ensuring that this occurs. Instead, EAC
has stated that until it defines procedures to supplement the program
manual, it will review each tool submitted. However, the commission has
yet to establish specific criteria for the assessment.
Without an established means for effectively and efficiently verifying
that acquired systems have the same system configurations as the
version that EAC certified, states and local jurisdictions will not
know whether they are using federally certified voting systems. The
absence of such tools unnecessarily increases the risk of a system
bearing EAC's mark of certification differing from the certified
version.
EAC's Mechanism for Tracking and Resolving Problems with Certified
Voting Systems Lacks Sufficient Definition, and Its Scope Is Limited:
As part of its voting system testing and certification program, EAC has
broadly described an approach for tracking and resolving problems with
certified voting systems, and using the information about these
problems to improve its program. This approach reflects some key
aspects of relevant guidance. However, other aspects are either missing
or not adequately defined, and although EAC officials stated that they
intend to address some of these gaps, the commission does not have
defined plans or time frames for doing so. Commission officials cited
limited resources and competing priorities as reasons for these gaps.
In addition, EAC's problem tracking and resolution approach does not
extend to any of the voting systems that are likely to be used in the
2008 elections, and it is uncertain when, and to what extent, this
situation will change. This is because its defined scope only includes
EAC-certified systems; it does not include NASED-qualified systems or
any other systems to be used in elections. EAC officials stated that
the reason for this is because HAVA does not explicitly assign the
commission responsibility for systems other than those it certifies.
This means that no federal entity is currently responsible for tracking
and facilitating the resolution of problems found with the vast
majority of voting systems that are used across the country today and
that could be used in the future, and thus states and local
jurisdictions must deal with problems with their systems on their own.
EAC's Approach for Tracking and Resolving Problems with Certified
Systems Partially Reflects Relevant Guidance:
According to published guidance,[Footnote 24] tracking and resolving
problems with certified products is important. This includes, among
other things, (1) withdrawing certification if a product becomes
noncompliant; (2) regularly monitoring the continued compliance of
products being produced and distributed; (3) investigating the validity
and scope of reports of noncompliance; (4) requiring the manufacturer
to take corrective actions when defects are discovered, and ensuring
that such actions are taken; and (5) using information gathered from
these activities to improve the certification program.
EAC's approach for tracking and resolving problems with certified
systems reflects some, but not all aspects of these five practices.
First, its certification program includes provisions for withdrawing
certification for noncompliant voting systems. For example, the program
manual describes procedures for decertifying a noncompliant voting
system if the manufacturer does not take timely or sufficient action to
correct instances of noncompliance. According to these procedures, the
decertification decision cannot be made until the manufacturer is
formally alerted to the noncompliance issue and provided with an
opportunity to correct the issue (problem) or to submit additional
information for consideration. Also, a manufacturer can dispute the
decision by requesting an appeal. The procedures also state that upon
decertification, the manufacturer cannot represent the system as
certified and the system may not be labeled with a mark of
certification. In addition, EAC is to remove the system from its list
of certified systems, alert state and local election officials to the
system's decertification via monthly newsletters and e-mail updates,
and post all correspondence regarding the decertification on its Web
site.
Second, EAC's certification program includes provisions for
postcertification oversight of voting systems and manufacturers.
Specifically, the program manual provides for reviewing and retesting
certified voting systems to ensure that they have not been modified,
and that they continue to comply with applicable standards and program
requirements. In addition, the program manual calls for periodic
inspections of manufacturers' facilities to evaluate their production
quality, internal test procedures, and overall compliance with program
requirements. However, the program manual states that reviewing and
retesting of certified systems is an optional step, and does not
specify the conditions under which this option is to be exercised.
Further, while the program manual provides for conducting periodic
inspections of manufacturers' facilities, it does not define, for
example, who is to conduct the inspections, what procedures and
evaluation criteria are to be used in conducting them, and how they are
to be documented.
Third, EAC's certification program includes provisions for
investigating reports of system defects. According to the program
manual, investigations of reports alleging defects with certified
systems begin as informal inquiries that can potentially become formal
investigations. Specifically, the Program Director is to conduct an
informal inquiry in which a determination is made regarding whether the
reported defect information is both credible and deserving of system
decertification if found to be credible. If both conditions are met,
then a formal investigation is to be conducted. Depending on the
outcome, decertification of the system could result. However, the
program manual does not call for assessing the scope or impact of any
defects identified, such as whether a defect is confined to an
individual unit or whether it applies to all such units. Further, the
program manual does not include procedures or criteria for determining
the credibility of reported defects, or any other aspect of the inquiry
or investigation, such as how EAC will gain access to systems once they
are purchased and fielded by states and local jurisdictions. This is
particularly important because EAC does not have regulatory authority
over state election authorities, and thus, it cannot compel their
cooperation during an inquiry or investigation.
Fourth, EAC's certification program does not address how it will verify
that manufacturers take required corrective actions to fix problems
identified with certified systems. Specifically, the program manual
states that the manufacturer is to provide EAC with a compliance plan
describing how it will address identified defects. However, the program
manual does not define an approach for evaluating the compliance plan
and confirming that a manufacturer actually implements the plan.
According to EAC officials, they see their role as certifying a system
and informing states of modifications. As a result, they do not intend
to monitor how or whether manufacturers implement changes to fielded
systems. In their view, the state ultimately decides if a system will
be fielded.
Fifth, EAC's certification program provides for using information
generated by its problem tracking and resolution activities to improve
the program. According to the program manual, information gathered
during quality monitoring activities will be used to, among other
things, identify improvements to the certification process and to
inform the related standards-setting process. Further, the program
manual states that information gathered from these activities will be
used to inform relevant stakeholders of issues associated with
operating a voting system in a real-world environment and to share
information with jurisdictions that use similar systems. However, the
program manual does not describe how EAC will compile and analyze the
information gathered to improve the program, or how it will coordinate
these functions with information gathered in performing its HAVA-
assigned clearinghouse function.[Footnote 25]
EAC officials attributed the state of their problem tracking and
resolution approach to the newness of the certification program and the
fact that the commission's limited resources have been devoted to other
priorities. In addition, while these officials said that they intend to
address some of these gaps, they do not have defined plans or time
frames for doing so. For example, while EAC officials stated that they
plan to develop procedures for investigating voting system problems and
for inspecting manufacturing facilities, they said that it is the
states' responsibility to ensure that corrective actions are
implemented on fielded systems. To illustrate their resource
challenges, these officials told us that three staff are assigned to
the testing and certification program and each is also supporting other
programs.[Footnote 26] In addition, they said that the commission's
technical reviewers are experts who, under Office of Personnel
Management regulation, work no more than one-half of the time of the
year.[Footnote 27]
Given that EAC has not yet certified a system, the impact of these
definitional limitations has yet to be realized. Nevertheless, with 12
systems currently undergoing certification, it is important for the
commission to address them quickly. If it does not, EAC will be
challenged in its ability to effectively track and resolve problems
with the systems that it certifies.
EAC's Problem Tracking and Resolution Efforts Do Not Include Most
Voting Systems to Be Used in Elections:
The scope of EAC's efforts to track and resolve problems with certified
voting systems does not extend to those systems that were either
qualified by NASED or were not endorsed by any national authority.
According to program officials, the commission does not have the
authority or the resources needed to undertake such a responsibility.
Instead of tracking and resolving problems with these systems, EAC
anticipates that they will eventually be replaced or upgraded with
certified systems. Our review of HAVA confirmed that the act does not
explicitly assign EAC any responsibilities for noncertified systems,
although it also does not preclude EAC from tracking and facilitating
the resolution of problems with these systems.
As a result, the commission's efforts to track and resolve problems
with voting systems do not include most of the voting systems that will
be used in the 2008 elections. More specifically, while EAC has efforts
under way relative to the certification of 12 voting systems, as we
have previously described in this report, commission officials stated
that it will be difficult to field any system that EAC anticipates
certifying before 2008 in time for the 2008 elections. Thus, voting
systems used in national elections will likely be either those
qualified under the now-discontinued NASED program, or those not
endorsed by any national entity. Moreover, this will continue to be the
case until states voluntarily begin to adopt EAC-certified systems,
which is currently unclear and uncertain because only 18 states
reported having requirements to use EAC-certified voting systems.
Restated, most states' voting systems will not be covered by EAC's
problem tracking and resolution efforts, and when and if they will is
not known. Moreover, manufacturers may or may not upgrade existing,
noncertified systems, and they may or may not seek EAC certification of
those systems. Thus, it is likely that many states, and their millions
of voters, will not use EAC-certified voting systems for the
foreseeable future.
Nevertheless, EAC has initiated efforts under the auspices of its HAVA-
assigned clearinghouse responsibility[Footnote 28] to receive
information that is volunteered by states and local jurisdictions on
problems and experiences with systems that it has not certified, and to
post this information on the commission's Web site to inform other
states and jurisdictions about the problems. In doing so, EAC's Web
site states that the commission does not review the information for
quality and does not endorse the reports and studies. Notwithstanding
this clearinghouse activity, this means that no national entity is
currently responsible for tracking and facilitating the resolution of
problems found with the vast majority of voting systems that are in use
across the country. This in turn leaves state and local jurisdictions
on their own to discover, disclose, and address any shared problems
with systems. While this increases the chances of states and local
jurisdictions duplicating efforts to get problems fixed, it also
increases the chances that problems addressed by one state or
jurisdiction may not even be known to another. A key to overcoming this
situation will be strong central leadership.
Conclusions:
The effectiveness of our nation's overall election system depends on
many interrelated and interdependent variables. Among these are the
security and reliability of the voting systems that are used to cast
and count votes, which in turn depend largely on the effectiveness with
which these systems are tested and certified. EAC plays a pivotal role
in testing and certifying voting systems. To its credit, EAC has
recently established and begun implementing a voting system testing and
certification program that is to both improve the quality of voting
systems in use across the country, and help foster public confidence in
the electoral process. While EAC has made important progress in
defining and executing its program, more can be done. Specifically, key
elements of its defined approach, such as the extent to which
certification activities are to be documented, are vague, while other
elements are wholly undefined--such as threshold criteria for making
certification-related decisions. Moreover, a key element that is
defined--namely, giving states and local jurisdictions an effective and
efficient means to access the certified version of a given voting
system software--has yet to be implemented. While EAC acknowledges the
need to address these gaps, it has yet to develop specific plans or
time frames for completing them that, among other things, ensure that
adequate resources for accomplishing them are sought.
Addressing these gaps is very important because their existence not
only increases the chances of testing and certification activities
being performed in a manner that is neither repeatable nor verifiable,
they also can create misunderstanding among manufacturers and VSTLs
that can lead to delays in the time needed to certify systems. Such
delays have already been experienced, to the point that needed upgrades
to current systems will likely not be fielded in time for use in the
2008 elections. Such situations ultimately detract from, and do not
enhance, election integrity and voter confidence. Moreover, by not
having established an effective means for states and local
jurisdictions to verify that the systems each acquires are the same as
the EAC-certified version, EAC is increasing the risk of noncertified
versions ultimately getting used in an election.
Beyond the state of EAC's efforts to define and follow an approach to
testing and certifying voting systems, including efforts to track and
resolve problems with certified systems and use this information to
improve the commission's testing and certification program, a void
exists relative to having a national focus on tracking and resolving
problems with voting systems that EAC has not certified, and thus has
not been assigned explicit responsibility or has the resources to
address. Unless this void is filled, state and local governments will
likely continue to be on their own for resolving performance and
maintenance issues for the vast majority of voting systems in use today
and the near future.
Recommendations for Executive Action:
To assist EAC in building upon and evolving its voting systems testing
and certification program, we recommend that the Chair of the EAC
direct the commission's Executive Director to ensure that plans are
prepared, approved, and implemented for developing and implementing:
* detailed procedures, review criteria, and documentation requirements
to ensure that voting system testing and certification review
activities are conducted thoroughly, consistently, and verifiably;
* an accessible and available software repository for testing
laboratories to deposit certified versions of voting system software,
as well as procedures and review criteria for evaluating related
manufacturer-provided tools to support stakeholders in comparing their
systems with this repository; and:
* detailed procedures, review criteria, and documentation requirements
to ensure that problems with certified voting systems are effectively
tracked and resolved, and that the lessons learned are effectively used
to improve the certification program.
Matter for Congressional Consideration:
To address the potentially longstanding void in centrally facilitated
problem identification and resolution for non-EAC-certified voting
systems, we are raising for congressional consideration expanding EAC's
role under HAVA such that, consistent with both the commission's
nonregulatory mission and the voluntary nature of its voting system
standards and certification program, EAC is assigned responsibility for
providing resources and services to facilitate understanding and
resolution of common voting system problems that are not otherwise
covered under EAC's certification program, and providing EAC with the
resources needed to accomplish this.
Agency Comments and Our Evaluation:
In written comments on a draft of this report, signed by the EAC
Executive Director, and reprinted in appendix II, the commission stated
that it agrees with the report's conclusion that more can be done to
build on the existing voting system certification program and ensure
that certifications are based on consistently performed reviews. In
addition, EAC stated that it has found our review and report helpful in
its efforts to fully implement and improve this program. It also stated
that it generally accepts our three recommendations with little
comment, adding that it will work hard to implement them. In this
regard, it cited efforts that are planned or underway to address the
recommendations.
EAC provided additional comments on the findings that underlie each of
the recommendations, which it described as needed to clarify and avoid
confusion about some aspects of its certification program. According to
EAC, these comments are intended to allay some of the concerns raised
in our findings. We summarize and evaluate these comments below, and to
avoid any misunderstanding of our findings and the recommendation
associated with one of them, we have modified the report, as
appropriate, in response to EAC's comments.
EAC also provided comments on our matter for congressional
consideration, including characterizing it as intending "to affect a
sea change in the way that EAC operates its testing and certification"
program. We agree that EAC does not have the authority to compel
manufacturers or states and local jurisdictions to submit to its
testing and certification program and that the wording of the matter in
our draft inadvertently led EAC to believe that our proposal would
require the commission to assume a more regulatory role. In response,
we have modified the wording that we used to clarify any
misunderstanding as to our intent.
With respect to our first recommendation for developing and
implementing plans for ensuring that voting system testing and
certification review activities are governed by detailed procedures,
criteria, and documentation requirements, EAC stated that it is
committed to having a program that is rigorous and thorough, and that
its program manual creates such a program. It also stated that it
agrees with our recommendation and that it will work to further the
process in its manual by implementing detailed procedures. In this
regard, however, it took issue with five aspects of our finding.
* With respect to our point that criteria and procedures have not been
adequately defined for reviewing the information in the manufacturer
registration application package, EAC stated that such criteria are not
necessary because the package does not require a determination of
sufficiency. We do not agree. According to section 2.4.2 of the program
manual, EAC is to review completed registration applications for
sufficiency. However, as our report states, the manual does not define
criteria as to what constitutes sufficiency and what this sufficiency
review should entail, and EAC officials stated this determination is
left up to the individual reviewer's judgment.
* Concerning our point that criteria and procedures have not been
adequately defined for reviewing the information in the system
certification application package, EAC stated that a technical review
of the package is not required. Rather, it said that the package simply
requires a determination that all necessary information is present,
adding that it has a checklist to assist a reviewer in determining
this. We do not agree with this comment for two reasons. First, our
report does not state that the package review is technical in nature,
but rather is a review to determine a package's completeness and
accuracy. Second, the checklist does not include any criteria upon
which to base a completeness and accuracy determination. As EAC's
comments confirm, this is important because the information in the
package is to be used by technical reviewers as they review test plans
and test reports to ensure that the testing covers all aspects of the
voting system. For example, EAC requires certification applications to
include a functional diagram depicting how the components for the
voting system function and interact, as well as a system overview that
includes a description of the functional and physical interfaces
between components. Although the checklist provides for determining
whether these items are part of the application package, it does not
provide for checking them for completeness and consistency. We have
clarified this finding in our report by including this example.
* As to our point that EAC has not defined how technical reviewers are
to determine the adequacy of system test plans and reports, the
commission stated that our report does not take into account what it
described as a certification requirements traceability matrix that its
technical reviewers use to assess the completeness and adequacy of the
plans and reports. However, EAC also acknowledged in its comments that
procedures have yet to be established relative to the use of the
matrix. Further, we reviewed this matrix, which we refer to in our
report as a checklist, and as we state in our report, this checklist
does not provide for capturing how decisions were reached, including
steps performed and criteria applied. For example, the VVSG requires
that systems permit authorized access and prevent unauthorized access,
and lists examples of measures to accomplish this, such as computer-
generated password keys and controlled access security. While the
checklist cites this requirement and provides for the reviewer to
indicate whether the test plan satisfies it, it does not provide
specific guidance on how to determine whether the access control
measures are adequate, and it does not provide for documenting how the
reviewer made such a decision. In response to EAC's comments, we have
added this access control example to clarify our finding.
* With regard to our point that the program manual does not include
defined test suites, EAC commented on the purpose of these test suites
and stated that it would not be appropriate to include them in the
manual because the manual is not intended to define technical
requirements for testing. We agree that the program manual should not
include actual test suites, and it was not our intent to suggest that
it should. Rather our point is that test suites do not yet exist.
Accordingly, we have modified our report to more clearly reflect this.
In addition, we acknowledge EAC's comment that NIST is currently in the
process of developing test suites, and that it recently sent several
test suites to the VSTLs and other stakeholders for review. However, as
we state in our report, NIST officials said that they are focused on
preparing test suites for the yet-to-be-released update to the VVSG and
not the 2005 version. Further, EAC has not yet established plans or
time frames for finalizing test suites for either versions of these
guidelines, and the program manual does not make reference to the
development of these test suites.
* In commenting on our point that differences in interpretation of
program requirements have resulted in test plan and report approval
delays, EAC stated that its interpretation process provides a means for
VSTLs and manufacturers to request clarification of the voting system
standards that are ambiguous. We agree with this statement. However, we
also believe that having the kind of defined procedures and established
criteria that are embodied in our recommendation will provide a common
understanding among EAC stakeholders around testing and certification
expectations, which should minimize the need to reconcile differences
in interpretations later in the process.
With respect to our second recommendation for developing and
implementing plans for an accessible and available software repository
for certified versions of voting system software, as well as the
related manufacturer-provided procedures and tools to support
stakeholders in using this repository, EAC stated that it agrees that
implementation of a repository is needed. However, it stated that there
is some misunderstanding regarding the purpose of the repository and
the creation of software identification tools. Specifically, the
commission stated that the repository is intended for the commission's
own use when conducting investigations of fielded systems, while the
manufacturer-provided system identification tools are for use by state
and local election officials to confirm that their systems are the same
as the one certified by EAC. In addition, it described steps taken or
under way to ensure that a repository and identification tools are in
place when needed. This includes "placing the onus" on system
manufacturers to create verification tools, investigating software
storage options, and discussing with another government agency and
outside vendors the possibility of providing secure storage for
certified software.
We agree with EAC that its repository serves as a tool for its internal
use. However, the repository is also to serve state and local election
officials in verifying that their respective systems are identical to
the EAC-certified versions of the systems. According to the 2005 VVSG
software distribution and setup validation requirements, the process
for voting system purchasers in verifying that the version of the
software that they receive from a manufacturer is the same as the
version certified by EAC is to be performed by comparing it with the
reference information generated by the designated repository. Further,
commission officials told us that EAC's repository needs to be
accessible and easy to use by state and local election officials. While
we understand that the manufacturer-provided system identification
tools serve a separate function from the repository, both tools
together are required for state and local election officials to verify
their systems. We also acknowledge that the commission has initiated
steps relative to establishing a repository and identification tools.
However, our point is that EAC does not have any plans or time frames
for accomplishing this. Further, while we agree that the manufacturers
are responsible for creating the identification tools, as we stated in
our report, EAC has not defined how it will evaluate the manufacturer-
provided tools. To avoid any misunderstanding as to these points, we
have slightly modified our finding and related recommendation.
Concerning our third recommendation for developing and implementing
detailed procedures, review criteria, and documentation requirements
for tracking and resolving problems with certified voting systems and
applying lessons learned to improve the certification program, EAC
stated that the report does not correctly represent its role in
confirming that manufacturers actually correct anomalies in all fielded
systems, and it added that the commission does not have the authority
or the human capital to do so. Accordingly, EAC stated that it informs
affected jurisdictions of system changes, but that it is at the
discretion of the states and local jurisdictions, and beyond the scope
of the commission, to determine whether fixes are made to individual
systems in the field.
We agree that the states and local jurisdictions have the
responsibility and authority to determine whether they will implement
EAC-approved fixes in the systems that they own. However, as we state
in our report, published ISO guidance on tracking and resolving
problems with certified products recognizes the importance of the
certification body's decision to require manufacturers to take
corrective actions when defects are discovered, and to ensure that such
actions are taken. Although this guidance acknowledges the difficulty
in ensuring corrective actions are implemented on all affected units,
it states that products should be corrected "to the maximum degree
feasible." Given EAC's authority over registered manufacturers, it can
play a larger role in ensuring that problems with fielded system are in
fact resolved, while maintaining the voluntary nature of its program,
by monitoring the manufacturers' efforts to fix systems for those
jurisdictions that choose to implement such corrections, and holding
manufacturers accountable for doing so. To avoid any confusion about
this point, we have slightly modified our finding.
As to our matter for congressional consideration to amend HAVA to give
EAC certain additional responsibilities relative to problem resolution
on voting systems not certified by EAC, the commission voiced several
concerns. Among other things, it stated that our proposal would "affect
a sea change in the way that EAC operates its testing and
certification" program, changing it from voluntary to mandatory.
Further, it stated that it would, in effect, place EAC in a position to
act in a regulatory capacity without having the specific authority to
do so, as it would necessitate making both the voluntary voting system
guidelines and the testing and certification program mandatory for all
states. It also stated that it would require EAC to have specific
authority to compel manufacturers of these noncertified voting systems
to submit their systems for testing, and to compel states and local
jurisdictions to report and resolve any identified system problems.
We recognize that both the voting system guidelines and the testing and
certification program are voluntary, and that EAC does not have the
authority to compel manufacturers or states and local jurisdictions to
submit to its testing and certification program, or to force them to
correct any known problems or report future problems. We further
acknowledge that the wording of the matter for congressional
consideration in our draft report resulted in EAC interpreting
accomplishment of it as requiring such unintended measures. Therefore,
we have modified it to clarify our intent and to avoid any possible
misunderstanding. In doing so, we have emphasized our intent for EAC to
continue to serve its existing role as a facilitator and provider of
resources and services to assist states and local jurisdictions in
understanding shared problems, as well as the voluntary nature of both
the system guidelines and the testing and certification program.
Further, we seek to capitalize on EAC's unique role as a national
coordination entity to address a potentially longstanding, situational
awareness void as it pertains to voting systems in use in our nation's
elections. As we state in our report, this void increases the chances
of states and local jurisdictions duplicating efforts to fix common
system problems, and of problems addressed by one state or local
jurisdiction being unknown to others. We believe that a key to
overcoming this will be strong central leadership, and that with the
appropriate resources, EAC is in the best position to serve this role.
We are sending a copy of this report to the Ranking Member of the House
Committee on House Administration, the Chairman and Ranking Member of
the Senate Committee on Rules and Administration, the Chairmen and
Ranking Members of the Subcommittees on Financial Services and General
Government, Senate and House Committees on Appropriations, and the
Chairman and Ranking Member of the House Committee on Oversight and
Government Reform. We are also sending copies to the Chair and
Executive Director of the EAC, the Secretary of Commerce, the Acting
Director of the NIST, and other interested parties. We will also make
copies available to others on request. In addition, this report will be
available at no charge on the GAO Web site at [hyperlink, www.gao.gov].
Should you or your staff have any questions on matters discussed in
this report, please contact me at (202) 512-3439 or at [email protected].
Contact points for our Offices of Congressional Relations and Public
Affairs may be found on the last page of this report. Key contributors
to this report are listed in appendix III.
Sincerely yours,
Signed by:
Randolph C. Hite:
Director, Information Technology Architecture and System Issues:
[End of section]
Appendix I: Objectives, Scope, and Methodology:
Our objectives were to determine whether the Election Assistance
Commission (EAC) has (1) defined an effective approach to testing and
certifying voting systems, (2) followed its defined approach, and (3)
developed an effective mechanism to track problems with certified
systems and use the results to improve its certification program.
To address the first and third objectives, we researched leading
practices relevant to certification testing/conformity assessment and
tracking and resolving problems with certified products, including
published guidance from the National Institute of Standards and
Technology (NIST), International Organization for Standardization, and
International Electrotechnical Commission, and legal requirements in
the Help America Vote Act. We obtained and reviewed relevant EAC
policies and procedures for testing, certifying, decertifying, and
recertifying voting systems. Specifically, we reviewed the EAC
Certification Program Manual and other EAC-provided documents. We
interviewed EAC officials and their technical reviewers, NIST
officials, representatives from the industry trade association for
voting system manufacturers, representatives from the voting system
test laboratories, and the National Association of State Election
Directors' point-of-contact for qualified systems. We then compared
this body of evidence with the leading practices and related guidance
we had researched, as well as applicable legal requirements, to
determine whether EAC's program had been effectively defined. In
addition, for the third objective, we reviewed the contents and policy
of EAC's clearinghouse.
To address our second objective, we obtained and reviewed actions and
artifacts from EAC's execution of its certification program to date. We
assessed this information against the policies, procedures, and
standards outlined in the EAC Certification Program Manual and the 2005
Voluntary Voting System Guidelines, and after discussing and confirming
our findings with EAC officials, we determined whether EAC had followed
its defined approach.
In addition, to determine the impact of federal certification time
frames, we included a question about EAC certification on a survey of
officials from all 50 states, the District of Columbia, and 4
territories. We also contacted officials from states that indicated
their intent to use EAC certification for the 2008 elections to better
understand how they plan to address voting system certification in
their state relative to EAC's program. To develop our survey, we
reviewed related previous and ongoing GAO work, and developed a
questionnaire in collaboration with GAO's survey and subject matter
experts. We conducted pretests in person and by telephone with election
officials from 5 states to refine and clarify our questions. Our Web-
based survey was conducted from December 2007 through April 2008. We
received responses from 47 states, the District of Columbia, and all 4
territories (a 95 percent response rate). Differences in the
interpretation of our questions among election officials, the sources
of information available to respondents, and the types of people who do
not respond may have introduced unwanted variability in the responses.
We examined the survey results and performed analyses to identify
inconsistencies and other indications of error, which were reviewed by
an independent analyst. We also contacted officials from those states
whose survey response indicated their intent to use EAC certification
for the 2008 elections to identify their plans and approaches for state
certification in the event that federal certification could not be
completed to meet their election preparation schedules.
We conducted this performance audit at EAC offices in Washington, D.C.,
from September 2007 to September 2008 in accordance with generally
accepted government auditing standards. Those standards require that we
plan and perform the audit to obtain sufficient, appropriate evidence
to provide a reasonable basis for our findings and conclusions based on
our audit objectives. We believe that the evidence obtained provides a
reasonable basis for our findings and conclusions based on our audit
objectives.
[End of section]
Appendix II: Comments from the Election Assistance Commission:
U.S. Election Assistance Commission:
1225 New York Ave. NW � Suite 1100:
Washington, DC 20005:
July 16, 2008:
Mr. Randolph C. Hite, Director:
Information Technology Architecture and Systems:
United States Government Accountability Office:
Delivered via e-mail:
Re: Draft Report GAO 08-814, Elections: Federal Program for Certifying
Voting Systems Needs to be Further Defined, Fully Implemented, and
Expended
Thank you for the opportunity to provide comment on GAO report 08-814,
regarding the Election Assistance Commission (EAC) voting system
testing and certification program. The EAC appreciates the time and
effort put forth by GAO during the preparation of this document and the
willingness of GAO staff to discuss pertinent issues at length with the
EAC. The EAC has found both the review process and report helpful as it
works to fully implement and improve its HAVA required mandate to
provide for testing, certification, decertification, and
recertification of voting system hardware and software.
GAO recognized that: "EAC has defined an approach to testing and
certifying voting systems that follows a range of relevant practices
and statutory requirements associated with a product certification
program, including those published by U.S. and international standards
organizations, and those reflected in HAVA." (pp. 5-6). The EAC
generally agrees with the report's conclusion that more can be done in
order to build on the EAC's existing certification program in order to
make sure that certifications are based upon consistent reviews.
However, EAC is concerned that GAO confuses some aspects of EAC's
Testing and Certification Program and therefore doesn't recognize
practices and procedures already in place as part of the EAC's program
that would quell some of GAO's concerns.
The report provides three recommendations for the EAC to better conform
to certification program management guidance published by National
Institute of Standards and Technology (NIST) and International
Standards Organization (ISO)/International Electrotechnical Commission
(IEC) . Generally, the EAC accepts the recommendations provided with
little comment. However, the EAC feels the need to clarify several
points related to the recommendations and the Matter for Congressional
Consideration. The following are EAC's comments in response to each
recommendation.
1. Detailed procedures, review criteria, and documentation requirements
to ensure that voting system testing and certification review
activities are conducted thoroughly, consistently, and verifiably;
GAO recognizes that EAC's Testing and Certification program has defined
a testing and certification process that follows many recognized and
accepted practices for conformance assessment and product certification
(pp. 20-21). Specifically, GAO found that EAC's Testing and
Certification Program covers the standard procedures established by
NIST, ISO and IEC for testing and certification programs, including:
(1) Defining roles and responsibilities for all parties involved in the
certification process; (2) defining a clear and transparent process for
applicants to follow; (3) ensuring that persons involved in the process
are impartial and independent; (4) establishing a process for handling
complaints and appeals; and (5) having testing conducted by competent
laboratories. These procedures are outlined in the EAC's Voting System
Testing and Certification Program Manual. As the GAO report notes, in
addition to the five items listed above, the EAC's program manual also
"clearly defines the program's administrative requirements that
manufacturers and Voting System Test Laboratories (VSTLs) are to
follow," addresses impartiality and independence of the testing
process, and outlines processes for the resolution of complaints,
appeals, and disputes received from manufacturers and laboratories (p.
22). The GAO report also states, ". the EAC has provided an important
foundation for having an effective voting system certification
program."(p.23).
A thorough testing and certification program is essential to ensuring
public confidence in our electoral system. As GAO found in its report,
prior efforts at testing and qualifying voting systems have left
election administrators to deal with complaints and questions relating
to the sufficiency and security of their voting systems. "As we
reported in 2005, these concerns include weak security controls, system
design flaws, inadequate system version control, inadequate security
testing, incorrect system configuration, poor security management, and
vague or incomplete voting system standards. Further, security experts
and some election officials have expressed concerns that tests
performed under the NASED program by independent testing authorities
and state and local election officials did not adequately assess voting
systems' security and reliability. Consistent with these concerns, most
of the security weaknesses that we identified in our prior report
related to systems that had previously been qualified by NASED. Our
report also recognized that security experts and others pointed to
these weaknesses as an indication that both the standards and the NASED
testing program were not rigorous enough with respect to security, and
that these concerns were amplified by what some described as a lack of
transparency in the testing process". (p. 12, footnote omitted)
EAC is committed to conducting a program that is rigorous and
thoroughly tests systems to high standards for operation and security.
EAC's Voting System Testing and Certification Manual creates such a
program and EAC will work to further this process by implementing
internal procedures consistent with the program manual.
Although GAO did not find any instances in which EAC's Voting System
Testing and Certification Program has been conducted in an inequitable
or discriminatory manner, GAO offered several areas where additional
internal procedures would further ensure consistency in the process.
GAO recommends procedures for reviewing manufacturer registration and
system application packages, procedures for assessing test plan and
test reports, including test suites in the program manual, and
providing a means to resolve differing interpretations of voting system
standards. While we agree with GAO's overall recommendations, we are
concerned that some of the examples cited in the report may cause
confusion as to the EAC's current policies and procedures.
GAO asserts that while the EAC requires manufacturers to register prior
to submitting a system for certification, it does not adequately define
the criteria for approval of the registration package. As stated in the
EAC's program manual, the EAC will review manufacturer registration
applications for completeness before approval (Chapter 2 of the
Certification manual). The registration package simply contains contact
information, a listing of manufacturing facilities, and a series of
agreements by the manufacturer to comply with program requirements.
Reviewing this information does not require the EAC to make a
determination of sufficiency, but instead simply requires confirmation
that all required information is present. To accomplish this, the EAC
has created a checklist for the review of each manufacturer
registration package.
Similarly, GAO notes that in order for a system to begin the EAC's
certification process, a manufacturer must submit a voting system
application package. The voting system application package includes
information related to the make up of the system. It is used by EAC
technical reviewers as they review testing plans and reports to assure
that those plans and reports cover all aspects of the voting system.
Thus, the voting system application package does not require a
technical review of the information provided but instead simply
requires a determination that all necessary information is present. As
such, the EAC has created a checklist for voting system application
packages that allows a reviewer to document whether or not all required
information is provided and if it is not provided to request the
information from the manufacturer.
The GAO report correctly finds that all test plans submitted to the EAC
for approval are reviewed for sufficiency. In conducting this review,
the EAC is looking to ensure that all requirements of the VSS or VVSG
that are applicable to the system will be tested. The GAO report states
that the EAC does not define how such reviews are to be performed.
However, this assessment does not take into account the certification
requirements traceability matrix used by EAC technical reviewers to
assess the quality and completeness of the test plan and the test
report. The requirements matrix lists the requirements to be tested and
the requirements which must be met by a system before it can receive
certification. Using the requirements matrix enables EAC to
consistently assess each test plan and test report for completeness and
adequacy.
The GAO report describes the EAC program manual as excluding defined
test suites. The EAC believes there is some confusion regarding exactly
what test suites are and the role they will play in the certification
process. As noted in the GAO report, NIST is currently in the process
of developing test suites for use by the VSTLs. Recently, NIST sent
several test suites to the laboratories and other stakeholders for
review. These test suites are designed to be high level test methods
that can be taken by a VSTL and adapted for the creation of system
specific test cases. Each test suite will encompass a set of voting
system standards to be tested on a given system. As such, it would not
be appropriate to include the actual test suites in the EAC's program
manuals because the manuals are designed to document the EAC's
procedural program requirements not to represent the technical
requirements for testing. However, the EAC does require use of the test
suites in its program. In the EAC's Voting System Testing Laboratory
Program Manual, which is scheduled to be voted on by the Commission at
the July 2008 public meeting, the EAC requires the use of test suites
in the creation of test plans and the testing of the system. Likewise,
in Chapter 4 of the Testing and Certification Manual, the EAC
establishes a requirement to submit a test plan and test report for EAC
approval. Test suites will be included and reviewed in the submitted
test plans and test reports.
GAO asserts that differences in how the EAC, VSTLs, and manufacturers
interpret voting system requirements have caused delays in the test
plan and test report approval process. As the GAO report notes, the EAC
provides a means by which the VSTLs and Manufacturers may request
clarification of the standard to be tested to (p. 22)(Chapter 9 Testing
and Certification Manual). As the EAC encounters ambiguities in the
standard the EAC issues interpretations of the standard in order to aid
VSTLs in their testing of the system. However, as GAO noted in its
report, the establishment of EAC's Voting System Testing and
Certification Program represents a paradigm shift from previous testing
efforts. In order for this to have its greatest effect, all players in
the process must participate and use the tools available to them. To
date the EAC has issued eight interpretations based upon requests by
system users all of which are available on the EAC's website at
[hyperlink, http://www.eac.gov].
The EAC is working to develop the internal procedures recommended by
GAO. To ensure consistent and verifiable review, EAC is creating
standard report formats, review tools and checklists. These documents
will include guidance regarding:
* Review of Technical Data Package elements.
* Review and use of the Requirements Traceability Matrix.
* Review of Test Plans.
* Review of Test Cases.
* Review of Test Reports.
* Identifying and reporting of common anomalies found during technical
reviews.
* Timelines and processes for extending timelines when necessary.
* Documenting review findings in a standard organized report.
* Protocols for communicating with VSTLs and manufacturers.
* Reporting process through which technical problems identified with
the Voluntary Voting System Guidelines (VVSG) can be identified and
reported to the EAC.
2. Ensure plans are prepared, approved and implemented for an
accessible and available software repository for testing laboratories
to deposit certified versions of voting system software as well as
related manufacturer-provided procedures and tools to support
stakeholders in using this repository.
As stated in the GAO report the EAC has, "...largely executed its
voting certification program as defined." (p. 28) GAO goes on to add
that for each of the 12 systems submitted for certification, "all
elements of each executed step in the certification process were
followed." Included in these steps were the approval of registration
applications, applications for system testing, approval of test plans
and the issuance of three notices of non-compliance to manufacturers
not conforming with the EAC's requirements (p. 29). However, GAO
pointed out the need to implement the program's requirement for a
voting system software repository.
While the EAC agrees that the implementation of a software repository
is needed, there is some misunderstanding regarding the purpose of the
repository and the creation of software identification tools. The EAC
requires two post certification steps as a part of its testing and
certification program: (1) submission of software in an approved
repository; and (2) creation of system identification tools by the
manufacturer. The purpose of the software repository is to create a
frozen image or picture of the system as certified through the EAC
program. This will allow the EAC to use the information stored in the
repository when conducting investigations of fielded, EAC-certified
systems and ensure that the fielded system is an exact match to the
system that was certified. The creation of system identification tools
by the manufacturer allows the voting system users (states) to review
and confirm that the systems and software that they purchase are the
same as those certified by EAC.
Thus, the software repository is a tool for EAC use, while the system
identification tools are for use by state and local governments. GAO
suggests that the repository is intended to serve the function of the
system identification tools and that this process should be
implemented. However, EAC must point out that it has made provision for
this function by requiring the manufacturer of the system to create
this type of verification tool. Because no system has successfully
completed the EAC certification process, EAC has yet to approve any
manufacturer's system identification tool. However, the EAC is
currently working with one manufacturer in the development of the
required system identification tools.
As noted in the GAO report, EAC had intended to use NIST's National
Software Reference Library (NSRL) as its software repository. However,
EAC quickly realized that NSRL could not serve the function of both a
repository and a system identification tool. This is because of the
limitation on comparing installed software (including the nonstatic
portions of that code) to the hashed code maintained by NSRL. EAC's
program needed more. In discussion with NIST, they agreed that NSRL's
functions could not meet the needs of EAC's program. As such, EAC
placed the onus on the manufacturer to develop the system
identification tools. EAC has also investigated other, simpler
solutions to its need for software storage. EAC has considered the
possibility of contracting with an outside vendor for the secure
storage of the certified software. Likewise, EAC has entered into
discussion with another government agency that can provide the same
service. Prior to the final certification of its first system, EAC will
have a mechanism in place to securely store the certified software.
3. Detailed procedures, review criteria, and documentation requirements
to ensure that problems with certified voting systems are effectively
tracked and resolved, and that the lessons learned are effectively used
to improve the certification program.
As GAO notes, EAC has already "broadly described an approach to allow
it to track and resolve problems with certified voting systems." The
GAO report cites five activities that a certifying body should provide
for in its monitoring of fielded certified equipment: (1) Withdrawing
certification if a product becomes non-compliant; (2) regularly
monitoring the continued compliance of products being produced and
distributed; (3) investigating the validity and scope of reports of non-
compliance; (4) requiring the manufacturer to take corrective actions
when defects are discovered and ensuring such actions are taken; and
(5) using information gathered from these activities to improve the
certification program (p. 33). The GAO report correctly identifies that
Chapter 8 of the EAC's program manual provides for aspects of all five
of these requirements (pp. 33-34). GAO cites the EAC's outlined
procedures for decertification, post-certification oversight including
periodic inspections of manufacturer facilities, investigations of
system defects, compliance management, and the use of information
regarding fielded systems to improve the program.
Additionally, GAO recognizes that the EAC compliance management process
requires voting system manufacturers to report anomalies with fielded
EAC-certified voting systems and to create a compliance plan to fix the
anomaly found. However, the GAO report does not correctly represent the
EAC's role in confirming that the manufacturer actually implements the
solution in all fielded systems. As the certifying body EAC does not
have the authority or the manpower to ensure that a manufacturer has
instituted its solution in all fielded versions of the certified
system. EAC has the responsibility to ensure that a noncompliant system
has come into compliance. The EAC does this through its compliance
management program and the compliance plan noted above. As noted in the
EAC's program manual, after a compliance plan and compliance test
report have been approved by the EAC, the EAC will make a decision on
the amended voting system. All compliance plans including test reports
and EAC decisions on amended systems will be made public. After a
decision has been made on an amended system, the EAC will inform all
jurisdictions impacted of the decision made. It is then up to the
individual election jurisdictions to ensure that the change noted in
the amended system decision is implemented. HAVA is explicit in stating
that the EAC's program is voluntary and that it is the states'
responsibility to determine if and how to use the EAC's program.
Therefore, fixes made to individual systems in the field are at the
discretion of the state and out of the scope of the EAC's program.
In addition to the compliance management program requirements already
noted by GAO, the EAC has already begun to develop specific procedures
for use in investigating anomalies with certified voting systems. The
procedures will include details on when and how the EAC will work with
State and local election officials to investigate these anomalies and
include general timeframes for all aspects of the investigations.
Included in these procedures will be details on conducting
manufacturing site audits. The information collected via the EAC's
Compliance Management Program will be used to:
* Identify areas for improvement in the EAC Testing and Certification
Program.
* Improve manufacturing quality and change control processes.
* Increase voter confidence in voting technology.
* Inform Manufacturers, election officials, and the EAC of issues
associated with voting systems in a real-world environment.
* Share information among jurisdictions that use similar voting
systems.
* Resolve problems associated with voting technology or manufacturing
in a timely manner by involving manufacturers, election officials, and
the EAC.
* Provide feedback to the EAC and the Technical Guidelines Development
Committee (TGDC) regarding issues that may need to be addressed through
a revision to the Voluntary Voting System Guidelines.
* Initiate an investigation when information suggests that
Decertification is warranted.
In addition to the three recommendations discussed above, GAO has
issued a Matter for Congressional Consideration. The Congressional
recommendation states:
To address the potentially longstanding void in centrally facilitated
problem identification and resolution for non-EAC certified voting
systems, we are raising for congressional consideration amending HAVA
to give EAC explicit responsibility for identifying, tracking,
reporting and facilitating the resolution of problems that states and
local jurisdictions experience with voting systems that are not covered
by EAC certification, and providing EAC with the resources needed to
accomplish this.
GAO is recommending to Congress that it give "EAC explicit
responsibility for identifying, tracking, reporting and facilitating
the resolution of problems that states and local jurisdictions
experience with voting systems that are not covered by EAC
certification..." With this recommendation GAO is effectively
requesting that EAC have the authority to regulate the voting systems
that are currently in the field as well as the users of those systems.
HAVA is explicit that both the voluntary voting system guidelines and
the testing and certification program are voluntary. (42 U.S.C. ��
15361, 15362, and 15371(a)(2)). States must act to adopt the guidelines
and participate in the program in order to require the use of EAC
certified systems in their respective jurisdictions. Furthermore, the
duties of the EAC as established by HAVA are prospective. HAVA tasks
EAC with developing a new set of voting system testing standards (42
U.S.C. � 15322(1)) and developing a testing and certification program
(42 U.S.C. � 15371). In fact, HAVA recognizes and provides for a period
of transition until these elements are in place:
"(d) TRANSITION. � Until such time as the Commission provides for the
testing, certification, decertification, and recertification of voting
system hardware and software by accredited laboratories under this
section, the accreditation of laboratories and the procedures for
testing, certification, decertification, and recertification of voting
system hardware and software used as of the date of the enactment of
this Act shall remain in effect." (42 U.S.C. � 15371(d)).
GAO's proposal would require making both the voluntary voting system
guidelines and the testing and certification program to become
mandatory. Furthermore, as written, the proposal would apply not only
to systems that were previously tested under another program, but also
those systems fielded by states that have chosen not to participate in
EAC's testing and certification program.
If Congress desires EAC to identify, track, report and facilitate the
resolution of problems with voting systems that are currently in the
field and which were tested and certified prior to the existence of
EAC's and its testing and certification program and systems that have
been fielded by states that chose not to participate in EAC's program,
specific authority must be given to EAC to compel the manufacturers of
these systems to provide those systems for testing, and to compel the
users of those systems (states and local units of government) to report
and resolve any identified problems. In order to accomplish this type
of mandate, EAC would have to subject the fielded systems to testing
against a set of standards. EAC would have to have a mechanism to force
states and local governments that have these systems to correct any
problems that were identified and make them report any future problems
to EAC.
Specifically, this would mean that EAC would have the authority to
compel both voting system manufacturers and state and local government
users of those systems to submit the systems for testing pursuant to
EAC's testing and certification program. Systems would be tested
against the existing voluntary voting system guidelines, 2005 VVSG.
This is in stark contrast to the current authority given this
Commission by HAVA, which is to operate a voluntary testing and
certification program against voluntary testing standards. Similarly,
EAC would have to be given the authority to regulate the users of these
fielded systems so that EAC could compel those users to resolve
identified problems and report any future problems with their voting
systems. Under the current authorities granted by HAVA, EAC obtains the
agreement of participating manufacturers to submit systems for testing
and to report anomalies, problems, or issues with the operation of the
systems that have been tested and certified through the EAC program.
While EAC will implement any program or set of requirements that
Congress desires, EAC must be given the appropriate authority to
conduct those programs and the human capital and financial resources
necessary to effectively implement any changes to its existing
operations. GAO's proposal intends to affect a sea change in the way
that EAC operates its testing and certification � from voluntary to
mandatory. GAO's proposed change to HAVA would place EAC in the
position of acting in a regulatory capacity without the specific
authority necessary to carry out the enumerated functions.
The EAC thanks GAO for its work in assisting the Commission in its
efforts to improve and develop the Voting System Testing and
Certification Program. The EAC is committed to continuous improvement
in all of it programs and will work hard to implement the
recommendations made in this report. The EAC is focused on developing a
world class testing and certification program to benefit election
officials and the voting public.
Sincerely,
Signed by:
Thomas R. Wilkey:
Executive Director:
[End of section]
Appendix III: GAO Contact and Staff Acknowledgments:
GAO Contact:
Randolph C. Hite, (202) 512-3439 or [email protected]:
Staff Acknowledgments:
In addition to the person named above, Paula Moore, Assistant Director;
Mathew Bader; Neil Doherty; Nancy Glover; Dan Gordon; David Hinchman;
Valerie Hopkins; Rebecca LaPaze; Jeanne Sung; and Shawn Ward made key
contributions to this report.
[End of section]
Footnotes:
[1] See, for example, GAO, Elections: All Levels of Government Are
Needed to Address Electronic Voting System Challenges, GAO-07-576T
(Washington, D.C.: Mar. 7, 2007); Elections: The Nation's Evolving
Election System as Reflected in the November 2004 General Election, GAO-
06-450 (Washington, D.C.: June 6, 2006); Elections: Federal Efforts to
Improve Security and Reliability of Electronic Voting Systems Are Under
Way, but Key Activities Need to Be Completed, GAO-05-956 (Washington,
D.C.: Sept. 21, 2005); Elections: Perspectives on Activities and
Challenges Across the Nation, GAO-02-3 (Washington, D.C.: Oct. 15,
2001); Elections: Status and Use of Federal Voting Equipment Standards,
GAO-02-52 (Washington, D.C.: Oct. 15, 2001); and Elections: A Framework
for Evaluating Reform Proposals, GAO-02-90 (Washington, D.C.: Oct. 15,
2001).
[2] GAO-02-52.
[3] 42 U.S.C. � 15371.
[4] NASED is an independent, nongovernmental organization consisting of
state election officials. It voluntarily formed the first national
program to test and qualify voting systems to federal standards.
[5] GAO-02-3.
[6] GAO-02-3.
[7] GAO-05-956.
[8] GAO-05-956.
[9] Help America Vote Act, Pub. L. No. 107-252 (Oct. 29, 2002).
[10] GAO-04-975T.
[11] The Technical Guidelines Development Committee was established
under HAVA to assist EAC in developing the VVSG.
[12] GAO, Elections: Federal Programs for Accrediting Laboratories That
Test Voting Systems Need to Be Better Defined and Implemented, GAO-08-
770 (Washington, D.C.: Sept. 9, 2008).
[13] The implementation statement provides a summary description of the
voting system's capabilities, including identification of voting system
hardware and software components, version numbers, and dates. It must
also include a checklist identifying all of the VVSG requirements that
the voting system implements.
[14] A signature of a file or set of files is produced using a HASH
algorithm. A file signature, sometimes called a HASH value, consists of
a value that is computationally infeasible of being produced by two
similar but different files. File signatures are used to verify that
files are unmodified from their original versions.
[15] Hardware is commonly verified by identifying the model and
revision numbers of the system's printed wiring boards and major
subunits, and comparing the system with detailed photographs of the
printed wiring boards and internal construction of the unit that was
tested and approved. Software is typically verified by using a self-
booting compact disk or similar device to verify the file signatures of
the system application files and of all nonvolatile files that the
application files access during operation, and comparing against the
file signatures of the original executable files that were placed in
the software repository.
[16] NIST, Conformance Testing and Certification Model for Software
Specifications, Lisa Carnahan, Lynne Rosenthal, and Mark Skall, March
1998; and ISO/IEC, Guide 65: 1996 (E) General Requirement for bodies
operating product certification systems, First Edition; and Guide 60:
2004 (E) Conformity Assessment - Code of good practice, Second Edition.
[17] 42 U.S.C. � 15371.
[18] NIST, Conformance Testing and Certification Framework, Lisa
Carnahan, Lynne Rosenthal, and Mark Skall, April 2001; Conformance
Testing, Martha Gray, Alan Goldfine, Lynne Rosenthal, and Lisa
Carnahan; and Conformance Testing and Certification Model; and ISO/IEC,
Guide 65: 1996 (E); Guide 60: 2004 (E); and Guide 7: 1994 (E)
Guidelines for drafting of standards suitable for use for conformity
assessment, Second Edition.
[19] EAC has not yet established a date for finalizing these
guidelines.
[20] According to the Program Director, the program has requested
additional certification resources for fiscal year 2009.
[21] In a May 2008 correspondence to state election officials, EAC
cited several issues that have contributed to delays in certifying
voting systems, such as manufacturers and VSTLs having to adjust to the
demands of the commission's program and the lack of standardized test
methods.
[22] The trusted build produces a file signature of the executable code
as well as file signatures of the installation disk(s). A file
signature is produced using a HASH algorithm, sometimes called a HASH
value, which creates a value that is computationally infeasible of
being produced by two similar but different files. These file
signatures are deposited into EAC's designated repository. Election
jurisdictions can refer to the file signatures of the installation
disk(s) to validate the software before installing it on the voting
system.
[23] According to Section 7.4.6 of Volume I of the 2005 VVSG, the
verification process should be able to be performed using commercial
off-the-shelf software and hardware available from sources other than
the voting system manufacturer. If such tools are commercially
available, then election officials will be able to independently
acquire the tools needed to verify that their systems are unmodified.
[24] ISO/IEC, Guide 65: 1996 (E); Guide 67: 2004 (E) Conformity
assessment - Fundamentals of product certification, First Edition;
Guide 28: 1982 (E) General rules for a model third-party certification
system for products, First Edition; and Guide 53: 2005 (E) Conformity
assessment - Guidance on the use of an organization's quality
management system in product certification, Second Edition; and ISO,
Guide 27: 1983 (E) Guidelines for corrective action to be taken by a
certification body in the event of misuse of its mark of conformity,
First Edition.
[25] EAC is responsible for, among other things, serving as a national
clearinghouse and resource for the compilation of information by, for
example, carrying out duties related to the testing, certification,
decertification, and recertification of voting system hardware and
software. 42 U.S.C. � 15322.
[26] For fiscal year 2009, EAC requested $16,679,000 in appropriated
funds for salaries and expenses. Of this amount, $4,202,000 was for
personnel compensation providing for 37 full-time-equivalent (FTE) EAC
staff salaries. The 37 FTEs include 29 full-time-permanent employees
and 8 experts, consultants, contract employees, students, and other
part-time employees.
[27] According to 5 C.F.R. � 304.103(c)(2)(i), an agency may reappoint
an expert or consultant, with no limit on the number of reappointments,
as long as the individual is paid for no more than 6 months of work in
a service year.
[28] According to HAVA, EAC shall serve as a national clearinghouse and
resource for the compilation of information, including information on
the experiences of state and local governments in, for example,
operating voting systems.
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office:
441 G Street NW, Room LM:
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: [email protected]:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, [email protected]:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, [email protected]:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
*** End of document. ***