Federal Register, Volume 64 Issue 56 (Wednesday, March 24, 1999)

[Federal Register Volume 64, Number 56 (Wednesday, March 24, 1999)]
[Notices]
[Pages 14336-14345]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 99-7148]

[[Page 14335]]

_______________________________________________________________________

Part III

Department of Labor

_______________________________________________________________________

Employment and Training Administration

_______________________________________________________________________

Consultation Paper on Performance Accountability Measurement for the
Workforce Investment System Under Title I of the Workforce Investment
Act; Notice

Federal Register / Vol. 64, No. 56 / Wednesday, March 24, 1999 /
Notices

[[Page 14336]]

DEPARTMENT OF LABOR

Employment and Training Administration

Consultation Paper on Performance Accountability Measurement for
the Workforce Investment System Under Title I of the Workforce
Investment Act

AGENCY: Employment and Training Administration, Labor.

ACTION: Notice.

-----------------------------------------------------------------------

SUMMARY: The purpose of this notice is to disseminate consultation
papers for interested parties on the Performance Accountability
Measurement System for Title I of the Workforce Investment Act. There
are two papers. The first presents the broad framework for Core
Measures of Performance and Customer Satisfaction specified in Title I,
Section 136. The second presents the framework for Negotiating State
Adjusted Levels of Performance as specified in Title I Section 136.
These papers are to be used by States intending to implement the
Workforce Investment Act as of July 1, 1999. The Department of Labor
will work with States individually to ensure that there are no negative
consequences if significant changes occur in these papers based on the
comments received. Interested parties have 30 days to provide comments
on these papers. Over the next several months additional consultation
papers will also be disseminated for comment.

FOR FURTHER INFORMATION CONTACT: Mr. Eric Johnson, Workforce Investment
Implementation Taskforce Office, U.S. Department of Labor, 200
Constitution Avenue, NW, Room S5513, Washington, DC, Telephone: (202)
219-0316 (voice) (This is not a toll-free number), or 1-800-326-2577
(TDD). Information may also be found, or comments provided, at the
website--http://usworkforce.org.

SUPPLEMENTARY INFORMATION: The Workforce Investment Act (WIA or Act),
Pub. L. 105-220 (August 7, 1998) provides the framework for a reformed
National workforce and employment system designed to meet the needs of
the Nation's employers, job seekers and those who want to further their
careers. Title I of WIA specifies Core Performance and Customer
Satisfaction measures. Each Governor must submit a five-year strategic
plan no later than April 1, 1999, to begin WIA programs by July 1,
1999, and no later than April 1, 2000, to begin WIA programs by July 1,
2000. The current law, the Job Training Partnership Act, is repealed
effective July 1, 2000. States planning to implement during PY 1999 are
to utilize these papers in addressing Performance Accountability in
their plans.
An important part of the five-year strategic plan is the
establishment of performance levels for each of the core performance
and customer satisfaction measures, which will be negotiated between
the Governor and the Secretary of Labor. These levels will form the
basis for incentives and sanctions as specified in Title I, section 136
and Title V, section 503 of the Workforce Investment Act.
The U. S. Department of Labor is establishing this performance
accountability measurement system, and the process for reaching
agreement on State adjusted levels of performance. These two
consultation papers are part of that effort. Some of the questions on
which the Department of Labor is seeking input are the following:
Which services would be appropriately defined as self-
service/informational and thus not included in the core measures, and
which services fall into the core services, intensive services or
training;
The point at which adult and youth registrants are counted
for different performance measures (e.g., at a certain time after
registration, during the reporting period, after completion of service,
after program exit);
The use of wage records for performance measurement
considering availability, completeness, accuracy, timeliness and when
wage records might be combined with supplemental sources (i.e.,
administrative records or survey data for performance purposes)
considering the need for consistency, comparability and cost
effectiveness;
Who will be counted in the numerator and/or denominator of
those measures expressed as rates, for example how should the employed
and underemployed who receive services be accounted for;
Identifying possible unintended effects resulting from
definitions/policies around performance measurements;
Identifying burdensome and unnecessary requirements that
will provide limited benefit, but will be costly in terms of both
record keeping requirements and processing;
Using adjustment models in (1) negotiating State adjusted
levels of performance to account for differences in service mix,
participant characteristics and labor markets and/or (2) determining
eligibility for incentives and consideration for sanctions comparing
negotiation assumptions with actual information.
Sources and types of information that would be useful in
negotiating State adjusted levels of performance;
The circumstances in which revisions to State adjusted
levels of performance will be required by the Department, including
special circumstances for early implementing States and the differences
between the first program year and subsequent years covered by a
State's 5-year plan. For example, if better data available in later
years or if actual performance data shows that the assumptions under
which State adjusted performance levels were negotiated are incorrect,
should the Department require that approved levels for later years be
changed.
Please consider these issues as you review these consultation
papers, and provide comments.

Signed at Washington, DC, this 17th day of March 1999.
Raymond L. Bramucci,
Assistant Secretary of Labor, Employment and Training Administration.

Attachment 1--Performance Accountability Measurement for the
Workforce Investment System

I. Introduction

The Workforce Investment Act (WIA) calls for a comprehensive
accountability system to assess the effectiveness of State and local
areas in providing employability and training services. The Act
requires:
A focus on results defined by ``core indicators'' of
performance;
Customer Satisfaction with programs and services measured
and related to results;
A strong emphases on Continuous Improvement of Services;
Annual performance levels and improvement plans developed
during negotiations among Federal, State and local partners;
Awards and Sanctions based on State performance; and
State reporting and record keeping.
In addition, States are required to provide annual reports to the
Secretary of Labor with respect to progress in achieving State
performance measures. The Act requires certain additional information
be provided, such as cost of workforce investment activities and
specified recipient data.
This paper presents a draft framework for the workforce investment
system core performance and customer service measures that apply to
States in Title I of the WIA and will be used in determining State
adjusted performance levels, eligibility for incentive grants or
imposing sanctions. Additional papers on all of the above listed
requirements

[[Page 14337]]

are being developed and will be provided for comment in the coming
weeks. This paper is intended to elicit discussion about how success
will be defined for workforce investment system activities and how it
can efficiently and effectively be measured Statewide. The concepts
within the paper build on previous Department of Labor (DOL) efforts
such as the Workforce Development Performance Measures Initiative, the
Labor Exchange Performance Measures Work Group, Simply Better!,
Employment Service (ES) Reinvention and the Enterprise. It also
incorporates input from State and local officials that was received at
recent consultations focusing on WIA accountability and from other WIA
briefings and communications.
Please keep in mind that this document presents an overall
framework. It does not fully address a number of the detailed technical
and operational issues that were raised at recent consultations, nor is
it intended to serve as reporting instructions. In addition to your
feedback on this framework, we are also interested in input on
technical and operational issues that may not have been addressed by
this paper. All of this input will be used to develop further guidance
and finalize the document for use by those States planning to implement
the Workforce Investment Act before July 2000. This paper includes
proposed definitions for the Core Indicators of Performance that will
be used for State incentive grant eligibility determinations and
sanctions.
The paper is divided into three sections:
Adult Performance Measures and Definitions that will apply
separately to: (1) Adult Services, (2) Dislocated Worker services, and
(3) Services to Eligible Youth 19 to 21 years old in Youth programs
under Section 129 of WIA.
Youth Performance Measures and Definitions for Services to
Eligible Youth 14 to 18 years old in the Youth Program.
Customer Satisfaction Measures for Participants and
Employers.
While the specific core performance indicators and customer
satisfaction indicators outlined in this paper only apply by law to
Title I of WIA, DOL may adopt them, as appropriate, for other DOL
programs, and will work in cooperation with other Federal partner
agencies to reach agreement where feasible on uniform measures. Thus,
all partners are encouraged to review and comment on this draft
framework. However, any changes to other programs whether internal or
external to DOL would require appropriate actions based upon present
Laws, Regulations and/or policies.
Definitions to be used by all States and localities are provided
for each of the core indicators and customer satisfaction measures to
ensure comparability. Comparability of measures among States is
important for two reasons. First, core indicators and the customer
satisfaction performance levels are to be negotiated between the States
and DOL. One of the factors affecting those negotiations are ``how the
levels compare with State-adjusted levels of performance established
for other States * * *'' Second, comparability also contributes to
continuous improvement. Having standard definitions will allow States
and localities to benchmark other States and localities to promote
continuous improvement. Comparability also will facilitate the sharing
of best practices within and among the States. Since the performance
and accountability system under WIA includes incentives and sanctions,
comparability is important to fairness and equity.
Continuous improvement is a significant and required element of
WIA. States and localities need to collect more substantial data than
the core measures or other required measures under the Act to function
in a continuous improvement environment. Therefore, it is important for
State and local leadership to take advantage of the opportunity when
developing their performance systems to go beyond Federal requirements.

II. Adult/Dislocated Worker Services

A. Workforce Investment Act (WIA) Requirements/Program Activity
Categories for Reporting

The WIA provides for a continuum of service delivery that includes
three levels of services: (1) core services; (2) intensive services;
and (3) training services. All persons will have access to core
employment-related information and self-service tools without
restrictions or additional eligibility requirements, assuming
sufficient funds are available. Those core services that are not
primarily informational and must be staff-assisted will require WIA
registration. Intensive services and training will also require WIA
registration. The intensive services are provided when a determination
is made that unemployed individuals are or would be unable to obtain
employment after receiving the basic core services, or when employed
individuals are determined to be in need of these intensive services to
obtain or retain employment that allows for self-sufficiency.
Similarly, training services are only available after a determination
that the individual is unable to obtain or retain employment that leads
to self-sufficiency through intensive services.
For accountability purposes, WIA establishes core indicators of
performance in State and local WIA financed systems for participants in
all workforce investment activities other than self-service and
informational activities. This exception recognizes the low cost per
unit of providing these services. WIA also requires that States and
localities report on how participants in workforce investment
activities other than training (except for self-service and
informational activities) compare to participants in training
activities. Therefore, the level of service individuals receive defines
whether the individual will be counted in the core indicators and if
so, how they will be categorized for reporting purposes.
Many of the core services generally will be low cost, self-service
and consist primarily of information and not require registration. In
contrast, intensive services will be more costly, require significantly
more staff investment, and thus, justify a different measurement system
that calls for registering and tracking individuals throughout their
program participation. For reporting, services are divided into--
Core Services (for registered participants)
Intensive Services
Training
Consistent with WIA, participants who use one-stop self-service
facilities or only access information do not need to be registered and
tracked. Access to some Core Services will be universally available
through the Internet, at a One-Stop center or through a One-Stop
partner. States and local Boards will be free to allow completely
anonymous access to core services that are primarily information and
universally available (for example, browsing a job bank or using a
computer in the resource room). However, States and local Boards may be
encouraged to request unique identifying information about customers
who use the Internet (for example, current America's Job Bank account)
and who use the universally available self-service capacity in One-Stop
centers.
For customers who are assessed for purposes of determining whether
they require services or training that are not universally available,
additional information will need to be collected as part of the
assessment process. This information will include demographic data
elements such as racial-ethnic

[[Page 14338]]

characteristics, veteran status and information on disabilities, and
will assist in referring individuals to other partners' services and
will be available for comparisons of program applicants and
registrants.

B. What Services Fall Into What Category?

The categorization of services is a State or local decision and
will depend on the nature of the service. To serve as a guide and to
assist in this identification, Table 1 includes most of the core,
intensive, and training services described in Section 134(d). Each of
the required WIA services is italicized. Frequently provided services
that are in addition to those required by the legislation are not
italicized. Given the wide variation in types of service that can be
categorized as job search and placement assistance, and career
counseling, finer distinctions have been made for these services. As
soon as a participant moves from the self-service/informational level
of service to registered service (core, intensive, training) core
measures apply.

Table 1.--Categories of Services
----------------------------------------------------------------------------------------------------------------
C. Intensive services D. Training
A. Core services-- Self-service and B. Other core services (registration (registration
information (registration required) required) required)
----------------------------------------------------------------------------------------------------------------
Determination of eligibility to Follow-up services, Comprehensive and Occupational skills
receive assistance. including counseling for specialized training.
registrants (those assessment, including
previously receiving diagnostic testing
intensive/training and interviewing.
services) after entering
employment.
Outreach, intake (which may include Individual job development. Development of On the Job Training.
profiling), and orientation to the individual employment
One-Stop center. plan.
Initial assessment of skill levels, Job clubs.................. Group counseling...... Workplace training and
aptitudes, abilities, and support cooperative education
service. programs.
Labor Market Information........... Screened referrals (testing Individual counseling Private sector
and background checks done and career planning. training programs.
before referral or when
operating as the employers
agent).
Consumer reports information and ........................... Case management....... Skill upgrading and
delivery system performance retraining.
information.
Information on other One-Stop ........................... Short term pre- Entrepreneurial
partner services and supportive vocational services. training.
services.
Information on filing UI claims.... ........................... ...................... Job readiness
training.
Assistance in establishing WtW ........................... ...................... Adult education and
eligibility and other non-WIA literacy activities
training and education. in combination with
training.
Resource Room usage................ ........................... ...................... Customized training.
``How to'' group sessions (e.g.
writing a resume).
Job referrals (informational, e.g.,
job scouts, ES referrals in non-
exclusive hiring arrangements,
short-term or seasonal
placements).
Internet browsing--job, info, and
training searches.
Internet accounts--Career Kit,
Personnel Kit.
Talent referrals (informational,
e.g., talent scouts, ES staff
referrals of resumes without
further screening).
----------------------------------------------------------------------------------------------------------------

The following considerations provided some of the rationale used in
preparing this guide. First, ``self-service and informational
activities'' are by their nature core services that do not require
registration and tracking. A second consideration is the likely per
unit cost of services. A number of placement activities are primarily
informational in nature and relatively low cost. In these instances,
the added cost of registration and participant tracking may not be
justifiable. Thirdly, some services benefit participants but are
undertaken primarily for their value to employers (e.g., assistance
with recruitment for seasonal work--summer or holiday) are intended
only to provide short term employment and do not necessarily increase
worker earnings, retention or occupational skill attainment. Fourthly,
these groupings of activities are intended to be clean, easy to
administer, and applicable to all programs.

[[Page 14339]]

C. How to Measure Core Services, Intensive Services and Training

1. Core Indicators of Performance

----------------------------------------------------------------------------------------------------------------
Measure Definition
----------------------------------------------------------------------------------------------------------------
Entry into Unsubsidized Employment:
Entered Employment Rate (Sec 136(b)(2)(A)(i)(I)) The rate will be defined for cohorts of registered
participants unemployed at the time of registration. The
numerator will be the number of these registered
participants that are shown to have paid employment in
the quarter following registration or service completion.
The denominator will be all registered participants
unemployed at the time of registration who were active
during the reporting period (received services or
continuing from a prior period) but who are no longer
actively receiving services, other than post-employment
services. This includes enrolled participants who (1)
have obtained unsubsidized employment; (2) have withdrawn
from participation; or (3) who have completed training or
services. Individuals should be considered no longer
active and to have completed service if they have
received no services in the last quarter of the reporting
period, and are not scheduled to receive services in the
future.
Note: State and local officials opposed using a program
exit or termination to trigger reporting.
Records of all registered participants unemployed at the
time of registration, and not enrolled in a training
program at the end of the reporting period would be drawn
and matched against wage records to identify employment.
Dislocated workers as defined in WIA, Title I, subtitle
A, sec.101 (9) are included in the definition of
unemployed. A person is considered employed if his/her
social security number appears in the employer wage
report (no minimal wage requirement) in the quarter
following the one in which the seed record for matching
is drawn.
Not all jobs are covered by State UI wage records.
Therefore, a State or locality may supplement the results
of the wage record review by other methods and count as
employed any of these individuals in jobs not covered by
the State's UI wage records. Again, employment would be
determined based on employment in the quarter following
the one in which the seed record is drawn. Supplementary
information could include: use of the New Hire index,
surveys, self-reported hires or staff-reported hires
through an administrative record system.
Seed records not shown as employed would be matched
against administrative records to exclude from the
computation individuals who remain active, i.e, have
received services in the last quarter or are scheduled to
receive them in the future.
Retention in unsubsidized employment Six Months
after entry into the employment:
Six Month Retention Rate (Sec The rate is computed using information on the total number
136(b)(2)(A)(i)(II)). of registered participants who have employment and who
appear in the wage records, and wage record information
for the second quarter thereafter (6 month rate). For
example, an individual completing training and placed
immediately in the first quarter of the program year,
would be recorded as employed in the second quarter. The
fourth quarter records would be queried to determine
retention.
Note: retention is not limited to the same employer.
Earnings Received in Unsubsidized Employment Six
Months After Entry into Employment:
Average Earnings Change in Six months (Sec The average earnings change is measured as follows: the
136(b)(2)(A)(i)(III)). wage record earnings for the registered participant in
the two quarters following employment (not counting the
quarter in which employment was recorded) less 50% of the
wage record earnings for the four quarters prior to
enrollment (not counting the quarter of enrollment). The
post-employment income can be with the same or other
employer in which the placement was first noted. The
measure is reported as an average (mean) gain. For
Incumbent workers and others who are employed at the time
of registration average earnings and retention would be
computed begining in the second quarter following the
quarter in which services are completed.
Educational Credential/Occupational Skills
Credential Attainment Rate (Training Services For adults entering employment after training and eligible
Only) (Sec 136(b)(2)(A)(i)(IV)). youth 19 through 21 entering employment, post-secondary
education or advanced training after training, the
percent who attained a State-recognized credential
related to educational skill attainment (diploma, degree
or certificate) or attainment of an occupational skill
(license or certification) recognized by a State or a
Nationally-recognized industry trade body. Information in
administrative records or information gathered through
electronic interfaces with other data bases available or
surveys may be used. Additional guidance on acceptable
credentials and certificates will be provided.
----------------------------------------------------------------------------------------------------------------

2. Why These Measures and Definitions?
Wage record versus surveys. WIA requires that wage records be used
but does not exclude the use of surveys to supplement information.
Where the same information is obtainable from both sources,
instructions will provide that wage records be used both to ensure
comparability among States and to minimize costs.
Earnings change versus absolute earnings level. A change--gain/
loss--measure was chosen because of the different circumstances within
the WIA service population and the difficulty of creating an absolute
earnings target for all adults.
Earnings change--prior period earnings. Six month comparisons are
made with a full twelve month period prior to registration. (Fifty
percent is used for the six month change for comparability of periods.)
The twelve month period is used to minimize, to the extent possible,
the impact on dislocated workers of reduced earnings as layoffs
approach and any severance pay that may be added to earnings.
Retention rate. While most of the measures use the registered
population as the base, this measure chooses only those who are
employed because the retention question makes no sense in another
context.

[[Page 14340]]

Credentials rate (Training Service only). Adults entering
employment after receipt of core or intensive services also may acquire
some credential as a result of WIA participation. However, given the
few relative to the number who will be employed after receiving
services, the cost of this added information collection is not
justified.

III. Youth 14 to 18 Years Old

A. WIA Requirements/Guiding Principles

WIA authorizes programs to provide services to prepare youth to
enter the workforce or to advance to postsecondary education or other
occupational skills training. Programs link academic and occupational
learning. Service providers will have strong ties to employers and must
also include tutoring, study skills training and instruction leading to
completion of secondary school (including dropout prevention). Other
elements of programs should include alternative school services,
mentoring by appropriate adults, paid and unpaid work experience (such
as internships and job shadowing, occupational skills training,
leadership development, and appropriate supportive services.) Youth
participants also will receive guidance and counseling and follow-up
services for at least one year. Programs must also provide summer job
opportunities linked to academic and occupational learning. The mix of
year-round and summer activities is left to local discretion.
Eligible youth are low-income, ages 14 through 21 with barriers to
employment. WIA specifies different youth core indicators for older
youth ages 19 through 21 and for youth ages 14 through 18. The older
youth performance measures are identical to the adult program measures
and were addressed earlier in the paper. The three required core
indicators for youth ages 14 through 18 are--
Attainment of basic skills and, as appropriate, work
readiness or occupational skills;
Attainment of secondary school diplomas and their
recognized equivalent;
Placement and retention in postsecondary education or
advanced training, or placement and retention in military service,
employment, or qualified apprenticeships.

B. Guiding Principles

Performance measures should reflect the same flexibility
available for program design and services and the varied successful
outcomes recognized. WIA allows a wide variety of services that are
offered to youth. This provides the opportunity for local programs to
design and operate programs that meet local needs and respond to gaps
in services as they are identified for their area. Indicators should
not force certain designs to remain competitive in terms of
measurement.
Performance measures should accommodate a variety of
different approaches to serving youth without forcing arbitrary time
limits or sequencing of services. Measures should not determine when
youth begin or end services by forcing measurements before participants
are ready to complete their goals or require that a youth leave the
program before credit can be taken for outcomes achieved. Research has
shown that programs that establish an ongoing relationship with youth,
and continue to serve them for several years while adjusting goals and
services to reflect needs as youth age have the greatest success.
Performance measures need to recognize that youth goals
change as youth mature and must be age appropriate. The denominator for
various rates should depend on the appropriateness of the goal as
determined through individual service plans. For example, younger youth
participants should receive services that encourage staying in or
returning to school and keeping up academically. As the participants
get older, goals will change and relate to getting a secondary school
diploma, and, ultimately, placement and retention in post-secondary
education, advanced training or employment.
Indicators must (1) recognize the differing goals
depending on the activities/services; (2) the age of the youth; and (3)
allow comparison between activities/services that include modest levels
of summer job opportunities and those that invest a large proportion of
resources during the summer. WIA provides for summer employment
opportunities directly linked to academic and occupational learning,
but intentionally did not provide a separate funding stream or
differing set of indicators. In addition, the summer employment
opportunities element is not intended to be a stand alone program (WIA
sec. 129(c)(2)(C)). Indicators should be designed so that achieving
every goal established for every youth--whether for a year-round or
summer employment opportunities only--would result in a 100 percent
rate.

C. Establishing a Basis for Performance Measurement

State and local officials with whom DOL consulted strongly believed
that the core indicators of performance for participants should reflect
completion of their activities, not necessarily completion of
participation. By design, youth programs are intended to provide a
continuum of services for youth resulting in attainment of several
interim outcomes such as the acquisition of basic skills, work
readiness, or occupational skills, award of a secondary school degree,
and then, placement and retention in employment or advanced education/
training. Participant goals are reflected in individual service
strategies and will be different from one youth to another, depending
on the needs and interests of the youth.
1. Core Indicators of Performance

[[Page 14341]]

Attainment of secondary school diplomas or
recognized equivalents
Diplomas or Equivalent Attainment Rate (Sec A rate computed by dividing the number of youth who
136(b)(2)(A)(ii)(II)). attained a secondary school diploma or equivalent divided
by the number of youth for whom attaining a diploma or
certificate was a goal to be achieved during the
reporting period. This goal will generally be appropriate
for older youth 16 or 18 years old.
Placement and retention in postsecondary education
or advanced training, or placement and retention in
military service, employment, or qualified
apprenticeships:
Retention Rate (Sec 136(b)(2)(A)(ii)(III))...... Of those who are receiving follow-up services and for whom
placement and retention is a goal, the percent with
retention status at 30 days, 90 days, 180 days and one
year from beginning follow-up. This overall rate would be
an average of measures for all four periods.
----------------------------------------------------------------------------------------------------------------

2. Age as the Basis for Outcomes.
These younger youth reporting requirements are for participants who
are 14 to 18 when enrolled in the youth program. They are appropriate
as long as the youth continues to receive services identified in their
individual service plans and has not attained the outcomes established
for that age group. If the youth were going to continue being served by
the youth program after achieving goals established prior to age 19,
the youth does not need to be ``terminated'' but instead the record
would indicate that the youth transferred to the age 19 to 21 youth
program. This would have the same effect as a ``termination'' for
performance measurement purposes but would not disrupt services to the
youth. Individuals ages 19 through 21 who have completed the goals
established for them prior to turning 19 and those who enter the youth
program as age 19 or older should have individual service plans leading
to the attainment of the goals appropriate for that age group. These
outcomes would be measured in accordance with the same principles
established for the adult program.

D. Why These Measures and Definitions?

Skill Attainment Rate; Attainment of secondary school diplomas and
their recognized equivalents: and Placement in postsecondary education
or advanced training, or placement in military service, employment, or
qualified apprenticeships. All three core indicators use as their
denominators the total number of youth age 14 through 18 for whom the
particular outcome to be measured was a goal. This recognizes that
depending on the age of the youth and their needs, individual goals
will differ significantly and that the effectiveness and quality of a
program should be measured by its experience in achieving goals
established for its participants. It allows comparisons of programs and
allows for program goals to be established without having to know the
exact mix of ages of participants and relative investments in different
activities. It also permits a 100 percent rate if all goals are
achieved.

Note: A youth participating in WIA youth activities/services for
multiple reporting periods (years), could be counted during each of
the periods (years) if goals established in each are achieved.

Placement Rate and Retention Rate. Unlike the adult program where
separate measures are provided for placement and retention, WIA calls
for a single youth indicator that includes both placement and retention
rates in one measure. The retention rate--for unsubsidized employment
and further education--was chosen as the core indicator because of the
difficulty in coming up with a single measure that measures placement
and retention. A meaningful measure with both would be difficult
because of complications coming up with a single denominator. Placement
will be measured, but outside the core indicators.
Unsubsidized employment. WIA specifies that adults be measured with
regard to placement in unsubsidized employment. The Youth indicator
specifies only placement in employment. Unsubsidized employment is
being used for consistency and because it, and not subsidized
employment, will lead to self-sufficiency.

IV. Customer Satisfaction

A. Workforce Investment Act Requirements

In addition to the core measures, WIA, at sec. 136(b)(2)(B), states
that ``the customer satisfaction indicator of performance shall consist
of customer satisfaction of employers and participants with services
received from the workforce investment activities authorized under this
subtitle.'' The Statute also requires, at sec. 136(b)(3)(A)(i), that
there be State adjusted levels of performance for customer satisfaction
and that ``the levels of performance established * * * shall, at a
minimum--
(I) Be expressed in an objective, quantifiable, and measurable
form; and
(II) Show the progress of the State toward continuously improving
on performance.''
The Act draws a link between the core measures and customer
satisfaction by indicating that negotiations between DOL and the States
must take into consideration ``* * * the extent to which the levels
involved will assist the State in attaining a high level of customer
satisfaction.'' WIA further suggests that ``customer satisfaction may
be measured through surveys conducted after the conclusion of
participation in workforce investment activities.''

B. Methods To Measure Customer Satisfaction

Customer satisfaction measures are important because they provide
valuable information from customers for strategic planning and program
operation. Such feedback to supervisors and staff can motivate high
performance and continuous improvement. They also send a clear message
to staff, management, and customers themselves that customers matter.

[[Page 14342]]

There are a number of different methods to collect customer
satisfaction information. The simplest approach is to train staff to
listen to the customers they serve and to ask questions that elicit
customer needs while they are providing service. Focus groups and group
interviews are another strategy. A trained manager or staff person can
circulate in the resource center where people are waiting and ask
questions informally to gain a better understanding of customer needs
and concerns. Suggestion boxes are also a way of gathering information.
As part of a comprehensive continuous improvement strategy,
organizations generally use a combination of strategies since each
serves a somewhat different purpose and provides different types of
information.
To meet the customer satisfaction requirements of WIA, the
recommended measures focus on customer satisfaction surveys. This is
the only method that allows State and National aggregation of
comparable, quantifiable data. The proposed measures present a general
framework for developing a National customer satisfaction index for
different customers.
An index is a single score that is created by combining the scores
from several questions that address different dimensions of the
customer experience. The customer satisfaction index will be described
in more detail in the proposed measures. Essentially, the index would
provide a way to capture common customer satisfaction information
across programs that could be aggregated to a State and National level.
The proposed measures will continue to be modified as DOL receives
feedback and validation through consultation with the workforce
investment system.

C. Proposed Customer Satisfaction Measurement Strategy

WIA requires measures of customer satisfaction for both
participants and employers that are quantifiable, comparable across
States, and that, along with the core indicators of performance,
promote continuous improvement.
The basic approach for gathering and reporting customer
satisfaction measures that meet these requirements will be as follows--
There will be separate surveys of participants and
employers;
Customer satisfaction indicators will be derived from
State or locally conducted surveys that include a few embedded
questions that enable comparisons across States, while allowing each
State to design its own instrument;
Guidelines will be issued to the States requiring the
States to validate survey methods to ensure comparability across
States;
Participants surveyed will include only registered
individuals who have completed their activity/service participation
(excluding follow-up services). This includes individuals who have
completed services and are now employed, those who have discontinued
receiving services or training, either because they have withdrawn or
completed;
Employer surveys can be based on a random sample of
employers using the system; and
Results on the embedded questions, for both employers and
participants, will be compiled on the State level and reported
annually.
While the Act requires reporting and comparisons across States, the
primary value of effective customer feedback is to drive strategic
planning and continuous improvement at the local level. DOL plans to
play a strong, proactive technical assistance role that will provide
States and localities with easy access to the information, instruments,
tools and other resources that will enable them to use this feedback as
a springboard toward high performance and quality services.
ETA will finance the design and development of customer
satisfaction instruments and methodologies that meet requirements, and
make these available to the States. However, States will have the
option of designing their own instruments and methods as long as the
few questions that enable comparisons are embedded in the State
instruments and the methodologies used to collect responses on those
questions are within guidelines that DOL will develop. These guidelines
will include suggestions for establishing State baseline levels to be
used where they do not already exist.

Attachment II--Reaching Agreement on State Adjusted Levels of
Performance Under Title I of the Workforce Investment Act of 1998;
A Consultation Paper

This Consultation Paper provides a framework regarding the approach
and process for reaching agreement on State adjusted levels of
performance under Title I of the Workforce Investment Act of 1998 to be
incorporated in States' five-year strategic plans required by that Act.
Comments are solicited on the overall framework and the approach. The
paper does not address the additional indicators a State may identify
in the State plan in accordance with Title I, because they are not
subject to the agreement process.

I. Introduction

A. Broad Legal Framework

In Section 136, the Workforce Investment Act (WIA) establishes a
comprehensive performance accountability system for the Statewide and
local workforce investment systems. Its purpose is to assess the
effectiveness of States and local areas in achieving continuous
improvement of their Federally-funded workforce investment activities,
in order to optimize the return on investment of Federal funds in those
activities.
As part of the performance accountability system, WIA requires the
Secretary and the Governor of each State to reach agreement on the
respective State performance levels for the core indicators of
performance and the customer satisfaction indicator. WIA requires that
the State performance levels be expressed in an objective,
quantifiable, and measurable form. The law also requires that the
levels show the State's progress toward continuously improving in
performance. The negotiated performance levels become ``State adjusted
levels of performance'' and must be incorporated into the State's 5-
year plan. They become the basis for sanctions for failed performance.
Together with adjusted levels of performance for adult education
literacy programs and performance levels for vocational education
programs, they become the basis for incentive grants.
State adjusted levels of performance for the first three program
years covered by a State 5-year plan are agreed to during the plan
review and approval process. In reaching agreement on those levels, the
Secretary and the Governor must take into account the expected levels
of performance identified by the State in its 5-year plan. The
Secretary and the Governor also must take into account the following
three factors: (1) the extent to which the levels will help the State
achieve a high level of customer satisfaction; (2) how the levels
compare to those of other States, with consideration of, at least,
differences in economic conditions, participant characteristics at
entry into the program, and services to be provided; and (3) the extent
to which the levels promote continuous improvement in performance and
ensure optimal return on investment. State adjusted levels of
performance for the fourth and fifth program years covered by a State
5-year plan must be agreed to before the beginning of the fourth
program year.

[[Page 14343]]

The State plan is not required to identify expected levels of
performance for those years. In reaching agreement, the Secretary and
the Governor must take into account the same three factors listed
above. The Governor may request revision of the agreed to performance
levels if unanticipated new circumstances in the State result in
significant change in any of factors for any year.

B. Guiding Principles

WIA reflects the Nation's commitment to accountability for results
in government programs that is expressed in the Government Performance
and Results Act. It embodies strong commitment to partnership among
government entities in serving people in areas of workforce
development. It promotes flexibility to provide services that provide
maximum benefit to customers and high levels of customer satisfaction.
These commitments are characterized by key principles that must be
reflected in the process of reaching agreement on target levels for
State performance measures.
1. Performance expectations for related workforce investment
programs and for related reporting purposes (e.g., reporting under the
Government Performance and Results Act) should be aligned to facilitate
better management of overall performance.
2. State and Federal partners share the commitment to high
performance and customer satisfaction, and continuous improvement in
both.
3. The process and considerations in reaching agreement on State
performance levels must be consistent for all the States so as to
assure fairness in the derivation of performance levels.
4. States should have maximum flexibility to target populations for
WIA services within the parameters of the law, and to develop and adopt
innovative methods for accomplishing workforce investment objectives;
this flexibility should be reflected in agreed target levels of
performance.
5. States should have maximum flexibility to develop reasonable
methodologies for setting performance goals in the core measurement
areas and customer satisfaction.

C. Assumptions

Critical assumptions underlie the discussion of the process for
reaching State/Federal agreement on State adjusted levels of
performance. These assumptions relate to performance measures,
incentives and sanctions, performance data and tools, and continuous
improvement.
1. A paper on Performance Accountability Measurement for the
Workforce Investment System is being circulated for comment. The
measures described here are drawn from that paper, and so are subject
to change. There are seven measurements for the core indicators of
performance. Four of these apply separately to activities under three
funding streams (adult, dislocated worker, and youth ages 19-22) and
three apply to activities for youth ages 14-18, for a total of fifteen
measures. In addition to those fifteen measures for the core indicators
of performance, there are two measures for the customer satisfaction
indicator. Each State plan will include one State adjusted performance
level for each of the seventeen measures. The indicators and their
measures, grouped by program, are:
Adult Program (four indicators)
Entry into unsubsidized employment, measured by Entered
Employment Rate,
Retention in unsubsidized employment after entry into the
employment, measured by Six Month Retention Rate,
Earnings received in unsubsidized employment six months
after entry into employment, measured by Average Earnings Change in Six
Months,
Attainment of educational credential/occupational skills
credential for adults entering employment after training, measured by
Educational Credential/Occupational Skills Credential Attainment Rate
(Training Services Only);
Dislocated Workers Program
(Four indicators--same as for the Adult Program)

Entry into unsubsidized employment, measured by Entered
Employment Rate,
Retention in unsubsidized employment after entry into the
employment, measured by Six Month Retention Rate,
Earnings received in unsubsidized employment six months
after entry into employment, measured by Average Earnings Change in Six
Months,
Attainment of educational credential/occupational skills
credential for adults entering employment after training, measured by
Educational Credential/Occupational Skills Credential Attainment Rate
(Training Services Only);
Youth Ages 19-22 Program
(four indicators--same as for the Adult Program, with a variation in
the credentials indicator)

Entry into unsubsidized employment, measured by Entered
Employment Rate,
Retention in unsubsidized employment after entry into the
employment, measured by Six Month Retention Rate,
Earnings received in unsubsidized employment six months
after entry into employment, measured by Average Earnings Change in Six
Months,
Attainment of educational credential/occupational skills
credential for youth ages 19-22 entering post-secondary education,
advanced training, or employment after training, measured by
Educational Credential/Occupational Skills Credential Attainment Rate
(Training Services Only),
Youth Ages 14-18 Program
(three indicators)

Attainment of basic skills and, as appropriate, work
readiness or occupational skills, measured by Skill Attainment Rate,
Attainment of secondary school diplomas and their
recognized equivalents, measured by Diplomas and Equivalents Attainment
Rate,
Placement and retention in postsecondary education or
advanced training, or placement and retention in military service,
employment, or qualified apprenticeships, measured by Placement Rate;
Customer Satisfaction for Combined Programs
(two indicators)

Participant satisfaction, measured by an index derived
from several questions on customer satisfaction surveys,
Employer satisfaction, measured by an index derived from
several questions on customer satisfaction surveys.
2. The Department of Labor expects that negotiations will lead to
high State adjusted levels of performance to encourage high
performance.
3. Incentives will be awarded and sanctions will apply based on
performance against State adjusted levels of performance beginning in
the first year of WIA implementation. Only States that exceed their
agreed to levels will be eligible for incentive awards. Sanctions based
on first year performance will not include monetary penalties.
4. The Department of Labor will establish a single estimate of
National average performance for each performance measure. Initially,
because only limited historical outcome data are available to develop
information on likely outcomes for WIA core

[[Page 14344]]

performance indicators, the information will be subject to serious
qualifications. For some indicators, rough estimates of the
distribution of performance among States and simple adjustment models
will be developed as tools for reaching agreement on State adjusted
levels of performance. For other core performance indicators and the
customer satisfaction indicator, the available data will not be
sufficient at the outset to develop such tools. The Department expects
that future negotiations will be informed by discussions with and the
actual performance of the early implementing States.
5. A commitment to continuous improvement is not simply an
agreement to raise the State adjusted levels of performance for
successive years and incrementally improve performance numbers.
Continuous improvement is the process of building dynamic, high
achieving systems within every organization through the ongoing
systematic improvement of products, programs, services, and processes
by both small increments and major breakthroughs. Continuous
improvement encompasses a commitment to a systematic approach to high
performance. Continuous improvement is driven by finding opportunities
to do better, as well as by solving problems that need immediate
correction. It becomes a regular part of daily work, and provides a
method of eliminating problems at their source. Performance measures
and customer satisfaction are integrated into a continuous improvement
approach to focus on where to concentrate resources, or redesign
programs or sequences of services in order to achieve better results.

II. Principal Stages of Agreement Process

There are three principal stages of the process for reaching
agreement on State adjusted levels of performance in State five-year
plans. These stages are defined in terms of plan submittal and
approval. The first stage precedes submittal of the State plan and
includes the Department's provision of information, the State's
planning and development of an ``expected level of performance'' for
each of the prescribed indicators of performance and customer
satisfaction, and preliminary State/Federal discussions. The second
stage begins with the State's submittal of its plan including the
expected levels of performance that serve as the starting point for
negotiations between the State and Federal partners, and ends when the
State adjusted levels of performance are agreed to by the State
Governor and the Secretary of Labor and incorporated into the State
plan. The third stage follows approval of the plan with the agreed
levels and encompasses possible modification of the plan to revise the
State adjusted levels of performance.

A. Stage One: Federal and State Information and Preliminary Discussion

Before any meaningful activity can be accomplished relative to
planning or setting State performance goals, the State and Federal
partners will need an understanding about the process and its
relationship to other processes in the performance accountability
system, e.g., reporting, incentive and sanction policies, GPRA goals,
etc. Ideally, this information would become available before a State
engages in its strategic planning process or sets its ``expected levels
of performance'' and must be available before the negotiation process
begins. Both the State and the Federal partners must gather and assess
information prior to States' submittal of their plans.
1. Federal Information
The Department expects to release information in the following
areas at the earliest possible time. Each of these items will be
covered in papers to be developed and issued for comment.
Specific measures will be identified for the core
performance and customer satisfaction indicators. Definitions will be
provided for those measures, and will include the scope of the
measures, e.g., are they only Title I, all WIA referenced activities,
etc.
Information will be provided about performance measurement
tools under development, such as estimates of the distribution of
performance among States and simple adjustment models. The information
will include sources of data, the expected usefulness of information to
be developed, and for which measures, if any, there will be State-
specific estimates or adjustment models.
The process for negotiating the measures will be
established and communicated to the system. This will include
expectations for how and when the discussions will occur as well as the
kinds of information that must be available from the State to
facilitate the discussions.
Guidance will be issued about the levels that will be
considered acceptable when negotiating adjusted performance levels,
including specific information on:

--Policies that set criteria for evaluating expected levels of
performance;
--Policies for award of incentives and related concepts for meeting and
exceeding WIA Title I performance goals (note that this does not
include the overall policy for consideration of performance WIA Title
II and Carl D. Perkins Act programs); and
--Policies defining sanctions and related definitions for failure to
meet standards.

Guidance will be issued on determining the impact of
adjusted levels of performance on attaining high levels of customer
satisfaction.
Policy will be set to describe the consideration of
continuous improvement in the goal setting process.
Specific data and tools including models, if available,
will be provided for comparing the adjusted levels of performance among
States. Data and tools will continue to be released as they are
developed.
2. State Information
WIA envisions an accountability process that takes into
consideration unique State and local requirements and circumstances. As
States engage in the planning process and develop the expected levels
of performance they will identify in their State plans, they will
gather information that will be useful in the subsequent negotiation
process. States will obtain preliminary information about the economy,
anticipate characteristics of the population to be served, and set
strategies for determining service mix, since these must be considered
in setting performance levels. States will explore pertinent data
sources related to sequenced services and one-stop service systems and
examine their utility in establishing a baseline for the negotiations
process. States will develop information gleaned from an environmental
scan to determine the progress local areas have made in developing a
service delivery system as required in WIA. States will consider the
strength of State/local partnerships among agencies and organizations
that will support the system and strategies under consideration to
strengthen and streamline the delivery system. States are encouraged to
take into account, in developing their expected levels of performance,
the results of the negotiation of local performance levels.
3. Preliminary Discussion
The Department recognizes, with the States, that time is short for
development of State plans, including performance levels, particularly
for those States that expect to implement WIA in July 1999. The
Department appreciates and continues to encourage

[[Page 14345]]

State and local involvement as WIA policies and procedures are
developed. In this spirit of cooperation, preliminary discussion of
performance management, including the development of performance
levels, is welcomed. States are encouraged to contact their Regional
Offices for discussion and technical assistance prior to plan
submittal. The benefits of early discussion could include:
Ensuring understanding of guidance, policy, data, and
technical material provided by the Department prior to plan submittal;
Benefitting from the experience of regional staff and
other States;
Tailoring the provision of technical assistance on
performance accountability to meet local and regional planning needs;
Developing a mutual understanding of State and Federal
expectations and assumptions prior to plan submittal, to ensure
development of a shared set of goals;
Allowing maximum time (in advance of the up-to-ninety-days
plan review period) for States and local areas to complete necessary
planning and consultation on performance levels; and
Smoothing the agreement process.

B. Stage Two: Formal Discussion and Agreement

A State's submittal of its five-year plan to the Secretary of Labor
triggers the up-to-ninety-days review period during which the Federal
and State partners are to reach agreement on the State adjusted levels
of performance for the core and customer satisfaction indicators. The
agreed to levels will be incorporated into the plan prior to its
approval. A State's plan will not be approved if agreement has not been
reached. It is expected that the negotiations will take place between
the States and the Department's Regional Offices, consistent with
guidelines to be issued by the Department to the workforce development
system.
The State plan will include the State's expected level of
performance for each core indicator and customer satisfaction
indicator. The plan will also include an explanation of the derivation
of each expected level. In the formal negotiations, States should be
prepared to provide support for any data and data analysis used in
arriving at the expected levels of performance. States should also be
prepared to discuss any environmental or strategic issues that are
expected to influence performance levels.
In addition to the expected levels of performance identified in the
State plan for the first three years covered by the plan, WIA specifies
three factors that the Governor and the Secretary must take into
consideration in the agreement process. The first is the extent to
which the State adjusted levels of performance will help the State
achieve a high level of customer satisfaction. The second is how those
levels compare to the adjusted levels of other States, considering
differences in economic conditions, characteristics of participants
upon entry into WIA programs, and services to be provided. The third
factor is the extent to which the adjusted levels promote continuous
improvement in performance and ensure optimal return on investment. As
mentioned earlier, the Department expects to issue guidelines for
consideration of all of these factors, to ensure consistency and
fairness in the agreement process. The performance levels, representing
the anticipated results of the comprehensive workforce development
plan, will be considered in the context of the entire plan.
Particularly because of the anticipated limitations of available data
at the outset of WIA implementation, the negotiation process itself is
expected to be a learning experience for the State and Federal
partners.

C. Stage Three: Modifications, and Years Four and Five

WIA specifies that Governors may request revisions to State
adjusted levels of performance for any of the five program years
included in a State plan, based on unanticipated circumstances in their
respective States that result in significant changes in factors
including economic conditions, characteristics of participants, and
services provided. WIA does not prohibit consideration of other
factors. The Secretary will issue guidelines establishing objective
criteria and methods for making revisions requested by Governors. These
guidelines also will specify the conditions under which a State is
required to revise the agreed to levels of performance. The revision
process will be addressed in a separate paper which is expected to be
issued for comment in late April 1999.
Because of the transitional nature of the program for the first
three program years and the lack of data from which predictions of WIA
performance can be derived for each State, there must be allowance for
changes in expected performance beyond the circumstances specified in
WIA. Allowance for changes in performance expectations is particularly
important because a State's performance measured against its State
adjusted levels of performance will affect its eligibility for
incentive grants and its susceptibility to sanctions. As the body of
WIA experience grows--over time in individual States and as more States
implement WIA--more information will become available to permit
development of more useful performance management tools, including
National figures, State-specific information, distributions of
performance data across States, and adjustment models. The effects of
continuous improvement approaches will be better understood and more
predictable as their application is tested. Because of these
anticipated changes, it is expected that State adjusted performance
levels included at the beginning of a State's five-year plan will be
able to be refined as time passes.
Federal guidance will delineate circumstances in which the State
adjusted levels of performance must or may be revised--upward or
downward--for individual States or for all States. Some possibilities
beyond those identified in the law are listed here.
Performance levels are set for all States based on the
pre-WIA wage record experience of a few States, and experience shows
that the predictions were not valid.
The operation of the one-stop system in a State varies
significantly from that discussed during performance negotiations.
Changes in State law, Statewide vision, or strategies have
a significant impact on performance outcomes.
Changes in Federal law or policy have a significant impact
on performance outcomes.
WIA requires that agreement be reached on State adjusted levels of
performance for the fourth and fifth program years covered by a five-
year plan prior to the beginning of the fourth year. The State does not
submit expected levels of performance for those years. The law seems to
contemplate that experience under WIA in the first three years will
provide a sufficient basis for setting levels for subsequent
performance.

[FR Doc. 99-7148 Filed 3-23-99; 8:45 am]
BILLING CODE 4510-30-P