[Federal Register Volume 75, Number 141 (Friday, July 23, 2010)]
[Notices]
[Pages 43172-43177]
From the Federal Register Online via the Government Printing Office [www.gpo.gov]
[FR Doc No: 2010-18013]
-----------------------------------------------------------------------
DEPARTMENT OF HEALTH AND HUMAN SERVICES
Health Resources and Services Administration
Administration for Children and Families
Maternal, Infant, and Early Childhood Home Visiting Program
AGENCY: Health Resources and Services Administration and Administration
for Children and Families, HHS.
ACTION: Request for public comment on criteria for evidence of
effectiveness of home visiting program models for pregnant women,
expectant fathers, and caregivers of children birth through
kindergarten entry.
-----------------------------------------------------------------------
SUMMARY: The Health Resources and Services Administration and
[[Page 43173]]
Administration for Children and Families, HHS, solicit comments by
August 17, 2010 on proposed criteria for evidence of effectiveness of
home visiting program models for pregnant women, expectant fathers, and
primary caregivers of children birth through kindergarten entry. Final
criteria for evidence of effectiveness will be included in the program
announcement inviting eligible entities to apply for funding under the
Affordable Care Act Maternal, Infant, and Early Childhood Home Visiting
Program.
SUPPLEMENTARY INFORMATION: Invitation to Comment: HHS invites comments
regarding this notice, both on the proposed criteria and proposed
methodology for HHS's systematic review of the evidence. To ensure that
your comments have maximum effect, please identify clearly the specific
criterion or other section of this notice that your comment addresses.
1.0 Purpose of Program
The Affordable Care Act (ACA) Maternal, Infant, and Early Childhood
Home Visiting program is designed to strengthen and improve home
visiting programs, improve service coordination for at risk
communities, and identify and provide comprehensive evidence-based home
visiting services to families who reside in at risk communities. The
legislation specifies that most program funds must be used for
``evidence-based'' home visiting program models. This Notice (1)
proposes criteria to be considered in assessing whether home visiting
models have evidence of effectiveness, and (2) describes the
methodology for a systematic review of evidence, applying the criteria
proposed in this Notice, which HHS is currently conducting. The Notice
solicits comments on both items.
2.0 Background
2.1 Legislative Context
On March 23, 2010, the President signed into law the Patient
Protection and Affordable Care Act of 2010 (Pub. L. 111-148) (also
known as the Affordable Care Act or ACA); historic legislation designed
to make quality, affordable health care available to all Americans,
reduce costs, improve health care quality, enhance disease prevention,
and strengthen the health care workforce. Through a provision adding
Section 511 to Title V of the Social Security Act to create the
Maternal, Infant, and Early Childhood Home Visiting program, the Act
responds to the diverse needs of children and families in at risk
communities and provides an historic and unique opportunity for
collaboration at the Federal, State, and local level to assure
coordination and delivery of critical health, development, early
learning, and child abuse and neglect prevention services to most
effectively serve these children and families. By supporting evidence-
based home visiting program models, the ACA Maternal, Infant, and Early
Childhood Home Visiting program plays a crucial role in national
efforts to build quality, comprehensive statewide early childhood
systems for pregnant women, parents, and caregivers, and children from
birth to 8 years of age.
The ACA Maternal, Infant, and Early Childhood Home Visiting Program
is designed: (1) To strengthen and improve the programs and activities
carried out under Title V; (2) to improve coordination of services for
at risk communities; and (3) to identify and provide comprehensive
services to improve outcomes for families who reside in at risk
communities. At risk communities will be identified through a statewide
assessment of needs and of existing resources to meet those needs. HHS
intends that the home visiting program will result in a coordinated
system of early childhood home visiting in every State that has the
capacity to provide infrastructure and supports to assure high-quality,
evidence-based practice.
The program enables eligible entities to utilize what is known
about effective home visiting services to provide evidence-based
programs to promote: improvements in prenatal, maternal and newborn
health; child health and development including prevention of child
injuries and maltreatment and improvements in cognitive, language,
social-emotional, and physical development; parenting skills; school
readiness; reductions in crime or domestic violence; improvements in
family economic self-sufficiency; and improvements in the coordination
and referrals for other community resources and supports.
2.2 Use of Funds for ``Evidence-Based'' Programs
Section 511(d)(3)(A) of Title V, as amended by the Affordable Care
Act, reserves the majority of grant funds for home visiting program
models with evidence of effectiveness based on rigorous evaluation
research. The legislation specifies that models must meet the following
requirements in order to be considered ``evidence-based'':
(I) The model conforms to a clear consistent home visitation
model that has been existence for at least 3 years and is research-
based, grounded in relevant empirically-based knowledge, linked to
program determined outcomes, associated with a national organization
or institution of higher education that has comprehensive home
visitation program standards that ensure high quality services
delivery and continuous program improvement, and has demonstrated
significant, (and in the case of the service delivery model
described in item (aa), sustained) positive outcomes, as described
in the benchmark areas specified in paragraph (1)(A) and the
participant outcomes described in paragraph (2)(B), when evaluated
using a well-designed and rigorous--
(aa) randomized controlled research designs, and the evaluation
results have been published in a peer-reviewed journal; or
(bb) quasi-experimental research designs.
The legislation charges the Secretary of Health and Human Services
with establishing criteria for evidence of effectiveness of the home
visiting program models and ensuring the process for establishing the
criteria is transparent and provides the opportunity for public
comment.
This Notice (1) proposes criteria to be considered in assessing
whether home visiting models have evidence of effectiveness and (2)
describes the methodology for a systematic review of evidence, applying
the criteria proposed in this Notice, which HHS is currently
conducting. The Notice solicits comments on both items. After comments
are received, HHS will finalize the criteria and methodology and
complete the systematic review of the available evidence of
effectiveness of selected home visiting program models.
It is expected that eligible entities will also have an opportunity
to present documentation in their applications for the ACA Maternal,
Infant, and Early Childhood Home Visiting program to demonstrate that
additional home visiting models meet the final criteria. Such
documentation will be reviewed by HHS using the same procedures applied
in HHS' systematic review and described below.
The criteria proposed in this notice apply only to the home
visiting program for States and territories authorized by Section
511(c) of Title V. Criteria for the ACA Tribal Maternal, Infant, and
Early Childhood Home Visiting Program authorized by Section
511(h)(2)(A) of Title V will be issued separately. Based on a careful
review of available research evidence on home visiting interventions
with Tribal populations, the Secretary will develop alternative
evidence-based criteria for identifying home visiting models likely to
improve outcomes for families in Tribal communities.
[[Page 43174]]
3.0 Proposed Criteria for Evidence of Effectiveness
A home visiting model must have been (1) evaluated using rigorous
methodology and (2) shown to have a positive impact on outcomes in
order to meet criteria for evidence of effectiveness. The following two
types of criteria (3.1 and 3.2) must be met in order for a home
visiting model to be considered evidence-based for the purposes of the
Maternal, Infant, and Early Childhood Home Visiting Program:
3.1 Criteria for Well-Designed, Rigorous Impact Research
In order to ensure the highest probability of producing unbiased
estimates of program impacts, there are a number of variables that
should be considered. These variables include study design (i.e.
randomized controlled trial [RCT] or quasi-experimental design [QED]),
level of attrition, baseline equivalence, reassignment of participants
from one condition to another in the trial, the reliability and
validity of outcome measures studied, and confounding factors.
Two types of impact study designs have the potential to be both
well-designed and rigorous: Randomized controlled trials and quasi-
experimental designs. HHS proposes to define randomized controlled
trials as a study design in which sample members are assigned to the
program and comparison groups by chance. Randomized control designs are
often considered the ``gold standard'' of research design because
personal characteristics (before the program begins) do not affect
whether someone is assigned to the program or control group. HHS
proposes to define a quasi-experimental design as a study design in
which sample members are selected for the program and comparison groups
in a nonrandom way. For example, families may self-select into groups
(deciding whether they want services or not) or an administrator may
assign families to groups based on family risk factors. Quasi-
experimental designs are considered weaker than randomized controlled
trials because characteristics that may be related to outcomes, such as
motivation or need, may also influence whether someone is in the
program or comparison group.
HHS proposes that an impact study will be considered high, moderate
or low quality depending on the study's capacity to provide unbiased
estimates of program impact. Studies that are rated ``high'' and
``moderate'' (see Table 1 below), therefore, would meet requirements to
be considered ``well-designed, rigorous impact research.'' In brief,
the high rating would be reserved for random assignment studies with
low attrition of sample members and no reassignment of sample members
after the original random assignment. The moderate rating would apply
to studies that use a quasi-experimental design and to random
assignment studies that, due to flaws in the study design or execution
(for example, high sample attrition), do not meet all the criteria for
the high rating. To receive the moderate rating, studies would have to
demonstrate that at the study's onset, the intervention and comparison
groups were well matched on specified measures (i.e. baseline
equivalence), such as a pretest measure of targeted outcomes or race
and maternal education. Studies that do not meet all of the criteria
for either high or moderate quality would be considered low quality.
As summarized in Table 1, the rating scheme would consider five
dimensions: (1) Study design, (2) attrition, (3) baseline equivalence,
(4) reassignment, and (5) confounding factors.
Table 1--Criteria for Well-Designed, Rigorous Impact Research
----------------------------------------------------------------------------------------------------------------
Rating
Rating Criteria --------------------------------------------------------------------------
High Moderate Low
----------------------------------------------------------------------------------------------------------------
Study design......................... Random assignment...... Quasi-experimental Studies that do not
design with a meet the requirements
comparison group; for a high or moderate
random assignment rating.
design with high
attrition or any
reassignment.
Attrition............................ Meets ``What Works No requirement.........
Clearinghouse'' (Dept.
of Education)
standards for
acceptable rates of
overall and
differential attrition.
Baseline equivalence................. No requirement other Must establish baseline
than random equivalence of study
assignment; arms on selected
Statistically measures (see Table 1,
significant Note 2 below).
differences must be
controlled.
Reassignment (see Table 1, Note 1 Analysis must be based No requirement.........
below). on original assignment
to study arms.
Confounding factors.................. Must have at least two Must have at least two
participants in each participants in each
study arm and no study arm and no
systematic differences systematic differences
in data collection in data collection
methods. methods.
----------------------------------------------------------------------------------------------------------------
Table 1, Note 1: In random assignment studies, deviation from the
original random assignment (for example, moving families from the
treatment to the control group) can also bias the impact estimates.
Therefore, in order for a RCT to meet our criteria for the high rating,
the analysis must be performed on the sample as originally assigned.
Subjects may not be reassigned for reasons such as contamination,
noncompliance, or level of exposure. RCTs that somehow alter the
original random assignment but otherwise meet the criteria for the high
rating are considered for a moderate study rating, provided they meet
the other criteria for that rating. Our criteria are similar to those
developed by the WWC, which allows a study to be downgraded as a result
of reassignment.
Table 1, Note 2: When possible, baseline equivalence should be
established on outcomes of interest. For some studies it is not
feasible to collect baseline measures on the outcome of interest, for
example, children's
[[Page 43175]]
outcomes when baseline is collected prenatally. For all studies,
baseline equivalence must be established on two demographic factors:
(1) The parent or child's race and ethnicity and (2) socioeconomic
status.
3.2 HHS Proposed Criteria for Evidence of Effectiveness of a Home
Visiting Service Delivery Model
In order to have confidence in the impact estimates created from a
high or moderate quality study design, a number of variables should be
considered. These variables include statistical significance, whether
impacts are sustained, and whether the impacts were found for the full
sample or only for non-replicated subgroups.
3.2.1 The ACA Maternal, Infant and Early Childhood Home Visitation
Program legislation includes a number of participant outcome and
benchmark areas. In determining program effectiveness HHS proposes to
examine programs for impacts in the following eight program domains:
(1) Maternal health
(2) Child health
(3) Child development and school readiness, including improvements
in cognitive, language, social-emotional or physical development
(4) Prevention of child injuries and maltreatment
(5) Parenting skills
(6) Reductions in crime or domestic violence
(7) Improvements in family economic self-sufficiency
(8) Improvements in the coordination and referrals for other
community resources and supports.
3.2.2 Taking into account the legislative language and the two
types of criteria discussed in 3.1 and 3.2 above, HHS proposes to
consider a program model eligible for evidence-based funding for the
purposes of the ACA Maternal, Infant, and Early Childhood Home Visiting
Program if it meets the following minimum criteria:
At least one high- or moderate-quality impact study (see
3.1) of the program model finds favorable, statistically significant
impacts in two or more of the eight outcome domains (see 3.2.1); or
At least two high- or moderate-quality impact studies
using different samples (see 3.1) of the program model find one or more
favorable, statistically significant impacts in the same domain (see
3.2.1).
In both cases, the impacts considered must be found either for the
full sample or, if found for subgroups but not for the full sample,
impacts must be replicated in the same domain in two or more studies
using different samples.
Additionally, if the program model meets the above criteria based
on findings from randomized control trial(s) only, then one or more
impacts in an outcome domain must be sustained for at least one year
after program enrollment, and one or more impacts in an outcome domain
must be reported in a peer-reviewed journal (consistent with section
511(d)(3)(A)(i)(I)).
Isolated positive findings, and impacts found only for a subgroup,
but not the full sample in a study, raise concerns about false
positives that may be artifacts of multiple statistical tests rather
than reflecting true impacts. The requirements for replication of
positive findings across samples or for findings in two or more outcome
domains are meant to guard against this problem. HHS recognizes the
importance of subgroup findings for determining impacts on subgroups of
the population of interest, including specific racial or ethnic groups,
and plans to report information on subgroup findings, whether
replicated or not.
4.0 Proposed Methods for HHS's Systematic Review of Evidence of
Effectiveness
HHS is conducting a comprehensive and detailed program model-by-
model review of the available evidence of effectiveness of home
visiting programs that support the following legislatively specified
benchmarks and outcomes: Maternal health; child health; child
development and school readiness including improvements in cognitive,
language, social-emotional, and physical development; prevention of
child injuries and maltreatment; parenting skills; reductions in crime
or domestic violence; improvements in family economic self-sufficiency;
and improvements in the coordination and referrals for other community
resources and supports.
The review is being carried out through a contract to Mathematica
Policy Research, Inc. and led by the Administration for Children and
Families in collaboration with the Health Resources and Services
Administration, the Office of the Assistant Secretary for Planning and
Evaluation, and the Centers for Disease Control and Prevention. The
review will apply the HHS criteria proposed above to determine which of
the program models reviewed meet the criteria for evidence of
effectiveness. The review will be completed after comments on this
notice are received and considered.
4.1 Review Process
To conduct a through and transparent review of the home visiting
program model research literature, the systematic review project is
following five main steps, the first three of which have been
provisionally completed. Comments on steps 4 and 5 are especially
encouraged.
1. Conduct a broad literature search;
2. Screen studies for relevance;
3. Prioritize program models for review;
4. Rate the quality of impact studies with eligible designs;
5. Assess the evidence of effectiveness for each program model.
In addition, the project plans to review and make available
implementation information for each program. Steps taken to address
potential conflicts of interest are also described below.
4.1.1 Step 1: Conduct a Broad Literature Search
The literature search included four main activities:
1. Database Searches. The project team searched on relevant key
words in a range of research databases. Key words included terms
related to the service delivery approach, target population, and
outcome domains emphasized in the Patient Protection and Affordable
Care Act. The initial search was limited to studies published since
1989; a more focused search on prioritized program models included
studies published since 1979 (see Prioritizing Programs below).
2. Web Site Searches. The project team used a custom Google search
engine to search more than 50 relevant government, university,
research, and nonprofit Web sites for unpublished reports and papers.
3. Call for Studies. In November 2009, Mathematica issued a call
for studies and sent it to approximately 40 relevant listservs for
dissemination.
4. Review of Existing Literature Reviews and Meta-Analyses. The
project team checked search results against the bibliographies of
recent literature reviews and meta-analyses of home visiting programs
and added relevant missing citations to the search results.
The literature search yielded approximately 8,200 unduplicated
citations, including 150 articles submitted through the call for
studies.
4.1.2 Step 2: Screen Studies for Relevance
The project team then screened all citations identified through the
literature search for relevance. Studies were screened out for the
following reasons:
[[Page 43176]]
The model under study did not use home visitation as a
primary service delivery strategy. Programs that are primarily center-
based with infrequent or supplemental home visiting were excluded. In
order to be considered a home visiting model, a program must offer home
visiting services to most or all participants and these services must
be integral to programmatic goals. Visits should occur solely or
primarily where participating families reside but occasionally may
occur elsewhere if the families are homeless or uncomfortable
conducting visits in the home. The services could be voluntary or
mandated (for example, court ordered).
The study did not use an eligible design as described in
3.1 above (randomized controlled trial, quasi-experimental design). The
project team also included any studies on the implementation of
specific home visiting models. These studies were used in the
implementation reports described in section 5.0 of this Notice.
The program did not include pregnant women and families
with children from birth to kindergarten entry.
The study did not examine any outcomes in the domains of:
Maternal health and/or child health; child development and school
readiness; reductions in child maltreatment; reductions in juvenile
delinquency, family violence, and crime; positive parenting practices;
and family economic factors. The legislatively specified domain of
improvement in coordination and referrals for community resources and
supports was not used in screening because of challenges in specifying
discrete measures.
The study did not examine a clear home visiting program
model. For example, the study might focus on a specific home visiting
strategy, such as comparing the use of professional and
paraprofessional home visiting staff within home visiting program model
broadly rather than a specific program model. Without a clearly
identified program model, the evidence review could not use the impact
study to assess the effectiveness of a specific program.
The study was not published in English. This limitation
reflects practical considerations, given the limited time available for
the review.
The study was published before 1989 for the initial search
or 1979 for the focused search on prioritized program models. These
limitations balance practical considerations, given limited time
available, and were designed to ensure that seminal research was
included.
4.1.3 Step 3: Prioritize Program Models for Review
After screening, the initial search yielded studies on more than
250 home visiting program models. Timing and resources do not allow for
a detailed review of all of these home visiting program models prior to
the implementation of the ACA Maternal, Infant, and Early Childhood
Home Visiting Program. For each model the team examined the number and
design of impact studies, sample sizes of the impact studies, the
availability of implementation information, whether the program was
currently in widespread use in the U.S., and whether the program had
been implemented only in a developing-world context. The project staff
eliminated programs that had no information available about
implementation, were implemented only in a developing-world context, or
were no longer in operation and provided no support for implementation.
This decision was made so that resources could be focused on reviewing
program models that States and territories would be readily able to
implement and that would be likely to meet other statutory
requirements.
4.1.4 Step 4: Rating the Quality of Impact Studies
For the purposes of the systematic review, HHS plans to assign each
impact study a rating of high, moderate, or low, per the criteria
described in 3.1 above.
4.1.5 Step 5: Assessing Evidence of Effectiveness
After rating the quality of all available impact studies for a
program, HHS plans to assess the evidence across all studies of the
program models that received a high or moderate rating and measured
outcomes in at least one of the legislatively specified participant
outcome domains utilizing the HHS proposed criteria for evidence for
effectiveness discussed in 3.2 above.
5.0 Implementation Reviews
To assist in implementation of the ACA Maternal, Infant and Early
Childhood Home Visiting Program, the project plans to collect and
publish information about implementation of the prioritized program
models. The project plans to provide two kinds of implementation
reports for each program model. One implementation report will focus on
the support available to assist interested entities to implement the
model (such as program model technical assistance staff or trainings)
or infrastructure required to implement the model (such as the purchase
of a specific data management system or curricula). The second kind of
implementation report will focus on implementation experiences during
the impact trials or in implementing the model in the field. These
reports will provide information on the study samples in the impact
trials, describe the locations where the specific model has been
implemented, the average number of visits the participants receive, any
available research on adaptations of the program models and lessons
learned about implementing the models that have been reported in the
available research.
6.0 Addressing Conflicts of Interest
All members of the review team have signed a conflict of interest
statement in which they declared any financial or personal connections
to developers, studies, or products being reviewed and confirmed their
understanding of the process by which they must inform the project
director if such conflicts arise. The review team's project director
assembled signed conflict of interest forms for all project staff and
subcontractors and monitors for possible conflicts over time. If a team
member is found to have a potential conflict of interest concerning a
particular home visiting program model being reviewed, that team member
is excluded from the review process for the studies of that program
model. In addition, reviews for two programs evaluated by Mathematica
Policy Research are being conducted by contracted reviewers who are not
Mathematica[reg] employees.
7.0 Future Allocations Based on Application Strength
To encourage exemplary programs and direct Federal funds where they
can have the greatest impact, HHS plans to allocate the ACA Maternal,
Infant and Early Childhood Home Visiting Program funding available in
future years that exceeds funding available in FY 2010 competitively
based upon States' capacity and commitment to improve child outcomes
specified in the statute through improvements in service coordination
and the implementation of home visiting programs with fidelity to high-
quality, evidence-based models. HHS plans to evaluate applications
based on multiple criteria and invites comments on what criteria are
appropriate. Among the criteria, HHS proposes to give significant
weight to the strength of the available evidence of effectiveness of
the model or models employed by the State. In this context, the use of
program models satisfying the
[[Page 43177]]
criteria outlined in section 3.2.2 would be a minimal requirement, but
HHS would consider additional criteria that further distinguish models
with greater and lesser support in evidence. HHS is committed to
ensuring that these criteria are transparent, methodologically sound,
and increase the likelihood that federal funds will contribute to
improved outcomes for at-risk children and families.
There are a number of different ways that such a system could be
structured. We invite comments on the proposal to distinguish among
evidence-based models based on a rubric that weighs factors relating to
research quality and findings. For example, one relatively simple
approach would rate models using an index constructed by weighting
several factors equally. Models might be given points for meeting each
of the following criteria: Favorable impacts sustained at least one
year after program completion, favorable impacts replicated in distinct
samples, favorable impacts in studies conducted by independent
evaluators, quality and relevance of outcome measures; and balance
between favorable and unfavorable and null findings. Additional factors
which might be considered could include further indicia of the quality
of the research design and implementation (as reflected in
randomization, sample size, attrition, and baseline equivalence). We
invite comments on HHS' proposal to use evidence for program models as
a factor in determining allocations of additional funds, how various
factors should be weighed in assessing the evidence of effectiveness,
how to define these categories, and any other role distinctions related
to the strength of the evidence should play in funding allocation. As
noted above, strength of evidence is proposed to be only one factor in
the evaluation of the strength of States' applications, and we invite
comments on other appropriate factors as well.
8.0 Future Considerations
We invite comment on the following:
HHS anticipates the criteria for evidence-based models
will likely need to be altered over time as the state of the field
changes. If HHS believes the criteria need to be changed in future
years, it is anticipated the public will have an opportunity to comment
on the proposed revisions.
HHS intends to review the evidence base for home visiting
models on an ongoing basis to ensure that new evidence is incorporated.
9.0 Submission of Comments
Comments may be submitted until August 17, 2010 by e-mail to:
mpr.com">HVEE@mathematica-mpr.com.
Dated: July 19, 2010.
Mary K. Wakefield,
Administrator, Health Resources and Services Administration.
Carmen R. Nazario,
Assistant Secretary, Administration for Children and Families.
[FR Doc. 2010-18013 Filed 7-22-10; 8:45 am]
BILLING CODE 4165-15-P