Att_OMB Supporting Statement CHANGE

What Works Clearinghouse
Database Forms Revisions
Supporting Statement
CONTROL NO. 1850-0788

Submitted to:
Office of Management and Budget

Submitted by:
Institute for Education Sciences
U.S. Department of Education

June 2009

INTRODUCTION
This submission is a request to continue a currently approved collection under OMB Control
Number 1850-0788 for the What Works Clearinghouse (WWC) [ED-07-CO-0062]. The U.S.
Department of Education (ED) established the WWC to develop, maintain, and make accessible a
system of high-quality reviews of studies of the effectiveness of education-related interventions.
In support of this effort, the WWC currently collects the following types of information:
Nominations for review
o

Interventions

o

Studies

o

Topics

Registry of evaluation researchers
During the course of its operation, the WWC determined that it could decrease the amount of
information collected on the nomination forms—and thus decrease respondent burden—without
compromising the review process. This supporting statement describes and justifies the proposed
form changes and includes as attachments the proposed versions of the nomination forms.
Similarly, the WWC has identified collections that no longer serve the needs of the WWC. This
supporting statement does not describe these forms, as the following information collected is no
longer necessary.

WWC user type query
WWC update subscription
This submission is also a request to supplement the WWC’s work by collecting the following
information:
Registry of randomized controlled trials
This supporting statement describes the justification for the WWC data collections, and the
sections of the forms are detailed in the attachments. This document, which addresses Office of
Management and Budget (OMB) concerns regarding respondent burden and paperwork control,
has been prepared according to guidelines for completing the justification statement to
accompany OMB Form 83-I.

1

JUSTIFICATION
1. Necessity of Information Collection
There is an increasing recognition among policymakers that decisions about educational
programs, products, and practices should be based on evidence of effectiveness from high-quality
scientific research. The No Child Left Behind Act (PL 107-110), for example, includes numerous
requirements to use the findings from ―scientifically based research‖ in designing and
implementing educational interventions. The implementation of this Act—with its promise to
serve the educational needs of all children and its emphasis on ―doing what works‖—creates new
opportunities and a strong impetus to identify, select, and implement effective school
improvement strategies.
However, meeting the challenge and achieving the promise of No Child Left Behind will prove
difficult unless education decision makers, including parents, are provided with the timely
information and support they need to make wise choices from among the ever-increasing options.
Those who have a stake in improving education urgently want credible and reliable information
that enables them to judge the effectiveness and quality of alternative approaches to raising
student achievement. Educators, policymakers, and the public cannot be expected to do ―what
works‖ until they actually know ―what works.‖ By transforming education into an evidence-based
field, the Department is working to fulfill its mission of improving student outcomes and
providing the information that decision makers need.
The WWC is a key part of this effort. Established by the Department’s Institute of Education
Sciences (IES) in 2002, the purpose of the WWC is to offer everyone—from public officials to
the public at large—a central, independent, and trusted source of scientific evidence of what
works in education. In order to be successful, the WWC involves key constituents in the process
of its activities, in the development of its products, and in the identification of areas for
improvement.
In support of this effort, the web-based submission forms currently posted on the What Works
Clearinghouse website (whatworks.ed.gov) (1) allow the public to nominate evidence-based
research studies, interventions, and topics for review by the WWC and (2) collect information
about researchers who want to supply the public with information about their evaluation services.
Accordingly, the WWC will continue to administer the following:
Nomination Forms (for studies, interventions, and topics)
Registry of Evaluation Researchers Form for registration
Registry of Evaluation Researchers Letter of Commitment
Clearance is requested to continue these information collection efforts that support the
development, operation, and evolution of the WWC and provide the public with information they
request. However, the WWC requests clearance to use a redesigned format for the Nomination
Forms. These revised forms allow individuals to nominate a replicable intervention, a research
study, or a topic for WWC review.
Additionally, the WWC seeks clearance to collect information from researchers and entities that
want to supply the public with information about their randomized controlled trials (RCTs),

2

which are either (1) funded and in progress or (2) completed, with a final report available. The
information from the online form will ensure greater coverage of research being conducted and
participation of researchers in the field. Accordingly, the WWC requests OMB clearance to
administer the following form:
Registry of Randomized Controlled Trials Form. Individuals and organizations that
would like to provide information about studies either funded and in progress or
completed, with a final report available, may submit study information via a registration
form. This information will be included in an online searchable database for users to find
RCTs concerning particular interventions.
Registry of Randomized Controlled Trials Letter of Commitment. To be listed in the
Registry of Evaluation Researchers, registrants must sign a letter acknowledging their
commitment to both WWC Evidence Standards and related professional standards and
ethical practices.
Clearance is requested to add this information collection to support the development, operation,
and evolution of the WWC. The WWC will continue to adhere to the language of the existing
privacy statement, which indicates that any individual identification will not be posted on the
public website without the user’s permission.

2. Purposes and Use of the WWC Data Collection Forms
Below we explain the purpose of each of the existing voluntary information collection forms and
the changes for which we are requesting review.
Existing forms collect detailed information that the WWC has determined are not needed to
decide whether a study, intervention, or topic should be reviewed. The WWC would like to alter
these forms and collect only the information necessary to consider interventions, studies, and
topics for review. Even though the existing forms collect detailed information from users, the new
forms collect comparable information using a smaller number of items.
Study Submission Form. The purpose of the Study Submission Form is to gather information
from members of the public who want to inform the WWC of studies that are relevant to WWC
Intervention Reports. The online version of this form enables users to enter and submit
information electronically. The WWC uses these submissions to supplement the studies located
by staff, and the studies will be considered for inclusion in WWC Intervention Reports.
The WWC would like to alter the existing Study Submission Form and collect only the
information necessary for the WWC to consider a study for review. The elements of the new
form (attachment Form 01) are
Name of study
User email address
Optional electronic document upload

3

The new Study Submission Form drops a number of items from collection because their accuracy
needs to be independently researched and verified by WWC staff before moving forward with a
review of the study. Dropping these items also decreases the burden on respondents. Dropped
items include:
Study details
Study profile (existing long form only)
Optional contact information
Optional electronic information release form
Optional user type categories
Intervention Nomination Form. The purpose of the Intervention Nomination Form is to collect
information from education producers and consumers about specific interventions within topic
areas that they would like reviewed. Nominations will be used to inform the WWC of
interventions that users would like considered for inclusion in WWC Intervention Reports.
Currently, the long Intervention Nomination Form includes three main elements that provide the
WWC with information to assess interventions’ relevance to WWC topic areas, as well as two
optional elements. The short Intervention Nomination Form is similar to the long form, but it
omits the In-Depth Description and Background of Intervention.
The WWC would like to replace both of these forms with a single form that collects only the
information that is most necessary for the WWC to consider an intervention to review and to
respond to the submitter. The elements of the new form (attachment Form 02) are:
Name of intervention
User email address
The new Intervention Nomination Form drops a number of items from collection. These items
were dropped because their accuracy needed to be independently researched and verified by
WWC staff before moving forward with a review of the intervention. Dropping these items also
decreases the burden on respondents. Dropped items include:
Intervention relevance to current topic areas
General description of the intervention
In-depth description and background of the intervention
Optional contact information
Optional user type categories
Topic Nomination Form. The purpose of the Topic Nomination Form is to gather in-depth
information from education producers and consumers about the topic areas and practice guide
topics they would like to suggest for review by the WWC. Information gathered via the Topic

4

Nomination Form will be used to help prioritize topics and practice guides proposed and
recommended to IES in the WWC Annual Plan.
Currently, the Topic Nomination Form includes three main elements that are designed to provide
the WWC with information to assess the significance of the topic in improving student
educational outcomes, as well as two optional elements.
The WWC would like to alter this form and collect only the information necessary for the WWC
to consider a topic for review. The elements of the new form (attachment Form 03) are:
Description of topic area
User email address
WWC Submission Process. The WWC Submission Flowchart shown in Exhibit 1 demonstrates
the path that website users will be prompted to take during the online submission process.
Responses to the Intervention, Study, and Topic Nomination forms will be captured in a database.
Case ID numbers will be used in association with respondents’ information. Individual
identifying information is accessible only to project staff and will be used only if there are
questions about a respondent’s submission.
Information will be stored in a secure database on the WWC private website. At the end of the
contract, the WWC will provide the U.S. Department of Education with the database that was
created during the project.

5

EXHIBIT 1
WWC Submission Process
Start
(WWC Nomination Form)

Select ―Suggest a Topic to
Review,‖ ―Suggest an
Intervention to Review,‖ or
―Submit a Study to Review‖

Suggest an
Intervention to
Review

Suggest a Topic to
Review

User Email
Address

Message Content
(length variable)

Done

Submit a Study to
Review

User Email
Address

Message Content
(length variable)

Optional
Electronic Upload

Done

6

Registry of Evaluation Researchers Form and Registry of Evaluation Researchers Letter of
Commitment. The online Registry of Evaluation Researchers is designed to provide potential
sponsors of effectiveness studies with information about individual and organization evaluators
who subscribe to the standards of evidence established by the WWC. The goal of this registry is
to contribute to advancing the quality of educational research by (1) assembling a substantial list
of committed outcome evaluation providers and (2) supplying tools and guides that help
consumers decide what they need, how to assess the qualifications of the evaluators, how to use
the information supplied to find potential evaluation providers, and how to supplement the help
provided by the registry with other sources of assistance. To be included, evaluators must
complete a registration form and sign a commitment letter on the password-protected site
(attachment Forms 04 [text documentation] and 05 [letter]). The WWC does not request changes
to either of these forms.
Below we explain the purpose of the proposed voluntary information collection form for which
the WWC seeks clearance.
Registry of Randomized Controlled Trials. The proposed Registry of Randomized Controlled
Trials is designed to provide users with information about RCTs of education interventions,
including those completed, with a final report available; those funded and in progress; and those
not yet reviewed by the WWC under one of its topic review areas. The goal of this registry is to
contribute to advancing the quality of educational research by (1) assembling a substantial list of
RCTs in education that are funded and in progress or completed, with a final report available, on
particular topics or interventions; and (2) supplying tools and guides that help consumers
understand how to use the information supplied in the Registry of Randomized Controlled Trials.
To be included, the researchers or entities conducting a RCT must complete the registration form
(attachment Form 06) on the password-protected site.
The WWC and the Department do not intend to assess the quality and accuracy of the information
supplied by researchers and entities wishing to list studies in the Registry of Randomized
Controlled Trials, only its completeness. This will be stated clearly in the searchable online
registry. Participation is voluntary. Because the registry will rely on self-reported data, sufficient
information is needed so that members of the user community can determine for themselves
whether the RCTs – either funded and in progress or completed, with an available final report –
meet their needs.
Information planned for this registry is not readily available to the field. We know of no other
RCT registries in the field of education that seek to provide information on RCTs either funded
and in progress or completed, with an available final report. The valuable role of RCT registries
has been well documented by health care researchers, and the Cochrane Collaboration, for
example, developed its trials registry, CENTRAL, to improve the quality of access to information
on RCTs and to support the production of systematic reviews of RCTs on the effectiveness of
interventions. By adapting this conceptual framework, the WWC Registry of Randomized
Controlled Trials will provide access to RCTs that are either funded and in progress or completed,
with an available final report. Additionally, the WWC Registry of Randomized Controlled Trials
will enable the WWC to develop a top-down approach that will track RCTs and look for reports
from these trials, a service currently not available. This new service will complement the bottomup approach of the current WWC, which begins with a complete search of the literature on a
given topic, winnows the literature down to identify studies of effectiveness, and sorts studies by
intervention.

7

The proposed Registry of Randomized Controlled Trials is designed for use by researchers and
organizations. It consists of elements that will provide end users with crucial information needed
to understand the purpose of the study. These elements (attachment Forms 06 [text
documentation] and 07 [details]) and are summarized below:
Contact information
Organization information
Study information
Study details
Study abstract
Registry of Randomized Controlled Trials Letter of Commitment. For a study to be listed in the
Registry of Randomized Controlled Trials, registrants must sign a letter acknowledging their
commitment to using the highest standards of scientific evidence, as defined by the WWC, and
make clear the degree to which the study will overcome or limit threats to construct, internal,
external, and statistical validity. This feature of the registry is intended to strengthen and improve
the quality and rigor of such evaluations and research studies. The commitment letter (attachment
Form 08) contains the following elements:
Commitment to the highest standards of scientific evidence, as defined by the WWC.
Make clear the degree to which the study will overcome or limit the threats to
construct, internal, external, and statistical validity and the resulting implications of
making appropriate causal claims.
In the commitment letter, evaluators will also provide their signed permission for the WWC to
release their information through the searchable registry. The form does not contain any questions
of a sensitive or controversial nature.
The registry will remain current in several ways. There will be an initial collection effort to build
the registry by identifying RCTs in design or sponsored by the following IES centers: National
Center for Education Evaluation and Regional Assistance (NCEE); National Center for Education
Research (NCER); and National Center for Special Education Research (NCSER).
Additionally, the WWC will develop a list of stakeholders, including developers, evaluators, and
policy research organizations, that are likely producers or sponsors of RCTs. The list will be used
to guide WWC staff who will periodically survey these stakeholders, as is consistent with the
OMB-approved WWC survey form, to identify RCTs that are in design or underway. To
encourage updates, the WWC will provide researchers and stakeholder entities in the registry
with automatic annual prompts about updates. In addition, incomplete entries will be returned for
additional information.
The registration form and the updates will be operated online directly through the website, with
the exception of those stakeholders, if any, who do not have Internet capacity or who choose to
submit a paper copy for other reasons. Information will be stored in a secure database on the
WWC’s private website. At the completion of the contract, the contractor will deliver all files,
including the Registry of Randomized Controlled Trials, to the Department.

8

3. Use of Technology in Information Collection
All WWC databases, customer surveys, and registry forms are web-based. Website users may
choose to print out the forms from the website and submit a paper copy, but we expect most users
to submit the forms electronically via the automated website interfaces. These interfaces have
been designed to impose as little collection burden as possible, while still gathering the
information needed to inform the WWC. For example, the existing nomination forms request
information from users on forms 5 to 12 pages in length. The proposed revised nomination forms
are only 1 to 3 pages, and each page requires no more than three items from the user.
To reduce the burden on those using the Registry of Evaluation Researchers and the Registry of
Randomized Controlled Trials, returning users may update their forms without having to resubmit
their information. Assistance is also provided in using the web-based forms. For example, the
registry forms link users to definitions of key terms and documents, where appropriate. In
addition, users have easy access to personal and individual guidance if needed, and information
on how to contact the WWC is visibly presented on the website.

4. Efforts to Identify Duplication
The information to be collected from the nomination forms, the Registry of Evaluation
Researchers database, the customer survey, and the Registry of Randomized Controlled Trials is
not available from any other source and will not duplicate any existing information collection
effort.

5. Burden on Small Business Organizations
Not applicable to this information collection.

6. Consequences of Less Frequent Information Collection
The nomination forms are voluntary and sought out by WWC website users; therefore, it is
assumed that these forms will be completed only when the user wishes to submit answers for a
survey or to nominate an intervention, study, or topic for review.
Those submitting information through the Registry of Evaluation Researchers and the Registry of
Randomized Controlled Trials also participate by self-selection, but they are prompted to update
their information on an annual basis. This is necessary to keep the contents of the database up to
date and useful to the public.
Information about the availability of the nomination forms will be disseminated periodically at
presentations to encourage users and potential users to visit the website and submit their requests
and feedback. If we did not disseminate information about these opportunities periodically,
existing users might forget that this feature of the website is available to them, and potential users
might not learn about how to submit their nominations and information needs.

9

7. Special Circumstances
Users may submit copyrighted publications. These publications will not be shared with the public
until the appropriate permissions are obtained. Users may also submit unpublished studies and
request that they be kept confidential. These studies also will not be shared with the public until
permission is obtained.

8. Outside Consultants
Since the contract award date, WWC staff members have given presentations to a wide range of
administrators, researchers, policymakers, practitioners, and others to inform them about the
WWC and to allow them to inform us of their information needs. Information provided through
these interactions and presentations has guided the redesign of the nomination forms.
No outside consultants have contributed to the design of the registries. No outside consultants
will contribute to the maintenance of the registries.

9. Explanation of Providing Payment or Gifts to Respondents
No payment or gift of any kind will be provided to respondents.

10. Assurance of Confidentiality
For the nomination forms, the website has a privacy policy statement indicating that no individual
identification will be posted on the public website without the user’s permission. Information
about users will be reported at the aggregate level only. Contact information on nomination
forms is used for the express purpose of clarifying the WWC user’s nomination and will not be
released beyond the WWC staff with an immediate need to clarify the submitted information.
The purpose of the Registry of Evaluation Researchers database is to allow evaluators to post
their credentials and contact information to the public; therefore, it is assumed when evaluators
submit their forms that they are not requesting confidentiality. The WWC does, however, offer
evaluators the option of limited disclosure for studies. For example, respondents can request that
proprietary information be kept confidential. This will typically only be requested when an
evaluator is in the process of getting a study published and would like the WWC to post the study
information only after it is publicly available. Additionally, the WWC will post only the provided
contact and experience information; biographies, resumes, and curricula vitae will not be
available through the WWC.
The purpose of the Registry of Randomized Controlled Trials is to enable researchers and other
stakeholders to post information to the public regarding RCTs that are funded and in progress or
completed, with an available final report; therefore, it is assumed when stakeholders submit their
forms that they are not requesting confidentiality. The WWC will post only the provided

10

bibliographic information and study abstract; the full text of the study will not be available
through the WWC.

11. Sensitive Questions
None of the requested information is sensitive in nature.

12. Estimates of Hour Burden
The total number of respondents for all WWC forms is estimated to be 1,600 across three years,
with the total annualized responses at 534. The total number of burden hours for all collections is
estimated to be 542.5, with the total annualized hours at 181. Hourly burden estimates by form
and totals by year are displayed in Exhibit 2.
In January 2007, the initial collection was submitted for approval. The total number of
respondents was estimated to be 21,820 across the three years, with the total annualized responses
at 7,273. The total number of burden hours for all collections is estimated to be 2,581.66, with the
total annualized hours at 861.
Experience operating the WWC has led to changes in necessary information requested from
users, which likewise resulted in anticipated changes to hourly burden estimates. The proposed
estimates for the revised collection reflect the following decreases:

Total respondents: less 20,220
Total annualized responses: less 6,739
Total burden hours: less 2,039.16
Total annualized burden hours: less 680
The estimated annualized cost to respondents, shown in Exhibit 3, is based on the average
hourly wage ($19.29) as of June 2007, as reported by the Bureau of Labor Statistics National
Bulletin.1

1

Available at http://www.bls.gov/ncs/ocs/sp/ncbl0910.pdf. Accessed June 16, 2009.

11

EXHIBIT 2
Estimates of Hourly Burden
Year

Time to
Complete

Number of
Respondents

Hourly
Burden

Intervention, Study, and Topic
Nomination Forms

Y1

3 minutes

300

15

Registry of Evaluation Researchers

Y1

120 minutes

60

120

Registry of Randomized
Controlled Trials

Y1

30 minutes

50

25

Total, Year 1

410

160

Form

Intervention, Study, and Topic
Nomination Forms

Y2

3 minutes

300

15

Registry of Evaluation Researchers

Y2

120 minutes

30

60

Registry of Evaluation Researchers
Update*

Y2

30 minutes

100

50

Registry of Randomized
Controlled Trials

Y2

30 minutes

75

37.5

Registry of Randomized
Controlled Trials Update*

Y2

15 minutes

40

10

Total, Year 2

545

172.5

Intervention, Study, and Topic
Nomination Forms

Y3

3 minutes

300

15

Registry of Evaluation Researchers

Y3

120 minutes

30

60

Registry of Evaluation Researchers
Update*

Y3

30 minutes

100

50

Registry of Randomized
Controlled Trials

Y3

30 minutes

125

62.5

Registry of Randomized
Controlled Trials Update*

Y3

15 minutes

90

22.5

645

210

1,600

542.5

Total, Year 3
Total, Years 1–3

* The Registry of Evaluation Researchers and the Registry of Randomized Controlled Trials updates are
not different forms. Rather, users are provided an opportunity to edit the information that already exists and
is prepopulated in the appropriate completed form.

12

EXHIBIT 3
Estimates of Hourly Burden
Year

Number of
Respondents

Hourly Burden

Cost

Intervention, Study, and Topic
Nomination Forms

Y1

300

15

$289.35

Registry of Evaluation
Researchers

Y1

60

120

$2,314.80

Registry of Randomized
Controlled Trials

Y1

50

25

$482.25

160

$3,086.40

Form

Total, Year 1
Intervention, Study, and Topic
Nomination Forms

Y2

300

15

$289.35

Registry of Evaluation
Researchers

Y2

30

60

$1,157.40

Registry of Evaluation
Researchers Update

Y2

100

50

$964.50

Registry of Randomized
Controlled Trials

Y2

75

37.5

$723.38

Registry of Randomized
Controlled Trials Update

Y2

40

10

$192.90

172.5

$3,327.53

Total, Year 2
Intervention, Study, and Topic
Nomination Forms
Registry of Evaluation
Researchers
Registry of Evaluation
Researchers Update
Registry of Randomized
Controlled Trials
Registry of Randomized
Controlled Trials Update

Y3

300

15

$289.35

Y3

30

60

$1,157.40

Y3

100

50

$964.50

Y3

125

62.5

$1,205.63

Y3

90

22.5

$434.03

210

$4,050.90

542.50

$10,464.83

Total, Year 3
Total, Years 1–3

13

13. Estimate of Annual Cost Burden to Respondents
There are no start-up costs to respondents related to this collection or other costs not
accounted for earlier.

14. Estimate of Annual Cost to the Federal Government
The total cost to the federal government for the nomination forms, Registry of Evaluation
Researchers, and Registry of Randomized Controlled Trials is estimated at $48,205.08 across
the three years, with the total annualized cost at $16,068.36. Exhibit 4 displays the costs for
developing the Registry of Randomized Controlled Trials, administering all online forms and
databases, and promoting and requesting input from the public in Year 1. Years 2 and 3
display the costs for maintaining and administering the online forms and databases,
promoting and requesting input from the public, and analyzing and tabulating results. The
total cost represents an $181,046.49 decrease from the original total cost estimate in January
2007.

EXHIBIT 4
Estimated Cost to Federal Government, by Year

Year
Year 1

Total Number
of Respondents
410

Maintenance and
Administration
$3,373.30

Development
$35,038.08

Total
$38,411.38

Year 2

545

$4,485.35

$0.00

$4,485.35

Year 3

645

$5,308.35

$0.00

$5,308.35

Total

1,600

$13,167.00

$35,038.08

$48,205.08

15. Program Changes or Adjustment
This set of information collections will result in an annual burden of 160 hours in Year 1, 172.5
hours in Year 2, and 210 hours in Year 3. The annualized hours total is 181, which is a decrease
of 680 hours from the initial collection estimate in January 2007. Based on the numbers of public
submissions we have received in the past two years, we have adjusted the figures down to
estimate the number of future responses more accurately.

16. Plans for Tabulation and Publication of Results
The majority of tabulations of the information collected under this request will be for internal use
by WWC contractors and Department of Education staff. The information will be used to
ascertain the needs of the public and to take their suggestions, as well as to monitor how well the
WWC is meeting their needs.

14

17. Approval to Not Display OMB Expiration
This approval is not requested.

18. Exceptions to Certification for Paperwork Reduction Act
Submissions
There are no exceptions.

15

Revised Forms for Existing and Approved Collection
01

Study Submission Form

02

Intervention Nomination Form

03

Topic Nomination Form

04

Registry of Evaluation Researchers Form

05

Registry of Evaluation Researchers Commitment Letter

Proposed New Forms
06

Registry of Randomized Controlled Trials (text documentation)

07

Registry of Randomized Controlled Trials (details)

08

Registry of Randomized Controlled Trials Commitment Letter

16