1. Circumstances Making the Information Collection Necessary
The Institute of Education Sciences (IES) within the U.S. Department of Education is proposing data collection activity as part of the What Works Clearinghouse (WWC) feedback task. The task and its associated efforts are being undertaken by the U.S. Department of Education, IES, and is being conducted by Mathematica Policy Research.
The intended purpose of the WWC feedback task is to collect users’ feedback on the relevance, timeliness, quality, and ease of use of the resources associated with the WWC website. The results of the data collection will inform improvements in WWC resources and services.
The WWC provides educators, policymakers, and the public with a central and trusted source of scientific evidence of what works in education. The WWC aims to make findings from education research easy and accessible through its searchable online repository of intervention reports, single-study reviews, and practice guides. Thousands of empirical studies claim to identify effective instructional approaches, many using complicated research methods and statistical analyses, yet this research often yields conflicting results, leaving educators wondering which approach to take. Given the large volume of education research and significant variation in quality, principals and other educators need help identifying reliable research and interpreting findings. Using systematic review processes and evidence standards, the WWC reviews all the research on a topic to identify the most rigorous studies and synthesize the findings from high quality education research.
The WWC has started to develop four new resources that focus on using the WWC when making key decisions in education. First, the WWC is developing several videos that describe the purpose of the WWC and how to understand specific materials on the website. For example, the WWC has already released a video that addresses how to select a mathematics curriculum. The WWC has also developed practice guide summaries that consolidate the information from practice guides into 8–10-page summaries that present expert recommendations from the field, along with tips on implementing the recommendations. The WWC has already released two of these summaries—Teaching Math to Young Children and Teaching Elementary School Students to Be Effective Writers. Third, topical blasts consolidate WWC content relevant to a specific education topic. Emails direct users to a dedicated landing page containing links to the relevant content. Finally, the WWC is redesigning the Find What Works tool on the website to improve how users locate and use information throughout the site. Findings from the case studies of these resources will facilitate improvement of these and other WWC resources.
The WWC feedback task will include the following data collection methods: virtual focus groups with WWC users, user feedback web surveys, and data analytics. Detailed information about these methods is below:
Virtual focus groups with WWC users
We will conduct six 60-minute virtual focus groups with individuals who have recently used the WWC. These groups will be organized by practitioner type, such as classroom instructors, school leadership, and district or state administrators. Two of the six groups will target classroom instructors, two will target school leadership, and two will target district or state administrators. We will send a brief demographic survey to interested participants in advance of the discussions so we can appropriately assign them to a focus group.
Web surveys
All users will receive invitations to complete a pop-up, or intercept, survey after viewing a video, downloading a practice guide summary, or accessing the topical blast landing page. The survey will include a pop-up message that clearly identifies the purpose of the survey and estimates the time required to complete the survey. The pop-up survey will be active for eight weeks.
After the pop-up survey has run for eight weeks, we will place it on the WWC website and make a link to it available to all users to provide feedback. The survey will be available for 36 months. Questions will focus on the following:
• User type
• Reason for accessing the resource
• Overall rating of the ease of website navigation
• Overall rating of resource clarity, presentation, and usefulness
• Additional comments and suggestions for improvement
The survey consists of eight questions. Based on trial tests of the survey, we anticipate that survey completion will take approximately four minutes.
Data analytics
Data analytics will provide basic information about resource use. Most of the data we analyze will reflect website use statistics available from the WWC’s website analytics contractor, Sanametrix. Specifically, for each of the resources, we will analyze the following website statistics:
Number of times the resource landing page is visited
Number of times the resource is downloaded (if PDF) or viewed (if video)
Number of times the WWC home page is visited
For each resource, we propose to track this information for two months following the resource release. In addition to website statistics, we will obtain data on the number of media mentions about each resource, the nature of those media mentions, and the types of questions and comments about these resources submitted to the WWC Help Desk. Mathematica and subcontractor staff oversee the WWC Help Desk and respond to inquiries posted through the Contact Us form on the WWC website.
The WWC requests clearance for the following data collection activities:
(1) Demographic questionnaire for the virtual focus group participants
The questionnaire contains demographic and WWC website use questions, which we will collect from participants before the start of the virtual focus group. Advance data collection will enable the virtual focus group moderator to assign participants to the appropriate virtual focus group on the basis of their role in education, as well as better prepare for the discussion by understanding participants’ experiences with and knowledge of the WWC.
(2) Protocol for the virtual focus groups
The protocol contains questions related to the resources that we will demonstrate during the virtual focus groups. We will solicit feedback from users about the clarity, presentation, and usefulness of the WWC resources.
(3) Web survey to document user feedback on the website and resources
The survey will focus on user feedback on the WWC website and its resources. The information requested in the survey will include user type, reason for accessing the WWC, source of information about the WWC, an overall rating of usability, ratings of individual aspects of the resource, overall satisfaction with the website, and additional comments and suggestions for improvement.
These three data collection instruments are submitted with this
request, along with the existing WWC instruments.
2. Purpose and Use of the Information Collection
The information we will obtain through the WWC feedback task is critical to determining the usability of the WWC website; the clarity, presentation, and usefulness of its resources; and how best to improve both the website and resources for users. We will obtain information for the WWC feedback task from virtual focus groups, a web survey, and data analytics. Next, we present details on the purpose and use of the information collection.
Drawing on the insights learned from this analysis, we will draft a report, as well as a summary of findings suitable for publication on the WWC website. For each of the four WWC target resources, the report will provide the following:
A summary of the users of each resource. Drawing on the survey data, for each resource we will identify the characteristics of the primary users. This will enable us to assess whether the WWC is reaching the intended groups with these resources.
An assessment of the accessibility of the resource. We will provide information on how people use the resource, why they use it in the way they do, what the WWC can do to better support that use, and how to develop resources that improve use of the site overall.
A discussion of implications for future resources. On the basis of the feedback received from the data collection, we will identify potential changes for the WWC website and for the next round of resources to be developed.
In addition to the final report, we will produce a high-level, succinct document describing the methods that inform the report, the findings of the task, and suggested implications for changes. This document will include an infographic summary of the findings and can potentially be posted on the WWC website, making it clear to visitors that the WWC is interested in knowing how to better respond to the needs of current and potential users.
Lastly, drawing on the findings and recommendations presented in the final report, we will develop a proposed action plan. This action plan will explain the specific next steps to using the lessons learned from the feedback effort.
3. Use of Improved Information Technology and Burden Reduction
Demographic questionnaire for the virtual focus group participants. The demographic questionnaire for focus group participants can be submitted electronically via email.
Virtual focus groups. The virtual focus groups will be conducted via screen-sharing technology. This will enable users to attend from their homes/offices without losing any visual aspect of the focus group. We will also use the technology to record these sessions for internal project use.
Web survey. The web survey will allow for electronic submission of responses.
4. Efforts to Identify Duplication and Use of Similar Information
The WWC feedback task is the first effort that will collect information on the clarity, presentation, and usefulness of WWC resources. No other similar information exists.
5. Impact on Small Businesses or Other Small Entities
No small businesses are expected to be involved in data collection.
6. Consequences of Not Collecting Information or Collecting Information Less Frequently
Not collecting information for the WWC feedback task would limit the government’s assessment of the effectiveness of the WWC and improvement of the WWC website and its resources for future users. In particular, the WWC feedback task represents an important opportunity for IES to learn how WWC users engage with the website resources.
7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5
There are no special circumstances for the proposed data collection.
8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency
The 60-day notice to solicit public comments was published in the Federal Register on May 1, 2015. There were no public comments.
9. Explanation of Any Payment or Gift to Respondents
We propose to offer a $30 Amazon.com gift card to virtual focus group participants. We suggest offering this payment for one reason:
Increased response to invitation for focus group participation. The focus group will take approximately 60 minutes. The knowledge that they will be paid for completion is expected to increase users’ likelihood of agreeing to participate in the WWC user focus group.
We also propose to offer a $50 Amazon.com gift card to one winning web survey participant via a drawing. Users will be entered into the drawing to win the gift card only during the eight weeks when the survey is active as a pop-up survey. We suggest offering this payment for one reason:
Increased response rate to the web survey. Pop-up surveys typically have low response rates, and the most recent WWC user satisfaction survey (an emailed survey conducted in 2011) achieved a response rate of 4.1 percent. We believe that offering the possibility of an incentive will increase users’ likelihood of responding to the survey. McClendon and Olson (2013)1 found that a random drawing for an incentive increased web-based survey response rates. We expect 4,700 responses in the first eight weeks. A single $50 incentive represents a cost of a fraction of a penny per survey, which we believe is a cost effective means to increase the response rate.
10. Assurance of Confidentiality Provided to Respondents
Respondents’ privacy will be protected to the fullest extent permitted by law. Study participants will be informed that their participation is voluntary and that information will be kept private to the extent permitted by law, and that information gathered will be presented in the aggregate format and used only for analytical purposes. The collection is voluntary.
Respondents will be informed of their privacy protection under the Education Science Reform Act (ESRA):
Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) Public Law 107-279, Section 183].
For the Contact Us forms, case ID numbers will be used in association with the respondent’s information. The website has a privacy policy statement indicating that no individual identification will be posted on the public website without the user’s permission. Information about users will be reported at the aggregate level only. Contact information on Contact Us forms is used for the express purpose of clarifying the WWC user’s submission and will not be released beyond the WWC staff with an immediate need to clarify the submitted information.
The purpose of the Registry of Evaluation Researchers database is to allow evaluators to post their credentials and contact information to the public; therefore, it is assumed that when evaluators submit their forms, they are not requesting confidentiality. The WWC does, however, offer evaluators the option of limited disclosure for studies. For example, respondents can request that proprietary information be kept confidential. This typically will be requested only when an evaluator is in the process of getting a study published and would like the WWC to post the study information after it is publicly available. Additionally, the WWC will post only the provided contact and experience information; biographies, resumes, and curricula vitae will not be available through the WWC. The purpose of the Registry of Randomized Controlled Trials is to enable researchers and other stakeholders to post information to the public regarding RCTs that are funded and in progress or completed, and with an available final report; therefore, it is assumed that when stakeholders submit their forms, they are not requesting confidentiality. The WWC will post only the provided bibliographic information and study abstract; the full text of the study will not be available through the WWC.
Focus group and survey responses will be anonymous and unidentifiable.
11. Justification for Sensitive Questions
We will not ask any sensitive questions for the WWC feedback task.
12. Estimates of Annualized Burden Hours and Costs
The following table summarizes the proposed total estimated reporting burden for the WWC feedback focus groups and web survey.
Instrument |
Total number of respondents |
Average burden hours |
Estimated total burden hours |
Total annual burden hours |
Questionnaire for focus group participants |
48 |
.25 |
12 |
12 |
Focus groups |
30 |
1 |
30 |
30 |
Web survey |
84,600 |
.06 (each survey is about four minutes) |
5,076 |
1,692 |
Questionnaire for virtual user focus groups. We expect to invite eight users to participate in each focus group with an estimated group size of four to six individuals; this would total approximately 48 respondents. Each questionnaire will take approximately 15 minutes to complete; thus, the estimated total burden is 12 hours annually (48 x .25).
Virtual focus groups. We expect to invite eight users to participate in each focus group with an estimated group size of four to six individuals. We plan to hold six user focus groups total with an average size of five participants; this would total approximately 30 respondents. The focus groups will take approximately 60 minutes; thus, the estimated total burden is 30 hours annually (30 x 1).
Web survey. The WWC receives approximately 235,000 visitors per month; thus, we estimate a total possible sample of 8,460,000 respondents over a 36-month period. We estimate a response rate of approximately 1 percent, which totals approximately 84,600 possible respondents. Each survey will take approximately four minutes to complete, which means the estimated total burden is approximately 5,076 hours. Because OMB clearance is valid for three years, this is approximately 1,692 hours per year.
Total burden. The following table summarizes the current ICR burden with the addition of the above new burden for the total burden for the WWC:
Instrument |
Total number of respondents |
Average burden hours |
Estimated total burden hours |
Total annual burden hours |
Contact Us Forms |
350 |
.05 |
18 |
18 |
Registry of Evaluation Researchers |
30 |
2 |
60 |
60 |
Registration of Evaluation Update Researchers Update |
60 |
.5 |
30 |
30 |
Registry of Randomized Control Trials Form |
80 |
.5 |
40 |
40 |
Registry of Randomized Control Trials Update |
60 |
.25 |
15 |
15 |
What Works Clearinghouse Formative Feedback (RELs, virtual focus groups, web survey) |
84,678 |
1.31 |
5,118 |
1,734 |
Total Burden |
85,258 |
4.55 |
5,281 |
1,897 |
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is 1850-0788. Public reporting burden for this collection of information is estimated to average xx minutes/hours per response, including time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. The obligation to respond to this collection is voluntary. If you have comments or concerns regarding the status of your individual submission of this form, application or survey, please contact: Institute of Education Sciences, US Department of Education, 555 New Jersey Ave, NW, Washington, DC 20208.
13. Estimates of Other Total Cost Burden to Respondents and Record Keepers
These information collection activities do not place any additional costs on respondents or record keepers.
14. Cost to the Federal Government
The total cost of the WWC feedback task to the federal government is estimated to be $155,277. Because the WWC feedback task will last three years, the total cost of the one-year request is $116,548; the annualized cost to the federal government is $116,548 in the first year; $18,908 in the second year; and $19,821 in the third year.
15. Explanation for Program Changes or Adjustments
This is a change due to the increase in respondents and the addition of the formative feedback focus group and web survey.
16. Plans for Tabulation and Publication and Project Time Schedule
Questionnaire for virtual focus groups. We will obtain the questionnaire responses and run cross-tabulations on the user types and WWC website use. The information will supplement data gathered from the virtual focus groups.
Virtual focus groups. We will transcribe each of the virtual focus group conversations and analyze the transcripts using Atlas.ti software to identify common themes and suggestions emerging from the focus groups.
Web survey. For the user feedback web survey, we will obtain the survey responses and run cross-tabulations on the user types and the accessibility of WWC resources. We will also complete a full analysis of all pop-up survey questions.
Web trends. We will examine the web trends for each resource, identifying similarities and differences in use patterns across resources and comparing them to overall WWC use patterns.
This WWC feedback task is expected to take place over one year beginning in September 2015.
Schedule for the feedback task
Activity |
Completion date |
Schedule user focus groups Execute web survey (pop-up version) Conduct user focus groups Execute web survey (link version) Submit memo on preliminary analysis Draft evaluation report and summary of findings for public Submit final evaluation report and summary of findings for public Submit memo summarizing action plan |
October 30, 2015 October 30, 2015 December 4, 2015 January 2, 2016 January 29, 2016 February 26, 2016 March 30, 2016
April 15, 2016 |
17. Reason(s) Display of OMB Expiration Date Is Inappropriate
All instruments will display the expiration date for OMB approval.
18. Exceptions to Certification for Paperwork Reduction Act Submissions
No exceptions are necessary for this information collection.
1 McClendon, Richard, and Danny Olsen. “Using the iPad as a Prize-Based Incentive to Boost Response Rates: A Case Study at Brigham Young University.” Presented at the American Association for Public Opinion Research conference, Boston, MA, May, 2013.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Subject | OMB Package Part A |
| Author | Sheena McConnell (formatted by Sheena Flowers) |
| File Modified | 0000-00-00 |
| File Created | 2021-01-25 |