Supporting Statement Part B: What Works Clearinghouse (WWC) Feedback Task
Respondent Universe and Sampling Methods
Virtual focus groups with WWC users
We will conduct six 60-minute focus groups with individuals who have recently used the WWC.
To recruit focus group members, we will conduct an outreach campaign informing WWC users that we want their feedback and encouraging them to sign up for an online virtual focus group. The outreach campaign will explain that we are seeking feedback focused specifically on the four new resources: videos, practice guide summaries, topical blasts, and the What Works tool.
We will distribute a two-part demographic survey to interested participants. The first part, drawing on their role in education, and potentially, their familiarity with the WWC, will be used to assign them to the appropriate virtual focus group. The second part, drawing on their experiences with and knowledge of the WWC, will be used to better understand the participants’ level of familiarity with the WWC in advance of the discussion. The virtual focus groups will be organized by practitioner type: classroom instructors, school leadership, and district or state administrators. We will conduct two focus groups with each practitioner type.
The outreach campaign will be disseminated to the following groups: (1) people who follow the WWC on Facebook, (2) people who follow the WWC on Twitter, (3) people who have contacted the WWC Help Desk, (4) authors of articles about the WWC, (5) people on the Institute of Education Sciences’ NewsFlash dissemination list, and (6) participants in previous WWC webinars. Users can also indicate their interest in participating in the focus group by checking a relevant box on the user feedback survey (detailed below).
User feedback survey
We will invite all WWC website users to complete a pop-up, or intercept, survey after they have watched a video, downloaded a practice guide summary, or accessed the topical blast landing page. The survey will include a pop-up message that clearly identifies the purpose of the survey and estimates the time required to complete the survey. It will also include a checkbox for users to indicate whether they are interested in participating in the virtual focus groups (this option will only exist for the first eight weeks of the survey). The pop-up survey will be active for eight weeks. After eight weeks, the survey will be available to all WWC website users via a web link on the WWC home page.
2. Procedures for the Collection of Information
Virtual focus groups with WWC users
We plan to invite eight WWC website users to each virtual focus group with the goal of having five users per group. The emailed sign-up form will ask WWC users to provide their name; email address; role in education (classroom instructor, school leadership, district or state administrator); and level of familiarity with the WWC. Using this information, we will assign them to the appropriate focus group. We will offer a $30 Amazon.com gift card as an incentive, which we will email to the participants after completion of the virtual focus group.
Because the participants will be geographically dispersed, we will convene virtual focus groups via a computer software program that enables users to hold online meetings and share screens. A Mathematica staff member will lead these focus groups and will schedule them in the evenings, or if necessary due to participant availability, on weekends.
User feedback survey
The pop-up, or intercept, survey questions will focus on the following topics:
User type
Reason for accessing the resource
Overall rating of the ease of website navigation
Overall rating of resource clarity, presentation, and usefulness
Additional comments and suggestions for improvement
The survey consists of eight questions. We anticipate the total survey time will take approximately four minutes.
3. Methods to Maximize Response Rates and Data Reliability
Virtual focus groups with WWC users
We will offer a $30 Amazon.com gift card as an incentive to individuals who participate in the virtual focus group. Mathematica staff members will serve as moderators and will conduct all focus groups according to the protocols developed to solicit feedback on four new resources.
User feedback survey
Pop-up surveys typically have low response rates, which was the case with the most recent WWC user satisfaction survey conducted in 2011. That survey, sent to users via email, achieved a response rate of only 4.1 percent.
To encourage greater response rates for this survey effort, we developed an online pop-up, rather than email, survey; frontloaded key questions in the beginning of the survey should a respondent terminate his or her response; and provided an incentive for completion, which is a common practice to increase online survey response rates.
For an online pop-up survey, offering incentives to all respondents could compromise data quality. Therefore, we plan to hold a drawing in which one user will win a $50 Amazon.com gift card. All respondents will have an equal chance of winning; users will be entered into the drawing only during the eight weeks when the survey is active as a pop-up survey.
4. Test of Procedures or Methods
We conducted a pre-test of the survey by selecting individuals similar to our target population and asking them to complete the survey. We asked respondents to examine the WWC resources and answer the survey with those resources in mind. The respondents gave us feedback on the survey and survey responses. Seven respondents noted that it took an average of 4.4 minutes to complete the survey.
5. Individuals Consulted on Statistical Methods
There were no consultations on the statistical methods.
Contact information for the key individual responsible for collecting and analyzing the data:
Neil Seftor, Senior Researcher, (609) 275-2246, nseftor@mathematica-mpr.com
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Cassandra Pickens Jewell |
| File Modified | 0000-00-00 |
| File Created | 2021-01-25 |