What Works Clearinghouse Formative Feedback

WWC Customer Service Focus Groups:

Protocol for Virtual Focus Groups

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB number. The valid OMB control number of this information collection is 1800-0011. The time required to complete this information collection is estimated to average 75 minutes per customer satisfaction focus group. If you have any comments concerning the accuracy of the time estimates(s) or suggestion for improving this form, please write to: U.S. Department of Education, Washington, 20202-4651. If you have comments or concerns regarding the status of your individual submission of this form, write directly to: Institute of Education Sciences, US Department of Education, 555 New Jersey Ave, NW, Washington, DC 20208.

A. Consent/introduction to the project

Thanks again for taking the time to speak with me this morning/afternoon. Before we start, I’d like to provide a little background on our work, and answer any questions you might have for me. As you probably know, I work for Mathematica Policy Research, which is a nonpartisan research organization. Mathematica is one of the contractors operating the What Works Clearinghouse (WWC) on behalf of IES. The mission of the WWC is to be a central source for evidence from high quality research in order to answer the question: of ‘What Works’ in Education to improve student outcomes. We are examining the usefulness and ease of use of the WWC, for the Institute of Education Sciences (IES) at the U.S. Department of Education. We will be conducting customer satisfaction focus groups like this one with different stakeholders throughout the country. This short conversation will ask you first to comment on your familiarity with the WWC in general and then ask you to engage and react to 3 specific products that the WWC recently produced. We anticipate using the ideas this group generates to improve the WWC products and services.

We will NOT report results in any way that would identify you. Your participation is voluntary, and you do not have to respond to any questions you do not want to. 

I’m recording the discussion so I don’t have to take detailed notes and can listen carefully to what you are saying. The recording is just to help me remember what you say. No one outside of the research team will have access to the tape. Do we have your permission to record? Do you have any questions before we begin?

Let’s get started. [HIT THE RECORD BUTTON].

I have hit the record button.

_______________________________________________________________________

B. Introductions

1. As you will see on my screen, there are four questions I’d like you to respond to. [The following questions will be listed ON THE SCREEN] I’ll call your name individually. Please take just a few moments and tell me the following things about yourself:

• Your name

• Your professional occupation ( (i.e. are you a teacher, curriculum director, building administrator, researcher, etc).

• Please also let me know, if you teach, which grade you teach and if you are an administrator, what level you work with. (i.e. district and/or building and/or state)

• How would you describe your level of familiarity with the What Works Clearinghouse? Please tell me if you are 1)Not very familiar, 2) somewhat familiar, or 3) quite familiar?

___________________________________________________________________________

Today I’ll be talking with you about a few topics. These include: products or types of information that may be helpful to you as a practitioner. Then we will spend some time reviewing specific What Works Clearinghouse products in order to gather your thoughts on the utility of the products and how they could be improved.

Links to these products were sent in advance of the meeting for your review but if you haven’t reviewed them, that is perfectly fine.

___________________________________________________________________________

C. Experiences with the WWC

2. We are interested in your experiences on the What Works Clearinghouse website. What usually is your motivation for visiting the What Works Clearinghouse?

1. Were you able to find what you were looking for?

• If so, was it easy to find the information? Was the site easy or hard to navigate?

Are some kinds of information easier or more difficult to find than others? Please explain.

• If not, what would help you find that information on the Website?

b) How do use the information found at the WWC?

c) The intervention reports, which are some of WWC’s products, have three different ratings (intervention rating, effectiveness rating, and extent of evidence); Do you understand the ratings? Are they useful? Is there a better way to describe the set of findings to be meaningful to you? [Note, on the shared screen there will be an example of an intervention report for the group to refer to]

3. What is your understanding of the resources available through the What Works Clearinghouse? Who do you feel the target audience(s) is/are?

_____________________________________________________________________________

I’d like to now take about 20 minutes to review three of the products that the What Works Clearinghouse has recently produced. We are very interested in getting your feedback as to their clarity and utility to better inform future products. After we engage these products, we’ll have a series of questions for you.

______________________________________________________________________________

D. Product Number 1: Video

The What Works Clearinghouse has produced one video on the ways that the WWC’s resources can help with the Selection of Math Instructional Materials. This video is intended to provide district-level decision-makers with information on what WWC resources can inform the selection process. This is a short, 4 minute video that I’m going to show you now.

[Show Sample Video (4 minutes): “How to Select Mathematics Instructional Materials”]

4. Let’s talk a bit about the video.

1. Is this video useful for district decision makers?

2. Is the content in the video clear? If not, in what ways could it be clearer or improved?

3. Would you recommend this particular video to others? If so, whom? If not, why not?

4. What changes, if any, would you make about the information the video contains and/or how the information is presented? Do you have suggestions for how the product could be improved?

5. Is the length of the video appropriate?

6. Are there other topics for videos that you could suggest that might be useful to you and/or other practitioners you work with?

_____________________________________________________________________________

E. Product number 2: A sample practice guide summary/By the Numbers

The WWC produces “practice guides” which provide evidence-based recommendations for educators on a range of topics. The recommendations are made by a panel of experts after a review of the research. Each recommendation includes implementation steps and solutions for common roadblocks. [To complement these often lengthy guides, the WWC has started to produce “practice guide summaries” products which synthesize the key recommendations from practice guides.] OR [To complement these often lengthy guides, the WWC produced a product which is essentially an orientation to the practice guide. One such product, “By the Numbers” orients the reader to the Teaching Math to Young Children practice guide]

[Interviewer to demonstrate/walk through the product on using a shared screen capability]

5. Let’s spend a few minutes now talking about the practice guide summary/by the numbers document.

1. What information in the summary did you find informative and useful?

2. Is the content clear? If not, in what ways could it be improved?

3. Would you recommend it to others? If so, whom? (i.e. would you show it to your school leaders? District staff?) If not, why not?

4. Does it pique your interest in the full guide?

5. What additional materials, if any, would make the summaries/By the Numbers product more useful?

___________________________________________________________________________

F. Product number 3: Topical Blast

The WWC also produces web pages and corresponding email blasts that consolidate information and resources from the What Works Clearinghouse related to a specific topic or event. Those who do not receive the email blast can access the materials in the ‘rotator’ at the top of the website or in the NEWS and EVENTS feature. The WWC has produced a handful of these blasts over the past year. They include information on going to college, challenging classroom behaviors and literacy, among other topics. I will now guide you through the Back to School blast, which was released in late August to coincide with the start of the academic semester. This consolidates many WWC products on instruction.

[Interviewer to demonstrate/walk through the product using a shared screen capability]]

6. Let’s spend a few minutes discussing the topical blast we just explored.

1. What information did you find informative and useful?

2. Is the content presented in the topical blast clear?

3. Would you recommend it to others? If so, whom? If not, why not?

4. What changes, if any, would you make about the information the blast contains and/or how the information is presented? Do you have suggestions for how the product could be improved?

5. Are there other topical blasts that you could suggest that might be useful to you and/or other practitioners you work with?

__________________________________________________________________________

G. Dissemination

7. You have been able to interact with several of the products this [afternoon/morning].The What Works Clearinghouse currently engages in quite a few dissemination strategies. In addition to the email blasts we mentioned, the WWC hosts webinars, posts materials on the website for users to find via other channels and uses facebook and twitter to alert users to new content.

a) Do you have any overall suggestions for ways in which the information on the WWC could be disseminated more broadly? Do you have any suggestions of how to get new people to the WWC?

b) Are there audiences that you feel the WWC (IES) may not be targeting? Which ones? What type of information would they want to access?

8. While we have viewed three new products, (a video, a practice guide summary and a topical blast) we would also be interested to hear your thoughts on the three types of reports the WWC produces, intervention reports, single study reviews/quick reviews and practice guides. The electronic brochure we shared ahead of time outlines these reports. For those of you who had a chance to review these reports are there particular reports you found more useful than others? Did you find the presentation of findings assessable and easy to understand?

a) Which report type was the most useful? Why?

9. Thinking about all of the products the WWC produces are there other formats (i.e. types of reports, etc) you think the What Works Clearinghouse should produce? Are there particularly products you feel are more useful than others?

a) What formats would be useful? Why?

b) Are there products you feel are more useful than others?

_____________________________________________________________________________

H. Wrap Up

10. As we close, I just wanted to remind you that the WWC feedback effort is working to evaluate ways in which the WWC and its associated products could be improved. With this in mind, is there anything that I haven’t asked you about regarding the What Works Clearinghouse and your experiences and interactions with the WWC that you’d like to comment on? Is there anything else we should be aware of?

DRAFT 4