13 Attachment 9 Evaluator Interview Protocol

Attachment 8

Evaluator Interview Protocol

OMB No: XXXX-XXXX

Expiration Date: XX/XX/20XX

OMB No.: 0925-XXXX

Expiration Date: XX/XX/20XX

Public reporting burden for this collection of information is estimated to average 30 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to: NIH, Project Clearance Branch, 6705 Rockledge Drive, MSC 7974, Bethesda, MD 20892-7974, ATTN: PRA (0925-XXXX). Do not return the completed form to this address.

Rights of study participants are protected by The Privacy Act of 1974. Participation is voluntary, and there are no penalties for not participating or withdrawing from the study at any time. Refusal to participate will not affect your benefits in any way. The information collected in this study will be kept private to the extent provided by law. Names and other identifiers will not appear in any report of the study. Information provided will be combined for all study participants and reported as summaries. You are being contacted to participate in this data collection so that we can evaluate the Science Education Partnership Award Program.

SEPA Evaluator Interview

Date and time of interview: INSERT HERE

Evaluator: PREPOPULATE

Institution: PREPOPULATE

Project title: PREPOPULATE

Funded Year: PREPOPULATE

End Date: PREPOPULATE

Project abstract: PREPOPULATE

Introduction

• Thank you for taking the time to talk with me today about your SEPA project.

• We are interested in hearing about your views related to the evaluation and about the SEPA program in general. Specifically, we are interested in learning about the following:

• Your level of involvement in designing the SEPA project evaluation

• Whether you needed or received evaluation technical assistance guidance from NIH

• Your experience conducting the evaluation for this SEPA program

• What the SEPA project accomplished.

• The interview will take about an hour and 15 minutes.

• Your participation is voluntary.

• We would like to audiotape this interview. It helps us take more accurate notes. Is that okay? At any point during the interview, we can stop or you can ask us to turn off the audio recorder. [If permission to tape is granted.] Recorded interviews will be stored in a secure location and will be destroyed after the end of the study.

• The data will be reported in aggregate and individuals associated with comments will not be disclosed.

• We are also interested in documents or other sources of information regarding any of the topics we discuss.

A. Evaluator’s Background

First, I would like to ask you a few questions about your background.

1. How did you become an evaluator? Has this been your principle line of work or were your engaged in other activities previously? What is your academic background?

[Probe: What degrees or certifications do you hold?]

2. How long have you worked as an evaluator (i.e., in years)?

3. Have you conducted similar evaluations of STEM Ed interventions before? Of other SEPA projects?

[Probe: Please describe the types of evaluations you have conducted. Confirm whether these

were evaluations of STEM Ed interventions.]

4. How well did your previous evaluation experience prepare you to conduct this SEPA evaluation?

[Probe: In what ways did you feel well prepared? In what ways did you feel under prepared?]

5. Do you have previous experience using the methods being used in the SEPA evaluation?

[Probe: Ask what methods are being used (i.e., observations, interviews, surveys etc.) and

which are similar/different than those they’ve used in other evaluations.]

B. Evaluation of [TITLE OF SEPA PROJECT]

The rest of the interview asks about your views regarding evaluation and your involvement designing the evaluation of the [TITLE OF SEPA PROJECT] project. The questions will touch on a number of topics, including your level of involvement in the evaluation design, any evaluation guidance you received from SEPA, your general views about the evaluation, your experience conducting the evaluation, and your perceptions of project impact.

Level of involvement designing the evaluation

6. How did you (or your team) become the evaluator for the SEPA project (e.g., open solicitation, previous connections, or referrals)?

7. At what point of the project did you get involved (e.g., during the preparation of the proposal, at the beginning of the SEPA project, mid-term, or the end of the SEPA project)?

8. Who was responsible for developing the evaluation study design?

[Probe: If someone else designed the evaluation, ask whether they had a role and what it was.]

9. To what extend do you think the evaluation influenced the design of the intervention?

[Probe: Did the evaluation design impact the extent of implementation in some sites? Did formative feedback from the evaluation influence development of the intervention as it matured? Please provide an example that illustrates your point.]

10. How would you describe your working relationship with the project team?

[Probe: Were you included in regularly scheduled meetings? Was the project team eager/reluctant to engage in evaluation-related activities and discussions? To what do you attribute their eagerness/reluctance? What makes you think so?]

Evaluation technical assistance

11. To what extent did the SEPA program solicitation provide clear and adequate guidance about key elements of the evaluation design?

[Probe: What was clear? What was unclear?]

12. During the evaluation, did you feel the need for technical assistance or other support?

[Probe: If yes, what TA or supports did you need? Did the SEPA program provide the TA or direct you to a TA provider? Who did you go to get this support? Was it useful? If not, why not.]

13. Do you think it is useful for SEPA to provide evaluation TA?

[Probe: If so, what kinds of TA would be most useful?]

Evaluation Design

14. Please briefly describe the key features of the evaluation design.

[Probe: Is there a comparison group? Were units randomly assigned? If not, were steps taken to insure the groups were equivalent at baseline? If so, please describe how.]

15. What were the major factors/considerations that shaped the evaluation design?

[Probes: Did the evaluation build on your experiences evaluating other STEM Ed programs? Did you borrow methods or protocols from other SEPA evaluations? Did you use STEM Ed evaluation resources developed by other programs (e.g., CAISE website’s repository of informal science ed projects)? In your view, did these strengthen or weaken the evaluation?]

16. Did the design of the evaluation address all the project’s goals?

[Probe: If not, how did you decide which ones to focus on? Why were some not included? Which ones?]

17. What do you see as the main strengths of the evaluation?

[Probe: Please explain why you see these as strengths.]

18. If the evaluation design had shortcomings, what were they, and why did they arise?

[Probe: Was the budget sufficient to support the evaluation? Is there anything that you were not able to do with your budget that would have been possible with more funding? How much more funding would have been needed? Was more time needed after the project ended to detect additional outcomes? What changes would have been necessary to address these shortcomings? How feasible was it to make these changes? What makes you think so?]

19. Were there aspects of the evaluation that were particularly unique?

[Probe: If yes, describe.]

Conducting the evaluation

20. Was the evaluation implemented as designed?

[Probe: If no, what were the major changes? Why did they occur?]

21. What challenges did you encounter conducting the evaluation?

[Probe: How did you handle them? Was the project team supportive of the evaluation? What makes you think so?]

22. Did the project team use the evaluation findings in their work?

[Probe: If yes, in what ways? If no, why do you think they did not?]

23. What aspects of the evaluation seemed to you to be most useful or least useful?

[Probe: Please briefly describe what makes you think so.]

Project impacts

The next set of questions asks about project impacts.

24. What were the project’s most significant accomplishments?

[Probe: Is there evidence from the evaluation that these impacts occurred?]

25. Where there project outcomes that were hard to measure?

[Probe: What were they and what made measurement difficult?]

26. Are there findings from the evaluation that have implications for others working on STEM Ed projects with goals that are similar to SEPA’s?

[Probe: If yes, ask what they are.]

27. Did the evaluation identify unanticipated impacts?

[Probe: How were these unanticipated impacts detected? What suggestions do you have for other SEPA evaluators about ways to capture information about unanticipated impacts?]

28. Were there impacts that the project was not able to accomplish or impacts that had lower than anticipated levels?

[Probe: If there were, what factors accounted for these impacts not being accomplished?]

29. Are there impacts that you believe occurred that you were not able to document through the evaluation?

[Probe: Why? What type of information would you need to evaluate these impacts?]

Lessons Learned

30. What lessons did you learn conducting the evaluation that would be helpful for other SEPA evaluators to know? For other PIs to know? For NIH to know?

31. If you had a chance to conduct this evaluation again, what would you do differently?

[Probe: Why? How?]

THANK YOU FOR TAKING TIME TO TALK WITH ME TODAY. OUR CONVERSATION WAS VERY HELPFUL. I ENJOYED TALKING WITH YOU.