Needs Assessment Protocol

Appendix

Needs Assessment Protocol

REL-Southeast at SERVE Center – University of North Carolina at Greensboro

Market Research Group Session

Needs Assessment Protocol

This needs assessment protocol is to be used in the REL-SE’s market research interactive group sessions, facilitated by Curtis Research Associates, a market research consulting firm. The protocol (with variations in topics and issues addressed depending on the group and year) will be utilized in settings of 20-40 participants who are engaged in other meetings or conferences. That is, we will use participants from existing state conferences or REL-SE events planned for other purposes. The purpose of the needs assessment is to ascertain from our regional clients their attitudes toward evidence-based education (EBE), any challenges they face in identifying and utilizing high quality information in their decision making, and the issues or topics for which they need high quality information.

The format of the two hour facilitated market research session consists of an introduction to the mission of the REL-SE and to the idea of evidence-based education, followed by the needs assessment protocol. The protocol consists of both qualitative and quantitative items. Participants use a hand-held electronic dial, the Perception Analyzer, to record their quantitative responses. The group’s quantitative responses can be graphically displayed and discussed with the participants, prompted by questions from the facilitator, as in a focus group discussion.

1. Introduction to REL-SE mission and Evidence-based Education

Following the introductory presentation by REL-SE staff, participants will be asked to respond in writing to the following prompt to help in getting them focused on their information needs. Their written responses will be collected at the end of the session.

Think about the three most pressing questions you have in your current role that you would like answered. The answers might come in the form of a written document or report such as a research synthesis, research or evaluation study, workshop, or other service. Write down the three questions with a little description for each question of why it is important to you.

2. Attitudes toward Evidence-based Education

The facilitator is introduced and explains the technology of the hand-held dials.

Facilitator: Using the Perception Analyzer, please rate how strongly you agree or disagree with each of the following statements.

1. In my organization, we face considerable challenges in trying to examine evidence of program effectiveness as part of the decision-making process.

0 = Strongly disagree

100 = Strongly agree

2. In my organization, we face considerable challenges in trying to conduct program evaluations when new initiatives are implemented.

0 = Strongly disagree

100 = Strongly agree

3. I am able to get the research information I need for various interventions/solutions my school/district/state is considering.

0 = Strongly disagree

100 = Strongly agree

4. I am where I need to be as an accomplished “evidence-based decision maker.”

0 = Strongly disagree

100 = Strongly agree

3. Priorities and Need for High-quality Information

The goal of this section is to understand the issues participants are currently grappling with so that the REL-SE can determine where it should focus its attention in providing products and services. The issues listed below are examples of topics we would explore with the various groups over the contract period. As new issues emerge, we will change this section of the needs assessment to explore these emerging issues.

Facilitator: As a decision maker in your school, district or state who is trying to improve learning in the classroom, we are interested in input from you on which issues are currently priorities for you. We realize that you have numerous issues competing for attention and action; however, we would like to get an idea of the issues that are currently commanding the highest priority for you and your organization in terms of your need for high-quality information. That is, do you have the information you need to inform your decisions or is this an area where you need more high-quality information to help you determine the optimal programs and policy decisions for your organization?

The issues we would like you to consider fall into five broad topic areas:

1. School improvement/reform

2. Curriculum/instruction

3. Teacher/leadership quality

4. Subgroups of students/meeting remediation needs

5. State policies/programs

Within each of these areas we have several issues that we would like to get your opinion about. We would like you to use your Perception Analyzer dial to rate how strong your need is for high-quality information on each of these issue areas. If any of the issue areas does not apply to you, please set your dial to zero, which means “not applicable.” Otherwise, you can use any number from 1 to 100 to indicate how great your need is for high-quality information.

As we are going along, if you notice any topics or issues that we have not included but you think are important, please jot them down. We will return to your topics later in the meeting.

Issue 1: School Improvement/Reform

5. How strong is your need for high-quality information on: The usefulness of “evidence-based budgeting” at the district or school level – where it’s being tried and how it works?

1 = No need

100 = Urgent need

6. How strong is your need for high-quality information on: The usefulness of literacy coaches at the middle and high school level?

1 = No need

100 = Urgent need

7. How strong is your need for high-quality information on: The range of initiatives in place in the Southeast to increase Advanced Placement course enrollment?

1 = No need

100 = Urgent need

8. How strong is your need for high-quality information on: Issues in defining dropout and graduation rates and trends for high schools in the Southeast?

1 = No need

100 = Urgent need

9. How strong is your need for high-quality information on: Processes district leaders can use to engage middle and high school principals in using data, research and evaluation in school improvement?

1 = No need

100 = Urgent need

10. How strong is your need for high-quality information on: The impact of various approaches to parental involvement in high school?

1 = No need

100 = Urgent need

11. How strong is your need for high-quality information on: The strengths and weaknesses of the school improvement planning process?

1 = No need

100 = Urgent need

12. Overall, for which of the following school improvement/reform issues do you have the greatest need for high-quality information? Pick one.

0 = None

1 = The usefulness of “evidence-based budgeting” at the district or school level – where it’s being tried and how it works

2 = The usefulness of literacy coaches at the middle and high school level

3 = The range of initiatives in place in the Southeast to increase AP course enrollment

4 = Issues in defining dropout and graduation rates and trends for high schools in the Southeast

5 = Processes district leaders can use to engage middle and high school principals in using data, research and evaluation in school improvement

6 = The impact of various approaches to parental involvement in high school

7 = The strengths and weaknesses of the school improvement planning process

13. For which of these school improvement/reform issues do you have the second greatest need for high-quality information? Pick one.

0 = None

1 = The usefulness of “evidence-based budgeting” at the district or school level – where it’s being tried and how it works

2 = The usefulness of literacy coaches at the middle and high school level

3 = The range of initiatives in place in the Southeast to increase Advanced Placement course enrollment

4 = Issues in defining dropout and graduation rates and trends for high schools in the Southeast

5 = Processes district leaders can use to engage middle and high school principals in using data, research and evaluation in school improvement

6 = The impact of various approaches to parental involvement in high school

7 = The strengths and weaknesses of the school improvement planning process

Issue 2: Curriculum/Instruction

14. How strong is your need for high-quality information on: The formative assessment techniques that are most promising for teachers in core content areas, particularly in low-achieving middle or high school settings?

1 = No need

100 = Urgent need

15. How strong is your need for high-quality information on: Research on the academic experiences, course-taking patterns, peer influences, and attitudes toward high school of African-American males/implications for instruction?

1 = No need

100 = Urgent need

16. How strong is your need for high-quality information on: Research on factors that influence high school students’ level of effort in completing assignments?

1 = No need

100 = Urgent need

17. Overall, for which of these curriculum/instruction issues do you have the greatest need for high-quality information? Pick one.

0 = None

1 = The formative assessment techniques that are most promising for teachers in core content areas, particularly in low-achieving middle or high school settings

2 = Research on the academic experiences, course-taking patterns, peer influences, and attitudes toward high school of African-American males/implications for instruction

3 = Research on factors that influence high school students’ level of effort in completing assignments

18. For which of these curriculum/instruction issues do you have the second greatest need for high-quality information? Pick one.

0 = None

1 = The formative assessment techniques that are most promising for teachers in core content areas, particularly in low-achieving middle or high school settings

2 = Research on the academic experiences, course-taking patterns, peer influences, and attitudes toward high school of African-American males/implications for instruction

3 = Research on factors that influence high school students’ level of effort in completing assignments

Issue 3: Teacher/Leadership Quality

19. How strong is your need for high-quality information on: The state of teacher preparation in the Southeast (trends in entering student demographics, courses, graduates, alternative certification program participation)?

1 = No need

100 = Urgent need

20. How strong is your need for high-quality information on: Approaches to improving content area instructional leadership in middle and high schools?

1 = No need

100 = Urgent need

21. How strong is your need for high-quality information on: The roles of the university, the state and the district in training and evaluating new principals in the Southeast?

1 = No need

100 = Urgent need

22. Overall, for which of these teacher/leadership quality issues do you have the greatest need for high-quality information? Pick one.

0 = None

1 = The state of teacher preparation in the Southeast (trends in entering student demographics, courses, graduates, alternative certification program participation)

2 = Approaches to improving content area instructional leadership in middle and high schools

3 = The roles of the university, the state and the district in training and evaluating new principals in the Southeast

23. For which of these teacher/leadership quality issues do you have the 2nd greatest need for high-quality information? Pick one.

0 = None

1 = The state of teacher preparation in the Southeast (trends in entering student demographics, courses, graduates, alternative certification program participation)

2 = Approaches to improving content area instructional leadership in middle and high schools

3 = The roles of the university, the state and the district in training and evaluating new principals in the Southeast

Issue 4: Subgroups of Students/Meeting Remediation Needs

24. How strong is your need for high-quality information on: Summarizing professional development programs for districts to choose from in helping teachers meet the needs of English language learners in their classrooms?

1 = No need

100 = Urgent need

25. How strong is your need for high-quality information on: The design and implementation of IDEA plans by school districts with disproportionality that focus on early intervening for students at risk of school failure (particularly the use of 15% of IDEA funds for regular education students)?

1 = No need

100 = Urgent need

26. How strong is your need for high-quality information on: Remediation for course failure in middle and high schools (what are states and districts doing with policies and programs; what approaches are most promising)?

1 = No need

100 = Urgent need

27. Overall, for which of these subgroups of students/meeting remediation needs issues do you have the greatest need for high-quality information? Pick one.

0 = None

1 = Summarizing professional development programs for districts to choose from in helping teachers meet the needs of English language learners in their classrooms

2 = The design and implementation of IDEA plans by school districts with disproportionality that focus on early intervening for students at risk of school failure (particularly the use of 15% of IDEA funds for regular education students)

3 = Remediation for course failure in middle and high schools (what are states and districts doing with policies and programs; what approaches are most promising)

28. For which of these subgroups of students/meeting remediation needs issues do you have the second greatest need for high-quality information? Pick one.

0 = None

1 = Summarizing professional development programs for districts to choose from in helping teachers meet the needs of English language learners in their classrooms

2 = The design and implementation of IDEA plans by school districts with disproportionality that focus on early intervening for students at risk of school failure (particularly the use of 15% of IDEA funds for regular education students)

3 = Remediation for course failure in middle and high schools (what are states and districts doing with policies and programs; what approaches are most promising)

Issue 5: Southeastern State Policies/Programs

29. How strong is your need for high-quality information on: Current status of southeastern state supported early childhood programs and accountability systems/approaches?

1 = No need

100 = Urgent need

30. How strong is your need for high-quality information on: Status of state policies/programs in Special Education (particularly inclusion and mainstreaming)?

1 = No need

100 = Urgent need

31. How strong is your need for high-quality information on: What states are doing to increase K-20 rigor and curriculum alignment, e.g., embedding 21^st Century Skills into the curriculum?

1 = No need

100 = Urgent need

32. How strong is your need for high-quality information on: Status of state efforts in implementing virtual school programs?

1 = No need

100 = Urgent need

33. Overall, for which of these state policy/program issues do you have the greatest need for high-quality information? Pick one.

0 = None

1 = Current status of southeastern state supported early childhood programs and accountability systems/approaches

2 = Status of state policies/programs in Special Education (particularly inclusion and mainstreaming)

3 = What states are doing to increase K-20 rigor and curriculum alignment, e.g., embedding 21^st Century Skills into the curriculum

4 = Status of state efforts in implementing virtual school programs

34. For which of these state policy/program issues do you have the second greatest need for high-quality information? Pick one.

0 = None

1 = Current status of southeastern state supported early childhood programs and accountability systems/approaches

2 = Status of state policies/programs in Special Education (particularly inclusion and mainstreaming)

3 = What states are doing to increase K-20 rigor and curriculum alignment, e.g., embedding 21^st Century Skills into the curriculum

4 = Status of state efforts in implementing virtual school programs

IV. Other Important Issues as Identified by Participants

Facilitator: Are there any issues that we didn’t include in the last section that you think are important – for which you have a need for high-quality information? (Take suggestions, and as appropriate, have the group rate their need for high-quality information on the issue.)

35. How strong is your need for high-quality information on: ?

1 = No need

100 = Urgent need

100 = Urgent need

36. How strong is your need for high-quality information on: ?

1 = No need

100 = Urgent need

5. Discussion Around Issues Rated High for “Need for Information”

Facilitator will show graphically the group’s quantitative results from the above items for each topic area and lead a discussion, as follows:

• What is going on in your organization that _(insert issue here)___ rated high on need for information? Can you give me an example?

• What challenges are you facing as you try to address this issues?

• Probe to understand if others in group feel similarly or have a different point of view on the issue.

• What kind of information would be useful? What form of information? (Try to get them to define what the product would look like.)

VI. Wrap up Discussion Led by Facilitator

• What are your challenges in using data, research and evaluation to make decisions?

• How far along is your organization in the process of using evidence-based information?

• What kinds of products or services might help you?

9

R

egional Educational Laboratory