REL Peer Review: Pilot Data Collection Methods for Examining the Use of Research Evidence

OMB Number:

Expiration Date:

REL Use of Research Evidence (URE) Survey

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is [XXXX]-[XXXX]. The time required to complete this information collection is estimated to average 20 minutes. If you have any comments concerning the accuracy of the time estimate or suggestions for improving this form, please write to: U.S. Department of Education, Washington, DC 20202. If you have comments or concerns regarding the content or the status of your individual submission of this form, write directly to: U.S. Department of Education, Institute of Education Sciences, 550 12th Street, SW, Washington, DC 20202.

SURVEY LANDING PAGE

REL Use of Research Evidence (URE) Survey

Welcome to the REL URE Survey. This study is sponsored by the United States Department of Education and is being conducted by Mathematica. The survey has been optimized for desktop computers, laptops, or tablets, and works best in current versions of Internet Explorer, Chrome, and Firefox.

To begin the survey, click the “Start” button below.

START

ALL

The U.S. Department of Education is sponsoring a study to help each of the ten Regional Educational Laboratories (RELs) learn more about the use of research evidence among REL partners and help the Institute of Education Sciences assess the validity and reliability of the survey items to be used in future feedback surveys. REL Peer Review is conducting this study for the Institute of Education Sciences within the U.S. Department of Education under contract 91990023C0008 with Mathematica.

This survey will take about 15-20 minutes to complete. Participation in the survey is voluntary and you may choose to skip any question you prefer not to answer. If you are unsure of how to answer a question, please give the best answer you can rather than leaving the question blank. After completing the survey, you will be able to select a $30 e-gift card or choose to donate your $30 incentive to your choice of one of several charities. There are no known risks associated with your participation.

In reporting the study’s results, your answers will be completely confidential; no information that identifies you or your organization will be reported. The researchers conducting this study follow the confidentiality and data protection requirements of the U.S. Department of Education’s Institute of Education Sciences (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). The reports prepared for the study will summarize findings across the sample and will not associate responses with a specific district, school, institution, or individual. All information you provide will be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

 By clicking this box, you are confirming that you understand that the information you provide will be kept private and used only for research purposes. You further understand that your answers will be combined with the responses of other respondents so that no individuals will be identified.

SOFT CHECK IF MISSING: Before moving forward, please review the information on screen and check the box to confirm that you understand that the information you provide will be kept private and only used for research purposes.

Instructions

Here are a few helpful tips for navigating the survey:

• To answer a question, click the box to choose your response or type your answer.

• To continue to the next webpage, click the "Next" button.

• To go back to the previous webpage, click the "Back" button. Please note that this command is only available in certain sections.

• For security purposes, you will be timed out if you are idle for longer than 30 minutes.

Definitions

Before you begin, please note that the terms “research” and “research evidence” are used in many questions throughout this survey. For the purposes of this study, we have adopted the following definitions of these terms and ask you to use them as you respond to survey questions:

• Research is the systematic data collection and analysis driven by research questions.

• Research evidence is the information produced by research. This could include evidence in reports, journal articles, and research summaries, as well as evidence embedded in curricula, practice tools, and assessments.

A. Instrumental Use of Research Evidence and Other Information

The following questions ask about how often you use different sources of evidence and information when engaged in certain actions or types of decision-making. You will be asked about the same sources of information for up to four different actions.

ALL

A1. To start, please select which actions or types of decision-making you ever engage in.

Select all that apply

o Make decisions about the content or direction of a policy or program 1

o Develop new activities, programs, or guidelines 2

o Implement activities and programs 3

IF A1 = 1

A2. When deciding about the content or direction of a policy or program, how often do you use the following sources of information:

	Never	Sometimes	Often	Always
a. Research evidence, including evidence provided by RELs	1 m	2 m	3 m	4 m
b. Research evidence limited to what RELs provide or produce	1 m	2 m	3 m	4 m
c. Data from your school, district, institution, and/or state	1 m	2 m	3 m	4 m
d. Your own experience and expertise	1 m	2 m	3 m	4 m
e. Colleague input, feedback, or consultation	1 m	2 m	3 m	4 m
f. Feedback and concerns from students, parents, and/or community members	1 m	2 m	3 m	4 m
g. Other professional organization or technical assistance resources	1 m	2 m	3 m	4 m

MISSING M

IF A1 = 2

A3. When developing new activities, programs, or guidelines in your role/agency, how often do you use the following sources of information:

	Never	Sometimes	Often	Always
a. Research evidence, including evidence provided by RELs	1 m	2 m	3 m	4 m
b. Research evidence limited to what RELs provide or produce	1 m	2 m	3 m	4 m
c. Data from your school, district, institution, and/or state	1 m	2 m	3 m	4 m
d. Your own experience and expertise	1 m	2 m	3 m	4 m
e. Colleague input, feedback, or consultation	1 m	2 m	3 m	4 m
f. Feedback and concerns from students, parents, and/or community members	1 m	2 m	3 m	4 m
g. Other professional organization or technical assistance resources	1 m	2 m	3 m	4 m

MISSING M

IF A1 = 3

A4. When implementing activities and programs, how often do you use the following sources of information:

	Never	Sometimes	Often	Always
a. Research evidence, including evidence provided by RELs	1 m	2 m	3 m	4 m
b. Research evidence limited to what RELs provide or produce	1 m	2 m	3 m	4 m
c. Data from your school, district, institution, and/or state	1 m	2 m	3 m	4 m
d. Your own experience and expertise	1 m	2 m	3 m	4 m
e. Colleague input, feedback, or consultation	1 m	2 m	3 m	4 m
f. Feedback and concerns from students, parents, and/or community members	1 m	2 m	3 m	4 m
g. Other professional organization or technical assistance resources	1 m	2 m	3 m	4 m

MISSING M

ALL

A5. When resolving problems in your daily professional practice, how often do you use the following sources of information:

	Never	Sometimes	Often	Always
a. Research evidence, including evidence provided by RELs	1 m	2 m	3 m	4 m
b. Research evidence limited to what RELs provide or produce	1 m	2 m	3 m	4 m
c. Data from your school, district, institution, and/or state	1 m	2 m	3 m	4 m
d. Your own experience and expertise	1 m	2 m	3 m	4 m
e. Colleague input, feedback, or consultation	1 m	2 m	3 m	4 m
f. Feedback and concerns from students, parents, and/or community members	1 m	2 m	3 m	4 m
g. Other professional organization or technical assistance resources	1 m	2 m	3 m	4 m

MISSING M

The next set of questions ask about how often you use research evidence and data.

ALL

A6. How often do you use research evidence provided or produced by RELs to inform decisions in your school, district, institution, or state?

m Never 1

m Rarely 2

m Occasionally 3

m Frequently 4

MISSING M

IF A6 = 1 OR 2

A6a. Do you plan to use research evidence provided or produced by RELs to inform decisions in your school, district, institution, or state?

m No 0

m Yes 1

MISSING M

ALL

A7. How often do you use data to inform decisions in your school, district, institution, or state?

m Never 1

m Rarely 2

m Occasionally 3

m Frequently 4

MISSING M

IF A7 = 1 OR 2

A7a. Do you plan to use data to inform decisions in your school, district, institution, or state?

m No 0

m Yes 1

MISSING M

B. Conceptual Use of Research Evidence

ALL

B1. Thinking about research evidence broadly (and not limited to what REL provides)…

How often do you use research evidence to:

	Never	Sometimes	Frequently	All the time
a. Change the way you look at problems facing your school, district, institution, or state	1 m	2 m	3 m	4 m
b. Expand your understanding of an issue	1 m	2 m	3 m	4 m
c. Suggest alternative solutions to problems facing your school, district, institution, or state	1 m	2 m	3 m	4 m
d. Bring attention to an issue that your school, district, institution, or state has not yet faced	1 m	2 m	3 m	4 m
e. Use a research-based framework when making improvements	1 m	2 m	3 m	4 m
f. Provide a common language and set of ideas for you and your colleagues	1 m	2 m	3 m	4 m

MISSING M

C. Quality of Research Use

ALL

C1. To what extent do you feel the following statements describe how you use research evidence in your professional practice? You may independently engage in these activities or collaborate with colleagues. Please include both types of engagement when answering these questions. Please think about research evidence broadly, and not limited to what REL provides.

	Not at all like me	Somewhat like me	Mostly like me	Exactly like me
a. I determine whether the research is of high quality	1 m	2 m	3 m	4 m
b. I determine whether the research is relevant to my school, district, institution, or state	1 m	2 m	3 m	4 m
c. I make connections between the research and my current practice	1 m	2 m	3 m	4 m
d. I work to develop a shared understanding of the research with others in my school, district, institution, or state	1 m	2 m	3 m	4 m
e. I take the time to reflect on my own practice with the help of research evidence	1 m	2 m	3 m	4 m
f. I use research in response to specific challenges or priorities	1 m	2 m	3 m	4 m
g. I use multiple sources of research to help me solve issues or problems	1 m	2 m	3 m	4 m
h. I take part in initiatives provided by my school or organization to improve my research use	1 m	2 m	3 m	4 m
i. I pilot or trial a new process or practice before I fully implement it, and evaluate early impacts	1 m	2 m	3 m	4 m

MISSING M

D. Value of REL Research Evidence and Services

When answering questions in this section, please think about your REL engagement over the past several months. This could include sustained involvement with the partnership, involvement in partnership meetings or trainings, attendance at dissemination events or webinars, or resources shared by the partnership during this period.

ALL

D1. Please rate the extent to which you agree or disagree with the following statements about REL research evidence and services.

REL research evidence and services…

	Not at all agree	Somewhat agree	Mostly agree	Strongly agree
a. Are relevant to the learners and communities I serve	1 m	2 m	3 m	4 m
b. Offer timely information	1 m	2 m	3 m	4 m
c. Are easy to transfer into my own professional practice	1 m	2 m	3 m	4 m
d. Are useful to guide my own professional practice	1 m	2 m	3 m	4 m
e. Are easy to understand	1 m	2 m	3 m	4 m
f. Are easy to find	1 m	2 m	3 m	4 m
g. Are reliable	1 m	2 m	3 m	4 m

MISSING M

ALL

D2. To what extent do you agree or disagree with the following statements.

	Not at all agree	Somewhat agree	Mostly agree	Strongly agree
a. I would like to participate in additional support activities offered by the REL	1 m	2 m	3 m	4 m
b. The format of REL engagement provides ample opportunity for participants to meaningfully interact with each other	1 m	2 m	3 m	4 m
c. REL engagement is relevant to my work	1 m	2 m	3 m	4 m
d. REL engagement is relevant to the issues faced by my school, district, institution, or state	1 m	2 m	3 m	4 m
e. The benefits from my REL engagement were worth the time and resources invested to participate	1 m	2 m	3 m	4 m
f. REL engagement improved my ability to support change in my school, district, institution, or state and make sure the change is sustained	1 m	2 m	3 m	4 m
g. REL engagement increased my desire to support and participate in change in my school, district, institution, or state	1 m	2 m	3 m	4 m
h. REL engagement is helping my school, district, institution, or state achieve its goals	1 m	2 m	3 m	4 m
i. I expect to apply information from my REL engagement in my work	1 m	2 m	3 m	4 m

MISSING M

E. Usefulness of Research

ALL

E1. Please rate your level of agreement with the following statements about research. Please think about research evidence broadly, and not limited to what REL provides.

	Not at all agree	Somewhat agree	Mostly agree	Strongly agree
a. Most research evidence suggests actionable steps to take in practice	1 m	2 m	3 m	4 m
b. Researchers have a solid grasp on evolving problems in schools/districts	1 m	2 m	3 m	4 m
c. Research evidence informs the most important issues schools/districts face	1 m	2 m	3 m	4 m
d. Research evidence is produced quickly enough for me to make use of it	1 m	2 m	3 m	4 m
e. Research takes into consideration the varying levels of resources available to schools/districts to implement research findings	1 m	2 m	3 m	4 m
f. I am not clear about how research evidence can be used to help change practice	1 m	2 m	3 m	4 m
g. I don’t believe research evidence will help to improve student outcomes	1 m	2 m	3 m	4 m
h. It is difficult to find research evidence that addresses my specific practice, context, or needs	1 m	2 m	3 m	4 m

MISSING M

F. Confidence in Ability to Use Research

ALL

F1. I feel confident that I have the knowledge and skills to:

	Not at all confident	Somewhat confident	Mostly confident	Very confident
a. Find research evidence to inform practice, policy, or program development	1 m	2 m	3 m	4 m
b. Evaluate the quality of research	1 m	2 m	3 m	4 m
c. Interpret the results of research	1 m	2 m	3 m	4 m
d. Apply research evidence to practice, policy, or program development	1 m	2 m	3 m	4 m
e. Design evaluations of practices, policies, or programs	1 m	2 m	3 m	4 m
f. Partner with others to generate research evidence	1 m	2 m	3 m	4 m

MISSING M

When answering the next two questions, please think about your REL engagement over the past several months. This could include sustained involvement with the partnership, involvement in partnership meetings or trainings, attendance at dissemination events or webinars, or resources shared by the partnership during this period.

ALL

F2. After working with the REL, I have greater capacity to use research evidence to inform decisions about policies or practices in my school, district, institution, or state.

m Not at all agree 1

m Somewhat agree 2

m

m Mostly agree 3

m Strongly agree 4

MISSING M

ALL

F3. After working with the REL, I have greater capacity to use data to inform decisions about policies or practices in my school, district, institution, or state.

m Not at all agree 1

m Somewhat agree 2

m Mostly agree 3

m Strongly agree 4

MISSING M

G. Organizational Support for Use of Research Evidence

ALL

G1. In my organization:

	Not at all agree	Somewhat agree	Mostly agree	Strongly agree
a. Leaders generally convey that it is important to use research evidence in policy or program development	1 m	2 m	3 m	4 m
b. It is expected that research evidence will be used in policy or program development	1 m	2 m	3 m	4 m
c. Generating new research evidence to inform policy or program development is encouraged	1 m	2 m	3 m	4 m
d. It is expected that policies/programs will be evaluated	1 m	2 m	3 m	4 m
e. Interaction or collaboration with researchers or research organizations is encouraged	1 m	2 m	3 m	4 m
f. We refer to evidence of what works when deciding which programs or initiatives to implement	1 m	2 m	3 m	4 m
g. We are encouraged to engage in informed risk-taking in educational practices	1 m	2 m	3 m	4 m

MISSING M

H. Organizational Infrastructure for Use of Research Evidence

ALL

H1. To what extent do you agree with the following statements about the current environment within your organization?

	Not at all agree	Somewhat agree	Mostly agree	Strongly agree
a. We have adequate time to access and review research evidence	1 m	2 m	3 m	4 m
b. We have sufficient access to research evidence	1 m	2 m	3 m	4 m
c. There are formal processes to help staff engage critically with different information sources	1 m	2 m	3 m	4 m
d. There are processes in place to support collaborative learning	1 m	2 m	3 m	4 m

MISSING M

I. Respondent Information

ALL

I1. Which of the following best describes your primary occupation?

Select all that apply

School Level

o Teacher/Educator 1

o Principal/Vice-Principal 2

o Instructional coach 3

o Other school-level administrator 4

o Librarian 5

o Other school-level role (please specify) 6

Local Education Agency

o School board member 7

o School district central office staff – research office 8

o School district central office staff – program office 9

o School superintendent/Assistant superintendent 10

State Education Agency

o State-level education administrator 11

o State-level advisor or board member 12

o Other state-level staff (please specify) 13

State Government

o State legislator or legislative staff member 14

Tribal Government

o Council member 15

o Tribal member 16

o Education specialist 17

o Other tribal government staff (please specify) 18

I1. CONTINUED Which of the following best describes your primary occupation?

Select all that apply

Other

o Adult Education staff 19

o Community Organization staff 20

o Higher Education/Postsecondary staff 21

o Other (please specify) 22

Options for Pacific Region only

o National-level education administrator 23

o National-level education central office staff 24

o State-level education administrator 25

o State-level education central office staff 26

MISSING M

ALL

I2. How long have you served in any education or research-related role?

LENGTH OF TIME (RANGE 1-75)	MONTHS, YEAR(S)
	▼

MISSING M

ALL

I3. In which state/territory/jurisdiction is your work based?

(STRING 150)

MISSING M

J. Additional Thoughts About REL Support

Please think about your recent engagement with the REL when answering these final questions about REL partnership support.

ALL

J1. What aspects of the REL partnership support were most helpful and why?

(STRING 250)

MISSING M

ALL

J2. What aspects of the REL partnership support were least helpful and why?

(STRING 250)

MISSING M

ALL

J3. How would you suggest improving the partnership support provided by the REL?

(STRING 250)

MISSING M

Thank you for completing the survey!